diff --git a/README.md b/README.md index 71ce334..4749299 100644 --- a/README.md +++ b/README.md @@ -233,7 +233,7 @@ libjxl-dev`, plus the Vimba X SDK under `/opt/VimbaX` for `-DWITH_VIMBA=ON`. |----------|----------| | [docs/architecture.md](docs/architecture.md) | Components, interfaces, threading, data flow, capture state machine | | [docs/build-and-setup.md](docs/build-and-setup.md) | CMake, dependencies, options, Vimba X / Paho, host setup | -| [docs/deployment.md](docs/deployment.md) | Step-by-step deploy to the gimbal LattePanda (systemd) | +| [docs/deployment.md](docs/deployment.md) | Running and updating a tower: `deploy.sh`, `scripts/fgc`, the boot service, the log | | [docs/configuration.md](docs/configuration.md) | `config.ini` keys, CLI flags, console commands | | [docs/mqtt-api.md](docs/mqtt-api.md) | MQTT topic catalog and payloads | | [docs/test-command.md](docs/test-command.md) | The `test` hardware self-test suite: subsystems, profiles, reports, baselines | diff --git a/docs/deployment.md b/docs/deployment.md index e5e473d..6fa062f 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -1,204 +1,163 @@ -# Deploying to the Gimbal LattePanda +# Running and updating a tower -Step-by-step guide to deploy `fire_gimbal_control` on a tower's LattePanda (Linux x86-64). Like the -firmware workflow, this is a repeatable **build → install → configure → enable → verify** cycle; the -[Update / redeploy](#9-update--redeploy) section at the end is the short loop you run for every new version. +How a tower that already works is operated and updated: pushing a new version +from the dev box, running the program detached, starting it at boot, reading its +log. A **first** install - a freshly imaged board, or a tower not yet migrated - +is [tower-installation.md](tower-installation.md), which `scripts/install-tower` +automates. -> **Which LattePanda.** The deployed fleet is the **3 Delta** — read from tower 111's DMI -> `product_name`, *not* the Sigma, which this page claimed until 2026-09-18. Tower 121 is a -> **LattePanda IOTA**; see [lattepanda-iota.md](lattepanda-iota.md) and its setup kit under -> `hardware/lattepanda-iota/`. Nothing in this guide is board-specific — it is x86-64 Linux either way — -> so the distinction matters for the device paths in `hardware.ini`, not for these steps. -> -> Assumes a Debian/Ubuntu-based Linux on the LattePanda. For Arch/Manjaro swap `apt` for `pacman` -> (see [build-and-setup.md](build-and-setup.md)). Run as a normal user with `sudo` rights; the service runs -> under a dedicated `ggs` user. +The fleet: tower **111** is a LattePanda 3 Delta (ATmega32U4 motor MCU, two +axes), tower **121** a LattePanda IOTA (RP2040, yaw only). Nothing here differs +between them; what differs lives in each tower's `hardware.ini`. -## 0. Prerequisites (one-time per device) +## Where things are on a tower -Install the toolchain and system libraries: +| Path | What | Written by | +|---|---|---| +| `~ggs/projects/fwt_2a/software/` | this repository, and the program's build in `build/` | `deploy.sh` (sync + build), or the installer | +| `~ggs/projects/fwt_2a/firmware/` | the firmware repository, built on the tower | `firmware/deploy.sh` | +| `.../software/config.ini` | what the tower DOES: scan, capture, logging, calibration | the installer (seeded once), `gimbal ... setup` (calibration), a person | +| `/etc/fire_gimbal_control/hardware.ini` | what the tower IS: board, axes, sensors, pins, drivetrain | the installer; a person when the hardware changes | +| `/etc/fire_gimbal_control/credentials.env` | `FGC_MQTT_USER=` / `FGC_MQTT_PW=`, mode 0640 root:ggs | the installer | +| `/etc/fire_gimbal_control/install.log` | one line per installer run | the installer | +| `~ggs/.local/share/fire_gimbal_control/logs/` | the program's log, test reports | the program | +| `.../software/config/camera_profiles/rgb/*.xml` | the camera settings, exported ON the tower from its camera | the installer, or `scripts/export_camera_profile.py` | -```bash -sudo apt update -sudo apt install -y git cmake build-essential \ - libopencv-dev libjxl-dev libboost-all-dev -``` +`deploy.sh` never touches `config.ini`, `hardware.ini`, the camera profiles or +the images: they exist only on the tower, and `rsync --delete` once erased a +freshly exported profile before they were excluded. -Install the **Allied Vision Vimba X SDK** (proprietary, needed for the cameras): +## 1. Updating the program from the dev box -1. Download Vimba X for Linux x86-64 from - . -2. Unpack to a fixed location, e.g. `/opt/VimbaX_2024-1`. -3. Run the SDK's transport-layer installer so cameras are discoverable, e.g.: - ```bash - sudo /opt/VimbaX_2024-1/cti/Install_GenTL_Path.sh - ``` -4. Make its runtime libraries findable at run time: - ```bash - echo /opt/VimbaX_2024-1/api/lib | sudo tee /etc/ld.so.conf.d/vimbax.conf - sudo ldconfig - ``` - -Ensure an **MQTT broker** is reachable from the tower (the ground-station/ZKMS broker, or a local Mosquitto -for bring-up). - -## 1. Get the code - -```bash -sudo mkdir -p /opt/src && sudo chown "$USER" /opt/src -cd /opt/src -git clone fire_gimbal_control_src -cd fire_gimbal_control_src -# (or: git pull, for an update) -``` - -## 2. Build (full, with cameras + MQTT) - -```bash -cmake -B build -DCMAKE_BUILD_TYPE=Release \ - -DWITH_VIMBA=ON -DWITH_MQTT=ON \ - -DVMB_HOME=/opt/VimbaX_2024-1 -cmake --build build -j"$(nproc)" -``` - -- `WITH_MQTT=ON` fetches + builds Eclipse Paho from source on first configure (needs network). -- If a camera build isn't needed yet, use `-DWITH_VIMBA=OFF` for a mock binary to test the pipeline. -- Smoke-test the core before installing: `ctest --test-dir build` (after adding `-DBUILD_TESTING=ON`). - -## 3. Install to a fixed location - -```bash -sudo useradd -r -s /usr/sbin/nologin ggs 2>/dev/null || true -sudo mkdir -p /opt/fire_gimbal_control -sudo cp build/fire_gimbal_control /opt/fire_gimbal_control/ -# demo-mode placeholder image (only needed if you ever run --demo) -sudo cp bin/x64/Release/test_smoke.jxl /opt/fire_gimbal_control/ 2>/dev/null || true -``` - -## 4. Configure - -```bash -sudo cp config/config.example.ini /opt/fire_gimbal_control/config.ini -sudo nano /opt/fire_gimbal_control/config.ini -``` - -Set at least: +On the dev box, in `software/`, with a `.deploy.env` naming the tower: ```ini -[General] -tower_name = ExampleTower ; this tower's name (drives all MQTT topics) -image_interval = 3 - -[Network] -zkms_server_ip = 10.11.12.13 ; the broker address - -[Serial] -device = /dev/ttyACM0 ; the motor-controller MCU -baud = 115200 - -[Camera.rgb] -id = DEV_1AB22C0AADED ; or a GigE IP like 192.168.11.101 -label = RGB -profile_dir = config/camera_profiles/rgb -profile = standard +REMOTE_HOST=ggs@10.11.12.121 +REMOTE_DIR=/home/ggs/projects/fwt_2a/software +CMAKE_ARGS="-DWITH_MQTT=ON -DWITH_VIMBA=ON" +RUN_ARGS="--start" ``` -**Credentials** go in the environment, not the config file. Create a root-only env file for the service: - ```bash -sudo install -m 600 /dev/null /etc/fire_gimbal_control/credentials.env 2>/dev/null || \ - { sudo mkdir -p /etc/fire_gimbal_control && sudo install -m 600 /dev/null /etc/fire_gimbal_control/credentials.env; } -printf 'FGC_MQTT_USER=fwt_gimbal\nFGC_MQTT_PW=CHANGE_ME\n' | sudo tee /etc/fire_gimbal_control/credentials.env >/dev/null -sudo chmod 600 /etc/fire_gimbal_control/credentials.env +./deploy.sh # sync the tree, build on the tower +./deploy.sh --check-deps # only check the tower's build dependencies +./deploy.sh --clean # wipe the tower's build dir first (fresh configure) ``` -Give the `ggs` user access to its files and the image output dir: +It syncs the tree (leaving `.git` behind, and writing the revision into +`.revision` beside it instead), then configures and builds **on the tower**. The +program already running keeps running the old binary until it is restarted: ```bash -sudo chown -R ggs:ggs /opt/fire_gimbal_control +# on a tower started by its boot service +ssh ggs@ sudo systemctl restart fire-gimbal-control +# on a tower started by hand +./deploy.sh --stop && ./deploy.sh --run # --run --home adds --init ``` -## 5. Hardware access +`--run` starts the program detached through `scripts/fgc` (section 2) and first +refuses a `config.ini` that still holds the example's placeholders +(`ExampleTower`, `CHANGE_ME`, `DEV_0000000000`); `--force` overrides that. On a +tower whose service is running, `--run` fails on the single-instance lock +(section 3) - restart it through systemd instead. -**Serial (motor controller):** let the service user open the port. +After an update, check it: ```bash -sudo usermod -aG dialout ggs +ssh ggs@ 'cd ~/projects/fwt_2a/software && scripts/install-tower --verify' ``` -For a stable device name across reboots, add a udev rule keyed on the MCU's USB IDs (find them with -`udevadm info -a -n /dev/ttyACM0 | grep -m1 idVendor`) and point `[Serial] device` at the symlink: +It changes nothing; it runs `test hardware` inside the running program (pausing +and then resuming capture) and checks the files, the firmware's pins and the +boot service. + +`--push-config` also overwrites the tower's `config.ini` with the dev box's, +backing the tower's copy up first. It exists for the development tower; a +tower's `config.ini` holds its own calibration, so do not push one tower's to +another. `hardware.ini` is never pushed. + +> **One-time: `git pull` can delete your `.deploy.env`.** It was tracked by +> mistake until `b0127db`, which stopped tracking it and added it to +> `.gitignore`. Pulling *across* that commit deletes the working copy, credentials +> and all, with no warning. Only checkouts that predate `b0127db` are exposed, +> and only once: copy it aside first (`cp .deploy.env ../deploy.env.bak`), pull, +> copy it back. To recover one already lost: `git show b0127db^:.deploy.env`. + +### Updating the firmware + +**Host first, firmware second**, always: a new firmware asks the host for its +drivetrain at boot, and an older host cannot answer. In `firmware/`, with its own +`.deploy.env` (`REMOTE_HOST`, `REMOTE_DIR`, `PIO_ENV` = `leonardo` or +`iota_rp2040`, `SERIAL_PORT` = the MCU's `/dev/serial/by-id/` path): ```bash -echo 'SUBSYSTEM=="tty", ATTRS{idVendor}=="XXXX", ATTRS{idProduct}=="YYYY", SYMLINK+="ttyGimbal"' | \ - sudo tee /etc/udev/rules.d/99-gimbal.rules -sudo udevadm control --reload && sudo udevadm trigger +./deploy.sh --compile-only # sync + build on the tower, flash nothing +./deploy.sh # build + flash +./deploy.sh --monitor # ... then watch it boot ``` -**GigE cameras (if used):** put the host NIC on the camera subnet (e.g. `192.168.11.0/24`), enable jumbo -frames, and confirm discovery with the Vimba X `VimbaXViewer`/`ListCameras` tool. +The build takes the pins from the tower's `hardware.ini` (`[Pins]`) and fails +without them. The flashing is `firmware/tools/flash.sh`, run on the tower: an +RP2040 is put in its bootloader by the firmware's own `BOOTSEL` command first; a +Leonardo uses its 1200-baud touch. **Stop the program first** - it holds the +serial port - and restart it after; the new firmware then asks, the host +answers, and `install-tower --verify` confirms the firmware holds what +`hardware.ini` declares. -## 6. First run by hand (verify before enabling the service) +## 2. Running detached (`scripts/fgc`) + +The tower is operated over ssh, so the program has to survive the connection +closing. `scripts/fgc` runs it inside a tmux session, which owns the pty: ```bash -cd /opt/fire_gimbal_control -set -a; . /etc/fire_gimbal_control/credentials.env; set +a -./fire_gimbal_control --config ./config.ini --init --start --log-level debug -``` - -Watch for: `Loaded config`, `Serial controller started`, `Opened camera ...`, `MQTT connected`, then -`Saved .../RGB/.jxl` after the first capture. Type `exit` (or Ctrl-D) to stop. To validate without -cameras first: add `--mock-camera --mock-serial --no-mqtt`. - -This run is attached to your terminal and dies with the ssh session — fine for a first check, not for -leaving it running. For that, use section 6b. - -### 6b. Running detached (`scripts/fgc`) - -The tower is operated over ssh, so the program has to survive the connection closing. `scripts/fgc` runs -it inside a tmux session, which owns the pty: - -```bash -cd /opt/fire_gimbal_control +cd ~/projects/fwt_2a/software scripts/fgc start --start # starts detached, returns as soon as it is up scripts/fgc status # running? since when? which PID? -scripts/fgc attach # the dashboard — Ctrl-b d detaches, leaves it running +scripts/fgc attach # the dashboard - Ctrl-b d detaches, leaves it running scripts/fgc stop # SIGTERM -> the same clean teardown as `exit` scripts/fgc peek # snapshot of the pane without attaching ``` -**`Ctrl-b d` detaches. The dashboard's own `q` key opens an exit prompt** offering `q` (quit the -program) or `b` (background — detach and leave it running, the same as `Ctrl-b d`), with `Esc` to -cancel. `b` is offered only inside a `scripts/fgc` session; outside one there is nothing to detach -into and it is shown dimmed. +The boot service (section 4) runs the program the same way, so all of this +works on a tower started at boot too. -`scripts/fgc stop` sends SIGTERM and waits up to 30s; the program handles it with the same shutdown path -as the `exit` command (stop capture, close the camera, flush the image queue). Do **not** `kill -9` — -an unclean exit leaves the camera wedged for the next run. If it will not stop, attach and look at why. +**`Ctrl-b d` detaches. The dashboard's own `q` key opens an exit prompt** +offering `q` (quit the program) or `b` (background - detach and leave it +running, the same as `Ctrl-b d`), with `Esc` to cancel. `b` is offered only +inside a `scripts/fgc` session; outside one there is nothing to detach into and +it is shown dimmed. -`scripts/fgc start` refuses if a run is already up, and prints the previous run's last output if it died -rather than silently discarding it. +`scripts/fgc stop` sends SIGTERM and waits up to 120 s; the program handles it +with the same shutdown path as the `exit` command (stop capture, close the +camera, flush the image queue). Do **not** `kill -9` - an unclean exit leaves the +camera wedged for the next run. If it will not stop, attach and look at why. -Choosing **quit** at that prompt while attached ends the tmux session too, so you land straight back in -your shell — `exit` then closes ssh. A run that *fails* is the exception: its pane is kept so you can read -why (`remain-on-exit failed`), and `fgc status` shows it until `fgc stop` or the next `fgc start` clears it. +`scripts/fgc start` refuses if a run is already up, and prints the previous +run's last output if it died rather than silently discarding it. -A kept pane has **no process behind it** — keys and Ctrl-C do nothing, which reads as a hung ssh session. -`Ctrl-b d` still detaches (tmux, not the pane, handles it); the session's status line spells that out along -the bottom of the screen. As a last resort `Enter ~ .` closes the ssh client itself from your end. +Choosing **quit** at that prompt while attached ends the tmux session too, so you +land straight back in your shell. A run that *fails* is the exception: its pane +is kept so you can read why (`remain-on-exit failed`), and `fgc status` shows it +until `fgc stop` or the next `fgc start` clears it. A kept pane has **no process +behind it** - keys and Ctrl-C do nothing, which reads as a hung ssh session. +`Ctrl-b d` still detaches; as a last resort `Enter ~ .` closes the ssh client. -> **Why `_run` waits on the app instead of `exec`ing it.** When the app is itself the pane's process, tmux -> never records an exit status for it — `#{pane_dead_status}` comes back empty and the process lingers as a -> zombie — so `remain-on-exit failed` cannot tell success from failure and keeps *every* finished pane, -> including a clean `q`. That is what used to strand an operator on a frozen dead pane. Interposing a shell -> that reaps the child and exits with its status gives tmux a real status. Measured on the device: 3/3 panes -> kept when exec'd, 4/4 sessions closed cleanly when parented. The wrapper forwards SIGTERM/SIGINT/SIGHUP, -> so `fgc stop` still reaches the app and runs the full teardown. +MQTT credentials are read from `/etc/fire_gimbal_control/credentials.env` by +`fgc` itself, inside the pane, as are the Vimba X paths: a tmux server that +already exists hands a new session its own environment, not the caller's. -### 6c. The single-instance lock +> **Why `_run` waits on the app instead of `exec`ing it.** When the app is itself +> the pane's process, tmux never records an exit status for it, so +> `remain-on-exit failed` cannot tell success from failure and keeps *every* +> finished pane. Interposing a shell that reaps the child and exits with its +> status gives tmux a real one. Measured on the device: 3/3 panes kept when +> exec'd, 4/4 sessions closed cleanly when parented. The wrapper forwards +> SIGTERM/SIGINT/SIGHUP, so `fgc stop` still reaches the app. -Two instances would fight over the motor's serial port and the camera. The binary -prevents that itself: at startup it takes an exclusive `flock` on +## 3. The single-instance lock + +Two instances would fight over the motor's serial port and the camera. The +binary prevents that itself: at startup it takes an exclusive `flock` on - `/run/user//fire_gimbal_control.lock` when that directory exists, else - `/tmp/fire_gimbal_control-.lock` @@ -209,24 +168,31 @@ and, if it is already held, exits with status **2** and: Cannot start: another instance is already running (PID 1234); attach to it with: scripts/fgc attach ``` -The path is derived from the uid and not from `$XDG_RUNTIME_DIR` on purpose: that variable is set in an -interactive ssh shell but not in a non-interactive `ssh host cmd` one, and a lock the two launch routes -disagree about would exclude nothing. +The path is derived from the uid and not from `$XDG_RUNTIME_DIR` on purpose: that +variable is set in an interactive ssh shell but not in a non-interactive +`ssh host cmd` one, and a lock the two launch routes disagree about would exclude +nothing. For the same reason a tower with a boot service needs **lingering** +(section 4). -Because the check lives in the binary, it covers every route in — `scripts/fgc`, systemd, or a bare -`./fire_gimbal_control`. Nothing needs cleaning up after a crash or a power cut: the kernel releases an -`flock` when the holder dies, so a leftover lockfile with a dead PID in it never blocks a launch. Use -`--no-lock` only to run a second mock-only instance during development, and `--lock-file ` to -point at a different lockfile. +Because the check lives in the binary, it covers every route in - `scripts/fgc`, +the service, or a bare `./build/fire_gimbal_control`. Nothing needs cleaning up +after a crash or a power cut: the kernel releases an `flock` when the holder +dies. Use `--no-lock` only to run a second mock-only instance during +development, and `--lock-file ` to point at a different lockfile. -## 7. Install the systemd service +## 4. Starting at boot -`scripts/install-tower` does this (it asks first, and shows the unit it will +`scripts/install-tower` installs this (it asks first, and shows the unit it will write). The unit is the template `scripts/fire-gimbal-control.service`, and it runs the program **through `scripts/fgc`, inside tmux** - so a tower started at -boot still has its dashboard (`scripts/fgc attach`), and `fgc status` / `peek` / -`stop` work from any ssh session. It is not an alternative to the tmux workflow -any more; it is how that workflow starts at boot. +boot still has its dashboard, and `fgc status` / `peek` / `stop` work from any +ssh session. + +```bash +sudo systemctl start|stop|restart fire-gimbal-control +systemctl is-enabled fire-gimbal-control # starts at boot? +sudo systemctl disable fire-gimbal-control # not any more +``` By hand, fill in `@USER@`, `@REPO@` and `@ARGS@` (normally `--init --start`) and: @@ -244,73 +210,27 @@ running", and a second copy could start beside the first. **`--init` homes the gimbal unattended at every boot.** That is what a tower in service needs - the firmware keeps no position across a power cycle, so without -it the gimbal sits un-homed and captures nothing - but it is a deliberate choice, -and the installer asks for the arguments. +it the gimbal sits un-homed and captures nothing - but it is a deliberate +choice, and the installer asks for the arguments. -MQTT credentials go in `/etc/fire_gimbal_control/credentials.env` -(`FGC_MQTT_USER=` / `FGC_MQTT_PW=`, mode 0640 root:ggs). `fgc` reads it inside -the tmux pane rather than taking it from the unit's environment: a tmux server -that already exists hands a new session its own environment, not the caller's. - -## 8. Enable and verify - -```bash -sudo systemctl start fire-gimbal-control -scripts/fgc status # running, lockfile under /run/user/ -scripts/install-tower --verify # every check, including test hardware -``` +| Tower | Service | Starts with | +|---|---|---| +| 111 | enabled 2026-09-24 | `--init --start`: homes and captures at boot. Not yet exercised by a reboot | +| 121 | enabled 2026-09-23 | no arguments: starts idle, neither homes nor captures (the development tower) | `[verified]` on tower 121, 2026-09-23: the service came up by itself after a reboot with nobody logged in, with its lock under `/run/user/1000`; `fgc status` and the dashboard worked from a later ssh session; credentials reached the program intact (a password with `$`, a quote and a space); `systemctl stop` shut it down cleanly; and `install-tower --verify` passed after the reboot, USB -preconditions included. 121 runs it with no arguments, so it does not home at -boot - see the note on `--init` above. +preconditions included. -Confirm end to end: -- Images accumulating under the configured `[Paths] output_dir` (default - `~ggs/.local/share/fire_gimbal_control/images/RGB/`). -- CamEvents on the broker: - `mosquitto_sub -h -t 'GGS/FWT//#' -v`. +A tower started this way is in service when: +- images accumulate under `[Paths] output_dir` (default + `~ggs/.local/share/fire_gimbal_control/images/RGB/`); +- CamEvents reach the broker: `mosquitto_sub -h -t 'GGS/FWT//#' -v`. -## 9. Update / redeploy - -The short loop for each new version (mirrors the firmware flash cycle): - -```bash -cd /opt/src/fire_gimbal_control_src -git pull -cmake --build build -j"$(nproc)" # reuses the configured build dir -sudo systemctl stop fire-gimbal-control -sudo cp build/fire_gimbal_control /opt/fire_gimbal_control/ -sudo systemctl start fire-gimbal-control -journalctl -u fire-gimbal-control -n 50 -f # confirm healthy -``` - -If `CMakeLists.txt` or options changed, re-run the `cmake -B build ...` configure step from -[section 2](#2-build-full-with-cameras--mqtt) first. To roll back, keep the previous binary -(`fire_gimbal_control.prev`) and swap it back. - -> **One-time: `git pull` can delete your `.deploy.env`.** It was tracked by -> mistake until `b0127db`, which stopped tracking it and added it to -> `.gitignore`. Pulling *across* that commit therefore does what git does to any -> file a commit removes — deletes the working copy, credentials and all. It is a -> checkout, not a merge conflict, so there is no warning and nothing to resolve. -> -> Only checkouts that predate `b0127db` are exposed, and only once. Copy it aside -> first: -> -> ```bash -> cp .deploy.env ../deploy.env.bak 2>/dev/null # then pull, then copy back -> ``` -> -> After that pull the file is ignored and stays put. `deploy.sh` already excludes -> it from the rsync, so deploying never overwrites the tower's own copy — this is -> purely about the machine you deploy *from*. To recover one already lost: -> `git show b0127db^:.deploy.env`. - -## 10. Where the log is +## 5. Where the log is The program writes a persistent log alongside the self-test and diagnostics reports: @@ -325,7 +245,7 @@ roughly 5 MB/day in steady state. This exists because a tower runs detached and, in TUI mode, every line otherwise goes to a 500-line on-screen ring buffer that dies with the tmux session. The -file is **more verbose than the dashboard** — `info` on screen, `debug` on disk — +file is **more verbose than the dashboard** - `info` on screen, `debug` on disk - so it holds detail the pane never showed. ```bash @@ -337,9 +257,8 @@ Use `tail -F`, not `-f`: rotation replaces the file and `-f` would follow the ol inode into a file nothing writes to any more. Each file opens with a header naming the build, tower, PID, config and hardware -paths and the full command line — repeated after every rotation, so an archive -found on its own still says where it came from. Lines carry a `YYYY-MM-DD` prefix -the console ones do not. +paths and the full command line - repeated after every rotation, so an archive +found on its own still says where it came from. What to grep when a tower will not come up: @@ -352,17 +271,22 @@ head -8 "$L" # which build, which > Wire traces (`[Logging] trace` / `--trace`) reach this file too. The serial > trace alone takes it from ~5 MB/day to ~41 MB/day, enough to age out the -> startup record within days — turn it on to chase something, off afterwards. +> startup record within days - turn it on to chase something, off afterwards. ## Troubleshooting +Start with `scripts/install-tower --verify` on the tower: it checks most of the +list below and says which. + | Symptom | Check | |---------|-------| -| `No config.ini found` | `FGC_CONFIG` / `--config` path; service `Environment=` line | -| `Failed to open serial port` | MCU plugged in, `dialout` group, `[Serial] device` path / udev symlink | -| `No camera found` / Vimba startup error | SDK installed, GenTL path set, `ldconfig`, NIC subnet, camera IDs in config | -| `MQTT connect failed` (degraded mode) | broker reachable, credentials env, firewall | -| Runs but no images | capture started (`--start` or `start` command), `output_dir` writable by `ggs` | -| `another instance is already running` (exit 2) | `scripts/fgc status`; a systemd unit and a tmux session both running (section 6c) | -| Program dies when ssh disconnects | started with `run.sh`/`--foreground` instead of `scripts/fgc start` (section 6b) | -| `No camera found` only when started via `scripts/fgc` | GenTL path not resolved in the pane — check `scripts/fgc peek`, and `/etc/profile.d/*Vimba*GenTL*.sh` or `VIMBA_CTI_PATH` | +| `No config.ini found` | run from `~/projects/fwt_2a/software`, or `--config` / `$FGC_CONFIG` | +| `Config error` naming a key | `build/fgc-config check --config config.ini --hardware /etc/fire_gimbal_control/hardware.ini` | +| `Failed to open serial port` | MCU plugged in, `ggs` in `dialout`, `[Serial] device` in `hardware.ini` - a `/dev/serial/by-id/` path, which **changes when the MCU is flashed** | +| An axis will not move, `ERR need config` | `hardware.ini` is missing a drivetrain value for it; the log names the key | +| `No camera found` / Vimba startup error | SDK installed, GenTL path set (`/etc/profile.d/*Vimba*GenTL*.sh` or `VIMBA_CTI_PATH`), `[Camera.rgb] id` in `hardware.ini` | +| `MQTT connect failed` (degraded mode) | broker reachable, `credentials.env` readable by `ggs`, firewall | +| Runs but no images | capture started (`--start`, or the `start` command) and the gimbal homed; `output_dir` writable | +| `another instance is already running` (exit 2) | `scripts/fgc status`; the service and a hand-started run both up (section 3) | +| `fgc status` says not running, but the program is | the service started it before anyone logged in, without lingering (section 4) | +| Program dies when ssh disconnects | started with `run.sh` / `deploy.sh --foreground` instead of `scripts/fgc start` | diff --git a/docs/tower-installation.md b/docs/tower-installation.md index 61534eb..6ca0bf3 100644 --- a/docs/tower-installation.md +++ b/docs/tower-installation.md @@ -1,56 +1,88 @@ -# Installing a tower from scratch +# Installing a tower -How a new fire-watch tower host goes from a freshly imaged machine to a running, -calibrated `fire_gimbal_control`. Written while doing exactly that on the IOTA -tower at `10.11.12.121` (2026-09-15) and kept honest about which steps have -actually been exercised and which are still on paper. +How a fire-watch tower host goes from a freshly imaged machine - or from an +older, unmigrated setup - to a running, calibrated `fire_gimbal_control`. +[deployment.md](deployment.md) is what comes after: updating and operating a +tower that already works. -> [deployment.md](deployment.md) is the *update* loop for a tower that already -> works. This is the *first* install, including the parts that only happen once: -> the board's preconditions, the Vimba X SDK, `hardware.ini`, and the motor -> firmware. +**`scripts/install-tower` does the whole of this.** The short way is the next +section. Everything after it, sections 1 to 11, is the reference for what each +step does and why, written while doing each one by hand, and still the way to +do one step on its own. -> **`scripts/install-tower` now does the steps below**, on the tower, in this -> order: it surveys the machine, asks for what it cannot work out (every -> question can also be answered with `--answer KEY=VALUE`), shows the plan, and -> applies it after one yes. `--dry-run` shows the plan and changes nothing; -> `--verify` runs the checks only. It never invents a drivetrain value: answer -> `unknown` and it writes a `; TODO`, and refuses to flash firmware an axis of -> which would then sit in `NEEDCFG`. Homing is offered only at the end, and only -> to someone who says they are at the rig. -> -> `[verified]` as a re-run on the already-configured tower 121 (2026-09-23): it -> changed only what was out of date and passed `test hardware`. `[untested]` on a -> freshly imaged board - the fresh path has been rehearsed with `--dry-run` only. -> What follows stays the reference for what each step does and why. - -**Status legend** — each step says where it stands: +**Status legend** - each step says where it stands: `[verified]` run end to end on a real tower and the result checked · `[partial]` run, with something still open · `[untested]` written from the design, not yet exercised. --- -## What you need before starting +## The short way: `scripts/install-tower` + +### Before starting | | | |---|---| -| The tower host | Powered, on the network, reachable by IP, with an OS installed and an `ssh` server running | -| An account on it | `ggs`, in the `sudo` and `dialout` groups | -| A dev box | This repository checked out, plus a copy of the Vimba X SDK (login-walled, so it cannot be fetched by a script) | -| The hardware | Camera plugged in, motor MCU plugged in, and a note of **what was assembled**: which axes, which IMU, which ambient sensor | +| The tower host | Powered, on the network, an OS installed, `ssh` running, reachable by key (section 1) | +| An account | `ggs`, in the `sudo` and `dialout` groups | +| Both repositories on it | `~/projects/fwt_2a/software` and `~/projects/fwt_2a/firmware` - cloned (`git.ggs-solutions.eu:GGS/fwt_software.git` and `fwt_firmware.git`, which needs a key on the tower), or synced by each repo's `deploy.sh` | +| The Vimba X SDK, staged on it | Login-walled, so it cannot be fetched by a script: copy it on (USB stick, `scp`) and give its directory when asked. `none` for a tower with no camera yet | +| The facts of the build | Which axes, which IMU, which ambient sensor, how the driver boards are wired, and - per axis - the drivetrain: sense resistor, currents, travel, `steps_per_rev`, `sw_mode`, `enc_deviation`, encoder or not. Anything not yet measured can be answered `unknown` | -Two facts decide most of what follows, so establish them first: - -```bash -ssh ggs@ 'cat /sys/class/dmi/id/product_name; . /etc/os-release && echo "$PRETTY_NAME"' -``` +The installer takes the board from DMI and picks its kit: | Board | MCU | Setup kit | PlatformIO env | |---|---|---|---| -| LattePanda 3 Delta / Sigma | ATmega32U4 (Leonardo), 5 V | *not written - the Delta fleet predates the kits* | `leonardo` | +| LattePanda 3 Delta | ATmega32U4 (Leonardo), 5 V | [`hardware/lattepanda-3-delta/`](../hardware/lattepanda-3-delta/) | `leonardo` | | LattePanda IOTA | RP2040, 3.3 V | [`hardware/lattepanda-iota/`](../hardware/lattepanda-iota/) | `iota_rp2040` | +### Running it + +On the tower, as `ggs`: + +```bash +cd ~/projects/fwt_2a/software +scripts/install-tower --dry-run # survey + questions + the plan; changes nothing +scripts/install-tower # the same, then does it after one yes +scripts/install-tower --verify # the checks only; changes nothing +``` + +It surveys the machine, asks only what it cannot work out, prints a plan with +each step marked `skip`, `create`, `change` or `blocked`, and applies it after +one yes. In order: + +1. packages, then the USB preconditions (section 3), then the Vimba X SDK (4) +2. builds the host +3. writes `hardware.ini` (5) - or, when one exists, only fills its gaps +4. seeds `config.ini` (9) and `credentials.env`, or edits only what is out of date +5. PlatformIO, then the camera profiles that are missing (8) +6. builds and flashes the firmware (6) - **blocked** while any drivetrain value + is still a `; TODO`, because that axis would refuse to move +7. `test hardware` +8. the boot service ([deployment.md](deployment.md) section 4), if wanted +9. asks whether someone is at the rig; only on a yes does it home the gimbal and + take a short capture - through the service, when the service is what homes it + +Every question has a key, so any can be answered up front - `--answer +tower_name=north`, `--answer yaw.r_sense_ohm=0.075` - or from a file of +`key=value` lines with `--answers FILE`. `--non-interactive` stops instead of +prompting. **It never invents a drivetrain value**: a wrong sense resistor cooks a +driver and a wrong homing current shears an endstop, and both have happened on +this project. Answer `unknown` and it writes a commented `; TODO` naming what to +measure. + +Re-running it is safe, and is how a configured tower is repaired: an existing +`hardware.ini` or `config.ini` is never regenerated, only filled in; an existing +camera profile is never overwritten; a firmware already running the declared +revision and pins is not reflashed; and if it fails after stopping the program, +it puts the program back the way it was. + +| Where | Result | +|---|---| +| Tower 121 (IOTA), 2026-09-23 | `[verified]` re-run on a configured tower: changed only what was out of date, `--verify` green, boot service installed and proven across a reboot | +| Tower 111 (Delta), 2026-09-23/24 | `[verified]` migration of a tower that predated all of this: host first, firmware once the drivetrain was known, `test hardware` green on every leaf (section 10b) | +| A freshly imaged board | `[untested]` - rehearsed with `--dry-run` and typed answers only. The first flash of a factory MicroPython RP2040 is automated but has not met a factory board | + --- ## 1. SSH by key `[verified]` @@ -158,9 +190,11 @@ first reboot, since surviving one is the entire point. > ssh -t ggs@ 'cd ~/projects/fwt_2a/software/hardware/lattepanda-iota && sudo ./install.sh --deps --preconditions' > ``` -> **The Delta fleet does not have these applied.** Checked on `10.11.12.111` -> 2026-09-15: `usbfs_memory_mb` is 16 and the kernel cmdline is stock. The -> preconditions were documented in known-issues but never persisted anywhere. +> **The Delta kit carries the same two files** (`hardware/lattepanda-3-delta/`, +> as symlinks to these). Tower 111 had neither - `usbfs_memory_mb` 16, camera +> LPM `u1_u2`, checked 2026-09-15 and again 2026-09-23 - and got both on +> 2026-09-23, reading 1000 and `0` afterwards. Not yet confirmed across a reboot +> on a Delta. ## 4. Vimba X SDK `[verified]` @@ -289,7 +323,8 @@ is what catches the second case, so it has to be set from the real travel. > `hardware.ini` is merged *over* `config.ini` and wins, so a shared key would > silently discard whatever `gimbal heading setup` had saved. What it does instead > is work the requirement out from `steps_per_rev` and print it as a `NOTE` in the -> block, because the default is wrong for any geared-up rig: +> block, because the default is wrong for any geared-up rig - and +> `scripts/install-tower` then sets it in `config.ini` when it is too small: > > ``` > ; NOTE: half this travel is 605389 counts, so config.ini needs @@ -315,9 +350,23 @@ yaw motor current: run 756 mA (GLOBALSCALER 80, IRUN 31), homing 189 mA (IRUN 7) -- R_sense 0.075 ohm from hardware.ini [Motor.yaw] ``` -Omit the block entirely on a tower whose endstops hold against full torque: no -`CURRENT` is sent, the firmware keeps its compile-time defaults, and nothing -changes. +There is no leaving the block out any more. The firmware holds no drivetrain +values of its own: at boot every axis stops its driver, asks the host (`NEED`), +and refuses to move until every value it asked for has arrived. An axis whose +block is missing or incomplete is an axis that will not turn - by design, since a +guessed value is how drivers get cooked and endstops sheared. (For a bench board +with no host, the firmware's `USE DEFAULTS` command loads deliberately gentle +limp-home values instead; `test hardware motor` fails on a tower running them.) + +The kit writes eight keys per axis - the four above plus `steps_per_rev`, +`sw_mode`, `enc_deviation` and `has_encoder` - each as a value or a `; TODO`; +`--motor AXIS.KEY=VALUE` gives any of them for any axis, where the older +`--r-sense`-style flags cover the first axis only. The three speeds +(`scan_speed_deg_s`, `homing_speed_deg_s`, `center_speed_deg_s`, all in degrees +per second) are written as gentle starting values: **on a tower that is being +migrated, set the homing speed to what its endstop detection is proven at**, as +was done for tower 111 (section 10b), or a stall-detected endstop may behave +differently. > **If an axis declares a reduced homing current and the firmware has not adopted > it, the host refuses to home** rather than driving into a stop at full torque. @@ -330,7 +379,27 @@ changes. > > **The by-id path changes when you flash the MCU**, because the name comes from > the running image: `usb-MicroPython_Board_in_FS_mode_*` before, a Pico/Arduino -> name after. Re-run `--probe` after step 6 and fix the file if it moved. +> name after. `scripts/install-tower` re-probes after flashing and updates +> `[Serial] device` itself; by hand, re-run `--probe` after step 6 and fix the +> file if it moved. + +### The wiring block: `[Pins]` + +The firmware is **built** from this section - which MCU pin reaches which driver +board - by `firmware/tools/gen_wiring.py`, reading the tower's own +`/etc/fire_gimbal_control/hardware.ini` when the firmware is built on the tower. +There is no copy in the firmware and no fallback: without `[Pins]` the firmware +build fails, naming what to add. Each kit's reference loom, with the table saying +which wire goes where, is its `pins.ini`; `install.sh --loom standard` copies it +in, and `--pin KEY=N` gives pins one at a time for a tower wired differently. + +The generator refuses what the board cannot do rather than building it: on the +RP2040, SPI pins outside SPI0's reach (SCK `{2,6,18,22}`, MOSI `{3,7,19,23}`, +MISO `{0,4,16,20}`); on the Leonardo, any `spi_*` key at all (its SPI is fixed on +the ICSP header) and the ICSP pins as chip-selects; duplicates; and a +`hardware.ini` written for the other board. The running firmware reports the +pins it was built with, and `test hardware motor` compares them with this +section - so a flash built from an older copy shows up there. ## 6. Motor firmware `[verified]` @@ -374,17 +443,20 @@ SERIAL_PORT=/dev/serial/by-id/usb-... ./deploy.sh --monitor # watch it boot ``` +`deploy.sh` syncs the tree and then runs `firmware/tools/flash.sh` on the tower, +which is also what `scripts/install-tower` runs, locally: one implementation of +the board-specific bootloader step for both. **Stop the program first**; it holds +the serial port. + **Confirm the pin map before flashing a board that is wired to motors.** A chip-select on the wrong pin writes into a driver that was never meant to hear -it. The pins are the `[Pins]` section of this tower's `hardware.ini`: the build -reads them from `/etc/fire_gimbal_control/hardware.ini` (`firmware/tools/gen_wiring.py`) -and **fails without them**, so add the section before the first build. The -reference loom, with the table saying which wire goes where, is -[`hardware/lattepanda-iota/pins.ini`](../hardware/lattepanda-iota/pins.ini) - -the loom as built on tower 121, which is **not** the map proposed in the porting -plan. Copy it in if the tower is wired the same way. The generator refuses pins -SPI0 cannot reach, and `test hardware motor` compares the declared pins with the -ones the running firmware reports it was built with. +it. The build takes the pins from this tower's `hardware.ini` and fails without +them - see "The wiring block" in section 5. The IOTA's reference loom is the one +as built on tower 121, which is **not** the map proposed in the porting plan. + +`[verified]` on both boards: the RP2040 on tower 121 (2026-09-23), the Leonardo on +tower 111 (2026-09-24) - the Leonardo through its own 1200-baud touch, which +`flash.sh` leaves alone. ### Getting a factory board to accept its first flash @@ -469,8 +541,9 @@ loop and unsupervised energised drivers would hold torque. | before | `usb-MicroPython_Board_in_FS_mode_-if00` | `2e8a:0005` | | after | `usb-Raspberry_Pi_Pico_-if00` | `2e8a:000a` | -So generate `hardware.ini` (step 5) **after** flashing, or re-run -`install.sh --hardware-ini --force` to pick up the new path. +So generate `hardware.ini` (step 5) **after** flashing, or update `[Serial] +device` once the name has moved - `scripts/install-tower` does that itself, +re-probing after every flash. ### Verify the flash @@ -513,10 +586,14 @@ RUN_ARGS="--start" ./deploy.sh # rsync + remote cmake + build ``` -The first deploy seeds `config.ini` from `config/config.example.ini` **only if +The first deploy copies `config/config.example.ini` to `config.ini` **only if the device has none** - see the warning in step 2 about a machine that arrived -with somebody else's config. +with somebody else's config. That copy still carries the example's placeholders +and the keys `hardware.ini` owns; `scripts/install-tower` seeds it instead with +`build/fgc-config seed`, which fills in the answers, comments out every key +`hardware.ini` sets, and refuses to write a file the program would not load. +`[verified]` on tower 111: builds on GCC 11 / Ubuntu 22.04. `[verified]` on tower 121: builds clean on GCC 15 / Ubuntu 26.04 with a single `-Wformat-truncation` warning in `utcStampNow()`, which is a false positive (the compiler cannot prove `tm_year` is four digits; the real output is 17 bytes into @@ -610,8 +687,12 @@ zkms_server_ip = Credentials belong in `/etc/fire_gimbal_control/credentials.env`, not here. Anything `hardware.ini` now owns - `[Serial] device`, `[IMU]`, `[Env]`, the -camera ids - should be **deleted** from `config.ini` rather than left to rot: -the program warns at startup about every key set in both. +camera ids - should not be in `config.ini` as well: the program warns at startup +about every key set in both. `scripts/install-tower` comments them out (value and +all), and `build/fgc-config set config.ini --config config.ini --hardware +/etc/fire_gimbal_control/hardware.ini --drop Section.key=note` does one by hand. +Every edit through `fgc-config` is loaded the way the program loads it before it +is written, and keeps the previous value as a `;prev` comment. > **`[Camera.] id` is required even when the camera is disabled.** Config > validation runs before any of the feature flags are consulted, so a tower with @@ -619,20 +700,12 @@ the program warns at startup about every key set in both. > `DEV_0000000000`, is what `deploy.sh`'s placeholder guard refuses. On a tower > that has its camera, the id lives in `hardware.ini` and this never comes up. -`[Features] enable_imu` and `enable_env` matter more than they look. Declaring -`imu = none` in `hardware.ini` does **not** yet switch them off - deriving them -is section 12 of the config plan and is not implemented. On tower 121 the first -clean-looking run did this: - -``` -[INFO ] MTi IMU started on /dev/ttyS0 @ 115200 -[ERROR] I2C: failed to open /dev/i2c-2 -[ERROR] SHT41: failed to open I2C bus; sensor disabled -``` - -It "started" an IMU on a tower that has none - `/dev/ttyS0` exists and accepts a -serial open whether or not anything is attached to it. Set both flags to `false` -by hand until reconciliation lands. +`[Features] enable_imu` and `enable_env` are for towers **without** a +`hardware.ini`. Where `hardware.ini` declares `imu` / `env`, the declaration +decides, and a flag that contradicts it is warned about and ignored. Before that +was so, tower 121's first clean-looking run "started" an IMU it does not have - +`/dev/ttyS0` accepts a serial open whether or not anything is attached - which is +why a declared `none` has to be able to switch a sensor off by itself. ## 10. First run: bring the subsystems up one at a time `[partial]` @@ -690,8 +763,16 @@ DUMP BEGIN build=aab1925-dirty uptime=7925212 mcusr=0x3 free_ram=252712 DUMP Y state=1 ... speed=3000 center=500000 has_encoder=1 ``` -Check `build=` matches the firmware you flashed and `speed=` matches the -drivetrain values you intended. Then, and only then, move the thing: +Check `build=` matches the firmware you flashed, and that the handshake +completed - the log says `firmware config: answered N keys for axis Y`, and the +axis leaves state 5 (`NEEDCFG`). The scripted form of all of this, which also +compares every value the firmware holds with `hardware.ini`, moves nothing: + +```bash +./build/fire_gimbal_control --no-mqtt --test hardware # exit 0 PASS, 4 FAIL, 5 could not start +``` + +Then, and only then, move the thing: ``` gimbal home # the endstop-finding sequence - see the warning below @@ -703,13 +784,14 @@ gimbal move 10,0 # a small deliberate move | Message | Why | |---|---| -| `Motor controller: ERR axis not present` | The host configures both axes at startup; a yaw-only firmware refuses the pitch half. Harmless until presence reconciliation lands (config plan section 10) | +| `Motor controller: ERR unknown command`, a burst at startup | Only on firmware older than the config handshake: it does not know `SET`. Seen on tower 111 before its flash; harmless there, and gone once the firmware is current | | `encoder tracking error on yaw: 500366 counts (limit 400)` | Before homing, the firmware presets `XACTUAL` to the centre count while the encoder sits at its own zero, so they disagree by the whole range. Homing resyncs them. **If it persists after a successful home, that is real** | > **On an unfamiliar drivetrain, homing is the dangerous step** - it finds the -> endstop by driving into it. Start from a deliberately slow `homing_speed` and -> walk it up, watching the free-running encoder gap that `ENC_DEVIATION_LIMIT` is -> measured against. Be at the rig with a hand on the power for the first one. +> endstop by driving into it. Start from a deliberately slow +> `homing_speed_deg_s` and walk it up, watching the free-running encoder gap that +> `enc_deviation` is measured against. Be at the rig with a hand on the power for +> the first one - which is why `scripts/install-tower` asks before it homes. ### Phase 2 - camera `[verified]` @@ -753,9 +835,10 @@ the ground station. That lives in `~/zkms-client/`, outside this repository. ### Phase 4 - IMU and ambient sensor -Only if fitted. `hardware.ini` should already say so, and `[Features] -enable_imu` / `enable_env` have to agree with it (see step 9 - declaring -`imu = none` does not yet switch the flag off by itself). +Only if fitted, and declared in `hardware.ini` (`imu`, `env`) - the declaration +is what switches them on. `test hardware imu` wants a `0xFA`-framed stream from +the MTi; `test hardware env` wants a real I2C adapter and a CRC-valid SHT41 +reading. `[verified]` on tower 111, 2026-09-23: both. ### Phase 5 - unattended @@ -766,9 +849,10 @@ scripts/fgc start --start # detached, inside tmux scripts/fgc status # running? since when? which PID? ``` -Close the ssh session, reconnect, and check it is still running. Then reboot the -tower and confirm the USB preconditions survived (`install.sh --probe`) and the -app comes back. +Close the ssh session, reconnect, and check it is still running. Then install +the boot service (`scripts/install-tower` asks; [deployment.md](deployment.md) +section 4), reboot the tower, and confirm with `scripts/install-tower --verify` +that the USB preconditions survived and the program came back by itself. ### Calibration @@ -791,7 +875,31 @@ turning. So the order matters. **Host first, firmware second.** Each step below either changes nothing or is reversible, and the step that could strand the tower comes -after the one that prevents it. +after the one that prevents it. `scripts/install-tower` follows this order by +itself, and blocks the flash while any drivetrain value is missing. + +### The worked example: tower 111 `[verified]` + +The Delta tower, which predated `hardware.ini`, the handshake and the kits. +2026-09-23 and -24, through the installer with the Delta kit: + +1. **A rollback copy** of the sources, the running binary and `config.ini`, in + `~ggs/rollback-20260923/`, before anything changed. +2. **The host half.** Preconditions, `hardware.ini` with every drivetrain value + a `; TODO`, `config.ini`'s duplicated keys commented out, the missing `night` + profile, a new host. `test hardware`: axes, IMU, env and camera PASS; motor + FAIL on the old firmware's missing echoes, as expected (step 2 below). Firmware + blocked. It homed and resumed capturing on the new host. +3. **The drivetrain, without inventing it.** The sense resistors were read off + both boards (75 mOhm). Everything else was converted from what the old + firmware ran - its revision is in `DUMP BEGIN build=`, and git history at + that revision has its register values - through the host's own planners, and + round-tripped back to the identical registers before being written. Homing + and centring speeds were set to what that firmware used, not the kit's + gentle defaults, since its endstop detection was proven at those. +4. **The flash**, by the installer: every gap filled, the firmware built from + `hardware.ini` and flashed, `test hardware` green on every leaf, and homing + within 0.03 % of the old firmware's travel. ### 0. Record what you are rolling back to @@ -806,9 +914,14 @@ the comparison that tells you whether the new one adopted the same values. ### 1. Update `hardware.ini` — nothing else, and no restart yet -Add the `[Motor.]` block (section 5). The five keys the new firmware will -not run without are `steps_per_rev`, `enc_deviation`, `sw_mode`, plus the two -speeds; `install.sh --stdout` generates the block if you would rather not type it. +Add the `[Motor.]` block (section 5) and `[Pins]`. The firmware asks for +eleven values per axis - `steps_per_rev`, `global_scaler`, `run_irun`, +`homing_irun`, `center_count`, `soft_margin`, `enc_deviation`, `normal_speed`, +`homing_speed`, `center_speed`, `sw_mode` - and the host derives them from the +block (currents from milliamps and the sense resistor, speeds from degrees per +second, centre and margin from `config.ini`). `install.sh --stdout` generates the +block; for values nobody has measured, convert what the old firmware runs, as +for tower 111 above. Doing this **before** flashing is what keeps the window at zero: the host is ready to answer the moment the firmware starts asking. @@ -825,16 +938,21 @@ scripts/fgc peek test hardware motor ``` -Expect: no handshake lines at all, and the motor leaf reporting a **version skew** -rather than a mismatch — +Expect: no handshake lines at all (and possibly a burst of `ERR unknown +command` - the old firmware refusing `SET`), and the motor leaf **failing** on +missing echoes, reported as a version skew rather than a mismatch: ``` -; yaw_current: yaw: firmware echoes only part of the current (2 of 3 fields - absent), so it cannot be confirmed -- it predates the full CURRENT echo. Re-flash it +; yaw_pins: yaw: the firmware does not report the pins it was built with, so its + wiring cannot be confirmed -- it predates the pin echo. Re-flash it ``` -If instead you see wrong *values*, stop: something in `hardware.ini` disagrees -with what the tower is really running, and flashing will not fix it. +That FAIL is by design - a check that could not happen must not read as a pass - +so on old firmware the gate is "every other leaf green, and `hw_motor` failing +only on missing echoes". A speed can also differ before the first home, since +the host sends it after homing. If instead you see wrong *values* for something +the firmware does echo, stop: `hardware.ini` disagrees with what the tower is +really running, and flashing will not fix it. ### 3. Flash @@ -923,37 +1041,43 @@ table and lets the tower move while you work out what `hardware.ini` is missing. ## 11. Verification checklist -The checks that move nothing can be run as one command, which is what a script -should use rather than typing into the dashboard. Stop the app first: it holds -the serial port. +Most of this is one command, which changes nothing: + +```bash +scripts/install-tower --verify +``` + +It checks the USB preconditions, that `config.ini` and `hardware.ini` load +together, that `hardware.ini` has no gaps, that the firmware reports the pins it +declares, and the boot service - and runs `test hardware`: inside the running +program when it is up (pausing capture, then resuming it), or as a one-shot run +when it is not. By hand, the one-shot form is: ```bash -scripts/fgc stop GENICAM_GENTL64_PATH=/opt/VimbaX/cti ./build/fire_gimbal_control --no-mqtt --test hardware echo $? # 0 PASS, 4 FAIL, 5 could not start (the log says why) -scripts/fgc start ``` It waits for the firmware to be talking and configured before it looks, so a freshly flashed board is not failed for values the host had not sent yet. -`build/fgc-config check --config config.ini --hardware /etc/fire_gimbal_control/hardware.ini` -checks that the two files load together, and changes nothing. -| Check | How | Tower 121 | -|---|---|---| -| Key-only ssh | `ssh ggs@ hostname`, no prompt | ✅ | -| Preconditions survive a reboot | reboot, then `install.sh --probe`: `usbfs_memory_mb 1000`, `lpm_permit 0` | ✅ confirmed across a real reboot | -| `hardware.ini` found unaided | run with no `FGC_HARDWARE`: startup names `/etc/fire_gimbal_control/hardware.ini` | ✅ | -| Camera enumerates | `ListCameras_VmbC` reports the `DEV_` id in `hardware.ini` | ✅ `DEV_1AB22C0B7D4E` | -| Camera profile applies | startup logs `profile 'standard' applied ... (N settings)` | ✅ 17206 settings | -| Firmware identity | `gimbal dump` reports the `GIT_REV` you flashed | ✅ `build=aab1925-dirty` | -| Driver answers over SPI | boot banner shows `IOIN Y=0x… version=0x30` | ✅ | -| Remote re-flash | `BOOTSEL`, then `./deploy.sh` - no hands on the board | ✅ | -| Declared hardware matches | startup logs no presence warnings | ✅ no shadowed-key warnings | -| Homing repeatable | `gimbal home` several times, endstop counts within `home_spread_max_counts` | ✅ three homes within 0.27 % (1,210,778 / 1,208,358 / 1,207,514 counts) | -| Endstop survives homing | inspect the stop after a home; the axis must stall against it, not push through | ✅ at 189 mA, after failing at full torque | -| Degree scale derived | startup logs `counts_per_deg` agreeing with the homed travel | ✅ 3552.87, saved with `;prev` history | -| Images land | `.jxl` files appear under the configured output dir | ❌ needs a capture run | -| Survives a disconnect | `scripts/fgc start`, close ssh, reconnect, `scripts/fgc status` | ✅ runs under the boot service, independent of any ssh session | -| Starts at boot | reboot with nobody logged in; `scripts/fgc status` from a later ssh | ✅ 2026-09-23, lock under `/run/user/1000` (lingering on) | -| Reports to a broker | topics arrive at the configured broker | ❌ no broker on this tower yet | +| Check | How | Tower 121 (IOTA) | Tower 111 (Delta) | +|---|---|---|---| +| Key-only ssh | `ssh ggs@ hostname`, no prompt | ✅ | ✅ | +| Preconditions applied | `--verify`: `usbfs_memory_mb 1000`, camera LPM `0` | ✅ | ✅ 2026-09-23 (were 16 and `u1_u2`) | +| Preconditions survive a reboot | reboot, then `--verify` | ✅ confirmed across a real reboot | ❓ not rebooted yet | +| `hardware.ini` found unaided | startup names `/etc/fire_gimbal_control/hardware.ini` | ✅ | ✅ | +| Declared hardware matches | startup logs `declared assembly confirmed`; `test hardware axes` | ✅ yaw | ✅ yaw, pitch | +| IMU / ambient sensor | `test hardware imu` / `env` | none declared | ✅ MTi streaming, SHT41 CRC-valid on a real I2C bus | +| Camera enumerates | `ListCameras_VmbC` reports the `DEV_` id in `hardware.ini` | ✅ `DEV_1AB22C0B7D4E` | ✅ `DEV_1AB22C049CD1` | +| Camera profile applies | startup logs `profile 'standard' applied ... (N settings)` | ✅ 17206 settings | ✅ 3743 settings | +| Firmware identity | `DUMP BEGIN build=` is the revision flashed | ✅ | ✅ `a0b1e4a` | +| Driver answers over SPI | `IOIN` reads version `0x30` for every axis | ✅ | ✅ both axes | +| Firmware holds `hardware.ini` | `test hardware motor`: values, current, pins | ✅ | ✅ every leaf, 2026-09-24 | +| Remote re-flash | no hands on the board | ✅ `BOOTSEL` | ✅ 1200-baud touch | +| Homing repeatable | endstop counts across homes | ✅ three homes within 0.27 % | ✅ within 0.03 %, old and new firmware | +| Endstop survives homing | the axis must stall against the stop, not push through | ✅ at 189 mA, after failing at full torque | not inspected; full current, as for its whole service life | +| Degree scale derived | startup logs `counts_per_deg` agreeing with the homed travel | ✅ 3552.87 | ✅ agrees, calibration left alone | +| Images land | `.jxl` files appear under the configured output dir | ❌ needs a capture run | ✅ | +| Starts at boot | reboot with nobody logged in; `scripts/fgc status` from a later ssh | ✅ lock under `/run/user/1000` | ❓ service enabled (`--init --start`), not rebooted yet | +| Reports to a broker | topics arrive at the configured broker | ❌ no broker on this tower yet | ❓ not checked | diff --git a/scripts/fire-gimbal-control.service b/scripts/fire-gimbal-control.service index 09721b5..e678c30 100644 --- a/scripts/fire-gimbal-control.service +++ b/scripts/fire-gimbal-control.service @@ -36,7 +36,9 @@ WorkingDirectory=@REPO@ ExecStart=@REPO@/scripts/fgc start @ARGS@ ExecStop=@REPO@/scripts/fgc stop TimeoutStartSec=120 -TimeoutStopSec=60 +# Longer than `fgc stop` itself waits (120 s) for the program's clean shutdown: +# past this systemd kills it outright, which wedges the camera for the next run. +TimeoutStopSec=150 [Install] WantedBy=multi-user.target