From 9b232c3f6061e40268b81e23a5e322e3e70f1db9 Mon Sep 17 00:00:00 2001 From: pgdalmeida Date: Wed, 16 Sep 2026 18:27:51 +0200 Subject: [PATCH] Add a from-scratch tower installation guide with phased bring-up --- docs/tower-installation.md | 668 +++++++++++++++++++++++++++++++++++++ 1 file changed, 668 insertions(+) create mode 100644 docs/tower-installation.md diff --git a/docs/tower-installation.md b/docs/tower-installation.md new file mode 100644 index 0000000..25bf403 --- /dev/null +++ b/docs/tower-installation.md @@ -0,0 +1,668 @@ +# Installing a tower from scratch + +How a new fire-watch tower host goes from a freshly imaged machine to a running, +calibrated `fire_gimbal_control`. Written while doing exactly that on the IOTA +tower at `10.11.12.121` (2026-09-15) and kept honest about which steps have +actually been exercised and which are still on paper. + +> [deployment.md](deployment.md) is the *update* loop for a tower that already +> works. This is the *first* install, including the parts that only happen once: +> the board's preconditions, the Vimba X SDK, `hardware.ini`, and the motor +> firmware. + +**Status legend** — each step says where it stands: +`[verified]` run end to end on a real tower and the result checked · +`[partial]` run, with something still open · `[untested]` written from the +design, not yet exercised. + +--- + +## What you need before starting + +| | | +|---|---| +| The tower host | Powered, on the network, reachable by IP, with an OS installed and an `ssh` server running | +| An account on it | `ggs`, in the `sudo` and `dialout` groups | +| A dev box | This repository checked out, plus a copy of the Vimba X SDK (login-walled, so it cannot be fetched by a script) | +| The hardware | Camera plugged in, motor MCU plugged in, and a note of **what was assembled**: which axes, which IMU, which ambient sensor | + +Two facts decide most of what follows, so establish them first: + +```bash +ssh ggs@ 'cat /sys/class/dmi/id/product_name; . /etc/os-release && echo "$PRETTY_NAME"' +``` + +| Board | MCU | Setup kit | PlatformIO env | +|---|---|---|---| +| LattePanda 3 Delta / Sigma | ATmega32U4 (Leonardo), 5 V | *not written - the Delta fleet predates the kits* | `leonardo` | +| LattePanda IOTA | RP2040, 3.3 V | [`hardware/lattepanda-iota/`](../hardware/lattepanda-iota/) | `iota_rp2040` | + +--- + +## 1. SSH by key `[verified]` + +Everything after this is driven over ssh from the dev box, and `deploy.sh` runs +several ssh commands per invocation, so password auth is not workable. + +```bash +# on the dev box +ssh-copy-id -i ~/.ssh/id_ed25519_ggs_pedro.pub ggs@ +``` + +Then give the host a block in `~/.ssh/config`, so every tool picks the right key +without being told: + +``` +Host 10.11.12.121 + User ggs + IdentityFile ~/.ssh/id_ed25519_ggs_pedro +``` + +Verify with `ssh ggs@ hostname` - it must not prompt. + +> **`ssh-copy-id` needs a terminal.** Run it in a real shell, not through a tool +> that gives it no tty: without one, ssh looks for a GUI `ssh-askpass`, and on a +> machine that has none it fails with `ssh_askpass: exec(...): No such file or +> directory` and then `Permission denied` - which looks like a rejected key +> rather than a prompt that never appeared. + +## 2. Survey the machine before changing it `[verified]` + +The setup kit can describe the machine without touching it, and this is the +cheapest moment to notice that the tower is not what you assumed: + +```bash +rsync -a hardware/ ggs@:~/projects/fwt_2a/software/hardware/ +ssh ggs@ '~/projects/fwt_2a/software/hardware/lattepanda-iota/install.sh --probe' +``` + +``` +board : lattepanda-iota +os : Ubuntu 26.04.1 LTS (7.0.0-31-generic) +motor MCU : /dev/serial/by-id/usb-MicroPython_Board_in_FS_mode_df61a863dfa02932-if00 +camera DEV_ id : NOT FOUND (needs the Vimba X SDK installed) +alvium on usb : 1 device(s) +usbfs_memory_mb : 16 (want >= 1000) +camera lpm_permit: u1_u2 (want 0) +``` + +Read every line. On tower 121 this pass found that the MCU was still running the +MicroPython image it shipped with, and that both USB preconditions were at their +(wrong) defaults. + +**Check what is already in `~/projects/`.** A machine cloned from another tower, +or one that has been used for development, can arrive with a `config.ini` that +belongs to a different rig. Tower 121 had a byte-identical copy of the dev box's +`config.ini` - `tower_name = dev`, broker `127.0.0.1` - which is worse than no +config at all, because **none of those values trip the placeholder guard in +`deploy.sh --run`**. That guard only catches `ExampleTower`, `CHANGE_ME` and +`DEV_0000000000`. + +```bash +ssh ggs@ 'grep -E "tower_name|zkms_server_ip|^id" ~/projects/fwt_2a/software/config.ini' +``` + +## 3. Host preconditions and build dependencies `[verified]` + +Both come from the board's setup kit, in one step, on the device: + +```bash +ssh ggs@ +cd ~/projects/fwt_2a/software/hardware/lattepanda-iota +sudo ./install.sh --deps --preconditions +``` + +`--deps` installs [`packages.txt`](../hardware/lattepanda-iota/packages.txt): the +toolchain, OpenCV, Boost, libjxl, tmux, and the Python bits PlatformIO needs. + +`--preconditions` installs two files and applies them immediately: + +| File | Why | +|---|---| +| `/etc/udev/rules.d/99-fwt-alvium-usb.rules` | Disables USB3 hardware LPM on the camera's port, which the host otherwise re-enables on every enumeration. With U1/U2 on, the camera logs link errors and frame delivery stalls (known-issues #19) | +| `/etc/tmpfiles.d/10-fwt-usbfs.conf` | Raises `usbfs_memory_mb` from 16 to 1000 at every boot. A single full-resolution frame does not fit in 16 MB | + +Both are settings that **revert on reboot** if nobody persists them, which is the +whole of known-issue #19. The script re-reads them after applying and says so: + +``` +>> usbfs_memory_mb = 1000 +>> camera USB3 LPM disabled (2-1) +``` + +If it instead warns that a value did not take, stop and find out why - a camera +on a host with these wrong looks like a flaky camera. + +Confirmed on tower 121 (2026-09-15): both went from wrong to right in one run, +`usbfs_memory_mb` 16 -> 1000 and `lpm_permit` `u1_u2` -> `0`. Re-check after the +first reboot, since surviving one is the entire point. + +> **Neither step can be driven from a tool without a terminal.** `sudo` needs a +> tty to prompt, so this runs from a real terminal window: +> +> ```bash +> ssh -t ggs@ 'cd ~/projects/fwt_2a/software/hardware/lattepanda-iota && sudo ./install.sh --deps --preconditions' +> ``` + +> **The Delta fleet does not have these applied.** Checked on `10.11.12.111` +> 2026-09-15: `usbfs_memory_mb` is 16 and the kernel cmdline is stock. The +> preconditions were documented in known-issues but never persisted anywhere. + +## 4. Vimba X SDK `[verified]` + +Proprietary and login-walled, so it is staged from a machine that already has it +rather than downloaded on the tower: + +```bash +# dev box +rsync -az ~/Programs/VimbaX_2025-3 ggs@:~/ +# tower +sudo ./install.sh --vimba ~ggs/VimbaX_2025-3 +``` + +That copies it to `/opt/VimbaX_`, points `/opt/VimbaX` at it, runs the +SDK's `Install_GenTL_Path.sh` (which registers `GENICAM_GENTL64_PATH` in +`/etc/profile.d` and writes the `1ab2` udev rule), registers `api/lib` with the +dynamic loader, and then asks the SDK to list cameras as a check. + +**The SDK asks for a reboot before the transport layers are usable.** If it +finds nothing, reboot and try again before suspecting the camera. On tower 121 +no reboot was needed - the camera answered immediately. + +Two things about that check, both found the hard way here: + +- **The binary is `ListCameras_VmbC`**, in `/opt/VimbaX/bin/`. There is no plain + `ListCameras`. +- **Run it with `GENICAM_GENTL64_PATH` set**, or it fails with `Could not start + system. Error code: -16` on a perfectly good installation - that is "no + transport layer found", not "no camera": + + ```bash + GENICAM_GENTL64_PATH=/opt/VimbaX/cti /opt/VimbaX/bin/ListCameras_VmbC + ``` + +> **The SDK ships a camera simulator, and it is on by default.** On tower 121 +> `ListCameras_VmbC` reported **four** cameras: the real Alvium plus three +> synthetic ones from `VimbaCameraSimulatorTL.cti`, with ids like `DEV_Cam1`. A +> tower configured with one of those would open its "camera" happily and capture +> synthetic images forever. Take the id only from a block whose transport layer +> path is `VimbaUSBTL` or `VimbaGigETL` - which is what `install.sh --probe` +> does, after this bit me. + +> Non-interactive ssh shells do not source `/etc/profile.d`, which is why +> `deploy.sh` resolves `GENICAM_GENTL64_PATH` itself before launching the app. +> Getting this wrong shows up as `VmbStartup()` failing with *"Could not start +> Vimba X API"* even though the viewer works when you are logged in. + +## 5. `hardware.ini` - what this machine IS `[verified]` + +```bash +sudo ./install.sh --hardware-ini --axes yaw --imu none --env none --cameras rgb +``` + +This writes `/etc/fire_gimbal_control/hardware.ini`, **outside** the directory +`deploy.sh` rsyncs, so no deploy can overwrite or delete it. + +The probed half - board model, firmware env, the MCU's stable `by-id` path, the +camera's `DEV_` id - the script fills in. The declared half is the flags, and +they are not a formality: the program degrades gracefully around missing +hardware, so **a part that was never fitted and a part that has failed look +identical** unless the file says which was expected. Declare what was assembled. + +Check the result, especially the serial path: + +```bash +ssh ggs@ 'cat /etc/fire_gimbal_control/hardware.ini' +``` + +> `[Serial] device` must be a `/dev/serial/by-id/` path, never `/dev/ttyACM0`. +> Tower 121 has a 5G modem that claims three `ttyUSB` nodes, and enumeration +> order is not stable across reboots. +> +> **The by-id path changes when you flash the MCU**, because the name comes from +> the running image: `usb-MicroPython_Board_in_FS_mode_*` before, a Pico/Arduino +> name after. Re-run `--probe` after step 6 and fix the file if it moved. + +## 6. Motor firmware `[verified]` + +The firmware is built and flashed **on the tower** - it is the machine wired to +the MCU - so PlatformIO is installed there, not on the dev box. + +```bash +# tower: PlatformIO core, in its own venv under ~/.platformio. No sudo needed. +curl -fsSL https://raw.githubusercontent.com/platformio/platformio-core-installer/master/get-platformio.py -o /tmp/get-platformio.py +python3 /tmp/get-platformio.py +~/.platformio/penv/bin/pio --version # check it actually runs +``` + +`[verified]` on tower 121: PlatformIO Core 6.2.0 on the system Python 3.14.4. + +> The installer ends with a `FileNotFoundError` traceback out of +> `multiprocessing/util.py` about a missing `/tmp/.piocore-installer-*` path. +> It is a temp-directory cleanup finalizer running after the work is done - +> cosmetic. Judge the install by whether `pio --version` answers. + +`deploy.sh` looks for `pio` in the usual places, but a non-interactive ssh shell +does not have `~/.platformio/penv/bin` on PATH, so name it explicitly: + +```ini +REMOTE_PIO=/home/ggs/.platformio/penv/bin/pio +``` + +Then, from the dev box, in `firmware/`: + +```ini +# .deploy.env +REMOTE_HOST=ggs@10.11.12.121 +REMOTE_DIR=/home/ggs/projects/fwt_2a/firmware +PIO_ENV=iota_rp2040 # leonardo on a Delta +SERIAL_PORT=/dev/serial/by-id/usb-... +``` + +```bash +./deploy.sh --compile-only # build on the tower, flash nothing +./deploy.sh # build + upload +./deploy.sh --monitor # watch it boot +``` + +**Confirm the pin map before flashing a board that is wired to motors.** The map +lives in `src/platform_iota.h` and a chip-select on the wrong pin writes into a +driver that was never meant to hear it. The numbers there are the loom as built +on tower 121, which is **not** the map proposed in the porting plan. + +### Getting a factory board to accept its first flash + +`[verified]` on tower 121, and this is the step that will bite you. These boards +ship running MicroPython, and **the normal upload does not work against it**: + +``` +Forcing reset using 1200bps open/close on port /dev/ttyACM0 +No new RPxxxx device found yet, waiting.. (x20, then it gives up) +``` + +| What | Factory board (MicroPython) | Our firmware | +|---|---|---| +| 1200-baud touch, i.e. a plain upload | no reset at all | resets, but **reboots into the application** | +| `picotool reboot -f -u` | refused: no reset interface | - | +| MicroPython REPL | **works** | n/a | + +So drop into MicroPython's own REPL and tell it to enter the bootloader. Write +this to a file and run it with `~/.platformio/penv/bin/python` (it has pyserial): + +```python +import serial, time +s = serial.Serial("/dev/serial/by-id/usb-MicroPython_Board_in_FS_mode_XXXX-if00", 115200, timeout=1) +s.write(b"\x03\x03"); time.sleep(0.3) # Ctrl-C: interrupt whatever is running +s.write(b"import machine\r\n"); time.sleep(0.3) +s.write(b"machine.bootloader()\r\n") # the port drops here - that is success +``` + +```bash +lsusb | grep 2e8a # expect: 2e8a:0003 Raspberry Pi RP2 Boot +``` + +Then run `./deploy.sh` and it flashes. A physical BOOTSEL press does the same +job if you are standing at the board. + +**Every board needs exactly one physical BOOTSEL press in its life** - the one +that installs the first image carrying the `BOOTSEL` command (below). After +that, updates are remote. + +**`sudo ./install.sh --preconditions` must have run first**, or picotool finds +the board and refuses it: + +``` +RP2040 device at bus 1, address 4 appears to be in BOOTSEL mode, but picotool + was unable to connect. Maybe try 'sudo' or check your permissions. +``` + +### Re-flashing later: the `BOOTSEL` command + +`[verified]` on tower 121. Neither documented remote path works on this board - +the 1200-baud touch resets the firmware but returns it to the application, and +`picotool reboot -f -u` answers `Unable to locate reset interface on the +device`. So the firmware carries a command that does it deliberately: + +```bash +# over the serial port, or from the app: gimbal raw "BOOTSEL" +BOOTSEL -> OK entering bootloader - reflash to return +lsusb | grep 2e8a # 2e8a:0003 Raspberry Pi RP2 Boot +./deploy.sh # flashes and reboots into the new firmware +``` + +The whole loop was run end to end from the dev box with nobody at the tower. +`BOOTSEL` disables every axis first, since the bootrom does not run the control +loop and unsupervised energised drivers would hold torque. + +> **It takes the gimbal out of service and the firmware cannot undo it** - once +> in the bootrom there is no command loop left to change its mind. Recovery is +> `deploy.sh`, or a power cycle: entering the bootrom erases nothing, so the +> existing image is still in flash. Anything that can write to the serial port +> can send it, including `gimbal raw`. Near misses (`BOOT`, `BOOTSE`) are +> rejected as unknown commands. + +### The identity changes when you flash + +| | by-id path | USB | +|---|---|---| +| before | `usb-MicroPython_Board_in_FS_mode_-if00` | `2e8a:0005` | +| after | `usb-Raspberry_Pi_Pico_-if00` | `2e8a:000a` | + +So generate `hardware.ini` (step 5) **after** flashing, or re-run +`install.sh --hardware-ini --force` to pick up the new path. + +### Verify the flash + +Read the boot banner over the serial port: + +``` +BOOT build=aab1925-dirty +IOIN Y=0x3000001E version=0x30 +ST Y:R,500000,18,80084000,0,8,8,Sd +``` + +`version=0x30` is the TMC5160 answering over SPI - the single most useful line +here, because it proves the pin map, the 3.3 V logic level and the driver board +all at once. `BOOT build=` must match the commit you flashed. One `Y` line and +no `P` line is a yaw-only build. + +> **Flashing an RP2040 remotely is not the same as a Leonardo.** The documented +> flow is *hold BOOTSEL, tap RST, drag a `.uf2`* - a person at the board. The +> two remote paths (the 1200-baud touch that the Arduino tooling already uses, +> and `picotool reboot -f -u`) both need the **running** image to present a USB +> CDC port. That is why `setup()` brings `Serial` up before any hardware init: +> an image that hangs before USB comes up cannot be reflashed without a site +> visit, while an interrupted flash recovers by itself - the bootrom falls back +> to mass storage when flash holds no valid image. + +## 7. Deploy the software `[verified]` + +On the dev box, in `software/`: + +```ini +# .deploy.env +REMOTE_HOST=ggs@10.11.12.121 +REMOTE_DIR=/home/ggs/projects/fwt_2a/software +CMAKE_ARGS="-DWITH_MQTT=ON -DWITH_VIMBA=ON" +RUN_ARGS="--start" +``` + +```bash +./deploy.sh --check-deps # must print "remote deps OK" +./deploy.sh # rsync + remote cmake + build +``` + +The first deploy seeds `config.ini` from `config/config.example.ini` **only if +the device has none** - see the warning in step 2 about a machine that arrived +with somebody else's config. + +`[verified]` on tower 121: builds clean on GCC 15 / Ubuntu 26.04 with a single +`-Wformat-truncation` warning in `utcStampNow()`, which is a false positive (the +compiler cannot prove `tm_year` is four digits; the real output is 17 bytes into +a 32-byte buffer). + +## 8. Camera profiles `[verified]` + +**Without a profile the program does not start** - `no camera profiles (*.xml) +in config/camera_profiles/rgb` is a startup error, by design. A fresh tower has +an empty profile directory, so this step is not optional. + +The documented way to make one is the VimbaX Viewer, which is a GUI - awkward on +a headless tower and impossible unattended. `scripts/export_camera_profile.py` +does the same job through the SDK's Python API: + +```bash +# once per tower: the SDK ships the wheel +python3 -m venv ~/.venvs/vmbpy +~/.venvs/vmbpy/bin/pip install /opt/VimbaX/api/python/vmbpy-*.whl + +cd ~/projects/fwt_2a/software +export GENICAM_GENTL64_PATH=/opt/VimbaX/cti +~/.venvs/vmbpy/bin/python scripts/export_camera_profile.py --profile standard --dry-run +~/.venvs/vmbpy/bin/python scripts/export_camera_profile.py --profile standard +~/.venvs/vmbpy/bin/python scripts/export_camera_profile.py --profile night +``` + +It writes each recipe value to the real camera **in dependency order**, lets the +camera validate it, and exports what was actually accepted. That preserves the +property the Viewer workflow exists to provide: GenICam validity is stateful, so +a hand-written XML can be individually plausible and collectively wrong. Here +the camera is the judge, and anything it refused is printed rather than assumed. + +Read the summary line it prints - it is the number that decides whether the +profile can be delivered at all: + +``` +result : 2664x1520 RGB8 12.1 MB/frame 4.95 fps = 60 MB/s +``` + +### Profiles do not transfer between cameras + +`[verified]` the hard way. The fleet runs several Alvium generations and their +feature vocabularies differ, so copying another tower's XML does not work: + +| | Delta tower (1800 U-1240c) | IOTA tower (1800 U-1620c) | +|---|---|---| +| `SensorBitDepth` | `Bpp8` | **no `Bpp8`** - only `Adaptive` / `Bpp12` | +| `IntensityAutoPrecedence` | `Minimizeexposuretime` / `Minimizenoise` | **`MinimizeBlur` / `MinimizeNoise`** | +| Sensor | 12 MP | 16 MP, 5328x3040 | + +### Binning is not optional, and the frame-rate ceiling is not what it looks like + +Unbinned RGB8 on the 16 MP sensor is a **48.6 MB frame** - over the ~20 MB +ceiling in known-issues #19. 2x2 binning brings it to 12.1 MB. + +The frame rate is then capped by `DeviceLinkThroughputLimit`, but the limit is +accounted against the **unbinned** payload, not the binned frame that crosses +the wire. Measured on tower 121: + +| `DeviceLinkThroughputLimit` | max `AcquisitionFrameRate` | actual USB traffic | +|---|---|---| +| 250 MB/s (the fleet's value) | 4.95 fps | ~60 MB/s | +| 450 MB/s (this camera's max) | 8.79 fps | ~106 MB/s | + +48.6 MB x 4.95 = 240 MB/s, which is the 250 limit - so the camera is budgeting +for data that binning discards. **The recipe ships the conservative 250 MB/s**, +giving ~5 fps rather than the Delta's 10, because the USB stalls behind +known-issues #19 are a host-side finding that has not been re-measured on the +N150 yet (porting plan phase 5). Raising it is a one-line change in the recipe +once that measurement exists. + +> **Export `standard` after `night`, not the other way round, or use the current +> script.** A refused write is not a no-op: the first version of this script +> asked for 10 fps, the camera refused, and the 1 fps left over from the night +> profile stayed - producing a "standard" profile that ran at night speed. The +> script now clamps an out-of-range numeric value to the camera's limit and says +> `~ ... (clamped)`. + +## 9. `config.ini` - what this tower DOES `[partial]` + +Device-local, never overwritten by a deploy. Edit it on the tower: + +```ini +[General] +tower_name = ; drives every MQTT topic +[Network] +zkms_server_ip = +``` + +Credentials belong in `/etc/fire_gimbal_control/credentials.env`, not here. + +Anything `hardware.ini` now owns - `[Serial] device`, `[IMU]`, `[Env]`, the +camera ids - should be **deleted** from `config.ini` rather than left to rot: +the program warns at startup about every key set in both. + +`[Features] enable_imu` and `enable_env` matter more than they look. Declaring +`imu = none` in `hardware.ini` does **not** yet switch them off - deriving them +is section 12 of the config plan and is not implemented. On tower 121 the first +clean-looking run did this: + +``` +[INFO ] MTi IMU started on /dev/ttyS0 @ 115200 +[ERROR] I2C: failed to open /dev/i2c-2 +[ERROR] SHT41: failed to open I2C bus; sensor disabled +``` + +It "started" an IMU on a tower that has none - `/dev/ttyS0` exists and accepts a +serial open whether or not anything is attached to it. Set both flags to `false` +by hand until reconciliation lands. + +## 10. First run: bring the subsystems up one at a time `[partial]` + +Do **not** start a new tower with everything switched on. Each phase below adds +exactly one subsystem, so that a failure names its own cause instead of arriving +as a wall of unrelated errors. It also keeps the motors - the only part that can +break itself - until you have proved the boring parts work. + +Each phase says what good looks like and what noise to expect. + +### Phase 0 - the host alone, no hardware `[verified]` + +```bash +./build/fire_gimbal_control --mock-serial --mock-camera --no-mqtt --no-tui +``` + +Proves config resolution, `hardware.ini` layering, the single-instance lock, the +scan grid and the output directory - with nothing physical involved. Neither a +camera nor its profile XMLs need exist: + +``` +[INFO ] camera profiles not loaded: no real camera in this run +``` + +`[Features] enable_camera = false` does the same thing from the config file, for +a tower that has no camera fitted yet, and says so rather than pretending: + +``` +[WARN ] camera disabled ([Features] enable_camera = false); using a mock source + - no images come from hardware in this run +``` + +``` +[INFO ] hardware: /etc/fire_gimbal_control/hardware.ini board lattepanda-iota axes yaw +[INFO ] Loaded config: config.ini +[INFO ] scan grid: 15 waypoints +``` + +Any `config.ini still sets X, which hardware.ini overrides` warning belongs here +and should be cleared before moving on (step 9). + +### Phase 1 - motors `[partial]` + +Real serial link, nothing else: + +```bash +./build/fire_gimbal_control --mock-camera --no-mqtt --no-tui +``` + +What proves the link is a **live `DUMP` from the board**, not just an open port: + +``` +[INFO ] Serial controller started on /dev/serial/by-id/usb-Raspberry_Pi_Pico_... +DUMP BEGIN build=aab1925-dirty uptime=7925212 mcusr=0x3 free_ram=252712 +DUMP Y state=1 ... speed=3000 center=500000 has_encoder=1 +``` + +Check `build=` matches the firmware you flashed and `speed=` matches the +drivetrain values you intended. Then, and only then, move the thing: + +``` +gimbal home # the endstop-finding sequence - see the warning below +gimbal diag y # firmware self-test, reports PASS/FAIL +gimbal move 10,0 # a small deliberate move +``` + +**Expected noise in this phase, neither of which is a fault:** + +| Message | Why | +|---|---| +| `Motor controller: ERR axis not present` | The host configures both axes at startup; a yaw-only firmware refuses the pitch half. Harmless until presence reconciliation lands (config plan section 10) | +| `encoder tracking error on yaw: 500366 counts (limit 400)` | Before homing, the firmware presets `XACTUAL` to the centre count while the encoder sits at its own zero, so they disagree by the whole range. Homing resyncs them. **If it persists after a successful home, that is real** | + +> **On an unfamiliar drivetrain, homing is the dangerous step** - it finds the +> endstop by driving into it. Start from a deliberately slow `homing_speed` and +> walk it up, watching the free-running encoder gap that `ENC_DEVIATION_LIMIT` is +> measured against. Be at the rig with a hand on the power for the first one. + +### Phase 2 - camera `[verified]` + +Add the camera, still no broker: + +```bash +./build/fire_gimbal_control --no-mqtt --no-tui +``` + +``` +[INFO ] Opened camera 'rgb' DEV_1AB22C0B7D4E (1800 U-1620c) +[INFO ] camera profile 'standard' applied from .../standard.xml (17206 settings) +``` + +Then `start`, and confirm `.jxl` files appear under the configured output dir. + +> **A camera-free phase needs no profile XMLs** - but this only became true on +> 2026-09-16. Before that, `[Features] enable_camera` was parsed and never used, +> and the profile directory was validated even for a mock camera, so a tower +> whose camera had not been configured yet could not start at all. If you are +> running an older binary, export the profiles (step 8) first or it will refuse +> with `no camera profiles (*.xml)`. + +### Phase 3 - the broker + +Set `[Features] enable_mqtt = true` and point `zkms_server_ip` at the broker. +Without one, the app says so and carries on: + +``` +[ERROR] MQTT connect failed (tcp://10.11.12.121:1883): TCP/TLS connect failure +[WARN ] Control channel not connected; continuing in degraded mode +``` + +**Degraded mode is not a failure to notice later** - the tower captures images +and reports to nobody. Confirm topics arrive at the broker before calling the +phase done. + +Each tower runs its own RabbitMQ, in Docker, with the `rabbitmq_mqtt` and +`rabbitmq_shovel` plugins - the shovel is what forwards this tower's messages to +the ground station. That lives in `~/zkms-client/`, outside this repository. + +### Phase 4 - IMU and ambient sensor + +Only if fitted. `hardware.ini` should already say so, and `[Features] +enable_imu` / `enable_env` have to agree with it (see step 9 - declaring +`imu = none` does not yet switch the flag off by itself). + +### Phase 5 - unattended + +Only once every phase above is green: + +```bash +scripts/fgc start --start # detached, inside tmux +scripts/fgc status # running? since when? which PID? +``` + +Close the ssh session, reconnect, and check it is still running. Then reboot the +tower and confirm the USB preconditions survived (`install.sh --probe`) and the +app comes back. + +### Calibration + +Per tower, and **cannot be inherited** - least of all from a tower with a +different gear ratio, where `yaw_counts_per_deg` is derived from a +`steps_per_rev` that no longer applies: + +``` +gimbal heading setup # ties yaw to true north by sighting landmarks; no IMU needed +gimbal pitch setup # needs an IMU; skip on a rig without one +``` + +## 11. Verification checklist + +| Check | How | +|---|---| +| Key-only ssh | `ssh ggs@ hostname`, no prompt | +| Preconditions survive a reboot | reboot, then `install.sh --probe`: `usbfs_memory_mb 1000`, `lpm_permit 0` | +| Camera enumerates | `/opt/VimbaX/bin/ListCameras` reports the `DEV_` id in `hardware.ini` | +| Declared hardware matches | startup logs no presence warnings | +| Firmware identity | `gimbal dump` reports the `GIT_REV` you flashed | +| Homing repeatable | `gimbal home` several times, endstop counts within `home_spread_max_counts` | +| Images land | `.jxl` files appear under the configured output dir | +| Survives a disconnect | `scripts/fgc start`, close ssh, reconnect, `scripts/fgc status` |