# Installing a tower from scratch How a new fire-watch tower host goes from a freshly imaged machine to a running, calibrated `fire_gimbal_control`. Written while doing exactly that on the IOTA tower at `10.11.12.121` (2026-09-15) and kept honest about which steps have actually been exercised and which are still on paper. > [deployment.md](deployment.md) is the *update* loop for a tower that already > works. This is the *first* install, including the parts that only happen once: > the board's preconditions, the Vimba X SDK, `hardware.ini`, and the motor > firmware. > **`scripts/install-tower` now does the steps below**, on the tower, in this > order: it surveys the machine, asks for what it cannot work out (every > question can also be answered with `--answer KEY=VALUE`), shows the plan, and > applies it after one yes. `--dry-run` shows the plan and changes nothing; > `--verify` runs the checks only. It never invents a drivetrain value: answer > `unknown` and it writes a `; TODO`, and refuses to flash firmware an axis of > which would then sit in `NEEDCFG`. Homing is offered only at the end, and only > to someone who says they are at the rig. > > `[verified]` as a re-run on the already-configured tower 121 (2026-09-23): it > changed only what was out of date and passed `test hardware`. `[untested]` on a > freshly imaged board - the fresh path has been rehearsed with `--dry-run` only. > What follows stays the reference for what each step does and why. **Status legend** — each step says where it stands: `[verified]` run end to end on a real tower and the result checked · `[partial]` run, with something still open · `[untested]` written from the design, not yet exercised. --- ## What you need before starting | | | |---|---| | The tower host | Powered, on the network, reachable by IP, with an OS installed and an `ssh` server running | | An account on it | `ggs`, in the `sudo` and `dialout` groups | | A dev box | This repository checked out, plus a copy of the Vimba X SDK (login-walled, so it cannot be fetched by a script) | | The hardware | Camera plugged in, motor MCU plugged in, and a note of **what was assembled**: which axes, which IMU, which ambient sensor | Two facts decide most of what follows, so establish them first: ```bash ssh ggs@ 'cat /sys/class/dmi/id/product_name; . /etc/os-release && echo "$PRETTY_NAME"' ``` | Board | MCU | Setup kit | PlatformIO env | |---|---|---|---| | LattePanda 3 Delta / Sigma | ATmega32U4 (Leonardo), 5 V | *not written - the Delta fleet predates the kits* | `leonardo` | | LattePanda IOTA | RP2040, 3.3 V | [`hardware/lattepanda-iota/`](../hardware/lattepanda-iota/) | `iota_rp2040` | --- ## 1. SSH by key `[verified]` Everything after this is driven over ssh from the dev box, and `deploy.sh` runs several ssh commands per invocation, so password auth is not workable. ```bash # on the dev box ssh-copy-id -i ~/.ssh/id_ed25519_ggs_pedro.pub ggs@ ``` Then give the host a block in `~/.ssh/config`, so every tool picks the right key without being told: ``` Host 10.11.12.121 User ggs IdentityFile ~/.ssh/id_ed25519_ggs_pedro ``` Verify with `ssh ggs@ hostname` - it must not prompt. > **`ssh-copy-id` needs a terminal.** Run it in a real shell, not through a tool > that gives it no tty: without one, ssh looks for a GUI `ssh-askpass`, and on a > machine that has none it fails with `ssh_askpass: exec(...): No such file or > directory` and then `Permission denied` - which looks like a rejected key > rather than a prompt that never appeared. ## 2. Survey the machine before changing it `[verified]` The setup kit can describe the machine without touching it, and this is the cheapest moment to notice that the tower is not what you assumed: ```bash rsync -a hardware/ ggs@:~/projects/fwt_2a/software/hardware/ ssh ggs@ '~/projects/fwt_2a/software/hardware/lattepanda-iota/install.sh --probe' ``` ``` board : lattepanda-iota os : Ubuntu 26.04.1 LTS (7.0.0-31-generic) motor MCU : /dev/serial/by-id/usb-MicroPython_Board_in_FS_mode_df61a863dfa02932-if00 camera DEV_ id : NOT FOUND (needs the Vimba X SDK installed) alvium on usb : 1 device(s) usbfs_memory_mb : 16 (want >= 1000) camera lpm_permit: u1_u2 (want 0) ``` Read every line. On tower 121 this pass found that the MCU was still running the MicroPython image it shipped with, and that both USB preconditions were at their (wrong) defaults. **Check what is already in `~/projects/`.** A machine cloned from another tower, or one that has been used for development, can arrive with a `config.ini` that belongs to a different rig. Tower 121 had a byte-identical copy of the dev box's `config.ini` - `tower_name = dev`, broker `127.0.0.1` - which is worse than no config at all, because **none of those values trip the placeholder guard in `deploy.sh --run`**. That guard only catches `ExampleTower`, `CHANGE_ME` and `DEV_0000000000`. ```bash ssh ggs@ 'grep -E "tower_name|zkms_server_ip|^id" ~/projects/fwt_2a/software/config.ini' ``` ## 3. Host preconditions and build dependencies `[verified]` Both come from the board's setup kit, in one step, on the device: ```bash ssh ggs@ cd ~/projects/fwt_2a/software/hardware/lattepanda-iota sudo ./install.sh --deps --preconditions ``` `--deps` installs [`packages.txt`](../hardware/lattepanda-iota/packages.txt): the toolchain, OpenCV, Boost, libjxl, tmux, and the Python bits PlatformIO needs. `--preconditions` installs two files and applies them immediately: | File | Why | |---|---| | `/etc/udev/rules.d/99-fwt-alvium-usb.rules` | Disables USB3 hardware LPM on the camera's port, which the host otherwise re-enables on every enumeration. With U1/U2 on, the camera logs link errors and frame delivery stalls (known-issues #19) | | `/etc/tmpfiles.d/10-fwt-usbfs.conf` | Raises `usbfs_memory_mb` from 16 to 1000 at every boot. A single full-resolution frame does not fit in 16 MB | Both are settings that **revert on reboot** if nobody persists them, which is the whole of known-issue #19. The script re-reads them after applying and says so: ``` >> usbfs_memory_mb = 1000 >> camera USB3 LPM disabled (2-1) ``` If it instead warns that a value did not take, stop and find out why - a camera on a host with these wrong looks like a flaky camera. Confirmed on tower 121 (2026-09-15): both went from wrong to right in one run, `usbfs_memory_mb` 16 -> 1000 and `lpm_permit` `u1_u2` -> `0`. Re-check after the first reboot, since surviving one is the entire point. > **Neither step can be driven from a tool without a terminal.** `sudo` needs a > tty to prompt, so this runs from a real terminal window: > > ```bash > ssh -t ggs@ 'cd ~/projects/fwt_2a/software/hardware/lattepanda-iota && sudo ./install.sh --deps --preconditions' > ``` > **The Delta fleet does not have these applied.** Checked on `10.11.12.111` > 2026-09-15: `usbfs_memory_mb` is 16 and the kernel cmdline is stock. The > preconditions were documented in known-issues but never persisted anywhere. ## 4. Vimba X SDK `[verified]` Proprietary and login-walled, so it is staged from a machine that already has it rather than downloaded on the tower: ```bash # dev box rsync -az ~/Programs/VimbaX_2025-3 ggs@:~/ # tower sudo ./install.sh --vimba ~ggs/VimbaX_2025-3 ``` That copies it to `/opt/VimbaX_`, points `/opt/VimbaX` at it, runs the SDK's `Install_GenTL_Path.sh` (which registers `GENICAM_GENTL64_PATH` in `/etc/profile.d` and writes the `1ab2` udev rule), registers `api/lib` with the dynamic loader, and then asks the SDK to list cameras as a check. **The SDK asks for a reboot before the transport layers are usable.** If it finds nothing, reboot and try again before suspecting the camera. On tower 121 no reboot was needed - the camera answered immediately. Two things about that check, both found the hard way here: - **The binary is `ListCameras_VmbC`**, in `/opt/VimbaX/bin/`. There is no plain `ListCameras`. - **Run it with `GENICAM_GENTL64_PATH` set**, or it fails with `Could not start system. Error code: -16` on a perfectly good installation - that is "no transport layer found", not "no camera": ```bash GENICAM_GENTL64_PATH=/opt/VimbaX/cti /opt/VimbaX/bin/ListCameras_VmbC ``` > **The SDK ships a camera simulator, and it is on by default.** On tower 121 > `ListCameras_VmbC` reported **four** cameras: the real Alvium plus three > synthetic ones from `VimbaCameraSimulatorTL.cti`, with ids like `DEV_Cam1`. A > tower configured with one of those would open its "camera" happily and capture > synthetic images forever. Take the id only from a block whose transport layer > path is `VimbaUSBTL` or `VimbaGigETL` - which is what `install.sh --probe` > does, after this bit me. > Non-interactive ssh shells do not source `/etc/profile.d`, which is why > `deploy.sh` resolves `GENICAM_GENTL64_PATH` itself before launching the app. > Getting this wrong shows up as `VmbStartup()` failing with *"Could not start > Vimba X API"* even though the viewer works when you are logged in. ## 5. `hardware.ini` - what this machine IS `[verified]` ```bash sudo ./install.sh --hardware-ini --axes yaw --imu none --env none --cameras rgb ``` This writes `/etc/fire_gimbal_control/hardware.ini`, **outside** the directory `deploy.sh` rsyncs, so no deploy can overwrite or delete it. The probed half - board model, firmware env, the MCU's stable `by-id` path, the camera's `DEV_` id - the script fills in. The declared half is the flags, and they are not a formality: the program degrades gracefully around missing hardware, so **a part that was never fitted and a part that has failed look identical** unless the file says which was expected. Declare what was assembled. Check the result, especially the serial path: ```bash ssh ggs@ 'cat /etc/fire_gimbal_control/hardware.ini' ``` Then prove the program finds it **with nothing in the environment** - no `FGC_HARDWARE`, no `--hardware` - because that is how the tower starts itself: ``` [INFO ] hardware: /etc/fire_gimbal_control/hardware.ini board lattepanda-iota axes yaw ``` > **Do not leave it anywhere under `/tmp`.** That is tempting during bring-up > because it needs no root, and it works right up until the first reboot, which > clears `/tmp` and leaves the program with `Config error: Camera.rgb.id is > required` - a message that points at the camera rather than at the file that > disappeared. Verified on tower 121 by rebooting it. ### The drivetrain block **`install.sh` now writes this block**, one `[Motor.]` per declared axis — it has to, because the firmware refuses to move until it has been told these (see the config handshake in [configuration.md](configuration.md)). What it still cannot do is *probe* them: no reading of a board tells you its sense resistor. So pass what you know and it writes the values; omit anything and it writes a commented `; TODO` line naming what to measure: ```bash ./install.sh --stdout --axes yaw \ --r-sense 0.075 --run-current 756 --homing-current 200 \ --travel-deg 340 --steps-per-rev 1210778 --sw-mode 0 ``` A `TODO` line is **commented, not a placeholder value**, and deliberately: these keys are numeric, and a non-numeric value makes the program refuse to start with a message about a config key. Left absent, it starts and names exactly which keys that axis is missing and that it will not move — the useful failure. The block it writes looks like this: ```ini [Motor.yaw] r_sense_ohm = 0.075 ; the resistor fitted to THIS driver board run_current_ma = 756 ; normal motion homing_current_ma = 200 ; the endstop search only travel_deg = 340 ; mechanical angle between the hard stops ``` **`homing_current_ma` is a safety setting, not a tuning one.** Homing finds a stop by driving into it until the shaft falls behind; that only works while the stop is stronger than the motor. A high gear ratio can shear a screw-head stop straight past its mate - on tower 121 the axis ran 1.8 turns past the stop with the encoder never falling behind, at full torque. Declare a reduced search current wherever the stop cannot survive one. Getting it wrong in either direction is visible: too strong and the stop yields again; too weak and the motor stalls on ordinary friction mid-sweep, trips the same latch, and records a phantom endstop. The firmware's `steps_per_rev` guard is what catches the second case, so it has to be set from the real travel. > **The centre lives in `config.ini`, not here.** `install.sh` cannot write it — > `hardware.ini` is merged *over* `config.ini` and wins, so a shared key would > silently discard whatever `gimbal heading setup` had saved. What it does instead > is work the requirement out from `steps_per_rev` and print it as a `NOTE` in the > block, because the default is wrong for any geared-up rig: > > ``` > ; NOTE: half this travel is 605389 counts, so config.ini needs > ; [Motor] yaw_center_count = 1000000 (or anything above 605389) > ; The 500000 default would put the negative end at -105389. > ``` > > The range is centre ± half the travel, so a centre below half-travel puts the > negative end under zero. On tower 121 the inherited `500000` homed to > `-104749..1104749`; at `1000000` it homes to `395194..1604807`. A warning names > it after every home if it is still too small. **`travel_deg` is a nameplate fact**, and the host divides the measured homing travel by it to derive `[Motor] _counts_per_deg` into `config.ini`. A rough angle gives a proportionally rough degree scale - 1 % out here is 1 % out on every bearing - until `gimbal heading setup` supersedes it with a landmark fit. The startup log reports what the registers **actually** deliver, which is not always what was asked for: ``` yaw motor current: run 756 mA (GLOBALSCALER 80, IRUN 31), homing 189 mA (IRUN 7) -- R_sense 0.075 ohm from hardware.ini [Motor.yaw] ``` Omit the block entirely on a tower whose endstops hold against full torque: no `CURRENT` is sent, the firmware keeps its compile-time defaults, and nothing changes. > **If an axis declares a reduced homing current and the firmware has not adopted > it, the host refuses to home** rather than driving into a stop at full torque. > The message names the skew; the fix is to flash the firmware, not to override > it. > `[Serial] device` must be a `/dev/serial/by-id/` path, never `/dev/ttyACM0`. > Tower 121 has a 5G modem that claims three `ttyUSB` nodes, and enumeration > order is not stable across reboots. > > **The by-id path changes when you flash the MCU**, because the name comes from > the running image: `usb-MicroPython_Board_in_FS_mode_*` before, a Pico/Arduino > name after. Re-run `--probe` after step 6 and fix the file if it moved. ## 6. Motor firmware `[verified]` The firmware is built and flashed **on the tower** - it is the machine wired to the MCU - so PlatformIO is installed there, not on the dev box. ```bash # tower: PlatformIO core, in its own venv under ~/.platformio. No sudo needed. curl -fsSL https://raw.githubusercontent.com/platformio/platformio-core-installer/master/get-platformio.py -o /tmp/get-platformio.py python3 /tmp/get-platformio.py ~/.platformio/penv/bin/pio --version # check it actually runs ``` `[verified]` on tower 121: PlatformIO Core 6.2.0 on the system Python 3.14.4. > The installer ends with a `FileNotFoundError` traceback out of > `multiprocessing/util.py` about a missing `/tmp/.piocore-installer-*` path. > It is a temp-directory cleanup finalizer running after the work is done - > cosmetic. Judge the install by whether `pio --version` answers. `deploy.sh` looks for `pio` in the usual places, but a non-interactive ssh shell does not have `~/.platformio/penv/bin` on PATH, so name it explicitly: ```ini REMOTE_PIO=/home/ggs/.platformio/penv/bin/pio ``` Then, from the dev box, in `firmware/`: ```ini # .deploy.env REMOTE_HOST=ggs@10.11.12.121 REMOTE_DIR=/home/ggs/projects/fwt_2a/firmware PIO_ENV=iota_rp2040 # leonardo on a Delta SERIAL_PORT=/dev/serial/by-id/usb-... ``` ```bash ./deploy.sh --compile-only # build on the tower, flash nothing ./deploy.sh # build + upload ./deploy.sh --monitor # watch it boot ``` **Confirm the pin map before flashing a board that is wired to motors.** A chip-select on the wrong pin writes into a driver that was never meant to hear it. The pins are the `[Pins]` section of this tower's `hardware.ini`: the build reads them from `/etc/fire_gimbal_control/hardware.ini` (`firmware/tools/gen_wiring.py`) and **fails without them**, so add the section before the first build. The reference loom, with the table saying which wire goes where, is [`hardware/lattepanda-iota/pins.ini`](../hardware/lattepanda-iota/pins.ini) - the loom as built on tower 121, which is **not** the map proposed in the porting plan. Copy it in if the tower is wired the same way. The generator refuses pins SPI0 cannot reach, and `test hardware motor` compares the declared pins with the ones the running firmware reports it was built with. ### Getting a factory board to accept its first flash `[verified]` on tower 121, and this is the step that will bite you. These boards ship running MicroPython, and **the normal upload does not work against it**: ``` Forcing reset using 1200bps open/close on port /dev/ttyACM0 No new RPxxxx device found yet, waiting.. (x20, then it gives up) ``` | What | Factory board (MicroPython) | Our firmware | |---|---|---| | 1200-baud touch, i.e. a plain upload | no reset at all | resets, but **reboots into the application** | | `picotool reboot -f -u` | refused: no reset interface | - | | MicroPython REPL | **works** | n/a | So drop into MicroPython's own REPL and tell it to enter the bootloader. `firmware/tools/flash.sh` (which `deploy.sh` runs on the tower) now does this by itself when the port's by-id name says `MicroPython_Board`: `./deploy.sh` with `SERIAL_PORT` set to that path should flash a factory board directly. `[untested]` - no factory board has been through that path yet; the manual way, verified on tower 121, is below. Write this to a file and run it with `~/.platformio/penv/bin/python` (it has pyserial): ```python import serial, time s = serial.Serial("/dev/serial/by-id/usb-MicroPython_Board_in_FS_mode_XXXX-if00", 115200, timeout=1) s.write(b"\x03\x03"); time.sleep(0.3) # Ctrl-C: interrupt whatever is running s.write(b"import machine\r\n"); time.sleep(0.3) s.write(b"machine.bootloader()\r\n") # the port drops here - that is success ``` ```bash lsusb | grep 2e8a # expect: 2e8a:0003 Raspberry Pi RP2 Boot ``` Then run `./deploy.sh` and it flashes. A physical BOOTSEL press does the same job if you are standing at the board. **Every board needs exactly one physical BOOTSEL press in its life** - the one that installs the first image carrying the `BOOTSEL` command (below). After that, updates are remote. **`sudo ./install.sh --preconditions` must have run first**, or picotool finds the board and refuses it: ``` RP2040 device at bus 1, address 4 appears to be in BOOTSEL mode, but picotool was unable to connect. Maybe try 'sudo' or check your permissions. ``` ### Re-flashing later: the `BOOTSEL` command `[verified]` on tower 121. Neither documented remote path works on this board - the 1200-baud touch resets the firmware but returns it to the application, and `picotool reboot -f -u` answers `Unable to locate reset interface on the device`. So the firmware carries a command that does it deliberately: ```bash # over the serial port, or from the app: gimbal raw "BOOTSEL" BOOTSEL -> OK entering bootloader - reflash to return lsusb | grep 2e8a # 2e8a:0003 Raspberry Pi RP2 Boot ./deploy.sh # flashes and reboots into the new firmware ``` The whole loop was run end to end from the dev box with nobody at the tower. `BOOTSEL` disables every axis first, since the bootrom does not run the control loop and unsupervised energised drivers would hold torque. > **It takes the gimbal out of service and the firmware cannot undo it** - once > in the bootrom there is no command loop left to change its mind. Recovery is > `deploy.sh`, or a power cycle: entering the bootrom erases nothing, so the > existing image is still in flash. Anything that can write to the serial port > can send it, including `gimbal raw`. Near misses (`BOOT`, `BOOTSE`) are > rejected as unknown commands. ### The identity changes when you flash | | by-id path | USB | |---|---|---| | before | `usb-MicroPython_Board_in_FS_mode_-if00` | `2e8a:0005` | | after | `usb-Raspberry_Pi_Pico_-if00` | `2e8a:000a` | So generate `hardware.ini` (step 5) **after** flashing, or re-run `install.sh --hardware-ini --force` to pick up the new path. ### Verify the flash Read the boot banner over the serial port: ``` BOOT build=aab1925-dirty IOIN Y=0x3000001E version=0x30 ST Y:R,500000,18,80084000,0,8,8,Sd ``` `version=0x30` is the TMC5160 answering over SPI - the single most useful line here, because it proves the pin map, the 3.3 V logic level and the driver board all at once. `BOOT build=` must match the commit you flashed. One `Y` line and no `P` line is a yaw-only build. > **Flashing an RP2040 remotely is not the same as a Leonardo.** The documented > flow is *hold BOOTSEL, tap RST, drag a `.uf2`* - a person at the board. The > two remote paths (the 1200-baud touch that the Arduino tooling already uses, > and `picotool reboot -f -u`) both need the **running** image to present a USB > CDC port. That is why `setup()` brings `Serial` up before any hardware init: > an image that hangs before USB comes up cannot be reflashed without a site > visit, while an interrupted flash recovers by itself - the bootrom falls back > to mass storage when flash holds no valid image. ## 7. Deploy the software `[verified]` On the dev box, in `software/`: ```ini # .deploy.env REMOTE_HOST=ggs@10.11.12.121 REMOTE_DIR=/home/ggs/projects/fwt_2a/software CMAKE_ARGS="-DWITH_MQTT=ON -DWITH_VIMBA=ON" RUN_ARGS="--start" ``` ```bash ./deploy.sh --check-deps # must print "remote deps OK" ./deploy.sh # rsync + remote cmake + build ``` The first deploy seeds `config.ini` from `config/config.example.ini` **only if the device has none** - see the warning in step 2 about a machine that arrived with somebody else's config. `[verified]` on tower 121: builds clean on GCC 15 / Ubuntu 26.04 with a single `-Wformat-truncation` warning in `utcStampNow()`, which is a false positive (the compiler cannot prove `tm_year` is four digits; the real output is 17 bytes into a 32-byte buffer). ## 8. Camera profiles `[verified]` **Without a profile the program does not start** - `no camera profiles (*.xml) in config/camera_profiles/rgb` is a startup error, by design. A fresh tower has an empty profile directory, so this step is not optional. The documented way to make one is the VimbaX Viewer, which is a GUI - awkward on a headless tower and impossible unattended. `scripts/export_camera_profile.py` does the same job through the SDK's Python API: ```bash # once per tower: the SDK ships the wheel python3 -m venv ~/.venvs/vmbpy ~/.venvs/vmbpy/bin/pip install /opt/VimbaX/api/python/vmbpy-*.whl cd ~/projects/fwt_2a/software export GENICAM_GENTL64_PATH=/opt/VimbaX/cti ~/.venvs/vmbpy/bin/python scripts/export_camera_profile.py --profile standard --dry-run ~/.venvs/vmbpy/bin/python scripts/export_camera_profile.py --profile standard ~/.venvs/vmbpy/bin/python scripts/export_camera_profile.py --profile night ``` It writes each recipe value to the real camera **in dependency order**, lets the camera validate it, and exports what was actually accepted. That preserves the property the Viewer workflow exists to provide: GenICam validity is stateful, so a hand-written XML can be individually plausible and collectively wrong. Here the camera is the judge, and anything it refused is printed rather than assumed. Read the summary line it prints - it is the number that decides whether the profile can be delivered at all: ``` result : 2664x1520 RGB8 12.1 MB/frame 4.95 fps = 60 MB/s ``` ### Profiles do not transfer between cameras `[verified]` the hard way. The fleet runs several Alvium generations and their feature vocabularies differ, so copying another tower's XML does not work: | | Delta tower (1800 U-1240c) | IOTA tower (1800 U-1620c) | |---|---|---| | `SensorBitDepth` | `Bpp8` | **no `Bpp8`** - only `Adaptive` / `Bpp12` | | `IntensityAutoPrecedence` | `Minimizeexposuretime` / `Minimizenoise` | **`MinimizeBlur` / `MinimizeNoise`** | | Sensor | 12 MP | 16 MP, 5328x3040 | ### Binning is not optional, and the frame-rate ceiling is not what it looks like Unbinned RGB8 on the 16 MP sensor is a **48.6 MB frame** - over the ~20 MB ceiling in known-issues #19. 2x2 binning brings it to 12.1 MB. The frame rate is then capped by `DeviceLinkThroughputLimit`, but the limit is accounted against the **unbinned** payload, not the binned frame that crosses the wire. Measured on tower 121: | `DeviceLinkThroughputLimit` | max `AcquisitionFrameRate` | actual USB traffic | |---|---|---| | 250 MB/s (the fleet's value) | 4.95 fps | ~60 MB/s | | 450 MB/s (this camera's max) | 8.79 fps | ~106 MB/s | 48.6 MB x 4.95 = 240 MB/s, which is the 250 limit - so the camera is budgeting for data that binning discards. **The recipe ships the conservative 250 MB/s**, giving ~5 fps rather than the Delta's 10, because the USB stalls behind known-issues #19 are a host-side finding that has not been re-measured on the N150 yet (porting plan phase 5). Raising it is a one-line change in the recipe once that measurement exists. > **Export `standard` after `night`, not the other way round, or use the current > script.** A refused write is not a no-op: the first version of this script > asked for 10 fps, the camera refused, and the 1 fps left over from the night > profile stayed - producing a "standard" profile that ran at night speed. The > script now clamps an out-of-range numeric value to the camera's limit and says > `~ ... (clamped)`. ## 9. `config.ini` - what this tower DOES `[partial]` Device-local, never overwritten by a deploy. Edit it on the tower: ```ini [General] tower_name = ; drives every MQTT topic [Network] zkms_server_ip = ``` Credentials belong in `/etc/fire_gimbal_control/credentials.env`, not here. Anything `hardware.ini` now owns - `[Serial] device`, `[IMU]`, `[Env]`, the camera ids - should be **deleted** from `config.ini` rather than left to rot: the program warns at startup about every key set in both. > **`[Camera.] id` is required even when the camera is disabled.** Config > validation runs before any of the feature flags are consulted, so a tower with > no camera fitted yet cannot start without one - and the obvious dummy, > `DEV_0000000000`, is what `deploy.sh`'s placeholder guard refuses. On a tower > that has its camera, the id lives in `hardware.ini` and this never comes up. `[Features] enable_imu` and `enable_env` matter more than they look. Declaring `imu = none` in `hardware.ini` does **not** yet switch them off - deriving them is section 12 of the config plan and is not implemented. On tower 121 the first clean-looking run did this: ``` [INFO ] MTi IMU started on /dev/ttyS0 @ 115200 [ERROR] I2C: failed to open /dev/i2c-2 [ERROR] SHT41: failed to open I2C bus; sensor disabled ``` It "started" an IMU on a tower that has none - `/dev/ttyS0` exists and accepts a serial open whether or not anything is attached to it. Set both flags to `false` by hand until reconciliation lands. ## 10. First run: bring the subsystems up one at a time `[partial]` Do **not** start a new tower with everything switched on. Each phase below adds exactly one subsystem, so that a failure names its own cause instead of arriving as a wall of unrelated errors. It also keeps the motors - the only part that can break itself - until you have proved the boring parts work. Each phase says what good looks like and what noise to expect. ### Phase 0 - the host alone, no hardware `[verified]` ```bash ./build/fire_gimbal_control --mock-serial --mock-camera --no-mqtt --no-tui ``` Proves config resolution, `hardware.ini` layering, the single-instance lock, the scan grid and the output directory - with nothing physical involved. Neither a camera nor its profile XMLs need exist: ``` [INFO ] camera profiles not loaded: no real camera in this run ``` `[Features] enable_camera = false` does the same thing from the config file, for a tower that has no camera fitted yet, and says so rather than pretending: ``` [WARN ] camera disabled ([Features] enable_camera = false); using a mock source - no images come from hardware in this run ``` ``` [INFO ] hardware: /etc/fire_gimbal_control/hardware.ini board lattepanda-iota axes yaw [INFO ] Loaded config: config.ini [INFO ] scan grid: 15 waypoints ``` Any `config.ini still sets X, which hardware.ini overrides` warning belongs here and should be cleared before moving on (step 9). ### Phase 1 - motors `[partial]` Real serial link, nothing else: ```bash ./build/fire_gimbal_control --mock-camera --no-mqtt --no-tui ``` What proves the link is a **live `DUMP` from the board**, not just an open port: ``` [INFO ] Serial controller started on /dev/serial/by-id/usb-Raspberry_Pi_Pico_... DUMP BEGIN build=aab1925-dirty uptime=7925212 mcusr=0x3 free_ram=252712 DUMP Y state=1 ... speed=3000 center=500000 has_encoder=1 ``` Check `build=` matches the firmware you flashed and `speed=` matches the drivetrain values you intended. Then, and only then, move the thing: ``` gimbal home # the endstop-finding sequence - see the warning below gimbal diag y # firmware self-test, reports PASS/FAIL gimbal move 10,0 # a small deliberate move ``` **Expected noise in this phase, neither of which is a fault:** | Message | Why | |---|---| | `Motor controller: ERR axis not present` | The host configures both axes at startup; a yaw-only firmware refuses the pitch half. Harmless until presence reconciliation lands (config plan section 10) | | `encoder tracking error on yaw: 500366 counts (limit 400)` | Before homing, the firmware presets `XACTUAL` to the centre count while the encoder sits at its own zero, so they disagree by the whole range. Homing resyncs them. **If it persists after a successful home, that is real** | > **On an unfamiliar drivetrain, homing is the dangerous step** - it finds the > endstop by driving into it. Start from a deliberately slow `homing_speed` and > walk it up, watching the free-running encoder gap that `ENC_DEVIATION_LIMIT` is > measured against. Be at the rig with a hand on the power for the first one. ### Phase 2 - camera `[verified]` Add the camera, still no broker: ```bash ./build/fire_gimbal_control --no-mqtt --no-tui ``` ``` [INFO ] Opened camera 'rgb' DEV_1AB22C0B7D4E (1800 U-1620c) [INFO ] camera profile 'standard' applied from .../standard.xml (17206 settings) ``` Then `start`, and confirm `.jxl` files appear under the configured output dir. > **A camera-free phase needs no profile XMLs** - but this only became true on > 2026-09-16. Before that, `[Features] enable_camera` was parsed and never used, > and the profile directory was validated even for a mock camera, so a tower > whose camera had not been configured yet could not start at all. If you are > running an older binary, export the profiles (step 8) first or it will refuse > with `no camera profiles (*.xml)`. ### Phase 3 - the broker Set `[Features] enable_mqtt = true` and point `zkms_server_ip` at the broker. Without one, the app says so and carries on: ``` [ERROR] MQTT connect failed (tcp://10.11.12.121:1883): TCP/TLS connect failure [WARN ] Control channel not connected; continuing in degraded mode ``` **Degraded mode is not a failure to notice later** - the tower captures images and reports to nobody. Confirm topics arrive at the broker before calling the phase done. Each tower runs its own RabbitMQ, in Docker, with the `rabbitmq_mqtt` and `rabbitmq_shovel` plugins - the shovel is what forwards this tower's messages to the ground station. That lives in `~/zkms-client/`, outside this repository. ### Phase 4 - IMU and ambient sensor Only if fitted. `hardware.ini` should already say so, and `[Features] enable_imu` / `enable_env` have to agree with it (see step 9 - declaring `imu = none` does not yet switch the flag off by itself). ### Phase 5 - unattended Only once every phase above is green: ```bash scripts/fgc start --start # detached, inside tmux scripts/fgc status # running? since when? which PID? ``` Close the ssh session, reconnect, and check it is still running. Then reboot the tower and confirm the USB preconditions survived (`install.sh --probe`) and the app comes back. ### Calibration Per tower, and **cannot be inherited** - least of all from a tower with a different gear ratio, where `yaw_counts_per_deg` is derived from a `steps_per_rev` that no longer applies: ``` gimbal heading setup # ties yaw to true north by sighting landmarks; no IMU needed gimbal pitch setup # needs an IMU; skip on a rig without one ``` ## 10b. Bringing up the config handshake on an existing tower This is the one upgrade that changes what an **unconfigured** tower does: the new firmware boots into `NEEDCFG` and refuses `HOME` / `MOVE` / `ENABLE` until the host has answered its `NEED` lines. Flashing it onto a tower whose `hardware.ini` predates the drivetrain keys leaves the gimbal inert — working, talking, and not turning. So the order matters. **Host first, firmware second.** Each step below either changes nothing or is reversible, and the step that could strand the tower comes after the one that prevents it. ### 0. Record what you are rolling back to ```bash gimbal dump # the running firmware's build id and register state cat /etc/fire_gimbal_control/hardware.ini git -C ~/projects/fwt_2a/firmware rev-parse --short HEAD ``` Keep the dump. It is the only record of what the old firmware was holding, and the comparison that tells you whether the new one adopted the same values. ### 1. Update `hardware.ini` — nothing else, and no restart yet Add the `[Motor.]` block (section 5). The five keys the new firmware will not run without are `steps_per_rev`, `enc_deviation`, `sw_mode`, plus the two speeds; `install.sh --stdout` generates the block if you would rather not type it. Doing this **before** flashing is what keeps the window at zero: the host is ready to answer the moment the firmware starts asking. ### 2. Deploy the new host against the OLD firmware A real test, and it risks nothing. The old firmware never sends `NEED`, so the host must behave exactly as it did before — which is also the guarantee the whole Delta fleet depends on. ```bash scripts/fgc start --start scripts/fgc peek test hardware motor ``` Expect: no handshake lines at all, and the motor leaf reporting a **version skew** rather than a mismatch — ``` ; yaw_current: yaw: firmware echoes only part of the current (2 of 3 fields absent), so it cannot be confirmed -- it predates the full CURRENT echo. Re-flash it ``` If instead you see wrong *values*, stop: something in `hardware.ini` disagrees with what the tower is really running, and flashing will not fix it. ### 3. Flash ```bash cd firmware && ./deploy.sh # uploads; --compile-only if you want a dry run first ``` Watch the banner. A `NEED` line per axis should follow it: ``` BOOT build= NEED Y steps_per_rev global_scaler run_irun homing_irun center_count soft_margin ... ``` **No `NEED` line means the axis is not in `NEEDCFG`** — either the flash did not take, or the image is older than you think. Check the build id before going on. ### 4. Confirm the handshake completed The host answers within a control-loop tick. Expect: ``` firmware config: answered 11 keys for axis Y ``` Then prove it landed rather than trusting the log: ``` gimbal dump # state should be 1 (RESET), NOT 5 (NEEDCFG) # cfg_rx should be the full required mask test hardware motor # both metrics PASS ``` `test hardware motor` is the real gate: it compares every declared value against what the firmware echoes, so a `SET` that was lost on the wire shows up here and nowhere else. ### 5. Prove the refusal actually refuses Worth doing once per firmware version, because a guard that silently stopped working looks exactly like a guard that is working. On a **bench** board, not a mast: power-cycle it without the host running, then send `HOME` over a serial monitor. Expect `ERR need config`. Send `RESET` and try again — still `ERR need config`, because a RESET is deliberately not a way out. `USE DEFAULTS` is, and should then let `HOME` through. ### 6. Only then, move the axis ``` gimbal home # at the reduced homing current ``` Check against the recorded figures: travel ~1,210,778 counts, repeatable to ~0.3 %, and the derived `counts_per_deg` near 3552.87. A travel that disagrees by more than a few percent means the axis homed against something that was not the endstop. ### Reading the log afterwards Everything below is recorded to `~/.local/share/fire_gimbal_control/logs/fgc_*.log`, not only to the dashboard — which matters here, because in TUI mode the pane holds 500 lines and dies with the tmux session. During this bring-up that cost us the handshake record four separate times before the file existed. ```bash L=$(ls -t ~/.local/share/fire_gimbal_control/logs/fgc_*.log | head -1) grep -E "NEED|firmware config|need config" "$L" # step 4: did the handshake complete grep -E "hardware:" "$L" # presence reconciliation grep -E "speed:|motor current" "$L" # what was actually sent to the firmware ``` The file is more verbose than the dashboard was, so it holds lines the pane never showed. `head -8` gives the build, tower and config that produced it. ### If it goes wrong The board is recoverable over the network as long as USB CDC comes up: `BOOTSEL` over the serial port, then `./deploy.sh` with the previous firmware checked out. Only an image that wedges *before* `Serial.begin()` needs a physical button press — which is why `Serial.begin()` is the first thing `setup()` does. `USE DEFAULTS` is the in-place escape: it puts every axis on the compile-time table and lets the tower move while you work out what `hardware.ini` is missing. ## 11. Verification checklist The checks that move nothing can be run as one command, which is what a script should use rather than typing into the dashboard. Stop the app first: it holds the serial port. ```bash scripts/fgc stop GENICAM_GENTL64_PATH=/opt/VimbaX/cti ./build/fire_gimbal_control --no-mqtt --test hardware echo $? # 0 PASS, 4 FAIL, 5 could not start (the log says why) scripts/fgc start ``` It waits for the firmware to be talking and configured before it looks, so a freshly flashed board is not failed for values the host had not sent yet. `build/fgc-config check --config config.ini --hardware /etc/fire_gimbal_control/hardware.ini` checks that the two files load together, and changes nothing. | Check | How | Tower 121 | |---|---|---| | Key-only ssh | `ssh ggs@ hostname`, no prompt | ✅ | | Preconditions survive a reboot | reboot, then `install.sh --probe`: `usbfs_memory_mb 1000`, `lpm_permit 0` | ✅ confirmed across a real reboot | | `hardware.ini` found unaided | run with no `FGC_HARDWARE`: startup names `/etc/fire_gimbal_control/hardware.ini` | ✅ | | Camera enumerates | `ListCameras_VmbC` reports the `DEV_` id in `hardware.ini` | ✅ `DEV_1AB22C0B7D4E` | | Camera profile applies | startup logs `profile 'standard' applied ... (N settings)` | ✅ 17206 settings | | Firmware identity | `gimbal dump` reports the `GIT_REV` you flashed | ✅ `build=aab1925-dirty` | | Driver answers over SPI | boot banner shows `IOIN Y=0x… version=0x30` | ✅ | | Remote re-flash | `BOOTSEL`, then `./deploy.sh` - no hands on the board | ✅ | | Declared hardware matches | startup logs no presence warnings | ✅ no shadowed-key warnings | | Homing repeatable | `gimbal home` several times, endstop counts within `home_spread_max_counts` | ✅ three homes within 0.27 % (1,210,778 / 1,208,358 / 1,207,514 counts) | | Endstop survives homing | inspect the stop after a home; the axis must stall against it, not push through | ✅ at 189 mA, after failing at full torque | | Degree scale derived | startup logs `counts_per_deg` agreeing with the homed travel | ✅ 3552.87, saved with `;prev` history | | Images land | `.jxl` files appear under the configured output dir | ❌ needs a capture run | | Survives a disconnect | `scripts/fgc start`, close ssh, reconnect, `scripts/fgc status` | ✅ runs under the boot service, independent of any ssh session | | Starts at boot | reboot with nobody logged in; `scripts/fgc status` from a later ssh | ✅ 2026-09-23, lock under `/run/user/1000` (lingering on) | | Reports to a broker | topics arrive at the configured broker | ❌ no broker on this tower yet |