40 KiB

Raw Blame History

Installing a tower from scratch

How a new fire-watch tower host goes from a freshly imaged machine to a running, calibrated fire_gimbal_control. Written while doing exactly that on the IOTA tower at 10.11.12.121 (2026-09-15) and kept honest about which steps have actually been exercised and which are still on paper.

deployment.md is the update loop for a tower that already works. This is the first install, including the parts that only happen once: the board's preconditions, the Vimba X SDK, hardware.ini, and the motor firmware.

scripts/install-tower now does the steps below, on the tower, in this order: it surveys the machine, asks for what it cannot work out (every question can also be answered with --answer KEY=VALUE), shows the plan, and applies it after one yes. --dry-run shows the plan and changes nothing; --verify runs the checks only. It never invents a drivetrain value: answer unknown and it writes a ; TODO, and refuses to flash firmware an axis of which would then sit in NEEDCFG. Homing is offered only at the end, and only to someone who says they are at the rig.

[verified] as a re-run on the already-configured tower 121 (2026-09-23): it changed only what was out of date and passed test hardware. [untested] on a freshly imaged board - the fresh path has been rehearsed with --dry-run only. What follows stays the reference for what each step does and why.

Status legend — each step says where it stands: [verified] run end to end on a real tower and the result checked · [partial] run, with something still open · [untested] written from the design, not yet exercised.

What you need before starting


The tower host	Powered, on the network, reachable by IP, with an OS installed and an `ssh` server running
An account on it	`ggs`, in the `sudo` and `dialout` groups
A dev box	This repository checked out, plus a copy of the Vimba X SDK (login-walled, so it cannot be fetched by a script)
The hardware	Camera plugged in, motor MCU plugged in, and a note of what was assembled: which axes, which IMU, which ambient sensor

Two facts decide most of what follows, so establish them first:

ssh ggs@<tower-ip> 'cat /sys/class/dmi/id/product_name; . /etc/os-release && echo "$PRETTY_NAME"'

Board	MCU	Setup kit	PlatformIO env
LattePanda 3 Delta / Sigma	ATmega32U4 (Leonardo), 5 V	not written - the Delta fleet predates the kits	`leonardo`
LattePanda IOTA	RP2040, 3.3 V	`hardware/lattepanda-iota/`	`iota_rp2040`

1. SSH by key `[verified]`

Everything after this is driven over ssh from the dev box, and deploy.sh runs several ssh commands per invocation, so password auth is not workable.

# on the dev box
ssh-copy-id -i ~/.ssh/id_ed25519_ggs_pedro.pub ggs@<tower-ip>

Then give the host a block in ~/.ssh/config, so every tool picks the right key without being told:

Host 10.11.12.121
    User ggs
    IdentityFile ~/.ssh/id_ed25519_ggs_pedro

Verify with ssh ggs@<tower-ip> hostname - it must not prompt.

ssh-copy-id needs a terminal. Run it in a real shell, not through a tool that gives it no tty: without one, ssh looks for a GUI ssh-askpass, and on a machine that has none it fails with ssh_askpass: exec(...): No such file or directory and then Permission denied - which looks like a rejected key rather than a prompt that never appeared.

2. Survey the machine before changing it `[verified]`

The setup kit can describe the machine without touching it, and this is the cheapest moment to notice that the tower is not what you assumed:

rsync -a hardware/ ggs@<tower-ip>:~/projects/fwt_2a/software/hardware/
ssh ggs@<tower-ip> '~/projects/fwt_2a/software/hardware/lattepanda-iota/install.sh --probe'

board            : lattepanda-iota
os               : Ubuntu 26.04.1 LTS (7.0.0-31-generic)
motor MCU        : /dev/serial/by-id/usb-MicroPython_Board_in_FS_mode_df61a863dfa02932-if00
camera DEV_ id   : NOT FOUND (needs the Vimba X SDK installed)
alvium on usb    : 1 device(s)
usbfs_memory_mb  : 16 (want >= 1000)
camera lpm_permit: u1_u2 (want 0)

Read every line. On tower 121 this pass found that the MCU was still running the MicroPython image it shipped with, and that both USB preconditions were at their (wrong) defaults.

Check what is already in ~/projects/. A machine cloned from another tower, or one that has been used for development, can arrive with a config.ini that belongs to a different rig. Tower 121 had a byte-identical copy of the dev box's config.ini - tower_name = dev, broker 127.0.0.1 - which is worse than no config at all, because none of those values trip the placeholder guard in deploy.sh --run. That guard only catches ExampleTower, CHANGE_ME and DEV_0000000000.

ssh ggs@<tower-ip> 'grep -E "tower_name|zkms_server_ip|^id" ~/projects/fwt_2a/software/config.ini'

3. Host preconditions and build dependencies `[verified]`

Both come from the board's setup kit, in one step, on the device:

ssh ggs@<tower-ip>
cd ~/projects/fwt_2a/software/hardware/lattepanda-iota
sudo ./install.sh --deps --preconditions

--deps installs packages.txt: the toolchain, OpenCV, Boost, libjxl, tmux, and the Python bits PlatformIO needs.

--preconditions installs two files and applies them immediately:

File	Why
`/etc/udev/rules.d/99-fwt-alvium-usb.rules`	Disables USB3 hardware LPM on the camera's port, which the host otherwise re-enables on every enumeration. With U1/U2 on, the camera logs link errors and frame delivery stalls (known-issues #19)
`/etc/tmpfiles.d/10-fwt-usbfs.conf`	Raises `usbfs_memory_mb` from 16 to 1000 at every boot. A single full-resolution frame does not fit in 16 MB

Both are settings that revert on reboot if nobody persists them, which is the whole of known-issue #19. The script re-reads them after applying and says so:

>> usbfs_memory_mb = 1000
>> camera USB3 LPM disabled (2-1)

If it instead warns that a value did not take, stop and find out why - a camera on a host with these wrong looks like a flaky camera.

Confirmed on tower 121 (2026-09-15): both went from wrong to right in one run, usbfs_memory_mb 16 -> 1000 and lpm_permit u1_u2 -> 0. Re-check after the first reboot, since surviving one is the entire point.

Neither step can be driven from a tool without a terminal. sudo needs a tty to prompt, so this runs from a real terminal window:
ssh -t ggs@<tower-ip> 'cd ~/projects/fwt_2a/software/hardware/lattepanda-iota && sudo ./install.sh --deps --preconditions'

The Delta fleet does not have these applied. Checked on 10.11.12.111 2026-09-15: usbfs_memory_mb is 16 and the kernel cmdline is stock. The preconditions were documented in known-issues but never persisted anywhere.

4. Vimba X SDK `[verified]`

Proprietary and login-walled, so it is staged from a machine that already has it rather than downloaded on the tower:

# dev box
rsync -az ~/Programs/VimbaX_2025-3 ggs@<tower-ip>:~/
# tower
sudo ./install.sh --vimba ~ggs/VimbaX_2025-3

That copies it to /opt/VimbaX_<version>, points /opt/VimbaX at it, runs the SDK's Install_GenTL_Path.sh (which registers GENICAM_GENTL64_PATH in /etc/profile.d and writes the 1ab2 udev rule), registers api/lib with the dynamic loader, and then asks the SDK to list cameras as a check.

The SDK asks for a reboot before the transport layers are usable. If it finds nothing, reboot and try again before suspecting the camera. On tower 121 no reboot was needed - the camera answered immediately.

Two things about that check, both found the hard way here:

The binary is ListCameras_VmbC, in /opt/VimbaX/bin/. There is no plain ListCameras.
Run it with GENICAM_GENTL64_PATH set, or it fails with Could not start system. Error code: -16 on a perfectly good installation - that is "no transport layer found", not "no camera":
```
GENICAM_GENTL64_PATH=/opt/VimbaX/cti /opt/VimbaX/bin/ListCameras_VmbC
```

The SDK ships a camera simulator, and it is on by default. On tower 121 ListCameras_VmbC reported four cameras: the real Alvium plus three synthetic ones from VimbaCameraSimulatorTL.cti, with ids like DEV_Cam1. A tower configured with one of those would open its "camera" happily and capture synthetic images forever. Take the id only from a block whose transport layer path is VimbaUSBTL or VimbaGigETL - which is what install.sh --probe does, after this bit me.

Non-interactive ssh shells do not source /etc/profile.d, which is why deploy.sh resolves GENICAM_GENTL64_PATH itself before launching the app. Getting this wrong shows up as VmbStartup() failing with "Could not start Vimba X API" even though the viewer works when you are logged in.

5. `hardware.ini` - what this machine IS `[verified]`

sudo ./install.sh --hardware-ini --axes yaw --imu none --env none --cameras rgb

This writes /etc/fire_gimbal_control/hardware.ini, outside the directory deploy.sh rsyncs, so no deploy can overwrite or delete it.

The probed half - board model, firmware env, the MCU's stable by-id path, the camera's DEV_ id - the script fills in. The declared half is the flags, and they are not a formality: the program degrades gracefully around missing hardware, so a part that was never fitted and a part that has failed look identical unless the file says which was expected. Declare what was assembled.

Check the result, especially the serial path:

ssh ggs@<tower-ip> 'cat /etc/fire_gimbal_control/hardware.ini'

Then prove the program finds it with nothing in the environment - no FGC_HARDWARE, no --hardware - because that is how the tower starts itself:

[INFO ] hardware: /etc/fire_gimbal_control/hardware.ini  board lattepanda-iota  axes yaw

Do not leave it anywhere under /tmp. That is tempting during bring-up because it needs no root, and it works right up until the first reboot, which clears /tmp and leaves the program with Config error: Camera.rgb.id is required - a message that points at the camera rather than at the file that disappeared. Verified on tower 121 by rebooting it.

The drivetrain block

install.sh now writes this block, one [Motor.<axis>] per declared axis — it has to, because the firmware refuses to move until it has been told these (see the config handshake in configuration.md). What it still cannot do is probe them: no reading of a board tells you its sense resistor.

So pass what you know and it writes the values; omit anything and it writes a commented ; TODO line naming what to measure:

./install.sh --stdout --axes yaw \
    --r-sense 0.075 --run-current 756 --homing-current 200 \
    --travel-deg 340 --steps-per-rev 1210778 --sw-mode 0

A TODO line is commented, not a placeholder value, and deliberately: these keys are numeric, and a non-numeric value makes the program refuse to start with a message about a config key. Left absent, it starts and names exactly which keys that axis is missing and that it will not move — the useful failure.

The block it writes looks like this:

[Motor.yaw]
r_sense_ohm       = 0.075   ; the resistor fitted to THIS driver board
run_current_ma    = 756     ; normal motion
homing_current_ma = 200     ; the endstop search only
travel_deg        = 340     ; mechanical angle between the hard stops

homing_current_ma is a safety setting, not a tuning one. Homing finds a stop by driving into it until the shaft falls behind; that only works while the stop is stronger than the motor. A high gear ratio can shear a screw-head stop straight past its mate - on tower 121 the axis ran 1.8 turns past the stop with the encoder never falling behind, at full torque. Declare a reduced search current wherever the stop cannot survive one.

Getting it wrong in either direction is visible: too strong and the stop yields again; too weak and the motor stalls on ordinary friction mid-sweep, trips the same latch, and records a phantom endstop. The firmware's steps_per_rev guard is what catches the second case, so it has to be set from the real travel.

The centre lives in config.ini, not here. install.sh cannot write it — hardware.ini is merged over config.ini and wins, so a shared key would silently discard whatever gimbal heading setup had saved. What it does instead is work the requirement out from steps_per_rev and print it as a NOTE in the block, because the default is wrong for any geared-up rig:
; NOTE: half this travel is 605389 counts, so config.ini needs
;   [Motor] yaw_center_count = 1000000    (or anything above 605389)
;   The 500000 default would put the negative end at -105389.
The range is centre ± half the travel, so a centre below half-travel puts the negative end under zero. On tower 121 the inherited 500000 homed to -104749..1104749; at 1000000 it homes to 395194..1604807. A warning names it after every home if it is still too small.

travel_deg is a nameplate fact, and the host divides the measured homing travel by it to derive [Motor] <axis>_counts_per_deg into config.ini. A rough angle gives a proportionally rough degree scale - 1 % out here is 1 % out on every bearing - until gimbal heading setup supersedes it with a landmark fit.

The startup log reports what the registers actually deliver, which is not always what was asked for:

yaw motor current: run 756 mA (GLOBALSCALER 80, IRUN 31), homing 189 mA (IRUN 7)
                   -- R_sense 0.075 ohm from hardware.ini [Motor.yaw]

Omit the block entirely on a tower whose endstops hold against full torque: no CURRENT is sent, the firmware keeps its compile-time defaults, and nothing changes.

If an axis declares a reduced homing current and the firmware has not adopted it, the host refuses to home rather than driving into a stop at full torque. The message names the skew; the fix is to flash the firmware, not to override it.

[Serial] device must be a /dev/serial/by-id/ path, never /dev/ttyACM0. Tower 121 has a 5G modem that claims three ttyUSB nodes, and enumeration order is not stable across reboots.

The by-id path changes when you flash the MCU, because the name comes from the running image: usb-MicroPython_Board_in_FS_mode_* before, a Pico/Arduino name after. Re-run --probe after step 6 and fix the file if it moved.

6. Motor firmware `[verified]`

The firmware is built and flashed on the tower - it is the machine wired to the MCU - so PlatformIO is installed there, not on the dev box.

# tower: PlatformIO core, in its own venv under ~/.platformio. No sudo needed.
curl -fsSL https://raw.githubusercontent.com/platformio/platformio-core-installer/master/get-platformio.py -o /tmp/get-platformio.py
python3 /tmp/get-platformio.py
~/.platformio/penv/bin/pio --version      # check it actually runs

[verified] on tower 121: PlatformIO Core 6.2.0 on the system Python 3.14.4.

The installer ends with a FileNotFoundError traceback out of multiprocessing/util.py about a missing /tmp/.piocore-installer-* path. It is a temp-directory cleanup finalizer running after the work is done - cosmetic. Judge the install by whether pio --version answers.

deploy.sh looks for pio in the usual places, but a non-interactive ssh shell does not have ~/.platformio/penv/bin on PATH, so name it explicitly:

REMOTE_PIO=/home/ggs/.platformio/penv/bin/pio

Then, from the dev box, in firmware/:

# .deploy.env
REMOTE_HOST=ggs@10.11.12.121
REMOTE_DIR=/home/ggs/projects/fwt_2a/firmware
PIO_ENV=iota_rp2040          # leonardo on a Delta
SERIAL_PORT=/dev/serial/by-id/usb-...

./deploy.sh --compile-only     # build on the tower, flash nothing
./deploy.sh                    # build + upload
./deploy.sh --monitor          # watch it boot

Confirm the pin map before flashing a board that is wired to motors. A chip-select on the wrong pin writes into a driver that was never meant to hear it. The pins are the [Pins] section of this tower's hardware.ini: the build reads them from /etc/fire_gimbal_control/hardware.ini (firmware/tools/gen_wiring.py) and fails without them, so add the section before the first build. The reference loom, with the table saying which wire goes where, is hardware/lattepanda-iota/pins.ini - the loom as built on tower 121, which is not the map proposed in the porting plan. Copy it in if the tower is wired the same way. The generator refuses pins SPI0 cannot reach, and test hardware motor compares the declared pins with the ones the running firmware reports it was built with.

Getting a factory board to accept its first flash

[verified] on tower 121, and this is the step that will bite you. These boards ship running MicroPython, and the normal upload does not work against it:

Forcing reset using 1200bps open/close on port /dev/ttyACM0
No new RPxxxx device found yet, waiting..          (x20, then it gives up)

What	Factory board (MicroPython)	Our firmware
1200-baud touch, i.e. a plain upload	no reset at all	resets, but reboots into the application
`picotool reboot -f -u`	refused: no reset interface	-
MicroPython REPL	works	n/a

So drop into MicroPython's own REPL and tell it to enter the bootloader. firmware/tools/flash.sh (which deploy.sh runs on the tower) now does this by itself when the port's by-id name says MicroPython_Board: ./deploy.sh with SERIAL_PORT set to that path should flash a factory board directly. [untested]

no factory board has been through that path yet; the manual way, verified on tower 121, is below. Write this to a file and run it with ~/.platformio/penv/bin/python (it has pyserial):

import serial, time
s = serial.Serial("/dev/serial/by-id/usb-MicroPython_Board_in_FS_mode_XXXX-if00", 115200, timeout=1)
s.write(b"\x03\x03"); time.sleep(0.3)        # Ctrl-C: interrupt whatever is running
s.write(b"import machine\r\n"); time.sleep(0.3)
s.write(b"machine.bootloader()\r\n")         # the port drops here - that is success

lsusb | grep 2e8a        # expect: 2e8a:0003 Raspberry Pi RP2 Boot

Then run ./deploy.sh and it flashes. A physical BOOTSEL press does the same job if you are standing at the board.

Every board needs exactly one physical BOOTSEL press in its life - the one that installs the first image carrying the BOOTSEL command (below). After that, updates are remote.

sudo ./install.sh --preconditions must have run first, or picotool finds the board and refuses it:

RP2040 device at bus 1, address 4 appears to be in BOOTSEL mode, but picotool
    was unable to connect. Maybe try 'sudo' or check your permissions.

Re-flashing later: the `BOOTSEL` command

[verified] on tower 121. Neither documented remote path works on this board - the 1200-baud touch resets the firmware but returns it to the application, and picotool reboot -f -u answers Unable to locate reset interface on the device. So the firmware carries a command that does it deliberately:

# over the serial port, or from the app: gimbal raw "BOOTSEL"
BOOTSEL   -> OK entering bootloader - reflash to return
lsusb | grep 2e8a        # 2e8a:0003 Raspberry Pi RP2 Boot
./deploy.sh              # flashes and reboots into the new firmware

The whole loop was run end to end from the dev box with nobody at the tower. BOOTSEL disables every axis first, since the bootrom does not run the control loop and unsupervised energised drivers would hold torque.

It takes the gimbal out of service and the firmware cannot undo it - once in the bootrom there is no command loop left to change its mind. Recovery is deploy.sh, or a power cycle: entering the bootrom erases nothing, so the existing image is still in flash. Anything that can write to the serial port can send it, including gimbal raw. Near misses (BOOT, BOOTSE) are rejected as unknown commands.

The identity changes when you flash

	by-id path	USB
before	`usb-MicroPython_Board_in_FS_mode_<serial>-if00`	`2e8a:0005`
after	`usb-Raspberry_Pi_Pico_<SERIAL>-if00`	`2e8a:000a`

So generate hardware.ini (step 5) after flashing, or re-run install.sh --hardware-ini --force to pick up the new path.

Verify the flash

Read the boot banner over the serial port:

BOOT build=aab1925-dirty
IOIN Y=0x3000001E version=0x30
ST Y:R,500000,18,80084000,0,8,8,Sd

version=0x30 is the TMC5160 answering over SPI - the single most useful line here, because it proves the pin map, the 3.3 V logic level and the driver board all at once. BOOT build= must match the commit you flashed. One Y line and no P line is a yaw-only build.

Flashing an RP2040 remotely is not the same as a Leonardo. The documented flow is hold BOOTSEL, tap RST, drag a .uf2 - a person at the board. The two remote paths (the 1200-baud touch that the Arduino tooling already uses, and picotool reboot -f -u) both need the running image to present a USB CDC port. That is why setup() brings Serial up before any hardware init: an image that hangs before USB comes up cannot be reflashed without a site visit, while an interrupted flash recovers by itself - the bootrom falls back to mass storage when flash holds no valid image.

7. Deploy the software `[verified]`

On the dev box, in software/:

# .deploy.env
REMOTE_HOST=ggs@10.11.12.121
REMOTE_DIR=/home/ggs/projects/fwt_2a/software
CMAKE_ARGS="-DWITH_MQTT=ON -DWITH_VIMBA=ON"
RUN_ARGS="--start"

./deploy.sh --check-deps    # must print "remote deps OK"
./deploy.sh                 # rsync + remote cmake + build

The first deploy seeds config.ini from config/config.example.ini only if the device has none - see the warning in step 2 about a machine that arrived with somebody else's config.

[verified] on tower 121: builds clean on GCC 15 / Ubuntu 26.04 with a single -Wformat-truncation warning in utcStampNow(), which is a false positive (the compiler cannot prove tm_year is four digits; the real output is 17 bytes into a 32-byte buffer).

8. Camera profiles `[verified]`

Without a profile the program does not start - no camera profiles (*.xml) in config/camera_profiles/rgb is a startup error, by design. A fresh tower has an empty profile directory, so this step is not optional.

The documented way to make one is the VimbaX Viewer, which is a GUI - awkward on a headless tower and impossible unattended. scripts/export_camera_profile.py does the same job through the SDK's Python API:

# once per tower: the SDK ships the wheel
python3 -m venv ~/.venvs/vmbpy
~/.venvs/vmbpy/bin/pip install /opt/VimbaX/api/python/vmbpy-*.whl

cd ~/projects/fwt_2a/software
export GENICAM_GENTL64_PATH=/opt/VimbaX/cti
~/.venvs/vmbpy/bin/python scripts/export_camera_profile.py --profile standard --dry-run
~/.venvs/vmbpy/bin/python scripts/export_camera_profile.py --profile standard
~/.venvs/vmbpy/bin/python scripts/export_camera_profile.py --profile night

It writes each recipe value to the real camera in dependency order, lets the camera validate it, and exports what was actually accepted. That preserves the property the Viewer workflow exists to provide: GenICam validity is stateful, so a hand-written XML can be individually plausible and collectively wrong. Here the camera is the judge, and anything it refused is printed rather than assumed.

Read the summary line it prints - it is the number that decides whether the profile can be delivered at all:

result  : 2664x1520 RGB8  12.1 MB/frame  4.95 fps  = 60 MB/s

Profiles do not transfer between cameras

[verified] the hard way. The fleet runs several Alvium generations and their feature vocabularies differ, so copying another tower's XML does not work:

	Delta tower (1800 U-1240c)	IOTA tower (1800 U-1620c)
`SensorBitDepth`	`Bpp8`	no `Bpp8` - only `Adaptive` / `Bpp12`
`IntensityAutoPrecedence`	`Minimizeexposuretime` / `Minimizenoise`	`MinimizeBlur` / `MinimizeNoise`
Sensor	12 MP	16 MP, 5328x3040

Binning is not optional, and the frame-rate ceiling is not what it looks like

Unbinned RGB8 on the 16 MP sensor is a 48.6 MB frame - over the ~20 MB ceiling in known-issues #19. 2x2 binning brings it to 12.1 MB.

The frame rate is then capped by DeviceLinkThroughputLimit, but the limit is accounted against the unbinned payload, not the binned frame that crosses the wire. Measured on tower 121:

`DeviceLinkThroughputLimit`	max `AcquisitionFrameRate`	actual USB traffic
250 MB/s (the fleet's value)	4.95 fps	~60 MB/s
450 MB/s (this camera's max)	8.79 fps	~106 MB/s

48.6 MB x 4.95 = 240 MB/s, which is the 250 limit - so the camera is budgeting for data that binning discards. The recipe ships the conservative 250 MB/s, giving ~5 fps rather than the Delta's 10, because the USB stalls behind known-issues #19 are a host-side finding that has not been re-measured on the N150 yet (porting plan phase 5). Raising it is a one-line change in the recipe once that measurement exists.

Export standard after night, not the other way round, or use the current script. A refused write is not a no-op: the first version of this script asked for 10 fps, the camera refused, and the 1 fps left over from the night profile stayed - producing a "standard" profile that ran at night speed. The script now clamps an out-of-range numeric value to the camera's limit and says ~ ... (clamped).

9. `config.ini` - what this tower DOES `[partial]`

Device-local, never overwritten by a deploy. Edit it on the tower:

[General]
tower_name = <this tower>          ; drives every MQTT topic
[Network]
zkms_server_ip = <broker>

Credentials belong in /etc/fire_gimbal_control/credentials.env, not here.

Anything hardware.ini now owns - [Serial] device, [IMU], [Env], the camera ids - should be deleted from config.ini rather than left to rot: the program warns at startup about every key set in both.

[Camera.<name>] id is required even when the camera is disabled. Config validation runs before any of the feature flags are consulted, so a tower with no camera fitted yet cannot start without one - and the obvious dummy, DEV_0000000000, is what deploy.sh's placeholder guard refuses. On a tower that has its camera, the id lives in hardware.ini and this never comes up.

[Features] enable_imu and enable_env matter more than they look. Declaring imu = none in hardware.ini does not yet switch them off - deriving them is section 12 of the config plan and is not implemented. On tower 121 the first clean-looking run did this:

[INFO ] MTi IMU started on /dev/ttyS0 @ 115200
[ERROR] I2C: failed to open /dev/i2c-2
[ERROR] SHT41: failed to open I2C bus; sensor disabled

It "started" an IMU on a tower that has none - /dev/ttyS0 exists and accepts a serial open whether or not anything is attached to it. Set both flags to false by hand until reconciliation lands.

10. First run: bring the subsystems up one at a time `[partial]`

Do not start a new tower with everything switched on. Each phase below adds exactly one subsystem, so that a failure names its own cause instead of arriving as a wall of unrelated errors. It also keeps the motors - the only part that can break itself - until you have proved the boring parts work.

Each phase says what good looks like and what noise to expect.

Phase 0 - the host alone, no hardware `[verified]`

./build/fire_gimbal_control --mock-serial --mock-camera --no-mqtt --no-tui

Proves config resolution, hardware.ini layering, the single-instance lock, the scan grid and the output directory - with nothing physical involved. Neither a camera nor its profile XMLs need exist:

[INFO ] camera profiles not loaded: no real camera in this run

[Features] enable_camera = false does the same thing from the config file, for a tower that has no camera fitted yet, and says so rather than pretending:

[WARN ] camera disabled ([Features] enable_camera = false); using a mock source
        - no images come from hardware in this run

[INFO ] hardware: /etc/fire_gimbal_control/hardware.ini  board lattepanda-iota  axes yaw
[INFO ] Loaded config: config.ini
[INFO ] scan grid: 15 waypoints

Any config.ini still sets X, which hardware.ini overrides warning belongs here and should be cleared before moving on (step 9).

Phase 1 - motors `[partial]`

Real serial link, nothing else:

./build/fire_gimbal_control --mock-camera --no-mqtt --no-tui

What proves the link is a live DUMP from the board, not just an open port:

[INFO ] Serial controller started on /dev/serial/by-id/usb-Raspberry_Pi_Pico_...
DUMP BEGIN build=aab1925-dirty uptime=7925212 mcusr=0x3 free_ram=252712
DUMP Y state=1 ... speed=3000 center=500000 has_encoder=1

Check build= matches the firmware you flashed and speed= matches the drivetrain values you intended. Then, and only then, move the thing:

gimbal home              # the endstop-finding sequence - see the warning below
gimbal diag y            # firmware self-test, reports PASS/FAIL
gimbal move 10,0         # a small deliberate move

Expected noise in this phase, neither of which is a fault:

Message	Why
`Motor controller: ERR axis not present`	The host configures both axes at startup; a yaw-only firmware refuses the pitch half. Harmless until presence reconciliation lands (config plan section 10)
`encoder tracking error on yaw: 500366 counts (limit 400)`	Before homing, the firmware presets `XACTUAL` to the centre count while the encoder sits at its own zero, so they disagree by the whole range. Homing resyncs them. If it persists after a successful home, that is real

On an unfamiliar drivetrain, homing is the dangerous step - it finds the endstop by driving into it. Start from a deliberately slow homing_speed and walk it up, watching the free-running encoder gap that ENC_DEVIATION_LIMIT is measured against. Be at the rig with a hand on the power for the first one.

Phase 2 - camera `[verified]`

Add the camera, still no broker:

./build/fire_gimbal_control --no-mqtt --no-tui

[INFO ] Opened camera 'rgb' DEV_1AB22C0B7D4E (1800 U-1620c)
[INFO ] camera profile 'standard' applied from .../standard.xml (17206 settings)

Then start, and confirm .jxl files appear under the configured output dir.

A camera-free phase needs no profile XMLs - but this only became true on 2026-09-16. Before that, [Features] enable_camera was parsed and never used, and the profile directory was validated even for a mock camera, so a tower whose camera had not been configured yet could not start at all. If you are running an older binary, export the profiles (step 8) first or it will refuse with no camera profiles (*.xml).

Phase 3 - the broker

Set [Features] enable_mqtt = true and point zkms_server_ip at the broker. Without one, the app says so and carries on:

[ERROR] MQTT connect failed (tcp://10.11.12.121:1883): TCP/TLS connect failure
[WARN ] Control channel not connected; continuing in degraded mode

Degraded mode is not a failure to notice later - the tower captures images and reports to nobody. Confirm topics arrive at the broker before calling the phase done.

Each tower runs its own RabbitMQ, in Docker, with the rabbitmq_mqtt and rabbitmq_shovel plugins - the shovel is what forwards this tower's messages to the ground station. That lives in ~/zkms-client/, outside this repository.

Phase 4 - IMU and ambient sensor

Only if fitted. hardware.ini should already say so, and [Features] enable_imu / enable_env have to agree with it (see step 9 - declaring imu = none does not yet switch the flag off by itself).

Phase 5 - unattended

Only once every phase above is green:

scripts/fgc start --start     # detached, inside tmux
scripts/fgc status            # running? since when? which PID?

Close the ssh session, reconnect, and check it is still running. Then reboot the tower and confirm the USB preconditions survived (install.sh --probe) and the app comes back.

Calibration

Per tower, and cannot be inherited - least of all from a tower with a different gear ratio, where yaw_counts_per_deg is derived from a steps_per_rev that no longer applies:

gimbal heading setup     # ties yaw to true north by sighting landmarks; no IMU needed
gimbal pitch setup       # needs an IMU; skip on a rig without one

10b. Bringing up the config handshake on an existing tower

This is the one upgrade that changes what an unconfigured tower does: the new firmware boots into NEEDCFG and refuses HOME / MOVE / ENABLE until the host has answered its NEED lines. Flashing it onto a tower whose hardware.ini predates the drivetrain keys leaves the gimbal inert — working, talking, and not turning.

So the order matters. Host first, firmware second. Each step below either changes nothing or is reversible, and the step that could strand the tower comes after the one that prevents it.

0. Record what you are rolling back to

gimbal dump                 # the running firmware's build id and register state
cat /etc/fire_gimbal_control/hardware.ini
git -C ~/projects/fwt_2a/firmware rev-parse --short HEAD

Keep the dump. It is the only record of what the old firmware was holding, and the comparison that tells you whether the new one adopted the same values.

1. Update `hardware.ini` — nothing else, and no restart yet

Add the [Motor.<axis>] block (section 5). The five keys the new firmware will not run without are steps_per_rev, enc_deviation, sw_mode, plus the two speeds; install.sh --stdout generates the block if you would rather not type it.

Doing this before flashing is what keeps the window at zero: the host is ready to answer the moment the firmware starts asking.

2. Deploy the new host against the OLD firmware

A real test, and it risks nothing. The old firmware never sends NEED, so the host must behave exactly as it did before — which is also the guarantee the whole Delta fleet depends on.

scripts/fgc start --start
scripts/fgc peek
test hardware motor

Expect: no handshake lines at all, and the motor leaf reporting a version skew rather than a mismatch —

; yaw_current: yaw: firmware echoes only part of the current (2 of 3 fields
  absent), so it cannot be confirmed -- it predates the full CURRENT echo. Re-flash it

If instead you see wrong values, stop: something in hardware.ini disagrees with what the tower is really running, and flashing will not fix it.

3. Flash

cd firmware && ./deploy.sh          # uploads; --compile-only if you want a dry run first

Watch the banner. A NEED line per axis should follow it:

BOOT build=<rev>
NEED Y steps_per_rev global_scaler run_irun homing_irun center_count soft_margin ...

No NEED line means the axis is not in NEEDCFG — either the flash did not take, or the image is older than you think. Check the build id before going on.

4. Confirm the handshake completed

The host answers within a control-loop tick. Expect:

firmware config: answered 11 keys for axis Y

Then prove it landed rather than trusting the log:

gimbal dump                 # state should be 1 (RESET), NOT 5 (NEEDCFG)
                            # cfg_rx should be the full required mask
test hardware motor         # both metrics PASS

test hardware motor is the real gate: it compares every declared value against what the firmware echoes, so a SET that was lost on the wire shows up here and nowhere else.

5. Prove the refusal actually refuses

Worth doing once per firmware version, because a guard that silently stopped working looks exactly like a guard that is working.

On a bench board, not a mast: power-cycle it without the host running, then send HOME over a serial monitor. Expect ERR need config. Send RESET and try again — still ERR need config, because a RESET is deliberately not a way out. USE DEFAULTS is, and should then let HOME through.

6. Only then, move the axis

gimbal home                 # at the reduced homing current

Check against the recorded figures: travel ~1,210,778 counts, repeatable to ~0.3 %, and the derived counts_per_deg near 3552.87. A travel that disagrees by more than a few percent means the axis homed against something that was not the endstop.

Reading the log afterwards

Everything below is recorded to ~/.local/share/fire_gimbal_control/logs/fgc_*.log, not only to the dashboard — which matters here, because in TUI mode the pane holds 500 lines and dies with the tmux session. During this bring-up that cost us the handshake record four separate times before the file existed.

L=$(ls -t ~/.local/share/fire_gimbal_control/logs/fgc_*.log | head -1)
grep -E "NEED|firmware config|need config" "$L"   # step 4: did the handshake complete
grep -E "hardware:"                        "$L"   # presence reconciliation
grep -E "speed:|motor current"             "$L"   # what was actually sent to the firmware

The file is more verbose than the dashboard was, so it holds lines the pane never showed. head -8 gives the build, tower and config that produced it.

If it goes wrong

The board is recoverable over the network as long as USB CDC comes up: BOOTSEL over the serial port, then ./deploy.sh with the previous firmware checked out. Only an image that wedges before Serial.begin() needs a physical button press — which is why Serial.begin() is the first thing setup() does.

USE DEFAULTS is the in-place escape: it puts every axis on the compile-time table and lets the tower move while you work out what hardware.ini is missing.

11. Verification checklist

The checks that move nothing can be run as one command, which is what a script should use rather than typing into the dashboard. Stop the app first: it holds the serial port.

scripts/fgc stop
GENICAM_GENTL64_PATH=/opt/VimbaX/cti ./build/fire_gimbal_control --no-mqtt --test hardware
echo $?      # 0 PASS, 4 FAIL, 5 could not start (the log says why)
scripts/fgc start

It waits for the firmware to be talking and configured before it looks, so a freshly flashed board is not failed for values the host had not sent yet. build/fgc-config check --config config.ini --hardware /etc/fire_gimbal_control/hardware.ini checks that the two files load together, and changes nothing.

Check	How	Tower 121
Key-only ssh	`ssh ggs@<tower> hostname`, no prompt	✅
Preconditions survive a reboot	reboot, then `install.sh --probe`: `usbfs_memory_mb 1000`, `lpm_permit 0`	✅ confirmed across a real reboot
`hardware.ini` found unaided	run with no `FGC_HARDWARE`: startup names `/etc/fire_gimbal_control/hardware.ini`	✅
Camera enumerates	`ListCameras_VmbC` reports the `DEV_` id in `hardware.ini`	✅ `DEV_1AB22C0B7D4E`
Camera profile applies	startup logs `profile 'standard' applied ... (N settings)`	✅ 17206 settings
Firmware identity	`gimbal dump` reports the `GIT_REV` you flashed	✅ `build=aab1925-dirty`
Driver answers over SPI	boot banner shows `IOIN Y=0x… version=0x30`	✅
Remote re-flash	`BOOTSEL`, then `./deploy.sh` - no hands on the board	✅
Declared hardware matches	startup logs no presence warnings	✅ no shadowed-key warnings
Homing repeatable	`gimbal home` several times, endstop counts within `home_spread_max_counts`	✅ three homes within 0.27 % (1,210,778 / 1,208,358 / 1,207,514 counts)
Endstop survives homing	inspect the stop after a home; the axis must stall against it, not push through	✅ at 189 mA, after failing at full torque
Degree scale derived	startup logs `counts_per_deg` agreeing with the homed travel	✅ 3552.87, saved with `;prev` history
Images land	`.jxl` files appear under the configured output dir	❌ needs a capture run
Survives a disconnect	`scripts/fgc start`, close ssh, reconnect, `scripts/fgc status`	✅ runs under the boot service, independent of any ssh session
Starts at boot	reboot with nobody logged in; `scripts/fgc status` from a later ssh	✅ 2026-09-23, lock under `/run/user/1000` (lingering on)
Reports to a broker	topics arrive at the configured broker	❌ no broker on this tower yet

40 KiB Raw Blame History