60 KiB

Raw Permalink Blame History

Installing a tower

How a fire-watch tower host goes from a freshly imaged machine - or from an older, unmigrated setup - to a running, calibrated fire_gimbal_control. deployment.md is what comes after: updating and operating a tower that already works.

scripts/install-tower does the whole of this. The short way is the next section. Everything after it, sections 1 to 11, is the reference for what each step does and why, written while doing each one by hand, and still the way to do one step on its own.

Status legend - each step says where it stands: [verified] run end to end on a real tower and the result checked · [partial] run, with something still open · [untested] written from the design, not yet exercised.

The short way: `scripts/install-tower`

Before starting


The tower host	Powered, on the network, an OS installed, `ssh` running, reachable by key (section 1)
An account	`ggs`, in the `sudo` and `dialout` groups
Both repositories on it	`~/projects/fwt_2a/software` and `~/projects/fwt_2a/firmware`. `scripts/bootstrap-tower` sets that up from one copied file (below), or each repo's `deploy.sh` syncs them from a dev box
The Vimba X SDK, staged on it	Login-walled, so it cannot be fetched by a script: copy it on (USB stick, `scp`) and give its directory when asked. `none` for a tower with no camera yet
The facts of the build	Which axes, which IMU, which ambient sensor, how the driver boards are wired, and - per axis - the drivetrain: sense resistor, currents, travel, `steps_per_rev`, `sw_mode`, `enc_deviation`, encoder or not. Anything not yet measured can be answered `unknown`

The installer takes the board from DMI and picks its kit:

Board	MCU	Setup kit	PlatformIO env
LattePanda 3 Delta	ATmega32U4 (Leonardo), 5 V	`hardware/lattepanda-3-delta/`	`leonardo`
LattePanda IOTA	RP2040, 3.3 V	`hardware/lattepanda-iota/`	`iota_rp2040`

Getting the code onto a tower: `scripts/bootstrap-tower`

One file to copy; it fetches the rest. It needs the Gitea deploy key (tower 111's ~/.ssh/id_ed25519_deploy_gitea) in the tower's ~/.ssh/:

scp scripts/bootstrap-tower ggs@<tower>:
scp -3 ggs@10.11.12.111:.ssh/id_ed25519_deploy_gitea ggs@<tower>:.ssh/   # -3: via this machine
ssh -t ggs@<tower> bash bootstrap-tower          # any install-tower options pass through

What it does:

adds Host gitea (port 2222) to ~/.ssh/config and records the server's host key;
clones both repositories side by side, or brings existing clones up to date (fast-forward only);
starts install-tower.

It stops, rather than overwrite, when it finds a checkout with local changes or a directory that isn't a clone.

A legacy tower: read it first, with `scripts/survey-legacy-tower`

The legacy towers run an older host program and a TMC-API Arduino sketch. Before migrating one, record what it runs, so its hardware.ini comes from its own values:

bash survey-legacy-tower --firmware      # pauses the old service ~20 s; changes nothing on disk

What it records:

the machine: board, OS, groups, disk, whether Gitea, GitHub, PyPI and PlatformIO are reachable;
the camera and the Vimba install;
the old service and its config.ini;
with --firmware, from the old firmware:
- the endstop travel (?);
- its stored north calibration (y);
- its TMC registers (i), with the run current worked out for the usual sense resistors.
the old firmware's source, from the newest build in the Arduino cache: its register table (encoder scale R3A, current R0B/R10, encoder limit R3D) and homing speed. The running firmware cannot report these: the registers are write-only, and it answers i for them with 0.

The report is saved as ~/legacy-survey-<date>.txt.

Running it

On the tower, as ggs:

cd ~/projects/fwt_2a/software
scripts/install-tower --dry-run     # survey + questions + the plan; changes nothing
scripts/install-tower               # the same, then does it after one yes
scripts/install-tower --verify      # the checks only; changes nothing

It surveys the machine, asks only what it cannot work out, prints a plan with each step marked skip, create, change or blocked, and applies it after one yes. In order:

packages, then the USB preconditions (section 3), then the Vimba X SDK (4)
builds the host
writes hardware.ini (5) - or, when one exists, only fills its gaps
seeds config.ini (9) and credentials.env, or edits only what is out of date. This is where it asks what the tower does as it starts - [Startup] home and mode - and where a tower whose boot service still passes the old --init / --start has them moved into config.ini
PlatformIO, then the camera profiles that are missing (8)
builds and flashes the firmware (6) - blocked while any drivetrain value is still a ; TODO, because that axis would refuse to move
test hardware
the boot service (deployment.md section 4), if wanted. It passes no startup flags; a new tower gets [Startup] home = true, mode = manual: it homes at every boot, then waits in manual and captures nothing until someone chooses auto
asks whether someone is at the rig; only on a yes does it home the gimbal and take one picture (a snapshot), then leave it in manual - through the service, when [Startup] home has the service home it, and otherwise with gimbal home, leaving config.ini as it is

Nothing starts capturing by itself. Choose auto yourself: key a on the dashboard, or mode auto over MQTT. Once the tower is proven, make it boot into auto, as 111 does:

scripts/install-tower --answer startup_mode=auto

That writes [Startup] mode = auto into config.ini; editing the file by hand does the same (with the program stopped - see the warning at the top of the file). --answer startup_home=y|n sets homing the same way. service_args now holds only other flags, and refuses --init / --start.

Every question has a key, so any can be answered up front - --answer tower_name=north, --answer yaw.r_sense_ohm=0.075 - or from a file of key=value lines with --answers FILE. --non-interactive stops instead of prompting. It never invents a drivetrain value: a wrong sense resistor cooks a driver and a wrong homing current shears an endstop, and both have happened on this project. Answer unknown and it writes a commented ; TODO naming what to measure.

Re-running it is safe, and is how a configured tower is repaired: an existing hardware.ini or config.ini is never regenerated, only filled in; an existing camera profile is never overwritten; a firmware already running the declared revision and pins is not reflashed; and if it fails after stopping the program, it puts the program back the way it was.

Where	Result
Tower 121 (IOTA), 2026-09-23	`[verified]` re-run on a configured tower: changed only what was out of date, `--verify` green, boot service installed and proven across a reboot
Tower 111 (Delta), 2026-09-23/24	`[verified]` migration of a tower that predated all of this: host first, firmware once the drivetrain was known, `test hardware` green on every leaf (section 10b)
A freshly imaged board	`[untested]` - rehearsed with `--dry-run` and typed answers only. The first flash of a factory MicroPython RP2040 is automated but has not met a factory board

1. SSH by key `[verified]`

Everything after this is driven over ssh from the dev box, and deploy.sh runs several ssh commands per invocation, so password auth is not workable.

# on the dev box
ssh-copy-id -i ~/.ssh/id_ed25519_ggs_pedro.pub ggs@<tower-ip>

Then give the host a block in ~/.ssh/config, so every tool picks the right key without being told:

Host 10.11.12.121
    User ggs
    IdentityFile ~/.ssh/id_ed25519_ggs_pedro

Verify with ssh ggs@<tower-ip> hostname - it must not prompt.

ssh-copy-id needs a terminal. Run it in a real shell, not through a tool that gives it no tty: without one, ssh looks for a GUI ssh-askpass, and on a machine that has none it fails with ssh_askpass: exec(...): No such file or directory and then Permission denied - which looks like a rejected key rather than a prompt that never appeared.

2. Survey the machine before changing it `[verified]`

The setup kit can describe the machine without touching it, and this is the cheapest moment to notice that the tower is not what you assumed:

rsync -a hardware/ ggs@<tower-ip>:~/projects/fwt_2a/software/hardware/
ssh ggs@<tower-ip> '~/projects/fwt_2a/software/hardware/lattepanda-iota/install.sh --probe'

board            : lattepanda-iota
os               : Ubuntu 26.04.1 LTS (7.0.0-31-generic)
motor MCU        : /dev/serial/by-id/usb-MicroPython_Board_in_FS_mode_df61a863dfa02932-if00
camera DEV_ id   : NOT FOUND (needs the Vimba X SDK installed)
alvium on usb    : 1 device(s)
usbfs_memory_mb  : 16 (want >= 1000)
camera lpm_permit: u1_u2 (want 0)

Read every line. On tower 121 this pass found that the MCU was still running the MicroPython image it shipped with, and that both USB preconditions were at their (wrong) defaults.

Check what is already in ~/projects/. A machine cloned from another tower, or one that has been used for development, can arrive with a config.ini that belongs to a different rig. Tower 121 had a byte-identical copy of the dev box's config.ini - tower_name = dev, broker 127.0.0.1 - which is worse than no config at all, because none of those values trip the placeholder guard in deploy.sh --run. That guard only catches ExampleTower, CHANGE_ME and DEV_0000000000.

ssh ggs@<tower-ip> 'grep -E "tower_name|zkms_server_ip|^id" ~/.config/fire_gimbal_control/config.ini'

3. Host preconditions and build dependencies `[verified]`

Both come from the board's setup kit, in one step, on the device:

ssh ggs@<tower-ip>
cd ~/projects/fwt_2a/software/hardware/lattepanda-iota
sudo ./install.sh --deps --preconditions

--deps installs packages.txt: the toolchain, OpenCV, Boost, libjxl, tmux, and the Python bits PlatformIO needs.

--preconditions installs two files and applies them immediately:

File	Why
`/etc/udev/rules.d/99-fwt-alvium-usb.rules`	Disables USB3 hardware LPM on the camera's port, which the host otherwise re-enables on every enumeration. With U1/U2 on, the camera logs link errors and frame delivery stalls (known-issues #19)
`/etc/tmpfiles.d/10-fwt-usbfs.conf`	Raises `usbfs_memory_mb` from 16 to 1000 at every boot. A single full-resolution frame does not fit in 16 MB

Both are settings that revert on reboot if nobody persists them, which is the whole of known-issue #19. The script re-reads them after applying and says so:

>> usbfs_memory_mb = 1000
>> camera USB3 LPM disabled (2-1)

If it instead warns that a value did not take, stop and find out why - a camera on a host with these wrong looks like a flaky camera.

Confirmed on tower 121 (2026-09-15): both went from wrong to right in one run, usbfs_memory_mb 16 -> 1000 and lpm_permit u1_u2 -> 0. Re-check after the first reboot, since surviving one is the entire point.

Neither step can be driven from a tool without a terminal. sudo needs a tty to prompt, so this runs from a real terminal window:
ssh -t ggs@<tower-ip> 'cd ~/projects/fwt_2a/software/hardware/lattepanda-iota && sudo ./install.sh --deps --preconditions'

The Delta kit carries the same two files (hardware/lattepanda-3-delta/, as symlinks to these). Tower 111 had neither - usbfs_memory_mb 16, camera LPM u1_u2, checked 2026-09-15 and again 2026-09-23 - and got both on 2026-09-23, reading 1000 and 0 afterwards - and still after a reboot on 2026-09-24.

4. Vimba X SDK `[verified]`

Proprietary and login-walled, so it is staged from a machine that already has it rather than downloaded on the tower:

# dev box
rsync -az ~/Programs/VimbaX_2025-3 ggs@<tower-ip>:~/
# tower
sudo ./install.sh --vimba ~ggs/VimbaX_2025-3

That copies it to /opt/VimbaX_<version>, points /opt/VimbaX at it, runs the SDK's Install_GenTL_Path.sh (which registers GENICAM_GENTL64_PATH in /etc/profile.d and writes the 1ab2 udev rule), registers api/lib with the dynamic loader, and then asks the SDK to list cameras as a check.

The SDK asks for a reboot before the transport layers are usable. If it finds nothing, reboot and try again before suspecting the camera. On tower 121 no reboot was needed - the camera answered immediately.

Two things about that check, both found the hard way here:

The binary is ListCameras_VmbC, in /opt/VimbaX/bin/. There is no plain ListCameras.
Run it with GENICAM_GENTL64_PATH set, or it fails with Could not start system. Error code: -16 on a perfectly good installation - that is "no transport layer found", not "no camera":
```
GENICAM_GENTL64_PATH=/opt/VimbaX/cti /opt/VimbaX/bin/ListCameras_VmbC
```

The SDK ships a camera simulator, and it is on by default. On tower 121 ListCameras_VmbC reported four cameras: the real Alvium plus three synthetic ones from VimbaCameraSimulatorTL.cti, with ids like DEV_Cam1. A tower configured with one of those would open its "camera" happily and capture synthetic images forever. Take the id only from a block whose transport layer path is VimbaUSBTL or VimbaGigETL - which is what install.sh --probe does, after this bit me.

Non-interactive ssh shells do not source /etc/profile.d, which is why deploy.sh resolves GENICAM_GENTL64_PATH itself before launching the app. Getting this wrong shows up as VmbStartup() failing with "Could not start Vimba X API" even though the viewer works when you are logged in.

5. `hardware.ini` - what this machine IS `[verified]`

sudo ./install.sh --hardware-ini --axes yaw --imu none --env none --cameras rgb

This writes /etc/fire_gimbal_control/hardware.ini, outside the directory deploy.sh rsyncs, so no deploy can overwrite or delete it.

The probed half - board model, firmware env, the MCU's stable by-id path, the camera's DEV_ id - the script fills in. The declared half is the flags, and they are not a formality: the program degrades gracefully around missing hardware, so a part that was never fitted and a part that has failed look identical unless the file says which was expected. Declare what was assembled.

Check the result, especially the serial path:

ssh ggs@<tower-ip> 'cat /etc/fire_gimbal_control/hardware.ini'

Then prove the program finds it with nothing in the environment - no FGC_HARDWARE, no --hardware - because that is how the tower starts itself:

[INFO ] hardware: /etc/fire_gimbal_control/hardware.ini  board lattepanda-iota  axes yaw

Do not leave it anywhere under /tmp. That is tempting during bring-up because it needs no root, and it works right up until the first reboot, which clears /tmp and leaves the program with Config error: Camera.rgb.id is required - a message that points at the camera rather than at the file that disappeared. Verified on tower 121 by rebooting it.

The drivetrain block

install.sh now writes this block, one [Motor.<axis>] per declared axis — it has to, because the firmware refuses to move until it has been told these (see the config handshake in configuration.md). What it still cannot do is probe them: no reading of a board tells you its sense resistor.

So pass what you know and it writes the values; omit anything and it writes a commented ; TODO line naming what to measure:

./install.sh --stdout --axes yaw \
    --r-sense 0.075 --run-current 756 --homing-current 200 \
    --travel-deg 340 --steps-per-rev 1210778 --sw-mode 0

A TODO line is commented, not a placeholder value, and deliberately: these keys are numeric, and a non-numeric value makes the program refuse to start with a message about a config key. Left absent, it starts and names exactly which keys that axis is missing and that it will not move — the useful failure.

The block it writes looks like this:

[Motor.yaw]
r_sense_ohm       = 0.075   ; the resistor fitted to THIS driver board
run_current_ma    = 756     ; normal motion
homing_current_ma = 200     ; the endstop search only
travel_deg        = 340     ; mechanical angle between the hard stops

homing_current_ma is a safety setting, not a tuning one. Homing finds a stop by driving into it until the shaft falls behind; that only works while the stop is stronger than the motor. A high gear ratio can shear a screw-head stop straight past its mate - on tower 121 the axis ran 1.8 turns past the stop with the encoder never falling behind, at full torque. Declare a reduced search current wherever the stop cannot survive one.

Getting it wrong in either direction is visible: too strong and the stop yields again; too weak and the motor stalls on ordinary friction mid-sweep, trips the same latch, and records a phantom endstop. The firmware's steps_per_rev guard is what catches the second case, so it has to be set from the real travel.

The centre lives in config.ini, not here. install.sh cannot write it — hardware.ini is merged over config.ini and wins, so a shared key would silently discard whatever gimbal heading setup had saved. What it does instead is work the requirement out from steps_per_rev and print it as a NOTE in the block, because the default is wrong for any geared-up rig - and scripts/install-tower then sets it in config.ini when it is too small:
; NOTE: half this travel is 605389 counts, so config.ini needs
;   [Motor] yaw_center_count = 1000000    (or anything above 605389)
;   The 500000 default would put the negative end at -105389.
The range is centre ± half the travel, so a centre below half-travel puts the negative end under zero. On tower 121 the inherited 500000 homed to -104749..1104749; at 1000000 it homes to 395194..1604807. A warning names it after every home if it is still too small.

travel_deg is a nameplate fact, and the host divides the measured homing travel by it to derive [Motor] <axis>_counts_per_deg into config.ini. A rough angle gives a proportionally rough degree scale - 1 % out here is 1 % out on every bearing - until gimbal heading setup supersedes it with a landmark fit.

The startup log reports what the registers actually deliver, which is not always what was asked for:

yaw motor current: run 756 mA (GLOBALSCALER 80, IRUN 31), homing 189 mA (IRUN 7)
                   -- R_sense 0.075 ohm from hardware.ini [Motor.yaw]

There is no leaving the block out any more. The firmware holds no drivetrain values of its own: at boot every axis stops its driver, asks the host (NEED), and refuses to move until every value it asked for has arrived. An axis whose block is missing or incomplete is an axis that will not turn - by design, since a guessed value is how drivers get cooked and endstops sheared. (For a bench board with no host, the firmware's USE DEFAULTS command loads deliberately gentle limp-home values instead; test hardware motor fails on a tower running them.)

The kit writes eight keys per axis - the four above plus steps_per_rev, sw_mode, enc_deviation and has_encoder - each as a value or a ; TODO; --motor AXIS.KEY=VALUE gives any of them for any axis, where the older --r-sense-style flags cover the first axis only. The three speeds (scan_speed_deg_s, homing_speed_deg_s, center_speed_deg_s, all in degrees per second) are written as gentle starting values: on a tower that is being migrated, set the homing speed to what its endstop detection is proven at, as was done for tower 111 (section 10b), or a stall-detected endstop may behave differently.

If an axis declares a reduced homing current and the firmware has not adopted it, the host refuses to home rather than driving into a stop at full torque. The message names the skew; the fix is to flash the firmware, not to override it.

[Serial] device must be a /dev/serial/by-id/ path, never /dev/ttyACM0. Tower 121 has a 5G modem that claims three ttyUSB nodes, and enumeration order is not stable across reboots.

The by-id path changes when you flash the MCU, because the name comes from the running image: usb-MicroPython_Board_in_FS_mode_* before, a Pico/Arduino name after. scripts/install-tower re-probes after flashing and updates [Serial] device itself; by hand, re-run --probe after step 6 and fix the file if it moved.

The wiring block: `[Pins]`

The firmware is built from this section - which MCU pin reaches which driver board - by firmware/tools/gen_wiring.py, reading the tower's own /etc/fire_gimbal_control/hardware.ini when the firmware is built on the tower. There is no copy in the firmware and no fallback: without [Pins] the firmware build fails, naming what to add. Each kit's reference loom, with the table saying which wire goes where, is its pins.ini; install.sh --loom standard copies it in, and --pin KEY=N gives pins one at a time for a tower wired differently.

The generator refuses what the board cannot do rather than building it: on the RP2040, SPI pins outside SPI0's reach (SCK {2,6,18,22}, MOSI {3,7,19,23}, MISO {0,4,16,20}); on the Leonardo, any spi_* key at all (its SPI is fixed on the ICSP header) and the ICSP pins as chip-selects; duplicates; and a hardware.ini written for the other board. The running firmware reports the pins it was built with, and test hardware motor compares them with this section - so a flash built from an older copy shows up there.

6. Motor firmware `[verified]`

The firmware is built and flashed on the tower - it is the machine wired to the MCU - so PlatformIO is installed there, not on the dev box.

# tower: PlatformIO core, in its own venv under ~/.platformio. No sudo needed.
curl -fsSL https://raw.githubusercontent.com/platformio/platformio-core-installer/master/get-platformio.py -o /tmp/get-platformio.py
python3 /tmp/get-platformio.py
~/.platformio/penv/bin/pio --version      # check it actually runs

[verified] on tower 121: PlatformIO Core 6.2.0 on the system Python 3.14.4.

The installer ends with a FileNotFoundError traceback out of multiprocessing/util.py about a missing /tmp/.piocore-installer-* path. It is a temp-directory cleanup finalizer running after the work is done - cosmetic. Judge the install by whether pio --version answers.

deploy.sh looks for pio in the usual places, but a non-interactive ssh shell does not have ~/.platformio/penv/bin on PATH, so name it explicitly:

REMOTE_PIO=/home/ggs/.platformio/penv/bin/pio

Then, from the dev box, in firmware/:

# .deploy.env
REMOTE_HOST=ggs@10.11.12.121
REMOTE_DIR=/home/ggs/projects/fwt_2a/firmware
PIO_ENV=iota_rp2040          # leonardo on a Delta
SERIAL_PORT=/dev/serial/by-id/usb-...

./deploy.sh --compile-only     # build on the tower, flash nothing
./deploy.sh                    # build + upload
./deploy.sh --monitor          # watch it boot

deploy.sh syncs the tree and then runs firmware/tools/flash.sh on the tower, which is also what scripts/install-tower runs, locally: one implementation of the board-specific bootloader step for both. Stop the program first; it holds the serial port.

Confirm the pin map before flashing a board that is wired to motors. A chip-select on the wrong pin writes into a driver that was never meant to hear it. The build takes the pins from this tower's hardware.ini and fails without them - see "The wiring block" in section 5. The IOTA's reference loom is the one as built on tower 121, which is not the map proposed in the porting plan.

[verified] on both boards: the RP2040 on tower 121 (2026-09-23), the Leonardo on tower 111 (2026-09-24) - the Leonardo through its own 1200-baud touch, which flash.sh leaves alone.

Getting a factory board to accept its first flash

[verified] on tower 121, and this is the step that will bite you. These boards ship running MicroPython, and the normal upload does not work against it:

Forcing reset using 1200bps open/close on port /dev/ttyACM0
No new RPxxxx device found yet, waiting..          (x20, then it gives up)

What	Factory board (MicroPython)	Our firmware
1200-baud touch, i.e. a plain upload	no reset at all	resets, but reboots into the application
`picotool reboot -f -u`	refused: no reset interface	-
MicroPython REPL	works	n/a

So drop into MicroPython's own REPL and tell it to enter the bootloader. firmware/tools/flash.sh (which deploy.sh runs on the tower) now does this by itself when the port's by-id name says MicroPython_Board: ./deploy.sh with SERIAL_PORT set to that path should flash a factory board directly. [untested]

no factory board has been through that path yet; the manual way, verified on tower 121, is below. Write this to a file and run it with ~/.platformio/penv/bin/python (it has pyserial):

import serial, time
s = serial.Serial("/dev/serial/by-id/usb-MicroPython_Board_in_FS_mode_XXXX-if00", 115200, timeout=1)
s.write(b"\x03\x03"); time.sleep(0.3)        # Ctrl-C: interrupt whatever is running
s.write(b"import machine\r\n"); time.sleep(0.3)
s.write(b"machine.bootloader()\r\n")         # the port drops here - that is success

lsusb | grep 2e8a        # expect: 2e8a:0003 Raspberry Pi RP2 Boot

Then run ./deploy.sh and it flashes. A physical BOOTSEL press does the same job if you are standing at the board.

Every board needs exactly one physical BOOTSEL press in its life - the one that installs the first image carrying the BOOTSEL command (below). After that, updates are remote.

sudo ./install.sh --preconditions must have run first, or picotool finds the board and refuses it:

RP2040 device at bus 1, address 4 appears to be in BOOTSEL mode, but picotool
    was unable to connect. Maybe try 'sudo' or check your permissions.

Re-flashing later: the `BOOTSEL` command

[verified] on tower 121. Neither documented remote path works on this board - the 1200-baud touch resets the firmware but returns it to the application, and picotool reboot -f -u answers Unable to locate reset interface on the device. So the firmware carries a command that does it deliberately:

# over the serial port, or from the app: gimbal raw "BOOTSEL"
BOOTSEL   -> OK entering bootloader - reflash to return
lsusb | grep 2e8a        # 2e8a:0003 Raspberry Pi RP2 Boot
./deploy.sh              # flashes and reboots into the new firmware

The whole loop was run end to end from the dev box with nobody at the tower. BOOTSEL disables every axis first, since the bootrom does not run the control loop and unsupervised energised drivers would hold torque.

It takes the gimbal out of service and the firmware cannot undo it - once in the bootrom there is no command loop left to change its mind. Recovery is deploy.sh, or a power cycle: entering the bootrom erases nothing, so the existing image is still in flash. Anything that can write to the serial port can send it, including gimbal raw. Near misses (BOOT, BOOTSE) are rejected as unknown commands.

The identity changes when you flash

	by-id path	USB
before	`usb-MicroPython_Board_in_FS_mode_<serial>-if00`	`2e8a:0005`
after	`usb-Raspberry_Pi_Pico_<SERIAL>-if00`	`2e8a:000a`

So generate hardware.ini (step 5) after flashing, or update [Serial] device once the name has moved - scripts/install-tower does that itself, re-probing after every flash.

Verify the flash

Read the boot banner over the serial port:

BOOT build=aab1925-dirty
IOIN Y=0x3000001E version=0x30
ST Y:R,500000,18,80084000,0,8,8,Sd

version=0x30 is the TMC5160 answering over SPI - the single most useful line here, because it proves the pin map, the 3.3 V logic level and the driver board all at once. BOOT build= must match the commit you flashed. One Y line and no P line is a yaw-only build.

Flashing an RP2040 remotely is not the same as a Leonardo. The documented flow is hold BOOTSEL, tap RST, drag a .uf2 - a person at the board. The two remote paths (the 1200-baud touch that the Arduino tooling already uses, and picotool reboot -f -u) both need the running image to present a USB CDC port. That is why setup() brings Serial up before any hardware init: an image that hangs before USB comes up cannot be reflashed without a site visit, while an interrupted flash recovers by itself - the bootrom falls back to mass storage when flash holds no valid image.

7. Deploy the software `[verified]`

On the dev box, in software/:

# .deploy.env
REMOTE_HOST=ggs@10.11.12.121
REMOTE_DIR=/home/ggs/projects/fwt_2a/software
CMAKE_ARGS="-DWITH_MQTT=ON -DWITH_VIMBA=ON"
RUN_ARGS=""                 # extra flags only; homing and mode are config.ini [Startup]

./deploy.sh --check-deps    # must print "remote deps OK"
./deploy.sh                 # rsync + remote cmake + build

The first deploy copies config/config.example.ini to config.ini only if the device has none - see the warning in step 2 about a machine that arrived with somebody else's config. That copy still carries the example's placeholders and the keys hardware.ini owns; scripts/install-tower seeds it instead with build/fgc-config seed, which fills in the answers, comments out every key hardware.ini sets, and refuses to write a file the program would not load.

[verified] on tower 111: builds on GCC 11 / Ubuntu 22.04. [verified] on tower 121: builds clean on GCC 15 / Ubuntu 26.04 with a single -Wformat-truncation warning in utcStampNow(), which is a false positive (the compiler cannot prove tm_year is four digits; the real output is 17 bytes into a 32-byte buffer).

8. Camera profiles `[verified]`

Without a profile the program does not start - no camera profiles (*.xml) in ~/.config/fire_gimbal_control/camera_profiles/rgb is a startup error, by design. A fresh tower has an empty profile directory, so this step is not optional.

The documented way to make one is the VimbaX Viewer, which is a GUI - awkward on a headless tower and impossible unattended. scripts/export_camera_profile.py does the same job through the SDK's Python API:

# once per tower: the SDK ships the wheel
python3 -m venv ~/.venvs/vmbpy
~/.venvs/vmbpy/bin/pip install /opt/VimbaX/api/python/vmbpy-*.whl

cd ~/projects/fwt_2a/software
export GENICAM_GENTL64_PATH=/opt/VimbaX/cti
~/.venvs/vmbpy/bin/python scripts/export_camera_profile.py --profile standard --dry-run
~/.venvs/vmbpy/bin/python scripts/export_camera_profile.py --profile standard
~/.venvs/vmbpy/bin/python scripts/export_camera_profile.py --profile night

It writes each recipe value to the real camera in dependency order, lets the camera validate it, and exports what was actually accepted. That preserves the property the Viewer workflow exists to provide: GenICam validity is stateful, so a hand-written XML can be individually plausible and collectively wrong. Here the camera is the judge, and anything it refused is printed rather than assumed.

Read the summary line it prints - it is the number that decides whether the profile can be delivered at all:

result  : 2664x1520 RGB8  12.1 MB/frame  4.95 fps  = 60 MB/s

Profiles do not transfer between cameras

[verified] the hard way. The fleet runs several Alvium generations and their feature vocabularies differ, so copying another tower's XML does not work:

	Delta tower (1800 U-1240c)	IOTA tower (1800 U-1620c)
`SensorBitDepth`	`Bpp8`	no `Bpp8` - only `Adaptive` / `Bpp12`
`IntensityAutoPrecedence`	`Minimizeexposuretime` / `Minimizenoise`	`MinimizeBlur` / `MinimizeNoise`
Sensor	12 MP	16 MP, 5328x3040

Older GigE cameras (Manta) `[partial]`

Rietschen has three Allied Vision Manta G-1236 on one GbE port (enp1s0, MTU 8000): RGB (G-1236C), NIR and ACR (both G-1236B). Their firmware (00.01.54) predates the names the program and the Alvium use:

Program / Alvium	Manta
`ExposureTime`	`ExposureTimeAbs`
`AcquisitionFrameRate` (+ `…Enable`)	`AcquisitionFrameRateAbs` - which a free-running Manta runs at, up to `AcquisitionFrameRateLimit`
`IntensityControllerTarget` / `…Tolerance`	`ExposureAutoTarget` / `ExposureAutoAdjustTol`
`DeviceLinkThroughputLimit`	`StreamBytesPerSecond` (max 124 MB/s, shared by every camera on the port)
`BayerRG12p`	`BayerRG12Packed`: the GigE Vision 1.x packing, p0's high bits first - its own unpacking, see `RawImage.h`
chunk data, `DeviceReset`	none

The program looks a missing feature up under its older name, so test camera and the telemetry work. export_camera_profile.py picks its gige recipe by itself when a camera has ExposureTimeAbs and no ExposureTime: frame trigger off (the legacy cameras booted with TriggerMode = On, Line1), BayerRG8 on the colour camera (BayerRG12Packed, 12 bits, when it has it) and Mono8 on a mono one, 60 MB/s each, the frame rate at the camera's maximum (rietschen's RGB camera came set to 0.955 fps), and auto-exposure on the frame mean (ExposureAutoAlg = Mean) at 30 % +- 5 %, down to 20 us by day. The keep rule waits for the frame mean to reach the camera's own target and tolerance, read once the camera is open. With the RGB camera's FitRange, which fits the histogram wherever its mean lands, no waypoint settled. 30 rather than the Alvium's 50: at 50 the sky over the horizon burnt out at noon, where smoke would rise, and at 35 it was still too bright; the old program ran these cameras at 23-25.

Give --camera the id: with several cameras the first one the SDK lists is not the colour one, and the order changes from run to run. install.sh --probe lists them all, install.sh --hardware-ini writes CHOOSE_ONE without --camera-id, and install-tower asks (camera_id).

What is still missing: a Manta has no DeviceReset, so the program cannot reset a stuck camera (it says so and goes on); and without chunk data the keep rule cannot tell a dark scene at the exposure limit, so at night it keeps the newest frame after frame_timeout_ms.

[verified] on rietschen's NIR camera, 2026-10-20, with the motor mocked and the old program running beside it: test camera PASS (5 frames 4112x3008 Mono8 in 1.7 s, dark 100 % <= 20, bright mean 255), snapshot writes a 3008x4112 grey JXL (rotation_deg = 270, as the old program rotated), and a scan keeps a picture every 5 s without a timeout. The colour camera is not yet tested: the old program has it open.

Binning is not optional, and the frame-rate ceiling is not what it looks like

Unbinned RGB8 on the 16 MP sensor is a 48.6 MB frame - over the ~20 MB ceiling in known-issues #19. 2x2 binning brings it to 12.1 MB.

The frame rate is then capped by DeviceLinkThroughputLimit, but the limit is accounted against the unbinned payload, not the binned frame that crosses the wire. Measured on tower 121:

`DeviceLinkThroughputLimit`	max `AcquisitionFrameRate`	actual USB traffic
250 MB/s (the fleet's value)	4.95 fps	~60 MB/s
450 MB/s (this camera's max)	8.79 fps	~106 MB/s

48.6 MB x 4.95 = 240 MB/s, which is the 250 limit - so the camera is budgeting for data that binning discards. The recipe ships the conservative 250 MB/s, giving ~5 fps rather than the Delta's 10, because the USB stalls behind known-issues #19 are a host-side finding that has not been re-measured on the N150 yet (porting plan phase 5). Raising it is a one-line change in the recipe once that measurement exists.

Export standard after night, not the other way round, or use the current script. A refused write is not a no-op: the first version of this script asked for 10 fps, the camera refused, and the 1 fps left over from the night profile stayed - producing a "standard" profile that ran at night speed. The script now clamps an out-of-range numeric value to the camera's limit and says ~ ... (clamped).

9. `config.ini` - what this tower DOES `[partial]`

Device-local, never overwritten by a deploy. Edit it on the tower:

[General]
tower_name = <this tower>          ; drives every MQTT topic
[Network]
zkms_server_ip = <broker>

Credentials belong in /etc/fire_gimbal_control/credentials.env, not here.

Anything hardware.ini now owns - [Serial] device, [IMU], [Env], the camera ids - should not be in config.ini as well: the program warns at startup about every key set in both. scripts/install-tower comments them out (value and all), and build/fgc-config set config.ini --config config.ini --hardware /etc/fire_gimbal_control/hardware.ini --drop Section.key=note does one by hand. Every edit through fgc-config is loaded the way the program loads it before it is written, and keeps the previous value as a ;prev comment.

[Camera.<name>] id is required even when the camera is disabled. Config validation runs before any of the feature flags are consulted, so a tower with no camera fitted yet cannot start without one - and the obvious dummy, DEV_0000000000, is what deploy.sh's placeholder guard refuses. On a tower that has its camera, the id lives in hardware.ini and this never comes up.

[Features] enable_imu and enable_env are for towers without a hardware.ini. Where hardware.ini declares imu / env, the declaration decides, and a flag that contradicts it is warned about and ignored. Before that was so, tower 121's first clean-looking run "started" an IMU it does not have - /dev/ttyS0 accepts a serial open whether or not anything is attached - which is why a declared none has to be able to switch a sensor off by itself.

10. First run: bring the subsystems up one at a time `[partial]`

Do not start a new tower with everything switched on. Each phase below adds exactly one subsystem, so that a failure names its own cause instead of arriving as a wall of unrelated errors. It also keeps the motors - the only part that can break itself - until you have proved the boring parts work.

Each phase says what good looks like and what noise to expect.

Phase 0 - the host alone, no hardware `[verified]`

./build/fire_gimbal_control --mock-serial --mock-camera --no-mqtt --no-tui

Proves config resolution, hardware.ini layering, the single-instance lock, the scan grid and the output directory - with nothing physical involved. Neither a camera nor its profile XMLs need exist:

[INFO ] camera profiles not loaded: no real camera in this run

[Features] enable_camera = false does the same thing from the config file, for a tower that has no camera fitted yet, and says so rather than pretending:

[WARN ] camera disabled ([Features] enable_camera = false); using a mock source
        - no images come from hardware in this run

[INFO ] hardware: /etc/fire_gimbal_control/hardware.ini  board lattepanda-iota  axes yaw
[INFO ] Loaded config: config.ini
[INFO ] scan grid: 15 waypoints

Any config.ini still sets X, which hardware.ini overrides warning belongs here and should be cleared before moving on (step 9).

Phase 1 - motors `[partial]`

Real serial link, nothing else:

./build/fire_gimbal_control --mock-camera --no-mqtt --no-tui

What proves the link is a live DUMP from the board, not just an open port:

[INFO ] Serial controller started on /dev/serial/by-id/usb-Raspberry_Pi_Pico_...
DUMP BEGIN build=aab1925-dirty uptime=7925212 mcusr=0x3 free_ram=252712
DUMP Y state=1 ... speed=3000 center=500000 has_encoder=1

Check build= matches the firmware you flashed, and that the handshake completed - the log says firmware config: answered N keys for axis Y, and the axis leaves state 5 (NEEDCFG). The scripted form of all of this, which also compares every value the firmware holds with hardware.ini, moves nothing:

./build/fire_gimbal_control --no-mqtt --test hardware    # exit 0 PASS, 4 FAIL, 5 could not start

Then, and only then, move the thing:

gimbal home              # the endstop-finding sequence - see the warning below
gimbal diag y            # firmware self-test, reports PASS/FAIL
gimbal move 10,0         # a small deliberate move (a yaw-only tower: gimbal move 10)

Expected noise in this phase, neither of which is a fault:

Message	Why
`Motor controller: ERR unknown command`, a burst at startup	Only on firmware older than the config handshake: it does not know `SET`. Seen on tower 111 before its flash; harmless there, and gone once the firmware is current
`encoder tracking error on yaw: 500366 counts (limit 400)`	Before homing, the firmware presets `XACTUAL` to the centre count while the encoder sits at its own zero, so they disagree by the whole range. Homing resyncs them. If it persists after a successful home, that is real

On an unfamiliar drivetrain, homing is the dangerous step - it finds the endstop by driving into it. Start from a deliberately slow homing_speed_deg_s and walk it up, watching the free-running encoder gap that enc_deviation is measured against. Be at the rig with a hand on the power for the first one - which is why scripts/install-tower asks before it homes.

Phase 2 - camera `[verified]`

Add the camera, still no broker:

./build/fire_gimbal_control --no-mqtt --no-tui

[INFO ] Opened camera 'rgb' DEV_1AB22C0B7D4E (1800 U-1620c)
[INFO ] camera profile 'standard' applied from .../standard.xml (17206 settings)

Then mode auto, and confirm .jxl files appear under the configured output dir.

A camera-free phase needs no profile XMLs - but this only became true on 2026-09-16. Before that, [Features] enable_camera was parsed and never used, and the profile directory was validated even for a mock camera, so a tower whose camera had not been configured yet could not start at all. If you are running an older binary, export the profiles (step 8) first or it will refuse with no camera profiles (*.xml).

Phase 2b - thermal camera

Only for a tower with an InfraTec TarisIR mini (tower 111 since 2026-10). The program runs as the tower's user; the SDK needs three things from the system.

SDK and licence. Packages infratec-sdk-irbgrab and infratec-itcamtarisirmini (amd64), then the licence for the camera's serial via the vendor's copycustomerdata.sh into /var/lib/infratec/.license/ (root:infratec, rw-rw----). Repeat after every OS update. Keep the source files on the tower (~/InfraTec/LINUX/ on 111) so it can be re-run. The tower's user must be in group infratec (id shows it; sudo usermod -aG infratec ggs and log in again otherwise). The library must be in the ld cache: ldconfig -p | grep irbgrab.
Network. Camera and port directly cabled, static address, no DHCP (dhcp4: false in netplan: DHCP on a link without a server keeps failing and takes the static address with it), MTU 1500 - the camera does no jumbo frames. The SDK finds the camera by serial and MAC on raw Ethernet; the IP only matters to ordinary GigE Vision tools. On 111: enp1s0, 192.168.0.111, camera 192.168.0.200.
cap_net_raw on the binary. The SDK opens a raw AF_PACKET socket. Rather than running the program as root, the binary gets this one capability. A rebuild drops it, so deploy.sh and install-tower run scripts/fgc-setcap --if-thermal after every build. Once per tower, let deploy do that without a password:
```
scripts/fgc-setcap --install-sudoers   # /etc/sudoers.d/fire-gimbal-control-setcap
scripts/fgc-setcap && getcap build/fire_gimbal_control
```
A capability binary runs in the loader's secure mode (LD_* ignored, no core dumps): check once that Vimba X still opens the colour camera, and that scripts/fgc status and stop still find the process.

Then hardware.ini (driver = taris, id = <serial>, interface = enp1s0, cameras = rgb, tir) and config.ini (label = TIR, see configuration.md), and a run with the real cameras but no motors:

./build/fire_gimbal_control --mock-serial --mock-env --no-mqtt --no-tui

[INFO ] IRBGRAB SDK 4.3.44
[INFO ] thermal camera connected via SN:2017744|...|192.168.0.200#0
[INFO ] thermal camera: profile 0, 0.01 K per count

snapshot then writes RGB/<t>.jxl and TIR/<t>.jxl with the same <t>. camera calib reports the profile. If the camera stays offline the log says why, once per change - no cap_net_raw, not found (found: ...), the port's state and MTU. A 0x80000017 that keeps coming is the licence before it is the NUC.

Measured on tower 111 (2026-10-14, SDK 4.3.44, SN 2017744):

cap_net_raw alone is enough; no root, no cap_net_admin. Vimba X still opens the colour camera from the capability binary.
Connecting takes about 15 s (SDK init and search); the scan does not wait.
Framerate_Hz is settable. The LUT is empty for the first frame after connecting and right from then on: 0.0099998 K/count (profile 0), 0.0199995 (profile 1).
A profile switch takes about 3 s; frames keep the old scale until Calib_ChangeInProgress reaches 0, and the program takes none meanwhile.
The camera's frame counter (ImageNum) is always 0; MissedPackets was always 0 on the direct link.
Frame rate (2026-10-15): from power-up the camera sends 50 fps, 31 MB/s and 22 000 packets a second. Framerate_Hz set right after connecting answers ok and changes nothing; set while grabbing it only thins out what the SDK hands over; set while the grab is stopped, after the stream has run, it holds on the wire. The program does that once the first frame is in and checks the packet rate of [Camera.tir] interface (5 Hz: about 2 200 packets a second); the log says 5.3 fps on enp1s0 (2224 packets/s), set on attempt 2.
NUC (2026-10-15): the camera runs one every 30 s (NUC_Interval) and sends no frame for 650-760 ms meanwhile. NUC_Interval = 0 stops that; NUC_Perform (1-point) runs one and returns when it is done, after about 0.67 s. NUC_INPROGRESS stays 0 throughout, also for those. The program now runs the NUC itself at the start of a move (nuc_interval_s).
The SDK switches the process's C locale to LC_NUMERIC=de_DE when it is loaded. The program puts it back after every SDK call and writes its JSON without the C library's number formatting.

Phase 3 - the broker

Set [Features] enable_mqtt = true and point zkms_server_ip at the broker. Without one, the app says so and carries on:

[ERROR] MQTT connect failed (tcp://10.11.12.121:1883): TCP/TLS connect failure
[WARN ] Control channel not connected; continuing in degraded mode

Degraded mode is not a failure to notice later - the tower captures images and reports to nobody. Confirm topics arrive at the broker before calling the phase done.

Each tower runs its own RabbitMQ, in Docker, with the rabbitmq_mqtt and rabbitmq_shovel plugins - the shovel is what forwards this tower's messages to the ground station. That lives in ~/zkms-client/, outside this repository.

A legacy tower (DHT11 and fan on the MCU)

Answer the installer's ambient sensor question with dht11 and the fan question with yes, and choose loom = legacy. The firmware is then built with the DHT11 and fan (see configuration.md, "The legacy towers").

The encoder scale. Take it from the survey's "old firmware's source": R3A is ENC_CONST, the whole part in the high 16 bits and 1/10000ths in the low 16, so 0x000C1388 is 12.5. If it is not 12.8, put it in the answers file as yaw.enc_const=12.5 (the installer never asks for it). On staeffelsberg, 2026-09-28, it was missing: homing took the 2% drift for a stop after 131°.

Check it on site after the first flash:

The fan runs at full speed from power-up.
It drops once readings come in, if the box is below 25 °C.
Warming the DHT11 by hand, or with a hair-dryer, spins it up.
test hardware env passes, fan leaf included.

Phase 4 - IMU and ambient sensor

Only if fitted, and declared in hardware.ini (imu, env) - the declaration is what switches them on. test hardware imu wants a 0xFA-framed stream from the MTi; test hardware env wants a real I2C adapter and a CRC-valid SHT41 reading. [verified] on tower 111, 2026-09-23: both.

Phase 5 - unattended

Only once every phase above is green:

# config.ini: [Startup] home = true, mode = auto
scripts/fgc start             # detached, inside tmux
scripts/fgc status            # running? since when? which PID?

Close the ssh session, reconnect, and check it is still running. Then install the boot service (scripts/install-tower asks; deployment.md section 4), reboot the tower, and confirm with scripts/install-tower --verify that the USB preconditions survived and the program came back by itself.

Calibration

Per tower, and cannot be inherited - least of all from a tower with a different gear ratio, where yaw_counts_per_deg is derived from a steps_per_rev that no longer applies:

gimbal heading setup     # ties yaw to true north by sighting landmarks; no IMU needed
gimbal pitch setup       # needs an IMU and a pitch axis; skip on a rig without them

10b. Bringing up the config handshake on an existing tower

This is the one upgrade that changes what an unconfigured tower does: the new firmware boots into NEEDCFG and refuses HOME / MOVE / ENABLE until the host has answered its NEED lines. Flashing it onto a tower whose hardware.ini predates the drivetrain keys leaves the gimbal inert — working, talking, and not turning.

So the order matters. Host first, firmware second. Each step below either changes nothing or is reversible, and the step that could strand the tower comes after the one that prevents it. scripts/install-tower follows this order by itself, and blocks the flash while any drivetrain value is missing.

The worked example: tower 111 `[verified]`

The Delta tower, which predated hardware.ini, the handshake and the kits. 2026-09-23 and -24, through the installer with the Delta kit:

A rollback copy of the sources, the running binary and config.ini, in ~ggs/rollback-20260923/, before anything changed.
The host half. Preconditions, hardware.ini with every drivetrain value a ; TODO, config.ini's duplicated keys commented out, the missing night profile, a new host. test hardware: axes, IMU, env and camera PASS; motor FAIL on the old firmware's missing echoes, as expected (step 2 below). Firmware blocked. It homed and resumed capturing on the new host.
The drivetrain, without inventing it. The sense resistors were read off both boards (75 mOhm). Everything else was converted from what the old firmware ran - its revision is in DUMP BEGIN build=, and git history at that revision has its register values - through the host's own planners, and round-tripped back to the identical registers before being written. Homing and centring speeds were set to what that firmware used, not the kit's gentle defaults, since its endstop detection was proven at those.
The flash, by the installer: every gap filled, the firmware built from hardware.ini and flashed, test hardware green on every leaf, and homing within 0.03 % of the old firmware's travel.

0. Record what you are rolling back to

gimbal dump                 # the running firmware's build id and register state
cat /etc/fire_gimbal_control/hardware.ini
git -C ~/projects/fwt_2a/firmware rev-parse --short HEAD

Keep the dump. It is the only record of what the old firmware was holding, and the comparison that tells you whether the new one adopted the same values.

1. Update `hardware.ini` — nothing else, and no restart yet

Add the [Motor.<axis>] block (section 5) and [Pins]. The firmware asks for eleven values per axis - steps_per_rev, global_scaler, run_irun, homing_irun, center_count, soft_margin, enc_deviation, normal_speed, homing_speed, center_speed, sw_mode - and the host derives them from the block (currents from milliamps and the sense resistor, speeds from degrees per second, centre and margin from config.ini). install.sh --stdout generates the block; for values nobody has measured, convert what the old firmware runs, as for tower 111 above.

Doing this before flashing is what keeps the window at zero: the host is ready to answer the moment the firmware starts asking.

2. Deploy the new host against the OLD firmware

A real test, and it risks nothing. The old firmware never sends NEED, so the host must behave exactly as it did before — which is also the guarantee the whole Delta fleet depends on.

scripts/fgc start
scripts/fgc peek
test hardware motor

Expect: no handshake lines at all (and possibly a burst of ERR unknown command - the old firmware refusing SET), and the motor leaf failing on missing echoes, reported as a version skew rather than a mismatch:

; yaw_pins: yaw: the firmware does not report the pins it was built with, so its
  wiring cannot be confirmed -- it predates the pin echo. Re-flash it

That FAIL is by design - a check that could not happen must not read as a pass - so on old firmware the gate is "every other leaf green, and hw_motor failing only on missing echoes". A speed can also differ before the first home, since the host sends it after homing. If instead you see wrong values for something the firmware does echo, stop: hardware.ini disagrees with what the tower is really running, and flashing will not fix it.

3. Flash

cd firmware && ./deploy.sh          # uploads; --compile-only if you want a dry run first

Watch the banner. A NEED line per axis should follow it:

BOOT build=<rev>
NEED Y steps_per_rev global_scaler run_irun homing_irun center_count soft_margin ...

No NEED line means the axis is not in NEEDCFG — either the flash did not take, or the image is older than you think. Check the build id before going on.

4. Confirm the handshake completed

The host answers within a control-loop tick. Expect:

firmware config: answered 11 keys for axis Y

Then prove it landed rather than trusting the log:

gimbal dump                 # state should be 1 (RESET), NOT 5 (NEEDCFG)
                            # cfg_rx should be the full required mask
test hardware motor         # both metrics PASS

test hardware motor is the real gate: it compares every declared value against what the firmware echoes, so a SET that was lost on the wire shows up here and nowhere else.

5. Prove the refusal actually refuses

Worth doing once per firmware version, because a guard that silently stopped working looks exactly like a guard that is working.

On a bench board, not a mast: power-cycle it without the host running, then send HOME over a serial monitor. Expect ERR need config. Send RESET and try again — still ERR need config, because a RESET is deliberately not a way out. USE DEFAULTS is, and should then let HOME through.

6. Only then, move the axis

gimbal home                 # at the reduced homing current

Check against the recorded figures: travel ~1,210,778 counts, repeatable to ~0.3 %, and the derived counts_per_deg near 3552.87. A travel that disagrees by more than a few percent means the axis homed against something that was not the endstop.

Reading the log afterwards

Everything below is recorded to ~/.local/share/fire_gimbal_control/logs/fgc_*.log, not only to the dashboard — which matters here, because in TUI mode the pane holds 500 lines and dies with the tmux session. During this bring-up that cost us the handshake record four separate times before the file existed.

L=$(ls -t ~/.local/share/fire_gimbal_control/logs/fgc_*.log | head -1)
grep -E "NEED|firmware config|need config" "$L"   # step 4: did the handshake complete
grep -E "hardware:"                        "$L"   # presence reconciliation
grep -E "speed:|motor current"             "$L"   # what was actually sent to the firmware

The file is more verbose than the dashboard was, so it holds lines the pane never showed. head -8 gives the build, tower and config that produced it.

If it goes wrong

The board is recoverable over the network as long as USB CDC comes up: BOOTSEL over the serial port, then ./deploy.sh with the previous firmware checked out. Only an image that wedges before Serial.begin() needs a physical button press — which is why Serial.begin() is the first thing setup() does.

USE DEFAULTS is the in-place escape: it puts every axis on the compile-time table and lets the tower move while you work out what hardware.ini is missing.

11. Verification checklist

Most of this is one command, which changes nothing:

scripts/install-tower --verify

It checks the USB preconditions, that config.ini and hardware.ini load together, that hardware.ini has no gaps, that the firmware reports the pins it declares, and the boot service - and runs test hardware and test camera: inside the running program when it is up (pausing capture, then resuming it), or as a one-shot run when it is not. By hand, the one-shot form is:

GENICAM_GENTL64_PATH=/opt/VimbaX/cti ./build/fire_gimbal_control --no-mqtt --test hardware
echo $?      # 0 PASS, 4 FAIL, 5 could not start (the log says why)

It waits for the firmware to be talking and configured before it looks, so a freshly flashed board is not failed for values the host had not sent yet.

Check	How	Tower 121 (IOTA)	Tower 111 (Delta)
Key-only ssh	`ssh ggs@<tower> hostname`, no prompt	✅	✅
Preconditions applied	`--verify`: `usbfs_memory_mb 1000`, camera LPM `0`	✅	✅ 2026-09-23 (were 16 and `u1_u2`)
Preconditions survive a reboot	reboot, then `--verify`	✅ confirmed across a real reboot	✅ 2026-09-24
`hardware.ini` found unaided	startup names `/etc/fire_gimbal_control/hardware.ini`	✅	✅
Declared hardware matches	startup logs `declared assembly confirmed`; `test hardware axes`	✅ yaw	✅ yaw, pitch
IMU / ambient sensor	`test hardware imu` / `env`	none declared	✅ MTi streaming, SHT41 CRC-valid on a real I2C bus
Camera enumerates	`ListCameras_VmbC` reports the `DEV_` id in `hardware.ini`	✅ `DEV_1AB22C0B7D4E`	✅ `DEV_1AB22C049CD1`
Camera profile applies	startup logs `profile 'standard' applied ... (N settings)`	✅ 17206 settings	✅ 3743 settings
Camera works, whatever it sees	`test camera`: whole frames; every pixel forced to black (minimum exposure and gain); the sensor responding at maximum gain and a long exposure; the profile restored	✅ 2026-09-24, lens covered: 100 % ≤ 20 dark, bright mean 0 → 255	✅ 2026-09-24, inside the running program: 5 frames of 4024x3036 in 9.2 s (a dark scene, ~2 s exposures), 100 % ≤ 20 dark (mean 5.6), bright mean 5.6 → 255; capture resumed. An earlier run correctly reported the camera delivering nothing at all - known-issues #15
Firmware identity	`DUMP BEGIN build=` is the revision flashed	✅	✅ `a0b1e4a`
Driver answers over SPI	`IOIN` reads version `0x30` for every axis	✅	✅ both axes
Firmware holds `hardware.ini`	`test hardware motor`: values, current, pins	✅	✅ every leaf, 2026-09-24
Remote re-flash	no hands on the board	✅ `BOOTSEL`	✅ 1200-baud touch
Homing repeatable	endstop counts across homes	✅ three homes within 0.27 %	✅ within 0.03 %, old and new firmware
Endstop survives homing	the axis must stall against the stop, not push through	✅ at 189 mA, after failing at full torque	not inspected; full current, as for its whole service life
Degree scale derived	startup logs `counts_per_deg` agreeing with the homed travel	✅ 3552.87	✅ agrees, calibration left alone
Images land	`.jxl` files appear under the configured output dir	❌ needs a capture run	✅
Starts at boot	reboot with nobody logged in; `scripts/fgc status` from a later ssh	✅ lock under `/run/user/1000`	✅ 2026-09-24: homed and captured by itself within ~35 s. The Delta logs a desktop session in at boot, so lingering was not what put the lock in `/run/user` here
Reports to a broker	topics arrive at the configured broker	❌ no broker on this tower yet	✅ connects to its own RabbitMQ (retrying until Docker has it up at boot); forwarding to the ground station not checked

60 KiB Raw Permalink Blame History

Installing a tower

The short way: scripts/install-tower

Before starting

Getting the code onto a tower: scripts/bootstrap-tower

A legacy tower: read it first, with scripts/survey-legacy-tower