fwt_software/docs/tower-installation.md

# Installing a tower

How a fire-watch tower host goes from a freshly imaged machine - or from an
older, unmigrated setup - to a running, calibrated `fire_gimbal_control`.
[deployment.md](deployment.md) is what comes after: updating and operating a
tower that already works.

**`scripts/install-tower` does the whole of this.** The short way is the next
section. Everything after it, sections 1 to 11, is the reference for what each
step does and why, written while doing each one by hand, and still the way to
do one step on its own.

**Status legend** - each step says where it stands:
`[verified]` run end to end on a real tower and the result checked ·
`[partial]` run, with something still open · `[untested]` written from the
design, not yet exercised.

---

## The short way: `scripts/install-tower`

### Before starting

| | |
|---|---|
| The tower host | Powered, on the network, an OS installed, `ssh` running, reachable by key (section 1) |
| An account | `ggs`, in the `sudo` and `dialout` groups |
| Both repositories on it | `~/projects/fwt_2a/software` and `~/projects/fwt_2a/firmware`. `scripts/bootstrap-tower` sets that up from one copied file (below), or each repo's `deploy.sh` syncs them from a dev box |
| The Vimba X SDK, staged on it | Login-walled, so it cannot be fetched by a script: copy it on (USB stick, `scp`) and give its directory when asked. `none` for a tower with no camera yet |
| The facts of the build | Which axes, which IMU, which ambient sensor, how the driver boards are wired, and - per axis - the drivetrain: sense resistor, currents, travel, `steps_per_rev`, `sw_mode`, `enc_deviation`, encoder or not. Anything not yet measured can be answered `unknown` |

The installer takes the board from DMI and picks its kit:

| Board | MCU | Setup kit | PlatformIO env |
|---|---|---|---|
| LattePanda 3 Delta | ATmega32U4 (Leonardo), 5 V | [`hardware/lattepanda-3-delta/`](../hardware/lattepanda-3-delta/) | `leonardo` |
| LattePanda IOTA | RP2040, 3.3 V | [`hardware/lattepanda-iota/`](../hardware/lattepanda-iota/) | `iota_rp2040` |

### Getting the code onto a tower: `scripts/bootstrap-tower`

One file to copy; it fetches the rest. It needs the Gitea deploy key (tower
111's `~/.ssh/id_ed25519_deploy_gitea`) in the tower's `~/.ssh/`:

```bash
scp scripts/bootstrap-tower ggs@<tower>:
scp -3 ggs@10.11.12.111:.ssh/id_ed25519_deploy_gitea ggs@<tower>:.ssh/   # -3: via this machine
ssh -t ggs@<tower> bash bootstrap-tower          # any install-tower options pass through
```

What it does:
- adds `Host gitea` (port 2222) to `~/.ssh/config` and records the server's
  host key;
- clones both repositories side by side, or brings existing clones up to date
  (fast-forward only);
- starts `install-tower`.

It stops, rather than overwrite, when it finds a checkout with local changes or
a directory that isn't a clone.

### A legacy tower: read it first, with `scripts/survey-legacy-tower`

The legacy towers run an older host program and a TMC-API Arduino sketch.
Before migrating one, record what it runs, so its `hardware.ini` comes from its
own values:

```bash
bash survey-legacy-tower --firmware      # pauses the old service ~20 s; changes nothing on disk
```

**What it records:**
- the machine: board, OS, groups, disk, whether Gitea, GitHub, PyPI and
  PlatformIO are reachable;
- the camera and the Vimba install;
- the old service and its `config.ini`;
- with `--firmware`, from the old firmware:
  - the endstop travel (`?`);
  - its stored north calibration (`y`);
  - its TMC registers (`i`), with the run current worked out for the usual
    sense resistors.
- the old firmware's source, from the newest build in the Arduino cache:
  its register table (encoder scale `R3A`, current `R0B`/`R10`, encoder limit
  `R3D`) and homing speed. The running firmware cannot report these: the
  registers are write-only, and it answers `i` for them with 0.

The report is saved as `~/legacy-survey-<date>.txt`.

### Running it

On the tower, as `ggs`:

```bash
cd ~/projects/fwt_2a/software
scripts/install-tower --dry-run     # survey + questions + the plan; changes nothing
scripts/install-tower               # the same, then does it after one yes
scripts/install-tower --verify      # the checks only; changes nothing
```

It surveys the machine, asks only what it cannot work out, prints a plan with
each step marked `skip`, `create`, `change` or `blocked`, and applies it after
one yes. In order:

1. packages, then the USB preconditions (section 3), then the Vimba X SDK (4)
2. builds the host
3. writes `hardware.ini` (5) - or, when one exists, only fills its gaps
4. seeds `config.ini` (9) and `credentials.env`, or edits only what is out of date.
   This is where it asks what the tower does as it starts - `[Startup] home` and
   `mode` - and where a tower whose boot service still passes the old `--init` /
   `--start` has them moved into `config.ini`
5. PlatformIO, then the camera profiles that are missing (8)
6. builds and flashes the firmware (6) - **blocked** while any drivetrain value
   is still a `; TODO`, because that axis would refuse to move
7. `test hardware`
8. the boot service ([deployment.md](deployment.md) section 4), if wanted. It
   passes no startup flags; a new tower gets `[Startup] home = true`,
   `mode = manual`: it homes at every boot, then **waits in manual and captures
   nothing** until someone chooses auto
9. asks whether someone is at the rig; only on a yes does it home the gimbal and
   take **one** picture (a snapshot), then leave it in manual - through the
   service, when `[Startup] home` has the service home it, and otherwise with
   `gimbal home`, leaving `config.ini` as it is

**Nothing starts capturing by itself.** Choose auto yourself: key `a` on the
dashboard, or `mode auto` over MQTT. Once the tower is proven, make it boot into
auto, as 111 does:

```bash
scripts/install-tower --answer startup_mode=auto
```

That writes `[Startup] mode = auto` into `config.ini`; editing the file by hand
does the same (with the program stopped - see the warning at the top of the
file). `--answer startup_home=y|n` sets homing the same way. `service_args` now
holds only other flags, and refuses `--init` / `--start`.

Every question has a key, so any can be answered up front - `--answer
tower_name=north`, `--answer yaw.r_sense_ohm=0.075` - or from a file of
`key=value` lines with `--answers FILE`. `--non-interactive` stops instead of
prompting. **It never invents a drivetrain value**: a wrong sense resistor cooks a
driver and a wrong homing current shears an endstop, and both have happened on
this project. Answer `unknown` and it writes a commented `; TODO` naming what to
measure.

Re-running it is safe, and is how a configured tower is repaired: an existing
`hardware.ini` or `config.ini` is never regenerated, only filled in; an existing
camera profile is never overwritten; a firmware already running the declared
revision and pins is not reflashed; and if it fails after stopping the program,
it puts the program back the way it was.

| Where | Result |
|---|---|
| Tower 121 (IOTA), 2026-09-23 | `[verified]` re-run on a configured tower: changed only what was out of date, `--verify` green, boot service installed and proven across a reboot |
| Tower 111 (Delta), 2026-09-23/24 | `[verified]` migration of a tower that predated all of this: host first, firmware once the drivetrain was known, `test hardware` green on every leaf (section 10b) |
| A freshly imaged board | `[untested]` - rehearsed with `--dry-run` and typed answers only. The first flash of a factory MicroPython RP2040 is automated but has not met a factory board |

---

## 1. SSH by key `[verified]`

Everything after this is driven over ssh from the dev box, and `deploy.sh` runs
several ssh commands per invocation, so password auth is not workable.

```bash
# on the dev box
ssh-copy-id -i ~/.ssh/id_ed25519_ggs_pedro.pub ggs@<tower-ip>
```

Then give the host a block in `~/.ssh/config`, so every tool picks the right key
without being told:

```
Host 10.11.12.121
    User ggs
    IdentityFile ~/.ssh/id_ed25519_ggs_pedro
```

Verify with `ssh ggs@<tower-ip> hostname` - it must not prompt.

> **`ssh-copy-id` needs a terminal.** Run it in a real shell, not through a tool
> that gives it no tty: without one, ssh looks for a GUI `ssh-askpass`, and on a
> machine that has none it fails with `ssh_askpass: exec(...): No such file or
> directory` and then `Permission denied` - which looks like a rejected key
> rather than a prompt that never appeared.

## 2. Survey the machine before changing it `[verified]`

The setup kit can describe the machine without touching it, and this is the
cheapest moment to notice that the tower is not what you assumed:

```bash
rsync -a hardware/ ggs@<tower-ip>:~/projects/fwt_2a/software/hardware/
ssh ggs@<tower-ip> '~/projects/fwt_2a/software/hardware/lattepanda-iota/install.sh --probe'
```

```
board            : lattepanda-iota
os               : Ubuntu 26.04.1 LTS (7.0.0-31-generic)
motor MCU        : /dev/serial/by-id/usb-MicroPython_Board_in_FS_mode_df61a863dfa02932-if00
camera DEV_ id   : NOT FOUND (needs the Vimba X SDK installed)
alvium on usb    : 1 device(s)
usbfs_memory_mb  : 16 (want >= 1000)
camera lpm_permit: u1_u2 (want 0)
```

Read every line. On tower 121 this pass found that the MCU was still running the
MicroPython image it shipped with, and that both USB preconditions were at their
(wrong) defaults.

**Check what is already in `~/projects/`.** A machine cloned from another tower,
or one that has been used for development, can arrive with a `config.ini` that
belongs to a different rig. Tower 121 had a byte-identical copy of the dev box's
`config.ini` - `tower_name = dev`, broker `127.0.0.1` - which is worse than no
config at all, because **none of those values trip the placeholder guard in
`deploy.sh --run`**. That guard only catches `ExampleTower`, `CHANGE_ME` and
`DEV_0000000000`.

```bash
ssh ggs@<tower-ip> 'grep -E "tower_name|zkms_server_ip|^id" ~/.config/fire_gimbal_control/config.ini'
```

## 3. Host preconditions and build dependencies `[verified]`

Both come from the board's setup kit, in one step, on the device:

```bash
ssh ggs@<tower-ip>
cd ~/projects/fwt_2a/software/hardware/lattepanda-iota
sudo ./install.sh --deps --preconditions
```

`--deps` installs [`packages.txt`](../hardware/lattepanda-iota/packages.txt): the
toolchain, OpenCV, Boost, libjxl, tmux, and the Python bits PlatformIO needs.

`--preconditions` installs two files and applies them immediately:

| File | Why |
|---|---|
| `/etc/udev/rules.d/99-fwt-alvium-usb.rules` | Disables USB3 hardware LPM on the camera's port, which the host otherwise re-enables on every enumeration. With U1/U2 on, the camera logs link errors and frame delivery stalls (known-issues #19) |
| `/etc/tmpfiles.d/10-fwt-usbfs.conf` | Raises `usbfs_memory_mb` from 16 to 1000 at every boot. A single full-resolution frame does not fit in 16 MB |

Both are settings that **revert on reboot** if nobody persists them, which is the
whole of known-issue #19. The script re-reads them after applying and says so:

```
>> usbfs_memory_mb = 1000
>> camera USB3 LPM disabled (2-1)
```

If it instead warns that a value did not take, stop and find out why - a camera
on a host with these wrong looks like a flaky camera.

Confirmed on tower 121 (2026-09-15): both went from wrong to right in one run,
`usbfs_memory_mb` 16 -> 1000 and `lpm_permit` `u1_u2` -> `0`. Re-check after the
first reboot, since surviving one is the entire point.

> **Neither step can be driven from a tool without a terminal.** `sudo` needs a
> tty to prompt, so this runs from a real terminal window:
>
> ```bash
> ssh -t ggs@<tower-ip> 'cd ~/projects/fwt_2a/software/hardware/lattepanda-iota && sudo ./install.sh --deps --preconditions'
> ```

> **The Delta kit carries the same two files** (`hardware/lattepanda-3-delta/`,
> as symlinks to these). Tower 111 had neither - `usbfs_memory_mb` 16, camera
> LPM `u1_u2`, checked 2026-09-15 and again 2026-09-23 - and got both on
> 2026-09-23, reading 1000 and `0` afterwards - and still after a reboot on
> 2026-09-24.

## 4. Vimba X SDK `[verified]`

Proprietary and login-walled, so it is staged from a machine that already has it
rather than downloaded on the tower:

```bash
# dev box
rsync -az ~/Programs/VimbaX_2025-3 ggs@<tower-ip>:~/
# tower
sudo ./install.sh --vimba ~ggs/VimbaX_2025-3
```

That copies it to `/opt/VimbaX_<version>`, points `/opt/VimbaX` at it, runs the
SDK's `Install_GenTL_Path.sh` (which registers `GENICAM_GENTL64_PATH` in
`/etc/profile.d` and writes the `1ab2` udev rule), registers `api/lib` with the
dynamic loader, and then asks the SDK to list cameras as a check.

**The SDK asks for a reboot before the transport layers are usable.** If it
finds nothing, reboot and try again before suspecting the camera. On tower 121
no reboot was needed - the camera answered immediately.

Two things about that check, both found the hard way here:

- **The binary is `ListCameras_VmbC`**, in `/opt/VimbaX/bin/`. There is no plain
  `ListCameras`.
- **Run it with `GENICAM_GENTL64_PATH` set**, or it fails with `Could not start
  system. Error code: -16` on a perfectly good installation - that is "no
  transport layer found", not "no camera":

  ```bash
  GENICAM_GENTL64_PATH=/opt/VimbaX/cti /opt/VimbaX/bin/ListCameras_VmbC
  ```

> **The SDK ships a camera simulator, and it is on by default.** On tower 121
> `ListCameras_VmbC` reported **four** cameras: the real Alvium plus three
> synthetic ones from `VimbaCameraSimulatorTL.cti`, with ids like `DEV_Cam1`. A
> tower configured with one of those would open its "camera" happily and capture
> synthetic images forever. Take the id only from a block whose transport layer
> path is `VimbaUSBTL` or `VimbaGigETL` - which is what `install.sh --probe`
> does, after this bit me.

> Non-interactive ssh shells do not source `/etc/profile.d`, which is why
> `deploy.sh` resolves `GENICAM_GENTL64_PATH` itself before launching the app.
> Getting this wrong shows up as `VmbStartup()` failing with *"Could not start
> Vimba X API"* even though the viewer works when you are logged in.

## 5. `hardware.ini` - what this machine IS `[verified]`

```bash
sudo ./install.sh --hardware-ini --axes yaw --imu none --env none --cameras rgb
```

This writes `/etc/fire_gimbal_control/hardware.ini`, **outside** the directory
`deploy.sh` rsyncs, so no deploy can overwrite or delete it.

The probed half - board model, firmware env, the MCU's stable `by-id` path, the
camera's `DEV_` id - the script fills in. The declared half is the flags, and
they are not a formality: the program degrades gracefully around missing
hardware, so **a part that was never fitted and a part that has failed look
identical** unless the file says which was expected. Declare what was assembled.

Check the result, especially the serial path:

```bash
ssh ggs@<tower-ip> 'cat /etc/fire_gimbal_control/hardware.ini'
```

Then prove the program finds it **with nothing in the environment** - no
`FGC_HARDWARE`, no `--hardware` - because that is how the tower starts itself:

```
[INFO ] hardware: /etc/fire_gimbal_control/hardware.ini  board lattepanda-iota  axes yaw
```

> **Do not leave it anywhere under `/tmp`.** That is tempting during bring-up
> because it needs no root, and it works right up until the first reboot, which
> clears `/tmp` and leaves the program with `Config error: Camera.rgb.id is
> required` - a message that points at the camera rather than at the file that
> disappeared. Verified on tower 121 by rebooting it.

### The drivetrain block

**`install.sh` now writes this block**, one `[Motor.<axis>]` per declared axis —
it has to, because the firmware refuses to move until it has been told these
(see the config handshake in [configuration.md](configuration.md)). What it still
cannot do is *probe* them: no reading of a board tells you its sense resistor.

So pass what you know and it writes the values; omit anything and it writes a
commented `; TODO` line naming what to measure:

```bash
./install.sh --stdout --axes yaw \
    --r-sense 0.075 --run-current 756 --homing-current 200 \
    --travel-deg 340 --steps-per-rev 1210778 --sw-mode 0
```

A `TODO` line is **commented, not a placeholder value**, and deliberately: these
keys are numeric, and a non-numeric value makes the program refuse to start with
a message about a config key. Left absent, it starts and names exactly which keys
that axis is missing and that it will not move — the useful failure.

The block it writes looks like this:

```ini
[Motor.yaw]
r_sense_ohm       = 0.075   ; the resistor fitted to THIS driver board
run_current_ma    = 756     ; normal motion
homing_current_ma = 200     ; the endstop search only
travel_deg        = 340     ; mechanical angle between the hard stops
```

**`homing_current_ma` is a safety setting, not a tuning one.** Homing finds a
stop by driving into it until the shaft falls behind; that only works while the
stop is stronger than the motor. A high gear ratio can shear a screw-head stop
straight past its mate - on tower 121 the axis ran 1.8 turns past the stop with
the encoder never falling behind, at full torque. Declare a reduced search
current wherever the stop cannot survive one.

Getting it wrong in either direction is visible: too strong and the stop yields
again; too weak and the motor stalls on ordinary friction mid-sweep, trips the
same latch, and records a phantom endstop. The firmware's `steps_per_rev` guard
is what catches the second case, so it has to be set from the real travel.

> **The centre lives in `config.ini`, not here.** `install.sh` cannot write it —
> `hardware.ini` is merged *over* `config.ini` and wins, so a shared key would
> silently discard whatever `gimbal heading setup` had saved. What it does instead
> is work the requirement out from `steps_per_rev` and print it as a `NOTE` in the
> block, because the default is wrong for any geared-up rig - and
> `scripts/install-tower` then sets it in `config.ini` when it is too small:
>
> ```
> ; NOTE: half this travel is 605389 counts, so config.ini needs
> ;   [Motor] yaw_center_count = 1000000    (or anything above 605389)
> ;   The 500000 default would put the negative end at -105389.
> ```
>
> The range is centre ± half the travel, so a centre below half-travel puts the
> negative end under zero. On tower 121 the inherited `500000` homed to
> `-104749..1104749`; at `1000000` it homes to `395194..1604807`. A warning names
> it after every home if it is still too small.

**`travel_deg` is a nameplate fact**, and the host divides the measured homing
travel by it to derive `[Motor] <axis>_counts_per_deg` into `config.ini`. A rough
angle gives a proportionally rough degree scale - 1 % out here is 1 % out on every
bearing - until `gimbal heading setup` supersedes it with a landmark fit.

The startup log reports what the registers **actually** deliver, which is not
always what was asked for:

```
yaw motor current: run 756 mA (GLOBALSCALER 80, IRUN 31), homing 189 mA (IRUN 7)
                   -- R_sense 0.075 ohm from hardware.ini [Motor.yaw]
```

There is no leaving the block out any more. The firmware holds no drivetrain
values of its own: at boot every axis stops its driver, asks the host (`NEED`),
and refuses to move until every value it asked for has arrived. An axis whose
block is missing or incomplete is an axis that will not turn - by design, since a
guessed value is how drivers get cooked and endstops sheared. (For a bench board
with no host, the firmware's `USE DEFAULTS` command loads deliberately gentle
limp-home values instead; `test hardware motor` fails on a tower running them.)

The kit writes eight keys per axis - the four above plus `steps_per_rev`,
`sw_mode`, `enc_deviation` and `has_encoder` - each as a value or a `; TODO`;
`--motor AXIS.KEY=VALUE` gives any of them for any axis, where the older
`--r-sense`-style flags cover the first axis only. The three speeds
(`scan_speed_deg_s`, `homing_speed_deg_s`, `center_speed_deg_s`, all in degrees
per second) are written as gentle starting values: **on a tower that is being
migrated, set the homing speed to what its endstop detection is proven at**, as
was done for tower 111 (section 10b), or a stall-detected endstop may behave
differently.

> **If an axis declares a reduced homing current and the firmware has not adopted
> it, the host refuses to home** rather than driving into a stop at full torque.
> The message names the skew; the fix is to flash the firmware, not to override
> it.

> `[Serial] device` must be a `/dev/serial/by-id/` path, never `/dev/ttyACM0`.
> Tower 121 has a 5G modem that claims three `ttyUSB` nodes, and enumeration
> order is not stable across reboots.
>
> **The by-id path changes when you flash the MCU**, because the name comes from
> the running image: `usb-MicroPython_Board_in_FS_mode_*` before, a Pico/Arduino
> name after. `scripts/install-tower` re-probes after flashing and updates
> `[Serial] device` itself; by hand, re-run `--probe` after step 6 and fix the
> file if it moved.

### The wiring block: `[Pins]`

The firmware is **built** from this section - which MCU pin reaches which driver
board - by `firmware/tools/gen_wiring.py`, reading the tower's own
`/etc/fire_gimbal_control/hardware.ini` when the firmware is built on the tower.
There is no copy in the firmware and no fallback: without `[Pins]` the firmware
build fails, naming what to add. Each kit's reference loom, with the table saying
which wire goes where, is its `pins.ini`; `install.sh --loom standard` copies it
in, and `--pin KEY=N` gives pins one at a time for a tower wired differently.

The generator refuses what the board cannot do rather than building it: on the
RP2040, SPI pins outside SPI0's reach (SCK `{2,6,18,22}`, MOSI `{3,7,19,23}`,
MISO `{0,4,16,20}`); on the Leonardo, any `spi_*` key at all (its SPI is fixed on
the ICSP header) and the ICSP pins as chip-selects; duplicates; and a
`hardware.ini` written for the other board. The running firmware reports the
pins it was built with, and `test hardware motor` compares them with this
section - so a flash built from an older copy shows up there.

## 6. Motor firmware `[verified]`

The firmware is built and flashed **on the tower** - it is the machine wired to
the MCU - so PlatformIO is installed there, not on the dev box.

```bash
# tower: PlatformIO core, in its own venv under ~/.platformio. No sudo needed.
curl -fsSL https://raw.githubusercontent.com/platformio/platformio-core-installer/master/get-platformio.py -o /tmp/get-platformio.py
python3 /tmp/get-platformio.py
~/.platformio/penv/bin/pio --version      # check it actually runs
```

`[verified]` on tower 121: PlatformIO Core 6.2.0 on the system Python 3.14.4.

> The installer ends with a `FileNotFoundError` traceback out of
> `multiprocessing/util.py` about a missing `/tmp/.piocore-installer-*` path.
> It is a temp-directory cleanup finalizer running after the work is done -
> cosmetic. Judge the install by whether `pio --version` answers.

`deploy.sh` looks for `pio` in the usual places, but a non-interactive ssh shell
does not have `~/.platformio/penv/bin` on PATH, so name it explicitly:

```ini
REMOTE_PIO=/home/ggs/.platformio/penv/bin/pio
```

Then, from the dev box, in `firmware/`:

```ini
# .deploy.env
REMOTE_HOST=ggs@10.11.12.121
REMOTE_DIR=/home/ggs/projects/fwt_2a/firmware
PIO_ENV=iota_rp2040          # leonardo on a Delta
SERIAL_PORT=/dev/serial/by-id/usb-...
```

```bash
./deploy.sh --compile-only     # build on the tower, flash nothing
./deploy.sh                    # build + upload
./deploy.sh --monitor          # watch it boot
```

`deploy.sh` syncs the tree and then runs `firmware/tools/flash.sh` on the tower,
which is also what `scripts/install-tower` runs, locally: one implementation of
the board-specific bootloader step for both. **Stop the program first**; it holds
the serial port.

**Confirm the pin map before flashing a board that is wired to motors.** A
chip-select on the wrong pin writes into a driver that was never meant to hear
it. The build takes the pins from this tower's `hardware.ini` and fails without
them - see "The wiring block" in section 5. The IOTA's reference loom is the one
as built on tower 121, which is **not** the map proposed in the porting plan.

`[verified]` on both boards: the RP2040 on tower 121 (2026-09-23), the Leonardo on
tower 111 (2026-09-24) - the Leonardo through its own 1200-baud touch, which
`flash.sh` leaves alone.

### Getting a factory board to accept its first flash

`[verified]` on tower 121, and this is the step that will bite you. These boards
ship running MicroPython, and **the normal upload does not work against it**:

```
Forcing reset using 1200bps open/close on port /dev/ttyACM0
No new RPxxxx device found yet, waiting..          (x20, then it gives up)
```

| What | Factory board (MicroPython) | Our firmware |
|---|---|---|
| 1200-baud touch, i.e. a plain upload | no reset at all | resets, but **reboots into the application** |
| `picotool reboot -f -u` | refused: no reset interface | - |
| MicroPython REPL | **works** | n/a |

So drop into MicroPython's own REPL and tell it to enter the bootloader.
`firmware/tools/flash.sh` (which `deploy.sh` runs on the tower) now does this by
itself when the port's by-id name says `MicroPython_Board`: `./deploy.sh` with
`SERIAL_PORT` set to that path should flash a factory board directly. `[untested]`
- no factory board has been through that path yet; the manual way, verified on
tower 121, is below. Write this to a file and run it with
`~/.platformio/penv/bin/python` (it has pyserial):

```python
import serial, time
s = serial.Serial("/dev/serial/by-id/usb-MicroPython_Board_in_FS_mode_XXXX-if00", 115200, timeout=1)
s.write(b"\x03\x03"); time.sleep(0.3)        # Ctrl-C: interrupt whatever is running
s.write(b"import machine\r\n"); time.sleep(0.3)
s.write(b"machine.bootloader()\r\n")         # the port drops here - that is success
```

```bash
lsusb | grep 2e8a        # expect: 2e8a:0003 Raspberry Pi RP2 Boot
```

Then run `./deploy.sh` and it flashes. A physical BOOTSEL press does the same
job if you are standing at the board.

**Every board needs exactly one physical BOOTSEL press in its life** - the one
that installs the first image carrying the `BOOTSEL` command (below). After
that, updates are remote.

**`sudo ./install.sh --preconditions` must have run first**, or picotool finds
the board and refuses it:

```
RP2040 device at bus 1, address 4 appears to be in BOOTSEL mode, but picotool
    was unable to connect. Maybe try 'sudo' or check your permissions.
```

### Re-flashing later: the `BOOTSEL` command

`[verified]` on tower 121. Neither documented remote path works on this board -
the 1200-baud touch resets the firmware but returns it to the application, and
`picotool reboot -f -u` answers `Unable to locate reset interface on the
device`. So the firmware carries a command that does it deliberately:

```bash
# over the serial port, or from the app: gimbal raw "BOOTSEL"
BOOTSEL   -> OK entering bootloader - reflash to return
lsusb | grep 2e8a        # 2e8a:0003 Raspberry Pi RP2 Boot
./deploy.sh              # flashes and reboots into the new firmware
```

The whole loop was run end to end from the dev box with nobody at the tower.
`BOOTSEL` disables every axis first, since the bootrom does not run the control
loop and unsupervised energised drivers would hold torque.

> **It takes the gimbal out of service and the firmware cannot undo it** - once
> in the bootrom there is no command loop left to change its mind. Recovery is
> `deploy.sh`, or a power cycle: entering the bootrom erases nothing, so the
> existing image is still in flash. Anything that can write to the serial port
> can send it, including `gimbal raw`. Near misses (`BOOT`, `BOOTSE`) are
> rejected as unknown commands.

### The identity changes when you flash

| | by-id path | USB |
|---|---|---|
| before | `usb-MicroPython_Board_in_FS_mode_<serial>-if00` | `2e8a:0005` |
| after | `usb-Raspberry_Pi_Pico_<SERIAL>-if00` | `2e8a:000a` |

So generate `hardware.ini` (step 5) **after** flashing, or update `[Serial]
device` once the name has moved - `scripts/install-tower` does that itself,
re-probing after every flash.

### Verify the flash

Read the boot banner over the serial port:

```
BOOT build=aab1925-dirty
IOIN Y=0x3000001E version=0x30
ST Y:R,500000,18,80084000,0,8,8,Sd
```

`version=0x30` is the TMC5160 answering over SPI - the single most useful line
here, because it proves the pin map, the 3.3 V logic level and the driver board
all at once. `BOOT build=` must match the commit you flashed. One `Y` line and
no `P` line is a yaw-only build.

> **Flashing an RP2040 remotely is not the same as a Leonardo.** The documented
> flow is *hold BOOTSEL, tap RST, drag a `.uf2`* - a person at the board. The
> two remote paths (the 1200-baud touch that the Arduino tooling already uses,
> and `picotool reboot -f -u`) both need the **running** image to present a USB
> CDC port. That is why `setup()` brings `Serial` up before any hardware init:
> an image that hangs before USB comes up cannot be reflashed without a site
> visit, while an interrupted flash recovers by itself - the bootrom falls back
> to mass storage when flash holds no valid image.

## 7. Deploy the software `[verified]`

On the dev box, in `software/`:

```ini
# .deploy.env
REMOTE_HOST=ggs@10.11.12.121
REMOTE_DIR=/home/ggs/projects/fwt_2a/software
CMAKE_ARGS="-DWITH_MQTT=ON -DWITH_VIMBA=ON"
RUN_ARGS=""                 # extra flags only; homing and mode are config.ini [Startup]
```

```bash
./deploy.sh --check-deps    # must print "remote deps OK"
./deploy.sh                 # rsync + remote cmake + build
```

The first deploy copies `config/config.example.ini` to `config.ini` **only if
the device has none** - see the warning in step 2 about a machine that arrived
with somebody else's config. That copy still carries the example's placeholders
and the keys `hardware.ini` owns; `scripts/install-tower` seeds it instead with
`build/fgc-config seed`, which fills in the answers, comments out every key
`hardware.ini` sets, and refuses to write a file the program would not load.

`[verified]` on tower 111: builds on GCC 11 / Ubuntu 22.04.
`[verified]` on tower 121: builds clean on GCC 15 / Ubuntu 26.04 with a single
`-Wformat-truncation` warning in `utcStampNow()`, which is a false positive (the
compiler cannot prove `tm_year` is four digits; the real output is 17 bytes into
a 32-byte buffer).

## 8. Camera profiles `[verified]`

**Without a profile the program does not start** - `no camera profiles (*.xml)
in ~/.config/fire_gimbal_control/camera_profiles/rgb` is a startup error, by design. A fresh tower has
an empty profile directory, so this step is not optional.

The documented way to make one is the VimbaX Viewer, which is a GUI - awkward on
a headless tower and impossible unattended. `scripts/export_camera_profile.py`
does the same job through the SDK's Python API:

```bash
# once per tower: the SDK ships the wheel
python3 -m venv ~/.venvs/vmbpy
~/.venvs/vmbpy/bin/pip install /opt/VimbaX/api/python/vmbpy-*.whl

cd ~/projects/fwt_2a/software
export GENICAM_GENTL64_PATH=/opt/VimbaX/cti
~/.venvs/vmbpy/bin/python scripts/export_camera_profile.py --profile standard --dry-run
~/.venvs/vmbpy/bin/python scripts/export_camera_profile.py --profile standard
~/.venvs/vmbpy/bin/python scripts/export_camera_profile.py --profile night
```

It writes each recipe value to the real camera **in dependency order**, lets the
camera validate it, and exports what was actually accepted. That preserves the
property the Viewer workflow exists to provide: GenICam validity is stateful, so
a hand-written XML can be individually plausible and collectively wrong. Here
the camera is the judge, and anything it refused is printed rather than assumed.

Read the summary line it prints - it is the number that decides whether the
profile can be delivered at all:

```
result  : 2664x1520 RGB8  12.1 MB/frame  4.95 fps  = 60 MB/s
```

### Profiles do not transfer between cameras

`[verified]` the hard way. The fleet runs several Alvium generations and their
feature vocabularies differ, so copying another tower's XML does not work:

| | Delta tower (1800 U-1240c) | IOTA tower (1800 U-1620c) |
|---|---|---|
| `SensorBitDepth` | `Bpp8` | **no `Bpp8`** - only `Adaptive` / `Bpp12` |
| `IntensityAutoPrecedence` | `Minimizeexposuretime` / `Minimizenoise` | **`MinimizeBlur` / `MinimizeNoise`** |
| Sensor | 12 MP | 16 MP, 5328x3040 |

### Binning is not optional, and the frame-rate ceiling is not what it looks like

Unbinned RGB8 on the 16 MP sensor is a **48.6 MB frame** - over the ~20 MB
ceiling in known-issues #19. 2x2 binning brings it to 12.1 MB.

The frame rate is then capped by `DeviceLinkThroughputLimit`, but the limit is
accounted against the **unbinned** payload, not the binned frame that crosses
the wire. Measured on tower 121:

| `DeviceLinkThroughputLimit` | max `AcquisitionFrameRate` | actual USB traffic |
|---|---|---|
| 250 MB/s (the fleet's value) | 4.95 fps | ~60 MB/s |
| 450 MB/s (this camera's max) | 8.79 fps | ~106 MB/s |

48.6 MB x 4.95 = 240 MB/s, which is the 250 limit - so the camera is budgeting
for data that binning discards. **The recipe ships the conservative 250 MB/s**,
giving ~5 fps rather than the Delta's 10, because the USB stalls behind
known-issues #19 are a host-side finding that has not been re-measured on the
N150 yet (porting plan phase 5). Raising it is a one-line change in the recipe
once that measurement exists.

> **Export `standard` after `night`, not the other way round, or use the current
> script.** A refused write is not a no-op: the first version of this script
> asked for 10 fps, the camera refused, and the 1 fps left over from the night
> profile stayed - producing a "standard" profile that ran at night speed. The
> script now clamps an out-of-range numeric value to the camera's limit and says
> `~ ... (clamped)`.

## 9. `config.ini` - what this tower DOES `[partial]`

Device-local, never overwritten by a deploy. Edit it on the tower:

```ini
[General]
tower_name = <this tower>          ; drives every MQTT topic
[Network]
zkms_server_ip = <broker>
```

Credentials belong in `/etc/fire_gimbal_control/credentials.env`, not here.

Anything `hardware.ini` now owns - `[Serial] device`, `[IMU]`, `[Env]`, the
camera ids - should not be in `config.ini` as well: the program warns at startup
about every key set in both. `scripts/install-tower` comments them out (value and
all), and `build/fgc-config set config.ini --config config.ini --hardware
/etc/fire_gimbal_control/hardware.ini --drop Section.key=note` does one by hand.
Every edit through `fgc-config` is loaded the way the program loads it before it
is written, and keeps the previous value as a `;prev` comment.

> **`[Camera.<name>] id` is required even when the camera is disabled.** Config
> validation runs before any of the feature flags are consulted, so a tower with
> no camera fitted yet cannot start without one - and the obvious dummy,
> `DEV_0000000000`, is what `deploy.sh`'s placeholder guard refuses. On a tower
> that has its camera, the id lives in `hardware.ini` and this never comes up.

`[Features] enable_imu` and `enable_env` are for towers **without** a
`hardware.ini`. Where `hardware.ini` declares `imu` / `env`, the declaration
decides, and a flag that contradicts it is warned about and ignored. Before that
was so, tower 121's first clean-looking run "started" an IMU it does not have -
`/dev/ttyS0` accepts a serial open whether or not anything is attached - which is
why a declared `none` has to be able to switch a sensor off by itself.

## 10. First run: bring the subsystems up one at a time `[partial]`

Do **not** start a new tower with everything switched on. Each phase below adds
exactly one subsystem, so that a failure names its own cause instead of arriving
as a wall of unrelated errors. It also keeps the motors - the only part that can
break itself - until you have proved the boring parts work.

Each phase says what good looks like and what noise to expect.

### Phase 0 - the host alone, no hardware `[verified]`

```bash
./build/fire_gimbal_control --mock-serial --mock-camera --no-mqtt --no-tui
```

Proves config resolution, `hardware.ini` layering, the single-instance lock, the
scan grid and the output directory - with nothing physical involved. Neither a
camera nor its profile XMLs need exist:

```
[INFO ] camera profiles not loaded: no real camera in this run
```

`[Features] enable_camera = false` does the same thing from the config file, for
a tower that has no camera fitted yet, and says so rather than pretending:

```
[WARN ] camera disabled ([Features] enable_camera = false); using a mock source
        - no images come from hardware in this run
```

```
[INFO ] hardware: /etc/fire_gimbal_control/hardware.ini  board lattepanda-iota  axes yaw
[INFO ] Loaded config: config.ini
[INFO ] scan grid: 15 waypoints
```

Any `config.ini still sets X, which hardware.ini overrides` warning belongs here
and should be cleared before moving on (step 9).

### Phase 1 - motors `[partial]`

Real serial link, nothing else:

```bash
./build/fire_gimbal_control --mock-camera --no-mqtt --no-tui
```

What proves the link is a **live `DUMP` from the board**, not just an open port:

```
[INFO ] Serial controller started on /dev/serial/by-id/usb-Raspberry_Pi_Pico_...
DUMP BEGIN build=aab1925-dirty uptime=7925212 mcusr=0x3 free_ram=252712
DUMP Y state=1 ... speed=3000 center=500000 has_encoder=1
```

Check `build=` matches the firmware you flashed, and that the handshake
completed - the log says `firmware config: answered N keys for axis Y`, and the
axis leaves state 5 (`NEEDCFG`). The scripted form of all of this, which also
compares every value the firmware holds with `hardware.ini`, moves nothing:

```bash
./build/fire_gimbal_control --no-mqtt --test hardware    # exit 0 PASS, 4 FAIL, 5 could not start
```

Then, and only then, move the thing:

```
gimbal home              # the endstop-finding sequence - see the warning below
gimbal diag y            # firmware self-test, reports PASS/FAIL
gimbal move 10,0         # a small deliberate move (a yaw-only tower: gimbal move 10)
```

**Expected noise in this phase, neither of which is a fault:**

| Message | Why |
|---|---|
| `Motor controller: ERR unknown command`, a burst at startup | Only on firmware older than the config handshake: it does not know `SET`. Seen on tower 111 before its flash; harmless there, and gone once the firmware is current |
| `encoder tracking error on yaw: 500366 counts (limit 400)` | Before homing, the firmware presets `XACTUAL` to the centre count while the encoder sits at its own zero, so they disagree by the whole range. Homing resyncs them. **If it persists after a successful home, that is real** |

> **On an unfamiliar drivetrain, homing is the dangerous step** - it finds the
> endstop by driving into it. Start from a deliberately slow
> `homing_speed_deg_s` and walk it up, watching the free-running encoder gap that
> `enc_deviation` is measured against. Be at the rig with a hand on the power for
> the first one - which is why `scripts/install-tower` asks before it homes.

### Phase 2 - camera `[verified]`

Add the camera, still no broker:

```bash
./build/fire_gimbal_control --no-mqtt --no-tui
```

```
[INFO ] Opened camera 'rgb' DEV_1AB22C0B7D4E (1800 U-1620c)
[INFO ] camera profile 'standard' applied from .../standard.xml (17206 settings)
```

Then `mode auto`, and confirm `.jxl` files appear under the configured output dir.

> **A camera-free phase needs no profile XMLs** - but this only became true on
> 2026-09-16. Before that, `[Features] enable_camera` was parsed and never used,
> and the profile directory was validated even for a mock camera, so a tower
> whose camera had not been configured yet could not start at all. If you are
> running an older binary, export the profiles (step 8) first or it will refuse
> with `no camera profiles (*.xml)`.

### Phase 2b - thermal camera

Only for a tower with an InfraTec TarisIR mini (tower 111 since 2026-10). The
program runs as the tower's user; the SDK needs three things from the system.

1. **SDK and licence.** Packages `infratec-sdk-irbgrab` and
   `infratec-itcamtarisirmini` (amd64), then the licence for the camera's serial
   via the vendor's `copycustomerdata.sh` into `/var/lib/infratec/.license/`
   (`root:infratec`, `rw-rw----`). **Repeat after every OS update.** Keep the
   source files on the tower (`~/InfraTec/LINUX/` on 111) so it can be re-run.
   The tower's user must be in group `infratec` (`id` shows it; `sudo usermod -aG
   infratec ggs` and log in again otherwise). The library must be in the ld
   cache: `ldconfig -p | grep irbgrab`.
2. **Network.** Camera and port directly cabled, static address, **no DHCP**
   (`dhcp4: false` in netplan: DHCP on a link without a server keeps failing and
   takes the static address with it), **MTU 1500** - the camera does no jumbo
   frames. The SDK finds the camera by serial and MAC on raw Ethernet; the IP
   only matters to ordinary GigE Vision tools. On 111: `enp1s0`, 192.168.0.111,
   camera 192.168.0.200.
3. **`cap_net_raw` on the binary.** The SDK opens a raw `AF_PACKET` socket.
   Rather than running the program as root, the binary gets this one
   capability. A rebuild drops it, so `deploy.sh` and `install-tower` run
   `scripts/fgc-setcap --if-thermal` after every build. Once per tower, let
   deploy do that without a password:

   ```bash
   scripts/fgc-setcap --install-sudoers   # /etc/sudoers.d/fire-gimbal-control-setcap
   scripts/fgc-setcap && getcap build/fire_gimbal_control
   ```

   A capability binary runs in the loader's secure mode (`LD_*` ignored, no core
   dumps): check once that Vimba X still opens the colour camera, and that
   `scripts/fgc status` and `stop` still find the process.

Then hardware.ini (`driver = taris`, `id = <serial>`, `interface = enp1s0`,
`cameras = rgb, tir`) and config.ini (`label = TIR`, see
[configuration.md](configuration.md#thermal-camera-driver--taris)), and a run
with the real cameras but no motors:

```bash
./build/fire_gimbal_control --mock-serial --mock-env --no-mqtt --no-tui
```

```
[INFO ] IRBGRAB SDK 4.3.44
[INFO ] thermal camera connected via SN:2017744|...|192.168.0.200#0
[INFO ] thermal camera: profile 0, 0.01 K per count
```

`snapshot` then writes `RGB/<t>.jxl` and `TIR/<t>.jxl` with the same `<t>`.
`camera calib` reports the profile. If the camera stays offline the log says
why, once per change - `no cap_net_raw`, `not found (found: ...)`, the port's
state and MTU. A `0x80000017` that keeps coming is the licence before it is the
NUC.

Measured on tower 111 (2026-10-14, SDK 4.3.44, SN 2017744):

- `cap_net_raw` alone is enough; no root, no `cap_net_admin`. Vimba X still
  opens the colour camera from the capability binary.
- Connecting takes about 15 s (SDK init and search); the scan does not wait.
- `Framerate_Hz` is settable. The LUT is empty for the first frame after
  connecting and right from then on: 0.0099998 K/count (profile 0), 0.0199995
  (profile 1).
- A profile switch takes about 3 s; frames keep the old scale until
  `Calib_ChangeInProgress` reaches 0, and the program takes none meanwhile.
- The camera's frame counter (`ImageNum`) is always 0; `MissedPackets` was
  always 0 on the direct link.
- Frame rate (2026-10-15): from power-up the camera sends 50 fps, 31 MB/s and
  22 000 packets a second. `Framerate_Hz` set right after connecting answers ok
  and changes nothing; set while grabbing it only thins out what the SDK hands
  over; set while the grab is stopped, after the stream has run, it holds on the
  wire. The program does that once the first frame is in and checks the packet
  rate of `[Camera.tir] interface` (5 Hz: about 2 200 packets a second); the log
  says `5.3 fps on enp1s0 (2224 packets/s), set on attempt 2`.
- NUC (2026-10-15): the camera runs one every 30 s (`NUC_Interval`) and sends no
  frame for 650-760 ms meanwhile. `NUC_Interval = 0` stops that; `NUC_Perform`
  (1-point) runs one and returns when it is done, after about 0.67 s.
  `NUC_INPROGRESS` stays 0 throughout, also for those. The program now runs the
  NUC itself at the start of a move (`nuc_interval_s`).
- The SDK switches the process's C locale to `LC_NUMERIC=de_DE` when it is
  loaded. The program puts it back after every SDK call and writes its JSON
  without the C library's number formatting.

### Phase 3 - the broker

Set `[Features] enable_mqtt = true` and point `zkms_server_ip` at the broker.
Without one, the app says so and carries on:

```
[ERROR] MQTT connect failed (tcp://10.11.12.121:1883): TCP/TLS connect failure
[WARN ] Control channel not connected; continuing in degraded mode
```

**Degraded mode is not a failure to notice later** - the tower captures images
and reports to nobody. Confirm topics arrive at the broker before calling the
phase done.

Each tower runs its own RabbitMQ, in Docker, with the `rabbitmq_mqtt` and
`rabbitmq_shovel` plugins - the shovel is what forwards this tower's messages to
the ground station. That lives in `~/zkms-client/`, outside this repository.

### A legacy tower (DHT11 and fan on the MCU)

Answer the installer's ambient sensor question with `dht11` and the fan
question with yes, and choose `loom = legacy`. The firmware is then built with
the DHT11 and fan (see configuration.md, "The legacy towers").

**The encoder scale.** Take it from the survey's "old firmware's source":
`R3A` is `ENC_CONST`, the whole part in the high 16 bits and 1/10000ths in the
low 16, so `0x000C1388` is 12.5. If it is not 12.8, put it in the answers file
as `yaw.enc_const=12.5` (the installer never asks for it). On staeffelsberg,
2026-09-28, it was missing: homing took the 2% drift for a stop after 131°.

Check it **on site** after the first flash:
1. The fan runs at full speed from power-up.
2. It drops once readings come in, if the box is below 25 °C.
3. Warming the DHT11 by hand, or with a hair-dryer, spins it up.
4. `test hardware env` passes, fan leaf included.

### Phase 4 - IMU and ambient sensor

Only if fitted, and declared in `hardware.ini` (`imu`, `env`) - the declaration
is what switches them on. `test hardware imu` wants a `0xFA`-framed stream from
the MTi; `test hardware env` wants a real I2C adapter and a CRC-valid SHT41
reading. `[verified]` on tower 111, 2026-09-23: both.

### Phase 5 - unattended

Only once every phase above is green:

```bash
# config.ini: [Startup] home = true, mode = auto
scripts/fgc start             # detached, inside tmux
scripts/fgc status            # running? since when? which PID?
```

Close the ssh session, reconnect, and check it is still running. Then install
the boot service (`scripts/install-tower` asks; [deployment.md](deployment.md)
section 4), reboot the tower, and confirm with `scripts/install-tower --verify`
that the USB preconditions survived and the program came back by itself.

### Calibration

Per tower, and **cannot be inherited** - least of all from a tower with a
different gear ratio, where `yaw_counts_per_deg` is derived from a
`steps_per_rev` that no longer applies:

```
gimbal heading setup     # ties yaw to true north by sighting landmarks; no IMU needed
gimbal pitch setup       # needs an IMU and a pitch axis; skip on a rig without them
```

## 10b. Bringing up the config handshake on an existing tower

This is the one upgrade that changes what an **unconfigured** tower does: the new
firmware boots into `NEEDCFG` and refuses `HOME` / `MOVE` / `ENABLE` until the
host has answered its `NEED` lines. Flashing it onto a tower whose `hardware.ini`
predates the drivetrain keys leaves the gimbal inert — working, talking, and not
turning.

So the order matters. **Host first, firmware second.** Each step below either
changes nothing or is reversible, and the step that could strand the tower comes
after the one that prevents it. `scripts/install-tower` follows this order by
itself, and blocks the flash while any drivetrain value is missing.

### The worked example: tower 111 `[verified]`

The Delta tower, which predated `hardware.ini`, the handshake and the kits.
2026-09-23 and -24, through the installer with the Delta kit:

1. **A rollback copy** of the sources, the running binary and `config.ini`, in
   `~ggs/rollback-20260923/`, before anything changed.
2. **The host half.** Preconditions, `hardware.ini` with every drivetrain value
   a `; TODO`, `config.ini`'s duplicated keys commented out, the missing `night`
   profile, a new host. `test hardware`: axes, IMU, env and camera PASS; motor
   FAIL on the old firmware's missing echoes, as expected (step 2 below). Firmware
   blocked. It homed and resumed capturing on the new host.
3. **The drivetrain, without inventing it.** The sense resistors were read off
   both boards (75 mOhm). Everything else was converted from what the old
   firmware ran - its revision is in `DUMP BEGIN build=`, and git history at
   that revision has its register values - through the host's own planners, and
   round-tripped back to the identical registers before being written. Homing
   and centring speeds were set to what that firmware used, not the kit's
   gentle defaults, since its endstop detection was proven at those.
4. **The flash**, by the installer: every gap filled, the firmware built from
   `hardware.ini` and flashed, `test hardware` green on every leaf, and homing
   within 0.03 % of the old firmware's travel.

### 0. Record what you are rolling back to

```bash
gimbal dump                 # the running firmware's build id and register state
cat /etc/fire_gimbal_control/hardware.ini
git -C ~/projects/fwt_2a/firmware rev-parse --short HEAD
```

Keep the dump. It is the only record of what the old firmware was holding, and
the comparison that tells you whether the new one adopted the same values.

### 1. Update `hardware.ini` — nothing else, and no restart yet

Add the `[Motor.<axis>]` block (section 5) and `[Pins]`. The firmware asks for
eleven values per axis - `steps_per_rev`, `global_scaler`, `run_irun`,
`homing_irun`, `center_count`, `soft_margin`, `enc_deviation`, `normal_speed`,
`homing_speed`, `center_speed`, `sw_mode` - and the host derives them from the
block (currents from milliamps and the sense resistor, speeds from degrees per
second, centre and margin from `config.ini`). `install.sh --stdout` generates the
block; for values nobody has measured, convert what the old firmware runs, as
for tower 111 above.

Doing this **before** flashing is what keeps the window at zero: the host is ready
to answer the moment the firmware starts asking.

### 2. Deploy the new host against the OLD firmware

A real test, and it risks nothing. The old firmware never sends `NEED`, so the
host must behave exactly as it did before — which is also the guarantee the whole
Delta fleet depends on.

```bash
scripts/fgc start
scripts/fgc peek
test hardware motor
```

Expect: no handshake lines at all (and possibly a burst of `ERR unknown
command` - the old firmware refusing `SET`), and the motor leaf **failing** on
missing echoes, reported as a version skew rather than a mismatch:

```
; yaw_pins: yaw: the firmware does not report the pins it was built with, so its
  wiring cannot be confirmed -- it predates the pin echo. Re-flash it
```

That FAIL is by design - a check that could not happen must not read as a pass -
so on old firmware the gate is "every other leaf green, and `hw_motor` failing
only on missing echoes". A speed can also differ before the first home, since
the host sends it after homing. If instead you see wrong *values* for something
the firmware does echo, stop: `hardware.ini` disagrees with what the tower is
really running, and flashing will not fix it.

### 3. Flash

```bash
cd firmware && ./deploy.sh          # uploads; --compile-only if you want a dry run first
```

Watch the banner. A `NEED` line per axis should follow it:

```
BOOT build=<rev>
NEED Y steps_per_rev global_scaler run_irun homing_irun center_count soft_margin ...
```

**No `NEED` line means the axis is not in `NEEDCFG`** — either the flash did not
take, or the image is older than you think. Check the build id before going on.

### 4. Confirm the handshake completed

The host answers within a control-loop tick. Expect:

```
firmware config: answered 11 keys for axis Y
```

Then prove it landed rather than trusting the log:

```
gimbal dump                 # state should be 1 (RESET), NOT 5 (NEEDCFG)
                            # cfg_rx should be the full required mask
test hardware motor         # both metrics PASS
```

`test hardware motor` is the real gate: it compares every declared value against
what the firmware echoes, so a `SET` that was lost on the wire shows up here and
nowhere else.

### 5. Prove the refusal actually refuses

Worth doing once per firmware version, because a guard that silently stopped
working looks exactly like a guard that is working.

On a **bench** board, not a mast: power-cycle it without the host running, then
send `HOME` over a serial monitor. Expect `ERR need config`. Send `RESET` and try
again — still `ERR need config`, because a RESET is deliberately not a way out.
`USE DEFAULTS` is, and should then let `HOME` through.

### 6. Only then, move the axis

```
gimbal home                 # at the reduced homing current
```

Check against the recorded figures: travel ~1,210,778 counts, repeatable to
~0.3 %, and the derived `counts_per_deg` near 3552.87. A travel that disagrees by
more than a few percent means the axis homed against something that was not the
endstop.

### Reading the log afterwards

Everything below is recorded to
`~/.local/share/fire_gimbal_control/logs/fgc_*.log`, not only to the dashboard —
which matters here, because in TUI mode the pane holds 500 lines and dies with
the tmux session. During this bring-up that cost us the handshake record four
separate times before the file existed.

```bash
L=$(ls -t ~/.local/share/fire_gimbal_control/logs/fgc_*.log | head -1)
grep -E "NEED|firmware config|need config" "$L"   # step 4: did the handshake complete
grep -E "hardware:"                        "$L"   # presence reconciliation
grep -E "speed:|motor current"             "$L"   # what was actually sent to the firmware
```

The file is more verbose than the dashboard was, so it holds lines the pane never
showed. `head -8` gives the build, tower and config that produced it.

### If it goes wrong

The board is recoverable over the network as long as USB CDC comes up:
`BOOTSEL` over the serial port, then `./deploy.sh` with the previous firmware
checked out. Only an image that wedges *before* `Serial.begin()` needs a physical
button press — which is why `Serial.begin()` is the first thing `setup()` does.

`USE DEFAULTS` is the in-place escape: it puts every axis on the compile-time
table and lets the tower move while you work out what `hardware.ini` is missing.

## 11. Verification checklist

Most of this is one command, which changes nothing:

```bash
scripts/install-tower --verify
```

It checks the USB preconditions, that `config.ini` and `hardware.ini` load
together, that `hardware.ini` has no gaps, that the firmware reports the pins it
declares, and the boot service - and runs `test hardware` and `test camera`: inside the running
program when it is up (pausing capture, then resuming it), or as a one-shot run
when it is not. By hand, the one-shot form is:

```bash
GENICAM_GENTL64_PATH=/opt/VimbaX/cti ./build/fire_gimbal_control --no-mqtt --test hardware
echo $?      # 0 PASS, 4 FAIL, 5 could not start (the log says why)
```

It waits for the firmware to be talking and configured before it looks, so a
freshly flashed board is not failed for values the host had not sent yet.

| Check | How | Tower 121 (IOTA) | Tower 111 (Delta) |
|---|---|---|---|
| Key-only ssh | `ssh ggs@<tower> hostname`, no prompt | ✅ | ✅ |
| Preconditions applied | `--verify`: `usbfs_memory_mb 1000`, camera LPM `0` | ✅ | ✅ 2026-09-23 (were 16 and `u1_u2`) |
| Preconditions survive a reboot | reboot, then `--verify` | ✅ confirmed across a real reboot | ✅ 2026-09-24 |
| `hardware.ini` found unaided | startup names `/etc/fire_gimbal_control/hardware.ini` | ✅ | ✅ |
| Declared hardware matches | startup logs `declared assembly confirmed`; `test hardware axes` | ✅ yaw | ✅ yaw, pitch |
| IMU / ambient sensor | `test hardware imu` / `env` | none declared | ✅ MTi streaming, SHT41 CRC-valid on a real I2C bus |
| Camera enumerates | `ListCameras_VmbC` reports the `DEV_` id in `hardware.ini` | ✅ `DEV_1AB22C0B7D4E` | ✅ `DEV_1AB22C049CD1` |
| Camera profile applies | startup logs `profile 'standard' applied ... (N settings)` | ✅ 17206 settings | ✅ 3743 settings |
| Camera works, whatever it sees | `test camera`: whole frames; every pixel forced to black (minimum exposure and gain); the sensor responding at maximum gain and a long exposure; the profile restored | ✅ 2026-09-24, lens covered: 100 % ≤ 20 dark, bright mean 0 → 255 | ✅ 2026-09-24, inside the running program: 5 frames of 4024x3036 in 9.2 s (a dark scene, ~2 s exposures), 100 % ≤ 20 dark (mean 5.6), bright mean 5.6 → 255; capture resumed. An earlier run correctly reported the camera delivering nothing at all - known-issues #15 |
| Firmware identity | `DUMP BEGIN build=` is the revision flashed | ✅ | ✅ `a0b1e4a` |
| Driver answers over SPI | `IOIN` reads version `0x30` for every axis | ✅ | ✅ both axes |
| Firmware holds `hardware.ini` | `test hardware motor`: values, current, pins | ✅ | ✅ every leaf, 2026-09-24 |
| Remote re-flash | no hands on the board | ✅ `BOOTSEL` | ✅ 1200-baud touch |
| Homing repeatable | endstop counts across homes | ✅ three homes within 0.27 % | ✅ within 0.03 %, old and new firmware |
| Endstop survives homing | the axis must stall against the stop, not push through | ✅ at 189 mA, after failing at full torque | not inspected; full current, as for its whole service life |
| Degree scale derived | startup logs `counts_per_deg` agreeing with the homed travel | ✅ 3552.87 | ✅ agrees, calibration left alone |
| Images land | `.jxl` files appear under the configured output dir | ❌ needs a capture run | ✅ |
| Starts at boot | reboot with nobody logged in; `scripts/fgc status` from a later ssh | ✅ lock under `/run/user/1000` | ✅ 2026-09-24: homed and captured by itself within ~35 s. The Delta logs a desktop session in at boot, so lingering was not what put the lock in `/run/user` here |
| Reports to a broker | topics arrive at the configured broker | ❌ no broker on this tower yet | ✅ connects to its own RabbitMQ (retrying until Docker has it up at boot); forwarding to the ground station not checked |