fwt_software/docs/deployment.md

# Running and updating a tower

How a tower that already works is operated and updated: pushing a new version
from the dev box, running the program detached, starting it at boot, reading its
log. A **first** install - a freshly imaged board, or a tower not yet migrated -
is [tower-installation.md](tower-installation.md), which `scripts/install-tower`
automates.

The fleet: tower **111** is a LattePanda 3 Delta (ATmega32U4 motor MCU, two
axes), tower **121** a LattePanda IOTA (RP2040, yaw only). Nothing here differs
between them; what differs lives in each tower's `hardware.ini`.

## Where things are on a tower

| Path | What | Written by |
|---|---|---|
| `~ggs/projects/fwt_2a/software/` | this repository, and the program's build in `build/` | `deploy.sh` (sync + build), or the installer |
| `~ggs/projects/fwt_2a/firmware/` | the firmware repository, built on the tower | `firmware/deploy.sh` |
| `~ggs/.config/fire_gimbal_control/config.ini` | what the tower DOES: scan, capture, logging, calibration | the installer (seeded once), `gimbal ... setup` (calibration), a person |
| `~ggs/.config/fire_gimbal_control/scan.csv` | the scan grid chosen for this tower (`[Scan] grid_file`) | the installer (the example grid), a person |
| `/etc/fire_gimbal_control/hardware.ini` | what the tower IS: board, axes, sensors, pins, drivetrain | the installer; a person when the hardware changes |
| `/etc/fire_gimbal_control/credentials.env` | `FGC_MQTT_USER=` / `FGC_MQTT_PW=`, mode 0640 root:ggs | the installer |
| `/etc/fire_gimbal_control/install.log` | one line per installer run | the installer |
| `~ggs/.local/share/fire_gimbal_control/logs/` | the program's log, test reports | the program |
| `~ggs/.config/fire_gimbal_control/camera_profiles/rgb/*.xml` | the camera settings, exported ON the tower from its camera | the installer, or `scripts/export_camera_profile.py` |

The rule: what a tower changes lives in `~/.config/fire_gimbal_control/`, what
it never changes in `/etc/fire_gimbal_control/`, and the project folder holds
only what a deploy or an update may replace. `deploy.sh` mirrors the project
folder with `rsync --delete`, so nothing of the tower's own may live there -
`rsync --delete` once erased a freshly exported profile that did.

### A tower set up before the tower folder existed

Its `config.ini`, scan grid and camera profiles are still in the project folder.
The program loads them from there and warns at every start, and every deploy
prints how to move them. Until then `deploy.sh`'s excludes keep them safe. Move
them once, on the tower:

```bash
ssh ggs@<tower>
cd ~/projects/fwt_2a/software
scripts/tower-config move --dry-run   # what moves where, and what config.ini will say
scripts/fgc stop                      # the program writes config.ini itself
scripts/tower-config move
scripts/fgc start                     # or: sudo systemctl start fire-gimbal-control
```

It copies `config.ini`, the scan grid `[Scan] grid_file` names and every
camera's profiles into the tower folder, and rewrites those two paths
(`config/scan.csv` becomes `scan.csv`). `fgc-config` checks the result loads
before it is written. The old `config.ini` is renamed
`config.ini.moved-<timestamp>`, and the old grid and profiles stay where they
were. `scripts/install-tower` does the same move as one of its steps.

## 1. Updating the program from the dev box

On the dev box, in `software/`, with a `.deploy.env` naming the tower:

```ini
REMOTE_HOST=ggs@10.11.12.121
REMOTE_DIR=/home/ggs/projects/fwt_2a/software
CMAKE_ARGS="-DWITH_MQTT=ON -DWITH_VIMBA=ON"
RUN_ARGS=""
```

`RUN_ARGS` holds only extra flags (`--no-mqtt`, say). Whether the program homes
and which mode it starts in are the tower's own `config.ini` `[Startup] home` /
`mode`, the same for `--run` as for the boot service; `deploy.sh` refuses a
`RUN_ARGS` that still has `--init` or `--start`.

```bash
./deploy.sh                  # sync the tree, build on the tower
./deploy.sh --check-deps     # only check the tower's build dependencies
./deploy.sh --clean          # wipe the tower's build dir first (fresh configure)
```

It syncs the tree (leaving `.git` behind, and writing the revision into
`.revision` beside it instead), then configures and builds **on the tower**. Only
files git knows about are synced - tracked ones and new ones not yet added, as
they are in the working tree - so nothing `.gitignore` names (build trees, editor
caches, `.claude/worktrees/`, the camera bench's example pictures) reaches the
tower, and a file left there by an older deploy is removed. The
program already running keeps running the old binary until it is restarted:

```bash
# on a tower started by its boot service
ssh ggs@<tower> sudo systemctl restart fire-gimbal-control
# on a tower started by hand
./deploy.sh --stop && ./deploy.sh --run          # homes / sweeps as its [Startup] says
```

To home once on a tower whose `[Startup] home` is off, attach and type
`gimbal home` (key `h`); `deploy.sh --home` is gone.

After building, `deploy.sh` also reads the tower's boot service: one that still
passes `--init`/`--start` gets a loud warning, because the new program refuses
them and **the tower would not start at its next boot**. Run
`scripts/install-tower` on the tower to fix it (section 4).

`--run` starts the program detached through `scripts/fgc` (section 2) and first
refuses a `config.ini` that still holds the example's placeholders
(`ExampleTower`, `CHANGE_ME`, `DEV_0000000000`); `--force` overrides that. On a
tower whose service is running, `--run` fails on the single-instance lock
(section 3) - restart it through systemd instead.

After an update, check it:

```bash
ssh ggs@<tower> 'cd ~/projects/fwt_2a/software && scripts/install-tower --verify'
```

It changes nothing; it runs `test hardware` and `test camera` inside the running
program (pausing and then resuming capture) and checks the files, the firmware's
pins and the boot service. `test camera` takes 15-30 s: it grabs frames, then
drives the camera's own settings - minimum exposure and gain must take every
pixel black, maximum gain and a long exposure must make the frame measurably
different - which holds whatever the camera is pointed at, and puts the profile
back.

`--push-config` also overwrites the tower's `config.ini` with the dev box's,
backing the tower's copy up first. It exists for the development tower; a
tower's `config.ini` holds its own calibration, so do not push one tower's to
another. `hardware.ini` is never pushed.

> **One-time: `git pull` can delete your `.deploy.env`.** It was tracked by
> mistake until `b0127db`, which stopped tracking it and added it to
> `.gitignore`. Pulling *across* that commit deletes the working copy, credentials
> and all, with no warning. Only checkouts that predate `b0127db` are exposed,
> and only once: copy it aside first (`cp .deploy.env ../deploy.env.bak`), pull,
> copy it back. To recover one already lost: `git show b0127db^:.deploy.env`.

### Updating the firmware

**Host first, firmware second**, always: a new firmware asks the host for its
drivetrain at boot, and an older host cannot answer. In `firmware/`, with its own
`.deploy.env` (`REMOTE_HOST`, `REMOTE_DIR`, `PIO_ENV` = `leonardo` or
`iota_rp2040`, `SERIAL_PORT` = the MCU's `/dev/serial/by-id/` path):

```bash
./deploy.sh --compile-only   # sync + build on the tower, flash nothing
./deploy.sh                  # build + flash
./deploy.sh --monitor        # ... then watch it boot
```

The build takes the pins from the tower's `hardware.ini` (`[Pins]`) and fails
without them. The flashing is `firmware/tools/flash.sh`, run on the tower: an
RP2040 is put in its bootloader by the firmware's own `BOOTSEL` command first; a
Leonardo uses its 1200-baud touch. **Stop the program first** - it holds the
serial port - and restart it after; the new firmware then asks, the host
answers, and `install-tower --verify` confirms the firmware holds what
`hardware.ini` declares.

## 2. Running detached (`scripts/fgc`)

The tower is operated over ssh, so the program has to survive the connection
closing. `scripts/fgc` runs it inside a tmux session, which owns the pty:

```bash
cd ~/projects/fwt_2a/software
scripts/fgc start                # starts detached, returns as soon as it is up
scripts/fgc status               # running? since when? which PID?
scripts/fgc attach               # the dashboard - Ctrl-b d detaches, leaves it running
scripts/fgc stop                 # SIGTERM -> the same clean teardown as `exit`
scripts/fgc peek                 # snapshot of the pane without attaching
```

The boot service (section 4) runs the program the same way, so all of this
works on a tower started at boot too.

**`Ctrl-b d` detaches. The dashboard's own `q` key opens an exit prompt**
offering `q` (quit the program) or `b` (background - detach and leave it
running, the same as `Ctrl-b d`), with `Esc` to cancel. `b` is offered only
inside a `scripts/fgc` session; outside one there is nothing to detach into and
it is shown dimmed.

`scripts/fgc stop` sends SIGTERM and waits up to 120 s; the program handles it
with the same shutdown path as the `exit` command (stop capture, close the
camera, flush the image queue). A stop while the gimbal is homing cancels the
homing rather than waiting for it - a full home takes minutes on some towers,
longer than this wait - so the gimbal is left un-homed, and the next start homes
it again. Do **not** `kill -9` - an unclean exit leaves the camera wedged for the
next run. If it will not stop, attach and look at why.

`scripts/fgc start` refuses if a run is already up, and prints the previous
run's last output if it died rather than silently discarding it.

Choosing **quit** at that prompt while attached ends the tmux session too, so you
land straight back in your shell. A run that *fails* is the exception: its pane
is kept so you can read why (`remain-on-exit failed`), and `fgc status` shows it
until `fgc stop` or the next `fgc start` clears it. A kept pane has **no process
behind it** - keys and Ctrl-C do nothing, which reads as a hung ssh session.
`Ctrl-b d` still detaches; as a last resort `Enter ~ .` closes the ssh client.

MQTT credentials are read from `/etc/fire_gimbal_control/credentials.env` by
`fgc` itself, inside the pane, as are the Vimba X paths: a tmux server that
already exists hands a new session its own environment, not the caller's.

> **Why `_run` waits on the app instead of `exec`ing it.** When the app is itself
> the pane's process, tmux never records an exit status for it, so
> `remain-on-exit failed` cannot tell success from failure and keeps *every*
> finished pane. Interposing a shell that reaps the child and exits with its
> status gives tmux a real one. Measured on the device: 3/3 panes kept when
> exec'd, 4/4 sessions closed cleanly when parented. The wrapper forwards
> SIGTERM/SIGINT/SIGHUP, so `fgc stop` still reaches the app.

## 3. The single-instance lock

Two instances would fight over the motor's serial port and the camera. The
binary prevents that itself: at startup it takes an exclusive `flock` on

- `/run/user/<uid>/fire_gimbal_control.lock` when that directory exists, else
- `/tmp/fire_gimbal_control-<uid>.lock`

and, if it is already held, exits with status **2** and:

```
Cannot start: another instance is already running (PID 1234); attach to it with: scripts/fgc attach
```

The path is derived from the uid and not from `$XDG_RUNTIME_DIR` on purpose: that
variable is set in an interactive ssh shell but not in a non-interactive
`ssh host cmd` one, and a lock the two launch routes disagree about would exclude
nothing. For the same reason a tower with a boot service needs **lingering**
(section 4).

Because the check lives in the binary, it covers every route in - `scripts/fgc`,
the service, or a bare `./build/fire_gimbal_control`. Nothing needs cleaning up
after a crash or a power cut: the kernel releases an `flock` when the holder
dies. Use `--no-lock` only to run a second mock-only instance during
development, and `--lock-file <path>` to point at a different lockfile.

## 4. Starting at boot

`scripts/install-tower` installs this (it asks first, and shows the unit it will
write). The unit is the template `scripts/fire-gimbal-control.service`, and it
runs the program **through `scripts/fgc`, inside tmux** - so a tower started at
boot still has its dashboard, and `fgc status` / `peek` / `stop` work from any
ssh session.

```bash
sudo systemctl start|stop|restart fire-gimbal-control
systemctl is-enabled fire-gimbal-control     # starts at boot?
sudo systemctl disable fire-gimbal-control   # not any more
```

By hand, fill in `@USER@`, `@REPO@` and `@ARGS@` (normally empty) and:

```bash
sudo cp fire-gimbal-control.service /etc/systemd/system/
sudo loginctl enable-linger ggs          # NOT optional - see below
sudo systemctl daemon-reload
sudo systemctl enable fire-gimbal-control
```

**Lingering is required.** Without it `/run/user/<uid>` does not exist until
someone logs in, so a program started at boot takes its single-instance lock in
`/tmp`; a later ssh session looks in `/run/user`, finds nothing, reports "not
running", and a second copy could start beside the first.

The service passes no startup flags. What the program does at boot is
`config.ini` `[Startup]` ([configuration.md](configuration.md#startup--homing-and-the-start-mode)):

**`home = true` homes the gimbal unattended at every boot.** That is what a tower
in service needs - the firmware keeps no position across a power cycle, so
without it the gimbal sits un-homed and captures nothing - but it is a
deliberate choice, and the installer asks for it (`startup_home`), as it does
for the start mode (`startup_mode`, `manual` or `auto`).

**Upgrading a tower from before `[Startup]`.** Its unit still passes `--init` /
`--start`, which the program now refuses. Run `scripts/install-tower` on the
tower **before its next reboot**: it moves them into `config.ini` (`--init` →
`home = true`, `--start` → `mode = auto`) and rewrites the unit without them.
`install-tower --verify` fails its `boot service args` check until that is done.

| Tower | Service | Starts with |
|---|---|---|
| 111 | enabled 2026-09-24 | `--init --start`: homes and captures at boot - `[verified]` by a reboot 2026-09-24. **To migrate**: becomes `[Startup] home = true`, `mode = auto` |
| 121 | enabled 2026-09-23 | no arguments and no `[Startup]`: starts idle, neither homes nor captures (the development tower) |

`[verified]` on tower 121, 2026-09-23: the service came up by itself after a
reboot with nobody logged in, with its lock under `/run/user/1000`; `fgc status`
and the dashboard worked from a later ssh session; credentials reached the
program intact (a password with `$`, a quote and a space); `systemctl stop` shut
it down cleanly; and `install-tower --verify` passed after the reboot, USB
preconditions included.

A tower started this way is in service when:
- images accumulate under `[Paths] output_dir` (default
  `~ggs/.local/share/fire_gimbal_control/images/RGB/`);
- CamEvents reach the broker: `mosquitto_sub -h <broker> -t 'GGS/FWT/<tower>/#' -v`.

## 5. Where the log is

The program writes a persistent log alongside the self-test and diagnostics
reports:

```
~/.local/share/fire_gimbal_control/logs/fgc_<YYYYMMDD-HHMMSS>.<NNN>.log
```

One file per run, rotated by size within a run (`.000`, `.001`, …), oldest pruned
by count. Bounded at `[Logging] file_keep x file_max_mb`, 200 MB by default, at
roughly 5 MB/day in steady state.

This exists because a tower runs detached and, in TUI mode, every line otherwise
goes to a 500-line on-screen ring buffer that dies with the tmux session. The
file is **more verbose than the dashboard** - `info` on screen, `debug` on disk -
so it holds detail the pane never showed.

```bash
ls -t ~/.local/share/fire_gimbal_control/logs/fgc_*.log | head
tail -F "$(ls -t ~/.local/share/fire_gimbal_control/logs/fgc_*.log | head -1)"
```

Use `tail -F`, not `-f`: rotation replaces the file and `-f` would follow the old
inode into a file nothing writes to any more.

Each file opens with a header naming the build, tower, PID, config and hardware
paths and the full command line - repeated after every rotation, so an archive
found on its own still says where it came from.

What to grep when a tower will not come up:

```bash
L=$(ls -t ~/.local/share/fire_gimbal_control/logs/fgc_*.log | head -1)
grep -E "hardware:|NEED|firmware config|need config" "$L"   # declaration + handshake
grep -E "\[WARN |\[ERROR" "$L"                              # everything it complained about
head -8 "$L"                                                # which build, which config
```

> Wire traces (`[Logging] trace` / `--trace`) reach this file too. The serial
> trace alone takes it from ~5 MB/day to ~41 MB/day, enough to age out the
> startup record within days - turn it on to chase something, off afterwards.

## Troubleshooting

Start with `scripts/install-tower --verify` on the tower: it checks most of the
list below and says which.

| Symptom | Check |
|---------|-------|
| `No config.ini found` | run from `~/projects/fwt_2a/software`, or `--config` / `$FGC_CONFIG` |
| `Config error` naming a key | `build/fgc-config check --config config.ini --hardware /etc/fire_gimbal_control/hardware.ini` |
| `Failed to open serial port` | MCU plugged in, `ggs` in `dialout`, `[Serial] device` in `hardware.ini` - a `/dev/serial/by-id/` path, which **changes when the MCU is flashed** |
| An axis will not move, `ERR need config` | `hardware.ini` is missing a drivetrain value for it; the log names the key |
| Camera opened, but no images and `no frame yet (stream warming up)` for minutes | the stream is stuck; the program restarts it and then resets the camera by itself (known-issues #15), or type `camera reset` |
| `No camera found` / Vimba startup error | SDK installed, GenTL path set (`/etc/profile.d/*Vimba*GenTL*.sh` or `VIMBA_CTI_PATH`), `[Camera.rgb] id` in `hardware.ini` |
| `MQTT connect failed` (degraded mode) | broker reachable, `credentials.env` readable by `ggs`, firewall |
| Runs but no images | in auto mode (`[Startup] mode = auto`, or `mode auto`) and the gimbal homed; `output_dir` writable |
| `another instance is already running` (exit 2) | `scripts/fgc status`; the service and a hand-started run both up (section 3) |
| `fgc status` says not running, but the program is | the service started it before anyone logged in, without lingering (section 4) |
| Program dies when ssh disconnects | started with `run.sh` / `deploy.sh --foreground` instead of `scripts/fgc start` |