# Running and updating a tower

How a tower that already works is operated and updated: pushing a new version
from the dev box, running the program detached, starting it at boot, reading its
log. A **first** install - a freshly imaged board, or a tower not yet migrated -
is [tower-installation.md](tower-installation.md), which `scripts/install-tower`
automates.

The fleet: tower **111** is a LattePanda 3 Delta (ATmega32U4 motor MCU, two
axes), tower **121** a LattePanda IOTA (RP2040, yaw only). Nothing here differs
between them; what differs lives in each tower's `hardware.ini`.

## Where things are on a tower

| Path | What | Written by |
|---|---|---|
| `~ggs/projects/fwt_2a/software/` | this repository, and the program's build in `build/` | `deploy.sh` (sync + build), or the installer |
| `~ggs/projects/fwt_2a/firmware/` | the firmware repository, built on the tower | `firmware/deploy.sh` |
| `.../software/config.ini` | what the tower DOES: scan, capture, logging, calibration | the installer (seeded once), `gimbal ... setup` (calibration), a person |
| `/etc/fire_gimbal_control/hardware.ini` | what the tower IS: board, axes, sensors, pins, drivetrain | the installer; a person when the hardware changes |
| `/etc/fire_gimbal_control/credentials.env` | `FGC_MQTT_USER=` / `FGC_MQTT_PW=`, mode 0640 root:ggs | the installer |
| `/etc/fire_gimbal_control/install.log` | one line per installer run | the installer |
| `~ggs/.local/share/fire_gimbal_control/logs/` | the program's log, test reports | the program |
| `.../software/config/camera_profiles/rgb/*.xml` | the camera settings, exported ON the tower from its camera | the installer, or `scripts/export_camera_profile.py` |

`deploy.sh` never touches `config.ini`, `hardware.ini`, the camera profiles or
the images: they exist only on the tower, and `rsync --delete` once erased a
freshly exported profile before they were excluded.

## 1. Updating the program from the dev box

On the dev box, in `software/`, with a `.deploy.env` naming the tower:

```ini
REMOTE_HOST=ggs@10.11.12.121
REMOTE_DIR=/home/ggs/projects/fwt_2a/software
CMAKE_ARGS="-DWITH_MQTT=ON -DWITH_VIMBA=ON"
RUN_ARGS=""
```

`RUN_ARGS` holds only extra flags (`--no-mqtt`, say). Whether the program homes
and which mode it starts in are the tower's own `config.ini` `[Startup] home` /
`mode`, the same for `--run` as for the boot service; `deploy.sh` refuses a
`RUN_ARGS` that still has `--init` or `--start`.

```bash
./deploy.sh                  # sync the tree, build on the tower
./deploy.sh --check-deps     # only check the tower's build dependencies
./deploy.sh --clean          # wipe the tower's build dir first (fresh configure)
```

It syncs the tree (leaving `.git` behind, and writing the revision into
`.revision` beside it instead), then configures and builds **on the tower**. The
program already running keeps running the old binary until it is restarted:

```bash
# on a tower started by its boot service
ssh ggs@<tower> sudo systemctl restart fire-gimbal-control
# on a tower started by hand
./deploy.sh --stop && ./deploy.sh --run          # homes / sweeps as its [Startup] says
```

To home once on a tower whose `[Startup] home` is off, attach and type
`gimbal home` (key `h`); `deploy.sh --home` is gone.

After building, `deploy.sh` also reads the tower's boot service: one that still
passes `--init`/`--start` gets a loud warning, because the new program refuses
them and **the tower would not start at its next boot**. Run
`scripts/install-tower` on the tower to fix it (section 4).

`--run` starts the program detached through `scripts/fgc` (section 2) and first
refuses a `config.ini` that still holds the example's placeholders
(`ExampleTower`, `CHANGE_ME`, `DEV_0000000000`); `--force` overrides that. On a
tower whose service is running, `--run` fails on the single-instance lock
(section 3) - restart it through systemd instead.

After an update, check it:

```bash
ssh ggs@<tower> 'cd ~/projects/fwt_2a/software && scripts/install-tower --verify'
```

It changes nothing; it runs `test hardware` and `test camera` inside the running
program (pausing and then resuming capture) and checks the files, the firmware's
pins and the boot service. `test camera` takes 15-30 s: it grabs frames, then
drives the camera's own settings - minimum exposure and gain must take every
pixel black, maximum gain and a long exposure must make the frame measurably
different - which holds whatever the camera is pointed at, and puts the profile
back.

`--push-config` also overwrites the tower's `config.ini` with the dev box's,
backing the tower's copy up first. It exists for the development tower; a
tower's `config.ini` holds its own calibration, so do not push one tower's to
another. `hardware.ini` is never pushed.

> **One-time: `git pull` can delete your `.deploy.env`.** It was tracked by
> mistake until `b0127db`, which stopped tracking it and added it to
> `.gitignore`. Pulling *across* that commit deletes the working copy, credentials
> and all, with no warning. Only checkouts that predate `b0127db` are exposed,
> and only once: copy it aside first (`cp .deploy.env ../deploy.env.bak`), pull,
> copy it back. To recover one already lost: `git show b0127db^:.deploy.env`.

### Updating the firmware

**Host first, firmware second**, always: a new firmware asks the host for its
drivetrain at boot, and an older host cannot answer. In `firmware/`, with its own
`.deploy.env` (`REMOTE_HOST`, `REMOTE_DIR`, `PIO_ENV` = `leonardo` or
`iota_rp2040`, `SERIAL_PORT` = the MCU's `/dev/serial/by-id/` path):

```bash
./deploy.sh --compile-only   # sync + build on the tower, flash nothing
./deploy.sh                  # build + flash
./deploy.sh --monitor        # ... then watch it boot
```

The build takes the pins from the tower's `hardware.ini` (`[Pins]`) and fails
without them. The flashing is `firmware/tools/flash.sh`, run on the tower: an
RP2040 is put in its bootloader by the firmware's own `BOOTSEL` command first; a
Leonardo uses its 1200-baud touch. **Stop the program first** - it holds the
serial port - and restart it after; the new firmware then asks, the host
answers, and `install-tower --verify` confirms the firmware holds what
`hardware.ini` declares.

## 2. Running detached (`scripts/fgc`)

The tower is operated over ssh, so the program has to survive the connection
closing. `scripts/fgc` runs it inside a tmux session, which owns the pty:

```bash
cd ~/projects/fwt_2a/software
scripts/fgc start                # starts detached, returns as soon as it is up
scripts/fgc status               # running? since when? which PID?
scripts/fgc attach               # the dashboard - Ctrl-b d detaches, leaves it running
scripts/fgc stop                 # SIGTERM -> the same clean teardown as `exit`
scripts/fgc peek                 # snapshot of the pane without attaching
```

The boot service (section 4) runs the program the same way, so all of this
works on a tower started at boot too.

**`Ctrl-b d` detaches. The dashboard's own `q` key opens an exit prompt**
offering `q` (quit the program) or `b` (background - detach and leave it
running, the same as `Ctrl-b d`), with `Esc` to cancel. `b` is offered only
inside a `scripts/fgc` session; outside one there is nothing to detach into and
it is shown dimmed.

`scripts/fgc stop` sends SIGTERM and waits up to 120 s; the program handles it
with the same shutdown path as the `exit` command (stop capture, close the
camera, flush the image queue). Do **not** `kill -9` - an unclean exit leaves the
camera wedged for the next run. If it will not stop, attach and look at why.

`scripts/fgc start` refuses if a run is already up, and prints the previous
run's last output if it died rather than silently discarding it.

Choosing **quit** at that prompt while attached ends the tmux session too, so you
land straight back in your shell. A run that *fails* is the exception: its pane
is kept so you can read why (`remain-on-exit failed`), and `fgc status` shows it
until `fgc stop` or the next `fgc start` clears it. A kept pane has **no process
behind it** - keys and Ctrl-C do nothing, which reads as a hung ssh session.
`Ctrl-b d` still detaches; as a last resort `Enter ~ .` closes the ssh client.

MQTT credentials are read from `/etc/fire_gimbal_control/credentials.env` by
`fgc` itself, inside the pane, as are the Vimba X paths: a tmux server that
already exists hands a new session its own environment, not the caller's.

> **Why `_run` waits on the app instead of `exec`ing it.** When the app is itself
> the pane's process, tmux never records an exit status for it, so
> `remain-on-exit failed` cannot tell success from failure and keeps *every*
> finished pane. Interposing a shell that reaps the child and exits with its
> status gives tmux a real one. Measured on the device: 3/3 panes kept when
> exec'd, 4/4 sessions closed cleanly when parented. The wrapper forwards
> SIGTERM/SIGINT/SIGHUP, so `fgc stop` still reaches the app.

## 3. The single-instance lock

Two instances would fight over the motor's serial port and the camera. The
binary prevents that itself: at startup it takes an exclusive `flock` on

- `/run/user/<uid>/fire_gimbal_control.lock` when that directory exists, else
- `/tmp/fire_gimbal_control-<uid>.lock`

and, if it is already held, exits with status **2** and:

```
Cannot start: another instance is already running (PID 1234); attach to it with: scripts/fgc attach
```

The path is derived from the uid and not from `$XDG_RUNTIME_DIR` on purpose: that
variable is set in an interactive ssh shell but not in a non-interactive
`ssh host cmd` one, and a lock the two launch routes disagree about would exclude
nothing. For the same reason a tower with a boot service needs **lingering**
(section 4).

Because the check lives in the binary, it covers every route in - `scripts/fgc`,
the service, or a bare `./build/fire_gimbal_control`. Nothing needs cleaning up
after a crash or a power cut: the kernel releases an `flock` when the holder
dies. Use `--no-lock` only to run a second mock-only instance during
development, and `--lock-file <path>` to point at a different lockfile.

## 4. Starting at boot

`scripts/install-tower` installs this (it asks first, and shows the unit it will
write). The unit is the template `scripts/fire-gimbal-control.service`, and it
runs the program **through `scripts/fgc`, inside tmux** - so a tower started at
boot still has its dashboard, and `fgc status` / `peek` / `stop` work from any
ssh session.

```bash
sudo systemctl start|stop|restart fire-gimbal-control
systemctl is-enabled fire-gimbal-control     # starts at boot?
sudo systemctl disable fire-gimbal-control   # not any more
```

By hand, fill in `@USER@`, `@REPO@` and `@ARGS@` (normally empty) and:

```bash
sudo cp fire-gimbal-control.service /etc/systemd/system/
sudo loginctl enable-linger ggs          # NOT optional - see below
sudo systemctl daemon-reload
sudo systemctl enable fire-gimbal-control
```

**Lingering is required.** Without it `/run/user/<uid>` does not exist until
someone logs in, so a program started at boot takes its single-instance lock in
`/tmp`; a later ssh session looks in `/run/user`, finds nothing, reports "not
running", and a second copy could start beside the first.

The service passes no startup flags. What the program does at boot is
`config.ini` `[Startup]` ([configuration.md](configuration.md#startup--homing-and-the-start-mode)):

**`home = true` homes the gimbal unattended at every boot.** That is what a tower
in service needs - the firmware keeps no position across a power cycle, so
without it the gimbal sits un-homed and captures nothing - but it is a
deliberate choice, and the installer asks for it (`startup_home`), as it does
for the start mode (`startup_mode`, `manual` or `auto`).

**Upgrading a tower from before `[Startup]`.** Its unit still passes `--init` /
`--start`, which the program now refuses. Run `scripts/install-tower` on the
tower **before its next reboot**: it moves them into `config.ini` (`--init` →
`home = true`, `--start` → `mode = auto`) and rewrites the unit without them.
`install-tower --verify` fails its `boot service args` check until that is done.

| Tower | Service | Starts with |
|---|---|---|
| 111 | enabled 2026-09-24 | `--init --start`: homes and captures at boot - `[verified]` by a reboot 2026-09-24. **To migrate**: becomes `[Startup] home = true`, `mode = auto` |
| 121 | enabled 2026-09-23 | no arguments and no `[Startup]`: starts idle, neither homes nor captures (the development tower) |

`[verified]` on tower 121, 2026-09-23: the service came up by itself after a
reboot with nobody logged in, with its lock under `/run/user/1000`; `fgc status`
and the dashboard worked from a later ssh session; credentials reached the
program intact (a password with `$`, a quote and a space); `systemctl stop` shut
it down cleanly; and `install-tower --verify` passed after the reboot, USB
preconditions included.

A tower started this way is in service when:
- images accumulate under `[Paths] output_dir` (default
  `~ggs/.local/share/fire_gimbal_control/images/RGB/`);
- CamEvents reach the broker: `mosquitto_sub -h <broker> -t 'GGS/FWT/<tower>/#' -v`.

## 5. Where the log is

The program writes a persistent log alongside the self-test and diagnostics
reports:

```
~/.local/share/fire_gimbal_control/logs/fgc_<YYYYMMDD-HHMMSS>.<NNN>.log
```

One file per run, rotated by size within a run (`.000`, `.001`, …), oldest pruned
by count. Bounded at `[Logging] file_keep x file_max_mb`, 200 MB by default, at
roughly 5 MB/day in steady state.

This exists because a tower runs detached and, in TUI mode, every line otherwise
goes to a 500-line on-screen ring buffer that dies with the tmux session. The
file is **more verbose than the dashboard** - `info` on screen, `debug` on disk -
so it holds detail the pane never showed.

```bash
ls -t ~/.local/share/fire_gimbal_control/logs/fgc_*.log | head
tail -F "$(ls -t ~/.local/share/fire_gimbal_control/logs/fgc_*.log | head -1)"
```

Use `tail -F`, not `-f`: rotation replaces the file and `-f` would follow the old
inode into a file nothing writes to any more.

Each file opens with a header naming the build, tower, PID, config and hardware
paths and the full command line - repeated after every rotation, so an archive
found on its own still says where it came from.

What to grep when a tower will not come up:

```bash
L=$(ls -t ~/.local/share/fire_gimbal_control/logs/fgc_*.log | head -1)
grep -E "hardware:|NEED|firmware config|need config" "$L"   # declaration + handshake
grep -E "\[WARN |\[ERROR" "$L"                              # everything it complained about
head -8 "$L"                                                # which build, which config
```

> Wire traces (`[Logging] trace` / `--trace`) reach this file too. The serial
> trace alone takes it from ~5 MB/day to ~41 MB/day, enough to age out the
> startup record within days - turn it on to chase something, off afterwards.

## Troubleshooting

Start with `scripts/install-tower --verify` on the tower: it checks most of the
list below and says which.

| Symptom | Check |
|---------|-------|
| `No config.ini found` | run from `~/projects/fwt_2a/software`, or `--config` / `$FGC_CONFIG` |
| `Config error` naming a key | `build/fgc-config check --config config.ini --hardware /etc/fire_gimbal_control/hardware.ini` |
| `Failed to open serial port` | MCU plugged in, `ggs` in `dialout`, `[Serial] device` in `hardware.ini` - a `/dev/serial/by-id/` path, which **changes when the MCU is flashed** |
| An axis will not move, `ERR need config` | `hardware.ini` is missing a drivetrain value for it; the log names the key |
| Camera opened, but no images and `no frame yet (stream warming up)` for minutes | the stream is stuck; the program restarts it and then resets the camera by itself (known-issues #15), or type `camera reset` |
| `No camera found` / Vimba startup error | SDK installed, GenTL path set (`/etc/profile.d/*Vimba*GenTL*.sh` or `VIMBA_CTI_PATH`), `[Camera.rgb] id` in `hardware.ini` |
| `MQTT connect failed` (degraded mode) | broker reachable, `credentials.env` readable by `ggs`, firewall |
| Runs but no images | in auto mode (`[Startup] mode = auto`, or `mode auto`) and the gimbal homed; `output_dir` writable |
| `another instance is already running` (exit 2) | `scripts/fgc status`; the service and a hand-started run both up (section 3) |
| `fgc status` says not running, but the program is | the service started it before anyone logged in, without lingering (section 4) |
| Program dies when ssh disconnects | started with `run.sh` / `deploy.sh --foreground` instead of `scripts/fgc start` |