fwt_software/docs/deployment.md

Running and updating a tower

How a tower that already works is operated and updated: pushing a new version from the dev box, running the program detached, starting it at boot, reading its log. A first install - a freshly imaged board, or a tower not yet migrated - is tower-installation.md, which scripts/install-tower automates.

The fleet: tower 111 is a LattePanda 3 Delta (ATmega32U4 motor MCU, two axes), tower 121 a LattePanda IOTA (RP2040, yaw only). Nothing here differs between them; what differs lives in each tower's hardware.ini.

Where things are on a tower

Path	What	Written by
~ggs/projects/fwt_2a/software/	this repository, and the program's build in build/	deploy.sh (sync + build), or the installer
~ggs/projects/fwt_2a/firmware/	the firmware repository, built on the tower	firmware/deploy.sh
~ggs/.config/fire_gimbal_control/config.ini	what the tower DOES: scan, capture, logging, calibration	the installer (seeded once), gimbal ... setup (calibration), a person
~ggs/.config/fire_gimbal_control/scan.csv	the scan grid chosen for this tower ([Scan] grid_file)	the installer (the example grid), a person
/etc/fire_gimbal_control/hardware.ini	what the tower IS: board, axes, sensors, pins, drivetrain	the installer; a person when the hardware changes
/etc/fire_gimbal_control/credentials.env	FGC_MQTT_USER= / FGC_MQTT_PW=, mode 0640 root:ggs	the installer
/etc/fire_gimbal_control/install.log	one line per installer run	the installer
~ggs/.local/share/fire_gimbal_control/logs/	the program's log, test reports	the program
~ggs/.config/fire_gimbal_control/camera_profiles/rgb/*.xml	the camera settings, exported ON the tower from its camera	the installer, or scripts/export_camera_profile.py

The rule: what a tower changes lives in ~/.config/fire_gimbal_control/, what it never changes in /etc/fire_gimbal_control/, and the project folder holds only what a deploy or an update may replace. deploy.sh mirrors the project folder with rsync --delete, so nothing of the tower's own may live there - rsync --delete once erased a freshly exported profile that did.

A tower set up before the tower folder existed

Its config.ini, scan grid and camera profiles are still in the project folder. The program loads them from there and warns at every start, and every deploy prints how to move them. Until then deploy.sh's excludes keep them safe. Move them once, on the tower:

ssh ggs@<tower>
cd ~/projects/fwt_2a/software
scripts/tower-config move --dry-run   # what moves where, and what config.ini will say
scripts/fgc stop                      # the program writes config.ini itself
scripts/tower-config move
scripts/fgc start                     # or: sudo systemctl start fire-gimbal-control

It copies config.ini, the scan grid [Scan] grid_file names and every camera's profiles into the tower folder, and rewrites those two paths (config/scan.csv becomes scan.csv). fgc-config checks the result loads before it is written. The old config.ini is renamed config.ini.moved-<timestamp>, and the old grid and profiles stay where they were. scripts/install-tower does the same move as one of its steps.

1. Updating the program from the dev box

On the dev box, in software/, with a .deploy.env naming the tower:

REMOTE_HOST=ggs@10.11.12.121
REMOTE_DIR=/home/ggs/projects/fwt_2a/software
CMAKE_ARGS="-DWITH_MQTT=ON -DWITH_VIMBA=ON"
RUN_ARGS=""

RUN_ARGS holds only extra flags (--no-mqtt, say). Whether the program homes and which mode it starts in are the tower's own config.ini [Startup] home / mode, the same for --run as for the boot service; deploy.sh refuses a RUN_ARGS that still has --init or --start.

./deploy.sh                  # sync the tree, build on the tower
./deploy.sh --check-deps     # only check the tower's build dependencies
./deploy.sh --clean          # wipe the tower's build dir first (fresh configure)

It syncs the tree (leaving .git behind, and writing the revision into .revision beside it instead), then configures and builds on the tower. The program already running keeps running the old binary until it is restarted:

# on a tower started by its boot service
ssh ggs@<tower> sudo systemctl restart fire-gimbal-control
# on a tower started by hand
./deploy.sh --stop && ./deploy.sh --run          # homes / sweeps as its [Startup] says

To home once on a tower whose [Startup] home is off, attach and type gimbal home (key h); deploy.sh --home is gone.

After building, deploy.sh also reads the tower's boot service: one that still passes --init/--start gets a loud warning, because the new program refuses them and the tower would not start at its next boot. Run scripts/install-tower on the tower to fix it (section 4).

--run starts the program detached through scripts/fgc (section 2) and first refuses a config.ini that still holds the example's placeholders (ExampleTower, CHANGE_ME, DEV_0000000000); --force overrides that. On a tower whose service is running, --run fails on the single-instance lock (section 3) - restart it through systemd instead.

After an update, check it:

ssh ggs@<tower> 'cd ~/projects/fwt_2a/software && scripts/install-tower --verify'

It changes nothing; it runs test hardware and test camera inside the running program (pausing and then resuming capture) and checks the files, the firmware's pins and the boot service. test camera takes 15-30 s: it grabs frames, then drives the camera's own settings - minimum exposure and gain must take every pixel black, maximum gain and a long exposure must make the frame measurably different - which holds whatever the camera is pointed at, and puts the profile back.

--push-config also overwrites the tower's config.ini with the dev box's, backing the tower's copy up first. It exists for the development tower; a tower's config.ini holds its own calibration, so do not push one tower's to another. hardware.ini is never pushed.

One-time: git pull can delete your .deploy.env. It was tracked by mistake until b0127db, which stopped tracking it and added it to .gitignore. Pulling across that commit deletes the working copy, credentials and all, with no warning. Only checkouts that predate b0127db are exposed, and only once: copy it aside first (cp .deploy.env ../deploy.env.bak), pull, copy it back. To recover one already lost: git show b0127db^:.deploy.env.

Updating the firmware

Host first, firmware second, always: a new firmware asks the host for its drivetrain at boot, and an older host cannot answer. In firmware/, with its own .deploy.env (REMOTE_HOST, REMOTE_DIR, PIO_ENV = leonardo or iota_rp2040, SERIAL_PORT = the MCU's /dev/serial/by-id/ path):

./deploy.sh --compile-only   # sync + build on the tower, flash nothing
./deploy.sh                  # build + flash
./deploy.sh --monitor        # ... then watch it boot

The build takes the pins from the tower's hardware.ini ([Pins]) and fails without them. The flashing is firmware/tools/flash.sh, run on the tower: an RP2040 is put in its bootloader by the firmware's own BOOTSEL command first; a Leonardo uses its 1200-baud touch. Stop the program first - it holds the serial port - and restart it after; the new firmware then asks, the host answers, and install-tower --verify confirms the firmware holds what hardware.ini declares.

2. Running detached (scripts/fgc)

The tower is operated over ssh, so the program has to survive the connection closing. scripts/fgc runs it inside a tmux session, which owns the pty:

cd ~/projects/fwt_2a/software
scripts/fgc start                # starts detached, returns as soon as it is up
scripts/fgc status               # running? since when? which PID?
scripts/fgc attach               # the dashboard - Ctrl-b d detaches, leaves it running
scripts/fgc stop                 # SIGTERM -> the same clean teardown as `exit`
scripts/fgc peek                 # snapshot of the pane without attaching

The boot service (section 4) runs the program the same way, so all of this works on a tower started at boot too.

Ctrl-b d detaches. The dashboard's own q key opens an exit prompt offering q (quit the program) or b (background - detach and leave it running, the same as Ctrl-b d), with Esc to cancel. b is offered only inside a scripts/fgc session; outside one there is nothing to detach into and it is shown dimmed.

scripts/fgc stop sends SIGTERM and waits up to 120 s; the program handles it with the same shutdown path as the exit command (stop capture, close the camera, flush the image queue). A stop while the gimbal is homing cancels the homing rather than waiting for it - a full home takes minutes on some towers, longer than this wait - so the gimbal is left un-homed, and the next start homes it again. Do not kill -9 - an unclean exit leaves the camera wedged for the next run. If it will not stop, attach and look at why.

scripts/fgc start refuses if a run is already up, and prints the previous run's last output if it died rather than silently discarding it.

Choosing quit at that prompt while attached ends the tmux session too, so you land straight back in your shell. A run that fails is the exception: its pane is kept so you can read why (remain-on-exit failed), and fgc status shows it until fgc stop or the next fgc start clears it. A kept pane has no process behind it - keys and Ctrl-C do nothing, which reads as a hung ssh session. Ctrl-b d still detaches; as a last resort Enter ~ . closes the ssh client.

MQTT credentials are read from /etc/fire_gimbal_control/credentials.env by fgc itself, inside the pane, as are the Vimba X paths: a tmux server that already exists hands a new session its own environment, not the caller's.

Why _run waits on the app instead of execing it. When the app is itself the pane's process, tmux never records an exit status for it, so remain-on-exit failed cannot tell success from failure and keeps every finished pane. Interposing a shell that reaps the child and exits with its status gives tmux a real one. Measured on the device: 3/3 panes kept when exec'd, 4/4 sessions closed cleanly when parented. The wrapper forwards SIGTERM/SIGINT/SIGHUP, so fgc stop still reaches the app.

3. The single-instance lock

Two instances would fight over the motor's serial port and the camera. The binary prevents that itself: at startup it takes an exclusive flock on

• /run/user/<uid>/fire_gimbal_control.lock when that directory exists, else
• /tmp/fire_gimbal_control-<uid>.lock

and, if it is already held, exits with status 2 and:

Cannot start: another instance is already running (PID 1234); attach to it with: scripts/fgc attach

The path is derived from the uid and not from $XDG_RUNTIME_DIR on purpose: that variable is set in an interactive ssh shell but not in a non-interactive ssh host cmd one, and a lock the two launch routes disagree about would exclude nothing. For the same reason a tower with a boot service needs lingering (section 4).

Because the check lives in the binary, it covers every route in - scripts/fgc, the service, or a bare ./build/fire_gimbal_control. Nothing needs cleaning up after a crash or a power cut: the kernel releases an flock when the holder dies. Use --no-lock only to run a second mock-only instance during development, and --lock-file <path> to point at a different lockfile.

4. Starting at boot

scripts/install-tower installs this (it asks first, and shows the unit it will write). The unit is the template scripts/fire-gimbal-control.service, and it runs the program through scripts/fgc, inside tmux - so a tower started at boot still has its dashboard, and fgc status / peek / stop work from any ssh session.

sudo systemctl start|stop|restart fire-gimbal-control
systemctl is-enabled fire-gimbal-control     # starts at boot?
sudo systemctl disable fire-gimbal-control   # not any more

By hand, fill in @USER@, @REPO@ and @ARGS@ (normally empty) and:

sudo cp fire-gimbal-control.service /etc/systemd/system/
sudo loginctl enable-linger ggs          # NOT optional - see below
sudo systemctl daemon-reload
sudo systemctl enable fire-gimbal-control

Lingering is required. Without it /run/user/<uid> does not exist until someone logs in, so a program started at boot takes its single-instance lock in /tmp; a later ssh session looks in /run/user, finds nothing, reports "not running", and a second copy could start beside the first.

The service passes no startup flags. What the program does at boot is config.ini [Startup] (configuration.md):

home = true homes the gimbal unattended at every boot. That is what a tower in service needs - the firmware keeps no position across a power cycle, so without it the gimbal sits un-homed and captures nothing - but it is a deliberate choice, and the installer asks for it (startup_home), as it does for the start mode (startup_mode, manual or auto).

Upgrading a tower from before [Startup]. Its unit still passes --init / --start, which the program now refuses. Run scripts/install-tower on the tower before its next reboot: it moves them into config.ini (--init → home = true, --start → mode = auto) and rewrites the unit without them. install-tower --verify fails its boot service args check until that is done.

Tower	Service	Starts with
111	enabled 2026-09-24	--init --start: homes and captures at boot - [verified] by a reboot 2026-09-24. To migrate: becomes [Startup] home = true, mode = auto
121	enabled 2026-09-23	no arguments and no [Startup]: starts idle, neither homes nor captures (the development tower)

[verified] on tower 121, 2026-09-23: the service came up by itself after a reboot with nobody logged in, with its lock under /run/user/1000; fgc status and the dashboard worked from a later ssh session; credentials reached the program intact (a password with $, a quote and a space); systemctl stop shut it down cleanly; and install-tower --verify passed after the reboot, USB preconditions included.

A tower started this way is in service when:

• images accumulate under [Paths] output_dir (default ~ggs/.local/share/fire_gimbal_control/images/RGB/);
• CamEvents reach the broker: mosquitto_sub -h <broker> -t 'GGS/FWT/<tower>/#' -v.

5. Where the log is

The program writes a persistent log alongside the self-test and diagnostics reports:

~/.local/share/fire_gimbal_control/logs/fgc_<YYYYMMDD-HHMMSS>.<NNN>.log

One file per run, rotated by size within a run (.000, .001, …), oldest pruned by count. Bounded at [Logging] file_keep x file_max_mb, 200 MB by default, at roughly 5 MB/day in steady state.

This exists because a tower runs detached and, in TUI mode, every line otherwise goes to a 500-line on-screen ring buffer that dies with the tmux session. The file is more verbose than the dashboard - info on screen, debug on disk - so it holds detail the pane never showed.

ls -t ~/.local/share/fire_gimbal_control/logs/fgc_*.log | head
tail -F "$(ls -t ~/.local/share/fire_gimbal_control/logs/fgc_*.log | head -1)"

Use tail -F, not -f: rotation replaces the file and -f would follow the old inode into a file nothing writes to any more.

Each file opens with a header naming the build, tower, PID, config and hardware paths and the full command line - repeated after every rotation, so an archive found on its own still says where it came from.

What to grep when a tower will not come up:

L=$(ls -t ~/.local/share/fire_gimbal_control/logs/fgc_*.log | head -1)
grep -E "hardware:|NEED|firmware config|need config" "$L"   # declaration + handshake
grep -E "\[WARN |\[ERROR" "$L"                              # everything it complained about
head -8 "$L"                                                # which build, which config

Wire traces ([Logging] trace / --trace) reach this file too. The serial trace alone takes it from ~5 MB/day to ~41 MB/day, enough to age out the startup record within days - turn it on to chase something, off afterwards.

Troubleshooting

Start with scripts/install-tower --verify on the tower: it checks most of the list below and says which.

Symptom	Check
No config.ini found	run from ~/projects/fwt_2a/software, or --config / $FGC_CONFIG
Config error naming a key	build/fgc-config check --config config.ini --hardware /etc/fire_gimbal_control/hardware.ini
Failed to open serial port	MCU plugged in, ggs in dialout, [Serial] device in hardware.ini - a /dev/serial/by-id/ path, which changes when the MCU is flashed
An axis will not move, ERR need config	hardware.ini is missing a drivetrain value for it; the log names the key
Camera opened, but no images and no frame yet (stream warming up) for minutes	the stream is stuck; the program restarts it and then resets the camera by itself (known-issues #15), or type camera reset
No camera found / Vimba startup error	SDK installed, GenTL path set (/etc/profile.d/*Vimba*GenTL*.sh or VIMBA_CTI_PATH), [Camera.rgb] id in hardware.ini
MQTT connect failed (degraded mode)	broker reachable, credentials.env readable by ggs, firewall
Runs but no images	in auto mode ([Startup] mode = auto, or mode auto) and the gimbal homed; output_dir writable
another instance is already running (exit 2)	scripts/fgc status; the service and a hand-started run both up (section 3)
fgc status says not running, but the program is	the service started it before anyone logged in, without lingering (section 4)
Program dies when ssh disconnects	started with run.sh / deploy.sh --foreground instead of scripts/fgc start