321 lines
16 KiB
Markdown
321 lines
16 KiB
Markdown
# Running and updating a tower
|
|
|
|
How a tower that already works is operated and updated: pushing a new version
|
|
from the dev box, running the program detached, starting it at boot, reading its
|
|
log. A **first** install - a freshly imaged board, or a tower not yet migrated -
|
|
is [tower-installation.md](tower-installation.md), which `scripts/install-tower`
|
|
automates.
|
|
|
|
The fleet: tower **111** is a LattePanda 3 Delta (ATmega32U4 motor MCU, two
|
|
axes), tower **121** a LattePanda IOTA (RP2040, yaw only). Nothing here differs
|
|
between them; what differs lives in each tower's `hardware.ini`.
|
|
|
|
## Where things are on a tower
|
|
|
|
| Path | What | Written by |
|
|
|---|---|---|
|
|
| `~ggs/projects/fwt_2a/software/` | this repository, and the program's build in `build/` | `deploy.sh` (sync + build), or the installer |
|
|
| `~ggs/projects/fwt_2a/firmware/` | the firmware repository, built on the tower | `firmware/deploy.sh` |
|
|
| `.../software/config.ini` | what the tower DOES: scan, capture, logging, calibration | the installer (seeded once), `gimbal ... setup` (calibration), a person |
|
|
| `/etc/fire_gimbal_control/hardware.ini` | what the tower IS: board, axes, sensors, pins, drivetrain | the installer; a person when the hardware changes |
|
|
| `/etc/fire_gimbal_control/credentials.env` | `FGC_MQTT_USER=` / `FGC_MQTT_PW=`, mode 0640 root:ggs | the installer |
|
|
| `/etc/fire_gimbal_control/install.log` | one line per installer run | the installer |
|
|
| `~ggs/.local/share/fire_gimbal_control/logs/` | the program's log, test reports | the program |
|
|
| `.../software/config/camera_profiles/rgb/*.xml` | the camera settings, exported ON the tower from its camera | the installer, or `scripts/export_camera_profile.py` |
|
|
|
|
`deploy.sh` never touches `config.ini`, `hardware.ini`, the camera profiles or
|
|
the images: they exist only on the tower, and `rsync --delete` once erased a
|
|
freshly exported profile before they were excluded.
|
|
|
|
## 1. Updating the program from the dev box
|
|
|
|
On the dev box, in `software/`, with a `.deploy.env` naming the tower:
|
|
|
|
```ini
|
|
REMOTE_HOST=ggs@10.11.12.121
|
|
REMOTE_DIR=/home/ggs/projects/fwt_2a/software
|
|
CMAKE_ARGS="-DWITH_MQTT=ON -DWITH_VIMBA=ON"
|
|
RUN_ARGS=""
|
|
```
|
|
|
|
`RUN_ARGS` holds only extra flags (`--no-mqtt`, say). Whether the program homes
|
|
and which mode it starts in are the tower's own `config.ini` `[Startup] home` /
|
|
`mode`, the same for `--run` as for the boot service; `deploy.sh` refuses a
|
|
`RUN_ARGS` that still has `--init` or `--start`.
|
|
|
|
```bash
|
|
./deploy.sh # sync the tree, build on the tower
|
|
./deploy.sh --check-deps # only check the tower's build dependencies
|
|
./deploy.sh --clean # wipe the tower's build dir first (fresh configure)
|
|
```
|
|
|
|
It syncs the tree (leaving `.git` behind, and writing the revision into
|
|
`.revision` beside it instead), then configures and builds **on the tower**. The
|
|
program already running keeps running the old binary until it is restarted:
|
|
|
|
```bash
|
|
# on a tower started by its boot service
|
|
ssh ggs@<tower> sudo systemctl restart fire-gimbal-control
|
|
# on a tower started by hand
|
|
./deploy.sh --stop && ./deploy.sh --run # homes / sweeps as its [Startup] says
|
|
```
|
|
|
|
To home once on a tower whose `[Startup] home` is off, attach and type
|
|
`gimbal home` (key `h`); `deploy.sh --home` is gone.
|
|
|
|
After building, `deploy.sh` also reads the tower's boot service: one that still
|
|
passes `--init`/`--start` gets a loud warning, because the new program refuses
|
|
them and **the tower would not start at its next boot**. Run
|
|
`scripts/install-tower` on the tower to fix it (section 4).
|
|
|
|
`--run` starts the program detached through `scripts/fgc` (section 2) and first
|
|
refuses a `config.ini` that still holds the example's placeholders
|
|
(`ExampleTower`, `CHANGE_ME`, `DEV_0000000000`); `--force` overrides that. On a
|
|
tower whose service is running, `--run` fails on the single-instance lock
|
|
(section 3) - restart it through systemd instead.
|
|
|
|
After an update, check it:
|
|
|
|
```bash
|
|
ssh ggs@<tower> 'cd ~/projects/fwt_2a/software && scripts/install-tower --verify'
|
|
```
|
|
|
|
It changes nothing; it runs `test hardware` and `test camera` inside the running
|
|
program (pausing and then resuming capture) and checks the files, the firmware's
|
|
pins and the boot service. `test camera` takes 15-30 s: it grabs frames, then
|
|
drives the camera's own settings - minimum exposure and gain must take every
|
|
pixel black, maximum gain and a long exposure must make the frame measurably
|
|
different - which holds whatever the camera is pointed at, and puts the profile
|
|
back.
|
|
|
|
`--push-config` also overwrites the tower's `config.ini` with the dev box's,
|
|
backing the tower's copy up first. It exists for the development tower; a
|
|
tower's `config.ini` holds its own calibration, so do not push one tower's to
|
|
another. `hardware.ini` is never pushed.
|
|
|
|
> **One-time: `git pull` can delete your `.deploy.env`.** It was tracked by
|
|
> mistake until `b0127db`, which stopped tracking it and added it to
|
|
> `.gitignore`. Pulling *across* that commit deletes the working copy, credentials
|
|
> and all, with no warning. Only checkouts that predate `b0127db` are exposed,
|
|
> and only once: copy it aside first (`cp .deploy.env ../deploy.env.bak`), pull,
|
|
> copy it back. To recover one already lost: `git show b0127db^:.deploy.env`.
|
|
|
|
### Updating the firmware
|
|
|
|
**Host first, firmware second**, always: a new firmware asks the host for its
|
|
drivetrain at boot, and an older host cannot answer. In `firmware/`, with its own
|
|
`.deploy.env` (`REMOTE_HOST`, `REMOTE_DIR`, `PIO_ENV` = `leonardo` or
|
|
`iota_rp2040`, `SERIAL_PORT` = the MCU's `/dev/serial/by-id/` path):
|
|
|
|
```bash
|
|
./deploy.sh --compile-only # sync + build on the tower, flash nothing
|
|
./deploy.sh # build + flash
|
|
./deploy.sh --monitor # ... then watch it boot
|
|
```
|
|
|
|
The build takes the pins from the tower's `hardware.ini` (`[Pins]`) and fails
|
|
without them. The flashing is `firmware/tools/flash.sh`, run on the tower: an
|
|
RP2040 is put in its bootloader by the firmware's own `BOOTSEL` command first; a
|
|
Leonardo uses its 1200-baud touch. **Stop the program first** - it holds the
|
|
serial port - and restart it after; the new firmware then asks, the host
|
|
answers, and `install-tower --verify` confirms the firmware holds what
|
|
`hardware.ini` declares.
|
|
|
|
## 2. Running detached (`scripts/fgc`)
|
|
|
|
The tower is operated over ssh, so the program has to survive the connection
|
|
closing. `scripts/fgc` runs it inside a tmux session, which owns the pty:
|
|
|
|
```bash
|
|
cd ~/projects/fwt_2a/software
|
|
scripts/fgc start # starts detached, returns as soon as it is up
|
|
scripts/fgc status # running? since when? which PID?
|
|
scripts/fgc attach # the dashboard - Ctrl-b d detaches, leaves it running
|
|
scripts/fgc stop # SIGTERM -> the same clean teardown as `exit`
|
|
scripts/fgc peek # snapshot of the pane without attaching
|
|
```
|
|
|
|
The boot service (section 4) runs the program the same way, so all of this
|
|
works on a tower started at boot too.
|
|
|
|
**`Ctrl-b d` detaches. The dashboard's own `q` key opens an exit prompt**
|
|
offering `q` (quit the program) or `b` (background - detach and leave it
|
|
running, the same as `Ctrl-b d`), with `Esc` to cancel. `b` is offered only
|
|
inside a `scripts/fgc` session; outside one there is nothing to detach into and
|
|
it is shown dimmed.
|
|
|
|
`scripts/fgc stop` sends SIGTERM and waits up to 120 s; the program handles it
|
|
with the same shutdown path as the `exit` command (stop capture, close the
|
|
camera, flush the image queue). Do **not** `kill -9` - an unclean exit leaves the
|
|
camera wedged for the next run. If it will not stop, attach and look at why.
|
|
|
|
`scripts/fgc start` refuses if a run is already up, and prints the previous
|
|
run's last output if it died rather than silently discarding it.
|
|
|
|
Choosing **quit** at that prompt while attached ends the tmux session too, so you
|
|
land straight back in your shell. A run that *fails* is the exception: its pane
|
|
is kept so you can read why (`remain-on-exit failed`), and `fgc status` shows it
|
|
until `fgc stop` or the next `fgc start` clears it. A kept pane has **no process
|
|
behind it** - keys and Ctrl-C do nothing, which reads as a hung ssh session.
|
|
`Ctrl-b d` still detaches; as a last resort `Enter ~ .` closes the ssh client.
|
|
|
|
MQTT credentials are read from `/etc/fire_gimbal_control/credentials.env` by
|
|
`fgc` itself, inside the pane, as are the Vimba X paths: a tmux server that
|
|
already exists hands a new session its own environment, not the caller's.
|
|
|
|
> **Why `_run` waits on the app instead of `exec`ing it.** When the app is itself
|
|
> the pane's process, tmux never records an exit status for it, so
|
|
> `remain-on-exit failed` cannot tell success from failure and keeps *every*
|
|
> finished pane. Interposing a shell that reaps the child and exits with its
|
|
> status gives tmux a real one. Measured on the device: 3/3 panes kept when
|
|
> exec'd, 4/4 sessions closed cleanly when parented. The wrapper forwards
|
|
> SIGTERM/SIGINT/SIGHUP, so `fgc stop` still reaches the app.
|
|
|
|
## 3. The single-instance lock
|
|
|
|
Two instances would fight over the motor's serial port and the camera. The
|
|
binary prevents that itself: at startup it takes an exclusive `flock` on
|
|
|
|
- `/run/user/<uid>/fire_gimbal_control.lock` when that directory exists, else
|
|
- `/tmp/fire_gimbal_control-<uid>.lock`
|
|
|
|
and, if it is already held, exits with status **2** and:
|
|
|
|
```
|
|
Cannot start: another instance is already running (PID 1234); attach to it with: scripts/fgc attach
|
|
```
|
|
|
|
The path is derived from the uid and not from `$XDG_RUNTIME_DIR` on purpose: that
|
|
variable is set in an interactive ssh shell but not in a non-interactive
|
|
`ssh host cmd` one, and a lock the two launch routes disagree about would exclude
|
|
nothing. For the same reason a tower with a boot service needs **lingering**
|
|
(section 4).
|
|
|
|
Because the check lives in the binary, it covers every route in - `scripts/fgc`,
|
|
the service, or a bare `./build/fire_gimbal_control`. Nothing needs cleaning up
|
|
after a crash or a power cut: the kernel releases an `flock` when the holder
|
|
dies. Use `--no-lock` only to run a second mock-only instance during
|
|
development, and `--lock-file <path>` to point at a different lockfile.
|
|
|
|
## 4. Starting at boot
|
|
|
|
`scripts/install-tower` installs this (it asks first, and shows the unit it will
|
|
write). The unit is the template `scripts/fire-gimbal-control.service`, and it
|
|
runs the program **through `scripts/fgc`, inside tmux** - so a tower started at
|
|
boot still has its dashboard, and `fgc status` / `peek` / `stop` work from any
|
|
ssh session.
|
|
|
|
```bash
|
|
sudo systemctl start|stop|restart fire-gimbal-control
|
|
systemctl is-enabled fire-gimbal-control # starts at boot?
|
|
sudo systemctl disable fire-gimbal-control # not any more
|
|
```
|
|
|
|
By hand, fill in `@USER@`, `@REPO@` and `@ARGS@` (normally empty) and:
|
|
|
|
```bash
|
|
sudo cp fire-gimbal-control.service /etc/systemd/system/
|
|
sudo loginctl enable-linger ggs # NOT optional - see below
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable fire-gimbal-control
|
|
```
|
|
|
|
**Lingering is required.** Without it `/run/user/<uid>` does not exist until
|
|
someone logs in, so a program started at boot takes its single-instance lock in
|
|
`/tmp`; a later ssh session looks in `/run/user`, finds nothing, reports "not
|
|
running", and a second copy could start beside the first.
|
|
|
|
The service passes no startup flags. What the program does at boot is
|
|
`config.ini` `[Startup]` ([configuration.md](configuration.md#startup--homing-and-the-start-mode)):
|
|
|
|
**`home = true` homes the gimbal unattended at every boot.** That is what a tower
|
|
in service needs - the firmware keeps no position across a power cycle, so
|
|
without it the gimbal sits un-homed and captures nothing - but it is a
|
|
deliberate choice, and the installer asks for it (`startup_home`), as it does
|
|
for the start mode (`startup_mode`, `manual` or `auto`).
|
|
|
|
**Upgrading a tower from before `[Startup]`.** Its unit still passes `--init` /
|
|
`--start`, which the program now refuses. Run `scripts/install-tower` on the
|
|
tower **before its next reboot**: it moves them into `config.ini` (`--init` →
|
|
`home = true`, `--start` → `mode = auto`) and rewrites the unit without them.
|
|
`install-tower --verify` fails its `boot service args` check until that is done.
|
|
|
|
| Tower | Service | Starts with |
|
|
|---|---|---|
|
|
| 111 | enabled 2026-09-24 | `--init --start`: homes and captures at boot - `[verified]` by a reboot 2026-09-24. **To migrate**: becomes `[Startup] home = true`, `mode = auto` |
|
|
| 121 | enabled 2026-09-23 | no arguments and no `[Startup]`: starts idle, neither homes nor captures (the development tower) |
|
|
|
|
`[verified]` on tower 121, 2026-09-23: the service came up by itself after a
|
|
reboot with nobody logged in, with its lock under `/run/user/1000`; `fgc status`
|
|
and the dashboard worked from a later ssh session; credentials reached the
|
|
program intact (a password with `$`, a quote and a space); `systemctl stop` shut
|
|
it down cleanly; and `install-tower --verify` passed after the reboot, USB
|
|
preconditions included.
|
|
|
|
A tower started this way is in service when:
|
|
- images accumulate under `[Paths] output_dir` (default
|
|
`~ggs/.local/share/fire_gimbal_control/images/RGB/`);
|
|
- CamEvents reach the broker: `mosquitto_sub -h <broker> -t 'GGS/FWT/<tower>/#' -v`.
|
|
|
|
## 5. Where the log is
|
|
|
|
The program writes a persistent log alongside the self-test and diagnostics
|
|
reports:
|
|
|
|
```
|
|
~/.local/share/fire_gimbal_control/logs/fgc_<YYYYMMDD-HHMMSS>.<NNN>.log
|
|
```
|
|
|
|
One file per run, rotated by size within a run (`.000`, `.001`, …), oldest pruned
|
|
by count. Bounded at `[Logging] file_keep x file_max_mb`, 200 MB by default, at
|
|
roughly 5 MB/day in steady state.
|
|
|
|
This exists because a tower runs detached and, in TUI mode, every line otherwise
|
|
goes to a 500-line on-screen ring buffer that dies with the tmux session. The
|
|
file is **more verbose than the dashboard** - `info` on screen, `debug` on disk -
|
|
so it holds detail the pane never showed.
|
|
|
|
```bash
|
|
ls -t ~/.local/share/fire_gimbal_control/logs/fgc_*.log | head
|
|
tail -F "$(ls -t ~/.local/share/fire_gimbal_control/logs/fgc_*.log | head -1)"
|
|
```
|
|
|
|
Use `tail -F`, not `-f`: rotation replaces the file and `-f` would follow the old
|
|
inode into a file nothing writes to any more.
|
|
|
|
Each file opens with a header naming the build, tower, PID, config and hardware
|
|
paths and the full command line - repeated after every rotation, so an archive
|
|
found on its own still says where it came from.
|
|
|
|
What to grep when a tower will not come up:
|
|
|
|
```bash
|
|
L=$(ls -t ~/.local/share/fire_gimbal_control/logs/fgc_*.log | head -1)
|
|
grep -E "hardware:|NEED|firmware config|need config" "$L" # declaration + handshake
|
|
grep -E "\[WARN |\[ERROR" "$L" # everything it complained about
|
|
head -8 "$L" # which build, which config
|
|
```
|
|
|
|
> Wire traces (`[Logging] trace` / `--trace`) reach this file too. The serial
|
|
> trace alone takes it from ~5 MB/day to ~41 MB/day, enough to age out the
|
|
> startup record within days - turn it on to chase something, off afterwards.
|
|
|
|
## Troubleshooting
|
|
|
|
Start with `scripts/install-tower --verify` on the tower: it checks most of the
|
|
list below and says which.
|
|
|
|
| Symptom | Check |
|
|
|---------|-------|
|
|
| `No config.ini found` | run from `~/projects/fwt_2a/software`, or `--config` / `$FGC_CONFIG` |
|
|
| `Config error` naming a key | `build/fgc-config check --config config.ini --hardware /etc/fire_gimbal_control/hardware.ini` |
|
|
| `Failed to open serial port` | MCU plugged in, `ggs` in `dialout`, `[Serial] device` in `hardware.ini` - a `/dev/serial/by-id/` path, which **changes when the MCU is flashed** |
|
|
| An axis will not move, `ERR need config` | `hardware.ini` is missing a drivetrain value for it; the log names the key |
|
|
| Camera opened, but no images and `no frame yet (stream warming up)` for minutes | the stream is stuck; the program restarts it and then resets the camera by itself (known-issues #15), or type `camera reset` |
|
|
| `No camera found` / Vimba startup error | SDK installed, GenTL path set (`/etc/profile.d/*Vimba*GenTL*.sh` or `VIMBA_CTI_PATH`), `[Camera.rgb] id` in `hardware.ini` |
|
|
| `MQTT connect failed` (degraded mode) | broker reachable, `credentials.env` readable by `ggs`, firewall |
|
|
| Runs but no images | in auto mode (`[Startup] mode = auto`, or `mode auto`) and the gimbal homed; `output_dir` writable |
|
|
| `another instance is already running` (exit 2) | `scripts/fgc status`; the service and a hand-started run both up (section 3) |
|
|
| `fgc status` says not running, but the program is | the service started it before anyone logged in, without lingering (section 4) |
|
|
| Program dies when ssh disconnects | started with `run.sh` / `deploy.sh --foreground` instead of `scripts/fgc start` |
|