GGS
/
fwt_software
8
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 1 Wiki Activity

Rewrite the running and installing guides for how towers are set up now

This commit is contained in:
pgdalmeida 2026-09-24 14:59:04 +02:00
parent 08c749ca06
commit 91cc8af915
Signed by: pedro.almeida
GPG Key ID: D4A6C394DF13F1D7
4 changed files with 413 additions and 363 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@ -233,7 +233,7 @@ libjxl-dev`, plus the Vimba X SDK under `/opt/VimbaX` for `-DWITH_VIMBA=ON`.
|----------|----------| |----------|----------|
| [docs/architecture.md](docs/architecture.md) | Components, interfaces, threading, data flow, capture state machine | | [docs/architecture.md](docs/architecture.md) | Components, interfaces, threading, data flow, capture state machine |
| [docs/build-and-setup.md](docs/build-and-setup.md) | CMake, dependencies, options, Vimba X / Paho, host setup | | [docs/build-and-setup.md](docs/build-and-setup.md) | CMake, dependencies, options, Vimba X / Paho, host setup |
| [docs/deployment.md](docs/deployment.md) | Step-by-step deploy to the gimbal LattePanda (systemd) | | [docs/deployment.md](docs/deployment.md) | Running and updating a tower: `deploy.sh`, `scripts/fgc`, the boot service, the log |
| [docs/configuration.md](docs/configuration.md) | `config.ini` keys, CLI flags, console commands | | [docs/configuration.md](docs/configuration.md) | `config.ini` keys, CLI flags, console commands |
| [docs/mqtt-api.md](docs/mqtt-api.md) | MQTT topic catalog and payloads | | [docs/mqtt-api.md](docs/mqtt-api.md) | MQTT topic catalog and payloads |
| [docs/test-command.md](docs/test-command.md) | The `test` hardware self-test suite: subsystems, profiles, reports, baselines | | [docs/test-command.md](docs/test-command.md) | The `test` hardware self-test suite: subsystems, profiles, reports, baselines |

408
docs/deployment.md
View File

@ -1,204 +1,163 @@
# Deploying to the Gimbal LattePanda # Running and updating a tower
Step-by-step guide to deploy `fire_gimbal_control` on a tower's LattePanda (Linux x86-64). Like the How a tower that already works is operated and updated: pushing a new version
firmware workflow, this is a repeatable **build → install → configure → enable → verify** cycle; the from the dev box, running the program detached, starting it at boot, reading its
[Update / redeploy](#9-update--redeploy) section at the end is the short loop you run for every new version. log. A **first** install - a freshly imaged board, or a tower not yet migrated -
is [tower-installation.md](tower-installation.md), which `scripts/install-tower`
automates.
> **Which LattePanda.** The deployed fleet is the **3 Delta** — read from tower 111's DMI The fleet: tower **111** is a LattePanda 3 Delta (ATmega32U4 motor MCU, two
> `product_name`, *not* the Sigma, which this page claimed until 2026-09-18. Tower 121 is a axes), tower **121** a LattePanda IOTA (RP2040, yaw only). Nothing here differs
> **LattePanda IOTA**; see [lattepanda-iota.md](lattepanda-iota.md) and its setup kit under between them; what differs lives in each tower's `hardware.ini`.
> `hardware/lattepanda-iota/`. Nothing in this guide is board-specific — it is x86-64 Linux either way —
> so the distinction matters for the device paths in `hardware.ini`, not for these steps.
>
> Assumes a Debian/Ubuntu-based Linux on the LattePanda. For Arch/Manjaro swap `apt` for `pacman`
> (see [build-and-setup.md](build-and-setup.md)). Run as a normal user with `sudo` rights; the service runs
> under a dedicated `ggs` user.
## 0. Prerequisites (one-time per device) ## Where things are on a tower
Install the toolchain and system libraries: | Path | What | Written by |
|---|---|---|
| `~ggs/projects/fwt_2a/software/` | this repository, and the program's build in `build/` | `deploy.sh` (sync + build), or the installer |
| `~ggs/projects/fwt_2a/firmware/` | the firmware repository, built on the tower | `firmware/deploy.sh` |
| `.../software/config.ini` | what the tower DOES: scan, capture, logging, calibration | the installer (seeded once), `gimbal ... setup` (calibration), a person |
| `/etc/fire_gimbal_control/hardware.ini` | what the tower IS: board, axes, sensors, pins, drivetrain | the installer; a person when the hardware changes |
| `/etc/fire_gimbal_control/credentials.env` | `FGC_MQTT_USER=` / `FGC_MQTT_PW=`, mode 0640 root:ggs | the installer |
| `/etc/fire_gimbal_control/install.log` | one line per installer run | the installer |
| `~ggs/.local/share/fire_gimbal_control/logs/` | the program's log, test reports | the program |
| `.../software/config/camera_profiles/rgb/*.xml` | the camera settings, exported ON the tower from its camera | the installer, or `scripts/export_camera_profile.py` |
```bash `deploy.sh` never touches `config.ini`, `hardware.ini`, the camera profiles or
sudo apt update the images: they exist only on the tower, and `rsync --delete` once erased a
sudo apt install -y git cmake build-essential \ freshly exported profile before they were excluded.
libopencv-dev libjxl-dev libboost-all-dev
```
Install the **Allied Vision Vimba X SDK** (proprietary, needed for the cameras): ## 1. Updating the program from the dev box
1. Download Vimba X for Linux x86-64 from On the dev box, in `software/`, with a `.deploy.env` naming the tower:
<https://www.alliedvision.com/en/products/software/vimba-x-sdk/>.
2. Unpack to a fixed location, e.g. `/opt/VimbaX_2024-1`.
3. Run the SDK's transport-layer installer so cameras are discoverable, e.g.:
```bash
sudo /opt/VimbaX_2024-1/cti/Install_GenTL_Path.sh
```
4. Make its runtime libraries findable at run time:
```bash
echo /opt/VimbaX_2024-1/api/lib | sudo tee /etc/ld.so.conf.d/vimbax.conf
sudo ldconfig
```
Ensure an **MQTT broker** is reachable from the tower (the ground-station/ZKMS broker, or a local Mosquitto
for bring-up).
## 1. Get the code
```bash
sudo mkdir -p /opt/src && sudo chown "$USER" /opt/src
cd /opt/src
git clone <repo-url> fire_gimbal_control_src
cd fire_gimbal_control_src
# (or: git pull, for an update)
```
## 2. Build (full, with cameras + MQTT)
```bash
cmake -B build -DCMAKE_BUILD_TYPE=Release \
-DWITH_VIMBA=ON -DWITH_MQTT=ON \
-DVMB_HOME=/opt/VimbaX_2024-1
cmake --build build -j"$(nproc)"
```
- `WITH_MQTT=ON` fetches + builds Eclipse Paho from source on first configure (needs network).
- If a camera build isn't needed yet, use `-DWITH_VIMBA=OFF` for a mock binary to test the pipeline.
- Smoke-test the core before installing: `ctest --test-dir build` (after adding `-DBUILD_TESTING=ON`).
## 3. Install to a fixed location
```bash
sudo useradd -r -s /usr/sbin/nologin ggs 2>/dev/null || true
sudo mkdir -p /opt/fire_gimbal_control
sudo cp build/fire_gimbal_control /opt/fire_gimbal_control/
# demo-mode placeholder image (only needed if you ever run --demo)
sudo cp bin/x64/Release/test_smoke.jxl /opt/fire_gimbal_control/ 2>/dev/null || true
```
## 4. Configure
```bash
sudo cp config/config.example.ini /opt/fire_gimbal_control/config.ini
sudo nano /opt/fire_gimbal_control/config.ini
```
Set at least:
```ini ```ini
[General] REMOTE_HOST=ggs@10.11.12.121
tower_name = ExampleTower ; this tower's name (drives all MQTT topics) REMOTE_DIR=/home/ggs/projects/fwt_2a/software
image_interval = 3 CMAKE_ARGS="-DWITH_MQTT=ON -DWITH_VIMBA=ON"
RUN_ARGS="--start"
[Network]
zkms_server_ip = 10.11.12.13 ; the broker address
[Serial]
device = /dev/ttyACM0 ; the motor-controller MCU
baud = 115200
[Camera.rgb]
id = DEV_1AB22C0AADED ; or a GigE IP like 192.168.11.101
label = RGB
profile_dir = config/camera_profiles/rgb
profile = standard
``` ```
**Credentials** go in the environment, not the config file. Create a root-only env file for the service:
```bash ```bash
sudo install -m 600 /dev/null /etc/fire_gimbal_control/credentials.env 2>/dev/null || \ ./deploy.sh # sync the tree, build on the tower
{ sudo mkdir -p /etc/fire_gimbal_control && sudo install -m 600 /dev/null /etc/fire_gimbal_control/credentials.env; } ./deploy.sh --check-deps # only check the tower's build dependencies
printf 'FGC_MQTT_USER=fwt_gimbal\nFGC_MQTT_PW=CHANGE_ME\n' | sudo tee /etc/fire_gimbal_control/credentials.env >/dev/null ./deploy.sh --clean # wipe the tower's build dir first (fresh configure)
sudo chmod 600 /etc/fire_gimbal_control/credentials.env
``` ```
Give the `ggs` user access to its files and the image output dir: It syncs the tree (leaving `.git` behind, and writing the revision into
`.revision` beside it instead), then configures and builds **on the tower**. The
program already running keeps running the old binary until it is restarted:
```bash ```bash
sudo chown -R ggs:ggs /opt/fire_gimbal_control # on a tower started by its boot service
ssh ggs@<tower> sudo systemctl restart fire-gimbal-control
# on a tower started by hand
./deploy.sh --stop && ./deploy.sh --run # --run --home adds --init
``` ```
## 5. Hardware access `--run` starts the program detached through `scripts/fgc` (section 2) and first
refuses a `config.ini` that still holds the example's placeholders
(`ExampleTower`, `CHANGE_ME`, `DEV_0000000000`); `--force` overrides that. On a
tower whose service is running, `--run` fails on the single-instance lock
(section 3) - restart it through systemd instead.
**Serial (motor controller):** let the service user open the port. After an update, check it:
```bash ```bash
sudo usermod -aG dialout ggs ssh ggs@<tower> 'cd ~/projects/fwt_2a/software && scripts/install-tower --verify'
``` ```
For a stable device name across reboots, add a udev rule keyed on the MCU's USB IDs (find them with It changes nothing; it runs `test hardware` inside the running program (pausing
`udevadm info -a -n /dev/ttyACM0 | grep -m1 idVendor`) and point `[Serial] device` at the symlink: and then resuming capture) and checks the files, the firmware's pins and the
boot service.
`--push-config` also overwrites the tower's `config.ini` with the dev box's,
backing the tower's copy up first. It exists for the development tower; a
tower's `config.ini` holds its own calibration, so do not push one tower's to
another. `hardware.ini` is never pushed.
> **One-time: `git pull` can delete your `.deploy.env`.** It was tracked by
> mistake until `b0127db`, which stopped tracking it and added it to
> `.gitignore`. Pulling *across* that commit deletes the working copy, credentials
> and all, with no warning. Only checkouts that predate `b0127db` are exposed,
> and only once: copy it aside first (`cp .deploy.env ../deploy.env.bak`), pull,
> copy it back. To recover one already lost: `git show b0127db^:.deploy.env`.
### Updating the firmware
**Host first, firmware second**, always: a new firmware asks the host for its
drivetrain at boot, and an older host cannot answer. In `firmware/`, with its own
`.deploy.env` (`REMOTE_HOST`, `REMOTE_DIR`, `PIO_ENV` = `leonardo` or
`iota_rp2040`, `SERIAL_PORT` = the MCU's `/dev/serial/by-id/` path):
```bash ```bash
echo 'SUBSYSTEM=="tty", ATTRS{idVendor}=="XXXX", ATTRS{idProduct}=="YYYY", SYMLINK+="ttyGimbal"' | \ ./deploy.sh --compile-only # sync + build on the tower, flash nothing
sudo tee /etc/udev/rules.d/99-gimbal.rules ./deploy.sh # build + flash
sudo udevadm control --reload && sudo udevadm trigger ./deploy.sh --monitor # ... then watch it boot
``` ```
**GigE cameras (if used):** put the host NIC on the camera subnet (e.g. `192.168.11.0/24`), enable jumbo The build takes the pins from the tower's `hardware.ini` (`[Pins]`) and fails
frames, and confirm discovery with the Vimba X `VimbaXViewer`/`ListCameras` tool. without them. The flashing is `firmware/tools/flash.sh`, run on the tower: an
RP2040 is put in its bootloader by the firmware's own `BOOTSEL` command first; a
Leonardo uses its 1200-baud touch. **Stop the program first** - it holds the
serial port - and restart it after; the new firmware then asks, the host
answers, and `install-tower --verify` confirms the firmware holds what
`hardware.ini` declares.
## 6. First run by hand (verify before enabling the service) ## 2. Running detached (`scripts/fgc`)
The tower is operated over ssh, so the program has to survive the connection
closing. `scripts/fgc` runs it inside a tmux session, which owns the pty:
```bash ```bash
cd /opt/fire_gimbal_control cd ~/projects/fwt_2a/software
set -a; . /etc/fire_gimbal_control/credentials.env; set +a
./fire_gimbal_control --config ./config.ini --init --start --log-level debug
```
Watch for: `Loaded config`, `Serial controller started`, `Opened camera ...`, `MQTT connected`, then
`Saved .../RGB/<timestamp>.jxl` after the first capture. Type `exit` (or Ctrl-D) to stop. To validate without
cameras first: add `--mock-camera --mock-serial --no-mqtt`.
This run is attached to your terminal and dies with the ssh session — fine for a first check, not for
leaving it running. For that, use section 6b.
### 6b. Running detached (`scripts/fgc`)
The tower is operated over ssh, so the program has to survive the connection closing. `scripts/fgc` runs
it inside a tmux session, which owns the pty:
```bash
cd /opt/fire_gimbal_control
scripts/fgc start --start # starts detached, returns as soon as it is up scripts/fgc start --start # starts detached, returns as soon as it is up
scripts/fgc status # running? since when? which PID? scripts/fgc status # running? since when? which PID?
scripts/fgc attach # the dashboard — Ctrl-b d detaches, leaves it running scripts/fgc attach # the dashboard - Ctrl-b d detaches, leaves it running
scripts/fgc stop # SIGTERM -> the same clean teardown as `exit` scripts/fgc stop # SIGTERM -> the same clean teardown as `exit`
scripts/fgc peek # snapshot of the pane without attaching scripts/fgc peek # snapshot of the pane without attaching
``` ```
**`Ctrl-b d` detaches. The dashboard's own `q` key opens an exit prompt** offering `q` (quit the The boot service (section 4) runs the program the same way, so all of this
program) or `b` (background — detach and leave it running, the same as `Ctrl-b d`), with `Esc` to works on a tower started at boot too.
cancel. `b` is offered only inside a `scripts/fgc` session; outside one there is nothing to detach
into and it is shown dimmed.
`scripts/fgc stop` sends SIGTERM and waits up to 30s; the program handles it with the same shutdown path **`Ctrl-b d` detaches. The dashboard's own `q` key opens an exit prompt**
as the `exit` command (stop capture, close the camera, flush the image queue). Do **not** `kill -9` — offering `q` (quit the program) or `b` (background - detach and leave it
an unclean exit leaves the camera wedged for the next run. If it will not stop, attach and look at why. running, the same as `Ctrl-b d`), with `Esc` to cancel. `b` is offered only
inside a `scripts/fgc` session; outside one there is nothing to detach into and
it is shown dimmed.
`scripts/fgc start` refuses if a run is already up, and prints the previous run's last output if it died `scripts/fgc stop` sends SIGTERM and waits up to 120 s; the program handles it
rather than silently discarding it. with the same shutdown path as the `exit` command (stop capture, close the
camera, flush the image queue). Do **not** `kill -9` - an unclean exit leaves the
camera wedged for the next run. If it will not stop, attach and look at why.
Choosing **quit** at that prompt while attached ends the tmux session too, so you land straight back in `scripts/fgc start` refuses if a run is already up, and prints the previous
your shell — `exit` then closes ssh. A run that *fails* is the exception: its pane is kept so you can read run's last output if it died rather than silently discarding it.
why (`remain-on-exit failed`), and `fgc status` shows it until `fgc stop` or the next `fgc start` clears it.
A kept pane has **no process behind it** — keys and Ctrl-C do nothing, which reads as a hung ssh session. Choosing **quit** at that prompt while attached ends the tmux session too, so you
`Ctrl-b d` still detaches (tmux, not the pane, handles it); the session's status line spells that out along land straight back in your shell. A run that *fails* is the exception: its pane
the bottom of the screen. As a last resort `Enter ~ .` closes the ssh client itself from your end. is kept so you can read why (`remain-on-exit failed`), and `fgc status` shows it
until `fgc stop` or the next `fgc start` clears it. A kept pane has **no process
behind it** - keys and Ctrl-C do nothing, which reads as a hung ssh session.
`Ctrl-b d` still detaches; as a last resort `Enter ~ .` closes the ssh client.
> **Why `_run` waits on the app instead of `exec`ing it.** When the app is itself the pane's process, tmux MQTT credentials are read from `/etc/fire_gimbal_control/credentials.env` by
> never records an exit status for it — `#{pane_dead_status}` comes back empty and the process lingers as a `fgc` itself, inside the pane, as are the Vimba X paths: a tmux server that
> zombie — so `remain-on-exit failed` cannot tell success from failure and keeps *every* finished pane, already exists hands a new session its own environment, not the caller's.
> including a clean `q`. That is what used to strand an operator on a frozen dead pane. Interposing a shell
> that reaps the child and exits with its status gives tmux a real status. Measured on the device: 3/3 panes
> kept when exec'd, 4/4 sessions closed cleanly when parented. The wrapper forwards SIGTERM/SIGINT/SIGHUP,
> so `fgc stop` still reaches the app and runs the full teardown.
### 6c. The single-instance lock > **Why `_run` waits on the app instead of `exec`ing it.** When the app is itself
> the pane's process, tmux never records an exit status for it, so
> `remain-on-exit failed` cannot tell success from failure and keeps *every*
> finished pane. Interposing a shell that reaps the child and exits with its
> status gives tmux a real one. Measured on the device: 3/3 panes kept when
> exec'd, 4/4 sessions closed cleanly when parented. The wrapper forwards
> SIGTERM/SIGINT/SIGHUP, so `fgc stop` still reaches the app.
Two instances would fight over the motor's serial port and the camera. The binary ## 3. The single-instance lock
prevents that itself: at startup it takes an exclusive `flock` on
Two instances would fight over the motor's serial port and the camera. The
binary prevents that itself: at startup it takes an exclusive `flock` on
- `/run/user/<uid>/fire_gimbal_control.lock` when that directory exists, else - `/run/user/<uid>/fire_gimbal_control.lock` when that directory exists, else
- `/tmp/fire_gimbal_control-<uid>.lock` - `/tmp/fire_gimbal_control-<uid>.lock`
@ -209,24 +168,31 @@ and, if it is already held, exits with status **2** and:
Cannot start: another instance is already running (PID 1234); attach to it with: scripts/fgc attach Cannot start: another instance is already running (PID 1234); attach to it with: scripts/fgc attach
``` ```
The path is derived from the uid and not from `$XDG_RUNTIME_DIR` on purpose: that variable is set in an The path is derived from the uid and not from `$XDG_RUNTIME_DIR` on purpose: that
interactive ssh shell but not in a non-interactive `ssh host cmd` one, and a lock the two launch routes variable is set in an interactive ssh shell but not in a non-interactive
disagree about would exclude nothing. `ssh host cmd` one, and a lock the two launch routes disagree about would exclude
nothing. For the same reason a tower with a boot service needs **lingering**
(section 4).
Because the check lives in the binary, it covers every route in — `scripts/fgc`, systemd, or a bare Because the check lives in the binary, it covers every route in - `scripts/fgc`,
`./fire_gimbal_control`. Nothing needs cleaning up after a crash or a power cut: the kernel releases an the service, or a bare `./build/fire_gimbal_control`. Nothing needs cleaning up
`flock` when the holder dies, so a leftover lockfile with a dead PID in it never blocks a launch. Use after a crash or a power cut: the kernel releases an `flock` when the holder
`--no-lock` only to run a second mock-only instance during development, and `--lock-file <path>` to dies. Use `--no-lock` only to run a second mock-only instance during
point at a different lockfile. development, and `--lock-file <path>` to point at a different lockfile.
## 7. Install the systemd service ## 4. Starting at boot
`scripts/install-tower` does this (it asks first, and shows the unit it will `scripts/install-tower` installs this (it asks first, and shows the unit it will
write). The unit is the template `scripts/fire-gimbal-control.service`, and it write). The unit is the template `scripts/fire-gimbal-control.service`, and it
runs the program **through `scripts/fgc`, inside tmux** - so a tower started at runs the program **through `scripts/fgc`, inside tmux** - so a tower started at
boot still has its dashboard (`scripts/fgc attach`), and `fgc status` / `peek` / boot still has its dashboard, and `fgc status` / `peek` / `stop` work from any
`stop` work from any ssh session. It is not an alternative to the tmux workflow ssh session.
any more; it is how that workflow starts at boot.
```bash
sudo systemctl start|stop|restart fire-gimbal-control
systemctl is-enabled fire-gimbal-control # starts at boot?
sudo systemctl disable fire-gimbal-control # not any more
```
By hand, fill in `@USER@`, `@REPO@` and `@ARGS@` (normally `--init --start`) and: By hand, fill in `@USER@`, `@REPO@` and `@ARGS@` (normally `--init --start`) and:
@ -244,73 +210,27 @@ running", and a second copy could start beside the first.
**`--init` homes the gimbal unattended at every boot.** That is what a tower in **`--init` homes the gimbal unattended at every boot.** That is what a tower in
service needs - the firmware keeps no position across a power cycle, so without service needs - the firmware keeps no position across a power cycle, so without
it the gimbal sits un-homed and captures nothing - but it is a deliberate choice, it the gimbal sits un-homed and captures nothing - but it is a deliberate
and the installer asks for the arguments. choice, and the installer asks for the arguments.
MQTT credentials go in `/etc/fire_gimbal_control/credentials.env` | Tower | Service | Starts with |
(`FGC_MQTT_USER=` / `FGC_MQTT_PW=`, mode 0640 root:ggs). `fgc` reads it inside |---|---|---|
the tmux pane rather than taking it from the unit's environment: a tmux server | 111 | enabled 2026-09-24 | `--init --start`: homes and captures at boot. Not yet exercised by a reboot |
that already exists hands a new session its own environment, not the caller's. | 121 | enabled 2026-09-23 | no arguments: starts idle, neither homes nor captures (the development tower) |
## 8. Enable and verify
```bash
sudo systemctl start fire-gimbal-control
scripts/fgc status # running, lockfile under /run/user/<uid>
scripts/install-tower --verify # every check, including test hardware
```
`[verified]` on tower 121, 2026-09-23: the service came up by itself after a `[verified]` on tower 121, 2026-09-23: the service came up by itself after a
reboot with nobody logged in, with its lock under `/run/user/1000`; `fgc status` reboot with nobody logged in, with its lock under `/run/user/1000`; `fgc status`
and the dashboard worked from a later ssh session; credentials reached the and the dashboard worked from a later ssh session; credentials reached the
program intact (a password with `$`, a quote and a space); `systemctl stop` shut program intact (a password with `$`, a quote and a space); `systemctl stop` shut
it down cleanly; and `install-tower --verify` passed after the reboot, USB it down cleanly; and `install-tower --verify` passed after the reboot, USB
preconditions included. 121 runs it with no arguments, so it does not home at preconditions included.
boot - see the note on `--init` above.
Confirm end to end: A tower started this way is in service when:
- Images accumulating under the configured `[Paths] output_dir` (default - images accumulate under `[Paths] output_dir` (default
`~ggs/.local/share/fire_gimbal_control/images/RGB/`). `~ggs/.local/share/fire_gimbal_control/images/RGB/`);
- CamEvents on the broker: - CamEvents reach the broker: `mosquitto_sub -h <broker> -t 'GGS/FWT/<tower>/#' -v`.
`mosquitto_sub -h <broker> -t 'GGS/FWT/<tower>/#' -v`.
## 9. Update / redeploy ## 5. Where the log is
The short loop for each new version (mirrors the firmware flash cycle):
```bash
cd /opt/src/fire_gimbal_control_src
git pull
cmake --build build -j"$(nproc)" # reuses the configured build dir
sudo systemctl stop fire-gimbal-control
sudo cp build/fire_gimbal_control /opt/fire_gimbal_control/
sudo systemctl start fire-gimbal-control
journalctl -u fire-gimbal-control -n 50 -f # confirm healthy
```
If `CMakeLists.txt` or options changed, re-run the `cmake -B build ...` configure step from
[section 2](#2-build-full-with-cameras--mqtt) first. To roll back, keep the previous binary
(`fire_gimbal_control.prev`) and swap it back.
> **One-time: `git pull` can delete your `.deploy.env`.** It was tracked by
> mistake until `b0127db`, which stopped tracking it and added it to
> `.gitignore`. Pulling *across* that commit therefore does what git does to any
> file a commit removes — deletes the working copy, credentials and all. It is a
> checkout, not a merge conflict, so there is no warning and nothing to resolve.
>
> Only checkouts that predate `b0127db` are exposed, and only once. Copy it aside
> first:
>
> ```bash
> cp .deploy.env ../deploy.env.bak 2>/dev/null # then pull, then copy back
> ```
>
> After that pull the file is ignored and stays put. `deploy.sh` already excludes
> it from the rsync, so deploying never overwrites the tower's own copy — this is
> purely about the machine you deploy *from*. To recover one already lost:
> `git show b0127db^:.deploy.env`.
## 10. Where the log is
The program writes a persistent log alongside the self-test and diagnostics The program writes a persistent log alongside the self-test and diagnostics
reports: reports:
@ -325,7 +245,7 @@ roughly 5 MB/day in steady state.
This exists because a tower runs detached and, in TUI mode, every line otherwise This exists because a tower runs detached and, in TUI mode, every line otherwise
goes to a 500-line on-screen ring buffer that dies with the tmux session. The goes to a 500-line on-screen ring buffer that dies with the tmux session. The
file is **more verbose than the dashboard** — `info` on screen, `debug` on disk — file is **more verbose than the dashboard** - `info` on screen, `debug` on disk -
so it holds detail the pane never showed. so it holds detail the pane never showed.
```bash ```bash
@ -337,9 +257,8 @@ Use `tail -F`, not `-f`: rotation replaces the file and `-f` would follow the ol
inode into a file nothing writes to any more. inode into a file nothing writes to any more.
Each file opens with a header naming the build, tower, PID, config and hardware Each file opens with a header naming the build, tower, PID, config and hardware
paths and the full command line — repeated after every rotation, so an archive paths and the full command line - repeated after every rotation, so an archive
found on its own still says where it came from. Lines carry a `YYYY-MM-DD` prefix found on its own still says where it came from.
the console ones do not.
What to grep when a tower will not come up: What to grep when a tower will not come up:
@ -352,17 +271,22 @@ head -8 "$L" # which build, which
> Wire traces (`[Logging] trace` / `--trace`) reach this file too. The serial > Wire traces (`[Logging] trace` / `--trace`) reach this file too. The serial
> trace alone takes it from ~5 MB/day to ~41 MB/day, enough to age out the > trace alone takes it from ~5 MB/day to ~41 MB/day, enough to age out the
> startup record within days — turn it on to chase something, off afterwards. > startup record within days - turn it on to chase something, off afterwards.
## Troubleshooting ## Troubleshooting
Start with `scripts/install-tower --verify` on the tower: it checks most of the
list below and says which.
| Symptom | Check | | Symptom | Check |
|---------|-------| |---------|-------|
| `No config.ini found` | `FGC_CONFIG` / `--config` path; service `Environment=` line | | `No config.ini found` | run from `~/projects/fwt_2a/software`, or `--config` / `$FGC_CONFIG` |
| `Failed to open serial port` | MCU plugged in, `dialout` group, `[Serial] device` path / udev symlink | | `Config error` naming a key | `build/fgc-config check --config config.ini --hardware /etc/fire_gimbal_control/hardware.ini` |
| `No camera found` / Vimba startup error | SDK installed, GenTL path set, `ldconfig`, NIC subnet, camera IDs in config | | `Failed to open serial port` | MCU plugged in, `ggs` in `dialout`, `[Serial] device` in `hardware.ini` - a `/dev/serial/by-id/` path, which **changes when the MCU is flashed** |
| `MQTT connect failed` (degraded mode) | broker reachable, credentials env, firewall | | An axis will not move, `ERR need config` | `hardware.ini` is missing a drivetrain value for it; the log names the key |
| Runs but no images | capture started (`--start` or `start` command), `output_dir` writable by `ggs` | | `No camera found` / Vimba startup error | SDK installed, GenTL path set (`/etc/profile.d/*Vimba*GenTL*.sh` or `VIMBA_CTI_PATH`), `[Camera.rgb] id` in `hardware.ini` |
| `another instance is already running` (exit 2) | `scripts/fgc status`; a systemd unit and a tmux session both running (section 6c) | | `MQTT connect failed` (degraded mode) | broker reachable, `credentials.env` readable by `ggs`, firewall |
| Program dies when ssh disconnects | started with `run.sh`/`--foreground` instead of `scripts/fgc start` (section 6b) | | Runs but no images | capture started (`--start`, or the `start` command) and the gimbal homed; `output_dir` writable |
| `No camera found` only when started via `scripts/fgc` | GenTL path not resolved in the pane — check `scripts/fgc peek`, and `/etc/profile.d/*Vimba*GenTL*.sh` or `VIMBA_CTI_PATH` | | `another instance is already running` (exit 2) | `scripts/fgc status`; the service and a hand-started run both up (section 3) |
| `fgc status` says not running, but the program is | the service started it before anyone logged in, without lingering (section 4) |
| Program dies when ssh disconnects | started with `run.sh` / `deploy.sh --foreground` instead of `scripts/fgc start` |

362
docs/tower-installation.md
View File

@ -1,56 +1,88 @@
# Installing a tower from scratch # Installing a tower
How a new fire-watch tower host goes from a freshly imaged machine to a running, How a fire-watch tower host goes from a freshly imaged machine - or from an
calibrated `fire_gimbal_control`. Written while doing exactly that on the IOTA older, unmigrated setup - to a running, calibrated `fire_gimbal_control`.
tower at `10.11.12.121` (2026-09-15) and kept honest about which steps have [deployment.md](deployment.md) is what comes after: updating and operating a
actually been exercised and which are still on paper. tower that already works.
> [deployment.md](deployment.md) is the *update* loop for a tower that already **`scripts/install-tower` does the whole of this.** The short way is the next
> works. This is the *first* install, including the parts that only happen once: section. Everything after it, sections 1 to 11, is the reference for what each
> the board's preconditions, the Vimba X SDK, `hardware.ini`, and the motor step does and why, written while doing each one by hand, and still the way to
> firmware. do one step on its own.
> **`scripts/install-tower` now does the steps below**, on the tower, in this **Status legend** - each step says where it stands:
> order: it surveys the machine, asks for what it cannot work out (every
> question can also be answered with `--answer KEY=VALUE`), shows the plan, and
> applies it after one yes. `--dry-run` shows the plan and changes nothing;
> `--verify` runs the checks only. It never invents a drivetrain value: answer
> `unknown` and it writes a `; TODO`, and refuses to flash firmware an axis of
> which would then sit in `NEEDCFG`. Homing is offered only at the end, and only
> to someone who says they are at the rig.
>
> `[verified]` as a re-run on the already-configured tower 121 (2026-09-23): it
> changed only what was out of date and passed `test hardware`. `[untested]` on a
> freshly imaged board - the fresh path has been rehearsed with `--dry-run` only.
> What follows stays the reference for what each step does and why.
**Status legend** — each step says where it stands:
`[verified]` run end to end on a real tower and the result checked · `[verified]` run end to end on a real tower and the result checked ·
`[partial]` run, with something still open · `[untested]` written from the `[partial]` run, with something still open · `[untested]` written from the
design, not yet exercised. design, not yet exercised.
--- ---
## What you need before starting ## The short way: `scripts/install-tower`
### Before starting
| | | | | |
|---|---| |---|---|
| The tower host | Powered, on the network, reachable by IP, with an OS installed and an `ssh` server running | | The tower host | Powered, on the network, an OS installed, `ssh` running, reachable by key (section 1) |
| An account on it | `ggs`, in the `sudo` and `dialout` groups | | An account | `ggs`, in the `sudo` and `dialout` groups |
| A dev box | This repository checked out, plus a copy of the Vimba X SDK (login-walled, so it cannot be fetched by a script) | | Both repositories on it | `~/projects/fwt_2a/software` and `~/projects/fwt_2a/firmware` - cloned (`git.ggs-solutions.eu:GGS/fwt_software.git` and `fwt_firmware.git`, which needs a key on the tower), or synced by each repo's `deploy.sh` |
| The hardware | Camera plugged in, motor MCU plugged in, and a note of **what was assembled**: which axes, which IMU, which ambient sensor | | The Vimba X SDK, staged on it | Login-walled, so it cannot be fetched by a script: copy it on (USB stick, `scp`) and give its directory when asked. `none` for a tower with no camera yet |
| The facts of the build | Which axes, which IMU, which ambient sensor, how the driver boards are wired, and - per axis - the drivetrain: sense resistor, currents, travel, `steps_per_rev`, `sw_mode`, `enc_deviation`, encoder or not. Anything not yet measured can be answered `unknown` |
Two facts decide most of what follows, so establish them first: The installer takes the board from DMI and picks its kit:
```bash
ssh ggs@<tower-ip> 'cat /sys/class/dmi/id/product_name; . /etc/os-release && echo "$PRETTY_NAME"'
```
| Board | MCU | Setup kit | PlatformIO env | | Board | MCU | Setup kit | PlatformIO env |
|---|---|---|---| |---|---|---|---|
| LattePanda 3 Delta / Sigma | ATmega32U4 (Leonardo), 5 V | *not written - the Delta fleet predates the kits* | `leonardo` | | LattePanda 3 Delta | ATmega32U4 (Leonardo), 5 V | [`hardware/lattepanda-3-delta/`](../hardware/lattepanda-3-delta/) | `leonardo` |
| LattePanda IOTA | RP2040, 3.3 V | [`hardware/lattepanda-iota/`](../hardware/lattepanda-iota/) | `iota_rp2040` | | LattePanda IOTA | RP2040, 3.3 V | [`hardware/lattepanda-iota/`](../hardware/lattepanda-iota/) | `iota_rp2040` |
### Running it
On the tower, as `ggs`:
```bash
cd ~/projects/fwt_2a/software
scripts/install-tower --dry-run # survey + questions + the plan; changes nothing
scripts/install-tower # the same, then does it after one yes
scripts/install-tower --verify # the checks only; changes nothing
```
It surveys the machine, asks only what it cannot work out, prints a plan with
each step marked `skip`, `create`, `change` or `blocked`, and applies it after
one yes. In order:
1. packages, then the USB preconditions (section 3), then the Vimba X SDK (4)
2. builds the host
3. writes `hardware.ini` (5) - or, when one exists, only fills its gaps
4. seeds `config.ini` (9) and `credentials.env`, or edits only what is out of date
5. PlatformIO, then the camera profiles that are missing (8)
6. builds and flashes the firmware (6) - **blocked** while any drivetrain value
is still a `; TODO`, because that axis would refuse to move
7. `test hardware`
8. the boot service ([deployment.md](deployment.md) section 4), if wanted
9. asks whether someone is at the rig; only on a yes does it home the gimbal and
take a short capture - through the service, when the service is what homes it
Every question has a key, so any can be answered up front - `--answer
tower_name=north`, `--answer yaw.r_sense_ohm=0.075` - or from a file of
`key=value` lines with `--answers FILE`. `--non-interactive` stops instead of
prompting. **It never invents a drivetrain value**: a wrong sense resistor cooks a
driver and a wrong homing current shears an endstop, and both have happened on
this project. Answer `unknown` and it writes a commented `; TODO` naming what to
measure.
Re-running it is safe, and is how a configured tower is repaired: an existing
`hardware.ini` or `config.ini` is never regenerated, only filled in; an existing
camera profile is never overwritten; a firmware already running the declared
revision and pins is not reflashed; and if it fails after stopping the program,
it puts the program back the way it was.
| Where | Result |
|---|---|
| Tower 121 (IOTA), 2026-09-23 | `[verified]` re-run on a configured tower: changed only what was out of date, `--verify` green, boot service installed and proven across a reboot |
| Tower 111 (Delta), 2026-09-23/24 | `[verified]` migration of a tower that predated all of this: host first, firmware once the drivetrain was known, `test hardware` green on every leaf (section 10b) |
| A freshly imaged board | `[untested]` - rehearsed with `--dry-run` and typed answers only. The first flash of a factory MicroPython RP2040 is automated but has not met a factory board |
--- ---
## 1. SSH by key `[verified]` ## 1. SSH by key `[verified]`
@ -158,9 +190,11 @@ first reboot, since surviving one is the entire point.
> ssh -t ggs@<tower-ip> 'cd ~/projects/fwt_2a/software/hardware/lattepanda-iota && sudo ./install.sh --deps --preconditions' > ssh -t ggs@<tower-ip> 'cd ~/projects/fwt_2a/software/hardware/lattepanda-iota && sudo ./install.sh --deps --preconditions'
> ``` > ```
> **The Delta fleet does not have these applied.** Checked on `10.11.12.111` > **The Delta kit carries the same two files** (`hardware/lattepanda-3-delta/`,
> 2026-09-15: `usbfs_memory_mb` is 16 and the kernel cmdline is stock. The > as symlinks to these). Tower 111 had neither - `usbfs_memory_mb` 16, camera
> preconditions were documented in known-issues but never persisted anywhere. > LPM `u1_u2`, checked 2026-09-15 and again 2026-09-23 - and got both on
> 2026-09-23, reading 1000 and `0` afterwards. Not yet confirmed across a reboot
> on a Delta.
## 4. Vimba X SDK `[verified]` ## 4. Vimba X SDK `[verified]`
@ -289,7 +323,8 @@ is what catches the second case, so it has to be set from the real travel.
> `hardware.ini` is merged *over* `config.ini` and wins, so a shared key would > `hardware.ini` is merged *over* `config.ini` and wins, so a shared key would
> silently discard whatever `gimbal heading setup` had saved. What it does instead > silently discard whatever `gimbal heading setup` had saved. What it does instead
> is work the requirement out from `steps_per_rev` and print it as a `NOTE` in the > is work the requirement out from `steps_per_rev` and print it as a `NOTE` in the
> block, because the default is wrong for any geared-up rig: > block, because the default is wrong for any geared-up rig - and
> `scripts/install-tower` then sets it in `config.ini` when it is too small:
> >
> ``` > ```
> ; NOTE: half this travel is 605389 counts, so config.ini needs > ; NOTE: half this travel is 605389 counts, so config.ini needs
@ -315,9 +350,23 @@ yaw motor current: run 756 mA (GLOBALSCALER 80, IRUN 31), homing 189 mA (IRUN 7)
-- R_sense 0.075 ohm from hardware.ini [Motor.yaw] -- R_sense 0.075 ohm from hardware.ini [Motor.yaw]
``` ```
Omit the block entirely on a tower whose endstops hold against full torque: no There is no leaving the block out any more. The firmware holds no drivetrain
`CURRENT` is sent, the firmware keeps its compile-time defaults, and nothing values of its own: at boot every axis stops its driver, asks the host (`NEED`),
changes. and refuses to move until every value it asked for has arrived. An axis whose
block is missing or incomplete is an axis that will not turn - by design, since a
guessed value is how drivers get cooked and endstops sheared. (For a bench board
with no host, the firmware's `USE DEFAULTS` command loads deliberately gentle
limp-home values instead; `test hardware motor` fails on a tower running them.)
The kit writes eight keys per axis - the four above plus `steps_per_rev`,
`sw_mode`, `enc_deviation` and `has_encoder` - each as a value or a `; TODO`;
`--motor AXIS.KEY=VALUE` gives any of them for any axis, where the older
`--r-sense`-style flags cover the first axis only. The three speeds
(`scan_speed_deg_s`, `homing_speed_deg_s`, `center_speed_deg_s`, all in degrees
per second) are written as gentle starting values: **on a tower that is being
migrated, set the homing speed to what its endstop detection is proven at**, as
was done for tower 111 (section 10b), or a stall-detected endstop may behave
differently.
> **If an axis declares a reduced homing current and the firmware has not adopted > **If an axis declares a reduced homing current and the firmware has not adopted
> it, the host refuses to home** rather than driving into a stop at full torque. > it, the host refuses to home** rather than driving into a stop at full torque.
@ -330,7 +379,27 @@ changes.
> >
> **The by-id path changes when you flash the MCU**, because the name comes from > **The by-id path changes when you flash the MCU**, because the name comes from
> the running image: `usb-MicroPython_Board_in_FS_mode_*` before, a Pico/Arduino > the running image: `usb-MicroPython_Board_in_FS_mode_*` before, a Pico/Arduino
> name after. Re-run `--probe` after step 6 and fix the file if it moved. > name after. `scripts/install-tower` re-probes after flashing and updates
> `[Serial] device` itself; by hand, re-run `--probe` after step 6 and fix the
> file if it moved.
### The wiring block: `[Pins]`
The firmware is **built** from this section - which MCU pin reaches which driver
board - by `firmware/tools/gen_wiring.py`, reading the tower's own
`/etc/fire_gimbal_control/hardware.ini` when the firmware is built on the tower.
There is no copy in the firmware and no fallback: without `[Pins]` the firmware
build fails, naming what to add. Each kit's reference loom, with the table saying
which wire goes where, is its `pins.ini`; `install.sh --loom standard` copies it
in, and `--pin KEY=N` gives pins one at a time for a tower wired differently.
The generator refuses what the board cannot do rather than building it: on the
RP2040, SPI pins outside SPI0's reach (SCK `{2,6,18,22}`, MOSI `{3,7,19,23}`,
MISO `{0,4,16,20}`); on the Leonardo, any `spi_*` key at all (its SPI is fixed on
the ICSP header) and the ICSP pins as chip-selects; duplicates; and a
`hardware.ini` written for the other board. The running firmware reports the
pins it was built with, and `test hardware motor` compares them with this
section - so a flash built from an older copy shows up there.
## 6. Motor firmware `[verified]` ## 6. Motor firmware `[verified]`
@ -374,17 +443,20 @@ SERIAL_PORT=/dev/serial/by-id/usb-...
./deploy.sh --monitor # watch it boot ./deploy.sh --monitor # watch it boot
``` ```
`deploy.sh` syncs the tree and then runs `firmware/tools/flash.sh` on the tower,
which is also what `scripts/install-tower` runs, locally: one implementation of
the board-specific bootloader step for both. **Stop the program first**; it holds
the serial port.
**Confirm the pin map before flashing a board that is wired to motors.** A **Confirm the pin map before flashing a board that is wired to motors.** A
chip-select on the wrong pin writes into a driver that was never meant to hear chip-select on the wrong pin writes into a driver that was never meant to hear
it. The pins are the `[Pins]` section of this tower's `hardware.ini`: the build it. The build takes the pins from this tower's `hardware.ini` and fails without
reads them from `/etc/fire_gimbal_control/hardware.ini` (`firmware/tools/gen_wiring.py`) them - see "The wiring block" in section 5. The IOTA's reference loom is the one
and **fails without them**, so add the section before the first build. The as built on tower 121, which is **not** the map proposed in the porting plan.
reference loom, with the table saying which wire goes where, is
[`hardware/lattepanda-iota/pins.ini`](../hardware/lattepanda-iota/pins.ini) - `[verified]` on both boards: the RP2040 on tower 121 (2026-09-23), the Leonardo on
the loom as built on tower 121, which is **not** the map proposed in the porting tower 111 (2026-09-24) - the Leonardo through its own 1200-baud touch, which
plan. Copy it in if the tower is wired the same way. The generator refuses pins `flash.sh` leaves alone.
SPI0 cannot reach, and `test hardware motor` compares the declared pins with the
ones the running firmware reports it was built with.
### Getting a factory board to accept its first flash ### Getting a factory board to accept its first flash
@ -469,8 +541,9 @@ loop and unsupervised energised drivers would hold torque.
| before | `usb-MicroPython_Board_in_FS_mode_<serial>-if00` | `2e8a:0005` | | before | `usb-MicroPython_Board_in_FS_mode_<serial>-if00` | `2e8a:0005` |
| after | `usb-Raspberry_Pi_Pico_<SERIAL>-if00` | `2e8a:000a` | | after | `usb-Raspberry_Pi_Pico_<SERIAL>-if00` | `2e8a:000a` |
So generate `hardware.ini` (step 5) **after** flashing, or re-run So generate `hardware.ini` (step 5) **after** flashing, or update `[Serial]
`install.sh --hardware-ini --force` to pick up the new path. device` once the name has moved - `scripts/install-tower` does that itself,
re-probing after every flash.
### Verify the flash ### Verify the flash
@ -513,10 +586,14 @@ RUN_ARGS="--start"
./deploy.sh # rsync + remote cmake + build ./deploy.sh # rsync + remote cmake + build
``` ```
The first deploy seeds `config.ini` from `config/config.example.ini` **only if The first deploy copies `config/config.example.ini` to `config.ini` **only if
the device has none** - see the warning in step 2 about a machine that arrived the device has none** - see the warning in step 2 about a machine that arrived
with somebody else's config. with somebody else's config. That copy still carries the example's placeholders
and the keys `hardware.ini` owns; `scripts/install-tower` seeds it instead with
`build/fgc-config seed`, which fills in the answers, comments out every key
`hardware.ini` sets, and refuses to write a file the program would not load.
`[verified]` on tower 111: builds on GCC 11 / Ubuntu 22.04.
`[verified]` on tower 121: builds clean on GCC 15 / Ubuntu 26.04 with a single `[verified]` on tower 121: builds clean on GCC 15 / Ubuntu 26.04 with a single
`-Wformat-truncation` warning in `utcStampNow()`, which is a false positive (the `-Wformat-truncation` warning in `utcStampNow()`, which is a false positive (the
compiler cannot prove `tm_year` is four digits; the real output is 17 bytes into compiler cannot prove `tm_year` is four digits; the real output is 17 bytes into
@ -610,8 +687,12 @@ zkms_server_ip = <broker>
Credentials belong in `/etc/fire_gimbal_control/credentials.env`, not here. Credentials belong in `/etc/fire_gimbal_control/credentials.env`, not here.
Anything `hardware.ini` now owns - `[Serial] device`, `[IMU]`, `[Env]`, the Anything `hardware.ini` now owns - `[Serial] device`, `[IMU]`, `[Env]`, the
camera ids - should be **deleted** from `config.ini` rather than left to rot: camera ids - should not be in `config.ini` as well: the program warns at startup
the program warns at startup about every key set in both. about every key set in both. `scripts/install-tower` comments them out (value and
all), and `build/fgc-config set config.ini --config config.ini --hardware
/etc/fire_gimbal_control/hardware.ini --drop Section.key=note` does one by hand.
Every edit through `fgc-config` is loaded the way the program loads it before it
is written, and keeps the previous value as a `;prev` comment.
> **`[Camera.<name>] id` is required even when the camera is disabled.** Config > **`[Camera.<name>] id` is required even when the camera is disabled.** Config
> validation runs before any of the feature flags are consulted, so a tower with > validation runs before any of the feature flags are consulted, so a tower with
@ -619,20 +700,12 @@ the program warns at startup about every key set in both.
> `DEV_0000000000`, is what `deploy.sh`'s placeholder guard refuses. On a tower > `DEV_0000000000`, is what `deploy.sh`'s placeholder guard refuses. On a tower
> that has its camera, the id lives in `hardware.ini` and this never comes up. > that has its camera, the id lives in `hardware.ini` and this never comes up.
`[Features] enable_imu` and `enable_env` matter more than they look. Declaring `[Features] enable_imu` and `enable_env` are for towers **without** a
`imu = none` in `hardware.ini` does **not** yet switch them off - deriving them `hardware.ini`. Where `hardware.ini` declares `imu` / `env`, the declaration
is section 12 of the config plan and is not implemented. On tower 121 the first decides, and a flag that contradicts it is warned about and ignored. Before that
clean-looking run did this: was so, tower 121's first clean-looking run "started" an IMU it does not have -
`/dev/ttyS0` accepts a serial open whether or not anything is attached - which is
``` why a declared `none` has to be able to switch a sensor off by itself.
[INFO ] MTi IMU started on /dev/ttyS0 @ 115200
[ERROR] I2C: failed to open /dev/i2c-2
[ERROR] SHT41: failed to open I2C bus; sensor disabled
```
It "started" an IMU on a tower that has none - `/dev/ttyS0` exists and accepts a
serial open whether or not anything is attached to it. Set both flags to `false`
by hand until reconciliation lands.
## 10. First run: bring the subsystems up one at a time `[partial]` ## 10. First run: bring the subsystems up one at a time `[partial]`
@ -690,8 +763,16 @@ DUMP BEGIN build=aab1925-dirty uptime=7925212 mcusr=0x3 free_ram=252712
DUMP Y state=1 ... speed=3000 center=500000 has_encoder=1 DUMP Y state=1 ... speed=3000 center=500000 has_encoder=1
``` ```
Check `build=` matches the firmware you flashed and `speed=` matches the Check `build=` matches the firmware you flashed, and that the handshake
drivetrain values you intended. Then, and only then, move the thing: completed - the log says `firmware config: answered N keys for axis Y`, and the
axis leaves state 5 (`NEEDCFG`). The scripted form of all of this, which also
compares every value the firmware holds with `hardware.ini`, moves nothing:
```bash
./build/fire_gimbal_control --no-mqtt --test hardware # exit 0 PASS, 4 FAIL, 5 could not start
```
Then, and only then, move the thing:
``` ```
gimbal home # the endstop-finding sequence - see the warning below gimbal home # the endstop-finding sequence - see the warning below
@ -703,13 +784,14 @@ gimbal move 10,0 # a small deliberate move
| Message | Why | | Message | Why |
|---|---| |---|---|
| `Motor controller: ERR axis not present` | The host configures both axes at startup; a yaw-only firmware refuses the pitch half. Harmless until presence reconciliation lands (config plan section 10) | | `Motor controller: ERR unknown command`, a burst at startup | Only on firmware older than the config handshake: it does not know `SET`. Seen on tower 111 before its flash; harmless there, and gone once the firmware is current |
| `encoder tracking error on yaw: 500366 counts (limit 400)` | Before homing, the firmware presets `XACTUAL` to the centre count while the encoder sits at its own zero, so they disagree by the whole range. Homing resyncs them. **If it persists after a successful home, that is real** | | `encoder tracking error on yaw: 500366 counts (limit 400)` | Before homing, the firmware presets `XACTUAL` to the centre count while the encoder sits at its own zero, so they disagree by the whole range. Homing resyncs them. **If it persists after a successful home, that is real** |
> **On an unfamiliar drivetrain, homing is the dangerous step** - it finds the > **On an unfamiliar drivetrain, homing is the dangerous step** - it finds the
> endstop by driving into it. Start from a deliberately slow `homing_speed` and > endstop by driving into it. Start from a deliberately slow
> walk it up, watching the free-running encoder gap that `ENC_DEVIATION_LIMIT` is > `homing_speed_deg_s` and walk it up, watching the free-running encoder gap that
> measured against. Be at the rig with a hand on the power for the first one. > `enc_deviation` is measured against. Be at the rig with a hand on the power for
> the first one - which is why `scripts/install-tower` asks before it homes.
### Phase 2 - camera `[verified]` ### Phase 2 - camera `[verified]`
@ -753,9 +835,10 @@ the ground station. That lives in `~/zkms-client/`, outside this repository.
### Phase 4 - IMU and ambient sensor ### Phase 4 - IMU and ambient sensor
Only if fitted. `hardware.ini` should already say so, and `[Features] Only if fitted, and declared in `hardware.ini` (`imu`, `env`) - the declaration
enable_imu` / `enable_env` have to agree with it (see step 9 - declaring is what switches them on. `test hardware imu` wants a `0xFA`-framed stream from
`imu = none` does not yet switch the flag off by itself). the MTi; `test hardware env` wants a real I2C adapter and a CRC-valid SHT41
reading. `[verified]` on tower 111, 2026-09-23: both.
### Phase 5 - unattended ### Phase 5 - unattended
@ -766,9 +849,10 @@ scripts/fgc start --start # detached, inside tmux
scripts/fgc status # running? since when? which PID? scripts/fgc status # running? since when? which PID?
``` ```
Close the ssh session, reconnect, and check it is still running. Then reboot the Close the ssh session, reconnect, and check it is still running. Then install
tower and confirm the USB preconditions survived (`install.sh --probe`) and the the boot service (`scripts/install-tower` asks; [deployment.md](deployment.md)
app comes back. section 4), reboot the tower, and confirm with `scripts/install-tower --verify`
that the USB preconditions survived and the program came back by itself.
### Calibration ### Calibration
@ -791,7 +875,31 @@ turning.
So the order matters. **Host first, firmware second.** Each step below either So the order matters. **Host first, firmware second.** Each step below either
changes nothing or is reversible, and the step that could strand the tower comes changes nothing or is reversible, and the step that could strand the tower comes
after the one that prevents it. after the one that prevents it. `scripts/install-tower` follows this order by
itself, and blocks the flash while any drivetrain value is missing.
### The worked example: tower 111 `[verified]`
The Delta tower, which predated `hardware.ini`, the handshake and the kits.
2026-09-23 and -24, through the installer with the Delta kit:
1. **A rollback copy** of the sources, the running binary and `config.ini`, in
`~ggs/rollback-20260923/`, before anything changed.
2. **The host half.** Preconditions, `hardware.ini` with every drivetrain value
a `; TODO`, `config.ini`'s duplicated keys commented out, the missing `night`
profile, a new host. `test hardware`: axes, IMU, env and camera PASS; motor
FAIL on the old firmware's missing echoes, as expected (step 2 below). Firmware
blocked. It homed and resumed capturing on the new host.
3. **The drivetrain, without inventing it.** The sense resistors were read off
both boards (75 mOhm). Everything else was converted from what the old
firmware ran - its revision is in `DUMP BEGIN build=`, and git history at
that revision has its register values - through the host's own planners, and
round-tripped back to the identical registers before being written. Homing
and centring speeds were set to what that firmware used, not the kit's
gentle defaults, since its endstop detection was proven at those.
4. **The flash**, by the installer: every gap filled, the firmware built from
`hardware.ini` and flashed, `test hardware` green on every leaf, and homing
within 0.03 % of the old firmware's travel.
### 0. Record what you are rolling back to ### 0. Record what you are rolling back to
@ -806,9 +914,14 @@ the comparison that tells you whether the new one adopted the same values.
### 1. Update `hardware.ini` — nothing else, and no restart yet ### 1. Update `hardware.ini` — nothing else, and no restart yet
Add the `[Motor.<axis>]` block (section 5). The five keys the new firmware will Add the `[Motor.<axis>]` block (section 5) and `[Pins]`. The firmware asks for
not run without are `steps_per_rev`, `enc_deviation`, `sw_mode`, plus the two eleven values per axis - `steps_per_rev`, `global_scaler`, `run_irun`,
speeds; `install.sh --stdout` generates the block if you would rather not type it. `homing_irun`, `center_count`, `soft_margin`, `enc_deviation`, `normal_speed`,
`homing_speed`, `center_speed`, `sw_mode` - and the host derives them from the
block (currents from milliamps and the sense resistor, speeds from degrees per
second, centre and margin from `config.ini`). `install.sh --stdout` generates the
block; for values nobody has measured, convert what the old firmware runs, as
for tower 111 above.
Doing this **before** flashing is what keeps the window at zero: the host is ready Doing this **before** flashing is what keeps the window at zero: the host is ready
to answer the moment the firmware starts asking. to answer the moment the firmware starts asking.
@ -825,16 +938,21 @@ scripts/fgc peek
test hardware motor test hardware motor
``` ```
Expect: no handshake lines at all, and the motor leaf reporting a **version skew** Expect: no handshake lines at all (and possibly a burst of `ERR unknown
rather than a mismatch — command` - the old firmware refusing `SET`), and the motor leaf **failing** on
missing echoes, reported as a version skew rather than a mismatch:
``` ```
; yaw_current: yaw: firmware echoes only part of the current (2 of 3 fields ; yaw_pins: yaw: the firmware does not report the pins it was built with, so its
absent), so it cannot be confirmed -- it predates the full CURRENT echo. Re-flash it wiring cannot be confirmed -- it predates the pin echo. Re-flash it
``` ```
If instead you see wrong *values*, stop: something in `hardware.ini` disagrees That FAIL is by design - a check that could not happen must not read as a pass -
with what the tower is really running, and flashing will not fix it. so on old firmware the gate is "every other leaf green, and `hw_motor` failing
only on missing echoes". A speed can also differ before the first home, since
the host sends it after homing. If instead you see wrong *values* for something
the firmware does echo, stop: `hardware.ini` disagrees with what the tower is
really running, and flashing will not fix it.
### 3. Flash ### 3. Flash
@ -923,37 +1041,43 @@ table and lets the tower move while you work out what `hardware.ini` is missing.
## 11. Verification checklist ## 11. Verification checklist
The checks that move nothing can be run as one command, which is what a script Most of this is one command, which changes nothing:
should use rather than typing into the dashboard. Stop the app first: it holds
the serial port. ```bash
scripts/install-tower --verify
```
It checks the USB preconditions, that `config.ini` and `hardware.ini` load
together, that `hardware.ini` has no gaps, that the firmware reports the pins it
declares, and the boot service - and runs `test hardware`: inside the running
program when it is up (pausing capture, then resuming it), or as a one-shot run
when it is not. By hand, the one-shot form is:
```bash ```bash
scripts/fgc stop
GENICAM_GENTL64_PATH=/opt/VimbaX/cti ./build/fire_gimbal_control --no-mqtt --test hardware GENICAM_GENTL64_PATH=/opt/VimbaX/cti ./build/fire_gimbal_control --no-mqtt --test hardware
echo $? # 0 PASS, 4 FAIL, 5 could not start (the log says why) echo $? # 0 PASS, 4 FAIL, 5 could not start (the log says why)
scripts/fgc start
``` ```
It waits for the firmware to be talking and configured before it looks, so a It waits for the firmware to be talking and configured before it looks, so a
freshly flashed board is not failed for values the host had not sent yet. freshly flashed board is not failed for values the host had not sent yet.
`build/fgc-config check --config config.ini --hardware /etc/fire_gimbal_control/hardware.ini`
checks that the two files load together, and changes nothing.
| Check | How | Tower 121 | | Check | How | Tower 121 (IOTA) | Tower 111 (Delta) |
|---|---|---| |---|---|---|---|
| Key-only ssh | `ssh ggs@<tower> hostname`, no prompt | ✅ | | Key-only ssh | `ssh ggs@<tower> hostname`, no prompt | ✅ | ✅ |
| Preconditions survive a reboot | reboot, then `install.sh --probe`: `usbfs_memory_mb 1000`, `lpm_permit 0` | ✅ confirmed across a real reboot | | Preconditions applied | `--verify`: `usbfs_memory_mb 1000`, camera LPM `0` | ✅ | ✅ 2026-09-23 (were 16 and `u1_u2`) |
| `hardware.ini` found unaided | run with no `FGC_HARDWARE`: startup names `/etc/fire_gimbal_control/hardware.ini` | ✅ | | Preconditions survive a reboot | reboot, then `--verify` | ✅ confirmed across a real reboot | ❓ not rebooted yet |
| Camera enumerates | `ListCameras_VmbC` reports the `DEV_` id in `hardware.ini` | ✅ `DEV_1AB22C0B7D4E` | | `hardware.ini` found unaided | startup names `/etc/fire_gimbal_control/hardware.ini` | ✅ | ✅ |
| Camera profile applies | startup logs `profile 'standard' applied ... (N settings)` | ✅ 17206 settings | | Declared hardware matches | startup logs `declared assembly confirmed`; `test hardware axes` | ✅ yaw | ✅ yaw, pitch |
| Firmware identity | `gimbal dump` reports the `GIT_REV` you flashed | ✅ `build=aab1925-dirty` | | IMU / ambient sensor | `test hardware imu` / `env` | none declared | ✅ MTi streaming, SHT41 CRC-valid on a real I2C bus |
| Driver answers over SPI | boot banner shows `IOIN Y=0x… version=0x30` | ✅ | | Camera enumerates | `ListCameras_VmbC` reports the `DEV_` id in `hardware.ini` | ✅ `DEV_1AB22C0B7D4E` | ✅ `DEV_1AB22C049CD1` |
| Remote re-flash | `BOOTSEL`, then `./deploy.sh` - no hands on the board | ✅ | | Camera profile applies | startup logs `profile 'standard' applied ... (N settings)` | ✅ 17206 settings | ✅ 3743 settings |
| Declared hardware matches | startup logs no presence warnings | ✅ no shadowed-key warnings | | Firmware identity | `DUMP BEGIN build=` is the revision flashed | ✅ | ✅ `a0b1e4a` |
| Homing repeatable | `gimbal home` several times, endstop counts within `home_spread_max_counts` | ✅ three homes within 0.27 % (1,210,778 / 1,208,358 / 1,207,514 counts) | | Driver answers over SPI | `IOIN` reads version `0x30` for every axis | ✅ | ✅ both axes |
| Endstop survives homing | inspect the stop after a home; the axis must stall against it, not push through | ✅ at 189 mA, after failing at full torque | | Firmware holds `hardware.ini` | `test hardware motor`: values, current, pins | ✅ | ✅ every leaf, 2026-09-24 |
| Degree scale derived | startup logs `counts_per_deg` agreeing with the homed travel | ✅ 3552.87, saved with `;prev` history | | Remote re-flash | no hands on the board | ✅ `BOOTSEL` | ✅ 1200-baud touch |
| Images land | `.jxl` files appear under the configured output dir | ❌ needs a capture run | | Homing repeatable | endstop counts across homes | ✅ three homes within 0.27 % | ✅ within 0.03 %, old and new firmware |
| Survives a disconnect | `scripts/fgc start`, close ssh, reconnect, `scripts/fgc status` | ✅ runs under the boot service, independent of any ssh session | | Endstop survives homing | the axis must stall against the stop, not push through | ✅ at 189 mA, after failing at full torque | not inspected; full current, as for its whole service life |
| Starts at boot | reboot with nobody logged in; `scripts/fgc status` from a later ssh | ✅ 2026-09-23, lock under `/run/user/1000` (lingering on) | | Degree scale derived | startup logs `counts_per_deg` agreeing with the homed travel | ✅ 3552.87 | ✅ agrees, calibration left alone |
| Reports to a broker | topics arrive at the configured broker | ❌ no broker on this tower yet | | Images land | `.jxl` files appear under the configured output dir | ❌ needs a capture run | ✅ |
| Starts at boot | reboot with nobody logged in; `scripts/fgc status` from a later ssh | ✅ lock under `/run/user/1000` | ❓ service enabled (`--init --start`), not rebooted yet |
| Reports to a broker | topics arrive at the configured broker | ❌ no broker on this tower yet | ❓ not checked |

4
scripts/fire-gimbal-control.service
View File

@ -36,7 +36,9 @@ WorkingDirectory=@REPO@
ExecStart=@REPO@/scripts/fgc start @ARGS@ ExecStart=@REPO@/scripts/fgc start @ARGS@
ExecStop=@REPO@/scripts/fgc stop ExecStop=@REPO@/scripts/fgc stop
TimeoutStartSec=120 TimeoutStartSec=120
TimeoutStopSec=60 # Longer than `fgc stop` itself waits (120 s) for the program's clean shutdown:
# past this systemd kills it outright, which wedges the camera for the next run.
TimeoutStopSec=150
[Install] [Install]
WantedBy=multi-user.target WantedBy=multi-user.target