# Deploying to the Gimbal LattePanda

Step-by-step guide to deploy `fire_gimbal_control` on a tower's LattePanda (Linux x86-64). Like the
firmware workflow, this is a repeatable **build → install → configure → enable → verify** cycle; the
[Update / redeploy](#9-update--redeploy) section at the end is the short loop you run for every new version.

> **Which LattePanda.** The deployed fleet is the **3 Delta** — read from tower 111's DMI
> `product_name`, *not* the Sigma, which this page claimed until 2026-09-18. Tower 121 is a
> **LattePanda IOTA**; see [lattepanda-iota.md](lattepanda-iota.md) and its setup kit under
> `hardware/lattepanda-iota/`. Nothing in this guide is board-specific — it is x86-64 Linux either way —
> so the distinction matters for the device paths in `hardware.ini`, not for these steps.
>
> Assumes a Debian/Ubuntu-based Linux on the LattePanda. For Arch/Manjaro swap `apt` for `pacman`
> (see [build-and-setup.md](build-and-setup.md)). Run as a normal user with `sudo` rights; the service runs
> under a dedicated `ggs` user.

## 0. Prerequisites (one-time per device)

Install the toolchain and system libraries:

```bash
sudo apt update
sudo apt install -y git cmake build-essential \
                    libopencv-dev libjxl-dev libboost-all-dev
```

Install the **Allied Vision Vimba X SDK** (proprietary, needed for the cameras):

1. Download Vimba X for Linux x86-64 from
   <https://www.alliedvision.com/en/products/software/vimba-x-sdk/>.
2. Unpack to a fixed location, e.g. `/opt/VimbaX_2024-1`.
3. Run the SDK's transport-layer installer so cameras are discoverable, e.g.:
   ```bash
   sudo /opt/VimbaX_2024-1/cti/Install_GenTL_Path.sh
   ```
4. Make its runtime libraries findable at run time:
   ```bash
   echo /opt/VimbaX_2024-1/api/lib | sudo tee /etc/ld.so.conf.d/vimbax.conf
   sudo ldconfig
   ```

Ensure an **MQTT broker** is reachable from the tower (the ground-station/ZKMS broker, or a local Mosquitto
for bring-up).

## 1. Get the code

```bash
sudo mkdir -p /opt/src && sudo chown "$USER" /opt/src
cd /opt/src
git clone <repo-url> fire_gimbal_control_src
cd fire_gimbal_control_src
# (or: git pull, for an update)
```

## 2. Build (full, with cameras + MQTT)

```bash
cmake -B build -DCMAKE_BUILD_TYPE=Release \
      -DWITH_VIMBA=ON -DWITH_MQTT=ON \
      -DVMB_HOME=/opt/VimbaX_2024-1
cmake --build build -j"$(nproc)"
```

- `WITH_MQTT=ON` fetches + builds Eclipse Paho from source on first configure (needs network).
- If a camera build isn't needed yet, use `-DWITH_VIMBA=OFF` for a mock binary to test the pipeline.
- Smoke-test the core before installing: `ctest --test-dir build` (after adding `-DBUILD_TESTING=ON`).

## 3. Install to a fixed location

```bash
sudo useradd -r -s /usr/sbin/nologin ggs 2>/dev/null || true
sudo mkdir -p /opt/fire_gimbal_control
sudo cp build/fire_gimbal_control /opt/fire_gimbal_control/
# demo-mode placeholder image (only needed if you ever run --demo)
sudo cp bin/x64/Release/test_smoke.jxl /opt/fire_gimbal_control/ 2>/dev/null || true
```

## 4. Configure

```bash
sudo cp config/config.example.ini /opt/fire_gimbal_control/config.ini
sudo nano /opt/fire_gimbal_control/config.ini
```

Set at least:

```ini
[General]
tower_name = ExampleTower           ; this tower's name (drives all MQTT topics)
image_interval = 3

[Network]
zkms_server_ip = 10.11.12.13        ; the broker address

[Serial]
device = /dev/ttyACM0               ; the motor-controller MCU
baud = 115200

[Camera.rgb]
id = DEV_1AB22C0AADED               ; or a GigE IP like 192.168.11.101
label = RGB
profile_dir = config/camera_profiles/rgb
profile = standard
```

**Credentials** go in the environment, not the config file. Create a root-only env file for the service:

```bash
sudo install -m 600 /dev/null /etc/fire_gimbal_control/credentials.env 2>/dev/null || \
  { sudo mkdir -p /etc/fire_gimbal_control && sudo install -m 600 /dev/null /etc/fire_gimbal_control/credentials.env; }
printf 'FGC_MQTT_USER=fwt_gimbal\nFGC_MQTT_PW=CHANGE_ME\n' | sudo tee /etc/fire_gimbal_control/credentials.env >/dev/null
sudo chmod 600 /etc/fire_gimbal_control/credentials.env
```

Give the `ggs` user access to its files and the image output dir:

```bash
sudo chown -R ggs:ggs /opt/fire_gimbal_control
```

## 5. Hardware access

**Serial (motor controller):** let the service user open the port.

```bash
sudo usermod -aG dialout ggs
```

For a stable device name across reboots, add a udev rule keyed on the MCU's USB IDs (find them with
`udevadm info -a -n /dev/ttyACM0 | grep -m1 idVendor`) and point `[Serial] device` at the symlink:

```bash
echo 'SUBSYSTEM=="tty", ATTRS{idVendor}=="XXXX", ATTRS{idProduct}=="YYYY", SYMLINK+="ttyGimbal"' | \
  sudo tee /etc/udev/rules.d/99-gimbal.rules
sudo udevadm control --reload && sudo udevadm trigger
```

**GigE cameras (if used):** put the host NIC on the camera subnet (e.g. `192.168.11.0/24`), enable jumbo
frames, and confirm discovery with the Vimba X `VimbaXViewer`/`ListCameras` tool.

## 6. First run by hand (verify before enabling the service)

```bash
cd /opt/fire_gimbal_control
set -a; . /etc/fire_gimbal_control/credentials.env; set +a
./fire_gimbal_control --config ./config.ini --init --start --log-level debug
```

Watch for: `Loaded config`, `Serial controller started`, `Opened camera ...`, `MQTT connected`, then
`Saved .../RGB/<timestamp>.jxl` after the first capture. Type `exit` (or Ctrl-D) to stop. To validate without
cameras first: add `--mock-camera --mock-serial --no-mqtt`.

This run is attached to your terminal and dies with the ssh session — fine for a first check, not for
leaving it running. For that, use section 6b.

### 6b. Running detached (`scripts/fgc`)

The tower is operated over ssh, so the program has to survive the connection closing. `scripts/fgc` runs
it inside a tmux session, which owns the pty:

```bash
cd /opt/fire_gimbal_control
scripts/fgc start --start        # starts detached, returns as soon as it is up
scripts/fgc status               # running? since when? which PID?
scripts/fgc attach               # the dashboard — Ctrl-b d detaches, leaves it running
scripts/fgc stop                 # SIGTERM -> the same clean teardown as `exit`
scripts/fgc peek                 # snapshot of the pane without attaching
```

**`Ctrl-b d` detaches. The dashboard's own `q` key opens an exit prompt** offering `q` (quit the
program) or `b` (background — detach and leave it running, the same as `Ctrl-b d`), with `Esc` to
cancel. `b` is offered only inside a `scripts/fgc` session; outside one there is nothing to detach
into and it is shown dimmed.

`scripts/fgc stop` sends SIGTERM and waits up to 30s; the program handles it with the same shutdown path
as the `exit` command (stop capture, close the camera, flush the image queue). Do **not** `kill -9` —
an unclean exit leaves the camera wedged for the next run. If it will not stop, attach and look at why.

`scripts/fgc start` refuses if a run is already up, and prints the previous run's last output if it died
rather than silently discarding it.

Choosing **quit** at that prompt while attached ends the tmux session too, so you land straight back in
your shell — `exit` then closes ssh. A run that *fails* is the exception: its pane is kept so you can read
why (`remain-on-exit failed`), and `fgc status` shows it until `fgc stop` or the next `fgc start` clears it.

A kept pane has **no process behind it** — keys and Ctrl-C do nothing, which reads as a hung ssh session.
`Ctrl-b d` still detaches (tmux, not the pane, handles it); the session's status line spells that out along
the bottom of the screen. As a last resort `Enter ~ .` closes the ssh client itself from your end.

> **Why `_run` waits on the app instead of `exec`ing it.** When the app is itself the pane's process, tmux
> never records an exit status for it — `#{pane_dead_status}` comes back empty and the process lingers as a
> zombie — so `remain-on-exit failed` cannot tell success from failure and keeps *every* finished pane,
> including a clean `q`. That is what used to strand an operator on a frozen dead pane. Interposing a shell
> that reaps the child and exits with its status gives tmux a real status. Measured on the device: 3/3 panes
> kept when exec'd, 4/4 sessions closed cleanly when parented. The wrapper forwards SIGTERM/SIGINT/SIGHUP,
> so `fgc stop` still reaches the app and runs the full teardown.

### 6c. The single-instance lock

Two instances would fight over the motor's serial port and the camera. The binary
prevents that itself: at startup it takes an exclusive `flock` on

- `/run/user/<uid>/fire_gimbal_control.lock` when that directory exists, else
- `/tmp/fire_gimbal_control-<uid>.lock`

and, if it is already held, exits with status **2** and:

```
Cannot start: another instance is already running (PID 1234); attach to it with: scripts/fgc attach
```

The path is derived from the uid and not from `$XDG_RUNTIME_DIR` on purpose: that variable is set in an
interactive ssh shell but not in a non-interactive `ssh host cmd` one, and a lock the two launch routes
disagree about would exclude nothing.

Because the check lives in the binary, it covers every route in — `scripts/fgc`, systemd, or a bare
`./fire_gimbal_control`. Nothing needs cleaning up after a crash or a power cut: the kernel releases an
`flock` when the holder dies, so a leftover lockfile with a dead PID in it never blocks a launch. Use
`--no-lock` only to run a second mock-only instance during development, and `--lock-file <path>` to
point at a different lockfile.

## 7. Install the systemd service

`scripts/install-tower` does this (it asks first, and shows the unit it will
write). The unit is the template `scripts/fire-gimbal-control.service`, and it
runs the program **through `scripts/fgc`, inside tmux** - so a tower started at
boot still has its dashboard (`scripts/fgc attach`), and `fgc status` / `peek` /
`stop` work from any ssh session. It is not an alternative to the tmux workflow
any more; it is how that workflow starts at boot.

By hand, fill in `@USER@`, `@REPO@` and `@ARGS@` (normally `--init --start`) and:

```bash
sudo cp fire-gimbal-control.service /etc/systemd/system/
sudo loginctl enable-linger ggs          # NOT optional - see below
sudo systemctl daemon-reload
sudo systemctl enable fire-gimbal-control
```

**Lingering is required.** Without it `/run/user/<uid>` does not exist until
someone logs in, so a program started at boot takes its single-instance lock in
`/tmp`; a later ssh session looks in `/run/user`, finds nothing, reports "not
running", and a second copy could start beside the first.

**`--init` homes the gimbal unattended at every boot.** That is what a tower in
service needs - the firmware keeps no position across a power cycle, so without
it the gimbal sits un-homed and captures nothing - but it is a deliberate choice,
and the installer asks for the arguments.

MQTT credentials go in `/etc/fire_gimbal_control/credentials.env`
(`FGC_MQTT_USER=` / `FGC_MQTT_PW=`, mode 0640 root:ggs). `fgc` reads it inside
the tmux pane rather than taking it from the unit's environment: a tmux server
that already exists hands a new session its own environment, not the caller's.

## 8. Enable and verify

```bash
sudo systemctl start fire-gimbal-control
scripts/fgc status                          # running, lockfile under /run/user/<uid>
scripts/install-tower --verify              # every check, including test hardware
```

`[verified]` on tower 121, 2026-09-23: the service came up by itself after a
reboot with nobody logged in, with its lock under `/run/user/1000`; `fgc status`
and the dashboard worked from a later ssh session; credentials reached the
program intact (a password with `$`, a quote and a space); `systemctl stop` shut
it down cleanly; and `install-tower --verify` passed after the reboot, USB
preconditions included. 121 runs it with no arguments, so it does not home at
boot - see the note on `--init` above.

Confirm end to end:
- Images accumulating under the configured `[Paths] output_dir` (default
  `~ggs/.local/share/fire_gimbal_control/images/RGB/`).
- CamEvents on the broker:
  `mosquitto_sub -h <broker> -t 'GGS/FWT/<tower>/#' -v`.

## 9. Update / redeploy

The short loop for each new version (mirrors the firmware flash cycle):

```bash
cd /opt/src/fire_gimbal_control_src
git pull
cmake --build build -j"$(nproc)"                 # reuses the configured build dir
sudo systemctl stop fire-gimbal-control
sudo cp build/fire_gimbal_control /opt/fire_gimbal_control/
sudo systemctl start fire-gimbal-control
journalctl -u fire-gimbal-control -n 50 -f       # confirm healthy
```

If `CMakeLists.txt` or options changed, re-run the `cmake -B build ...` configure step from
[section 2](#2-build-full-with-cameras--mqtt) first. To roll back, keep the previous binary
(`fire_gimbal_control.prev`) and swap it back.

> **One-time: `git pull` can delete your `.deploy.env`.** It was tracked by
> mistake until `b0127db`, which stopped tracking it and added it to
> `.gitignore`. Pulling *across* that commit therefore does what git does to any
> file a commit removes — deletes the working copy, credentials and all. It is a
> checkout, not a merge conflict, so there is no warning and nothing to resolve.
>
> Only checkouts that predate `b0127db` are exposed, and only once. Copy it aside
> first:
>
> ```bash
> cp .deploy.env ../deploy.env.bak 2>/dev/null   # then pull, then copy back
> ```
>
> After that pull the file is ignored and stays put. `deploy.sh` already excludes
> it from the rsync, so deploying never overwrites the tower's own copy — this is
> purely about the machine you deploy *from*. To recover one already lost:
> `git show b0127db^:.deploy.env`.

## 10. Where the log is

The program writes a persistent log alongside the self-test and diagnostics
reports:

```
~/.local/share/fire_gimbal_control/logs/fgc_<YYYYMMDD-HHMMSS>.<NNN>.log
```

One file per run, rotated by size within a run (`.000`, `.001`, …), oldest pruned
by count. Bounded at `[Logging] file_keep x file_max_mb`, 200 MB by default, at
roughly 5 MB/day in steady state.

This exists because a tower runs detached and, in TUI mode, every line otherwise
goes to a 500-line on-screen ring buffer that dies with the tmux session. The
file is **more verbose than the dashboard** — `info` on screen, `debug` on disk —
so it holds detail the pane never showed.

```bash
ls -t ~/.local/share/fire_gimbal_control/logs/fgc_*.log | head
tail -F "$(ls -t ~/.local/share/fire_gimbal_control/logs/fgc_*.log | head -1)"
```

Use `tail -F`, not `-f`: rotation replaces the file and `-f` would follow the old
inode into a file nothing writes to any more.

Each file opens with a header naming the build, tower, PID, config and hardware
paths and the full command line — repeated after every rotation, so an archive
found on its own still says where it came from. Lines carry a `YYYY-MM-DD` prefix
the console ones do not.

What to grep when a tower will not come up:

```bash
L=$(ls -t ~/.local/share/fire_gimbal_control/logs/fgc_*.log | head -1)
grep -E "hardware:|NEED|firmware config|need config" "$L"   # declaration + handshake
grep -E "\[WARN |\[ERROR" "$L"                              # everything it complained about
head -8 "$L"                                                # which build, which config
```

> Wire traces (`[Logging] trace` / `--trace`) reach this file too. The serial
> trace alone takes it from ~5 MB/day to ~41 MB/day, enough to age out the
> startup record within days — turn it on to chase something, off afterwards.

## Troubleshooting

| Symptom | Check |
|---------|-------|
| `No config.ini found` | `FGC_CONFIG` / `--config` path; service `Environment=` line |
| `Failed to open serial port` | MCU plugged in, `dialout` group, `[Serial] device` path / udev symlink |
| `No camera found` / Vimba startup error | SDK installed, GenTL path set, `ldconfig`, NIC subnet, camera IDs in config |
| `MQTT connect failed` (degraded mode) | broker reachable, credentials env, firewall |
| Runs but no images | capture started (`--start` or `start` command), `output_dir` writable by `ggs` |
| `another instance is already running` (exit 2) | `scripts/fgc status`; a systemd unit and a tmux session both running (section 6c) |
| Program dies when ssh disconnects | started with `run.sh`/`--foreground` instead of `scripts/fgc start` (section 6b) |
| `No camera found` only when started via `scripts/fgc` | GenTL path not resolved in the pane — check `scripts/fgc peek`, and `/etc/profile.d/*Vimba*GenTL*.sh` or `VIMBA_CTI_PATH` |