# Deploying to the Gimbal LattePanda

Step-by-step guide to deploy `fire_gimbal_control` on a tower's LattePanda Sigma (Linux x86-64). Like the
firmware workflow, this is a repeatable **build → install → configure → enable → verify** cycle; the
[Update / redeploy](#9-update--redeploy) section at the end is the short loop you run for every new version.

> Assumes a Debian/Ubuntu-based Linux on the LattePanda. For Arch/Manjaro swap `apt` for `pacman`
> (see [build-and-setup.md](build-and-setup.md)). Run as a normal user with `sudo` rights; the service runs
> under a dedicated `ggs` user.

## 0. Prerequisites (one-time per device)

Install the toolchain and system libraries:

```bash
sudo apt update
sudo apt install -y git cmake build-essential \
                    libopencv-dev libjxl-dev libboost-all-dev
```

Install the **Allied Vision Vimba X SDK** (proprietary, needed for the cameras):

1. Download Vimba X for Linux x86-64 from
   <https://www.alliedvision.com/en/products/software/vimba-x-sdk/>.
2. Unpack to a fixed location, e.g. `/opt/VimbaX_2024-1`.
3. Run the SDK's transport-layer installer so cameras are discoverable, e.g.:
   ```bash
   sudo /opt/VimbaX_2024-1/cti/Install_GenTL_Path.sh
   ```
4. Make its runtime libraries findable at run time:
   ```bash
   echo /opt/VimbaX_2024-1/api/lib | sudo tee /etc/ld.so.conf.d/vimbax.conf
   sudo ldconfig
   ```

Ensure an **MQTT broker** is reachable from the tower (the ground-station/ZKMS broker, or a local Mosquitto
for bring-up).

## 1. Get the code

```bash
sudo mkdir -p /opt/src && sudo chown "$USER" /opt/src
cd /opt/src
git clone <repo-url> fire_gimbal_control_src
cd fire_gimbal_control_src
# (or: git pull, for an update)
```

## 2. Build (full, with cameras + MQTT)

```bash
cmake -B build -DCMAKE_BUILD_TYPE=Release \
      -DWITH_VIMBA=ON -DWITH_MQTT=ON \
      -DVMB_HOME=/opt/VimbaX_2024-1
cmake --build build -j"$(nproc)"
```

- `WITH_MQTT=ON` fetches + builds Eclipse Paho from source on first configure (needs network).
- If a camera build isn't needed yet, use `-DWITH_VIMBA=OFF` for a mock binary to test the pipeline.
- Smoke-test the core before installing: `ctest --test-dir build` (after adding `-DBUILD_TESTING=ON`).

## 3. Install to a fixed location

```bash
sudo useradd -r -s /usr/sbin/nologin ggs 2>/dev/null || true
sudo mkdir -p /opt/fire_gimbal_control
sudo cp build/fire_gimbal_control /opt/fire_gimbal_control/
# demo-mode placeholder image (only needed if you ever run --demo)
sudo cp bin/x64/Release/test_smoke.jxl /opt/fire_gimbal_control/ 2>/dev/null || true
```

## 4. Configure

```bash
sudo cp config/config.example.ini /opt/fire_gimbal_control/config.ini
sudo nano /opt/fire_gimbal_control/config.ini
```

Set at least:

```ini
[General]
tower_name = ExampleTower           ; this tower's name (drives all MQTT topics)
image_interval = 3

[Network]
zkms_server_ip = 10.11.12.13        ; the broker address

[Serial]
device = /dev/ttyACM0               ; the motor-controller MCU
baud = 115200

[Camera]
id_Cam1 = DEV_1AB22C0AADED          ; or GigE IPs like 192.168.11.101
```

**Credentials** go in the environment, not the config file. Create a root-only env file for the service:

```bash
sudo install -m 600 /dev/null /etc/fire_gimbal_control/credentials.env 2>/dev/null || \
  { sudo mkdir -p /etc/fire_gimbal_control && sudo install -m 600 /dev/null /etc/fire_gimbal_control/credentials.env; }
printf 'FGC_MQTT_USER=fwt_gimbal\nFGC_MQTT_PW=CHANGE_ME\n' | sudo tee /etc/fire_gimbal_control/credentials.env >/dev/null
sudo chmod 600 /etc/fire_gimbal_control/credentials.env
```

Give the `ggs` user access to its files and the image output dir:

```bash
sudo chown -R ggs:ggs /opt/fire_gimbal_control
```

## 5. Hardware access

**Serial (motor controller):** let the service user open the port.

```bash
sudo usermod -aG dialout ggs
```

For a stable device name across reboots, add a udev rule keyed on the MCU's USB IDs (find them with
`udevadm info -a -n /dev/ttyACM0 | grep -m1 idVendor`) and point `[Serial] device` at the symlink:

```bash
echo 'SUBSYSTEM=="tty", ATTRS{idVendor}=="XXXX", ATTRS{idProduct}=="YYYY", SYMLINK+="ttyGimbal"' | \
  sudo tee /etc/udev/rules.d/99-gimbal.rules
sudo udevadm control --reload && sudo udevadm trigger
```

**GigE cameras (if used):** put the host NIC on the camera subnet (e.g. `192.168.11.0/24`), enable jumbo
frames, and confirm discovery with the Vimba X `VimbaXViewer`/`ListCameras` tool.

## 6. First run by hand (verify before enabling the service)

```bash
cd /opt/fire_gimbal_control
set -a; . /etc/fire_gimbal_control/credentials.env; set +a
./fire_gimbal_control --config ./config.ini --start --log-level debug
```

Watch for: `Loaded config`, `Serial controller started`, `Opened camera ...`, `MQTT connected`, then
`Saved .../RGB/<timestamp>.jxl` after the first capture. Type `exit` (or Ctrl-D) to stop. To validate without
cameras first: add `--mock-camera --mock-serial --no-mqtt`.

This run is attached to your terminal and dies with the ssh session — fine for a first check, not for
leaving it running. For that, use section 6b.

### 6b. Running detached (`scripts/fgc`)

The tower is operated over ssh, so the program has to survive the connection closing. `scripts/fgc` runs
it inside a tmux session, which owns the pty:

```bash
cd /opt/fire_gimbal_control
scripts/fgc start --start        # starts detached, returns as soon as it is up
scripts/fgc status               # running? since when? which PID?
scripts/fgc attach               # the dashboard — Ctrl-b d detaches, leaves it running
scripts/fgc stop                 # SIGTERM -> the same clean teardown as `exit`
scripts/fgc peek                 # snapshot of the pane without attaching
```

**`Ctrl-b d` detaches. The dashboard's own `q` key exits the program.** That is the one thing to get
right while attached.

`scripts/fgc stop` sends SIGTERM and waits up to 30s; the program handles it with the same shutdown path
as the `exit` command (stop capture, close the camera, save the exposure store). Do **not** `kill -9` —
an unclean exit leaves the camera wedged for the next run. If it will not stop, attach and look at why.

`scripts/fgc start` refuses if a run is already up, and prints the previous run's last output if it died
rather than silently discarding it.

Quitting the dashboard with `q` while attached ends the tmux session too, so you land straight back in
your shell — `exit` then closes ssh. A run that *fails* is the exception: its pane is kept so you can read
why (`remain-on-exit failed`), and `fgc status` shows it until `fgc stop` or the next `fgc start` clears it.

A kept pane has **no process behind it** — keys and Ctrl-C do nothing, which reads as a hung ssh session.
`Ctrl-b d` still detaches (tmux, not the pane, handles it); the session's status line spells that out along
the bottom of the screen. As a last resort `Enter ~ .` closes the ssh client itself from your end.

> **Why `_run` waits on the app instead of `exec`ing it.** When the app is itself the pane's process, tmux
> never records an exit status for it — `#{pane_dead_status}` comes back empty and the process lingers as a
> zombie — so `remain-on-exit failed` cannot tell success from failure and keeps *every* finished pane,
> including a clean `q`. That is what used to strand an operator on a frozen dead pane. Interposing a shell
> that reaps the child and exits with its status gives tmux a real status. Measured on the device: 3/3 panes
> kept when exec'd, 4/4 sessions closed cleanly when parented. The wrapper forwards SIGTERM/SIGINT/SIGHUP,
> so `fgc stop` still reaches the app and runs the full teardown.

### 6c. The single-instance lock

Two instances would fight over the motor's serial port, the camera, and the exposure store. The binary
prevents that itself: at startup it takes an exclusive `flock` on

- `/run/user/<uid>/fire_gimbal_control.lock` when that directory exists, else
- `/tmp/fire_gimbal_control-<uid>.lock`

and, if it is already held, exits with status **2** and:

```
Cannot start: another instance is already running (PID 1234); attach to it with: scripts/fgc attach
```

The path is derived from the uid and not from `$XDG_RUNTIME_DIR` on purpose: that variable is set in an
interactive ssh shell but not in a non-interactive `ssh host cmd` one, and a lock the two launch routes
disagree about would exclude nothing.

Because the check lives in the binary, it covers every route in — `scripts/fgc`, systemd, or a bare
`./fire_gimbal_control`. Nothing needs cleaning up after a crash or a power cut: the kernel releases an
`flock` when the holder dies, so a leftover lockfile with a dead PID in it never blocks a launch. Use
`--no-lock` only to run a second mock-only instance during development, and `--lock-file <path>` to
point at a different lockfile.

## 7. Install the systemd service

An **alternative** to section 6b, not a companion to it: systemd gives you start-at-boot and
restart-on-crash, but no attachable dashboard — you get `journalctl` instead. Pick one. If you run both by
accident the second one to start fails on the single-instance lock (section 6c), so the failure is loud
rather than a pair of processes corrupting each other's state.

```bash
sudo cp scripts/fire-gimbal-control.service /etc/systemd/system/
sudoedit /etc/systemd/system/fire-gimbal-control.service
```

Confirm/adjust these lines:

```ini
User=ggs
WorkingDirectory=/opt/fire_gimbal_control
ExecStart=/opt/fire_gimbal_control/fire_gimbal_control --start
Environment=FGC_CONFIG=/opt/fire_gimbal_control/config.ini
EnvironmentFile=/etc/fire_gimbal_control/credentials.env
```

(Use `ExecStart=... --init --start` for the first power-on if endstops must be found at boot.)

## 8. Enable and verify

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now fire-gimbal-control
systemctl status fire-gimbal-control
journalctl -u fire-gimbal-control -f          # live logs
```

Confirm end to end:
- Images accumulating under the configured `[Paths] output_dir` (default
  `~ggs/.local/share/fire_gimbal_control/images/RGB/`).
- CamEvents on the broker:
  `mosquitto_sub -h <broker> -t 'GGS/FWT/<tower>/#' -v`.

## 9. Update / redeploy

The short loop for each new version (mirrors the firmware flash cycle):

```bash
cd /opt/src/fire_gimbal_control_src
git pull
cmake --build build -j"$(nproc)"                 # reuses the configured build dir
sudo systemctl stop fire-gimbal-control
sudo cp build/fire_gimbal_control /opt/fire_gimbal_control/
sudo systemctl start fire-gimbal-control
journalctl -u fire-gimbal-control -n 50 -f       # confirm healthy
```

If `CMakeLists.txt` or options changed, re-run the `cmake -B build ...` configure step from
[section 2](#2-build-full-with-cameras--mqtt) first. To roll back, keep the previous binary
(`fire_gimbal_control.prev`) and swap it back.

## Troubleshooting

| Symptom | Check |
|---------|-------|
| `No config.ini found` | `FGC_CONFIG` / `--config` path; service `Environment=` line |
| `Failed to open serial port` | MCU plugged in, `dialout` group, `[Serial] device` path / udev symlink |
| `No camera found` / Vimba startup error | SDK installed, GenTL path set, `ldconfig`, NIC subnet, camera IDs in config |
| `MQTT connect failed` (degraded mode) | broker reachable, credentials env, firewall |
| Runs but no images | capture started (`--start` or `start` command), `output_dir` writable by `ggs` |
| `another instance is already running` (exit 2) | `scripts/fgc status`; a systemd unit and a tmux session both running (section 6c) |
| Program dies when ssh disconnects | started with `run.sh`/`--foreground` instead of `scripts/fgc start` (section 6b) |
| `No camera found` only when started via `scripts/fgc` | GenTL path not resolved in the pane — check `scripts/fgc peek`, and `/etc/profile.d/*Vimba*GenTL*.sh` or `VIMBA_CTI_PATH` |