GGS
/
fwt_software
8
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 1 Wiki Activity
fwt_software/docs/deployment.md

16 KiB
Raw Blame History

Deploying to the Gimbal LattePanda

Step-by-step guide to deploy fire_gimbal_control on a tower's LattePanda (Linux x86-64). Like the firmware workflow, this is a repeatable build → install → configure → enable → verify cycle; the Update / redeploy section at the end is the short loop you run for every new version.

Which LattePanda. The deployed fleet is the 3 Delta — read from tower 111's DMI product_name, not the Sigma, which this page claimed until 2026-09-18. Tower 121 is a LattePanda IOTA; see lattepanda-iota.md and its setup kit under hardware/lattepanda-iota/. Nothing in this guide is board-specific — it is x86-64 Linux either way — so the distinction matters for the device paths in hardware.ini, not for these steps.

Assumes a Debian/Ubuntu-based Linux on the LattePanda. For Arch/Manjaro swap apt for pacman (see build-and-setup.md). Run as a normal user with sudo rights; the service runs under a dedicated ggs user.

0. Prerequisites (one-time per device)

Install the toolchain and system libraries:

sudo apt update
sudo apt install -y git cmake build-essential \
                    libopencv-dev libjxl-dev libboost-all-dev

Install the Allied Vision Vimba X SDK (proprietary, needed for the cameras):

  1. Download Vimba X for Linux x86-64 from https://www.alliedvision.com/en/products/software/vimba-x-sdk/.
  2. Unpack to a fixed location, e.g. /opt/VimbaX_2024-1.
  3. Run the SDK's transport-layer installer so cameras are discoverable, e.g.: 
    sudo /opt/VimbaX_2024-1/cti/Install_GenTL_Path.sh
  4. Make its runtime libraries findable at run time: 
    echo /opt/VimbaX_2024-1/api/lib | sudo tee /etc/ld.so.conf.d/vimbax.conf
sudo ldconfig

Ensure an MQTT broker is reachable from the tower (the ground-station/ZKMS broker, or a local Mosquitto for bring-up).

1. Get the code

sudo mkdir -p /opt/src && sudo chown "$USER" /opt/src
cd /opt/src
git clone <repo-url> fire_gimbal_control_src
cd fire_gimbal_control_src
# (or: git pull, for an update)

2. Build (full, with cameras + MQTT)

cmake -B build -DCMAKE_BUILD_TYPE=Release \
      -DWITH_VIMBA=ON -DWITH_MQTT=ON \
      -DVMB_HOME=/opt/VimbaX_2024-1
cmake --build build -j"$(nproc)"
  • WITH_MQTT=ON fetches + builds Eclipse Paho from source on first configure (needs network).
  • If a camera build isn't needed yet, use -DWITH_VIMBA=OFF for a mock binary to test the pipeline.
  • Smoke-test the core before installing: ctest --test-dir build (after adding -DBUILD_TESTING=ON).

3. Install to a fixed location

sudo useradd -r -s /usr/sbin/nologin ggs 2>/dev/null || true
sudo mkdir -p /opt/fire_gimbal_control
sudo cp build/fire_gimbal_control /opt/fire_gimbal_control/
# demo-mode placeholder image (only needed if you ever run --demo)
sudo cp bin/x64/Release/test_smoke.jxl /opt/fire_gimbal_control/ 2>/dev/null || true

4. Configure

sudo cp config/config.example.ini /opt/fire_gimbal_control/config.ini
sudo nano /opt/fire_gimbal_control/config.ini

Set at least:

[General]
tower_name = ExampleTower           ; this tower's name (drives all MQTT topics)
image_interval = 3

[Network]
zkms_server_ip = 10.11.12.13        ; the broker address

[Serial]
device = /dev/ttyACM0               ; the motor-controller MCU
baud = 115200

[Camera.rgb]
id = DEV_1AB22C0AADED               ; or a GigE IP like 192.168.11.101
label = RGB
profile_dir = config/camera_profiles/rgb
profile = standard

Credentials go in the environment, not the config file. Create a root-only env file for the service:

sudo install -m 600 /dev/null /etc/fire_gimbal_control/credentials.env 2>/dev/null || \
  { sudo mkdir -p /etc/fire_gimbal_control && sudo install -m 600 /dev/null /etc/fire_gimbal_control/credentials.env; }
printf 'FGC_MQTT_USER=fwt_gimbal\nFGC_MQTT_PW=CHANGE_ME\n' | sudo tee /etc/fire_gimbal_control/credentials.env >/dev/null
sudo chmod 600 /etc/fire_gimbal_control/credentials.env

Give the ggs user access to its files and the image output dir:

sudo chown -R ggs:ggs /opt/fire_gimbal_control

5. Hardware access

Serial (motor controller): let the service user open the port.

sudo usermod -aG dialout ggs

For a stable device name across reboots, add a udev rule keyed on the MCU's USB IDs (find them with udevadm info -a -n /dev/ttyACM0 | grep -m1 idVendor) and point [Serial] device at the symlink:

echo 'SUBSYSTEM=="tty", ATTRS{idVendor}=="XXXX", ATTRS{idProduct}=="YYYY", SYMLINK+="ttyGimbal"' | \
  sudo tee /etc/udev/rules.d/99-gimbal.rules
sudo udevadm control --reload && sudo udevadm trigger

GigE cameras (if used): put the host NIC on the camera subnet (e.g. 192.168.11.0/24), enable jumbo frames, and confirm discovery with the Vimba X VimbaXViewer/ListCameras tool.

6. First run by hand (verify before enabling the service)

cd /opt/fire_gimbal_control
set -a; . /etc/fire_gimbal_control/credentials.env; set +a
./fire_gimbal_control --config ./config.ini --init --start --log-level debug

Watch for: Loaded config, Serial controller started, Opened camera ..., MQTT connected, then Saved .../RGB/<timestamp>.jxl after the first capture. Type exit (or Ctrl-D) to stop. To validate without cameras first: add --mock-camera --mock-serial --no-mqtt.

This run is attached to your terminal and dies with the ssh session — fine for a first check, not for leaving it running. For that, use section 6b.

6b. Running detached (scripts/fgc)

The tower is operated over ssh, so the program has to survive the connection closing. scripts/fgc runs it inside a tmux session, which owns the pty:

cd /opt/fire_gimbal_control
scripts/fgc start --start        # starts detached, returns as soon as it is up
scripts/fgc status               # running? since when? which PID?
scripts/fgc attach               # the dashboard — Ctrl-b d detaches, leaves it running
scripts/fgc stop                 # SIGTERM -> the same clean teardown as `exit`
scripts/fgc peek                 # snapshot of the pane without attaching

Ctrl-b d detaches. The dashboard's own q key opens an exit prompt offering q (quit the program) or b (background — detach and leave it running, the same as Ctrl-b d), with Esc to cancel. b is offered only inside a scripts/fgc session; outside one there is nothing to detach into and it is shown dimmed.

scripts/fgc stop sends SIGTERM and waits up to 30s; the program handles it with the same shutdown path as the exit command (stop capture, close the camera, flush the image queue). Do not kill -9 — an unclean exit leaves the camera wedged for the next run. If it will not stop, attach and look at why.

scripts/fgc start refuses if a run is already up, and prints the previous run's last output if it died rather than silently discarding it.

Choosing quit at that prompt while attached ends the tmux session too, so you land straight back in your shell — exit then closes ssh. A run that fails is the exception: its pane is kept so you can read why (remain-on-exit failed), and fgc status shows it until fgc stop or the next fgc start clears it.

A kept pane has no process behind it — keys and Ctrl-C do nothing, which reads as a hung ssh session. Ctrl-b d still detaches (tmux, not the pane, handles it); the session's status line spells that out along the bottom of the screen. As a last resort Enter ~ . closes the ssh client itself from your end.

Why _run waits on the app instead of execing it. When the app is itself the pane's process, tmux never records an exit status for it — #{pane_dead_status} comes back empty and the process lingers as a zombie — so remain-on-exit failed cannot tell success from failure and keeps every finished pane, including a clean q. That is what used to strand an operator on a frozen dead pane. Interposing a shell that reaps the child and exits with its status gives tmux a real status. Measured on the device: 3/3 panes kept when exec'd, 4/4 sessions closed cleanly when parented. The wrapper forwards SIGTERM/SIGINT/SIGHUP, so fgc stop still reaches the app and runs the full teardown.

6c. The single-instance lock

Two instances would fight over the motor's serial port and the camera. The binary prevents that itself: at startup it takes an exclusive flock on

  • /run/user/<uid>/fire_gimbal_control.lock when that directory exists, else
  • /tmp/fire_gimbal_control-<uid>.lock

and, if it is already held, exits with status 2 and:

Cannot start: another instance is already running (PID 1234); attach to it with: scripts/fgc attach

The path is derived from the uid and not from $XDG_RUNTIME_DIR on purpose: that variable is set in an interactive ssh shell but not in a non-interactive ssh host cmd one, and a lock the two launch routes disagree about would exclude nothing.

Because the check lives in the binary, it covers every route in — scripts/fgc, systemd, or a bare ./fire_gimbal_control. Nothing needs cleaning up after a crash or a power cut: the kernel releases an flock when the holder dies, so a leftover lockfile with a dead PID in it never blocks a launch. Use --no-lock only to run a second mock-only instance during development, and --lock-file <path> to point at a different lockfile.

7. Install the systemd service

scripts/install-tower does this (it asks first, and shows the unit it will write). The unit is the template scripts/fire-gimbal-control.service, and it runs the program through scripts/fgc, inside tmux - so a tower started at boot still has its dashboard (scripts/fgc attach), and fgc status / peek / stop work from any ssh session. It is not an alternative to the tmux workflow any more; it is how that workflow starts at boot.

By hand, fill in @USER@, @REPO@ and @ARGS@ (normally --init --start) and:

sudo cp fire-gimbal-control.service /etc/systemd/system/
sudo loginctl enable-linger ggs          # NOT optional - see below
sudo systemctl daemon-reload
sudo systemctl enable fire-gimbal-control

Lingering is required. Without it /run/user/<uid> does not exist until someone logs in, so a program started at boot takes its single-instance lock in /tmp; a later ssh session looks in /run/user, finds nothing, reports "not running", and a second copy could start beside the first.

--init homes the gimbal unattended at every boot. That is what a tower in service needs - the firmware keeps no position across a power cycle, so without it the gimbal sits un-homed and captures nothing - but it is a deliberate choice, and the installer asks for the arguments.

MQTT credentials go in /etc/fire_gimbal_control/credentials.env (FGC_MQTT_USER= / FGC_MQTT_PW=, mode 0640 root:ggs). fgc reads it inside the tmux pane rather than taking it from the unit's environment: a tmux server that already exists hands a new session its own environment, not the caller's.

8. Enable and verify

sudo systemctl start fire-gimbal-control
scripts/fgc status                          # running, lockfile under /run/user/<uid>
scripts/install-tower --verify              # every check, including test hardware

[verified] on tower 121, 2026-09-23: the service came up by itself after a reboot with nobody logged in, with its lock under /run/user/1000; fgc status and the dashboard worked from a later ssh session; credentials reached the program intact (a password with $, a quote and a space); systemctl stop shut it down cleanly; and install-tower --verify passed after the reboot, USB preconditions included. 121 runs it with no arguments, so it does not home at boot - see the note on --init above.

Confirm end to end:

  • Images accumulating under the configured [Paths] output_dir (default ~ggs/.local/share/fire_gimbal_control/images/RGB/).
  • CamEvents on the broker: mosquitto_sub -h <broker> -t 'GGS/FWT/<tower>/#' -v.

9. Update / redeploy

The short loop for each new version (mirrors the firmware flash cycle):

cd /opt/src/fire_gimbal_control_src
git pull
cmake --build build -j"$(nproc)"                 # reuses the configured build dir
sudo systemctl stop fire-gimbal-control
sudo cp build/fire_gimbal_control /opt/fire_gimbal_control/
sudo systemctl start fire-gimbal-control
journalctl -u fire-gimbal-control -n 50 -f       # confirm healthy

If CMakeLists.txt or options changed, re-run the cmake -B build ... configure step from section 2 first. To roll back, keep the previous binary (fire_gimbal_control.prev) and swap it back.

One-time: git pull can delete your .deploy.env. It was tracked by mistake until b0127db, which stopped tracking it and added it to .gitignore. Pulling across that commit therefore does what git does to any file a commit removes — deletes the working copy, credentials and all. It is a checkout, not a merge conflict, so there is no warning and nothing to resolve.

Only checkouts that predate b0127db are exposed, and only once. Copy it aside first:

cp .deploy.env ../deploy.env.bak 2>/dev/null   # then pull, then copy back

After that pull the file is ignored and stays put. deploy.sh already excludes it from the rsync, so deploying never overwrites the tower's own copy — this is purely about the machine you deploy from. To recover one already lost: git show b0127db^:.deploy.env.

10. Where the log is

The program writes a persistent log alongside the self-test and diagnostics reports:

~/.local/share/fire_gimbal_control/logs/fgc_<YYYYMMDD-HHMMSS>.<NNN>.log

One file per run, rotated by size within a run (.000, .001, …), oldest pruned by count. Bounded at [Logging] file_keep x file_max_mb, 200 MB by default, at roughly 5 MB/day in steady state.

This exists because a tower runs detached and, in TUI mode, every line otherwise goes to a 500-line on-screen ring buffer that dies with the tmux session. The file is more verbose than the dashboard — info on screen, debug on disk — so it holds detail the pane never showed.

ls -t ~/.local/share/fire_gimbal_control/logs/fgc_*.log | head
tail -F "$(ls -t ~/.local/share/fire_gimbal_control/logs/fgc_*.log | head -1)"

Use tail -F, not -f: rotation replaces the file and -f would follow the old inode into a file nothing writes to any more.

Each file opens with a header naming the build, tower, PID, config and hardware paths and the full command line — repeated after every rotation, so an archive found on its own still says where it came from. Lines carry a YYYY-MM-DD prefix the console ones do not.

What to grep when a tower will not come up:

L=$(ls -t ~/.local/share/fire_gimbal_control/logs/fgc_*.log | head -1)
grep -E "hardware:|NEED|firmware config|need config" "$L"   # declaration + handshake
grep -E "\[WARN |\[ERROR" "$L"                              # everything it complained about
head -8 "$L"                                                # which build, which config

Wire traces ([Logging] trace / --trace) reach this file too. The serial trace alone takes it from ~5 MB/day to ~41 MB/day, enough to age out the startup record within days — turn it on to chase something, off afterwards.

Troubleshooting

Symptom Check
No config.ini found FGC_CONFIG / --config path; service Environment= line
Failed to open serial port MCU plugged in, dialout group, [Serial] device path / udev symlink
No camera found / Vimba startup error SDK installed, GenTL path set, ldconfig, NIC subnet, camera IDs in config
MQTT connect failed (degraded mode) broker reachable, credentials env, firewall
Runs but no images capture started (--start or start command), output_dir writable by ggs
another instance is already running (exit 2) scripts/fgc status; a systemd unit and a tmux session both running (section 6c)
Program dies when ssh disconnects started with run.sh/--foreground instead of scripts/fgc start (section 6b)
No camera found only when started via scripts/fgc GenTL path not resolved in the pane — check scripts/fgc peek, and /etc/profile.d/*Vimba*GenTL*.sh or VIMBA_CTI_PATH