fwt_software/docs/architecture.md

Architecture

Overview

The program rotates a pan gimbal carrying up to four cameras, periodically stops/points it, triggers the cameras, compresses frames to JPEG XL, writes them to disk, and announces each capture over MQTT. Remote operators can override behaviour over MQTT.

The design separates policy (the control logic) from mechanism (the I/O to hardware/broker):

• fgc_core — an SDK-independent static library: typed configuration, path resolution, logging, the telemetry/command parsers, and the CaptureScheduler (control state machine). Depends on nothing proprietary, so it builds and unit-tests anywhere.
• Five interfaces abstract the outside world, each with a real and a mock/null implementation:

Interface	Real	Mock / Null
IMotorController	SerialMotorController (Boost.Asio)	MockMotorController (simulated sweep)
IControlChannel	MqttControlChannel (Paho)	NullControlChannel (no broker)
ICameraSource	VimbaCameraSource (Vimba X), TarisCameraSource (InfraTec TarisIR, thermal), both behind CompositeCameraSource	MockCameraSource (synthetic frames, also thermal counts)
IImuSource	MtiImuSource (Xsens MTi, Boost.Asio)	MockImuSource (synthetic orientation)
IEnvSensor	Sht41EnvSensor (SHT41 over I2C)	MockEnvSensor (synthetic temp/humidity)

Both sensors are optional ([Features] enable_imu / enable_env) and feed the Sensors panel: the IMU supplies orientation and the IMU-referenced gimbal pitch setup, the SHT41 the ambient temperature and humidity. Long-running operations (gimbal pitch setup on a worker thread, gimbal diag captured from the firmware) publish progress + results into the UiSnapshot activity strip, polled on the control thread so all geometry/state mutation stays single-threaded.

Application picks real vs mock from config + CLI, wires everything to the ImagePipeline and CaptureScheduler, and runs the loop. Selecting mocks lets the whole system run with no hardware or broker.

                         ┌──────────────── fgc_core (no SDKs) ───────────────┐
   main.cpp ──► Application ──► CaptureScheduler ──► (interfaces below)       │
                  │  Config · Logger · Paths · TelemetryParser · CommandParser│
                  └───────────────────────────────────────────────────────────┘
                         │ builds + owns
        ┌────────────────┼───────────────────────────┬──────────────────────┐
   IMotorController   IControlChannel             ICameraSource          ImagePipeline
   Serial/Mock        Mqtt/Null                   Vimba/Mock      ◄── frames ──┘ encode→.jxl→CamEvent
        │                  │                           │
     serial            MQTT broker                  cameras

Threading model

Thread	Where	Role
Main / control loop	Application::run	10 ms tick: drain UI commands, scheduler.tick(), publish a UiSnapshot
UI input (headless)	HeadlessUi	reads stdin lines into the command sink
UI render + input (TUI)	TuiUi	FTXUI event loop + 10 Hz refresher; pulls snapshots, pushes commands
Serial I/O	SerialMotorController	Boost.Asio io_context; async read-until parses telemetry
Image worker	ImagePipeline	drains the frame queue: rotate → encode → write → publish
MQTT client	Paho (internal)	delivers callbacks; auto-reconnects (no busy-wait loop)
Camera acquisition	Vimba X (internal)	free-runs at the mode's frame rate and delivers frames via the observer, which keeps the latest complete one
Thermal acquisition	TarisCameraSource worker	owns every SDK call: connects (and reconnects), polls the newest frame of counts, switches the calibration profile; trigger() only takes the kept frame

Shared state is mutex-guarded: latest MotorTelemetry (serial), RemoteCommand queue (channel), the frame queue (pipeline), the console command queue, and the latest UiSnapshot. The CaptureScheduler runs only on the main thread; the UI is a pure observer + command source (it copies a snapshot to render and pushes command strings back through the same queue the console uses, so it never touches live logic objects). In TUI mode all Logger output is diverted to an on-screen pane via a Logger::setSink callback so the screen is never corrupted.

Data flow

1. Startup — main() parses CLI, resolves + loads config, constructs Application, which builds the motor/channel/camera (real or mock), the ImagePipeline, and the CaptureScheduler.
2. Telemetry — the real motor controller streams firmware ST Y:...[ P:...] lines; parseTelemetryLine turns each into a per-axis MotorTelemetry snapshot (state + encoder counts + flags). The mock synthesizes one. Encoder counts are mapped to/from degrees by Geometry ([Motor] calibration).
3. Mode — auto, pan or manual (OperatingMode.h), changed only by the mode command (console, dashboard keys, or MQTT Command) through Application::setMode, which publishes it on StatusCode. The same header's allowedIn() gates every command by mode.
4. Capture cycle (CaptureScheduler::tick, per 10 ms) — a move → settle → wait → keep machine:
   • auto: MOVE <yaw>,<pitch> to the next ScanGrid waypoint (front to back, then wrap). A yaw-only tower (no P: segment in ST) gets MOVE Y <yaw> instead, as its firmware refuses the pair.
   • pan: MOVE to the next of panHeadings() (left to right, then back to the left edge) at the pan's pitch, converted to counts via Geometry.
   • manual: nothing, bar a requested snapshot or continuous capture (capture on), which skip the move and wait only for standstill.
   • Once both axes report standstill at the target, wait settle_wait_ms (and, with settle_rule = brightness, until the brightness has settled), then whatever is left of image_interval since the last frame kept; keep the newest frame and move straight on to the next grid or pan position - unless saving is behind, when the scan holds until it catches up.
5. Acquisition — the camera free-runs at the active profile's AcquisitionFrameRate with its own auto-exposure tracking the light; trigger() hands over the freshest complete frame. See Image capture below.
6. Frame handling — the kept Frame goes to the callback, which submit()s it to the ImagePipeline. submit() samples the gimbal pose there and then — yaw from the encoder, pitch from the IMU (no IMU: the pitch encoder; no pitch axis either: the fixed pitch_angle_at_center) — and the pose travels through the queue with the frame, so encode latency cannot attribute a later pose to earlier pixels. The worker rotates it by Camera.rotation_deg, encodes JPEG XL, writes <output_dir>/<label>/<unix_ms>.jxl, and publishes a CamEvent carrying that pose.

Capture state machine

        mode auto or pan AND not already moving (straight after the last frame)
        AND room to save (else hold here until saving catches up)
              │
              ▼
   ┌──────────────────────┐  auto → next ScanGrid (yaw,pitch); pan → next pan heading
   │ MOVE <yaw>,<pitch>   │  unreachable targets skipped in the same tick
   │ (yaw-only: MOVE Y)   │  deg→counts via Geometry; set moving
   └──────────┬───────────┘
              │ (both axes standstill AND |xenc − target| ≤ tol)
              ▼
   ┌──────────────────────┐  the camera keeps streaming throughout; this pause is
   │ wait settle_wait_ms, │  what makes the kept frame one exposed at rest.
   │ then the keep rule   │  settle_rule = brightness: until brightness is in band
   │ if on                │  and steady, at most frame_timeout_ms (then keep, warn)
   └──────────┬───────────┘  move + wait > interval: warn (not on the return to the
              │              first waypoint, nor the first move after a mode starts)
              ▼
   ┌──────────────────────┐
   │ wait out the rest of │  interval counted from the last frame kept
   │ image_interval       │
   └──────────┬───────────┘
              ▼
   ┌──────────────────────┐  on success: clear moving, advance, restart the interval
   │  camera.trigger()    │  no fresh frame: retry next tick
   └──────────────────────┘  still none after frame_timeout_ms: warn, skip, advance

Triggering only at the settled target replaces the original's trigger-while-moving behaviour (see known-issues.md #7). The scheduler is unit-tested with mock doubles and an injected clock (tests/test_scheduler.cpp).

Note the scheduler knows nothing about image quality. It asks for an image and advances when it gets one; everything below is an implementation detail of the camera source. That is what keeps it testable.

Image capture

Two cameras, one array: CompositeCameraSource

With a thermal camera the scheduler and the pipeline still see one ICameraSource. The composite numbers the colour cameras first (camera 0 is the one the keep rule reads) and the thermal one after them. Its trigger() asks the colour source first; only if that delivered does it ask the thermal source, which hands over its newest frame if it is no older than max_frame_age_ms and otherwise says no at once. Each frame keeps the time it reached the host, so the two of one waypoint differ by up to that age. The colour camera failing stops the run as before; the thermal one failing never does (offline() says why, Health cams_offline names it). Thermal frames are Mono16 counts: the pipeline rotates them, takes tmax/tmed/hot spot on the rotated image (ThermalStats.h), and saves them 16-bit lossless whatever jxl_distance says. The SDK is declared in IrbgrabApi.h (not copied: its licence forbids that) and loaded with dlopen, so every build has the thermal code and its tests run against a fake SDK.

The model: free-run, then hold still

The camera runs continuously at the mode's frame rate with its own auto-exposure and auto-gain in Continuous. Almost every frame it produces is thrown away. Their job is not to be saved — it is to keep the camera's exposure controller converged on the current light, so that the one frame we do keep is already correct without anyone waiting for it.

At each waypoint the scheduler therefore:

1. moves, and waits for both axes to report standstill at the target;
2. waits a further settle_wait_ms (1 s in standard, 2.5 s in night);
3. with settle_rule = brightness, waits on until brightness is in the camera's target band and steady for 3 frames (configuration.md);
4. waits out what is left of image_interval, counted from the last frame kept, and keeps the newest complete frame.

Saving runs in background workers and never blocks this loop; if more than save_queue_max frames are waiting, the scan holds at the waypoint until there is room. Raw profiles (BayerRG8, BayerRG12p) are converted to colour on the tower and saved at their own bit depth (raw12-pipeline-measurements.md).

The wait is the whole mechanism. It guarantees the kept frame was exposed with the gimbal at rest (no slew blur) and after the auto-exposure has seen this waypoint's light for several frames. It also sets the floor on a profile's frame rate: a profile must produce comfortably more than one frame inside its own settle wait, which is why night at 1 Hz waits 2.5 s where standard at 10 Hz waits 1 s.

If no fresh frame arrives within frame_timeout_ms of the frame falling due, the waypoint is logged and skipped. A stalled camera costs single images, never the whole sweep. A merely slow camera is not stalled: with multi-second exposures the next frame may not be due yet, so the stream is only restarted once nothing newer has come for twice the longer of the exposure and the frame period, plus a second (at least 2 s; CameraStall.h). Restarting earlier throws the exposure in progress away.

Camera profiles

Everything on the camera comes from a profile: a settings XML exported from the VimbaX Viewer, at <profile_dir>/<name>.xml. The program hands that file to the Vimba X SDK (LoadSettings, restricted to the export's RemoteDevice block) and nothing else writes camera settings.

The reason is validity, not convenience. GenICam writability is stateful — AcquisitionFrameRate is only writable once AcquisitionFrameRateEnable is set, Width's maximum depends on binning, IntensityAutoPrecedence takes Minimizenoise but not MinimizeNoise — so a list of features applied line by line can be individually plausible and collectively wrong, and a line that fails leaves the camera quietly on its old value. The Viewer greys out invalid combinations as the operator configures; replaying its export is how that guarantee is inherited. An earlier design applied a hand-written INI feature list; it was removed rather than repaired, because the failure mode was silent.

Beside each XML the program writes <name>.txt, a generated human-readable record of the same settings. It configures nothing — it exists so a profile can be read and diffed without opening the Viewer. At start-up the XML is hashed and compared with the hash stored in the record; a mismatch regenerates the record and logs every setting that changed, so a camera behaving differently from the last run says so rather than being discovered later. Every *.xml in the directory is loaded at start-up, so a profile nobody uses today still fails at launch rather than the first night someone switches to it.

Each camera has its own [Camera.<name>] block and its own profile directory, so profiles are per camera rather than shared. Switch live with set camera profile [<camera>] <name>; that stops acquisition, reloads the XML and restarts, because binning, ROI and pixel format cannot be changed on a running stream.

Why not judge each frame instead

An earlier design software-triggered one frame per waypoint, measured it (mean luma, clipping, sharpness), corrected exposure and re-shot until it passed, seeded from a per-angle exposure store. It was removed. It duplicated — and fought — the camera's own exposure controller, which meters continuously in hardware and does not need the scan to stop for it. Its premise was that the camera's auto-exposure needs several frames to converge and a scan changes scene at every waypoint; free-running at 10 Hz supplies those frames for free, during the slew, at no cost to scan rate.

Bandwidth: why both modes bin 2×2

At PixelFormat = RGB8 a full-resolution 4512×4512 frame is 61 MB. Under the 250 MB/s DeviceLinkThroughputLimit that caps out at ~4 fps, and even at the interface maximum only ~7 — so 10 Hz at full resolution is not reachable in RGB8 at all. (The datasheet's "21 fps at full resolution" is 450 MB/s Mono8.) With BinningHorizontal/Vertical = 2 the frame is 2256×2256 ≈ 5 MP, 15.3 MB, and 10 Hz needs ~153 MB/s — comfortably inside the cap. Binning also sums four photosites, which is worth having at night. Raising the throughput limit to 450 is not the fix: it produced incomplete frames on this host (known-issues.md).

Tower 121 (1800 U-1620c, 5328×3040) measured differently on 2026-09-30: its camera budgets the limit against the unbinned frame, so 2×2 binning gives no frame rate there (RGB8 is 4.95 fps at 250 MB/s either way); BayerRG8 at full resolution reaches 14.75 fps at 250 MB/s and 26.2 at 450, and 450 MB/s streamed without loss. See camera-capture-measurements.md.

Full parameter reference: [Camera] in configuration.md.

Why this shape

Decoupling the control logic from the SDKs makes the core testable and the binary buildable/runnable without proprietary dependencies, while preserving the original real-time behaviour. Persistence remains deliberately limited to image files plus fire-and-forget MQTT.