fwt_software/docs/raw12-pipeline-implementation-plan.md

# Implementation plan — raw capture, colour on the tower, 12-bit JPEG XL

Based on [raw12-pipeline-measurements.md](raw12-pipeline-measurements.md) (towers 121 and 111,
2026-10-02/04). Not started; decisions marked **[decide]** are the user's.

## Goal

The camera sends raw Bayer (`BayerRG12p`, or `BayerRG8` where chosen) at 250 MB/s, so the stream
runs 2–3× faster than RGB8 and auto-exposure settles in roughly half the time. The tower unpacks,
converts to colour (edge-aware) and saves JPEG XL at the frame's bit depth. A waypoint's frame is
kept when the image has settled, not after a fixed wait. Profiles that ask for RGB8 keep working
exactly as today.

## What the measurements fix

| Decision | Value | Why (measured) |
|---|---|---|
| Pixel format | from the profile XML: `BayerRG12p` (proposed) or `BayerRG8`; `RGB8` still accepted | 9.86 / 14.75 / 4.95 fps; no losses at 250 MB/s on either tower |
| Colour conversion | OpenCV edge-aware, 16-bit for 12-bit input | 30–35 ms; SSIM 0.973 vs 0.968 bilinear at the same size |
| Save | colour JXL, **distance 1.0, effort 3, 4 threads**; 12-bit via `JxlEncoderSetFrameBitDepth` | 2.7 MB/frame; 0.57 s (IOTA), 0.85 s (Delta) on the real frame |
| Workers | 1 × 4 threads on the IOTA, **2 × 2 on the Delta** | IOTA keeps up at 0.7 s/frame; Delta at 1.0 s only with 2 workers |
| Queue | bounded, ~4 frames (24 MB each raw) | unbounded today; pile-up measured at 23–97 frames |
| Keep rule | brightness in the camera's target band and steady (≤2%) over 3 frames, with a time limit | right in 21 of 24 steps; "exposure unchanged" 0 of 24 |
| Lossless | must set `uses_original_profile` | known issue #20 (production's distance 0 is lossy) |

## Processing budget the design must fit

Measured on the **identical frame** on both computers (5328×3040, real scene; 4 threads; see the
report's [Where the time goes](raw12-pipeline-measurements.md#where-the-time-goes)).

### Time per saved frame, step by step

| Step | IOTA (121) | 3 Delta (111) | Where it runs in the new code |
|---|---|---|---|
| Receive frame (SDK buffer → copy) | 1.6 ms (8-bit raw, measured); ~2.4 ms for 12p (estimated from size) | not measured | `VimbaCameraSource::FrameReceived` |
| Unpack `BayerRG12p` → 16-bit | 10 ms | 17 ms | `RawImage::unpack12p` (core) |
| De-Bayer, edge-aware, 16-bit (bilinear) | 28 ms (17) | 33 ms (22) | `ImagePipeline::process` (OpenCV) |
| Rotate (only if `rotation_deg` ≠ 0) | ~30 ms (measured on RGB8 8-bit) | not measured | `ImagePipeline::process` |
| JPEG XL encode, 12-bit colour, d1.0 e3 | 480 ms | 765 ms | `JpegXlEncoder` |
| Write file (2.7 MB) | 1 ms | 4 ms | `JpegXlEncoder` |
| **Total per frame** | **~520 ms** | **~820 ms** | one save-worker job |
| Today, for comparison (RGB8 → JXL d0.8 e4) | 567 ms | 872 ms | |

Unpacking and colour conversion are ~7% of a save; the encoder is ~90%. Nothing in the new steps
needs optimising before the encoder does.

### Space

| Saved as | Per frame | Per day, one frame / 2 s | / 5 s | / 10 s |
|---|---|---|---|---|
| Today: RGB8 → JXL d0.8 e4 (camera RGB8) | 3.3–4.0 MB | 141–172 GB | 57–69 GB | 28–34 GB |
| **12-bit colour JXL d1.0 e3 (proposed)** | **2.65 MB** | **114 GB** | **46 GB** | **23 GB** |
| 12-bit colour JXL d0.5 | 5.15 MB | 222 GB | 89 GB | 44 GB |
| 12-bit raw planes, lossless (option) | 16.8–17.7 MB | 727–763 GB | 291–305 GB | 145–153 GB |
| 12-bit colour, lossless | 44.8 MB | 1.9 TB | 774 GB | 387 GB |
| Working memory per save worker (12-bit colour) | ~0.35 GB; 0.7 GB for 1 worker, 1.1 GB for 2 (measured) | | | |
| Queued raw frame | 24.3 MB each | | | |

The towers have 11–22 GB free: saved images must leave the tower continuously.

### Pace

| | IOTA (121) | 3 Delta (111) |
|---|---|---|
| Camera frame interval at 250 MB/s (12-bit / 8-bit raw / RGB8) | 0.10 / 0.07 / 0.20 s | same camera, same |
| Exposure settle after a large change (12-bit) | ~3 s (2.2–6.6) | same camera, same |
| Saves per second, 1 worker × 4 threads | 1.79 (0.56 s each) | 1.19 (0.84 s each) |
| Saves per second, 2 workers × 2 threads | 2.15 (0.47 s) | 1.53 (0.65 s) |
| Keeps up with one frame every 0.7 s | yes, 1 worker | only with 2 workers |

A waypoint takes ~1.5–5 s (slew + exposure settle), so saving occupies 10–55% of each tower's
capacity at scan pace. That is why the design keeps saving in the background, sizes workers per
computer, and still bounds the queue for bursts.

## Phases

Each phase builds and passes `ctest` on its own; RGB8 profiles behave as today after every phase.

### 1. Encoder: correct lossless, 12-bit, configurable threads

`src/camera/JpegXlEncoder.cpp`, `include/fgc/JpegXlEncoder.h`
- One entry point taking `bits` (8 or 12) as well as channels: 16-bit input words when `bits > 8`,
  `bits_per_sample = bits`, and `JxlEncoderSetFrameBitDepth(JXL_BIT_DEPTH_FROM_CODESTREAM)`.
- `uses_original_profile = JXL_TRUE` when `distance == 0` (fixes #20).
- `threads` from config instead of the hard-coded 3 (3 is slower than 2 on the IOTA).
- The 8-bit lossy path must stay byte-identical to today's output (the benchmark checked its copy of
  these settings against production).
- Reuse: `encodeMem` in `tools/camera-bench/raw12_bench.cpp` is the tested version of all of this.

### 2. Frames that say what they hold

`include/fgc/ICameraSource.h`, `src/camera/VimbaCameraSource.cpp`, `include/fgc/mock/MockCameraSource.h`
- `Frame` gains `PixelLayout layout` (`RGB8`, `Mono8`, `BayerRG8`, `BayerRG12p`) and the per-frame
  camera data: exposure (µs), gain (dB), white-balance ratios, camera timestamp, and a brightness
  mean and clipped share.
- `FrameReceived` takes the layout from `pFrame->GetPixelFormat()` instead of `channels = sz/(w*h)`,
  which gives 1 for 12p (wrong). It reads the chunk data (`ChunkExposureTime`, `ChunkGain`,
  `ChunkBalanceRatioRed/Blue`) and samples brightness from the buffer (every ~1000th byte: under 1 ms,
  as `sample_brightness` in `camera_bench.py`).
- Optional: keep the newest frame in the SDK buffer instead of copying every frame (measured
  1.6–4 ms per frame, 2–4% of a core; low priority).
- `MockCameraSource` can produce `BayerRG8` and `BayerRG12p` frames, so the pipeline runs without a
  camera.

### 3. Raw handling in the core library (testable without OpenCV)

New `include/fgc/RawImage.h`, `src/core/RawImage.cpp` (in `fgc_core`)
- `unpack12p(const uint8_t*, uint16_t*, n)` (GenICam 12p, two pixels in three bytes).
- `BrightnessSample` (mean on a 0–255 scale, clipped share) per layout.
- The camera's `BayerRG` is OpenCV's `BayerBG` constant; that mapping and the layouts live here.

### 4. Pipeline: convert, encode, several workers, bounded queue

`src/camera/ImagePipeline.cpp`, `include/fgc/ImagePipeline.h`
- `process()` by layout: RGB8 as today; Bayer → (unpack) → `cv::cvtColor(..., COLOR_BayerBG2RGB_EA)`
  in 8 or 16 bit → rotate (16-bit works) → encode with the frame's bit depth.
- Optional `save_as = raw_planes`: the mosaic rearranged into four colour planes, lossless
  (17–18 MB/frame; for a detector that wants raw values). **[decide]** whether to offer it at all.
- N worker threads (`save_workers`), each encoding with `jxl_threads`.
- Bounded queue (`save_queue_max`). **[decide]** what happens when it is full:
  - *backpressure (recommended)*: `ImagePipeline::canAccept()`, and `CaptureScheduler` waits before
    moving to the next waypoint, so no waypoint is lost and the scan slows down;
  - *drop*: the frame is discarded with a warning and a counter.
- Pose still sampled at `submit()`.
- Peak memory per worker: ~0.35 GB (16-bit colour), measured 0.7 GB for 1 worker, 1.1 GB for 2.

### 5. Keep the right frame

`include/fgc/CaptureScheduler.h`, `src/core/CaptureScheduler.cpp`, `ICameraSource`
- After standstill, wait a short mechanical settle (`settle_wait_ms`, now a minimum; default ~200 ms),
  then keep the first frame for which the last 3 frames are all in the camera's target band
  (`IntensityControllerTarget ± IntensityControllerTolerance`, read from the camera when a profile
  is applied) and brightness changed ≤2% frame to frame.
- Exposure and gain at their limits with brightness still out of band count as settled (waiting
  cannot help); log it.
- `frame_timeout_ms` stays the upper bound; then keep the best frame seen (closest to the band),
  not nothing.
- Needs the camera source to expose recent frames' data (`recentFrames(since)` or a callback); the
  rule itself goes in `fgc_core` (`SettleRule`), so it can be tested on recorded series.
- Falls back to today's fixed wait when frames carry no brightness (mock without data, old profile).

### 6. Configuration

`include/fgc/Config.h`, `src/core/Config.cpp`, `config/config.example.ini`, hardware kit `hardware.ini`
- `[Camera]`: `jxl_distance = 1.0`, `jxl_effort = 3` (new defaults, **[decide]**), `jxl_threads`,
  `demosaic = ea | bilinear`, `save_as = colour | raw_planes`.
- `hardware.ini` (what the tower *is*): `save_workers` (1 on IOTA, 2 on Delta), `jxl_threads`
  (4 / 2), `save_queue_max`. These depend on the computer, not on what the tower does.
- `[Camera.<name>.<profile>]`: `settle_wait_ms` becomes the minimum mechanical settle; add
  `settle_rule = brightness | fixed` so a tower can be switched back.
- Validation and tests in `tests/test_config.cpp`, as for the existing keys.

### 7. Camera profiles

`scripts/export_camera_profile.py` (recipes), `config/camera_profiles/rgb/README.md`
- `standard`: `PixelFormat = BayerRG12p` (**[decide]** vs `BayerRG8`), binning 1, full sensor,
  `DeviceLinkThroughputLimit = 250000000`, `AcquisitionFrameRate` = max,
  `ChunkModeActive` with ExposureTime, Gain, BalanceRatioRed/Blue, Timestamp enabled (enable each
  item before activating chunk mode), `ExposureAutoMax = 20 ms`.
- `night`: `BayerRG12p`, long exposure allowed, where 12 bits carry real shadow detail.
- White balance: **[decide]** keep fixed ratios, or `BalanceWhiteAuto = Continuous` (converges in
  2–4 frames, stable within ±3%).
- Profiles stay the only thing that writes camera settings (the existing rule).

### 8. Downstream and docs

- **[decide]** The saved files become 12-bit JPEG XL. Anything that reads them (ground station,
  detection) must decode at 16 bits; check before switching a tower. Consider adding the bit depth
  to `CamEvent` (`docs/mqtt-api.md`).
- `docs/architecture.md` (Image capture), `docs/configuration.md` (new keys),
  `docs/known-issues.md` (#16 settle, #20 lossless), profile README.

## Tests

- `fgc_core` unit tests: `unpack12p` against hand-packed bytes; brightness sampling per layout; the
  settle rule replayed on the 24 recorded auto-exposure series in
  `tools/camera-bench/results-*/ae.jsonl` (it must pick a correct frame in ≥ 21 of 24); config keys;
  scheduler backpressure with the mock pipeline.
- A small separate test target linking OpenCV + libjxl (`fgc_pipeline_tests`): 12-bit lossless round
  trip is exact; distance-0 8-bit is exact (#20); de-Bayer colour order matches a known patch; 8-bit
  lossy output is byte-identical to the old encoder for a fixed input.
- Mock run: the program in auto mode with `MockCameraSource` producing `BayerRG12p`.

## Verification on the towers

1. Deploy to 121 with the new `standard` profile, run auto mode for 30 min in daylight: per-waypoint
   log (move, standstill, settle-rule wait, save time), queue depth, CPU temperature.
2. Same on 111 with `save_workers = 2`.
3. Compare with the benchmark: save time per frame within ~10% of `raw12_bench pipeline`, no backlog,
   temperatures as measured (53 °C IOTA, 66 °C Delta at one frame per 2 s).
4. Check saved files decode at 12 bits and match the camera's colour (channel means).
5. One dusk or night run to confirm the 12-bit shadow benefit before deciding the night profile.

## Order and size

| Phase | Depends on | Rough size |
|---|---|---|
| 1 Encoder | – | small |
| 2 Frames | – | medium |
| 3 Raw core | – | small |
| 4 Pipeline | 1, 2, 3 | medium |
| 5 Keep rule | 2, 3 | medium |
| 6 Config | 4, 5 | small |
| 7 Profiles | 2 | small (plus a tower session per profile) |
| 8 Downstream, docs | all | small; downstream check is outside this repo |

Phases 1–3 are independent and can go first; 4 and 5 can then proceed in parallel.