fwt_software/docs/raw12-pipeline-implementation-plan.md

Implementation plan — raw capture, colour on the tower, 12-bit JPEG XL

Based on raw12-pipeline-measurements.md (towers 121 and 111, 2026-10-02/04).

Status (branch raw12-capture, 2026-10-05)

Decisions taken: BayerRG12p for both profiles; colour JPEG XL only (no raw-planes save); white balance stays fixed; a full save queue holds the scan (no lost waypoints).

Phase	State
1 Encoder	done: 12-bit input, true lossless (#20), threads from config; 8-bit lossy byte-identical to before (72 of 72)
2 Frames	done: pixel layout from the camera, chunk data (exposure, gain, white balance), brightness sample per frame. Not done: keeping frames in the SDK buffer instead of copying (optional)
3 Raw core	done: RawImage (layouts, unpack12p, brightness sample)
4 Pipeline	done: de-Bayer (edge-aware/bilinear), save at the frame's bit depth, N workers, canAccept(); scan holds when full
5 Keep rule	done for scan waypoints (SettleRule, settle_rule = brightness); snapshots and continuous capture keep the fixed wait
6 Config	done: [Camera] demosaic; [Hardware] save_workers, jxl_threads, save_queue_max (kit writes 1 / 2 per board); per-profile settle_rule
7 Profiles	recipe done (scripts/export_camera_profile.py, dry-run checked on the 1620c); profiles still to be exported on each tower
8 Docs	done (configuration, architecture, known issues, profile README). Downstream 12-bit decode still to check

Tests: 457, of which 36 new, covering the encoder, the pipeline, raw handling, the keep rule (including a replay of the 24 recorded steps), the scheduler and config. Builds with the camera SDK on tower 121.

Still to do before switching a tower: export the new profiles on the tower; check the ground station and detection read 12-bit JPEG XL; a daylight run in auto mode on each tower (verification below); tune settle_wait_ms (500 ms is a guess) and frame_timeout_ms (7 s from the worst measured step).

Goal

The camera sends raw Bayer (BayerRG12p, or BayerRG8 where chosen) at 250 MB/s, so the stream runs 2–3× faster than RGB8 and auto-exposure settles in roughly half the time. The tower unpacks, converts to colour (edge-aware) and saves JPEG XL at the frame's bit depth. A waypoint's frame is kept when the image has settled, not after a fixed wait. Profiles that ask for RGB8 keep working exactly as today.

What the measurements fix

Decision	Value	Why (measured)
Pixel format	from the profile XML: BayerRG12p (proposed) or BayerRG8; RGB8 still accepted	9.86 / 14.75 / 4.95 fps; no losses at 250 MB/s on either tower
Colour conversion	OpenCV edge-aware, 16-bit for 12-bit input	30–35 ms; SSIM 0.973 vs 0.968 bilinear at the same size
Save	colour JXL, distance 1.0, effort 3, 4 threads; 12-bit via JxlEncoderSetFrameBitDepth	2.7 MB/frame; 0.57 s (IOTA), 0.85 s (Delta) on the real frame
Workers	1 × 4 threads on the IOTA, 2 × 2 on the Delta	IOTA keeps up at 0.7 s/frame; Delta at 1.0 s only with 2 workers
Queue	bounded, ~4 frames (24 MB each raw)	unbounded today; pile-up measured at 23–97 frames
Keep rule	brightness in the camera's target band and steady (≤2%) over 3 frames, with a time limit	right in 21 of 24 steps; "exposure unchanged" 0 of 24
Lossless	must set uses_original_profile	known issue #20 (production's distance 0 is lossy)

Processing budget the design must fit

Measured on the identical frame on both computers (5328×3040, real scene; 4 threads; see the report's Section 3.5).

Time per saved frame, step by step

Step	IOTA (121)	3 Delta (111)	Where it runs in the new code
Receive frame (SDK buffer → copy)	1.6 ms (8-bit raw, measured); ~2.4 ms for 12p (estimated from size)	not measured	VimbaCameraSource::FrameReceived
Unpack BayerRG12p → 16-bit	10 ms	17 ms	RawImage::unpack12p (core)
De-Bayer, edge-aware, 16-bit (bilinear)	28 ms (17)	33 ms (22)	ImagePipeline::process (OpenCV)
Rotate (only if rotation_deg ≠ 0)	~30 ms (measured on RGB8 8-bit)	not measured	ImagePipeline::process
JPEG XL encode, 12-bit colour, d1.0 e3	480 ms	765 ms	JpegXlEncoder
Write file (2.7 MB)	1 ms	4 ms	JpegXlEncoder
Total per frame	~520 ms	~820 ms	one save-worker job
Today, for comparison (RGB8 → JXL d0.8 e4)	567 ms	872 ms

Unpacking and colour conversion are ~7% of a save; the encoder is ~90%. Nothing in the new steps needs optimising before the encoder does.

Space

Saved as	Per frame	Per day, one frame / 2 s	/ 5 s	/ 10 s
Today: RGB8 → JXL d0.8 e4 (camera RGB8)	3.3–4.0 MB	141–172 GB	57–69 GB	28–34 GB
12-bit colour JXL d1.0 e3 (proposed)	2.65 MB	114 GB	46 GB	23 GB
12-bit colour JXL d0.5	5.15 MB	222 GB	89 GB	44 GB
12-bit raw planes, lossless (option)	16.8–17.7 MB	727–763 GB	291–305 GB	145–153 GB
12-bit colour, lossless	44.8 MB	1.9 TB	774 GB	387 GB
Working memory per save worker (12-bit colour)	~0.35 GB; 0.7 GB for 1 worker, 1.1 GB for 2 (measured)
Queued raw frame	24.3 MB each

The towers have 11–22 GB free: saved images must leave the tower continuously.

Pace

	IOTA (121)	3 Delta (111)
Camera frame interval at 250 MB/s (12-bit / 8-bit raw / RGB8)	0.10 / 0.07 / 0.20 s	same camera, same
Exposure settle after a large change (12-bit)	~3 s (2.2–6.6)	same camera, same
Saves per second, 1 worker × 4 threads	1.79 (0.56 s each)	1.19 (0.84 s each)
Saves per second, 2 workers × 2 threads	2.15 (0.47 s)	1.53 (0.65 s)
Keeps up with one frame every 0.7 s	yes, 1 worker	only with 2 workers

A waypoint takes ~1.5–5 s (slew + exposure settle), so saving occupies 10–55% of each tower's capacity at scan pace. That is why the design keeps saving in the background, sizes workers per computer, and still bounds the queue for bursts.

Phases

Each phase builds and passes ctest on its own; RGB8 profiles behave as today after every phase.

1. Encoder: correct lossless, 12-bit, configurable threads

src/camera/JpegXlEncoder.cpp, include/fgc/JpegXlEncoder.h

• One entry point taking bits (8 or 12) as well as channels: 16-bit input words when bits > 8, bits_per_sample = bits, and JxlEncoderSetFrameBitDepth(JXL_BIT_DEPTH_FROM_CODESTREAM).
• uses_original_profile = JXL_TRUE when distance == 0 (fixes #20).
• threads from config instead of the hard-coded 3 (3 is slower than 2 on the IOTA).
• The 8-bit lossy path must stay byte-identical to today's output (the benchmark checked its copy of these settings against production).
• Reuse: encodeMem in tools/camera-bench/raw12_bench.cpp is the tested version of all of this.

2. Frames that say what they hold

include/fgc/ICameraSource.h, src/camera/VimbaCameraSource.cpp, include/fgc/mock/MockCameraSource.h

• Frame gains PixelLayout layout (RGB8, Mono8, BayerRG8, BayerRG12p) and the per-frame camera data: exposure (µs), gain (dB), white-balance ratios, camera timestamp, and a brightness mean and clipped share.
• FrameReceived takes the layout from pFrame->GetPixelFormat() instead of channels = sz/(w*h), which gives 1 for 12p (wrong). It reads the chunk data (ChunkExposureTime, ChunkGain, ChunkBalanceRatioRed/Blue) and samples brightness from the buffer (every ~1000th byte: under 1 ms, as sample_brightness in camera_bench.py).
• Optional: keep the newest frame in the SDK buffer instead of copying every frame (measured 1.6–4 ms per frame, 2–4% of a core; low priority).
• MockCameraSource can produce BayerRG8 and BayerRG12p frames, so the pipeline runs without a camera.

3. Raw handling in the core library (testable without OpenCV)

New include/fgc/RawImage.h, src/core/RawImage.cpp (in fgc_core)

• unpack12p(const uint8_t*, uint16_t*, n) (GenICam 12p, two pixels in three bytes).
• BrightnessSample (mean on a 0–255 scale, clipped share) per layout.
• The camera's BayerRG is OpenCV's BayerBG constant; that mapping and the layouts live here.

4. Pipeline: convert, encode, several workers, bounded queue

src/camera/ImagePipeline.cpp, include/fgc/ImagePipeline.h

• process() by layout: RGB8 as today; Bayer → (unpack) → cv::cvtColor(..., COLOR_BayerBG2RGB_EA) in 8 or 16 bit → rotate (16-bit works) → encode with the frame's bit depth.
• Optional save_as = raw_planes: the mosaic rearranged into four colour planes, lossless (17–18 MB/frame; for a detector that wants raw values). [decide] whether to offer it at all.
• N worker threads (save_workers), each encoding with jxl_threads.
• Bounded queue (save_queue_max). [decide] what happens when it is full:
  • backpressure (recommended): ImagePipeline::canAccept(), and CaptureScheduler waits before moving to the next waypoint, so no waypoint is lost and the scan slows down;
  • drop: the frame is discarded with a warning and a counter.
• Pose still sampled at submit().
• Peak memory per worker: ~0.35 GB (16-bit colour), measured 0.7 GB for 1 worker, 1.1 GB for 2.

5. Keep the right frame

include/fgc/CaptureScheduler.h, src/core/CaptureScheduler.cpp, ICameraSource

• After standstill, wait a short mechanical settle (settle_wait_ms, now a minimum; default ~200 ms), then keep the first frame for which the last 3 frames are all in the camera's target band (IntensityControllerTarget ± IntensityControllerTolerance, read from the camera when a profile is applied) and brightness changed ≤2% frame to frame.
• Exposure and gain at their limits with brightness still out of band count as settled (waiting cannot help); log it.
• frame_timeout_ms stays the upper bound; then keep the best frame seen (closest to the band), not nothing.
• Needs the camera source to expose recent frames' data (recentFrames(since) or a callback); the rule itself goes in fgc_core (SettleRule), so it can be tested on recorded series.
• Falls back to today's fixed wait when frames carry no brightness (mock without data, old profile).

6. Configuration

include/fgc/Config.h, src/core/Config.cpp, config/config.example.ini, hardware kit hardware.ini

• [Camera]: jxl_distance = 1.0, jxl_effort = 3 (new defaults, [decide]), jxl_threads, demosaic = ea | bilinear, save_as = colour | raw_planes.
• hardware.ini (what the tower is): save_workers (1 on IOTA, 2 on Delta), jxl_threads (4 / 2), save_queue_max. These depend on the computer, not on what the tower does.
• [Camera.<name>.<profile>]: settle_wait_ms becomes the minimum mechanical settle; add settle_rule = brightness | fixed so a tower can be switched back.
• Validation and tests in tests/test_config.cpp, as for the existing keys.

7. Camera profiles

scripts/export_camera_profile.py (recipes), config/camera_profiles/rgb/README.md

• standard: PixelFormat = BayerRG12p ([decide] vs BayerRG8), binning 1, full sensor, DeviceLinkThroughputLimit = 250000000, AcquisitionFrameRate = max, ChunkModeActive with ExposureTime, Gain, BalanceRatioRed/Blue, Timestamp enabled (enable each item before activating chunk mode), ExposureAutoMax = 20 ms.
• night: BayerRG12p, long exposure allowed, where 12 bits carry real shadow detail.
• White balance: [decide] keep fixed ratios, or BalanceWhiteAuto = Continuous (converges in 2–4 frames, stable within ±3%).
• Profiles stay the only thing that writes camera settings (the existing rule).

8. Downstream and docs

• [decide] The saved files become 12-bit JPEG XL. Anything that reads them (ground station, detection) must decode at 16 bits; check before switching a tower. Consider adding the bit depth to CamEvent (docs/mqtt-api.md).
• docs/architecture.md (Image capture), docs/configuration.md (new keys), docs/known-issues.md (#16 settle, #20 lossless), profile README.

Tests

• fgc_core unit tests: unpack12p against hand-packed bytes; brightness sampling per layout; the settle rule replayed on the 24 recorded auto-exposure series in tools/camera-bench/results-*/ae.jsonl (it must pick a correct frame in ≥ 21 of 24); config keys; scheduler backpressure with the mock pipeline.
• A small separate test target linking OpenCV + libjxl (fgc_pipeline_tests): 12-bit lossless round trip is exact; distance-0 8-bit is exact (#20); de-Bayer colour order matches a known patch; 8-bit lossy output is byte-identical to the old encoder for a fixed input.
• Mock run: the program in auto mode with MockCameraSource producing BayerRG12p.

Verification on the towers

1. Deploy to 121 with the new standard profile, run auto mode for 30 min in daylight: per-waypoint log (move, standstill, settle-rule wait, save time), queue depth, CPU temperature.
2. Same on 111 with save_workers = 2.
3. Compare with the benchmark: save time per frame within ~10% of raw12_bench pipeline, no backlog, temperatures as measured (53 °C IOTA, 66 °C Delta at one frame per 2 s).
4. Check saved files decode at 12 bits and match the camera's colour (channel means).
5. One dusk or night run to confirm the 12-bit shadow benefit before deciding the night profile.

Order and size

Phase	Depends on	Rough size
1 Encoder	–	small
2 Frames	–	medium
3 Raw core	–	small
4 Pipeline	1, 2, 3	medium
5 Keep rule	2, 3	medium
6 Config	4, 5	small
7 Profiles	2	small (plus a tower session per profile)
8 Downstream, docs	all	small; downstream check is outside this repo

Phases 1–3 are independent and can go first; 4 and 5 can then proceed in parallel.