GGS
/
fwt_software
8
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 1 Wiki Activity
fwt_software/docs/raw12-pipeline-implementati...

12 KiB
Raw Blame History

Implementation plan — raw capture, colour on the tower, 12-bit JPEG XL

Based on raw12-pipeline-measurements.md (towers 121 and 111, 2026-10-02/04). Not started; decisions marked [decide] are the user's.

Goal

The camera sends raw Bayer (BayerRG12p, or BayerRG8 where chosen) at 250 MB/s, so the stream runs 2–3× faster than RGB8 and auto-exposure settles in roughly half the time. The tower unpacks, converts to colour (edge-aware) and saves JPEG XL at the frame's bit depth. A waypoint's frame is kept when the image has settled, not after a fixed wait. Profiles that ask for RGB8 keep working exactly as today.

What the measurements fix

Decision Value Why (measured)
Pixel format from the profile XML: BayerRG12p (proposed) or BayerRG8; RGB8 still accepted 9.86 / 14.75 / 4.95 fps; no losses at 250 MB/s on either tower
Colour conversion OpenCV edge-aware, 16-bit for 12-bit input 30–35 ms; SSIM 0.973 vs 0.968 bilinear at the same size
Save colour JXL, distance 1.0, effort 3, 4 threads; 12-bit via JxlEncoderSetFrameBitDepth 2.7 MB/frame; 0.57 s (IOTA), 0.85 s (Delta) on the real frame
Workers 1 × 4 threads on the IOTA, 2 × 2 on the Delta IOTA keeps up at 0.7 s/frame; Delta at 1.0 s only with 2 workers
Queue bounded, ~4 frames (24 MB each raw) unbounded today; pile-up measured at 23–97 frames
Keep rule brightness in the camera's target band and steady (≤2%) over 3 frames, with a time limit right in 21 of 24 steps; "exposure unchanged" 0 of 24
Lossless must set uses_original_profile known issue #20 (production's distance 0 is lossy)

Processing budget the design must fit

Measured on the identical frame on both computers (5328×3040, real scene; 4 threads; see the report's Where the time goes).

Time per saved frame, step by step

Step IOTA (121) 3 Delta (111) Where it runs in the new code
Receive frame (SDK buffer → copy) 1.6 ms (8-bit raw, measured); ~2.4 ms for 12p (estimated from size) not measured VimbaCameraSource::FrameReceived
Unpack BayerRG12p → 16-bit 10 ms 17 ms RawImage::unpack12p (core)
De-Bayer, edge-aware, 16-bit (bilinear) 28 ms (17) 33 ms (22) ImagePipeline::process (OpenCV)
Rotate (only if rotation_deg ≠ 0) ~30 ms (measured on RGB8 8-bit) not measured ImagePipeline::process
JPEG XL encode, 12-bit colour, d1.0 e3 480 ms 765 ms JpegXlEncoder
Write file (2.7 MB) 1 ms 4 ms JpegXlEncoder
Total per frame ~520 ms ~820 ms one save-worker job
Today, for comparison (RGB8 → JXL d0.8 e4) 567 ms 872 ms

Unpacking and colour conversion are ~7% of a save; the encoder is ~90%. Nothing in the new steps needs optimising before the encoder does.

Space

Saved as Per frame Per day, one frame / 2 s / 5 s / 10 s
Today: RGB8 → JXL d0.8 e4 (camera RGB8) 3.3–4.0 MB 141–172 GB 57–69 GB 28–34 GB
12-bit colour JXL d1.0 e3 (proposed) 2.65 MB 114 GB 46 GB 23 GB
12-bit colour JXL d0.5 5.15 MB 222 GB 89 GB 44 GB
12-bit raw planes, lossless (option) 16.8–17.7 MB 727–763 GB 291–305 GB 145–153 GB
12-bit colour, lossless 44.8 MB 1.9 TB 774 GB 387 GB
Working memory per save worker (12-bit colour) ~0.35 GB; 0.7 GB for 1 worker, 1.1 GB for 2 (measured)
Queued raw frame 24.3 MB each

The towers have 11–22 GB free: saved images must leave the tower continuously.

Pace

IOTA (121) 3 Delta (111)
Camera frame interval at 250 MB/s (12-bit / 8-bit raw / RGB8) 0.10 / 0.07 / 0.20 s same camera, same
Exposure settle after a large change (12-bit) ~3 s (2.2–6.6) same camera, same
Saves per second, 1 worker × 4 threads 1.79 (0.56 s each) 1.19 (0.84 s each)
Saves per second, 2 workers × 2 threads 2.15 (0.47 s) 1.53 (0.65 s)
Keeps up with one frame every 0.7 s yes, 1 worker only with 2 workers

A waypoint takes ~1.5–5 s (slew + exposure settle), so saving occupies 10–55% of each tower's capacity at scan pace. That is why the design keeps saving in the background, sizes workers per computer, and still bounds the queue for bursts.

Phases

Each phase builds and passes ctest on its own; RGB8 profiles behave as today after every phase.

1. Encoder: correct lossless, 12-bit, configurable threads

src/camera/JpegXlEncoder.cpp, include/fgc/JpegXlEncoder.h

  • One entry point taking bits (8 or 12) as well as channels: 16-bit input words when bits > 8, bits_per_sample = bits, and JxlEncoderSetFrameBitDepth(JXL_BIT_DEPTH_FROM_CODESTREAM).
  • uses_original_profile = JXL_TRUE when distance == 0 (fixes #20).
  • threads from config instead of the hard-coded 3 (3 is slower than 2 on the IOTA).
  • The 8-bit lossy path must stay byte-identical to today's output (the benchmark checked its copy of these settings against production).
  • Reuse: encodeMem in tools/camera-bench/raw12_bench.cpp is the tested version of all of this.

2. Frames that say what they hold

include/fgc/ICameraSource.h, src/camera/VimbaCameraSource.cpp, include/fgc/mock/MockCameraSource.h

  • Frame gains PixelLayout layout (RGB8, Mono8, BayerRG8, BayerRG12p) and the per-frame camera data: exposure (µs), gain (dB), white-balance ratios, camera timestamp, and a brightness mean and clipped share.
  • FrameReceived takes the layout from pFrame->GetPixelFormat() instead of channels = sz/(w*h), which gives 1 for 12p (wrong). It reads the chunk data (ChunkExposureTime, ChunkGain, ChunkBalanceRatioRed/Blue) and samples brightness from the buffer (every ~1000th byte: under 1 ms, as sample_brightness in camera_bench.py).
  • Optional: keep the newest frame in the SDK buffer instead of copying every frame (measured 1.6–4 ms per frame, 2–4% of a core; low priority).
  • MockCameraSource can produce BayerRG8 and BayerRG12p frames, so the pipeline runs without a camera.

3. Raw handling in the core library (testable without OpenCV)

New include/fgc/RawImage.h, src/core/RawImage.cpp (in fgc_core)

  • unpack12p(const uint8_t*, uint16_t*, n) (GenICam 12p, two pixels in three bytes).
  • BrightnessSample (mean on a 0–255 scale, clipped share) per layout.
  • The camera's BayerRG is OpenCV's BayerBG constant; that mapping and the layouts live here.

4. Pipeline: convert, encode, several workers, bounded queue

src/camera/ImagePipeline.cpp, include/fgc/ImagePipeline.h

  • process() by layout: RGB8 as today; Bayer → (unpack) → cv::cvtColor(..., COLOR_BayerBG2RGB_EA) in 8 or 16 bit → rotate (16-bit works) → encode with the frame's bit depth.
  • Optional save_as = raw_planes: the mosaic rearranged into four colour planes, lossless (17–18 MB/frame; for a detector that wants raw values). [decide] whether to offer it at all.
  • N worker threads (save_workers), each encoding with jxl_threads.
  • Bounded queue (save_queue_max). [decide] what happens when it is full:
    • backpressure (recommended): ImagePipeline::canAccept(), and CaptureScheduler waits before moving to the next waypoint, so no waypoint is lost and the scan slows down;
    • drop: the frame is discarded with a warning and a counter.
  • Pose still sampled at submit().
  • Peak memory per worker: ~0.35 GB (16-bit colour), measured 0.7 GB for 1 worker, 1.1 GB for 2.

5. Keep the right frame

include/fgc/CaptureScheduler.h, src/core/CaptureScheduler.cpp, ICameraSource

  • After standstill, wait a short mechanical settle (settle_wait_ms, now a minimum; default ~200 ms), then keep the first frame for which the last 3 frames are all in the camera's target band (IntensityControllerTarget ± IntensityControllerTolerance, read from the camera when a profile is applied) and brightness changed ≤2% frame to frame.
  • Exposure and gain at their limits with brightness still out of band count as settled (waiting cannot help); log it.
  • frame_timeout_ms stays the upper bound; then keep the best frame seen (closest to the band), not nothing.
  • Needs the camera source to expose recent frames' data (recentFrames(since) or a callback); the rule itself goes in fgc_core (SettleRule), so it can be tested on recorded series.
  • Falls back to today's fixed wait when frames carry no brightness (mock without data, old profile).

6. Configuration

include/fgc/Config.h, src/core/Config.cpp, config/config.example.ini, hardware kit hardware.ini

  • [Camera]: jxl_distance = 1.0, jxl_effort = 3 (new defaults, [decide]), jxl_threads, demosaic = ea | bilinear, save_as = colour | raw_planes.
  • hardware.ini (what the tower is): save_workers (1 on IOTA, 2 on Delta), jxl_threads (4 / 2), save_queue_max. These depend on the computer, not on what the tower does.
  • [Camera.<name>.<profile>]: settle_wait_ms becomes the minimum mechanical settle; add settle_rule = brightness | fixed so a tower can be switched back.
  • Validation and tests in tests/test_config.cpp, as for the existing keys.

7. Camera profiles

scripts/export_camera_profile.py (recipes), config/camera_profiles/rgb/README.md

  • standard: PixelFormat = BayerRG12p ([decide] vs BayerRG8), binning 1, full sensor, DeviceLinkThroughputLimit = 250000000, AcquisitionFrameRate = max, ChunkModeActive with ExposureTime, Gain, BalanceRatioRed/Blue, Timestamp enabled (enable each item before activating chunk mode), ExposureAutoMax = 20 ms.
  • night: BayerRG12p, long exposure allowed, where 12 bits carry real shadow detail.
  • White balance: [decide] keep fixed ratios, or BalanceWhiteAuto = Continuous (converges in 2–4 frames, stable within ±3%).
  • Profiles stay the only thing that writes camera settings (the existing rule).

8. Downstream and docs

  • [decide] The saved files become 12-bit JPEG XL. Anything that reads them (ground station, detection) must decode at 16 bits; check before switching a tower. Consider adding the bit depth to CamEvent (docs/mqtt-api.md).
  • docs/architecture.md (Image capture), docs/configuration.md (new keys), docs/known-issues.md (#16 settle, #20 lossless), profile README.

Tests

  • fgc_core unit tests: unpack12p against hand-packed bytes; brightness sampling per layout; the settle rule replayed on the 24 recorded auto-exposure series in tools/camera-bench/results-*/ae.jsonl (it must pick a correct frame in ≥ 21 of 24); config keys; scheduler backpressure with the mock pipeline.
  • A small separate test target linking OpenCV + libjxl (fgc_pipeline_tests): 12-bit lossless round trip is exact; distance-0 8-bit is exact (#20); de-Bayer colour order matches a known patch; 8-bit lossy output is byte-identical to the old encoder for a fixed input.
  • Mock run: the program in auto mode with MockCameraSource producing BayerRG12p.

Verification on the towers

  1. Deploy to 121 with the new standard profile, run auto mode for 30 min in daylight: per-waypoint log (move, standstill, settle-rule wait, save time), queue depth, CPU temperature.
  2. Same on 111 with save_workers = 2.
  3. Compare with the benchmark: save time per frame within ~10% of raw12_bench pipeline, no backlog, temperatures as measured (53 °C IOTA, 66 °C Delta at one frame per 2 s).
  4. Check saved files decode at 12 bits and match the camera's colour (channel means).
  5. One dusk or night run to confirm the 12-bit shadow benefit before deciding the night profile.

Order and size

Phase Depends on Rough size
1 Encoder – small
2 Frames – medium
3 Raw core – small
4 Pipeline 1, 2, 3 medium
5 Keep rule 2, 3 medium
6 Config 4, 5 small
7 Profiles 2 small (plus a tower session per profile)
8 Downstream, docs all small; downstream check is outside this repo

Phases 1–3 are independent and can go first; 4 and 5 can then proceed in parallel.