fwt_software/docs/raw12-pipeline-measurements_llm_context.md

Context for discussing raw12-pipeline-measurements.md

This file gives an AI assistant the background needed to discuss the report raw12-pipeline-measurements.md: what was asked, how the tests were run, what the numbers mean, what was decided, which points have been questioned, and what is still open. Read the report itself for the full tables and figures; this file explains around them. State as of 2026-10-07, branch raw12-capture (dark-scene comparison added 2026-10-07).

1. The system in one paragraph

Fire-watch towers scan waypoints with a gimbal-mounted Allied Vision Alvium camera and keep one picture per waypoint, saved as JPEG XL on the tower's computer. After the gimbal stops, the camera's continuous auto-exposure needs frames to adapt to the new view, so the scan waits ("settling"). The tower program is C++ (fgc), using Vimba X (VmbCPP), OpenCV and libjxl. Two computers are in use: LattePanda IOTA (Intel N150, tower 121) and LattePanda 3 Delta (Celeron N5105, tower 111).

2. The question and how it evolved

• Origin: speeding up the per-waypoint capture. Today the camera sends RGB8 (colour made in the camera, 48.6 MB per full frame), which limits the frame rate over USB to 4.95 fps at the agreed 250 MB/s link cap, so exposure settles slowly.
• Proposal tested: camera sends raw BayerRG12p (24.3 MB, 9.86 fps), the tower unpacks, de-Bayers (converts to colour) and saves 12-bit colour JPEG XL.
• An earlier campaign (camera-capture-measurements.md) used blurred dim scenes; its compression numbers were unrepresentative. This campaign used a focused, real scene (spruce, birch, leaves, building, ~13% clipped sky).
• The report went through several rounds of requested edits: example images per compression type, methodology detail (how settings were forced), processing on one identical frame on both towers, an abstract, a pixel-format description, an academic structure (numbered sections, figure and table captions, units in every header, references), captions that explain every term, an explanation of raw mosaic vs raw planes, a more careful treatment of what 12 bits add, and a fuller explanation of the keep rules.

3. User decisions (fixed constraints, do not re-litigate)

• Full resolution always; no binning; one quality setting (no switching between a fast low quality mode and a slow high quality one); continuous acquisition.
• No exposure memory across passes or from neighbouring waypoints.
• USB link cap 250 MB/s (DeviceLinkThroughputLimit), conservative.
• BayerRG12p everywhere, day and night profiles.
• Save 12-bit colour JPEG XL only; no raw-planes option in the program for now.
• Fixed white balance ratios kept in the profiles (auto white balance was measured but not adopted).
• When saving falls behind, pause the scan (save gate) rather than drop frames.
• Writing style for the report: academic structure, captions that explain their terms, units given wherever possible, plain language. Commit messages: one plain-language line, no trailers.

4. Test setup essentials

• Same camera for both towers: Alvium 1800 U-1620c DEV_1AB22C0B7D4E, 5328 × 3040, firmware 14.3. Moved from 121 to 111 so only the computer differs. Tower 111's own camera (1800 U-1240c, 12 MP, 4024 × 3036) was not tested; its frames are 25% smaller, so the Delta results are conservative.
• Dates: tower 121 on 2 Oct 2026 (sunny afternoon); tower 111 on 4 Oct (afternoon into evening); shared-frame processing on both towers simultaneously on 4 Oct 23:03–23:32.
• The tower program was stopped during all tests.
• Storage write speed: IOTA 218 MB/s, Delta 76 MB/s (eMMC). Libraries: 121 libjxl 0.11.1 / OpenCV 4.10; 111 libjxl 0.11.0 / OpenCV 4.5.4.
• Tools in tools/camera-bench/ (camera_bench.py with vmbpy; raw12_bench.cpp; run scripts; plot_results.py, plot_common.py, make_figures.py, compare_examples.py). Results in results-2026-10-02-tower121/, results-2026-10-04-tower111/, results-common-tower121/111/. Full-size example .jxl files in tools/camera-bench/examples-* (git-ignored, ~115 MB per set). The recorded auto-exposure steps are also in tests/data/ae_steps.csv (used by a unit test).

5. Methods that are easy to misunderstand

• Forcing settings: feature writes over USB while streaming. ExposureAuto/GainAuto set to Off freezes current values, then ExposureTime/Gain are written; for the 2 s "force" phase the camera delivers 10–30 frames at the forced values; then auto is set back to Continuous (time 0). Dark step: exposure E₀/16 with 0 dB gain. Bright step: exposure at the 20 ms cap plus gain up to +24.1 dB (×16). Bright start is fully clipped (97–100%), so the camera cannot see how far over it is.
• Brightness = mean of a sparse sample of the frame buffer on 0–255 (12-bit: high byte). Settled = stays within ±10% of the median brightness of the run's last 2 s.
• Frames in transit: the first 1–5 frames after release still carry the forced values.
• Shared frame: tower 121's real frame rebuilt bit-exactly from its lossless 12-bit file and copied to both towers (identical MD5). Its 8-bit raw version is the 12-bit value >> 4; its "RGB8" version is the tower's own edge-aware conversion scaled to 8 bits, so on the shared frame the RGB8 file size equals raw 8-bit's (2.66 MB). The camera's real RGB8 files were 3.7–4.0 MB.
• Synthetic frame: "dead leaves" texture plus noise (ISO 19567-2 style), sharper than any real scene; worst case for the encoder; used for pile-up and heat.
• Quality is against the exact lossless 12-bit colour image of the same frame: PSNR on the 12-bit scale over 3 channels; SSIM on luma. JPEG XL is perceptual, so PSNR is moderate while SSIM is high.
• Camera BayerRG = OpenCV COLOR_BayerBG2RGB (verified). The camera applies white balance to raw.
• 12-bit JPEG XL from 16-bit words needs bit depth "from codestream", or values get rescaled; lossless needs the original colour profile. An earlier mistake on both points was found and the 12-bit numbers corrected before the final report.

6. Key results (numbers to quote)

Frame rate at 250 MB/s (fps): RGB8 4.95, BayerRG8 14.75, BayerRG10p 11.81, BayerRG12p 9.86, BayerRG12 7.41. 60 s streams: zero lost or incomplete frames in every format on both towers, also while saving continuously. On 111 the camera shares the USB 3 controller with a 5G modem, a USB network adapter and the idle second camera.

Auto-exposure settling after ×16 steps: ~25–35 frames, so time scales with 1/frame rate. BayerRG8 1.8–2.7 s; BayerRG12p 2.2–6.6 s (typically ≈ 3 s; long runs contain mid-way pauses); RGB8 4.6–6.6 s. With exposure at its 20 ms cap the camera adjusts gain by 1–2 dB per frame and sometimes pauses up to 2.8 s while still far too bright.

Keep rules (24 runs = 3 formats × 2 directions × 2 repeats × 2 towers; good = within ±10% of final level):

• "Exposure and gain unchanged 3 frames": 0/24. Replay detail (added 2026-10-05 from tests/data/ae_steps.csv): 16 runs fired on the first frames (0.2–0.5 s, frames in transit, image black 9–13 or white 253–255 vs ≈ 120); 8 fired during mid-way pauses at brightness 33–227.
• "In band 2 frames": 12/24 (passes through the band ~15% above final level when descending).
• "In band and steady (≤ 2%/frame) 3 frames": 21/24 (3 slightly early on descents); median time when correct 3.61 s (121) / 5.45 s (111). Implemented as settle_rule = brightness.
• Band = IntensityControllerTarget ± Tolerance = 50 ± 5% → 114.75–140.25 on 0–255.

Auto white balance: recovers in 2–4 frames (0.2–0.8 s), stable ±0.8–3.1%. Exposure, not white balance, sets the wait. Auto ratios ≈ red 2.36–2.46, blue 1.87–1.97 vs fixed profile 2.87 / 2.0.

Processing on the shared frame (4 threads; IOTA / Delta, ms): unpack 10 / 17; de-Bayer 12-bit bilinear 17 / 22, edge-aware 28 / 33; encode 12-bit d1.0 e3 480 / 765; today's RGB8 encode d0.8 e4 566 / 867. Totals: 12-bit edge-aware d1.0 e3 518 / 820 ms, 2.65 MB; today RGB8 567 / 872 ms, 3.27 MB; raw 8-bit → colour 468 / 741 ms, 2.66 MB. Delta ≈ 1.6× slower. Unpack + de-Bayer = 7% of a save; encode ≈ 90%. Effort 1–4 within ~15%; effort 3 smallest; 4 threads fastest; on the IOTA 3 threads is oddly slower than 2.

Distance sweep (12-bit colour, shared frame, MB): lossless 44.8, d0.3 7.19, d0.5 5.15, d0.8 3.16, d1.0 2.65, d1.5 1.63, d2.0 1.32. Quality at d1.0: PSNR 36.6 dB, SSIM 0.973. Raw planes d0.5: 6.0 MB, PSNR 47.2 dB. Lossless: raw planes 17.6, raw mosaic 20.9, colour 47.4 MB.

Throughput (saves/s back to back, shared frame): 1 worker × 4 threads IOTA 1.79, Delta 1.19; 2 × 2: 2.15 / 1.53. On the shared frame the IOTA keeps up with one frame per 0.7 s with one worker; the Delta needs two. Pile-up details on the synthetic frame are in Table 18.

Heat (10 min each): one frame every 2 s → IOTA 53 °C mean / 5.1 W, Delta 66 °C / 5.7 W. Back to back (2 × 2): IOTA 80 °C, 12.5 W, loses 16% throughput as it heats; Delta 78 °C, at its 8 W limit, steady.

Storage: 12-bit colour d1.0 ≈ 2.65–2.75 MB/frame → 114–119 GB/day at one frame every 2 s. Towers have 11–22 GB free, so images must leave the tower continuously.

7. The 12-bit question (discussed at length; current position)

• Noise floor (12-bit DN, green pixels): at ≈ 700 DN 26 (0 dB) / 42 (15 dB); at ≈ 1500 35 / 58; at ≈ 3000 37 / 80. One 8-bit step = 16 DN; rounding error RMS = 16/√12 = 4.6 DN.
• Errors add in quadrature: √(26² + 4.6²) = 26.4 → rounding raises noise by at most 1.5% in the measured daylight scene (0.6% at 15 dB, < 0.2% in bright areas).
• Averaging: while noise > one 8-bit step, rounding error is random (dither) and averages down at the same √N rate as noise, so 8-bit averages resolve sub-step differences as well as 12-bit. A simulation (not in the report, by user request) showed a 3 DN difference over 100 × 100 pixels recovered exactly by both at noise 26, but only 0.2 of 3 by 8-bit at noise 2.
• 12 bits always contain at least as much information; at noise 26 the dropped bits carry about 0.02 bits/pixel of scene information.
• Compression dominates: PSNR 36.6 dB at d1.0 ≈ 61 DN RMS error (13× the rounding error); d0.3 ≈ 40 DN; raw planes d0.5 ≈ 18 DN. Much of that is discarded noise, and JPEG XL keeps relatively more precision in dark tones; shadow-only error was not measured.
• Where 12 bits should matter (Table 17b): dark areas at low gain where noise < one 8-bit step (below ≈ 300 DN at 0 dB, ≈ 100 DN at 15 dB), i.e. high-contrast scenes such as shaded forest or backlit ridges under a bright sky at dawn or dusk; thin smoke over dark terrain is the relevant fire-detection case. At night with high gain, noise is amplified above the step again, so little benefit. Strong shadow lifting (≥ 3 stops) untested; 2 stops showed no difference.
• The October scene had nothing darker than ≈ 700 DN (a dusk attempt on 121 needed 24 dB and was discarded). On 7 Oct the user took a dark, high-contrast pair at the Stäffelsberg tower (shaded forest under hazy sky, same 1620c model): BayerRG12p converted to colour by the camera viewer and saved as 16-bit TIFF (12-bit values << 4), and a separate RGB8 capture as PNG; files in ../8bit_vs_12bit_test/ (outside the repo). Gain/exposure not recorded. Report Section 3.9.2, script tools/camera-bench/compare_8_vs_12.py, results.json in docs/images/raw12/examples/staeffelsberg/.
• Stäffelsberg findings: 33% of the frame < 300 DN; noise in the colour image (std around a local mean, quietest 10% of 24 × 24 patches, calibrated) 5–15 DN below ≈ 1300 DN, i.e. below one 8-bit step over ~70% of the frame. Rounding to 8 bits: +10–26% noise in the forest; dark crop 272 vs 25 green levels (camera RGB8 22); local-average error (Gaussian σ 4 px) only 0.6 DN; no visible banding at +4 stops, 8-bit slightly grainier. Camera RGB8 1.4–1.8× noisier than the 12-bit colour image (camera's own processing).
• Compression is where it shows: at d1.0 the 8-bit file has 3.7 DN local-average error in the dark vs 1.6 DN for 12-bit (bright: 4.0 vs 6.0), visibly blocky shadows; today's RGB8 d0.8 e4 file (0.92 MB) is equally blocky. The same 8-bit values stored in a 12-bit file give 1.8 DN and 0.94 MB: the effect is libjxl spending bits by declared bit depth, not the four extra bits. Sizes on this scene at d1.0: 12-bit 0.92 MB, 8-bit 0.63 MB (+46%; equal in daylight). Implication: if BayerRG8 were ever used, save its colour as a 12/16-bit JPEG XL.
• Same setting for both (user request, Table 17e, Figures 13g/13h): camera RGB8 and BayerRG12p both at d0.8 e4, each scored against its own uncompressed frame (separate captures). As stored (linear, near black) no visible difference; at +4 stops RGB8 is blocky, 12-bit keeps detail. Local-average error dark/mid/bright: RGB8 2.2/2.3/2.8, 12-bit 1.2/1.9/4.4 DN; sizes 0.92 vs 1.22 MB (+33%).
• Not yet tested: real smoke; night at high gain; the tower's own de-Bayer on a dark frame.
• 12-bit vs 8-bit raw costs little in saving (same size, ~10% more time) but the camera runs at 9.86 vs 14.75 fps, ~1 s slower settling in daylight. The main gain over today is frame rate, not bit depth.

8. Limitations to keep in mind

One scene, two afternoons, main runs at ≈ 15 dB gain plus one 0 dB frame (124 ms exposure, cap raised to 200 ms); no night data. Steps forced on a still camera, not real slews; real waypoint-to-waypoint changes are usually smaller. Tower program stopped, so its own load is not included. Detection quality not measured (PSNR/SSIM stand in). Slew time (≈ 1–1.5 s) and mechanical settling (≈ 0.2 s) in the pace table are estimates.

9. Implementation status (branch raw12-capture)

Implemented and unit-tested (457 tests pass locally): 12-bit JPEG XL encoding with the bit-depth fix and true lossless (known issue #20 fixed); frames carry pixel layout, chunk exposure/gain/white balance and a brightness sample; ImagePipeline unpacks and de-Bayers (edge-aware default), N workers, bounded queue; scheduler save gate pauses the scan when the queue is full; SettleRule (settle_rule = brightness) with a time limit and an "at limit" exit; config keys save_workers, jxl_threads, save_queue_max, demosaic, settle_rule; profile export script recipe for raw standard and night profiles. Plan: raw12-pipeline-implementation-plan.md.

Not done yet: exporting the new camera profiles on a tower (the 1620c is currently on tower 111 and must be moved back to 121 first); field verification with real slews; checking that downstream consumers decode 12-bit files. The branch is deployed on tower 111, whose boot service still passes old --init/--start flags and needs scripts/install-tower.

10. Report structure (for orientation)

Abstract; Units and abbreviations; 1 Introduction (1.1 question, 1.2 pixel formats incl. raw mosaic vs raw planes, Figures 1a/1b, Table 1; 1.3 earlier work); 2 Materials and methods (2.1–2.12, Tables 2–8, Figure 2); 3 Results (3.1 frame rate, 3.2 auto-exposure, 3.3 keep rules, 3.4 white balance, 3.5 processing per step, 3.6 effort and threads, 3.7 quality setting, 3.8 example images, 3.9 information content of 12-bit data (3.9.1 daylight, 3.9.2 Stäffelsberg dark scene, 3.9.3 summary; Figures 12, 13a–13h, Tables 17a–17e), 3.10 throughput, 3.11 heat, 3.12 storage; Figures 3–17, Tables 9–20); 4 Discussion (summary Table 21, pace Table 22, limitations); 5 Conclusions and six recommendations; References [1]–[7]; Appendix A raw data and tools; Appendix B per-tower processing times. Figures live in docs/images/raw12/.