From 2e6609eb1d9eb43cf913ebab971072f5d730f72f Mon Sep 17 00:00:00 2001 From: pgdalmeida Date: Mon, 5 Oct 2026 12:25:05 +0200 Subject: [PATCH] Explain in the 12-bit report why the brightness rule picks the right picture, and add a background file for discussing the report with Claude --- docs/raw12-pipeline-measurements.md | 31 ++- ...raw12-pipeline-measurements_llm_context.md | 193 ++++++++++++++++++ 2 files changed, 221 insertions(+), 3 deletions(-) create mode 100644 docs/raw12-pipeline-measurements_llm_context.md diff --git a/docs/raw12-pipeline-measurements.md b/docs/raw12-pipeline-measurements.md index a9ac66c..05e4965 100644 --- a/docs/raw12-pipeline-measurements.md +++ b/docs/raw12-pipeline-measurements.md @@ -402,6 +402,11 @@ Settling typically takes 25–35 frames, so **settling time scales with 1/frame ### 3.3 Keep rules +When the gimbal stops at a new waypoint the view changes, often to a much brighter or darker one, +and auto-exposure needs a few seconds to adapt; frames taken in that time are too dark or too bright. +The program must recognise when the image has settled and keep that frame. Each candidate rule was +replayed on the recordings of the 24 forced steps (Section 2.6) to find the frame it would have kept. + ***Table 11.** Candidate keep rules scored on the 24 recorded steps (12 per tower). A keep rule decides which frame after a gimbal stop is saved. Correct: the kept frame's brightness is within ±10% of the final level; early: the rule kept a frame before that. Target band: brightness between @@ -417,9 +422,29 @@ kept frame, over the correct runs.* | Target band 3 frames | 8 / 4 | 8 / 4 | 3.40 / 4.84 | | **Target band and brightness steady (≤ 2% change) over 3 frames** | **11 / 1** | **10 / 2** | 3.61 / 5.45 | -Descending from over-exposure, brightness enters the band about 15% above its final level, so "in -band" alone fires early; also requiring the brightness to have stopped changing corrects most of -these cases. +**Exposure and gain unchanged.** This rule assumes that once the camera stops adjusting, the image +is right. It kept a frame early in every run, for two reasons: + +- In 16 of 24 runs it fired on the first frames after release (0.2–0.5 s). These frames were already + on their way from the camera and still carried the forced settings, so the settings looked + unchanged while the image was black (brightness 9–13 DN) or white (253–255 DN), against a settled + level of about 120 DN. At a real waypoint the same happens: the settings have not changed yet + because the camera has not reacted to the new view. +- In the other 8 runs it fired during a pause: auto-exposure adjusts in steps and sometimes holds + still for several frames mid-way, here at brightness 33–227 DN. + +Settings that hold still therefore do not show that the image is right. + +**Brightness in the target band and steady.** This rule judges the image itself: it keeps a frame +when, for 3 frames in a row, the brightness is within the band the camera's auto-exposure aims for +and changes by no more than 2% between frames. Both conditions are needed. Descending from +over-exposure, brightness passes through the band about 15% above its final level, so "in band" +alone fired early in half the runs; "steady" alone would fire during the pauses. Combined, the rule +kept a good frame in 21 of 24 runs; the 3 failures were slightly early on a descent. The program +implements this rule as `settle_rule = brightness`, with a time limit. + +These were ×16 steps forced on a still camera. Changes between neighbouring waypoints are usually +smaller, so the rule remains to be confirmed on a real scan. ### 3.4 Auto white balance diff --git a/docs/raw12-pipeline-measurements_llm_context.md b/docs/raw12-pipeline-measurements_llm_context.md new file mode 100644 index 0000000..c644f61 --- /dev/null +++ b/docs/raw12-pipeline-measurements_llm_context.md @@ -0,0 +1,193 @@ +# Context for discussing `raw12-pipeline-measurements.md` + +This file gives an AI assistant the background needed to discuss the report +[raw12-pipeline-measurements.md](raw12-pipeline-measurements.md): what was asked, how the tests were +run, what the numbers mean, what was decided, which points have been questioned, and what is still +open. Read the report itself for the full tables and figures; this file explains around them. +State as of 2026-10-05, branch `raw12-capture`. + +## 1. The system in one paragraph + +Fire-watch towers scan waypoints with a gimbal-mounted Allied Vision Alvium camera and keep one +picture per waypoint, saved as JPEG XL on the tower's computer. After the gimbal stops, the camera's +continuous auto-exposure needs frames to adapt to the new view, so the scan waits ("settling"). The +tower program is C++ (`fgc`), using Vimba X (VmbCPP), OpenCV and libjxl. Two computers are in use: +LattePanda IOTA (Intel N150, tower 121) and LattePanda 3 Delta (Celeron N5105, tower 111). + +## 2. The question and how it evolved + +- Origin: speeding up the per-waypoint capture. Today the camera sends RGB8 (colour made in the + camera, 48.6 MB per full frame), which limits the frame rate over USB to 4.95 fps at the agreed + 250 MB/s link cap, so exposure settles slowly. +- Proposal tested: camera sends raw `BayerRG12p` (24.3 MB, 9.86 fps), the tower unpacks, + de-Bayers (converts to colour) and saves 12-bit colour JPEG XL. +- An earlier campaign ([camera-capture-measurements.md](camera-capture-measurements.md)) used + blurred dim scenes; its compression numbers were unrepresentative. This campaign used a focused, + real scene (spruce, birch, leaves, building, ~13% clipped sky). +- The report went through several rounds of requested edits: example images per compression type, + methodology detail (how settings were forced), processing on one identical frame on both towers, + an abstract, a pixel-format description, an academic structure (numbered sections, figure and + table captions, units in every header, references), captions that explain every term, an + explanation of raw mosaic vs raw planes, a more careful treatment of what 12 bits add, and a fuller + explanation of the keep rules. + +## 3. User decisions (fixed constraints, do not re-litigate) + +- Full resolution always; **no binning**; **one quality setting** (no switching between a fast low + quality mode and a slow high quality one); continuous acquisition. +- **No exposure memory** across passes or from neighbouring waypoints. +- USB link cap **250 MB/s** (`DeviceLinkThroughputLimit`), conservative. +- **`BayerRG12p` everywhere**, day and night profiles. +- Save **12-bit colour JPEG XL only**; no raw-planes option in the program for now. +- **Fixed white balance** ratios kept in the profiles (auto white balance was measured but not adopted). +- When saving falls behind, **pause the scan** (save gate) rather than drop frames. +- Writing style for the report: academic structure, captions that explain their terms, units given + wherever possible, plain language. Commit messages: one plain-language line, no trailers. + +## 4. Test setup essentials + +- Same camera for both towers: Alvium 1800 U-1620c `DEV_1AB22C0B7D4E`, 5328 × 3040, firmware 14.3. + Moved from 121 to 111 so only the computer differs. Tower 111's own camera (1800 U-1240c, 12 MP, + 4024 × 3036) was not tested; its frames are 25% smaller, so the Delta results are conservative. +- Dates: tower 121 on 2 Oct 2026 (sunny afternoon); tower 111 on 4 Oct (afternoon into evening); + shared-frame processing on both towers simultaneously on 4 Oct 23:03–23:32. +- The tower program was stopped during all tests. +- Storage write speed: IOTA 218 MB/s, Delta 76 MB/s (eMMC). Libraries: 121 libjxl 0.11.1 / + OpenCV 4.10; 111 libjxl 0.11.0 / OpenCV 4.5.4. +- Tools in `tools/camera-bench/` (camera_bench.py with vmbpy; raw12_bench.cpp; run scripts; + plot_results.py, plot_common.py, make_figures.py, compare_examples.py). Results in + `results-2026-10-02-tower121/`, `results-2026-10-04-tower111/`, `results-common-tower121/111/`. + Full-size example `.jxl` files in `tools/camera-bench/examples-*` (git-ignored, ~115 MB per set). + The recorded auto-exposure steps are also in `tests/data/ae_steps.csv` (used by a unit test). + +## 5. Methods that are easy to misunderstand + +- **Forcing settings:** feature writes over USB while streaming. `ExposureAuto`/`GainAuto` set to + `Off` freezes current values, then `ExposureTime`/`Gain` are written; for the 2 s "force" phase the + camera delivers 10–30 frames at the forced values; then auto is set back to `Continuous` (time 0). + Dark step: exposure E₀/16 with 0 dB gain. Bright step: exposure at the 20 ms cap plus gain up to + +24.1 dB (×16). Bright start is fully clipped (97–100%), so the camera cannot see how far over it is. +- **Brightness** = mean of a sparse sample of the frame buffer on 0–255 (12-bit: high byte). + **Settled** = stays within ±10% of the median brightness of the run's last 2 s. +- **Frames in transit:** the first 1–5 frames after release still carry the forced values. +- **Shared frame:** tower 121's real frame rebuilt bit-exactly from its lossless 12-bit file and + copied to both towers (identical MD5). Its 8-bit raw version is the 12-bit value >> 4; its "RGB8" + version is the tower's own edge-aware conversion scaled to 8 bits, so on the shared frame the RGB8 + file size equals raw 8-bit's (2.66 MB). The camera's real RGB8 files were 3.7–4.0 MB. +- **Synthetic frame:** "dead leaves" texture plus noise (ISO 19567-2 style), sharper than any real + scene; worst case for the encoder; used for pile-up and heat. +- **Quality** is against the exact lossless 12-bit colour image of the same frame: PSNR on the + 12-bit scale over 3 channels; SSIM on luma. JPEG XL is perceptual, so PSNR is moderate while SSIM + is high. +- Camera `BayerRG` = OpenCV `COLOR_BayerBG2RGB` (verified). The camera applies white balance to raw. +- 12-bit JPEG XL from 16-bit words needs bit depth "from codestream", or values get rescaled; + lossless needs the original colour profile. An earlier mistake on both points was found and the + 12-bit numbers corrected before the final report. + +## 6. Key results (numbers to quote) + +Frame rate at 250 MB/s (fps): RGB8 4.95, BayerRG8 14.75, BayerRG10p 11.81, BayerRG12p 9.86, +BayerRG12 7.41. 60 s streams: zero lost or incomplete frames in every format on both towers, also +while saving continuously. On 111 the camera shares the USB 3 controller with a 5G modem, a USB +network adapter and the idle second camera. + +Auto-exposure settling after ×16 steps: ~25–35 frames, so time scales with 1/frame rate. +BayerRG8 1.8–2.7 s; BayerRG12p 2.2–6.6 s (typically ≈ 3 s; long runs contain mid-way pauses); +RGB8 4.6–6.6 s. With exposure at its 20 ms cap the camera adjusts gain by 1–2 dB per frame and +sometimes pauses up to 2.8 s while still far too bright. + +Keep rules (24 runs = 3 formats × 2 directions × 2 repeats × 2 towers; good = within ±10% of +final level): +- "Exposure and gain unchanged 3 frames": 0/24. Replay detail (added 2026-10-05 from + `tests/data/ae_steps.csv`): 16 runs fired on the first frames (0.2–0.5 s, frames in transit, image + black 9–13 or white 253–255 vs ≈ 120); 8 fired during mid-way pauses at brightness 33–227. +- "In band 2 frames": 12/24 (passes through the band ~15% above final level when descending). +- "In band and steady (≤ 2%/frame) 3 frames": 21/24 (3 slightly early on descents); median time + when correct 3.61 s (121) / 5.45 s (111). Implemented as `settle_rule = brightness`. +- Band = `IntensityControllerTarget` ± `Tolerance` = 50 ± 5% → 114.75–140.25 on 0–255. + +Auto white balance: recovers in 2–4 frames (0.2–0.8 s), stable ±0.8–3.1%. Exposure, not white +balance, sets the wait. Auto ratios ≈ red 2.36–2.46, blue 1.87–1.97 vs fixed profile 2.87 / 2.0. + +Processing on the shared frame (4 threads; IOTA / Delta, ms): unpack 10 / 17; de-Bayer 12-bit +bilinear 17 / 22, edge-aware 28 / 33; encode 12-bit d1.0 e3 480 / 765; today's RGB8 encode +d0.8 e4 566 / 867. Totals: 12-bit edge-aware d1.0 e3 **518 / 820 ms, 2.65 MB**; today RGB8 +567 / 872 ms, 3.27 MB; raw 8-bit → colour 468 / 741 ms, 2.66 MB. Delta ≈ 1.6× slower. +Unpack + de-Bayer = 7% of a save; encode ≈ 90%. Effort 1–4 within ~15%; effort 3 smallest; +4 threads fastest; on the IOTA 3 threads is oddly slower than 2. + +Distance sweep (12-bit colour, shared frame, MB): lossless 44.8, d0.3 7.19, d0.5 5.15, d0.8 3.16, +d1.0 2.65, d1.5 1.63, d2.0 1.32. Quality at d1.0: PSNR 36.6 dB, SSIM 0.973. Raw planes d0.5: +6.0 MB, PSNR 47.2 dB. Lossless: raw planes 17.6, raw mosaic 20.9, colour 47.4 MB. + +Throughput (saves/s back to back, shared frame): 1 worker × 4 threads IOTA 1.79, Delta 1.19; +2 × 2: 2.15 / 1.53. On the shared frame the IOTA keeps up with one frame per 0.7 s with one worker; +the Delta needs two. Pile-up details on the synthetic frame are in Table 18. + +Heat (10 min each): one frame every 2 s → IOTA 53 °C mean / 5.1 W, Delta 66 °C / 5.7 W. Back to +back (2 × 2): IOTA 80 °C, 12.5 W, loses 16% throughput as it heats; Delta 78 °C, at its 8 W limit, +steady. + +Storage: 12-bit colour d1.0 ≈ 2.65–2.75 MB/frame → 114–119 GB/day at one frame every 2 s. Towers +have 11–22 GB free, so images must leave the tower continuously. + +## 7. The 12-bit question (discussed at length; current position) + +- Noise floor (12-bit DN, green pixels): at ≈ 700 DN 26 (0 dB) / 42 (15 dB); at ≈ 1500 35 / 58; + at ≈ 3000 37 / 80. One 8-bit step = 16 DN; rounding error RMS = 16/√12 = 4.6 DN. +- Errors add in quadrature: √(26² + 4.6²) = 26.4 → rounding raises noise by at most 1.5% in the + measured daylight scene (0.6% at 15 dB, < 0.2% in bright areas). +- Averaging: while noise > one 8-bit step, rounding error is random (dither) and averages down at + the same √N rate as noise, so 8-bit averages resolve sub-step differences as well as 12-bit. A + simulation (not in the report, by user request) showed a 3 DN difference over 100 × 100 pixels + recovered exactly by both at noise 26, but only 0.2 of 3 by 8-bit at noise 2. +- 12 bits always contain at least as much information; at noise 26 the dropped bits carry about + 0.02 bits/pixel of scene information. +- Compression dominates: PSNR 36.6 dB at d1.0 ≈ 61 DN RMS error (13× the rounding error); d0.3 + ≈ 40 DN; raw planes d0.5 ≈ 18 DN. Much of that is discarded noise, and JPEG XL keeps relatively + more precision in dark tones; shadow-only error was not measured. +- Where 12 bits should matter (Table 17b): dark areas at low gain where noise < one 8-bit step + (below ≈ 300 DN at 0 dB, ≈ 100 DN at 15 dB), i.e. high-contrast scenes such as shaded forest or + backlit ridges under a bright sky at dawn or dusk; thin smoke over dark terrain is the relevant + fire-detection case. At night with high gain, noise is amplified above the step again, so little + benefit. Strong shadow lifting (≥ 3 stops) untested; 2 stops showed no difference. +- None of the dark cases were measured (the darkest area in the scene was ≈ 700 DN; a dusk attempt + on 121 needed 24 dB and was discarded). Proposed test: a dawn/dusk frame with dark terrain under a + bright sky, compared as in Figures 12 and 13. +- 12-bit vs 8-bit raw costs little in saving (same size, ~10% more time) but the camera runs at + 9.86 vs 14.75 fps, ~1 s slower settling in daylight. The main gain over today is frame rate, not + bit depth. + +## 8. Limitations to keep in mind + +One scene, two afternoons, main runs at ≈ 15 dB gain plus one 0 dB frame (124 ms exposure, cap +raised to 200 ms); no night data. Steps forced on a still camera, not real slews; real +waypoint-to-waypoint changes are usually smaller. Tower program stopped, so its own load is not +included. Detection quality not measured (PSNR/SSIM stand in). Slew time (≈ 1–1.5 s) and +mechanical settling (≈ 0.2 s) in the pace table are estimates. + +## 9. Implementation status (branch `raw12-capture`) + +Implemented and unit-tested (457 tests pass locally): 12-bit JPEG XL encoding with the bit-depth fix +and true lossless (known issue #20 fixed); frames carry pixel layout, chunk exposure/gain/white +balance and a brightness sample; `ImagePipeline` unpacks and de-Bayers (edge-aware default), +N workers, bounded queue; scheduler save gate pauses the scan when the queue is full; `SettleRule` +(`settle_rule = brightness`) with a time limit and an "at limit" exit; config keys `save_workers`, +`jxl_threads`, `save_queue_max`, `demosaic`, `settle_rule`; profile export script recipe for raw +`standard` and `night` profiles. Plan: [raw12-pipeline-implementation-plan.md](raw12-pipeline-implementation-plan.md). + +Not done yet: exporting the new camera profiles on a tower (the 1620c is currently on tower 111 +and must be moved back to 121 first); field verification with real slews; checking that downstream +consumers decode 12-bit files; the dawn/dusk 12-bit test. The branch is deployed on tower 111, +whose boot service still passes old `--init/--start` flags and needs `scripts/install-tower`. + +## 10. Report structure (for orientation) + +Abstract; Units and abbreviations; 1 Introduction (1.1 question, 1.2 pixel formats incl. raw +mosaic vs raw planes, Figures 1a/1b, Table 1; 1.3 earlier work); 2 Materials and methods (2.1–2.12, +Tables 2–8, Figure 2); 3 Results (3.1 frame rate, 3.2 auto-exposure, 3.3 keep rules, 3.4 white +balance, 3.5 processing per step, 3.6 effort and threads, 3.7 quality setting, 3.8 example images, +3.9 information content of 12-bit data with Tables 17a/17b, 3.10 throughput, 3.11 heat, +3.12 storage; Figures 3–17, Tables 9–20); 4 Discussion (summary Table 21, pace Table 22, +limitations); 5 Conclusions and six recommendations; References [1]–[7]; Appendix A raw data and +tools; Appendix B per-tower processing times. Figures live in `docs/images/raw12/`.