Explain in the 12-bit report why the brightness rule picks the right picture, and add a background file for discussing the report with Claude

docs/raw12-pipeline-measurements.md

@@ -402,6 +402,11 @@ Settling typically takes 25–35 frames, so **settling time scales with 1/frame
 
 ### 3.3 Keep rules
 
+When the gimbal stops at a new waypoint the view changes, often to a much brighter or darker one,
+and auto-exposure needs a few seconds to adapt; frames taken in that time are too dark or too bright.
+The program must recognise when the image has settled and keep that frame. Each candidate rule was
+replayed on the recordings of the 24 forced steps (Section 2.6) to find the frame it would have kept.
+
 ***Table 11.** Candidate keep rules scored on the 24 recorded steps (12 per tower). A keep rule
 decides which frame after a gimbal stop is saved. Correct: the kept frame's brightness is within
 ±10% of the final level; early: the rule kept a frame before that. Target band: brightness between
@@ -417,9 +422,29 @@ kept frame, over the correct runs.*
 | Target band 3 frames | 8 / 4 | 8 / 4 | 3.40 / 4.84 |
 | **Target band and brightness steady (≤ 2% change) over 3 frames** | **11 / 1** | **10 / 2** | 3.61 / 5.45 |
 
-Descending from over-exposure, brightness enters the band about 15% above its final level, so "in
-band" alone fires early; also requiring the brightness to have stopped changing corrects most of
-these cases.
+**Exposure and gain unchanged.** This rule assumes that once the camera stops adjusting, the image
+is right. It kept a frame early in every run, for two reasons:
+
+- In 16 of 24 runs it fired on the first frames after release (0.2–0.5 s). These frames were already
+  on their way from the camera and still carried the forced settings, so the settings looked
+  unchanged while the image was black (brightness 9–13 DN) or white (253–255 DN), against a settled
+  level of about 120 DN. At a real waypoint the same happens: the settings have not changed yet
+  because the camera has not reacted to the new view.
+- In the other 8 runs it fired during a pause: auto-exposure adjusts in steps and sometimes holds
+  still for several frames mid-way, here at brightness 33–227 DN.
+
+Settings that hold still therefore do not show that the image is right.
+
+**Brightness in the target band and steady.** This rule judges the image itself: it keeps a frame
+when, for 3 frames in a row, the brightness is within the band the camera's auto-exposure aims for
+and changes by no more than 2% between frames. Both conditions are needed. Descending from
+over-exposure, brightness passes through the band about 15% above its final level, so "in band"
+alone fired early in half the runs; "steady" alone would fire during the pauses. Combined, the rule
+kept a good frame in 21 of 24 runs; the 3 failures were slightly early on a descent. The program
+implements this rule as `settle_rule = brightness`, with a time limit.
+
+These were ×16 steps forced on a still camera. Changes between neighbouring waypoints are usually
+smaller, so the rule remains to be confirmed on a real scan.
 
 ### 3.4 Auto white balance
 

docs/raw12-pipeline-measurements_llm_context.md

@@ -0,0 +1,193 @@
+# Context for discussing `raw12-pipeline-measurements.md`
+
+This file gives an AI assistant the background needed to discuss the report
+[raw12-pipeline-measurements.md](raw12-pipeline-measurements.md): what was asked, how the tests were
+run, what the numbers mean, what was decided, which points have been questioned, and what is still
+open. Read the report itself for the full tables and figures; this file explains around them.
+State as of 2026-10-05, branch `raw12-capture`.
+
+## 1. The system in one paragraph
+
+Fire-watch towers scan waypoints with a gimbal-mounted Allied Vision Alvium camera and keep one
+picture per waypoint, saved as JPEG XL on the tower's computer. After the gimbal stops, the camera's
+continuous auto-exposure needs frames to adapt to the new view, so the scan waits ("settling"). The
+tower program is C++ (`fgc`), using Vimba X (VmbCPP), OpenCV and libjxl. Two computers are in use:
+LattePanda IOTA (Intel N150, tower 121) and LattePanda 3 Delta (Celeron N5105, tower 111).
+
+## 2. The question and how it evolved
+
+- Origin: speeding up the per-waypoint capture. Today the camera sends RGB8 (colour made in the
+  camera, 48.6 MB per full frame), which limits the frame rate over USB to 4.95 fps at the agreed
+  250 MB/s link cap, so exposure settles slowly.
+- Proposal tested: camera sends raw `BayerRG12p` (24.3 MB, 9.86 fps), the tower unpacks,
+  de-Bayers (converts to colour) and saves 12-bit colour JPEG XL.
+- An earlier campaign ([camera-capture-measurements.md](camera-capture-measurements.md)) used
+  blurred dim scenes; its compression numbers were unrepresentative. This campaign used a focused,
+  real scene (spruce, birch, leaves, building, ~13% clipped sky).
+- The report went through several rounds of requested edits: example images per compression type,
+  methodology detail (how settings were forced), processing on one identical frame on both towers,
+  an abstract, a pixel-format description, an academic structure (numbered sections, figure and
+  table captions, units in every header, references), captions that explain every term, an
+  explanation of raw mosaic vs raw planes, a more careful treatment of what 12 bits add, and a fuller
+  explanation of the keep rules.
+
+## 3. User decisions (fixed constraints, do not re-litigate)
+
+- Full resolution always; **no binning**; **one quality setting** (no switching between a fast low
+  quality mode and a slow high quality one); continuous acquisition.
+- **No exposure memory** across passes or from neighbouring waypoints.
+- USB link cap **250 MB/s** (`DeviceLinkThroughputLimit`), conservative.
+- **`BayerRG12p` everywhere**, day and night profiles.
+- Save **12-bit colour JPEG XL only**; no raw-planes option in the program for now.
+- **Fixed white balance** ratios kept in the profiles (auto white balance was measured but not adopted).
+- When saving falls behind, **pause the scan** (save gate) rather than drop frames.
+- Writing style for the report: academic structure, captions that explain their terms, units given
+  wherever possible, plain language. Commit messages: one plain-language line, no trailers.
+
+## 4. Test setup essentials
+
+- Same camera for both towers: Alvium 1800 U-1620c `DEV_1AB22C0B7D4E`, 5328 × 3040, firmware 14.3.
+  Moved from 121 to 111 so only the computer differs. Tower 111's own camera (1800 U-1240c, 12 MP,
+  4024 × 3036) was not tested; its frames are 25% smaller, so the Delta results are conservative.
+- Dates: tower 121 on 2 Oct 2026 (sunny afternoon); tower 111 on 4 Oct (afternoon into evening);
+  shared-frame processing on both towers simultaneously on 4 Oct 23:03–23:32.
+- The tower program was stopped during all tests.
+- Storage write speed: IOTA 218 MB/s, Delta 76 MB/s (eMMC). Libraries: 121 libjxl 0.11.1 /
+  OpenCV 4.10; 111 libjxl 0.11.0 / OpenCV 4.5.4.
+- Tools in `tools/camera-bench/` (camera_bench.py with vmbpy; raw12_bench.cpp; run scripts;
+  plot_results.py, plot_common.py, make_figures.py, compare_examples.py). Results in
+  `results-2026-10-02-tower121/`, `results-2026-10-04-tower111/`, `results-common-tower121/111/`.
+  Full-size example `.jxl` files in `tools/camera-bench/examples-*` (git-ignored, ~115 MB per set).
+  The recorded auto-exposure steps are also in `tests/data/ae_steps.csv` (used by a unit test).
+
+## 5. Methods that are easy to misunderstand
+
+- **Forcing settings:** feature writes over USB while streaming. `ExposureAuto`/`GainAuto` set to
+  `Off` freezes current values, then `ExposureTime`/`Gain` are written; for the 2 s "force" phase the
+  camera delivers 10–30 frames at the forced values; then auto is set back to `Continuous` (time 0).
+  Dark step: exposure E₀/16 with 0 dB gain. Bright step: exposure at the 20 ms cap plus gain up to
+  +24.1 dB (×16). Bright start is fully clipped (97–100%), so the camera cannot see how far over it is.
+- **Brightness** = mean of a sparse sample of the frame buffer on 0–255 (12-bit: high byte).
+  **Settled** = stays within ±10% of the median brightness of the run's last 2 s.
+- **Frames in transit:** the first 1–5 frames after release still carry the forced values.
+- **Shared frame:** tower 121's real frame rebuilt bit-exactly from its lossless 12-bit file and
+  copied to both towers (identical MD5). Its 8-bit raw version is the 12-bit value >> 4; its "RGB8"
+  version is the tower's own edge-aware conversion scaled to 8 bits, so on the shared frame the RGB8
+  file size equals raw 8-bit's (2.66 MB). The camera's real RGB8 files were 3.7–4.0 MB.
+- **Synthetic frame:** "dead leaves" texture plus noise (ISO 19567-2 style), sharper than any real
+  scene; worst case for the encoder; used for pile-up and heat.
+- **Quality** is against the exact lossless 12-bit colour image of the same frame: PSNR on the
+  12-bit scale over 3 channels; SSIM on luma. JPEG XL is perceptual, so PSNR is moderate while SSIM
+  is high.
+- Camera `BayerRG` = OpenCV `COLOR_BayerBG2RGB` (verified). The camera applies white balance to raw.
+- 12-bit JPEG XL from 16-bit words needs bit depth "from codestream", or values get rescaled;
+  lossless needs the original colour profile. An earlier mistake on both points was found and the
+  12-bit numbers corrected before the final report.
+
+## 6. Key results (numbers to quote)
+
+Frame rate at 250 MB/s (fps): RGB8 4.95, BayerRG8 14.75, BayerRG10p 11.81, BayerRG12p 9.86,
+BayerRG12 7.41. 60 s streams: zero lost or incomplete frames in every format on both towers, also
+while saving continuously. On 111 the camera shares the USB 3 controller with a 5G modem, a USB
+network adapter and the idle second camera.
+
+Auto-exposure settling after ×16 steps: ~25–35 frames, so time scales with 1/frame rate.
+BayerRG8 1.8–2.7 s; BayerRG12p 2.2–6.6 s (typically ≈ 3 s; long runs contain mid-way pauses);
+RGB8 4.6–6.6 s. With exposure at its 20 ms cap the camera adjusts gain by 1–2 dB per frame and
+sometimes pauses up to 2.8 s while still far too bright.
+
+Keep rules (24 runs = 3 formats × 2 directions × 2 repeats × 2 towers; good = within ±10% of
+final level):
+- "Exposure and gain unchanged 3 frames": 0/24. Replay detail (added 2026-10-05 from
+  `tests/data/ae_steps.csv`): 16 runs fired on the first frames (0.2–0.5 s, frames in transit, image
+  black 9–13 or white 253–255 vs ≈ 120); 8 fired during mid-way pauses at brightness 33–227.
+- "In band 2 frames": 12/24 (passes through the band ~15% above final level when descending).
+- "In band and steady (≤ 2%/frame) 3 frames": 21/24 (3 slightly early on descents); median time
+  when correct 3.61 s (121) / 5.45 s (111). Implemented as `settle_rule = brightness`.
+- Band = `IntensityControllerTarget` ± `Tolerance` = 50 ± 5% → 114.75–140.25 on 0–255.
+
+Auto white balance: recovers in 2–4 frames (0.2–0.8 s), stable ±0.8–3.1%. Exposure, not white
+balance, sets the wait. Auto ratios ≈ red 2.36–2.46, blue 1.87–1.97 vs fixed profile 2.87 / 2.0.
+
+Processing on the shared frame (4 threads; IOTA / Delta, ms): unpack 10 / 17; de-Bayer 12-bit
+bilinear 17 / 22, edge-aware 28 / 33; encode 12-bit d1.0 e3 480 / 765; today's RGB8 encode
+d0.8 e4 566 / 867. Totals: 12-bit edge-aware d1.0 e3 **518 / 820 ms, 2.65 MB**; today RGB8
+567 / 872 ms, 3.27 MB; raw 8-bit → colour 468 / 741 ms, 2.66 MB. Delta ≈ 1.6× slower.
+Unpack + de-Bayer = 7% of a save; encode ≈ 90%. Effort 1–4 within ~15%; effort 3 smallest;
+4 threads fastest; on the IOTA 3 threads is oddly slower than 2.
+
+Distance sweep (12-bit colour, shared frame, MB): lossless 44.8, d0.3 7.19, d0.5 5.15, d0.8 3.16,
+d1.0 2.65, d1.5 1.63, d2.0 1.32. Quality at d1.0: PSNR 36.6 dB, SSIM 0.973. Raw planes d0.5:
+6.0 MB, PSNR 47.2 dB. Lossless: raw planes 17.6, raw mosaic 20.9, colour 47.4 MB.
+
+Throughput (saves/s back to back, shared frame): 1 worker × 4 threads IOTA 1.79, Delta 1.19;
+2 × 2: 2.15 / 1.53. On the shared frame the IOTA keeps up with one frame per 0.7 s with one worker;
+the Delta needs two. Pile-up details on the synthetic frame are in Table 18.
+
+Heat (10 min each): one frame every 2 s → IOTA 53 °C mean / 5.1 W, Delta 66 °C / 5.7 W. Back to
+back (2 × 2): IOTA 80 °C, 12.5 W, loses 16% throughput as it heats; Delta 78 °C, at its 8 W limit,
+steady.
+
+Storage: 12-bit colour d1.0 ≈ 2.65–2.75 MB/frame → 114–119 GB/day at one frame every 2 s. Towers
+have 11–22 GB free, so images must leave the tower continuously.
+
+## 7. The 12-bit question (discussed at length; current position)
+
+- Noise floor (12-bit DN, green pixels): at ≈ 700 DN 26 (0 dB) / 42 (15 dB); at ≈ 1500 35 / 58;
+  at ≈ 3000 37 / 80. One 8-bit step = 16 DN; rounding error RMS = 16/√12 = 4.6 DN.
+- Errors add in quadrature: √(26² + 4.6²) = 26.4 → rounding raises noise by at most 1.5% in the
+  measured daylight scene (0.6% at 15 dB, < 0.2% in bright areas).
+- Averaging: while noise > one 8-bit step, rounding error is random (dither) and averages down at
+  the same √N rate as noise, so 8-bit averages resolve sub-step differences as well as 12-bit. A
+  simulation (not in the report, by user request) showed a 3 DN difference over 100 × 100 pixels
+  recovered exactly by both at noise 26, but only 0.2 of 3 by 8-bit at noise 2.
+- 12 bits always contain at least as much information; at noise 26 the dropped bits carry about
+  0.02 bits/pixel of scene information.
+- Compression dominates: PSNR 36.6 dB at d1.0 ≈ 61 DN RMS error (13× the rounding error); d0.3
+  ≈ 40 DN; raw planes d0.5 ≈ 18 DN. Much of that is discarded noise, and JPEG XL keeps relatively
+  more precision in dark tones; shadow-only error was not measured.
+- Where 12 bits should matter (Table 17b): dark areas at low gain where noise < one 8-bit step
+  (below ≈ 300 DN at 0 dB, ≈ 100 DN at 15 dB), i.e. high-contrast scenes such as shaded forest or
+  backlit ridges under a bright sky at dawn or dusk; thin smoke over dark terrain is the relevant
+  fire-detection case. At night with high gain, noise is amplified above the step again, so little
+  benefit. Strong shadow lifting (≥ 3 stops) untested; 2 stops showed no difference.
+- None of the dark cases were measured (the darkest area in the scene was ≈ 700 DN; a dusk attempt
+  on 121 needed 24 dB and was discarded). Proposed test: a dawn/dusk frame with dark terrain under a
+  bright sky, compared as in Figures 12 and 13.
+- 12-bit vs 8-bit raw costs little in saving (same size, ~10% more time) but the camera runs at
+  9.86 vs 14.75 fps, ~1 s slower settling in daylight. The main gain over today is frame rate, not
+  bit depth.
+
+## 8. Limitations to keep in mind
+
+One scene, two afternoons, main runs at ≈ 15 dB gain plus one 0 dB frame (124 ms exposure, cap
+raised to 200 ms); no night data. Steps forced on a still camera, not real slews; real
+waypoint-to-waypoint changes are usually smaller. Tower program stopped, so its own load is not
+included. Detection quality not measured (PSNR/SSIM stand in). Slew time (≈ 1–1.5 s) and
+mechanical settling (≈ 0.2 s) in the pace table are estimates.
+
+## 9. Implementation status (branch `raw12-capture`)
+
+Implemented and unit-tested (457 tests pass locally): 12-bit JPEG XL encoding with the bit-depth fix
+and true lossless (known issue #20 fixed); frames carry pixel layout, chunk exposure/gain/white
+balance and a brightness sample; `ImagePipeline` unpacks and de-Bayers (edge-aware default),
+N workers, bounded queue; scheduler save gate pauses the scan when the queue is full; `SettleRule`
+(`settle_rule = brightness`) with a time limit and an "at limit" exit; config keys `save_workers`,
+`jxl_threads`, `save_queue_max`, `demosaic`, `settle_rule`; profile export script recipe for raw
+`standard` and `night` profiles. Plan: [raw12-pipeline-implementation-plan.md](raw12-pipeline-implementation-plan.md).
+
+Not done yet: exporting the new camera profiles on a tower (the 1620c is currently on tower 111
+and must be moved back to 121 first); field verification with real slews; checking that downstream
+consumers decode 12-bit files; the dawn/dusk 12-bit test. The branch is deployed on tower 111,
+whose boot service still passes old `--init/--start` flags and needs `scripts/install-tower`.
+
+## 10. Report structure (for orientation)
+
+Abstract; Units and abbreviations; 1 Introduction (1.1 question, 1.2 pixel formats incl. raw
+mosaic vs raw planes, Figures 1a/1b, Table 1; 1.3 earlier work); 2 Materials and methods (2.1–2.12,
+Tables 2–8, Figure 2); 3 Results (3.1 frame rate, 3.2 auto-exposure, 3.3 keep rules, 3.4 white
+balance, 3.5 processing per step, 3.6 effort and threads, 3.7 quality setting, 3.8 example images,
+3.9 information content of 12-bit data with Tables 17a/17b, 3.10 throughput, 3.11 heat,
+3.12 storage; Figures 3–17, Tables 9–20); 4 Discussion (summary Table 21, pace Table 22,
+limitations); 5 Conclusions and six recommendations; References [1]–[7]; Appendix A raw data and
+tools; Appendix B per-tower processing times. Figures live in `docs/images/raw12/`.