From 1fc87bf1eecca02efad0605ffe758ad3a25b456e Mon Sep 17 00:00:00 2001 From: pgdalmeida Date: Mon, 5 Oct 2026 11:28:52 +0200 Subject: [PATCH] Rewrite the 12-bit camera test report as a numbered paper with an abstract, figure and table captions, and units --- docs/raw12-pipeline-implementation-plan.md | 2 +- docs/raw12-pipeline-measurements.md | 983 ++++++++++++--------- include/fgc/SettleRule.h | 2 +- 3 files changed, 580 insertions(+), 407 deletions(-) diff --git a/docs/raw12-pipeline-implementation-plan.md b/docs/raw12-pipeline-implementation-plan.md index 4da8e8d..ad3df23 100644 --- a/docs/raw12-pipeline-implementation-plan.md +++ b/docs/raw12-pipeline-implementation-plan.md @@ -51,7 +51,7 @@ exactly as today. ## Processing budget the design must fit Measured on the **identical frame** on both computers (5328×3040, real scene; 4 threads; see the -report's [Where the time goes](raw12-pipeline-measurements.md#where-the-time-goes)). +report's [Section 3.5](raw12-pipeline-measurements.md#35-processing-time-per-step)). ### Time per saved frame, step by step diff --git a/docs/raw12-pipeline-measurements.md b/docs/raw12-pipeline-measurements.md index dc99313..6926b3c 100644 --- a/docs/raw12-pipeline-measurements.md +++ b/docs/raw12-pipeline-measurements.md @@ -1,561 +1,734 @@ -# 12-bit raw capture pipeline — final report +# Raw 12-bit capture with on-tower colour conversion: measurements on the LattePanda IOTA and 3 Delta -**Question:** should the towers stop asking the camera for 8-bit colour (RGB8) and take raw -12-bit Bayer (`BayerRG12p`) instead, with the LattePanda converting it to colour and saving 12-bit -JPEG XL? The aim is a higher frame rate, so auto-exposure and auto white balance settle faster, -and more tonal detail for the smoke-detection algorithms. Can the IOTA (tower 121) and the -3 Delta (tower 111) carry the work, how long does each step take, and how much space do the -images need? +*Fire-watch tower camera system — technical report, October 2026* -**Campaign:** 2–5 October 2026. Camera tests on tower 121 (2 Oct) and tower 111 (4 Oct) **with the -same camera** (Alvium 1800 U-1620c, 5328×3040), moved from 121 to 111 and looking at the same -scene, so the two towers differ only in their computer. Processing tests repeated on **one -identical frame** on both towers (4 Oct). Earlier blurred-scene measurements: -[camera-capture-measurements.md](camera-capture-measurements.md). What to build from this: +## Abstract + +The towers' camera currently delivers 8-bit colour images (RGB8), converted to colour inside the +camera. This study tested whether the camera should instead deliver its raw 12-bit sensor data +(`BayerRG12p`), with the tower's computer converting it to colour and saving 12-bit JPEG XL. The +same camera (Allied Vision Alvium 1800 U-1620c, 16.2 MP) was measured on both tower computers, the +LattePanda IOTA (tower 121) and the LattePanda 3 Delta (tower 111). Frame rate, auto-exposure and +auto-white-balance settling, the duration of every processing step, file sizes, CPU temperature and +power were recorded, and the processing was repeated on one identical image on both computers. + +Raw 12-bit doubles the camera's frame rate over the same USB link (9.86 fps against 4.95 fps), so +auto-exposure settles in about 60% of the time, with no lost frames on either tower. Converting to +colour on the tower costs 38–50 ms; the JPEG XL encode dominates, and the complete 12-bit save takes +slightly less time than today's RGB8 save (0.52 s on the IOTA, 0.82 s on the Delta, which is 1.6× +slower). At a realistic scan pace both computers keep up and stay cool (53 °C and 66 °C); the Delta +needs two save workers to match the IOTA's single one. A 12-bit colour file at quality setting +(distance) 1.0 is 2.7 MB, smaller than today's 3.3–4.0 MB. In daylight, however, sensor noise +exceeds one 8-bit quantisation step, so the extra bits mainly carry information in deep shadow, at +dusk and at night. A rule for when to keep a frame (brightness in the camera's target band and +steady for 3 frames) selected a good frame in 21 of 24 tests; waiting for exposure and gain to stop +changing never did. + +## Units and abbreviations + +| Symbol | Meaning | +|---|---| +| MB, GB, TB | 10⁶, 10⁹, 10¹² bytes | +| MB/s | 10⁶ bytes per second | +| fps | frames per second | +| ms, s | milliseconds, seconds | +| dB | decibel (camera gain; image quality PSNR) | +| DN | digital number: a pixel value, on a 0–255 scale (8-bit) or 0–4095 (12-bit) as stated | +| °C, W, GHz | CPU package temperature, package power, average core clock | +| MP | megapixels (10⁶ pixels) | +| d, e | JPEG XL distance (quality: 0 = lossless, higher = smaller and lossier) and effort (1–9) | +| PSNR | peak signal-to-noise ratio, in dB | +| SSIM | structural similarity index, 0–1 (1 = identical) [6] | +| IOTA, Delta | LattePanda IOTA (tower 121) and LattePanda 3 Delta (tower 111) | + +--- + +## 1. Introduction + +### 1.1 Background and question + +Each tower scans a set of waypoints and keeps one picture per waypoint. After the gimbal stops, the +camera's auto-exposure needs a number of frames to adapt to the new view, so the scan waits; and +each kept picture is compressed to JPEG XL [2] on the tower. Today the camera converts its sensor +data to 8-bit colour (RGB8) itself. That format is three times larger than the raw sensor data, which +limits the frame rate over the camera's USB link and thereby lengthens the exposure settling. + +This report answers: should the towers take raw 12-bit data from the camera instead, and convert it +to colour on the tower? Specifically: (1) how much faster is the camera, and does exposure settle +faster; (2) how long does each processing step take on each tower computer; (3) how large are the +files; (4) can both computers keep up without piling up images or overheating; and (5) what do the +extra bits add. + +### 1.2 The camera's pixel formats + +The sensor sees through a colour-filter mosaic: each pixel measures only red, green or blue, in the +pattern of Figure 1 ("BayerRG": a red pixel first). Producing a colour image means estimating the +two missing colours at every pixel from its neighbours, called **de-Bayering** (demosaicing). The +camera can either send the raw mosaic or de-Bayer it itself. + +``` +R G R G R G ... +G B G B G B ... +R G R G R G ... +``` + +***Figure 1.** The BayerRG colour-filter pattern of the sensor: each pixel records one colour.* + +***Table 1.** Pixel formats considered, at the 1620c's full resolution (5328 × 3040 pixels) and a USB +link cap of 250 MB/s [4].* + +| Format | What the camera sends | Bits per value | Bytes per pixel | Frame size (MB) | Maximum rate (fps) | Colour made by | +|---|---|---|---|---|---|---| +| `RGB8` (today) | three values per pixel, two of them estimated | 8 | 3 | 48.6 | 4.95 | the camera | +| `BayerRG8` | the raw mosaic, top 8 of the sensor's 12 bits | 8 | 1 | 16.2 | 14.75 | the tower | +| `BayerRG10p` | the raw mosaic, 10 bits, packed (4 pixels in 5 bytes) | 10 | 1.25 | 20.2 | 11.81 | the tower | +| **`BayerRG12p`** (proposed) | the raw mosaic, all 12 bits, packed (2 pixels in 3 bytes) | 12 | 1.5 | 24.3 | **9.86** | the tower | +| `BayerRG12` | the same 12 bits, each in a 16-bit word (4 bits of padding) | 12 | 2 | 32.4 | 7.41 | the tower | + +RGB8 is the largest because two of its three values per pixel are estimates; it carries no more +measured information than `BayerRG8`. `BayerRG12p` and `BayerRG12` hold identical data; packing saves +25% of the bytes and so allows 33% more frames. The camera applies its white balance to the raw +formats too, so the tower does not need to. The camera also offers grey (`Mono8/10/12`, which +discards colour), `BGR8` (RGB8 in reverse order) and video formats (`YCbCr…`, which reduce colour +detail); none of these was considered. + +### 1.3 Earlier work + +An earlier campaign ([camera-capture-measurements.md](camera-capture-measurements.md)) used blurred, +dim indoor scenes and found that the camera link is stable at full resolution once the USB settings +of known issue #19 are applied; its compression figures were not representative of real scenes. +The implementation built on this report is described in [raw12-pipeline-implementation-plan.md](raw12-pipeline-implementation-plan.md). -## Conclusions +--- -| Topic | Finding | Evidence | -|---|---|---| -| Frame rate | `BayerRG12p` runs at **9.86 fps**, 2× RGB8's 4.95 fps, at the 250 MB/s cap; `BayerRG8` 14.75 fps. Unpacked `BayerRG12` only 7.4 fps: use the packed format. | [streaming](#frame-rate-and-streaming) | -| Link | No lost or incomplete frames in any format on either tower, also while saving 12-bit flat out. | [streaming](#frame-rate-and-streaming) | -| Auto-exposure | From **under-exposure** (1/16, brightness ~11 of 255) or **over-exposure** (×16, 97–100% blown out), settling takes **~25–35 frames**, so time scales with 1/fps: raw 8-bit 1.8–2.7 s, **12-bit 2.2–6.6 s (typically ~3 s)**, RGB8 4.6–6.6 s. | [method](#method), [auto-exposure](#auto-exposure) | -| Which frame to keep | **"Brightness in the camera's target band and steady for 3 frames"**: right in 21 of 24 steps. "Exposure and gain unchanged" was never right. | [settle rules](#which-frame-to-keep) | -| Auto white balance | Recovers in **2–4 frames** (0.2–0.8 s) at every rate; stable within ±1–3% on a still scene. Exposure, not white balance, sets the wait. | [white balance](#auto-white-balance) | -| Processing cost | Same frame on both computers: unpack + colour conversion **38 ms (IOTA) / 50 ms (Delta)**; JPEG XL encode 480 / 765 ms. **A 12-bit save (518 / 820 ms) costs a little less than today's RGB8 save (567 / 872 ms).** | [stages](#where-the-time-goes) | -| IOTA vs 3 Delta | **The Delta takes 1.6× as long** on the identical frame (820 vs 518 ms for a 12-bit save); it saves 1.19 frames/s with one worker, 1.53 with two (IOTA 1.79 / 2.15). | [stages](#where-the-time-goes) | -| Pace | The camera side sets the pace: a waypoint needs ~1.5–5 s (move + exposure), a 12-bit save 0.56 s (IOTA) or 0.65 s (Delta, two workers). On the shared frame the IOTA keeps up with one frame every **0.7 s** with one worker; the Delta needs **two workers** for that. | [pace](#overall-pace-how-fast-can-a-tower-capture), [pile-up](#keeping-up-with-the-scan) | -| Heat | One frame every 2 s: IOTA **53 °C / 5.1 W**, Delta **66 °C / 5.7 W**. Flat out: ~80 °C on both; the IOTA slowed 16% over 10 min, the Delta held steady at its 8 W power limit. | [heat](#heat-and-power) | -| Storage | 12-bit colour at distance 1.0: **2.7 MB/frame**, less than today's 3.7–4.0 MB. Lossless 12-bit: 44–47 MB as colour, **17–18 MB as raw planes**. | [quality](#quality-setting-size-and-quality) | -| **Value of 12 bits** | Sensor noise in this sunlit scene is **2–6 steps of 8-bit even at 0 dB gain**, so 8-bit already holds what the sensor delivers in mid and bright tones. 12-bit adds information only below about **300 of 4095** (deep shadow), which daylight scenes barely have and dusk or night scenes do. | [bit depth](#what-12-bit-adds) | +## 2. Materials and methods -### Recommendation +### 2.1 Hardware and software -1. **Take raw from the camera** (`BayerRG12p`, 250 MB/s) and convert to colour on the tower, - edge-aware. It doubles the frame rate and roughly halves the settle time, at no extra CPU, - heat or storage cost compared with today, on either computer. -2. **Save 12-bit colour JPEG XL at distance 1.0, effort 3, 4 threads.** 2.7 MB per frame. -3. **Workers per computer:** one worker on the IOTA; two workers × 2 threads on the Delta, which - then keeps up with a frame every 1.0 s. With the measured settle times (~2–3 s per waypoint), - the scan pace stays below either limit. The save queue still needs a bound. -4. **Keep the frame by "target band and steady for 3 frames",** with a time limit. -5. **12-bit vs raw 8-bit is a choice, not a cost.** `BayerRG8` (14.75 fps) settles about 1 s faster - in daylight and loses nothing measurable there; `BayerRG12p` keeps the deep shadows at dusk and - night. The pixel format lives in the camera profile, so the code should accept both, and - `standard` and `night` can each use the right one. -6. **Store raw 12-bit planes losslessly (17–18 MB) only if the detection pipeline wants exact sensor - values**; it is 6× the storage. - -## Setup +***Table 2.** Test systems. The same camera was moved from tower 121 to tower 111.* | | Tower 121 | Tower 111 | |---|---|---| -| Computer | LattePanda IOTA, Intel N150, 4 cores, up to 3.6 GHz, 7 GB | LattePanda 3 Delta, Celeron N5105, 4 cores, up to 2.9 GHz, 7.6 GB | -| Storage write | eMMC, 218 MB/s | eMMC, 76 MB/s | -| Libraries | libjxl 0.11.1, OpenCV 4.10 | libjxl 0.11.0, OpenCV 4.5.4 | -| Date, light | 2 Oct, 15:01–16:50, sun | 4 Oct, 16:19–18:19, sun then evening | -| Camera | Alvium 1800 U-1620c `DEV_1AB22C0B7D4E`, 5328×3040, link cap 250 MB/s | same camera | -| Exposure | capped at 20 ms (the `standard` profile), so ~15 dB gain in this scene | same | +| Computer | LattePanda IOTA: Intel N150, 4 cores, ≤ 3.6 GHz, 7 GB RAM | LattePanda 3 Delta: Intel Celeron N5105, 4 cores, ≤ 2.9 GHz, 7.6 GB RAM | +| Storage write speed (MB/s) | 218 (eMMC) | 76 (eMMC) | +| Libraries | libjxl 0.11.1 [2], OpenCV 4.10 [3] | libjxl 0.11.0, OpenCV 4.5.4 | +| Camera | Alvium 1800 U-1620c `DEV_1AB22C0B7D4E` [5], 5328 × 3040 pixels | same camera | +| Link cap (MB/s) | 250 | 250 | +| Test dates and light | 2 Oct, 15:01–16:50, sun | 4 Oct, 16:19–18:19, sun, then evening | -The scene, through the camera: spruce, birch trunks, leaves, a building, ~13% blown-out sky. +The tower control program was stopped during every test (the camera accepts one program at a time, +and the CPU figures should have a single cause). Afterwards both cameras were returned to their +`standard` profile. + +### 2.2 Scene ![The scene](images/raw12/examples/121/thumbs/c12_ea_lossless.jpg) -The tower program was stopped during every test, and both cameras went back to their `standard` -profiles afterwards. +***Figure 2.** The test scene from tower 121's camera (2 Oct, 15:07; exposure 20 ms, gain ≈ 15 dB): +spruce, birch trunks, leaves and a building; about 13% of the pixels are clipped sky.* -## Method +### 2.3 Camera control -Everything ran on the towers themselves, with the tower program stopped (the camera accepts one -program at a time, and the CPU figures should have a single cause). The test programs are in -[tools/camera-bench](../tools/camera-bench/README.md); the raw results, one JSON line per -measurement, are linked at the end. +A Python program (`camera_bench.py`, Allied Vision vmbpy) drives the camera through the same GenICam +interface [4] the tower program uses. Each setting change is a GenICam feature write sent over the +USB control channel **while the camera keeps streaming**. `ExposureAuto`, `GainAuto` and +`BalanceWhiteAuto` take the values `Off`, `Once` or `Continuous`; setting one to `Off` freezes the +value the automatic control had reached, after which `ExposureTime`, `Gain` and `BalanceRatio` can be +written directly and take effect from the next frame exposed. No profile, user set or file on the +camera is changed. -### How the camera is driven +***Table 3.** Camera settings common to all camera tests.* -`camera_bench.py` talks to the camera through Allied Vision's Python SDK (vmbpy), the same -GenICam interface the tower program uses through C++. Every setting change is a **GenICam feature -write** (`feature.set(value)`), sent over the USB control channel **while the camera keeps -streaming**. `ExposureAuto`, `GainAuto` and `BalanceWhiteAuto` are switches with the values `Off`, -`Once` and `Continuous`. Setting one to `Off` freezes the value the automatic control had reached; -`ExposureTime`, `Gain` and `BalanceRatio` can then be written directly and take effect from the next -frame the camera exposes. No profile, user set or file on the camera is changed; afterwards the -camera is put back by reloading its `standard.xml` profile, exactly as the tower program does. - -Common settings for every camera test: - -| Setting | Value | Why | +| Setting | Value | Reason | |---|---|---| -| Resolution | 5328×3040, binning 1 | full resolution, as decided | -| `DeviceLinkThroughputLimit` | 250 MB/s | the agreed conservative link cap | -| `AcquisitionFrameRate` | the camera's maximum for the format | 9.86 fps 12-bit, 14.75 fps 8-bit raw, 4.95 fps RGB8 | -| `ExposureAutoMax` | 20 ms | as in the `standard` profile; keeps long exposures from limiting the frame rate | -| `IntensityAutoPrecedence` | `MinimizeNoise` | as the camera was left (favours exposure over gain; with exposure capped at 20 ms, the camera then uses gain) | +| Resolution | 5328 × 3040 pixels, binning 1 | full resolution, as decided | +| `DeviceLinkThroughputLimit` | 250 MB/s | agreed conservative link cap | +| `AcquisitionFrameRate` | the camera's maximum for the format (fps) | 9.86 (12-bit), 14.75 (8-bit raw), 4.95 (RGB8) | +| `ExposureAutoMax` | 20 ms | as in the `standard` profile | +| `IntensityAutoPrecedence` | `MinimizeNoise` | favours exposure over gain; with exposure at its 20 ms cap, the camera then uses gain | | `IntensityControllerTarget` / `Tolerance` / `Rate` | 50% / 5% / 100 | camera defaults; rate 100 is the fastest | -| `GainAutoMax` | as left in the camera (≥ 43.5 dB) | the bright steps reached 43.5 dB | -| Chunk data | on: `ExposureTime`, `Gain`, `BalanceRatioRed`, `BalanceRatioBlue`, `Timestamp` | each frame carries the exact settings it was exposed with | -| Buffers | 5 | as production | +| `GainAutoMax` | ≥ 43.5 dB (as left in the camera) | the bright steps reached 43.5 dB | +| Chunk data (per-frame metadata) | `ExposureTime`, `Gain`, `BalanceRatioRed`, `BalanceRatioBlue`, `Timestamp` | each frame carries the settings it was exposed with | +| Buffers | 5 | as in production | -**Per-frame record.** For every frame the script keeps the host arrival time, the camera's frame -number, complete or not, the chunk data above, and the frame's **brightness and blown-out share**, -sampled from the image buffer itself: every 997th byte for RGB8 and 8-bit raw (mixed colours), the -high byte of every 331st pixel pair for `BayerRG12p` (green and blue sites), each scaled to 0–255. -"Blown out" means a sample ≥ 250 of 255. +For every frame the program records the host arrival time, the camera's frame number, completeness, +the chunk data, and the frame's **brightness** (DN, 0–255) and **clipped share** (% of samples +≥ 250 DN), sampled from the image buffer: every 997th byte for RGB8 and 8-bit raw, and the high byte +of every 331st pixel pair for `BayerRG12p`, scaled to 0–255 DN. -### Frame rate and streaming +### 2.4 Frame rate and streaming -For each format: set it, read the camera's maximum frame rate (the camera computes it from the -link cap and frame size), stream 60 s at that rate, and count frames received, incomplete frames -(the SDK's frame status) and lost frames (gaps in the camera's frame counter). Once more for -`BayerRG12p` while the tower saved 12-bit images flat out in parallel. +For each format, the camera's maximum frame rate was read, then the camera streamed for 60 s at that +rate. Counted: frames received, incomplete frames (SDK frame status) and lost frames (gaps in the +camera's frame counter). `BayerRG12p` was streamed once more while the tower saved 12-bit images +continuously in parallel. -### Auto-exposure step test +### 2.5 Auto-exposure step test -One run, per format and direction (`camera_bench.py ae`): +***Table 4.** One auto-exposure step run.* -| Phase | Duration | Exactly what is done | +| Phase | Duration (s) | Procedure | |---|---|---| -| 1. Settle | 8 s | Stream with `ExposureAuto = Continuous`, `GainAuto = Continuous`. "Before" = median exposure and gain of the last 5 frames (chunk data). The brightness the camera steers is exposure × linear gain: E₀ = exposure × 10^(gain/20). | -| 2. Force | 2 s | `ExposureAuto = Off`, `GainAuto = Off` (freezes them). Then write a new exposure and gain. **Dark:** `ExposureTime` = max(`ExposureAutoMin`, E₀/16), `Gain` = 0 dB. **Bright:** `ExposureTime` = min(20 ms, 16·E₀), and the rest as gain: `Gain` = 20·log10(16·E₀ / exposure), up to `GainAutoMax`. Streaming continues; the camera delivers 10–30 frames at the forced settings. | -| 3. Release | 15 s | Time zero is taken, then `ExposureAuto = Continuous` and `GainAuto = Continuous` are written. Every frame for 15 s is recorded. | +| 1. Settle | 8 | `ExposureAuto` and `GainAuto` = `Continuous`. "Before" = median exposure and gain of the last 5 frames. The brightness the camera steers is E₀ = exposure × 10^(gain/20). | +| 2. Force | 2 | `ExposureAuto` = `GainAuto` = `Off`. **Dark:** `ExposureTime` = max(`ExposureAutoMin`, E₀/16), `Gain` = 0 dB. **Bright:** `ExposureTime` = min(20 ms, 16·E₀); the remainder as `Gain` = 20·log₁₀(16·E₀ / exposure) dB, up to `GainAutoMax`. The camera delivers 10–30 frames at the forced settings. | +| 3. Release | 15 | Time zero is taken, then `ExposureAuto` = `GainAuto` = `Continuous`; every frame is recorded. | -With exposure at its 20 ms cap before the step, the forced values came out as: +***Table 5.** Starting conditions of the auto-exposure steps: the first frame after release, over all +24 runs.* -| | Before (settled) | Forced "dark" (start) | Forced "bright" (start) | +| | Before (settled) | Start of "dark" step | Start of "bright" step | |---|---|---|---| -| Exposure | 20 ms (at its cap) | 7.4–11.5 ms | 20 ms | -| Gain | 15.4–19.5 dB | 0 dB | 39.6–43.5 dB (before + 24.1 dB) | -| Brightness (0–255), first frame after release | 117–122 | **9–13: under-exposed** | **254–255: over-exposed** | -| Blown-out share | 10–15% (sky) | 0% | **97–100%** | +| Exposure (ms) | 20 (at its cap) | 7.4–11.5 | 20 | +| Gain (dB) | 15.4–19.5 | 0 | 39.6–43.5 (before + 24.1) | +| Brightness (DN, 0–255) | 117–122 | **9–13 (under-exposed)** | **254–255 (over-exposed)** | +| Clipped share (%) | 10–15 (sky) | 0 | **97–100** | -So settling was measured from **both sides, under-exposure and over-exposure**. The bright start is -fully saturated: the camera cannot see how far over it is and can only step down until detail -reappears. +Settling was therefore measured from both under- and over-exposure. The bright start is fully +saturated, so the camera cannot measure how far over it is and can only step down until detail +reappears. Each format and direction was run twice: 12 runs per tower, 24 in total. -Each format × direction was run twice: 12 runs per tower, 24 in all. **"Settled"** is judged on the -image: the time from release after which the frame's brightness stays within ±10% of where it -finally settles (median brightness of the run's last 2 s), with the number of frames that took. -The first 1–5 frames after release still carry the forced values (frames already in the -camera-to-host pipeline); they count towards the settle time. +**Settling time** is judged on the image: the time from release after which brightness stays within +±10% of its final level (median over the run's last 2 s), and the number of frames this took. The +first 1–5 frames after release still carry the forced values (frames in transit); they count +towards the settling time. -**Keep rules** were then scored offline on the same 24 per-frame logs: for each rule, the first frame -at which it would have kept a picture, and whether that frame's brightness was within 10% of the -final level ("correct") or not ("early"). The camera's target band is 50% ± 5% of full scale, i.e. -114.8–140.3 on the 0–255 scale used here. +### 2.6 Evaluation of keep rules -### Auto white balance step test +A keep rule decides which frame a waypoint keeps. Each candidate rule was applied offline to the 24 +per-frame logs; for each, the first frame at which it would keep a picture was found, and counted as +**correct** if that frame's brightness was within 10% of the final level, otherwise **early**. The +camera's target band is 50 ± 5% of full scale, i.e. 114.8–140.3 DN on the 0–255 scale. -One run (`camera_bench.py awb`), per format and direction: +### 2.7 Auto white balance step test -| Phase | Duration | Exactly what is done | +***Table 6.** One auto-white-balance step run.* + +| Phase | Duration (s) | Procedure | |---|---|---| -| 1. Exposure | 6 s | Auto-exposure on, settle; then `ExposureAuto = Off`, `GainAuto = Off`, and `ExposureTime`/`Gain` written to the median of the last 5 frames, so only white balance moves from here on. `BalanceWhiteAutoRate = 100` (fastest). | -| 2. Reference | 8 s | `BalanceWhiteAuto = Continuous`. The reference = median red and blue ratios of the frames in the last 3 s (the camera's own answer for this scene: red 2.34–2.46, blue 1.86–1.97). | -| 3. Force | 2 s | `BalanceWhiteAuto = Off`; `BalanceRatioSelector = Red`, `BalanceRatio` = reference × factor; the same for Blue (green is the camera's fixed reference). **×0.5**: red ≈ 1.2, blue ≈ 0.95 (image too green); **×2**: red ≈ 4.8, blue ≈ 3.8 (too magenta). | -| 4. Release | 15 s | Time zero, then `BalanceWhiteAuto = Continuous`; every frame's ratios recorded (chunk data). | +| 1. Exposure | 6 | Auto-exposure settles; then `ExposureAuto` = `GainAuto` = `Off` with `ExposureTime` and `Gain` fixed at the median of the last 5 frames, so only white balance can change. `BalanceWhiteAutoRate` = 100 (fastest). | +| 2. Reference | 8 | `BalanceWhiteAuto` = `Continuous`. Reference = median red and blue ratios over the last 3 s (red 2.34–2.46, blue 1.86–1.97). | +| 3. Force | 2 | `BalanceWhiteAuto` = `Off`; red and blue `BalanceRatio` = reference × factor (green is the fixed reference). ×0.5: red ≈ 1.2, blue ≈ 0.95 (image too green); ×2: red ≈ 4.8, blue ≈ 3.8 (too magenta). | +| 4. Release | 15 | Time zero, then `BalanceWhiteAuto` = `Continuous`; every frame's ratios recorded. | -"Settled" = both ratios within ±5% of the reference and staying there. Each case twice; also the -spread of the reference over phase 2 on the still scene. +Settled = both ratios within ±5% of the reference and remaining there. Each case was run twice. -### Processing tests +### 2.8 Processing tests -`raw12_bench.cpp`, built on each tower against its own libjxl and OpenCV, runs the steps the tower -program would run for one saved frame and **times each with a monotonic clock**: +A C++ program (`raw12_bench.cpp`), built on each tower against its own libraries, performs the steps +the tower program would perform for one saved frame and times each with a monotonic clock. -| Step | What it does | +***Table 7.** Processing steps timed.* + +| Step | Operation | |---|---| | Unpack | `BayerRG12p` (two 12-bit pixels in three bytes) → one 16-bit word per pixel | -| Colour conversion | OpenCV `cvtColor`, `COLOR_BayerBG2RGB` (bilinear) or `…_EA` (edge-aware), 8- or 16-bit; the camera's `BayerRG` is OpenCV's `BayerBG` (checked against the camera's own RGB8) | -| JPEG XL encode | libjxl into memory; threads via libjxl's thread pool; effort and distance as stated; 12-bit input as 16-bit words with the bit depth taken from the file header; lossless with `uses_original_profile` set | +| Colour conversion | OpenCV `cvtColor`, bilinear (`COLOR_BayerBG2RGB`) or edge-aware (`…_EA`), 8- or 16-bit [3]; the camera's `BayerRG` corresponds to OpenCV's `BayerBG` (verified against the camera's RGB8) | +| JPEG XL encode | libjxl [2], into memory; effort and distance as stated; 12-bit input as 16-bit words with the bit depth taken from the file header; lossless with the original colour profile | | File write | `fopen`/`fwrite`/`fflush`/`fclose` to the tower's eMMC | -Each figure is the median of 3 runs (2 in the quality sweep). Before every run on every tower the -program checks that its 8-bit encoder produces **byte for byte** the file production's encoder -produces, that 12-bit lossless decodes to **exactly** the input, and that its colour order matches -the camera's. +Each time is the median of 3 runs (2 in the quality sweep). Before every run, the program verified +that its 8-bit encoder produces byte-for-byte the file of the production encoder, that 12-bit +lossless decodes exactly to the input, and that its colour order matches the camera's. **Input frames.** -- **Shared frame, identical on both towers (the tower comparison uses this):** tower 121's frame of - 2 Oct (15:07, 20 ms, ~15 dB), rebuilt bit-exactly from its lossless 12-bit file and repacked to - `BayerRG12p`; its 8-bit raw is the 12-bit value shifted down 4 bits; its "RGB8" is the edge-aware - colour conversion of the 12-bit frame shifted to 8 bits (the camera's own RGB8 of that instant - does not exist). Copied to both towers; MD5 checksums identical. -- **Synthetic frame, identical on both towers:** a "dead leaves" texture (the camera texture chart - of ISO 19567-2) with sensor noise, same generator and seed: sharper than any real scene, the worst - case for the encoder. +- **Shared frame** (identical on both towers; used for the tower comparison): tower 121's frame of + 2 Oct (15:07, 20 ms, ≈ 15 dB), rebuilt bit-exactly from its lossless 12-bit file. Its 8-bit raw + version is the 12-bit value shifted down by 4 bits; its "RGB8" version is the edge-aware colour + conversion of the 12-bit frame, scaled to 8 bits (the camera's own RGB8 of that instant does not + exist). Identical MD5 checksums on both towers. +- **Synthetic frame** (identical on both towers): a "dead leaves" texture [1] with added sensor + noise; sharper than any real scene and therefore the worst case for the encoder. - Each tower's own camera capture from its test day. -**Quality.** Every 12-bit result is decoded and compared with the exact lossless 12-bit colour image -of the same frame (mosaic results are converted to colour the same way first): PSNR on the 12-bit -scale, and SSIM on luma, which follows structure and is less fooled by noise. +### 2.9 Image quality metrics -**Load.** *Back to back:* workers save the same frame from memory continuously for 60 s. -*Paced:* a producer queues one frame every 2.0, 1.5, 1.0, 0.7 or 0.5 s for 2 minutes, like a scan -keeping one frame per waypoint; each job runs every step above; recorded: time from queueing to file -written, frames still queued at the end, peak memory. *Heat:* 10 minutes flat out and 10 minutes at -one frame every 2 s, each after a 5-minute cool-down, with once a second: CPU package temperature -(`x86_pkg_temp`), average core clock, CPU busy share (`/proc/stat`) and package power (Intel RAPL -energy counter, read through a root helper). +Every 12-bit result was decoded and compared with the exact lossless 12-bit colour image of the same +frame (mosaic results first converted to colour in the same way): PSNR (dB) over all three channels on +the 12-bit scale, and SSIM [6] on luma. -### Noise test +### 2.10 Throughput, pile-up and heat -To see whether the 12th bit carries information, sensor noise was measured on the raw 12-bit -values: the green sites split into 24×24-pixel patches; in each, noise = the robust spread (median -absolute deviation) of differences between horizontal neighbours ÷ √2; patches binned by brightness; -the quietest 10% in each bin taken as the noise floor (texture only adds to it). Once for the -~15 dB frame and once for a frame taken at **0 dB gain** (124 ms exposure, 4 Oct 16:19, exposure -cap temporarily raised to 200 ms). +*Back to back:* save workers process the same frame from memory continuously for 60 s. +*Paced:* one frame is queued every 2.0, 1.5, 1.0, 0.7 or 0.5 s for 120 s, as a scan keeps one frame +per waypoint; recorded are the time from queueing to file written, the frames still queued at the end +("behind") and the peak memory. *Heat:* 10 minutes back to back and 10 minutes at one frame every 2 s, +each after a 5-minute cool-down; recorded once per second: CPU package temperature, average core +clock, CPU busy share and package power (Intel RAPL energy counter [7]). -### Campaign log +### 2.11 Noise measurement -| When | Tower | What | Duration | Notes | +Sensor noise was measured on the raw 12-bit values: the green sites were divided into 24 × 24-pixel +patches; in each, noise σ = median absolute deviation of differences between horizontal neighbours, +÷ √2; patches were grouped by brightness, and the quietest 10% of each group taken as the noise floor +(scene texture can only add to it). This was done for the ≈ 15 dB frame and for a frame taken at +**0 dB gain** (exposure 124 ms, 4 Oct 16:19, exposure cap temporarily raised to 200 ms). + +### 2.12 Campaign log + +***Table 8.** Campaign log.* + +| Date and time | Tower | Activity | Duration (min) | Notes | |---|---|---|---|---| -| 2 Oct 15:01–15:22 | 121 | camera tests | 21 min | sunny | -| 2 Oct 15:22–16:50 | 121 | processing, pile-up, heat on its own frame and the synthetic frame | 88 min | | -| 2 Oct 19:01 | 121 | attempt at a 0 dB frame | – | too dark (24 dB needed); discarded | -| 4 Oct 16:19 | 111 | 0 dB frame (camera moved from 121) | 2 min | | -| 4 Oct 16:20–16:41 | 111 | camera tests | 21 min | 111's own 1240c camera connected but idle | -| 4 Oct 16:41–18:19 | 111 | processing, pile-up, heat | 98 min | | -| 4 Oct 23:03–23:32 | 121 + 111 | processing on the shared frame, both towers at once | 29 min | no camera needed | +| 2 Oct 15:01–15:22 | 121 | camera tests | 21 | sunny | +| 2 Oct 15:22–16:50 | 121 | processing, pile-up, heat (own and synthetic frames) | 88 | | +| 2 Oct 19:01 | 121 | attempted 0 dB frame | – | too dark (24 dB needed); discarded | +| 4 Oct 16:19 | 111 | 0 dB frame (camera moved from 121) | 2 | | +| 4 Oct 16:20–16:41 | 111 | camera tests | 21 | tower 111's own camera connected but idle | +| 4 Oct 16:41–18:19 | 111 | processing, pile-up, heat | 98 | | +| 4 Oct 23:03–23:32 | 121 and 111 | processing on the shared frame, both towers simultaneously | 29 | no camera needed | -## Frame rate and streaming +--- + +## 3. Results + +### 3.1 Frame rate and streaming ![Frame rate per format](images/raw12/fps_per_format.png) -Same camera on both towers, so identical maximum rates. 60 s at the maximum: +***Figure 3.** Maximum full-resolution frame rate (fps) per pixel format at the 250 MB/s link cap. +The same camera was used on both towers.* -| Format | Frame | Max fps | 121: complete / incomplete / lost | 111: complete / incomplete / lost | +***Table 9.** 60 s streams at each format's maximum rate.* + +| Format | Frame size (MB) | Maximum rate (fps) | 121: complete / incomplete / lost (frames) | 111: complete / incomplete / lost (frames) | |---|---|---|---|---| -| BayerRG12p | 24.3 MB | 9.86 | 593 / 0 / 0 | 592 / 0 / 0 | -| BayerRG12 (unpacked) | 32.4 MB | 7.41 | 445 / 0 / 0 | 445 / 0 / 0 | -| BayerRG10p | 20.2 MB | 11.81 | 711 / 0 / 0 | 709 / 0 / 0 | -| BayerRG8 | 16.2 MB | 14.75 | 887 / 0 / 0 | 887 / 0 / 0 | -| RGB8 (today) | 48.6 MB | 4.95 | 298 / 0 / 0 | 297 / 0 / 0 | -| **BayerRG12p while saving 12-bit flat out** | 24.3 MB | 9.86 | 591 / 0 / 0 | 592 / 0 / 0 | +| BayerRG12p | 24.3 | 9.86 | 593 / 0 / 0 | 592 / 0 / 0 | +| BayerRG12 (unpacked) | 32.4 | 7.41 | 445 / 0 / 0 | 445 / 0 / 0 | +| BayerRG10p | 20.2 | 11.81 | 711 / 0 / 0 | 709 / 0 / 0 | +| BayerRG8 | 16.2 | 14.75 | 887 / 0 / 0 | 887 / 0 / 0 | +| RGB8 (today) | 48.6 | 4.95 | 298 / 0 / 0 | 297 / 0 / 0 | +| BayerRG12p while saving 12-bit continuously | 24.3 | 9.86 | 591 / 0 / 0 | 592 / 0 / 0 | -On tower 111 the camera shares the USB 3 controller with the 5G modem, a USB network adapter and -the tower's second camera (idle during the test). +No frame was lost or incomplete in any format on either tower. On tower 111 the camera shares the +USB 3 controller with a 5G modem, a USB network adapter and a second (idle) camera. -## Auto-exposure +### 3.2 Auto-exposure settling ![Brightness after a step, 121](images/raw12/ae_brightness_121.png) ![Brightness after a step, 111](images/raw12/ae_brightness_111.png) -With exposure at its 20 ms cap, auto-exposure works through gain, 1–2 dB per frame, and sometimes -pauses mid-way (the flat stretches, up to 2.8 s while still far too bright). +***Figure 4.** Frame brightness (DN, 0–255) after auto-exposure is released, for each format: (a, +top) tower 121; (b, bottom) tower 111. Left: from under-exposure (×1/16); right: from over-exposure +(×16). Dashed line: settled level.* -Time until brightness stays within ±10% of where it settles (two runs each): +With exposure at its 20 ms cap, auto-exposure adjusts through gain, by 1–2 dB per frame, and +sometimes pauses mid-way (flat segments in Figure 4, up to 2.8 s while still far too bright). -| Format | fps | Step | Tower 121 | Tower 111 | +***Table 10.** Time (s) and frames until brightness stays within ±10% of its final level; two runs +each.* + +| Format | Rate (fps) | Step | Tower 121: time (s) (frames) | Tower 111: time (s) (frames) | |---|---|---|---|---| -| BayerRG8 | 14.75 | dark ×1/16 | 1.94 s (28 frames), 2.21 s (32) | 1.79 s (26), 1.87 s (27) | -| BayerRG8 | 14.75 | bright ×16 | 2.69 s (39), 2.55 s (37) | 2.00 s (29), 1.86 s (27) | -| **BayerRG12p** | 9.86 | dark ×1/16 | 2.90 s (28), 3.00 s (29) | 6.64 s (65), 4.51 s (44) | -| **BayerRG12p** | 9.86 | bright ×16 | 6.45 s (63), 2.69 s (26) | 2.18 s (21), 5.73 s (56) | -| RGB8 (today) | 4.95 | dark ×1/16 | 4.56 s (22), 5.17 s (25) | 5.97 s (29), 5.77 s (28) | -| RGB8 (today) | 4.95 | bright ×16 | 5.17 s (25), 5.17 s (25) | 6.57 s (32), 4.96 s (24) | +| BayerRG8 | 14.75 | dark ×1/16 | 1.94 (28), 2.21 (32) | 1.79 (26), 1.87 (27) | +| BayerRG8 | 14.75 | bright ×16 | 2.69 (39), 2.55 (37) | 2.00 (29), 1.86 (27) | +| **BayerRG12p** | 9.86 | dark ×1/16 | 2.90 (28), 3.00 (29) | 6.64 (65), 4.51 (44) | +| **BayerRG12p** | 9.86 | bright ×16 | 6.45 (63), 2.69 (26) | 2.18 (21), 5.73 (56) | +| RGB8 (today) | 4.95 | dark ×1/16 | 4.56 (22), 5.17 (25) | 5.97 (29), 5.77 (28) | +| RGB8 (today) | 4.95 | bright ×16 | 5.17 (25), 5.17 (25) | 6.57 (32), 4.96 (24) | -Typically ~25–35 frames, so **settling time scales with 1/fps**. The longer 12-bit runs (44–65 -frames) are the mid-way pauses; they happened at every rate but hit the 12-bit runs more often here. +Settling typically takes 25–35 frames, so **settling time scales with 1/frame rate**. The longer +12-bit runs (44–65 frames) contain mid-way pauses, which occurred at every rate but more often in the +12-bit runs here. -### Which frame to keep +### 3.3 Keep rules -Each rule applied to the per-frame log of all 24 steps. "Correct" means the frame it picks is within -10% of the brightness the camera finally settles at. +***Table 11.** Candidate keep rules scored on the 24 steps (12 per tower). Correct: the kept frame's +brightness is within 10% of the final level.* -| Rule | 121 correct / early | 111 correct / early | Median time when correct (121 / 111) | +| Rule | 121 correct / early (runs) | 111 correct / early (runs) | Median time when correct, 121 / 111 (s) | |---|---|---|---| | Exposure and gain unchanged for 3 frames | 0 / 12 | 0 / 12 | – | -| Brightness in the camera's target band (target 50% ± 5%), 2 frames | 6 / 6 | 6 / 6 | 3.30 / 6.10 s | -| Target band 2 frames and exposure/gain unchanged 2 frames | 6 / 6 | 7 / 5 | 3.35 / 5.02 s | -| Target band 3 frames | 8 / 4 | 8 / 4 | 3.40 / 4.84 s | -| **Target band and brightness steady (≤2% change) over 3 frames** | **11 / 1** | **10 / 2** | 3.61 / 5.45 s | +| Brightness in the target band (50 ± 5%), 2 frames | 6 / 6 | 6 / 6 | 3.30 / 6.10 | +| Target band 2 frames and exposure/gain unchanged 2 frames | 6 / 6 | 7 / 5 | 3.35 / 5.02 | +| Target band 3 frames | 8 / 4 | 8 / 4 | 3.40 / 4.84 | +| **Target band and brightness steady (≤ 2% change) over 3 frames** | **11 / 1** | **10 / 2** | 3.61 / 5.45 | -Coming down from too bright, brightness enters the band about 15% above where it settles, so "in -band" alone fires early; requiring it to have stopped changing fixes most of that. +Descending from over-exposure, brightness enters the band about 15% above its final level, so "in +band" alone fires early; also requiring the brightness to have stopped changing corrects most of +these cases. -## Auto white balance +### 3.4 Auto white balance -The profiles fix white balance today (red 2.87, blue 2.0). With `BalanceWhiteAuto = Continuous`, the -camera settled this scene at red ≈ 2.36–2.46, blue ≈ 1.87–1.97. +The profiles currently fix white balance (red 2.87, blue 2.0). With `BalanceWhiteAuto` = +`Continuous` the camera settled this scene at red ≈ 2.36–2.46, blue ≈ 1.87–1.97. ![Auto white balance, 121](images/raw12/awb_121.png) ![Auto white balance, 111](images/raw12/awb_111.png) -| Format | fps | Within ±5% and stays, 121 | Within ±5% and stays, 111 | +***Figure 5.** White-balance ratios relative to their settled values after release, for each +format: (a, top) tower 121; (b, bottom) tower 111. Left: ratios forced to ×0.5; right: ×2. Shaded: +±5% band.* + +***Table 12.** Time until both ratios are within ±5% of the reference and remain there.* + +| Format | Rate (fps) | Tower 121: time (s) (frames) | Tower 111: time (s) (frames) | |---|---|---|---| -| BayerRG12p | 9.86 | 0.21–0.32 s (2–3 frames); once 1.33 s | 0.22–0.41 s (2–4) | -| RGB8 | 4.95 | 0.57–0.77 s (2–3) | 0.55–0.76 s (2–3) | -| BayerRG8 | 14.75 | 0.26–0.27 s (3) | 0.19–0.26 s (2–3) | +| BayerRG12p | 9.86 | 0.21–0.32 (2–3); once 1.33 | 0.22–0.41 (2–4) | +| RGB8 | 4.95 | 0.57–0.77 (2–3) | 0.55–0.76 (2–3) | +| BayerRG8 | 14.75 | 0.26–0.27 (3) | 0.19–0.26 (2–3) | -On a still scene the auto ratios held within ±0.8–3.1%. +On a still scene the automatic ratios held within ±0.8–3.1%. White balance settles much faster than +exposure. -## Where the time goes - -### Same frame on both computers - -The fairest comparison: the **identical shared frame** (checksums verified on both towers), every -step timed on its own, 4 threads. Colour paths at distance 1.0 and effort 3; today's RGB8 path at -its production settings (distance 0.8, effort 4). +### 3.5 Processing time per step ![Same frame, time per step](images/raw12/stages_common.png) -| Step | IOTA (121) | 3 Delta (111) | Delta / IOTA | +***Figure 6.** Time per saved frame (ms) by step on the identical shared frame, for both computers; +4 threads; colour paths at d1.0 e3, today's RGB8 path at d0.8 e4.* + +***Table 13.** Duration of each step on the shared frame (4 threads).* + +| Step | IOTA (ms) | Delta (ms) | Delta / IOTA | |---|---|---|---| -| Unpack 12-bit packed → 16-bit | 10 ms | 17 ms | 1.7× | -| Colour conversion, 8-bit raw, bilinear / edge-aware | 8 / 9 ms | 11 / 12 ms | 1.3× | -| Colour conversion, 12-bit raw, bilinear / edge-aware | 17 / 28 ms | 22 / 33 ms | 1.2× | -| JPEG XL encode, 12-bit colour, d1.0 e3 | 480 ms | 765 ms | 1.6× | -| JPEG XL encode, today's RGB8, d0.8 e4 | 566 ms | 867 ms | 1.5× | -| File write, 2.7 MB colour file | 1 ms | 4 ms | | -| File write, 17–21 MB lossless raw file | 7–8 ms | 162–173 ms | (storage: 218 vs 76 MB/s) | +| Unpack 12-bit packed → 16-bit | 10 | 17 | 1.7 | +| Colour conversion, 8-bit raw, bilinear / edge-aware | 8 / 9 | 11 / 12 | 1.3 | +| Colour conversion, 12-bit raw, bilinear / edge-aware | 17 / 28 | 22 / 33 | 1.2 | +| JPEG XL encode, 12-bit colour, d1.0 e3 | 480 | 765 | 1.6 | +| JPEG XL encode, today's RGB8, d0.8 e4 | 566 | 867 | 1.5 | +| File write, 2.7 MB colour file | 1 | 4 | | +| File write, 17–21 MB lossless raw file | 7–8 | 162–173 | (storage 218 vs 76 MB/s) | -Per saved frame, all steps together, and the file each produces (sizes are identical on both -computers, as they must be for the same input): +***Table 14.** Total time per saved frame and resulting file size, shared frame (file sizes are +identical on both computers).* -| Path | IOTA total | Delta total | File | +| Path | IOTA (ms) | Delta (ms) | File size (MB) | |---|---|---|---| -| Today: RGB8 → JPEG XL, d0.8 e4 | 567 ms | 872 ms | 3.27 MB | -| Raw 8 → colour → JPEG XL, d1.0 e3 | 468 ms | 741 ms | 2.66 MB | -| Raw 12 → colour (bilinear) → JPEG XL, d1.0 e3 | 476 ms | 778 ms | 2.53 MB | -| **Raw 12 → colour (edge-aware) → JPEG XL, d1.0 e3** | **518 ms** | **820 ms** | **2.65 MB** | -| Raw 12 planes, lossless (e1) | 91 ms | 297 ms | 17.65 MB | -| Raw 12 mosaic, lossless (e1) | 87 ms | 303 ms | 20.94 MB | +| Today: RGB8 → JPEG XL, d0.8 e4 | 567 | 872 | 3.27 | +| Raw 8-bit → colour → JPEG XL, d1.0 e3 | 468 | 741 | 2.66 | +| Raw 12-bit → colour (bilinear) → JPEG XL, d1.0 e3 | 476 | 778 | 2.53 | +| **Raw 12-bit → colour (edge-aware) → JPEG XL, d1.0 e3** | **518** | **820** | **2.65** | +| Raw 12-bit planes, lossless, e1 | 91 | 297 | 17.65 | +| Raw 12-bit mosaic, lossless, e1 | 87 | 303 | 20.94 | -Unpacking and colour conversion together cost **38 ms on the IOTA and 50 ms on the Delta**, 7% of -the save. The JPEG XL encode is 90% of it. +Unpacking and colour conversion together cost 38 ms on the IOTA and 50 ms on the Delta (7% of the +save); the JPEG XL encode accounts for 90%. Here the "RGB8" input is the tower's own colour +conversion of the frame, so its file equals the raw 8-bit path's; the camera's real RGB8 files were +somewhat larger (3.7–4.0 MB, Appendix B). -The "RGB8" input here is the edge-aware colour conversion of the same frame (the camera's own RGB8 -of that instant does not exist), so its file matches the raw 8-bit path's. The camera's real RGB8 -files were a little larger in the per-tower runs below (3.7–4.0 MB). - -### Size and time per quality setting, same frame - -![Same frame, size and time by quality](images/raw12/sizes_common.png) - -| Distance | File (12-bit colour) | IOTA | Delta | File (raw 12 planes) | -|---|---|---|---|---| -| lossless | 44.8 MB | 1,225 ms | 2,906 ms | 16.8 MB (455 / 860 ms) | -| 0.3 | 7.19 MB | 742 ms | 1,197 ms | 7.31 MB | -| 0.5 | 5.15 MB | 678 ms | 1,033 ms | 5.99 MB | -| 0.8 | 3.16 MB | 570 ms | 848 ms | 4.32 MB | -| **1.0** | **2.65 MB** | **558 ms** | **847 ms** | 3.78 MB | -| 1.5 | 1.63 MB | 515 ms | 751 ms | 2.44 MB | -| 2.0 | 1.32 MB | 558 ms | 738 ms | 1.96 MB | - -(Effort 3, 4 threads, file write included. A separate pass of 2 runs per point, so it differs from -the stage table above by run-to-run variation, up to ~40 ms.) - -### Each tower's own capture - -Same camera and scene, different day and light; consistent with the shared-frame result. - -![Stage times, real frame](images/raw12/stages_real.png) -![Stage times, synthetic frame](images/raw12/stages_synth.png) - -| Path (effort 3, 4 threads, d1.0) | IOTA, own frame / synthetic | Delta, own frame / synthetic | -|---|---|---| -| Today: camera RGB8 | 463 / 652 ms | 788 / 1,071 ms | -| Raw 12 → colour (edge-aware) | 569 / 727 ms | 849 / 1,088 ms | -| Raw 12 planes, lossless (e1) | 90 / 96 ms | 276 / 298 ms | - -### Effort and threads +### 3.6 Effort and threads ![Effort × threads, 121](images/raw12/effort_threads_121.png) ![Effort × threads, 111](images/raw12/effort_threads_111.png) -- Effort 1–4 are within ~15% of each other; **effort 3 gives the smallest files**. -- On the IOTA, **3 threads is slower than 2** at every effort (e.g. 761 / 582 / 672 / 512 ms for - 1–4 threads at effort 1); the Delta scales normally (1,440 / 985 / 886 / 802 ms). 4 is fastest on - both computers. +***Figure 7.** Time per frame (ms) by JPEG XL effort (rows) and thread count (columns), each tower's +real frame, distance 1.0: (a, top) tower 121; (b, bottom) tower 111.* -## Quality setting: size and quality +Effort 1–4 lie within about 15% of each other, and effort 3 gives the smallest files. On the IOTA, +3 threads is slower than 2 at every effort (761 / 582 / 672 / 512 ms for 1–4 threads at effort 1); +the Delta scales normally (1440 / 985 / 886 / 802 ms). Four threads is fastest on both computers. + +### 3.7 Quality setting: size, time and quality + +![Same frame, size and time by quality](images/raw12/sizes_common.png) + +***Figure 8.** File size (MB, left) and time per frame (ms, right) versus JPEG XL distance on the +shared frame, effort 3, 4 threads, both towers.* + +***Table 15.** Size and time per distance on the shared frame (effort 3, 4 threads, file write +included; 2 runs per point).* + +| Distance | 12-bit colour size (MB) | IOTA time (ms) | Delta time (ms) | Raw 12-bit planes size (MB) | +|---|---|---|---|---| +| 0 (lossless) | 44.8 | 1225 | 2906 | 16.8 (IOTA 455 ms, Delta 860 ms) | +| 0.3 | 7.19 | 742 | 1197 | 7.31 | +| 0.5 | 5.15 | 678 | 1033 | 5.99 | +| 0.8 | 3.16 | 570 | 848 | 4.32 | +| **1.0** | **2.65** | **558** | **847** | 3.78 | +| 1.5 | 1.63 | 515 | 751 | 2.44 | +| 2.0 | 1.32 | 558 | 738 | 1.96 | ![Size and quality, real frame](images/raw12/distance_real.png) -Against the exact lossless 12-bit colour image of the same frame (real plant frame, ~15 dB gain; -the two towers agree within 0.3 dB): +***Figure 9.** File size (MB), PSNR (dB) and fine-detail retention against each method's own +uncompressed image, versus distance; each tower's real frame.* -| Saved as | File 121 / 111 | PSNR | SSIM | +***Table 16.** Quality against the exact lossless 12-bit colour image of the same frame (real frame, +≈ 15 dB gain; the towers agree within 0.3 dB).* + +| Saved as | File size, 121 / 111 (MB) | PSNR (dB) | SSIM | |---|---|---|---| -| 12-bit colour, distance 0.3 | 7.2 / 7.4 MB | 40.3 dB | 0.994 | -| 12-bit colour, distance 0.5 | 5.1 / 5.3 MB | 38.8 dB | 0.989 | -| 12-bit colour, distance 0.8 | 3.2 / 3.3 MB | 37.1 dB | 0.978 | -| **12-bit colour, distance 1.0** | **2.6 / 2.7 MB** | 36.6 dB | 0.973 | -| 12-bit colour, distance 1.5 | 1.6 / 1.7 MB | 35.4 dB | 0.960 | -| 12-bit colour, distance 2.0 | 1.3 / 1.3 MB | 34.8 dB | 0.953 | -| 12-bit colour, bilinear, distance 1.0 | 2.5 / 2.6 MB | 36.6 dB | 0.968 | -| 12-bit raw planes, lossy 0.5 | 6.0 / 6.0 MB | **47.2 dB** | 0.990 | -| 12-bit raw planes, lossless | 17.6 / 17.7 MB | exact | 1 | -| 12-bit raw mosaic, lossless | 20.9 / 19.0 MB | exact | 1 | -| 12-bit colour, lossless | 47.4 / 46.3 MB | exact | 1 | +| 12-bit colour, d0.3 | 7.2 / 7.4 | 40.3 | 0.994 | +| 12-bit colour, d0.5 | 5.1 / 5.3 | 38.8 | 0.989 | +| 12-bit colour, d0.8 | 3.2 / 3.3 | 37.1 | 0.978 | +| **12-bit colour, d1.0** | **2.6 / 2.7** | 36.6 | 0.973 | +| 12-bit colour, d1.5 | 1.6 / 1.7 | 35.4 | 0.960 | +| 12-bit colour, d2.0 | 1.3 / 1.3 | 34.8 | 0.953 | +| 12-bit colour, bilinear, d1.0 | 2.5 / 2.6 | 36.6 | 0.968 | +| 12-bit raw planes, d0.5 | 6.0 / 6.0 | **47.2** | 0.990 | +| 12-bit raw planes, lossless | 17.6 / 17.7 | exact | 1 | +| 12-bit raw mosaic, lossless | 20.9 / 19.0 | exact | 1 | +| 12-bit colour, lossless | 47.4 / 46.3 | exact | 1 | -- **Colour JPEG XL is tuned to the eye:** it drops sensor noise and fine colour, so PSNR is - moderate while SSIM (structure) stays high. -- **Raw planes keep the sensor values much closer** at the same size (47 vs 39 dB, equal SSIM). - They suit a detector that wants raw values and can convert to colour itself. Lossless, they take - a third of the space of lossless colour. +Colour JPEG XL is tuned to human vision: it discards sensor noise and fine colour, so PSNR is +moderate while SSIM stays high. Raw planes preserve the sensor values much more closely at the same +size (47 vs 39 dB, equal SSIM); lossless, they take a third of the space of lossless colour. -## Example images +### 3.8 Example images -Every compression type, from tower 121's frame: a 100% crop of spruce needles over bright leaves; -its difference from lossless (×8, black = identical); and a darker crop brightened 2 stops. -Full-size `.jxl` files are kept outside git in `tools/camera-bench/examples-2026-10-02-tower121/`, -`examples-2026-10-04-tower111/` and `examples-2026-10-04-lowgain/` (~115 MB each). -Crop positions: ![crops](images/raw12/examples/121/crops.jpg) +![crops](images/raw12/examples/121/crops.jpg) -| Saved as | File, time (121) | Detail (100%) | Difference ×8 | Shadow, +2 stops | +***Figure 10.** Positions of the detail crop (red) and shadow crop (blue) used in Figure 11.* + +***Figure 11.** One example per compression method, tower 121's frame (480 × 320-pixel crops at +100%): detail crop of spruce needles over bright leaves; its absolute difference from the lossless +image (amplified ×8; black = identical); and a darker crop brightened by 2 stops. File size and save +time are single runs at 4 threads. The two 8-bit rows are separate captures seconds apart (branches +move), so they have no difference map.* + +| Method | File size (MB), time (s) | Detail (100%) | Difference (×8) | Shadow (+2 stops) | |---|---|---|---|---| -| Camera RGB8, d0.8 e4 (today; another capture) | 3.8 MB, 0.62 s | ![](images/raw12/examples/121/detail/rgb8_d0.8_e4_today.png) | – | ![](images/raw12/examples/121/shadow/rgb8_d0.8_e4_today.png) | -| Raw 8 → colour, d1.0 (another capture) | 2.7 MB, 0.45 s | ![](images/raw12/examples/121/detail/c8_ea_d1.0.png) | – | ![](images/raw12/examples/121/shadow/c8_ea_d1.0.png) | -| 12-bit colour, lossless (reference) | 47.4 MB, 0.23 s | ![](images/raw12/examples/121/detail/c12_ea_lossless.png) | ![](images/raw12/examples/121/diff/c12_ea_lossless.png) | ![](images/raw12/examples/121/shadow/c12_ea_lossless.png) | -| 12-bit colour, d0.3 | 7.2 MB, 0.77 s | ![](images/raw12/examples/121/detail/c12_ea_d0.3.png) | ![](images/raw12/examples/121/diff/c12_ea_d0.3.png) | ![](images/raw12/examples/121/shadow/c12_ea_d0.3.png) | -| 12-bit colour, d0.5 | 5.1 MB, 0.69 s | ![](images/raw12/examples/121/detail/c12_ea_d0.5.png) | ![](images/raw12/examples/121/diff/c12_ea_d0.5.png) | ![](images/raw12/examples/121/shadow/c12_ea_d0.5.png) | -| 12-bit colour, d0.8 | 3.2 MB, 0.58 s | ![](images/raw12/examples/121/detail/c12_ea_d0.8.png) | ![](images/raw12/examples/121/diff/c12_ea_d0.8.png) | ![](images/raw12/examples/121/shadow/c12_ea_d0.8.png) | -| **12-bit colour, d1.0** | 2.6 MB, 0.53 s | ![](images/raw12/examples/121/detail/c12_ea_d1.0.png) | ![](images/raw12/examples/121/diff/c12_ea_d1.0.png) | ![](images/raw12/examples/121/shadow/c12_ea_d1.0.png) | -| 12-bit colour, d1.5 | 1.6 MB, 0.46 s | ![](images/raw12/examples/121/detail/c12_ea_d1.5.png) | ![](images/raw12/examples/121/diff/c12_ea_d1.5.png) | ![](images/raw12/examples/121/shadow/c12_ea_d1.5.png) | -| 12-bit colour, d2.0 | 1.3 MB, 0.42 s | ![](images/raw12/examples/121/detail/c12_ea_d2.0.png) | ![](images/raw12/examples/121/diff/c12_ea_d2.0.png) | ![](images/raw12/examples/121/shadow/c12_ea_d2.0.png) | -| 12-bit colour, bilinear, d1.0 | 2.5 MB, 0.48 s | ![](images/raw12/examples/121/detail/c12_bil_d1.0.png) | ![](images/raw12/examples/121/diff/c12_bil_d1.0.png) | ![](images/raw12/examples/121/shadow/c12_bil_d1.0.png) | -| 12-bit raw planes, d0.5 | 6.0 MB, 0.54 s | ![](images/raw12/examples/121/detail/raw12_planes_d0.5.png) | ![](images/raw12/examples/121/diff/raw12_planes_d0.5.png) | ![](images/raw12/examples/121/shadow/raw12_planes_d0.5.png) | -| 12-bit raw planes, lossless | 17.6 MB, 0.09 s | ![](images/raw12/examples/121/detail/raw12_planes_lossless.png) | ![](images/raw12/examples/121/diff/raw12_planes_lossless.png) | ![](images/raw12/examples/121/shadow/raw12_planes_lossless.png) | -| 12-bit raw mosaic, lossless | 20.9 MB, 0.09 s | ![](images/raw12/examples/121/detail/raw12_lossless.png) | ![](images/raw12/examples/121/diff/raw12_lossless.png) | ![](images/raw12/examples/121/shadow/raw12_lossless.png) | +| Camera RGB8, d0.8 e4 (today) | 3.8, 0.62 | ![](images/raw12/examples/121/detail/rgb8_d0.8_e4_today.png) | – | ![](images/raw12/examples/121/shadow/rgb8_d0.8_e4_today.png) | +| Raw 8-bit → colour, d1.0 | 2.7, 0.45 | ![](images/raw12/examples/121/detail/c8_ea_d1.0.png) | – | ![](images/raw12/examples/121/shadow/c8_ea_d1.0.png) | +| 12-bit colour, lossless (reference) | 47.4, 0.23 | ![](images/raw12/examples/121/detail/c12_ea_lossless.png) | ![](images/raw12/examples/121/diff/c12_ea_lossless.png) | ![](images/raw12/examples/121/shadow/c12_ea_lossless.png) | +| 12-bit colour, d0.3 | 7.2, 0.77 | ![](images/raw12/examples/121/detail/c12_ea_d0.3.png) | ![](images/raw12/examples/121/diff/c12_ea_d0.3.png) | ![](images/raw12/examples/121/shadow/c12_ea_d0.3.png) | +| 12-bit colour, d0.5 | 5.1, 0.69 | ![](images/raw12/examples/121/detail/c12_ea_d0.5.png) | ![](images/raw12/examples/121/diff/c12_ea_d0.5.png) | ![](images/raw12/examples/121/shadow/c12_ea_d0.5.png) | +| 12-bit colour, d0.8 | 3.2, 0.58 | ![](images/raw12/examples/121/detail/c12_ea_d0.8.png) | ![](images/raw12/examples/121/diff/c12_ea_d0.8.png) | ![](images/raw12/examples/121/shadow/c12_ea_d0.8.png) | +| **12-bit colour, d1.0** | 2.6, 0.53 | ![](images/raw12/examples/121/detail/c12_ea_d1.0.png) | ![](images/raw12/examples/121/diff/c12_ea_d1.0.png) | ![](images/raw12/examples/121/shadow/c12_ea_d1.0.png) | +| 12-bit colour, d1.5 | 1.6, 0.46 | ![](images/raw12/examples/121/detail/c12_ea_d1.5.png) | ![](images/raw12/examples/121/diff/c12_ea_d1.5.png) | ![](images/raw12/examples/121/shadow/c12_ea_d1.5.png) | +| 12-bit colour, d2.0 | 1.3, 0.42 | ![](images/raw12/examples/121/detail/c12_ea_d2.0.png) | ![](images/raw12/examples/121/diff/c12_ea_d2.0.png) | ![](images/raw12/examples/121/shadow/c12_ea_d2.0.png) | +| 12-bit colour, bilinear, d1.0 | 2.5, 0.48 | ![](images/raw12/examples/121/detail/c12_bil_d1.0.png) | ![](images/raw12/examples/121/diff/c12_bil_d1.0.png) | ![](images/raw12/examples/121/shadow/c12_bil_d1.0.png) | +| 12-bit raw planes, d0.5 | 6.0, 0.54 | ![](images/raw12/examples/121/detail/raw12_planes_d0.5.png) | ![](images/raw12/examples/121/diff/raw12_planes_d0.5.png) | ![](images/raw12/examples/121/shadow/raw12_planes_d0.5.png) | +| 12-bit raw planes, lossless | 17.6, 0.09 | ![](images/raw12/examples/121/detail/raw12_planes_lossless.png) | ![](images/raw12/examples/121/diff/raw12_planes_lossless.png) | ![](images/raw12/examples/121/shadow/raw12_planes_lossless.png) | +| 12-bit raw mosaic, lossless | 20.9, 0.09 | ![](images/raw12/examples/121/detail/raw12_lossless.png) | ![](images/raw12/examples/121/diff/raw12_lossless.png) | ![](images/raw12/examples/121/shadow/raw12_lossless.png) | -The two 8-bit rows are separate captures seconds apart (branches move), so they have no difference -map. Times are single runs at 4 threads. Tower 111's examples (same camera and scene) and the -0 dB examples are kept as full-size `.jxl` only; `tools/camera-bench/make_figures.py` makes the same -crops from them. +Full-size `.jxl` files are kept outside the repository in `tools/camera-bench/examples-*` +(about 115 MB per set), including tower 111's and the 0 dB examples. -## What 12-bit adds - -To see whether the extra 4 bits carry information or noise, a frame was also taken at **0 dB gain** -(124 ms exposure, 4 Oct). The noise floor, from the quietest patches at each brightness: +### 3.9 Information content of the 12-bit data ![Noise vs brightness](images/raw12/noise_vs_signal.png) -| Signal (12-bit, green) | Noise at 0 dB | Noise at ~15 dB | In 8-bit steps (0 dB / 15 dB) | +***Figure 12.** Sensor noise σ (DN, 12-bit) versus signal (DN, 12-bit), green sites, at 0 dB gain +and at ≈ 15 dB gain. Dashed: one 8-bit quantisation step (16 DN); dotted: the RMS error added by +rounding to 8 bits (4.6 DN).* + +***Table 17.** Noise floor at three signal levels.* + +| Signal (DN, 12-bit) | Noise at 0 dB (DN) | Noise at ≈ 15 dB (DN) | Noise in 8-bit steps, 0 dB / 15 dB | |---|---|---|---| -| ~700 (darkest in this scene) | 26 | 42 | 1.6 / 2.6 | -| ~1,500 | 35 | 58 | 2.2 / 3.6 | -| ~3,000 | 37 | 80 | 2.3 / 5.0 | +| ≈ 700 (darkest in this scene) | 26 | 42 | 1.6 / 2.6 | +| ≈ 1500 | 35 | 58 | 2.2 / 3.6 | +| ≈ 3000 | 37 | 80 | 2.3 / 5.0 | -Rounding to 8 bits adds an error of about 4.6 (12-bit units): small next to this noise. **In -daylight, 8-bit already holds what the sensor delivers.** By extrapolation, 12-bit starts to carry -information below about 300 (0 dB) or 100 (15 dB) of 4095, i.e. in deep shadow: dusk, night, the -dark side of a ridge. That is where a 12-bit night profile would pay off. Not measured at night. - -Same frame, a darker crop brightened 2 stops, 8-bit (left, the 12-bit value >> 4) vs 12-bit (right), -at ~15 dB and at 0 dB. 3,138 vs 227 distinct levels at 15 dB, yet no visible difference: +Rounding to 8 bits adds an RMS error of 4.6 DN (12-bit scale), small compared with this noise. In +this daylight scene, 8-bit data therefore already holds the information the sensor delivers. By +extrapolation, 12-bit data carries additional information below about 300 DN (0 dB) or 100 DN +(15 dB), i.e. in deep shadow: at dusk, at night, or on the dark side of a ridge. ![8 vs 12 bit at 15 dB](images/raw12/examples/121/shadow_8_vs_12.png) ![8 vs 12 bit at 0 dB](images/raw12/examples/lowgain/shadow_8_vs_12.png) -At 0 dB the same settings give **25–30% smaller files and ~3 dB higher PSNR** (less noise to -encode): 12-bit colour at distance 1.0 is 1.9 MB, PSNR 40.0 dB, SSIM 0.987. +***Figure 13.** The same darker crop brightened by 2 stops, 8-bit (left; the 12-bit value shifted +down by 4 bits) versus 12-bit (right): (a, top) ≈ 15 dB gain, 3138 vs 227 distinct green levels; +(b, bottom) 0 dB gain. No visible difference.* -## Overall pace: how fast can a tower capture? +At 0 dB gain the same settings produce 25–30% smaller files and about 3 dB higher PSNR (less noise +to encode): 12-bit colour at d1.0 is 1.9 MB, PSNR 40.0 dB, SSIM 0.987. -A waypoint goes through these steps; saving runs in the background, overlapping the next move: - -| Step | 12-bit raw (proposed) | Raw 8-bit | RGB8 (today) | Source | -|---|---|---|---|---| -| Camera delivers a frame every | 0.10 s (9.86 fps) | 0.07 s (14.75 fps) | 0.20 s (4.95 fps) | measured | -| Slew to the next waypoint | ~1–1.5 s (6° step at 5–8°/s) | same | same | estimate from the scan speed; not measured here | -| Mechanical settle after standstill | ~0.2 s | same | same | to be measured in the field | -| Exposure settle after a **large** change (×16) | ~3 s (2.2–6.6) | ~2 s (1.8–2.7) | ~5 s (4.6–6.6) | measured, 24 steps per tower | -| Exposure settle after a small change (neighbouring waypoints) | well under the above | | | not measured | -| **Save, one worker × 4 threads (IOTA / Delta)** | **0.56 / 0.84 s** | 0.52 / 0.85 s | 0.59 / 0.88 s | measured, shared frame, back to back | -| **Save, two workers × 2 threads (IOTA / Delta)** | **0.47 / 0.65 s** | 0.42 / 0.60 s | 0.43 / 0.64 s | measured, shared frame, back to back | +### 3.10 Saving throughput and pile-up ![Saves per second, same frame](images/raw12/pace_common.png) -- **The camera side sets the pace.** A waypoint needs roughly 1.5–5 s (move, then exposure), - while a 12-bit save takes 0.56 s on the IOTA and 0.65 s on the Delta with two workers. Saving - uses a fraction of the time between waypoints on either computer. -- **Saving becomes the limit only below about one waypoint per 0.6–0.85 s**, which a real scan does - not reach. On the shared frame, the IOTA kept up with a frame every 0.7 s with one worker; the - Delta needed two workers for 0.7 s (with one it fell 27 frames behind in 2 minutes). -- Settling after small changes, the common case between neighbouring waypoints, still needs a field - measurement; it is what decides the real scan pace. - -## Keeping up with the scan +***Figure 14.** Frames saved per second, back to back, on the shared frame, for one worker × 4 +threads and two workers × 2 threads.* ![Pile-up](images/raw12/pileup.png) -Synthetic detailed frame (worst case). *ok* = no backlog after 2 minutes; otherwise frames behind -and the worst capture-to-file time: +***Figure 15.** Worst time from queueing to file written (s, logarithmic) versus the interval +between queued frames (s), synthetic frame. A flat line means saving keeps up; a rising line means a +backlog builds.* -| Path | Workers × threads | Tower | Saves/s back to back | One frame every 2.0 / 1.5 / 1.0 / 0.7 / 0.5 s | Peak memory | +***Table 18.** Throughput and pile-up on the synthetic frame (worst case). "ok" = no backlog after +120 s; otherwise frames behind at the end.* + +| Path | Workers × threads | Tower | Back to back (frames/s) | One frame every 2.0 / 1.5 / 1.0 / 0.7 / 0.5 s | Peak memory (GB) | |---|---|---|---|---|---| -| Today: camera RGB8, d0.8 e4 | 1 × 4 | 121 | 1.33 | ok / – / ok / **3 behind** / – | 0.5 GB | -| Today: camera RGB8, d0.8 e4 | 1 × 4 | 111 | 0.88 | ok / – / **12 behind** / **65 behind** / – | 0.5 GB | -| **12-bit colour, d1.0 e3** | 1 × 4 | 121 | 1.44 | ok / ok / ok / **ok** / – | 0.7 GB | -| **12-bit colour, d1.0 e3** | 1 × 4 | 111 | 0.93 | ok / **ok** / **8 behind** / 60 behind / – | 0.7 GB | -| **12-bit colour, d1.0 e3** | 2 × 2 | 121 | 1.74 | – / – / ok / ok / **34 behind** | 1.1 GB | -| **12-bit colour, d1.0 e3** | 2 × 2 | 111 | 1.18 | – / – / **ok** / **28 behind** / 97 behind | 1.1 GB | -| 12-bit colour, d1.0 e3 | 4 × 1 | 121 / 111 | 1.99 / 1.37 | (back to back only; each save waits 2–3 s) | 2.0 GB | -| 12-bit colour, d0.5 e3 | 1 × 4 | 121 | 1.19 | – / – / ok / 24 behind / – | 0.8 GB | -| 12-bit colour, d0.5 e3 | 1 × 4 | 111 | 0.77 | – / – / 27 behind / 79 behind / – | 0.7 GB | -| 12-bit raw mosaic, lossless | 1 × 4 | 121 / 111 | 10.2 / 1.8 | ok at 1.0 and 0.7 on both | 0.3 GB | +| Today: camera RGB8, d0.8 e4 | 1 × 4 | 121 | 1.33 | ok / – / ok / **3 behind** / – | 0.5 | +| Today: camera RGB8, d0.8 e4 | 1 × 4 | 111 | 0.88 | ok / – / **12 behind** / **65 behind** / – | 0.5 | +| **12-bit colour, d1.0 e3** | 1 × 4 | 121 | 1.44 | ok / ok / ok / **ok** / – | 0.7 | +| **12-bit colour, d1.0 e3** | 1 × 4 | 111 | 0.93 | ok / **ok** / **8 behind** / 60 behind / – | 0.7 | +| **12-bit colour, d1.0 e3** | 2 × 2 | 121 | 1.74 | – / – / ok / ok / **34 behind** | 1.1 | +| **12-bit colour, d1.0 e3** | 2 × 2 | 111 | 1.18 | – / – / **ok** / **28 behind** / 97 behind | 1.1 | +| 12-bit colour, d1.0 e3 | 4 × 1 | 121 / 111 | 1.99 / 1.37 | (back to back only; each save waits 2–3 s) | 2.0 | +| 12-bit colour, d0.5 e3 | 1 × 4 | 121 | 1.19 | – / – / ok / 24 behind / – | 0.8 | +| 12-bit colour, d0.5 e3 | 1 × 4 | 111 | 0.77 | – / – / 27 behind / 79 behind / – | 0.7 | +| 12-bit raw mosaic, lossless | 1 × 4 | 121 / 111 | 10.2 / 1.8 | ok at 1.0 and 0.7 on both | 0.3 | -On the real plant frame (lighter), one worker kept up with a frame every 1.0 s on both towers -(saved within 0.63 s on 121, 0.96 s on 111). +On the shared real frame, back to back, a 12-bit save took 0.56 s (IOTA, 1 × 4) and 0.65 s (Delta, +2 × 2); the IOTA kept up with a frame every 0.7 s with one worker, the Delta with two (with one it +fell 27 frames behind in 120 s). -## Heat and power +### 3.11 Heat and power ![Flat out](images/raw12/heat_max.png) ![Paced](images/raw12/heat_paced.png) -10 minutes each: +***Figure 16.** CPU package temperature (°C) and package power (W) over 10 minutes of 12-bit saving: +(a, top) back to back with 2 × 2 workers; (b, bottom) one frame every 2 s with 1 × 4. Solid: tower +121; dashed: tower 111.* -| Load | Tower | Temp mean / max | Clock | CPU busy | Power | Saves per minute, first → last | +***Table 19.** Temperature, clock, load and power, 10 minutes per condition.* + +| Load | Tower | Temperature mean / max (°C) | Clock (GHz) | CPU busy (%) | Power (W) | Saves per minute, first → last minute | |---|---|---|---|---|---|---| -| Idle | 121 | 38 / 40 °C | 0.79 GHz | 2% | 2.2 W | – | -| Idle | 111 | 52 / 54 °C | 0.96 GHz | 4% | 2.9 W | – | -| 12-bit, one frame every 2 s (1 × 4) | 121 | **53 / 84 °C** | 1.71 GHz | 21% | **5.1 W** | 30 → 29 | -| 12-bit, one frame every 2 s (1 × 4) | 111 | **66 / 86 °C** | 1.84 GHz | 33% | **5.7 W** | 30 → 30 | -| 12-bit, flat out (2 × 2) | 121 | 80 / 87 °C | 2.88 GHz | 81% | 12.5 W | 122 → 103 (−16%) | -| 12-bit, flat out (2 × 2) | 111 | 78 / 83 °C | 2.44 GHz | 81% | 8.1 W | 74 → 73 | +| Idle | 121 | 38 / 40 | 0.79 | 2 | 2.2 | – | +| Idle | 111 | 52 / 54 | 0.96 | 4 | 2.9 | – | +| 12-bit, one frame every 2 s (1 × 4) | 121 | **53 / 84** | 1.71 | 21 | **5.1** | 30 → 29 | +| 12-bit, one frame every 2 s (1 × 4) | 111 | **66 / 86** | 1.84 | 33 | **5.7** | 30 → 30 | +| 12-bit, back to back (2 × 2) | 121 | 80 / 87 | 2.88 | 81 | 12.5 | 122 → 103 (−16%) | +| 12-bit, back to back (2 × 2) | 111 | 78 / 83 | 2.44 | 81 | 8.1 | 74 → 73 | -At scan pace both computers stay far from their limits. The Delta runs ~13 °C warmer; flat out it -sits at an 8 W power limit, so it does not slow further, while the IOTA starts fast (12.5 W) and -settles 16% lower as it heats. +At scan pace both computers stay far from their limits; the Delta runs about 13 °C warmer. Back to +back, the Delta sits at an 8 W power limit and does not slow further, whereas the IOTA starts at +12.5 W and loses 16% of its throughput as it heats. -## Storage +### 3.12 Storage ![Storage per day](images/raw12/storage.png) -Real plant frame sizes, 24 hours of scanning: +***Figure 17.** Storage per day (GB) at one frame every 2 s, per saving method, each tower's real +frame.* -| Saved as | Per frame (121 / 111) | One frame / 2 s | / 5 s | / 10 s | +***Table 20.** Storage for 24 hours of scanning at three frame intervals.* + +| Saved as | Per frame, 121 / 111 (MB) | Every 2 s (GB/day) | Every 5 s (GB/day) | Every 10 s (GB/day) | |---|---|---|---|---| -| Camera RGB8, d0.8 (today's quality) | 3.7 / 4.0 MB | 160–172 GB | 64–69 GB | 32–34 GB | -| **12-bit colour, d1.0** | **2.65 / 2.75 MB** | **114–119 GB** | **46–48 GB** | **23–24 GB** | -| 12-bit colour, d0.5 | 5.2 / 5.3 MB | 222–230 GB | 89–92 GB | 44–46 GB | -| 12-bit raw planes, lossless | 16.8 / 17.0 MB | 727–736 GB | 291–294 GB | 145–147 GB | -| 12-bit colour, lossless | 44.8 / 44.1 MB | ~1.9 TB | ~770 GB | ~385 GB | +| Camera RGB8, d0.8 (today's quality) | 3.7 / 4.0 | 160–172 | 64–69 | 32–34 | +| **12-bit colour, d1.0** | **2.65 / 2.75** | **114–119** | **46–48** | **23–24** | +| 12-bit colour, d0.5 | 5.2 / 5.3 | 222–230 | 89–92 | 44–46 | +| 12-bit raw planes, lossless | 16.8 / 17.0 | 727–736 | 291–294 | 145–147 | +| 12-bit colour, lossless | 44.8 / 44.1 | ≈ 1900 | ≈ 770 | ≈ 385 | -The towers have 11–22 GB free, so only a short buffer can stay on the tower. +The towers have 11–22 GB of free storage, so images must leave the tower continuously. -## Scope and caveats +--- -- **One scene, two afternoons,** ~15 dB gain in the main runs; one 0 dB frame. Night is not measured. -- **Auto-exposure steps on a still camera,** not after real slews. Real waypoint-to-waypoint changes - are usually much smaller than ×16, so real settling should be shorter. -- **The tower program was stopped,** so its own load (camera stream, control loop) is not included. - The stream-while-saving test shows the camera link is unaffected by saving. -- **Detection quality is not measured.** PSNR and SSIM stand in for "what the algorithms receive". -- **Tower 111's own camera (1240c, 12 MP) was not tested here.** Its frames are 25% smaller, so the +## 4. Discussion + +### 4.1 Summary of findings + +***Table 21.** Summary of findings.* + +| Topic | Finding | Section | +|---|---|---| +| Frame rate | `BayerRG12p`: 9.86 fps, twice RGB8's 4.95 fps at 250 MB/s; `BayerRG8`: 14.75 fps. Unpacked `BayerRG12`: 7.41 fps. | 3.1 | +| Link | No lost or incomplete frames in any format on either tower, also while saving continuously. | 3.1 | +| Auto-exposure | From under- or over-exposure, settling takes about 25–35 frames: raw 8-bit 1.8–2.7 s, 12-bit 2.2–6.6 s (typically ≈ 3 s), RGB8 4.6–6.6 s. | 3.2 | +| Keep rule | Target band and steady for 3 frames: correct in 21 of 24 steps; "exposure and gain unchanged": 0 of 24. | 3.3 | +| White balance | Recovers in 2–4 frames (0.2–0.8 s), stable within ±1–3%. Exposure, not white balance, determines the wait. | 3.4 | +| Processing | Unpack + colour conversion 38 ms (IOTA) / 50 ms (Delta); a 12-bit save 518 / 820 ms, slightly faster than today's RGB8 save (567 / 872 ms). | 3.5 | +| Computers | The Delta takes 1.6 times as long on the identical frame. | 3.5 | +| Quality | d1.0: 2.65 MB, SSIM 0.973. Raw planes keep sensor values much closer (47 vs 39 dB at 6 MB). | 3.7 | +| 12 bits | In daylight, noise exceeds one 8-bit step even at 0 dB; extra information only below ≈ 300 DN (deep shadow). | 3.9 | +| Throughput | IOTA keeps up with a frame every 0.7 s with one worker; the Delta needs two. | 3.10 | +| Heat | One frame every 2 s: IOTA 53 °C / 5.1 W, Delta 66 °C / 5.7 W. | 3.11 | +| Storage | 12-bit colour d1.0: 2.7 MB per frame, below today's 3.3–4.0 MB. | 3.12 | + +### 4.2 Overall pace + +Saving runs in the background and overlaps the next movement, so the scan pace is set by the +slowest of the camera-side steps and the save rate. + +***Table 22.** Time budget per waypoint.* + +| Step | 12-bit raw (proposed) | Raw 8-bit | RGB8 (today) | Source | +|---|---|---|---|---| +| Camera frame interval (s) | 0.10 (9.86 fps) | 0.07 (14.75 fps) | 0.20 (4.95 fps) | measured | +| Slew to the next waypoint (s) | ≈ 1–1.5 (6° at 5–8 °/s) | same | same | estimate, not measured | +| Mechanical settling after standstill (s) | ≈ 0.2 | same | same | estimate, to be measured | +| Exposure settling after a large (×16) change (s) | ≈ 3 (2.2–6.6) | ≈ 2 (1.8–2.7) | ≈ 5 (4.6–6.6) | measured, 24 steps per tower | +| Exposure settling after a small change (s) | below the above | | | not measured | +| Save, 1 worker × 4 threads, IOTA / Delta (s) | **0.56 / 0.84** | 0.52 / 0.85 | 0.59 / 0.88 | measured, shared frame | +| Save, 2 workers × 2 threads, IOTA / Delta (s) | **0.47 / 0.65** | 0.42 / 0.60 | 0.43 / 0.64 | measured, shared frame | + +A waypoint takes roughly 1.5–5 s, whereas a 12-bit save takes 0.56 s on the IOTA and 0.65 s on the +Delta with two workers. Saving only becomes the limit below about one waypoint per 0.6–0.85 s, which +a real scan does not reach. The camera side therefore sets the pace; settling after small changes +between neighbouring waypoints, which decides the real pace, remains to be measured. + +### 4.3 Limitations + +- One scene on two afternoons, at ≈ 15 dB gain in the main runs, plus one 0 dB frame. Night + conditions were not measured. +- Auto-exposure steps were forced on a still camera, not caused by real slews. Real waypoint-to- + waypoint changes are usually much smaller than ×16, so real settling should be shorter. +- The tower program was stopped, so its own load (camera stream, control loop) is not included; the + stream-while-saving test shows the camera link is unaffected by saving. +- Detection quality was not measured; PSNR and SSIM stand in for what the algorithms receive. +- Tower 111's own camera (1800 U-1240c, 12 MP) was not tested; its frames are 25% smaller, so the Delta's limits above are conservative for it. -- **Slew time and mechanical settle** in the pace table are estimates, not measurements. +- Slew time and mechanical settling in Table 22 are estimates. -## Raw data and tools +--- -- Programs and how to rerun every test: [tools/camera-bench](../tools/camera-bench/README.md). -- Camera tests, processing on own frames, heat: [tower 121](../tools/camera-bench/results-2026-10-02-tower121/), +## 5. Conclusions and recommendations + +1. **Take raw data from the camera** (`BayerRG12p`, 250 MB/s) and convert to colour on the tower, + edge-aware. This doubles the frame rate and roughly halves the settling time, at no additional + CPU, heat or storage cost compared with today, on either computer. +2. **Save 12-bit colour JPEG XL at distance 1.0, effort 3, 4 threads**: 2.7 MB per frame. +3. **Use one save worker on the IOTA and two workers × 2 threads on the Delta.** With the measured + settling times (≈ 2–3 s per waypoint) the scan stays below either limit; the save queue still + needs a bound. +4. **Keep a waypoint's frame when brightness is in the camera's target band and steady for 3 + frames**, with a time limit. +5. **12-bit versus raw 8-bit is a choice, not a cost.** `BayerRG8` (14.75 fps) settles about 1 s + faster in daylight and loses nothing measurable there; `BayerRG12p` preserves deep shadows at dusk + and night. The pixel format is part of the camera profile, so the program should accept both. +6. **Store raw 12-bit planes losslessly (17–18 MB per frame) only if the detection pipeline needs + exact sensor values**; it requires about 6 times the storage. + +--- + +## References + +1. ISO 19567-2:2019. *Photography — Digital cameras — Texture reproduction measurements — Part 2: + Measurements using dead leaves pattern.* International Organization for Standardization. +2. ISO/IEC 18181. *Information technology — JPEG XL image coding system*; reference implementation + libjxl, versions 0.11.0 and 0.11.1, https://github.com/libjxl/libjxl. +3. OpenCV, versions 4.5.4 and 4.10: `cvtColor` colour conversions, Bayer demosaicing (bilinear and + edge-aware), https://docs.opencv.org. +4. EMVA. *GenICam Pixel Format Naming Convention (PFNC)* and *GenICam Standard Features Naming + Convention (SFNC)*. European Machine Vision Association. +5. Allied Vision. *Alvium 1800 U series technical manual* and *Vimba X SDK* (VmbCPP, vmbpy 1.0.5 and + 1.2.0). +6. Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli (2004). Image quality assessment: from error + visibility to structural similarity. *IEEE Transactions on Image Processing* 13(4), 600–612. +7. Intel Running Average Power Limit (RAPL) energy counters, Linux `powercap` interface. + +--- + +## Appendix A. Raw data and tools + +- Programs and instructions to rerun every test: [tools/camera-bench](../tools/camera-bench/README.md). +- Camera tests, processing on each tower's own frame, heat: + [tower 121](../tools/camera-bench/results-2026-10-02-tower121/), [tower 111](../tools/camera-bench/results-2026-10-04-tower111/). - Shared-frame processing: [tower 121](../tools/camera-bench/results-common-tower121/), [tower 111](../tools/camera-bench/results-common-tower111/) (each with `frame_md5.txt`). -- Full-size example `.jxl` files (git-ignored, ~115 MB per set): `tools/camera-bench/examples-*`. -- Charts: `plot_results.py` and `plot_common.py`; example figures: `make_figures.py`; +- Full-size example `.jxl` files (not in the repository, about 115 MB per set): + `tools/camera-bench/examples-*`. +- Figures: `plot_results.py`, `plot_common.py` (charts), `make_figures.py` (example crops); quality scores: `compare_examples.py`. + +## Appendix B. Processing times on each tower's own frame + +![Stage times, real frame](images/raw12/stages_real.png) +![Stage times, synthetic frame](images/raw12/stages_synth.png) + +***Figure B1.** Time per saved frame (ms) by step: (a, top) each tower's own real frame; (b, +bottom) the synthetic frame. Effort 3 (raw lossless: effort 1), 4 threads, distance 1.0.* + +***Table B1.** Total time per saved frame (ms), effort 3, 4 threads, distance 1.0 (raw lossless: +effort 1).* + +| Path | IOTA, own frame (ms) | IOTA, synthetic (ms) | Delta, own frame (ms) | Delta, synthetic (ms) | +|---|---|---|---|---| +| Today: camera RGB8 | 463 | 652 | 788 | 1071 | +| Raw 12-bit → colour (edge-aware) | 569 | 727 | 849 | 1088 | +| Raw 12-bit planes, lossless | 90 | 96 | 276 | 298 | + +The camera's real RGB8 files on these frames were 3.7–4.0 MB. The results agree with the shared-frame +comparison (Section 3.5). diff --git a/include/fgc/SettleRule.h b/include/fgc/SettleRule.h index e96aad8..e241c8f 100644 --- a/include/fgc/SettleRule.h +++ b/include/fgc/SettleRule.h @@ -17,7 +17,7 @@ struct FrameStats { }; // When has the image settled after the gimbal stops? Measured on 24 forced -// exposure steps (docs/raw12-pipeline-measurements.md, "Which frame to keep"): +// exposure steps (docs/raw12-pipeline-measurements.md, section 3.3 "Keep rules"): // // "exposure and gain unchanged for 3 frames" right 0 of 24 times // "brightness in the camera's target band, 2 frames" right 12 of 24