GGS
/
fwt_software
8
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 1 Wiki Activity

Explain the raw mosaic and planes layouts in the 12-bit report, and spell out every term in its figure and table captions

This commit is contained in:
pgdalmeida 2026-10-05 11:39:31 +02:00
parent 1fc87bf1ee
commit 4abfd7cbb2
Signed by: pedro.almeida
GPG Key ID: D4A6C394DF13F1D7
1 changed files with 177 additions and 72 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

249
docs/raw12-pipeline-measurements.md
View File

@ -26,6 +26,8 @@ changing never did.
## Units and abbreviations ## Units and abbreviations
*Units, symbols and abbreviations used in this report.*
| Symbol | Meaning | | Symbol | Meaning |
|---|---| |---|---|
| MB, GB, TB | 10⁶, 10⁹, 10¹² bytes | | MB, GB, TB | 10⁶, 10⁹, 10¹² bytes |
@ -39,6 +41,7 @@ changing never did.
| d, e | JPEG XL distance (quality: 0 = lossless, higher = smaller and lossier) and effort (1–9) | | d, e | JPEG XL distance (quality: 0 = lossless, higher = smaller and lossier) and effort (1–9) |
| PSNR | peak signal-to-noise ratio, in dB | | PSNR | peak signal-to-noise ratio, in dB |
| SSIM | structural similarity index, 0–1 (1 = identical) [6] | | SSIM | structural similarity index, 0–1 (1 = identical) [6] |
| stop | a factor of 2 in light or brightness: brightened by 2 stops = pixel values ×4; ×16 = 4 stops |
| IOTA, Delta | LattePanda IOTA (tower 121) and LattePanda 3 Delta (tower 111) | | IOTA, Delta | LattePanda IOTA (tower 121) and LattePanda 3 Delta (tower 111) |
--- ---
@ -62,7 +65,7 @@ extra bits add.
### 1.2 The camera's pixel formats ### 1.2 The camera's pixel formats
The sensor sees through a colour-filter mosaic: each pixel measures only red, green or blue, in the The sensor sees through a colour-filter mosaic: each pixel measures only red, green or blue, in the
pattern of Figure 1 ("BayerRG": a red pixel first). Producing a colour image means estimating the pattern of Figure 1a ("BayerRG": a red pixel first). Producing a colour image means estimating the
two missing colours at every pixel from its neighbours, called **de-Bayering** (demosaicing). The two missing colours at every pixel from its neighbours, called **de-Bayering** (demosaicing). The
camera can either send the raw mosaic or de-Bayer it itself. camera can either send the raw mosaic or de-Bayer it itself.
@ -72,10 +75,12 @@ G B G B G B ...
R G R G R G ... R G R G R G ...
``` ```
***Figure 1.** The BayerRG colour-filter pattern of the sensor: each pixel records one colour.* ***Figure 1a.** The BayerRG colour-filter pattern of the sensor: each pixel records one colour.*
***Table 1.** Pixel formats considered, at the 1620c's full resolution (5328 × 3040 pixels) and a USB ***Table 1.** Pixel formats considered, at the 1620c's full resolution (5328 × 3040 pixels) and a
link cap of 250 MB/s [4].* USB link cap of 250 MB/s [4]. Bits per value: the precision of each stored measurement (8 bits = 256
levels, 10 bits = 1024, 12 bits = 4096). Packed: values stored back to back without padding bits.
Maximum rate: the highest frame rate the camera accepts at this link cap.*
| Format | What the camera sends | Bits per value | Bytes per pixel | Frame size (MB) | Maximum rate (fps) | Colour made by | | Format | What the camera sends | Bits per value | Bytes per pixel | Frame size (MB) | Maximum rate (fps) | Colour made by |
|---|---|---|---|---|---|---| |---|---|---|---|---|---|---|
@ -92,6 +97,40 @@ formats too, so the tower does not need to. The camera also offers grey (`Mono8/
discards colour), `BGR8` (RGB8 in reverse order) and video formats (`YCbCr…`, which reduce colour discards colour), `BGR8` (RGB8 in reverse order) and video formats (`YCbCr…`, which reduce colour
detail); none of these was considered. detail); none of these was considered.
**Saving raw data: mosaic and planes.** Besides converting to colour, the tower can save the raw
12-bit data as received, each pixel keeping the single value measured through its filter. Two
layouts were tested; they hold identical values and differ only in their arrangement in the file:
- **Raw mosaic:** the sensor data as the camera sends it, a single-channel (grey) image of
5328 × 3040 values still in the R G / G B pattern. Viewed directly it looks like a grey image with
a fine checkerboard texture. Because colours alternate from pixel to pixel, the encoder sees the
pattern as extremely fine detail: it compresses poorly (20.9 MB lossless), and lossy compression
would mix values of different colours.
- **Raw planes:** the same values reordered into four quarter-size images (2664 × 1520 each), one per
filter position (red, first green, second green, blue), tiled 2 × 2 into one grey image of the
same total size (Figure 1b). Within each tile neighbouring values belong to the same colour and are
similar, so the image is smooth to the encoder: 17.6 MB lossless, and lossy compression works
(distance 0.5: 6.0 MB at a PSNR of 47.2 dB, against 38.8 dB for 12-bit colour of similar size;
Section 3.7). The reordering is exactly reversible, so the mosaic can always be rebuilt.
```
Mosaic (5328 × 3040) Planes (2 × 2 tiles of 2664 × 1520)
R G R G R G ... R R R ... | G G G ...
G B G B G B ... → R R R ... | G G G ...
R G R G R G ... ----------+----------
G B G B G B ... G G G ... | B B B ...
G G G ... | B B B ...
```
***Figure 1b.** Rearranging the raw mosaic into four single-colour planes.*
Neither layout is a viewable colour picture: the reader must de-Bayer it, as the tower does before
saving colour. Their value is that they preserve the measured sensor values, with nothing estimated
and no colour processing applied, which matters only if the detection algorithms need exact sensor
data. Lossless planes take about 6 times the storage of 12-bit colour at distance 1.0 (17.6 MB
against 2.65 MB per frame), so they are an option (recommendation 6, Section 5) and are not part of
the current implementation.
### 1.3 Earlier work ### 1.3 Earlier work
An earlier campaign ([camera-capture-measurements.md](camera-capture-measurements.md)) used blurred, An earlier campaign ([camera-capture-measurements.md](camera-capture-measurements.md)) used blurred,
@ -106,7 +145,9 @@ The implementation built on this report is described in
### 2.1 Hardware and software ### 2.1 Hardware and software
***Table 2.** Test systems. The same camera was moved from tower 121 to tower 111.* ***Table 2.** Test systems. The same camera was moved from tower 121 to tower 111, so only the
computer differs between the towers. Storage write speed: sequential writing to the built-in eMMC
flash storage.*
| | Tower 121 | Tower 111 | | | Tower 121 | Tower 111 |
|---|---|---| |---|---|---|
@ -126,7 +167,8 @@ and the CPU figures should have a single cause). Afterwards both cameras were re
![The scene](images/raw12/examples/121/thumbs/c12_ea_lossless.jpg) ![The scene](images/raw12/examples/121/thumbs/c12_ea_lossless.jpg)
***Figure 2.** The test scene from tower 121's camera (2 Oct, 15:07; exposure 20 ms, gain ≈ 15 dB): ***Figure 2.** The test scene from tower 121's camera (2 Oct, 15:07; exposure 20 ms, gain ≈ 15 dB):
spruce, birch trunks, leaves and a building; about 13% of the pixels are clipped sky.* spruce, birch trunks, leaves and a building. About 13% of the pixels are clipped sky: saturated at
the maximum value, so they show pure white without detail.*
### 2.3 Camera control ### 2.3 Camera control
@ -138,7 +180,9 @@ value the automatic control had reached, after which `ExposureTime`, `Gain` and
written directly and take effect from the next frame exposed. No profile, user set or file on the written directly and take effect from the next frame exposed. No profile, user set or file on the
camera is changed. camera is changed.
***Table 3.** Camera settings common to all camera tests.* ***Table 3.** Camera settings common to all camera tests. Names in code font are the camera's
GenICam features [4]. Chunk data: values the camera attaches to each frame, describing the settings
it was taken with.*
| Setting | Value | Reason | | Setting | Value | Reason |
|---|---|---| |---|---|---|
@ -166,7 +210,10 @@ continuously in parallel.
### 2.5 Auto-exposure step test ### 2.5 Auto-exposure step test
***Table 4.** One auto-exposure step run.* ***Table 4.** The three phases of one auto-exposure step run. Exposure is the time the sensor
collects light (ms); gain amplifies the sensor signal (dB; +6 dB doubles the brightness, +20 dB
multiplies it by 10). A step of ×16 makes the image 16 times brighter (×1/16: 16 times darker), i.e.
4 stops, where one stop is a factor of 2.*
| Phase | Duration (s) | Procedure | | Phase | Duration (s) | Procedure |
|---|---|---| |---|---|---|
@ -174,8 +221,10 @@ continuously in parallel.
| 2. Force | 2 | `ExposureAuto` = `GainAuto` = `Off`. **Dark:** `ExposureTime` = max(`ExposureAutoMin`, E₀/16), `Gain` = 0 dB. **Bright:** `ExposureTime` = min(20 ms, 16·E₀); the remainder as `Gain` = 20·log₁₀(16·E₀ / exposure) dB, up to `GainAutoMax`. The camera delivers 10–30 frames at the forced settings. | | 2. Force | 2 | `ExposureAuto` = `GainAuto` = `Off`. **Dark:** `ExposureTime` = max(`ExposureAutoMin`, E₀/16), `Gain` = 0 dB. **Bright:** `ExposureTime` = min(20 ms, 16·E₀); the remainder as `Gain` = 20·log₁₀(16·E₀ / exposure) dB, up to `GainAutoMax`. The camera delivers 10–30 frames at the forced settings. |
| 3. Release | 15 | Time zero is taken, then `ExposureAuto` = `GainAuto` = `Continuous`; every frame is recorded. | | 3. Release | 15 | Time zero is taken, then `ExposureAuto` = `GainAuto` = `Continuous`; every frame is recorded. |
***Table 5.** Starting conditions of the auto-exposure steps: the first frame after release, over all ***Table 5.** Starting conditions of the auto-exposure steps: the first frame after release, ranges
24 runs.* over all 24 runs. Before: the settled state before forcing. Dark / bright step: the settings forced
16 times darker / brighter. Brightness: mean pixel value on the 0–255 scale (DN). Clipped share:
percentage of sampled pixels at or above 250 DN, i.e. saturated white without detail.*
| | Before (settled) | Start of "dark" step | Start of "bright" step | | | Before (settled) | Start of "dark" step | Start of "bright" step |
|---|---|---|---| |---|---|---|---|
@ -202,7 +251,9 @@ camera's target band is 50 ± 5% of full scale, i.e. 114.8–140.3 DN on the 0
### 2.7 Auto white balance step test ### 2.7 Auto white balance step test
***Table 6.** One auto-white-balance step run.* ***Table 6.** The four phases of one auto-white-balance step run. White-balance ratios are the gains
the camera applies to the red and blue values relative to green (no unit; 1 = unchanged); ×0.5
halves them and ×2 doubles them.*
| Phase | Duration (s) | Procedure | | Phase | Duration (s) | Procedure |
|---|---|---| |---|---|---|
@ -218,7 +269,10 @@ Settled = both ratios within ±5% of the reference and remaining there. Each cas
A C++ program (`raw12_bench.cpp`), built on each tower against its own libraries, performs the steps A C++ program (`raw12_bench.cpp`), built on each tower against its own libraries, performs the steps
the tower program would perform for one saved frame and times each with a monotonic clock. the tower program would perform for one saved frame and times each with a monotonic clock.
***Table 7.** Processing steps timed.* ***Table 7.** Processing steps timed for one saved frame. Bilinear conversion estimates each missing
colour as the average of the nearest pixels of that colour; edge-aware conversion interpolates along
edges rather than across them, giving fewer false colour fringes at a slightly higher cost. Distance
and effort are the JPEG XL quality and speed settings (Units and abbreviations).*
| Step | Operation | | Step | Operation |
|---|---| |---|---|
@ -266,7 +320,8 @@ patches; in each, noise σ = median absolute deviation of differences between ho
### 2.12 Campaign log ### 2.12 Campaign log
***Table 8.** Campaign log.* ***Table 8.** Campaign log: date and local time, tower, activity and duration (min) of each test
session.*
| Date and time | Tower | Activity | Duration (min) | Notes | | Date and time | Tower | Activity | Duration (min) | Notes |
|---|---|---|---|---| |---|---|---|---|---|
@ -289,7 +344,10 @@ patches; in each, noise σ = median absolute deviation of differences between ho
***Figure 3.** Maximum full-resolution frame rate (fps) per pixel format at the 250 MB/s link cap. ***Figure 3.** Maximum full-resolution frame rate (fps) per pixel format at the 250 MB/s link cap.
The same camera was used on both towers.* The same camera was used on both towers.*
***Table 9.** 60 s streams at each format's maximum rate.* ***Table 9.** Results of 60 s streams at each format's maximum rate. Complete: frames received
intact. Incomplete: frames whose data arrived only partly (SDK frame status). Lost: frames missing
from the camera's frame counter. Last row: the same stream while the tower saved 12-bit images
continuously in parallel.*
| Format | Frame size (MB) | Maximum rate (fps) | 121: complete / incomplete / lost (frames) | 111: complete / incomplete / lost (frames) | | Format | Frame size (MB) | Maximum rate (fps) | 121: complete / incomplete / lost (frames) | 111: complete / incomplete / lost (frames) |
|---|---|---|---|---| |---|---|---|---|---|
@ -308,15 +366,18 @@ USB 3 controller with a 5G modem, a USB network adapter and a second (idle) came
![Brightness after a step, 121](images/raw12/ae_brightness_121.png) ![Brightness after a step, 121](images/raw12/ae_brightness_121.png)
![Brightness after a step, 111](images/raw12/ae_brightness_111.png) ![Brightness after a step, 111](images/raw12/ae_brightness_111.png)
***Figure 4.** Frame brightness (DN, 0–255) after auto-exposure is released, for each format: (a, ***Figure 4.** Frame brightness (DN, 0–255) against time (s) after auto-exposure is switched back on
top) tower 121; (b, bottom) tower 111. Left: from under-exposure (×1/16); right: from over-exposure at time 0, for each format: (a, top) tower 121; (b, bottom) tower 111. Left: starting from
(×16). Dashed line: settled level.* under-exposure, 16 times (4 stops) too dark; right: from over-exposure, 16 times too bright. Dashed
line: settled level, the median brightness over the run's last 2 s.*
With exposure at its 20 ms cap, auto-exposure adjusts through gain, by 1–2 dB per frame, and With exposure at its 20 ms cap, auto-exposure adjusts through gain, by 1–2 dB per frame, and
sometimes pauses mid-way (flat segments in Figure 4, up to 2.8 s while still far too bright). sometimes pauses mid-way (flat segments in Figure 4, up to 2.8 s while still far too bright).
***Table 10.** Time (s) and frames until brightness stays within ±10% of its final level; two runs ***Table 10.** Settling time after a forced exposure step: time (s) from release, and in parentheses
each.* the number of frames, until brightness stays within ±10% of its final level (median of the run's
last 2 s). Dark ×1/16: started 16 times (4 stops) too dark; bright ×16: 16 times too bright. Two
runs per cell.*
| Format | Rate (fps) | Step | Tower 121: time (s) (frames) | Tower 111: time (s) (frames) | | Format | Rate (fps) | Step | Tower 121: time (s) (frames) | Tower 111: time (s) (frames) |
|---|---|---|---|---| |---|---|---|---|---|
@ -333,8 +394,12 @@ Settling typically takes 25–35 frames, so **settling time scales with 1/frame
### 3.3 Keep rules ### 3.3 Keep rules
***Table 11.** Candidate keep rules scored on the 24 steps (12 per tower). Correct: the kept frame's ***Table 11.** Candidate keep rules scored on the 24 recorded steps (12 per tower). A keep rule
brightness is within 10% of the final level.* decides which frame after a gimbal stop is saved. Correct: the kept frame's brightness is within
±10% of the final level; early: the rule kept a frame before that. Target band: brightness between
45% and 55% of full scale (114.8–140.3 DN), the range the camera's auto-exposure aims for. Steady:
brightness changes by at most 2% from one frame to the next. Median time (s) from release to the
kept frame, over the correct runs.*
| Rule | 121 correct / early (runs) | 111 correct / early (runs) | Median time when correct, 121 / 111 (s) | | Rule | 121 correct / early (runs) | 111 correct / early (runs) | Median time when correct, 121 / 111 (s) |
|---|---|---|---| |---|---|---|---|
@ -356,11 +421,14 @@ The profiles currently fix white balance (red 2.87, blue 2.0). With `BalanceWhit
![Auto white balance, 121](images/raw12/awb_121.png) ![Auto white balance, 121](images/raw12/awb_121.png)
![Auto white balance, 111](images/raw12/awb_111.png) ![Auto white balance, 111](images/raw12/awb_111.png)
***Figure 5.** White-balance ratios relative to their settled values after release, for each ***Figure 5.** White-balance ratios (the red and blue gains relative to green) divided by their
format: (a, top) tower 121; (b, bottom) tower 111. Left: ratios forced to ×0.5; right: ×2. Shaded: settled values, against time (s) after automatic white balance is switched back on at time 0, for
±5% band.* each format: (a, top) tower 121; (b, bottom) tower 111. A value of 1.0 means settled. Left: ratios
forced to half (×0.5); right: to double (×2). Shaded: the ±5% band counted as settled.*
***Table 12.** Time until both ratios are within ±5% of the reference and remain there.* ***Table 12.** Settling time after a forced white-balance step: time (s), and in parentheses frames,
until both the red and the blue ratio are within ±5% of the reference and remain there. Ranges over
the ×0.5 and ×2 steps, two runs each.*
| Format | Rate (fps) | Tower 121: time (s) (frames) | Tower 111: time (s) (frames) | | Format | Rate (fps) | Tower 121: time (s) (frames) | Tower 111: time (s) (frames) |
|---|---|---|---| |---|---|---|---|
@ -375,10 +443,14 @@ exposure.
![Same frame, time per step](images/raw12/stages_common.png) ![Same frame, time per step](images/raw12/stages_common.png)
***Figure 6.** Time per saved frame (ms) by step on the identical shared frame, for both computers; ***Figure 6.** Time per saved frame (ms), split by step, on the shared frame (tower 121's frame
4 threads; colour paths at d1.0 e3, today's RGB8 path at d0.8 e4.* copied bit-exactly to both towers, so only the computer differs). 4 encoder threads. Colour paths at
d1.0 e3 (JPEG XL distance 1.0, effort 3); today's RGB8 path at its current setting d0.8 e4.*
***Table 13.** Duration of each step on the shared frame (4 threads).* ***Table 13.** Duration (ms) of each processing step for one frame on the shared frame, 4 encoder
threads, median of 3 runs. Delta / IOTA: ratio of the two times (above 1 = the Delta is slower).
d1.0 e3: JPEG XL distance 1.0, effort 3. The last two rows differ mainly through the towers' storage
write speeds.*
| Step | IOTA (ms) | Delta (ms) | Delta / IOTA | | Step | IOTA (ms) | Delta (ms) | Delta / IOTA |
|---|---|---|---| |---|---|---|---|
@ -390,8 +462,9 @@ exposure.
| File write, 2.7 MB colour file | 1 | 4 | | | File write, 2.7 MB colour file | 1 | 4 | |
| File write, 17–21 MB lossless raw file | 7–8 | 162–173 | (storage 218 vs 76 MB/s) | | File write, 17–21 MB lossless raw file | 7–8 | 162–173 | (storage 218 vs 76 MB/s) |
***Table 14.** Total time per saved frame and resulting file size, shared frame (file sizes are ***Table 14.** Total time per saved frame (ms: unpack, colour conversion, encode and file write) and
identical on both computers).* file size (MB), on the shared frame, 4 encoder threads. File sizes are identical on both computers.
Raw planes and raw mosaic: raw data saved without colour conversion (Section 1.2).*
| Path | IOTA (ms) | Delta (ms) | File size (MB) | | Path | IOTA (ms) | Delta (ms) | File size (MB) |
|---|---|---|---| |---|---|---|---|
@ -412,8 +485,10 @@ somewhat larger (3.7–4.0 MB, Appendix B).
![Effort × threads, 121](images/raw12/effort_threads_121.png) ![Effort × threads, 121](images/raw12/effort_threads_121.png)
![Effort × threads, 111](images/raw12/effort_threads_111.png) ![Effort × threads, 111](images/raw12/effort_threads_111.png)
***Figure 7.** Time per frame (ms) by JPEG XL effort (rows) and thread count (columns), each tower's ***Figure 7.** Time per saved frame (ms) by JPEG XL effort (rows) and thread count (columns), each
real frame, distance 1.0: (a, top) tower 121; (b, bottom) tower 111.* tower's own real frame, distance 1.0: (a, top) tower 121; (b, bottom) tower 111. Effort: how hard
the encoder searches for a compact encoding (1 = fastest; higher = slower, usually smaller).
Threads: processor threads one encode may use.*
Effort 1–4 lie within about 15% of each other, and effort 3 gives the smallest files. On the IOTA, Effort 1–4 lie within about 15% of each other, and effort 3 gives the smallest files. On the IOTA,
3 threads is slower than 2 at every effort (761 / 582 / 672 / 512 ms for 1–4 threads at effort 1); 3 threads is slower than 2 at every effort (761 / 582 / 672 / 512 ms for 1–4 threads at effort 1);
@ -423,11 +498,14 @@ the Delta scales normally (1440 / 985 / 886 / 802 ms). Four threads is fastest o
![Same frame, size and time by quality](images/raw12/sizes_common.png) ![Same frame, size and time by quality](images/raw12/sizes_common.png)
***Figure 8.** File size (MB, left) and time per frame (ms, right) versus JPEG XL distance on the ***Figure 8.** File size (MB, left) and time per saved frame (ms, right) against JPEG XL distance on
shared frame, effort 3, 4 threads, both towers.* the shared frame, effort 3, 4 threads, both towers. Distance is the encoder's quality setting: 0 is
lossless, 1.0 is intended to be visually indistinguishable from the original, and higher values give
smaller files with more loss.*
***Table 15.** Size and time per distance on the shared frame (effort 3, 4 threads, file write ***Table 15.** File size (MB) and time per saved frame (ms) per JPEG XL distance on the shared frame
included; 2 runs per point).* (effort 3, 4 threads, file write included, median of 2 runs). Last column: raw planes (Section 1.2)
at the same distance.*
| Distance | 12-bit colour size (MB) | IOTA time (ms) | Delta time (ms) | Raw 12-bit planes size (MB) | | Distance | 12-bit colour size (MB) | IOTA time (ms) | Delta time (ms) | Raw 12-bit planes size (MB) |
|---|---|---|---|---| |---|---|---|---|---|
@ -441,11 +519,16 @@ included; 2 runs per point).*
![Size and quality, real frame](images/raw12/distance_real.png) ![Size and quality, real frame](images/raw12/distance_real.png)
***Figure 9.** File size (MB), PSNR (dB) and fine-detail retention against each method's own ***Figure 9.** File size (MB), PSNR (dB) and fine-detail retention against JPEG XL distance, each
uncompressed image, versus distance; each tower's real frame.* tower's own real frame, each method compared with its own uncompressed image. Fine-detail retention:
the amount of fine detail (Laplacian energy) in the decoded image relative to the uncompressed one;
1 = all detail kept, below 1 = smoothed.*
***Table 16.** Quality against the exact lossless 12-bit colour image of the same frame (real frame, ***Table 16.** Image quality against the exact lossless 12-bit colour image of the same frame (real
≈ 15 dB gain; the towers agree within 0.3 dB).* frame, ≈ 15 dB gain; the towers agree within 0.3 dB). PSNR (dB): peak signal-to-noise ratio, higher
means closer to the reference; +6 dB means half the error. SSIM: structural similarity, comparing
local patterns of brightness and contrast, 0–1 with 1 = identical. Exact: decodes bit for bit to the
original. Colour rows use edge-aware conversion unless stated.*
| Saved as | File size, 121 / 111 (MB) | PSNR (dB) | SSIM | | Saved as | File size, 121 / 111 (MB) | PSNR (dB) | SSIM |
|---|---|---|---| |---|---|---|---|
@ -469,12 +552,15 @@ size (47 vs 39 dB, equal SSIM); lossless, they take a third of the space of loss
![crops](images/raw12/examples/121/crops.jpg) ![crops](images/raw12/examples/121/crops.jpg)
***Figure 10.** Positions of the detail crop (red) and shadow crop (blue) used in Figure 11.* ***Figure 10.** Tower 121's frame (downscaled), with the positions of the detail crop (red) and
shadow crop (blue) used in Figure 11.*
***Figure 11.** One example per compression method, tower 121's frame (480 × 320-pixel crops at ***Figure 11.** One example per compression method, tower 121's frame, as 480 × 320-pixel crops at
100%): detail crop of spruce needles over bright leaves; its absolute difference from the lossless 100% (one image pixel per screen pixel). Detail: spruce needles over bright leaves. Difference: the
image (amplified ×8; black = identical); and a darker crop brightened by 2 stops. File size and save absolute difference from the lossless image, multiplied by 8 so small errors become visible (black =
time are single runs at 4 threads. The two 8-bit rows are separate captures seconds apart (branches identical). Shadow: a darker crop brightened by 2 stops, i.e. every pixel value multiplied by 4
(each stop doubles), so detail in the dark parts becomes visible. File size (MB) and save time (s)
are single runs at 4 threads. The two 8-bit rows are separate captures taken seconds apart (branches
move), so they have no difference map.* move), so they have no difference map.*
| Method | File size (MB), time (s) | Detail (100%) | Difference (×8) | Shadow (+2 stops) | | Method | File size (MB), time (s) | Detail (100%) | Difference (×8) | Shadow (+2 stops) |
@ -500,11 +586,15 @@ Full-size `.jxl` files are kept outside the repository in `tools/camera-bench/ex
![Noise vs brightness](images/raw12/noise_vs_signal.png) ![Noise vs brightness](images/raw12/noise_vs_signal.png)
***Figure 12.** Sensor noise σ (DN, 12-bit) versus signal (DN, 12-bit), green sites, at 0 dB gain ***Figure 12.** Sensor noise σ (DN, 12-bit scale) against signal level (DN, 12-bit scale), measured
and at ≈ 15 dB gain. Dashed: one 8-bit quantisation step (16 DN); dotted: the RMS error added by on the green pixels, at 0 dB gain and at ≈ 15 dB gain. σ is the standard deviation of the random
rounding to 8 bits (4.6 DN).* pixel-to-pixel variation. Dashed: one 8-bit quantisation step, the smallest change an 8-bit value
can record (16 DN on the 12-bit scale). Dotted: the RMS error added by rounding to 8 bits (16/√12 =
4.6 DN). Noise above these lines means rounding to 8 bits loses nothing measurable.*
***Table 17.** Noise floor at three signal levels.* ***Table 17.** Sensor noise floor σ (DN, 12-bit scale, 0–4095) at three signal levels, green pixels,
at 0 dB and ≈ 15 dB gain. Last column: the same noise in 8-bit steps (one step = 16 DN on the 12-bit
scale); above 1, the 8-bit values already resolve all the sensor delivers.*
| Signal (DN, 12-bit) | Noise at 0 dB (DN) | Noise at ≈ 15 dB (DN) | Noise in 8-bit steps, 0 dB / 15 dB | | Signal (DN, 12-bit) | Noise at 0 dB (DN) | Noise at ≈ 15 dB (DN) | Noise in 8-bit steps, 0 dB / 15 dB |
|---|---|---|---| |---|---|---|---|
@ -520,9 +610,10 @@ extrapolation, 12-bit data carries additional information below about 300 DN (0
![8 vs 12 bit at 15 dB](images/raw12/examples/121/shadow_8_vs_12.png) ![8 vs 12 bit at 15 dB](images/raw12/examples/121/shadow_8_vs_12.png)
![8 vs 12 bit at 0 dB](images/raw12/examples/lowgain/shadow_8_vs_12.png) ![8 vs 12 bit at 0 dB](images/raw12/examples/lowgain/shadow_8_vs_12.png)
***Figure 13.** The same darker crop brightened by 2 stops, 8-bit (left; the 12-bit value shifted ***Figure 13.** The same darker crop brightened by 2 stops (pixel values multiplied by 4), saved as
down by 4 bits) versus 12-bit (right): (a, top) ≈ 15 dB gain, 3138 vs 227 distinct green levels; 8-bit (left; the 12-bit value shifted down by 4 bits, i.e. divided by 16) and as 12-bit (right): (a,
(b, bottom) 0 dB gain. No visible difference.* top) ≈ 15 dB gain, 227 distinct green levels in 8-bit against 3138 in 12-bit; (b, bottom) 0 dB gain.
No visible difference: the noise is larger than the extra levels.*
At 0 dB gain the same settings produce 25–30% smaller files and about 3 dB higher PSNR (less noise At 0 dB gain the same settings produce 25–30% smaller files and about 3 dB higher PSNR (less noise
to encode): 12-bit colour at d1.0 is 1.9 MB, PSNR 40.0 dB, SSIM 0.987. to encode): 12-bit colour at d1.0 is 1.9 MB, PSNR 40.0 dB, SSIM 0.987.
@ -531,17 +622,21 @@ to encode): 12-bit colour at d1.0 is 1.9 MB, PSNR 40.0 dB, SSIM 0.987.
![Saves per second, same frame](images/raw12/pace_common.png) ![Saves per second, same frame](images/raw12/pace_common.png)
***Figure 14.** Frames saved per second, back to back, on the shared frame, for one worker × 4 ***Figure 14.** Frames saved per second, back to back (the next save starts as soon as one
threads and two workers × 2 threads.* finishes), on the shared frame, for one save worker with 4 threads and two workers with 2 threads
each. A worker saves one frame at a time; two workers save two frames in parallel.*
![Pile-up](images/raw12/pileup.png) ![Pile-up](images/raw12/pileup.png)
***Figure 15.** Worst time from queueing to file written (s, logarithmic) versus the interval ***Figure 15.** Worst time (s, logarithmic axis) from queueing a frame to its file being written,
between queued frames (s), synthetic frame. A flat line means saving keeps up; a rising line means a against the interval (s) between queued frames, synthetic frame. A flat line means saving keeps up;
backlog builds.* a rising line means a backlog builds.*
***Table 18.** Throughput and pile-up on the synthetic frame (worst case). "ok" = no backlog after ***Table 18.** Throughput and pile-up on the synthetic frame, the worst case for the encoder.
120 s; otherwise frames behind at the end.* Workers × threads: frames saved in parallel × encoder threads each. Back to back: frames saved per
second when saving continuously. Paced columns: one frame queued every given interval for 120 s;
"ok" = no frames waiting at the end, "N behind" = N frames still waiting, "–" = not tested. Peak
memory: the most memory (GB) used by the queue and encoders.*
| Path | Workers × threads | Tower | Back to back (frames/s) | One frame every 2.0 / 1.5 / 1.0 / 0.7 / 0.5 s | Peak memory (GB) | | Path | Workers × threads | Tower | Back to back (frames/s) | One frame every 2.0 / 1.5 / 1.0 / 0.7 / 0.5 s | Peak memory (GB) |
|---|---|---|---|---|---| |---|---|---|---|---|---|
@ -565,11 +660,14 @@ fell 27 frames behind in 120 s).
![Flat out](images/raw12/heat_max.png) ![Flat out](images/raw12/heat_max.png)
![Paced](images/raw12/heat_paced.png) ![Paced](images/raw12/heat_paced.png)
***Figure 16.** CPU package temperature (°C) and package power (W) over 10 minutes of 12-bit saving: ***Figure 16.** CPU package temperature (°C) and package power (W) over 10 minutes of 12-bit saving;
(a, top) back to back with 2 × 2 workers; (b, bottom) one frame every 2 s with 1 × 4. Solid: tower the package is the whole processor chip. (a, top) back to back with 2 workers × 2 threads; (b,
121; dashed: tower 111.* bottom) one frame every 2 s with 1 worker × 4 threads. Solid: tower 121; dashed: tower 111.*
***Table 19.** Temperature, clock, load and power, 10 minutes per condition.* ***Table 19.** Processor temperature (mean and maximum, °C), average core clock (GHz), CPU busy (%
of time the cores were working), and package power (W, read from the Intel RAPL energy counter [7]),
10 minutes per condition after a 5-minute cool-down. Saves per minute in the first and last minute
show any slowdown as the processor heats up and lowers its clock.*
| Load | Tower | Temperature mean / max (°C) | Clock (GHz) | CPU busy (%) | Power (W) | Saves per minute, first → last minute | | Load | Tower | Temperature mean / max (°C) | Clock (GHz) | CPU busy (%) | Power (W) | Saves per minute, first → last minute |
|---|---|---|---|---|---|---| |---|---|---|---|---|---|---|
@ -588,10 +686,11 @@ back, the Delta sits at an 8 W power limit and does not slow further, whereas th
![Storage per day](images/raw12/storage.png) ![Storage per day](images/raw12/storage.png)
***Figure 17.** Storage per day (GB) at one frame every 2 s, per saving method, each tower's real ***Figure 17.** Storage per day (GB) at one saved frame every 2 s, per saving method, file sizes
frame.* from each tower's own real frame.*
***Table 20.** Storage for 24 hours of scanning at three frame intervals.* ***Table 20.** Storage (GB/day) for 24 hours of scanning at one saved frame every 2, 5 or 10 s.
Per-frame sizes (MB) are from each tower's own real frame; ranges span the two towers.*
| Saved as | Per frame, 121 / 111 (MB) | Every 2 s (GB/day) | Every 5 s (GB/day) | Every 10 s (GB/day) | | Saved as | Per frame, 121 / 111 (MB) | Every 2 s (GB/day) | Every 5 s (GB/day) | Every 10 s (GB/day) |
|---|---|---|---|---| |---|---|---|---|---|
@ -609,7 +708,7 @@ The towers have 11–22 GB of free storage, so images must leave the tower conti
### 4.1 Summary of findings ### 4.1 Summary of findings
***Table 21.** Summary of findings.* ***Table 21.** Summary of findings, with the section where each is shown.*
| Topic | Finding | Section | | Topic | Finding | Section |
|---|---|---| |---|---|---|
@ -631,7 +730,10 @@ The towers have 11–22 GB of free storage, so images must leave the tower conti
Saving runs in the background and overlaps the next movement, so the scan pace is set by the Saving runs in the background and overlaps the next movement, so the scan pace is set by the
slowest of the camera-side steps and the save rate. slowest of the camera-side steps and the save rate.
***Table 22.** Time budget per waypoint.* ***Table 22.** Time budget per waypoint (s) for each pixel format. Slew: the gimbal turning to the
next waypoint. Mechanical settling: vibration dying down after it stops. Exposure settling:
auto-exposure adapting to the new view. Save: runs in the background, overlapping the next waypoint.
Source: whether the value was measured or estimated.*
| Step | 12-bit raw (proposed) | Raw 8-bit | RGB8 (today) | Source | | Step | 12-bit raw (proposed) | Raw 8-bit | RGB8 (today) | Source |
|---|---|---|---|---| |---|---|---|---|---|
@ -718,11 +820,14 @@ between neighbouring waypoints, which decides the real pace, remains to be measu
![Stage times, real frame](images/raw12/stages_real.png) ![Stage times, real frame](images/raw12/stages_real.png)
![Stage times, synthetic frame](images/raw12/stages_synth.png) ![Stage times, synthetic frame](images/raw12/stages_synth.png)
***Figure B1.** Time per saved frame (ms) by step: (a, top) each tower's own real frame; (b, ***Figure B1.** Time per saved frame (ms) split by step: (a, top) each tower's own real frame; (b,
bottom) the synthetic frame. Effort 3 (raw lossless: effort 1), 4 threads, distance 1.0.* bottom) the synthetic frame (a computer-generated "dead leaves" texture of overlapping discs with
added noise [1], harder to compress than any real scene). Effort 3 (raw lossless: effort 1), 4
threads, distance 1.0.*
***Table B1.** Total time per saved frame (ms), effort 3, 4 threads, distance 1.0 (raw lossless: ***Table B1.** Total time per saved frame (ms) on each tower's own real frame and on the synthetic
effort 1).* frame, effort 3, 4 threads, distance 1.0 (raw lossless: effort 1). The real frames differ between
towers, so Table 14 is the fair comparison.*
| Path | IOTA, own frame (ms) | IOTA, synthetic (ms) | Delta, own frame (ms) | Delta, synthetic (ms) | | Path | IOTA, own frame (ms) | IOTA, synthetic (ms) | Delta, own frame (ms) | Delta, synthetic (ms) |
|---|---|---|---|---| |---|---|---|---|---|