126 lines
7.4 KiB
Markdown
126 lines
7.4 KiB
Markdown
# camera-bench
|
||
|
||
Measures a tower's camera and CPU for the capture design: maximum and sustained frame rate per pixel
|
||
format, losses while streaming, auto-exposure settle time, whether white balance applies to raw
|
||
output, and the per-frame cost of copying, de-Bayering and JPEG XL encoding. The results of the first
|
||
run, and what they mean, are in [docs/camera-capture-measurements.md](../../docs/camera-capture-measurements.md).
|
||
|
||
Not part of the build: CMake only compiles the tools it names.
|
||
|
||
| File | What |
|
||
|---|---|
|
||
| `camera_bench.py` | drives the camera through vmbpy; one JSON line per measurement |
|
||
| `bench.cpp` | CPU cost per frame, using the production `src/camera/JpegXlEncoder.cpp` unchanged |
|
||
| `lossless_check.cpp` | encodes at `jxl_distance = 0` through the production encoder, decodes, and compares pixels |
|
||
| `encode_bench.cpp` | JPEG XL cost per input format, effort, threads and quality; saving throughput and pile-up; CPU temperature, clock, load and power |
|
||
| `make_synthetic.py` | a sharp, detailed stand-in frame (dead-leaves texture with sensor noise) in the camera's formats |
|
||
| `run_camera_tests.sh` | the camera sequence: streaming per format and link limit, under load, auto-exposure, white balance (~25 min) |
|
||
| `run_encode_tests.sh` | the compression, pile-up and heat sequence (~90 min) |
|
||
| `summarize_encode.py` | Markdown tables from a `run_encode_tests.sh` result folder |
|
||
| `raw12_bench.cpp` | the 12-bit pipeline: unpack, de-Bayer, encode and write timed per stage; quality against the frame's own lossless image; pile-up, heat; example files (2026-10-02) |
|
||
| `run_raw12_camera.sh` / `run_raw12_encode.sh` | the 12-bit camera sequence (~25 min) and processing sequence (~1.5 h) |
|
||
| `run_common_frame.sh` | the processing tests on one shared frame (`common-frame/`, git-ignored, rebuilt from 121's lossless 12-bit example), so towers compare exactly (~25–40 min) |
|
||
| `plot_common.py` | the shared-frame tables and charts (per-step time, size per quality setting, saves per second); laptop venv |
|
||
| `compare_examples.py` | scores every 12-bit example against the lossless 12-bit colour image (PSNR, SSIM); laptop venv |
|
||
| `compare_8_vs_12.py` | 8-bit against 12-bit on a dark scene: noise, levels, JPEG XL error per brightness region, crop figures (2026-10-07); laptop venv |
|
||
| `make_figures.py` | example-image figures (crops, difference maps, 8 vs 12-bit shadow) from the example `.jxl` files; laptop venv |
|
||
| `plot_results.py` | every chart and table of `docs/raw12-pipeline-measurements.md` from the results folders; laptop venv |
|
||
| `results-*/` | raw results of each run, one folder per tower and day |
|
||
| `examples-*/` | full-size example `.jxl` files (git-ignored, 100+ MB per tower) |
|
||
|
||
## Running it on a tower
|
||
|
||
The camera accepts one program at a time, so **the tower program must be stopped**, and the tower
|
||
captures nothing until it is started again.
|
||
|
||
```bash
|
||
ssh ggs@<tower>
|
||
sudo systemctl stop fire-gimbal-control
|
||
cd ~/projects/fwt_2a/software/tools/camera-bench
|
||
PY="env GENICAM_GENTL64_PATH=/opt/VimbaX/cti $HOME/.venvs/vmbpy/bin/python camera_bench.py"
|
||
|
||
$PY info # current settings and ranges
|
||
$PY maxfps BayerRG8 250000000 # full res; also BayerRG12p, RGB8; also 450000000
|
||
$PY stream BayerRG8 250000000 14.75 60 # FPS = the max from maxfps; 60 s
|
||
$PY ae BayerRG8 250000000 14.75 dark 100 # dark|bright step, IntensityControllerRate
|
||
$PY wb BayerRG8 250000000 # also RGB8, as the reference
|
||
$PY save BayerRG8 250000000 3 frames # also BayerRG12p and RGB8: bench.cpp needs all three
|
||
```
|
||
|
||
Then the CPU benchmark, built from the repo so it uses the production encoder:
|
||
|
||
```bash
|
||
cd ~/projects/fwt_2a/software
|
||
g++ -O2 -std=c++17 -Iinclude tools/camera-bench/bench.cpp src/camera/JpegXlEncoder.cpp \
|
||
src/core/Logger.cpp -o /tmp/bench $(pkg-config --cflags --libs libjxl libjxl_threads opencv4) -pthread
|
||
cd tools/camera-bench
|
||
/tmp/bench all frames /tmp/bench-out # every measurement
|
||
/tmp/bench raw8true frames /tmp/bench-out # lossless raw 8-bit through a correct lossless path
|
||
/tmp/bench load frames /tmp/bench-out 62 & # encode in a loop, while running `$PY stream ...`
|
||
```
|
||
|
||
`lossless_check.cpp` and `encode_bench.cpp` build the same way. `lossless_check` runs from the
|
||
directory holding `frames/`.
|
||
|
||
### The full sequences (2026-10-01)
|
||
|
||
Copy the programs into one working directory on the tower (for example `~/fgc-bench`), with the
|
||
binaries `bench` and `encode_bench` built there. Then, with the tower program stopped:
|
||
|
||
```bash
|
||
nohup sh run_camera_tests.sh results > camera.log 2>&1 & # ~25 min; also saves frames/
|
||
python3 make_synthetic.py synth 5328 3040 # the camera's resolution; needs numpy + OpenCV
|
||
nohup sh run_encode_tests.sh results-enc > enc.log 2>&1 & # ~90 min
|
||
python3 summarize_encode.py results-enc
|
||
```
|
||
|
||
`make_synthetic.py` needs Python's numpy and OpenCV; where the tower lacks them (tower 111), generate
|
||
the frame elsewhere and copy it. `run_encode_tests.sh` reads package power through a root loop
|
||
(`sudo -n`), which it starts and stops itself. In `run_camera_tests.sh`, auto-exposure is capped at
|
||
20 ms (`EXPOSURE_CAP_US`), so long exposures cannot limit the frame rate being measured.
|
||
|
||
Finally, put the camera back on its profile, confirm frames flow, and restart the program:
|
||
|
||
```bash
|
||
$PY restore ~/projects/fwt_2a/software/config/camera_profiles/rgb/standard.xml
|
||
sudo systemctl start fire-gimbal-control
|
||
rm -rf frames /tmp/bench-out # full-resolution frames are 16-49 MB each
|
||
```
|
||
|
||
The script never saves a camera user set or writes an XML. The program reloads its profile at start
|
||
anyway, so nothing a run changes survives it.
|
||
|
||
### The 12-bit pipeline run (2026-10-02)
|
||
|
||
On the tower, with `raw12_bench` built like `bench` and the tower program stopped:
|
||
|
||
```bash
|
||
python3 make_synthetic.py synth <width> <height> # or copy one in
|
||
nohup sh run_raw12_camera.sh results-r12 > camera-r12.log 2>&1 &
|
||
nohup sh run_raw12_encode.sh results-r12 > encode-r12.log 2>&1 & # after the first ends
|
||
```
|
||
|
||
Then on the laptop, in a venv with matplotlib, numpy, imagecodecs and opencv-python-headless
|
||
(`~/.venvs/fgc-plots`), after copying `results-r12/` and `examples/jxl/` off the tower:
|
||
|
||
```bash
|
||
P=~/.venvs/fgc-plots/bin/python
|
||
$P compare_examples.py examples-<date>-<tower> results-<date>-<tower>/examples_compared.json
|
||
$P make_figures.py examples-<date>-<tower> ../../docs/images/raw12/examples/<tower> [dx,dy sx,sy]
|
||
$P plot_results.py ../../docs/images/raw12 121=results-2026-10-02-tower121 [111=...]
|
||
```
|
||
|
||
## Things to know
|
||
|
||
- **vmbpy 1.2.0 cannot stream under Python 3.14 as installed:** its runtime type check raises
|
||
`NameError: name 'Camera' is not defined`. `camera_bench.py` bypasses that check in its own process.
|
||
- **Per-frame data:** `ChunkEnable` is writable only while `ChunkModeActive` is off, so the script
|
||
enables the items first and activates chunk mode last.
|
||
- **Colour order:** the camera's `BayerRG8` is OpenCV's `COLOR_BayerBG2RGB`.
|
||
- **The camera is the first real one found.** Set `CAM_ID=DEV_...` to choose one.
|
||
- **12-bit JPEG XL from 16-bit words needs `JxlEncoderSetFrameBitDepth(JXL_BIT_DEPTH_FROM_CODESTREAM)`**,
|
||
or libjxl rescales the values and "lossless" is not. Both C++ programs set it.
|
||
- **The `ae` bright step is limited by the camera's limits:** if gain is already at `GainAutoMax`
|
||
and exposure near `ExposureAutoMax`, it cannot force a full ×16. The result records the factor
|
||
actually reached (`forced_factor`).
|