248 lines
14 KiB
Markdown
248 lines
14 KiB
Markdown
# When the tower computer runs hot
|
||
|
||
In summer the sun and the gimbal heat the LattePanda until its CPU slows itself
|
||
down (throttles). Since the 12-bit raw pipeline, the tower also de-Bayers and
|
||
encodes every picture, ~1.4 CPU-seconds each
|
||
([raw12-pipeline-measurements.md](raw12-pipeline-measurements.md)). This page
|
||
says what happens then, what the program does about it, and how to simulate it.
|
||
|
||
**The rule:** heat or overload only ever **pauses** the program - it never
|
||
exits, never changes the mode, never touches picture quality or resolution. It
|
||
slows the scan, then parks it, and carries on by itself once the computer has
|
||
cooled. An exit would leave the tower dark until someone restarted it; a paused
|
||
tower costs some pictures and comes back alone.
|
||
|
||
## What was already safe, and what was not
|
||
|
||
| | Before | Now |
|
||
|---|---|---|
|
||
| Saving falls behind | The scan holds at its waypoint (`save_queue_max`), so frames never pile up in memory | Same, and a long hold now slows, then pauses the scan |
|
||
| CPU temperature | Read only by `test host` | Read every 2 s; on the Sensors panel, in the log, on MQTT `Health` |
|
||
| Throttling | Invisible (`ThermalInfo::throttled` was never set) | Logged when it starts, once a minute while it lasts, and when it ends |
|
||
| A hot, struggling CPU | Kept at full pace: the camera streaming, the encoder busy, the modem starved | Pictures spaced out (to fit the measured save time), then paused with the camera stream off |
|
||
| Slow 5G link | Each picture's CamEvent held the save worker up to 3 s | CamEvent and Health leave through their own queue and thread |
|
||
| Save workers | Same priority as the control loop, serial, IMU, MQTT | nice +5 (libjxl's threads inherit it) |
|
||
| Camera streaming, nothing saved | ~40% of one core on tower 111, always | Stopped while paused (buffer reuse was tried and measured no gain, so dropped) |
|
||
|
||
## The CPU guard
|
||
|
||
[ThermalGuard.h](../include/fgc/ThermalGuard.h). Every `sample_ms` (2 s) the
|
||
control loop reads the hottest thermal zone, the cores' clock, the kernel's
|
||
throttle counters and how busy the CPU was, plus whether saving is holding the
|
||
scan. The guard is pure - a reading in, a state and log lines out - so every
|
||
rule is unit-tested ([test_thermalguard.cpp](../tests/test_thermalguard.cpp)).
|
||
|
||
```
|
||
CPU >= slow_c (85), or throttling, or the scan waited for saving
|
||
25% of the last minute (or 30 s in one go)
|
||
NORMAL -----------------------------------------------------------> SLOWED
|
||
(as <-- 120 s all clear: CPU < 80, no throttling, waiting under pictures 2x further
|
||
configured) 12%, and saving fast enough for the normal pace --------- apart, or 1.5x the
|
||
| save time if longer
|
||
| CPU >= pause_c (92) for 20 s, or one 120 s wait for saving, |
|
||
| or saving still the bottleneck 120 s after slowing |
|
||
+--------------------------------> PAUSED <-------------------------+
|
||
gimbal parked where it is,
|
||
camera stream stopped, saving drains
|
||
|
|
||
| after 300 s, CPU <= resume_c (75)
|
||
v and nothing left to save
|
||
SLOWED
|
||
```
|
||
|
||
- **Slowed** stretches the interval (`CaptureScheduler::setIntervalScale`):
|
||
at least `slow_factor`, and as far as 1.5 × the measured save time per worker
|
||
when that is longer - so a CPU that saves slowly runs about two-thirds busy
|
||
instead of flat out. Fewer pictures per minute is less encoding and less motor
|
||
work. It returns to the normal pace only once that pace fits what saving
|
||
takes, so a saver that is simply too slow does not flap between the two.
|
||
- **Saving as the bottleneck** is judged by the share of each minute the scan
|
||
waits for it, not only by one long wait: a saver a little too slow holds the
|
||
scan 2-3 s at every waypoint and never once for long, yet keeps the CPU
|
||
pegged (see the results below).
|
||
- **Paused** finishes the waypoint under way, then starts no new move
|
||
(`setPaused`); once parked, the camera stream stops, so the CPU all but idles.
|
||
A snapshot an operator asks for is still taken.
|
||
- **Throttling** counts two ways: the kernel's `thermal_throttle` counters rising
|
||
(heat), or the clock below 50% of the chip's top while at least 50% busy, two
|
||
samples running. The second catches the power-limit slow-downs the counters
|
||
never count - and a clock capped by `cap-cpu.sh`.
|
||
- Missing readings (no thermal zone, no cpufreq) are left out of the rules, never
|
||
read as 0 °C or as hot.
|
||
- The mode does not change: StatusCode still says `0` for a paused auto scan.
|
||
Health says `paused`, and why.
|
||
|
||
All thresholds are `[Thermal]` keys, normally in `hardware.ini`
|
||
([configuration.md](configuration.md#when-the-computer-runs-hot-thermal)).
|
||
|
||
### Seeing it
|
||
|
||
- **Sensors panel** (`s`): a *LattePanda CPU* group - CPU temp, clock, load,
|
||
throttled - with a `HOT` / `THROTTLED` / `SLOWED` / `PAUSED` badge. The
|
||
expanded view adds the per-core clocks and any cap, load averages, throttle
|
||
counts, save time per picture, and the guard's state and reason.
|
||
- **Log**, in plain words:
|
||
```
|
||
CPU is slowing itself to stay cool: 94 °C, running at 1.2 GHz of 3.6 GHz
|
||
CPU still slowed: 14 times in the last minute, 94 °C, 1.1 GHz
|
||
CPU at 94 °C for 20 s: pausing pictures for at least 5 min, until the CPU is at 75 °C or below and saving has caught up
|
||
camera stream stopped while pictures are paused
|
||
CPU at 72 °C: pictures resume, one picture every 10.0 s instead of 5.0 until it has been calm for 2 min
|
||
Saving kept the scan waiting 27% of the last minute: one picture every 6.9 s instead of 2.0 until saving is fast enough for the normal pace, for 2 min
|
||
CPU no longer slowed: 72 °C
|
||
saving: 28 pictures in the last minute, 653 ms each on average (longest 817 ms)
|
||
```
|
||
- **MQTT** `GGS/FWT/<tower>/Health`, retained ([mqtt-api.md](mqtt-api.md#health-payload)).
|
||
- **Console / MQTT Command**: `thermal` (readings and pace), `thermal resume`,
|
||
`thermal off|on`. Allowed in every mode.
|
||
|
||
## Simulating it
|
||
|
||
[tools/thermal-sim/](../tools/thermal-sim/):
|
||
|
||
| Script | What it does | Root |
|
||
|---|---|---|
|
||
| `fake-temp.sh <root> init\|<C> [throttle]\|clock <MHz>\|ramp ...` | A fake `/sys` tree (busy % stays the real `/proc/stat`); the program reads it with `[Thermal] sysfs_root = <root>` | no |
|
||
| `cap-cpu.sh <MHz>\|restore` | Caps every core's clock (`scaling_max_freq`) | yes |
|
||
| `cap-power.sh <W>\|restore` | Caps package power (RAPL), like the 6 W TDP mode | yes |
|
||
| `heat-load.sh <cores> <secs>` | Busy cores competing with the program | no |
|
||
|
||
With `--mock-camera-full` (or `[Features] mock_camera_size = full`) the mock
|
||
camera delivers the real camera's 5328×3040 12-bit raw frames, so the program
|
||
saves - and loads the CPU - as on a tower, on any machine:
|
||
|
||
```
|
||
fire_gimbal_control -c sim/config.ini --mock-camera-full --mock-serial --no-mqtt
|
||
```
|
||
|
||
A CPU that cannot keep up, without root: `systemd-run --user --scope -p CPUQuota=50% fire_gimbal_control ...`.
|
||
|
||
## Results on the desk (2026-10-08)
|
||
|
||
i5-7300U (4 threads, 3.5 GHz - near the IOTA's N150), mock gimbal, tower-size
|
||
12-bit mock frames, `image_interval = 2`, short guard timings for the test
|
||
(`pause_after_s 10`, `calm_s 20`, `min_pause_s 30`, `held_pause_s 60`). A save
|
||
took 0.65-0.85 s.
|
||
|
||
**A. Heat (fake temperatures, real program).** `fake-temp.sh` walked the CPU
|
||
70 → 88 → 94 (+ throttle events) → 72 °C while the scan ran:
|
||
|
||
| Time | CPU | What the program did | Picture spacing |
|
||
|---|---|---|---|
|
||
| 0:00 | 70 °C | normal | 2.1-2.2 s |
|
||
| +0:01 | 88 °C | slowed (within one sample) | 4.2 s |
|
||
| +0:30 | 94 °C, throttling | "CPU is slowing itself to stay cool" at once; paused after 11 s, camera stream off 2 s later once the waypoint finished | none for 29 s |
|
||
| +1:11 | 72 °C | resumed, still slowed, after the 30 s minimum pause | 4.2 s |
|
||
| +2:09 | 72 °C | "CPU no longer slowed" one quiet minute after the last throttle event | 4.2 s |
|
||
| +2:51 | 72 °C | back to the normal pace after 20 s calm | 2.1 s |
|
||
|
||
No waypoint skipped, no overrun warnings, clean shutdown, 68 pictures.
|
||
|
||
**B. A CPU that cannot keep up (no heat).** The program held to half of one core
|
||
(`CPUQuota=50%`), so a save took 4.5-5 s against the 2 s interval:
|
||
|
||
- *Before* the share rule: the save gate held the scan 2-3 s at every waypoint,
|
||
pictures came every ~4.6 s, and the CPU sat **pegged for all 5 minutes**. The
|
||
guard never acted - no single wait reached 30 s - and the log said "saving is
|
||
behind / caught up" twice per picture.
|
||
- *After:* "Saving kept the scan waiting 27% of the last minute: one picture every
|
||
6.9 s instead of 2.0" after the first minute; from then on no holds at all and
|
||
the spacing followed the save time (7-10 s), stable for the rest of the run -
|
||
no flapping back to the 2 s pace it could not hold, no pause needed. The hold
|
||
warning is now said once, not at every waypoint.
|
||
|
||
The dashboard was checked in a real terminal: the Sensors panel's *LattePanda
|
||
CPU* group (88.0 °C, 2.8 GHz, 37 %, "yes, 2x last min", `SLOWED`) and the
|
||
expanded view's CPU section.
|
||
|
||
## Results on tower 111 (3 Delta, N5105, 2026-10-08)
|
||
|
||
Ambient: the enclosure sensor read 21 °C. The tower program was stopped for the
|
||
benchmark and replaced by the test build for the run; the tower's own build and
|
||
config were not touched. Raw data: `tools/camera-bench/results-2026-10-08-tower111-throttle/`.
|
||
|
||
### What a throttled Delta can sustain (`raw12_bench`)
|
||
|
||
The synthetic 5328×3040 12-bit frame (`make_synthetic.py`), edge-aware
|
||
de-Bayer, JPEG XL distance 1.0 effort 3, the tower's 1 worker × 4 threads. Each
|
||
condition: 60 s flat out, then 120 s at one picture per 2 s. `cap-cpu.sh` and
|
||
`cap-power.sh` set the limits; `stress-ng --cpu 2` is "2 busy cores".
|
||
|
||
| Condition | Save time | Most per second | CPU-s / picture | Pile-up after 2 min at one per 2 s | Temp mean / max |
|
||
|---|---|---|---|---|---|
|
||
| none (2.9 GHz) | 1.0 s | 0.98 | 2.3 | 0 | 63 / 80 °C |
|
||
| clock capped 1.6 GHz | 1.7 s | 0.58 | 4.0 | 0 | 57 / 59 °C |
|
||
| clock capped 1.2 GHz | 2.2 s | 0.45 | 5.2 | 6 frames (latency to 14.6 s) | 56 / 58 °C |
|
||
| clock capped 0.8 GHz | 3.3 s | 0.30 | 7.6 | 23 frames (latency to 51 s) | 54 / 56 °C |
|
||
| power capped 6 W | 1.0 s | 0.87 | 2.4-2.7 | 0 | 61 / 74 °C |
|
||
| 2 busy cores | 1.4 s | - | 2.3 | 0 | 79 / **88** °C |
|
||
| 0.8 GHz + 2 busy cores | 4.4 s | 0.22 | 6.9 | 33 frames (latency to 69 s) | 52 / 54 °C |
|
||
|
||
- At the tower's real pace, a picture every **5 s**, it keeps up in every case:
|
||
at 0.8 GHz saving takes two-thirds of the time, and with 2 busy cores on top
|
||
(4.4 s per save) it is on the edge - where the guard's headroom rule would
|
||
space pictures to ~6.6 s.
|
||
- The bare benchmark has no brake, so a slow CPU piles frames up (23-33 in two
|
||
minutes, ~24 MB each). In the program the save gate stops that and the guard
|
||
then slows the scan.
|
||
- The 6 W power cap hardly matters: saving draws only 6-8 W of the 10 W limit.
|
||
- The hottest run was competing load, not saving: 88 °C with 2 busy cores.
|
||
- **The kernel never counted a throttle event** in any run (all counters 0) -
|
||
21 °C was not hot enough. The counters exist on this kernel, so real heat
|
||
will be counted; the clock caps here are seen by the clock rule instead.
|
||
- Tower 111's camera is a 1800 U-1240c, 4024×3036: its real saves are 0.64-0.68 s,
|
||
about two-thirds of the synthetic frame's.
|
||
|
||
### The program on the tower, CPU capped to 0.8 GHz with 2 busy cores
|
||
|
||
The test build with the tower's own config, camera, gimbal and local MQTT
|
||
broker: 3 min normal, 6 min capped + loaded, 3 min restored.
|
||
|
||
| Time | What happened | Pictures | Save time |
|
||
|---|---|---|---|
|
||
| 16:57 | homed, auto scan, Health `normal`, CPU 51 °C | one per 5 s | 0.64-0.68 s |
|
||
| 16:59:32 | clock capped, 2 cores busy | | |
|
||
| 16:59:36 | "CPU is running slow while busy (judged from its clock...): 53 °C, running at 0.8 GHz of 2.9 GHz"; Health `slowed` | one per 10 s | 3.6-3.8 s |
|
||
| every minute | "CPU still slowed: clock low all minute" | | |
|
||
| 17:05:32 | cap lifted | | 0.64-0.72 s |
|
||
| 17:06:31 | "CPU no longer slowed" | one per 10 s | |
|
||
| 17:08:31 | "back to the normal picture pace"; Health `normal` | one per 5 s | |
|
||
|
||
The scan never waited for saving (`held_s` 0 throughout), the CPU ran 66-91%
|
||
busy while capped and loaded, and the program shut down cleanly; 34 pictures,
|
||
13 Health messages on `GGS/FWT/dev/Health`.
|
||
|
||
### What the camera stream costs on its own
|
||
|
||
Manual mode, one snapshot to start the stream, then 60 s of the program's own
|
||
CPU time with nothing saved (`stream_cost.sh`):
|
||
|
||
| Build | CPU while streaming |
|
||
|---|---|
|
||
| installed (main) | 39.2% of one core |
|
||
| reusing the frame buffer instead of a fresh 24 MB per frame | 40.8% of one core |
|
||
|
||
Reusing the buffer made no difference, so it was dropped. The stream itself
|
||
costs **about 40% of one core - a tenth of the computer - all the time it runs**,
|
||
whether or not pictures are saved: the program copies every frame it streams.
|
||
That is why a pause stops the stream. A lower streaming frame rate, or copying
|
||
only the frame that is kept, would cut it while scanning too; both touch the
|
||
fragile camera stream, so they are left for a separate change.
|
||
|
||
### Found on the way
|
||
|
||
- `x86_pkg_temp` is not the only thermal zone on the towers: tower 111 also lists
|
||
the Wi-Fi card (`iwlwifi_1`) and `acpitz`. The guard now reads the CPU package
|
||
zone, so a hot Wi-Fi card cannot pause the scan; `test host` still reports the
|
||
hottest zone.
|
||
- Tower 111's camera refuses its profile at every start since 2026-10-07 17:22
|
||
("applying profile 'standard' ... failed (error -39)"), with the installed
|
||
build as with this one, and captures anyway. Unrelated to this work; worth a
|
||
look.
|
||
|
||
## Still to do
|
||
|
||
- Tower 121 (the IOTA), when cleared: the same matrix and program run.
|
||
- A real hot day: the clock caps stand in for throttling; the kernel counters
|
||
have not yet seen any.
|