GGS
/
fwt_software
8
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 1 Wiki Activity
fwt_software/docs/thermal-throttling.md

14 KiB
Raw Blame History

When the tower computer runs hot

In summer the sun and the gimbal heat the LattePanda until its CPU slows itself down (throttles). Since the 12-bit raw pipeline, the tower also de-Bayers and encodes every picture, ~1.4 CPU-seconds each (raw12-pipeline-measurements.md). This page says what happens then, what the program does about it, and how to simulate it.

The rule: heat or overload only ever pauses the program - it never exits, never changes the mode, never touches picture quality or resolution. It slows the scan, then parks it, and carries on by itself once the computer has cooled. An exit would leave the tower dark until someone restarted it; a paused tower costs some pictures and comes back alone.

What was already safe, and what was not

Before Now
Saving falls behind The scan holds at its waypoint (save_queue_max), so frames never pile up in memory Same, and a long hold now slows, then pauses the scan
CPU temperature Read only by test host Read every 2 s; on the Sensors panel, in the log, on MQTT Health
Throttling Invisible (ThermalInfo::throttled was never set) Logged when it starts, once a minute while it lasts, and when it ends
A hot, struggling CPU Kept at full pace: the camera streaming, the encoder busy, the modem starved Pictures spaced out (to fit the measured save time), then paused with the camera stream off
Slow 5G link Each picture's CamEvent held the save worker up to 3 s CamEvent and Health leave through their own queue and thread
Save workers Same priority as the control loop, serial, IMU, MQTT nice +5 (libjxl's threads inherit it)
Camera streaming, nothing saved ~40% of one core on tower 111, always Stopped while paused (buffer reuse was tried and measured no gain, so dropped)

The CPU guard

ThermalGuard.h. Every sample_ms (2 s) the control loop reads the hottest thermal zone, the cores' clock, the kernel's throttle counters and how busy the CPU was, plus whether saving is holding the scan. The guard is pure - a reading in, a state and log lines out - so every rule is unit-tested (test_thermalguard.cpp).

         CPU >= slow_c (85), or throttling, or the scan waited for saving
         25% of the last minute (or 30 s in one go)
  NORMAL  ----------------------------------------------------------->  SLOWED
  (as      <-- 120 s all clear: CPU < 80, no throttling, waiting under   pictures 2x further
  configured)  12%, and saving fast enough for the normal pace ---------  apart, or 1.5x the
     |                                                                   save time if longer
     |   CPU >= pause_c (92) for 20 s, or one 120 s wait for saving,       |
     |   or saving still the bottleneck 120 s after slowing                |
     +-------------------------------->  PAUSED  <-------------------------+
                                         gimbal parked where it is,
                                         camera stream stopped, saving drains
                                            |
                                            | after 300 s, CPU <= resume_c (75)
                                            v  and nothing left to save
                                          SLOWED
  • Slowed stretches the interval (CaptureScheduler::setIntervalScale): at least slow_factor, and as far as 1.5 × the measured save time per worker when that is longer - so a CPU that saves slowly runs about two-thirds busy instead of flat out. Fewer pictures per minute is less encoding and less motor work. It returns to the normal pace only once that pace fits what saving takes, so a saver that is simply too slow does not flap between the two.
  • Saving as the bottleneck is judged by the share of each minute the scan waits for it, not only by one long wait: a saver a little too slow holds the scan 2-3 s at every waypoint and never once for long, yet keeps the CPU pegged (see the results below).
  • Paused finishes the waypoint under way, then starts no new move (setPaused); once parked, the camera stream stops, so the CPU all but idles. A snapshot an operator asks for is still taken.
  • Throttling counts two ways: the kernel's thermal_throttle counters rising (heat), or the clock below 50% of the chip's top while at least 50% busy, two samples running. The second catches the power-limit slow-downs the counters never count - and a clock capped by cap-cpu.sh.
  • Missing readings (no thermal zone, no cpufreq) are left out of the rules, never read as 0 °C or as hot.
  • The mode does not change: StatusCode still says 0 for a paused auto scan. Health says paused, and why.

All thresholds are [Thermal] keys, normally in hardware.ini (configuration.md).

Seeing it

  • Sensors panel (s): a LattePanda CPU group - CPU temp, clock, load, throttled - with a HOT / THROTTLED / SLOWED / PAUSED badge. The expanded view adds the per-core clocks and any cap, load averages, throttle counts, save time per picture, and the guard's state and reason.
  • Log, in plain words: 
    CPU is slowing itself to stay cool: 94 °C, running at 1.2 GHz of 3.6 GHz
CPU still slowed: 14 times in the last minute, 94 °C, 1.1 GHz
CPU at 94 °C for 20 s: pausing pictures for at least 5 min, until the CPU is at 75 °C or below and saving has caught up
camera stream stopped while pictures are paused
CPU at 72 °C: pictures resume, one picture every 10.0 s instead of 5.0 until it has been calm for 2 min
Saving kept the scan waiting 27% of the last minute: one picture every 6.9 s instead of 2.0 until saving is fast enough for the normal pace, for 2 min
CPU no longer slowed: 72 °C
saving: 28 pictures in the last minute, 653 ms each on average (longest 817 ms)
  • MQTT GGS/FWT/<tower>/Health, retained (mqtt-api.md).
  • Console / MQTT Command: thermal (readings and pace), thermal resume, thermal off|on. Allowed in every mode.

Simulating it

tools/thermal-sim/:

Script What it does Root
fake-temp.sh <root> init|<C> [throttle]|clock <MHz>|ramp ... A fake /sys tree (busy % stays the real /proc/stat); the program reads it with [Thermal] sysfs_root = <root> no
cap-cpu.sh <MHz>|restore Caps every core's clock (scaling_max_freq) yes
cap-power.sh <W>|restore Caps package power (RAPL), like the 6 W TDP mode yes
heat-load.sh <cores> <secs> Busy cores competing with the program no

With --mock-camera-full (or [Features] mock_camera_size = full) the mock camera delivers the real camera's 5328×3040 12-bit raw frames, so the program saves - and loads the CPU - as on a tower, on any machine:

fire_gimbal_control -c sim/config.ini --mock-camera-full --mock-serial --no-mqtt

A CPU that cannot keep up, without root: systemd-run --user --scope -p CPUQuota=50% fire_gimbal_control ....

Results on the desk (2026-10-08)

i5-7300U (4 threads, 3.5 GHz - near the IOTA's N150), mock gimbal, tower-size 12-bit mock frames, image_interval = 2, short guard timings for the test (pause_after_s 10, calm_s 20, min_pause_s 30, held_pause_s 60). A save took 0.65-0.85 s.

A. Heat (fake temperatures, real program). fake-temp.sh walked the CPU 70 → 88 → 94 (+ throttle events) → 72 °C while the scan ran:

Time CPU What the program did Picture spacing
0:00 70 °C normal 2.1-2.2 s
+0:01 88 °C slowed (within one sample) 4.2 s
+0:30 94 °C, throttling "CPU is slowing itself to stay cool" at once; paused after 11 s, camera stream off 2 s later once the waypoint finished none for 29 s
+1:11 72 °C resumed, still slowed, after the 30 s minimum pause 4.2 s
+2:09 72 °C "CPU no longer slowed" one quiet minute after the last throttle event 4.2 s
+2:51 72 °C back to the normal pace after 20 s calm 2.1 s

No waypoint skipped, no overrun warnings, clean shutdown, 68 pictures.

B. A CPU that cannot keep up (no heat). The program held to half of one core (CPUQuota=50%), so a save took 4.5-5 s against the 2 s interval:

  • Before the share rule: the save gate held the scan 2-3 s at every waypoint, pictures came every ~4.6 s, and the CPU sat pegged for all 5 minutes. The guard never acted - no single wait reached 30 s - and the log said "saving is behind / caught up" twice per picture.
  • After: "Saving kept the scan waiting 27% of the last minute: one picture every 6.9 s instead of 2.0" after the first minute; from then on no holds at all and the spacing followed the save time (7-10 s), stable for the rest of the run - no flapping back to the 2 s pace it could not hold, no pause needed. The hold warning is now said once, not at every waypoint.

The dashboard was checked in a real terminal: the Sensors panel's LattePanda CPU group (88.0 °C, 2.8 GHz, 37 %, "yes, 2x last min", SLOWED) and the expanded view's CPU section.

Results on tower 111 (3 Delta, N5105, 2026-10-08)

Ambient: the enclosure sensor read 21 °C. The tower program was stopped for the benchmark and replaced by the test build for the run; the tower's own build and config were not touched. Raw data: tools/camera-bench/results-2026-10-08-tower111-throttle/.

What a throttled Delta can sustain (raw12_bench)

The synthetic 5328×3040 12-bit frame (make_synthetic.py), edge-aware de-Bayer, JPEG XL distance 1.0 effort 3, the tower's 1 worker × 4 threads. Each condition: 60 s flat out, then 120 s at one picture per 2 s. cap-cpu.sh and cap-power.sh set the limits; stress-ng --cpu 2 is "2 busy cores".

Condition Save time Most per second CPU-s / picture Pile-up after 2 min at one per 2 s Temp mean / max
none (2.9 GHz) 1.0 s 0.98 2.3 0 63 / 80 °C
clock capped 1.6 GHz 1.7 s 0.58 4.0 0 57 / 59 °C
clock capped 1.2 GHz 2.2 s 0.45 5.2 6 frames (latency to 14.6 s) 56 / 58 °C
clock capped 0.8 GHz 3.3 s 0.30 7.6 23 frames (latency to 51 s) 54 / 56 °C
power capped 6 W 1.0 s 0.87 2.4-2.7 0 61 / 74 °C
2 busy cores 1.4 s - 2.3 0 79 / 88 °C
0.8 GHz + 2 busy cores 4.4 s 0.22 6.9 33 frames (latency to 69 s) 52 / 54 °C
  • At the tower's real pace, a picture every 5 s, it keeps up in every case: at 0.8 GHz saving takes two-thirds of the time, and with 2 busy cores on top (4.4 s per save) it is on the edge - where the guard's headroom rule would space pictures to ~6.6 s.
  • The bare benchmark has no brake, so a slow CPU piles frames up (23-33 in two minutes, ~24 MB each). In the program the save gate stops that and the guard then slows the scan.
  • The 6 W power cap hardly matters: saving draws only 6-8 W of the 10 W limit.
  • The hottest run was competing load, not saving: 88 °C with 2 busy cores.
  • The kernel never counted a throttle event in any run (all counters 0) - 21 °C was not hot enough. The counters exist on this kernel, so real heat will be counted; the clock caps here are seen by the clock rule instead.
  • Tower 111's camera is a 1800 U-1240c, 4024×3036: its real saves are 0.64-0.68 s, about two-thirds of the synthetic frame's.

The program on the tower, CPU capped to 0.8 GHz with 2 busy cores

The test build with the tower's own config, camera, gimbal and local MQTT broker: 3 min normal, 6 min capped + loaded, 3 min restored.

Time What happened Pictures Save time
16:57 homed, auto scan, Health normal, CPU 51 °C one per 5 s 0.64-0.68 s
16:59:32 clock capped, 2 cores busy
16:59:36 "CPU is running slow while busy (judged from its clock...): 53 °C, running at 0.8 GHz of 2.9 GHz"; Health slowed one per 10 s 3.6-3.8 s
every minute "CPU still slowed: clock low all minute"
17:05:32 cap lifted 0.64-0.72 s
17:06:31 "CPU no longer slowed" one per 10 s
17:08:31 "back to the normal picture pace"; Health normal one per 5 s

The scan never waited for saving (held_s 0 throughout), the CPU ran 66-91% busy while capped and loaded, and the program shut down cleanly; 34 pictures, 13 Health messages on GGS/FWT/dev/Health.

What the camera stream costs on its own

Manual mode, one snapshot to start the stream, then 60 s of the program's own CPU time with nothing saved (stream_cost.sh):

Build CPU while streaming
installed (main) 39.2% of one core
reusing the frame buffer instead of a fresh 24 MB per frame 40.8% of one core

Reusing the buffer made no difference, so it was dropped. The stream itself costs about 40% of one core - a tenth of the computer - all the time it runs, whether or not pictures are saved: the program copies every frame it streams. That is why a pause stops the stream. A lower streaming frame rate, or copying only the frame that is kept, would cut it while scanning too; both touch the fragile camera stream, so they are left for a separate change.

Found on the way

  • x86_pkg_temp is not the only thermal zone on the towers: tower 111 also lists the Wi-Fi card (iwlwifi_1) and acpitz. The guard now reads the CPU package zone, so a hot Wi-Fi card cannot pause the scan; test host still reports the hottest zone.
  • Tower 111's camera refuses its profile at every start since 2026-10-07 17:22 ("applying profile 'standard' ... failed (error -39)"), with the installed build as with this one, and captures anyway. Unrelated to this work; worth a look.

Still to do

  • Tower 121 (the IOTA), when cleared: the same matrix and program run.
  • A real hot day: the clock caps stand in for throttling; the kernel counters have not yet seen any.