Make the thermal camera's frame rate hold on the wire

Setting Framerate_Hz while grabbing only thinned out what the SDK handed over;
the camera went on sending 10 fps (and 50 from power-up). Set while the grab
is stopped, once the stream has run, it holds. So the grab is restarted with
the rate set when the first frame is in, and the packet rate of the camera's
port is counted: if it still carries more than 1.5x the rate's frames, the
restart is repeated (up to three times). On tower 111: 5.3 fps, 2 224 packets
a second instead of 22 234, on the second attempt; system load 18 % down from
about 29 %.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

config/config.example.ini

@@ -100,7 +100,8 @@ frame_timeout_ms = 5000
 
 ; --- A thermal camera (InfraTec TarisIR mini), beside the colour one ---------
 ; It takes its picture at the same waypoints and intervals, right after the
-; colour camera, and both carry the same timestamp: RGB/<t>.jxl and TIR/<t>.jxl.
+; colour camera. Each file and CamEvent has the time its own frame arrived; the
+; two of one waypoint share hdg/pit and are at most max_frame_age_ms apart.
 ; The counts are saved 16-bit and always lossless (jxl_distance must stay 0);
 ; T[K] = count x 0.01 under profile 0, x 0.02 under profile 1, emissivity 1.
 ; Its driver, id (the serial) and interface belong in hardware.ini; [Hardware]

docs/architecture.md

@@ -140,8 +140,9 @@ With a thermal camera the scheduler and the pipeline still see one
 the one the keep rule reads) and the thermal one after them. Its `trigger()`
 asks the colour source first; only if that delivered does it ask the thermal
 source, which hands over its newest frame if it is no older than
-`max_frame_age_ms` and otherwise says no at once. Every frame of one trigger
+`max_frame_age_ms` and otherwise says no at once. Each frame keeps the time it
-takes the colour frame's timestamp. The colour camera failing stops the run as
+reached the host, so the two of one waypoint differ by up to that age. The
+colour camera failing stops the run as
 before; the thermal one failing never does (`offline()` says why, Health
 `cams_offline` names it). Thermal frames are `Mono16` counts: the pipeline
 rotates them, takes `tmax`/`tmed`/hot spot on the rotated image

docs/configuration.md

@@ -1065,9 +1065,9 @@ firmware `ST` lines (per-axis state + encoder counts), parsed by
 
 A TarisIR mini beside the colour camera takes its picture at the same waypoints
 and intervals: the colour camera is asked first, and only once it has delivered
-is the thermal one asked for its newest frame. Both frames carry the colour
+is the thermal one asked for its newest frame. Each file and CamEvent has the
-frame's timestamp, so `RGB/<t>.jxl` and `TIR/<t>.jxl` are a pair (and so are
+time its own frame arrived: the two of one waypoint share `hdg`/`pit` and are at
-their CamEvents, see [mqtt-api.md](mqtt-api.md#thermal-camera-cam-tir)).
+most `max_frame_age_ms` apart (see [mqtt-api.md](mqtt-api.md#thermal-camera-cam-tir)).
 
 - **Best effort.** A thermal camera that is missing, unlicensed or has no fresh
   frame costs that picture its thermal image, nothing more: the scan goes on, the

docs/known-issues.md

@@ -173,9 +173,10 @@ scripts for this are worth keeping; note that wrapping them in `timeout` SIGTERM
   were the camera's own NUC every 30 s (a 0.7 s pause in its stream); since 2026-10-15 the program runs the
   NUC at the start of a move instead. A NIR unit would be a third driver
   behind the same key.
-- The InfraTec SDK costs about a quarter of one core on the 3 Delta even between pictures: three threads
+- The InfraTec SDK keeps the CPU awake: about 15 of its threads sleep 1 ms at a time, some 15 000 wake-ups
-  it starts in `dll_init` and one per connection poll the raw socket in `nanosleep` loops (measured on
+  a second, so the cores never settle and the CPU clocks them at turbo (2.8 GHz on the N5105) whenever they
-  tower 111, 2026-10-14). The frame rate does not change it. In the first minutes after deploying, the
+  run. Until 2026-10-15 the camera also sent 50 fps (31 MB/s) because the frame rate was not taken; at the
+  intended 5 fps the system load fell from about 29 % to 18 %, but the wake-ups remain. In the first minutes after deploying, the
   CPU guard slowed the scan once (86 °C, back to normal after 2 min). Watch Health `cpu_c` over a warm
   day before a tower without air flow gets a thermal camera.
 - Tower 111's colour profile does not apply: every start since 2026-10-07 logs `applying profile 'standard'

docs/mqtt-api.md

@@ -153,9 +153,12 @@ their payload is unchanged.
 | `tmed` | int/null | Median of the image, tenths of a °C (the anchor for a display window) |
 | `hotx`, `hoty` | number/null | Where the hottest pixel is: its centre, 0..1 across and down the image **as stored** (after `rotation_deg`), as for detection boxes |
 
-- **Same `time` as the colour picture.** The thermal camera is asked right after
+- **Its own `time`.** The thermal camera is asked right after the colour one, at
-  the colour one, at the same waypoint, and its event and file take the colour
+  the same waypoint, and hands over the newest frame it has. `time` (and the file
-  frame's timestamp: `RGB/<time>.jxl` and `TIR/<time>.jxl` are one pair. A
+  name) is when that frame reached the tower computer, so it is up to
+  `max_frame_age_ms` (400) off the colour picture's, earlier or later. The `RGB`
+  and `TIR` event of one waypoint share `hdg` and `pit` (sampled together) and
+  follow each other in the stream; pair them by that, not by an equal `time`. A
   picture the thermal camera had no fresh frame for has no `TIR` event.
 - **The file holds counts**, 16-bit single channel, lossless:
   `T[K] = count × 0.01` under `cal` 0 (`FF (-40-120C)`), `× 0.02` under `cal` 1

docs/tower-installation.md

@@ -950,8 +950,14 @@ Measured on tower 111 (2026-10-14, SDK 4.3.44, SN 2017744):
 - A profile switch takes about 3 s; frames keep the old scale until
   `Calib_ChangeInProgress` reaches 0, and the program takes none meanwhile.
 - The camera's frame counter (`ImageNum`) is always 0; `MissedPackets` was
-  always 0 on the direct link. It delivers about 50 fps whatever `Framerate_Hz`
+  always 0 on the direct link.
-  says; the program polls it at 10 Hz.
+- Frame rate (2026-10-15): from power-up the camera sends 50 fps, 31 MB/s and
+  22 000 packets a second. `Framerate_Hz` set right after connecting answers ok
+  and changes nothing; set while grabbing it only thins out what the SDK hands
+  over; set while the grab is stopped, after the stream has run, it holds on the
+  wire. The program does that once the first frame is in and checks the packet
+  rate of `[Camera.tir] interface` (5 Hz: about 2 200 packets a second); the log
+  says `5.3 fps on enp1s0 (2224 packets/s), set on attempt 2`.
 - NUC (2026-10-15): the camera runs one every 30 s (`NUC_Interval`) and sends no
   frame for 650-760 ms meanwhile. `NUC_Interval = 0` stops that; `NUC_Perform`
   (1-point) runs one and returns when it is done, after about 0.67 s.

include/fgc/CompositeCameraSource.h

@@ -19,8 +19,9 @@ namespace fgc {
 // frame for the same waypoint. The others are best effort - a follower with
 // no fresh frame costs that picture its thermal image, nothing more.
 //
-// Every frame of one trigger carries the leader's timestamp, so the pair is
+// Every frame keeps its own time, when it reached the host: the thermal frame
-// RGB/<t>.jxl and TIR/<t>.jxl with the same <t> in both CamEvents.
+// is the newest the camera had, so its time is up to max_frame_age_ms off the
+// colour one, and that is what the file name and the CamEvent say.
 //
 // A follower that is briefly without a fresh frame (frameReady() false: the
 // thermal camera's NUC) holds the picture back for up to `follower_wait_ms`,
@@ -71,8 +72,6 @@ private:
     std::vector<int> counts_;      // cameras each child is configured with
     std::vector<int> offsets_;     // global index of each child's camera 0
     FrameCallback    callback_;
-    bool             in_trigger_ = false;
-    long long        shot_ts_    = 0;   // the leader's frame time, during a trigger
     int              follower_wait_ms_;
     long long        waiting_since_ = 0;   // steady ms a picture began waiting for a follower
 };

src/camera/CompositeCameraSource.cpp

@@ -76,12 +76,9 @@ bool CompositeCameraSource::trigger() {
                   << " ms; taking the picture without it";
     }
     waiting_since_ = 0;
-    in_trigger_ = true;
-    shot_ts_    = 0;
     const bool taken = children_[0]->trigger();
     if (taken)
         for (size_t k = 1; k < children_.size(); ++k) children_[k]->trigger();
-    in_trigger_ = false;
     return taken;
 }
 
@@ -100,7 +97,6 @@ void CompositeCameraSource::setFrameCallback(FrameCallback cb) {
     // The leader's frames pass through untouched (they can be 24 MB); its
     // camera 0 is global camera 0, so not even the index changes.
     children_[0]->setFrameCallback([this](const Frame& f) {
-        if (in_trigger_ && shot_ts_ == 0) shot_ts_ = f.timestamp_ms;
         if (callback_) callback_(f);
     });
     for (size_t k = 1; k < children_.size(); ++k) {
@@ -109,7 +105,6 @@ void CompositeCameraSource::setFrameCallback(FrameCallback cb) {
             if (!callback_) return;
             Frame g  = f;
             g.cam_id = f.cam_id + offset;
-            if (in_trigger_ && shot_ts_ != 0) g.timestamp_ms = shot_ts_;
             callback_(g);
         });
     }

src/camera/TarisCameraSource.cpp

@@ -138,6 +138,14 @@ struct TarisCameraSource::Impl {
     double      last_cam_ts       = -1.0;  // its MemInfo time, ms
     long long   polls_since_frame = 0;
     std::map<Ret, int> codes_since_frame;  // getdata answers other than a new frame
+    // The frame rate (see setFrameRate()): set by restarting the grab once the
+    // first frame is in, then checked on the wire.
+    int         rate_attempts     = 0;
+    bool        rate_pending      = false; // restart the grab with it at the next frame
+    long long   wire_check_ms     = 0;     // steady: when to count packets; 0 = none
+    long long   wire_packets0     = -1;
+    long long   wire_start_ms     = 0;
+    size_t      frame_bytes       = 0;
     // NUC run by the program (Options::nuc_interval_ms).
     int         nuc_interval_found = -1;   // the camera's own timer, put back on close
     bool        nuc_ours          = false; // the camera's timer is off and we run it
@@ -408,12 +416,6 @@ struct TarisCameraSource::Impl {
             }
         }
 
-        float rate = static_cast<float>(o.frame_rate_hz);
-        r = api->dev_setparam(dev, param::kFramerateHz, &rate, kTypeSingle);
-        if (!ok(r))
-            LOG_WARN << "thermal camera: cannot set " << o.frame_rate_hz
-                     << " Hz: " << describe(r) << "; it runs at its own rate";
-
         // Polling, before and again after starting: the SDK resets it on startgrab.
         setInt32(param::kPollingEnable, 1);
         r = api->dev_startgrab(dev, 0);
@@ -424,6 +426,10 @@ struct TarisCameraSource::Impl {
         }
         setInt32(param::kPollingEnable, 1);
 
+        rate_attempts = 0;
+        rate_pending  = true;
+        wire_check_ms = 0;
+
         std::string model;
         getString(param::kCameraName, model);
         {
@@ -482,6 +488,7 @@ struct TarisCameraSource::Impl {
             return;
         const size_t need = static_cast<size_t>(w) * h * 2;
         if (bytes < 0 || static_cast<size_t>(bytes) < need) return;
+        frame_bytes = need;
 
         // Polling hands out the newest frame again until a newer one arrives. The
         // camera's frame counter stays 0 (TarisIR mini, SDK 4.3.44), so a new
@@ -580,6 +587,84 @@ struct TarisCameraSource::Impl {
         }
     }
 
+    // ---- frame rate ----------------------------------------------------------
+    // The camera sends 50 fps from power-up: 31 MB/s and 22 000 packets a second
+    // for the SDK and the CPU to take in, for one picture every few seconds. Set
+    // right after connecting, Framerate_Hz answers ok and changes nothing; set
+    // while grabbing, it thins out what the SDK hands over but the camera keeps
+    // sending. Set while the grab is stopped, after the stream has run, it holds
+    // on the wire (tower 111: 5 Hz, 2 100 packets a second). So that is what is
+    // done once the first frame is in, and checked on the port's packet count.
+    void setFrameRate() {
+        float rate = static_cast<float>(o.frame_rate_hz);
+        const Ret r = api->dev_setparam(dev, param::kFramerateHz, &rate, kTypeSingle);
+        ++rate_attempts;
+        if (!ok(r))
+            LOG_WARN << "thermal camera: cannot set " << o.frame_rate_hz << " Hz: " << describe(r)
+                     << "; it runs at its own rate";
+    }
+
+    static long long rxPackets(const std::string& interface) {
+        if (interface.empty()) return -1;
+        std::ifstream f("/sys/class/net/" + interface + "/statistics/rx_packets");
+        long long v = -1;
+        f >> v;
+        return f ? v : -1;
+    }
+
+    void restartForRate() {
+        rate_pending = false;
+        api->dev_stopgrab(dev, 0);
+        setFrameRate();
+        setInt32(param::kPollingEnable, 1);
+        const Ret r = api->dev_startgrab(dev, 0);
+        setInt32(param::kPollingEnable, 1);
+        last_frame_steady = steadyMs();   // grace for the restart
+        if (!ok(r)) {
+            setOffline("cannot restart grabbing: " + describe(r));
+            disconnect();
+            return;
+        }
+        wire_check_ms = steadyMs() + 1000;   // count packets once the stream is up again
+        wire_packets0 = -1;
+    }
+
+    // Packets per second on the camera's port against what the rate should send.
+    void checkWire() {
+        if (wire_check_ms == 0 || steadyMs() < wire_check_ms) return;
+        if (o.interface.empty()) {
+            wire_check_ms = 0;
+            LOG_INFO << "thermal camera: " << o.frame_rate_hz
+                     << " fps set; not checked on the wire (no [Camera.tir] interface)";
+            return;
+        }
+        const long long now = steadyMs(), pk = rxPackets(o.interface);
+        if (pk < 0) { wire_check_ms = 0; return; }
+        if (wire_packets0 < 0) {            // start counting
+            wire_packets0 = pk;
+            wire_start_ms = now;
+            wire_check_ms = now + 3000;
+            return;
+        }
+        wire_check_ms = 0;
+        const double pps      = (pk - wire_packets0) * 1000.0 / std::max<long long>(1, now - wire_start_ms);
+        const double per_fr   = frame_bytes > 0 ? frame_bytes / 1464.0 : 420.0;   // GVSP payload
+        const double fps_wire = pps / per_fr;
+        if (fps_wire <= o.frame_rate_hz * 1.5) {
+            LOG_INFO << "thermal camera: " << std::lround(fps_wire * 10) / 10.0 << " fps on "
+                     << o.interface << " (" << std::lround(pps) << " packets/s), set on attempt "
+                     << rate_attempts;
+        } else if (rate_attempts >= 3) {
+            LOG_WARN << "thermal camera: still " << std::lround(fps_wire) << " fps on " << o.interface
+                     << " (" << std::lround(pps) << " packets/s), not " << o.frame_rate_hz
+                     << "; frame rate not taken";
+        } else {
+            LOG_INFO << "thermal camera: " << std::lround(fps_wire) << " fps on " << o.interface
+                     << ", not " << o.frame_rate_hz << "; restarting the grab again";
+            rate_pending = true;
+        }
+    }
+
     // ---- NUC ---------------------------------------------------------------
     // Switch the camera's own NUC timer off: from now on the program runs the
     // NUC, while the gimbal moves. The timer's value goes back on close. (Read
@@ -783,6 +868,9 @@ struct TarisCameraSource::Impl {
             if (!connected) continue;
             serviceNuc();
             poll();
+            if (rate_pending && last_cam_ts >= 0.0) restartForRate();   // once a frame is in
+            if (!connected) continue;
+            checkWire();
 
             const long long now = steadyMs();
             if (now - last_frame_steady > o.stall_ms) {

tests/test_composite.cpp

@@ -100,12 +100,12 @@ TEST_CASE("the composite numbers the cameras leader first, and counts them all")
     CHECK(r.frames[2].cam_id == 2);   // the thermal camera
 }
 
-TEST_CASE("one trigger gives every frame the leader's time") {
+TEST_CASE("every frame keeps the time it arrived, the follower's too") {
     Rig r;
     REQUIRE(r.src->trigger());
     REQUIRE(r.frames.size() == 2);
     CHECK(r.frames[0].timestamp_ms == 1000);
-    CHECK(r.frames[1].timestamp_ms == 1000);   // not its own 5000
+    CHECK(r.frames[1].timestamp_ms == 5000);   // its own, not the leader's
 }
 
 TEST_CASE("the followers are asked only once the leader has delivered") {

tests/test_tariscamerasource.cpp

@@ -233,6 +233,8 @@ TEST_CASE("the thermal camera connects, trying adapters until one answers") {
     t.cam.start();
     t.sdk.push(29315, 57315);
     REQUIRE(t.online());
+    // The rate is set by restarting the grab once the first frame is in.
+    CHECK(waitFor([&] { std::lock_guard<std::mutex> l(t.sdk.m); return t.sdk.rate > 49.f; }));
     std::lock_guard<std::mutex> l(t.sdk.m);
     REQUIRE(t.sdk.connect_attempts.size() >= 4);
     CHECK(t.sdk.connect_attempts.front() == "SN:2017744|00:0e:c6:aa:bb:cc|192.168.0.200#0");