Migrate tower 111's host, and resume capture after a test instead of leaving it stopped

docs/hardware-config-plan.md

@@ -402,26 +402,35 @@ Deploy host-before-firmware (A.6). Fix the `Application.cpp:1040` literals in th
 existing fleet that drops yaw 50000 → 30000, a real behaviour change that wants a deliberate rig check
 rather than a silent correction.
 
-> ### ⚠️ The Delta fleet is NOT migrated, and flashing it first would strand it
+> ### ⚠️ Tower 111 is half migrated: host yes, firmware no - and must stay so for now
 >
-> Checked 2026-09-23: tower **111 has no `/etc/fire_gimbal_control/hardware.ini`**.
+> 2026-09-23, through `scripts/install-tower` with the Delta kit: 111 now has a
-> Only tower 121 has been migrated. This is a live hazard rather than a leftover
+> `hardware.ini` (yaw + pitch, MTi on `/dev/ttyS0`, SHT41 on `/dev/i2c-2`,
-> chore, because the firmware half is now written: **an axis with no declared
+> declared and confirmed by `test hardware`), its USB preconditions, the current
-> drivetrain sits in `NEEDCFG` and refuses to move.** Flashing 111 with current
+> host, and a `config.ini` with the keys `hardware.ini` owns commented out. It
-> firmware today would leave it unable to home, with nothing wrong that an
+> homed and resumed capturing on the new host. A rollback copy of the previous
-> operator could see on the board.
+> sources, binary and `config.ini` is in `~ggs/rollback-20260923/` on the tower.
 >
-> The ordering rule above is what prevents it, and it is not advisory:
+> **Every drivetrain value is still a `; TODO`**, because the Delta firmware held
+> register values (GLOBALSCALER / IRUN) and converting them to milliamps needs
+> each driver board's sense resistor, which has to be read off the board. So
+> **do not flash 111 yet**: the new firmware would sit in `NEEDCFG`, refusing to
+> move. The installer blocks the flash for exactly this reason.
 >
-> 1. Write 111 a `hardware.ini` — `hardware/lattepanda-iota/install.sh --stdout`
+> Two corrections to what this section used to say:
->    generates the shape; the Delta's own values come from its `config.ini`.
-> 2. Deploy the **host** and run `test hardware`. It must pass on the old
->    firmware, which is the point of the `_current`/`_declared` metrics reporting
->    an absent field as unconfirmable rather than as a mismatch.
-> 3. Only then flash the firmware, and re-run `test hardware`.
 >
-> Until step 1 is done for every Delta tower, phase 1's gate — "`test hardware`
+> - `test hardware` does **not** pass on the old firmware. An absent echo is
-> green on every existing tower" — is **not met**, whatever 121 reports.
+>   reported as unconfirmable, not as a mismatch - but unconfirmable is still a
+>   FAIL, by design (`checkDeclaredValues`, `checkPinsBuilt`): a check that could
+>   not happen must not read as a pass. On 111 `hw_axes`, `hw_imu`, `hw_env` and
+>   `hw_camera` pass; `hw_motor` fails as "predates the pin echo". Phase 1's gate
+>   for a Delta is therefore "everything but `hw_motor` green, and `hw_motor`
+>   failing only on missing echoes" until the firmware is flashed.
+> - Speed: `scan_speed_deg_s = 8.0` applies to **both** axes. On 111 that is VMAX
+>   5638 for yaw but **157861 for pitch** (14114 counts/deg), above the 150000 it
+>   ran at and far above the firmware's tuned-down 60000; pitch logged encoder
+>   tracking errors of 424 and 473 counts (limit 400) in its first minutes. Set
+>   `[Motor.pitch] scan_speed_deg_s` lower if that persists.
 
 ## 14. Firmware changes, collected
 

docs/known-issues.md

@@ -31,7 +31,7 @@ doctest unit-test suite (`ctest`).
 | 14 | Capture sweep on hardware | **Mechanically verified on the rig (2026-08-07).** A full `--init --start` run homed both axes and completed `MOVE → settle → trigger` for 15/15 waypoints with 0 dropped frames, 0 stalls and 0 acquisition failures. `kSettleTolCounts` (600) and the per-interval timing in [CaptureScheduler.cpp](../src/core/CaptureScheduler.cpp) have **not** been tuned against observed `ST` behaviour yet, and the settle tolerance has not been checked against a sharp image (see #16). |
 | 15 | **Camera profiles unverified on a lit scene** | The `standard`/`night` profiles must be exported from the VimbaX Viewer (see `config/camera_profiles/rgb/README.md`) and have not been measured against real light. Confirm on the rig that `LoadSettings` succeeds, that `standard` sustains 10 fps with `frames_dropped` at 0, and that the auto-exposure lands on a sensible `ExposureTime` outdoors. The only previous rig test ran with the lens covered. |
 | 16 | **`settle_wait_ms` untuned** | `standard` waits 1 s and `night` 2.5 s after standstill before keeping a frame. Both are reasoned (≫ one frame period; long enough for the auto-exposure to see the new scene) but not measured. Check on a lit scene that the kept frame shows no slew blur and that its exposure has actually converged for that heading, and shorten the wait if it has. |
-| 19 | **USB preconditions do not survive a reboot** | `usbcore.usbfs_memory_mb` reverts to 16 (needs 1000) and USB3 LPM re-enables on the camera's port. **Both were wrong on the LattePanda when tested on 2026-08-07**, and frame delivery stalls without them regardless of application code. **Solved on tower 121 by the setup kit** (`hardware/lattepanda-iota/preconditions/`): a systemd-tmpfiles drop-in for usbfs and two udev rules, all installed by `install.sh --preconditions`. Verified 2026-09-23 — `usbfs_memory_mb` reads 1000 after **6 days of uptime**, so it survived the boot, and both rules are in `/etc/udev/rules.d/`; and again across a deliberate reboot the same day, when `install-tower --verify` read `usbfs_memory_mb` 1000 and the camera port's `usb3_lpm_permit` 0 (a check `test hardware camera` now makes too, once `[Test] usb3_lpm_path` is set - the installer sets it). Note it is **tmpfiles, not the kernel cmdline** the original entry proposed. **Still open on the Delta fleet**: tower 111 read 16 and `u1_u2` on 2026-09-23. The Delta kit (`hardware/lattepanda-3-delta/`) carries the same two files, not yet applied. |
+| 19 | **USB preconditions do not survive a reboot** | `usbcore.usbfs_memory_mb` reverts to 16 (needs 1000) and USB3 LPM re-enables on the camera's port. **Both were wrong on the LattePanda when tested on 2026-08-07**, and frame delivery stalls without them regardless of application code. **Solved on tower 121 by the setup kit** (`hardware/lattepanda-iota/preconditions/`): a systemd-tmpfiles drop-in for usbfs and two udev rules, all installed by `install.sh --preconditions`. Verified 2026-09-23 — `usbfs_memory_mb` reads 1000 after **6 days of uptime**, so it survived the boot, and both rules are in `/etc/udev/rules.d/`; and again across a deliberate reboot the same day, when `install-tower --verify` read `usbfs_memory_mb` 1000 and the camera port's `usb3_lpm_permit` 0 (a check `test hardware camera` now makes too, once `[Test] usb3_lpm_path` is set - the installer sets it). Note it is **tmpfiles, not the kernel cmdline** the original entry proposed. **On the Delta fleet**: tower 111 read 16 and `u1_u2` on 2026-09-23, and the Delta kit (`hardware/lattepanda-3-delta/`, the same two files) was applied the same day - 1000 and `0` afterwards, confirmed by `test hardware camera`. **Not yet confirmed across a reboot on a Delta.** |
 
 ## Camera acquisition (USB3) — root cause + approach
 

hardware/lattepanda-3-delta/NOTES.md

@@ -17,9 +17,19 @@ For the Delta fleet: a LattePanda 3 Delta host, an onboard ATmega32U4
 | Preconditions | NOT applied: `usbfs_memory_mb` 16, camera `usb3_lpm_permit` `u1_u2` |
 | `hardware.ini` | none - the tower has never been migrated |
 
-## What this kit has NOT done yet `[untested]`
+## Applied to tower 111, 2026-09-23 `[partial]`
 
-Nothing here has been applied to a Delta. In particular:
+Through `scripts/install-tower`, host half only: preconditions (usbfs 16 -> 1000,
+camera LPM `u1_u2` -> 0), `hardware.ini` (yaw + pitch, MTi, SHT41, standard
+`pins.ini`, every drivetrain value `; TODO`), `config.ini` keys that `hardware.ini`
+now owns commented out, the missing `night` camera profile exported, the host
+rebuilt on Ubuntu 22.04 (GCC 11) and restarted with `--init --start`. It homed and
+resumed capturing. `test hardware`: axes, IMU, env and camera PASS; motor FAIL on
+the old firmware's missing echoes, as expected. Firmware NOT flashed.
+
+## What this kit has NOT done yet
+
+In particular:
 
 - **`pins.ini` is not a traced loom.** It carries the numbers the firmware ran
   on before pins moved into `hardware.ini`. Trace one Delta's wiring and replace

scripts/install-tower

@@ -243,7 +243,11 @@ declared_pins() {
     echo "spi=$spi Y=$y P=$p"
 }
 
-hw_todos() { [[ -r $HW ]] && grep -E '^; TODO' "$HW" | sed 's/^; TODO //' || true; }
+# Each gap in hardware.ini, named with its section: "[Motor.pitch] sw_mode".
+hw_todos() {
+    [[ -r $HW ]] || return 0
+    awk '/^\[/ { sec = $0; sub(/\].*/, "]", sec) } /^; TODO / { sub(/^; TODO /, ""); print sec " " $0 }' "$HW"
+}
 
 # ------------------------------------------------------------------ survey ---
 hr "survey"
@@ -315,7 +319,15 @@ run_hardware_test() {
             [[ -n $after && $after != "$before" ]] && break
             sleep 1
         done
-        [[ -n $after && $after != "$before" ]] || { warn "no test report appeared"; TEST_RC=5; return; }
+        if [[ -z $after || $after == "$before" ]]; then
+            # The program refuses a test while it homes; say that, not "nothing happened".
+            if grep -q "ignored 'test'" "$(latest_log)" 2>/dev/null; then
+                warn "the program refused the test: $(grep "ignored 'test'" "$(latest_log)" | tail -1 | sed 's/.*ignored/ignored/')"
+            else
+                warn "no test report appeared"
+            fi
+            TEST_RC=5; return
+        fi
         grep -E '^\[|  ; ' "$after" | sed 's/^/   /'
         grep -q '^result = PASS' "$after" && TEST_RC=0 || TEST_RC=4
     else
@@ -499,7 +511,7 @@ else
         echo; echo "The motor MCU is now at $S_MCU, but hardware.ini says ${cur_serial:-nothing}."
         ask_yn fix_serial "Update [Serial] device?" y && HW_SETS+=("Serial.device=$S_MCU")
     fi
-    mapfile -t TODOS < <(hw_todos | grep -v '^\[Pins\]' || true)
+    mapfile -t TODOS < <(hw_todos | grep -v ' \[Pins\]' || true)
     TODO_LEFT=${#TODOS[@]}
     if (( ${#TODOS[@]} )); then
         echo; echo "hardware.ini still has gaps. An axis with any of these missing will not move."
@@ -541,9 +553,14 @@ if [[ ! -r $CFG ]]; then
     echo; echo "There is no config.ini yet."
     while :; do
         ask tower_name "Tower name (it is in every MQTT topic)"
-        case "${REPLY_VAL,,}" in ""|dev|exampletower|change_me) warn "that is a placeholder, not a tower name"; unset "ANS[tower_name]"
+        case "${REPLY_VAL,,}" in
-                                 [[ $NONINTERACTIVE -eq 1 ]] && die "tower_name is a placeholder" ;; *) break ;; esac
+            ""|exampletower|change_me)
+                warn "that is a placeholder, not a tower name"; unset "ANS[tower_name]"
+                [[ $NONINTERACTIVE -eq 1 ]] && die "tower_name is a placeholder" ;;
+            *) break ;;
+        esac
     done
+    [[ ${REPLY_VAL,,} == dev ]] && warn "tower_name 'dev' is also what a development box uses - make sure it is this tower's"
     CFG_SETS+=("General.tower_name=$REPLY_VAL")
     if ask_yn mqtt "Report over MQTT?" y; then
         CFG_SETS+=("Features.enable_mqtt=true")
@@ -560,8 +577,11 @@ if [[ ! -r $CFG ]]; then
 else
     tn="$(ini_get "$CFG" General tower_name 2>/dev/null || true)"
     case "${tn,,}" in ""|dev|exampletower)
-        echo; echo "config.ini's tower_name is '$tn' - a placeholder, or another rig's copy."
+        # 'dev' can be real (tower 111 is called that), so it is asked, not refused.
-        ask tower_name "Tower name"; CFG_SETS+=("General.tower_name=$REPLY_VAL") ;;
+        echo; echo "config.ini's tower_name is '$tn' - a placeholder, another rig's copy, or real."
+        ask tower_name "Tower name" "$tn"
+        [[ -z $REPLY_VAL || ${REPLY_VAL,,} == exampletower ]] && die "tower_name '$REPLY_VAL' is a placeholder"
+        [[ $REPLY_VAL != "$tn" ]] && CFG_SETS+=("General.tower_name=$REPLY_VAL") ;;
     esac
 fi
 # The centre each axis needs: its range is centre +/- half the travel, so a
@@ -576,8 +596,24 @@ done
 if [[ -n $S_LPM ]] && [[ -z $(ini_get "$CFG" Test usb3_lpm_path 2>/dev/null || true) ]]; then
     CFG_SETS+=("Test.usb3_lpm_path=$S_LPM")
 fi
+# In an existing config.ini, the keys hardware.ini now owns are dead copies the
+# program warns about at every start. They are commented out, value and all.
+declare -a CFG_DROPS=()
+if [[ $CFG_SEED -eq 0 && -r $HW_VIEW ]]; then
+    ini_keys() { awk '/^[[:space:]]*[;#]/ { next }
+                      /^[[:space:]]*\[/ { sec = $0; sub(/^[[:space:]]*\[/, "", sec); sub(/\].*$/, "", sec); next }
+                      index($0, "=") { k = substr($0, 1, index($0, "=") - 1); gsub(/^[ \t]+|[ \t]+$/, "", k); print sec "." k }' "$1" | sort -u; }
+    while read -r k; do
+        [[ -n $k ]] && CFG_DROPS+=("$k=is set in hardware.ini, which wins over this file")
+    done < <(comm -12 <(ini_keys "$HW_VIEW") <(ini_keys "$CFG"))
+    [[ -n $(ini_get "$HW_VIEW" Hardware imu 2>/dev/null) && -n $(ini_get "$CFG" Features enable_imu 2>/dev/null) ]] &&
+        CFG_DROPS+=("Features.enable_imu=follows [Hardware] imu in hardware.ini")
+    [[ -n $(ini_get "$HW_VIEW" Hardware env 2>/dev/null) && -n $(ini_get "$CFG" Features enable_env 2>/dev/null) ]] &&
+        CFG_DROPS+=("Features.enable_env=follows [Hardware] env in hardware.ini")
+fi
 if [[ $CFG_SEED -eq 1 ]]; then plan config.ini create "seed from the example: ${CFG_SETS[*]}"
-elif (( ${#CFG_SETS[@]} )); then plan config.ini change "set ${CFG_SETS[*]}"
+elif (( ${#CFG_SETS[@]} + ${#CFG_DROPS[@]} )); then
+    plan config.ini change "${CFG_SETS[*]:+set ${CFG_SETS[*]}; }${CFG_DROPS[*]:+comment out ${#CFG_DROPS[@]} key(s) hardware.ini owns (${CFG_DROPS[*]%%=*})}"
 else plan config.ini skip "present, nothing to change"; fi
 [[ $CRED_WRITE -eq 1 ]] && plan credentials create "$CRED (0640 root:$(id -gn))"
 
@@ -668,10 +704,27 @@ sudo -n true 2>/dev/null || { tty_ok && sudo true < /dev/tty; } || die "sudo is
 # The app holds the serial port and the camera. Stopped once, restarted once at
 # the end, the way it was running.
 WAS_RUNNING=$S_RUNNING
+STOPPED_BY_US=0 STOPPED_SVC=0
 stop_app() {
-    if [[ $S_SVC_ACTIVE == active ]]; then sudo systemctl stop fire-gimbal-control; S_SVC_ACTIVE=inactive
+    if [[ $S_SVC_ACTIVE == active ]]; then
-    elif app_running; then "$FGC" stop; fi
+        sudo systemctl stop fire-gimbal-control; S_SVC_ACTIVE=inactive; STOPPED_SVC=1; STOPPED_BY_US=1
+    elif app_running; then "$FGC" stop; STOPPED_BY_US=1; fi
 }
+# If anything fails after the program was stopped, put it back as it was: a
+# tower left stopped by a half-finished install captures nothing, silently.
+put_back() {
+    local rc=$?
+    rm -rf "$WORK"
+    [[ $STOPPED_BY_US -eq 1 ]] && ! app_running || return $rc
+    warn "stopped before finishing - restarting the program as it was"
+    if [[ $STOPPED_SVC -eq 1 ]]; then sudo systemctl start fire-gimbal-control
+    else
+        local a; read -r -a a <<< "${S_ARGS:-}"
+        ( cd "$REPO" && "$FGC" start "${a[@]}" ) > /dev/null
+    fi
+    return $rc
+}
+trap put_back EXIT
 
 if [[ ${DO[packages]} == change ]]; then hr packages; sudo "$KIT/install.sh" --deps; done_step packages; fi
 if [[ ${DO[preconditions]} == change ]]; then hr preconditions; sudo "$KIT/install.sh" --preconditions; done_step preconditions; fi
@@ -735,6 +788,8 @@ if [[ ${DO[config.ini]} != skip ]]; then
             --drop "Network.mqtt_user=comes from $CRED (FGC_MQTT_USER)" \
             --drop "Network.mqtt_pw=comes from $CRED (FGC_MQTT_PW)"
     else
+        for d in "${CFG_DROPS[@]}"; do args+=(--drop "$d"); done
+        cp -p "$CFG" "$CFG.bak-$(date +%Y%m%d-%H%M%S)"
         "$FGC_CONFIG" set "$CFG" --config "$CFG" --hardware "$HW" "${args[@]}" \
             --provenance "$(date -u +%Y-%m-%dT%H:%MZ), install-tower"
     fi
@@ -754,15 +809,19 @@ fi
 if [[ ${DO[profiles]} == change ]]; then
     hr "camera profiles"
     stop_app
-    [[ -x $HOME/.venvs/vmbpy/bin/python ]] || {
+    # A missing profile is worth trying for, not worth abandoning the install over.
-        python3 -m venv "$HOME/.venvs/vmbpy"
+    if { [[ -x $HOME/.venvs/vmbpy/bin/python ]] || {
-        "$HOME/.venvs/vmbpy/bin/pip" install -q /opt/VimbaX/api/python/vmbpy-*.whl
+             python3 -m venv "$HOME/.venvs/vmbpy" &&
-    }
+             "$HOME/.venvs/vmbpy/bin/pip" install -q /opt/VimbaX/api/python/vmbpy-*.whl; }; }; then
         for prof in "${PROF_MISSING[@]}"; do
             ( cd "$REPO" && GENICAM_GENTL64_PATH=/opt/VimbaX/cti "$HOME/.venvs/vmbpy/bin/python" \
-              scripts/export_camera_profile.py --profile "$prof" )
+                  scripts/export_camera_profile.py --profile "$prof" ) ||
+                warn "could not export the $prof profile; export it later (docs/tower-installation.md section 8)"
         done
         done_step profiles
+    else
+        warn "no vmbpy for this Vimba install; camera profiles ${PROF_MISSING[*]} not exported"
+    fi
 fi
 
 hr "config check"

src/core/Application.cpp

@@ -282,6 +282,7 @@ struct Application::Impl {
     std::atomic<bool>       running{true};
     // --test: what run() returns, and where the one-shot test has got to.
     int                     exit_code_ = 0;
+    bool                    resume_capture_after_test_ = false;  // was capturing when a test began
     bool                    oneshot_started_ = false;
     std::chrono::steady_clock::time_point oneshot_since_{};
     std::chrono::steady_clock::time_point oneshot_link_ok_{};
@@ -2187,6 +2188,9 @@ struct Application::Impl {
         test_drift_warn_ = profile.get("drift_warn_pct", 15.0);
         test_drift_gate_ = profile.getBool("drift_gates_pass", false);
 
+        // Paused, not ended: a test run on a tower in service (install-tower
+        // --verify does exactly that) must not leave it idle afterwards.
+        resume_capture_after_test_ = scheduler->captureActive();
         stopCapture();  // free the serial link; don't fight the routine's moves
         LOG_INFO << "starting hardware test (profile '" << profile.name << "')";
         // What `test hardware` needs beyond the motor and IMU: the declaration to
@@ -2334,6 +2338,11 @@ struct Application::Impl {
                     exit_code_ = r.all_pass ? 0 : kExitTestFail;
                     running    = false;
                 }
+                if (resume_capture_after_test_) {
+                    resume_capture_after_test_ = false;
+                    LOG_INFO << "resuming capture, paused for the test";
+                    startCapture();
+                }
             }
         }
         unsigned seq = motor->diagSeq();

tools/fgc_config.cpp

@@ -5,7 +5,8 @@
 //                    [--hardware <hardware.ini>] [--set Section.key=value]...
 //                    [--drop Section.key=note]... [--dry-run]
 //   fgc-config set   <file> --config <config.ini> [--hardware <hardware.ini>]
-//                    --set Section.key=value... [--provenance <text>] [--dry-run]
+//                    [--set Section.key=value]... [--drop Section.key=note]...
+//                    [--provenance <text>] [--dry-run]
 //   fgc-config get   <file> Section.key
 //   fgc-config check --config <config.ini> [--hardware <hardware.ini>]
 //
@@ -14,7 +15,8 @@
 //        saying so. Refuses if the output exists - a tower's config.ini holds
 //        its calibration, and is edited with `set`, never regenerated.
 // set    edits one of the two files in place: comments and order are kept, and
-//        each changed key keeps its previous value as a ";prev" comment.
+//        each changed key keeps its previous value as a ";prev" comment. --drop
+//        comments a key out (without its value), e.g. one hardware.ini now owns.
 // get    prints a value as the program reads it (nothing, exit 1, if unset).
 // check  validates the pair and changes nothing.
 //
@@ -51,8 +53,8 @@ int usage(const std::string& why) {
     std::cerr << "usage:\n"
                  "  fgc-config seed  --example E --out O [--hardware H] [--set S.k=v]... "
                  "[--drop S.k=note]... [--dry-run]\n"
-                 "  fgc-config set   FILE --config C [--hardware H] --set S.k=v... "
+                 "  fgc-config set   FILE --config C [--hardware H] [--set S.k=v]... "
-                 "[--provenance P] [--dry-run]\n"
+                 "[--drop S.k=note]... [--provenance P] [--dry-run]\n"
                  "  fgc-config get   FILE S.k\n"
                  "  fgc-config check --config C [--hardware H]\n";
     return kUsage;
@@ -198,7 +200,7 @@ int cmdSeed(const Args& a) {
 
 int cmdSet(const Args& a) {
     if (a.file.empty() || a.config.empty()) return usage("set needs FILE and --config");
-    if (a.sets.empty()) return usage("set needs at least one --set");
+    if (a.sets.empty() && a.drops.empty()) return usage("set needs at least one --set or --drop");
     std::error_code ec;
     const bool is_cfg = fs::equivalent(a.file, a.config, ec);
     const bool is_hw  = !a.hardware.empty() && fs::equivalent(a.file, a.hardware, ec);
@@ -206,6 +208,10 @@ int cmdSet(const Args& a) {
 
     std::string text;
     if (!readFile(a.file, text)) return usage("cannot read " + a.file);
+    std::vector<std::string> dropped;
+    text = dropIniKeys(text, a.drops, &dropped);
+    for (const auto& k : dropped)
+        std::cerr << "fgc-config: commented out " << k << " (" << a.drops.at(k) << ")\n";
     try {
         text = applyIniSets(text, a.sets, a.provenance);
     } catch (const std::exception& e) {