diff --git a/docs/hardware-config-plan.md b/docs/hardware-config-plan.md index ad2ece8..0b5e11d 100644 --- a/docs/hardware-config-plan.md +++ b/docs/hardware-config-plan.md @@ -402,26 +402,35 @@ Deploy host-before-firmware (A.6). Fix the `Application.cpp:1040` literals in th existing fleet that drops yaw 50000 → 30000, a real behaviour change that wants a deliberate rig check rather than a silent correction. -> ### ⚠️ The Delta fleet is NOT migrated, and flashing it first would strand it +> ### ⚠️ Tower 111 is half migrated: host yes, firmware no - and must stay so for now > -> Checked 2026-09-23: tower **111 has no `/etc/fire_gimbal_control/hardware.ini`**. -> Only tower 121 has been migrated. This is a live hazard rather than a leftover -> chore, because the firmware half is now written: **an axis with no declared -> drivetrain sits in `NEEDCFG` and refuses to move.** Flashing 111 with current -> firmware today would leave it unable to home, with nothing wrong that an -> operator could see on the board. +> 2026-09-23, through `scripts/install-tower` with the Delta kit: 111 now has a +> `hardware.ini` (yaw + pitch, MTi on `/dev/ttyS0`, SHT41 on `/dev/i2c-2`, +> declared and confirmed by `test hardware`), its USB preconditions, the current +> host, and a `config.ini` with the keys `hardware.ini` owns commented out. It +> homed and resumed capturing on the new host. A rollback copy of the previous +> sources, binary and `config.ini` is in `~ggs/rollback-20260923/` on the tower. > -> The ordering rule above is what prevents it, and it is not advisory: +> **Every drivetrain value is still a `; TODO`**, because the Delta firmware held +> register values (GLOBALSCALER / IRUN) and converting them to milliamps needs +> each driver board's sense resistor, which has to be read off the board. So +> **do not flash 111 yet**: the new firmware would sit in `NEEDCFG`, refusing to +> move. The installer blocks the flash for exactly this reason. > -> 1. Write 111 a `hardware.ini` — `hardware/lattepanda-iota/install.sh --stdout` -> generates the shape; the Delta's own values come from its `config.ini`. -> 2. Deploy the **host** and run `test hardware`. It must pass on the old -> firmware, which is the point of the `_current`/`_declared` metrics reporting -> an absent field as unconfirmable rather than as a mismatch. -> 3. Only then flash the firmware, and re-run `test hardware`. +> Two corrections to what this section used to say: > -> Until step 1 is done for every Delta tower, phase 1's gate — "`test hardware` -> green on every existing tower" — is **not met**, whatever 121 reports. +> - `test hardware` does **not** pass on the old firmware. An absent echo is +> reported as unconfirmable, not as a mismatch - but unconfirmable is still a +> FAIL, by design (`checkDeclaredValues`, `checkPinsBuilt`): a check that could +> not happen must not read as a pass. On 111 `hw_axes`, `hw_imu`, `hw_env` and +> `hw_camera` pass; `hw_motor` fails as "predates the pin echo". Phase 1's gate +> for a Delta is therefore "everything but `hw_motor` green, and `hw_motor` +> failing only on missing echoes" until the firmware is flashed. +> - Speed: `scan_speed_deg_s = 8.0` applies to **both** axes. On 111 that is VMAX +> 5638 for yaw but **157861 for pitch** (14114 counts/deg), above the 150000 it +> ran at and far above the firmware's tuned-down 60000; pitch logged encoder +> tracking errors of 424 and 473 counts (limit 400) in its first minutes. Set +> `[Motor.pitch] scan_speed_deg_s` lower if that persists. ## 14. Firmware changes, collected diff --git a/docs/known-issues.md b/docs/known-issues.md index 0bafa7b..afaeec7 100644 --- a/docs/known-issues.md +++ b/docs/known-issues.md @@ -31,7 +31,7 @@ doctest unit-test suite (`ctest`). | 14 | Capture sweep on hardware | **Mechanically verified on the rig (2026-08-07).** A full `--init --start` run homed both axes and completed `MOVE → settle → trigger` for 15/15 waypoints with 0 dropped frames, 0 stalls and 0 acquisition failures. `kSettleTolCounts` (600) and the per-interval timing in [CaptureScheduler.cpp](../src/core/CaptureScheduler.cpp) have **not** been tuned against observed `ST` behaviour yet, and the settle tolerance has not been checked against a sharp image (see #16). | | 15 | **Camera profiles unverified on a lit scene** | The `standard`/`night` profiles must be exported from the VimbaX Viewer (see `config/camera_profiles/rgb/README.md`) and have not been measured against real light. Confirm on the rig that `LoadSettings` succeeds, that `standard` sustains 10 fps with `frames_dropped` at 0, and that the auto-exposure lands on a sensible `ExposureTime` outdoors. The only previous rig test ran with the lens covered. | | 16 | **`settle_wait_ms` untuned** | `standard` waits 1 s and `night` 2.5 s after standstill before keeping a frame. Both are reasoned (≫ one frame period; long enough for the auto-exposure to see the new scene) but not measured. Check on a lit scene that the kept frame shows no slew blur and that its exposure has actually converged for that heading, and shorten the wait if it has. | -| 19 | **USB preconditions do not survive a reboot** | `usbcore.usbfs_memory_mb` reverts to 16 (needs 1000) and USB3 LPM re-enables on the camera's port. **Both were wrong on the LattePanda when tested on 2026-08-07**, and frame delivery stalls without them regardless of application code. **Solved on tower 121 by the setup kit** (`hardware/lattepanda-iota/preconditions/`): a systemd-tmpfiles drop-in for usbfs and two udev rules, all installed by `install.sh --preconditions`. Verified 2026-09-23 — `usbfs_memory_mb` reads 1000 after **6 days of uptime**, so it survived the boot, and both rules are in `/etc/udev/rules.d/`; and again across a deliberate reboot the same day, when `install-tower --verify` read `usbfs_memory_mb` 1000 and the camera port's `usb3_lpm_permit` 0 (a check `test hardware camera` now makes too, once `[Test] usb3_lpm_path` is set - the installer sets it). Note it is **tmpfiles, not the kernel cmdline** the original entry proposed. **Still open on the Delta fleet**: tower 111 read 16 and `u1_u2` on 2026-09-23. The Delta kit (`hardware/lattepanda-3-delta/`) carries the same two files, not yet applied. | +| 19 | **USB preconditions do not survive a reboot** | `usbcore.usbfs_memory_mb` reverts to 16 (needs 1000) and USB3 LPM re-enables on the camera's port. **Both were wrong on the LattePanda when tested on 2026-08-07**, and frame delivery stalls without them regardless of application code. **Solved on tower 121 by the setup kit** (`hardware/lattepanda-iota/preconditions/`): a systemd-tmpfiles drop-in for usbfs and two udev rules, all installed by `install.sh --preconditions`. Verified 2026-09-23 — `usbfs_memory_mb` reads 1000 after **6 days of uptime**, so it survived the boot, and both rules are in `/etc/udev/rules.d/`; and again across a deliberate reboot the same day, when `install-tower --verify` read `usbfs_memory_mb` 1000 and the camera port's `usb3_lpm_permit` 0 (a check `test hardware camera` now makes too, once `[Test] usb3_lpm_path` is set - the installer sets it). Note it is **tmpfiles, not the kernel cmdline** the original entry proposed. **On the Delta fleet**: tower 111 read 16 and `u1_u2` on 2026-09-23, and the Delta kit (`hardware/lattepanda-3-delta/`, the same two files) was applied the same day - 1000 and `0` afterwards, confirmed by `test hardware camera`. **Not yet confirmed across a reboot on a Delta.** | ## Camera acquisition (USB3) — root cause + approach diff --git a/hardware/lattepanda-3-delta/NOTES.md b/hardware/lattepanda-3-delta/NOTES.md index d576d4e..9d57cb2 100644 --- a/hardware/lattepanda-3-delta/NOTES.md +++ b/hardware/lattepanda-3-delta/NOTES.md @@ -17,9 +17,19 @@ For the Delta fleet: a LattePanda 3 Delta host, an onboard ATmega32U4 | Preconditions | NOT applied: `usbfs_memory_mb` 16, camera `usb3_lpm_permit` `u1_u2` | | `hardware.ini` | none - the tower has never been migrated | -## What this kit has NOT done yet `[untested]` +## Applied to tower 111, 2026-09-23 `[partial]` -Nothing here has been applied to a Delta. In particular: +Through `scripts/install-tower`, host half only: preconditions (usbfs 16 -> 1000, +camera LPM `u1_u2` -> 0), `hardware.ini` (yaw + pitch, MTi, SHT41, standard +`pins.ini`, every drivetrain value `; TODO`), `config.ini` keys that `hardware.ini` +now owns commented out, the missing `night` camera profile exported, the host +rebuilt on Ubuntu 22.04 (GCC 11) and restarted with `--init --start`. It homed and +resumed capturing. `test hardware`: axes, IMU, env and camera PASS; motor FAIL on +the old firmware's missing echoes, as expected. Firmware NOT flashed. + +## What this kit has NOT done yet + +In particular: - **`pins.ini` is not a traced loom.** It carries the numbers the firmware ran on before pins moved into `hardware.ini`. Trace one Delta's wiring and replace diff --git a/scripts/install-tower b/scripts/install-tower index f0ee551..7c829d7 100755 --- a/scripts/install-tower +++ b/scripts/install-tower @@ -243,7 +243,11 @@ declared_pins() { echo "spi=$spi Y=$y P=$p" } -hw_todos() { [[ -r $HW ]] && grep -E '^; TODO' "$HW" | sed 's/^; TODO //' || true; } +# Each gap in hardware.ini, named with its section: "[Motor.pitch] sw_mode". +hw_todos() { + [[ -r $HW ]] || return 0 + awk '/^\[/ { sec = $0; sub(/\].*/, "]", sec) } /^; TODO / { sub(/^; TODO /, ""); print sec " " $0 }' "$HW" +} # ------------------------------------------------------------------ survey --- hr "survey" @@ -315,7 +319,15 @@ run_hardware_test() { [[ -n $after && $after != "$before" ]] && break sleep 1 done - [[ -n $after && $after != "$before" ]] || { warn "no test report appeared"; TEST_RC=5; return; } + if [[ -z $after || $after == "$before" ]]; then + # The program refuses a test while it homes; say that, not "nothing happened". + if grep -q "ignored 'test'" "$(latest_log)" 2>/dev/null; then + warn "the program refused the test: $(grep "ignored 'test'" "$(latest_log)" | tail -1 | sed 's/.*ignored/ignored/')" + else + warn "no test report appeared" + fi + TEST_RC=5; return + fi grep -E '^\[| ; ' "$after" | sed 's/^/ /' grep -q '^result = PASS' "$after" && TEST_RC=0 || TEST_RC=4 else @@ -499,7 +511,7 @@ else echo; echo "The motor MCU is now at $S_MCU, but hardware.ini says ${cur_serial:-nothing}." ask_yn fix_serial "Update [Serial] device?" y && HW_SETS+=("Serial.device=$S_MCU") fi - mapfile -t TODOS < <(hw_todos | grep -v '^\[Pins\]' || true) + mapfile -t TODOS < <(hw_todos | grep -v ' \[Pins\]' || true) TODO_LEFT=${#TODOS[@]} if (( ${#TODOS[@]} )); then echo; echo "hardware.ini still has gaps. An axis with any of these missing will not move." @@ -541,9 +553,14 @@ if [[ ! -r $CFG ]]; then echo; echo "There is no config.ini yet." while :; do ask tower_name "Tower name (it is in every MQTT topic)" - case "${REPLY_VAL,,}" in ""|dev|exampletower|change_me) warn "that is a placeholder, not a tower name"; unset "ANS[tower_name]" - [[ $NONINTERACTIVE -eq 1 ]] && die "tower_name is a placeholder" ;; *) break ;; esac + case "${REPLY_VAL,,}" in + ""|exampletower|change_me) + warn "that is a placeholder, not a tower name"; unset "ANS[tower_name]" + [[ $NONINTERACTIVE -eq 1 ]] && die "tower_name is a placeholder" ;; + *) break ;; + esac done + [[ ${REPLY_VAL,,} == dev ]] && warn "tower_name 'dev' is also what a development box uses - make sure it is this tower's" CFG_SETS+=("General.tower_name=$REPLY_VAL") if ask_yn mqtt "Report over MQTT?" y; then CFG_SETS+=("Features.enable_mqtt=true") @@ -560,8 +577,11 @@ if [[ ! -r $CFG ]]; then else tn="$(ini_get "$CFG" General tower_name 2>/dev/null || true)" case "${tn,,}" in ""|dev|exampletower) - echo; echo "config.ini's tower_name is '$tn' - a placeholder, or another rig's copy." - ask tower_name "Tower name"; CFG_SETS+=("General.tower_name=$REPLY_VAL") ;; + # 'dev' can be real (tower 111 is called that), so it is asked, not refused. + echo; echo "config.ini's tower_name is '$tn' - a placeholder, another rig's copy, or real." + ask tower_name "Tower name" "$tn" + [[ -z $REPLY_VAL || ${REPLY_VAL,,} == exampletower ]] && die "tower_name '$REPLY_VAL' is a placeholder" + [[ $REPLY_VAL != "$tn" ]] && CFG_SETS+=("General.tower_name=$REPLY_VAL") ;; esac fi # The centre each axis needs: its range is centre +/- half the travel, so a @@ -576,8 +596,24 @@ done if [[ -n $S_LPM ]] && [[ -z $(ini_get "$CFG" Test usb3_lpm_path 2>/dev/null || true) ]]; then CFG_SETS+=("Test.usb3_lpm_path=$S_LPM") fi +# In an existing config.ini, the keys hardware.ini now owns are dead copies the +# program warns about at every start. They are commented out, value and all. +declare -a CFG_DROPS=() +if [[ $CFG_SEED -eq 0 && -r $HW_VIEW ]]; then + ini_keys() { awk '/^[[:space:]]*[;#]/ { next } + /^[[:space:]]*\[/ { sec = $0; sub(/^[[:space:]]*\[/, "", sec); sub(/\].*$/, "", sec); next } + index($0, "=") { k = substr($0, 1, index($0, "=") - 1); gsub(/^[ \t]+|[ \t]+$/, "", k); print sec "." k }' "$1" | sort -u; } + while read -r k; do + [[ -n $k ]] && CFG_DROPS+=("$k=is set in hardware.ini, which wins over this file") + done < <(comm -12 <(ini_keys "$HW_VIEW") <(ini_keys "$CFG")) + [[ -n $(ini_get "$HW_VIEW" Hardware imu 2>/dev/null) && -n $(ini_get "$CFG" Features enable_imu 2>/dev/null) ]] && + CFG_DROPS+=("Features.enable_imu=follows [Hardware] imu in hardware.ini") + [[ -n $(ini_get "$HW_VIEW" Hardware env 2>/dev/null) && -n $(ini_get "$CFG" Features enable_env 2>/dev/null) ]] && + CFG_DROPS+=("Features.enable_env=follows [Hardware] env in hardware.ini") +fi if [[ $CFG_SEED -eq 1 ]]; then plan config.ini create "seed from the example: ${CFG_SETS[*]}" -elif (( ${#CFG_SETS[@]} )); then plan config.ini change "set ${CFG_SETS[*]}" +elif (( ${#CFG_SETS[@]} + ${#CFG_DROPS[@]} )); then + plan config.ini change "${CFG_SETS[*]:+set ${CFG_SETS[*]}; }${CFG_DROPS[*]:+comment out ${#CFG_DROPS[@]} key(s) hardware.ini owns (${CFG_DROPS[*]%%=*})}" else plan config.ini skip "present, nothing to change"; fi [[ $CRED_WRITE -eq 1 ]] && plan credentials create "$CRED (0640 root:$(id -gn))" @@ -668,10 +704,27 @@ sudo -n true 2>/dev/null || { tty_ok && sudo true < /dev/tty; } || die "sudo is # The app holds the serial port and the camera. Stopped once, restarted once at # the end, the way it was running. WAS_RUNNING=$S_RUNNING +STOPPED_BY_US=0 STOPPED_SVC=0 stop_app() { - if [[ $S_SVC_ACTIVE == active ]]; then sudo systemctl stop fire-gimbal-control; S_SVC_ACTIVE=inactive - elif app_running; then "$FGC" stop; fi + if [[ $S_SVC_ACTIVE == active ]]; then + sudo systemctl stop fire-gimbal-control; S_SVC_ACTIVE=inactive; STOPPED_SVC=1; STOPPED_BY_US=1 + elif app_running; then "$FGC" stop; STOPPED_BY_US=1; fi } +# If anything fails after the program was stopped, put it back as it was: a +# tower left stopped by a half-finished install captures nothing, silently. +put_back() { + local rc=$? + rm -rf "$WORK" + [[ $STOPPED_BY_US -eq 1 ]] && ! app_running || return $rc + warn "stopped before finishing - restarting the program as it was" + if [[ $STOPPED_SVC -eq 1 ]]; then sudo systemctl start fire-gimbal-control + else + local a; read -r -a a <<< "${S_ARGS:-}" + ( cd "$REPO" && "$FGC" start "${a[@]}" ) > /dev/null + fi + return $rc +} +trap put_back EXIT if [[ ${DO[packages]} == change ]]; then hr packages; sudo "$KIT/install.sh" --deps; done_step packages; fi if [[ ${DO[preconditions]} == change ]]; then hr preconditions; sudo "$KIT/install.sh" --preconditions; done_step preconditions; fi @@ -735,6 +788,8 @@ if [[ ${DO[config.ini]} != skip ]]; then --drop "Network.mqtt_user=comes from $CRED (FGC_MQTT_USER)" \ --drop "Network.mqtt_pw=comes from $CRED (FGC_MQTT_PW)" else + for d in "${CFG_DROPS[@]}"; do args+=(--drop "$d"); done + cp -p "$CFG" "$CFG.bak-$(date +%Y%m%d-%H%M%S)" "$FGC_CONFIG" set "$CFG" --config "$CFG" --hardware "$HW" "${args[@]}" \ --provenance "$(date -u +%Y-%m-%dT%H:%MZ), install-tower" fi @@ -754,15 +809,19 @@ fi if [[ ${DO[profiles]} == change ]]; then hr "camera profiles" stop_app - [[ -x $HOME/.venvs/vmbpy/bin/python ]] || { - python3 -m venv "$HOME/.venvs/vmbpy" - "$HOME/.venvs/vmbpy/bin/pip" install -q /opt/VimbaX/api/python/vmbpy-*.whl - } - for prof in "${PROF_MISSING[@]}"; do - ( cd "$REPO" && GENICAM_GENTL64_PATH=/opt/VimbaX/cti "$HOME/.venvs/vmbpy/bin/python" \ - scripts/export_camera_profile.py --profile "$prof" ) - done - done_step profiles + # A missing profile is worth trying for, not worth abandoning the install over. + if { [[ -x $HOME/.venvs/vmbpy/bin/python ]] || { + python3 -m venv "$HOME/.venvs/vmbpy" && + "$HOME/.venvs/vmbpy/bin/pip" install -q /opt/VimbaX/api/python/vmbpy-*.whl; }; }; then + for prof in "${PROF_MISSING[@]}"; do + ( cd "$REPO" && GENICAM_GENTL64_PATH=/opt/VimbaX/cti "$HOME/.venvs/vmbpy/bin/python" \ + scripts/export_camera_profile.py --profile "$prof" ) || + warn "could not export the $prof profile; export it later (docs/tower-installation.md section 8)" + done + done_step profiles + else + warn "no vmbpy for this Vimba install; camera profiles ${PROF_MISSING[*]} not exported" + fi fi hr "config check" diff --git a/src/core/Application.cpp b/src/core/Application.cpp index b7fef7c..6b2bc71 100644 --- a/src/core/Application.cpp +++ b/src/core/Application.cpp @@ -282,6 +282,7 @@ struct Application::Impl { std::atomic running{true}; // --test: what run() returns, and where the one-shot test has got to. int exit_code_ = 0; + bool resume_capture_after_test_ = false; // was capturing when a test began bool oneshot_started_ = false; std::chrono::steady_clock::time_point oneshot_since_{}; std::chrono::steady_clock::time_point oneshot_link_ok_{}; @@ -2187,6 +2188,9 @@ struct Application::Impl { test_drift_warn_ = profile.get("drift_warn_pct", 15.0); test_drift_gate_ = profile.getBool("drift_gates_pass", false); + // Paused, not ended: a test run on a tower in service (install-tower + // --verify does exactly that) must not leave it idle afterwards. + resume_capture_after_test_ = scheduler->captureActive(); stopCapture(); // free the serial link; don't fight the routine's moves LOG_INFO << "starting hardware test (profile '" << profile.name << "')"; // What `test hardware` needs beyond the motor and IMU: the declaration to @@ -2334,6 +2338,11 @@ struct Application::Impl { exit_code_ = r.all_pass ? 0 : kExitTestFail; running = false; } + if (resume_capture_after_test_) { + resume_capture_after_test_ = false; + LOG_INFO << "resuming capture, paused for the test"; + startCapture(); + } } } unsigned seq = motor->diagSeq(); diff --git a/tools/fgc_config.cpp b/tools/fgc_config.cpp index 2bf9bb5..3574212 100644 --- a/tools/fgc_config.cpp +++ b/tools/fgc_config.cpp @@ -5,7 +5,8 @@ // [--hardware ] [--set Section.key=value]... // [--drop Section.key=note]... [--dry-run] // fgc-config set --config [--hardware ] -// --set Section.key=value... [--provenance ] [--dry-run] +// [--set Section.key=value]... [--drop Section.key=note]... +// [--provenance ] [--dry-run] // fgc-config get Section.key // fgc-config check --config [--hardware ] // @@ -14,7 +15,8 @@ // saying so. Refuses if the output exists - a tower's config.ini holds // its calibration, and is edited with `set`, never regenerated. // set edits one of the two files in place: comments and order are kept, and -// each changed key keeps its previous value as a ";prev" comment. +// each changed key keeps its previous value as a ";prev" comment. --drop +// comments a key out (without its value), e.g. one hardware.ini now owns. // get prints a value as the program reads it (nothing, exit 1, if unset). // check validates the pair and changes nothing. // @@ -51,8 +53,8 @@ int usage(const std::string& why) { std::cerr << "usage:\n" " fgc-config seed --example E --out O [--hardware H] [--set S.k=v]... " "[--drop S.k=note]... [--dry-run]\n" - " fgc-config set FILE --config C [--hardware H] --set S.k=v... " - "[--provenance P] [--dry-run]\n" + " fgc-config set FILE --config C [--hardware H] [--set S.k=v]... " + "[--drop S.k=note]... [--provenance P] [--dry-run]\n" " fgc-config get FILE S.k\n" " fgc-config check --config C [--hardware H]\n"; return kUsage; @@ -198,7 +200,7 @@ int cmdSeed(const Args& a) { int cmdSet(const Args& a) { if (a.file.empty() || a.config.empty()) return usage("set needs FILE and --config"); - if (a.sets.empty()) return usage("set needs at least one --set"); + if (a.sets.empty() && a.drops.empty()) return usage("set needs at least one --set or --drop"); std::error_code ec; const bool is_cfg = fs::equivalent(a.file, a.config, ec); const bool is_hw = !a.hardware.empty() && fs::equivalent(a.file, a.hardware, ec); @@ -206,6 +208,10 @@ int cmdSet(const Args& a) { std::string text; if (!readFile(a.file, text)) return usage("cannot read " + a.file); + std::vector dropped; + text = dropIniKeys(text, a.drops, &dropped); + for (const auto& k : dropped) + std::cerr << "fgc-config: commented out " << k << " (" << a.drops.at(k) << ")\n"; try { text = applyIniSets(text, a.sets, a.provenance); } catch (const std::exception& e) {