Add a thermal guard that slows the robot before the Pi throttles

hermes_thermal reads the CPU temperature and throttle flags, publishes /health/*, and at
72 / 77 / 82 C scales the camera, face and detector rates down through their live
parameters, restoring them as it cools one level at a time with a margin and a dwell so it
does not flap. The face and detector gained live parameter callbacks for this, the bridge's
robot://status carries a health block (temperature, level, throttle flags now and since
boot), and a bringup test asserts the guard's normal rates equal the real ones.

Live testing on the robot (forcing 73 / 78 / 85 C and back) caught a bug the unit tests
missed: one logging line whose severity changed between warn and info raised in rclpy and
killed the guard on its first cooling step, and the respawn hid it. Fixed with separate call
sites; the tick now survives any error; a full heat-then-cool test drives the real node, and
a mutation check confirmed it fails on the original bug.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
This commit is contained in:
2026-10-05 01:56:16 -07:00
co-authored by Claude Sonnet 5.5
parent 9e730de70e
commit ce7063f647
25 changed files with 583 additions and 15 deletions
+25
View File
@@ -321,6 +321,31 @@ unrelated photo scored 0.16. **Nobody is enrolled yet**; the owner enrols themse
4. A new Omarchy host with a GPU (it runs the local `ornith` model) is arriving: it needs only
`host/omarchy/robot-usb-link.sh` and its SSH key added to the Pi.
## Thermal guard (`ros/hermes_thermal`)
Built because the Pi has no fan (about 60 C open-air with everything running) and an enclosure will be
hotter. `thermal_guard` reads the CPU temperature (`/sys/class/thermal`) and `vcgencmd get_throttled`,
publishes `/health/temperature`, `/health/thermal_level` (ok | warm | hot | critical) and
`/health/throttled`, and sheds load through the target nodes' own live parameters: the camera's
`FrameDurationLimits`, the face's `fps` and the detector's `rate_hz` (the face and detector gained live
parameter callbacks for this). Levels enter at 72 / 77 / 82 C (the Pi throttles at about 80); heating is
acted on at once, cooling goes one level at a time, only below the threshold minus 3 C and after a 20 s
dwell. Rates scale by 1.0 / 0.7 / 0.45 / 0.2 with floors (camera 2 fps, face 8, detector 1 Hz). The
"normal" rates are parameters of the guard and a bringup test asserts they equal the real ones in
`robot.yaml`, so restoring is exact and survives the guard restarting. `fake_temp_c` forces a
temperature for testing. `robot://status` now carries a `health` block (temperature, level, throttle
flags now and since boot, which doubles as a power-supply check).
**Verified live on the robot** by forcing 73 / 78 / 85 C and back: the camera, face and detector rates
followed (measured camera 16.5 -> 11.5 -> 7.2 -> 3.5 Hz and back to 16.5), cooling stepped down critical
-> hot -> warm -> ok exactly one dwell apart, and the guard stayed the same process throughout.
**A live bug the unit tests missed:** one logging call that warned when heating and informed when
cooling raised `ValueError: Logger severity cannot be changed between calls` in rclpy, killing the
guard on its first cooling step (the launch file respawned it, which only looked like a recovery). Fixed
with separate call sites, the tick now catches and logs any error instead of dying, and a test drives
the real node through a full heat-then-cool run; a mutation check (the old logging in a throwaway
copy) confirmed that test fails on exactly that error. Not yet tested under a real thermal load.
When the servos arrive, watch the under-voltage flag in `robot://status` during a sweep.
## Arrival day: the neck (servos), planned and deliberately not built ahead of the hardware
The servos are expected Thursday or Friday at the earliest. The real unknowns cannot be faked (which