kyle/hermes-bot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Bring every doc up to date, and give the repo a Wi-Fi template that exists

Browse Source 
CONTEXT.md had grown by appending and contradicted itself (all four nodes vs six, a retracted CPU
figure, "no calibration", "nobody enrolled", "audio untested as a service", the Wi-Fi-off boot as
not done, a class the tracker replaced). Rewritten as one current-state document organised by
subsystem, with fresh measurements, the owner's decisions, the gotchas, the neck's arrival-day plan
and what is genuinely still open.

README: status, speakers not fitted, new-session note for Hermes, the calibration section (shipped
calibration, coverage hints, outlier dropping, --from-views), running the tests, the docs row.
HANDOFF.md gets a status banner listing where the original brief has drifted. CLAUDE.md starts at
CONTEXT.md. The echo-cancel config no longer talks about a Pi Zero, and the enclosure guide no longer
tells you to centre a servo with a tool that does not exist yet.

pi/network-config.example is a tracked, credentials-free first-boot template: the README pointed at
pi/50-cloud-init.yaml, which is gitignored and so missing from a fresh clone. The dead IdentityCache
(replaced by the tracker) is removed. Docstrings corrected: audio scope publishing, the MCP health
topics, and what the calibration tool does.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
This commit is contained in:
kyle
2026-10-05 08:38:32 -07:00
co-authored by Claude Sonnet 5.5
parent ce7063f647
commit 720a7cfc34
14 changed files with 390 additions and 468 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CONTEXT.md
+284 -369
View File
@@ -1,401 +1,316 @@
# Current state
Last updated 2026-10-04. `HANDOFF.md` is the brief; this file tracks where the work stands.
Last updated 2026-10-05. This file says where the work really stands. `HANDOFF.md` is the original
brief and has drifted (see its banner); where they differ, this file wins. `README.md` is the guide for a
person using the robot.
## Hardware on hand
Raspberry Pi 4 (8 GB), BrainCraft HAT, Arducam UC-346 camera, and nothing else: **no servos, no
fan, no speakers**, no USB-C splitter, no printed enclosure (the camera hangs loose on its ribbon and is
held by hand). The Pi runs at about 62 C open-air with everything on and no fan; an enclosure will be
hotter, so fit the fan before enclosing it. Playback and the servos are untested. Work until those arrive is limited to what the Pi, HAT and camera can do.
Raspberry Pi 4 (8 GB), BrainCraft HAT, Arducam UC-346 camera, and **nothing else: no servos, no fan, no
speakers**, no USB-C splitter, no printed enclosure. The camera hangs loose on its ribbon and is held by
hand. The Pi is powered from talos's USB port (a bench setup, fine without servos). It runs at about
60 °C open-air with everything on; an enclosure will be hotter, so fit the fan before enclosing it.
Playback and the servos are untested because they cannot be.
## What works (verified on the real robot)
- **Face** on the display, started at power-up, with a `sleeping` pose that means "no agent connected".
The eyes glance toward detected faces; the mouth draws the live voice. Gaze direction and the voice
trace were confirmed by the owner on the real display.
- **Camera**, **face detection**, **opt-in recognition** (the owner is enrolled and recognised live).
- **MCP bridge**: `hermes mcp test robot` passes; real Hermes turns on the local `ornith` model call
`look()` and `faces()`.
- **USB network link**, **firewall**, and the whole robot **boots and works with Wi-Fi off**.
- **Thermal guard**, and a reboot-tested **single installer**, `pi/install.sh` (re-run on the live robot,
it changed nothing; **not yet run on a blank card**).
- 122 unit tests on the Pi: face 50, vision 30, audio 10, MCP 16, thermal 12, bringup 4.
## The Pi (`hermes-bot`)
- Ubuntu 24.04.5 server, kernel `6.8.0-1065-raspi` (fully upgraded), flashed from `talos`. User
`hermes` (key-only; passwordless sudo), SSH keys for `kyle@vade` (talos) and `kyle@straylight`.
- On Wi-Fi at `192.168.88.152` (MAC `d8:3a:dd:0d:66:f5`); `eth0` is not cabled. The owner says it has
a static IP on the rack switch; that address is not recorded here yet. **USB link: `10.55.0.2`**
(see package 0b below); `ssh hermes@10.55.0.2` works from talos. `avahi-daemon` is
installed, but mDNS is off in `systemd-resolved` on talos, so `hermes-bot.local` doesn't resolve
there; use the IP.
- **cloud-init applies `network-config` and `user-data` on the first boot only.** To change Wi-Fi
later, edit `/etc/netplan/50-cloud-init.yaml` on the Pi, or bump `instance_id` in `meta-data` on
`system-boot`. `pi/50-cloud-init.yaml` is a template for the Wi-Fi block; it is gitignored
because it holds credentials.
- Installed: ROS 2 Jazzy (`ros-base`, `ros-dev-tools`, colcon, `robot_state_publisher`, `xacro`,
`camera_ros`); `python3-numpy/pil/spidev/lgpio/rpi-lgpio`; a venv `~/venv` (system site packages
visible) with `adafruit-circuitpython-rgb-display` and Blinka. Workspace: `~/ros_ws`, source synced
from `ros/` with `rsync -a --delete --exclude __pycache__ ros/ hermes@<ip>:~/ros_ws/src/`, then
`colcon build --symlink-install`.
- **One unit runs the whole robot:** `robot.service` runs `ros2 launch hermes_bringup robot.launch.py`
as `hermes` (face, camera, face detection, audio levels), so everything is ROS and the face is up
whenever the robot is powered; each node is its own process, respawned by the launch file. Install
with `pi/install-robot-service.sh`. `RuntimeDirectory=robot` gives lgpio a writable working
directory (it drops a pipe in its cwd; systemd's default `/` is not writable). The face starts
first and the camera, vision and audio nodes follow `start_delay` (6 s) later, so the Python
processes do not fight over the CPU. Parameters are in `ros/hermes_bringup/config/robot.yaml`.
`robot-cpu-performance.service` holds the governor at `performance` (the face costs about half the
CPU at full clock). `pi/trim-boot.sh` disables snapd and ModemManager (unused; ~9 s of boot CPU).
- **Boot timing** (reboot-tested, everything returns by itself): the face draws at about 28 s on the
boot clock (kernel 6.9 s, `robot.service` at ~10 s into userspace, then `ros2 launch` takes ~10 s
to start on a cold Pi, then the face ~3 s); camera, audio and vision are up by ~37 s. The owner is
fine with up to a minute, and prefers everything in ROS to a faster non-ROS face; a split
`robot-face.service` that skipped the launch framework only saved 3.7 s and was dropped.
- **Steady-state CPU** with everything running (4 cores): face ~76%, camera ~57% (15 fps, from 30
at ~40% alone), detector ~40% (was 131%), audio ~12%; about half the machine idle, 62 C, no
throttling, no fan fitted. Camera `FrameDurationLimits` is the lever (15 fps ~20% alone, 10 fps
~14%); JPEG quality barely matters. OpenCV's DNN spins every core and burns 3-4x its elapsed
time in CPU, so the detector node pins it to one thread (YuNet 86 ms, SFace 161 ms of CPU), decodes
the JPEG at half size (`IMREAD_REDUCED_COLOR_2`, boxes scaled back to full-size pixels) and
re-identifies a tracked face at most once a second (`IdentityCache`).
- `ros2 node list` with its default short discovery window often shows only some of the nodes; use
`ros2 node list --no-daemon --spin-time 6` (all four appear), or check topics.
- **ROS never leaves the Pi** (owner's decision): `ROS_AUTOMATIC_DISCOVERY_RANGE=LOCALHOST` (set in
`pi/robot-env.sh`, which `source`s everything) and the firewall drops domain 42's DDS ports
(17900-18149) on every interface but loopback. `hermes_mcp` is the only bridge to the outside; a
host that wants data from the robot gets it through an MCP tool, not DDS. This supersedes
HANDOFF.md's `host/ros-shell` talking to the robot. (ROS's setup script exports `SUBNET`, so the
variable must be assigned, not defaulted.)
- **PipeWire** (1.0.5, WirePlumber 0.4.17) runs as `hermes`'s user services (`loginctl enable-linger`).
`~/.config/pipewire/pipewire-pulse.conf.d/50-network-server.conf` (from `pi/pipewire/`) makes
pipewire-pulse listen on TCP 4713 on all interfaces; the firewall is the protection. Defaults are
the WM8960 source and sink (`alsa_input/output.platform-soc_sound.stereo-fallback`). Verified from
talos: `pactl -s tcp:10.55.0.2:4713 info` works; the same on the Wi-Fi address times out.
- Ubuntu 24.04.5 server, kernel `6.8.0-1065-raspi`, flashed from talos. User `hermes` (key-only login,
passwordless sudo; no password is set). SSH keys for `kyle@vade` (talos) and `kyle@straylight`.
- Network: USB link `10.55.0.2` (host `10.55.0.1`); Wi-Fi `192.168.88.152` (MAC `d8:3a:dd:0d:66:f5`);
`eth0` is not cabled (the owner mentioned a static address on the rack switch; not recorded here).
`avahi-daemon` is installed, but mDNS is off in `systemd-resolved` on talos, so use the IP. `rfkill`
is installed.
- **cloud-init applies `network-config` and `user-data` on the first boot only.** To change Wi-Fi later,
edit `/etc/netplan/50-cloud-init.yaml` on the Pi, or bump `instance_id` in `meta-data` on
`system-boot`. `pi/network-config.example` is a credentials-free template for the Wi-Fi block (`pi/50-cloud-init.yaml`
is gitignored, for a filled copy).
- **Install:** `rsync` this repo to the Pi and `sudo ./pi/install.sh`. It installs ROS 2 Jazzy, builds
the Raspberry Pi libcamera fork (pinned commit) into `/opt/libcamera-rpi` and `camera_ros` against it
(`~/camera_ws`), makes `~/venv` (display libraries, OpenCV 4.10, the MCP SDK), downloads the face
models to `~/models`, restores the camera calibration, builds `~/ros_ws`, sets up the USB gadget, the
audio overlay and mixer, the firewall (before anything listens), PipeWire and the services, and
creates the MCP token. Update just the code with `rsync -a --delete --exclude __pycache__ ros/
hermes@<ip>:~/ros_ws/src/`, `colcon build --symlink-install` in `~/ros_ws`, `sudo systemctl restart
robot`. Any shell that runs a node by hand must `source ~/robot-env.sh` first.
- **Services:** `robot.service` runs `ros2 launch hermes_bringup robot.launch.py` as `hermes`: the face,
camera, face detector, audio levels, MCP bridge and thermal guard, each its own process respawned by
the launch file. The face starts first; the others follow `start_delay` (6 s). Parameters:
`ros/hermes_bringup/config/robot.yaml`. `RuntimeDirectory=robot` gives lgpio a writable working
directory (it drops a pipe in its cwd; systemd's default `/` is not writable). Also
`robot-cpu-performance.service` (governor `performance`; the face costs about half the CPU at full
clock), `robot-firewall.service`, `robot-usb-gadget.service`. `pi/trim-boot.sh` disables snapd and
ModemManager (unused; ~9 s of boot CPU).
- **Boot:** the face draws about 28 s after the kernel starts (kernel ~7 s, `robot.service` ~10 s into
userspace, `ros2 launch` itself ~10 s on a cold Pi, the face ~3 s); the other nodes are up by ~37 s.
The owner is fine with up to a minute and prefers everything in ROS: a split face unit that skipped
`ros2 launch` saved only 3.7 s and was dropped. Reboot-tested: everything returns by itself.
- **CPU**, full stack, 4 cores: face ~84%, camera ~63% (15 fps; 30 fps alone is ~40%, 15 is ~20%),
detector ~45%, audio ~15%, MCP ~15%, PipeWire ~12%; about 40% of the machine idle, 60 °C, no
throttling. OpenCV's DNN spins every core and burns 3-4x its elapsed time in CPU, so the detector pins
it to one thread (YuNet 86 ms, SFace 161 ms of CPU), decodes the JPEG at half size and re-identifies a
tracked face at most once a second.
- **ROS never leaves the Pi** (owner's decision): `ROS_AUTOMATIC_DISCOVERY_RANGE=LOCALHOST` (ROS's
setup script exports `SUBNET`, so `pi/robot-env.sh` assigns it) and the firewall drops domain 42's DDS
ports (17900-18149) on every interface but loopback. `hermes_mcp` is the only bridge out. This
supersedes HANDOFF's `host/ros-shell`.
- **Firewall** (`pi/nftables-robot.nft`, loaded at boot before networking by `robot-firewall.service`):
accepts 4713 (audio), 8765 (MCP) and 8766 (reserved, nothing listens) only on `usb0` from `10.55.0.1`
and drops them elsewhere. SSH is deliberately open on all interfaces (key-only), for V2: the owner's call.
- **PipeWire** 1.0.5 / WirePlumber 0.4.17 run as `hermes`'s user services (`loginctl enable-linger`).
`pi/pipewire/50-network-server.conf` makes pipewire-pulse listen on TCP 4713 on all interfaces; the
firewall is the protection. Defaults are the WM8960 source and sink.
## Package 0a (platform proof on Ubuntu 24.04)
## Display and the face (`ros/hermes_face`)
- Step 1 done: `host/check-pi-platform.sh` passes.
- Step 2 done: flashed, booted, upgraded; ROS installed; `ros2 topic list` works.
- **Step 5 (display) done.** ST7789 works on `lgpio` through Blinka (`rpi-lgpio`, no `RPi.GPIO`).
Two gotchas: pass `cs=None` (the kernel drives CE0; claiming GPIO8 raises "GPIO busy"), and the
240x240 glass sits at `y_offset=80` in the controller's RAM, for every rotation (the driver
rotates in software and leaves the scan direction fixed). Mounted pose (microSD end up, USB-C edge
on the left) needs content turned 90 degrees clockwise: `rotation=270`. Full-frame push is
~23 fps through the driver; the face pushes raw RGB565 with `_block` and reaches ~27-28 fps.
- **Camera: works with the Raspberry Pi libcamera fork, not with the stock stacks.** The OV5647 is
detected (`ov5647 10-0036`) and Ubuntu's libcamera 0.2.0 lists it once `libcamera-ipa` is
installed, but capture dies in the Pi IPA (`assertion "it != buffers_.end()" in prepareIsp()`),
and ROS Jazzy's libcamera 0.7.2 dies with `FATAL … A list of V4L2 controls requires a
ControlInfoMap` right after `configure`. Both are deterministic whatever the stream format or
size. Fix: build the fork (`github.com/raspberrypi/libcamera`, commit `6c1dd9d`, 2026-08-17) with
`-Dpipelines=rpi/vc4 -Dipas=rpi/vc4 -Dcam=enabled -Dv4l2=false …` into `/opt/libcamera-rpi`; its
`cam` then captures 1296x972 RGB888 at ~16 fps (dim room). `pi/capture_still.py` takes a still
with it. **`camera_ros` still has to be built against this fork** (the apt one uses ROS's broken
libcamera). Frames are padded (stride 3936) and B,G,R in memory.
- **`camera_ros` 0.7.0 is built from source against the fork** in `~/camera_ws` (overlay on ROS; at
runtime the fork must come first in `LD_LIBRARY_PATH`, which `source ~/robot-env.sh` does; copy
from `pi/robot-env.sh`). Verified: `ros2 run camera_ros camera_node --ros-args -p width:=1296 -p
height:=972 -p format:=RGB888` publishes `/camera/image_raw`, `/camera/image_raw/compressed` (~26
Hz) and `/camera/camera_info`. It costs ~140% of a core with raw and compressed both on; lower the
frame rate or size before running everything together. No calibration file yet.
- **`ros/hermes_vision`**: `hermes_vision.detect` (YuNet, Haar fallback; no ROS imports) and a node
`face_detector` that subscribes to the compressed image and publishes `/vision/faces`
(`vision_msgs/Detection2DArray`, bbox centre/size in pixels, `class_id: face`, score), at `rate_hz`
(default 4, ~47% of a core on a Pi 4); `publish_debug:=true` adds an annotated image. CLI:
`python -m hermes_vision.detect_cli IMAGE`. YuNet 2023mar needs OpenCV 4.8+, so `~/venv` has
`opencv-python-headless==4.10.0.84` with `numpy<2` (system NumPy stays 1.26.4 for ROS), and
`robot-env.sh` puts the venv first on `PYTHONPATH`; YuNet also needs input sides that are multiples
of 32. Tested on a sample image and on a live frame (one face found, upright, no flip needed).
- **Audio capture works (verified 2026-10-04).** The stock `wm8960-soundcard` overlay fails every
stream with `wm8960: No MCLK configured` on Ubuntu 24.04: it puts the 12.288 MHz fixed clock on the
sound card's codec link, but the codec driver reads `mclk` from its own node.
`pi/overlays/hermes-wm8960.dts` is the stock overlay plus `clocks`/`clock-names` on the codec node;
`pi/install-audio-overlay.sh` compiles it to `/boot/firmware/overlays/hermes-wm8960.dtbo`, sets
`dtparam=audio=off` and `dtoverlay=hermes-wm8960`. Capture then opens at 8-48 kHz. Address the
card by name (`hw:CARD=wm8960soundcard,DEV=0`): its index changes (it was card 1, then 0).
- **The HAT's hardware Audio ON/OFF slide switch (right edge, by the "Audio" label) must be ON.**
With it off, the codec still answers on I2C and every software check looks right, but all
recordings are exactly zero. That cost an hour of debugging; check it first.
- Mixer: `Left/Right Input Mixer Boost Switch` default to off and must be on. Settings in use:
Capture 47/47 (+18 dB), `Left/Right Input Boost Mixer LINPUT1/RINPUT1` volume 1, input mixer boost
on. Singing at arm's length: RMS -34 to -49 dBFS, peaks -18 dBFS, no clipping, channels within
1-2 dB. Saved with `alsactl store` and exported as `pi/wm8960_asound.state`; restoring at boot is
not yet verified. No speakers on hand, so playback is untested.
- Not done: PWM (waits for servos). Ask before rebooting the Pi.
- **Display:** the ST7789 works on `lgpio` through Blinka (`rpi-lgpio`, not `RPi.GPIO`). Pass `cs=None`
(the kernel drives CE0; claiming GPIO8 raises "GPIO busy"). The 240x240 glass sits 80 rows into the
controller's RAM, `y_offset=80`, for every rotation (the driver rotates in software and leaves the
scan direction fixed). The mounted pose (microSD end up, USB-C edge on the left) needs the content
turned 90° clockwise: `rotation=270`. The face pushes raw RGB565 with `_block` at ~27-28 fps.
- **Renderer** (`render.py`, no ROS imports): a numpy port of `docs/face-prototype.html`, flat face only
(head and low-poly shapes are not ported). Phosphor and vector styles; tone map is a lookup table,
glow at half resolution, rotation a free `np.rot90`; parameters are clamped. States: `sleeping idle
listening thinking looking speaking`; `looking` returns to the state it interrupted.
- **Node** (`face_node.py`): subscribes `/hermes/state`, `/joint_states`, `/vision/faces`,
`/camera/camera_info`, `/audio/activity`, `/audio/scope/{mic,speaker}`; starts in `sleeping`; logs
every state change (`journalctl -u robot`), so behaviour can be checked without watching the panel.
`fps` can be changed live (the thermal guard does).
- **State arbitration** (`arbiter.py`): sleeping (agent side only) > speaking > listening > thinking >
idle. **Room sound never wakes a sleeping face.**
- **Agent heartbeat:** `/hermes/state` doubles as the heartbeat. After `agent_timeout_s` (10) without
it the face logs `agent gone, sleeping` and sleeps.
- **Gaze:** eyes glance toward the best face (a named one first, then the biggest), smoothed, held
2.5 s, then drift back; sleeping ignores it. Image x flips on the display (a person on the image's
right is on the robot's right, the viewer's left). The renderer's pan sign was wrong (inherited from
the prototype) and is fixed: positive pan = the robot's left = the viewer's right; positive tilt = up,
as the MCP `turn_head` documents. Never seen with a moving head.
- **Voice trace:** while fresh scope data exists, the mouth draws the real waveform (speaker when
speaking, mic when listening), auto-scaled; the owner confirmed it follows how close and loud they are.
- Not done: head/low-poly shapes, DotStar LEDs, a button mute. Open: the IPS panel's black reads dark
gray on camera at full backlight; dimming needs a software PWM on GPIO26.
## Package 0b (USB link): Pi side and talos side working
## Camera and vision (`ros/hermes_vision`, `camera_ros`)
- Pi: `dtoverlay=dwc2,dr_mode=peripheral`; `robot-usb-gadget.service` builds a configfs gadget
(NCM `usb0` + ACM `ttyGS0`, host MAC `02:68:65:72:6d:01`, robot MAC `02:68:65:72:6d:02`) before
networking; `serial-getty@ttyGS0` is enabled; `/etc/netplan/60-usb0.yaml` gives `usb0`
`10.55.0.2/30` with no gateway, so the default route stays on Wi-Fi. Installed by
`pi/install-usb-gadget.sh` (a backup of `config.txt` is `config.txt.bak-usbgadget`). NCM works on
the Pi 4 with kernel 1065; ECM was not needed.
- **netplan gotcha:** match the gadget interface by `name: usb0`. A `macaddress:` match becomes
`PermanentMACAddress=` in networkd, which a gadget interface lacks, so it stays unmanaged.
- talos (Omarchy, NetworkManager): `host/omarchy/robot-usb-link.sh` (needs sudo) adds profile
`hermes-robot-usb`: `10.55.0.1/30` on the interface with MAC `02:68:65:72:6d:01`, priority 100,
never-default. Verified from talos: ping ~2.4 ms, SSH over the link.
- **Firewall:** `robot-firewall.service` loads `pi/nftables-robot.nft` before networking at every
boot (`pi/install-firewall.sh`; it must be loaded before the PipeWire network server is enabled).
It accepts 4713/8765/8766 only on `usb0` from `10.55.0.1`, drops them elsewhere, and drops DDS on
every interface but loopback. Verified: from talos, 4713 connects over `usb0` and times out over
Wi-Fi, while an unused port on the Pi's Wi-Fi address refuses instantly (so only 4713 is blocked).
SSH is deliberately left open on all interfaces (key-only), for V2; the owner's call.
- Not yet verified: serial login prompt (talos's `/dev/ttyACM0` is `root:uucp`; use
`sudo screen /dev/ttyACM0 115200`). The `hermes` account has no password, so the console shows a
login prompt but cannot log in until the owner sets one; no autologin by design. Also not done:
the Pi booting with Wi-Fi off, replug and second-host recovery,
`get_throttled` after a servo sweep on the splitter, the NixOS snippet, and straylight/other-host
testing. Currently the Pi is powered from talos's USB port (bench test, no servos).
- **Camera: only the Raspberry Pi libcamera fork works.** The OV5647 is detected (`ov5647 10-0036`), but
Ubuntu's libcamera 0.2.0 dies in the Pi IPA (`assertion "it != buffers_.end()" in prepareIsp()`) and
ROS Jazzy's 0.7.2 with `FATAL … A list of V4L2 controls requires a ControlInfoMap`, deterministically
for every format and size. The fork (commit `6c1dd9d`, built with `-Dpipelines=rpi/vc4 -Dipas=rpi/vc4
-Dv4l2=false …` into `/opt/libcamera-rpi`) captures; `camera_ros` 0.7.0 is built from source against
it, and the fork must come first in `LD_LIBRARY_PATH` (`robot-env.sh` does it). Stream: 1296x972 at
15 fps (`FrameDurationLimits` in `robot.yaml`); `/camera/image_raw`, `…/compressed`, `camera_info`.
`pi/capture_still.py` takes a still with the fork's `cam`. Frames are padded and B,G,R in memory.
- **Calibration done** (`pi/calibration/…_1296x972.yaml`, restored by the installer): 18 of 24 views,
reprojection error 0.296 px, focal length ~1153 px, centre (653, 463), small distortion with `k3` held
at zero. The first pass fit 0.41 px but with cancelling k1/k2/k3 because the board never reached the
edges; the tool now fixes `k3`, reports coverage, drops blurred outlier views and can re-solve saved
views (`--from-views`). **The top-left and bottom-right corners were never reached**, so the model is
least certain there. The result is valid for the current focus and image size; a changed focus
invalidates it. Tool: `python3 -m hermes_vision.calibrate --cols 9 --rows 6 --square-mm <mm>`
(`docs/checkerboard-9x6.png` can be shown on a screen; the square size only scales distance, not the
intrinsics).
- **Detection** (`detect.py`, no ROS imports): OpenCV YuNet 2023mar (Haar fallback). It needs OpenCV
4.8+ (apt has 4.6), so `~/venv` carries `opencv-python-headless==4.10.0.84` with `numpy<2` (the system
NumPy stays 1.26.4 for ROS), and the venv is first on `PYTHONPATH`; input sides must be multiples of 32.
- **Tracking** (`tracking.py`): faces are matched across frames by box overlap, so a person keeps one
track id; identity is re-resolved at most once a second.
- **Node** `face_detector` publishes `/vision/faces` (`vision_msgs/Detection2DArray`): boxes in
full-size camera pixels, `Detection2D.id` = the track id, `results[0]` the face, `results[1]` the
identity if recognised (class_id = name, score = cosine similarity, not a probability). An unknown
face has no `results[1]`: a name is never made up. `rate_hz` (4) can be changed live.
- **Recognition is opt-in** (`recognize.py`, `people.py`): SFace embeddings on YuNet's landmarks. Nothing
is stored until someone is enrolled with `python3 -m hermes_vision.people enroll NAME`; only
128-number embeddings are kept, in `~/.local/share/hermes/faces.json` (mode 0600, dir 0700, written
atomically), never images. `list`, `forget`, `check`, `enroll-images` too. **The owner is enrolled
(10 samples) and was recognised in 29 of 31 detections live (similarity ~0.9).** A throwaway-database
test recognised 8 of 9 unenrolled frames of the same person, down to side profiles; an unrelated photo
scored 0.16. Not tested: a second person live. Names are never logged (a count is).
## Face (`ros/hermes_face`)
## Audio (`ros/hermes_audio`, `pi/overlays`, PipeWire)
- `render.py`: numpy port of `docs/face-prototype.html`, **flat face only** (head and lowpoly not
ported). Phosphor and vector styles. States: `sleeping idle listening thinking looking speaking`.
`sleeping` (new) is the "no agent connected" pose: eyes closed to thin lines, mouth short, dim and
slowly breathing. `looking` returns to the state it interrupted. Tone mapping is a lookup table
(RGB and RGB565 tables), glow is computed at half resolution, rotation is a free `np.rot90`.
- `display.py`: ST7789 driver plus an async pusher (render overlaps the SPI transfer).
- `face_node.py`: node `hermes_face`; subscribes `/hermes/state` (`std_msgs/String`) and
`/joint_states` (`pan_joint`, `tilt_joint`, radians); starts in `initial_state` (`sleeping`), so it
runs with nothing else up. Parameters: `rotation` (270), `x_offset` (0), `y_offset` (80),
`baudrate`, `fps`, `output` (`st7789` or `none`), `phosphor` (P7), `style`, `persistence_ms`,
`focus`, `jitter`, `intensity`, `initial_state`.
- `demo.py`: no ROS. `python -m hermes_face.demo --sequence` cycles the states on the panel;
`--png-dir DIR` writes a PNG per state without hardware.
- Tests: `python3 -m pytest test` in `ros/hermes_face` (25 pass on the Pi). Renderer parameters
(`focus`, `persistence_ms`, `intensity`, `jitter`) are clamped; `fps` is clamped to 1-60.
- Measured on the Pi 4: ~9 ms per frame in isolation. With the CPU at full clock: 24% of a core at
20 fps and 40% at 30 fps (target was under 40% at 20 fps or more). Under `ondemand` the same
load reads 40% at 20 fps and ~67% at 30 fps. No C++ rewrite was needed.
- **Gaze:** the node subscribes to `/vision/faces` (and `/camera/camera_info` for the image size)
and the eyes glance toward the best face: a recognised one first, then the biggest. Image x flips
on the display (a person on the image's right is on the robot's right, the viewer's left); the
glance is smoothed, held 2.5 s, then drifts back to centre. Sleeping ignores it. The pan sign in
the renderer was wrong (carried over from the prototype) and is fixed: positive pan = the robot's
left = the viewer's right, positive tilt = looks up, as the MCP `turn_head` tool documents.
**Confirmed by the owner on the real display (2026-10-04):** the glance directions look right.
An earlier run showed nothing because the one-off wake message was lost (volatile QoS drops a
sample published before discovery matches); test publishers must republish, as `hermes_mcp`
will. The node logs every state change (`state: idle (agent=..., audio=...)`), so behaviour can be
checked from `journalctl -u robot-face` without watching the panel.
- **Agent heartbeat:** `/hermes/state` doubles as the agent's heartbeat. `hermes_mcp` must republish
its state about once a second; after `agent_timeout_s` (default 10, 0 disables) with none, the
face logs `agent gone, sleeping` and sleeps, so "sleeping" means "no agent connected". Verified:
woke on messages, slept 10 s after the last.
- Not yet: audio levels for the mouth (`/audio/levels`), `hermes_mcp` publishing states, launch
file in `hermes_bringup`, head/lowpoly shapes, DotStar LEDs, button mute.
- Open: IPS black level. The backlight is on at full brightness, so black reads dark gray on
camera. Dimming it needs a software PWM on GPIO26 (not a hardware PWM pin).
- **The stock `wm8960-soundcard` overlay fails on Ubuntu 24.04** (`wm8960: No MCLK configured`): it puts
the 12.288 MHz fixed clock on the card's codec link, but the codec driver reads `mclk` from its own
node. `pi/overlays/hermes-wm8960.dts` adds it there; `pi/install-audio-overlay.sh` compiles it, sets
`dtparam=audio=off` and `dtoverlay=hermes-wm8960`. Capture then opens at 8-48 kHz. Address the card
by name (`hw:CARD=wm8960soundcard,DEV=0`): its index changes.
- **The HAT's hardware Audio ON/OFF switch (right edge) must be ON.** With it off, the codec still
answers on I²C and every software check looks right, but all recordings are exactly zero.
- **Mixer:** `Left/Right Input Mixer Boost` default to off and must be on. `pi/wm8960_asound.state` holds
the settings (capture 47, boost 1). **WirePlumber resets the codec's capture gain to +30 dB** when it
takes the device, so the installer sets PipeWire's source volume to 63% (its scale is cubic: -12 dB,
ALSA `Capture` 46, about +17 dB), which WirePlumber remembers; survives reboots (checked). Quiet-room
floor at this gain: about -43 dBFS; singing at arm's length peaks around -18 dBFS.
- **Node** `audio_state` taps the default source and sink monitor with `parec` (16 kHz mono) and
publishes `/audio/levels`, `/audio/activity` (speaking | listening | quiet) and trigger-locked
waveform windows `/audio/scope/{mic,speaker}` (only while listening or speaking). Audio itself never
goes through ROS. `Activity`: 3 s start-up calibration of an adaptive noise floor, margin 7 dB, 1.2 s
hold so pauses between words do not flicker, steady noise is absorbed after 8 s, the mic is ignored
while speaking. **The flicker retune is unit-tested but not re-confirmed live.**
- Verified: the `speaking` path (a tone played to the sink, no speaker attached, shows on the monitor);
the mic path with the owner's voice. Playback to a real speaker, echo cancel (`pi/pipewire/
60-echo-cancel.conf.optional`) and the host's `robot_mic` naming wait for the speakers.
## Audio levels and the literal voice trace (`ros/hermes_audio`, face integration)
## The MCP bridge and Hermes (`ros/hermes_mcp`)
- `hermes_audio` node `audio_state` taps the default source and the default sink's monitor with
`parec` (16 kHz mono) and publishes `/audio/levels` ([mic_dbfs, speaker_dbfs]), `/audio/activity`
(`speaking` | `listening` | `quiet`) and trigger-locked waveform windows `/audio/scope/{mic,speaker}`
(256 samples). Audio itself never goes through ROS. `hermes_audio.levels.Activity` debounces and
keeps an adaptive noise floor: a 3 s start-up calibration, then it drops fast and rises slowly, and
anything 'listening' without a gap for 8 s is absorbed as ambient noise. The microphone is ignored
while speaking (it would hear the robot).
- The face arbitrates (`hermes_face.arbiter`): sleeping (agent side only) > speaking > listening >
thinking > idle. **Room sound never wakes a sleeping face**: sleeping means "no agent connected".
`looking` stays a timed event owned by the renderer. While fresh (< 0.35 s) scope data exists, the
mouth draws the real waveform (speaker when speaking, mic when listening), auto-scaled.
- **WirePlumber resets the codec's capture gain** to 100% (+30 dB) when it takes the device,
overriding `alsactl`'s saved level. Fix: `pactl set-source-volume @DEFAULT_SOURCE@ 63%` (PipeWire's
scale is cubic: 63% is -12 dB from the +30 dB maximum, so ALSA `Capture` lands at 46, about +17
dB). WirePlumber remembers it; recheck after a reboot. The quiet-room floor at this gain is about
-43 dBFS (preamp and mic noise).
- **Confirmed by the owner:** the mouth draws their live voice and its size follows how close and
loud they are (proportional). The activity detector was retuned after the log showed the face
flickering between listening and idle about twice a second: margin 7 dB (quiet-room noise peaks
sit at about +3 dB over the floor), hold 1.2 s so pauses between words do not drop it. The
flicker fix is covered by unit tests but not yet re-confirmed live (the retest window was silent).
- Verified on the Pi: the `speaking` path (a tone played to the sink, with no speaker attached, shows
on the monitor at -13.5 dBFS, `speaking` for its length, mic ignored); levels and scope publish;
49 face tests, 10 audio tests. **Not yet verified:** the audio node as a service (it was run as a
transient unit).
## The MCP bridge (`ros/hermes_mcp`)
The only way in or out of the ROS graph. Serves MCP (SDK 2.3, streamable HTTP) on
`http://10.55.0.2:8765/mcp`, the USB address only (retrying the bind if the link is not up yet), with
a bearer token. **Tool names are the stable contract with Hermes**: `look()`, `faces()`, `list_people()`, and `turn_head` /
`head_position` only when a head is fitted. With no servos those two are not registered, by design
(`head: false`; the node refuses `head: true` until `hermes_head` exists). `look()` waits for the next
camera frame taken after the call (never a stale one) and passes the camera's JPEG through (1296x972,
~155 KB); it also publishes `looking` on `/hermes/state` so the face squints and flashes.
While an authenticated client is connected (in flight, or heard within `connected_grace_s` = 30) it
republishes `idle` on `/hermes/state` once a second: the agent heartbeat that wakes the face. Verified
end to end from talos over the USB link with a real MCP client: connects, lists `look`, `look()`
returns a JPEG in 0.08 s, a wrong token is refused, the Wi-Fi path is dropped by the firewall, and
the face went `looking` -> `idle` on connect and `agent gone, sleeping` about 40 s after the client
left. The token is in `/etc/robot-mcp.env` (root:hermes, mode 640), generated by
`pi/install-robot-service.sh` and never printed; read it with `sudo cat`. The MCP SDK is in `~/venv`
(`pip install "mcp>=2"`). The old Pi Zero files (`pi/robot_mcp_server.py`, `pi/robot-mcp.service`,
`pi/requirements.txt`, the old `pi/install.sh`, and `tools/`) were deleted (git has them). **Not yet verified:** a real Hermes client, and
whether Hermes passes MCP image content to the model (HANDOFF package 6).
**Hermes on talos is wired to it (2026-10-05, with the owner's authorisation).** Hermes Agent v0.21.5 is
installed on talos (`~/.hermes`; hosted main model `grok-4.6` via xai-oauth; vision is auxiliary and
runs locally, `qwen3-vl-8b-abliterated` on straylight; the `ornith-*` models are the local ones on
straylight/dixie). `~/.hermes/config.yaml` gained an `mcp_servers.robot` block (URL
`http://10.55.0.2:8765/mcp`, `Authorization: Bearer ${ROBOT_MCP_TOKEN}`) and `~/.hermes/.env` gained
`ROBOT_MCP_TOKEN`; backups are `config.yaml.bak-20261005T002659-robot-mcp` and
`.env.bak-20261005T002659-robot-mcp`. `hermes mcp list` shows it enabled and **`hermes mcp test
robot` passes: connected in 789 ms, 1 tool (`look`)**, which is the HANDOFF acceptance check. A fresh
`HERMES_HOME` hangs on first-run setup, so test against the real one.
**A real agent turn works, entirely on local models (2026-10-05):** `hermes chat -q "Call the look tool
once ... what you see" --oneshot -Q --provider custom:llama-router -m ornith-1.5-35b-a3b --max-turns 4`
(straylight's router; no hosted provider, so the image stayed on the LAN). Hermes called
`mcp__robot__look` (0.07 s), the model described the live photo correctly (blurry close-up of a board
with blue LEDs, a red label, camouflage fabric), and the robot's face went `looking` -> `idle`.
**Hermes does not hand MCP image content to the main model.** It saves the JPEG to
`~/.hermes/cache/images/img_<hash>.jpg` on talos (157,147 bytes, the same as the robot sent) and
returns a 72-character reference; the model then calls Hermes's `vision_analyze`, which ran on the local
vision model (`qwen3-vl-8b-abliterated` on straylight) in 29 s. That is HANDOFF package 6's fallback and
Hermes does it itself, so `look()` needs no change. Two consequences: each look() leaves a photo of the
room in talos's Hermes image cache (Hermes's retention, not the robot's), and a look-and-describe turn
takes ~70 s on the local models, mostly the vision call.
### Hermes's feedback on the bridge (2026-10-05) and what was done
Hermes (on talos, from `~/.hermes/cache/scratch/hermes-bot-v0-face-identity.md`) asked for recognitions
instead of raw photos. **Built:** `faces()` returns the faces in view as structured JSON with no photo
(`track_id`, pixel and normalised `bbox`, detection `confidence`, `seen_for_s`, and `identity`
`{name, similarity}` or `null`); `look(send_image="always"|"never")` now returns that same JSON as a
text block plus the photo (default kept as before: the image, so scene description still works);
`list_people()` gives names and sample counts; a `robot://status` resource (hardware present or
absent, what is detected, enrolled count, tools) and prompts `who_is_here` and `look_around` fill the
previously empty `list_resources` / `list_prompts`; `/vision/faces` carries a persistent track id
(`hermes_vision.tracking.FaceTracker`, matching by box overlap, identity re-checked once a second;
`Detection2D.id` is the track id, `results[0]` the face, `results[1]` the identity if recognised).
Verified with a real Hermes turn on the local `ornith` model: it called `faces()` (0.03 s, 593
characters, no image, no vision call) and answered that Kyle was there at 0.78 similarity.
**Declined, deliberately:** embeddings in responses (biometric templates, meaningless to an LLM,
and the registry stays on the robot); age/gender/hat attributes (no model for them here, and guessing
age and gender from faces is unreliable and sensitive); auto-updating a person's face data from new
captures (it lets a stranger slowly become "Kyle" with nobody deciding that); a remote `enroll` tool
(a prompt could make the robot call anyone "Kyle": enrolment stays the owner's step on the robot,
`python3 -m hermes_vision.people enroll NAME`; a gated enrol tool, off by default, is possible if the
owner wants it); detecting people, objects or motion (only faces exist, and `robot://status` says so).
`look()`'s default was not flipped to JSON-only, because the image is what lets Hermes describe a
scene; that default is the owner's call.
### Hermes bug report: "faces and list_people are not registered" (2026-10-05) — not a bridge bug
Checked, not assumed: a raw MCP client lists `faces`, `look`, `list_people`; `hermes mcp test robot`
(fresh connection) discovers all three; there is no tool filter in `~/.hermes/config.yaml`. Hermes's own
`agent.log` explains the report: my robot restart at 01:17 ended the MCP session (`keepalive failed ...
Session not found`), Hermes reconnected and logged `registered 7 tool(s): mcp__robot__faces, look,
list_people, ...` at 01:18:46, and a session that started afterwards (`...011845...`) called
`mcp__robot__faces` fine. The reporting session (`...003532...`) started at 00:35, before `faces`
existed, and keeps its start-time snapshot of the tool table: it still reaches live things
(`list_prompts`, `read_resource`) but gets `'mcp__robot__faces' is not a known tool name`. Fix on the
owner's side: start a new Hermes session after the robot's tool set changes. For Hermes: refresh a
session's tool table (and its deferred-tool index) when the MCP server is re-registered. We keep the
bridge's stateful sessions on purpose: the `Session not found` after a restart is what makes Hermes
reconnect and re-register (stateless HTTP would hide a changed tool set from a connected client).
## Face recognition (opt-in)
`hermes_vision.recognize` / `people`: SFace embeddings on YuNet's landmarks. Nothing is stored until
someone is enrolled (`python -m hermes_vision.people enroll NAME`, live from the camera topic, or
`enroll-images`); only 128-number embeddings are kept, in `~/.local/share/hermes/faces.json` (mode
0600, directory 0700, written atomically), never images. `list`, `forget NAME` and `check IMAGE`
manage and test it. The detector node puts a recognised name in `Detection2D.id` and picks up
database changes without a restart; its log prints a count, not names. Checked with a throwaway
database: 8 of 9 unenrolled frames of the same person were recognised, down to side profiles
(similarity 0.52-0.66), the miss being a head tipped back (0.34 against the 0.363 threshold); an
unrelated photo scored 0.16. **Nobody is enrolled yet**; the owner enrols themselves live.
## Next
1. **Camera calibration: tool built and tested, not yet run.** `python3 -m hermes_vision.calibrate
--cols 9 --rows 6 --square-mm <measured>` (after `source ~/robot-env.sh`, robot running) collects 20
varied views of a checkerboard (`docs/checkerboard-9x6.png` can be shown on a screen; measure a
square), solves with OpenCV and writes the camera_info YAML camera_ros looks for
(`~/.ros/camera_info/ov5647__base_soc_i2c0mux_i2c_1_ov5647_36_1296x972.yaml`: the name includes the
resolution). The solver recovers a known synthetic camera to within 1%. It needs the owner holding a
board in front of the camera, and the lens focus set first (the lens turns if not glued; earlier
close-range frames were blurry). V2's perception needs the result.
2. Waiting on hardware: the servos (see "Arrival day" above), the speakers
(playback, echo cancel, the `robot_mic` naming on the host, Hermes talking aloud), the fan (before
any enclosure), the splitter, the printed enclosure (measure first, see `enclosure/README.md`).
3. Still open: replug and second-host recovery of the USB link, the serial console login, an
`agent_timeout_s` check against a long idle Hermes session, the retuned listening detector live
(unit-tested only), a second person in frame, running `pi/install.sh` on a blank card, and the
NixOS host config (skipped for now by the owner).
4. A new Omarchy host with a GPU (it runs the local `ornith` model) is arriving: it needs only
`host/omarchy/robot-usb-link.sh` and its SSH key added to the Pi.
- Serves MCP (SDK 2.3, streamable HTTP) on `http://10.55.0.2:8765/mcp`: the USB address only (it retries
the bind until the link is up), bearer token in `/etc/robot-mcp.env` (root:hermes, mode 640, made by
the installer, never printed; read it with `sudo cat`). **Tool names are the stable contract with
Hermes.** Tools: `look(send_image="always"|"never")` (the faces summary as JSON text plus the next
fresh JPEG, never a stale frame; it also makes the face squint), `faces()` (the faces in view, no
photo: `track_id`, `bbox`, `confidence`, `seen_for_s`, `identity` `{name, similarity}` or `null`),
`list_people()` (names and sample counts only), and `turn_head` / `head_position` **only when a head
is fitted** (`head: false` now; the node refuses `head: true` until `hermes_head` exists). Resource
`robot://status` (hardware present or absent, what is detected, enrolled count, tools, and health:
temperature, thermal level, throttle flags now and since boot) and prompts `who_is_here`,
`look_around` make it discoverable. Sessions are stateful on purpose (see the Hermes note below).
- **Heartbeat:** while an authenticated client is connected (in flight, or heard within
`connected_grace_s` = 30) it republishes `idle` on `/hermes/state` once a second; the face wakes on
connect and sleeps about 40 s after the client leaves.
- **Hermes on talos is wired to it** (with the owner's authorisation). Hermes Agent v0.21.5 is in
`~/.hermes` (hosted main model `grok-4.6` via xai-oauth; vision is auxiliary and local,
`qwen3-vl-8b-abliterated` on straylight; the `ornith-*` models are the local ones on straylight and
dixie). `config.yaml` has an `mcp_servers.robot` block and `.env` has `ROBOT_MCP_TOKEN`; backups:
`config.yaml.bak-20261005T002659-robot-mcp`, `.env.bak-20261005T002659-robot-mcp`. `hermes mcp test
robot` passes. A fresh `HERMES_HOME` hangs on first-run setup, so test against the real one.
- **Real agent turns, entirely on local models:** `hermes chat -q "..." --oneshot -Q --provider
custom:llama-router -m ornith-1.5-35b-a3b --max-turns 5`. `look()` works; **Hermes does not hand MCP
images to the main model**: it saves the JPEG to `~/.hermes/cache/images/` on talos and returns a short
reference, and the model calls `vision_analyze` (local vision model, ~29 s). So a look-and-describe turn
takes ~70 s, and each look leaves a photo in talos's Hermes cache (Hermes's retention). `faces()` is
the cheap path: 0.03 s, no image, and Hermes answered "Kyle is at the desk, similarity 0.78".
- **Hermes's feedback (`~/.hermes/cache/scratch/hermes-bot-v0-face-identity.md`).** Built: `faces()`,
`look` returning that JSON, `list_people`, `robot://status` and prompts (previously empty), track ids.
**Declined, deliberately:** embeddings in responses (biometric templates, meaningless to an LLM; the
registry stays on the robot); age/gender/hat (no model, and unreliable and sensitive); auto-updating
face data from new captures (lets a stranger slowly become "Kyle"); a remote `enroll` tool (a prompt
could make the robot call anyone "Kyle": enrolment stays the owner's step; a gated tool, off by
default, is possible if wanted); detecting people, objects or motion (only faces exist). `look()`'s
default stays photo-plus-faces, because the photo is what lets Hermes describe a scene: **flipping it
to JSON-only is the owner's call.**
- **Hermes bug report "faces and list_people are not registered" was not a bridge bug.** A raw client and a
fresh `hermes mcp test` list all three; Hermes's own `agent.log` shows it registering all seven tools
at 01:18:46 after a robot restart, and a later session calling `faces()` fine. The reporting session
started at 00:35, before `faces` existed, and keeps a start-time snapshot of its tool table (live calls
like `list_prompts` still work, which matches the report). Fix: start a new Hermes session after the
robot's tool set changes; Hermes should refresh a session's tool table on re-registration. The bridge
keeps stateful sessions on purpose: the `Session not found` after a restart is what makes Hermes
reconnect and re-register.
## Thermal guard (`ros/hermes_thermal`)
Built because the Pi has no fan (about 60 C open-air with everything running) and an enclosure will be
hotter. `thermal_guard` reads the CPU temperature (`/sys/class/thermal`) and `vcgencmd get_throttled`,
publishes `/health/temperature`, `/health/thermal_level` (ok | warm | hot | critical) and
`/health/throttled`, and sheds load through the target nodes' own live parameters: the camera's
`FrameDurationLimits`, the face's `fps` and the detector's `rate_hz` (the face and detector gained live
parameter callbacks for this). Levels enter at 72 / 77 / 82 C (the Pi throttles at about 80); heating is
acted on at once, cooling goes one level at a time, only below the threshold minus 3 C and after a 20 s
dwell. Rates scale by 1.0 / 0.7 / 0.45 / 0.2 with floors (camera 2 fps, face 8, detector 1 Hz). The
"normal" rates are parameters of the guard and a bringup test asserts they equal the real ones in
`robot.yaml`, so restoring is exact and survives the guard restarting. `fake_temp_c` forces a
temperature for testing. `robot://status` now carries a `health` block (temperature, level, throttle
flags now and since boot, which doubles as a power-supply check).
**Verified live on the robot** by forcing 73 / 78 / 85 C and back: the camera, face and detector rates
followed (measured camera 16.5 -> 11.5 -> 7.2 -> 3.5 Hz and back to 16.5), cooling stepped down critical
-> hot -> warm -> ok exactly one dwell apart, and the guard stayed the same process throughout.
**A live bug the unit tests missed:** one logging call that warned when heating and informed when
cooling raised `ValueError: Logger severity cannot be changed between calls` in rclpy, killing the
guard on its first cooling step (the launch file respawned it, which only looked like a recovery). Fixed
with separate call sites, the tick now catches and logs any error instead of dying, and a test drives
the real node through a full heat-then-cool run; a mutation check (the old logging in a throwaway
copy) confirmed that test fails on exactly that error. Not yet tested under a real thermal load.
When the servos arrive, watch the under-voltage flag in `robot://status` during a sweep.
The Pi has no fan. `thermal_guard` reads the CPU temperature and `vcgencmd get_throttled`, publishes
`/health/temperature`, `/health/thermal_level` (ok | warm | hot | critical) and `/health/throttled`, and
sheds load through the target nodes' live parameters: the camera's `FrameDurationLimits`, the face's `fps`,
the detector's `rate_hz`. Levels enter at 72 / 77 / 82 °C (the Pi throttles at ~80); heating is acted on
at once, cooling goes one level at a time only below the threshold minus 3 °C and after a 20 s dwell.
Rates scale by 1.0 / 0.7 / 0.45 / 0.2 with floors (camera 2 fps, face 8, detector 1 Hz). Its "normal"
rates are its own parameters, and a bringup test asserts they equal the real ones in `robot.yaml`, so
restoring is exact and survives a restart. `fake_temp_c` (`ros2 param set /hermes_thermal fake_temp_c
85.0`, -1 to stop) forces a temperature. **Verified live** by forcing 73 / 78 / 85 °C and back: measured
camera rate 16.5 -> 11.5 -> 7.2 -> 3.5 Hz and back, cooling stepped down one dwell apart, same process
throughout. **A live bug the unit tests missed:** one log line that warned when heating and informed when
cooling raised rclpy's `Logger severity cannot be changed between calls` and killed the guard on its
first cooling step (the respawn hid it). Fixed (separate call sites), the tick now survives any error, and
a test drives the real node through heat-then-cool; a mutation check confirmed it fails on the original
bug. Not yet tested under a real thermal load.
## USB link (HANDOFF package 0b)
- Pi: `dtoverlay=dwc2,dr_mode=peripheral`; `robot-usb-gadget.service` builds a configfs gadget (NCM `usb0`
+ ACM `ttyGS0`; host MAC `02:68:65:72:6d:01`, robot MAC `…:02`) before networking;
`/etc/netplan/60-usb0.yaml` gives `usb0` `10.55.0.2/30` with no gateway (the default route stays on
Wi-Fi). NCM works on the Pi 4 (ECM was not needed). **netplan must match `usb0` by name, not MAC:** a
`macaddress:` match becomes `PermanentMACAddress=`, which a gadget interface lacks, so it stays unmanaged.
- Host (Omarchy, NetworkManager): `sudo host/omarchy/robot-usb-link.sh` adds profile `hermes-robot-usb`
(`10.55.0.1/30`, matched by the MAC, never-default). Ping ~2 ms.
- **Wi-Fi-off acceptance passed:** with Wi-Fi blocked (`rfkill block wifi`, remembered across reboots),
live and after a reboot, the Pi came up on the USB link alone and everything worked. With no internet
there is no NTP, so the clock is unsynchronised (roughly right).
- Not yet verified: replug and second-host recovery, and the serial console login (talos's `/dev/ttyACM0`
is `root:uucp`; `hermes` has no password, so the console shows a prompt but cannot log in; no autologin
by design).
## Owner's decisions
ROS never leaves the Pi (`hermes_mcp` is the only bridge); SSH stays open for V2; NixOS host config is
skipped for now; no remote `enroll`; `look()`'s default is open; the face may take up to a minute to
appear at boot; everything should be ROS. A new Omarchy host with a GPU (it runs the local `ornith`
model) is arriving: it needs only `host/omarchy/robot-usb-link.sh` and its SSH key on the Pi.
## Gotchas
- `ros2 node list` with its default short window shows only some nodes: use `--no-daemon --spin-time 6`.
- A one-off message published before ROS discovery matches is lost (volatile QoS): republish, as
`hermes_mcp` does every second. Anything that sets `PYTHONPATH` by hand must prepend, not replace.
- A shell that sourced `robot-env.sh` before a build cannot import the newly built package: re-source.
- rclpy needs each logging call site to keep one severity.
## Arrival day: the neck (servos), planned and deliberately not built ahead of the hardware
The servos are expected Thursday or Friday at the earliest. The real unknowns cannot be faked (which
`pwmchip` the overlay creates, its permissions on Ubuntu, each servo's real pulse range, whether the
servos brown out the Pi), so `hermes_head` waits for the hardware. The order, with what to verify at
each step:
Expected Thursday or Friday at the earliest. What matters cannot be faked (which `pwmchip` the overlay
creates, its permissions on Ubuntu, each servo's real pulse range, whether they brown out the Pi), so
`hermes_head` waits. Order, with what to verify:
1. **Wire it:** pan servo to the HAT's GPIO12 port, tilt to GPIO13; the "3pin Volt" jumper is already
verified at 5 V. Power matters: servo spikes on a host USB port may brown the Pi out, so use the
5.1 V 3 A supply through the splitter if it has arrived. Horns off for the first moves.
verified at 5 V. Servo spikes on a host USB port may brown the Pi out: use the 5.1 V 3 A supply
through the splitter if it has arrived. Horns off for the first moves.
2. **Overlay (needs a reboot, ask first):** add `dtoverlay=pwm-2chan,pin=12,func=4,pin2=13,func2=4` under
`[all]` in `/boot/firmware/config.txt` (`dtparam=audio=off` is already set; the Pi's headphone audio
shares PWM hardware). Add it to `pi/install-*.sh`, not by hand.
3. **sysfs:** find the chip (`/sys/class/pwm/pwmchip*`; it was not assumed to be 0), export channels 0
and 1, `period=20000000`. Add a udev rule that gives a group (not root) write access, as Ubuntu's
defaults differ from Raspberry Pi OS.
4. **Measure each servo:** centre (1.5 ms), then find the safe min and max pulse with the horn off; set
`ROBOT_SERVO_MIN_US` / `MAX_US` per servo (start 500 / 2400), the pan and tilt trims and invert flags.
5. **Write `hermes_head`:** `PwmHead` on sysfs, a ~0.3 s interpolated move, PWM off after ~1 s idle (stops
hold-jitter), limits (pan 80, tilt 45), `/joint_states` for `pan_joint` / `tilt_joint` in radians.
Convention (the MCP contract): positive pan = the robot's left, positive tilt = up. Unit-test the
logic against a fake sysfs tree, then verify on the Pi: `vcgencmd get_throttled` must stay `0x0`
through a full sweep while the face is running.
6. **Wire it in:** the bridge refuses `head: true` today. Give it a head backend that calls
`hermes_head`, set `head: true` in `robot.yaml`, and check `turn_head` / `head_position` appear and
that a new Hermes session sees them. Check the face's eyes follow the real joints the right way
(the sign was fixed in software, never seen with a moving head).
7. **URDF:** `hermes_description` from the enclosure STLs' `pose_*` parts, frames `base_link`,
`pan_link`, `tilt_link`, `camera_link`, `camera_optical_frame`, so `/tf` follows the head.
8. **Mounted camera:** set the flip (`orientation` parameter) for how it is actually mounted, re-check
the focus (a changed focus invalidates `pi/calibration/`), and set `ROBOT_FACE_ROTATION` if the
display ends up mounted differently from the portrait pose.
`[all]` in `config.txt` (`dtparam=audio=off` is already set: the Pi's headphone audio shares PWM
hardware). Add it to `pi/install-*.sh`, not by hand.
3. **sysfs:** find the chip (`/sys/class/pwm/pwmchip*`; do not assume 0), export channels 0 and 1,
`period=20000000`. A udev rule should give a group, not root, write access.
4. **Measure each servo:** centre (1.5 ms), then the safe min and max pulse with the horn off; set
`ROBOT_SERVO_MIN_US` / `MAX_US` per servo (start 500 / 2400), the trims and invert flags.
5. **Write `hermes_head`:** `PwmHead` on sysfs, a ~0.3 s interpolated move, PWM off after ~1 s idle,
limits (pan 80, tilt 45), `/joint_states` for `pan_joint` / `tilt_joint` in radians (positive pan =
the robot's left, positive tilt = up). Unit-test against a fake sysfs tree, then verify on the Pi:
`vcgencmd get_throttled` stays `0x0` through a full sweep with the face running; watch the
under-voltage flag in `robot://status`.
6. **Wire it in:** give the bridge a head backend that calls `hermes_head`, set `head: true`, check
`turn_head` / `head_position` appear to a new Hermes session, and that the face's eyes follow the real
joints the right way.
7. **URDF:** `hermes_description` from the enclosure STLs' `pose_*` parts, frames `base_link`, `pan_link`,
`tilt_link`, `camera_link`, `camera_optical_frame`.
8. **Mounted camera:** set the flip (`orientation`) for how it is mounted, re-check the focus (a changed
focus invalidates `pi/calibration/`), and the display rotation if it is mounted differently.
## Open items
1. Waiting on hardware: servos (above), speakers (playback, echo cancel, Hermes talking aloud, the host's
`robot_mic` naming), the fan, the splitter, the printed enclosure (measure first: `enclosure/README.md`).
2. Unconfirmed: the retuned listening detector live; a second person in frame; replug and second-host
recovery; the serial console; `agent_timeout_s` over a long idle Hermes session; the calibration's
uncovered corners (retake with the board in the top-left and bottom-right); `pi/install.sh` on a
blank card (back up the working SD card first: it holds all of this).
3. Optional: a gated `enroll` tool, if the owner wants one.
## Repository
Branch `master`. `.gitignore` covers `.remember/`, caches, `.venv/`, `*.orig`, `*.rej` and
`pi/50-cloud-init.yaml`. Layout: `ros/` (five packages), `pi/` (everything installed on the Pi),
`host/` (the owner's machines: `host/omarchy/` has the USB link script, Hermes and PipeWire config and
the launcher; `host/check-pi-platform.sh`), `enclosure/`, `docs/face-prototype.html`. `README.md` is
rewritten for this build. **`pi/install.sh` is the one installer** (re-runnable: run twice on the live
robot, the second run changed nothing, exit 0): it installs ROS, builds the libcamera fork (pinned
commit) and `camera_ros`, sets up the venv, models, workspace, USB gadget, audio overlay, firewall,
PipeWire, mixer and services. It has not been run on a blank card (no spare card). HANDOFF.md is the
original brief and has drifted; where they differ, this file wins.
`pi/50-cloud-init.yaml`. `ros/` has `hermes_face`, `hermes_vision`, `hermes_audio`, `hermes_mcp`,
`hermes_thermal` and `hermes_bringup`; `pi/` is everything installed on the Pi; `host/` is the owner's
machines (`host/omarchy/`: the USB link script, Hermes and PipeWire config, the launcher;
`host/check-pi-platform.sh`); `enclosure/`; `docs/` (`face-prototype.html`, `checkerboard-9x6.png`).
Run a package's tests on the Pi with `python3 -m pytest test` in its directory after `source
~/robot-env.sh` (the face suite takes about a minute).
## Security notes (all resolved)
The automated review of the initial commit flagged three items, all from the Pi Zero design:
`ROBOT_MCP_BIND` defaulting to `0.0.0.0` (now the USB address only, with a bearer token), PipeWire on
TCP 4713 on every interface (now protected by the boot-time firewall, which loads first), and the
single-IP any-interface nftables file (rewritten: usb0 from 10.55.0.1 only). Later reviews added:
DDS is blocked on every interface and discovery is local-only; recognised names are not logged; node
parameters are clamped. SSH stays open on all interfaces (key-only) by the owner's choice, for V2.
The automated review of the initial commit flagged `ROBOT_MCP_BIND` defaulting to `0.0.0.0` (now the USB
address only, with a bearer token), PipeWire on TCP 4713 on every interface (now behind the boot-time
firewall, which loads first), and a single-IP any-interface nftables file (rewritten). Later reviews led
to: DDS blocked on every interface and local-only discovery; recognised names never logged; node
parameters clamped. The token and the owner's Wi-Fi password never go through the repo or logs (one
`netplan get` once printed the Wi-Fi password into a session; it was flagged to the owner).