hermes-bot/CONTEXT.md

Current state

Last updated 2026-10-05. This file says where the work really stands. HANDOFF.md is the original brief and has drifted (see its banner); where they differ, this file wins. README.md is the guide for a person using the robot.

Hardware on hand

Raspberry Pi 4 (8 GB), BrainCraft HAT, Arducam UC-346 camera, and nothing else: no servos, no fan, no speakers, no USB-C splitter, no printed enclosure. The camera hangs loose on its ribbon and is held by hand. The Pi is powered from talos's USB port (a bench setup, fine without servos). It runs at about 60 °C open-air with everything on; an enclosure will be hotter, so fit the fan before enclosing it. Playback and the servos are untested because they cannot be.

What works (verified on the real robot)

• Face on the display, started at power-up, with a sleeping pose that means "no agent connected". The eyes glance toward detected faces; the mouth draws the live voice. Gaze direction and the voice trace were confirmed by the owner on the real display.
• Camera, face detection, opt-in recognition (the owner is enrolled and recognised live).
• MCP bridge: hermes mcp test robot passes; real Hermes turns on the local ornith model call look() and faces().
• USB network link, firewall, and the whole robot boots and works with Wi-Fi off.
• Thermal guard, and a reboot-tested single installer, pi/install.sh (re-run on the live robot, it changed nothing; not yet run on a blank card).
• 122 unit tests on the Pi: face 50, vision 30, audio 10, MCP 16, thermal 12, bringup 4.

The Pi (hermes-bot)

• Ubuntu 24.04.5 server, kernel 6.8.0-1065-raspi, flashed from talos. User hermes (key-only login, passwordless sudo; no password is set). SSH keys for kyle@vade (talos) and kyle@straylight.
• Network: USB link 10.55.0.2 (host 10.55.0.1); Wi-Fi 192.168.88.152 (MAC d8:3a:dd:0d:66:f5); eth0 is not cabled (the owner mentioned a static address on the rack switch; not recorded here). avahi-daemon is installed, but mDNS is off in systemd-resolved on talos, so use the IP. rfkill is installed.
• cloud-init applies network-config and user-data on the first boot only. To change Wi-Fi later, edit /etc/netplan/50-cloud-init.yaml on the Pi, or bump instance_id in meta-data on system-boot. pi/network-config.example is a credentials-free template for the Wi-Fi block (pi/50-cloud-init.yaml is gitignored, for a filled copy).
• Install: rsync this repo to the Pi and sudo ./pi/install.sh. It installs ROS 2 Jazzy, builds the Raspberry Pi libcamera fork (pinned commit) into /opt/libcamera-rpi and camera_ros against it (~/camera_ws), makes ~/venv (display libraries, OpenCV 4.10, the MCP SDK), downloads the face models to ~/models, restores the camera calibration, builds ~/ros_ws, sets up the USB gadget, the audio overlay and mixer, the firewall (before anything listens), PipeWire and the services, and creates the MCP token. Update just the code with rsync -a --delete --exclude __pycache__ ros/ hermes@<ip>:~/ros_ws/src/, colcon build --symlink-install in ~/ros_ws, sudo systemctl restart robot. Any shell that runs a node by hand must source ~/robot-env.sh first.
• Services: robot.service runs ros2 launch hermes_bringup robot.launch.py as hermes: the face, camera, face detector, audio levels, MCP bridge and thermal guard, each its own process respawned by the launch file. The face starts first; the others follow start_delay (6 s). Parameters: ros/hermes_bringup/config/robot.yaml. RuntimeDirectory=robot gives lgpio a writable working directory (it drops a pipe in its cwd; systemd's default / is not writable). Also robot-cpu-performance.service (governor performance; the face costs about half the CPU at full clock), robot-firewall.service, robot-usb-gadget.service. pi/trim-boot.sh disables snapd and ModemManager (unused; ~9 s of boot CPU).
• Boot: the face draws about 28 s after the kernel starts (kernel ~7 s, robot.service ~10 s into userspace, ros2 launch itself ~10 s on a cold Pi, the face ~3 s); the other nodes are up by ~37 s. The owner is fine with up to a minute and prefers everything in ROS: a split face unit that skipped ros2 launch saved only 3.7 s and was dropped. Reboot-tested: everything returns by itself.
• CPU, full stack, 4 cores: face ~84%, camera ~63% (15 fps; 30 fps alone is ~40%, 15 is ~20%), detector ~45%, audio ~15%, MCP ~15%, PipeWire ~12%; about 40% of the machine idle, 60 °C, no throttling. OpenCV's DNN spins every core and burns 3-4x its elapsed time in CPU, so the detector pins it to one thread (YuNet 86 ms, SFace 161 ms of CPU), decodes the JPEG at half size and re-identifies a tracked face at most once a second.
• ROS never leaves the Pi (owner's decision): ROS_AUTOMATIC_DISCOVERY_RANGE=LOCALHOST (ROS's setup script exports SUBNET, so pi/robot-env.sh assigns it) and the firewall drops domain 42's DDS ports (17900-18149) on every interface but loopback. hermes_mcp is the only bridge out. This supersedes HANDOFF's host/ros-shell.
• Firewall (pi/nftables-robot.nft, loaded at boot before networking by robot-firewall.service): accepts 4713 (audio), 8765 (MCP) and 8766 (reserved, nothing listens) only on usb0 from 10.55.0.1 and drops them elsewhere. SSH is deliberately open on all interfaces (key-only), for V2: the owner's call.
• PipeWire 1.0.5 / WirePlumber 0.4.17 run as hermes's user services (loginctl enable-linger). pi/pipewire/50-network-server.conf makes pipewire-pulse listen on TCP 4713 on all interfaces; the firewall is the protection. Defaults are the WM8960 source and sink.

Display and the face (ros/hermes_face)

• Display: the ST7789 works on lgpio through Blinka (rpi-lgpio, not RPi.GPIO). Pass cs=None (the kernel drives CE0; claiming GPIO8 raises "GPIO busy"). The 240x240 glass sits 80 rows into the controller's RAM, y_offset=80, for every rotation (the driver rotates in software and leaves the scan direction fixed). The mounted pose (microSD end up, USB-C edge on the left) needs the content turned 90° clockwise: rotation=270. The face pushes raw RGB565 with _block at ~27-28 fps.
• Renderer (render.py, no ROS imports): a numpy port of docs/face-prototype.html, flat face only (head and low-poly shapes are not ported). Phosphor and vector styles; tone map is a lookup table, glow at half resolution, rotation a free np.rot90; parameters are clamped. States: sleeping idle listening thinking looking speaking; looking returns to the state it interrupted.
• Node (face_node.py): subscribes /hermes/state, /joint_states, /vision/faces, /camera/camera_info, /audio/activity, /audio/scope/{mic,speaker}; starts in sleeping; logs every state change (journalctl -u robot), so behaviour can be checked without watching the panel. fps can be changed live (the thermal guard does).
• State arbitration (arbiter.py): sleeping (agent side only) > speaking > listening > thinking > idle. Room sound never wakes a sleeping face.
• Agent heartbeat: /hermes/state doubles as the heartbeat. After agent_timeout_s (10) without it the face logs agent gone, sleeping and sleeps.
• Gaze: eyes glance toward the best face (a named one first, then the biggest), smoothed, held 2.5 s, then drift back; sleeping ignores it. Image x flips on the display (a person on the image's right is on the robot's right, the viewer's left). The renderer's pan sign was wrong (inherited from the prototype) and is fixed: positive pan = the robot's left = the viewer's right; positive tilt = up, as the MCP turn_head documents. Never seen with a moving head.
• Voice trace: while fresh scope data exists, the mouth draws the real waveform (speaker when speaking, mic when listening), auto-scaled; the owner confirmed it follows how close and loud they are.
• Not done: head/low-poly shapes, DotStar LEDs, a button mute. Open: the IPS panel's black reads dark gray on camera at full backlight; dimming needs a software PWM on GPIO26.

Camera and vision (ros/hermes_vision, camera_ros)

• Camera: only the Raspberry Pi libcamera fork works. The OV5647 is detected (ov5647 10-0036), but Ubuntu's libcamera 0.2.0 dies in the Pi IPA (assertion "it != buffers_.end()" in prepareIsp()) and ROS Jazzy's 0.7.2 with FATAL … A list of V4L2 controls requires a ControlInfoMap, deterministically for every format and size. The fork (commit 6c1dd9d, built with -Dpipelines=rpi/vc4 -Dipas=rpi/vc4 -Dv4l2=false … into /opt/libcamera-rpi) captures; camera_ros 0.7.0 is built from source against it, and the fork must come first in LD_LIBRARY_PATH (robot-env.sh does it). Stream: 1296x972 at 15 fps (FrameDurationLimits in robot.yaml); /camera/image_raw, …/compressed, camera_info. pi/capture_still.py takes a still with the fork's cam. Frames are padded and B,G,R in memory.
• Calibration done (pi/calibration/…_1296x972.yaml, restored by the installer): 18 of 24 views, reprojection error 0.296 px, focal length ~1153 px, centre (653, 463), small distortion with k3 held at zero. The first pass fit 0.41 px but with cancelling k1/k2/k3 because the board never reached the edges; the tool now fixes k3, reports coverage, drops blurred outlier views and can re-solve saved views (--from-views). The top-left and bottom-right corners were never reached, so the model is least certain there. The result is valid for the current focus and image size; a changed focus invalidates it. Tool: python3 -m hermes_vision.calibrate --cols 9 --rows 6 --square-mm <mm> (docs/checkerboard-9x6.png can be shown on a screen; the square size only scales distance, not the intrinsics).
• Detection (detect.py, no ROS imports): OpenCV YuNet 2023mar (Haar fallback). It needs OpenCV 4.8+ (apt has 4.6), so ~/venv carries opencv-python-headless==4.10.0.84 with numpy<2 (the system NumPy stays 1.26.4 for ROS), and the venv is first on PYTHONPATH; input sides must be multiples of 32.
• Tracking (tracking.py): faces are matched across frames by box overlap, so a person keeps one track id; identity is re-resolved at most once a second.
• Node face_detector publishes /vision/faces (vision_msgs/Detection2DArray): boxes in full-size camera pixels, Detection2D.id = the track id, results[0] the face, results[1] the identity if recognised (class_id = name, score = cosine similarity, not a probability). An unknown face has no results[1]: a name is never made up. rate_hz (4) can be changed live.
• Recognition is opt-in (recognize.py, people.py): SFace embeddings on YuNet's landmarks. Nothing is stored until someone is enrolled with python3 -m hermes_vision.people enroll NAME; only 128-number embeddings are kept, in ~/.local/share/hermes/faces.json (mode 0600, dir 0700, written atomically), never images. list, forget, check, enroll-images too. The owner is enrolled (10 samples) and was recognised in 29 of 31 detections live (similarity ~0.9). A throwaway-database test recognised 8 of 9 unenrolled frames of the same person, down to side profiles; an unrelated photo scored 0.16. Not tested: a second person live. Names are never logged (a count is).

Audio (ros/hermes_audio, pi/overlays, PipeWire)

• The stock wm8960-soundcard overlay fails on Ubuntu 24.04 (wm8960: No MCLK configured): it puts the 12.288 MHz fixed clock on the card's codec link, but the codec driver reads mclk from its own node. pi/overlays/hermes-wm8960.dts adds it there; pi/install-audio-overlay.sh compiles it, sets dtparam=audio=off and dtoverlay=hermes-wm8960. Capture then opens at 8-48 kHz. Address the card by name (hw:CARD=wm8960soundcard,DEV=0): its index changes.
• The HAT's hardware Audio ON/OFF switch (right edge) must be ON. With it off, the codec still answers on I²C and every software check looks right, but all recordings are exactly zero.
• Mixer: Left/Right Input Mixer Boost default to off and must be on. pi/wm8960_asound.state holds the settings (capture 47, boost 1). WirePlumber resets the codec's capture gain to +30 dB when it takes the device, so the installer sets PipeWire's source volume to 63% (its scale is cubic: -12 dB, ALSA Capture 46, about +17 dB), which WirePlumber remembers; survives reboots (checked). Quiet-room floor at this gain: about -43 dBFS; singing at arm's length peaks around -18 dBFS.
• Node audio_state taps the default source and sink monitor with parec (16 kHz mono) and publishes /audio/levels, /audio/activity (speaking | listening | quiet) and trigger-locked waveform windows /audio/scope/{mic,speaker} (only while listening or speaking). Audio itself never goes through ROS. Activity: 3 s start-up calibration of an adaptive noise floor, margin 7 dB, 1.2 s hold so pauses between words do not flicker, steady noise is absorbed after 8 s, the mic is ignored while speaking. The flicker retune is unit-tested but not re-confirmed live.
• Verified: the speaking path (a tone played to the sink, no speaker attached, shows on the monitor); the mic path with the owner's voice. Playback to a real speaker, echo cancel (pi/pipewire/ 60-echo-cancel.conf.optional) and the host's robot_mic naming wait for the speakers.

The MCP bridge and Hermes (ros/hermes_mcp)

• Serves MCP (SDK 2.3, streamable HTTP) on http://10.55.0.2:8765/mcp: the USB address only (it retries the bind until the link is up), bearer token in /etc/robot-mcp.env (root:hermes, mode 640, made by the installer, never printed; read it with sudo cat). Tool names are the stable contract with Hermes. Tools: look(send_image="always"|"never") (the faces summary as JSON text plus the next fresh JPEG, never a stale frame; it also makes the face squint), faces() (the faces in view, no photo: track_id, bbox, confidence, seen_for_s, identity {name, similarity} or null), list_people() (names and sample counts only), and turn_head / head_position only when a head is fitted (head: false now; the node refuses head: true until hermes_head exists). Resource robot://status (hardware present or absent, what is detected, enrolled count, tools, and health: temperature, thermal level, throttle flags now and since boot) and prompts who_is_here, look_around make it discoverable. Sessions are stateful on purpose (see the Hermes note below).
• Heartbeat: while an authenticated client is connected (in flight, or heard within connected_grace_s = 30) it republishes idle on /hermes/state once a second; the face wakes on connect and sleeps about 40 s after the client leaves.
• Hermes on talos is wired to it (with the owner's authorisation). Hermes Agent v0.21.5 is in ~/.hermes (hosted main model grok-4.6 via xai-oauth; vision is auxiliary and local, qwen3-vl-8b-abliterated on straylight; the ornith-* models are the local ones on straylight and dixie). config.yaml has an mcp_servers.robot block and .env has ROBOT_MCP_TOKEN; backups: config.yaml.bak-20261005T002659-robot-mcp, .env.bak-20261005T002659-robot-mcp. hermes mcp test robot passes. A fresh HERMES_HOME hangs on first-run setup, so test against the real one.
• Real agent turns, entirely on local models: hermes chat -q "..." --oneshot -Q --provider custom:llama-router -m ornith-1.5-35b-a3b --max-turns 5. look() works; Hermes does not hand MCP images to the main model: it saves the JPEG to ~/.hermes/cache/images/ on talos and returns a short reference, and the model calls vision_analyze (local vision model, ~29 s). So a look-and-describe turn takes ~70 s, and each look leaves a photo in talos's Hermes cache (Hermes's retention). faces() is the cheap path: 0.03 s, no image, and Hermes answered "Kyle is at the desk, similarity 0.78".
• Hermes's feedback (~/.hermes/cache/scratch/hermes-bot-v0-face-identity.md). Built: faces(), look returning that JSON, list_people, robot://status and prompts (previously empty), track ids. Declined, deliberately: embeddings in responses (biometric templates, meaningless to an LLM; the registry stays on the robot); age/gender/hat (no model, and unreliable and sensitive); auto-updating face data from new captures (lets a stranger slowly become "Kyle"); a remote enroll tool (a prompt could make the robot call anyone "Kyle": enrolment stays the owner's step; a gated tool, off by default, is possible if wanted); detecting people, objects or motion (only faces exist). look()'s default stays photo-plus-faces, because the photo is what lets Hermes describe a scene: flipping it to JSON-only is the owner's call.
• Hermes bug report "faces and list_people are not registered" was not a bridge bug. A raw client and a fresh hermes mcp test list all three; Hermes's own agent.log shows it registering all seven tools at 01:18:46 after a robot restart, and a later session calling faces() fine. The reporting session started at 00:35, before faces existed, and keeps a start-time snapshot of its tool table (live calls like list_prompts still work, which matches the report). Fix: start a new Hermes session after the robot's tool set changes; Hermes should refresh a session's tool table on re-registration. The bridge keeps stateful sessions on purpose: the Session not found after a restart is what makes Hermes reconnect and re-register.

Thermal guard (ros/hermes_thermal)

The Pi has no fan. thermal_guard reads the CPU temperature and vcgencmd get_throttled, publishes /health/temperature, /health/thermal_level (ok | warm | hot | critical) and /health/throttled, and sheds load through the target nodes' live parameters: the camera's FrameDurationLimits, the face's fps, the detector's rate_hz. Levels enter at 72 / 77 / 82 °C (the Pi throttles at ~80); heating is acted on at once, cooling goes one level at a time only below the threshold minus 3 °C and after a 20 s dwell. Rates scale by 1.0 / 0.7 / 0.45 / 0.2 with floors (camera 2 fps, face 8, detector 1 Hz). Its "normal" rates are its own parameters, and a bringup test asserts they equal the real ones in robot.yaml, so restoring is exact and survives a restart. fake_temp_c (ros2 param set /hermes_thermal fake_temp_c 85.0, -1 to stop) forces a temperature. Verified live by forcing 73 / 78 / 85 °C and back: measured camera rate 16.5 -> 11.5 -> 7.2 -> 3.5 Hz and back, cooling stepped down one dwell apart, same process throughout. A live bug the unit tests missed: one log line that warned when heating and informed when cooling raised rclpy's Logger severity cannot be changed between calls and killed the guard on its first cooling step (the respawn hid it). Fixed (separate call sites), the tick now survives any error, and a test drives the real node through heat-then-cool; a mutation check confirmed it fails on the original bug. Not yet tested under a real thermal load.

USB link (HANDOFF package 0b)

• Pi: dtoverlay=dwc2,dr_mode=peripheral; robot-usb-gadget.service builds a configfs gadget (NCM usb0
  • ACM ttyGS0; host MAC 02:68:65:72:6d:01, robot MAC …:02) before networking; /etc/netplan/60-usb0.yaml gives usb0 10.55.0.2/30 with no gateway (the default route stays on Wi-Fi). NCM works on the Pi 4 (ECM was not needed). netplan must match usb0 by name, not MAC: a macaddress: match becomes PermanentMACAddress=, which a gadget interface lacks, so it stays unmanaged.
• Host (Omarchy, NetworkManager): sudo host/omarchy/robot-usb-link.sh adds profile hermes-robot-usb (10.55.0.1/30, matched by the MAC, never-default). Ping ~2 ms.
• Wi-Fi-off acceptance passed: with Wi-Fi blocked (rfkill block wifi, remembered across reboots), live and after a reboot, the Pi came up on the USB link alone and everything worked. With no internet there is no NTP, so the clock is unsynchronised (roughly right).
• Not yet verified: replug and second-host recovery, and the serial console login (talos's /dev/ttyACM0 is root:uucp; hermes has no password, so the console shows a prompt but cannot log in; no autologin by design).

Owner's decisions

ROS never leaves the Pi (hermes_mcp is the only bridge); SSH stays open for V2; NixOS host config is skipped for now; no remote enroll; look()'s default is open; the face may take up to a minute to appear at boot; everything should be ROS. A new Omarchy host with a GPU (it runs the local ornith model) is arriving: it needs only host/omarchy/robot-usb-link.sh and its SSH key on the Pi.

Gotchas

• ros2 node list with its default short window shows only some nodes: use --no-daemon --spin-time 6.
• A one-off message published before ROS discovery matches is lost (volatile QoS): republish, as hermes_mcp does every second. Anything that sets PYTHONPATH by hand must prepend, not replace.
• A shell that sourced robot-env.sh before a build cannot import the newly built package: re-source.
• rclpy needs each logging call site to keep one severity.

Arrival day: the neck (servos), planned and deliberately not built ahead of the hardware

Expected Thursday or Friday at the earliest. What matters cannot be faked (which pwmchip the overlay creates, its permissions on Ubuntu, each servo's real pulse range, whether they brown out the Pi), so hermes_head waits. Order, with what to verify:

1. Wire it: pan servo to the HAT's GPIO12 port, tilt to GPIO13; the "3pin Volt" jumper is already verified at 5 V. Servo spikes on a host USB port may brown the Pi out: use the 5.1 V 3 A supply through the splitter if it has arrived. Horns off for the first moves.
2. Overlay (needs a reboot, ask first): add dtoverlay=pwm-2chan,pin=12,func=4,pin2=13,func2=4 under [all] in config.txt (dtparam=audio=off is already set: the Pi's headphone audio shares PWM hardware). Add it to pi/install-*.sh, not by hand.
3. sysfs: find the chip (/sys/class/pwm/pwmchip*; do not assume 0), export channels 0 and 1, period=20000000. A udev rule should give a group, not root, write access.
4. Measure each servo: centre (1.5 ms), then the safe min and max pulse with the horn off; set ROBOT_SERVO_MIN_US / MAX_US per servo (start 500 / 2400), the trims and invert flags.
5. Write hermes_head: PwmHead on sysfs, a ~0.3 s interpolated move, PWM off after ~1 s idle, limits (pan 80, tilt 45), /joint_states for pan_joint / tilt_joint in radians (positive pan = the robot's left, positive tilt = up). Unit-test against a fake sysfs tree, then verify on the Pi: vcgencmd get_throttled stays 0x0 through a full sweep with the face running; watch the under-voltage flag in robot://status.
6. Wire it in: give the bridge a head backend that calls hermes_head, set head: true, check turn_head / head_position appear to a new Hermes session, and that the face's eyes follow the real joints the right way.
7. URDF: hermes_description from the enclosure STLs' pose_* parts, frames base_link, pan_link, tilt_link, camera_link, camera_optical_frame.
8. Mounted camera: set the flip (orientation) for how it is mounted, re-check the focus (a changed focus invalidates pi/calibration/), and the display rotation if it is mounted differently.

Open items

1. Waiting on hardware: servos (above), speakers (playback, echo cancel, Hermes talking aloud, the host's robot_mic naming), the fan, the splitter, the printed enclosure (measure first: enclosure/README.md).
2. Unconfirmed: the retuned listening detector live; a second person in frame; replug and second-host recovery; the serial console; agent_timeout_s over a long idle Hermes session; the calibration's uncovered corners (retake with the board in the top-left and bottom-right); pi/install.sh on a blank card (back up the working SD card first: it holds all of this).
3. Optional: a gated enroll tool, if the owner wants one.

Repository

Branch master. .gitignore covers .remember/, caches, .venv/, *.orig, *.rej and pi/50-cloud-init.yaml. ros/ has hermes_face, hermes_vision, hermes_audio, hermes_mcp, hermes_thermal and hermes_bringup; pi/ is everything installed on the Pi; host/ is the owner's machines (host/omarchy/: the USB link script, Hermes and PipeWire config, the launcher; host/check-pi-platform.sh); enclosure/; docs/ (face-prototype.html, checkerboard-9x6.png). Run a package's tests on the Pi with python3 -m pytest test in its directory after source ~/robot-env.sh (the face suite takes about a minute).

Security notes (all resolved)

The automated review of the initial commit flagged ROBOT_MCP_BIND defaulting to 0.0.0.0 (now the USB address only, with a bearer token), PipeWire on TCP 4713 on every interface (now behind the boot-time firewall, which loads first), and a single-IP any-interface nftables file (rewritten). Later reviews led to: DDS blocked on every interface and local-only discovery; recognised names never logged; node parameters clamped. The token and the owner's Wi-Fi password never go through the repo or logs (one netplan get once printed the Wi-Fi password into a session; it was flagged to the owner).