Hermes asked for structured detections instead of a bare JPEG. faces() returns the
faces in view with no photo: a persistent track id, bbox, detection confidence, how
long the person has been in view, and identity {name, similarity} or null, never a
made-up name. look() returns that same JSON as text plus the photo (send_image can
skip it); list_people() gives names and counts; robot://status and two prompts make
the server discoverable, where list_resources and list_prompts were empty. The
detector now tracks faces across frames (FaceTracker) and publishes the track id and
the identity as a second hypothesis.
Deliberately not built: embeddings in responses, age/gender attributes, auto-updating
face data, and a remote enroll tool (a prompt could make the robot call anyone "Kyle").
Checked end to end: a real Hermes turn on the local ornith model called faces() (0.03 s,
no image) and answered that Kyle was there at 0.78 similarity. Also: the camera is
calibrated (18 of 24 views, 0.296 px, outlier rejection added) and the calibration is
kept in pi/calibration and restored by the installer.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
hermes_vision.calibrate collects 20 varied checkerboard views from the camera topic
(rejecting repeats of a pose, which constrain nothing), solves with OpenCV and writes
the camera_info YAML camera_ros loads, named with the resolution as it expects.
Tested: the solver recovers a known synthetic camera (focal length within 1%), the
YAML round-trips, and real board detection works on a rendered board. Not yet run on
the real camera: it needs someone holding a board in front of it, with the focus set.
docs/checkerboard-9x6.png is a 10x7-square board that can be shown on a screen.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
The MCP server is now a ROS node and the only way in or out, per the owner's
decision that ROS never leaves the robot. It serves MCP on the USB address only
with a bearer token, keeps the stable tool name look() (which waits for the next
camera frame, so it is never stale, and passes the camera's JPEG through), and
registers turn_head/head_position only when a head is fitted, which today it is
not. While an authenticated client is connected it republishes the agent
heartbeat on /hermes/state, so the face wakes on connect and sleeps about 40 s
after the client leaves; look() also makes the face squint.
The token is generated on the Pi into /etc/robot-mcp.env (root:hermes, 640) and
never printed. Verified end to end from talos with a real MCP client over the USB
link: tools listed, a 157 KB JPEG back in 0.08 s, a wrong token refused, the Wi-Fi
path dropped by the firewall. Host config snippets now point at 10.55.0.2.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
hermes_bringup starts the face, camera, face detector and audio levels from one
launch file with their parameters in robot.yaml, each node its own respawned
process, the face first and the others after start_delay. robot.service runs it
at boot, so everything is ROS and the face is up whenever the robot is powered.
Reboot-tested: all services and nodes return by themselves, the face draws at
about 28 s on the boot clock, the mic gain survives, nothing throttles.
Cut the CPU: the camera runs at 15 fps (20% of a core, vs 40% at 30), and the
detector went from 131% to 40% by pinning OpenCV to one thread (its DNN burns 3-4x
its elapsed time across cores), decoding the JPEG at half size, and re-identifying
a tracked face at most once a second. The audio scope is published only while
someone is speaking or listening.
A standalone face unit that skipped `ros2 launch` saved only 3.7 s and was dropped:
the owner prefers everything in ROS and is fine with up to a minute. trim-boot.sh
disables the unused snapd and ModemManager.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
The face now treats /hermes/state as the agent's heartbeat: when nothing has been
heard for agent_timeout_s (10 s) it logs "agent gone" and sleeps, so "sleeping"
really means "no agent connected". It also logs every state change, which is how
a missed wake message was found: a one-off publish before discovery matches is
lost under volatile QoS.
The live log showed the face flickering between listening and idle about twice a
second, because the detector let go after 0.4 s. Hold is now 1.2 s so pauses
between words do not drop it, and the margin 7 dB over the adaptive floor (quiet
room noise peaks sit about 3 dB over it), with tests for both.
Confirmed by the owner on the real display: gaze directions, and a mouth that
follows their live voice proportionally.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
recognition_model was declared twice (once for the existence check, once for the
recognizer), so the node raised ParameterAlreadyDeclared at start-up and never
published. Declare each parameter once, and add a test that constructs the node,
since nothing else exercised it.
Recognition then works live: with the owner enrolled, 29 of 31 detections over
14 s were named, at detector scores around 0.9.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
audio_state taps the microphone and the speaker monitor with parec and publishes
levels, a speaking/listening/quiet activity, and trigger-locked waveform windows;
audio itself stays out of ROS. The activity detector calibrates its noise floor
at start-up and absorbs steady noise, so a room at -43 dBFS does not read as
"listening" forever, and it ignores the mic while the robot speaks.
The face arbitrates the state it shows: sleeping (agent side only, so room sound
never wakes a face that means "no agent connected") > speaking > listening >
thinking > idle, and draws the mouth from the real samples while fresh scope data
exists, auto-scaled so a quiet voice is still visible.
Checked: a tone played to the sink shows as `speaking` with the mic ignored. The
microphone-to-mouth path with a live voice is not yet confirmed.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
DDS has no authentication, and /camera/* and /vision/faces carry video and
recognised names. Discovery now defaults to this machine only
(ROBOT_ROS_DISCOVERY=SUBNET to opt in; a ${VAR:-default} on ROS's own variable
never applied, since its setup script already exports SUBNET), and the firewall
drops domain 42's DDS ports on every interface but usb0 and loopback. The
detector's startup log prints a count of enrolled people, not their names.
robot-firewall.service loads the rules before networking at every boot: they had
only been loaded at runtime and were gone after each reboot, which has to be fixed
before the PipeWire network server opens port 4713.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
SFace embeddings on top of YuNet's landmarks. Nothing is stored until someone is
enrolled with `python -m hermes_vision.people enroll NAME`; only 128-number
embeddings are kept, never images, in a 0600 JSON file under a 0700 directory,
written atomically. `list` and `forget` manage it. The detector node puts a
recognised name in Detection2D.id (empty if unknown) and picks up database
changes without a restart.
Checked with a throwaway database on the burst frames: eight of nine unenrolled
frames were recognised, down to side profiles (similarity 0.52-0.66), the miss
being a head tipped back at 0.34 against the 0.363 threshold, and an unrelated
photo scored 0.16 and stayed unknown.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
hermes_face now subscribes to /vision/faces and glances toward the best face (a
recognised one first, then the biggest), smoothed, holding 2.5 s and drifting
back to centre when none is seen. Sleeping ignores gaze. Image x flips on the
display: the camera looks the way the robot faces, so a person on the image's
right is on the robot's right, the viewer's left.
Fix the pan sign carried over from the prototype: positive pan is the robot's
left (the MCP contract), which is the viewer's right on the display.
robot-face.service starts the face at boot and restarts it on failure, with a
private working directory because lgpio drops a notification pipe in its cwd and
systemd's default / is not writable. robot-cpu-performance.service holds the CPU
at full clock, which halves the face's CPU cost.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
hermes_vision wraps OpenCV YuNet (Haar fallback) as a plain module plus a node
that publishes /vision/faces as vision_msgs/Detection2DArray. It replaces the
standalone detect_faces.py. pi/robot-env.sh puts the Raspberry Pi libcamera fork
ahead of ROS's own copy, the venv's OpenCV 4.10 ahead of the apt one, and fixes
the ROS domain and subnet-only discovery. camera_ros 0.7.0 now runs against the
fork and publishes raw and compressed images.
The stock wm8960-soundcard overlay fails on Ubuntu 24.04 with "No MCLK
configured": it hangs the clock on the sound card's codec link, but the codec
driver reads mclk from its own node. hermes-wm8960.dts adds it there, and
capture opens at 8-48 kHz. The microphones still read digital zeros; see
CONTEXT.md for what has been ruled out.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
The Pi presents NCM Ethernet and ACM serial from a configfs gadget, started by
robot-usb-gadget.service before networking. usb0 gets 10.55.0.2/30 with no
gateway through netplan, matched by interface name: a MAC match turns into
PermanentMACAddress= in networkd, which a gadget interface does not have, so it
stayed unmanaged. talos reaches the Pi at 10.55.0.2 (ping ~2.4 ms, SSH works)
through a NetworkManager profile, host/omarchy/robot-usb-link.sh, matched by the
gadget's fixed host MAC.
Also clamp the face's tunable parameters (focus sets a blur radius, persistence
divides) and fps, after a review flagged unvalidated node parameters.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
The Pi (hermes-bot) is flashed, upgraded and running ROS 2 Jazzy. The ST7789
works through Blinka on lgpio: pass cs=None (the kernel owns CE0), and the glass
sits at y_offset=80 in the controller RAM for every rotation, since the driver
rotates in software. The mounted pose needs rotation=270.
hermes_face is a numpy port of the flat face from docs/face-prototype.html, with
phosphor and vector styles, plus a new `sleeping` pose for "no agent connected".
The renderer has no ROS or hardware imports. The tone map is a lookup table and
the glow runs at half resolution, so the face reaches ~28 fps on the panel at
24% of a core (20 fps) to 40% (30 fps) with the CPU at full clock.
CONTEXT.md now records the Pi, the cloud-init first-boot-only gotcha, and what
is left. pi/50-cloud-init.yaml is a gitignored Wi-Fi template.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
kyleandClaude Sonnet 5.5