The servos are due Thursday or Friday at the earliest. What matters about them cannot
be faked: the pwmchip number and permissions on Ubuntu, each servo's real pulse range,
and whether they brown out the Pi. Write the order of work with what to verify at each
step, rather than a large untested node that would look finished and not be.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
A raw MCP client and a fresh `hermes mcp test robot` both list faces, look and
list_people, and Hermes's own log shows it registering all of them at 01:18:46 after
the robot restart and a later session calling faces() fine. The reporting session
started at 00:35, before faces existed, and keeps its start-time snapshot of the tool
table. Record the cause and the fix (start a new Hermes session) in the README
troubleshooting and CONTEXT.md, and note why the bridge keeps stateful sessions.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
Hermes asked for structured detections instead of a bare JPEG. faces() returns the
faces in view with no photo: a persistent track id, bbox, detection confidence, how
long the person has been in view, and identity {name, similarity} or null, never a
made-up name. look() returns that same JSON as text plus the photo (send_image can
skip it); list_people() gives names and counts; robot://status and two prompts make
the server discoverable, where list_resources and list_prompts were empty. The
detector now tracks faces across frames (FaceTracker) and publishes the track id and
the identity as a second hypothesis.
Deliberately not built: embeddings in responses, age/gender attributes, auto-updating
face data, and a remote enroll tool (a prompt could make the robot call anyone "Kyle").
Checked end to end: a real Hermes turn on the local ornith model called faces() (0.03 s,
no image) and answered that Kyle was there at 0.78 similarity. Also: the camera is
calibrated (18 of 24 views, 0.296 px, outlier rejection added) and the calibration is
kept in pi/calibration and restored by the installer.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
hermes_vision.calibrate collects 20 varied checkerboard views from the camera topic
(rejecting repeats of a pose, which constrain nothing), solves with OpenCV and writes
the camera_info YAML camera_ros loads, named with the resolution as it expects.
Tested: the solver recovers a known synthetic camera (focal length within 1%), the
YAML round-trips, and real board detection works on a rendered board. Not yet run on
the real camera: it needs someone holding a board in front of it, with the focus set.
docs/checkerboard-9x6.png is a 10x7-square board that can be shown on a screen.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
Wi-Fi-off acceptance passed: with Wi-Fi blocked, live and after a reboot, the Pi
came up on the USB link alone, every node ran, and hermes mcp test robot and look()
worked. (A temporary guard that would have unblocked Wi-Fi was the safety net and is
removed; Wi-Fi is back on.)
Delete what the ROS design superseded: the Picamera2 MCP server and its unit, the old
installer and requirements, and the duplicate platform check. Move omarchy/ to
host/omarchy/. Rewrite README.md for this build: parts, physical steps (including the
Audio switch), setup, daily use, the troubleshooting learned the hard way, and the
security model.
pi/install.sh is now the single installer for a fresh Ubuntu 24.04 image, built from the
steps done by hand: ROS, the libcamera fork at a pinned commit, camera_ros, the venv and
models, the workspace, USB gadget, audio overlay, firewall (before PipeWire opens a
port), PipeWire, mixer and services. Run twice on the live robot, the second run changed
nothing and everything stayed healthy. It has not been run on a blank card.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
hermes chat on talos, pinned to the local ornith-1.5-35b-a3b on straylight, called
the robot's look() and described the live frame correctly, with nothing sent to a
hosted provider. Hermes does not pass MCP images to the main model: it saves the
JPEG to ~/.hermes/cache/images and returns a short reference, and the model then
calls vision_analyze, which ran on the local qwen3-vl. That is the HANDOFF package 6
fallback, handled by Hermes itself, so look() needs no change.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
With the owner's authorisation, ~/.hermes/config.yaml and .env on talos gained
the robot's MCP server and token (backed up first). `hermes mcp test robot`
connects in 789 ms and discovers look(). A real agent turn, which would send a
camera image to the hosted model, is left for the owner to approve.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
The MCP server is now a ROS node and the only way in or out, per the owner's
decision that ROS never leaves the robot. It serves MCP on the USB address only
with a bearer token, keeps the stable tool name look() (which waits for the next
camera frame, so it is never stale, and passes the camera's JPEG through), and
registers turn_head/head_position only when a head is fitted, which today it is
not. While an authenticated client is connected it republishes the agent
heartbeat on /hermes/state, so the face wakes on connect and sleeps about 40 s
after the client leaves; look() also makes the face squint.
The token is generated on the Pi into /etc/robot-mcp.env (root:hermes, 640) and
never printed. Verified end to end from talos with a real MCP client over the USB
link: tools listed, a 157 KB JPEG back in 0.08 s, a wrong token refused, the Wi-Fi
path dropped by the firewall. Host config snippets now point at 10.55.0.2.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
hermes_bringup starts the face, camera, face detector and audio levels from one
launch file with their parameters in robot.yaml, each node its own respawned
process, the face first and the others after start_delay. robot.service runs it
at boot, so everything is ROS and the face is up whenever the robot is powered.
Reboot-tested: all services and nodes return by themselves, the face draws at
about 28 s on the boot clock, the mic gain survives, nothing throttles.
Cut the CPU: the camera runs at 15 fps (20% of a core, vs 40% at 30), and the
detector went from 131% to 40% by pinning OpenCV to one thread (its DNN burns 3-4x
its elapsed time across cores), decoding the JPEG at half size, and re-identifying
a tracked face at most once a second. The audio scope is published only while
someone is speaking or listening.
A standalone face unit that skipped `ros2 launch` saved only 3.7 s and was dropped:
the owner prefers everything in ROS and is fine with up to a minute. trim-boot.sh
disables the unused snapd and ModemManager.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
The face now treats /hermes/state as the agent's heartbeat: when nothing has been
heard for agent_timeout_s (10 s) it logs "agent gone" and sleeps, so "sleeping"
really means "no agent connected". It also logs every state change, which is how
a missed wake message was found: a one-off publish before discovery matches is
lost under volatile QoS.
The live log showed the face flickering between listening and idle about twice a
second, because the detector let go after 0.4 s. Hold is now 1.2 s so pauses
between words do not drop it, and the margin 7 dB over the adaptive floor (quiet
room noise peaks sit about 3 dB over it), with tests for both.
Confirmed by the owner on the real display: gaze directions, and a mouth that
follows their live voice proportionally.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
audio_state taps the microphone and the speaker monitor with parec and publishes
levels, a speaking/listening/quiet activity, and trigger-locked waveform windows;
audio itself stays out of ROS. The activity detector calibrates its noise floor
at start-up and absorbs steady noise, so a room at -43 dBFS does not read as
"listening" forever, and it ignores the mic while the robot speaks.
The face arbitrates the state it shows: sleeping (agent side only, so room sound
never wakes a face that means "no agent connected") > speaking > listening >
thinking > idle, and draws the mouth from the real samples while fresh scope data
exists, auto-scaled so a quiet voice is still visible.
Checked: a tone played to the sink shows as `speaking` with the mic ignored. The
microphone-to-mouth path with a live voice is not yet confirmed.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
ROS never leaves the robot, by the owner's decision: the firewall drops domain
42's DDS ports on every interface but loopback (usb0 included), discovery is
pinned to LOCALHOST, and hermes_mcp is the only bridge to the outside. PipeWire
1.0.5 runs as hermes's user services with its network server on TCP 4713; the
firewall, loaded at boot before it, is the protection. Verified from talos: 4713
connects over usb0 and times out over Wi-Fi.
CONTEXT.md brought up to date: the face service, gaze, recognition, the firewall,
and what is left.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
Capture works: singing at arm's length reads -34 to -49 dBFS RMS with peaks at
-18 dBFS and no clipping. The earlier all-zero recordings were the HAT's hardware
Audio ON/OFF switch being off; the codec answers on I2C and looks fully
configured either way. Save the stored mixer state (capture +18 dB, input mixer
boost on) in pi/wm8960_asound.state.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
hermes_vision wraps OpenCV YuNet (Haar fallback) as a plain module plus a node
that publishes /vision/faces as vision_msgs/Detection2DArray. It replaces the
standalone detect_faces.py. pi/robot-env.sh puts the Raspberry Pi libcamera fork
ahead of ROS's own copy, the venv's OpenCV 4.10 ahead of the apt one, and fixes
the ROS domain and subnet-only discovery. camera_ros 0.7.0 now runs against the
fork and publishes raw and compressed images.
The stock wm8960-soundcard overlay fails on Ubuntu 24.04 with "No MCLK
configured": it hangs the clock on the sound card's codec link, but the codec
driver reads mclk from its own node. hermes-wm8960.dts adds it there, and
capture opens at 8-48 kHz. The microphones still read digital zeros; see
CONTEXT.md for what has been ruled out.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
Stock libcamera 0.2.0 (Ubuntu) and ROS Jazzy's 0.7.2 both detect the OV5647 but
crash in the Pi IPA when capturing, for every stream format and size. The
Raspberry Pi fork, built into /opt/libcamera-rpi, captures 1296x972 frames.
capture_still.py takes a still with it, and detect_faces.py runs OpenCV YuNet
(Haar fallback) on images. YuNet needs OpenCV 4.8+ and input sides in multiples
of 32, so the venv carries opencv-python-headless 4.10 pinned with numpy<2.
camera_ros still has to be built against the fork.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
Audio (4713), MCP (8765) and the face webhook (8766) are accepted only on usb0
from 10.55.0.1 and dropped on every other interface, Wi-Fi included. SSH is left
open on all interfaces for V2, by the owner's decision. Tested on the Pi with
throwaway listeners: 8765 answers over usb0 and is unreachable over Wi-Fi, while
a control port answers on both.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
The Pi presents NCM Ethernet and ACM serial from a configfs gadget, started by
robot-usb-gadget.service before networking. usb0 gets 10.55.0.2/30 with no
gateway through netplan, matched by interface name: a MAC match turns into
PermanentMACAddress= in networkd, which a gadget interface does not have, so it
stayed unmanaged. talos reaches the Pi at 10.55.0.2 (ping ~2.4 ms, SSH works)
through a NetworkManager profile, host/omarchy/robot-usb-link.sh, matched by the
gadget's fixed host MAC.
Also clamp the face's tunable parameters (focus sets a blur radius, persistence
divides) and fps, after a review flagged unvalidated node parameters.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
The Pi (hermes-bot) is flashed, upgraded and running ROS 2 Jazzy. The ST7789
works through Blinka on lgpio: pass cs=None (the kernel owns CE0), and the glass
sits at y_offset=80 in the controller RAM for every rotation, since the driver
rotates in software. The mounted pose needs rotation=270.
hermes_face is a numpy port of the flat face from docs/face-prototype.html, with
phosphor and vector styles, plus a new `sleeping` pose for "no agent connected".
The renderer has no ROS or hardware imports. The tone map is a lookup table and
the glow runs at half resolution, so the face reaches ~28 fps on the panel at
24% of a core (20 fps) to 40% (30 fps) with the CPU at full clock.
CONTEXT.md now records the Pi, the cloud-init first-boot-only gotcha, and what
is left. pi/50-cloud-init.yaml is a gitignored Wi-Fi template.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
kyleandClaude Sonnet 5.5