Hermes robot

A small desk robot that gives Hermes Agent (Nous Research's open-source agent) eyes, ears, a voice and a face. It does no thinking of its own: Hermes runs on your computer and uses the robot through MCP.

• Eyes: a camera. Hermes calls look() and gets a current photo.
• Face: an oscilloscope-style display that shows what the agent is really doing: asleep (no agent connected), idle, listening, thinking, looking, speaking. The mouth draws the actual voice, and the eyes glance toward whoever the camera sees.
• Ears and voice: the HAT's two microphones and speakers, shared with your computer as ordinary audio devices.
• Neck: pan and tilt servos (turn_head), once they are fitted.
• One cable: USB-C carries power and a private network link to your computer. Wi-Fi is a fallback.

It runs ROS 2 Jazzy on Ubuntu 24.04, so a mobile version can grow out of it. Hermes needs no ROS: it only speaks MCP.

Status

Works and is tested	Waiting on hardware
Face on the display, always on at power-up, with a sleeping pose	Servos: turn_head, head_position, the pan/tilt URDF
Camera, face detection, opt-in recognition of people you enrol	Speakers: playback, echo cancel, Hermes talking out loud
Eyes follow faces; mouth draws your live voice	Fan (the Pi runs at about 62 °C open-air without one)
MCP bridge: hermes mcp test robot passes, real Hermes turns call look() and faces()	USB-C splitter (the Pi is powered from your computer's port for now)
Thermal guard: slows the camera, face and detector before the Pi throttles
USB network link, firewall, boots and works with Wi-Fi off	Printed enclosure (enclosure/)

How it fits together

Your computer (Omarchy)                          Raspberry Pi 4 + BrainCraft HAT (Ubuntu 24.04)
  Hermes Agent ── MCP over HTTP, USB link ────►  hermes_mcp  (the only way in or out of ROS)
  PipeWire     ◄── audio, TCP 4713 ──────────►  PipeWire (not ROS)
                                                   │ ROS 2 graph, this machine only
                      camera_ros ► hermes_face_detector ► /vision/faces ─┐
                      hermes_audio_state ► levels, speaking/listening ───┤
                      hermes_mcp ► /hermes/state (the agent heartbeat) ──┴► hermes_face ► display

Every node is its own process, started by one launch file (ros/hermes_bringup) and restarted if it dies. ROS never leaves the Pi. The firewall blocks it on every interface, so hermes_mcp is the one door.

Parts

Part	Notes
Raspberry Pi 4	8 GB here. Ubuntu Server 24.04 arm64.
Adafruit BrainCraft HAT	1.54″ ST7789 display, WM8960 audio codec with two mics, speaker and servo ports.
Arducam UC-346	OV5647 camera, 15-pin ribbon, fixed focus.
2 × MG90S servos	Pan (in the base) on the HAT's GPIO12 port, tilt (camera yoke) on GPIO13.
2 × speakers	8 Ω, up to 1 W, JST PH 2-pin.
USB-C power/data splitter	Power from a 5.1 V 3 A supply, data only to your computer. A Pi 4 cannot negotiate USB PD.
Fan	On the HAT's fan header.
Enclosure	enclosure/: printed, with a turntable.

Physical steps

1. The HAT's "3pin Volt" jumper must be on 5 V, which is the factory setting. Check it with a meter.
2. Pan servo into the GPIO12 port, tilt servo into GPIO13.
3. Camera ribbon: leave a gentle loop, because only the tilt flexes it.
4. Slide the HAT's "Audio" switch (right edge) to ON. With it off, everything looks configured and every recording is exactly silent. This cost an hour once.
5. Power the Pi from the splitter, not straight from a port that cannot give 3 A. Never power it from a laptop or tablet you care about the battery of.

Setting up the Pi

1. Flash Ubuntu Server 24.04 for the Pi (arm64, preinstalled image) and check its SHA256.

2. Configure first boot on the card's system-boot partition. In user-data, create the user hermes with key-only login and the hostname hermes-bot. For Wi-Fi, put the access point in network-config (a template is pi/50-cloud-init.yaml; it holds your password, so it is gitignored). cloud-init applies these on the first boot only: to change Wi-Fi later, edit /etc/netplan/50-cloud-init.yaml on the Pi.

3. Boot, then copy this repo to the Pi and run the installer:

   rsync -a ./ hermes@<pi>:~/hermes-bot/
   ssh hermes@<pi> 'cd hermes-bot && sudo ./pi/install.sh'
   

   It installs ROS, builds the camera driver against the Raspberry Pi libcamera fork, builds the workspace, sets up audio, the USB link, the firewall and the services. It can be run again safely. The first run takes a while (the libcamera build alone is about 15 minutes). Reboot afterwards.

4. Plug the USB-C data cable into your computer.

Setting up your computer (Omarchy)

1. Network link: sudo host/omarchy/robot-usb-link.sh gives your end of the link 10.55.0.1/30, matched by the gadget's fixed MAC, so any port and any machine works. The robot is 10.55.0.2.
2. Hermes: the token is in /etc/robot-mcp.env on the Pi. Put it in ~/.hermes/.env as ROBOT_MCP_TOKEN=, and merge host/omarchy/hermes-config-snippet.yaml into ~/.hermes/config.yaml.
3. Audio (optional, needs the speakers): host/omarchy/50-robot-tunnel.conf makes the robot's microphone and speaker appear as devices; host/omarchy/hermes-robot starts Hermes with them as the defaults.

First test

ping 10.55.0.2
hermes mcp test robot          # connects, lists the tools
hermes chat -q "Who is at the robot's desk right now?"          # calls faces(): instant, no photo
hermes chat -q "Use the look tool and tell me what you see."    # sends a photo

What Hermes can use: faces() (who is in view: names, positions, tracking ids; instant, no photo), look() (the same faces summary plus a current photo), list_people(), and, once a head is fitted, turn_head and head_position. robot://status says what hardware this robot actually has. Only faces are detected. Face data never leaves the robot: Hermes gets names and similarity scores, and an unrecognised face is null, never a guess.

The robot's face wakes when Hermes connects and falls asleep about 40 seconds after it leaves. A look-and-describe turn takes a minute or so on local models: look() itself is instant. Hermes does not give MCP images to the main model: it saves the photo to its image cache and the model calls vision_analyze on it.

Day to day

State	Face	Comes from
sleeping	eyes closed, dim, slowly breathing	no agent connected (no heartbeat)
idle	blinks, smiles	the agent is connected
listening	mouth draws your voice, dimmer	the microphone (never wakes a sleeping face)
thinking	eyes trace looping curves	the agent
looking	squint, then a flash	look()
speaking	mouth draws the outgoing audio	the speaker

• Heat: hermes_thermal watches the CPU temperature and, at 72, 77 and 82 °C (the Pi starts throttling at about 80), scales the camera, face and detector rates down, then restores them as it cools (one step at a time, so it does not flap). The temperature and the Pi's throttle and under-voltage flags are in robot://status and on /health/*. To test it without heating the Pi, on the Pi: ros2 param set /hermes_thermal fake_temp_c 85.0 (and -1 to stop).
• Logs and control: journalctl -u robot -f (the face logs every state change), sudo systemctl restart robot.
• Enrol yourself so the eyes prefer you (numbers only are stored, never photos, in ~/.local/share/hermes/faces.json): on the Pi, with the robot running and after source ~/robot-env.sh, python3 -m hermes_vision.people enroll YOURNAME. list shows who is enrolled, forget YOURNAME removes you.
• Update the code: rsync -a --delete --exclude __pycache__ ros/ hermes@<pi>:~/ros_ws/src/, then colcon build --symlink-install in ~/ros_ws, then sudo systemctl restart robot.
• Run a node by hand: source ~/robot-env.sh first: it puts the libcamera fork and the right OpenCV on the path, and keeps ROS local.

Calibrating the camera

Needed once, and again if you change the lens focus or the image size. Set the focus first (the lens turns if it is not glued: focus at the distance you will use), then show docs/checkerboard-9x6.png on a screen (or print it) and measure one square with a ruler. With the robot running, on the Pi, after source ~/robot-env.sh:

python3 -m hermes_vision.calibrate --cols 9 --rows 6 --square-mm <measured>

Hold the camera 25 to 60 cm from the board and vary distance, tilt and position, keeping all of it in view. It collects 20 different views, solves with OpenCV and writes ~/.ros/camera_info/ov5647__base_soc_i2c0mux_i2c_1_ov5647_36_1296x972.yaml. A reprojection error under about 0.5 px is good. sudo systemctl restart robot to load it.

Tuning and troubleshooting

• Recordings are exactly silent: the Audio switch is off (see above).
• Microphone too noisy or too quiet: WirePlumber resets the codec's capture gain to +30 dB. The installer sets PipeWire's source volume to 63% (about +17 dB); change it with pactl set-source-volume @DEFAULT_SOURCE@ 63%. Remember that PipeWire's scale is cubic.
• The face flickers between listening and idle: raise hold_s in ros/hermes_audio/hermes_audio/levels.py.
• Stock libcamera crashes when capturing: that is expected on this kernel. The installer builds the Raspberry Pi fork into /opt/libcamera-rpi and builds camera_ros against it. Anything that uses the camera must run after source ~/robot-env.sh.
• Display shows noise or a band: the 240×240 glass sits 80 rows into the controller's memory (y_offset: 80), and rotation: 270 is the mounted pose. Both are in ros/hermes_bringup/config/robot.yaml.
• GPIO busy or a crash at start from systemd: the display's GPIO backend needs a writable working directory; the unit sets one.
• The USB interface is unmanaged: netplan must match usb0 by name, not by MAC.
• ros2 node list shows only some nodes: use ros2 node list --no-daemon --spin-time 6.
• The Pi's clock is wrong with Wi-Fi off: there is no internet to sync from; it stays roughly right but unsynchronised.
• Hermes says faces (or another new tool) is "not a known tool name": a Hermes session takes a snapshot of its tool table when it starts. If it began before the robot offered the tool (or before a robot restart that added it), it never sees it, even though robot://status, list_prompts and a fresh hermes mcp test robot all show it. Start a new Hermes session. (Hermes's log shows the reconnect and "registered N tool(s)" for the whole process; only that old session keeps its stale table.)
• A one-off message to the face is ignored: ROS drops a message published before the connection matches. Republish it, as hermes_mcp does every second.

Security

• The robot's ports are open only over the USB link, only to 10.55.0.1: audio (4713), MCP (8765) and a reserved webhook port (8766). Everything else of those is dropped on Wi-Fi.
• ROS is blocked on every interface, including USB. It has no authentication and carries the camera and recognised names.
• MCP needs a bearer token, kept in /etc/robot-mcp.env (root and the service user only).
• SSH is open on all interfaces (key-only), by choice, for the mobile version.
• Face recognition is opt-in and stores only numbers. Hermes keeps each look() photo in its image cache on your computer.
• Never power the robot from a device whose battery you care about.

Files

Path	What
ros/	The ROS 2 workspace: hermes_face, hermes_vision, hermes_audio, hermes_mcp, hermes_thermal, hermes_bringup.
pi/	Everything installed on the Pi: install.sh and its parts, services, firewall, overlays.
host/	Your computer's side: the USB link script, Hermes and PipeWire config, a platform check.
enclosure/	OpenSCAD model, STLs and the print guide. Built from estimates: measure before printing.
docs/face-prototype.html	The browser prototype the face was ported from.
HANDOFF.md	The original implementation brief. CONTEXT.md tracks where things really stand.