223 lines
14 KiB
Markdown
223 lines
14 KiB
Markdown
# Hermes robot
|
||
|
||
|
||
|
||
A small desk robot that gives Hermes Agent (Nous Research's open-source agent) eyes, ears,
|
||
a voice and a face. It does no thinking of its own: Hermes runs on your computer and uses the robot
|
||
through MCP.
|
||
|
||
- **Eyes:** a camera. Hermes calls `look()` and gets a current photo.
|
||
- **Face:** an oscilloscope-style display that shows what the agent is really doing: asleep (no agent
|
||
connected), idle, listening, thinking, looking, speaking. The mouth draws the actual voice, and the
|
||
eyes glance toward whoever the camera sees.
|
||
- **Ears and voice:** the HAT's two microphones and speakers, shared with your computer as ordinary
|
||
audio devices (the speakers are not fitted yet; the microphones work).
|
||
- **Neck:** pan and tilt servos (`turn_head`), once they are fitted.
|
||
- **One cable:** USB-C carries power and a private network link to your computer. Wi-Fi is a fallback.
|
||
|
||
It runs ROS 2 Jazzy on Ubuntu 24.04, so a mobile version can grow out of it. Hermes needs no ROS: it
|
||
only speaks MCP.
|
||
|
||
## Status
|
||
|
||
| Works and is tested | Waiting on hardware |
|
||
|---|---|
|
||
| Face on the display, always on at power-up, with a sleeping pose | Servos: `turn_head`, `head_position`, the pan/tilt URDF |
|
||
| Camera (calibrated), face detection, opt-in recognition of people you enrol | Speakers: playback, echo cancel, Hermes talking out loud |
|
||
| Eyes follow faces; mouth draws your live voice | Fan (the Pi runs at about 60 °C open-air without one) |
|
||
| MCP bridge: `hermes mcp test robot` passes, real Hermes turns call `look()` and `faces()` | USB-C splitter (the Pi is powered from your computer's port for now) |
|
||
| Thermal guard: slows the camera, face and detector before the Pi throttles | |
|
||
| USB network link, firewall, boots and works with Wi-Fi off | Printed enclosure (`enclosure/`) |
|
||
|
||
## How it fits together
|
||
|
||
```
|
||
Your computer (Omarchy) Raspberry Pi 4 + BrainCraft HAT (Ubuntu 24.04)
|
||
Hermes Agent ── MCP over HTTP, USB link ────► hermes_mcp (the only way in or out of ROS)
|
||
PipeWire ◄── audio, TCP 4713 ──────────► PipeWire (not ROS)
|
||
│ ROS 2 graph, this machine only
|
||
camera_ros ► hermes_face_detector ► /vision/faces ─┐
|
||
hermes_audio_state ► levels, speaking/listening ───┤
|
||
hermes_mcp ► /hermes/state (the agent heartbeat) ──┴► hermes_face ► display
|
||
```
|
||
|
||
Every node is its own process, started by one launch file (`ros/hermes_bringup`) and restarted if it
|
||
dies. ROS never leaves the Pi. The firewall blocks it on every interface, so `hermes_mcp` is the one
|
||
door.
|
||
|
||
## Parts
|
||
|
||
| Part | Notes |
|
||
|---|---|
|
||
| Raspberry Pi 4 | 8 GB here. Ubuntu Server 24.04 arm64. |
|
||
| Adafruit BrainCraft HAT | 1.54″ ST7789 display, WM8960 audio codec with two mics, speaker and servo ports. |
|
||
| Arducam UC-346 | OV5647 camera, 15-pin ribbon, fixed focus. |
|
||
| 2 × MG90S servos | Pan (in the base) on the HAT's GPIO12 port, tilt (camera yoke) on GPIO13. |
|
||
| 2 × speakers | 8 Ω, up to 1 W, JST PH 2-pin. |
|
||
| USB-C power/data splitter | Power from a 5.1 V 3 A supply, data only to your computer. A Pi 4 cannot negotiate USB PD. |
|
||
| Fan | On the HAT's fan header. |
|
||
| Enclosure | `enclosure/`: printed, with a turntable. |
|
||
|
||
## Physical steps
|
||
|
||
1. The HAT's "3pin Volt" jumper must be on 5 V, which is the factory setting. Check it with a meter.
|
||
2. Pan servo into the GPIO12 port, tilt servo into GPIO13.
|
||
3. Camera ribbon: leave a gentle loop, because only the tilt flexes it.
|
||
4. **Slide the HAT's "Audio" switch (right edge) to ON.** With it off, everything looks configured and
|
||
every recording is exactly silent. This cost an hour once.
|
||
5. Power the Pi from the splitter, not straight from a port that cannot give 3 A. Never power it from a
|
||
laptop or tablet you care about the battery of.
|
||
|
||
## Setting up the Pi
|
||
|
||
1. **Flash** Ubuntu Server 24.04 for the Pi (arm64, preinstalled image) and check its SHA256.
|
||
2. **Configure first boot** on the card's `system-boot` partition. In `user-data`, create the user
|
||
`hermes` with key-only login and the hostname `hermes-bot`. For Wi-Fi, put the access point in
|
||
`network-config` (copy `pi/network-config.example` and fill it in; keep a filled copy out of the repo: `pi/50-cloud-init.yaml`
|
||
is gitignored for that). cloud-init applies these on the *first* boot only: to change Wi-Fi later, edit
|
||
`/etc/netplan/50-cloud-init.yaml` on the Pi.
|
||
3. **Boot, then copy this repo to the Pi and run the installer:**
|
||
|
||
```
|
||
rsync -a ./ hermes@<pi>:~/hermes-bot/
|
||
ssh hermes@<pi> 'cd hermes-bot && sudo ./pi/install.sh'
|
||
```
|
||
|
||
It installs ROS, builds the camera driver against the Raspberry Pi libcamera fork, builds the
|
||
workspace, sets up audio, the USB link, the firewall and the services. It can be run again safely.
|
||
The first run takes a while (the libcamera build alone is about 15 minutes). **Reboot afterwards.**
|
||
4. Plug the USB-C data cable into your computer.
|
||
|
||
## Setting up your computer (Omarchy)
|
||
|
||
1. **Network link:** `sudo host/omarchy/robot-usb-link.sh` gives your end of the link `10.55.0.1/30`,
|
||
matched by the gadget's fixed MAC, so any port and any machine works. The robot is `10.55.0.2`.
|
||
2. **Hermes:** the token is in `/etc/robot-mcp.env` on the Pi (`ssh hermes@10.55.0.2 sudo cat
|
||
/etc/robot-mcp.env`). Put it in `~/.hermes/.env` as `ROBOT_MCP_TOKEN=`, and merge
|
||
`host/omarchy/hermes-config-snippet.yaml` into `~/.hermes/config.yaml`. Start a new Hermes session
|
||
afterwards: a session keeps the tool list it started with.
|
||
3. **Audio (optional, needs the speakers):** `host/omarchy/50-robot-tunnel.conf` makes the robot's
|
||
microphone and speaker appear as devices; `host/omarchy/hermes-robot` starts Hermes with them as
|
||
the defaults.
|
||
|
||
## First test
|
||
|
||
```
|
||
ping 10.55.0.2
|
||
hermes mcp test robot # connects, lists the tools
|
||
hermes chat -q "Who is at the robot's desk right now?" # calls faces(): instant, no photo
|
||
hermes chat -q "Use the look tool and tell me what you see." # sends a photo
|
||
```
|
||
|
||
What Hermes can use: `faces()` (who is in view: names, positions, tracking ids; instant, no photo),
|
||
`look()` (the same faces summary plus a current photo), `list_people()`, and, once a head is fitted,
|
||
`turn_head` and `head_position`. `robot://status` says what hardware this robot actually has. Only faces
|
||
are detected. Face data never leaves the robot: Hermes gets names and similarity scores, and an
|
||
unrecognised face is `null`, never a guess.
|
||
|
||
The robot's face wakes when Hermes connects and falls asleep about 40 seconds after it leaves. A
|
||
look-and-describe turn takes a minute or so on local models: `look()` itself is instant. Hermes does not
|
||
give MCP images to the main model: it saves the photo to its image cache and the model calls
|
||
`vision_analyze` on it.
|
||
|
||
## Day to day
|
||
|
||
| State | Face | Comes from |
|
||
|---|---|---|
|
||
| sleeping | eyes closed, dim, slowly breathing | no agent connected (no heartbeat) |
|
||
| idle | blinks, smiles | the agent is connected |
|
||
| listening | mouth draws your voice, dimmer | the microphone (never wakes a sleeping face) |
|
||
| thinking | eyes trace looping curves | the agent |
|
||
| looking | squint, then a flash | `look()` |
|
||
| speaking | mouth draws the outgoing audio | the speaker |
|
||
|
||
- **Heat:** `hermes_thermal` watches the CPU temperature and, at 72, 77 and 82 °C (the Pi starts
|
||
throttling at about 80), scales the camera, face and detector rates down, then restores them as it cools
|
||
(one step at a time, so it does not flap). The temperature and the Pi's throttle and under-voltage
|
||
flags are in `robot://status` and on `/health/*`. To test it without heating the Pi, on the Pi:
|
||
`ros2 param set /hermes_thermal fake_temp_c 85.0` (and `-1` to stop).
|
||
- **Logs and control:** `journalctl -u robot -f` (the face logs every state change), `sudo systemctl
|
||
restart robot`.
|
||
- **Enrol yourself** so the eyes prefer you (numbers only are stored, never photos, in
|
||
`~/.local/share/hermes/faces.json`): on the Pi, with the robot running and after `source
|
||
~/robot-env.sh`, `python3 -m hermes_vision.people enroll YOURNAME`. `list` shows who is enrolled;
|
||
`forget YOURNAME` removes you. Enrolling is deliberately not an MCP tool: a prompt must not be able to
|
||
teach the robot a face.
|
||
- **Update the code:** `rsync -a --delete --exclude __pycache__ ros/ hermes@<pi>:~/ros_ws/src/`, then
|
||
`colcon build --symlink-install` in `~/ros_ws`, then `sudo systemctl restart robot`.
|
||
- **Run a node by hand:** `source ~/robot-env.sh` first: it puts the libcamera fork and the right
|
||
OpenCV on the path, and keeps ROS local.
|
||
- **Run the tests:** on the Pi, after `source ~/robot-env.sh`, `python3 -m pytest test` in each package
|
||
under `~/ros_ws/src/` (the face suite takes about a minute). A shell that sourced the environment
|
||
before a package was built cannot import it: source again.
|
||
|
||
## Calibrating the camera
|
||
|
||
This camera's calibration ships in `pi/calibration/` and the installer restores it, so a fresh install is
|
||
already calibrated. Redo it if you change the lens focus, the image size or the camera: a changed focus
|
||
invalidates it. Set the focus first (the lens turns if it is not glued: focus at the distance you will
|
||
use), then show `docs/checkerboard-9x6.png` on a screen (or print it) and measure one square with a
|
||
ruler. With the robot running, on the Pi, after `source ~/robot-env.sh`:
|
||
|
||
```
|
||
python3 -m hermes_vision.calibrate --cols 9 --rows 6 --square-mm <measured>
|
||
```
|
||
|
||
Hold the camera 25 to 60 cm from the board and vary distance, tilt and position, **including the edges
|
||
and corners of the camera's view** (the tool says which regions are still uncovered), and hold still:
|
||
blurred views are dropped as outliers. It collects 24 different views, solves with OpenCV (holding the
|
||
`k3` distortion term at zero, which a lens this mild does not need) and writes
|
||
`~/.ros/camera_info/ov5647__base_soc_i2c0mux_i2c_1_ov5647_36_1296x972.yaml`, plus the collected views so
|
||
`--from-views FILE` can re-solve them without recapturing. A reprojection error under about 0.5 px is good.
|
||
The square size only scales distances, not the lens model, so an approximate measurement is fine.
|
||
`sudo systemctl restart robot` to load it, and copy the YAML into `pi/calibration/` to keep it.
|
||
|
||
## Tuning and troubleshooting
|
||
|
||
- **Recordings are exactly silent:** the Audio switch is off (see above).
|
||
- **Microphone too noisy or too quiet:** WirePlumber resets the codec's capture gain to +30 dB. The
|
||
installer sets PipeWire's source volume to 63% (about +17 dB); change it with `pactl
|
||
set-source-volume @DEFAULT_SOURCE@ 63%`. Remember that PipeWire's scale is cubic.
|
||
- **The face flickers between listening and idle:** raise `hold_s` in `ros/hermes_audio/hermes_audio/levels.py`.
|
||
- **Stock libcamera crashes when capturing:** that is expected on this kernel. The installer builds the
|
||
Raspberry Pi fork into `/opt/libcamera-rpi` and builds `camera_ros` against it. Anything that uses
|
||
the camera must run after `source ~/robot-env.sh`.
|
||
- **Display shows noise or a band:** the 240×240 glass sits 80 rows into the controller's memory
|
||
(`y_offset: 80`), and `rotation: 270` is the mounted pose. Both are in `ros/hermes_bringup/config/robot.yaml`.
|
||
- **`GPIO busy` or a crash at start from systemd:** the display's GPIO backend needs a writable working
|
||
directory; the unit sets one.
|
||
- **The USB interface is unmanaged:** netplan must match `usb0` by *name*, not by MAC.
|
||
- **`ros2 node list` shows only some nodes:** use `ros2 node list --no-daemon --spin-time 6`.
|
||
- **The Pi's clock is wrong with Wi-Fi off:** there is no internet to sync from; it stays roughly right
|
||
but unsynchronised.
|
||
- **Hermes says `faces` (or another new tool) is "not a known tool name":** a Hermes session takes a
|
||
snapshot of its tool table when it starts. If it began before the robot offered the tool (or before a
|
||
robot restart that added it), it never sees it, even though `robot://status`, `list_prompts` and a fresh
|
||
`hermes mcp test robot` all show it. Start a new Hermes session. (Hermes's log shows the reconnect
|
||
and "registered N tool(s)" for the whole process; only that old session keeps its stale table.)
|
||
- **A one-off message to the face is ignored:** ROS drops a message published before the connection
|
||
matches. Republish it, as `hermes_mcp` does every second.
|
||
|
||
## Security
|
||
|
||
- The robot's ports are open **only over the USB link, only to `10.55.0.1`**: audio (4713), MCP (8765)
|
||
and a reserved webhook port (8766). Everything else of those is dropped on Wi-Fi.
|
||
- **ROS is blocked on every interface**, including USB. It has no authentication and carries the camera
|
||
and recognised names.
|
||
- MCP needs a bearer token, kept in `/etc/robot-mcp.env` (root and the service user only).
|
||
- **SSH is open on all interfaces** (key-only), by choice, for the mobile version.
|
||
- Face recognition is opt-in and stores only numbers. Hermes keeps each `look()` photo in its image
|
||
cache on your computer.
|
||
- Never power the robot from a device whose battery you care about.
|
||
|
||
## Files
|
||
|
||
| Path | What |
|
||
|---|---|
|
||
| `ros/` | The ROS 2 workspace: `hermes_face`, `hermes_vision`, `hermes_audio`, `hermes_mcp`, `hermes_thermal`, `hermes_bringup`. |
|
||
| `pi/` | Everything installed on the Pi: `install.sh` and its parts, services, firewall, overlays. |
|
||
| `host/` | Your computer's side: the USB link script, Hermes and PipeWire config, a platform check. |
|
||
| `enclosure/` | OpenSCAD model, STLs and the print guide. Built from estimates: measure before printing. |
|
||
| `docs/` | `face-prototype.html` (the browser prototype the face was ported from) and `checkerboard-9x6.png` (for calibration). |
|
||
| `HANDOFF.md` | The original implementation brief. `CONTEXT.md` tracks where things really stand. |
|