hermes-bot/README.md

# Hermes robot

A small desk robot that gives Hermes Agent (Nous Research's open-source agent) eyes, ears,
a voice and a face. It does no thinking of its own: Hermes runs on your computer and uses the robot
through MCP.

- **Eyes:** a camera. Hermes calls `look()` and gets a current photo.
- **Face:** an oscilloscope-style display that shows what the agent is really doing: asleep (no agent
  connected), idle, listening, thinking, looking, speaking. The mouth draws the actual voice, and the
  eyes glance toward whoever the camera sees.
- **Ears and voice:** the HAT's two microphones and speakers, shared with your computer as ordinary
  audio devices (the speakers are not fitted yet; the microphones work).
- **Neck:** pan and tilt servos (`turn_head`), once they are fitted.
- **One cable:** USB-C carries power and a private network link to your computer. Wi-Fi is a fallback.

It runs ROS 2 Jazzy on Ubuntu 24.04, so a mobile version can grow out of it. Hermes needs no ROS: it
only speaks MCP.

## Status

| Works and is tested | Waiting on hardware |
|---|---|
| Face on the display, always on at power-up, with a sleeping pose | Servos: `turn_head`, `head_position`, the pan/tilt URDF |
| Camera (calibrated), face detection, opt-in recognition of people you enrol | Speakers: playback, echo cancel, Hermes talking out loud |
| Eyes follow faces; mouth draws your live voice | Fan (the Pi runs at about 60 °C open-air without one) |
| MCP bridge: `hermes mcp test robot` passes, real Hermes turns call `look()` and `faces()` | USB-C splitter (the Pi is powered from your computer's port for now) |
| Thermal guard: slows the camera, face and detector before the Pi throttles | |
| USB network link, firewall, boots and works with Wi-Fi off | Printed enclosure (`enclosure/`) |

## How it fits together

```
Your computer (Omarchy)                          Raspberry Pi 4 + BrainCraft HAT (Ubuntu 24.04)
  Hermes Agent ── MCP over HTTP, USB link ────►  hermes_mcp  (the only way in or out of ROS)
  PipeWire     ◄── audio, TCP 4713 ──────────►  PipeWire (not ROS)
                                                   │ ROS 2 graph, this machine only
                      camera_ros ► hermes_face_detector ► /vision/faces ─┐
                      hermes_audio_state ► levels, speaking/listening ───┤
                      hermes_mcp ► /hermes/state (the agent heartbeat) ──┴► hermes_face ► display
```

Every node is its own process, started by one launch file (`ros/hermes_bringup`) and restarted if it
dies. ROS never leaves the Pi. The firewall blocks it on every interface, so `hermes_mcp` is the one
door.

## Parts

| Part | Notes |
|---|---|
| Raspberry Pi 4 | 8 GB here. Ubuntu Server 24.04 arm64. |
| Adafruit BrainCraft HAT | 1.54″ ST7789 display, WM8960 audio codec with two mics, speaker and servo ports. |
| Arducam UC-346 | OV5647 camera, 15-pin ribbon, fixed focus. |
| 2 × MG90S servos | Pan (in the base) on the HAT's GPIO12 port, tilt (camera yoke) on GPIO13. |
| 2 × speakers | 8 Ω, up to 1 W, JST PH 2-pin. |
| USB-C power/data splitter | Power from a 5.1 V 3 A supply, data only to your computer. A Pi 4 cannot negotiate USB PD. |
| Fan | On the HAT's fan header. |
| Enclosure | `enclosure/`: printed, with a turntable. |

## Physical steps

1. The HAT's "3pin Volt" jumper must be on 5 V, which is the factory setting. Check it with a meter.
2. Pan servo into the GPIO12 port, tilt servo into GPIO13.
3. Camera ribbon: leave a gentle loop, because only the tilt flexes it.
4. **Slide the HAT's "Audio" switch (right edge) to ON.** With it off, everything looks configured and
   every recording is exactly silent. This cost an hour once.
5. Power the Pi from the splitter, not straight from a port that cannot give 3 A. Never power it from a
   laptop or tablet you care about the battery of.

## Setting up the Pi

1. **Flash** Ubuntu Server 24.04 for the Pi (arm64, preinstalled image) and check its SHA256.
2. **Configure first boot** on the card's `system-boot` partition. In `user-data`, create the user
   `hermes` with key-only login and the hostname `hermes-bot`. For Wi-Fi, put the access point in
   `network-config` (copy `pi/network-config.example` and fill it in; keep a filled copy out of the repo: `pi/50-cloud-init.yaml`
   is gitignored for that). cloud-init applies these on the *first* boot only: to change Wi-Fi later, edit
   `/etc/netplan/50-cloud-init.yaml` on the Pi.
3. **Boot, then copy this repo to the Pi and run the installer:**

   ```
   rsync -a ./ hermes@<pi>:~/hermes-bot/
   ssh hermes@<pi> 'cd hermes-bot && sudo ./pi/install.sh'
   ```

   It installs ROS, builds the camera driver against the Raspberry Pi libcamera fork, builds the
   workspace, sets up audio, the USB link, the firewall and the services. It can be run again safely.
   The first run takes a while (the libcamera build alone is about 15 minutes). **Reboot afterwards.**
4. Plug the USB-C data cable into your computer.

## Setting up your computer (Omarchy)

1. **Network link:** `sudo host/omarchy/robot-usb-link.sh` gives your end of the link `10.55.0.1/30`,
   matched by the gadget's fixed MAC, so any port and any machine works. The robot is `10.55.0.2`.
2. **Hermes:** the token is in `/etc/robot-mcp.env` on the Pi (`ssh hermes@10.55.0.2 sudo cat
   /etc/robot-mcp.env`). Put it in `~/.hermes/.env` as `ROBOT_MCP_TOKEN=`, and merge
   `host/omarchy/hermes-config-snippet.yaml` into `~/.hermes/config.yaml`. Start a new Hermes session
   afterwards: a session keeps the tool list it started with.
3. **Audio (optional, needs the speakers):** `host/omarchy/50-robot-tunnel.conf` makes the robot's
   microphone and speaker appear as devices; `host/omarchy/hermes-robot` starts Hermes with them as
   the defaults.

## First test

```
ping 10.55.0.2
hermes mcp test robot          # connects, lists the tools
hermes chat -q "Who is at the robot's desk right now?"          # calls faces(): instant, no photo
hermes chat -q "Use the look tool and tell me what you see."    # sends a photo
```

What Hermes can use: `faces()` (who is in view: names, positions, tracking ids; instant, no photo),
`look()` (the same faces summary plus a current photo), `list_people()`, and, once a head is fitted,
`turn_head` and `head_position`. `robot://status` says what hardware this robot actually has. Only faces
are detected. Face data never leaves the robot: Hermes gets names and similarity scores, and an
unrecognised face is `null`, never a guess.

The robot's face wakes when Hermes connects and falls asleep about 40 seconds after it leaves. A
look-and-describe turn takes a minute or so on local models: `look()` itself is instant. Hermes does not
give MCP images to the main model: it saves the photo to its image cache and the model calls
`vision_analyze` on it.

## Day to day

| State | Face | Comes from |
|---|---|---|
| sleeping | eyes closed, dim, slowly breathing | no agent connected (no heartbeat) |
| idle | blinks, smiles | the agent is connected |
| listening | mouth draws your voice, dimmer | the microphone (never wakes a sleeping face) |
| thinking | eyes trace looping curves | the agent |
| looking | squint, then a flash | `look()` |
| speaking | mouth draws the outgoing audio | the speaker |

- **Heat:** `hermes_thermal` watches the CPU temperature and, at 72, 77 and 82 °C (the Pi starts
  throttling at about 80), scales the camera, face and detector rates down, then restores them as it cools
  (one step at a time, so it does not flap). The temperature and the Pi's throttle and under-voltage
  flags are in `robot://status` and on `/health/*`. To test it without heating the Pi, on the Pi:
  `ros2 param set /hermes_thermal fake_temp_c 85.0` (and `-1` to stop).
- **Logs and control:** `journalctl -u robot -f` (the face logs every state change), `sudo systemctl
  restart robot`.
- **Enrol yourself** so the eyes prefer you (numbers only are stored, never photos, in
  `~/.local/share/hermes/faces.json`): on the Pi, with the robot running and after `source
  ~/robot-env.sh`, `python3 -m hermes_vision.people enroll YOURNAME`. `list` shows who is enrolled;
  `forget YOURNAME` removes you. Enrolling is deliberately not an MCP tool: a prompt must not be able to
  teach the robot a face.
- **Update the code:** `rsync -a --delete --exclude __pycache__ ros/ hermes@<pi>:~/ros_ws/src/`, then
  `colcon build --symlink-install` in `~/ros_ws`, then `sudo systemctl restart robot`.
- **Run a node by hand:** `source ~/robot-env.sh` first: it puts the libcamera fork and the right
  OpenCV on the path, and keeps ROS local.
- **Run the tests:** on the Pi, after `source ~/robot-env.sh`, `python3 -m pytest test` in each package
  under `~/ros_ws/src/` (the face suite takes about a minute). A shell that sourced the environment
  before a package was built cannot import it: source again.

## Calibrating the camera

This camera's calibration ships in `pi/calibration/` and the installer restores it, so a fresh install is
already calibrated. Redo it if you change the lens focus, the image size or the camera: a changed focus
invalidates it. Set the focus first (the lens turns if it is not glued: focus at the distance you will
use), then show `docs/checkerboard-9x6.png` on a screen (or print it) and measure one square with a
ruler. With the robot running, on the Pi, after `source ~/robot-env.sh`:

```
python3 -m hermes_vision.calibrate --cols 9 --rows 6 --square-mm <measured>
```

Hold the camera 25 to 60 cm from the board and vary distance, tilt and position, **including the edges
and corners of the camera's view** (the tool says which regions are still uncovered), and hold still:
blurred views are dropped as outliers. It collects 24 different views, solves with OpenCV (holding the
`k3` distortion term at zero, which a lens this mild does not need) and writes
`~/.ros/camera_info/ov5647__base_soc_i2c0mux_i2c_1_ov5647_36_1296x972.yaml`, plus the collected views so
`--from-views FILE` can re-solve them without recapturing. A reprojection error under about 0.5 px is good.
The square size only scales distances, not the lens model, so an approximate measurement is fine.
`sudo systemctl restart robot` to load it, and copy the YAML into `pi/calibration/` to keep it.

## Tuning and troubleshooting

- **Recordings are exactly silent:** the Audio switch is off (see above).
- **Microphone too noisy or too quiet:** WirePlumber resets the codec's capture gain to +30 dB. The
  installer sets PipeWire's source volume to 63% (about +17 dB); change it with `pactl
  set-source-volume @DEFAULT_SOURCE@ 63%`. Remember that PipeWire's scale is cubic.
- **The face flickers between listening and idle:** raise `hold_s` in `ros/hermes_audio/hermes_audio/levels.py`.
- **Stock libcamera crashes when capturing:** that is expected on this kernel. The installer builds the
  Raspberry Pi fork into `/opt/libcamera-rpi` and builds `camera_ros` against it. Anything that uses
  the camera must run after `source ~/robot-env.sh`.
- **Display shows noise or a band:** the 240×240 glass sits 80 rows into the controller's memory
  (`y_offset: 80`), and `rotation: 270` is the mounted pose. Both are in `ros/hermes_bringup/config/robot.yaml`.
- **`GPIO busy` or a crash at start from systemd:** the display's GPIO backend needs a writable working
  directory; the unit sets one.
- **The USB interface is unmanaged:** netplan must match `usb0` by *name*, not by MAC.
- **`ros2 node list` shows only some nodes:** use `ros2 node list --no-daemon --spin-time 6`.
- **The Pi's clock is wrong with Wi-Fi off:** there is no internet to sync from; it stays roughly right
  but unsynchronised.
- **Hermes says `faces` (or another new tool) is "not a known tool name":** a Hermes session takes a
  snapshot of its tool table when it starts. If it began before the robot offered the tool (or before a
  robot restart that added it), it never sees it, even though `robot://status`, `list_prompts` and a fresh
  `hermes mcp test robot` all show it. Start a new Hermes session. (Hermes's log shows the reconnect
  and "registered N tool(s)" for the whole process; only that old session keeps its stale table.)
- **A one-off message to the face is ignored:** ROS drops a message published before the connection
  matches. Republish it, as `hermes_mcp` does every second.

## Security

- The robot's ports are open **only over the USB link, only to `10.55.0.1`**: audio (4713), MCP (8765)
  and a reserved webhook port (8766). Everything else of those is dropped on Wi-Fi.
- **ROS is blocked on every interface**, including USB. It has no authentication and carries the camera
  and recognised names.
- MCP needs a bearer token, kept in `/etc/robot-mcp.env` (root and the service user only).
- **SSH is open on all interfaces** (key-only), by choice, for the mobile version.
- Face recognition is opt-in and stores only numbers. Hermes keeps each `look()` photo in its image
  cache on your computer.
- Never power the robot from a device whose battery you care about.

## Files

| Path | What |
|---|---|
| `ros/` | The ROS 2 workspace: `hermes_face`, `hermes_vision`, `hermes_audio`, `hermes_mcp`, `hermes_thermal`, `hermes_bringup`. |
| `pi/` | Everything installed on the Pi: `install.sh` and its parts, services, firewall, overlays. |
| `host/` | Your computer's side: the USB link script, Hermes and PipeWire config, a platform check. |
| `enclosure/` | OpenSCAD model, STLs and the print guide. Built from estimates: measure before printing. |
| `docs/` | `face-prototype.html` (the browser prototype the face was ported from) and `checkerboard-9x6.png` (for calibration). |
| `HANDOFF.md` | The original implementation brief. `CONTEXT.md` tracks where things really stand. |