kyle/hermes-bot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Clean out the Pi Zero leftovers, rewrite the README, and add the one installer

Browse Source 
Wi-Fi-off acceptance passed: with Wi-Fi blocked, live and after a reboot, the Pi
came up on the USB link alone, every node ran, and hermes mcp test robot and look()
worked. (A temporary guard that would have unblocked Wi-Fi was the safety net and is
removed; Wi-Fi is back on.)

Delete what the ROS design superseded: the Picamera2 MCP server and its unit, the old
installer and requirements, and the duplicate platform check. Move omarchy/ to
host/omarchy/. Rewrite README.md for this build: parts, physical steps (including the
Audio switch), setup, daily use, the troubleshooting learned the hard way, and the
security model.

pi/install.sh is now the single installer for a fresh Ubuntu 24.04 image, built from the
steps done by hand: ROS, the libcamera fork at a pinned commit, camera_ros, the venv and
models, the workspace, USB gadget, audio overlay, firewall (before PipeWire opens a
port), PipeWire, mixer and services. Run twice on the live robot, the second run changed
nothing and everything stayed healthy. It has not been run on a blank card.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
This commit is contained in:
kyle
2026-10-05 00:49:21 -07:00
co-authored by Claude Sonnet 5.5
parent 841ddc3573
commit 9952c92bc5
11 changed files with 330 additions and 645 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+144 -174
View File
@@ -1,205 +1,175 @@
# Hermes robot
A small Wi-Fi robot that gives Hermes Agent eyes, ears, a voice and (optionally) a
neck. All thinking happens on your Omarchy machine. The robot only takes pictures,
moves its head, and carries audio.
A small desk robot that gives Hermes Agent (Nous Research's open-source agent) eyes, ears,
a voice and a face. It does no thinking of its own: Hermes runs on your computer and uses the robot
through MCP.
- **Eyes:** a camera. Hermes calls `look()` and gets a current photo.
- **Face:** an oscilloscope-style display that shows what the agent is really doing: asleep (no agent
connected), idle, listening, thinking, looking, speaking. The mouth draws the actual voice, and the
eyes glance toward whoever the camera sees.
- **Ears and voice:** the HAT's two microphones and speakers, shared with your computer as ordinary
audio devices.
- **Neck:** pan and tilt servos (`turn_head`), once they are fitted.
- **One cable:** USB-C carries power and a private network link to your computer. Wi-Fi is a fallback.
It runs ROS 2 Jazzy on Ubuntu 24.04, so a mobile version can grow out of it. Hermes needs no ROS: it
only speaks MCP.
## Status
| Works and is tested | Waiting on hardware |
|---|---|
| Face on the display, always on at power-up, with a sleeping pose | Servos: `turn_head`, `head_position`, the pan/tilt URDF |
| Camera, face detection, opt-in recognition of people you enrol | Speakers: playback, echo cancel, Hermes talking out loud |
| Eyes follow faces; mouth draws your live voice | Fan (the Pi runs at about 62 °C open-air without one) |
| MCP bridge: `hermes mcp test robot` passes, a real Hermes turn calls `look()` | USB-C splitter (the Pi is powered from your computer's port for now) |
| USB network link, firewall, boots and works with Wi-Fi off | Printed enclosure (`enclosure/`) |
## How it fits together
```
Omarchy machine Robot (Pi Zero 2 W)
─────────────── ───────────────────
hermes-robot ─► Hermes Agent
│ voice mode / wake word
├── default mic = "Robot microphone" ◄─┐
├── default sink = "Robot speaker" ─────┤ PipeWire pulse tunnel, TCP 4713
│ └─► pipewire-pulse ─► I²S mic(s) + amp
└── MCP tools: look(), turn_head() ──────► robot_mcp_server.py, HTTP 8765
├─ Camera Module 3
└─ PCA9685 ─► 2 servos (optional)
Your computer (Omarchy) Raspberry Pi 4 + BrainCraft HAT (Ubuntu 24.04)
Hermes Agent ── MCP over HTTP, USB link ────► hermes_mcp (the only way in or out of ROS)
PipeWire ◄── audio, TCP 4713 ──────────► PipeWire (not ROS)
│ ROS 2 graph, this machine only
camera_ros ► hermes_face_detector ► /vision/faces ─┐
hermes_audio_state ► levels, speaking/listening ───┤
hermes_mcp ► /hermes/state (the agent heartbeat) ──┴► hermes_face ► display
```
Every node is its own process, started by one launch file (`ros/hermes_bringup`) and restarted if it
dies. ROS never leaves the Pi. The firewall blocks it on every interface, so `hermes_mcp` is the one
door.
## Parts
Prices are rough single-unit figures and will vary by vendor.
| Part | Notes |
|---|---|
| Raspberry Pi 4 | 8 GB here. Ubuntu Server 24.04 arm64. |
| Adafruit BrainCraft HAT | 1.54″ ST7789 display, WM8960 audio codec with two mics, speaker and servo ports. |
| Arducam UC-346 | OV5647 camera, 15-pin ribbon, fixed focus. |
| 2 × MG90S servos | Pan (in the base) on the HAT's GPIO12 port, tilt (camera yoke) on GPIO13. |
| 2 × speakers | 8 Ω, up to 1 W, JST PH 2-pin. |
| USB-C power/data splitter | Power from a 5.1 V 3 A supply, data only to your computer. A Pi 4 cannot negotiate USB PD. |
| Fan | On the HAT's fan header. |
| Enclosure | `enclosure/`: printed, with a turntable. |
| Part | Notes | ≈ USD |
|---|---|---|
| Raspberry Pi Zero 2 W (the "WH" has headers soldered) | 2.4 GHz Wi-Fi only | 15–18 |
| microSD card, 16–32 GB | | 8 |
| Raspberry Pi Camera Module 3 | Autofocus; the "Wide" variant suits a desk robot | 25 |
| **Pi Zero camera cable** (22-pin to 15-pin) | The camera ships with the wrong cable for a Zero | 3 |
| INMP441 I²S MEMS microphone breakout, ×2 | One works; two give a stereo pair and a better signal | 3 each |
| MAX98357A I²S amplifier breakout | | 2–6 |
| Speaker, 4–8 Ω, 2–3 W, ~40 mm | | 3 |
| 5 V 3 A USB power supply | | 10 |
| *Optional head:* PCA9685 16-channel servo board | Clean hardware PWM over I²C | 6 |
| *Optional head:* 2× SG90 servos + pan/tilt bracket | | 8 |
| *Optional head:* 1000 µF electrolytic capacitor | Across the servo supply, against brown-outs | 1 |
| Dupont jumper wires, a bit of perfboard | | 5 |
## Physical steps
Without the head, the robot is roughly $70; with it, about $85.
## Wiring
Pin numbers are **physical header pins** (1–40), not GPIO numbers. The microphone(s)
and amplifier share the I²S clock lines; that's expected.
**Microphones (INMP441)**
| INMP441 | Pi pin | |
|---|---|---|
| VDD | 1 (3.3 V) | **3.3 V only**, never 5 V |
| GND | 6 (GND) | |
| SCK | 12 (GPIO18, I²S BCLK) | |
| WS | 35 (GPIO19, I²S LRCLK) | |
| SD | 38 (GPIO20, I²S DIN) | |
| L/R | GND for mic A; 3.3 V for mic B | Selects left or right channel |
With two microphones, wire both identically except L/R, so their SD lines share pin 38.
**Amplifier (MAX98357A)**
| MAX98357A | Pi pin | |
|---|---|---|
| Vin | 2 (5 V) | |
| GND | 9 (GND) | |
| BCLK | 12 (GPIO18) | Shared with the mics |
| LRC | 35 (GPIO19) | Shared with the mics |
| DIN | 40 (GPIO21, I²S DOUT) | |
| SD, GAIN | leave unconnected | Defaults: enabled, 9 dB gain |
| Speaker + / − | to the speaker | |
**Optional head (PCA9685)**
| PCA9685 | Connects to | |
|---|---|---|
| VCC | Pi pin 17 (3.3 V) | Logic supply |
| GND | Pi pin 14 (GND) | |
| SDA | Pi pin 3 (GPIO2) | |
| SCL | Pi pin 5 (GPIO3) | |
| V+ terminal | **5 V from the power supply**, not from the Pi's header | Servos' power; add the capacitor here |
| Channel 0 | pan servo | |
| Channel 1 | tilt servo | |
Servos draw current spikes that will brown out the Pi if they share its header pins. Feed
the supply's 5 V to both the Pi (micro-USB) and the PCA9685's V+ terminal, with a common ground.
**Camera:** the Zero cable's narrow end goes in the Pi, contacts facing the board.
The `googlevoicehat-soundcard` overlay is the standard trick for this exact mic + amp pairing.
Google's AIY Voice HAT used the same chips, so its driver drives both in one full-duplex sound card.
1. The HAT's "3pin Volt" jumper must be on 5 V, which is the factory setting. Check it with a meter.
2. Pan servo into the GPIO12 port, tilt servo into GPIO13.
3. Camera ribbon: leave a gentle loop, because only the tilt flexes it.
4. **Slide the HAT's "Audio" switch (right edge) to ON.** With it off, everything looks configured and
every recording is exactly silent. This cost an hour once.
5. Power the Pi from the splitter, not straight from a port that cannot give 3 A. Never power it from a
laptop or tablet you care about the battery of.
## Setting up the Pi
1. Flash **Raspberry Pi OS Lite (64-bit)** with Raspberry Pi Imager. In its settings, set the
hostname (e.g. `robot`), your user, your Wi-Fi, and enable SSH.
2. Boot, find the Pi's IP, and give it a **DHCP reservation** on your router.
3. Copy the `pi/` folder over and run the installer, passing your Omarchy machine's IP:
1. **Flash** Ubuntu Server 24.04 for the Pi (arm64, preinstalled image) and check its SHA256.
2. **Configure first boot** on the card's `system-boot` partition. In `user-data`, create the user
`hermes` with key-only login and the hostname `hermes-bot`. For Wi-Fi, put the access point in
`network-config` (a template is `pi/50-cloud-init.yaml`; it holds your password, so it is
gitignored). cloud-init applies these on the *first* boot only: to change Wi-Fi later, edit
`/etc/netplan/50-cloud-init.yaml` on the Pi.
3. **Boot, then copy this repo to the Pi and run the installer:**
```bash
scp -r pi robot.local:~/robot-setup
ssh robot.local
cd ~/robot-setup && chmod +x install.sh
./install.sh 192.168.1.20 # add --servos if you built the head
sudo reboot
```
rsync -a ./ hermes@<pi>:~/hermes-bot/
ssh hermes@<pi> 'cd hermes-bot && sudo ./pi/install.sh'
```
The installer prints a token at the end. Keep it for the Hermes step.
4. After the reboot, check things on the Pi:
It installs ROS, builds the camera driver against the Raspberry Pi libcamera fork, builds the
workspace, sets up audio, the USB link, the firewall and the services. It can be run again safely.
The first run takes a while (the libcamera build alone is about 15 minutes). **Reboot afterwards.**
4. Plug the USB-C data cable into your computer.
```bash
arecord -l # a "googlevoicehat" card should be listed
wpctl status # its source and sink should appear under Audio
rpicam-still -o test.jpg # camera works
systemctl status robot-mcp # MCP server running
```
## Setting up your computer (Omarchy)
5. If the camera is mounted upside down, set both `ROBOT_CAM_HFLIP=1` and `ROBOT_CAM_VFLIP=1`
in `/etc/robot-mcp.env`, then run `sudo systemctl restart robot-mcp`.
## Setting up Omarchy
1. **Audio tunnel.** Copy `omarchy/50-robot-tunnel.conf` to
`~/.config/pipewire/pipewire.conf.d/`, replace both `ROBOT_IP`s, then:
```bash
systemctl --user restart pipewire pipewire-pulse wireplumber
pactl list short sources | grep robot_mic # should show the robot
```
Omarchy's audio mixer (`wiremix`) will now list "Robot speaker" and "Robot microphone".
If `pactl` is missing, install it with `sudo pacman -S libpulse`.
2. **Hermes.** Add `ROBOT_MCP_TOKEN=<token>` to `~/.hermes/.env`. Merge
`omarchy/hermes-config-snippet.yaml` into `~/.hermes/config.yaml`, with `ROBOT_IP` replaced.
Then check the connection:
```bash
hermes mcp test robot
```
The tools will appear in Hermes as `mcp_robot_look`, `mcp_robot_turn_head` and so on.
3. **Launcher.** Install the wrapper, which switches your default audio to the robot for the
length of the session and restores it afterwards:
```bash
install -Dm755 omarchy/hermes-robot ~/.local/bin/hermes-robot
hermes-robot
```
Inside Hermes, `/voice on` starts the conversation. Set up the wake word as described in the
Hermes docs ("Wake Word") so you never need the keyboard.
1. **Network link:** `sudo host/omarchy/robot-usb-link.sh` gives your end of the link `10.55.0.1/30`,
matched by the gadget's fixed MAC, so any port and any machine works. The robot is `10.55.0.2`.
2. **Hermes:** the token is in `/etc/robot-mcp.env` on the Pi. Put it in `~/.hermes/.env` as
`ROBOT_MCP_TOKEN=`, and merge `host/omarchy/hermes-config-snippet.yaml` into
`~/.hermes/config.yaml`.
3. **Audio (optional, needs the speakers):** `host/omarchy/50-robot-tunnel.conf` makes the robot's
microphone and speaker appear as devices; `host/omarchy/hermes-robot` starts Hermes with them as
the defaults.
## First test
Say or type: *"Look around and tell me what's on my desk."* Hermes should call `look`, then
describe the scene.
```
ping 10.55.0.2
hermes mcp test robot # connects, finds the look tool
hermes chat -q "Use the look tool and tell me what you see."
```
**One thing to confirm early:** that the image actually reaches the model. The server returns a
standard MCP image block, and the test client sees it correctly. If Hermes answers as though it
saw nothing, check `~/.hermes/logs/agent.log`, and make sure your model supports vision. If
Hermes turns out to drop image blocks from MCP results, the fallback is a small change to
`look()` to save the JPEG and return a path for Hermes's own vision tool. Tell me if that happens.
The robot's face wakes when Hermes connects and falls asleep about 40 seconds after it leaves. A
look-and-describe turn takes a minute or so on local models: `look()` itself is instant. Hermes does not
give MCP images to the main model: it saves the photo to its image cache and the model calls
`vision_analyze` on it.
## Day to day
| State | Face | Comes from |
|---|---|---|
| sleeping | eyes closed, dim, slowly breathing | no agent connected (no heartbeat) |
| idle | blinks, smiles | the agent is connected |
| listening | mouth draws your voice, dimmer | the microphone (never wakes a sleeping face) |
| thinking | eyes trace looping curves | the agent |
| looking | squint, then a flash | `look()` |
| speaking | mouth draws the outgoing audio | the speaker |
- **Logs and control:** `journalctl -u robot -f` (the face logs every state change), `sudo systemctl
restart robot`.
- **Enrol yourself** so the eyes prefer you (numbers only are stored, never photos, in
`~/.local/share/hermes/faces.json`): on the Pi, with the robot running and after `source
~/robot-env.sh`, `python3 -m hermes_vision.people enroll YOURNAME`. `list` shows who is enrolled, `forget YOURNAME` removes you.
- **Update the code:** `rsync -a --delete --exclude __pycache__ ros/ hermes@<pi>:~/ros_ws/src/`, then
`colcon build --symlink-install` in `~/ros_ws`, then `sudo systemctl restart robot`.
- **Run a node by hand:** `source ~/robot-env.sh` first: it puts the libcamera fork and the right
OpenCV on the path, and keeps ROS local.
## Tuning and troubleshooting
- **Mic too quiet** (the level bar barely moves): on the Pi, run
`wpctl set-volume @DEFAULT_AUDIO_SOURCE@ 2.5`. Alternatively, lower `voice.silence_threshold`
in Hermes. A single mic is about 6 dB quieter than a pair, because the unused channel is
averaged in.
- **Hermes interrupts itself:** this is the robot hearing its own speaker. Raise
`voice.barge_in_threshold_multiplier`, point the speaker away from the mics, or enable
`pi/pipewire/60-echo-cancel.conf.optional` and uncomment the `pulse.device` lines in the
tunnel config.
- **Audio crackles:** this means weak Wi-Fi. Raise `pulse.latency` to 300–400. The Zero 2 W has
only 2.4 GHz Wi-Fi and a small antenna, so keep it within reasonable range of the router.
- **Robot devices vanish after the Pi reboots:** the tunnels reconnect every 5 s by themselves.
Start `hermes-robot` again once they reappear.
- **Head turns the wrong way:** set `ROBOT_PAN_INVERT=1` or `ROBOT_TILT_INVERT=1` in
`/etc/robot-mcp.env`, then restart `robot-mcp`.
- **Recordings are exactly silent:** the Audio switch is off (see above).
- **Microphone too noisy or too quiet:** WirePlumber resets the codec's capture gain to +30 dB. The
installer sets PipeWire's source volume to 63% (about +17 dB); change it with `pactl
set-source-volume @DEFAULT_SOURCE@ 63%`. Remember that PipeWire's scale is cubic.
- **The face flickers between listening and idle:** raise `hold_s` in `ros/hermes_audio/hermes_audio/levels.py`.
- **Stock libcamera crashes when capturing:** that is expected on this kernel. The installer builds the
Raspberry Pi fork into `/opt/libcamera-rpi` and builds `camera_ros` against it. Anything that uses
the camera must run after `source ~/robot-env.sh`.
- **Display shows noise or a band:** the 240×240 glass sits 80 rows into the controller's memory
(`y_offset: 80`), and `rotation: 270` is the mounted pose. Both are in `ros/hermes_bringup/config/robot.yaml`.
- **`GPIO busy` or a crash at start from systemd:** the display's GPIO backend needs a writable working
directory; the unit sets one.
- **The USB interface is unmanaged:** netplan must match `usb0` by *name*, not by MAC.
- **`ros2 node list` shows only some nodes:** use `ros2 node list --no-daemon --spin-time 6`.
- **The Pi's clock is wrong with Wi-Fi off:** there is no internet to sync from; it stays roughly right
but unsynchronised.
- **A one-off message to the face is ignored:** ROS drops a message published before the connection
matches. Republish it, as `hermes_mcp` does every second.
## Security
- The microphone stream (port 4713) and the camera server (port 8765) are open only to your
Omarchy machine's IP, via nftables on the Pi.
- The camera additionally requires the bearer token.
- Traffic is unencrypted on your LAN. If that matters to you, put both machines on Tailscale and
use the Pi's tailnet address in both configs.
- If your Omarchy machine's IP changes, re-run `install.sh` with the new one.
- The robot's ports are open **only over the USB link, only to `10.55.0.1`**: audio (4713), MCP (8765)
and a reserved webhook port (8766). Everything else of those is dropped on Wi-Fi.
- **ROS is blocked on every interface**, including USB. It has no authentication and carries the camera
and recognised names.
- MCP needs a bearer token, kept in `/etc/robot-mcp.env` (root and the service user only).
- **SSH is open on all interfaces** (key-only), by choice, for the mobile version.
- Face recognition is opt-in and stores only numbers. Hermes keeps each `look()` photo in its image
cache on your computer.
- Never power the robot from a device whose battery you care about.
## Files
```
pi/
install.sh one-shot Pi setup
robot_mcp_server.py MCP server: look, turn_head, head_position
requirements.txt MCP Python SDK 2.x
robot-mcp.service systemd unit
nftables-robot.nft port restriction
pipewire/50-network-server.conf exposes audio to the network
pipewire/60-echo-cancel.conf.optional
omarchy/
50-robot-tunnel.conf robot audio as local devices
hermes-config-snippet.yaml MCP and voice settings for Hermes
hermes-robot launcher that swaps default audio devices
```
| Path | What |
|---|---|
| `ros/` | The ROS 2 workspace: `hermes_face`, `hermes_vision`, `hermes_audio`, `hermes_mcp`, `hermes_bringup`. |
| `pi/` | Everything installed on the Pi: `install.sh` and its parts, services, firewall, overlays. |
| `host/` | Your computer's side: the USB link script, Hermes and PipeWire config, a platform check. |
| `enclosure/` | OpenSCAD model, STLs and the print guide. Built from estimates: measure before printing. |
| `docs/face-prototype.html` | The browser prototype the face was ported from. |
| `HANDOFF.md` | The original implementation brief. `CONTEXT.md` tracks where things really stand. |