kyle/hermes-bot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Clean out the Pi Zero leftovers, rewrite the README, and add the one installer

Browse Source 
Wi-Fi-off acceptance passed: with Wi-Fi blocked, live and after a reboot, the Pi
came up on the USB link alone, every node ran, and hermes mcp test robot and look()
worked. (A temporary guard that would have unblocked Wi-Fi was the safety net and is
removed; Wi-Fi is back on.)

Delete what the ROS design superseded: the Picamera2 MCP server and its unit, the old
installer and requirements, and the duplicate platform check. Move omarchy/ to
host/omarchy/. Rewrite README.md for this build: parts, physical steps (including the
Audio switch), setup, daily use, the troubleshooting learned the hard way, and the
security model.

pi/install.sh is now the single installer for a fresh Ubuntu 24.04 image, built from the
steps done by hand: ROS, the libcamera fork at a pinned commit, camera_ros, the venv and
models, the workspace, USB gadget, audio overlay, firewall (before PipeWire opens a
port), PipeWire, mixer and services. Run twice on the live robot, the second run changed
nothing and everything stayed healthy. It has not been run on a blank card.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
This commit is contained in:
kyle
2026-10-05 00:49:21 -07:00
co-authored by Claude Sonnet 5.5
parent 841ddc3573
commit 9952c92bc5
11 changed files with 330 additions and 645 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+4 -2
View File
@@ -8,11 +8,13 @@ the Pi; on the hosts, ROS runs only in podman containers.
**Start with `HANDOFF.md`.** It is the implementation brief: hardware facts, what exists, the
work packages in order, acceptance checks, and the list of unverified assumptions.
- `pi/` and `ros/` run on the robot; `omarchy/` (to become `host/`) runs on the owner's machines.
- `pi/` and `ros/` run on the robot; `host/` runs on the owner's machines. `CONTEXT.md` tracks where
the work stands; `HANDOFF.md` is the original brief and has drifted (ROS stays on the Pi, and
`hermes_mcp` is the only bridge out; see `CONTEXT.md`).
- Package 0a (platform proof) comes before everything; start with `host/check-pi-platform.sh`.
- `docs/face-prototype.html` is the reference for the face renderer; port it, don't ship it.
- `enclosure/` holds the OpenSCAD model and STLs. It was built from estimates; refine it from the
owner's caliper measurements and test prints before anyone prints the large parts.
- `README.md` still describes the earlier Pi Zero build until package 7 rewrites it.
- `README.md` is the guide for this build; keep it a guide for a person, and keep `CONTEXT.md` honest.
- The MCP server uses the MCP Python SDK 2.x (`mcp.server.mcpserver.MCPServer`), not 1.x `FastMCP`.
- Ask before rebooting the Pi, upgrading or holding kernel packages, or opening network ports.
CONTEXT.md
+29 -27
View File
@@ -223,8 +223,8 @@ returns a JPEG in 0.08 s, a wrong token is refused, the Wi-Fi path is dropped by
the face went `looking` -> `idle` on connect and `agent gone, sleeping` about 40 s after the client
left. The token is in `/etc/robot-mcp.env` (root:hermes, mode 640), generated by
`pi/install-robot-service.sh` and never printed; read it with `sudo cat`. The MCP SDK is in `~/venv`
(`pip install "mcp>=2"`). The old `pi/robot_mcp_server.py`, `pi/robot-mcp.service` and `pi/install.sh`
are superseded (package 5 rewrites the installer). **Not yet verified:** a real Hermes client, and
(`pip install "mcp>=2"`). The old Pi Zero files (`pi/robot_mcp_server.py`, `pi/robot-mcp.service`,
`pi/requirements.txt`, the old `pi/install.sh`, and `tools/`) were deleted (git has them). **Not yet verified:** a real Hermes client, and
whether Hermes passes MCP image content to the model (HANDOFF package 6).
**Hermes on talos is wired to it (2026-10-05, with the owner's authorisation).** Hermes Agent v0.21.5 is
@@ -264,34 +264,36 @@ unrelated photo scored 0.16. **Nobody is enrolled yet**; the owner enrols themse
## Next
1. Finish package 0b: Wi-Fi-off boot test, replug recovery, serial login check, NixOS snippet in
`host/nixos/`. A new Omarchy host with a GPU (runs the local
model "ornith") arrives this week: it needs only `host/omarchy/robot-usb-link.sh` and its SSH key
added to the Pi.
2. `hermes_head` and the servos when they arrive (then `head: true`), echo cancel and the `robot_mic`
naming once speakers arrive, a fan before any enclosure, the Wi-Fi-off boot test, a real Hermes
client against `hermes_mcp`, and the rest of `HANDOFF.md`.
3. Confirm the retuned listening detector live (the flicker fix is unit-tested only); a second person
in frame for the recognition test (not possible on 2026-10-04).
1. **Camera calibration** (needs the owner and a flat checkerboard, printed or on a screen, with the
square size measured): collect ~20 views at different poses, solve with OpenCV, write the
camera_info YAML so `camera_ros` stops logging the missing-file error. V2's perception needs it.
2. Waiting on hardware: the servos (`hermes_head`, the pan/tilt URDF, then `head: true`), the speakers
(playback, echo cancel, the `robot_mic` naming on the host, Hermes talking aloud), the fan (before
any enclosure), the splitter, the printed enclosure (measure first, see `enclosure/README.md`).
3. Still open: replug and second-host recovery of the USB link, the serial console login, an
`agent_timeout_s` check against a long idle Hermes session, the retuned listening detector live
(unit-tested only), a second person in frame, running `pi/install.sh` on a blank card, and the
NixOS host config (skipped for now by the owner).
4. A new Omarchy host with a GPU (it runs the local `ornith` model) is arriving: it needs only
`host/omarchy/robot-usb-link.sh` and its SSH key added to the Pi.
## Repository
Branch `master`. `.gitignore` covers `.remember/`, caches, `.venv/`, `*.orig`, `*.rej` and
`pi/50-cloud-init.yaml`. `tools/check-pi-platform.sh` duplicates `host/check-pi-platform.sh` and can
probably go.
`pi/50-cloud-init.yaml`. Layout: `ros/` (five packages), `pi/` (everything installed on the Pi),
`host/` (the owner's machines: `host/omarchy/` has the USB link script, Hermes and PipeWire config and
the launcher; `host/check-pi-platform.sh`), `enclosure/`, `docs/face-prototype.html`. `README.md` is
rewritten for this build. **`pi/install.sh` is the one installer** (re-runnable: run twice on the live
robot, the second run changed nothing, exit 0): it installs ROS, builds the libcamera fork (pinned
commit) and `camera_ros`, sets up the venv, models, workspace, USB gadget, audio overlay, firewall,
PipeWire, mixer and services. It has not been run on a blank card (no spare card). HANDOFF.md is the
original brief and has drifted; where they differ, this file wins.
## Security review notes (parked until the USB link is up)
## Security notes (all resolved)
An automated review of the initial commit flagged three items. All stem from the code predating
the USB-gadget design (HANDOFF.md package 0b); none is exposed until the Pi runs the services.
- `pi/robot_mcp_server.py`: plain HTTP, `ROBOT_MCP_BIND` defaults to `0.0.0.0`. Plan: default to
`10.55.0.2`, start after `usb0`.
- `pi/pipewire/50-network-server.conf`: TCP 4713 on all interfaces, unrestricted. Plan: keep,
comment that the firewall is the protection.
- `pi/nftables-robot.nft`: still the Pi Zero version (single host IP, any interface). Package 0b
rewrites it to accept only on the USB interface from `10.55.0.1`.
- `pi/install.sh` permissions looked fine (`/etc/robot-mcp.env` is `root:$USER`, 640).
The owner wants to confirm the real IPs on the link before changing any of this. If the USB
link doesn't work, revisit: TLS or a Tailscale-only bind for MCP and audio.
The automated review of the initial commit flagged three items, all from the Pi Zero design:
`ROBOT_MCP_BIND` defaulting to `0.0.0.0` (now the USB address only, with a bearer token), PipeWire on
TCP 4713 on every interface (now protected by the boot-time firewall, which loads first), and the
single-IP any-interface nftables file (rewritten: usb0 from 10.55.0.1 only). Later reviews added:
DDS is blocked on every interface and discovery is local-only; recognised names are not logged; node
parameters are clamped. SSH stays open on all interfaces (key-only) by the owner's choice, for V2.
README.md
+144 -174
View File
@@ -1,205 +1,175 @@
# Hermes robot
A small Wi-Fi robot that gives Hermes Agent eyes, ears, a voice and (optionally) a
neck. All thinking happens on your Omarchy machine. The robot only takes pictures,
moves its head, and carries audio.
A small desk robot that gives Hermes Agent (Nous Research's open-source agent) eyes, ears,
a voice and a face. It does no thinking of its own: Hermes runs on your computer and uses the robot
through MCP.
- **Eyes:** a camera. Hermes calls `look()` and gets a current photo.
- **Face:** an oscilloscope-style display that shows what the agent is really doing: asleep (no agent
connected), idle, listening, thinking, looking, speaking. The mouth draws the actual voice, and the
eyes glance toward whoever the camera sees.
- **Ears and voice:** the HAT's two microphones and speakers, shared with your computer as ordinary
audio devices.
- **Neck:** pan and tilt servos (`turn_head`), once they are fitted.
- **One cable:** USB-C carries power and a private network link to your computer. Wi-Fi is a fallback.
It runs ROS 2 Jazzy on Ubuntu 24.04, so a mobile version can grow out of it. Hermes needs no ROS: it
only speaks MCP.
## Status
| Works and is tested | Waiting on hardware |
|---|---|
| Face on the display, always on at power-up, with a sleeping pose | Servos: `turn_head`, `head_position`, the pan/tilt URDF |
| Camera, face detection, opt-in recognition of people you enrol | Speakers: playback, echo cancel, Hermes talking out loud |
| Eyes follow faces; mouth draws your live voice | Fan (the Pi runs at about 62 °C open-air without one) |
| MCP bridge: `hermes mcp test robot` passes, a real Hermes turn calls `look()` | USB-C splitter (the Pi is powered from your computer's port for now) |
| USB network link, firewall, boots and works with Wi-Fi off | Printed enclosure (`enclosure/`) |
## How it fits together
```
Omarchy machine Robot (Pi Zero 2 W)
─────────────── ───────────────────
hermes-robot ─► Hermes Agent
│ voice mode / wake word
├── default mic = "Robot microphone" ◄─┐
├── default sink = "Robot speaker" ─────┤ PipeWire pulse tunnel, TCP 4713
│ └─► pipewire-pulse ─► I²S mic(s) + amp
└── MCP tools: look(), turn_head() ──────► robot_mcp_server.py, HTTP 8765
├─ Camera Module 3
└─ PCA9685 ─► 2 servos (optional)
Your computer (Omarchy) Raspberry Pi 4 + BrainCraft HAT (Ubuntu 24.04)
Hermes Agent ── MCP over HTTP, USB link ────► hermes_mcp (the only way in or out of ROS)
PipeWire ◄── audio, TCP 4713 ──────────► PipeWire (not ROS)
│ ROS 2 graph, this machine only
camera_ros ► hermes_face_detector ► /vision/faces ─┐
hermes_audio_state ► levels, speaking/listening ───┤
hermes_mcp ► /hermes/state (the agent heartbeat) ──┴► hermes_face ► display
```
Every node is its own process, started by one launch file (`ros/hermes_bringup`) and restarted if it
dies. ROS never leaves the Pi. The firewall blocks it on every interface, so `hermes_mcp` is the one
door.
## Parts
Prices are rough single-unit figures and will vary by vendor.
| Part | Notes |
|---|---|
| Raspberry Pi 4 | 8 GB here. Ubuntu Server 24.04 arm64. |
| Adafruit BrainCraft HAT | 1.54″ ST7789 display, WM8960 audio codec with two mics, speaker and servo ports. |
| Arducam UC-346 | OV5647 camera, 15-pin ribbon, fixed focus. |
| 2 × MG90S servos | Pan (in the base) on the HAT's GPIO12 port, tilt (camera yoke) on GPIO13. |
| 2 × speakers | 8 Ω, up to 1 W, JST PH 2-pin. |
| USB-C power/data splitter | Power from a 5.1 V 3 A supply, data only to your computer. A Pi 4 cannot negotiate USB PD. |
| Fan | On the HAT's fan header. |
| Enclosure | `enclosure/`: printed, with a turntable. |
| Part | Notes | ≈ USD |
|---|---|---|
| Raspberry Pi Zero 2 W (the "WH" has headers soldered) | 2.4 GHz Wi-Fi only | 15–18 |
| microSD card, 16–32 GB | | 8 |
| Raspberry Pi Camera Module 3 | Autofocus; the "Wide" variant suits a desk robot | 25 |
| **Pi Zero camera cable** (22-pin to 15-pin) | The camera ships with the wrong cable for a Zero | 3 |
| INMP441 I²S MEMS microphone breakout, ×2 | One works; two give a stereo pair and a better signal | 3 each |
| MAX98357A I²S amplifier breakout | | 2–6 |
| Speaker, 4–8 Ω, 2–3 W, ~40 mm | | 3 |
| 5 V 3 A USB power supply | | 10 |
| *Optional head:* PCA9685 16-channel servo board | Clean hardware PWM over I²C | 6 |
| *Optional head:* 2× SG90 servos + pan/tilt bracket | | 8 |
| *Optional head:* 1000 µF electrolytic capacitor | Across the servo supply, against brown-outs | 1 |
| Dupont jumper wires, a bit of perfboard | | 5 |
## Physical steps
Without the head, the robot is roughly $70; with it, about $85.
## Wiring
Pin numbers are **physical header pins** (1–40), not GPIO numbers. The microphone(s)
and amplifier share the I²S clock lines; that's expected.
**Microphones (INMP441)**
| INMP441 | Pi pin | |
|---|---|---|
| VDD | 1 (3.3 V) | **3.3 V only**, never 5 V |
| GND | 6 (GND) | |
| SCK | 12 (GPIO18, I²S BCLK) | |
| WS | 35 (GPIO19, I²S LRCLK) | |
| SD | 38 (GPIO20, I²S DIN) | |
| L/R | GND for mic A; 3.3 V for mic B | Selects left or right channel |
With two microphones, wire both identically except L/R, so their SD lines share pin 38.
**Amplifier (MAX98357A)**
| MAX98357A | Pi pin | |
|---|---|---|
| Vin | 2 (5 V) | |
| GND | 9 (GND) | |
| BCLK | 12 (GPIO18) | Shared with the mics |
| LRC | 35 (GPIO19) | Shared with the mics |
| DIN | 40 (GPIO21, I²S DOUT) | |
| SD, GAIN | leave unconnected | Defaults: enabled, 9 dB gain |
| Speaker + / − | to the speaker | |
**Optional head (PCA9685)**
| PCA9685 | Connects to | |
|---|---|---|
| VCC | Pi pin 17 (3.3 V) | Logic supply |
| GND | Pi pin 14 (GND) | |
| SDA | Pi pin 3 (GPIO2) | |
| SCL | Pi pin 5 (GPIO3) | |
| V+ terminal | **5 V from the power supply**, not from the Pi's header | Servos' power; add the capacitor here |
| Channel 0 | pan servo | |
| Channel 1 | tilt servo | |
Servos draw current spikes that will brown out the Pi if they share its header pins. Feed
the supply's 5 V to both the Pi (micro-USB) and the PCA9685's V+ terminal, with a common ground.
**Camera:** the Zero cable's narrow end goes in the Pi, contacts facing the board.
The `googlevoicehat-soundcard` overlay is the standard trick for this exact mic + amp pairing.
Google's AIY Voice HAT used the same chips, so its driver drives both in one full-duplex sound card.
1. The HAT's "3pin Volt" jumper must be on 5 V, which is the factory setting. Check it with a meter.
2. Pan servo into the GPIO12 port, tilt servo into GPIO13.
3. Camera ribbon: leave a gentle loop, because only the tilt flexes it.
4. **Slide the HAT's "Audio" switch (right edge) to ON.** With it off, everything looks configured and
every recording is exactly silent. This cost an hour once.
5. Power the Pi from the splitter, not straight from a port that cannot give 3 A. Never power it from a
laptop or tablet you care about the battery of.
## Setting up the Pi
1. Flash **Raspberry Pi OS Lite (64-bit)** with Raspberry Pi Imager. In its settings, set the
hostname (e.g. `robot`), your user, your Wi-Fi, and enable SSH.
2. Boot, find the Pi's IP, and give it a **DHCP reservation** on your router.
3. Copy the `pi/` folder over and run the installer, passing your Omarchy machine's IP:
1. **Flash** Ubuntu Server 24.04 for the Pi (arm64, preinstalled image) and check its SHA256.
2. **Configure first boot** on the card's `system-boot` partition. In `user-data`, create the user
`hermes` with key-only login and the hostname `hermes-bot`. For Wi-Fi, put the access point in
`network-config` (a template is `pi/50-cloud-init.yaml`; it holds your password, so it is
gitignored). cloud-init applies these on the *first* boot only: to change Wi-Fi later, edit
`/etc/netplan/50-cloud-init.yaml` on the Pi.
3. **Boot, then copy this repo to the Pi and run the installer:**
```bash
scp -r pi robot.local:~/robot-setup
ssh robot.local
cd ~/robot-setup && chmod +x install.sh
./install.sh 192.168.1.20 # add --servos if you built the head
sudo reboot
```
rsync -a ./ hermes@<pi>:~/hermes-bot/
ssh hermes@<pi> 'cd hermes-bot && sudo ./pi/install.sh'
```
The installer prints a token at the end. Keep it for the Hermes step.
4. After the reboot, check things on the Pi:
It installs ROS, builds the camera driver against the Raspberry Pi libcamera fork, builds the
workspace, sets up audio, the USB link, the firewall and the services. It can be run again safely.
The first run takes a while (the libcamera build alone is about 15 minutes). **Reboot afterwards.**
4. Plug the USB-C data cable into your computer.
```bash
arecord -l # a "googlevoicehat" card should be listed
wpctl status # its source and sink should appear under Audio
rpicam-still -o test.jpg # camera works
systemctl status robot-mcp # MCP server running
```
## Setting up your computer (Omarchy)
5. If the camera is mounted upside down, set both `ROBOT_CAM_HFLIP=1` and `ROBOT_CAM_VFLIP=1`
in `/etc/robot-mcp.env`, then run `sudo systemctl restart robot-mcp`.
## Setting up Omarchy
1. **Audio tunnel.** Copy `omarchy/50-robot-tunnel.conf` to
`~/.config/pipewire/pipewire.conf.d/`, replace both `ROBOT_IP`s, then:
```bash
systemctl --user restart pipewire pipewire-pulse wireplumber
pactl list short sources | grep robot_mic # should show the robot
```
Omarchy's audio mixer (`wiremix`) will now list "Robot speaker" and "Robot microphone".
If `pactl` is missing, install it with `sudo pacman -S libpulse`.
2. **Hermes.** Add `ROBOT_MCP_TOKEN=<token>` to `~/.hermes/.env`. Merge
`omarchy/hermes-config-snippet.yaml` into `~/.hermes/config.yaml`, with `ROBOT_IP` replaced.
Then check the connection:
```bash
hermes mcp test robot
```
The tools will appear in Hermes as `mcp_robot_look`, `mcp_robot_turn_head` and so on.
3. **Launcher.** Install the wrapper, which switches your default audio to the robot for the
length of the session and restores it afterwards:
```bash
install -Dm755 omarchy/hermes-robot ~/.local/bin/hermes-robot
hermes-robot
```
Inside Hermes, `/voice on` starts the conversation. Set up the wake word as described in the
Hermes docs ("Wake Word") so you never need the keyboard.
1. **Network link:** `sudo host/omarchy/robot-usb-link.sh` gives your end of the link `10.55.0.1/30`,
matched by the gadget's fixed MAC, so any port and any machine works. The robot is `10.55.0.2`.
2. **Hermes:** the token is in `/etc/robot-mcp.env` on the Pi. Put it in `~/.hermes/.env` as
`ROBOT_MCP_TOKEN=`, and merge `host/omarchy/hermes-config-snippet.yaml` into
`~/.hermes/config.yaml`.
3. **Audio (optional, needs the speakers):** `host/omarchy/50-robot-tunnel.conf` makes the robot's
microphone and speaker appear as devices; `host/omarchy/hermes-robot` starts Hermes with them as
the defaults.
## First test
Say or type: *"Look around and tell me what's on my desk."* Hermes should call `look`, then
describe the scene.
```
ping 10.55.0.2
hermes mcp test robot # connects, finds the look tool
hermes chat -q "Use the look tool and tell me what you see."
```
**One thing to confirm early:** that the image actually reaches the model. The server returns a
standard MCP image block, and the test client sees it correctly. If Hermes answers as though it
saw nothing, check `~/.hermes/logs/agent.log`, and make sure your model supports vision. If
Hermes turns out to drop image blocks from MCP results, the fallback is a small change to
`look()` to save the JPEG and return a path for Hermes's own vision tool. Tell me if that happens.
The robot's face wakes when Hermes connects and falls asleep about 40 seconds after it leaves. A
look-and-describe turn takes a minute or so on local models: `look()` itself is instant. Hermes does not
give MCP images to the main model: it saves the photo to its image cache and the model calls
`vision_analyze` on it.
## Day to day
| State | Face | Comes from |
|---|---|---|
| sleeping | eyes closed, dim, slowly breathing | no agent connected (no heartbeat) |
| idle | blinks, smiles | the agent is connected |
| listening | mouth draws your voice, dimmer | the microphone (never wakes a sleeping face) |
| thinking | eyes trace looping curves | the agent |
| looking | squint, then a flash | `look()` |
| speaking | mouth draws the outgoing audio | the speaker |
- **Logs and control:** `journalctl -u robot -f` (the face logs every state change), `sudo systemctl
restart robot`.
- **Enrol yourself** so the eyes prefer you (numbers only are stored, never photos, in
`~/.local/share/hermes/faces.json`): on the Pi, with the robot running and after `source
~/robot-env.sh`, `python3 -m hermes_vision.people enroll YOURNAME`. `list` shows who is enrolled, `forget YOURNAME` removes you.
- **Update the code:** `rsync -a --delete --exclude __pycache__ ros/ hermes@<pi>:~/ros_ws/src/`, then
`colcon build --symlink-install` in `~/ros_ws`, then `sudo systemctl restart robot`.
- **Run a node by hand:** `source ~/robot-env.sh` first: it puts the libcamera fork and the right
OpenCV on the path, and keeps ROS local.
## Tuning and troubleshooting
- **Mic too quiet** (the level bar barely moves): on the Pi, run
`wpctl set-volume @DEFAULT_AUDIO_SOURCE@ 2.5`. Alternatively, lower `voice.silence_threshold`
in Hermes. A single mic is about 6 dB quieter than a pair, because the unused channel is
averaged in.
- **Hermes interrupts itself:** this is the robot hearing its own speaker. Raise
`voice.barge_in_threshold_multiplier`, point the speaker away from the mics, or enable
`pi/pipewire/60-echo-cancel.conf.optional` and uncomment the `pulse.device` lines in the
tunnel config.
- **Audio crackles:** this means weak Wi-Fi. Raise `pulse.latency` to 300–400. The Zero 2 W has
only 2.4 GHz Wi-Fi and a small antenna, so keep it within reasonable range of the router.
- **Robot devices vanish after the Pi reboots:** the tunnels reconnect every 5 s by themselves.
Start `hermes-robot` again once they reappear.
- **Head turns the wrong way:** set `ROBOT_PAN_INVERT=1` or `ROBOT_TILT_INVERT=1` in
`/etc/robot-mcp.env`, then restart `robot-mcp`.
- **Recordings are exactly silent:** the Audio switch is off (see above).
- **Microphone too noisy or too quiet:** WirePlumber resets the codec's capture gain to +30 dB. The
installer sets PipeWire's source volume to 63% (about +17 dB); change it with `pactl
set-source-volume @DEFAULT_SOURCE@ 63%`. Remember that PipeWire's scale is cubic.
- **The face flickers between listening and idle:** raise `hold_s` in `ros/hermes_audio/hermes_audio/levels.py`.
- **Stock libcamera crashes when capturing:** that is expected on this kernel. The installer builds the
Raspberry Pi fork into `/opt/libcamera-rpi` and builds `camera_ros` against it. Anything that uses
the camera must run after `source ~/robot-env.sh`.
- **Display shows noise or a band:** the 240×240 glass sits 80 rows into the controller's memory
(`y_offset: 80`), and `rotation: 270` is the mounted pose. Both are in `ros/hermes_bringup/config/robot.yaml`.
- **`GPIO busy` or a crash at start from systemd:** the display's GPIO backend needs a writable working
directory; the unit sets one.
- **The USB interface is unmanaged:** netplan must match `usb0` by *name*, not by MAC.
- **`ros2 node list` shows only some nodes:** use `ros2 node list --no-daemon --spin-time 6`.
- **The Pi's clock is wrong with Wi-Fi off:** there is no internet to sync from; it stays roughly right
but unsynchronised.
- **A one-off message to the face is ignored:** ROS drops a message published before the connection
matches. Republish it, as `hermes_mcp` does every second.
## Security
- The microphone stream (port 4713) and the camera server (port 8765) are open only to your
Omarchy machine's IP, via nftables on the Pi.
- The camera additionally requires the bearer token.
- Traffic is unencrypted on your LAN. If that matters to you, put both machines on Tailscale and
use the Pi's tailnet address in both configs.
- If your Omarchy machine's IP changes, re-run `install.sh` with the new one.
- The robot's ports are open **only over the USB link, only to `10.55.0.1`**: audio (4713), MCP (8765)
and a reserved webhook port (8766). Everything else of those is dropped on Wi-Fi.
- **ROS is blocked on every interface**, including USB. It has no authentication and carries the camera
and recognised names.
- MCP needs a bearer token, kept in `/etc/robot-mcp.env` (root and the service user only).
- **SSH is open on all interfaces** (key-only), by choice, for the mobile version.
- Face recognition is opt-in and stores only numbers. Hermes keeps each `look()` photo in its image
cache on your computer.
- Never power the robot from a device whose battery you care about.
## Files
```
pi/
install.sh one-shot Pi setup
robot_mcp_server.py MCP server: look, turn_head, head_position
requirements.txt MCP Python SDK 2.x
robot-mcp.service systemd unit
nftables-robot.nft port restriction
pipewire/50-network-server.conf exposes audio to the network
pipewire/60-echo-cancel.conf.optional
omarchy/
50-robot-tunnel.conf robot audio as local devices
hermes-config-snippet.yaml MCP and voice settings for Hermes
hermes-robot launcher that swaps default audio devices
```
| Path | What |
|---|---|
| `ros/` | The ROS 2 workspace: `hermes_face`, `hermes_vision`, `hermes_audio`, `hermes_mcp`, `hermes_bringup`. |
| `pi/` | Everything installed on the Pi: `install.sh` and its parts, services, firewall, overlays. |
| `host/` | Your computer's side: the USB link script, Hermes and PipeWire config, a platform check. |
| `enclosure/` | OpenSCAD model, STLs and the print guide. Built from estimates: measure before printing. |
| `docs/face-prototype.html` | The browser prototype the face was ported from. |
| `HANDOFF.md` | The original implementation brief. `CONTEXT.md` tracks where things really stand. |
omarchy/50-robot-tunnel.conf → host/omarchy/50-robot-tunnel.conf
View File
omarchy/hermes-config-snippet.yaml → host/omarchy/hermes-config-snippet.yaml
View File
omarchy/hermes-robot → host/omarchy/hermes-robot
+1 -1
View File
@@ -12,7 +12,7 @@ SOURCE=robot_mic
has() { pactl list short "$1" | awk '{print $2}' | grep -qxF "$2"; }
if ! has sinks "$SINK" || ! has sources "$SOURCE"; then
echo "hermes-robot: the robot's audio devices aren't present." >&2
echo " Is the robot powered and on Wi-Fi? Check with: pactl list short sources" >&2
echo " Is the robot powered and plugged in over USB (ping 10.55.0.2)? Check with: pactl list short sources" >&2
exit 1
fi
pi/install.sh
+152 -83
View File
@@ -1,96 +1,165 @@
#!/usr/bin/env bash
# Robot setup for Raspberry Pi OS Lite (64-bit) on a Pi Zero 2 W.
# Run as your normal user (not root) from inside this "pi" folder:
#!/bin/bash
# Set up the robot's Raspberry Pi 4 from a fresh Ubuntu Server 24.04 (arm64) image.
#
# ./install.sh <omarchy-machine-ip> [--servos]
# rsync -a ./ hermes@<pi>:~/hermes-bot/
# ssh hermes@<pi> 'cd hermes-bot && sudo ./pi/install.sh'
#
# Safe to re-run. Reboot afterwards so the I2S sound card overlay takes effect.
# Re-runnable: every step checks before it changes anything, and the heavy builds skip themselves if
# they are already done. Needs the internet (apt, git, pip, model downloads). The first run takes a
# while (the libcamera build alone is about 15 minutes). Reboot afterwards: it changes /boot/firmware/
# config.txt for the USB gadget and the audio codec.
#
# The user (default "hermes", set ROBOT_USER) must already exist: create it in cloud-init user-data.
# Order matters: the firewall goes in before PipeWire's network server, which listens on every
# interface and is protected only by that firewall.
set -euo pipefail
HOST_IP="${1:-}"
SERVOS=0
[[ "${2:-}" == "--servos" ]] && SERVOS=1
ROBOT_USER="${ROBOT_USER:-hermes}"
PI_DIR="$(cd "$(dirname "$0")" && pwd)"
REPO="$(dirname "$PI_DIR")"
# Raspberry Pi's fork of libcamera, at a commit known to capture on this kernel. Upstream libcamera,
# Ubuntu's 0.2.0 and ROS's 0.7.2 all crash in the Pi image-processing plugin when capturing.
LIBCAMERA_REF=6c1dd9d55573010f710c9e190a73e7e76f0d9432
CAMERA_ROS_TAG=0.7.0
MODEL_BASE=https://github.com/opencv/opencv_zoo/raw/main/models # a plain raw URL returns a Git LFS pointer
if [[ -z "$HOST_IP" || ! "$HOST_IP" =~ ^[0-9]{1,3}(\.[0-9]{1,3}){3}$ ]]; then
echo "usage: $0 <omarchy-machine-ipv4> [--servos]" >&2
exit 1
fi
if [[ $EUID -eq 0 ]]; then
echo "Run this as your normal user; it calls sudo where needed." >&2
exit 1
export DEBIAN_FRONTEND=noninteractive NEEDRESTART_MODE=a
REBOOT=0
[ "$(id -u)" = 0 ] || { echo "run with sudo" >&2; exit 1; }
HOME_DIR="$(getent passwd "$ROBOT_USER" | cut -d: -f6)"
[ -n "$HOME_DIR" ] || { echo "no user '$ROBOT_USER': create it in cloud-init user-data first" >&2; exit 1; }
grep -qs "Ubuntu 24.04" /etc/os-release || echo "warning: this is written for Ubuntu 24.04" >&2
step() { printf '\n==> %s\n' "$*"; }
as_user() { sudo -u "$ROBOT_USER" -H "$@"; }
cfg_sum() { sha256sum /boot/firmware/config.txt | cut -d' ' -f1; }
USER_UID="$(id -u "$ROBOT_USER")"
step "Base packages"
apt-get update -q
apt-get -y -q install git curl rsync software-properties-common openssl rfkill device-tree-compiler i2c-tools \
alsa-utils python3-numpy python3-pil python3-spidev python3-lgpio python3-rpi-lgpio python3-pip \
python3-venv pipewire pipewire-pulse wireplumber libspa-0.2-modules pulseaudio-utils \
meson ninja-build pkg-config python3-yaml python3-ply python3-jinja2 libyaml-dev libgnutls28-dev \
libssl-dev libudev-dev libevent-dev libdrm-dev libjpeg-dev libpng-dev libtiff-dev libboost-dev v4l-utils
step "Groups for hardware access"
for g in video audio i2c spi gpio dialout plugdev; do
getent group "$g" >/dev/null && usermod -aG "$g" "$ROBOT_USER" || echo " (no group $g on this image)"
done
step "ROS 2 Jazzy"
if ! dpkg -s ros-jazzy-ros-base >/dev/null 2>&1; then
add-apt-repository -y universe
. /etc/os-release
V="$(curl -s https://api.github.com/repos/ros-infrastructure/ros-apt-source/releases/latest | grep -F tag_name | awk -F\" '{print $4}')"
curl -fL -o /tmp/ros2-apt-source.deb \
"https://github.com/ros-infrastructure/ros-apt-source/releases/download/${V}/ros2-apt-source_${V}.${UBUNTU_CODENAME:-$VERSION_CODENAME}_all.deb"
dpkg -i /tmp/ros2-apt-source.deb
apt-get update -q
fi
# ros-jazzy-camera-ros is installed for its dependencies; the camera driver actually used is built below
apt-get -y -q install ros-jazzy-ros-base ros-dev-tools python3-colcon-common-extensions \
ros-jazzy-robot-state-publisher ros-jazzy-xacro ros-jazzy-camera-ros ros-jazzy-vision-msgs
HERE="$(cd "$(dirname "$0")" && pwd)"
ME="$(id -un)"
CONFIG_TXT=/boot/firmware/config.txt
[[ -f $CONFIG_TXT ]] || CONFIG_TXT=/boot/config.txt
echo "==> Installing packages"
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
python3-picamera2 python3-venv \
pipewire pipewire-pulse pipewire-alsa wireplumber libspa-0.2-modules \
pulseaudio-utils alsa-utils nftables openssl
echo "==> Enabling the I2S microphone/amplifier overlay in $CONFIG_TXT"
add_line() { grep -qxF "$1" "$CONFIG_TXT" || echo "$1" | sudo tee -a "$CONFIG_TXT" >/dev/null; }
add_line "dtparam=i2s=on"
add_line "dtoverlay=googlevoicehat-soundcard"
if [[ $SERVOS -eq 1 ]]; then
add_line "dtparam=i2c_arm=on"
fi
echo "==> Installing the MCP server to /opt/robot-mcp"
sudo mkdir -p /opt/robot-mcp
sudo chown "$ME:$ME" /opt/robot-mcp
cp "$HERE/robot_mcp_server.py" /opt/robot-mcp/
# --system-site-packages so the venv can see the apt-installed picamera2/libcamera
[[ -d /opt/robot-mcp/venv ]] || python3 -m venv --system-site-packages /opt/robot-mcp/venv
/opt/robot-mcp/venv/bin/pip install --upgrade pip >/dev/null
/opt/robot-mcp/venv/bin/pip install -r "$HERE/requirements.txt"
if [[ $SERVOS -eq 1 ]]; then
/opt/robot-mcp/venv/bin/pip install adafruit-circuitpython-servokit
fi
echo "==> Writing /etc/robot-mcp.env"
if ! sudo test -f /etc/robot-mcp.env; then
TOKEN="$(openssl rand -hex 32)"
sudo tee /etc/robot-mcp.env >/dev/null <<EOF
ROBOT_MCP_TOKEN=$TOKEN
ROBOT_MCP_PORT=8765
ROBOT_CAM_SIZE=1280x720
ROBOT_CAM_HFLIP=0
ROBOT_CAM_VFLIP=0
ROBOT_SERVOS=$SERVOS
EOF
sudo chown root:"$ME" /etc/robot-mcp.env
sudo chmod 640 /etc/robot-mcp.env
step "Raspberry Pi libcamera fork -> /opt/libcamera-rpi"
if [ -x /opt/libcamera-rpi/bin/cam ]; then
echo " already built"
else
sudo sed -i "s/^ROBOT_SERVOS=.*/ROBOT_SERVOS=$SERVOS/" /etc/robot-mcp.env
rm -rf /var/tmp/libcamera-rpi
git clone -q --no-checkout https://github.com/raspberrypi/libcamera.git /var/tmp/libcamera-rpi
git -C /var/tmp/libcamera-rpi fetch -q --depth 1 origin "$LIBCAMERA_REF"
git -C /var/tmp/libcamera-rpi checkout -q FETCH_HEAD
(cd /var/tmp/libcamera-rpi \
&& meson setup build --buildtype=release -Dprefix=/opt/libcamera-rpi -Dpipelines=rpi/vc4 -Dipas=rpi/vc4 \
-Dcam=enabled -Dqcam=disabled -Dgstreamer=disabled -Dpycamera=disabled -Dlc-compliance=disabled \
-Dtest=false -Ddocumentation=disabled -Dv4l2=false \
&& ninja -C build -j4 && ninja -C build install)
fi
echo "==> Installing the systemd service"
sed "s/__USER__/$ME/g" "$HERE/robot-mcp.service" | sudo tee /etc/systemd/system/robot-mcp.service >/dev/null
getent group i2c >/dev/null || sudo groupadd --system i2c
sudo systemctl daemon-reload
sudo systemctl enable robot-mcp.service
step "Python environment (~/venv: display, OpenCV 4.10 for YuNet, MCP SDK)"
# OpenCV's DNN needs 4.8+ for the YuNet model and apt has 4.6; numpy stays below 2 because the ROS
# Python packages are built against 1.x.
[ -d "$HOME_DIR/venv" ] || as_user python3 -m venv --system-site-packages "$HOME_DIR/venv"
as_user "$HOME_DIR/venv/bin/pip" install -q adafruit-circuitpython-rgb-display adafruit-blinka \
"opencv-python-headless==4.10.0.84" "numpy<2" "mcp>=2.3,<3"
echo "==> Configuring PipeWire to serve audio to $HOST_IP"
mkdir -p ~/.config/pipewire/pipewire-pulse.conf.d
cp "$HERE/pipewire/50-network-server.conf" ~/.config/pipewire/pipewire-pulse.conf.d/
# Run the audio stack at boot without anyone logging in.
sudo loginctl enable-linger "$ME"
systemctl --user enable pipewire.socket pipewire-pulse.socket wireplumber.service 2>/dev/null || true
step "Face models (~/models)"
as_user mkdir -p "$HOME_DIR/models"
fetch_model() { # name, path in the zoo
local dest="$HOME_DIR/models/$1"
if [ "$(stat -c %s "$dest" 2>/dev/null || echo 0)" -lt 100000 ]; then
as_user curl -fsSL -o "$dest" "$MODEL_BASE/$2/$1"
fi
[ "$(stat -c %s "$dest")" -gt 100000 ] || { echo " $1 looks like an LFS pointer, not the model" >&2; exit 1; }
}
fetch_model face_detection_yunet_2023mar.onnx face_detection_yunet
fetch_model face_recognition_sface_2021dec.onnx face_recognition_sface
echo "==> Restricting ports 4713 and 8765 to $HOST_IP"
sed "s/__HOST_IP__/$HOST_IP/g" "$HERE/nftables-robot.nft" | sudo tee /etc/nftables-robot.nft >/dev/null
grep -qxF 'include "/etc/nftables-robot.nft"' /etc/nftables.conf \
|| echo 'include "/etc/nftables-robot.nft"' | sudo tee -a /etc/nftables.conf >/dev/null
sudo systemctl enable nftables.service
sudo systemctl restart nftables.service
step "camera_ros $CAMERA_ROS_TAG, built against the libcamera fork (~/camera_ws)"
if [ -d "$HOME_DIR/camera_ws/install/camera_ros" ]; then
echo " already built"
else
as_user mkdir -p "$HOME_DIR/camera_ws/src"
[ -d "$HOME_DIR/camera_ws/src/camera_ros" ] || as_user git clone -q --depth 1 --branch "$CAMERA_ROS_TAG" \
https://github.com/christianrauch/camera_ros.git "$HOME_DIR/camera_ws/src/camera_ros"
as_user bash -c "source /opt/ros/jazzy/setup.bash && export PKG_CONFIG_PATH=/opt/libcamera-rpi/lib/aarch64-linux-gnu/pkgconfig \
&& cd '$HOME_DIR/camera_ws' && colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release"
fi
echo
echo "Done. Reboot now: sudo reboot"
echo
echo "Your MCP token (put this in ~/.hermes/.env on the Omarchy machine as ROBOT_MCP_TOKEN):"
sudo grep '^ROBOT_MCP_TOKEN=' /etc/robot-mcp.env | cut -d= -f2
step "Environment script and the robot's ROS workspace (~/ros_ws)"
install -o "$ROBOT_USER" -g "$ROBOT_USER" -m 644 "$PI_DIR/robot-env.sh" "$HOME_DIR/robot-env.sh"
as_user mkdir -p "$HOME_DIR/ros_ws/src"
rsync -a --delete --exclude __pycache__ "$REPO/ros/" "$HOME_DIR/ros_ws/src/"
chown -R "$ROBOT_USER:$ROBOT_USER" "$HOME_DIR/ros_ws/src"
as_user bash -c "source /opt/ros/jazzy/setup.bash && source '$HOME_DIR/camera_ws/install/setup.bash' \
&& cd '$HOME_DIR/ros_ws' && colcon build --symlink-install"
step "USB gadget (NCM Ethernet + ACM serial) and the audio codec overlay"
before="$(cfg_sum)"
bash "$PI_DIR/install-usb-gadget.sh"
bash "$PI_DIR/install-audio-overlay.sh"
[ "$before" = "$(cfg_sum)" ] || REBOOT=1
step "Firewall (before anything listens on the network)"
bash "$PI_DIR/install-firewall.sh"
step "PipeWire for $ROBOT_USER (audio server, reachable only over the USB link)"
loginctl enable-linger "$ROBOT_USER"
as_user mkdir -p "$HOME_DIR/.config/pipewire/pipewire-pulse.conf.d"
install -o "$ROBOT_USER" -g "$ROBOT_USER" -m 644 "$PI_DIR/pipewire/50-network-server.conf" \
"$HOME_DIR/.config/pipewire/pipewire-pulse.conf.d/50-network-server.conf"
for _ in $(seq 20); do [ -S "/run/user/$USER_UID/bus" ] && break; sleep 1; done
as_user env XDG_RUNTIME_DIR="/run/user/$USER_UID" systemctl --user enable --now \
pipewire.socket pipewire-pulse.socket wireplumber.service \
|| echo " (the user session is not up yet; PipeWire starts at the next boot)"
step "Audio codec mixer and microphone gain"
if aplay -l 2>/dev/null | grep -q wm8960; then
alsactl -f "$PI_DIR/wm8960_asound.state" restore wm8960soundcard && alsactl store wm8960soundcard
# WirePlumber resets the codec's capture gain to +30 dB when it takes the device; 63% on PipeWire's
# cubic scale is -12 dB from that, about +17 dB.
as_user env XDG_RUNTIME_DIR="/run/user/$USER_UID" pactl set-source-volume @DEFAULT_SOURCE@ 63% \
|| echo " (could not set the source volume: run pactl set-source-volume @DEFAULT_SOURCE@ 63% later)"
else
echo " no WM8960 card yet: reboot, then run this script again"
REBOOT=1
fi
step "Services: the robot (face, camera, vision, audio, MCP bridge), the CPU governor, trimmed boot"
bash "$PI_DIR/install-robot-service.sh"
bash "$PI_DIR/trim-boot.sh"
cat <<EOF
Done.$([ "$REBOOT" = 1 ] && echo " REBOOT NOW: the USB gadget and the audio overlay only take effect then.")
Next:
- Slide the HAT's "Audio" switch (right edge) to ON, or every recording is silent.
- Plug the USB-C data cable into your computer, then on it: sudo host/omarchy/robot-usb-link.sh
- Hermes needs the MCP token. It is not printed here; read it with:
ssh $ROBOT_USER@10.55.0.2 sudo cat /etc/robot-mcp.env
then see host/omarchy/hermes-config-snippet.yaml.
- Check with: hermes mcp test robot
EOF
pi/requirements.txt
-3
View File
@@ -1,3 +0,0 @@
# picamera2 and libcamera come from apt (python3-picamera2), not pip.
# Written against the MCP Python SDK 2.x (MCPServer); 1.x used the older FastMCP name.
mcp>=2.3,<3
pi/robot-mcp.service
-17
View File
@@ -1,17 +0,0 @@
[Unit]
Description=Robot MCP server (camera and head for Hermes Agent)
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=__USER__
Group=__USER__
SupplementaryGroups=video i2c
EnvironmentFile=/etc/robot-mcp.env
ExecStart=/opt/robot-mcp/venv/bin/python /opt/robot-mcp/robot_mcp_server.py
Restart=on-failure
RestartSec=3
[Install]
WantedBy=multi-user.target
pi/robot_mcp_server.py
-229
View File
@@ -1,229 +0,0 @@
#!/usr/bin/env python3
"""
Robot MCP server: runs on the Raspberry Pi and gives Hermes Agent the robot's eyes
(and, optionally, a neck).
Tools exposed
look() -> a JPEG still from the camera
turn_head(pan, tilt) -> aim the pan/tilt servos (only if ROBOT_SERVOS=1)
head_position() -> current pan/tilt (only if ROBOT_SERVOS=1)
Transport: MCP streamable HTTP at http://<pi>:<port>/mcp, guarded by a bearer token.
Audio is NOT handled here; it travels over PipeWire (see README).
Configuration is by environment variable (normally set in /etc/robot-mcp.env):
ROBOT_MCP_TOKEN required shared secret; Hermes sends "Authorization: Bearer <token>"
ROBOT_MCP_PORT 8765
ROBOT_MCP_BIND 0.0.0.0
ROBOT_CAM_SIZE 1280x720 still resolution handed to the model
ROBOT_CAM_QUALITY 85 JPEG quality
ROBOT_CAM_HFLIP 0 set 1 if the camera is mounted mirrored
ROBOT_CAM_VFLIP 0 set 1 (with HFLIP=1) if mounted upside down
ROBOT_SERVOS 0 set 1 to enable turn_head via a PCA9685 board
ROBOT_PAN_CHANNEL 0
ROBOT_TILT_CHANNEL 1
ROBOT_PAN_LIMIT 80 degrees either side of centre
ROBOT_TILT_LIMIT 45
ROBOT_PAN_INVERT 0 set 1 if "positive = left" comes out backwards
ROBOT_TILT_INVERT 0
"""
from __future__ import annotations
import hmac
import io
import logging
import os
import sys
import threading
import time
import uvicorn
from mcp.server.mcpserver import Image, MCPServer
from mcp.server.transport_security import TransportSecuritySettings
log = logging.getLogger("robot-mcp")
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s: %(message)s")
def env_flag(name: str, default: str = "0") -> bool:
return os.environ.get(name, default).strip().lower() in ("1", "true", "yes", "on")
def env_int(name: str, default: int) -> int:
return int(os.environ.get(name, str(default)))
TOKEN = os.environ.get("ROBOT_MCP_TOKEN", "").strip()
PORT = env_int("ROBOT_MCP_PORT", 8765)
BIND = os.environ.get("ROBOT_MCP_BIND", "0.0.0.0")
CAM_W, CAM_H = (int(v) for v in os.environ.get("ROBOT_CAM_SIZE", "1280x720").lower().split("x"))
CAM_QUALITY = env_int("ROBOT_CAM_QUALITY", 85)
# --------------------------------------------------------------------------- camera
class Camera:
"""Keeps Picamera2 running so a still is ready in well under a second."""
def __init__(self) -> None:
from libcamera import Transform # type: ignore[import-not-found]
from picamera2 import Picamera2 # type: ignore[import-not-found]
self._lock = threading.Lock()
self.cam = Picamera2()
config = self.cam.create_still_configuration(
main={"size": (CAM_W, CAM_H)},
transform=Transform(hflip=env_flag("ROBOT_CAM_HFLIP"), vflip=env_flag("ROBOT_CAM_VFLIP")),
buffer_count=2,
)
self.cam.configure(config)
self.cam.options["quality"] = CAM_QUALITY
self.has_autofocus = "AfMode" in self.cam.camera_controls
self.cam.start()
time.sleep(1.0) # let auto-exposure settle once at startup
log.info("camera ready at %dx%d (autofocus: %s)", CAM_W, CAM_H, self.has_autofocus)
def snapshot(self) -> bytes:
with self._lock:
if self.has_autofocus:
# Camera Module 3: run one focus sweep so whatever is in front is sharp.
self.cam.autofocus_cycle()
buf = io.BytesIO()
self.cam.capture_file(buf, format="jpeg")
return buf.getvalue()
# --------------------------------------------------------------------------- servos
class Head:
"""Pan/tilt on a PCA9685 servo board. Angles are degrees from centre."""
def __init__(self) -> None:
from adafruit_servokit import ServoKit # type: ignore[import-not-found]
self._lock = threading.Lock()
self.kit = ServoKit(channels=16)
self.pan_ch = env_int("ROBOT_PAN_CHANNEL", 0)
self.tilt_ch = env_int("ROBOT_TILT_CHANNEL", 1)
self.pan_limit = env_int("ROBOT_PAN_LIMIT", 80)
self.tilt_limit = env_int("ROBOT_TILT_LIMIT", 45)
self.pan_sign = -1 if env_flag("ROBOT_PAN_INVERT") else 1
self.tilt_sign = -1 if env_flag("ROBOT_TILT_INVERT") else 1
self.pan = 0.0
self.tilt = 0.0
self.move(0.0, 0.0)
log.info("servos ready (pan ch %d, tilt ch %d)", self.pan_ch, self.tilt_ch)
def move(self, pan: float, tilt: float) -> tuple[float, float]:
pan = max(-self.pan_limit, min(self.pan_limit, pan))
tilt = max(-self.tilt_limit, min(self.tilt_limit, tilt))
with self._lock:
self.kit.servo[self.pan_ch].angle = 90 + self.pan_sign * pan
self.kit.servo[self.tilt_ch].angle = 90 + self.tilt_sign * tilt
self.pan, self.tilt = pan, tilt
time.sleep(0.4) # let the servos arrive and stop jittering before any photo
return pan, tilt
# --------------------------------------------------------------------------- MCP server
mcp = MCPServer(
"robot",
instructions=(
"This server is a small physical robot on the user's desk. "
"Call look() whenever you need to see what is in front of it; the image is current. "
"If turn_head is available, aim first and then look."
),
)
camera: Camera | None = None
head: Head | None = None
@mcp.tool()
def look() -> Image:
"""Take a photograph with the robot's camera and return it. Use this to see the room,
the desk, the person, or anything the user asks you to look at."""
if camera is None:
raise RuntimeError("camera is not available on the robot")
t0 = time.monotonic()
data = camera.snapshot()
log.info("look(): %d bytes in %.2fs", len(data), time.monotonic() - t0)
return Image(data=data, format="jpeg")
def register_head_tools() -> None:
@mcp.tool()
def turn_head(pan: float = 0.0, tilt: float = 0.0) -> str:
"""Aim the robot's head (and so its camera). Angles are in degrees from straight ahead.
pan: positive turns to the robot's left, negative to its right.
tilt: positive looks up, negative looks down.
Values beyond the mechanical limits are clamped. Call look() afterwards to see."""
assert head is not None
p, t = head.move(pan, tilt)
return f"head now at pan {p:+.0f} degrees, tilt {t:+.0f} degrees"
@mcp.tool()
def head_position() -> str:
"""Report where the robot's head is currently pointing."""
assert head is not None
return f"pan {head.pan:+.0f} degrees, tilt {head.tilt:+.0f} degrees"
# --------------------------------------------------------------------------- auth wrapper
class BearerAuth:
"""Minimal ASGI middleware: reject any HTTP request without the right bearer token.
Lifespan events pass straight through so the MCP session manager starts properly."""
def __init__(self, app, token: str) -> None:
self.app = app
self.expected = f"Bearer {token}".encode()
async def __call__(self, scope, receive, send):
if scope["type"] == "http":
supplied = dict(scope.get("headers") or []).get(b"authorization", b"")
if not hmac.compare_digest(supplied, self.expected):
await send({"type": "http.response.start", "status": 401,
"headers": [(b"content-type", b"text/plain"),
(b"www-authenticate", b"Bearer")]})
await send({"type": "http.response.body", "body": b"unauthorized\n"})
return
await self.app(scope, receive, send)
def build_app():
app = mcp.streamable_http_app(
host=BIND,
# Bearer auth already stops drive-by requests; DNS-rebinding checks would only
# get in the way of reaching the Pi by IP or by robot.local.
transport_security=TransportSecuritySettings(enable_dns_rebinding_protection=False),
)
return BearerAuth(app, TOKEN)
def main() -> None:
global camera, head
if len(TOKEN) < 16:
sys.exit("ROBOT_MCP_TOKEN is missing or too short (generate one with: openssl rand -hex 32)")
try:
camera = Camera()
except Exception as exc: # keep serving servos even if the camera ribbon is loose
log.error("camera failed to start: %s", exc)
if env_flag("ROBOT_SERVOS"):
try:
head = Head()
register_head_tools()
except Exception as exc:
log.error("servos failed to start, turn_head disabled: %s", exc)
log.info("serving MCP on http://%s:%d/mcp", BIND, PORT)
uvicorn.run(build_app(), host=BIND, port=PORT, log_level="warning")
if __name__ == "__main__":
main()
tools/check-pi-platform.sh
-109
View File
@@ -1,109 +0,0 @@
#!/usr/bin/env bash
# check-pi-platform.sh: confirm, without the Pi, that Ubuntu 24.04's Raspberry Pi kernel ships the
# drivers and overlays the robot needs. It downloads the current linux-modules-*-raspi package for
# arm64 (about 50 MB) and inspects it. Nothing is installed; everything happens in a temp folder.
#
# Needs: curl, gzip, awk, ar (binutils), zstd, tar
# NixOS: nix-shell -p curl binutils zstd gnutar gawk --run ./check-pi-platform.sh
# Omarchy: sudo pacman -S --needed binutils zstd && ./check-pi-platform.sh
#
# Usage: ./check-pi-platform.sh [suite] (default suite: noble-updates)
# MIRROR=<url> overrides http://ports.ubuntu.com/ubuntu-ports
set -euo pipefail
SUITE="${1:-noble-updates}"
MIRROR="${MIRROR:-http://ports.ubuntu.com/ubuntu-ports}"
for tool in curl gzip awk ar zstd tar; do
command -v "$tool" >/dev/null || { echo "Missing tool: $tool (see the header of this script)" >&2; exit 2; }
done
work="$(mktemp -d)"
trap 'rm -rf "$work"' EXIT
cd "$work"
echo "==> Reading the arm64 package index for $SUITE"
curl -fsSL "$MIRROR/dists/$SUITE/main/binary-arm64/Packages.gz" | gzip -dc > Packages
# field <package> <Field>: the value of Field in that package's stanza (last stanza wins)
field() {
awk -v p="$1" -v f="$2" '
/^Package: / { inpkg = ($2 == p) }
inpkg && index($0, f ": ") == 1 { v = substr($0, length(f) + 3) }
END { if (v != "") print v }' Packages
}
meta_deps="$(field linux-image-raspi Depends)"
[[ -n "$meta_deps" ]] || { echo "linux-image-raspi not found in $SUITE" >&2; exit 1; }
abi="$(grep -oE 'linux-image-[0-9][^ ,(]*-raspi' <<<"$meta_deps" | head -1 | sed 's/^linux-image-//')"
modpkg="linux-modules-$abi"
debpath="$(field "$modpkg" Filename)"
[[ -n "$debpath" ]] || { echo "$modpkg not found in $SUITE" >&2; exit 1; }
echo "==> Current Pi kernel for Ubuntu 24.04: $abi"
echo "==> Downloading $modpkg"
curl -fsSL "$MIRROR/$debpath" -o modules.deb
member="$(ar t modules.deb | grep '^data\.tar' | head -1)"
case "$member" in
*.zst) decomp=(zstd -dc) ;;
*.xz) decomp=(xz -dc) ;;
*.gz) decomp=(gzip -dc) ;;
*.tar) decomp=(cat) ;;
*) echo "Unexpected package layout: $member" >&2; exit 1 ;;
esac
ar p modules.deb "$member" | "${decomp[@]}" | tar -t > files
ar p modules.deb "$member" | "${decomp[@]}" | tar -xO "./boot/config-$abi" > config 2>/dev/null || : > config
missing=0
# check <label> <module file name> <CONFIG_ key>
check() {
local label="$1" mod="$2" key="$3" state
if grep -qE "/${mod}\.ko(\.zst|\.xz|\.gz)?$" files; then
state="ok (module)"
elif grep -qx "${key}=y" config; then
state="ok (built in)"
else
state="MISSING"; missing=$((missing + 1))
fi
printf ' %-40s %s\n' "$label" "$state"
}
# overlay <name>
overlay() {
if grep -qE "/overlays/$1\.dtbo$" files; then
printf ' %-40s ok\n' "$1.dtbo"
else
printf ' %-40s MISSING\n' "$1.dtbo"; missing=$((missing + 1))
fi
}
echo
echo "Audio (BrainCraft HAT, WM8960 codec at I2C 0x1A)"
check "WM8960 codec driver" snd-soc-wm8960 CONFIG_SND_SOC_WM8960
check "Simple sound card" snd-soc-simple-card CONFIG_SND_SIMPLE_CARD
check "Pi I2S interface" snd-soc-bcm2835-i2s CONFIG_SND_BCM2835_SOC_I2S
overlay wm8960-soundcard
echo "Camera (Arducam UC-346, OV5647)"
check "OV5647 sensor driver" ov5647 CONFIG_VIDEO_OV5647
check "Pi camera receiver (Unicam)" bcm2835-unicam CONFIG_VIDEO_BCM2835_UNICAM
overlay ov5647
echo "Servos (hardware PWM on GPIO12/13)"
check "Pi PWM driver" pwm-bcm2835 CONFIG_PWM_BCM2835
overlay pwm-2chan
echo "Display (ST7789 on SPI)"
check "Pi SPI driver" spi-bcm2835 CONFIG_SPI_BCM2835
echo "USB link (gadget mode on the USB-C port)"
check "DWC2 USB controller" dwc2 CONFIG_USB_DWC2
check "Gadget framework (configfs)" libcomposite CONFIG_USB_LIBCOMPOSITE
check "NCM Ethernet function" usb_f_ncm CONFIG_USB_F_NCM
check "ECM Ethernet function (fallback)" usb_f_ecm CONFIG_USB_F_ECM
check "ACM serial function" usb_f_acm CONFIG_USB_F_ACM
overlay dwc2
echo
[[ -s config ]] || echo "Note: no kernel config in the package, so built-in drivers would show as MISSING."
if (( missing == 0 )); then
echo "Everything the robot needs is in Ubuntu's $abi kernel."
else
echo "$missing item(s) missing from $abi. Ubuntu 24.04 may not suit this robot as-is; see HANDOFF.md package 0a."
exit 1
fi