Add hermes_mcp, the bridge between Hermes and the ROS graph
The MCP server is now a ROS node and the only way in or out, per the owner's decision that ROS never leaves the robot. It serves MCP on the USB address only with a bearer token, keeps the stable tool name look() (which waits for the next camera frame, so it is never stale, and passes the camera's JPEG through), and registers turn_head/head_position only when a head is fitted, which today it is not. While an authenticated client is connected it republishes the agent heartbeat on /hermes/state, so the face wakes on connect and sleeps about 40 s after the client leaves; look() also makes the face squint. The token is generated on the Pi into /etc/robot-mcp.env (root:hermes, 640) and never printed. Verified end to end from talos with a real MCP client over the USB link: tools listed, a 157 KB JPEG back in 0.08 s, a wrong token refused, the Wi-Fi path dropped by the firewall. Host config snippets now point at 10.55.0.2. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
This commit is contained in:
+27
-7
@@ -4,8 +4,10 @@ Last updated 2026-10-04. `HANDOFF.md` is the brief; this file tracks where the w
|
|||||||
|
|
||||||
## Hardware on hand
|
## Hardware on hand
|
||||||
|
|
||||||
Raspberry Pi 4 (8 GB), BrainCraft HAT, Arducam UC-346 camera. Not yet: servos, USB-C splitter,
|
Raspberry Pi 4 (8 GB), BrainCraft HAT, Arducam UC-346 camera, and nothing else: **no servos, no
|
||||||
printed enclosure, speakers. Work until those arrive is limited to what the Pi, HAT and camera can do.
|
fan, no speakers**, no USB-C splitter, no printed enclosure (the camera hangs loose on its ribbon and is
|
||||||
|
held by hand). The Pi runs at about 62 C open-air with everything on and no fan; an enclosure will be
|
||||||
|
hotter, so fit the fan before enclosing it. Playback and the servos are untested. Work until those arrive is limited to what the Pi, HAT and camera can do.
|
||||||
|
|
||||||
## The Pi (`hermes-bot`)
|
## The Pi (`hermes-bot`)
|
||||||
|
|
||||||
@@ -205,6 +207,26 @@ printed enclosure, speakers. Work until those arrive is limited to what the Pi,
|
|||||||
49 face tests, 10 audio tests. **Not yet verified:** the audio node as a service (it was run as a
|
49 face tests, 10 audio tests. **Not yet verified:** the audio node as a service (it was run as a
|
||||||
transient unit).
|
transient unit).
|
||||||
|
|
||||||
|
## The MCP bridge (`ros/hermes_mcp`)
|
||||||
|
|
||||||
|
The only way in or out of the ROS graph. Serves MCP (SDK 2.3, streamable HTTP) on
|
||||||
|
`http://10.55.0.2:8765/mcp`, the USB address only (retrying the bind if the link is not up yet), with
|
||||||
|
a bearer token. **Tool names are the stable contract with Hermes**: `look()`, and `turn_head` /
|
||||||
|
`head_position` only when a head is fitted. With no servos only `look` is registered, by design
|
||||||
|
(`head: false`; the node refuses `head: true` until `hermes_head` exists). `look()` waits for the next
|
||||||
|
camera frame taken after the call (never a stale one) and passes the camera's JPEG through (1296x972,
|
||||||
|
~155 KB); it also publishes `looking` on `/hermes/state` so the face squints and flashes.
|
||||||
|
While an authenticated client is connected (in flight, or heard within `connected_grace_s` = 30) it
|
||||||
|
republishes `idle` on `/hermes/state` once a second: the agent heartbeat that wakes the face. Verified
|
||||||
|
end to end from talos over the USB link with a real MCP client: connects, lists `look`, `look()`
|
||||||
|
returns a JPEG in 0.08 s, a wrong token is refused, the Wi-Fi path is dropped by the firewall, and
|
||||||
|
the face went `looking` -> `idle` on connect and `agent gone, sleeping` about 40 s after the client
|
||||||
|
left. The token is in `/etc/robot-mcp.env` (root:hermes, mode 640), generated by
|
||||||
|
`pi/install-robot-service.sh` and never printed; read it with `sudo cat`. The MCP SDK is in `~/venv`
|
||||||
|
(`pip install "mcp>=2"`). The old `pi/robot_mcp_server.py`, `pi/robot-mcp.service` and `pi/install.sh`
|
||||||
|
are superseded (package 5 rewrites the installer). **Not yet verified:** a real Hermes client, and
|
||||||
|
whether Hermes passes MCP image content to the model (HANDOFF package 6).
|
||||||
|
|
||||||
## Face recognition (opt-in)
|
## Face recognition (opt-in)
|
||||||
|
|
||||||
`hermes_vision.recognize` / `people`: SFace embeddings on YuNet's landmarks. Nothing is stored until
|
`hermes_vision.recognize` / `people`: SFace embeddings on YuNet's landmarks. Nothing is stored until
|
||||||
@@ -223,11 +245,9 @@ unrelated photo scored 0.16. **Nobody is enrolled yet**; the owner enrols themse
|
|||||||
`host/nixos/`. A new Omarchy host with a GPU (runs the local
|
`host/nixos/`. A new Omarchy host with a GPU (runs the local
|
||||||
model "ornith") arrives this week: it needs only `host/omarchy/robot-usb-link.sh` and its SSH key
|
model "ornith") arrives this week: it needs only `host/omarchy/robot-usb-link.sh` and its SSH key
|
||||||
added to the Pi.
|
added to the Pi.
|
||||||
2. `hermes_mcp`: the MCP server as an `rclpy` node (stable tool names; `look()` returns the latest
|
2. `hermes_head` and the servos when they arrive (then `head: true`), echo cancel and the `robot_mic`
|
||||||
camera image, `turn_head()` calls the head action), the only bridge to the outside; it must
|
naming once speakers arrive, a fan before any enclosure, the Wi-Fi-off boot test, a real Hermes
|
||||||
republish `/hermes/state` about once a second as the agent heartbeat. Then `hermes_head` and
|
client against `hermes_mcp`, and the rest of `HANDOFF.md`.
|
||||||
the servos when they arrive, echo cancel and the `robot_mic` naming once speakers arrive, the
|
|
||||||
Wi-Fi-off boot test, and the rest of `HANDOFF.md`.
|
|
||||||
3. Confirm the retuned listening detector live (the flicker fix is unit-tested only); a second person
|
3. Confirm the retuned listening detector live (the flicker fix is unit-tested only); a second person
|
||||||
in frame for the recognition test (not possible on 2026-10-04).
|
in frame for the recognition test (not possible on 2026-10-04).
|
||||||
|
|
||||||
|
|||||||
@@ -1,8 +1,8 @@
|
|||||||
# ~/.config/pipewire/pipewire.conf.d/50-robot-tunnel.conf (on Omarchy)
|
# ~/.config/pipewire/pipewire.conf.d/50-robot-tunnel.conf (on Omarchy)
|
||||||
#
|
#
|
||||||
# Makes the robot's speaker and microphone appear here as ordinary audio devices,
|
# Makes the robot's speaker and microphone appear here as ordinary audio devices,
|
||||||
# "Robot speaker" and "Robot microphone". Replace ROBOT_IP with the Pi's address
|
# "Robot speaker" and "Robot microphone". The robot is 10.55.0.2 on the USB link, a fixed address;
|
||||||
# (give it a DHCP reservation on your router so it never changes).
|
# the robot's firewall accepts port 4713 only over that link, from 10.55.0.1.
|
||||||
#
|
#
|
||||||
# Apply with: systemctl --user restart pipewire pipewire-pulse wireplumber
|
# Apply with: systemctl --user restart pipewire pipewire-pulse wireplumber
|
||||||
|
|
||||||
@@ -10,7 +10,7 @@ context.modules = [
|
|||||||
{ name = libpipewire-module-pulse-tunnel
|
{ name = libpipewire-module-pulse-tunnel
|
||||||
args = {
|
args = {
|
||||||
tunnel.mode = sink
|
tunnel.mode = sink
|
||||||
pulse.server.address = "tcp:ROBOT_IP:4713"
|
pulse.server.address = "tcp:10.55.0.2:4713"
|
||||||
#pulse.device = "robot_ec_sink" # only with the Pi's optional echo-cancel config
|
#pulse.device = "robot_ec_sink" # only with the Pi's optional echo-cancel config
|
||||||
pulse.latency = 200 # ms; lower is snappier but may crackle on weak Wi-Fi
|
pulse.latency = 200 # ms; lower is snappier but may crackle on weak Wi-Fi
|
||||||
reconnect.interval.ms = 5000 # keep retrying while the robot is off
|
reconnect.interval.ms = 5000 # keep retrying while the robot is off
|
||||||
@@ -23,7 +23,7 @@ context.modules = [
|
|||||||
{ name = libpipewire-module-pulse-tunnel
|
{ name = libpipewire-module-pulse-tunnel
|
||||||
args = {
|
args = {
|
||||||
tunnel.mode = source
|
tunnel.mode = source
|
||||||
pulse.server.address = "tcp:ROBOT_IP:4713"
|
pulse.server.address = "tcp:10.55.0.2:4713"
|
||||||
#pulse.device = "robot_ec_source" # only with the Pi's optional echo-cancel config
|
#pulse.device = "robot_ec_source" # only with the Pi's optional echo-cancel config
|
||||||
pulse.latency = 200
|
pulse.latency = 200
|
||||||
reconnect.interval.ms = 5000
|
reconnect.interval.ms = 5000
|
||||||
|
|||||||
@@ -1,10 +1,14 @@
|
|||||||
# Merge these into ~/.hermes/config.yaml on Omarchy.
|
# Merge these into ~/.hermes/config.yaml on Omarchy.
|
||||||
# Put the token from the Pi's install.sh output in ~/.hermes/.env as:
|
# The robot is at 10.55.0.2 on the USB link (the host is 10.55.0.1; see host/omarchy/robot-usb-link.sh).
|
||||||
# ROBOT_MCP_TOKEN=<64 hex characters>
|
# Put the token in ~/.hermes/.env as: ROBOT_MCP_TOKEN=<64 hex characters>
|
||||||
|
# It lives only on the Pi, in /etc/robot-mcp.env (read it with: ssh hermes@10.55.0.2 sudo cat /etc/robot-mcp.env).
|
||||||
|
# The robot's firewall accepts this port only over the USB link, from 10.55.0.1.
|
||||||
|
# While Hermes holds a connection the robot's face is awake; when it stops, the face falls asleep after
|
||||||
|
# about 40 seconds. A robot with no head fitted offers only look(); turn_head appears when servos do.
|
||||||
|
|
||||||
mcp_servers:
|
mcp_servers:
|
||||||
robot:
|
robot:
|
||||||
url: "http://ROBOT_IP:8765/mcp"
|
url: "http://10.55.0.2:8765/mcp"
|
||||||
headers:
|
headers:
|
||||||
Authorization: "Bearer ${ROBOT_MCP_TOKEN}"
|
Authorization: "Bearer ${ROBOT_MCP_TOKEN}"
|
||||||
connect_timeout: 15
|
connect_timeout: 15
|
||||||
|
|||||||
@@ -1,10 +1,17 @@
|
|||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
# One unit runs the whole robot (face, camera, vision, audio) via the hermes_bringup launch file, so
|
# One unit runs the whole robot (face, camera, vision, audio, MCP bridge) via the hermes_bringup launch file, so
|
||||||
# the face is up whenever the robot is powered, and everything is ROS. Each node is its own process
|
# the face is up whenever the robot is powered, and everything is ROS. Each node is its own process
|
||||||
# and is respawned by the launch file, so one crash cannot take the others down.
|
# and is respawned by the launch file, so one crash cannot take the others down.
|
||||||
# Replaces the earlier standalone robot-face.service. Run as root from this directory. Re-runnable.
|
# Replaces the earlier standalone robot-face.service. Run as root from this directory. Re-runnable.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
cd "$(dirname "$0")"
|
cd "$(dirname "$0")"
|
||||||
|
# The MCP bridge's bearer token: created once, readable only by root and the service user, and never
|
||||||
|
# printed (read it yourself with `sudo cat /etc/robot-mcp.env` when configuring the host).
|
||||||
|
if [ ! -s /etc/robot-mcp.env ]; then
|
||||||
|
(umask 077; printf 'ROBOT_MCP_TOKEN=%s\n' "$(openssl rand -hex 32)" > /etc/robot-mcp.env)
|
||||||
|
fi
|
||||||
|
chown root:hermes /etc/robot-mcp.env
|
||||||
|
chmod 640 /etc/robot-mcp.env
|
||||||
install -m 644 robot-cpu-performance.service robot.service /etc/systemd/system/
|
install -m 644 robot-cpu-performance.service robot.service /etc/systemd/system/
|
||||||
systemctl disable --now robot-face.service 2>/dev/null || true
|
systemctl disable --now robot-face.service 2>/dev/null || true
|
||||||
rm -f /etc/systemd/system/robot-face.service
|
rm -f /etc/systemd/system/robot-face.service
|
||||||
|
|||||||
+3
-1
@@ -1,5 +1,5 @@
|
|||||||
[Unit]
|
[Unit]
|
||||||
Description=Robot (hermes_bringup): face, camera, face detection, audio levels
|
Description=Robot (hermes_bringup): face, camera, face detection, audio levels, MCP bridge
|
||||||
# Nothing here needs the user session: audio_state retries until the user's PipeWire is up.
|
# Nothing here needs the user session: audio_state retries until the user's PipeWire is up.
|
||||||
After=local-fs.target robot-cpu-performance.service robot-firewall.service
|
After=local-fs.target robot-cpu-performance.service robot-firewall.service
|
||||||
|
|
||||||
@@ -9,6 +9,8 @@ User=hermes
|
|||||||
# be writable: systemd's default, /, is not.
|
# be writable: systemd's default, /, is not.
|
||||||
RuntimeDirectory=robot
|
RuntimeDirectory=robot
|
||||||
WorkingDirectory=/run/robot
|
WorkingDirectory=/run/robot
|
||||||
|
# hermes_mcp reads its bearer token from here (root:hermes, mode 640); it is never in a parameter
|
||||||
|
EnvironmentFile=-/etc/robot-mcp.env
|
||||||
# audio_state reaches the user's PipeWire through this
|
# audio_state reaches the user's PipeWire through this
|
||||||
Environment=XDG_RUNTIME_DIR=/run/user/1000
|
Environment=XDG_RUNTIME_DIR=/run/user/1000
|
||||||
ExecStart=/bin/bash -c 'source /home/hermes/robot-env.sh && exec ros2 launch hermes_bringup robot.launch.py'
|
ExecStart=/bin/bash -c 'source /home/hermes/robot-env.sh && exec ros2 launch hermes_bringup robot.launch.py'
|
||||||
|
|||||||
@@ -30,3 +30,11 @@ hermes_face_detector:
|
|||||||
hermes_audio_state:
|
hermes_audio_state:
|
||||||
ros__parameters:
|
ros__parameters:
|
||||||
rate_hz: 30.0
|
rate_hz: 30.0
|
||||||
|
|
||||||
|
hermes_mcp:
|
||||||
|
ros__parameters:
|
||||||
|
bind: 10.55.0.2 # the USB link; the firewall accepts 8765 only from 10.55.0.1 on usb0
|
||||||
|
port: 8765
|
||||||
|
head: false # no servos fitted yet: turn_head and head_position stay unregistered
|
||||||
|
connected_grace_s: 30.0 # an authenticated client heard within this long keeps the heartbeat going
|
||||||
|
look_timeout_s: 3.0
|
||||||
|
|||||||
@@ -1,12 +1,12 @@
|
|||||||
"""Bring up the robot: camera, face, face detection, audio levels.
|
"""Bring up the robot: camera, face, face detection, audio levels, and the MCP bridge.
|
||||||
|
|
||||||
ros2 launch hermes_bringup robot.launch.py [camera:=false] [face:=false] [vision:=false] [audio:=false]
|
ros2 launch hermes_bringup robot.launch.py [camera:=false] [face:=false] [vision:=false] [audio:=false] [mcp:=false]
|
||||||
[start_delay:=6.0]
|
[start_delay:=6.0]
|
||||||
|
|
||||||
Every node is its own process and is respawned if it exits, so one crash cannot take the others
|
Every node is its own process and is respawned if it exits, so one crash cannot take the others
|
||||||
down: the face keeps running when the camera does not. Parameters are in config/robot.yaml.
|
down: the face keeps running when the camera does not. Parameters are in config/robot.yaml.
|
||||||
|
|
||||||
The face starts at once; the camera, vision and audio nodes wait start_delay seconds, so the four
|
The face starts at once; the camera, vision, audio and MCP nodes wait start_delay seconds, so the four
|
||||||
Python processes do not fight over the CPU while the face imports and draws its first frame.
|
Python processes do not fight over the CPU while the face imports and draws its first frame.
|
||||||
"""
|
"""
|
||||||
import os
|
import os
|
||||||
@@ -24,10 +24,11 @@ NODES = [
|
|||||||
('face', 'hermes_face', 'face_node', 'hermes_face'),
|
('face', 'hermes_face', 'face_node', 'hermes_face'),
|
||||||
('vision', 'hermes_vision', 'face_detector', 'hermes_face_detector'),
|
('vision', 'hermes_vision', 'face_detector', 'hermes_face_detector'),
|
||||||
('audio', 'hermes_audio', 'audio_state', 'hermes_audio_state'),
|
('audio', 'hermes_audio', 'audio_state', 'hermes_audio_state'),
|
||||||
|
('mcp', 'hermes_mcp', 'mcp_server', 'hermes_mcp'),
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
DELAYED = {'camera', 'vision', 'audio'}
|
DELAYED = {'camera', 'vision', 'audio', 'mcp'}
|
||||||
|
|
||||||
|
|
||||||
def generate_launch_description():
|
def generate_launch_description():
|
||||||
@@ -35,7 +36,7 @@ def generate_launch_description():
|
|||||||
actions = [DeclareLaunchArgument(arg, default_value='true', description=f'run the {name} node')
|
actions = [DeclareLaunchArgument(arg, default_value='true', description=f'run the {name} node')
|
||||||
for arg, _, _, name in NODES]
|
for arg, _, _, name in NODES]
|
||||||
actions.append(DeclareLaunchArgument('start_delay', default_value='6.0',
|
actions.append(DeclareLaunchArgument('start_delay', default_value='6.0',
|
||||||
description='seconds the camera, vision and audio wait for the face'))
|
description='seconds the other nodes wait for the face'))
|
||||||
for arg, package, executable, name in NODES:
|
for arg, package, executable, name in NODES:
|
||||||
node = Node(package=package, executable=executable, name=name, output='screen',
|
node = Node(package=package, executable=executable, name=name, output='screen',
|
||||||
parameters=[params], respawn=True, respawn_delay=2.0,
|
parameters=[params], respawn=True, respawn_delay=2.0,
|
||||||
|
|||||||
@@ -14,6 +14,7 @@
|
|||||||
<exec_depend>hermes_face</exec_depend>
|
<exec_depend>hermes_face</exec_depend>
|
||||||
<exec_depend>hermes_vision</exec_depend>
|
<exec_depend>hermes_vision</exec_depend>
|
||||||
<exec_depend>hermes_audio</exec_depend>
|
<exec_depend>hermes_audio</exec_depend>
|
||||||
|
<exec_depend>hermes_mcp</exec_depend>
|
||||||
|
|
||||||
<test_depend>python3-pytest</test_depend>
|
<test_depend>python3-pytest</test_depend>
|
||||||
|
|
||||||
|
|||||||
@@ -19,8 +19,8 @@ def test_launch_description_has_one_respawning_process_per_node():
|
|||||||
ld = mod.generate_launch_description()
|
ld = mod.generate_launch_description()
|
||||||
nodes = [e for e in ld.entities if isinstance(e, Node)]
|
nodes = [e for e in ld.entities if isinstance(e, Node)]
|
||||||
delayed = [n for t in ld.entities if isinstance(t, TimerAction) for n in t.actions if isinstance(n, Node)]
|
delayed = [n for t in ld.entities if isinstance(t, TimerAction) for n in t.actions if isinstance(n, Node)]
|
||||||
assert len(nodes) + len(delayed) == len(mod.NODES) == 4
|
assert len(nodes) + len(delayed) == len(mod.NODES) == 5
|
||||||
assert len(nodes) == 1 and 'face' in mod.NODES[1] and len(delayed) == 3 # the face starts at once
|
assert len(nodes) == 1 and 'face' in mod.NODES[1] and len(delayed) == 4 # the face starts at once
|
||||||
# respawn lives on a private attribute that differs between ROS releases, so check the source
|
# respawn lives on a private attribute that differs between ROS releases, so check the source
|
||||||
src = open(os.path.join(ROOT, 'launch', 'robot.launch.py')).read()
|
src = open(os.path.join(ROOT, 'launch', 'robot.launch.py')).read()
|
||||||
assert 'respawn=True' in src and 'respawn_delay' in src
|
assert 'respawn=True' in src and 'respawn_delay' in src
|
||||||
@@ -28,7 +28,13 @@ def test_launch_description_has_one_respawning_process_per_node():
|
|||||||
|
|
||||||
def test_every_node_in_the_launch_file_has_parameters_and_the_face_is_mounted_portrait():
|
def test_every_node_in_the_launch_file_has_parameters_and_the_face_is_mounted_portrait():
|
||||||
cfg = yaml.safe_load(open(os.path.join(ROOT, 'config', 'robot.yaml')))
|
cfg = yaml.safe_load(open(os.path.join(ROOT, 'config', 'robot.yaml')))
|
||||||
for name in ('camera', 'hermes_face', 'hermes_face_detector', 'hermes_audio_state'):
|
for name in ('camera', 'hermes_face', 'hermes_face_detector', 'hermes_audio_state', 'hermes_mcp'):
|
||||||
assert 'ros__parameters' in cfg[name], name
|
assert 'ros__parameters' in cfg[name], name
|
||||||
assert cfg['hermes_face']['ros__parameters']['rotation'] == 270
|
assert cfg['hermes_face']['ros__parameters']['rotation'] == 270
|
||||||
assert cfg['camera']['ros__parameters']['FrameDurationLimits'] == [66666, 66666]
|
assert cfg['camera']['ros__parameters']['FrameDurationLimits'] == [66666, 66666]
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_mcp_bridge_binds_the_usb_link_and_has_no_head_yet():
|
||||||
|
cfg = yaml.safe_load(open(os.path.join(ROOT, 'config', 'robot.yaml')))['hermes_mcp']['ros__parameters']
|
||||||
|
assert cfg['bind'] == '10.55.0.2' and cfg['port'] == 8765 and cfg['head'] is False
|
||||||
|
assert 'token' not in ' '.join(cfg) # the secret lives in an env file, not here
|
||||||
|
|||||||
@@ -0,0 +1,150 @@
|
|||||||
|
"""The MCP side of the bridge: the tools, bearer-token auth and connection tracking. No ROS imports,
|
||||||
|
so it is testable with a stub camera.
|
||||||
|
|
||||||
|
MCP tool names are the stable contract with Hermes and must not change when the internals do:
|
||||||
|
look() -> the next fresh JPEG from the camera
|
||||||
|
turn_head(pan, tilt) -> only registered if a head backend is given (servos fitted)
|
||||||
|
head_position() -> likewise
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hmac
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
from typing import Callable, Optional, Protocol
|
||||||
|
|
||||||
|
from mcp.server.mcpserver import Image, MCPServer
|
||||||
|
from mcp.server.transport_security import TransportSecuritySettings
|
||||||
|
|
||||||
|
|
||||||
|
class LatestFrame:
|
||||||
|
"""The newest camera frame, and a way to wait for one taken after a given moment: look() must
|
||||||
|
return what the camera sees now, not a frame that was already sitting there."""
|
||||||
|
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self._cv = threading.Condition()
|
||||||
|
self._jpeg: Optional[bytes] = None
|
||||||
|
self._t = -1.0
|
||||||
|
|
||||||
|
def put(self, jpeg: bytes, t: Optional[float] = None) -> None:
|
||||||
|
with self._cv:
|
||||||
|
self._jpeg, self._t = jpeg, time.monotonic() if t is None else t
|
||||||
|
self._cv.notify_all()
|
||||||
|
|
||||||
|
def wait_newer(self, t0: float, timeout: float) -> Optional[bytes]:
|
||||||
|
"""The first frame received after t0, or None if none arrives within timeout seconds."""
|
||||||
|
deadline = time.monotonic() + timeout
|
||||||
|
with self._cv:
|
||||||
|
while self._jpeg is None or self._t <= t0:
|
||||||
|
remaining = deadline - time.monotonic()
|
||||||
|
if remaining <= 0:
|
||||||
|
return None
|
||||||
|
self._cv.wait(remaining)
|
||||||
|
return self._jpeg
|
||||||
|
|
||||||
|
|
||||||
|
class Head(Protocol):
|
||||||
|
def move(self, pan: float, tilt: float) -> tuple[float, float]: ...
|
||||||
|
def position(self) -> tuple[float, float]: ...
|
||||||
|
|
||||||
|
|
||||||
|
def build_mcp(frames: LatestFrame, announce_look: Callable[[], None] = lambda: None,
|
||||||
|
head: Optional[Head] = None, look_timeout_s: float = 3.0) -> MCPServer:
|
||||||
|
mcp = MCPServer(
|
||||||
|
"robot",
|
||||||
|
instructions=(
|
||||||
|
"This server is a small physical robot on the user's desk. "
|
||||||
|
"Call look() whenever you need to see what is in front of it; the image is current. "
|
||||||
|
"If turn_head is available, aim first and then look."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
@mcp.tool()
|
||||||
|
def look() -> Image:
|
||||||
|
"""Take a photograph with the robot's camera and return it. Use this to see the room,
|
||||||
|
the desk, the person, or anything the user asks you to look at."""
|
||||||
|
t0 = time.monotonic()
|
||||||
|
announce_look() # the face squints and flashes
|
||||||
|
jpeg = frames.wait_newer(t0, look_timeout_s)
|
||||||
|
if jpeg is None:
|
||||||
|
raise RuntimeError("the robot's camera is not producing images")
|
||||||
|
return Image(data=jpeg, format="jpeg")
|
||||||
|
|
||||||
|
if head is not None:
|
||||||
|
@mcp.tool()
|
||||||
|
def turn_head(pan: float = 0.0, tilt: float = 0.0) -> str:
|
||||||
|
"""Aim the robot's head (and so its camera). Angles are in degrees from straight ahead.
|
||||||
|
pan: positive turns to the robot's left, negative to its right.
|
||||||
|
tilt: positive looks up, negative looks down.
|
||||||
|
Values beyond the mechanical limits are clamped. Call look() afterwards to see."""
|
||||||
|
p, t = head.move(pan, tilt)
|
||||||
|
return f"head now at pan {p:+.0f} degrees, tilt {t:+.0f} degrees"
|
||||||
|
|
||||||
|
@mcp.tool()
|
||||||
|
def head_position() -> str:
|
||||||
|
"""Report where the robot's head is currently pointing."""
|
||||||
|
p, t = head.position()
|
||||||
|
return f"pan {p:+.0f} degrees, tilt {t:+.0f} degrees"
|
||||||
|
|
||||||
|
return mcp
|
||||||
|
|
||||||
|
|
||||||
|
class BearerAuth:
|
||||||
|
"""ASGI middleware: reject any HTTP request without the right bearer token. Lifespan events pass
|
||||||
|
straight through so the MCP session manager starts properly."""
|
||||||
|
|
||||||
|
def __init__(self, app, token: str) -> None:
|
||||||
|
self.app = app
|
||||||
|
self.expected = f"Bearer {token}".encode()
|
||||||
|
|
||||||
|
async def __call__(self, scope, receive, send):
|
||||||
|
if scope["type"] == "http":
|
||||||
|
supplied = dict(scope.get("headers") or []).get(b"authorization", b"")
|
||||||
|
if not hmac.compare_digest(supplied, self.expected):
|
||||||
|
await send({"type": "http.response.start", "status": 401,
|
||||||
|
"headers": [(b"content-type", b"text/plain"), (b"www-authenticate", b"Bearer")]})
|
||||||
|
await send({"type": "http.response.body", "body": b"unauthorized\n"})
|
||||||
|
return
|
||||||
|
await self.app(scope, receive, send)
|
||||||
|
|
||||||
|
|
||||||
|
class Activity:
|
||||||
|
"""ASGI middleware, placed inside BearerAuth so that only authenticated requests count: tracks
|
||||||
|
in-flight requests (a held-open event stream stays in flight) and when the last one finished.
|
||||||
|
A client that has been heard from recently is "connected"; that is the agent heartbeat."""
|
||||||
|
|
||||||
|
def __init__(self, app) -> None:
|
||||||
|
self.app = app
|
||||||
|
self._lock = threading.Lock()
|
||||||
|
self.inflight = 0
|
||||||
|
self.last = -1e9
|
||||||
|
|
||||||
|
async def __call__(self, scope, receive, send):
|
||||||
|
if scope["type"] != "http":
|
||||||
|
await self.app(scope, receive, send)
|
||||||
|
return
|
||||||
|
with self._lock:
|
||||||
|
self.inflight += 1
|
||||||
|
self.last = time.monotonic()
|
||||||
|
try:
|
||||||
|
await self.app(scope, receive, send)
|
||||||
|
finally:
|
||||||
|
with self._lock:
|
||||||
|
self.inflight -= 1
|
||||||
|
self.last = time.monotonic()
|
||||||
|
|
||||||
|
def connected(self, now: float, grace_s: float) -> bool:
|
||||||
|
with self._lock:
|
||||||
|
return self.inflight > 0 or now - self.last <= grace_s
|
||||||
|
|
||||||
|
|
||||||
|
def build_app(mcp: MCPServer, token: str, host: str):
|
||||||
|
"""The ASGI app: token check outside, activity tracking inside. Returns (app, activity)."""
|
||||||
|
inner = mcp.streamable_http_app(
|
||||||
|
host=host,
|
||||||
|
# Bearer auth already stops drive-by requests; DNS-rebinding checks would only get in the way
|
||||||
|
# of reaching the Pi by IP.
|
||||||
|
transport_security=TransportSecuritySettings(enable_dns_rebinding_protection=False),
|
||||||
|
)
|
||||||
|
activity = Activity(inner)
|
||||||
|
return BearerAuth(activity, token), activity
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
"""ROS node: the bridge between the outside world and the ROS graph, and the only way in or out.
|
||||||
|
|
||||||
|
Serves MCP (streamable HTTP, bearer token) to Hermes on the USB link and talks to the graph locally.
|
||||||
|
Subscribes: image_topic (sensor_msgs/CompressedImage, default /camera/image_raw/compressed)
|
||||||
|
Publishes: /hermes/state (std_msgs/String):
|
||||||
|
`idle` once a second while an authenticated MCP client is connected: the agent heartbeat
|
||||||
|
the face uses to wake and, when it stops, to fall asleep ("no agent connected");
|
||||||
|
`looking` when look() is called, so the face squints and flashes.
|
||||||
|
The token comes from the ROBOT_MCP_TOKEN environment variable (an env file readable only by the
|
||||||
|
service user), never from a parameter.
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
|
||||||
|
import rclpy
|
||||||
|
import uvicorn
|
||||||
|
from rclpy.node import Node
|
||||||
|
from sensor_msgs.msg import CompressedImage
|
||||||
|
from std_msgs.msg import String
|
||||||
|
|
||||||
|
from .bridge import LatestFrame, build_app, build_mcp
|
||||||
|
|
||||||
|
|
||||||
|
class McpNode(Node):
|
||||||
|
def __init__(self):
|
||||||
|
super().__init__('hermes_mcp')
|
||||||
|
param = lambda name, default: self.declare_parameter(name, default).value # noqa: E731
|
||||||
|
topic = param('image_topic', '/camera/image_raw/compressed')
|
||||||
|
self.port = int(param('port', 8765))
|
||||||
|
self.host = str(param('bind', '10.55.0.2')) # the USB link; the firewall allows only 10.55.0.1
|
||||||
|
self.grace_s = float(param('connected_grace_s', 30.0))
|
||||||
|
look_timeout = float(param('look_timeout_s', 3.0))
|
||||||
|
if bool(param('head', False)):
|
||||||
|
self.get_logger().error('head:=true but hermes_head is not built yet: turn_head stays off')
|
||||||
|
self.state_pub = self.create_publisher(String, '/hermes/state', 10)
|
||||||
|
self.frames = LatestFrame()
|
||||||
|
self.create_subscription(CompressedImage, topic, lambda m: self.frames.put(bytes(m.data)), 1)
|
||||||
|
|
||||||
|
token = os.environ.get('ROBOT_MCP_TOKEN', '').strip()
|
||||||
|
self.activity = None
|
||||||
|
self._server = None
|
||||||
|
self._stopping = False
|
||||||
|
if len(token) < 16:
|
||||||
|
self.get_logger().error('ROBOT_MCP_TOKEN is missing or too short (openssl rand -hex 32): '
|
||||||
|
'not serving MCP, and no heartbeat')
|
||||||
|
return
|
||||||
|
mcp = build_mcp(self.frames, self._announce_look, head=None, look_timeout_s=look_timeout)
|
||||||
|
self.app, self.activity = build_app(mcp, token, self.host)
|
||||||
|
threading.Thread(target=self._serve, name='mcp-http', daemon=True).start()
|
||||||
|
self.create_timer(1.0, self._heartbeat)
|
||||||
|
|
||||||
|
def _announce_look(self) -> None:
|
||||||
|
self.state_pub.publish(String(data='looking'))
|
||||||
|
|
||||||
|
def _heartbeat(self) -> None:
|
||||||
|
if self.activity.connected(time.monotonic(), self.grace_s):
|
||||||
|
self.state_pub.publish(String(data='idle'))
|
||||||
|
|
||||||
|
def _serve(self) -> None:
|
||||||
|
"""Serve until stopped. The USB address may not exist yet at boot, so a failed bind retries."""
|
||||||
|
while not self._stopping:
|
||||||
|
self._server = uvicorn.Server(uvicorn.Config(self.app, host=self.host, port=self.port,
|
||||||
|
log_level='warning'))
|
||||||
|
try:
|
||||||
|
self.get_logger().info(f'serving MCP on http://{self.host}:{self.port}/mcp')
|
||||||
|
self._server.run()
|
||||||
|
except (SystemExit, OSError) as exc:
|
||||||
|
self.get_logger().warn(f'MCP server stopped ({exc!r}); retrying in 3 s', throttle_duration_sec=30.0)
|
||||||
|
time.sleep(3.0)
|
||||||
|
|
||||||
|
def destroy_node(self):
|
||||||
|
self._stopping = True
|
||||||
|
if self._server is not None:
|
||||||
|
self._server.should_exit = True
|
||||||
|
super().destroy_node()
|
||||||
|
|
||||||
|
|
||||||
|
def main(args=None):
|
||||||
|
rclpy.init(args=args)
|
||||||
|
node = McpNode()
|
||||||
|
try:
|
||||||
|
rclpy.spin(node)
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
pass
|
||||||
|
finally:
|
||||||
|
node.destroy_node()
|
||||||
|
if rclpy.ok():
|
||||||
|
rclpy.shutdown()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
<?xml version="1.0"?>
|
||||||
|
<?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
|
||||||
|
<package format="3">
|
||||||
|
<name>hermes_mcp</name>
|
||||||
|
<version>0.1.0</version>
|
||||||
|
<description>The MCP bridge: Hermes's only way in or out of the robot's ROS graph.</description>
|
||||||
|
<maintainer email="noreply@example.com">K. Isom</maintainer>
|
||||||
|
<license>UNLICENSED</license>
|
||||||
|
|
||||||
|
<depend>rclpy</depend>
|
||||||
|
<depend>sensor_msgs</depend>
|
||||||
|
<depend>std_msgs</depend>
|
||||||
|
|
||||||
|
<test_depend>python3-pytest</test_depend>
|
||||||
|
|
||||||
|
<export>
|
||||||
|
<build_type>ament_python</build_type>
|
||||||
|
</export>
|
||||||
|
</package>
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
[develop]
|
||||||
|
script_dir=$base/lib/hermes_mcp
|
||||||
|
[install]
|
||||||
|
install_scripts=$base/lib/hermes_mcp
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
from setuptools import find_packages, setup
|
||||||
|
|
||||||
|
package_name = 'hermes_mcp'
|
||||||
|
|
||||||
|
setup(
|
||||||
|
name=package_name,
|
||||||
|
version='0.1.0',
|
||||||
|
packages=find_packages(exclude=['test']),
|
||||||
|
data_files=[
|
||||||
|
('share/ament_index/resource_index/packages', ['resource/' + package_name]),
|
||||||
|
('share/' + package_name, ['package.xml']),
|
||||||
|
],
|
||||||
|
install_requires=['setuptools'],
|
||||||
|
zip_safe=True,
|
||||||
|
maintainer='K. Isom',
|
||||||
|
maintainer_email='noreply@example.com',
|
||||||
|
description='MCP bridge for the Hermes robot.',
|
||||||
|
license='UNLICENSED',
|
||||||
|
entry_points={
|
||||||
|
'console_scripts': [
|
||||||
|
'mcp_server = hermes_mcp.mcp_node:main',
|
||||||
|
],
|
||||||
|
},
|
||||||
|
)
|
||||||
@@ -0,0 +1,99 @@
|
|||||||
|
import asyncio
|
||||||
|
import base64
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
pytest.importorskip('mcp')
|
||||||
|
from starlette.testclient import TestClient # noqa: E402
|
||||||
|
|
||||||
|
from hermes_mcp.bridge import Activity, BearerAuth, LatestFrame, build_app, build_mcp # noqa: E402
|
||||||
|
|
||||||
|
TOKEN = 'x' * 32
|
||||||
|
|
||||||
|
|
||||||
|
class FakeHead:
|
||||||
|
def move(self, pan, tilt):
|
||||||
|
return max(-80, min(80, pan)), max(-45, min(45, tilt))
|
||||||
|
|
||||||
|
def position(self):
|
||||||
|
return 0.0, 0.0
|
||||||
|
|
||||||
|
|
||||||
|
def tool_names(mcp):
|
||||||
|
return {t.name for t in asyncio.run(mcp.list_tools())}
|
||||||
|
|
||||||
|
|
||||||
|
def test_tools_are_look_only_without_a_head_and_keep_their_names_with_one():
|
||||||
|
assert tool_names(build_mcp(LatestFrame())) == {'look'}
|
||||||
|
assert tool_names(build_mcp(LatestFrame(), head=FakeHead())) == {'look', 'turn_head', 'head_position'}
|
||||||
|
|
||||||
|
|
||||||
|
def test_look_returns_the_next_frame_not_a_stale_one():
|
||||||
|
frames = LatestFrame()
|
||||||
|
frames.put(b'stale', time.monotonic() - 5)
|
||||||
|
announced = []
|
||||||
|
mcp = build_mcp(frames, announce_look=lambda: announced.append(1), look_timeout_s=2.0)
|
||||||
|
threading.Timer(0.3, lambda: frames.put(b'\xff\xd8fresh')).start()
|
||||||
|
result = asyncio.run(mcp.call_tool('look', {}))
|
||||||
|
image = result.content[0]
|
||||||
|
assert announced == [1]
|
||||||
|
assert image.type == 'image' and image.mime_type == 'image/jpeg'
|
||||||
|
assert base64.b64decode(image.data) == b'\xff\xd8fresh' # the frame that arrived after the call
|
||||||
|
|
||||||
|
|
||||||
|
def test_look_raises_when_the_camera_is_silent():
|
||||||
|
mcp = build_mcp(LatestFrame(), look_timeout_s=0.3)
|
||||||
|
with pytest.raises(Exception) as err:
|
||||||
|
asyncio.run(mcp.call_tool('look', {}))
|
||||||
|
assert 'camera' in str(err.value.__cause__) # the SDK wraps it; the cause is ours
|
||||||
|
|
||||||
|
|
||||||
|
def test_wait_newer_semantics():
|
||||||
|
f = LatestFrame()
|
||||||
|
t0 = time.monotonic()
|
||||||
|
assert f.wait_newer(t0, 0.1) is None # nothing yet
|
||||||
|
f.put(b'a', t0 - 1)
|
||||||
|
assert f.wait_newer(t0, 0.1) is None # older than t0
|
||||||
|
threading.Timer(0.1, lambda: f.put(b'b')).start()
|
||||||
|
assert f.wait_newer(t0, 1.0) == b'b'
|
||||||
|
|
||||||
|
|
||||||
|
def test_auth_rejects_missing_and_wrong_tokens_and_counts_only_good_requests():
|
||||||
|
app, activity = build_app(build_mcp(LatestFrame()), TOKEN, '127.0.0.1')
|
||||||
|
with TestClient(app) as client:
|
||||||
|
assert client.get('/mcp').status_code == 401
|
||||||
|
assert client.get('/mcp', headers={'Authorization': 'Bearer wrong'}).status_code == 401
|
||||||
|
assert not activity.connected(time.monotonic(), 30.0) # rejected requests are not a client
|
||||||
|
r = client.get('/mcp', headers={'Authorization': f'Bearer {TOKEN}'})
|
||||||
|
assert r.status_code != 401
|
||||||
|
assert activity.connected(time.monotonic(), 30.0) # an authenticated one is
|
||||||
|
assert not activity.connected(time.monotonic() + 31.0, 30.0) # ...until the grace runs out
|
||||||
|
|
||||||
|
|
||||||
|
def test_activity_counts_inflight_requests_as_connected():
|
||||||
|
async def run():
|
||||||
|
release = asyncio.Event()
|
||||||
|
|
||||||
|
async def slow_app(scope, receive, send):
|
||||||
|
await release.wait()
|
||||||
|
|
||||||
|
act = Activity(slow_app)
|
||||||
|
task = asyncio.create_task(act({'type': 'http'}, None, None))
|
||||||
|
await asyncio.sleep(0.05)
|
||||||
|
assert act.inflight == 1 and act.connected(time.monotonic() + 1000, 1.0) # a held stream stays connected
|
||||||
|
release.set()
|
||||||
|
await task
|
||||||
|
assert act.inflight == 0
|
||||||
|
asyncio.run(run())
|
||||||
|
|
||||||
|
|
||||||
|
def test_bearer_auth_passes_lifespan_through():
|
||||||
|
seen = []
|
||||||
|
|
||||||
|
async def inner(scope, receive, send):
|
||||||
|
seen.append(scope['type'])
|
||||||
|
|
||||||
|
asyncio.run(BearerAuth(inner, TOKEN)({'type': 'lifespan'}, None, None))
|
||||||
|
assert seen == ['lifespan']
|
||||||
Reference in New Issue
Block a user