|
|
|
@@ -1,401 +1,316 @@
|
|
|
|
|
# Current state
|
|
|
|
|
|
|
|
|
|
Last updated 2026-10-04. `HANDOFF.md` is the brief; this file tracks where the work stands.
|
|
|
|
|
Last updated 2026-10-05. This file says where the work really stands. `HANDOFF.md` is the original
|
|
|
|
|
brief and has drifted (see its banner); where they differ, this file wins. `README.md` is the guide for a
|
|
|
|
|
person using the robot.
|
|
|
|
|
|
|
|
|
|
## Hardware on hand
|
|
|
|
|
|
|
|
|
|
Raspberry Pi 4 (8 GB), BrainCraft HAT, Arducam UC-346 camera, and nothing else: **no servos, no
|
|
|
|
|
fan, no speakers**, no USB-C splitter, no printed enclosure (the camera hangs loose on its ribbon and is
|
|
|
|
|
held by hand). The Pi runs at about 62 C open-air with everything on and no fan; an enclosure will be
|
|
|
|
|
hotter, so fit the fan before enclosing it. Playback and the servos are untested. Work until those arrive is limited to what the Pi, HAT and camera can do.
|
|
|
|
|
Raspberry Pi 4 (8 GB), BrainCraft HAT, Arducam UC-346 camera, and **nothing else: no servos, no fan, no
|
|
|
|
|
speakers**, no USB-C splitter, no printed enclosure. The camera hangs loose on its ribbon and is held by
|
|
|
|
|
hand. The Pi is powered from talos's USB port (a bench setup, fine without servos). It runs at about
|
|
|
|
|
60 °C open-air with everything on; an enclosure will be hotter, so fit the fan before enclosing it.
|
|
|
|
|
Playback and the servos are untested because they cannot be.
|
|
|
|
|
|
|
|
|
|
## What works (verified on the real robot)
|
|
|
|
|
|
|
|
|
|
- **Face** on the display, started at power-up, with a `sleeping` pose that means "no agent connected".
|
|
|
|
|
The eyes glance toward detected faces; the mouth draws the live voice. Gaze direction and the voice
|
|
|
|
|
trace were confirmed by the owner on the real display.
|
|
|
|
|
- **Camera**, **face detection**, **opt-in recognition** (the owner is enrolled and recognised live).
|
|
|
|
|
- **MCP bridge**: `hermes mcp test robot` passes; real Hermes turns on the local `ornith` model call
|
|
|
|
|
`look()` and `faces()`.
|
|
|
|
|
- **USB network link**, **firewall**, and the whole robot **boots and works with Wi-Fi off**.
|
|
|
|
|
- **Thermal guard**, and a reboot-tested **single installer**, `pi/install.sh` (re-run on the live robot,
|
|
|
|
|
it changed nothing; **not yet run on a blank card**).
|
|
|
|
|
- 122 unit tests on the Pi: face 50, vision 30, audio 10, MCP 16, thermal 12, bringup 4.
|
|
|
|
|
|
|
|
|
|
## The Pi (`hermes-bot`)
|
|
|
|
|
|
|
|
|
|
- Ubuntu 24.04.5 server, kernel `6.8.0-1065-raspi` (fully upgraded), flashed from `talos`. User
|
|
|
|
|
`hermes` (key-only; passwordless sudo), SSH keys for `kyle@vade` (talos) and `kyle@straylight`.
|
|
|
|
|
- On Wi-Fi at `192.168.88.152` (MAC `d8:3a:dd:0d:66:f5`); `eth0` is not cabled. The owner says it has
|
|
|
|
|
a static IP on the rack switch; that address is not recorded here yet. **USB link: `10.55.0.2`**
|
|
|
|
|
(see package 0b below); `ssh hermes@10.55.0.2` works from talos. `avahi-daemon` is
|
|
|
|
|
installed, but mDNS is off in `systemd-resolved` on talos, so `hermes-bot.local` doesn't resolve
|
|
|
|
|
there; use the IP.
|
|
|
|
|
- **cloud-init applies `network-config` and `user-data` on the first boot only.** To change Wi-Fi
|
|
|
|
|
later, edit `/etc/netplan/50-cloud-init.yaml` on the Pi, or bump `instance_id` in `meta-data` on
|
|
|
|
|
`system-boot`. `pi/50-cloud-init.yaml` is a template for the Wi-Fi block; it is gitignored
|
|
|
|
|
because it holds credentials.
|
|
|
|
|
- Installed: ROS 2 Jazzy (`ros-base`, `ros-dev-tools`, colcon, `robot_state_publisher`, `xacro`,
|
|
|
|
|
`camera_ros`); `python3-numpy/pil/spidev/lgpio/rpi-lgpio`; a venv `~/venv` (system site packages
|
|
|
|
|
visible) with `adafruit-circuitpython-rgb-display` and Blinka. Workspace: `~/ros_ws`, source synced
|
|
|
|
|
from `ros/` with `rsync -a --delete --exclude __pycache__ ros/ hermes@<ip>:~/ros_ws/src/`, then
|
|
|
|
|
`colcon build --symlink-install`.
|
|
|
|
|
- **One unit runs the whole robot:** `robot.service` runs `ros2 launch hermes_bringup robot.launch.py`
|
|
|
|
|
as `hermes` (face, camera, face detection, audio levels), so everything is ROS and the face is up
|
|
|
|
|
whenever the robot is powered; each node is its own process, respawned by the launch file. Install
|
|
|
|
|
with `pi/install-robot-service.sh`. `RuntimeDirectory=robot` gives lgpio a writable working
|
|
|
|
|
directory (it drops a pipe in its cwd; systemd's default `/` is not writable). The face starts
|
|
|
|
|
first and the camera, vision and audio nodes follow `start_delay` (6 s) later, so the Python
|
|
|
|
|
processes do not fight over the CPU. Parameters are in `ros/hermes_bringup/config/robot.yaml`.
|
|
|
|
|
`robot-cpu-performance.service` holds the governor at `performance` (the face costs about half the
|
|
|
|
|
CPU at full clock). `pi/trim-boot.sh` disables snapd and ModemManager (unused; ~9 s of boot CPU).
|
|
|
|
|
- **Boot timing** (reboot-tested, everything returns by itself): the face draws at about 28 s on the
|
|
|
|
|
boot clock (kernel 6.9 s, `robot.service` at ~10 s into userspace, then `ros2 launch` takes ~10 s
|
|
|
|
|
to start on a cold Pi, then the face ~3 s); camera, audio and vision are up by ~37 s. The owner is
|
|
|
|
|
fine with up to a minute, and prefers everything in ROS to a faster non-ROS face; a split
|
|
|
|
|
`robot-face.service` that skipped the launch framework only saved 3.7 s and was dropped.
|
|
|
|
|
- **Steady-state CPU** with everything running (4 cores): face ~76%, camera ~57% (15 fps, from 30
|
|
|
|
|
at ~40% alone), detector ~40% (was 131%), audio ~12%; about half the machine idle, 62 C, no
|
|
|
|
|
throttling, no fan fitted. Camera `FrameDurationLimits` is the lever (15 fps ~20% alone, 10 fps
|
|
|
|
|
~14%); JPEG quality barely matters. OpenCV's DNN spins every core and burns 3-4x its elapsed
|
|
|
|
|
time in CPU, so the detector node pins it to one thread (YuNet 86 ms, SFace 161 ms of CPU), decodes
|
|
|
|
|
the JPEG at half size (`IMREAD_REDUCED_COLOR_2`, boxes scaled back to full-size pixels) and
|
|
|
|
|
re-identifies a tracked face at most once a second (`IdentityCache`).
|
|
|
|
|
- `ros2 node list` with its default short discovery window often shows only some of the nodes; use
|
|
|
|
|
`ros2 node list --no-daemon --spin-time 6` (all four appear), or check topics.
|
|
|
|
|
- **ROS never leaves the Pi** (owner's decision): `ROS_AUTOMATIC_DISCOVERY_RANGE=LOCALHOST` (set in
|
|
|
|
|
`pi/robot-env.sh`, which `source`s everything) and the firewall drops domain 42's DDS ports
|
|
|
|
|
(17900-18149) on every interface but loopback. `hermes_mcp` is the only bridge to the outside; a
|
|
|
|
|
host that wants data from the robot gets it through an MCP tool, not DDS. This supersedes
|
|
|
|
|
HANDOFF.md's `host/ros-shell` talking to the robot. (ROS's setup script exports `SUBNET`, so the
|
|
|
|
|
variable must be assigned, not defaulted.)
|
|
|
|
|
- **PipeWire** (1.0.5, WirePlumber 0.4.17) runs as `hermes`'s user services (`loginctl enable-linger`).
|
|
|
|
|
`~/.config/pipewire/pipewire-pulse.conf.d/50-network-server.conf` (from `pi/pipewire/`) makes
|
|
|
|
|
pipewire-pulse listen on TCP 4713 on all interfaces; the firewall is the protection. Defaults are
|
|
|
|
|
the WM8960 source and sink (`alsa_input/output.platform-soc_sound.stereo-fallback`). Verified from
|
|
|
|
|
talos: `pactl -s tcp:10.55.0.2:4713 info` works; the same on the Wi-Fi address times out.
|
|
|
|
|
- Ubuntu 24.04.5 server, kernel `6.8.0-1065-raspi`, flashed from talos. User `hermes` (key-only login,
|
|
|
|
|
passwordless sudo; no password is set). SSH keys for `kyle@vade` (talos) and `kyle@straylight`.
|
|
|
|
|
- Network: USB link `10.55.0.2` (host `10.55.0.1`); Wi-Fi `192.168.88.152` (MAC `d8:3a:dd:0d:66:f5`);
|
|
|
|
|
`eth0` is not cabled (the owner mentioned a static address on the rack switch; not recorded here).
|
|
|
|
|
`avahi-daemon` is installed, but mDNS is off in `systemd-resolved` on talos, so use the IP. `rfkill`
|
|
|
|
|
is installed.
|
|
|
|
|
- **cloud-init applies `network-config` and `user-data` on the first boot only.** To change Wi-Fi later,
|
|
|
|
|
edit `/etc/netplan/50-cloud-init.yaml` on the Pi, or bump `instance_id` in `meta-data` on
|
|
|
|
|
`system-boot`. `pi/network-config.example` is a credentials-free template for the Wi-Fi block (`pi/50-cloud-init.yaml`
|
|
|
|
|
is gitignored, for a filled copy).
|
|
|
|
|
- **Install:** `rsync` this repo to the Pi and `sudo ./pi/install.sh`. It installs ROS 2 Jazzy, builds
|
|
|
|
|
the Raspberry Pi libcamera fork (pinned commit) into `/opt/libcamera-rpi` and `camera_ros` against it
|
|
|
|
|
(`~/camera_ws`), makes `~/venv` (display libraries, OpenCV 4.10, the MCP SDK), downloads the face
|
|
|
|
|
models to `~/models`, restores the camera calibration, builds `~/ros_ws`, sets up the USB gadget, the
|
|
|
|
|
audio overlay and mixer, the firewall (before anything listens), PipeWire and the services, and
|
|
|
|
|
creates the MCP token. Update just the code with `rsync -a --delete --exclude __pycache__ ros/
|
|
|
|
|
hermes@<ip>:~/ros_ws/src/`, `colcon build --symlink-install` in `~/ros_ws`, `sudo systemctl restart
|
|
|
|
|
robot`. Any shell that runs a node by hand must `source ~/robot-env.sh` first.
|
|
|
|
|
- **Services:** `robot.service` runs `ros2 launch hermes_bringup robot.launch.py` as `hermes`: the face,
|
|
|
|
|
camera, face detector, audio levels, MCP bridge and thermal guard, each its own process respawned by
|
|
|
|
|
the launch file. The face starts first; the others follow `start_delay` (6 s). Parameters:
|
|
|
|
|
`ros/hermes_bringup/config/robot.yaml`. `RuntimeDirectory=robot` gives lgpio a writable working
|
|
|
|
|
directory (it drops a pipe in its cwd; systemd's default `/` is not writable). Also
|
|
|
|
|
`robot-cpu-performance.service` (governor `performance`; the face costs about half the CPU at full
|
|
|
|
|
clock), `robot-firewall.service`, `robot-usb-gadget.service`. `pi/trim-boot.sh` disables snapd and
|
|
|
|
|
ModemManager (unused; ~9 s of boot CPU).
|
|
|
|
|
- **Boot:** the face draws about 28 s after the kernel starts (kernel ~7 s, `robot.service` ~10 s into
|
|
|
|
|
userspace, `ros2 launch` itself ~10 s on a cold Pi, the face ~3 s); the other nodes are up by ~37 s.
|
|
|
|
|
The owner is fine with up to a minute and prefers everything in ROS: a split face unit that skipped
|
|
|
|
|
`ros2 launch` saved only 3.7 s and was dropped. Reboot-tested: everything returns by itself.
|
|
|
|
|
- **CPU**, full stack, 4 cores: face ~84%, camera ~63% (15 fps; 30 fps alone is ~40%, 15 is ~20%),
|
|
|
|
|
detector ~45%, audio ~15%, MCP ~15%, PipeWire ~12%; about 40% of the machine idle, 60 °C, no
|
|
|
|
|
throttling. OpenCV's DNN spins every core and burns 3-4x its elapsed time in CPU, so the detector pins
|
|
|
|
|
it to one thread (YuNet 86 ms, SFace 161 ms of CPU), decodes the JPEG at half size and re-identifies a
|
|
|
|
|
tracked face at most once a second.
|
|
|
|
|
- **ROS never leaves the Pi** (owner's decision): `ROS_AUTOMATIC_DISCOVERY_RANGE=LOCALHOST` (ROS's
|
|
|
|
|
setup script exports `SUBNET`, so `pi/robot-env.sh` assigns it) and the firewall drops domain 42's DDS
|
|
|
|
|
ports (17900-18149) on every interface but loopback. `hermes_mcp` is the only bridge out. This
|
|
|
|
|
supersedes HANDOFF's `host/ros-shell`.
|
|
|
|
|
- **Firewall** (`pi/nftables-robot.nft`, loaded at boot before networking by `robot-firewall.service`):
|
|
|
|
|
accepts 4713 (audio), 8765 (MCP) and 8766 (reserved, nothing listens) only on `usb0` from `10.55.0.1`
|
|
|
|
|
and drops them elsewhere. SSH is deliberately open on all interfaces (key-only), for V2: the owner's call.
|
|
|
|
|
- **PipeWire** 1.0.5 / WirePlumber 0.4.17 run as `hermes`'s user services (`loginctl enable-linger`).
|
|
|
|
|
`pi/pipewire/50-network-server.conf` makes pipewire-pulse listen on TCP 4713 on all interfaces; the
|
|
|
|
|
firewall is the protection. Defaults are the WM8960 source and sink.
|
|
|
|
|
|
|
|
|
|
## Package 0a (platform proof on Ubuntu 24.04)
|
|
|
|
|
## Display and the face (`ros/hermes_face`)
|
|
|
|
|
|
|
|
|
|
- Step 1 done: `host/check-pi-platform.sh` passes.
|
|
|
|
|
- Step 2 done: flashed, booted, upgraded; ROS installed; `ros2 topic list` works.
|
|
|
|
|
- **Step 5 (display) done.** ST7789 works on `lgpio` through Blinka (`rpi-lgpio`, no `RPi.GPIO`).
|
|
|
|
|
Two gotchas: pass `cs=None` (the kernel drives CE0; claiming GPIO8 raises "GPIO busy"), and the
|
|
|
|
|
240x240 glass sits at `y_offset=80` in the controller's RAM, for every rotation (the driver
|
|
|
|
|
rotates in software and leaves the scan direction fixed). Mounted pose (microSD end up, USB-C edge
|
|
|
|
|
on the left) needs content turned 90 degrees clockwise: `rotation=270`. Full-frame push is
|
|
|
|
|
~23 fps through the driver; the face pushes raw RGB565 with `_block` and reaches ~27-28 fps.
|
|
|
|
|
- **Camera: works with the Raspberry Pi libcamera fork, not with the stock stacks.** The OV5647 is
|
|
|
|
|
detected (`ov5647 10-0036`) and Ubuntu's libcamera 0.2.0 lists it once `libcamera-ipa` is
|
|
|
|
|
installed, but capture dies in the Pi IPA (`assertion "it != buffers_.end()" in prepareIsp()`),
|
|
|
|
|
and ROS Jazzy's libcamera 0.7.2 dies with `FATAL … A list of V4L2 controls requires a
|
|
|
|
|
ControlInfoMap` right after `configure`. Both are deterministic whatever the stream format or
|
|
|
|
|
size. Fix: build the fork (`github.com/raspberrypi/libcamera`, commit `6c1dd9d`, 2026-08-17) with
|
|
|
|
|
`-Dpipelines=rpi/vc4 -Dipas=rpi/vc4 -Dcam=enabled -Dv4l2=false …` into `/opt/libcamera-rpi`; its
|
|
|
|
|
`cam` then captures 1296x972 RGB888 at ~16 fps (dim room). `pi/capture_still.py` takes a still
|
|
|
|
|
with it. **`camera_ros` still has to be built against this fork** (the apt one uses ROS's broken
|
|
|
|
|
libcamera). Frames are padded (stride 3936) and B,G,R in memory.
|
|
|
|
|
- **`camera_ros` 0.7.0 is built from source against the fork** in `~/camera_ws` (overlay on ROS; at
|
|
|
|
|
runtime the fork must come first in `LD_LIBRARY_PATH`, which `source ~/robot-env.sh` does; copy
|
|
|
|
|
from `pi/robot-env.sh`). Verified: `ros2 run camera_ros camera_node --ros-args -p width:=1296 -p
|
|
|
|
|
height:=972 -p format:=RGB888` publishes `/camera/image_raw`, `/camera/image_raw/compressed` (~26
|
|
|
|
|
Hz) and `/camera/camera_info`. It costs ~140% of a core with raw and compressed both on; lower the
|
|
|
|
|
frame rate or size before running everything together. No calibration file yet.
|
|
|
|
|
- **`ros/hermes_vision`**: `hermes_vision.detect` (YuNet, Haar fallback; no ROS imports) and a node
|
|
|
|
|
`face_detector` that subscribes to the compressed image and publishes `/vision/faces`
|
|
|
|
|
(`vision_msgs/Detection2DArray`, bbox centre/size in pixels, `class_id: face`, score), at `rate_hz`
|
|
|
|
|
(default 4, ~47% of a core on a Pi 4); `publish_debug:=true` adds an annotated image. CLI:
|
|
|
|
|
`python -m hermes_vision.detect_cli IMAGE`. YuNet 2023mar needs OpenCV 4.8+, so `~/venv` has
|
|
|
|
|
`opencv-python-headless==4.10.0.84` with `numpy<2` (system NumPy stays 1.26.4 for ROS), and
|
|
|
|
|
`robot-env.sh` puts the venv first on `PYTHONPATH`; YuNet also needs input sides that are multiples
|
|
|
|
|
of 32. Tested on a sample image and on a live frame (one face found, upright, no flip needed).
|
|
|
|
|
- **Audio capture works (verified 2026-10-04).** The stock `wm8960-soundcard` overlay fails every
|
|
|
|
|
stream with `wm8960: No MCLK configured` on Ubuntu 24.04: it puts the 12.288 MHz fixed clock on the
|
|
|
|
|
sound card's codec link, but the codec driver reads `mclk` from its own node.
|
|
|
|
|
`pi/overlays/hermes-wm8960.dts` is the stock overlay plus `clocks`/`clock-names` on the codec node;
|
|
|
|
|
`pi/install-audio-overlay.sh` compiles it to `/boot/firmware/overlays/hermes-wm8960.dtbo`, sets
|
|
|
|
|
`dtparam=audio=off` and `dtoverlay=hermes-wm8960`. Capture then opens at 8-48 kHz. Address the
|
|
|
|
|
card by name (`hw:CARD=wm8960soundcard,DEV=0`): its index changes (it was card 1, then 0).
|
|
|
|
|
- **The HAT's hardware Audio ON/OFF slide switch (right edge, by the "Audio" label) must be ON.**
|
|
|
|
|
With it off, the codec still answers on I2C and every software check looks right, but all
|
|
|
|
|
recordings are exactly zero. That cost an hour of debugging; check it first.
|
|
|
|
|
- Mixer: `Left/Right Input Mixer Boost Switch` default to off and must be on. Settings in use:
|
|
|
|
|
Capture 47/47 (+18 dB), `Left/Right Input Boost Mixer LINPUT1/RINPUT1` volume 1, input mixer boost
|
|
|
|
|
on. Singing at arm's length: RMS -34 to -49 dBFS, peaks -18 dBFS, no clipping, channels within
|
|
|
|
|
1-2 dB. Saved with `alsactl store` and exported as `pi/wm8960_asound.state`; restoring at boot is
|
|
|
|
|
not yet verified. No speakers on hand, so playback is untested.
|
|
|
|
|
- Not done: PWM (waits for servos). Ask before rebooting the Pi.
|
|
|
|
|
- **Display:** the ST7789 works on `lgpio` through Blinka (`rpi-lgpio`, not `RPi.GPIO`). Pass `cs=None`
|
|
|
|
|
(the kernel drives CE0; claiming GPIO8 raises "GPIO busy"). The 240x240 glass sits 80 rows into the
|
|
|
|
|
controller's RAM, `y_offset=80`, for every rotation (the driver rotates in software and leaves the
|
|
|
|
|
scan direction fixed). The mounted pose (microSD end up, USB-C edge on the left) needs the content
|
|
|
|
|
turned 90° clockwise: `rotation=270`. The face pushes raw RGB565 with `_block` at ~27-28 fps.
|
|
|
|
|
- **Renderer** (`render.py`, no ROS imports): a numpy port of `docs/face-prototype.html`, flat face only
|
|
|
|
|
(head and low-poly shapes are not ported). Phosphor and vector styles; tone map is a lookup table,
|
|
|
|
|
glow at half resolution, rotation a free `np.rot90`; parameters are clamped. States: `sleeping idle
|
|
|
|
|
listening thinking looking speaking`; `looking` returns to the state it interrupted.
|
|
|
|
|
- **Node** (`face_node.py`): subscribes `/hermes/state`, `/joint_states`, `/vision/faces`,
|
|
|
|
|
`/camera/camera_info`, `/audio/activity`, `/audio/scope/{mic,speaker}`; starts in `sleeping`; logs
|
|
|
|
|
every state change (`journalctl -u robot`), so behaviour can be checked without watching the panel.
|
|
|
|
|
`fps` can be changed live (the thermal guard does).
|
|
|
|
|
- **State arbitration** (`arbiter.py`): sleeping (agent side only) > speaking > listening > thinking >
|
|
|
|
|
idle. **Room sound never wakes a sleeping face.**
|
|
|
|
|
- **Agent heartbeat:** `/hermes/state` doubles as the heartbeat. After `agent_timeout_s` (10) without
|
|
|
|
|
it the face logs `agent gone, sleeping` and sleeps.
|
|
|
|
|
- **Gaze:** eyes glance toward the best face (a named one first, then the biggest), smoothed, held
|
|
|
|
|
2.5 s, then drift back; sleeping ignores it. Image x flips on the display (a person on the image's
|
|
|
|
|
right is on the robot's right, the viewer's left). The renderer's pan sign was wrong (inherited from
|
|
|
|
|
the prototype) and is fixed: positive pan = the robot's left = the viewer's right; positive tilt = up,
|
|
|
|
|
as the MCP `turn_head` documents. Never seen with a moving head.
|
|
|
|
|
- **Voice trace:** while fresh scope data exists, the mouth draws the real waveform (speaker when
|
|
|
|
|
speaking, mic when listening), auto-scaled; the owner confirmed it follows how close and loud they are.
|
|
|
|
|
- Not done: head/low-poly shapes, DotStar LEDs, a button mute. Open: the IPS panel's black reads dark
|
|
|
|
|
gray on camera at full backlight; dimming needs a software PWM on GPIO26.
|
|
|
|
|
|
|
|
|
|
## Package 0b (USB link): Pi side and talos side working
|
|
|
|
|
## Camera and vision (`ros/hermes_vision`, `camera_ros`)
|
|
|
|
|
|
|
|
|
|
- Pi: `dtoverlay=dwc2,dr_mode=peripheral`; `robot-usb-gadget.service` builds a configfs gadget
|
|
|
|
|
(NCM `usb0` + ACM `ttyGS0`, host MAC `02:68:65:72:6d:01`, robot MAC `02:68:65:72:6d:02`) before
|
|
|
|
|
networking; `serial-getty@ttyGS0` is enabled; `/etc/netplan/60-usb0.yaml` gives `usb0`
|
|
|
|
|
`10.55.0.2/30` with no gateway, so the default route stays on Wi-Fi. Installed by
|
|
|
|
|
`pi/install-usb-gadget.sh` (a backup of `config.txt` is `config.txt.bak-usbgadget`). NCM works on
|
|
|
|
|
the Pi 4 with kernel 1065; ECM was not needed.
|
|
|
|
|
- **netplan gotcha:** match the gadget interface by `name: usb0`. A `macaddress:` match becomes
|
|
|
|
|
`PermanentMACAddress=` in networkd, which a gadget interface lacks, so it stays unmanaged.
|
|
|
|
|
- talos (Omarchy, NetworkManager): `host/omarchy/robot-usb-link.sh` (needs sudo) adds profile
|
|
|
|
|
`hermes-robot-usb`: `10.55.0.1/30` on the interface with MAC `02:68:65:72:6d:01`, priority 100,
|
|
|
|
|
never-default. Verified from talos: ping ~2.4 ms, SSH over the link.
|
|
|
|
|
- **Firewall:** `robot-firewall.service` loads `pi/nftables-robot.nft` before networking at every
|
|
|
|
|
boot (`pi/install-firewall.sh`; it must be loaded before the PipeWire network server is enabled).
|
|
|
|
|
It accepts 4713/8765/8766 only on `usb0` from `10.55.0.1`, drops them elsewhere, and drops DDS on
|
|
|
|
|
every interface but loopback. Verified: from talos, 4713 connects over `usb0` and times out over
|
|
|
|
|
Wi-Fi, while an unused port on the Pi's Wi-Fi address refuses instantly (so only 4713 is blocked).
|
|
|
|
|
SSH is deliberately left open on all interfaces (key-only), for V2; the owner's call.
|
|
|
|
|
- Not yet verified: serial login prompt (talos's `/dev/ttyACM0` is `root:uucp`; use
|
|
|
|
|
`sudo screen /dev/ttyACM0 115200`). The `hermes` account has no password, so the console shows a
|
|
|
|
|
login prompt but cannot log in until the owner sets one; no autologin by design. Also not done:
|
|
|
|
|
the Pi booting with Wi-Fi off, replug and second-host recovery,
|
|
|
|
|
`get_throttled` after a servo sweep on the splitter, the NixOS snippet, and straylight/other-host
|
|
|
|
|
testing. Currently the Pi is powered from talos's USB port (bench test, no servos).
|
|
|
|
|
- **Camera: only the Raspberry Pi libcamera fork works.** The OV5647 is detected (`ov5647 10-0036`), but
|
|
|
|
|
Ubuntu's libcamera 0.2.0 dies in the Pi IPA (`assertion "it != buffers_.end()" in prepareIsp()`) and
|
|
|
|
|
ROS Jazzy's 0.7.2 with `FATAL … A list of V4L2 controls requires a ControlInfoMap`, deterministically
|
|
|
|
|
for every format and size. The fork (commit `6c1dd9d`, built with `-Dpipelines=rpi/vc4 -Dipas=rpi/vc4
|
|
|
|
|
-Dv4l2=false …` into `/opt/libcamera-rpi`) captures; `camera_ros` 0.7.0 is built from source against
|
|
|
|
|
it, and the fork must come first in `LD_LIBRARY_PATH` (`robot-env.sh` does it). Stream: 1296x972 at
|
|
|
|
|
15 fps (`FrameDurationLimits` in `robot.yaml`); `/camera/image_raw`, `…/compressed`, `camera_info`.
|
|
|
|
|
`pi/capture_still.py` takes a still with the fork's `cam`. Frames are padded and B,G,R in memory.
|
|
|
|
|
- **Calibration done** (`pi/calibration/…_1296x972.yaml`, restored by the installer): 18 of 24 views,
|
|
|
|
|
reprojection error 0.296 px, focal length ~1153 px, centre (653, 463), small distortion with `k3` held
|
|
|
|
|
at zero. The first pass fit 0.41 px but with cancelling k1/k2/k3 because the board never reached the
|
|
|
|
|
edges; the tool now fixes `k3`, reports coverage, drops blurred outlier views and can re-solve saved
|
|
|
|
|
views (`--from-views`). **The top-left and bottom-right corners were never reached**, so the model is
|
|
|
|
|
least certain there. The result is valid for the current focus and image size; a changed focus
|
|
|
|
|
invalidates it. Tool: `python3 -m hermes_vision.calibrate --cols 9 --rows 6 --square-mm <mm>`
|
|
|
|
|
(`docs/checkerboard-9x6.png` can be shown on a screen; the square size only scales distance, not the
|
|
|
|
|
intrinsics).
|
|
|
|
|
- **Detection** (`detect.py`, no ROS imports): OpenCV YuNet 2023mar (Haar fallback). It needs OpenCV
|
|
|
|
|
4.8+ (apt has 4.6), so `~/venv` carries `opencv-python-headless==4.10.0.84` with `numpy<2` (the system
|
|
|
|
|
NumPy stays 1.26.4 for ROS), and the venv is first on `PYTHONPATH`; input sides must be multiples of 32.
|
|
|
|
|
- **Tracking** (`tracking.py`): faces are matched across frames by box overlap, so a person keeps one
|
|
|
|
|
track id; identity is re-resolved at most once a second.
|
|
|
|
|
- **Node** `face_detector` publishes `/vision/faces` (`vision_msgs/Detection2DArray`): boxes in
|
|
|
|
|
full-size camera pixels, `Detection2D.id` = the track id, `results[0]` the face, `results[1]` the
|
|
|
|
|
identity if recognised (class_id = name, score = cosine similarity, not a probability). An unknown
|
|
|
|
|
face has no `results[1]`: a name is never made up. `rate_hz` (4) can be changed live.
|
|
|
|
|
- **Recognition is opt-in** (`recognize.py`, `people.py`): SFace embeddings on YuNet's landmarks. Nothing
|
|
|
|
|
is stored until someone is enrolled with `python3 -m hermes_vision.people enroll NAME`; only
|
|
|
|
|
128-number embeddings are kept, in `~/.local/share/hermes/faces.json` (mode 0600, dir 0700, written
|
|
|
|
|
atomically), never images. `list`, `forget`, `check`, `enroll-images` too. **The owner is enrolled
|
|
|
|
|
(10 samples) and was recognised in 29 of 31 detections live (similarity ~0.9).** A throwaway-database
|
|
|
|
|
test recognised 8 of 9 unenrolled frames of the same person, down to side profiles; an unrelated photo
|
|
|
|
|
scored 0.16. Not tested: a second person live. Names are never logged (a count is).
|
|
|
|
|
|
|
|
|
|
## Face (`ros/hermes_face`)
|
|
|
|
|
## Audio (`ros/hermes_audio`, `pi/overlays`, PipeWire)
|
|
|
|
|
|
|
|
|
|
- `render.py`: numpy port of `docs/face-prototype.html`, **flat face only** (head and lowpoly not
|
|
|
|
|
ported). Phosphor and vector styles. States: `sleeping idle listening thinking looking speaking`.
|
|
|
|
|
`sleeping` (new) is the "no agent connected" pose: eyes closed to thin lines, mouth short, dim and
|
|
|
|
|
slowly breathing. `looking` returns to the state it interrupted. Tone mapping is a lookup table
|
|
|
|
|
(RGB and RGB565 tables), glow is computed at half resolution, rotation is a free `np.rot90`.
|
|
|
|
|
- `display.py`: ST7789 driver plus an async pusher (render overlaps the SPI transfer).
|
|
|
|
|
- `face_node.py`: node `hermes_face`; subscribes `/hermes/state` (`std_msgs/String`) and
|
|
|
|
|
`/joint_states` (`pan_joint`, `tilt_joint`, radians); starts in `initial_state` (`sleeping`), so it
|
|
|
|
|
runs with nothing else up. Parameters: `rotation` (270), `x_offset` (0), `y_offset` (80),
|
|
|
|
|
`baudrate`, `fps`, `output` (`st7789` or `none`), `phosphor` (P7), `style`, `persistence_ms`,
|
|
|
|
|
`focus`, `jitter`, `intensity`, `initial_state`.
|
|
|
|
|
- `demo.py`: no ROS. `python -m hermes_face.demo --sequence` cycles the states on the panel;
|
|
|
|
|
`--png-dir DIR` writes a PNG per state without hardware.
|
|
|
|
|
- Tests: `python3 -m pytest test` in `ros/hermes_face` (25 pass on the Pi). Renderer parameters
|
|
|
|
|
(`focus`, `persistence_ms`, `intensity`, `jitter`) are clamped; `fps` is clamped to 1-60.
|
|
|
|
|
- Measured on the Pi 4: ~9 ms per frame in isolation. With the CPU at full clock: 24% of a core at
|
|
|
|
|
20 fps and 40% at 30 fps (target was under 40% at 20 fps or more). Under `ondemand` the same
|
|
|
|
|
load reads 40% at 20 fps and ~67% at 30 fps. No C++ rewrite was needed.
|
|
|
|
|
- **Gaze:** the node subscribes to `/vision/faces` (and `/camera/camera_info` for the image size)
|
|
|
|
|
and the eyes glance toward the best face: a recognised one first, then the biggest. Image x flips
|
|
|
|
|
on the display (a person on the image's right is on the robot's right, the viewer's left); the
|
|
|
|
|
glance is smoothed, held 2.5 s, then drifts back to centre. Sleeping ignores it. The pan sign in
|
|
|
|
|
the renderer was wrong (carried over from the prototype) and is fixed: positive pan = the robot's
|
|
|
|
|
left = the viewer's right, positive tilt = looks up, as the MCP `turn_head` tool documents.
|
|
|
|
|
**Confirmed by the owner on the real display (2026-10-04):** the glance directions look right.
|
|
|
|
|
An earlier run showed nothing because the one-off wake message was lost (volatile QoS drops a
|
|
|
|
|
sample published before discovery matches); test publishers must republish, as `hermes_mcp`
|
|
|
|
|
will. The node logs every state change (`state: idle (agent=..., audio=...)`), so behaviour can be
|
|
|
|
|
checked from `journalctl -u robot-face` without watching the panel.
|
|
|
|
|
- **Agent heartbeat:** `/hermes/state` doubles as the agent's heartbeat. `hermes_mcp` must republish
|
|
|
|
|
its state about once a second; after `agent_timeout_s` (default 10, 0 disables) with none, the
|
|
|
|
|
face logs `agent gone, sleeping` and sleeps, so "sleeping" means "no agent connected". Verified:
|
|
|
|
|
woke on messages, slept 10 s after the last.
|
|
|
|
|
- Not yet: audio levels for the mouth (`/audio/levels`), `hermes_mcp` publishing states, launch
|
|
|
|
|
file in `hermes_bringup`, head/lowpoly shapes, DotStar LEDs, button mute.
|
|
|
|
|
- Open: IPS black level. The backlight is on at full brightness, so black reads dark gray on
|
|
|
|
|
camera. Dimming it needs a software PWM on GPIO26 (not a hardware PWM pin).
|
|
|
|
|
- **The stock `wm8960-soundcard` overlay fails on Ubuntu 24.04** (`wm8960: No MCLK configured`): it puts
|
|
|
|
|
the 12.288 MHz fixed clock on the card's codec link, but the codec driver reads `mclk` from its own
|
|
|
|
|
node. `pi/overlays/hermes-wm8960.dts` adds it there; `pi/install-audio-overlay.sh` compiles it, sets
|
|
|
|
|
`dtparam=audio=off` and `dtoverlay=hermes-wm8960`. Capture then opens at 8-48 kHz. Address the card
|
|
|
|
|
by name (`hw:CARD=wm8960soundcard,DEV=0`): its index changes.
|
|
|
|
|
- **The HAT's hardware Audio ON/OFF switch (right edge) must be ON.** With it off, the codec still
|
|
|
|
|
answers on I²C and every software check looks right, but all recordings are exactly zero.
|
|
|
|
|
- **Mixer:** `Left/Right Input Mixer Boost` default to off and must be on. `pi/wm8960_asound.state` holds
|
|
|
|
|
the settings (capture 47, boost 1). **WirePlumber resets the codec's capture gain to +30 dB** when it
|
|
|
|
|
takes the device, so the installer sets PipeWire's source volume to 63% (its scale is cubic: -12 dB,
|
|
|
|
|
ALSA `Capture` 46, about +17 dB), which WirePlumber remembers; survives reboots (checked). Quiet-room
|
|
|
|
|
floor at this gain: about -43 dBFS; singing at arm's length peaks around -18 dBFS.
|
|
|
|
|
- **Node** `audio_state` taps the default source and sink monitor with `parec` (16 kHz mono) and
|
|
|
|
|
publishes `/audio/levels`, `/audio/activity` (speaking | listening | quiet) and trigger-locked
|
|
|
|
|
waveform windows `/audio/scope/{mic,speaker}` (only while listening or speaking). Audio itself never
|
|
|
|
|
goes through ROS. `Activity`: 3 s start-up calibration of an adaptive noise floor, margin 7 dB, 1.2 s
|
|
|
|
|
hold so pauses between words do not flicker, steady noise is absorbed after 8 s, the mic is ignored
|
|
|
|
|
while speaking. **The flicker retune is unit-tested but not re-confirmed live.**
|
|
|
|
|
- Verified: the `speaking` path (a tone played to the sink, no speaker attached, shows on the monitor);
|
|
|
|
|
the mic path with the owner's voice. Playback to a real speaker, echo cancel (`pi/pipewire/
|
|
|
|
|
60-echo-cancel.conf.optional`) and the host's `robot_mic` naming wait for the speakers.
|
|
|
|
|
|
|
|
|
|
## Audio levels and the literal voice trace (`ros/hermes_audio`, face integration)
|
|
|
|
|
## The MCP bridge and Hermes (`ros/hermes_mcp`)
|
|
|
|
|
|
|
|
|
|
- `hermes_audio` node `audio_state` taps the default source and the default sink's monitor with
|
|
|
|
|
`parec` (16 kHz mono) and publishes `/audio/levels` ([mic_dbfs, speaker_dbfs]), `/audio/activity`
|
|
|
|
|
(`speaking` | `listening` | `quiet`) and trigger-locked waveform windows `/audio/scope/{mic,speaker}`
|
|
|
|
|
(256 samples). Audio itself never goes through ROS. `hermes_audio.levels.Activity` debounces and
|
|
|
|
|
keeps an adaptive noise floor: a 3 s start-up calibration, then it drops fast and rises slowly, and
|
|
|
|
|
anything 'listening' without a gap for 8 s is absorbed as ambient noise. The microphone is ignored
|
|
|
|
|
while speaking (it would hear the robot).
|
|
|
|
|
- The face arbitrates (`hermes_face.arbiter`): sleeping (agent side only) > speaking > listening >
|
|
|
|
|
thinking > idle. **Room sound never wakes a sleeping face**: sleeping means "no agent connected".
|
|
|
|
|
`looking` stays a timed event owned by the renderer. While fresh (< 0.35 s) scope data exists, the
|
|
|
|
|
mouth draws the real waveform (speaker when speaking, mic when listening), auto-scaled.
|
|
|
|
|
- **WirePlumber resets the codec's capture gain** to 100% (+30 dB) when it takes the device,
|
|
|
|
|
overriding `alsactl`'s saved level. Fix: `pactl set-source-volume @DEFAULT_SOURCE@ 63%` (PipeWire's
|
|
|
|
|
scale is cubic: 63% is -12 dB from the +30 dB maximum, so ALSA `Capture` lands at 46, about +17
|
|
|
|
|
dB). WirePlumber remembers it; recheck after a reboot. The quiet-room floor at this gain is about
|
|
|
|
|
-43 dBFS (preamp and mic noise).
|
|
|
|
|
- **Confirmed by the owner:** the mouth draws their live voice and its size follows how close and
|
|
|
|
|
loud they are (proportional). The activity detector was retuned after the log showed the face
|
|
|
|
|
flickering between listening and idle about twice a second: margin 7 dB (quiet-room noise peaks
|
|
|
|
|
sit at about +3 dB over the floor), hold 1.2 s so pauses between words do not drop it. The
|
|
|
|
|
flicker fix is covered by unit tests but not yet re-confirmed live (the retest window was silent).
|
|
|
|
|
- Verified on the Pi: the `speaking` path (a tone played to the sink, with no speaker attached, shows
|
|
|
|
|
on the monitor at -13.5 dBFS, `speaking` for its length, mic ignored); levels and scope publish;
|
|
|
|
|
49 face tests, 10 audio tests. **Not yet verified:** the audio node as a service (it was run as a
|
|
|
|
|
transient unit).
|
|
|
|
|
|
|
|
|
|
## The MCP bridge (`ros/hermes_mcp`)
|
|
|
|
|
|
|
|
|
|
The only way in or out of the ROS graph. Serves MCP (SDK 2.3, streamable HTTP) on
|
|
|
|
|
`http://10.55.0.2:8765/mcp`, the USB address only (retrying the bind if the link is not up yet), with
|
|
|
|
|
a bearer token. **Tool names are the stable contract with Hermes**: `look()`, `faces()`, `list_people()`, and `turn_head` /
|
|
|
|
|
`head_position` only when a head is fitted. With no servos those two are not registered, by design
|
|
|
|
|
(`head: false`; the node refuses `head: true` until `hermes_head` exists). `look()` waits for the next
|
|
|
|
|
camera frame taken after the call (never a stale one) and passes the camera's JPEG through (1296x972,
|
|
|
|
|
~155 KB); it also publishes `looking` on `/hermes/state` so the face squints and flashes.
|
|
|
|
|
While an authenticated client is connected (in flight, or heard within `connected_grace_s` = 30) it
|
|
|
|
|
republishes `idle` on `/hermes/state` once a second: the agent heartbeat that wakes the face. Verified
|
|
|
|
|
end to end from talos over the USB link with a real MCP client: connects, lists `look`, `look()`
|
|
|
|
|
returns a JPEG in 0.08 s, a wrong token is refused, the Wi-Fi path is dropped by the firewall, and
|
|
|
|
|
the face went `looking` -> `idle` on connect and `agent gone, sleeping` about 40 s after the client
|
|
|
|
|
left. The token is in `/etc/robot-mcp.env` (root:hermes, mode 640), generated by
|
|
|
|
|
`pi/install-robot-service.sh` and never printed; read it with `sudo cat`. The MCP SDK is in `~/venv`
|
|
|
|
|
(`pip install "mcp>=2"`). The old Pi Zero files (`pi/robot_mcp_server.py`, `pi/robot-mcp.service`,
|
|
|
|
|
`pi/requirements.txt`, the old `pi/install.sh`, and `tools/`) were deleted (git has them). **Not yet verified:** a real Hermes client, and
|
|
|
|
|
whether Hermes passes MCP image content to the model (HANDOFF package 6).
|
|
|
|
|
|
|
|
|
|
**Hermes on talos is wired to it (2026-10-05, with the owner's authorisation).** Hermes Agent v0.21.5 is
|
|
|
|
|
installed on talos (`~/.hermes`; hosted main model `grok-4.6` via xai-oauth; vision is auxiliary and
|
|
|
|
|
runs locally, `qwen3-vl-8b-abliterated` on straylight; the `ornith-*` models are the local ones on
|
|
|
|
|
straylight/dixie). `~/.hermes/config.yaml` gained an `mcp_servers.robot` block (URL
|
|
|
|
|
`http://10.55.0.2:8765/mcp`, `Authorization: Bearer ${ROBOT_MCP_TOKEN}`) and `~/.hermes/.env` gained
|
|
|
|
|
`ROBOT_MCP_TOKEN`; backups are `config.yaml.bak-20261005T002659-robot-mcp` and
|
|
|
|
|
`.env.bak-20261005T002659-robot-mcp`. `hermes mcp list` shows it enabled and **`hermes mcp test
|
|
|
|
|
robot` passes: connected in 789 ms, 1 tool (`look`)**, which is the HANDOFF acceptance check. A fresh
|
|
|
|
|
`HERMES_HOME` hangs on first-run setup, so test against the real one.
|
|
|
|
|
|
|
|
|
|
**A real agent turn works, entirely on local models (2026-10-05):** `hermes chat -q "Call the look tool
|
|
|
|
|
once ... what you see" --oneshot -Q --provider custom:llama-router -m ornith-1.5-35b-a3b --max-turns 4`
|
|
|
|
|
(straylight's router; no hosted provider, so the image stayed on the LAN). Hermes called
|
|
|
|
|
`mcp__robot__look` (0.07 s), the model described the live photo correctly (blurry close-up of a board
|
|
|
|
|
with blue LEDs, a red label, camouflage fabric), and the robot's face went `looking` -> `idle`.
|
|
|
|
|
**Hermes does not hand MCP image content to the main model.** It saves the JPEG to
|
|
|
|
|
`~/.hermes/cache/images/img_<hash>.jpg` on talos (157,147 bytes, the same as the robot sent) and
|
|
|
|
|
returns a 72-character reference; the model then calls Hermes's `vision_analyze`, which ran on the local
|
|
|
|
|
vision model (`qwen3-vl-8b-abliterated` on straylight) in 29 s. That is HANDOFF package 6's fallback and
|
|
|
|
|
Hermes does it itself, so `look()` needs no change. Two consequences: each look() leaves a photo of the
|
|
|
|
|
room in talos's Hermes image cache (Hermes's retention, not the robot's), and a look-and-describe turn
|
|
|
|
|
takes ~70 s on the local models, mostly the vision call.
|
|
|
|
|
|
|
|
|
|
### Hermes's feedback on the bridge (2026-10-05) and what was done
|
|
|
|
|
|
|
|
|
|
Hermes (on talos, from `~/.hermes/cache/scratch/hermes-bot-v0-face-identity.md`) asked for recognitions
|
|
|
|
|
instead of raw photos. **Built:** `faces()` returns the faces in view as structured JSON with no photo
|
|
|
|
|
(`track_id`, pixel and normalised `bbox`, detection `confidence`, `seen_for_s`, and `identity`
|
|
|
|
|
`{name, similarity}` or `null`); `look(send_image="always"|"never")` now returns that same JSON as a
|
|
|
|
|
text block plus the photo (default kept as before: the image, so scene description still works);
|
|
|
|
|
`list_people()` gives names and sample counts; a `robot://status` resource (hardware present or
|
|
|
|
|
absent, what is detected, enrolled count, tools) and prompts `who_is_here` and `look_around` fill the
|
|
|
|
|
previously empty `list_resources` / `list_prompts`; `/vision/faces` carries a persistent track id
|
|
|
|
|
(`hermes_vision.tracking.FaceTracker`, matching by box overlap, identity re-checked once a second;
|
|
|
|
|
`Detection2D.id` is the track id, `results[0]` the face, `results[1]` the identity if recognised).
|
|
|
|
|
Verified with a real Hermes turn on the local `ornith` model: it called `faces()` (0.03 s, 593
|
|
|
|
|
characters, no image, no vision call) and answered that Kyle was there at 0.78 similarity.
|
|
|
|
|
**Declined, deliberately:** embeddings in responses (biometric templates, meaningless to an LLM,
|
|
|
|
|
and the registry stays on the robot); age/gender/hat attributes (no model for them here, and guessing
|
|
|
|
|
age and gender from faces is unreliable and sensitive); auto-updating a person's face data from new
|
|
|
|
|
captures (it lets a stranger slowly become "Kyle" with nobody deciding that); a remote `enroll` tool
|
|
|
|
|
(a prompt could make the robot call anyone "Kyle": enrolment stays the owner's step on the robot,
|
|
|
|
|
`python3 -m hermes_vision.people enroll NAME`; a gated enrol tool, off by default, is possible if the
|
|
|
|
|
owner wants it); detecting people, objects or motion (only faces exist, and `robot://status` says so).
|
|
|
|
|
`look()`'s default was not flipped to JSON-only, because the image is what lets Hermes describe a
|
|
|
|
|
scene; that default is the owner's call.
|
|
|
|
|
|
|
|
|
|
### Hermes bug report: "faces and list_people are not registered" (2026-10-05) — not a bridge bug
|
|
|
|
|
|
|
|
|
|
Checked, not assumed: a raw MCP client lists `faces`, `look`, `list_people`; `hermes mcp test robot`
|
|
|
|
|
(fresh connection) discovers all three; there is no tool filter in `~/.hermes/config.yaml`. Hermes's own
|
|
|
|
|
`agent.log` explains the report: my robot restart at 01:17 ended the MCP session (`keepalive failed ...
|
|
|
|
|
Session not found`), Hermes reconnected and logged `registered 7 tool(s): mcp__robot__faces, look,
|
|
|
|
|
list_people, ...` at 01:18:46, and a session that started afterwards (`...011845...`) called
|
|
|
|
|
`mcp__robot__faces` fine. The reporting session (`...003532...`) started at 00:35, before `faces`
|
|
|
|
|
existed, and keeps its start-time snapshot of the tool table: it still reaches live things
|
|
|
|
|
(`list_prompts`, `read_resource`) but gets `'mcp__robot__faces' is not a known tool name`. Fix on the
|
|
|
|
|
owner's side: start a new Hermes session after the robot's tool set changes. For Hermes: refresh a
|
|
|
|
|
session's tool table (and its deferred-tool index) when the MCP server is re-registered. We keep the
|
|
|
|
|
bridge's stateful sessions on purpose: the `Session not found` after a restart is what makes Hermes
|
|
|
|
|
reconnect and re-register (stateless HTTP would hide a changed tool set from a connected client).
|
|
|
|
|
|
|
|
|
|
## Face recognition (opt-in)
|
|
|
|
|
|
|
|
|
|
`hermes_vision.recognize` / `people`: SFace embeddings on YuNet's landmarks. Nothing is stored until
|
|
|
|
|
someone is enrolled (`python -m hermes_vision.people enroll NAME`, live from the camera topic, or
|
|
|
|
|
`enroll-images`); only 128-number embeddings are kept, in `~/.local/share/hermes/faces.json` (mode
|
|
|
|
|
0600, directory 0700, written atomically), never images. `list`, `forget NAME` and `check IMAGE`
|
|
|
|
|
manage and test it. The detector node puts a recognised name in `Detection2D.id` and picks up
|
|
|
|
|
database changes without a restart; its log prints a count, not names. Checked with a throwaway
|
|
|
|
|
database: 8 of 9 unenrolled frames of the same person were recognised, down to side profiles
|
|
|
|
|
(similarity 0.52-0.66), the miss being a head tipped back (0.34 against the 0.363 threshold); an
|
|
|
|
|
unrelated photo scored 0.16. **Nobody is enrolled yet**; the owner enrols themselves live.
|
|
|
|
|
|
|
|
|
|
## Next
|
|
|
|
|
|
|
|
|
|
1. **Camera calibration: tool built and tested, not yet run.** `python3 -m hermes_vision.calibrate
|
|
|
|
|
--cols 9 --rows 6 --square-mm <measured>` (after `source ~/robot-env.sh`, robot running) collects 20
|
|
|
|
|
varied views of a checkerboard (`docs/checkerboard-9x6.png` can be shown on a screen; measure a
|
|
|
|
|
square), solves with OpenCV and writes the camera_info YAML camera_ros looks for
|
|
|
|
|
(`~/.ros/camera_info/ov5647__base_soc_i2c0mux_i2c_1_ov5647_36_1296x972.yaml`: the name includes the
|
|
|
|
|
resolution). The solver recovers a known synthetic camera to within 1%. It needs the owner holding a
|
|
|
|
|
board in front of the camera, and the lens focus set first (the lens turns if not glued; earlier
|
|
|
|
|
close-range frames were blurry). V2's perception needs the result.
|
|
|
|
|
2. Waiting on hardware: the servos (see "Arrival day" above), the speakers
|
|
|
|
|
(playback, echo cancel, the `robot_mic` naming on the host, Hermes talking aloud), the fan (before
|
|
|
|
|
any enclosure), the splitter, the printed enclosure (measure first, see `enclosure/README.md`).
|
|
|
|
|
3. Still open: replug and second-host recovery of the USB link, the serial console login, an
|
|
|
|
|
`agent_timeout_s` check against a long idle Hermes session, the retuned listening detector live
|
|
|
|
|
(unit-tested only), a second person in frame, running `pi/install.sh` on a blank card, and the
|
|
|
|
|
NixOS host config (skipped for now by the owner).
|
|
|
|
|
4. A new Omarchy host with a GPU (it runs the local `ornith` model) is arriving: it needs only
|
|
|
|
|
`host/omarchy/robot-usb-link.sh` and its SSH key added to the Pi.
|
|
|
|
|
- Serves MCP (SDK 2.3, streamable HTTP) on `http://10.55.0.2:8765/mcp`: the USB address only (it retries
|
|
|
|
|
the bind until the link is up), bearer token in `/etc/robot-mcp.env` (root:hermes, mode 640, made by
|
|
|
|
|
the installer, never printed; read it with `sudo cat`). **Tool names are the stable contract with
|
|
|
|
|
Hermes.** Tools: `look(send_image="always"|"never")` (the faces summary as JSON text plus the next
|
|
|
|
|
fresh JPEG, never a stale frame; it also makes the face squint), `faces()` (the faces in view, no
|
|
|
|
|
photo: `track_id`, `bbox`, `confidence`, `seen_for_s`, `identity` `{name, similarity}` or `null`),
|
|
|
|
|
`list_people()` (names and sample counts only), and `turn_head` / `head_position` **only when a head
|
|
|
|
|
is fitted** (`head: false` now; the node refuses `head: true` until `hermes_head` exists). Resource
|
|
|
|
|
`robot://status` (hardware present or absent, what is detected, enrolled count, tools, and health:
|
|
|
|
|
temperature, thermal level, throttle flags now and since boot) and prompts `who_is_here`,
|
|
|
|
|
`look_around` make it discoverable. Sessions are stateful on purpose (see the Hermes note below).
|
|
|
|
|
- **Heartbeat:** while an authenticated client is connected (in flight, or heard within
|
|
|
|
|
`connected_grace_s` = 30) it republishes `idle` on `/hermes/state` once a second; the face wakes on
|
|
|
|
|
connect and sleeps about 40 s after the client leaves.
|
|
|
|
|
- **Hermes on talos is wired to it** (with the owner's authorisation). Hermes Agent v0.21.5 is in
|
|
|
|
|
`~/.hermes` (hosted main model `grok-4.6` via xai-oauth; vision is auxiliary and local,
|
|
|
|
|
`qwen3-vl-8b-abliterated` on straylight; the `ornith-*` models are the local ones on straylight and
|
|
|
|
|
dixie). `config.yaml` has an `mcp_servers.robot` block and `.env` has `ROBOT_MCP_TOKEN`; backups:
|
|
|
|
|
`config.yaml.bak-20261005T002659-robot-mcp`, `.env.bak-20261005T002659-robot-mcp`. `hermes mcp test
|
|
|
|
|
robot` passes. A fresh `HERMES_HOME` hangs on first-run setup, so test against the real one.
|
|
|
|
|
- **Real agent turns, entirely on local models:** `hermes chat -q "..." --oneshot -Q --provider
|
|
|
|
|
custom:llama-router -m ornith-1.5-35b-a3b --max-turns 5`. `look()` works; **Hermes does not hand MCP
|
|
|
|
|
images to the main model**: it saves the JPEG to `~/.hermes/cache/images/` on talos and returns a short
|
|
|
|
|
reference, and the model calls `vision_analyze` (local vision model, ~29 s). So a look-and-describe turn
|
|
|
|
|
takes ~70 s, and each look leaves a photo in talos's Hermes cache (Hermes's retention). `faces()` is
|
|
|
|
|
the cheap path: 0.03 s, no image, and Hermes answered "Kyle is at the desk, similarity 0.78".
|
|
|
|
|
- **Hermes's feedback (`~/.hermes/cache/scratch/hermes-bot-v0-face-identity.md`).** Built: `faces()`,
|
|
|
|
|
`look` returning that JSON, `list_people`, `robot://status` and prompts (previously empty), track ids.
|
|
|
|
|
**Declined, deliberately:** embeddings in responses (biometric templates, meaningless to an LLM; the
|
|
|
|
|
registry stays on the robot); age/gender/hat (no model, and unreliable and sensitive); auto-updating
|
|
|
|
|
face data from new captures (lets a stranger slowly become "Kyle"); a remote `enroll` tool (a prompt
|
|
|
|
|
could make the robot call anyone "Kyle": enrolment stays the owner's step; a gated tool, off by
|
|
|
|
|
default, is possible if wanted); detecting people, objects or motion (only faces exist). `look()`'s
|
|
|
|
|
default stays photo-plus-faces, because the photo is what lets Hermes describe a scene: **flipping it
|
|
|
|
|
to JSON-only is the owner's call.**
|
|
|
|
|
- **Hermes bug report "faces and list_people are not registered" was not a bridge bug.** A raw client and a
|
|
|
|
|
fresh `hermes mcp test` list all three; Hermes's own `agent.log` shows it registering all seven tools
|
|
|
|
|
at 01:18:46 after a robot restart, and a later session calling `faces()` fine. The reporting session
|
|
|
|
|
started at 00:35, before `faces` existed, and keeps a start-time snapshot of its tool table (live calls
|
|
|
|
|
like `list_prompts` still work, which matches the report). Fix: start a new Hermes session after the
|
|
|
|
|
robot's tool set changes; Hermes should refresh a session's tool table on re-registration. The bridge
|
|
|
|
|
keeps stateful sessions on purpose: the `Session not found` after a restart is what makes Hermes
|
|
|
|
|
reconnect and re-register.
|
|
|
|
|
|
|
|
|
|
## Thermal guard (`ros/hermes_thermal`)
|
|
|
|
|
|
|
|
|
|
Built because the Pi has no fan (about 60 C open-air with everything running) and an enclosure will be
|
|
|
|
|
hotter. `thermal_guard` reads the CPU temperature (`/sys/class/thermal`) and `vcgencmd get_throttled`,
|
|
|
|
|
publishes `/health/temperature`, `/health/thermal_level` (ok | warm | hot | critical) and
|
|
|
|
|
`/health/throttled`, and sheds load through the target nodes' own live parameters: the camera's
|
|
|
|
|
`FrameDurationLimits`, the face's `fps` and the detector's `rate_hz` (the face and detector gained live
|
|
|
|
|
parameter callbacks for this). Levels enter at 72 / 77 / 82 C (the Pi throttles at about 80); heating is
|
|
|
|
|
acted on at once, cooling goes one level at a time, only below the threshold minus 3 C and after a 20 s
|
|
|
|
|
dwell. Rates scale by 1.0 / 0.7 / 0.45 / 0.2 with floors (camera 2 fps, face 8, detector 1 Hz). The
|
|
|
|
|
"normal" rates are parameters of the guard and a bringup test asserts they equal the real ones in
|
|
|
|
|
`robot.yaml`, so restoring is exact and survives the guard restarting. `fake_temp_c` forces a
|
|
|
|
|
temperature for testing. `robot://status` now carries a `health` block (temperature, level, throttle
|
|
|
|
|
flags now and since boot, which doubles as a power-supply check).
|
|
|
|
|
**Verified live on the robot** by forcing 73 / 78 / 85 C and back: the camera, face and detector rates
|
|
|
|
|
followed (measured camera 16.5 -> 11.5 -> 7.2 -> 3.5 Hz and back to 16.5), cooling stepped down critical
|
|
|
|
|
-> hot -> warm -> ok exactly one dwell apart, and the guard stayed the same process throughout.
|
|
|
|
|
**A live bug the unit tests missed:** one logging call that warned when heating and informed when
|
|
|
|
|
cooling raised `ValueError: Logger severity cannot be changed between calls` in rclpy, killing the
|
|
|
|
|
guard on its first cooling step (the launch file respawned it, which only looked like a recovery). Fixed
|
|
|
|
|
with separate call sites, the tick now catches and logs any error instead of dying, and a test drives
|
|
|
|
|
the real node through a full heat-then-cool run; a mutation check (the old logging in a throwaway
|
|
|
|
|
copy) confirmed that test fails on exactly that error. Not yet tested under a real thermal load.
|
|
|
|
|
When the servos arrive, watch the under-voltage flag in `robot://status` during a sweep.
|
|
|
|
|
The Pi has no fan. `thermal_guard` reads the CPU temperature and `vcgencmd get_throttled`, publishes
|
|
|
|
|
`/health/temperature`, `/health/thermal_level` (ok | warm | hot | critical) and `/health/throttled`, and
|
|
|
|
|
sheds load through the target nodes' live parameters: the camera's `FrameDurationLimits`, the face's `fps`,
|
|
|
|
|
the detector's `rate_hz`. Levels enter at 72 / 77 / 82 °C (the Pi throttles at ~80); heating is acted on
|
|
|
|
|
at once, cooling goes one level at a time only below the threshold minus 3 °C and after a 20 s dwell.
|
|
|
|
|
Rates scale by 1.0 / 0.7 / 0.45 / 0.2 with floors (camera 2 fps, face 8, detector 1 Hz). Its "normal"
|
|
|
|
|
rates are its own parameters, and a bringup test asserts they equal the real ones in `robot.yaml`, so
|
|
|
|
|
restoring is exact and survives a restart. `fake_temp_c` (`ros2 param set /hermes_thermal fake_temp_c
|
|
|
|
|
85.0`, -1 to stop) forces a temperature. **Verified live** by forcing 73 / 78 / 85 °C and back: measured
|
|
|
|
|
camera rate 16.5 -> 11.5 -> 7.2 -> 3.5 Hz and back, cooling stepped down one dwell apart, same process
|
|
|
|
|
throughout. **A live bug the unit tests missed:** one log line that warned when heating and informed when
|
|
|
|
|
cooling raised rclpy's `Logger severity cannot be changed between calls` and killed the guard on its
|
|
|
|
|
first cooling step (the respawn hid it). Fixed (separate call sites), the tick now survives any error, and
|
|
|
|
|
a test drives the real node through heat-then-cool; a mutation check confirmed it fails on the original
|
|
|
|
|
bug. Not yet tested under a real thermal load.
|
|
|
|
|
|
|
|
|
|
## USB link (HANDOFF package 0b)
|
|
|
|
|
|
|
|
|
|
- Pi: `dtoverlay=dwc2,dr_mode=peripheral`; `robot-usb-gadget.service` builds a configfs gadget (NCM `usb0`
|
|
|
|
|
+ ACM `ttyGS0`; host MAC `02:68:65:72:6d:01`, robot MAC `…:02`) before networking;
|
|
|
|
|
`/etc/netplan/60-usb0.yaml` gives `usb0` `10.55.0.2/30` with no gateway (the default route stays on
|
|
|
|
|
Wi-Fi). NCM works on the Pi 4 (ECM was not needed). **netplan must match `usb0` by name, not MAC:** a
|
|
|
|
|
`macaddress:` match becomes `PermanentMACAddress=`, which a gadget interface lacks, so it stays unmanaged.
|
|
|
|
|
- Host (Omarchy, NetworkManager): `sudo host/omarchy/robot-usb-link.sh` adds profile `hermes-robot-usb`
|
|
|
|
|
(`10.55.0.1/30`, matched by the MAC, never-default). Ping ~2 ms.
|
|
|
|
|
- **Wi-Fi-off acceptance passed:** with Wi-Fi blocked (`rfkill block wifi`, remembered across reboots),
|
|
|
|
|
live and after a reboot, the Pi came up on the USB link alone and everything worked. With no internet
|
|
|
|
|
there is no NTP, so the clock is unsynchronised (roughly right).
|
|
|
|
|
- Not yet verified: replug and second-host recovery, and the serial console login (talos's `/dev/ttyACM0`
|
|
|
|
|
is `root:uucp`; `hermes` has no password, so the console shows a prompt but cannot log in; no autologin
|
|
|
|
|
by design).
|
|
|
|
|
|
|
|
|
|
## Owner's decisions
|
|
|
|
|
|
|
|
|
|
ROS never leaves the Pi (`hermes_mcp` is the only bridge); SSH stays open for V2; NixOS host config is
|
|
|
|
|
skipped for now; no remote `enroll`; `look()`'s default is open; the face may take up to a minute to
|
|
|
|
|
appear at boot; everything should be ROS. A new Omarchy host with a GPU (it runs the local `ornith`
|
|
|
|
|
model) is arriving: it needs only `host/omarchy/robot-usb-link.sh` and its SSH key on the Pi.
|
|
|
|
|
|
|
|
|
|
## Gotchas
|
|
|
|
|
|
|
|
|
|
- `ros2 node list` with its default short window shows only some nodes: use `--no-daemon --spin-time 6`.
|
|
|
|
|
- A one-off message published before ROS discovery matches is lost (volatile QoS): republish, as
|
|
|
|
|
`hermes_mcp` does every second. Anything that sets `PYTHONPATH` by hand must prepend, not replace.
|
|
|
|
|
- A shell that sourced `robot-env.sh` before a build cannot import the newly built package: re-source.
|
|
|
|
|
- rclpy needs each logging call site to keep one severity.
|
|
|
|
|
|
|
|
|
|
## Arrival day: the neck (servos), planned and deliberately not built ahead of the hardware
|
|
|
|
|
|
|
|
|
|
The servos are expected Thursday or Friday at the earliest. The real unknowns cannot be faked (which
|
|
|
|
|
`pwmchip` the overlay creates, its permissions on Ubuntu, each servo's real pulse range, whether the
|
|
|
|
|
servos brown out the Pi), so `hermes_head` waits for the hardware. The order, with what to verify at
|
|
|
|
|
each step:
|
|
|
|
|
Expected Thursday or Friday at the earliest. What matters cannot be faked (which `pwmchip` the overlay
|
|
|
|
|
creates, its permissions on Ubuntu, each servo's real pulse range, whether they brown out the Pi), so
|
|
|
|
|
`hermes_head` waits. Order, with what to verify:
|
|
|
|
|
|
|
|
|
|
1. **Wire it:** pan servo to the HAT's GPIO12 port, tilt to GPIO13; the "3pin Volt" jumper is already
|
|
|
|
|
verified at 5 V. Power matters: servo spikes on a host USB port may brown the Pi out, so use the
|
|
|
|
|
5.1 V 3 A supply through the splitter if it has arrived. Horns off for the first moves.
|
|
|
|
|
verified at 5 V. Servo spikes on a host USB port may brown the Pi out: use the 5.1 V 3 A supply
|
|
|
|
|
through the splitter if it has arrived. Horns off for the first moves.
|
|
|
|
|
2. **Overlay (needs a reboot, ask first):** add `dtoverlay=pwm-2chan,pin=12,func=4,pin2=13,func2=4` under
|
|
|
|
|
`[all]` in `/boot/firmware/config.txt` (`dtparam=audio=off` is already set; the Pi's headphone audio
|
|
|
|
|
shares PWM hardware). Add it to `pi/install-*.sh`, not by hand.
|
|
|
|
|
3. **sysfs:** find the chip (`/sys/class/pwm/pwmchip*`; it was not assumed to be 0), export channels 0
|
|
|
|
|
and 1, `period=20000000`. Add a udev rule that gives a group (not root) write access, as Ubuntu's
|
|
|
|
|
defaults differ from Raspberry Pi OS.
|
|
|
|
|
4. **Measure each servo:** centre (1.5 ms), then find the safe min and max pulse with the horn off; set
|
|
|
|
|
`ROBOT_SERVO_MIN_US` / `MAX_US` per servo (start 500 / 2400), the pan and tilt trims and invert flags.
|
|
|
|
|
5. **Write `hermes_head`:** `PwmHead` on sysfs, a ~0.3 s interpolated move, PWM off after ~1 s idle (stops
|
|
|
|
|
hold-jitter), limits (pan 80, tilt 45), `/joint_states` for `pan_joint` / `tilt_joint` in radians.
|
|
|
|
|
Convention (the MCP contract): positive pan = the robot's left, positive tilt = up. Unit-test the
|
|
|
|
|
logic against a fake sysfs tree, then verify on the Pi: `vcgencmd get_throttled` must stay `0x0`
|
|
|
|
|
through a full sweep while the face is running.
|
|
|
|
|
6. **Wire it in:** the bridge refuses `head: true` today. Give it a head backend that calls
|
|
|
|
|
`hermes_head`, set `head: true` in `robot.yaml`, and check `turn_head` / `head_position` appear and
|
|
|
|
|
that a new Hermes session sees them. Check the face's eyes follow the real joints the right way
|
|
|
|
|
(the sign was fixed in software, never seen with a moving head).
|
|
|
|
|
7. **URDF:** `hermes_description` from the enclosure STLs' `pose_*` parts, frames `base_link`,
|
|
|
|
|
`pan_link`, `tilt_link`, `camera_link`, `camera_optical_frame`, so `/tf` follows the head.
|
|
|
|
|
8. **Mounted camera:** set the flip (`orientation` parameter) for how it is actually mounted, re-check
|
|
|
|
|
the focus (a changed focus invalidates `pi/calibration/`), and set `ROBOT_FACE_ROTATION` if the
|
|
|
|
|
display ends up mounted differently from the portrait pose.
|
|
|
|
|
`[all]` in `config.txt` (`dtparam=audio=off` is already set: the Pi's headphone audio shares PWM
|
|
|
|
|
hardware). Add it to `pi/install-*.sh`, not by hand.
|
|
|
|
|
3. **sysfs:** find the chip (`/sys/class/pwm/pwmchip*`; do not assume 0), export channels 0 and 1,
|
|
|
|
|
`period=20000000`. A udev rule should give a group, not root, write access.
|
|
|
|
|
4. **Measure each servo:** centre (1.5 ms), then the safe min and max pulse with the horn off; set
|
|
|
|
|
`ROBOT_SERVO_MIN_US` / `MAX_US` per servo (start 500 / 2400), the trims and invert flags.
|
|
|
|
|
5. **Write `hermes_head`:** `PwmHead` on sysfs, a ~0.3 s interpolated move, PWM off after ~1 s idle,
|
|
|
|
|
limits (pan 80, tilt 45), `/joint_states` for `pan_joint` / `tilt_joint` in radians (positive pan =
|
|
|
|
|
the robot's left, positive tilt = up). Unit-test against a fake sysfs tree, then verify on the Pi:
|
|
|
|
|
`vcgencmd get_throttled` stays `0x0` through a full sweep with the face running; watch the
|
|
|
|
|
under-voltage flag in `robot://status`.
|
|
|
|
|
6. **Wire it in:** give the bridge a head backend that calls `hermes_head`, set `head: true`, check
|
|
|
|
|
`turn_head` / `head_position` appear to a new Hermes session, and that the face's eyes follow the real
|
|
|
|
|
joints the right way.
|
|
|
|
|
7. **URDF:** `hermes_description` from the enclosure STLs' `pose_*` parts, frames `base_link`, `pan_link`,
|
|
|
|
|
`tilt_link`, `camera_link`, `camera_optical_frame`.
|
|
|
|
|
8. **Mounted camera:** set the flip (`orientation`) for how it is mounted, re-check the focus (a changed
|
|
|
|
|
focus invalidates `pi/calibration/`), and the display rotation if it is mounted differently.
|
|
|
|
|
|
|
|
|
|
## Open items
|
|
|
|
|
|
|
|
|
|
1. Waiting on hardware: servos (above), speakers (playback, echo cancel, Hermes talking aloud, the host's
|
|
|
|
|
`robot_mic` naming), the fan, the splitter, the printed enclosure (measure first: `enclosure/README.md`).
|
|
|
|
|
2. Unconfirmed: the retuned listening detector live; a second person in frame; replug and second-host
|
|
|
|
|
recovery; the serial console; `agent_timeout_s` over a long idle Hermes session; the calibration's
|
|
|
|
|
uncovered corners (retake with the board in the top-left and bottom-right); `pi/install.sh` on a
|
|
|
|
|
blank card (back up the working SD card first: it holds all of this).
|
|
|
|
|
3. Optional: a gated `enroll` tool, if the owner wants one.
|
|
|
|
|
|
|
|
|
|
## Repository
|
|
|
|
|
|
|
|
|
|
Branch `master`. `.gitignore` covers `.remember/`, caches, `.venv/`, `*.orig`, `*.rej` and
|
|
|
|
|
`pi/50-cloud-init.yaml`. Layout: `ros/` (five packages), `pi/` (everything installed on the Pi),
|
|
|
|
|
`host/` (the owner's machines: `host/omarchy/` has the USB link script, Hermes and PipeWire config and
|
|
|
|
|
the launcher; `host/check-pi-platform.sh`), `enclosure/`, `docs/face-prototype.html`. `README.md` is
|
|
|
|
|
rewritten for this build. **`pi/install.sh` is the one installer** (re-runnable: run twice on the live
|
|
|
|
|
robot, the second run changed nothing, exit 0): it installs ROS, builds the libcamera fork (pinned
|
|
|
|
|
commit) and `camera_ros`, sets up the venv, models, workspace, USB gadget, audio overlay, firewall,
|
|
|
|
|
PipeWire, mixer and services. It has not been run on a blank card (no spare card). HANDOFF.md is the
|
|
|
|
|
original brief and has drifted; where they differ, this file wins.
|
|
|
|
|
`pi/50-cloud-init.yaml`. `ros/` has `hermes_face`, `hermes_vision`, `hermes_audio`, `hermes_mcp`,
|
|
|
|
|
`hermes_thermal` and `hermes_bringup`; `pi/` is everything installed on the Pi; `host/` is the owner's
|
|
|
|
|
machines (`host/omarchy/`: the USB link script, Hermes and PipeWire config, the launcher;
|
|
|
|
|
`host/check-pi-platform.sh`); `enclosure/`; `docs/` (`face-prototype.html`, `checkerboard-9x6.png`).
|
|
|
|
|
Run a package's tests on the Pi with `python3 -m pytest test` in its directory after `source
|
|
|
|
|
~/robot-env.sh` (the face suite takes about a minute).
|
|
|
|
|
|
|
|
|
|
## Security notes (all resolved)
|
|
|
|
|
|
|
|
|
|
The automated review of the initial commit flagged three items, all from the Pi Zero design:
|
|
|
|
|
`ROBOT_MCP_BIND` defaulting to `0.0.0.0` (now the USB address only, with a bearer token), PipeWire on
|
|
|
|
|
TCP 4713 on every interface (now protected by the boot-time firewall, which loads first), and the
|
|
|
|
|
single-IP any-interface nftables file (rewritten: usb0 from 10.55.0.1 only). Later reviews added:
|
|
|
|
|
DDS is blocked on every interface and discovery is local-only; recognised names are not logged; node
|
|
|
|
|
parameters are clamped. SSH stays open on all interfaces (key-only) by the owner's choice, for V2.
|
|
|
|
|
The automated review of the initial commit flagged `ROBOT_MCP_BIND` defaulting to `0.0.0.0` (now the USB
|
|
|
|
|
address only, with a bearer token), PipeWire on TCP 4713 on every interface (now behind the boot-time
|
|
|
|
|
firewall, which loads first), and a single-IP any-interface nftables file (rewritten). Later reviews led
|
|
|
|
|
to: DDS blocked on every interface and local-only discovery; recognised names never logged; node
|
|
|
|
|
parameters clamped. The token and the owner's Wi-Fi password never go through the repo or logs (one
|
|
|
|
|
`netplan get` once printed the Wi-Fi password into a session; it was flagged to the owner).
|
|
|
|
|
Claude Sonnet 5.5