CONTEXT.md had grown by appending and contradicted itself (all four nodes vs six, a retracted CPU figure, "no calibration", "nobody enrolled", "audio untested as a service", the Wi-Fi-off boot as not done, a class the tracker replaced). Rewritten as one current-state document organised by subsystem, with fresh measurements, the owner's decisions, the gotchas, the neck's arrival-day plan and what is genuinely still open. README: status, speakers not fitted, new-session note for Hermes, the calibration section (shipped calibration, coverage hints, outlier dropping, --from-views), running the tests, the docs row. HANDOFF.md gets a status banner listing where the original brief has drifted. CLAUDE.md starts at CONTEXT.md. The echo-cancel config no longer talks about a Pi Zero, and the enclosure guide no longer tells you to centre a servo with a tool that does not exist yet. pi/network-config.example is a tracked, credentials-free first-boot template: the README pointed at pi/50-cloud-init.yaml, which is gitignored and so missing from a fresh clone. The dead IdentityCache (replaced by the tracker) is removed. Docstrings corrected: audio scope publishing, the MCP health topics, and what the calibration tool does. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
317 lines
25 KiB
Markdown
317 lines
25 KiB
Markdown
# Current state
|
|
|
|
Last updated 2026-10-05. This file says where the work really stands. `HANDOFF.md` is the original
|
|
brief and has drifted (see its banner); where they differ, this file wins. `README.md` is the guide for a
|
|
person using the robot.
|
|
|
|
## Hardware on hand
|
|
|
|
Raspberry Pi 4 (8 GB), BrainCraft HAT, Arducam UC-346 camera, and **nothing else: no servos, no fan, no
|
|
speakers**, no USB-C splitter, no printed enclosure. The camera hangs loose on its ribbon and is held by
|
|
hand. The Pi is powered from talos's USB port (a bench setup, fine without servos). It runs at about
|
|
60 °C open-air with everything on; an enclosure will be hotter, so fit the fan before enclosing it.
|
|
Playback and the servos are untested because they cannot be.
|
|
|
|
## What works (verified on the real robot)
|
|
|
|
- **Face** on the display, started at power-up, with a `sleeping` pose that means "no agent connected".
|
|
The eyes glance toward detected faces; the mouth draws the live voice. Gaze direction and the voice
|
|
trace were confirmed by the owner on the real display.
|
|
- **Camera**, **face detection**, **opt-in recognition** (the owner is enrolled and recognised live).
|
|
- **MCP bridge**: `hermes mcp test robot` passes; real Hermes turns on the local `ornith` model call
|
|
`look()` and `faces()`.
|
|
- **USB network link**, **firewall**, and the whole robot **boots and works with Wi-Fi off**.
|
|
- **Thermal guard**, and a reboot-tested **single installer**, `pi/install.sh` (re-run on the live robot,
|
|
it changed nothing; **not yet run on a blank card**).
|
|
- 122 unit tests on the Pi: face 50, vision 30, audio 10, MCP 16, thermal 12, bringup 4.
|
|
|
|
## The Pi (`hermes-bot`)
|
|
|
|
- Ubuntu 24.04.5 server, kernel `6.8.0-1065-raspi`, flashed from talos. User `hermes` (key-only login,
|
|
passwordless sudo; no password is set). SSH keys for `kyle@vade` (talos) and `kyle@straylight`.
|
|
- Network: USB link `10.55.0.2` (host `10.55.0.1`); Wi-Fi `192.168.88.152` (MAC `d8:3a:dd:0d:66:f5`);
|
|
`eth0` is not cabled (the owner mentioned a static address on the rack switch; not recorded here).
|
|
`avahi-daemon` is installed, but mDNS is off in `systemd-resolved` on talos, so use the IP. `rfkill`
|
|
is installed.
|
|
- **cloud-init applies `network-config` and `user-data` on the first boot only.** To change Wi-Fi later,
|
|
edit `/etc/netplan/50-cloud-init.yaml` on the Pi, or bump `instance_id` in `meta-data` on
|
|
`system-boot`. `pi/network-config.example` is a credentials-free template for the Wi-Fi block (`pi/50-cloud-init.yaml`
|
|
is gitignored, for a filled copy).
|
|
- **Install:** `rsync` this repo to the Pi and `sudo ./pi/install.sh`. It installs ROS 2 Jazzy, builds
|
|
the Raspberry Pi libcamera fork (pinned commit) into `/opt/libcamera-rpi` and `camera_ros` against it
|
|
(`~/camera_ws`), makes `~/venv` (display libraries, OpenCV 4.10, the MCP SDK), downloads the face
|
|
models to `~/models`, restores the camera calibration, builds `~/ros_ws`, sets up the USB gadget, the
|
|
audio overlay and mixer, the firewall (before anything listens), PipeWire and the services, and
|
|
creates the MCP token. Update just the code with `rsync -a --delete --exclude __pycache__ ros/
|
|
hermes@<ip>:~/ros_ws/src/`, `colcon build --symlink-install` in `~/ros_ws`, `sudo systemctl restart
|
|
robot`. Any shell that runs a node by hand must `source ~/robot-env.sh` first.
|
|
- **Services:** `robot.service` runs `ros2 launch hermes_bringup robot.launch.py` as `hermes`: the face,
|
|
camera, face detector, audio levels, MCP bridge and thermal guard, each its own process respawned by
|
|
the launch file. The face starts first; the others follow `start_delay` (6 s). Parameters:
|
|
`ros/hermes_bringup/config/robot.yaml`. `RuntimeDirectory=robot` gives lgpio a writable working
|
|
directory (it drops a pipe in its cwd; systemd's default `/` is not writable). Also
|
|
`robot-cpu-performance.service` (governor `performance`; the face costs about half the CPU at full
|
|
clock), `robot-firewall.service`, `robot-usb-gadget.service`. `pi/trim-boot.sh` disables snapd and
|
|
ModemManager (unused; ~9 s of boot CPU).
|
|
- **Boot:** the face draws about 28 s after the kernel starts (kernel ~7 s, `robot.service` ~10 s into
|
|
userspace, `ros2 launch` itself ~10 s on a cold Pi, the face ~3 s); the other nodes are up by ~37 s.
|
|
The owner is fine with up to a minute and prefers everything in ROS: a split face unit that skipped
|
|
`ros2 launch` saved only 3.7 s and was dropped. Reboot-tested: everything returns by itself.
|
|
- **CPU**, full stack, 4 cores: face ~84%, camera ~63% (15 fps; 30 fps alone is ~40%, 15 is ~20%),
|
|
detector ~45%, audio ~15%, MCP ~15%, PipeWire ~12%; about 40% of the machine idle, 60 °C, no
|
|
throttling. OpenCV's DNN spins every core and burns 3-4x its elapsed time in CPU, so the detector pins
|
|
it to one thread (YuNet 86 ms, SFace 161 ms of CPU), decodes the JPEG at half size and re-identifies a
|
|
tracked face at most once a second.
|
|
- **ROS never leaves the Pi** (owner's decision): `ROS_AUTOMATIC_DISCOVERY_RANGE=LOCALHOST` (ROS's
|
|
setup script exports `SUBNET`, so `pi/robot-env.sh` assigns it) and the firewall drops domain 42's DDS
|
|
ports (17900-18149) on every interface but loopback. `hermes_mcp` is the only bridge out. This
|
|
supersedes HANDOFF's `host/ros-shell`.
|
|
- **Firewall** (`pi/nftables-robot.nft`, loaded at boot before networking by `robot-firewall.service`):
|
|
accepts 4713 (audio), 8765 (MCP) and 8766 (reserved, nothing listens) only on `usb0` from `10.55.0.1`
|
|
and drops them elsewhere. SSH is deliberately open on all interfaces (key-only), for V2: the owner's call.
|
|
- **PipeWire** 1.0.5 / WirePlumber 0.4.17 run as `hermes`'s user services (`loginctl enable-linger`).
|
|
`pi/pipewire/50-network-server.conf` makes pipewire-pulse listen on TCP 4713 on all interfaces; the
|
|
firewall is the protection. Defaults are the WM8960 source and sink.
|
|
|
|
## Display and the face (`ros/hermes_face`)
|
|
|
|
- **Display:** the ST7789 works on `lgpio` through Blinka (`rpi-lgpio`, not `RPi.GPIO`). Pass `cs=None`
|
|
(the kernel drives CE0; claiming GPIO8 raises "GPIO busy"). The 240x240 glass sits 80 rows into the
|
|
controller's RAM, `y_offset=80`, for every rotation (the driver rotates in software and leaves the
|
|
scan direction fixed). The mounted pose (microSD end up, USB-C edge on the left) needs the content
|
|
turned 90° clockwise: `rotation=270`. The face pushes raw RGB565 with `_block` at ~27-28 fps.
|
|
- **Renderer** (`render.py`, no ROS imports): a numpy port of `docs/face-prototype.html`, flat face only
|
|
(head and low-poly shapes are not ported). Phosphor and vector styles; tone map is a lookup table,
|
|
glow at half resolution, rotation a free `np.rot90`; parameters are clamped. States: `sleeping idle
|
|
listening thinking looking speaking`; `looking` returns to the state it interrupted.
|
|
- **Node** (`face_node.py`): subscribes `/hermes/state`, `/joint_states`, `/vision/faces`,
|
|
`/camera/camera_info`, `/audio/activity`, `/audio/scope/{mic,speaker}`; starts in `sleeping`; logs
|
|
every state change (`journalctl -u robot`), so behaviour can be checked without watching the panel.
|
|
`fps` can be changed live (the thermal guard does).
|
|
- **State arbitration** (`arbiter.py`): sleeping (agent side only) > speaking > listening > thinking >
|
|
idle. **Room sound never wakes a sleeping face.**
|
|
- **Agent heartbeat:** `/hermes/state` doubles as the heartbeat. After `agent_timeout_s` (10) without
|
|
it the face logs `agent gone, sleeping` and sleeps.
|
|
- **Gaze:** eyes glance toward the best face (a named one first, then the biggest), smoothed, held
|
|
2.5 s, then drift back; sleeping ignores it. Image x flips on the display (a person on the image's
|
|
right is on the robot's right, the viewer's left). The renderer's pan sign was wrong (inherited from
|
|
the prototype) and is fixed: positive pan = the robot's left = the viewer's right; positive tilt = up,
|
|
as the MCP `turn_head` documents. Never seen with a moving head.
|
|
- **Voice trace:** while fresh scope data exists, the mouth draws the real waveform (speaker when
|
|
speaking, mic when listening), auto-scaled; the owner confirmed it follows how close and loud they are.
|
|
- Not done: head/low-poly shapes, DotStar LEDs, a button mute. Open: the IPS panel's black reads dark
|
|
gray on camera at full backlight; dimming needs a software PWM on GPIO26.
|
|
|
|
## Camera and vision (`ros/hermes_vision`, `camera_ros`)
|
|
|
|
- **Camera: only the Raspberry Pi libcamera fork works.** The OV5647 is detected (`ov5647 10-0036`), but
|
|
Ubuntu's libcamera 0.2.0 dies in the Pi IPA (`assertion "it != buffers_.end()" in prepareIsp()`) and
|
|
ROS Jazzy's 0.7.2 with `FATAL … A list of V4L2 controls requires a ControlInfoMap`, deterministically
|
|
for every format and size. The fork (commit `6c1dd9d`, built with `-Dpipelines=rpi/vc4 -Dipas=rpi/vc4
|
|
-Dv4l2=false …` into `/opt/libcamera-rpi`) captures; `camera_ros` 0.7.0 is built from source against
|
|
it, and the fork must come first in `LD_LIBRARY_PATH` (`robot-env.sh` does it). Stream: 1296x972 at
|
|
15 fps (`FrameDurationLimits` in `robot.yaml`); `/camera/image_raw`, `…/compressed`, `camera_info`.
|
|
`pi/capture_still.py` takes a still with the fork's `cam`. Frames are padded and B,G,R in memory.
|
|
- **Calibration done** (`pi/calibration/…_1296x972.yaml`, restored by the installer): 18 of 24 views,
|
|
reprojection error 0.296 px, focal length ~1153 px, centre (653, 463), small distortion with `k3` held
|
|
at zero. The first pass fit 0.41 px but with cancelling k1/k2/k3 because the board never reached the
|
|
edges; the tool now fixes `k3`, reports coverage, drops blurred outlier views and can re-solve saved
|
|
views (`--from-views`). **The top-left and bottom-right corners were never reached**, so the model is
|
|
least certain there. The result is valid for the current focus and image size; a changed focus
|
|
invalidates it. Tool: `python3 -m hermes_vision.calibrate --cols 9 --rows 6 --square-mm <mm>`
|
|
(`docs/checkerboard-9x6.png` can be shown on a screen; the square size only scales distance, not the
|
|
intrinsics).
|
|
- **Detection** (`detect.py`, no ROS imports): OpenCV YuNet 2023mar (Haar fallback). It needs OpenCV
|
|
4.8+ (apt has 4.6), so `~/venv` carries `opencv-python-headless==4.10.0.84` with `numpy<2` (the system
|
|
NumPy stays 1.26.4 for ROS), and the venv is first on `PYTHONPATH`; input sides must be multiples of 32.
|
|
- **Tracking** (`tracking.py`): faces are matched across frames by box overlap, so a person keeps one
|
|
track id; identity is re-resolved at most once a second.
|
|
- **Node** `face_detector` publishes `/vision/faces` (`vision_msgs/Detection2DArray`): boxes in
|
|
full-size camera pixels, `Detection2D.id` = the track id, `results[0]` the face, `results[1]` the
|
|
identity if recognised (class_id = name, score = cosine similarity, not a probability). An unknown
|
|
face has no `results[1]`: a name is never made up. `rate_hz` (4) can be changed live.
|
|
- **Recognition is opt-in** (`recognize.py`, `people.py`): SFace embeddings on YuNet's landmarks. Nothing
|
|
is stored until someone is enrolled with `python3 -m hermes_vision.people enroll NAME`; only
|
|
128-number embeddings are kept, in `~/.local/share/hermes/faces.json` (mode 0600, dir 0700, written
|
|
atomically), never images. `list`, `forget`, `check`, `enroll-images` too. **The owner is enrolled
|
|
(10 samples) and was recognised in 29 of 31 detections live (similarity ~0.9).** A throwaway-database
|
|
test recognised 8 of 9 unenrolled frames of the same person, down to side profiles; an unrelated photo
|
|
scored 0.16. Not tested: a second person live. Names are never logged (a count is).
|
|
|
|
## Audio (`ros/hermes_audio`, `pi/overlays`, PipeWire)
|
|
|
|
- **The stock `wm8960-soundcard` overlay fails on Ubuntu 24.04** (`wm8960: No MCLK configured`): it puts
|
|
the 12.288 MHz fixed clock on the card's codec link, but the codec driver reads `mclk` from its own
|
|
node. `pi/overlays/hermes-wm8960.dts` adds it there; `pi/install-audio-overlay.sh` compiles it, sets
|
|
`dtparam=audio=off` and `dtoverlay=hermes-wm8960`. Capture then opens at 8-48 kHz. Address the card
|
|
by name (`hw:CARD=wm8960soundcard,DEV=0`): its index changes.
|
|
- **The HAT's hardware Audio ON/OFF switch (right edge) must be ON.** With it off, the codec still
|
|
answers on I²C and every software check looks right, but all recordings are exactly zero.
|
|
- **Mixer:** `Left/Right Input Mixer Boost` default to off and must be on. `pi/wm8960_asound.state` holds
|
|
the settings (capture 47, boost 1). **WirePlumber resets the codec's capture gain to +30 dB** when it
|
|
takes the device, so the installer sets PipeWire's source volume to 63% (its scale is cubic: -12 dB,
|
|
ALSA `Capture` 46, about +17 dB), which WirePlumber remembers; survives reboots (checked). Quiet-room
|
|
floor at this gain: about -43 dBFS; singing at arm's length peaks around -18 dBFS.
|
|
- **Node** `audio_state` taps the default source and sink monitor with `parec` (16 kHz mono) and
|
|
publishes `/audio/levels`, `/audio/activity` (speaking | listening | quiet) and trigger-locked
|
|
waveform windows `/audio/scope/{mic,speaker}` (only while listening or speaking). Audio itself never
|
|
goes through ROS. `Activity`: 3 s start-up calibration of an adaptive noise floor, margin 7 dB, 1.2 s
|
|
hold so pauses between words do not flicker, steady noise is absorbed after 8 s, the mic is ignored
|
|
while speaking. **The flicker retune is unit-tested but not re-confirmed live.**
|
|
- Verified: the `speaking` path (a tone played to the sink, no speaker attached, shows on the monitor);
|
|
the mic path with the owner's voice. Playback to a real speaker, echo cancel (`pi/pipewire/
|
|
60-echo-cancel.conf.optional`) and the host's `robot_mic` naming wait for the speakers.
|
|
|
|
## The MCP bridge and Hermes (`ros/hermes_mcp`)
|
|
|
|
- Serves MCP (SDK 2.3, streamable HTTP) on `http://10.55.0.2:8765/mcp`: the USB address only (it retries
|
|
the bind until the link is up), bearer token in `/etc/robot-mcp.env` (root:hermes, mode 640, made by
|
|
the installer, never printed; read it with `sudo cat`). **Tool names are the stable contract with
|
|
Hermes.** Tools: `look(send_image="always"|"never")` (the faces summary as JSON text plus the next
|
|
fresh JPEG, never a stale frame; it also makes the face squint), `faces()` (the faces in view, no
|
|
photo: `track_id`, `bbox`, `confidence`, `seen_for_s`, `identity` `{name, similarity}` or `null`),
|
|
`list_people()` (names and sample counts only), and `turn_head` / `head_position` **only when a head
|
|
is fitted** (`head: false` now; the node refuses `head: true` until `hermes_head` exists). Resource
|
|
`robot://status` (hardware present or absent, what is detected, enrolled count, tools, and health:
|
|
temperature, thermal level, throttle flags now and since boot) and prompts `who_is_here`,
|
|
`look_around` make it discoverable. Sessions are stateful on purpose (see the Hermes note below).
|
|
- **Heartbeat:** while an authenticated client is connected (in flight, or heard within
|
|
`connected_grace_s` = 30) it republishes `idle` on `/hermes/state` once a second; the face wakes on
|
|
connect and sleeps about 40 s after the client leaves.
|
|
- **Hermes on talos is wired to it** (with the owner's authorisation). Hermes Agent v0.21.5 is in
|
|
`~/.hermes` (hosted main model `grok-4.6` via xai-oauth; vision is auxiliary and local,
|
|
`qwen3-vl-8b-abliterated` on straylight; the `ornith-*` models are the local ones on straylight and
|
|
dixie). `config.yaml` has an `mcp_servers.robot` block and `.env` has `ROBOT_MCP_TOKEN`; backups:
|
|
`config.yaml.bak-20261005T002659-robot-mcp`, `.env.bak-20261005T002659-robot-mcp`. `hermes mcp test
|
|
robot` passes. A fresh `HERMES_HOME` hangs on first-run setup, so test against the real one.
|
|
- **Real agent turns, entirely on local models:** `hermes chat -q "..." --oneshot -Q --provider
|
|
custom:llama-router -m ornith-1.5-35b-a3b --max-turns 5`. `look()` works; **Hermes does not hand MCP
|
|
images to the main model**: it saves the JPEG to `~/.hermes/cache/images/` on talos and returns a short
|
|
reference, and the model calls `vision_analyze` (local vision model, ~29 s). So a look-and-describe turn
|
|
takes ~70 s, and each look leaves a photo in talos's Hermes cache (Hermes's retention). `faces()` is
|
|
the cheap path: 0.03 s, no image, and Hermes answered "Kyle is at the desk, similarity 0.78".
|
|
- **Hermes's feedback (`~/.hermes/cache/scratch/hermes-bot-v0-face-identity.md`).** Built: `faces()`,
|
|
`look` returning that JSON, `list_people`, `robot://status` and prompts (previously empty), track ids.
|
|
**Declined, deliberately:** embeddings in responses (biometric templates, meaningless to an LLM; the
|
|
registry stays on the robot); age/gender/hat (no model, and unreliable and sensitive); auto-updating
|
|
face data from new captures (lets a stranger slowly become "Kyle"); a remote `enroll` tool (a prompt
|
|
could make the robot call anyone "Kyle": enrolment stays the owner's step; a gated tool, off by
|
|
default, is possible if wanted); detecting people, objects or motion (only faces exist). `look()`'s
|
|
default stays photo-plus-faces, because the photo is what lets Hermes describe a scene: **flipping it
|
|
to JSON-only is the owner's call.**
|
|
- **Hermes bug report "faces and list_people are not registered" was not a bridge bug.** A raw client and a
|
|
fresh `hermes mcp test` list all three; Hermes's own `agent.log` shows it registering all seven tools
|
|
at 01:18:46 after a robot restart, and a later session calling `faces()` fine. The reporting session
|
|
started at 00:35, before `faces` existed, and keeps a start-time snapshot of its tool table (live calls
|
|
like `list_prompts` still work, which matches the report). Fix: start a new Hermes session after the
|
|
robot's tool set changes; Hermes should refresh a session's tool table on re-registration. The bridge
|
|
keeps stateful sessions on purpose: the `Session not found` after a restart is what makes Hermes
|
|
reconnect and re-register.
|
|
|
|
## Thermal guard (`ros/hermes_thermal`)
|
|
|
|
The Pi has no fan. `thermal_guard` reads the CPU temperature and `vcgencmd get_throttled`, publishes
|
|
`/health/temperature`, `/health/thermal_level` (ok | warm | hot | critical) and `/health/throttled`, and
|
|
sheds load through the target nodes' live parameters: the camera's `FrameDurationLimits`, the face's `fps`,
|
|
the detector's `rate_hz`. Levels enter at 72 / 77 / 82 °C (the Pi throttles at ~80); heating is acted on
|
|
at once, cooling goes one level at a time only below the threshold minus 3 °C and after a 20 s dwell.
|
|
Rates scale by 1.0 / 0.7 / 0.45 / 0.2 with floors (camera 2 fps, face 8, detector 1 Hz). Its "normal"
|
|
rates are its own parameters, and a bringup test asserts they equal the real ones in `robot.yaml`, so
|
|
restoring is exact and survives a restart. `fake_temp_c` (`ros2 param set /hermes_thermal fake_temp_c
|
|
85.0`, -1 to stop) forces a temperature. **Verified live** by forcing 73 / 78 / 85 °C and back: measured
|
|
camera rate 16.5 -> 11.5 -> 7.2 -> 3.5 Hz and back, cooling stepped down one dwell apart, same process
|
|
throughout. **A live bug the unit tests missed:** one log line that warned when heating and informed when
|
|
cooling raised rclpy's `Logger severity cannot be changed between calls` and killed the guard on its
|
|
first cooling step (the respawn hid it). Fixed (separate call sites), the tick now survives any error, and
|
|
a test drives the real node through heat-then-cool; a mutation check confirmed it fails on the original
|
|
bug. Not yet tested under a real thermal load.
|
|
|
|
## USB link (HANDOFF package 0b)
|
|
|
|
- Pi: `dtoverlay=dwc2,dr_mode=peripheral`; `robot-usb-gadget.service` builds a configfs gadget (NCM `usb0`
|
|
+ ACM `ttyGS0`; host MAC `02:68:65:72:6d:01`, robot MAC `…:02`) before networking;
|
|
`/etc/netplan/60-usb0.yaml` gives `usb0` `10.55.0.2/30` with no gateway (the default route stays on
|
|
Wi-Fi). NCM works on the Pi 4 (ECM was not needed). **netplan must match `usb0` by name, not MAC:** a
|
|
`macaddress:` match becomes `PermanentMACAddress=`, which a gadget interface lacks, so it stays unmanaged.
|
|
- Host (Omarchy, NetworkManager): `sudo host/omarchy/robot-usb-link.sh` adds profile `hermes-robot-usb`
|
|
(`10.55.0.1/30`, matched by the MAC, never-default). Ping ~2 ms.
|
|
- **Wi-Fi-off acceptance passed:** with Wi-Fi blocked (`rfkill block wifi`, remembered across reboots),
|
|
live and after a reboot, the Pi came up on the USB link alone and everything worked. With no internet
|
|
there is no NTP, so the clock is unsynchronised (roughly right).
|
|
- Not yet verified: replug and second-host recovery, and the serial console login (talos's `/dev/ttyACM0`
|
|
is `root:uucp`; `hermes` has no password, so the console shows a prompt but cannot log in; no autologin
|
|
by design).
|
|
|
|
## Owner's decisions
|
|
|
|
ROS never leaves the Pi (`hermes_mcp` is the only bridge); SSH stays open for V2; NixOS host config is
|
|
skipped for now; no remote `enroll`; `look()`'s default is open; the face may take up to a minute to
|
|
appear at boot; everything should be ROS. A new Omarchy host with a GPU (it runs the local `ornith`
|
|
model) is arriving: it needs only `host/omarchy/robot-usb-link.sh` and its SSH key on the Pi.
|
|
|
|
## Gotchas
|
|
|
|
- `ros2 node list` with its default short window shows only some nodes: use `--no-daemon --spin-time 6`.
|
|
- A one-off message published before ROS discovery matches is lost (volatile QoS): republish, as
|
|
`hermes_mcp` does every second. Anything that sets `PYTHONPATH` by hand must prepend, not replace.
|
|
- A shell that sourced `robot-env.sh` before a build cannot import the newly built package: re-source.
|
|
- rclpy needs each logging call site to keep one severity.
|
|
|
|
## Arrival day: the neck (servos), planned and deliberately not built ahead of the hardware
|
|
|
|
Expected Thursday or Friday at the earliest. What matters cannot be faked (which `pwmchip` the overlay
|
|
creates, its permissions on Ubuntu, each servo's real pulse range, whether they brown out the Pi), so
|
|
`hermes_head` waits. Order, with what to verify:
|
|
|
|
1. **Wire it:** pan servo to the HAT's GPIO12 port, tilt to GPIO13; the "3pin Volt" jumper is already
|
|
verified at 5 V. Servo spikes on a host USB port may brown the Pi out: use the 5.1 V 3 A supply
|
|
through the splitter if it has arrived. Horns off for the first moves.
|
|
2. **Overlay (needs a reboot, ask first):** add `dtoverlay=pwm-2chan,pin=12,func=4,pin2=13,func2=4` under
|
|
`[all]` in `config.txt` (`dtparam=audio=off` is already set: the Pi's headphone audio shares PWM
|
|
hardware). Add it to `pi/install-*.sh`, not by hand.
|
|
3. **sysfs:** find the chip (`/sys/class/pwm/pwmchip*`; do not assume 0), export channels 0 and 1,
|
|
`period=20000000`. A udev rule should give a group, not root, write access.
|
|
4. **Measure each servo:** centre (1.5 ms), then the safe min and max pulse with the horn off; set
|
|
`ROBOT_SERVO_MIN_US` / `MAX_US` per servo (start 500 / 2400), the trims and invert flags.
|
|
5. **Write `hermes_head`:** `PwmHead` on sysfs, a ~0.3 s interpolated move, PWM off after ~1 s idle,
|
|
limits (pan 80, tilt 45), `/joint_states` for `pan_joint` / `tilt_joint` in radians (positive pan =
|
|
the robot's left, positive tilt = up). Unit-test against a fake sysfs tree, then verify on the Pi:
|
|
`vcgencmd get_throttled` stays `0x0` through a full sweep with the face running; watch the
|
|
under-voltage flag in `robot://status`.
|
|
6. **Wire it in:** give the bridge a head backend that calls `hermes_head`, set `head: true`, check
|
|
`turn_head` / `head_position` appear to a new Hermes session, and that the face's eyes follow the real
|
|
joints the right way.
|
|
7. **URDF:** `hermes_description` from the enclosure STLs' `pose_*` parts, frames `base_link`, `pan_link`,
|
|
`tilt_link`, `camera_link`, `camera_optical_frame`.
|
|
8. **Mounted camera:** set the flip (`orientation`) for how it is mounted, re-check the focus (a changed
|
|
focus invalidates `pi/calibration/`), and the display rotation if it is mounted differently.
|
|
|
|
## Open items
|
|
|
|
1. Waiting on hardware: servos (above), speakers (playback, echo cancel, Hermes talking aloud, the host's
|
|
`robot_mic` naming), the fan, the splitter, the printed enclosure (measure first: `enclosure/README.md`).
|
|
2. Unconfirmed: the retuned listening detector live; a second person in frame; replug and second-host
|
|
recovery; the serial console; `agent_timeout_s` over a long idle Hermes session; the calibration's
|
|
uncovered corners (retake with the board in the top-left and bottom-right); `pi/install.sh` on a
|
|
blank card (back up the working SD card first: it holds all of this).
|
|
3. Optional: a gated `enroll` tool, if the owner wants one.
|
|
|
|
## Repository
|
|
|
|
Branch `master`. `.gitignore` covers `.remember/`, caches, `.venv/`, `*.orig`, `*.rej` and
|
|
`pi/50-cloud-init.yaml`. `ros/` has `hermes_face`, `hermes_vision`, `hermes_audio`, `hermes_mcp`,
|
|
`hermes_thermal` and `hermes_bringup`; `pi/` is everything installed on the Pi; `host/` is the owner's
|
|
machines (`host/omarchy/`: the USB link script, Hermes and PipeWire config, the launcher;
|
|
`host/check-pi-platform.sh`); `enclosure/`; `docs/` (`face-prototype.html`, `checkerboard-9x6.png`).
|
|
Run a package's tests on the Pi with `python3 -m pytest test` in its directory after `source
|
|
~/robot-env.sh` (the face suite takes about a minute).
|
|
|
|
## Security notes (all resolved)
|
|
|
|
The automated review of the initial commit flagged `ROBOT_MCP_BIND` defaulting to `0.0.0.0` (now the USB
|
|
address only, with a bearer token), PipeWire on TCP 4713 on every interface (now behind the boot-time
|
|
firewall, which loads first), and a single-IP any-interface nftables file (rewritten). Later reviews led
|
|
to: DDS blocked on every interface and local-only discovery; recognised names never logged; node
|
|
parameters clamped. The token and the owner's Wi-Fi password never go through the repo or logs (one
|
|
`netplan get` once printed the Wi-Fi password into a session; it was flagged to the owner).
|