hermes-bot/CONTEXT.md

# Current state

Last updated 2026-10-04. `HANDOFF.md` is the brief; this file tracks where the work stands.

## Hardware on hand

Raspberry Pi 4 (8 GB), BrainCraft HAT, Arducam UC-346 camera. Not yet: servos, USB-C splitter,
printed enclosure, speakers. Work until those arrive is limited to what the Pi, HAT and camera can do.

## The Pi (`hermes-bot`)

- Ubuntu 24.04.5 server, kernel `6.8.0-1065-raspi` (fully upgraded), flashed from `talos`. User
  `hermes` (key-only; passwordless sudo), SSH keys for `kyle@vade` (talos) and `kyle@straylight`.
- On Wi-Fi at `192.168.88.152` (MAC `d8:3a:dd:0d:66:f5`); `eth0` is not cabled. The owner says it has
  a static IP on the rack switch; that address is not recorded here yet. **USB link: `10.55.0.2`**
  (see package 0b below); `ssh hermes@10.55.0.2` works from talos. `avahi-daemon` is
  installed, but mDNS is off in `systemd-resolved` on talos, so `hermes-bot.local` doesn't resolve
  there; use the IP.
- **cloud-init applies `network-config` and `user-data` on the first boot only.** To change Wi-Fi
  later, edit `/etc/netplan/50-cloud-init.yaml` on the Pi, or bump `instance_id` in `meta-data` on
  `system-boot`. `pi/50-cloud-init.yaml` is a template for the Wi-Fi block; it is gitignored
  because it holds credentials.
- Installed: ROS 2 Jazzy (`ros-base`, `ros-dev-tools`, colcon, `robot_state_publisher`, `xacro`,
  `camera_ros`); `python3-numpy/pil/spidev/lgpio/rpi-lgpio`; a venv `~/venv` (system site packages
  visible) with `adafruit-circuitpython-rgb-display` and Blinka. Workspace: `~/ros_ws`, source synced
  from `ros/` with `rsync -a --delete --exclude __pycache__ ros/ hermes@<ip>:~/ros_ws/src/`, then
  `colcon build --symlink-install`.
- CPU governor is held at `performance` by `robot-cpu-performance.service` (the face costs about
  half as much CPU at full clock). The face starts at boot: `robot-face.service` runs
  `ros2 run hermes_face face_node` as `hermes` and restarts it on failure (it needs a writable
  working directory, `RuntimeDirectory=robot-face`, because lgpio drops a pipe in its cwd). It comes
  up in the `sleeping` pose. Install with `pi/install-face-service.sh`.
- **ROS never leaves the Pi** (owner's decision): `ROS_AUTOMATIC_DISCOVERY_RANGE=LOCALHOST` (set in
  `pi/robot-env.sh`, which `source`s everything) and the firewall drops domain 42's DDS ports
  (17900-18149) on every interface but loopback. `hermes_mcp` is the only bridge to the outside; a
  host that wants data from the robot gets it through an MCP tool, not DDS. This supersedes
  HANDOFF.md's `host/ros-shell` talking to the robot. (ROS's setup script exports `SUBNET`, so the
  variable must be assigned, not defaulted.)
- **PipeWire** (1.0.5, WirePlumber 0.4.17) runs as `hermes`'s user services (`loginctl enable-linger`).
  `~/.config/pipewire/pipewire-pulse.conf.d/50-network-server.conf` (from `pi/pipewire/`) makes
  pipewire-pulse listen on TCP 4713 on all interfaces; the firewall is the protection. Defaults are
  the WM8960 source and sink (`alsa_input/output.platform-soc_sound.stereo-fallback`). Verified from
  talos: `pactl -s tcp:10.55.0.2:4713 info` works; the same on the Wi-Fi address times out.

## Package 0a (platform proof on Ubuntu 24.04)

- Step 1 done: `host/check-pi-platform.sh` passes.
- Step 2 done: flashed, booted, upgraded; ROS installed; `ros2 topic list` works.
- **Step 5 (display) done.** ST7789 works on `lgpio` through Blinka (`rpi-lgpio`, no `RPi.GPIO`).
  Two gotchas: pass `cs=None` (the kernel drives CE0; claiming GPIO8 raises "GPIO busy"), and the
  240x240 glass sits at `y_offset=80` in the controller's RAM, for every rotation (the driver
  rotates in software and leaves the scan direction fixed). Mounted pose (microSD end up, USB-C edge
  on the left) needs content turned 90 degrees clockwise: `rotation=270`. Full-frame push is
  ~23 fps through the driver; the face pushes raw RGB565 with `_block` and reaches ~27-28 fps.
- **Camera: works with the Raspberry Pi libcamera fork, not with the stock stacks.** The OV5647 is
  detected (`ov5647 10-0036`) and Ubuntu's libcamera 0.2.0 lists it once `libcamera-ipa` is
  installed, but capture dies in the Pi IPA (`assertion "it != buffers_.end()" in prepareIsp()`),
  and ROS Jazzy's libcamera 0.7.2 dies with `FATAL … A list of V4L2 controls requires a
  ControlInfoMap` right after `configure`. Both are deterministic whatever the stream format or
  size. Fix: build the fork (`github.com/raspberrypi/libcamera`, commit `6c1dd9d`, 2026-08-17) with
  `-Dpipelines=rpi/vc4 -Dipas=rpi/vc4 -Dcam=enabled -Dv4l2=false …` into `/opt/libcamera-rpi`; its
  `cam` then captures 1296x972 RGB888 at ~16 fps (dim room). `pi/capture_still.py` takes a still
  with it. **`camera_ros` still has to be built against this fork** (the apt one uses ROS's broken
  libcamera). Frames are padded (stride 3936) and B,G,R in memory.
- **`camera_ros` 0.7.0 is built from source against the fork** in `~/camera_ws` (overlay on ROS; at
  runtime the fork must come first in `LD_LIBRARY_PATH`, which `source ~/robot-env.sh` does; copy
  from `pi/robot-env.sh`). Verified: `ros2 run camera_ros camera_node --ros-args -p width:=1296 -p
  height:=972 -p format:=RGB888` publishes `/camera/image_raw`, `/camera/image_raw/compressed` (~26
  Hz) and `/camera/camera_info`. It costs ~140% of a core with raw and compressed both on; lower the
  frame rate or size before running everything together. No calibration file yet.
- **`ros/hermes_vision`**: `hermes_vision.detect` (YuNet, Haar fallback; no ROS imports) and a node
  `face_detector` that subscribes to the compressed image and publishes `/vision/faces`
  (`vision_msgs/Detection2DArray`, bbox centre/size in pixels, `class_id: face`, score), at `rate_hz`
  (default 4, ~47% of a core on a Pi 4); `publish_debug:=true` adds an annotated image. CLI:
  `python -m hermes_vision.detect_cli IMAGE`. YuNet 2023mar needs OpenCV 4.8+, so `~/venv` has
  `opencv-python-headless==4.10.0.84` with `numpy<2` (system NumPy stays 1.26.4 for ROS), and
  `robot-env.sh` puts the venv first on `PYTHONPATH`; YuNet also needs input sides that are multiples
  of 32. Tested on a sample image and on a live frame (one face found, upright, no flip needed).
- **Audio capture works (verified 2026-10-04).** The stock `wm8960-soundcard` overlay fails every
  stream with `wm8960: No MCLK configured` on Ubuntu 24.04: it puts the 12.288 MHz fixed clock on the
  sound card's codec link, but the codec driver reads `mclk` from its own node.
  `pi/overlays/hermes-wm8960.dts` is the stock overlay plus `clocks`/`clock-names` on the codec node;
  `pi/install-audio-overlay.sh` compiles it to `/boot/firmware/overlays/hermes-wm8960.dtbo`, sets
  `dtparam=audio=off` and `dtoverlay=hermes-wm8960`. Capture then opens at 8-48 kHz. Address the
  card by name (`hw:CARD=wm8960soundcard,DEV=0`): its index changes (it was card 1, then 0).
- **The HAT's hardware Audio ON/OFF slide switch (right edge, by the "Audio" label) must be ON.**
  With it off, the codec still answers on I2C and every software check looks right, but all
  recordings are exactly zero. That cost an hour of debugging; check it first.
- Mixer: `Left/Right Input Mixer Boost Switch` default to off and must be on. Settings in use:
  Capture 47/47 (+18 dB), `Left/Right Input Boost Mixer LINPUT1/RINPUT1` volume 1, input mixer boost
  on. Singing at arm's length: RMS -34 to -49 dBFS, peaks -18 dBFS, no clipping, channels within
  1-2 dB. Saved with `alsactl store` and exported as `pi/wm8960_asound.state`; restoring at boot is
  not yet verified. No speakers on hand, so playback is untested.
- Not done: PWM (waits for servos). Ask before rebooting the Pi.

## Package 0b (USB link): Pi side and talos side working

- Pi: `dtoverlay=dwc2,dr_mode=peripheral`; `robot-usb-gadget.service` builds a configfs gadget
  (NCM `usb0` + ACM `ttyGS0`, host MAC `02:68:65:72:6d:01`, robot MAC `02:68:65:72:6d:02`) before
  networking; `serial-getty@ttyGS0` is enabled; `/etc/netplan/60-usb0.yaml` gives `usb0`
  `10.55.0.2/30` with no gateway, so the default route stays on Wi-Fi. Installed by
  `pi/install-usb-gadget.sh` (a backup of `config.txt` is `config.txt.bak-usbgadget`). NCM works on
  the Pi 4 with kernel 1065; ECM was not needed.
- **netplan gotcha:** match the gadget interface by `name: usb0`. A `macaddress:` match becomes
  `PermanentMACAddress=` in networkd, which a gadget interface lacks, so it stays unmanaged.
- talos (Omarchy, NetworkManager): `host/omarchy/robot-usb-link.sh` (needs sudo) adds profile
  `hermes-robot-usb`: `10.55.0.1/30` on the interface with MAC `02:68:65:72:6d:01`, priority 100,
  never-default. Verified from talos: ping ~2.4 ms, SSH over the link.
- **Firewall:** `robot-firewall.service` loads `pi/nftables-robot.nft` before networking at every
  boot (`pi/install-firewall.sh`; it must be loaded before the PipeWire network server is enabled).
  It accepts 4713/8765/8766 only on `usb0` from `10.55.0.1`, drops them elsewhere, and drops DDS on
  every interface but loopback. Verified: from talos, 4713 connects over `usb0` and times out over
  Wi-Fi, while an unused port on the Pi's Wi-Fi address refuses instantly (so only 4713 is blocked).
  SSH is deliberately left open on all interfaces (key-only), for V2; the owner's call.
- Not yet verified: serial login prompt (talos's `/dev/ttyACM0` is `root:uucp`; use
  `sudo screen /dev/ttyACM0 115200`). The `hermes` account has no password, so the console shows a
  login prompt but cannot log in until the owner sets one; no autologin by design. Also not done:
  the Pi booting with Wi-Fi off, replug and second-host recovery,
  `get_throttled` after a servo sweep on the splitter, the NixOS snippet, and straylight/other-host
  testing. Currently the Pi is powered from talos's USB port (bench test, no servos).

## Face (`ros/hermes_face`)

- `render.py`: numpy port of `docs/face-prototype.html`, **flat face only** (head and lowpoly not
  ported). Phosphor and vector styles. States: `sleeping idle listening thinking looking speaking`.
  `sleeping` (new) is the "no agent connected" pose: eyes closed to thin lines, mouth short, dim and
  slowly breathing. `looking` returns to the state it interrupted. Tone mapping is a lookup table
  (RGB and RGB565 tables), glow is computed at half resolution, rotation is a free `np.rot90`.
- `display.py`: ST7789 driver plus an async pusher (render overlaps the SPI transfer).
- `face_node.py`: node `hermes_face`; subscribes `/hermes/state` (`std_msgs/String`) and
  `/joint_states` (`pan_joint`, `tilt_joint`, radians); starts in `initial_state` (`sleeping`), so it
  runs with nothing else up. Parameters: `rotation` (270), `x_offset` (0), `y_offset` (80),
  `baudrate`, `fps`, `output` (`st7789` or `none`), `phosphor` (P7), `style`, `persistence_ms`,
  `focus`, `jitter`, `intensity`, `initial_state`.
- `demo.py`: no ROS. `python -m hermes_face.demo --sequence` cycles the states on the panel;
  `--png-dir DIR` writes a PNG per state without hardware.
- Tests: `python3 -m pytest test` in `ros/hermes_face` (25 pass on the Pi). Renderer parameters
  (`focus`, `persistence_ms`, `intensity`, `jitter`) are clamped; `fps` is clamped to 1-60.
- Measured on the Pi 4: ~9 ms per frame in isolation. With the CPU at full clock: 24% of a core at
  20 fps and 40% at 30 fps (target was under 40% at 20 fps or more). Under `ondemand` the same
  load reads 40% at 20 fps and ~67% at 30 fps. No C++ rewrite was needed.
- **Gaze:** the node subscribes to `/vision/faces` (and `/camera/camera_info` for the image size)
  and the eyes glance toward the best face: a recognised one first, then the biggest. Image x flips
  on the display (a person on the image's right is on the robot's right, the viewer's left); the
  glance is smoothed, held 2.5 s, then drifts back to centre. Sleeping ignores it. The pan sign in
  the renderer was wrong (carried over from the prototype) and is fixed: positive pan = the robot's
  left = the viewer's right, positive tilt = looks up, as the MCP `turn_head` tool documents.
  Not yet confirmed by eye on the real display.
- Not yet: audio levels for the mouth (`/audio/levels`), `hermes_mcp` publishing states, launch
  file in `hermes_bringup`, head/lowpoly shapes, DotStar LEDs, button mute.
- Open: IPS black level. The backlight is on at full brightness, so black reads dark gray on
  camera. Dimming it needs a software PWM on GPIO26 (not a hardware PWM pin).

## Audio levels and the literal voice trace (`ros/hermes_audio`, face integration)

- `hermes_audio` node `audio_state` taps the default source and the default sink's monitor with
  `parec` (16 kHz mono) and publishes `/audio/levels` ([mic_dbfs, speaker_dbfs]), `/audio/activity`
  (`speaking` | `listening` | `quiet`) and trigger-locked waveform windows `/audio/scope/{mic,speaker}`
  (256 samples). Audio itself never goes through ROS. `hermes_audio.levels.Activity` debounces and
  keeps an adaptive noise floor: a 3 s start-up calibration, then it drops fast and rises slowly, and
  anything 'listening' without a gap for 8 s is absorbed as ambient noise. The microphone is ignored
  while speaking (it would hear the robot).
- The face arbitrates (`hermes_face.arbiter`): sleeping (agent side only) > speaking > listening >
  thinking > idle. **Room sound never wakes a sleeping face**: sleeping means "no agent connected".
  `looking` stays a timed event owned by the renderer. While fresh (< 0.35 s) scope data exists, the
  mouth draws the real waveform (speaker when speaking, mic when listening), auto-scaled.
- **WirePlumber resets the codec's capture gain** to 100% (+30 dB) when it takes the device,
  overriding `alsactl`'s saved level. Fix: `pactl set-source-volume @DEFAULT_SOURCE@ 63%` (PipeWire's
  scale is cubic: 63% is -12 dB from the +30 dB maximum, so ALSA `Capture` lands at 46, about +17
  dB). WirePlumber remembers it; recheck after a reboot. The quiet-room floor at this gain is about
  -43 dBFS (preamp and mic noise).
- Verified on the Pi: the `speaking` path (a tone played to the sink, with no speaker attached, shows
  on the monitor at -13.5 dBFS, `speaking` for its length, mic ignored); levels and scope publish;
  47 face tests, 8 audio tests. **Not yet verified:** the microphone-to-mouth path with a real voice
  (one live attempt saw no speech: the cue and the talking did not overlap), and the audio node as a
  service (it was run as a transient unit).

## Face recognition (opt-in)

`hermes_vision.recognize` / `people`: SFace embeddings on YuNet's landmarks. Nothing is stored until
someone is enrolled (`python -m hermes_vision.people enroll NAME`, live from the camera topic, or
`enroll-images`); only 128-number embeddings are kept, in `~/.local/share/hermes/faces.json` (mode
0600, directory 0700, written atomically), never images. `list`, `forget NAME` and `check IMAGE`
manage and test it. The detector node puts a recognised name in `Detection2D.id` and picks up
database changes without a restart; its log prints a count, not names. Checked with a throwaway
database: 8 of 9 unenrolled frames of the same person were recognised, down to side profiles
(similarity 0.52-0.66), the miss being a head tipped back (0.34 against the 0.363 threshold); an
unrelated photo scored 0.16. **Nobody is enrolled yet**; the owner enrols themselves live.

## Next

1. Finish package 0b: Wi-Fi-off boot test, replug recovery, serial login check, NixOS snippet in
   `host/nixos/`. A new Omarchy host with a GPU (runs the local
   model "ornith") arrives this week: it needs only `host/omarchy/robot-usb-link.sh` and its SSH key
   added to the Pi.
2. Audio: the `hermes_audio_state` node (levels, speaking/listening, a trigger-locked scope for the
   literal voice trace), echo cancel and `robot_mic` naming once speakers arrive. Then a launch file
   that brings up camera, vision, face and audio together, and CPU tuning (camera ~140% of a core).
3. Second-person detection test; enrol the owner live; the rest of `HANDOFF.md`.

## Repository

Branch `master`. `.gitignore` covers `.remember/`, caches, `.venv/`, `*.orig`, `*.rej` and
`pi/50-cloud-init.yaml`. `tools/check-pi-platform.sh` duplicates `host/check-pi-platform.sh` and can
probably go.

## Security review notes (parked until the USB link is up)

An automated review of the initial commit flagged three items. All stem from the code predating
the USB-gadget design (HANDOFF.md package 0b); none is exposed until the Pi runs the services.

- `pi/robot_mcp_server.py`: plain HTTP, `ROBOT_MCP_BIND` defaults to `0.0.0.0`. Plan: default to
  `10.55.0.2`, start after `usb0`.
- `pi/pipewire/50-network-server.conf`: TCP 4713 on all interfaces, unrestricted. Plan: keep,
  comment that the firewall is the protection.
- `pi/nftables-robot.nft`: still the Pi Zero version (single host IP, any interface). Package 0b
  rewrites it to accept only on the USB interface from `10.55.0.1`.
- `pi/install.sh` permissions looked fine (`/etc/robot-mcp.env` is `root:$USER`, 640).

The owner wants to confirm the real IPs on the link before changing any of this. If the USB
link doesn't work, revisit: TLS or a Tailscale-only bind for MCP and audio.