hermes-bot/CONTEXT.md

# Current state

Last updated 2026-10-04. `HANDOFF.md` is the brief; this file tracks where the work stands.

## Hardware on hand

Raspberry Pi 4 (8 GB), BrainCraft HAT, Arducam UC-346 camera. Not yet: servos, USB-C splitter,
printed enclosure, speakers. Work until those arrive is limited to what the Pi, HAT and camera can do.

## The Pi (`hermes-bot`)

- Ubuntu 24.04.5 server, kernel `6.8.0-1065-raspi` (fully upgraded), flashed from `talos`. User
  `hermes` (key-only; passwordless sudo), SSH keys for `kyle@vade` (talos) and `kyle@straylight`.
- On Wi-Fi at `192.168.88.152` (MAC `d8:3a:dd:0d:66:f5`); `eth0` is not cabled. The owner says it has
  a static IP on the rack switch; that address is not recorded here yet. **USB link: `10.55.0.2`**
  (see package 0b below); `ssh hermes@10.55.0.2` works from talos. `avahi-daemon` is
  installed, but mDNS is off in `systemd-resolved` on talos, so `hermes-bot.local` doesn't resolve
  there; use the IP.
- **cloud-init applies `network-config` and `user-data` on the first boot only.** To change Wi-Fi
  later, edit `/etc/netplan/50-cloud-init.yaml` on the Pi, or bump `instance_id` in `meta-data` on
  `system-boot`. `pi/50-cloud-init.yaml` is a template for the Wi-Fi block; it is gitignored
  because it holds credentials.
- Installed: ROS 2 Jazzy (`ros-base`, `ros-dev-tools`, colcon, `robot_state_publisher`, `xacro`,
  `camera_ros`); `python3-numpy/pil/spidev/lgpio/rpi-lgpio`; a venv `~/venv` (system site packages
  visible) with `adafruit-circuitpython-rgb-display` and Blinka. Workspace: `~/ros_ws`, source synced
  from `ros/` with `rsync -a --delete --exclude __pycache__ ros/ hermes@<ip>:~/ros_ws/src/`, then
  `colcon build --symlink-install`.
- CPU governor is `ondemand`; at full clock the face costs about half as much CPU. Set
  `performance` in the installer (package 5). It is not persistent yet.

## Package 0a (platform proof on Ubuntu 24.04)

- Step 1 done: `host/check-pi-platform.sh` passes.
- Step 2 done: flashed, booted, upgraded; ROS installed; `ros2 topic list` works.
- **Step 5 (display) done.** ST7789 works on `lgpio` through Blinka (`rpi-lgpio`, no `RPi.GPIO`).
  Two gotchas: pass `cs=None` (the kernel drives CE0; claiming GPIO8 raises "GPIO busy"), and the
  240x240 glass sits at `y_offset=80` in the controller's RAM, for every rotation (the driver
  rotates in software and leaves the scan direction fixed). Mounted pose (microSD end up, USB-C edge
  on the left) needs content turned 90 degrees clockwise: `rotation=270`. Full-frame push is
  ~23 fps through the driver; the face pushes raw RGB565 with `_block` and reaches ~27-28 fps.
- **Camera: works with the Raspberry Pi libcamera fork, not with the stock stacks.** The OV5647 is
  detected (`ov5647 10-0036`) and Ubuntu's libcamera 0.2.0 lists it once `libcamera-ipa` is
  installed, but capture dies in the Pi IPA (`assertion "it != buffers_.end()" in prepareIsp()`),
  and ROS Jazzy's libcamera 0.7.2 dies with `FATAL … A list of V4L2 controls requires a
  ControlInfoMap` right after `configure`. Both are deterministic whatever the stream format or
  size. Fix: build the fork (`github.com/raspberrypi/libcamera`, commit `6c1dd9d`, 2026-08-17) with
  `-Dpipelines=rpi/vc4 -Dipas=rpi/vc4 -Dcam=enabled -Dv4l2=false …` into `/opt/libcamera-rpi`; its
  `cam` then captures 1296x972 RGB888 at ~16 fps (dim room). `pi/capture_still.py` takes a still
  with it. **`camera_ros` still has to be built against this fork** (the apt one uses ROS's broken
  libcamera). Frames are padded (stride 3936) and B,G,R in memory.
- **`camera_ros` 0.7.0 is built from source against the fork** in `~/camera_ws` (overlay on ROS; at
  runtime the fork must come first in `LD_LIBRARY_PATH`, which `source ~/robot-env.sh` does; copy
  from `pi/robot-env.sh`). Verified: `ros2 run camera_ros camera_node --ros-args -p width:=1296 -p
  height:=972 -p format:=RGB888` publishes `/camera/image_raw`, `/camera/image_raw/compressed` (~26
  Hz) and `/camera/camera_info`. It costs ~140% of a core with raw and compressed both on; lower the
  frame rate or size before running everything together. No calibration file yet.
- **`ros/hermes_vision`**: `hermes_vision.detect` (YuNet, Haar fallback; no ROS imports) and a node
  `face_detector` that subscribes to the compressed image and publishes `/vision/faces`
  (`vision_msgs/Detection2DArray`, bbox centre/size in pixels, `class_id: face`, score), at `rate_hz`
  (default 4, ~47% of a core on a Pi 4); `publish_debug:=true` adds an annotated image. CLI:
  `python -m hermes_vision.detect_cli IMAGE`. YuNet 2023mar needs OpenCV 4.8+, so `~/venv` has
  `opencv-python-headless==4.10.0.84` with `numpy<2` (system NumPy stays 1.26.4 for ROS), and
  `robot-env.sh` puts the venv first on `PYTHONPATH`; YuNet also needs input sides that are multiples
  of 32. Tested on a sample image and on a live frame (one face found, upright, no flip needed).
- **Audio: card works, microphones read digital silence (open).** The stock `wm8960-soundcard`
  overlay fails every stream with `wm8960: No MCLK configured` on Ubuntu 24.04: it puts the 12.288 MHz
  fixed clock on the sound card's codec link, but the codec driver reads `mclk` from its own node.
  `pi/overlays/hermes-wm8960.dts` is the stock overlay plus `clocks`/`clock-names` on the codec node;
  `pi/install-audio-overlay.sh` compiles it to `/boot/firmware/overlays/hermes-wm8960.dtbo`, sets
  `dtparam=audio=off` and `dtoverlay=hermes-wm8960`. With it, capture opens at 8-48 kHz. Address
  the card by name (`hw:CARD=wm8960soundcard,DEV=0`): its index changes (it was card 1, then 0).
  But every recording is exactly zero (no noise floor), with every mixer path on and gains maxed.
  Checked: DAPM shows ADCs, input mixers, boost mixers and LINPUT1/RINPUT1 on; registers
  `R25=0xfc`, `R47=0x30`, `R32/33=0x108`; MCLK 12.288 MHz and BCLK 1.536 MHz running; GPIO18-21 in
  `alt0` for the I2S; setting `ALRCGPIO` (R9 bit 6) over I2C made no difference. Not yet checked:
  the HAT's hardware **Audio ON/OFF slide switch** position, MICBIAS, a DAC-to-ADC loopback.
  `Left/Right Input Mixer Boost Switch` default to off and must be turned on for the mic path.
  No speakers on hand, so playback is untested.
- Not done: PWM (waits for servos). Ask before rebooting the Pi.

## Package 0b (USB link): Pi side and talos side working

- Pi: `dtoverlay=dwc2,dr_mode=peripheral`; `robot-usb-gadget.service` builds a configfs gadget
  (NCM `usb0` + ACM `ttyGS0`, host MAC `02:68:65:72:6d:01`, robot MAC `02:68:65:72:6d:02`) before
  networking; `serial-getty@ttyGS0` is enabled; `/etc/netplan/60-usb0.yaml` gives `usb0`
  `10.55.0.2/30` with no gateway, so the default route stays on Wi-Fi. Installed by
  `pi/install-usb-gadget.sh` (a backup of `config.txt` is `config.txt.bak-usbgadget`). NCM works on
  the Pi 4 with kernel 1065; ECM was not needed.
- **netplan gotcha:** match the gadget interface by `name: usb0`. A `macaddress:` match becomes
  `PermanentMACAddress=` in networkd, which a gadget interface lacks, so it stays unmanaged.
- talos (Omarchy, NetworkManager): `host/omarchy/robot-usb-link.sh` (needs sudo) adds profile
  `hermes-robot-usb`: `10.55.0.1/30` on the interface with MAC `02:68:65:72:6d:01`, priority 100,
  never-default. Verified from talos: ping ~2.4 ms, SSH over the link.
- Not yet verified: serial login prompt (talos's `/dev/ttyACM0` is `root:uucp`; use
  `sudo screen /dev/ttyACM0 115200`). The `hermes` account has no password, so the console shows a
  login prompt but cannot log in until the owner sets one; no autologin by design. Also not done:
  the nftables rewrite, the Pi booting with Wi-Fi off, replug and second-host recovery,
  `get_throttled` after a servo sweep on the splitter, the NixOS snippet, and straylight/other-host
  testing. Currently the Pi is powered from talos's USB port (bench test, no servos).

## Face (`ros/hermes_face`)

- `render.py`: numpy port of `docs/face-prototype.html`, **flat face only** (head and lowpoly not
  ported). Phosphor and vector styles. States: `sleeping idle listening thinking looking speaking`.
  `sleeping` (new) is the "no agent connected" pose: eyes closed to thin lines, mouth short, dim and
  slowly breathing. `looking` returns to the state it interrupted. Tone mapping is a lookup table
  (RGB and RGB565 tables), glow is computed at half resolution, rotation is a free `np.rot90`.
- `display.py`: ST7789 driver plus an async pusher (render overlaps the SPI transfer).
- `face_node.py`: node `hermes_face`; subscribes `/hermes/state` (`std_msgs/String`) and
  `/joint_states` (`pan_joint`, `tilt_joint`, radians); starts in `initial_state` (`sleeping`), so it
  runs with nothing else up. Parameters: `rotation` (270), `x_offset` (0), `y_offset` (80),
  `baudrate`, `fps`, `output` (`st7789` or `none`), `phosphor` (P7), `style`, `persistence_ms`,
  `focus`, `jitter`, `intensity`, `initial_state`.
- `demo.py`: no ROS. `python -m hermes_face.demo --sequence` cycles the states on the panel;
  `--png-dir DIR` writes a PNG per state without hardware.
- Tests: `python3 -m pytest test` in `ros/hermes_face` (25 pass on the Pi). Renderer parameters
  (`focus`, `persistence_ms`, `intensity`, `jitter`) are clamped; `fps` is clamped to 1-60.
- Measured on the Pi 4: ~9 ms per frame in isolation. With the CPU at full clock: 24% of a core at
  20 fps and 40% at 30 fps (target was under 40% at 20 fps or more). Under `ondemand` the same
  load reads 40% at 20 fps and ~67% at 30 fps. No C++ rewrite was needed.
- Not yet: audio levels for the mouth (`/audio/levels`), `hermes_mcp` publishing states, launch
  file in `hermes_bringup`, head/lowpoly shapes, DotStar LEDs, button mute.
- Open: IPS black level. The backlight is on at full brightness, so black reads dark gray on
  camera. Dimming it needs a software PWM on GPIO26 (not a hardware PWM pin).

## Next

1. Finish package 0b: Wi-Fi-off boot test, replug recovery, serial login check, NixOS snippet in
   `host/nixos/`, and make the firewall load at boot. A new Omarchy host with a GPU (runs the local
   model "ornith") arrives this week: it needs only `host/omarchy/robot-usb-link.sh` and its SSH key
   added to the Pi.
2. Audio (package 1; needs the `wm8960-soundcard` overlay and a reboot), camera (package 3), then
   the rest of the packages in `HANDOFF.md`.

## Repository

Branch `master`. `.gitignore` covers `.remember/`, caches, `.venv/`, `*.orig`, `*.rej` and
`pi/50-cloud-init.yaml`. `tools/check-pi-platform.sh` duplicates `host/check-pi-platform.sh` and can
probably go.

## Security review notes (parked until the USB link is up)

An automated review of the initial commit flagged three items. All stem from the code predating
the USB-gadget design (HANDOFF.md package 0b); none is exposed until the Pi runs the services.

- `pi/robot_mcp_server.py`: plain HTTP, `ROBOT_MCP_BIND` defaults to `0.0.0.0`. Plan: default to
  `10.55.0.2`, start after `usb0`.
- `pi/pipewire/50-network-server.conf`: TCP 4713 on all interfaces, unrestricted. Plan: keep,
  comment that the firewall is the protection.
- `pi/nftables-robot.nft`: still the Pi Zero version (single host IP, any interface). Package 0b
  rewrites it to accept only on the USB interface from `10.55.0.1`.
- `pi/install.sh` permissions looked fine (`/etc/robot-mcp.env` is `root:$USER`, 640).

The owner wants to confirm the real IPs on the link before changing any of this. If the USB
link doesn't work, revisit: TLS or a Tailscale-only bind for MCP and audio.