diff --git a/CONTEXT.md b/CONTEXT.md index b4e7cfc..b36459d 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -25,8 +25,22 @@ printed enclosure, speakers. Work until those arrive is limited to what the Pi, visible) with `adafruit-circuitpython-rgb-display` and Blinka. Workspace: `~/ros_ws`, source synced from `ros/` with `rsync -a --delete --exclude __pycache__ ros/ hermes@:~/ros_ws/src/`, then `colcon build --symlink-install`. -- CPU governor is `ondemand`; at full clock the face costs about half as much CPU. Set - `performance` in the installer (package 5). It is not persistent yet. +- CPU governor is held at `performance` by `robot-cpu-performance.service` (the face costs about + half as much CPU at full clock). The face starts at boot: `robot-face.service` runs + `ros2 run hermes_face face_node` as `hermes` and restarts it on failure (it needs a writable + working directory, `RuntimeDirectory=robot-face`, because lgpio drops a pipe in its cwd). It comes + up in the `sleeping` pose. Install with `pi/install-face-service.sh`. +- **ROS never leaves the Pi** (owner's decision): `ROS_AUTOMATIC_DISCOVERY_RANGE=LOCALHOST` (set in + `pi/robot-env.sh`, which `source`s everything) and the firewall drops domain 42's DDS ports + (17900-18149) on every interface but loopback. `hermes_mcp` is the only bridge to the outside; a + host that wants data from the robot gets it through an MCP tool, not DDS. This supersedes + HANDOFF.md's `host/ros-shell` talking to the robot. (ROS's setup script exports `SUBNET`, so the + variable must be assigned, not defaulted.) +- **PipeWire** (1.0.5, WirePlumber 0.4.17) runs as `hermes`'s user services (`loginctl enable-linger`). + `~/.config/pipewire/pipewire-pulse.conf.d/50-network-server.conf` (from `pi/pipewire/`) makes + pipewire-pulse listen on TCP 4713 on all interfaces; the firewall is the protection. Defaults are + the WM8960 source and sink (`alsa_input/output.platform-soc_sound.stereo-fallback`). Verified from + talos: `pactl -s tcp:10.55.0.2:4713 info` works; the same on the Wi-Fi address times out. ## Package 0a (platform proof on Ubuntu 24.04) @@ -92,10 +106,16 @@ printed enclosure, speakers. Work until those arrive is limited to what the Pi, - talos (Omarchy, NetworkManager): `host/omarchy/robot-usb-link.sh` (needs sudo) adds profile `hermes-robot-usb`: `10.55.0.1/30` on the interface with MAC `02:68:65:72:6d:01`, priority 100, never-default. Verified from talos: ping ~2.4 ms, SSH over the link. +- **Firewall:** `robot-firewall.service` loads `pi/nftables-robot.nft` before networking at every + boot (`pi/install-firewall.sh`; it must be loaded before the PipeWire network server is enabled). + It accepts 4713/8765/8766 only on `usb0` from `10.55.0.1`, drops them elsewhere, and drops DDS on + every interface but loopback. Verified: from talos, 4713 connects over `usb0` and times out over + Wi-Fi, while an unused port on the Pi's Wi-Fi address refuses instantly (so only 4713 is blocked). + SSH is deliberately left open on all interfaces (key-only), for V2; the owner's call. - Not yet verified: serial login prompt (talos's `/dev/ttyACM0` is `root:uucp`; use `sudo screen /dev/ttyACM0 115200`). The `hermes` account has no password, so the console shows a login prompt but cannot log in until the owner sets one; no autologin by design. Also not done: - the nftables rewrite, the Pi booting with Wi-Fi off, replug and second-host recovery, + the Pi booting with Wi-Fi off, replug and second-host recovery, `get_throttled` after a servo sweep on the splitter, the NixOS snippet, and straylight/other-host testing. Currently the Pi is powered from talos's USB port (bench test, no servos). @@ -119,19 +139,40 @@ printed enclosure, speakers. Work until those arrive is limited to what the Pi, - Measured on the Pi 4: ~9 ms per frame in isolation. With the CPU at full clock: 24% of a core at 20 fps and 40% at 30 fps (target was under 40% at 20 fps or more). Under `ondemand` the same load reads 40% at 20 fps and ~67% at 30 fps. No C++ rewrite was needed. +- **Gaze:** the node subscribes to `/vision/faces` (and `/camera/camera_info` for the image size) + and the eyes glance toward the best face: a recognised one first, then the biggest. Image x flips + on the display (a person on the image's right is on the robot's right, the viewer's left); the + glance is smoothed, held 2.5 s, then drifts back to centre. Sleeping ignores it. The pan sign in + the renderer was wrong (carried over from the prototype) and is fixed: positive pan = the robot's + left = the viewer's right, positive tilt = looks up, as the MCP `turn_head` tool documents. + Not yet confirmed by eye on the real display. - Not yet: audio levels for the mouth (`/audio/levels`), `hermes_mcp` publishing states, launch file in `hermes_bringup`, head/lowpoly shapes, DotStar LEDs, button mute. - Open: IPS black level. The backlight is on at full brightness, so black reads dark gray on camera. Dimming it needs a software PWM on GPIO26 (not a hardware PWM pin). +## Face recognition (opt-in) + +`hermes_vision.recognize` / `people`: SFace embeddings on YuNet's landmarks. Nothing is stored until +someone is enrolled (`python -m hermes_vision.people enroll NAME`, live from the camera topic, or +`enroll-images`); only 128-number embeddings are kept, in `~/.local/share/hermes/faces.json` (mode +0600, directory 0700, written atomically), never images. `list`, `forget NAME` and `check IMAGE` +manage and test it. The detector node puts a recognised name in `Detection2D.id` and picks up +database changes without a restart; its log prints a count, not names. Checked with a throwaway +database: 8 of 9 unenrolled frames of the same person were recognised, down to side profiles +(similarity 0.52-0.66), the miss being a head tipped back (0.34 against the 0.363 threshold); an +unrelated photo scored 0.16. **Nobody is enrolled yet**; the owner enrols themselves live. + ## Next 1. Finish package 0b: Wi-Fi-off boot test, replug recovery, serial login check, NixOS snippet in - `host/nixos/`, and make the firewall load at boot. A new Omarchy host with a GPU (runs the local + `host/nixos/`. A new Omarchy host with a GPU (runs the local model "ornith") arrives this week: it needs only `host/omarchy/robot-usb-link.sh` and its SSH key added to the Pi. -2. Audio (package 1; needs the `wm8960-soundcard` overlay and a reboot), camera (package 3), then - the rest of the packages in `HANDOFF.md`. +2. Audio: the `hermes_audio_state` node (levels, speaking/listening, a trigger-locked scope for the + literal voice trace), echo cancel and `robot_mic` naming once speakers arrive. Then a launch file + that brings up camera, vision, face and audio together, and CPU tuning (camera ~140% of a core). +3. Second-person detection test; enrol the owner live; the rest of `HANDOFF.md`. ## Repository diff --git a/pi/nftables-robot.nft b/pi/nftables-robot.nft index 5fb5979..6624b72 100644 --- a/pi/nftables-robot.nft +++ b/pi/nftables-robot.nft @@ -1,9 +1,10 @@ #!/usr/sbin/nft -f # The robot's audio (4713), MCP (8765) and face webhook (8766) ports are reachable only over the # USB link, and only from the host at 10.55.0.1. Every other interface, Wi-Fi included, is dropped. -# ROS 2's DDS has no authentication, and /camera/* and /vision/faces carry video and recognised -# names, so its ports (domain 42: 17900-18149 with Fast DDS's default offsets) are dropped on every -# interface but usb0 and loopback too. +# ROS 2 never leaves the robot: its DDS has no authentication, and /camera/* and /vision/faces +# carry video and recognised names. Domain 42's ports (17900-18149 with Fast DDS's default offsets) +# are dropped on every interface but loopback, usb0 included. The only way in or out is hermes_mcp, +# which talks to the ROS graph locally and to the host over MCP. # SSH and everything else are left alone: that policy is the owner's call. table inet robot @@ -15,6 +16,6 @@ table inet robot { iif "lo" accept iifname "usb0" ip saddr 10.55.0.1 tcp dport { 4713, 8765, 8766 } accept tcp dport { 4713, 8765, 8766 } drop - iifname != { "lo", "usb0" } udp dport 17900-18149 drop + iifname != "lo" udp dport 17900-18149 drop } } diff --git a/pi/pipewire/50-network-server.conf b/pi/pipewire/50-network-server.conf index 89f2e4a..6c1a926 100644 --- a/pi/pipewire/50-network-server.conf +++ b/pi/pipewire/50-network-server.conf @@ -1,11 +1,13 @@ # ~/.config/pipewire/pipewire-pulse.conf.d/50-network-server.conf (on the Pi) # -# Lets your Omarchy machine reach the robot's microphone and speaker over the -# network, using the PulseAudio protocol that pipewire-pulse speaks. +# Lets the host reach the robot's microphone and speaker over the USB link, using the +# PulseAudio protocol that pipewire-pulse speaks. # # This replaces the default address list, so "unix:native" must stay in it. -# Anyone on the LAN who can reach port 4713 could listen to the microphone; -# the nftables rules installed alongside restrict it to your machine's IP. +# It listens on every interface and is unauthenticated: anyone who can reach port 4713 could +# listen to the microphone. The protection is the firewall (pi/nftables-robot.nft, loaded at boot +# by robot-firewall.service), which accepts 4713 only on usb0 from 10.55.0.1 and drops it on every +# other interface, Wi-Fi included. Do not enable this without that firewall loaded. pulse.properties = { server.address = [ diff --git a/pi/robot-env.sh b/pi/robot-env.sh index d56aa29..605806f 100644 --- a/pi/robot-env.sh +++ b/pi/robot-env.sh @@ -3,14 +3,14 @@ # - the Raspberry Pi libcamera fork ahead of ROS's own copy (same soname, but ROS's crashes in the # Pi image plugin: see CONTEXT.md) # - ~/venv's site-packages first: Blinka/Adafruit for the display, OpenCV 4.10 for YuNet -# - a fixed ROS domain (42), discovery on this machine only. DDS has no authentication and the -# camera and recognised names travel on it: set ROBOT_ROS_DISCOVERY=SUBNET only to -# reach it from a host over the USB link (the firewall drops DDS ports on every other interface) +# - a fixed ROS domain (42), discovery on this machine only. ROS never leaves the robot: DDS has +# no authentication and carries the camera and recognised names, so the firewall drops its ports +# on every interface but loopback. hermes_mcp is the only bridge to the outside world. source /opt/ros/jazzy/setup.bash [ -f "$HOME/camera_ws/install/setup.bash" ] && source "$HOME/camera_ws/install/setup.bash" [ -f "$HOME/ros_ws/install/setup.bash" ] && source "$HOME/ros_ws/install/setup.bash" export LD_LIBRARY_PATH="/opt/libcamera-rpi/lib/aarch64-linux-gnu:${LD_LIBRARY_PATH:-}" export PYTHONPATH="$HOME/venv/lib/python3.12/site-packages:${PYTHONPATH:-}" export ROS_DOMAIN_ID="${ROS_DOMAIN_ID:-42}" -# ROS's own setup script exports SUBNET, so a ${VAR:-default} on the ROS variable would never apply -export ROS_AUTOMATIC_DISCOVERY_RANGE="${ROBOT_ROS_DISCOVERY:-LOCALHOST}" +# ROS's own setup script exports SUBNET, so this must be assigned, not defaulted +export ROS_AUTOMATIC_DISCOVERY_RANGE=LOCALHOST