Add hermes_face ROS package and verify the display on the Pi
The Pi (hermes-bot) is flashed, upgraded and running ROS 2 Jazzy. The ST7789 works through Blinka on lgpio: pass cs=None (the kernel owns CE0), and the glass sits at y_offset=80 in the controller RAM for every rotation, since the driver rotates in software. The mounted pose needs rotation=270. hermes_face is a numpy port of the flat face from docs/face-prototype.html, with phosphor and vector styles, plus a new `sleeping` pose for "no agent connected". The renderer has no ROS or hardware imports. The tone map is a lookup table and the glow runs at half resolution, so the face reaches ~28 fps on the panel at 24% of a core (20 fps) to 40% (30 fps) with the CPU at full clock. CONTEXT.md now records the Pi, the cloud-init first-boot-only gotcha, and what is left. pi/50-cloud-init.yaml is a gitignored Wi-Fi template. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AFN8nRRCiDiMN7ChBpHJhs
This commit is contained in:
@@ -4,3 +4,4 @@ __pycache__/
|
|||||||
*.orig
|
*.orig
|
||||||
*.rej
|
*.rej
|
||||||
.venv/
|
.venv/
|
||||||
|
pi/50-cloud-init.yaml
|
||||||
|
|||||||
+67
-27
@@ -4,43 +4,83 @@ Last updated 2026-10-04. `HANDOFF.md` is the brief; this file tracks where the w
|
|||||||
|
|
||||||
## Hardware on hand
|
## Hardware on hand
|
||||||
|
|
||||||
Raspberry Pi 4, BrainCraft HAT, Arducam UC-346 camera. Not yet: servos, USB-C splitter, printed
|
Raspberry Pi 4 (8 GB), BrainCraft HAT, Arducam UC-346 camera. Not yet: servos, USB-C splitter,
|
||||||
enclosure, speakers. Work until those arrive is limited to what the Pi, HAT and camera can do.
|
printed enclosure, speakers. Work until those arrive is limited to what the Pi, HAT and camera can do.
|
||||||
|
|
||||||
|
## The Pi (`hermes-bot`)
|
||||||
|
|
||||||
|
- Ubuntu 24.04.5 server, kernel `6.8.0-1065-raspi` (fully upgraded), flashed from `talos`. User
|
||||||
|
`hermes` (key-only; passwordless sudo), SSH keys for `kyle@vade` (talos) and `kyle@straylight`.
|
||||||
|
- On Wi-Fi at `192.168.88.152` (MAC `d8:3a:dd:0d:66:f5`); `eth0` is not cabled. The owner says it has
|
||||||
|
a static IP on the rack switch; that address is not recorded here yet. `avahi-daemon` is
|
||||||
|
installed, but mDNS is off in `systemd-resolved` on talos, so `hermes-bot.local` doesn't resolve
|
||||||
|
there; use the IP.
|
||||||
|
- **cloud-init applies `network-config` and `user-data` on the first boot only.** To change Wi-Fi
|
||||||
|
later, edit `/etc/netplan/50-cloud-init.yaml` on the Pi, or bump `instance_id` in `meta-data` on
|
||||||
|
`system-boot`. `pi/50-cloud-init.yaml` is a template for the Wi-Fi block; it is gitignored
|
||||||
|
because it holds credentials.
|
||||||
|
- Installed: ROS 2 Jazzy (`ros-base`, `ros-dev-tools`, colcon, `robot_state_publisher`, `xacro`,
|
||||||
|
`camera_ros`); `python3-numpy/pil/spidev/lgpio/rpi-lgpio`; a venv `~/venv` (system site packages
|
||||||
|
visible) with `adafruit-circuitpython-rgb-display` and Blinka. Workspace: `~/ros_ws`, source synced
|
||||||
|
from `ros/` with `rsync -a --delete --exclude __pycache__ ros/ hermes@<ip>:~/ros_ws/src/`, then
|
||||||
|
`colcon build --symlink-install`.
|
||||||
|
- CPU governor is `ondemand`; at full clock the face costs about half as much CPU. Set
|
||||||
|
`performance` in the installer (package 5). It is not persistent yet.
|
||||||
|
|
||||||
## Package 0a (platform proof on Ubuntu 24.04)
|
## Package 0a (platform proof on Ubuntu 24.04)
|
||||||
|
|
||||||
- **Step 1 done.** `host/check-pi-platform.sh` passes: Ubuntu's `6.8.0-1065-raspi` kernel ships
|
- Step 1 done: `host/check-pi-platform.sh` passes.
|
||||||
every driver and overlay needed (WM8960 audio, OV5647, Unicam, PWM, SPI, dwc2 gadget with
|
- Step 2 done: flashed, booted, upgraded; ROS installed; `ros2 topic list` works.
|
||||||
NCM/ECM/ACM). The script needed a fix for uncompressed `data.tar` (applied to `host/` and the
|
- **Step 5 (display) done.** ST7789 works on `lgpio` through Blinka (`rpi-lgpio`, no `RPi.GPIO`).
|
||||||
duplicate in `tools/`; committed).
|
Two gotchas: pass `cs=None` (the kernel drives CE0; claiming GPIO8 raises "GPIO busy"), and the
|
||||||
- **Step 2 in progress: flashing the SD card.** Image chosen:
|
240x240 glass sits at `y_offset=80` in the controller's RAM, for every rotation (the driver
|
||||||
`ubuntu-24.04.5-preinstalled-server-arm64+raspi.img.xz`, SHA256
|
rotates in software and leaves the scan direction fixed). Mounted pose (microSD end up, USB-C edge
|
||||||
`b23371a5c8d02f612c26e7c18cacb81c5a48f968c7c66a2645391320aa94351e` (from
|
on the left) needs content turned 90 degrees clockwise: `rotation=270`. Full-frame push is
|
||||||
cdimage.ubuntu.com/releases/24.04/release/SHA256SUMS). It was downloaded to the session
|
~23 fps through the driver; the face pushes raw RGB565 with `_block` and reaches ~27-28 fps.
|
||||||
scratchpad under `/tmp`, which a reboot likely cleared; re-download and re-verify.
|
- Not done: audio (`wm8960-soundcard` overlay + mixer), camera (`cam -l`, `camera_ros`), PWM
|
||||||
- The Pi is not running yet. No card has been written. `straylight` (NixOS; has `xz`, `dd`,
|
(waits for servos). Ask before rebooting the Pi.
|
||||||
`udisksctl`, no `rpi-imager`) never detected the Framework SD expansion card: no kernel event
|
|
||||||
in two bays, so the flashing moves to **`talos`**, where the owner confirmed the card mounts.
|
## Face (`ros/hermes_face`)
|
||||||
Continue the session there: clone the repo, read this file, re-download and re-verify the
|
|
||||||
image (the copy on straylight is in its `/tmp`). Check talos's OS and available tools first.
|
- `render.py`: numpy port of `docs/face-prototype.html`, **flat face only** (head and lowpoly not
|
||||||
- **Open questions for the owner before flashing:** Wi-Fi credentials or Ethernet for first boot;
|
ported). Phosphor and vector styles. States: `sleeping idle listening thinking looking speaking`.
|
||||||
hostname (proposed `hermes-robot`); username; SSH key (default `~/.ssh/id_ed25519.pub`).
|
`sleeping` (new) is the "no agent connected" pose: eyes closed to thin lines, mouth short, dim and
|
||||||
Plan: write the image after the owner confirms the device from `lsblk`, then edit cloud-init
|
slowly breathing. `looking` returns to the state it interrupted. Tone mapping is a lookup table
|
||||||
on the `system-boot` partition (user, key, hostname, SSH, network).
|
(RGB and RGB565 tables), glow is computed at half resolution, rotation is a free `np.rot90`.
|
||||||
- Remaining 0a steps after boot: ROS 2 Jazzy, audio (`wm8960-soundcard` overlay + mixer),
|
- `display.py`: ST7789 driver plus an async pusher (render overlaps the SPI transfer).
|
||||||
camera (`cam -l`, `camera_ros`), display test pattern. PWM test waits for the servos. Ask
|
- `face_node.py`: node `hermes_face`; subscribes `/hermes/state` (`std_msgs/String`) and
|
||||||
before rebooting the Pi.
|
`/joint_states` (`pan_joint`, `tilt_joint`, radians); starts in `initial_state` (`sleeping`), so it
|
||||||
|
runs with nothing else up. Parameters: `rotation` (270), `x_offset` (0), `y_offset` (80),
|
||||||
|
`baudrate`, `fps`, `output` (`st7789` or `none`), `phosphor` (P7), `style`, `persistence_ms`,
|
||||||
|
`focus`, `jitter`, `intensity`, `initial_state`.
|
||||||
|
- `demo.py`: no ROS. `python -m hermes_face.demo --sequence` cycles the states on the panel;
|
||||||
|
`--png-dir DIR` writes a PNG per state without hardware.
|
||||||
|
- Tests: `python3 -m pytest test` in `ros/hermes_face` (23 pass on the Pi).
|
||||||
|
- Measured on the Pi 4: ~9 ms per frame in isolation. With the CPU at full clock: 24% of a core at
|
||||||
|
20 fps and 40% at 30 fps (target was under 40% at 20 fps or more). Under `ondemand` the same
|
||||||
|
load reads 40% at 20 fps and ~67% at 30 fps. No C++ rewrite was needed.
|
||||||
|
- Not yet: audio levels for the mouth (`/audio/levels`), `hermes_mcp` publishing states, launch
|
||||||
|
file in `hermes_bringup`, head/lowpoly shapes, DotStar LEDs, button mute.
|
||||||
|
- Open: IPS black level. The backlight is on at full brightness, so black reads dark gray on
|
||||||
|
camera. Dimming it needs a software PWM on GPIO26 (not a hardware PWM pin).
|
||||||
|
|
||||||
|
## Next
|
||||||
|
|
||||||
|
1. Package 0b: USB gadget Ethernet (`dwc2` peripheral, NCM + ACM, `usb0` at `10.55.0.2/30`). Needs a
|
||||||
|
reboot and a USB-C data cable to a host; the splitter has not arrived, so a bench test would
|
||||||
|
power the Pi from the host's port (fine without servos). Keep Wi-Fi up throughout.
|
||||||
|
2. Audio (package 1), camera (package 3), then the rest of the packages in `HANDOFF.md`.
|
||||||
|
|
||||||
## Repository
|
## Repository
|
||||||
|
|
||||||
Git repo initialised on `main` with one commit ("Initial commit"). `.gitignore` covers
|
Branch `master`. `.gitignore` covers `.remember/`, caches, `.venv/`, `*.orig`, `*.rej` and
|
||||||
`.remember/`, caches, `.venv/`, `*.orig`, `*.rej`. The earlier zip and patch files were deleted.
|
`pi/50-cloud-init.yaml`. `tools/check-pi-platform.sh` duplicates `host/check-pi-platform.sh` and can
|
||||||
`tools/check-pi-platform.sh` duplicates `host/check-pi-platform.sh` and can probably go.
|
probably go.
|
||||||
|
|
||||||
## Security review notes (parked until the USB link is up)
|
## Security review notes (parked until the USB link is up)
|
||||||
|
|
||||||
An automated review of the initial commit flagged three items. All stem from the code predating
|
An automated review of the initial commit flagged three items. All stem from the code predating
|
||||||
the USB-gadget design (HANDOFF.md package 0b); none is exposed until the Pi runs.
|
the USB-gadget design (HANDOFF.md package 0b); none is exposed until the Pi runs the services.
|
||||||
|
|
||||||
- `pi/robot_mcp_server.py`: plain HTTP, `ROBOT_MCP_BIND` defaults to `0.0.0.0`. Plan: default to
|
- `pi/robot_mcp_server.py`: plain HTTP, `ROBOT_MCP_BIND` defaults to `0.0.0.0`. Plan: default to
|
||||||
`10.55.0.2`, start after `usb0`.
|
`10.55.0.2`, start after `usb0`.
|
||||||
|
|||||||
@@ -0,0 +1,64 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Draw a test pattern on the BrainCraft HAT's ST7789 (240x240, SPI CE0, DC=GPIO25, BL=GPIO26).
|
||||||
|
|
||||||
|
display_test.py [ROTATION] one pattern, then a frame-rate probe
|
||||||
|
|
||||||
|
The driver rotates in software (PIL, counter-clockwise), and the panel's 80-row RAM offset belongs
|
||||||
|
to the glass, so y_offset=80 holds for every rotation. 270 is the mounted pose (microSD end up).
|
||||||
|
A correct picture has the frame complete on all four edges, no noise, the white triangle in the
|
||||||
|
top-left corner as you look at the panel in its mounted pose, and upright text.
|
||||||
|
"""
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
|
||||||
|
import board
|
||||||
|
import digitalio
|
||||||
|
from adafruit_rgb_display import st7789
|
||||||
|
from PIL import Image, ImageDraw, ImageFont
|
||||||
|
|
||||||
|
W = H = 240
|
||||||
|
|
||||||
|
|
||||||
|
def make_display(rotation, x_offset, y_offset, baudrate=48_000_000):
|
||||||
|
return st7789.ST7789(
|
||||||
|
board.SPI(),
|
||||||
|
cs=None, # CE0 is driven by the kernel SPI driver; claiming GPIO8 fails with 'GPIO busy'
|
||||||
|
dc=digitalio.DigitalInOut(board.D25),
|
||||||
|
rst=None,
|
||||||
|
baudrate=baudrate,
|
||||||
|
width=W,
|
||||||
|
height=H,
|
||||||
|
x_offset=x_offset,
|
||||||
|
y_offset=y_offset,
|
||||||
|
rotation=rotation,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def pattern(label):
|
||||||
|
img = Image.new("RGB", (W, H), "black")
|
||||||
|
d = ImageDraw.Draw(img)
|
||||||
|
for i, c in enumerate(["red", "lime", "blue", "yellow", "cyan", "magenta"]):
|
||||||
|
d.rectangle([i * 40, 90, i * 40 + 39, 150], fill=c)
|
||||||
|
d.rectangle([0, 0, W - 1, H - 1], outline="white") # edge frame: shows offset/clipping
|
||||||
|
d.rectangle([3, 3, W - 4, H - 4], outline="gray")
|
||||||
|
d.polygon([(8, 8), (58, 8), (8, 58)], fill="white") # top-left marker: shows rotation
|
||||||
|
d.text((70, 12), label[0], fill="white", font=ImageFont.load_default(size=48))
|
||||||
|
d.text((70, 66), label[1], fill="white", font=ImageFont.load_default(size=14))
|
||||||
|
for x in range(0, W, 30):
|
||||||
|
d.line([(x, 160), (x, H - 8)], fill="gray")
|
||||||
|
for y in range(160, H - 8, 20):
|
||||||
|
d.line([(0, y), (W, y)], fill="gray")
|
||||||
|
return img
|
||||||
|
|
||||||
|
|
||||||
|
backlight = digitalio.DigitalInOut(board.D26)
|
||||||
|
backlight.switch_to_output(value=True)
|
||||||
|
|
||||||
|
rotation = int(sys.argv[1]) if len(sys.argv) > 1 else 270
|
||||||
|
disp = make_display(rotation, 0, 80)
|
||||||
|
img = pattern((str(rotation), f"rot={rotation} y_off=80"))
|
||||||
|
disp.image(img)
|
||||||
|
t0, n = time.monotonic(), 30
|
||||||
|
for _ in range(n):
|
||||||
|
disp.image(img)
|
||||||
|
print(f"pattern shown; full-frame push rate {n / (time.monotonic() - t0):.1f} fps at 48 MHz")
|
||||||
@@ -0,0 +1,94 @@
|
|||||||
|
"""Run the face without ROS: on the Pi's display, or to PNG files for checking the look.
|
||||||
|
|
||||||
|
python -m hermes_face.demo --sequence # cycle through every state on the panel
|
||||||
|
python -m hermes_face.demo --state speaking --seconds 10
|
||||||
|
python -m hermes_face.demo --png-dir /tmp/face # one PNG per state, no hardware
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
from .render import STATES, FaceRenderer
|
||||||
|
|
||||||
|
SEQUENCE = (('sleeping', 4.0), ('idle', 3.5), ('listening', 3.6), ('thinking', 2.4),
|
||||||
|
('looking', 1.3), ('thinking', 1.4), ('speaking', 6.5))
|
||||||
|
|
||||||
|
|
||||||
|
def write_pngs(args) -> None:
|
||||||
|
from PIL import Image
|
||||||
|
os.makedirs(args.png_dir, exist_ok=True)
|
||||||
|
for state in STATES:
|
||||||
|
r = FaceRenderer(args.phosphor, args.style, state=state, seed=1, rotation=args.rotation)
|
||||||
|
r.set_pose(args.pan, args.tilt)
|
||||||
|
dt, t = 1.0 / args.fps, 0.0
|
||||||
|
settle = 5.0 if state in ('sleeping', 'thinking') else 3.0
|
||||||
|
for _ in range(int(settle * args.fps)):
|
||||||
|
t += dt
|
||||||
|
frame = r.frame(t, dt)
|
||||||
|
Image.fromarray(frame).resize((480, 480), Image.NEAREST).save(
|
||||||
|
os.path.join(args.png_dir, f'{args.style}-{state}.png'))
|
||||||
|
print('wrote', state)
|
||||||
|
|
||||||
|
|
||||||
|
def run(args) -> None:
|
||||||
|
from .display import AsyncDisplay, St7789
|
||||||
|
panel = AsyncDisplay(St7789(args.x_offset, args.y_offset, args.baud))
|
||||||
|
r = FaceRenderer(args.phosphor, args.style, state=args.state, now=time.monotonic(),
|
||||||
|
rotation=args.rotation)
|
||||||
|
r.set_pose(args.pan, args.tilt)
|
||||||
|
period = 1.0 / args.fps
|
||||||
|
start = last = time.monotonic()
|
||||||
|
seq_i, seq_t0 = 0, start
|
||||||
|
if args.sequence:
|
||||||
|
r.set_state(SEQUENCE[0][0], start)
|
||||||
|
frames, stat_t, cpu0, render_s = 0, start, time.process_time(), 0.0
|
||||||
|
try:
|
||||||
|
while args.seconds <= 0 or time.monotonic() - start < args.seconds:
|
||||||
|
now = time.monotonic()
|
||||||
|
if args.sequence and now - seq_t0 >= SEQUENCE[seq_i][1]:
|
||||||
|
seq_i, seq_t0 = (seq_i + 1) % len(SEQUENCE), now
|
||||||
|
r.set_state(SEQUENCE[seq_i][0], now)
|
||||||
|
t0 = time.perf_counter()
|
||||||
|
rgb = r.frame(now, now - last, 'rgb565')
|
||||||
|
render_s += time.perf_counter() - t0
|
||||||
|
last = now
|
||||||
|
panel.show(rgb)
|
||||||
|
frames += 1
|
||||||
|
if now - stat_t >= 5.0:
|
||||||
|
wall = now - stat_t
|
||||||
|
print(f'{r.state:>9}: render {frames / wall:5.1f} fps ({1000 * render_s / frames:4.1f} ms/frame), '
|
||||||
|
f'panel {panel.pushed / wall:5.1f} fps, cpu {100 * (time.process_time() - cpu0) / wall:3.0f}%')
|
||||||
|
frames, stat_t, cpu0, render_s, panel.pushed = 0, now, time.process_time(), 0.0, 0
|
||||||
|
time.sleep(max(0.0, period - (time.monotonic() - now)))
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
pass
|
||||||
|
finally:
|
||||||
|
panel.close()
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||||
|
p.add_argument('--state', choices=STATES, default='idle')
|
||||||
|
p.add_argument('--sequence', action='store_true', help='cycle through the states')
|
||||||
|
p.add_argument('--seconds', type=float, default=0, help='stop after this long (0 = until Ctrl-C)')
|
||||||
|
p.add_argument('--fps', type=float, default=30)
|
||||||
|
p.add_argument('--phosphor', default='P7', choices=['P31', 'P1', 'P7', 'P3'])
|
||||||
|
p.add_argument('--style', default='phosphor', choices=['phosphor', 'vector'])
|
||||||
|
p.add_argument('--pan', type=float, default=0, help='degrees')
|
||||||
|
p.add_argument('--tilt', type=float, default=0, help='degrees')
|
||||||
|
p.add_argument('--rotation', type=int, default=270, choices=[0, 90, 180, 270],
|
||||||
|
help='degrees counter-clockwise; 270 is the mounted pose (microSD end up)')
|
||||||
|
p.add_argument('--x-offset', type=int, default=0)
|
||||||
|
p.add_argument('--y-offset', type=int, default=80)
|
||||||
|
p.add_argument('--baud', type=int, default=48_000_000)
|
||||||
|
p.add_argument('--png-dir', help='write one PNG per state here and exit (no hardware)')
|
||||||
|
args = p.parse_args()
|
||||||
|
write_pngs(args) if args.png_dir else run(args)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
"""BrainCraft HAT display: ST7789, 240x240, SPI CE0, DC=GPIO25, backlight=GPIO26.
|
||||||
|
|
||||||
|
Hardware imports are deferred so the renderer and tests run on hosts without a Pi.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import threading
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
|
||||||
|
class St7789:
|
||||||
|
"""Frames arrive already rotated (FaceRenderer(rotation=...)), so the driver's own rotation is
|
||||||
|
left at 0. The 80-row offset belongs to the panel, not to a rotation: a 240x240 glass sits in a
|
||||||
|
240x320 controller RAM, and the driver rotates in software, leaving the scan direction fixed."""
|
||||||
|
|
||||||
|
def __init__(self, x_offset: int = 0, y_offset: int = 80, baudrate: int = 48_000_000):
|
||||||
|
import board
|
||||||
|
import digitalio
|
||||||
|
from adafruit_rgb_display import st7789
|
||||||
|
self._backlight = digitalio.DigitalInOut(board.D26)
|
||||||
|
self._backlight.switch_to_output(value=True)
|
||||||
|
self._disp = st7789.ST7789(
|
||||||
|
board.SPI(),
|
||||||
|
cs=None, # the kernel drives CE0; claiming GPIO8 fails with 'GPIO busy'
|
||||||
|
dc=digitalio.DigitalInOut(board.D25),
|
||||||
|
rst=None,
|
||||||
|
baudrate=baudrate,
|
||||||
|
width=240, height=240,
|
||||||
|
x_offset=x_offset, y_offset=y_offset, # 240x240 panel in a 240x320 controller
|
||||||
|
rotation=0,
|
||||||
|
)
|
||||||
|
|
||||||
|
def show(self, frame) -> None:
|
||||||
|
"""Push a frame: big-endian RGB565 bytes (fast path), or an (H, W, 3) uint8 array."""
|
||||||
|
if not isinstance(frame, (bytes, bytearray)):
|
||||||
|
from PIL import Image
|
||||||
|
self._disp.image(Image.fromarray(frame, 'RGB'))
|
||||||
|
return
|
||||||
|
self._disp._block(0, 0, 239, 239, frame) # the driver's own windowing, minus its PIL conversion
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
self.show(bytes(240 * 240 * 2))
|
||||||
|
self._backlight.value = False
|
||||||
|
|
||||||
|
|
||||||
|
class AsyncDisplay:
|
||||||
|
"""Pushes frames on a worker thread so rendering the next one overlaps the SPI transfer.
|
||||||
|
Only the newest pending frame is kept; if the panel is slower than the renderer, frames drop."""
|
||||||
|
|
||||||
|
def __init__(self, display: St7789):
|
||||||
|
self._display = display
|
||||||
|
self._latest = None
|
||||||
|
self._cv = threading.Condition()
|
||||||
|
self._stop = False
|
||||||
|
self.pushed = 0
|
||||||
|
self._thread = threading.Thread(target=self._run, name='display-push', daemon=True)
|
||||||
|
self._thread.start()
|
||||||
|
|
||||||
|
def show(self, frame) -> None:
|
||||||
|
with self._cv:
|
||||||
|
self._latest = frame
|
||||||
|
self._cv.notify()
|
||||||
|
|
||||||
|
def _run(self) -> None:
|
||||||
|
while True:
|
||||||
|
with self._cv:
|
||||||
|
while self._latest is None and not self._stop:
|
||||||
|
self._cv.wait()
|
||||||
|
if self._stop:
|
||||||
|
return
|
||||||
|
frame, self._latest = self._latest, None
|
||||||
|
self._display.show(frame)
|
||||||
|
self.pushed += 1
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
with self._cv:
|
||||||
|
self._stop = True
|
||||||
|
self._cv.notify()
|
||||||
|
self._thread.join(timeout=2)
|
||||||
|
self._display.close()
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
"""ROS node for the face. Draws on the BrainCraft HAT's display.
|
||||||
|
|
||||||
|
Subscribes: /hermes/state (std_msgs/String): sleeping idle listening thinking looking speaking
|
||||||
|
/joint_states (sensor_msgs/JointState): pan_joint, tilt_joint in radians (REP 103)
|
||||||
|
The face starts in `initial_state` (default `sleeping`), so it runs with nothing else up; whichever
|
||||||
|
node knows the agent is connected (hermes_mcp) publishes `idle` to wake it.
|
||||||
|
"""
|
||||||
|
import math
|
||||||
|
import time
|
||||||
|
|
||||||
|
import rclpy
|
||||||
|
from rclpy.node import Node
|
||||||
|
from sensor_msgs.msg import JointState
|
||||||
|
from std_msgs.msg import String
|
||||||
|
|
||||||
|
from .render import STATES, FaceRenderer
|
||||||
|
|
||||||
|
|
||||||
|
class FaceNode(Node):
|
||||||
|
def __init__(self):
|
||||||
|
super().__init__('hermes_face')
|
||||||
|
param = lambda name, default: self.declare_parameter(name, default).value # noqa: E731
|
||||||
|
self.fps = float(param('fps', 30.0))
|
||||||
|
output = param('output', 'st7789') # 'st7789' or 'none' (render only)
|
||||||
|
self.pan_joint = param('pan_joint', 'pan_joint')
|
||||||
|
self.tilt_joint = param('tilt_joint', 'tilt_joint')
|
||||||
|
initial = param('initial_state', 'sleeping')
|
||||||
|
|
||||||
|
self.renderer = FaceRenderer(
|
||||||
|
phosphor=param('phosphor', 'P7'), style=param('style', 'phosphor'),
|
||||||
|
persist_ms=float(param('persistence_ms', 70.0)), focus=float(param('focus', 0.35)),
|
||||||
|
jitter=float(param('jitter', 0.3)), intensity=float(param('intensity', 1.0)),
|
||||||
|
state=initial if initial in STATES else 'sleeping', now=time.monotonic(),
|
||||||
|
rotation=int(param('rotation', 270))) # 270: the mounted pose, microSD end up
|
||||||
|
|
||||||
|
self.display = None
|
||||||
|
if output == 'st7789':
|
||||||
|
from .display import AsyncDisplay, St7789
|
||||||
|
self.display = AsyncDisplay(St7789(
|
||||||
|
x_offset=int(param('x_offset', 0)), y_offset=int(param('y_offset', 80)),
|
||||||
|
baudrate=int(param('baudrate', 48_000_000))))
|
||||||
|
|
||||||
|
self.create_subscription(String, '/hermes/state', self._on_state, 10)
|
||||||
|
self.create_subscription(JointState, '/joint_states', self._on_joints, 10)
|
||||||
|
self._last = time.monotonic()
|
||||||
|
self._frames, self._stat_t = 0, self._last
|
||||||
|
self.create_timer(1.0 / self.fps, self._tick)
|
||||||
|
self.get_logger().info(f'face up: {output}, {self.fps:g} fps, state {self.renderer.state}')
|
||||||
|
|
||||||
|
def _on_state(self, msg: String) -> None:
|
||||||
|
if msg.data not in STATES:
|
||||||
|
self.get_logger().warn(f'ignoring unknown state {msg.data!r}; known: {", ".join(STATES)}')
|
||||||
|
return
|
||||||
|
self.renderer.set_state(msg.data, time.monotonic())
|
||||||
|
|
||||||
|
def _on_joints(self, msg: JointState) -> None:
|
||||||
|
pos = dict(zip(msg.name, msg.position))
|
||||||
|
if self.pan_joint in pos or self.tilt_joint in pos:
|
||||||
|
self.renderer.set_pose(math.degrees(pos.get(self.pan_joint, 0.0)),
|
||||||
|
math.degrees(pos.get(self.tilt_joint, 0.0)))
|
||||||
|
|
||||||
|
def _tick(self) -> None:
|
||||||
|
now = time.monotonic()
|
||||||
|
rgb = self.renderer.frame(now, now - self._last, 'rgb565')
|
||||||
|
self._last = now
|
||||||
|
if self.display:
|
||||||
|
self.display.show(rgb)
|
||||||
|
self._frames += 1
|
||||||
|
if now - self._stat_t >= 30.0:
|
||||||
|
self.get_logger().debug(f'{self._frames / (now - self._stat_t):.1f} fps')
|
||||||
|
self._frames, self._stat_t = 0, now
|
||||||
|
|
||||||
|
def destroy_node(self):
|
||||||
|
if self.display:
|
||||||
|
self.display.close()
|
||||||
|
super().destroy_node()
|
||||||
|
|
||||||
|
|
||||||
|
def main(args=None):
|
||||||
|
rclpy.init(args=args)
|
||||||
|
node = FaceNode()
|
||||||
|
try:
|
||||||
|
rclpy.spin(node)
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
pass
|
||||||
|
finally:
|
||||||
|
node.destroy_node()
|
||||||
|
if rclpy.ok():
|
||||||
|
rclpy.shutdown()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()
|
||||||
@@ -0,0 +1,386 @@
|
|||||||
|
"""Oscilloscope-phosphor face: a numpy port of docs/face-prototype.html (flat face).
|
||||||
|
|
||||||
|
No ROS or hardware imports, so it runs on any host and in tests. The model is the prototype's:
|
||||||
|
a 240x240 float32 energy buffer that decays each frame, a beam traced at constant speed that
|
||||||
|
deposits energy with bilinear splatting, two box-blur passes for glow, and a tone map tinted
|
||||||
|
with the phosphor colour. Only the flat shape is ported so far (head and lowpoly are not).
|
||||||
|
|
||||||
|
Beyond the prototype: a `sleeping` state for "no agent connected", and `looking` returns to the
|
||||||
|
state it interrupted instead of always going to idle.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import math
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
W = H = 240
|
||||||
|
N = W * H
|
||||||
|
|
||||||
|
PHOSPHORS = {
|
||||||
|
'P31': (0.30, 1.00, 0.45), # bright green, the usual scope tube
|
||||||
|
'P1': (0.45, 1.00, 0.25), # yellower green, longer-glowing
|
||||||
|
'P7': (0.62, 0.80, 1.00), # blue-white flash, radar screens
|
||||||
|
'P3': (1.00, 0.68, 0.18), # amber
|
||||||
|
}
|
||||||
|
|
||||||
|
STATES = ('sleeping', 'idle', 'listening', 'thinking', 'looking', 'speaking')
|
||||||
|
LOOK_HOLD_S = 1.2 # how long `looking` holds before returning to the prior state
|
||||||
|
SQUINT_S = 0.18 # squint before the shutter flash
|
||||||
|
FLASH_AT_S = 0.16 # the shutter bloom lands this long after entering `looking`
|
||||||
|
SLEEP_OPEN = 0.07 # eye openness while asleep: a thin line
|
||||||
|
|
||||||
|
|
||||||
|
def _approach(cur: float, target: float, rate: float, dt: float) -> float:
|
||||||
|
return cur + (target - cur) * (1.0 - math.exp(-rate * dt))
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- synthetic voices
|
||||||
|
def _hash(i: float, seed: float) -> float:
|
||||||
|
x = math.sin(i * 127.1 + seed * 311.7) * 43758.5453
|
||||||
|
return x - math.floor(x)
|
||||||
|
|
||||||
|
|
||||||
|
_VOWELS = ((730, 1090), (270, 2290), (300, 870), (530, 1840), (570, 840), (660, 1720))
|
||||||
|
_K = np.arange(1, 10, dtype=np.float64)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Voice:
|
||||||
|
env: float
|
||||||
|
f0: float
|
||||||
|
w: np.ndarray
|
||||||
|
norm: float
|
||||||
|
noise: float
|
||||||
|
|
||||||
|
|
||||||
|
def voice(T: float, seed: float, base_f0: float, rate: float) -> Voice:
|
||||||
|
u = T * rate
|
||||||
|
i = math.floor(u)
|
||||||
|
frac = u - i
|
||||||
|
pause = _hash(i, seed) < 0.16
|
||||||
|
amp = 0.0 if pause else 0.55 + 0.45 * _hash(i + 7, seed)
|
||||||
|
env = amp * math.sin(math.pi * frac) ** 0.7
|
||||||
|
v0 = _VOWELS[int(math.floor(_hash(i, seed + 3) * len(_VOWELS)))]
|
||||||
|
v1 = _VOWELS[int(math.floor(_hash(i + 1, seed + 3) * len(_VOWELS)))]
|
||||||
|
m = frac * frac * (3 - 2 * frac)
|
||||||
|
f1 = v0[0] + (v1[0] - v0[0]) * m
|
||||||
|
f2 = v0[1] + (v1[1] - v0[1]) * m
|
||||||
|
f0 = base_f0 * (1 + 0.12 * math.sin(T * 1.7 + seed) + 0.06 * math.sin(T * 4.3))
|
||||||
|
f = _K * f0
|
||||||
|
d1 = (f - f1) / 160.0
|
||||||
|
d2 = (f - f2) / 260.0
|
||||||
|
a = np.exp(-d1 * d1) + 0.65 * np.exp(-d2 * d2) + np.where(_K == 1, 0.35, 0.04)
|
||||||
|
noise = (0.14 - frac) * 3.2 if frac < 0.14 and not pause else 0.0
|
||||||
|
return Voice(env, f0, a, float(a.sum()), noise)
|
||||||
|
|
||||||
|
|
||||||
|
def voice_sample(v: Voice, t: np.ndarray, phase: float) -> np.ndarray:
|
||||||
|
"""The voice's waveform at local times t (seconds) inside a trigger-locked window."""
|
||||||
|
arg = 2 * math.pi * _K[:, None] * v.f0 * t[None, :] + phase * _K[:, None]
|
||||||
|
return (v.w[:, None] * np.sin(arg)).sum(axis=0) / v.norm
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- blur
|
||||||
|
def box_blur(a: np.ndarray, r: int) -> np.ndarray:
|
||||||
|
"""Separable box blur, zero beyond the edges, via cumulative sums (conserves total energy)."""
|
||||||
|
k = 2 * r + 1
|
||||||
|
for axis in (1, 0):
|
||||||
|
n = a.shape[axis]
|
||||||
|
shape = list(a.shape)
|
||||||
|
shape[axis] = n + 2 * r + 1
|
||||||
|
p = np.zeros(shape, np.float32)
|
||||||
|
body = [slice(None), slice(None)]
|
||||||
|
tail = [slice(None), slice(None)]
|
||||||
|
last = [slice(None), slice(None)]
|
||||||
|
lo = [slice(None), slice(None)]
|
||||||
|
hi = [slice(None), slice(None)]
|
||||||
|
body[axis] = slice(r + 1, r + 1 + n)
|
||||||
|
tail[axis] = slice(r + 1 + n, None)
|
||||||
|
last[axis] = slice(r + n, r + n + 1)
|
||||||
|
lo[axis] = slice(0, n)
|
||||||
|
hi[axis] = slice(k, k + n)
|
||||||
|
np.cumsum(a, axis=axis, dtype=np.float32, out=p[tuple(body)])
|
||||||
|
p[tuple(tail)] = p[tuple(last)]
|
||||||
|
a = (p[tuple(hi)] - p[tuple(lo)]) * np.float32(1.0 / k)
|
||||||
|
return a
|
||||||
|
|
||||||
|
|
||||||
|
_LUT_SCALE = 512 # tone-map table entries per unit of energy
|
||||||
|
_LUT_SIZE = 8 * _LUT_SCALE # covers energy 0..8, far beyond any overdriven spot
|
||||||
|
|
||||||
|
|
||||||
|
def _rgb565_be(rgb: np.ndarray) -> np.ndarray:
|
||||||
|
r, g, b = (rgb[..., i].astype(np.uint16) for i in range(3))
|
||||||
|
return (((r & 0xF8) << 8) | ((g & 0xFC) << 3) | (b >> 3)).astype('>u2') # big-endian for SPI
|
||||||
|
|
||||||
|
|
||||||
|
class FaceRenderer:
|
||||||
|
def __init__(self, phosphor: str = 'P7', style: str = 'phosphor', persist_ms: float = 70.0,
|
||||||
|
intensity: float = 1.0, focus: float = 0.35, jitter: float = 0.3,
|
||||||
|
state: str = 'idle', now: float = 0.0, seed: int | None = None, rotation: int = 0):
|
||||||
|
if style not in ('phosphor', 'vector'):
|
||||||
|
raise ValueError(f'unknown style {style!r}')
|
||||||
|
if rotation not in (0, 90, 180, 270):
|
||||||
|
raise ValueError(f'rotation must be 0, 90, 180 or 270, not {rotation!r}')
|
||||||
|
self.rotation = rotation # output turned this many degrees counter-clockwise
|
||||||
|
self.rgb = PHOSPHORS[phosphor]
|
||||||
|
self.style = style
|
||||||
|
self.persist_ms = persist_ms
|
||||||
|
self.intensity = intensity
|
||||||
|
self.focus = focus
|
||||||
|
self.jitter = jitter
|
||||||
|
self.rng = np.random.default_rng(seed)
|
||||||
|
|
||||||
|
self.buf = np.zeros((H, W), np.float32)
|
||||||
|
self._xi: list[np.ndarray] = []
|
||||||
|
self._yi: list[np.ndarray] = []
|
||||||
|
self._w: list[np.ndarray] = []
|
||||||
|
self.beam = (120.0, 120.0)
|
||||||
|
self.flash = 0.0
|
||||||
|
|
||||||
|
self.state = state
|
||||||
|
self._state_t0 = now
|
||||||
|
self._return_state = 'idle'
|
||||||
|
self._flash_pending = state == 'looking'
|
||||||
|
self.pan = 0.0 # degrees, from the servos
|
||||||
|
self.tilt = 0.0
|
||||||
|
self.a = dict(think=0.0, open=SLEEP_OPEN if state == 'sleeping' else 1.0, eye_scale=1.0,
|
||||||
|
mouth_speak=0.0, mouth_listen=0.0, smile=1.0, pan_x=0.0, tilt_y=0.0,
|
||||||
|
sleep=1.0 if state == 'sleeping' else 0.0)
|
||||||
|
self._next_blink = now + 2.5
|
||||||
|
self._blink_t = -1.0
|
||||||
|
self._build_lut()
|
||||||
|
|
||||||
|
def _build_lut(self) -> None:
|
||||||
|
"""Tone curve as a table indexed by energy, so a frame is one gather, not exp/clip/stack."""
|
||||||
|
v = np.arange(_LUT_SIZE, dtype=np.float64) / _LUT_SCALE
|
||||||
|
pr = np.array(self.rgb)
|
||||||
|
if self.style == 'vector':
|
||||||
|
c = pr + (1 - pr) * 0.35 # pale the tint so lines read as light, not phosphor
|
||||||
|
out = np.minimum(1.0, c[None, :] * v[:, None])
|
||||||
|
else:
|
||||||
|
base = 1 - np.exp(-v * 1.6)
|
||||||
|
hot = np.clip((v - 1.1) * 0.35, 0, 0.75) # overdriven spots whiten
|
||||||
|
out = np.minimum(1.0, pr[None, :] * base[:, None] + hot[:, None])
|
||||||
|
self._lut_rgb = (out * 255).astype(np.uint8)
|
||||||
|
self._lut_565 = _rgb565_be(self._lut_rgb)
|
||||||
|
|
||||||
|
# ------------------------------------------------------------ inputs
|
||||||
|
def set_state(self, s: str, now: float) -> None:
|
||||||
|
if s not in STATES:
|
||||||
|
raise ValueError(f'unknown state {s!r}')
|
||||||
|
if s == self.state and s != 'looking':
|
||||||
|
return
|
||||||
|
if s == 'looking' and self.state != 'looking':
|
||||||
|
self._return_state = self.state
|
||||||
|
self.state = s
|
||||||
|
self._state_t0 = now
|
||||||
|
if s == 'looking':
|
||||||
|
self._flash_pending = True
|
||||||
|
|
||||||
|
def set_pose(self, pan_deg: float, tilt_deg: float) -> None:
|
||||||
|
self.pan, self.tilt = pan_deg, tilt_deg
|
||||||
|
|
||||||
|
# ------------------------------------------------------------ beam
|
||||||
|
def _queue(self, x, y, e) -> None:
|
||||||
|
x = np.atleast_1d(np.asarray(x, np.float64))
|
||||||
|
y = np.atleast_1d(np.asarray(y, np.float64))
|
||||||
|
if self.jitter > 0 and self.style == 'phosphor':
|
||||||
|
x = x + (self.rng.random(x.shape) - 0.5) * 2 * self.jitter
|
||||||
|
y = y + (self.rng.random(y.shape) - 0.5) * 2 * self.jitter
|
||||||
|
xi = np.floor(x).astype(np.int64)
|
||||||
|
yi = np.floor(y).astype(np.int64)
|
||||||
|
ok = (xi >= 0) & (yi >= 0) & (xi < W - 1) & (yi < H - 1)
|
||||||
|
e = np.broadcast_to(np.asarray(e, np.float64), x.shape)
|
||||||
|
self._xi.append(xi[ok])
|
||||||
|
self._yi.append(yi[ok])
|
||||||
|
fx = (x - xi)[ok]
|
||||||
|
fy = (y - yi)[ok]
|
||||||
|
self._w.append(np.stack([e[ok] * (1 - fx) * (1 - fy), e[ok] * fx * (1 - fy),
|
||||||
|
e[ok] * (1 - fx) * fy, e[ok] * fx * fy]))
|
||||||
|
|
||||||
|
def _flush(self) -> None:
|
||||||
|
if not self._xi:
|
||||||
|
return
|
||||||
|
xi = np.concatenate(self._xi)
|
||||||
|
yi = np.concatenate(self._yi)
|
||||||
|
w = np.concatenate(self._w, axis=1)
|
||||||
|
idx = yi * W + xi
|
||||||
|
acc = np.bincount(np.concatenate([idx, idx + 1, idx + W, idx + W + 1]),
|
||||||
|
weights=w.reshape(-1), minlength=N)
|
||||||
|
self.buf += acc.reshape(H, W).astype(np.float32)
|
||||||
|
self._xi.clear()
|
||||||
|
self._yi.clear()
|
||||||
|
self._w.clear()
|
||||||
|
|
||||||
|
def _stroke(self, x: np.ndarray, y: np.ndarray, e: float) -> None:
|
||||||
|
if self.style == 'vector' and len(x) > 1: # keep line weight even: fill the gaps
|
||||||
|
gaps = np.floor(np.hypot(np.diff(x), np.diff(y)) / 0.7).astype(np.int64)
|
||||||
|
reps = gaps + 1
|
||||||
|
seg = np.repeat(np.arange(len(reps)), reps)
|
||||||
|
within = np.arange(len(seg)) - np.repeat(np.cumsum(reps) - reps, reps)
|
||||||
|
t = within / np.repeat(reps, reps)
|
||||||
|
x, y = (np.append(x[seg] + (x[seg + 1] - x[seg]) * t, x[-1]),
|
||||||
|
np.append(y[seg] + (y[seg + 1] - y[seg]) * t, y[-1]))
|
||||||
|
self._queue(x, y, e)
|
||||||
|
|
||||||
|
def _jump_to(self, x: float, y: float, e: float) -> None:
|
||||||
|
if self.style == 'phosphor': # a vector display blanks the beam on retrace
|
||||||
|
bx, by = self.beam
|
||||||
|
n = max(4, round(math.hypot(x - bx, y - by) / 1.5))
|
||||||
|
j = np.arange(1, n + 1) / n
|
||||||
|
self._queue(bx + (x - bx) * j, by + (y - by) * j, e * 0.045)
|
||||||
|
self.beam = (x, y)
|
||||||
|
|
||||||
|
def _trace(self, fn, n: int, e: float) -> None:
|
||||||
|
x, y = fn(np.linspace(0.0, 1.0, n + 1))
|
||||||
|
self._jump_to(float(x[0]), float(y[0]), e)
|
||||||
|
self._stroke(x, y, e)
|
||||||
|
self.beam = (float(x[-1]), float(y[-1]))
|
||||||
|
|
||||||
|
# ------------------------------------------------------------ the face
|
||||||
|
def _draw(self, T: float, dt: float) -> None:
|
||||||
|
a = self.a
|
||||||
|
sleep = a['sleep']
|
||||||
|
e = 0.45 * (dt * 30) * (1.0 if self.style == 'vector' else
|
||||||
|
self.intensity * (70.0 / self.persist_ms) ** 0.6)
|
||||||
|
if sleep > 0.001: # asleep: dim, with a slow breathing swell
|
||||||
|
breath = 0.5 + 0.5 * math.sin(2 * math.pi * 0.22 * T)
|
||||||
|
e *= 1.0 - sleep * (0.55 - 0.18 * breath)
|
||||||
|
|
||||||
|
ex, ey = a['pan_x'], a['tilt_y'] + 8.0 * sleep
|
||||||
|
eyes = [(82 + ex, 98 + ey), (158 + ex, 98 + ey)]
|
||||||
|
rx = 21 * a['eye_scale']
|
||||||
|
ry = 25 * a['eye_scale'] * a['open']
|
||||||
|
listening = self.state == 'listening'
|
||||||
|
lv = voice(T, 9, 190, 4.2) if listening or a['mouth_listen'] > 0.01 else None
|
||||||
|
listen_pulse = 0.12 * lv.env if listening else 0.0
|
||||||
|
|
||||||
|
# eyes: an ellipse morphing to a rotating 3:2 Lissajous figure while thinking
|
||||||
|
k = a['think']
|
||||||
|
for idx, (cx, cy) in enumerate(eyes):
|
||||||
|
ph = T * 1.6 + idx * math.pi * 0.5
|
||||||
|
|
||||||
|
def eye(u, cx=cx, cy=cy, ph=ph):
|
||||||
|
th = u * 2 * math.pi
|
||||||
|
ox, oy = rx * np.cos(th), ry * np.sin(th)
|
||||||
|
lx, ly = rx * 0.95 * np.sin(3 * th + ph), ry * 0.95 * np.sin(2 * th)
|
||||||
|
return cx + ox + (lx - ox) * k, cy + oy + (ly - oy) * k
|
||||||
|
self._trace(eye, 260 + round(160 * k), e * (1 + listen_pulse))
|
||||||
|
|
||||||
|
# mouth: a smile at rest, the live waveform when speaking or listening
|
||||||
|
mx0, mx1 = 70 + ex * 0.6, 170 + ex * 0.6
|
||||||
|
my = 166 + ey * 0.6 + 4.0 * sleep
|
||||||
|
sv = voice(T, 1, 118, 4.6) if a['mouth_speak'] > 0.01 else None
|
||||||
|
win_s, win_l = (2.4 / sv.f0 if sv else 0.0), (3.2 / lv.f0 if lv else 0.0)
|
||||||
|
speak_amp = 26 * a['mouth_speak'] * sv.env if sv else 0.0
|
||||||
|
listen_amp = 12 * a['mouth_listen'] * lv.env if lv else 0.0
|
||||||
|
smile = 7 * a['smile'] * (1 - a['mouth_speak']) * (1 - k * 0.8)
|
||||||
|
mouth_w = (mx1 - mx0) * (1 - 0.45 * k) * (1 - 0.5 * sleep)
|
||||||
|
m_start = (mx0 + mx1) / 2 - mouth_w / 2
|
||||||
|
|
||||||
|
def mouth(u):
|
||||||
|
x = m_start + u * mouth_w
|
||||||
|
c = 2 * u - 1
|
||||||
|
y = my + smile * (1 - c * c) - smile * 0.4
|
||||||
|
if speak_amp > 0.05:
|
||||||
|
y = y - speak_amp * voice_sample(sv, u * win_s, 0.0)
|
||||||
|
if sv.noise:
|
||||||
|
y = y - sv.noise * 5 * a['mouth_speak'] * (np.sin(u * 97 + T * 53) *
|
||||||
|
np.sin(u * 61 - T * 31))
|
||||||
|
if listen_amp > 0.05:
|
||||||
|
y = y - listen_amp * voice_sample(lv, u * win_l, 0.4)
|
||||||
|
return x, y
|
||||||
|
self._trace(mouth, 320, e * 0.65 * (0.95 - 0.3 * a['mouth_listen']))
|
||||||
|
self._jump_to(eyes[0][0] - rx, eyes[0][1], e)
|
||||||
|
|
||||||
|
# ------------------------------------------------------------ per-frame update
|
||||||
|
def _update(self, T: float, dt: float) -> None:
|
||||||
|
a = self.a
|
||||||
|
if self.state == 'looking' and T - self._state_t0 > LOOK_HOLD_S:
|
||||||
|
self.set_state(self._return_state, T)
|
||||||
|
st = self.state
|
||||||
|
in_look = st == 'looking'
|
||||||
|
tl = T - self._state_t0
|
||||||
|
|
||||||
|
a['sleep'] = _approach(a['sleep'], 1.0 if st == 'sleeping' else 0.0, 2.5, dt)
|
||||||
|
a['think'] = _approach(a['think'], 1.0 if st == 'thinking' else 0.0, 6, dt)
|
||||||
|
a['mouth_speak'] = _approach(a['mouth_speak'], 1.0 if st == 'speaking' else 0.0, 10, dt)
|
||||||
|
a['mouth_listen'] = _approach(a['mouth_listen'], 1.0 if st == 'listening' else 0.0, 8, dt)
|
||||||
|
a['smile'] = _approach(a['smile'], {'idle': 1.0, 'speaking': 0.6, 'sleeping': 0.0}.get(st, 0.25),
|
||||||
|
4, dt)
|
||||||
|
scale_t = 1.08 if st == 'listening' else 1.0
|
||||||
|
if in_look:
|
||||||
|
scale_t = 0.92 if tl < SQUINT_S else 1.14
|
||||||
|
a['eye_scale'] = _approach(a['eye_scale'], scale_t, 12, dt)
|
||||||
|
a['pan_x'] = _approach(a['pan_x'], -self.pan * 0.22, 7, dt) # +pan = robot's left = viewer's right
|
||||||
|
a['tilt_y'] = _approach(a['tilt_y'], -self.tilt * 0.3, 7, dt)
|
||||||
|
|
||||||
|
# blink, and the squint before a photo; none while thinking or asleep
|
||||||
|
open_t = SLEEP_OPEN if st == 'sleeping' else 1.0
|
||||||
|
if in_look and tl < SQUINT_S:
|
||||||
|
open_t = 0.35
|
||||||
|
if st not in ('thinking', 'sleeping') and not in_look:
|
||||||
|
if self._blink_t < 0 and T >= self._next_blink:
|
||||||
|
self._blink_t = T
|
||||||
|
if self._blink_t >= 0:
|
||||||
|
b = (T - self._blink_t) / 0.17
|
||||||
|
if b >= 1:
|
||||||
|
self._blink_t = -1.0
|
||||||
|
self._next_blink = T + 2.2 + float(self.rng.random()) * 4
|
||||||
|
else:
|
||||||
|
open_t = max(0.06, abs(1 - 2 * b))
|
||||||
|
if self._blink_t >= 0:
|
||||||
|
a['open'] = open_t
|
||||||
|
else:
|
||||||
|
rate = 3.0 if st == 'sleeping' else (6.0 if a['sleep'] > 0.1 else 18.0) # drift shut, wake gently
|
||||||
|
a['open'] = _approach(a['open'], open_t, rate, dt)
|
||||||
|
|
||||||
|
if self._flash_pending and tl > FLASH_AT_S: # shutter: bloom the whole tube once
|
||||||
|
if self.style == 'phosphor':
|
||||||
|
self.buf += 0.55
|
||||||
|
else:
|
||||||
|
self.flash = 0.6
|
||||||
|
self._flash_pending = False
|
||||||
|
self.flash = max(0.0, self.flash - dt * 2.5)
|
||||||
|
|
||||||
|
# ------------------------------------------------------------ tone map
|
||||||
|
def _glow(self, r: int) -> np.ndarray:
|
||||||
|
"""Two blur passes, at half resolution (the glow is smooth) and repeated back up."""
|
||||||
|
b = self.buf
|
||||||
|
half = (b[0::2, 0::2] + b[1::2, 0::2] + b[0::2, 1::2] + b[1::2, 1::2]) * np.float32(0.25)
|
||||||
|
rh = max(1, int(r / 2 + 0.5))
|
||||||
|
g = box_blur(box_blur(half, rh), rh)
|
||||||
|
return np.repeat(np.repeat(g, 2, axis=0), 2, axis=1)
|
||||||
|
|
||||||
|
def _energy_index(self) -> np.ndarray:
|
||||||
|
if self.style == 'vector':
|
||||||
|
k = np.float32(2.6 * self.intensity)
|
||||||
|
glow = box_blur(self.buf, 1)
|
||||||
|
v = (np.minimum(1, self.buf * k) * np.float32(0.9) + np.minimum(1, glow * k) * np.float32(0.25)
|
||||||
|
+ np.float32(self.flash * 0.35))
|
||||||
|
else:
|
||||||
|
r = 1 + round(self.focus * 5)
|
||||||
|
v = self.buf * np.float32(1.15 - self.focus * 0.55) + self._glow(r) * np.float32(0.6 + self.focus * 2.2)
|
||||||
|
v *= np.float32(_LUT_SCALE)
|
||||||
|
np.minimum(v, _LUT_SIZE - 1, out=v)
|
||||||
|
idx = v.astype(np.int32)
|
||||||
|
return np.rot90(idx, self.rotation // 90) if self.rotation else idx # free: the gather re-packs it
|
||||||
|
|
||||||
|
def frame(self, T: float, dt: float, fmt: str = 'rgb'):
|
||||||
|
"""Advance by dt seconds (clock at T). Returns an (H, W, 3) uint8 array for fmt='rgb', or
|
||||||
|
the frame as big-endian RGB565 bytes (what the ST7789 wants) for fmt='rgb565'."""
|
||||||
|
dt = min(dt, 0.1)
|
||||||
|
if self.style == 'vector':
|
||||||
|
self.buf.fill(0)
|
||||||
|
else:
|
||||||
|
self.buf *= np.float32(math.exp(-(dt * 1000) / self.persist_ms))
|
||||||
|
self._update(T, dt)
|
||||||
|
self._draw(T, dt)
|
||||||
|
self._flush()
|
||||||
|
idx = self._energy_index()
|
||||||
|
if fmt == 'rgb565':
|
||||||
|
return self._lut_565[idx].tobytes()
|
||||||
|
return self._lut_rgb[idx]
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
<?xml version="1.0"?>
|
||||||
|
<?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
|
||||||
|
<package format="3">
|
||||||
|
<name>hermes_face</name>
|
||||||
|
<version>0.1.0</version>
|
||||||
|
<description>Oscilloscope-phosphor face for the Hermes robot's ST7789 display.</description>
|
||||||
|
<maintainer email="noreply@example.com">K. Isom</maintainer>
|
||||||
|
<license>UNLICENSED</license>
|
||||||
|
|
||||||
|
<depend>rclpy</depend>
|
||||||
|
<depend>std_msgs</depend>
|
||||||
|
<depend>sensor_msgs</depend>
|
||||||
|
<exec_depend>python3-numpy</exec_depend>
|
||||||
|
<exec_depend>python3-pil</exec_depend>
|
||||||
|
|
||||||
|
<test_depend>python3-pytest</test_depend>
|
||||||
|
|
||||||
|
<export>
|
||||||
|
<build_type>ament_python</build_type>
|
||||||
|
</export>
|
||||||
|
</package>
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
[develop]
|
||||||
|
script_dir=$base/lib/hermes_face
|
||||||
|
[install]
|
||||||
|
install_scripts=$base/lib/hermes_face
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
from setuptools import find_packages, setup
|
||||||
|
|
||||||
|
package_name = 'hermes_face'
|
||||||
|
|
||||||
|
setup(
|
||||||
|
name=package_name,
|
||||||
|
version='0.1.0',
|
||||||
|
packages=find_packages(exclude=['test']),
|
||||||
|
data_files=[
|
||||||
|
('share/ament_index/resource_index/packages', ['resource/' + package_name]),
|
||||||
|
('share/' + package_name, ['package.xml']),
|
||||||
|
],
|
||||||
|
install_requires=['setuptools'],
|
||||||
|
zip_safe=True,
|
||||||
|
maintainer='K. Isom',
|
||||||
|
maintainer_email='noreply@example.com',
|
||||||
|
description='Oscilloscope-phosphor face for the Hermes robot.',
|
||||||
|
license='UNLICENSED',
|
||||||
|
entry_points={
|
||||||
|
'console_scripts': [
|
||||||
|
'face_node = hermes_face.face_node:main',
|
||||||
|
'demo = hermes_face.demo:main',
|
||||||
|
],
|
||||||
|
},
|
||||||
|
)
|
||||||
@@ -0,0 +1,101 @@
|
|||||||
|
import numpy as np
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from hermes_face.render import H, STATES, W, FaceRenderer, box_blur
|
||||||
|
|
||||||
|
|
||||||
|
def run(r, seconds, t0=0.0, fps=30):
|
||||||
|
t, frame = t0, None
|
||||||
|
for _ in range(int(seconds * fps)):
|
||||||
|
t += 1 / fps
|
||||||
|
frame = r.frame(t, 1 / fps)
|
||||||
|
return frame, t
|
||||||
|
|
||||||
|
|
||||||
|
def brightness(frame):
|
||||||
|
return float(frame.astype(np.float32).mean())
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('state', STATES)
|
||||||
|
@pytest.mark.parametrize('style', ['phosphor', 'vector'])
|
||||||
|
def test_every_state_draws_something(state, style):
|
||||||
|
frame, _ = run(FaceRenderer(style=style, state=state, seed=0), 3)
|
||||||
|
assert frame.shape == (H, W, 3) and frame.dtype == np.uint8
|
||||||
|
assert frame.max() > 100
|
||||||
|
|
||||||
|
|
||||||
|
def test_sleeping_is_dimmer_than_idle():
|
||||||
|
idle, _ = run(FaceRenderer(state='idle', seed=0), 4)
|
||||||
|
asleep, _ = run(FaceRenderer(state='sleeping', seed=0), 4)
|
||||||
|
assert brightness(asleep) < 0.8 * brightness(idle)
|
||||||
|
|
||||||
|
|
||||||
|
def test_wakes_up():
|
||||||
|
r = FaceRenderer(state='sleeping', seed=0)
|
||||||
|
_, t = run(r, 2)
|
||||||
|
r.set_state('idle', t)
|
||||||
|
frame, _ = run(r, 3, t)
|
||||||
|
idle, _ = run(FaceRenderer(state='idle', seed=0), 3)
|
||||||
|
assert r.a['open'] > 0.9 and abs(brightness(frame) - brightness(idle)) < 0.2 * brightness(idle)
|
||||||
|
|
||||||
|
|
||||||
|
def test_looking_returns_to_the_state_it_interrupted():
|
||||||
|
r = FaceRenderer(state='thinking', seed=0)
|
||||||
|
_, t = run(r, 1)
|
||||||
|
r.set_state('looking', t)
|
||||||
|
assert r.state == 'looking'
|
||||||
|
run(r, 2, t)
|
||||||
|
assert r.state == 'thinking'
|
||||||
|
|
||||||
|
|
||||||
|
def test_unknown_state_rejected():
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
FaceRenderer().set_state('dancing', 0.0)
|
||||||
|
|
||||||
|
|
||||||
|
def test_pan_moves_the_eyes():
|
||||||
|
def centroid_x(pan):
|
||||||
|
r = FaceRenderer(state='idle', seed=0, jitter=0)
|
||||||
|
r.set_pose(pan, 0)
|
||||||
|
frame, _ = run(r, 2)
|
||||||
|
lum = frame.astype(np.float32).sum(axis=(0, 2))
|
||||||
|
return float((lum * np.arange(W)).sum() / lum.sum())
|
||||||
|
assert centroid_x(-60) > centroid_x(0) > centroid_x(60) # +pan = robot's left = viewer's right
|
||||||
|
|
||||||
|
|
||||||
|
def test_box_blur_conserves_energy():
|
||||||
|
b = np.zeros((H, W), np.float32)
|
||||||
|
b[120, 120] = 1.0
|
||||||
|
out = box_blur(b, 2)
|
||||||
|
assert abs(out.sum() - 1.0) < 1e-3
|
||||||
|
assert np.allclose(out[118:123, 118:123], 1 / 25, atol=1e-5)
|
||||||
|
|
||||||
|
|
||||||
|
def test_rgb565_frame_matches_rgb_frame():
|
||||||
|
a = FaceRenderer(state='idle', seed=3)
|
||||||
|
b = FaceRenderer(state='idle', seed=3)
|
||||||
|
rgb, raw = None, None
|
||||||
|
for i in range(1, 40):
|
||||||
|
rgb = a.frame(i / 30, 1 / 30)
|
||||||
|
raw = b.frame(i / 30, 1 / 30, 'rgb565')
|
||||||
|
assert len(raw) == H * W * 2
|
||||||
|
px = np.frombuffer(raw, '>u2').reshape(H, W)
|
||||||
|
assert np.array_equal(px >> 11, rgb[..., 0] >> 3)
|
||||||
|
assert np.array_equal((px >> 5) & 0x3F, rgb[..., 1] >> 2)
|
||||||
|
assert np.array_equal(px & 0x1F, rgb[..., 2] >> 3)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('rotation', [90, 180, 270])
|
||||||
|
def test_rotation_turns_the_frame_counter_clockwise(rotation):
|
||||||
|
base = FaceRenderer(state='idle', seed=5, rotation=0)
|
||||||
|
turned = FaceRenderer(state='idle', seed=5, rotation=rotation)
|
||||||
|
a = b = None
|
||||||
|
for i in range(1, 30):
|
||||||
|
a = base.frame(i / 30, 1 / 30)
|
||||||
|
b = turned.frame(i / 30, 1 / 30)
|
||||||
|
assert np.array_equal(b, np.rot90(a, rotation // 90))
|
||||||
|
|
||||||
|
|
||||||
|
def test_bad_rotation_rejected():
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
FaceRenderer(rotation=45)
|
||||||
Reference in New Issue
Block a user