From a4cde40e85eed0673dd728a96f0c0602a1fdc312 Mon Sep 17 00:00:00 2001 From: "K. Isom" Date: Tue, 20 Oct 2026 01:03:08 -0700 Subject: [PATCH] docs: sweep after the fleet fixes; approve the 1.0 spec MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - README: custody chain and receipts replace epidemic forwarding; Ack frame; running on a robot (host crates, current limits); security; crate layout, constants and reading list. - AGENTS.md: current spec list, receipt constraints and review focus 7-8, first_seq rule, constants, host crates and the ros2 gate. - THREAT_MODEL.md: re-checked at 89041ea. F1, F2, F5, F17 fixed; F3 (single configured peer) and F4 partly fixed; new F18 (no scheduled flush, stale clock marker). - CHANGELOG: Unreleased section for the fleet milestone and fixes. - tines-bt README: single-peer limit; manual BLE check not yet recorded. - tines service rustdoc: oversized-frame behaviour, JSON tagging, clock. - 1.0 spec: approved (2026-10-20), including §10 and the adversarial fuzz mode. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 37 +++- CHANGELOG.md | 79 +++++++ README.md | 198 +++++++++++++++--- THREAT_MODEL.md | 161 ++++++++++---- .../specs/2026-10-18-tines-1-0-design.md | 9 +- tines-bt/README.md | 10 + tines/src/ipc.rs | 11 +- tines/src/main.rs | 6 +- 8 files changed, 424 insertions(+), 87 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 07362916..b56736f6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -18,11 +18,17 @@ architecture in the **spec first**: - **Core design (authoritative):** `docs/superpowers/specs/2026-10-16-tines-design.md` - **Addressed delivery (authoritative):** `docs/superpowers/specs/2026-10-18-tines-addressed-delivery.md` - **Fleet deployment v1 (authoritative):** `docs/superpowers/specs/2026-10-18-tines-fleet-deployment-design.md` — supersedes the addressed-delivery spec's epidemic forwarding (custody chain with handoff receipts). +- **Fleet fixes (done):** `docs/superpowers/specs/2026-10-19-tines-fleet-fixes.md` — security-assessment findings F1–F5 and F17. +- **tines-core 1.0 (next, core):** `docs/superpowers/specs/2026-10-18-tines-1-0-design.md` — wire version, public API, signature slot, core hardening (§10). +- **Host hardening (next, hosts):** `docs/superpowers/specs/2026-10-19-tines-host-hardening.md` — findings F10–F12. - **Implementation plans:** `docs/superpowers/plans/` The plans argue from the specs. Where a plan is silent, the **spec rules**. Do not edit the specs or the plans — they are design history, not source. `README.md` is the user-facing overview and should track the code. +`THREAT_MODEL.md` is the living security assessment: when a change fixes or +changes a finding, update its status there (§1 and §10) with the commit. +`CHANGELOG.md` records each release. ## Architecture (three layers) @@ -68,9 +74,14 @@ call `poll()` in its loop, and `observe()` when it sees something. - Addressed observations (`to = Some`) never appear in a summary Bloom filter, and an addressed observation past its `lifetime_ms` is never stored or forwarded. -- `Node::next_seq()` must be persisted by the application across reboots - (`boot_count << 32 | counter`); a repeated seq after reboot collides in the - ID hash and must dedupe silently (`insert` returns `Ok(false)`). +- `first_seq` passed to `Node::new` must not repeat across reboots + (`boot_count << 32`, with a persisted boot counter; `LogStore::first_seq` + provides it). A repeated seq after reboot collides in the ID hash and must + dedupe silently (`insert` returns `Ok(false)`). +- Receipts: a receive the store rejects (`Err`, e.g. `Full`) is never acked, + addressed or broadcast; an addressed duplicate whose local copy is released + is never acked; an `Ack` is accepted only from a peer the id was sent to + this contact. - `NodeId = [u8; 16]`, `ObservationId = [u8; 16]`, `Tick = u64` (local monotonic ms, supplied on every call). @@ -82,12 +93,17 @@ call `poll()` in its loop, and `observe()` when it sees something. 4. Frame from an address with no prior `Contact` → treated as implicit contact. 5. Tick going backwards → age saturates at stored value, never underflow. 6. Every emitted frame fits the transport's `mtu()`, including summaries. +7. Store-full receive → no `Ack`; custody stays with the sender (unit test and + the full-store sim property). +8. Released addressed duplicate → no `Ack`. ### Compile-time constants (`tines-core/src/consts.rs`) `MAX_TOPIC=64`, `MAX_PAYLOAD=1024`, `MAX_PEERS=8`, `SUMMARY_BYTES=512`, `BLOOM_K=3`, `SUMMARY_INTERVAL_MS=2000`, `REASSEMBLY_TIMEOUT_MS=5000`, -`SEND_QUEUE_CAP=256`, `MAX_FRAME=1500`. Match the exact values above. +`SEND_QUEUE_CAP=256`, `MAX_FRAME=1500`, `ACK_IDS_CAP=90`, +`EVENT_QUEUE_CAP=64`, `BCAST_SENT_CAP=256`, `ACK_BUFFER_CAP=256`. Match the +exact values above. ## Crate layout @@ -96,10 +112,21 @@ tines/ tines-core/ no_std core: Observation, Store/Transport/BeliefPolicy traits, exchange state machine, Node facade, in-memory store, reference policies - tines-sim/ std: simulator, contact traces, convergence and relay + tines-sim/ std: simulator, contact traces, convergence, relay and custody property tests + tines-store/ std: LogStore, the persistent append-only backend + tines-bt/ std: BLE transport over BlueZ (bluer) + tines/ std: standalone service, Unix-socket JSON IPC + tines-ros2/ std: ROS2 node; rclrs behind the `ros2` feature ``` +Only `tines-core` is headed for 1.0; the host crates stay 0.x (1.0 spec, D4). +They're std-only and may use threads and I/O, but they're still the trust +boundary for radio, IPC and disk input (see `THREAT_MODEL.md`). + +`tines-ros2` with `--features ros2` is outside the default gate: check it +with the ROS2 Docker job (`Dockerfile.ros2`, `scripts/check-ros2.sh`). + ## Not started `tines-udp`, `tines-sqlite`, `tines-cli`, and `tines-serial` (from the core diff --git a/CHANGELOG.md b/CHANGELOG.md index b70c0da5..61f1f01c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,85 @@ All notable changes to the tines crates. Versions follow [Semantic Versioning](https://semver.org/); before 1.0, a minor bump may break the API. +## Unreleased + +The fleet-deployment milestone and the fleet fixes (security assessment +findings F1–F5 and F17; see `THREAT_MODEL.md`). The workspace version is +still 0.3.0. + +### Added + +- **Handoff receipts** (tines-core). The new `Ack` frame (the last `Frame` + variant) carries up to `ACK_IDS_CAP` (90) ids. Acks are sent as the + exchange runs and are accepted only from a peer the id was sent to during + the current contact. + - An ack for an addressed observation moves custody: the sender releases + its copy (`Store::release`) and never offers it again. + - An ack for a broadcast is confidence only. It surfaces as a + `ConfidenceEvent { peer, id }` from the new `Node::drain_events`. +- New constants: `ACK_IDS_CAP`, `EVENT_QUEUE_CAP`, `BCAST_SENT_CAP`, + `ACK_BUFFER_CAP`. +- `wire::frame_sender(bytes) -> Option`: a frame's claimed sender, + for transports that bind a connection to an identity. +- `Node::store_mut`, so a host can call backend methods such as + `LogStore::flush`. +- **New host crates:** + - `tines-store`: `LogStore`, a persistent append-only backend. + - `tines-bt`: a BLE transport over BlueZ. + - `tines`: a standalone service with Unix-socket JSON IPC. + - `tines-ros2`: a ROS2 node; rclrs is behind the `ros2` feature. + +### Changed + +- **Addressed delivery is a custody chain.** It used to be epidemic + forwarding. A message has one custodian at a time, which hands it on and + releases it when the next holder acks. A lost ack can leave two + custodians for a while; a message is never dropped by the protocol. +- Bounce records carry a direction, so only records of an actual send + validate an ack. +- **`LogStore` log format v1** (incompatible; there's no migration, so delete + old logs): + - The file starts with a `"TINESLOG"` header and a format version. + - Each record has a sync marker, a length and a CRC-32C. + - Ids are recomputed on load, not stored. +- `LogStore` persists a boot counter, and `first_seq()` returns + `boot_count << 32`. Both hosts seed `Node::new` from it, and the + `first_seq` config field is gone. +- `LogStore::release` writes a `Release` record immediately. `flush` writes + a clock marker, so ages carry across a restart. +- `LogStore::open` takes an exclusive lock and creates the file `0600`. +- tines-bt rewrite: + - L2CAP SeqPacket channels, one frame per packet. + - Every node advertises and scans; the smaller `NodeId` dials and the + larger accepts. + - Connections are bound to the advertised `NodeId`, and frames whose + claimed sender differs are dropped. + - Connections emit `Contact` and `Lost`. + +### Fixed + +- A receiver whose store returned `Err` (for example `Full`) still acked, so + the sender released custody of a message nobody kept. Now nothing is + acked unless it was stored, addressed or broadcast (F1). +- The standalone service never polled its `Node`, so it never exchanged + anything. It now runs the exchange on a poll loop (F2). +- tines-bt had no listener, lost frame boundaries on a stream socket, and + stamped every frame with the configured peer (F3). +- Sequence numbers repeated after every restart, and released flags were + never persisted (F4). +- One corrupt record in the middle of the log truncated everything after it. + Recovery now skips bad records and truncates only a torn tail, with an + fsync of the file and its directory (F5). +- The service's tick was wall-clock *seconds*. Both hosts now use one + monotonic millisecond clock (F17). + +### Known issues + +- The service talks to exactly one configured `peer`. The multi-peer + allowlist the fleet-fixes spec describes isn't wired up (F3). +- Neither host flushes on a timer, and tines-ros2 never flushes or trims + (F18). + ## 0.3.0 — 2026-10-18 ### Breaking diff --git a/README.md b/README.md index 0d46b9b1..85992f75 100644 --- a/README.md +++ b/README.md @@ -21,12 +21,15 @@ hear each other. - [Observations](#observations) - [The exchange protocol](#the-exchange-protocol) - [Addressed delivery](#addressed-delivery) +- [Receipts](#receipts) - [Usage](#usage) - [Writing a transport](#writing-a-transport) - [Writing a store](#writing-a-store) - [The simulator](#the-simulator) +- [Running on a robot](#running-on-a-robot) - [Crate layout](#crate-layout) - [Building and CI](#building-and-ci) +- [Security](#security) - [Where to read more](#where-to-read-more) ## Design at a glance @@ -43,6 +46,8 @@ hear each other. defensively; malformed bytes are dropped. - **The application sets the pace.** `Node::poll` emits at most one frame per call, so duty cycle and airtime are the application's call. +- **No trust layer yet.** Anyone in radio range can read and inject. See + [Security](#security). ## Architecture @@ -109,9 +114,11 @@ saturates rather than underflowing. ### Sequence numbers must survive reboots `seq` is part of the id, so a robot must not reuse one for different content. -Persist `Node::next_seq()` and pass it back as `first_seq`, typically as -`boot_count << 32 | counter`. A repeated `seq` with identical content simply -dedupes (`insert` returns `Ok(false)`). +Pass `Node::new` a `first_seq` that is higher than any seq used before, +typically `boot_count << 32`, with a boot counter persisted across reboots. +`tines-store`'s `LogStore` keeps that counter for you: seed the node with +`store.first_seq()`. A repeated `seq` with identical content simply dedupes +(`insert` returns `Ok(false)`). ## The exchange protocol @@ -169,8 +176,16 @@ classDiagram count: u8 data: bytes } + class Ack { + from: NodeId + ids: ≤90 ObservationIds + } ``` +A transport that binds a connection to a peer's identity can check a frame's +claimed sender with `tines_core::wire::frame_sender`, without decoding the +whole frame. + ### Fragmentation An observation that doesn't fit the transport's MTU is split into up to 64 @@ -217,9 +232,9 @@ flowchart TD Most observations are broadcasts: everyone gets everything. An observation can also be **addressed** to one robot with `Node::observe_to`, e.g. a tasking from -a base station. Addressed messages travel by store-and-forward: any robot that -meets a carrier becomes a carrier too, until the message reaches its recipient -or its lifetime runs out. +a base station. Addressed messages travel along a **custody chain**: a message +has one custodian at a time, and each handoff is confirmed by a receipt before +the old custodian lets go. ```mermaid sequenceDiagram @@ -229,28 +244,60 @@ sequenceDiagram participant R3 as Robot 3 (recipient) Base->>Base: observe_to(R3, lifetime 60 s, "task", …) Note over Base,R1: contact - Base->>R1: Obs to=R3 (epidemic forward) + Base->>R1: Obs to=R3 (forward) + R1->>Base: Ack + Note over Base: releases its copy:
no longer offered to anyone Note over R1,R2: later contact - R1->>R2: Obs to=R3 (epidemic forward) + R1->>R2: Obs to=R3 (forward) + R2->>R1: Ack Note over R2,R3: later contact R2->>R3: Obs to=R3 (delivery) + R3->>R2: Ack Note over R3: shows up in R3's belief() - Note over R1,R2: carriers drop it once age > lifetime + Note over Base,R3: anyone still holding it drops it once age > lifetime ``` +- A custodian offers the message to the recipient when it meets it + (**delivery**), and otherwise to any peer it meets (**forwarding**). + Delivery is preferred. Custody and broadcast take turns for each peer, so + neither can starve the other. +- The receiver acks only once it has stored the message. When the ack comes + back, the sender **releases** its copy: the copy stays in its store until + it expires or is trimmed, but is never offered again. +- If an ack is lost, the sender keeps offering the message, so a message can + briefly have two custodians. That costs airtime, never the message. A + custodian that fails after handing the message on does lose it; that's + accepted. - Addressed messages **never appear in a summary**, so their existence doesn't leak to unrelated robots through gossip. -- **Delivery** (the peer is the recipient) is preferred over **forwarding**. - Custody and broadcast take turns for each peer, so neither can starve the - other. - **Bounce suppression:** within one contact, a robot never offers a message back to the peer it came from, or to a peer that already has it. - **Expiry is local and silent.** A message older than its `lifetime_ms` is - never forwarded and never stored. There are no acks and no cancellation - messages. + never forwarded and never stored. There are no end-to-end delivery reports + and no cancellation messages. - `belief()` sees broadcasts and messages addressed *to me*. Messages carried for others exist only to be forwarded. +## Receipts + +Each robot acknowledges what it receives with `Ack` frames, sent as it goes +during a contact (up to 90 ids per frame), for both kinds of observation: + +| Kind | What an ack means to the sender | +|-----------|-------------------------------------------------------------------------| +| addressed | custody has moved: release my copy | +| broadcast | confidence only: "peer P has observation X"; nothing changes in storage | + +An ack is accepted only from a peer the id was actually sent to during the +current contact. A robot never acks an observation it failed to store (for +example, because its store is full), and never acks an addressed duplicate +it has already released, since that would hand custody back to a robot that +no longer carries the message. + +Broadcast confirmations surface as `ConfidenceEvent { peer, id }` from +`Node::drain_events`. The event queue is bounded (64): events are telemetry, +and losing one never loses data. + ## Usage `tines-core` exposes a single application-facing facade, `Node`, generic @@ -260,18 +307,13 @@ over any `Store` and `Transport`: use tines_core::{LatestWins, MemStore, Node, NodeId, TrimPolicy}; let me: NodeId = [7; 16]; -let boot_count = 0u64; -let counter = 3u64; +let boot_count = 0u64; // persisted and incremented on every boot let store = MemStore::new(); let transport = /* any Transport implementation */; -// first_seq must survive reboot: boot_count << 32 | counter -let mut node: Node = Node::new( - me, - boot_count << 32 | counter, - store, - transport, -); +// first_seq must not repeat across reboots. With tines-store's LogStore, +// pass store.first_seq() instead. +let mut node: Node = Node::new(me, boot_count << 32, store, transport); // Record something you saw. observed_at is an optional absolute clock // (ms since epoch); `now` is a local monotonic Tick on every call. @@ -291,6 +333,11 @@ let stored = node.poll(500); // (e.g. the youngest observation wins). let belief = node.belief(b"door", &LatestWins, 500); +// Which peers have confirmed holding which of your broadcasts. +for event in node.drain_events() { + println!("peer {:?} has {:?}", event.peer, event.id); +} + // Keep the store bounded. node.trim( &TrimPolicy { @@ -370,11 +417,15 @@ flash or SQLite backend implements the `Store` trait: | `for_each_id` | every stored id, any order | | `for_each_broadcast_id` | ids with `to == None`; these feed the summary | | `for_each_addressed_id` | ids with `to == Some(_)`; scanned for custody on every poll | +| `release` | mark an id as handed off, so it's never offered again; a no-op if the id isn't stored | +| `is_released` | whether an id has been released; a re-inserted observation starts un-released | | `for_each_topic` | age-adjusted observations whose topic starts with a prefix | | `trim` | apply a `TrimPolicy` (per-topic, then age, then count); return how many were removed | The store must remember when each observation was inserted (`stored_at`, in -the caller's `Tick`s) so it can report current ages. +the caller's `Tick`s) so it can report current ages. `insert` may return +`Err(StoreError::Full)`; the exchange then doesn't ack, so the sender keeps +custody. `for_each_addressed_id` has a default implementation that visits every id, which is correct but makes the exchange read every broadcast observation on @@ -420,10 +471,72 @@ check that: - no store depends on the order of contacts; - a store only ever grows during a run (union is monotone); - garbage frames never panic and never change a store; -- addressed messages reach their recipient along a connected path. +- an unexpired addressed message always has a holder, a custodian or its + recipient, including when some robots' stores are full. -`tines-sim/tests/relay.rs` adds scripted relay scenarios, including a message -that expires before it reaches its recipient and is never stored. +`tines-sim/tests/relay.rs` adds scripted relay scenarios: a message relayed +down a custody chain to its recipient, one that expires before it gets there +and is never stored, and a released duplicate that isn't acked. + +## Running on a robot + +`tines-core` is only the library. The host crates put it on a Linux robot +(a Pi, or a Jetson running ROS2): + +```mermaid +flowchart LR + subgraph robot["One robot"] + direction TB + clients(["IPC clients"]) -- "Unix socket,
length-prefixed JSON" --> svc["tines
(service)"] + ros(["ROS2 nodes"]) -- "/tines/… topics" --> rosnode["tines-ros2
(TinesNode)"] + svc & rosnode --> core["tines-core
Node"] + core --> log[("tines-store
LogStore")] + core --> bt["tines-bt
BtTransport"] + end + bt <-- "BLE L2CAP" --> other(["other robots"]) +``` + +A robot runs either the service or the ROS2 node, not both: each owns its +`Node` and its log. + +- **`tines-store`**: `LogStore`, an append-only log file. + - Each record is checksummed (CRC-32C). Recovery skips a corrupt record and + truncates only a torn tail. + - Releases are written as they happen. + - A boot counter provides `first_seq()`. + - `flush()` writes a clock marker, so ages carry across a restart, and + fsyncs the log. + - The file is created `0600` and locked, so only one process can open it. +- **`tines-bt`**: BLE over BlueZ, using L2CAP SeqPacket channels (one + frame per packet). + - Every robot advertises its `NodeId` and scans. The smaller `NodeId` + dials and the larger accepts. + - A connection is bound to the advertised `NodeId`, and frames claiming + any other sender are dropped. + - Connections produce `Contact` and `Lost` events. See + [`tines-bt/README.md`](tines-bt/README.md). +- **`tines`**: the standalone service, for robots without ROS2. + - It runs the exchange on a poll loop and answers `Observe`, `Belief`, + `Events` and `Status` requests on a Unix socket. + - It's configured by a JSON file: `node_id`, `peer`, `log_path`, + `socket_path` and `poll_interval_ms`. +- **`tines-ros2`**: a ROS2 node. + - It publishes each tines topic under `/tines/` and confidence events on + `/tines/receipts`. + - It turns messages on those topics into observations. + - rclrs is behind the `ros2` feature, so the crate builds without a ROS2 + toolchain. + +The hosts use a monotonic millisecond clock that starts at 0 when the +process does. Neither host sets `observed_at` yet. + +**Current limits** (tracked in [`THREAT_MODEL.md`](THREAT_MODEL.md)): +- The service talks to one configured `peer`. +- The hosts don't yet flush on a timer. +- The host-hardening work (trim defaults, bounded channels, socket + permissions, ROS2 isolation) hasn't landed. + +The two-device BLE check in `tines-bt/README.md` hasn't been recorded yet. ## Crate layout @@ -435,7 +548,11 @@ tines/ `alloc` is a default feature; without it, tines-core builds for thumbv7em-none-eabihf (only MemStore needs alloc). tines-sim/ std: the simulator, contact traces, and the convergence / - order-independence / relay property tests. + order-independence / custody property tests. + tines-store/ std: LogStore, the persistent append-only backend. + tines-bt/ std: the BLE transport (BlueZ via bluer). + tines/ std: the standalone service with Unix-socket IPC. + tines-ros2/ std: the ROS2 node (rclrs behind the `ros2` feature). ``` Fixed limits, all compile-time constants in `tines-core/src/consts.rs`: @@ -451,6 +568,10 @@ Fixed limits, all compile-time constants in `tines-core/src/consts.rs`: | `REASSEMBLY_TIMEOUT_MS` | 5000 | partial observation discard time | | `SEND_QUEUE_CAP` | 256 | ids queued per peer | | `MAX_FRAME` | 1500 | largest frame (and MTU) supported | +| `ACK_IDS_CAP` | 90 | ids in one `Ack` frame | +| `EVENT_QUEUE_CAP` | 64 | confidence events awaiting `drain_events`| +| `BCAST_SENT_CAP` | 256 | broadcast sends remembered per peer, to validate acks | +| `ACK_BUFFER_CAP` | 256 | received ids waiting to be acked, per kind | ## Building and CI @@ -465,13 +586,28 @@ sh scripts/ci.sh GitHub Actions runs the same script on every push to `master` and on every pull request. +## Security + +tines has **no trust layer yet**, by design. Anyone in radio range can read +everything, inject observations under any identity, and take custody of +addressed messages. That's acceptable only where the radio environment is +trusted. Signatures are planned for 1.1, with room for them reserved in the +1.0 wire format. + +[`THREAT_MODEL.md`](THREAT_MODEL.md) has the full assessment: trust +boundaries, adversaries, findings, and which spec fixes each one. + ## Where to read more The specs are authoritative; the plans argue from them. - **Core design:** `docs/superpowers/specs/2026-10-16-tines-design.md` - **Addressed delivery:** `docs/superpowers/specs/2026-10-18-tines-addressed-delivery.md` + (its epidemic forwarding is superseded by the fleet spec's custody chain) +- **Fleet deployment:** `docs/superpowers/specs/2026-10-18-tines-fleet-deployment-design.md` +- **Fleet fixes:** `docs/superpowers/specs/2026-10-19-tines-fleet-fixes.md` +- **Host hardening (next for the hosts):** `docs/superpowers/specs/2026-10-19-tines-host-hardening.md` +- **tines-core 1.0 (next for the core):** `docs/superpowers/specs/2026-10-18-tines-1-0-design.md` +- **Threat model:** [`THREAT_MODEL.md`](THREAT_MODEL.md) - **Plans:** `docs/superpowers/plans/` - -Not yet built (a second plan): `tines-udp`, `tines-sqlite`, `tines-cli`, and -`tines-serial`. +- **Changes:** [`CHANGELOG.md`](CHANGELOG.md) diff --git a/THREAT_MODEL.md b/THREAT_MODEL.md index e36689e3..d61e3d75 100644 --- a/THREAT_MODEL.md +++ b/THREAT_MODEL.md @@ -3,13 +3,16 @@ - **Assessed:** `master` at `3b1a644` (tines-core / tines-store / tines-bt / tines-ros2 / tines 0.3.0, fleet-deployment milestone mostly landed) - **Date:** 2026-10-19 -- **Status:** first version. This is a living document, not a spec: update - it when the code or the trust assumptions change. +- **Updated:** 2026-10-20 at `89041ea`, after the fleet fixes landed. The + findings keep their original text; each fixed one ends with a **Fixed** + or **Partly fixed** line, and §6 adds F18, found in the re-check. +- **Status:** living document.It's not a spec: update it when the code or the + trust assumptions change. - **Related:** the 1.0 spec (`docs/superpowers/specs/2026-10-18-tines-1-0-design.md`), especially §2a (signature slot) and §2b (age and the 1.1 threat model). Every finding below was checked against the source; `file:line` references -are to `3b1a644`. +are to `3b1a644` unless a line says otherwise. ## 1. Summary @@ -43,10 +46,22 @@ Recommended order: fix F1–F5 in the fleet milestone (they're bugs), harden the hosts (F10–F12) before any robot leaves the lab, and settle the 1.1 threat questions (§8) before designing signing. -**Resolved 2026-10-19** (the fleet-fixes milestone, through `7df9f41`): -F1 `2738de4` · F2 `d98895b` · F3 `7df9f41` · F4 `848721a` · -F5 `4718b6f`+`24b5a78` · F17 `6e2dc6f`. F6 and F8–F14 remain — F6/F8/F9/F13 -are in the 1.0 spec §10; the rest in the host-hardening spec. +**Fleet fixes, 2026-10-19** (through `89041ea`): + +| # | Status | Commit | +|---|---|---| +| F1 | Fixed | `2738de4` | +| F2 | Fixed | `d98895b` | +| F3 | **Partly fixed**: framing, roles, binding and `Contact`/`Lost` are in, but the host is still limited to one configured peer (see F3) | `7df9f41` | +| F4 | **Partly fixed**: seqs and releases are durable, but flushing isn't on a schedule (F18) | `848721a` | +| F5 | Fixed | `4718b6f`, `24b5a78` | +| F17 | Fixed | `6e2dc6f` | + +Still open: +- F6, F8, F9 and F13: the 1.0 spec §10. +- F10–F12: the host-hardening spec. +- F7 and F14: 1.1. +- F18 (new) and the rest of F3: host work, not yet specced (§10). ## 2. System and trust boundaries @@ -72,9 +87,9 @@ flowchart LR | Boundary | What crosses it | Who can be on the other side today | |---|---|---| -| **B1 radio** | Advertisements (NodeId in clear), L2CAP frames: Summary, Obs, Frag, Ack | Any BLE device in range. There's no pairing, encryption or allowlist. | +| **B1 radio** | Advertisements (NodeId in clear), L2CAP frames: Summary, Obs, Frag, Ack | Any BLE device in range. There's no pairing or encryption. Since `7df9f41` a connection is bound to an advertised NodeId on an allowlist, but advertisements are unauthenticated, so anyone can advertise an allowed NodeId. | | **B2 local IPC / ROS2** | Observations in, beliefs/payloads and events out | Any local user who can reach the socket (permissions come from umask); any participant on the ROS2 DDS domain (no SROS2). | -| **B3 storage** | The observation log, released set, `next_seq` | Anyone with write access to the log file (mode from umask). | +| **B3 storage** | The observation log: observations, releases, the boot counter | Anyone with write access to the log file. Since `4718b6f` the file is created `0600` (its directory `0700` when the store creates it) and locked against a second opener. | | Core ↔ transport | `Addr`, raw frames | Trusted code. Core relies on the transport to bind `Addr` to a NodeId (AGENTS.md: "core never maps Addr to NodeId"). | ## 3. Assets @@ -110,16 +125,19 @@ in software), compromise of BlueZ/the kernel, and supply chain. ## 5. Security properties: claimed vs actual -| Property | Claimed (specs/AGENTS.md) | Actual at `3b1a644` | -|---|---|---| -| No panic on untrusted input (core) | Hard constraint | **Holds** as far as reviewed: saturating age arithmetic, guarded shifts and indexing, `expect`s unreachable from wire input (exchange.rs:437, 1092; frag.rs:147). The sim's `garbage_frames_never_panic` covers random bytes only (§9). | -| Malformed frames dropped | Hard constraint | **Holds** in core (wire.rs:64-69). The 1.0 spec adds reporting. | -| Bounded memory in core | Fixed-capacity buffers | **Holds** for the exchange (§6 table). **Doesn't hold for the system**: MemStore and LogStore are unbounded without `trim`, and no host calls `trim` (F10). | -| Custody: ≥ 1 holder (no carrier failure) | Fleet spec, sim property | **Breaks without an attacker** when a store is full (F1). Breaks trivially under A1 (F7). | -| Frames bound to the advertised NodeId | Fleet spec ("frames whose `from` differs … are dropped at the transport") | **Doesn't hold** (F3, F6). | -| Released messages not re-forwarded after reboot | Fleet spec | **Doesn't hold**: release flags are never flushed (F4). Safe direction (duplicate custodian), but wasteful. | -| `next_seq` persisted across reboots | Core contract | **Doesn't hold** in either host (F4). | -| Age continues across reboot | Fleet spec | Partly. Hosts open the store at tick 0, and the service's tick is wall-clock *seconds* (F17). | +The third column is the original assessment. The fourth is the state after +the fleet fixes. + +| Property | Claimed (specs/AGENTS.md) | Actual at `3b1a644` | At `89041ea` | +|---|---|---|---| +| No panic on untrusted input (core) | Hard constraint | **Holds** as far as reviewed: saturating age arithmetic, guarded shifts and indexing, `expect`s unreachable from wire input (exchange.rs:437, 1092; frag.rs:147). The sim's `garbage_frames_never_panic` covers random bytes only (§9). | Unchanged. | +| Malformed frames dropped | Hard constraint | **Holds** in core (wire.rs:64-69). The 1.0 spec adds reporting. | Unchanged; 1.0 adds reporting. | +| Bounded memory in core | Fixed-capacity buffers | **Holds** for the exchange (§6 table). **Doesn't hold for the system**: MemStore and LogStore are unbounded without `trim`, and no host calls `trim` (F10). | Unchanged in practice: the service calls `trim` with an empty default policy, tines-ros2 never does (F10, F18). | +| Custody: ≥ 1 holder (no carrier failure) | Fleet spec, sim property | **Breaks without an attacker** when a store is full (F1). Breaks trivially under A1 (F7). | Holds without an attacker (F1 fixed, with a sim property). Still breaks under A1 (F7). | +| Frames bound to the advertised NodeId | Fleet spec ("frames whose `from` differs … are dropped at the transport") | **Doesn't hold** (F3, F6). | **Holds at the transport** for tines-bt (F3), against unauthenticated adverts. Core still re-reads `from` per frame (F6). | +| Released messages not re-forwarded after reboot | Fleet spec | **Doesn't hold**: release flags are never flushed (F4). Safe direction (duplicate custodian), but wasteful. | **Holds**: `Release` records are appended at once (not fsynced until the next flush, so a power cut can still lose one). | +| `next_seq` persisted across reboots | Core contract | **Doesn't hold** in either host (F4). | **Holds**: a persisted boot counter seeds `first_seq` in both hosts. | +| Age continues across reboot | Fleet spec | Partly. Hosts open the store at tick 0, and the service's tick is wall-clock *seconds* (F17). | Partly. Monotonic ms (F17 fixed) and clock markers fold age across a restart, but only up to the last flush, which may be long ago or never (F18). | ## 6. Findings @@ -192,8 +210,20 @@ for a poll loop, `docs/superpowers/plans/2026-10-18-tines-fleet-deployment.md:20 - Decode the advertised NodeId and bind the connection to it. - Drop frames whose `from` differs from the bound NodeId. - Emit `Contact`/`Lost`. -- **Fixed:** `7df9f41` (SeqPacket framing, advertise+scan roles, UUID-checked - NodeId binding, `frame_sender` from-check, `Contact`/`Lost`, multi-peer). +- **Partly fixed:** `7df9f41` (SeqPacket framing, advertise+scan roles, + UUID-checked NodeId binding, `frame_sender` from-check, `Contact`/`Lost`). + Re-checked at `89041ea`: + - The fleet-fixes spec's allowlist (empty = any tines advertiser) and + multi-peer support aren't reachable. The service config still takes one + required `peer`. `BluerRadio::new` builds the allowlist as exactly + `{peer}`, and `BtTransport::poll` drops every frame and event whose + address isn't that peer (tines-bt/src/transport.rs:186-202, 274). The + radio loop can hold several connections, but only one is ever used, so + each robot exchanges with one configured partner. + - Inbound and outbound channels are still unbounded (F10). + - No BLE security level is requested (unchanged; 1.1). + - No two-radio check of the SeqPacket path is recorded yet. + tines-bt/README.md gives the procedure but no result. **Later (1.1):** authenticate the link (spec §2b). Advertised NodeIds are unauthenticated, so the binding only becomes meaningful once there's a @@ -221,7 +251,9 @@ handshake. `Node` from it. - Flush on release and on a timer. - Add a host-level restart test. -- **Fixed:** `848721a`. +- **Partly fixed:** `848721a`. The boot counter seeds `first_seq` in both + hosts, and releases are appended as records immediately. Flushing isn't + on a schedule, though; see F18. ### F5 — Log recovery can destroy good data (Medium, A0/A5) @@ -290,6 +322,8 @@ custody of every addressed message in range and drop it. frame, and removed only on `Lost` (exchange.rs:674-708). A full table drops all frames from new addresses. There's no idle eviction. With BLE never emitting `Lost` (F3), entries effectively never leave on real hardware. +(Since `7df9f41` tines-bt emits `Lost`, so honest peers leave; spoofed +addresses on other transports still can't be evicted.) Eight spoofed addresses lock a robot out of exchange until restart. **Fix:** idle-timeout eviction in core (a local constant), and real @@ -315,6 +349,10 @@ re-queues what was already sent. - `MemStore` and `LogStore` are bounded only by `Node::trim`. Neither host calls it. The log also never compacts (log.rs: append-only, and every flush appends the full released list). + - *At `89041ea`:* the service calls `trim` with `TrimPolicy::default()`, + which removes nothing; tines-ros2 never calls it. Releases are now one + small record each rather than a growing list, but the log still never + compacts. - Unbounded channels: tines-bt's inbound and outbound `mpsc` (transport.rs:207-208), and tines-ros2's inbound `mpsc` (node.rs:152). - The IPC server allocates up to 1 MiB per request frame. @@ -433,6 +471,37 @@ full preimage, not the id. everywhere in the host, and open the store with the same clock. - **Fixed:** `6e2dc6f`. +### F18 — Hosts don't flush on a schedule (Medium, A0) + +*Found in the re-check at `89041ea`.* `LogStore::flush` writes the clock +marker that carries age across a restart, and it's the only `fsync` for +inserts and trims (log.rs module docs, "Durability contract"). + +- The service runs `trim` and `flush` only after `POLL_CAP` (64) polls that + each stored something new (tines/src/ipc.rs:199-217). The counter + advances only while observations arrive, so a robot that is mostly + offline, the normal case, may never flush. +- tines-ros2 never calls `flush` or `trim`. +- Even when it runs, `flush` writes `LogStore`'s own tick as the clock + marker, and that tick advances only when something new is inserted + (tines-store/src/log.rs:698). On a quiet robot the marker is stale, and + the time since the last insert is lost. +- **Effect:** + - After a restart, every stored observation's age is folded only up to the + last clock marker. Time since that marker is lost, so facts look fresher + than they are and addressed messages outlive their `lifetime_ms`, by up + to the whole previous uptime. That is F17's effect again, by another + route. + - On a power cut, observations inserted since the last flush can be lost. + That includes a robot's own observations, though their seqs aren't + reused. +- **Fix:** give flush the caller's `now` (proposed for 1.0: `Store::flush(now)` + and `Node::flush(now)`, replacing `Node::store_mut`), and + flush and trim on a wall-time schedule (every + `flush_interval_ms`, from the host's monotonic clock) in both hosts, and + on clean shutdown. The fleet-fixes spec asked for "a scheduled flush". The + trim defaults belong to host hardening (F10). + ## 7. Threats by STRIDE, mapped to findings | | Threat | Status today | Findings | Planned mitigation | @@ -477,32 +546,44 @@ The sim covers honest-node custody properties and random garbage frames - well-formed malicious frames: forged acks, spoofed `from`/`observer`, NodeId changes mid-contact; -- a store returning `Full` (F1); +- ~~a store returning `Full` (F1)~~ (covered since `2738de4`: a unit test + and the full-store custody sim property); - summary floods and peer-table exhaustion (F8, F9); - reassembly abuse; -- `LogStore` corruption in the middle, an oversized `Meta`, concurrent - openers, a full disk, and restart and seq continuity (F4, F5); -- anything in the hosts: IPC abuse, ROS2 injection and self-echo - (F11, F12). +- a full disk (`LogStore` corruption in the middle, oversized records, + concurrent openers, and restart and seq continuity are covered since + `848721a` and `4718b6f`); +- a quiet host restarting without having flushed (F18); +- most of the hosts: IPC abuse, ROS2 injection and self-echo (F11, F12). + The service now has a two-instance exchange test (`d98895b`), and + tines-bt has mock-radio tests for framing, binding and contact events + (`7df9f41`). The 1.0 fuzz harness (spec §7) should take an *adversarial peer* mode, not just random bytes: structured frames with chosen ids and identities. ## 10. Where each finding is being fixed -| Findings | Where | Spec | -|---|---|---| -| F1–F5, F17 | Fleet milestone (M3), now | `docs/superpowers/specs/2026-10-19-tines-fleet-fixes.md` | -| F10–F12 | Host hardening, before a robot leaves the lab | `docs/superpowers/specs/2026-10-19-tines-host-hardening.md` | -| F6, F8, F9, F13 | tines-core 1.0 | `docs/superpowers/specs/2026-10-18-tines-1-0-design.md` §10 | -| F7, F14, §8 | 1.1 (threat model, link authentication, signing) | 1.0 spec §2a, §2b | -| F15, F16 | Accepted for now | — | +| Findings | Where | Spec | Status | +|---|---|---|---| +| F1, F2, F5, F17 | Fleet milestone (M3) | `docs/superpowers/specs/2026-10-19-tines-fleet-fixes.md` | Fixed (§1) | +| F3, F4 | Fleet milestone (M3) | same | Partly fixed; the rest has no spec yet | +| F18 | Host work, with or before host hardening | none yet | Open | +| F10–F12 | Host hardening, before a robot leaves the lab | `docs/superpowers/specs/2026-10-19-tines-host-hardening.md` | Open | +| F6, F8, F9, F13 | tines-core 1.0 | `docs/superpowers/specs/2026-10-18-tines-1-0-design.md` §10 | Open | +| F7, F14, §8 | 1.1 (threat model, link authentication, signing) | 1.0 spec §2a, §2b | Open | +| F15, F16 | Accepted for now | — | — | Mark each finding fixed here, with its commit, when it lands. +The remaining F3 work (one configured peer) and F18 are host-only. Neither +blocks the tines-core 1.0 work, but both block a fleet of more than two +robots. + ## 11. Recommendations, in order -1. **Fleet milestone bugs:** +1. **Fleet milestone bugs** (done 2026-10-19, except the F3 and F4 + remainders): - F1 (ack only what was stored). - F2 (poll loop). - F3 (framing, listener, NodeId binding, Contact/Lost). @@ -511,20 +592,22 @@ Mark each finding fixed here, with its commit, when it lands. locking). - F17 (monotonic ms clock). - Each with a regression test. -2. **Host hardening before any robot leaves the lab:** +2. **Host follow-ups:** the F3 multi-peer allowlist and F18 scheduled + flush. Small, and needed before more than two robots share a fleet. +3. **Host hardening before any robot leaves the lab:** - F10 (trim, bounded channels, compaction). - F11 (socket permissions, peer creds, timeouts). - F12 (ignore own publications, SROS2 guidance, rate limits). -3. **Core hardening (can ride with 1.0):** +4. **Core hardening (rides with 1.0):** - F8 (idle eviction). - F9 (summary rate limit, no re-queue). - F13 (event drop-oldest, purge acks on peer loss, reject trailing bytes). - F6 (refuse NodeId change within a contact). -4. **1.1:** +5. **1.1:** - Write the full threat model from §4/§8. - Choose link authentication. - Add signing per spec §2a (F14, F7). - Add "replace if better signed" dedupe. -5. **Later:** confidentiality and rotating rendezvous identifiers (F15), +6. **Later:** confidentiality and rotating rendezvous identifiers (F15), only if the deployment needs them. diff --git a/docs/superpowers/specs/2026-10-18-tines-1-0-design.md b/docs/superpowers/specs/2026-10-18-tines-1-0-design.md index e61018af..35e0217e 100644 --- a/docs/superpowers/specs/2026-10-18-tines-1-0-design.md +++ b/docs/superpowers/specs/2026-10-18-tines-1-0-design.md @@ -1,10 +1,11 @@ # tines-core 1.0 — design Date: 2026-10-18 -Status: draft v6. Kyle decided D1–D5 and Q1–Q3 (2026-10-18) and the -review decisions R1–R3 and the `observed_at` definition (2026-10-19); -review fixes applied. v6 adds §10 (core hardening from `THREAT_MODEL.md`) -and the adversarial fuzz mode; these additions aren't reviewed yet. +Status: approved by Kyle (2026-10-20). Kyle decided D1–D5 and Q1–Q3 +(2026-10-18) and the review decisions R1–R3 and the `observed_at` +definition (2026-10-19); review fixes applied. v6 added §10 (core hardening +from `THREAT_MODEL.md`) and the adversarial fuzz mode; Kyle reviewed and +approved both (2026-10-20). Builds on: `2026-10-16-tines-design.md`, `2026-10-18-tines-addressed-delivery.md`, `2026-10-18-tines-fleet-deployment-design.md` (assumed fully implemented) diff --git a/tines-bt/README.md b/tines-bt/README.md index 055b68c4..759db2dd 100644 --- a/tines-bt/README.md +++ b/tines-bt/README.md @@ -52,6 +52,12 @@ violation). Up to [`MAX_PEERS`] connections may be live at once, one per bound [`NodeId`]; a second connection to the same [`NodeId`] replaces the first. +**Current limit:** `BluerRadio::new` and `BtTransport::new` still take a single +`peer`. The allowlist is exactly that one `NodeId`, and `BtTransport` drops +frames and events from any other peer, so in practice each node exchanges with +one configured partner. A configurable allowlist (empty = any tines advertiser) +is still to come; see `THREAT_MODEL.md` F3. + ## Manual verification The SeqPacket framing can't be exercised in CI (it needs two real radios), so @@ -64,6 +70,10 @@ verify it by hand with two devices running the `tines` service: 3. send a burst of observations on one device and confirm the other receives each as a separate, intact frame (no merged or truncated frames). +No result has been recorded yet. Once it's done, note the date, hardware and +BlueZ version here, along with whether SeqPacket worked or the length-prefix +fallback was needed. + [`Transport`]: https://docs.rs/tines-core/latest/tines_core/transport/trait.Transport.html [`NodeId`]: https://docs.rs/tines-core/latest/tines_core/observation/struct.NodeId.html [`Radio`]: crate::Radio diff --git a/tines/src/ipc.rs b/tines/src/ipc.rs index 86821bcb..76a98762 100644 --- a/tines/src/ipc.rs +++ b/tines/src/ipc.rs @@ -15,16 +15,17 @@ //! ``` //! //! Each frame is one of the [`Request`] or [`Response`] enums, tagged by a -//! `"kind"` field (serde externally-tagged). Frames are capped at 1 MiB; a -//! longer frame is treated as a malformed request and the connection is -//! dropped. There is no framing negotiation — the first byte after the length -//! prefix is `"`. +//! `"kind"` field (serde internally tagged). Frames are capped at 1 MiB. A +//! longer length prefix gets an error response, but its body isn't consumed, +//! so the connection is out of sync afterwards; the host-hardening spec makes +//! it close the connection instead (`THREAT_MODEL.md` F11). There is no +//! framing negotiation: the first byte after the length prefix is `{`. //! //! Observations are content-addressed, so IDs are exchanged as lowercase hex //! (the hex form of the 16-byte `ObservationId`). Topics and payloads are //! passed to `observe` the same way: as hex. //! -//! # Sequence numbers +//! # Clock //! //! `now` is the tines tick passed to `observe`/`belief`. The service reads it //! from its own monotonic millisecond [`HostClock`] (an `Instant` captured at diff --git a/tines/src/main.rs b/tines/src/main.rs index f9c91896..8fbbda0a 100644 --- a/tines/src/main.rs +++ b/tines/src/main.rs @@ -4,9 +4,9 @@ //! `bluer`), and serves the IPC ([`tines::serve`]) over a Unix-domain socket. //! One instance per device — the typical deployment on robots without ROS2. //! -//! The node/peer ids are 32 lowercase hex chars (the 16-byte `NodeId`), and -//! The tines sequence numbers are seeded from the LogStore's boot counter, so -//! they persist across reboots without a separate config field. +//! The node and peer ids are 32 lowercase hex chars (the 16-byte `NodeId`). +//! Sequence numbers are seeded from the LogStore's boot counter, so they don't +//! repeat across reboots and need no config field. use std::error::Error; use std::path::Path;