diff --git a/AGENTS.md b/AGENTS.md index e2bd64ac..d6de70f3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -19,7 +19,7 @@ architecture in the **spec first**: - **Addressed delivery (authoritative):** `docs/superpowers/specs/2026-10-18-tines-addressed-delivery.md` - **Fleet deployment v1 (authoritative):** `docs/superpowers/specs/2026-10-18-tines-fleet-deployment-design.md` — supersedes the addressed-delivery spec's epidemic forwarding (custody chain with handoff receipts). - **Fleet fixes (done):** `docs/superpowers/specs/2026-10-19-tines-fleet-fixes.md` — security-assessment findings F1–F5 and F17. -- **tines-core 1.0 (done — shipped as tines-core 1.0.0, 2026-10-21):** `docs/superpowers/specs/2026-10-18-tines-1-0-design.md` — wire version, public API, signature slot, core hardening (§10). +- **tines-core 1.0 (implemented on `feat/tines-core-1-0`; publish pending):** `docs/superpowers/specs/2026-10-18-tines-1-0-design.md` — wire version, public API, signature slot, core hardening (§10). - **Host hardening (next, hosts):** `docs/superpowers/specs/2026-10-19-tines-host-hardening.md` — findings F10–F12. - **Implementation plans:** `docs/superpowers/plans/` @@ -111,15 +111,28 @@ Split into **wire** (frozen for wire v1 — changing one changes how a v1 peer decodes or interprets a frame) and **local** (never crosses the wire; may change in a minor release): -- **Wire:** `WIRE_VERSION=4`, `MAX_TOPIC=64`, `MAX_PAYLOAD=1024`, +- **Wire** (public): `WIRE_VERSION=4`, `MAX_TOPIC=64`, `MAX_PAYLOAD=1024`, `MAX_FRAME=1500`, `SUMMARY_BYTES=512`, `ACK_IDS_CAP=90`, `MAX_SIG=64`, - `BLOOM_K=3`, plus the private `MAX_FRAGMENTS=64`. -- **Local:** `MAX_PEERS=8`, `SUMMARY_INTERVAL_MS=2000`, - `REASSEMBLY_TIMEOUT_MS=5000`, `SEND_QUEUE_CAP=256`, `MIN_MTU`, - `EVENT_QUEUE_CAP=64`, `DROP_TABLE_CAP=32`, `BCAST_SENT_CAP=256`, - `ACK_BUFFER_CAP=256`. + `BLOOM_K=3`. Also wire, but private: `MAX_FRAGMENTS=64`, which lives in + `tines-core/src/frag.rs`, not `consts.rs`. +- **Local** (`pub(crate)`, spec decision K: defaults that a later 1.x makes + configurable): `MAX_PEERS=8`, `SUMMARY_INTERVAL_MS=2000`, + `REASSEMBLY_TIMEOUT_MS=5000`, `SEND_QUEUE_CAP=256`, `EVENT_QUEUE_CAP=64`, + `DROP_TABLE_CAP=32`, `BCAST_SENT_CAP=256`, `ACK_BUFFER_CAP=256`, + `PEER_IDLE_MS=30_000`, and `MIN_MTU=23`. `MIN_MTU` is the lower bound of the + fragmentation MTU sweep in tests (BLE ATT's default), not the smallest + usable MTU; `Transport::mtu`'s rustdoc states those floors (85 bytes for + every valid observation). -Match the exact values above. +Match the exact values above. A crate outside core that needs a matching +number keeps its own constant (as `tines-bt` does for `MAX_PEERS` and +`MIN_MTU`). + +The `conformance` feature also compiles a `#[doc(hidden)]` +`tines_core::conformance` module: byte-level frame encoders and decoders, and +mirrors of the local limits, for the fuzz harness and tests outside core. It +isn't supported public API and isn't covered by semver. Don't build on it +outside the workspace and `fuzz/`. ## Crate layout diff --git a/CHANGELOG.md b/CHANGELOG.md index 8f5cf9a0..04a5557c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,119 +1,314 @@ # Changelog All notable changes to the tines crates. Versions follow -[Semantic Versioning](https://semver.org/); before 1.0, a minor bump may break -the API. +[Semantic Versioning](https://semver.org/). `tines-core` is 1.x and keeps the +compatibility promise in README's "Stability" section. The host crates +(`tines`, `tines-store`, `tines-bt`, `tines-ros2`, `tines-sim`) stay 0.x, +where a minor bump may break the API. -## 1.0.0 — 2026-10-21 +## tines-core 1.0.0 — Unreleased -tines-core 1.0: the wire format is frozen (wire v1), the public API is -narrowed and documented, and the core hardening from the 1.0 spec is in. -This release ships the fleet-deployment milestone (handoff receipts) and the -remaining fleet fixes (F1–F5, F6, F8, F9, F13, F17; see `THREAT_MODEL.md`). +Implemented on `feat/tines-core-1-0`; not yet tagged or published. The wire +format is frozen as wire v1, the public API is narrowed and documented +(`#![deny(missing_docs)]`), and the core hardening from the 1.0 spec +(`docs/superpowers/specs/2026-10-18-tines-1-0-design.md` §10) is in. It also +carries the fleet-deployment milestone (handoff receipts) and the fleet fixes, +which were never released on their own. Findings F1, F6, F8, F9 and F13 are +fixed in core; see `THREAT_MODEL.md`. ### Breaking (from 0.3.0) -- **Wire v1** (tines-core). Every frame now leads with a `WIRE_VERSION` byte - (4). 0.x-shaped frames are rejected as `WireError::VersionMismatch`. -- **`Dest` replaces `to`/`lifetime_ms`** (tines-core). Addressed delivery now - takes `dest: Option` where `Dest { to, lifetime_ms }`; the lifetime is - excluded from observation identity. -- **Signature slot** (tines-core). `Observation` carries a reserved - `sig: Option` (algorithm + up to 64 bytes). 1.0 enforces the - wire shape but does not sign or verify (1.1). -- **Public API narrowing** (tines-core). Nine items moved `pub(crate)`, - eleven types are `#[non_exhaustive]` (build via constructors, not struct - literals). `Frame` and `WireError` are now *public*: the wire format is the - 1.x compatibility contract. -- **`Store` trait settled** (tines-core). `flush(&mut self, now)` and - `is_empty` joined the trait; `Node::store_mut` is gone. A new opt-in - `WallClockStore` trait carries the trusted-time operations - (`set_wall_clock`, `peer_time_estimate`) that only `LogStore` implements. -- **`NodeEvent` + counted drops** (tines-core). Exchange outcomes surface as - `NodeEvent` (`Delivered`, `Dropped { reason, count }`) through - `Node::drain_events`; `DropReason` is `VersionMismatch`, `Malformed`, or - `IdentityChanged`. -- **`Node::poll` is the only pump** (tines-core). The separate custody - re-offer pass folded into `poll`. -- **Log format v2** (tines-store). `FORMAT_VERSION` bumped for the new - `Observation` encoding; opening an older log is a hard error that names - the fix (delete the log). -- **Quarantine + wall-clock fold** (tines-store). `LogStore::open(path, now)` - takes no wall time; observations held across a power-off are quarantined - until the host feeds trusted time via `set_wall_clock`. -- **Trusted wall time in the hosts** (tines). A `trusted_clock = "system"` - config flag or a peer-derived estimate (smallest `observed_at + age_ms`) - anchors the wall clock once; observations carry `observed_at: None` until - then. -- **MSRV** (tines-core). `rust-version = "1.82"`; CI builds and tests - tines-core alone on it. +- **Wire v1.** + - Every frame now starts with a `WIRE_VERSION` byte (4). A 0.x-shaped frame + is dropped and counted as `DropReason::VersionMismatch`. + - The new `Ack` frame is the last `Frame` variant (see Added). + - Frames must be canonical. Trailing bytes after a complete frame, or after + a reassembled observation, are rejected and counted as `Malformed`. +- **`Dest` replaces `to`/`lifetime_ms`.** + - An addressed observation carries `dest: Option`, where + `Dest { to, lifetime_ms }`. + - The identity hash covers `dest.to`. It still excludes `lifetime_ms`, + `age_ms` and `observed_at`. +- **Signature slot.** + - `Observation` carries `sig: Option` (version, `SigAlgo`, and up + to `MAX_SIG` = 64 bytes). + - 1.0 checks the wire shape and carries signatures verbatim. It neither + signs nor verifies; that's 1.1. `sig` is outside the identity hash. +- **Nine items are now `pub(crate)`:** + - `Exchange` (and with it `Exchange::on_send_error`; drive the exchange + through `Node`); + - `Peer`; + - `InFlight`; + - `Bloom` (and its public `salt` field); + - `Reassembler`; + - `fragment`; + - `Frame`; + - `WireError`; + - `FRAG_OVERHEAD`. +- **The `bloom`, `exchange`, `frag` and `wire` modules are private**, so paths + through them are gone: + - `tines_core::wire::frame_sender` is now `tines_core::frame_sender`; + - `DropReason` and `NodeEvent` are reached at the crate root. +- **Local constants are `pub(crate)`** (spec decision K). They are defaults + that a later 1.x release will make configurable: + - `MAX_PEERS`, `SUMMARY_INTERVAL_MS`, `REASSEMBLY_TIMEOUT_MS`, + `SEND_QUEUE_CAP`, `EVENT_QUEUE_CAP`, `BCAST_SENT_CAP` and `ACK_BUFFER_CAP` + are no longer in `tines_core::consts`; + - the new local constants, `PEER_IDLE_MS`, `DROP_TABLE_CAP` and `MIN_MTU`, + were never public; + - only the wire constants stay public (see Added). + - `Transport::mtu`'s rustdoc now states the smallest MTU core can use (85 + bytes to carry every valid observation). +- **Eleven types are `#[non_exhaustive]`.** + - Enums (match with a wildcard): `ObserveError`, `StoreError`, `SendError`, + `Event`, `NodeEvent`, `DropReason`, `SigAlgo`. + - Structs (no struct literals outside core): `Observation`, `Dest`, + `Signature`, `TrimPolicy`. + - Build an `Observation` with `Observation::broadcast` or + `Observation::addressed`, and a `TrimPolicy` with + `TrimPolicy::default()` plus field assignment. +- **`Node::drain_events` returns `NodeEvent`.** This replaces the fleet + milestone's unreleased `ConfidenceEvent`. +- **`Node::store_mut` is removed.** It was added by the unreleased fleet fixes + so the service could reach `LogStore::flush`. Use `Node::flush`, and + `Node::set_wall_clock` / `Node::peer_time_estimate` for a `WallClockStore`. + Read access through `Node::store` stays. +- **MSRV.** `rust-version = "1.82.0"`. CI builds tines-core on 1.82, with and + without default features and for `thumbv7em-none-eabihf`. It runs no tests + there, because a dev-dependency needs edition 2024. From now on an MSRV bump + is a minor release. ### Added -- **Handoff receipts** (tines-core). The new `Ack` frame (the last `Frame` - variant) carries up to `ACK_IDS_CAP` (90) ids. Acks are sent as the - exchange runs and are accepted only from a peer the id was sent to during - the current contact. +- **Handoff receipts.** The new `Ack` frame carries up to `ACK_IDS_CAP` (90) + ids. Acks are sent as the exchange runs, and are accepted only from a peer + the id was sent to during the current contact. - An ack for an addressed observation moves custody: the sender releases its copy (`Store::release`) and never offers it again. - - An ack for a broadcast is confidence only. It surfaces as a - `NodeEvent::Delivered { peer, id }` from the new `Node::drain_events`. - - Broadcast confirmations and counted drops both come through - `Node::drain_events`; a confirmed broadcast arrives as - `NodeEvent::Delivered`, and a frame the peer dropped arrives as - `NodeEvent::Dropped { reason, count }` (§3). -- **Counted drop reporting** (tines-core). A frame the peer could not use is no - dropped silently: the exchange counts it under a `DropReason` (`VersionMismatch` - or `Malformed`) keyed by the peer's transport address and reports the count - through `Node::drain_events` as `NodeEvent::Dropped`. Counters are held in a - bounded table (`consts::DROP_TABLE_CAP`) with a single overflow slot for the - case where the table is full, so an adversarial peer can't spin forever — it - just reports `addr: None`. Expired addressed observations dropped by policy are - *not* counted. -- New constants: `ACK_IDS_CAP`, `EVENT_QUEUE_CAP`, `BCAST_SENT_CAP`, - `ACK_BUFFER_CAP`. -- `wire::frame_sender(bytes) -> Option`: a frame's claimed sender, - for transports that bind a connection to an identity. -- `Node::store_mut`, so a host can call backend methods such as - `LogStore::flush`. -- **New host crates:** - - `tines-store`: `LogStore`, a persistent append-only backend. - - `tines-bt`: a BLE transport over BlueZ. - - `tines`: a standalone service with Unix-socket JSON IPC. - - `tines-ros2`: a ROS2 node; rclrs is behind the `ros2` feature. + - An ack for a broadcast is confidence only. It surfaces as + `NodeEvent::Delivered { peer, id }` from `Node::drain_events`. +- **Counted drop reporting.** A frame this node can't use isn't dropped + silently. + - The exchange counts it under a `DropReason`: `VersionMismatch`, + `Malformed`, or `IdentityChanged` (F6). The count is keyed by the + sender's transport address. + - `Node::drain_events` reports each counter as + `NodeEvent::Dropped { addr, reason, count }`, after the queued + `Delivered` events. + - The counter table is bounded. When it's full, further drops go to one + overflow counter, reported with `addr: None` and the reason of the first + drop that overflowed since the last drain. Draining empties the table, + keys included. + - An expired addressed observation that's refused by policy is *not* + counted. +- **`Store::flush(now) -> Result<(), StoreError>`** and **`Store::is_empty`**, + both with safe defaults, plus **`Node::flush(now)`**. `flush` is the store's + durability point and its time anchor for ages after a restart. Flushing + twice is harmless, and an earlier `now` is clamped. Core never flushes on + its own. +- **`WallClockStore`** (spec decision H). An opt-in supertrait of `Store`, for + backends that hold data across power-off. It has an associated + `FoldReport`, `set_wall_clock(wall_now, now)` and `peer_time_estimate(now)`. + `LogStore` implements it; `MemStore` doesn't. +- **`Node::set_wall_clock`** and **`Node::peer_time_estimate`**. They exist + when `S: WallClockStore` and delegate to it, so no `&mut Store` escapes the + facade. Core forwards the wall time and never reads it. +- **`Dest`, `Signature` and `SigAlgo`**, and the constructors + `Observation::broadcast`, `Observation::addressed` and + `Observation::with_observed_at`. The two constructors panic on an + oversized topic or payload; `Node::observe` and `Node::observe_to` return + `ObserveError::TooLarge` instead. +- **`tines_core::frame_sender(bytes) -> Option`**: a frame's claimed + sender, for transports that bind a connection to an identity. +- **Public wire constants:** `WIRE_VERSION`, `MAX_TOPIC`, `MAX_PAYLOAD`, + `MAX_FRAME`, `SUMMARY_BYTES`, `ACK_IDS_CAP`, `MAX_SIG` and `BLOOM_K`. +- **`tines_core::store_conformance`**, behind the `conformance` feature and + `std`-free. A suite that lets any backend prove the §4 `Store` contract. + `MemStore` and `LogStore` both run it. +- The `conformance` feature also enables a `#[doc(hidden)]` `conformance` + module of frame encoders and local limits for the fuzz harness. It isn't + part of the supported API. ### Changed - **Addressed delivery is a custody chain.** It used to be epidemic - forwarding. A message has one custodian at a time, which hands it on and - releases it when the next holder acks. A lost ack can leave two - custodians for a while; a message is never dropped by the protocol. -- Bounce records carry a direction, so only records of an actual send - validate an ack. -- **`LogStore` log format v1** (incompatible; there's no migration, so delete - old logs): - - The file starts with a `"TINESLOG"` header and a format version. - - Each record has a sync marker, a length and a CRC-32C. - - Ids are recomputed on load, not stored. + forwarding. + - A message has one custodian at a time, which hands it on and releases it + when the next holder acks. + - A lost ack can leave two custodians for a while; the protocol never drops + a message. + - Bounce records carry a direction, so only a record of an actual send + validates an ack. +- **Core never trusts a backend's filtering.** It re-checks `dest` and + `is_released` on every custody candidate. It re-checks `dest` when building + a summary, rebuilding a broadcast diff, and sending a broadcast. +- **Core hardening** (spec §10): + - **F6:** a peer's `NodeId` is pinned by its first frame in a contact. A + later frame with another `from` is dropped and counted as + `IdentityChanged`. The pin resets on `Lost`, and on a repeat `Contact` + for a known peer. + - **F8:** a peer with no inbound frame for `PEER_IDLE_MS` (30 s) is evicted + as if `Lost`. That includes a peer that only ever sent `Contact`. Idle + peers are swept before a peer is added and at the start of every + `next_frame`, and a peer that resumes later is a new implicit contact. + - **F9:** at most one received Summary per peer is acted on per + `SUMMARY_INTERVAL_MS`. A summary is accepted once at least half the + interval has passed, to allow for poll jitter (spec decision L). The + send-queue rebuild skips ids the peer's filter says it holds + (`Bloom::might_contain`). + - **F13:** + - the event queue drops the oldest event when full; + - a removed peer's pending acks are purged, and only that peer's; + - frames, and reassembled observations, must be canonical. +- **Receipts are scoped to the contact.** Broadcast-send records for an + address are dropped when its contact starts or ends. A broadcast whose send + fails has its record undone, so its ack isn't accepted. + +### Fixed + +- A receiver whose store returned `Err` (for example `Full`) still acked, so + the sender released custody of a message nobody kept. Now nothing is acked + unless it was stored, addressed or broadcast (F1). +- From the 1.0 review: + - Silent and contact-only peers were never evicted, so eight spoofed + addresses could still lock out every newcomer (S1, F8). + - The drop table never freed its keys. After 32 distinct + `(addr, reason)` pairs, every drop went to the overflow slot until reboot + (S7). + - Losing one peer purged every peer's pending acks (S9). + - Broadcast receipts weren't limited to this contact, and a failed send + could still be confirmed (S10). + - A store whose broadcast iterator yielded addressed ids could leak them + into a summary or forward them as broadcasts (S11). A test store that + overrides no filtering now pins the re-checks (B8). + - A repeat `Contact` didn't reset the F6 pin. + - A reassembled observation was decoded with trailing bytes allowed. + - The summary rate limit could ignore an on-time summary measured a + millisecond early. + - `DropReason::IdentityChanged` carried a variant-level `#[non_exhaustive]`, + so code outside the crate couldn't match it. + +## Hosts 0.4.0 — Unreleased + +`tines`, `tines-store`, `tines-bt`, `tines-ros2` and `tines-sim` move to 0.4.0 +and depend on `tines-core = "1"`. Like tines-core 1.0.0, nothing here is +tagged or published yet. + +The 0.3.0 release shipped only `tines-core` and `tines-sim`. The four other +host crates were added afterwards, during the fleet milestone, and carried the +workspace version 0.3.0 in the tree without being released. The breaks below +are measured against that in-tree 0.3.0. This release also carries the fleet +milestone's host work (F2–F5, F17). + +### Breaking + +- **Log format v2** (tines-store). + - The file starts with a `"TINESLOG"` header and a format version (now 2, + for tines-core 1.0's `Observation` encoding). + - Each record has a sync marker, a length and a CRC-32C, and ids are + recomputed on load. + - Opening a log of any other version is an error that names the fix + (delete the log). There's no migration. +- **`LogStore::flush` is the `Store::flush(now)` trait method** (tines-store). + The inherent `flush()` is gone. Call `Node::flush(now)`; the marker now + uses the caller's tick. +- **`LogStore::open(path, now)` quarantines what it loads** (tines-store). + Observations already in the log are held back until the host feeds + trusted wall time (see Added). +- **`tines::serve`** takes a fifth argument, `peer_time_window_ms`. + **`tines::handle`** takes `observed_at: Option` before `now`. +- **Service config** (tines): + - unknown keys are rejected (`deny_unknown_fields`); + - `trusted_clock` accepts only `"system"`; any other value is a config + error; + - the `first_seq` field is gone (seqs come from the store's boot counter). + +### Added + +- **Trusted wall time in the hosts** (spec §9, decisions G and J). A host + opens its store before it knows the time, then adopts a wall time once per + run. + - **Local source first.** The service's `trusted_clock = "system"`, or + tines-ros2's `trust_system_clock` / `trust_wall_clock`. + - **Otherwise a peer, by decision J.** + - The host collects `Node::peer_time_estimate` for `peer_time_window_ms` + (default 10 000) from the first plausible estimate. + - It then adopts the latest estimate. + - The store only offers estimates within its + `LogStoreBuilder::plausible_wall_ms` range, so a forged one can't hide + the honest ones. + - **Plausible range.** A wall time is adopted only if it's between + 2026-01-01Z (`tines::trust::WALL_FLOOR_MS`) and about 2126 + (`WALL_CEILING_MS`). That applies to the system clock and to peers. + - **Flush on trust.** Adopting the time flushes the store at once, so a + marker carrying the wall time reaches disk. + - Until trust, observations carry `observed_at: None`; after it, the + host's wall time. +- **`tines::trust`**: `WALL_FLOOR_MS`, `WALL_CEILING_MS`, + `PLAUSIBLE_WALL_MS`, `DEFAULT_PEER_TIME_WINDOW_MS`, `is_plausible_wall_ms`, + `system_wall_ms`, `adopt_wall_clock` and `PeerTimeWindow`. `HostClock` + gains `anchor_wall`, `wall_ms` and `is_trusted`. +- **Service config keys** (tines): + - `trusted_clock`; + - `peer_time_window_ms` (default 10000); + - `unknown_downtime_age_ms` (default: the LogStore default, 24 h). +- **`TinesNode` trust methods** (tines-ros2): `trust_system_clock`, + `trust_wall_clock`, `set_peer_time_window_ms` and `has_trusted_time`. They + give a ROS2-only fleet a local trusted-time source. +- **LogStore quarantine and fold** (tines-store). + - `LogStore` implements `WallClockStore`. + - `set_wall_clock` folds the downtime into held ages, once, and returns a + `FoldReport`, which `fold_report` reads back later. + - **Downtime known:** held ages gain the wall-clock downtime. + - **Downtime unknown:** held addressed observations expire, and held + broadcasts are floored at `unknown_downtime_age_ms`. + - Quarantined entries are exempt from every trim rule (spec decision L). + - `peer_time_estimate` returns the latest estimate among observations + stored since this open. +- **`LogStoreBuilder`** (tines-store): + - `set_unknown_downtime_age_ms` (default 24 h); + - `plausible_wall_ms` (default unbounded; both hosts pass + `tines::trust::PLAUSIBLE_WALL_MS`). +- **Snapshots** (tines-store, spec decision I). + - `open`, after its tick fold, rewrites the log as a snapshot. So does + `set_wall_clock`, after its fold. A snapshot holds every live observation + at its current age, a `Release` for every released id still held, and a + closing clock marker. + - The snapshot goes to `.snapshot-tmp` in the same directory. That + file is fsynced, locked, renamed over the log, and then the directory is + fsynced. A leftover temp file is removed on open. + - The fold's results and age lineage then survive any number of reboots. + Each snapshot also compacts the log. +- **New host crates** (first released in this line, see above): + - `tines-store`: `LogStore`, a persistent backend; + - `tines-bt`: a BLE transport over BlueZ; + - `tines`: a standalone service with Unix-socket JSON IPC; + - `tines-ros2`: a ROS2 node, with rclrs behind the `ros2` feature. + +### Changed + - `LogStore` persists a boot counter, and `first_seq()` returns - `boot_count << 32`. Both hosts seed `Node::new` from it, and the - `first_seq` config field is gone. -- `LogStore::release` writes a `Release` record immediately. `flush` writes - a clock marker, so ages carry across a restart. + `boot_count << 32`. Both hosts seed `Node::new` from it. +- `LogStore::release` writes a `Release` record immediately. - `LogStore::open` takes an exclusive lock and creates the file `0600`. +- `LogStore::flush` and `set_wall_clock` clamp an earlier `now` to the store's + time anchor, so `stored_at` and clock markers never move backwards. +- A log whose first header was torn while it was being created (shorter than + the header, starting with the magic) opens as a new log. Any other short + file is still refused. - tines-bt rewrite: - L2CAP SeqPacket channels, one frame per packet. - Every node advertises and scans; the smaller `NodeId` dials and the larger accepts. - Connections are bound to the advertised `NodeId`, and frames whose - claimed sender differs are dropped. + claimed sender differs (`tines_core::frame_sender`) are dropped. - Connections emit `Contact` and `Lost`. + - Live connections are capped by tines-bt's own `MAX_PEERS` (8). +- tines-sim's property tests run 64 cases again, and the bounce-overflow + property checks that the table really overflowed. ### Fixed -- A receiver whose store returned `Err` (for example `Full`) still acked, so - the sender released custody of a message nobody kept. Now nothing is - acked unless it was stored, addressed or broadcast (F1). - The standalone service never polled its `Node`, so it never exchanged anything. It now runs the exchange on a poll loop (F2). - tines-bt had no listener, lost frame boundaries on a stream socket, and @@ -125,13 +320,33 @@ remaining fleet fixes (F1–F5, F6, F8, F9, F13, F17; see `THREAT_MODEL.md`). fsync of the file and its directory (F5). - The service's tick was wall-clock *seconds*. Both hosts now use one monotonic millisecond clock (F17). +- From the 1.0 review: + - `LogStore` didn't implement `Store::flush`, so `Node::flush` wrote no + marker and fsynced nothing (B1). + - Fold results and age lineage lived only in memory. An expired message + came back after the next reboot, and ages reset on a third boot (B2). + Snapshots fix both. + - A forged `observed_at = u64::MAX` overflowed: a panic in debug, a time + near 1970 in release (B3). The arithmetic saturates, and the hosts + bound every wall time they adopt. + - A flush between open and the fold made the fold count that time twice + (S2). + - `flush` didn't clamp an earlier `now` (S3). + - tines-ros2 had no local trusted-time source (S4). + - A quiet reboot lost everything, because nothing flushed after trust + was set (S5). + - The service trusted a 1970 system clock and ignored config typos (S6). ### Known issues - The service talks to exactly one configured `peer`. The multi-peer allowlist the fleet-fixes spec describes isn't wired up (F3). -- Neither host flushes on a timer, and tines-ros2 never flushes or trims - (F18). +- Neither host flushes on a timer or at shutdown (F18). + - The service trims and flushes only after 64 poll rounds that each stored + something new. + - tines-ros2 flushes only when it adopts trusted time, and never trims. +- Peer time is unauthenticated until 1.1. An adopted time isn't corrected + while the host runs (F19). ## 0.3.0 — 2026-10-18 diff --git a/README.md b/README.md index d62e04d4..f65a020c 100644 --- a/README.md +++ b/README.md @@ -29,6 +29,7 @@ hear each other. - [Running on a robot](#running-on-a-robot) - [Crate layout](#crate-layout) - [Building and CI](#building-and-ci) +- [Stability](#stability) - [Security](#security) - [Where to read more](#where-to-read-more) @@ -88,14 +89,18 @@ An `Observation` is the unit of storage and exchange: | `seq` | the observer's sequence number | ✓ | | `topic` | what it is about, ≤ 64 bytes (`b"door/3"`) | ✓ | | `payload` | opaque bytes, ≤ 1024 | ✓ | -| `to` | `None` for broadcast, or a recipient `NodeId` | ✓ | -| `age_ms` | how old the observation is | | -| `observed_at` | optional absolute timestamp, if the robot has a clock | | -| `lifetime_ms` | validity window, present exactly when `to` is | | +| `dest` | `None` for broadcast, or `Some(Dest { to, lifetime_ms })` for an addressed observation | `to` only | +| `age_ms` | how old the observation is: a lower bound that every holder advances | | +| `observed_at` | the observer's wall-clock time (ms since the Unix epoch), or `None` if it had no trusted clock | | +| `sig` | a reserved `Signature` slot (algorithm + up to 64 bytes); carried verbatim, signed and verified from 1.1 | | -The **content id** is the first 16 bytes of Blake2s-256 over the ✓ fields. -Because age and timestamps are excluded, the same fact arriving by two -different paths, at two different ages, dedupes to one entry. +The **content id** is the first 16 bytes of Blake2s-256 over the ✓ fields +(`dest.to`, but not `dest.lifetime_ms`). Because age, timestamps, lifetimes +and signatures are excluded, the same fact arriving by two different paths, at +two different ages, dedupes to one entry. + +`Observation` is `#[non_exhaustive]`: build one with `Observation::broadcast` +or `Observation::addressed`, or let `Node::observe` / `Node::observe_to` do it. ### Time is relative @@ -154,9 +159,11 @@ sequenceDiagram ### Wire frames -Frames are [postcard](https://docs.rs/postcard)-encoded and each carries the +Every frame starts with a `WIRE_VERSION` byte (4 for wire v1), followed by +the [postcard](https://docs.rs/postcard)-encoded frame. Each frame carries the sender's `NodeId`, so the core never needs to map transport addresses to -robots. +robots. A frame with another version byte, trailing bytes, or anything else it +can't decode is dropped and counted (see [Receipts](#receipts)). ```mermaid classDiagram @@ -183,13 +190,14 @@ classDiagram ``` A transport that binds a connection to a peer's identity can check a frame's -claimed sender with `tines_core::wire::frame_sender`, without decoding the +claimed sender with `tines_core::frame_sender`, without decoding the whole frame. ### Fragmentation An observation that doesn't fit the transport's MTU is split into up to 64 -`Frag` frames (64 bytes of each frame are reserved for the header). Each peer +`Frag` frames (65 bytes of each frame are reserved for the version byte and +the header). Each peer has one stream in flight at a time, and it is finished before anything else starts for that peer. All fragments of a stream are cut from one encoding, so the receiver can always reassemble them. A partial observation is never @@ -295,12 +303,15 @@ it has already released, since that would hand custody back to a robot that no longer carries the message. Broadcast confirmations surface as `NodeEvent::Delivered { peer, id }` from -`Node::drain_events`. Frames the peer could not use are counted under a -`DropReason` (`VersionMismatch` or `Malformed`) and surface as -`NodeEvent::Dropped { reason, count }` (see the `drop` reporting in the `tines` -service and the ROS2 node). The event queue is bounded (64) plus a small -counted-drop table, so drops are always reported even under a chatty peer — -events are telemetry, and losing one never loses data. +`Node::drain_events`. Frames this node received but couldn't use are counted +under a `DropReason` (`VersionMismatch`, `Malformed`, or `IdentityChanged` for +a peer whose `from` changed mid-contact) and surface as +`NodeEvent::Dropped { addr, reason, count }`, keyed by the sender's transport +address (see the `drop` reporting in the `tines` service and the ROS2 node). +The event queue is bounded (64, dropping the oldest), and drops go to a small +counted table with one overflow counter (`addr: None`), so drops are always +reported even under a chatty peer. Events are telemetry: losing one never +loses data. ## Usage @@ -347,6 +358,7 @@ for event in node.drain_events() { tines_core::NodeEvent::Dropped { addr, reason, count } => { println!("dropped {count} frame(s) from {addr:?}: {reason:?}"); } + _ => {} // NodeEvent is #[non_exhaustive] } } @@ -407,10 +419,11 @@ pub trait Transport { - **Report back-pressure honestly.** Return `SendError::Busy` when the radio can't take a frame right now; the exchange will offer the same frame again on the next poll. `TooLarge` tells the exchange to give that frame up. -- **Driving `Exchange` without `Node`?** `Exchange::next_frame` assumes the - frame it returns gets sent. If `send` fails, call - `Exchange::on_send_error(&err)` before the next `next_frame`, or the frame - is treated as delivered. `Node::poll` does this for you. +- **Mind the MTU.** `mtu()` must be at most 1500. Core can carry every valid + observation from 85 bytes up; below that it never sends an oversized frame, + but it sends less (see `Transport::mtu`'s docs for each floor). +- **Binding identities?** A transport that binds a connection to a `NodeId` + can check each frame's claimed sender with `tines_core::frame_sender`. ## Writing a store @@ -424,12 +437,14 @@ flash or SQLite backend implements the `Store` trait: | `get` | the observation with `age_ms` advanced by `now − stored_at`, saturating if `now` is earlier | | `len` | number of stored observations | | `for_each_id` | every stored id, any order | -| `for_each_broadcast_id` | ids with `to == None`; these feed the summary | -| `for_each_addressed_id` | ids with `to == Some(_)`; scanned for custody on every poll | +| `for_each_broadcast_id` | ids with `dest == None`; these feed the summary | +| `for_each_addressed_id` | ids with `dest == Some(_)`; scanned for custody on every poll | | `release` | mark an id as handed off, so it's never offered again; a no-op if the id isn't stored | | `is_released` | whether an id has been released; a re-inserted observation starts un-released | | `for_each_topic` | age-adjusted observations whose topic starts with a prefix | | `trim` | apply a `TrimPolicy` (per-topic, then age, then count); return how many were removed | +| `is_empty` | `len() == 0` (default provided) | +| `flush` | the durability point and the time anchor for ages after a restart; harmless twice, an earlier `now` is clamped; returns `Err(StoreError::Backend)` on failure and leaves the store usable (default: no-op `Ok(())`) | The store must remember when each observation was inserted (`stored_at`, in the caller's `Tick`s) so it can report current ages. `insert` may return @@ -439,7 +454,19 @@ custody. `for_each_addressed_id` has a default implementation that visits every id, which is correct but makes the exchange read every broadcast observation on each poll just to find the few addressed ones. Override it if your backend can -filter by `to` cheaply, e.g. with an index or a separate table. +filter by `dest` cheaply, e.g. with an index or a separate table. The +filtering is only an optimisation: core re-checks `dest` and `is_released` on +every id a backend yields. + +A store that keeps `sig` must return it verbatim. Core never calls `flush`; +the application decides how often, because on flash every flush costs wear. +A backend that holds data across power-off can also implement the opt-in +`WallClockStore` trait, which lets the host pass in trusted wall time through +`Node::set_wall_clock` (see [Trusted time](#trusted-time)). + +With the `conformance` feature, `tines_core::store_conformance` checks a +backend against the trait contract (dedupe, age lineage, saturating ticks, +release, `sig` round-trip, trim order, no tombstones, `flush`). ## The simulator @@ -508,13 +535,21 @@ flowchart LR A robot runs either the service or the ROS2 node, not both: each owns its `Node` and its log. -- **`tines-store`**: `LogStore`, an append-only log file. +- **`tines-store`**: `LogStore`, an append-only log file (format v2; an + older log is refused with an error that says to delete it). - Each record is checksummed (CRC-32C). Recovery skips a corrupt record and truncates only a torn tail. - Releases are written as they happen. - A boot counter provides `first_seq()`. - - `flush()` writes a clock marker, so ages carry across a restart, and - fsyncs the log. + - `Store::flush(now)`, reached through `Node::flush`, writes a clock marker + at `now` and fsyncs the log, so ages carry across a restart. An earlier + `now` is clamped. + - What the log held at open is quarantined until the host has trusted + wall time (see [Trusted time](#trusted-time)). + - The log is rewritten as a snapshot at open and after the wall-clock + fold, through a `.snapshot-tmp` file that's fsynced and renamed + over it. That keeps the fold's results and ages across any number of + reboots, and compacts the log. - The file is created `0600` and locked, so only one process can open it. - **`tines-bt`**: BLE over BlueZ, using L2CAP SeqPacket channels (one frame per packet). @@ -527,8 +562,9 @@ A robot runs either the service or the ROS2 node, not both: each owns its - **`tines`**: the standalone service, for robots without ROS2. - It runs the exchange on a poll loop and answers `Observe`, `Belief`, `Events` and `Status` requests on a Unix socket. - - It's configured by a JSON file: `node_id`, `peer`, `log_path`, - `socket_path` and `poll_interval_ms`. + - It's configured by a JSON file (below). Library users call + `tines::serve(node, socket_path, clock, poll_interval_ms, + peer_time_window_ms)`. - **`tines-ros2`**: a ROS2 node. - It publishes each tines topic under `/tines/` and confirmed deliveries on `/tines/receipts`. @@ -536,12 +572,73 @@ A robot runs either the service or the ROS2 node, not both: each owns its - rclrs is behind the `ros2` feature, so the crate builds without a ROS2 toolchain. +The service's config file is a JSON object. An unknown key is an error, so a +typo can't silently fall back to a default. + +| Key | Default | Meaning | +|---------------------------|--------------|---------| +| `node_id` | required | this robot's `NodeId`, 32 hex chars | +| `peer` | required | the one peer's `NodeId`, 32 hex chars | +| `log_path` | required | the `LogStore` log file | +| `socket_path` | required | the IPC Unix socket | +| `poll_interval_ms` | 20 | exchange poll cadence | +| `trusted_clock` | unset | `"system"` to trust the system clock; the only accepted value | +| `peer_time_window_ms` | 10000 | how long to collect peer time estimates before adopting the latest | +| `unknown_downtime_age_ms` | 24 h | age floor for held broadcasts when the downtime is unknown | + The hosts use a monotonic millisecond clock that starts at 0 when the -process does. Neither host sets `observed_at` yet. +process does. + +### Trusted time + +Ages are relative, but a robot that was powered off can't tell from ticks how +long it was off. So the hosts open the store before they know the wall time, +quarantine what it held (not offered, delivered or acked, but still deduped), +and adopt a trusted wall time **once per run**: + +1. **A local source.** The service adopts the system clock at startup when + `trusted_clock = "system"`. tines-ros2's `TinesNode` has + `trust_system_clock(now)`, and `trust_wall_clock(wall_ms, now)` for an RTC + or GPS reading; call one right after construction. +2. **Otherwise, peers.** From the first plausible estimate a peer's + observations give (`observed_at + age_ms`), the host collects estimates + for `peer_time_window_ms` (10 s by default; `set_peer_time_window_ms` in + tines-ros2), then adopts the latest. +3. **Otherwise** the quarantine stays. The robot's own fail-safe is outside + tines. + +Only a wall time between 2026-01-01 and about 2126 is adopted +(`tines::trust::WALL_FLOOR_MS` and `WALL_CEILING_MS`), from a peer or from +the system clock. A board that booted before NTP and reads 1970 is logged and +ignored, and the host falls back to peers. Adopting a time runs the store's +fold and flushes at once. If the log's last clock marker carries a wall time +no later than the adopted one, the fold adds the downtime to held ages. +Otherwise the downtime is unknown: the fold expires held addressed messages +and floors held broadcasts at `unknown_downtime_age_ms`. From then on +the host stamps its observations' `observed_at`; before, they carry `None`. +`TinesNode::has_trusted_time` reports whether a time was adopted. + +The system clock is read once, at startup. Under systemd, order the service +after clock sync: + +```ini +[Unit] +Wants=time-sync.target +After=time-sync.target +``` + +`time-sync.target` only waits for a synced clock when the sync service +supports it (`systemd-timesyncd` with `systemd-time-wait-sync.service` +enabled, or chrony's equivalent). + +Peer time is unauthenticated until 1.1, and an adopted time isn't corrected +while the host runs; see F19 in [`THREAT_MODEL.md`](THREAT_MODEL.md). **Current limits** (tracked in [`THREAT_MODEL.md`](THREAT_MODEL.md)): - The service talks to one configured `peer`. -- The hosts don't yet flush on a timer. +- The hosts don't flush on a timer or at shutdown. The service flushes after + 64 productive poll rounds, and both hosts flush when they adopt trusted + time. - The host-hardening work (trim defaults, bounded channels, socket permissions, ROS2 isolation) hasn't landed. @@ -564,23 +661,24 @@ tines/ tines-ros2/ std: the ROS2 node (rclrs behind the `ros2` feature). ``` -Fixed limits, all compile-time constants in `tines-core/src/consts.rs`: +The public constants, in `tines_core::consts`, are the wire constants. They +are frozen for wire v1: changing one is a `WIRE_VERSION` bump. -| Constant | Value | Meaning | -|-------------------------|--------|------------------------------------------| -| `MAX_TOPIC` | 64 | topic bytes | -| `MAX_PAYLOAD` | 1024 | payload bytes | -| `MAX_PEERS` | 8 | peers tracked at once | -| `SUMMARY_BYTES` | 512 | full Bloom summary size | -| `BLOOM_K` | 3 | hash functions per id | -| `SUMMARY_INTERVAL_MS` | 2000 | re-summary period per peer | -| `REASSEMBLY_TIMEOUT_MS` | 5000 | partial observation discard time | -| `SEND_QUEUE_CAP` | 256 | ids queued per peer | -| `MAX_FRAME` | 1500 | largest frame (and MTU) supported | -| `ACK_IDS_CAP` | 90 | ids in one `Ack` frame | -| `EVENT_QUEUE_CAP` | 64 | confirmed deliveries awaiting `drain_events` | -| `BCAST_SENT_CAP` | 256 | broadcast sends remembered per peer, to validate acks | -| `ACK_BUFFER_CAP` | 256 | received ids waiting to be acked, per kind | +| Constant | Value | Meaning | +|-----------------|-------|------------------------------------------| +| `WIRE_VERSION` | 4 | the first byte of every frame | +| `MAX_TOPIC` | 64 | topic bytes | +| `MAX_PAYLOAD` | 1024 | payload bytes | +| `MAX_FRAME` | 1500 | largest frame (and MTU) supported | +| `SUMMARY_BYTES` | 512 | full Bloom summary size | +| `ACK_IDS_CAP` | 90 | ids in one `Ack` frame | +| `MAX_SIG` | 64 | signature bytes | +| `BLOOM_K` | 3 | hash functions per id | + +Local capacities and timings (peers tracked at once, the re-summary interval, +queue sizes, the idle-eviction window, and so on) are internal defaults, not +public API. They can change in a minor release, and a later 1.x release will +make them configurable. ## Building and CI @@ -593,7 +691,15 @@ sh scripts/ci.sh ``` GitHub Actions runs the same script on every push to `master` and on every -pull request. +pull request. Two more jobs run there: + +- **MSRV:** builds `tines-core` on Rust 1.82 (its `rust-version`), with and + without default features and for `thumbv7em-none-eabihf`. It builds only, + because a test dependency needs a newer compiler. +- **Fuzz:** a scheduled nightly job runs `cargo-fuzz` on the `fuzz/` crate's + `node` target, in both its random-bytes and adversarial-peer modes, and + keeps any crash reproducer. A bounded proptest twin of the same harness runs + in the normal test suite. ## Stability @@ -637,7 +743,7 @@ The specs are authoritative; the plans argue from them. - **Fleet deployment:** `docs/superpowers/specs/2026-10-18-tines-fleet-deployment-design.md` - **Fleet fixes:** `docs/superpowers/specs/2026-10-19-tines-fleet-fixes.md` - **Host hardening (next for the hosts):** `docs/superpowers/specs/2026-10-19-tines-host-hardening.md` -- **tines-core 1.0 (next for the core):** `docs/superpowers/specs/2026-10-18-tines-1-0-design.md` +- **tines-core 1.0 (implemented on `feat/tines-core-1-0`; publish pending):** `docs/superpowers/specs/2026-10-18-tines-1-0-design.md` - **Threat model:** [`THREAT_MODEL.md`](THREAT_MODEL.md) - **Plans:** `docs/superpowers/plans/` - **Changes:** [`CHANGELOG.md`](CHANGELOG.md) diff --git a/THREAT_MODEL.md b/THREAT_MODEL.md index e32c425c..d004016d 100644 --- a/THREAT_MODEL.md +++ b/THREAT_MODEL.md @@ -3,10 +3,13 @@ - **Assessed:** `master` at `3b1a644` (tines-core / tines-store / tines-bt / tines-ros2 / tines 0.3.0, fleet-deployment milestone mostly landed) - **Date:** 2026-10-19 -- **Updated:** 2026-10-20 at `89041ea`, after the fleet fixes landed. The - findings keep their original text; each fixed one ends with a **Fixed** - or **Partly fixed** line, and §6 adds F18, found in the re-check. -- **Status:** living document.It's not a spec: update it when the code or the +- **Updated:** 2026-10-21 at `411daa0` (`feat/tines-core-1-0`), after the + tines-core 1.0 review fixes landed. Earlier update: 2026-10-20 at + `89041ea`, after the fleet fixes. The findings keep their original text; + each fixed one ends with a **Fixed** or **Partly fixed** line. §6 adds + F18, found in the fleet re-check, and F19, found with the 1.0 trusted-time + work. +- **Status:** living document. It's not a spec: update it when the code or the trust assumptions change. - **Related:** the 1.0 spec (`docs/superpowers/specs/2026-10-18-tines-1-0-design.md`), especially §2a (signature slot) and §2b (age and the 1.1 threat model). @@ -57,22 +60,24 @@ threat questions (§8) before designing signing. | F5 | Fixed | `4718b6f`, `24b5a78` | | F17 | Fixed | `6e2dc6f` | -**1.0 fixes, 2026-10-21** (through `941a7ce`): +**1.0 fixes, 2026-10-21** (through `411daa0`, on `feat/tines-core-1-0`; not +yet merged or published): | # | Status | Commit | |---|---|---| -| F6 | Fixed | `fb71810` | -| F8 | Fixed | `fb71810` | -| F9 | Fixed | `fb71810`, `a180ec2` | -| F13 | Fixed | `fb71810` | -| F18 | **Partly fixed**: the stale clock-marker tick is fixed (`Store::flush(now)` takes the caller's `now`); hosts still don't flush on a schedule | `c42f5de` | -| F19 | **Open** (new, 2026-10-21): peer-derived trusted time is unauthenticated | — | +| F6 | Fixed | `fb71810`, `506d880` (the pin resets on a repeat `Contact`) | +| F8 | Fixed | `fb71810`, `84c79fa` (silent and contact-only peers are evicted) | +| F9 | Fixed | `fb71810`, `a180ec2`, `53f592f` (rate-limit slack) | +| F13 | Fixed (Reassembler displacement accepted, per spec) | `fb71810`, `506d880`, `9362e9f` | +| F18 | **Partly fixed**: the store side is done (flush through the `Store` trait, the caller's tick as the marker, clamping), and both hosts flush right after they first adopt trusted time. Hosts still don't flush on a timer or at shutdown | `c42f5de`, `06a0577`, `039fbf7`, `e4544bf` | +| F19 | **Partly mitigated** (new, 2026-10-21): peer time is chosen by decision J and bounded to a plausible range, but it stays unauthenticated until 1.1 | `039fbf7`, `e4544bf`, `4690bf9`, `03b75f2` | Still open: - F10–F12: the host-hardening spec. - F7 and F14: 1.1. -- F18 (remaining): scheduled flushes, host work (§10). -- F19: 1.1 (signatures), or a local clock source before peers (§6). +- F18 (remaining): timed and shutdown flushes, host work (§10). +- F19 (remaining): 1.1 (signatures); until then, a local clock source + wherever one exists (§6). ## 2. System and trust boundaries @@ -145,10 +150,10 @@ the fleet fixes. | Malformed frames dropped | Hard constraint | **Holds** in core (wire.rs:64-69). The 1.0 spec adds reporting. | Unchanged; 1.0 adds reporting. | | Bounded memory in core | Fixed-capacity buffers | **Holds** for the exchange (§6 table). **Doesn't hold for the system**: MemStore and LogStore are unbounded without `trim`, and no host calls `trim` (F10). | Unchanged in practice: the service calls `trim` with an empty default policy, tines-ros2 never does (F10, F18). | | Custody: ≥ 1 holder (no carrier failure) | Fleet spec, sim property | **Breaks without an attacker** when a store is full (F1). Breaks trivially under A1 (F7). | Holds without an attacker (F1 fixed, with a sim property). Still breaks under A1 (F7). | -| Frames bound to the advertised NodeId | Fleet spec ("frames whose `from` differs … are dropped at the transport") | **Doesn't hold** (F3, F6). | **Holds at the transport** for tines-bt (F3), against unauthenticated adverts. Core still re-reads `from` per frame (F6). | +| Frames bound to the advertised NodeId | Fleet spec ("frames whose `from` differs … are dropped at the transport") | **Doesn't hold** (F3, F6). | **Holds at the transport** for tines-bt (F3), against unauthenticated adverts. Since 1.0 (`fb71810`, `506d880`) core also pins a peer's NodeId to its first frame in a contact and drops, and counts, a frame whose `from` differs (F6). | | Released messages not re-forwarded after reboot | Fleet spec | **Doesn't hold**: release flags are never flushed (F4). Safe direction (duplicate custodian), but wasteful. | **Holds**: `Release` records are appended at once (not fsynced until the next flush, so a power cut can still lose one). | | `next_seq` persisted across reboots | Core contract | **Doesn't hold** in either host (F4). | **Holds**: a persisted boot counter seeds `first_seq` in both hosts. | -| Age continues across reboot | Fleet spec | Partly. Hosts open the store at tick 0, and the service's tick is wall-clock *seconds* (F17). | Partly. Monotonic ms (F17 fixed) and clock markers fold age across a restart, but only up to the last flush, which may be long ago or never (F18). | +| Age continues across reboot | Fleet spec | Partly. Hosts open the store at tick 0, and the service's tick is wall-clock *seconds* (F17). | Partly. Monotonic ms (F17 fixed) and clock markers fold age across a restart, but only up to the last flush, which may be long ago or never (F18). Since 1.0 the downtime itself is folded in once the host has trusted wall time; without it, held addressed messages expire and broadcasts are floored (F18, F19). | ## 6. Findings @@ -312,6 +317,10 @@ handshake. the transport, so this is F3's consequence. Still, core could cheaply refuse a NodeId *change* for an existing peer within a contact, and should, once the transport binds identities. +- **Fixed in `fb71810`, `506d880`:** the first decoded frame in a contact + pins `peer.id`. A later frame with another `from` is dropped and counted as + `DropReason::IdentityChanged`. The pin resets on `Lost` and on a repeat + `Contact`. ### F7 — Custody can be taken by anyone the carrier sends to (Medium; by design v1) @@ -339,6 +348,12 @@ Eight spoofed addresses lock a robot out of exchange until restart. **Fix:** idle-timeout eviction in core (a local constant), and real `Lost` events from tines-bt. +- **Fixed in `84c79fa`** (first attempt `fb71810` evicted only the peer that + had just sent): a peer with no inbound frame for `PEER_IDLE_MS` (30 s), + counted from its contact if it never sent, is removed as if `Lost`. Idle + peers are swept before a peer is added and at the start of every + `next_frame`, so eight silent or contact-only addresses can't hold the + table. ### F9 — Summary amplification (Medium, A1) @@ -351,9 +366,18 @@ re-queues what was already sent. - Each rebuild also scans every broadcast and clones payloads, which costs CPU and allocation per summary, with no rate limit. -- **Fix:** rate-limit summary processing per peer (honour at most one per - `SUMMARY_INTERVAL_MS`), and remember what was already sent this contact - (`bcast_sent` already exists) so a repeated summary doesn't re-queue it. +- **Fix** (as amended in spec §10, 2026-10-20 and decision L): + - Rate-limit summary processing per peer: act on at most one summary per + `SUMMARY_INTERVAL_MS`. To allow for poll jitter, a summary is accepted + once at least `SUMMARY_INTERVAL_MS / 2` has passed since the last one + acted on, which still bounds replay to two per interval. + - The send-queue rebuild skips ids the peer's filter says it holds + (`Bloom::might_contain`). Output per received summary stays bounded by + `SEND_QUEUE_CAP`. + - It doesn't skip ids already sent this contact. "Sent" doesn't mean + "received", so a peer that lost an id must get it again on its next + summary. +- **Fixed in `fb71810`, `a180ec2`, `53f592f`.** ### F10 — Unbounded growth (Medium, A0/A1/A4) @@ -364,6 +388,9 @@ re-queues what was already sent. which removes nothing; tines-ros2 never calls it. Releases are now one small record each rather than a growing list, but the log still never compacts. + - *At `411daa0`:* `LogStore` rewrites the log as a snapshot at every open + and after the wall-clock fold (`06a0577`), which compacts it. Between + those points it only grows, and the trim policy is still empty. - Unbounded channels: tines-bt's inbound and outbound `mpsc` (transport.rs:207-208), and tines-ros2's inbound `mpsc` (node.rs:152). - The IPC server allocates up to 1 MiB per request frame. @@ -435,6 +462,15 @@ re-queues what was already sent. - **Trailing bytes:** `postcard::from_bytes` accepts trailing bytes after a valid frame. That's harmless now, but the 1.0 vectors should pin rejection so the wire format is canonical. +- **Fixed in `fb71810`, `506d880`, `9362e9f`**, with Reassembler + displacement accepted, per spec §10: + - the event queue drops the oldest event; + - removing a peer purges that peer's pending acks, and only its own + (`506d880`; the first fix purged every peer's); + - a frame, and a reassembled observation (`9362e9f`), must decode with + no trailing bytes, or it's counted as `Malformed`; + - displacement stays as it is: it's same-address only, and the transport + binding (F3) plus the F6 pin limit who can do it. ### F14 — Belief policies trust attacker-controlled fields (Medium; by design v1) @@ -489,8 +525,8 @@ marker that carries age across a restart, and it's the only `fsync` for inserts and trims (log.rs module docs, "Durability contract"). - The service runs `trim` and `flush` only after `POLL_CAP` (64) polls that - each stored something new (tines/src/ipc.rs:199-217). The counter - advances only while observations arrive, so a robot that is mostly + each stored something new (tines/src/ipc.rs:269-279 at `411daa0`). The + counter advances only while observations arrive, so a robot that is mostly offline, the normal case, may never flush. - tines-ros2 never calls `flush` or `trim`. - Even when it runs, `flush` wrote `LogStore`'s own tick as the clock @@ -514,30 +550,81 @@ inserts and trims (log.rs module docs, "Durability contract"). `flush_interval_ms`, from the host's monotonic clock) in both hosts, and on clean shutdown. The fleet-fixes spec asked for "a scheduled flush". The trim defaults belong to host hardening (F10). +- **Partly fixed in `c42f5de`, `06a0577`, `039fbf7`, `e4544bf`.** + - **Store side, done.** `Store::flush(now)` takes the caller's tick + (`c42f5de`). `LogStore` implements it as the trait method, so + `Node::flush` really writes the marker and fsyncs; before `06a0577` it + hit the trait's no-op default. An earlier `now` is clamped to the + store's time anchor. + - **Flush after first trusted time, done.** Both hosts adopt trusted time + through `tines::trust::adopt_wall_clock` (tines/src/trust.rs:90-108), + which flushes right after the fold. So the first marker with a wall time + reaches disk at once. + - **Snapshots.** `LogStore` also rewrites the log at open and after the + fold (decision I), so the fold's results survive a reboot. + - **Still open (host work):** + - The service's scheduled trim and flush are still tied to `POLL_CAP` + productive polls (tines/src/ipc.rs:269-279). + - tines-ros2 flushes only when it adopts trusted time, and never trims. + - Neither host flushes on a timer or at clean shutdown. A quiet robot + that loses power long after its last flush still under-counts the + time since that flush. -### F19 — Peer-derived trusted time is unauthenticated (Medium, A0) +### F19 — Peer-derived trusted time is unauthenticated (Medium, A0/A1) *Added 2026-10-21, when the hosts gained the peer-time path (Task 10b).* -When a host has no `trusted_clock = "system"` source, it estimates wall time -from a peer's observation: `observed_at + age_ms` from the youngest such -observation, then folds it into the store once. Nothing authenticates that -estimate. +When a host has no local trusted clock, it estimates wall time from peers' +observations (`observed_at + age_ms`, advanced by local ticks since the +observation arrived), then folds it into the store once. Nothing +authenticates that estimate. - **Effect:** - - A forged time **ahead** of the true time only makes observations expire - early: the fold ages everything faster, addressed messages die sooner, - and `observed_at` stamps run fast. Availability loss, not integrity loss. - - A forged time **behind** the true time takes the "downtime unknown" - path (or under-folds), leaving the store quarantined longer or ages - younger — the same effect as F17/F18 by another route. - - Worst case: the robot's own `observed_at` stamps are wrong until a - local source corrects them, so every observation it publishes between - the forged fold and the correction carries a bad wall time. -- **Mitigation:** prefer a local source whenever one exists (the config - flag is exactly that); otherwise accept that a peer's time is a - lower-bound hint, not a trusted clock. Signing and verification (1.1) - closes it; a monotonic-host sanity check (reject a wall time behind the - last fold) would contain the "behind" case today. + - A time too far **ahead** makes the fold over-count downtime: addressed + messages expire early, facts look stale, and `observed_at` stamps run + fast. That's availability loss, not integrity loss. + - A time **behind** the last clock marker reads as "clock went backwards", + which takes the unknown-downtime path: held addressed messages expire + and broadcasts are floored. Also the safe direction. + - A time behind the truth but still ahead of the marker under-counts the + downtime, so held ages come out too young. That's the dangerous + direction (spec §2b, §9). Honest estimates can land here, because age + is a lower bound. + - Before the 1.0 review, a forged `observed_at = u64::MAX` overflowed + `wall + elapsed` in the hosts and the store: a panic in debug, a time + near 1970 in release (review B3). +- **Mitigated since `039fbf7`, `e4544bf`, `4690bf9`, `03b75f2`:** + - **Decision J: the latest estimate in a window.** From the first + plausible estimate, the host collects for `peer_time_window_ms` + (default 10 s) and then adopts the latest one, so an honest undershoot + loses to a better estimate. `LogStore::peer_time_estimate` returns the + latest estimate among observations stored since this open. A forger + gains nothing over the old smallest-age rule. + - **Plausibility bounds (review B3).** + - The hosts adopt only wall times between `WALL_FLOOR_MS` + (2026-01-01Z) and `WALL_CEILING_MS` (floor + 100 years, about 2126). + That applies to peers and to the system clock (`tines::trust`). + - The store drops estimates outside + `LogStoreBuilder::plausible_wall_ms` before choosing the latest, so a + forged far-future estimate can't hide the honest ones. Both hosts set + it. + - The wall-time arithmetic saturates. + - **Local sources first.** The service's `trusted_clock = "system"`, and + tines-ros2's new `trust_system_clock` / `trust_wall_clock`, both take + priority over peers. A ROS2-only fleet now has a local source. +- **Still open:** + - Peer time is unauthenticated until 1.1. Within the plausible range, a + forger can still choose the time, and the window takes the latest + estimate, forged or not. + - A wrong-but-plausible time spreads. The robot stamps its own + observations' `observed_at` from it, and peers without a trusted clock + can adopt those stamps in turn. + - The adopted time is corrected only at the next restart, or when a + local trusted source arrives. Nothing corrects it while the host runs + (decision J), and on the peer path it's adopted once per run. +- **Mitigation for deployments:** give every robot a local trusted source + (RTC, GPS, or an NTP-synced clock with the service ordered after + `time-sync.target`). Signing and verification in 1.1 close the forged + case. ## 7. Threats by STRIDE, mapped to findings @@ -582,22 +669,34 @@ The sim covers honest-node custody properties and random garbage frames (`tests/props.rs:281`, `tests/relay.rs`). It doesn't cover: - well-formed malicious frames: forged acks, spoofed `from`/`observer`, - NodeId changes mid-contact; + NodeId changes mid-contact (since `37ca04d` the fuzz harness generates + all three, but in the core harness only; the sim and the hosts don't); - ~~a store returning `Full` (F1)~~ (covered since `2738de4`: a unit test and the full-store custody sim property); -- summary floods and peer-table exhaustion (F8, F9); -- reassembly abuse; +- ~~summary floods and peer-table exhaustion (F8, F9)~~ (covered since + `37ca04d`, `8ba1e86`: the fuzz harness's adversarial mode replays + summaries and floods the peer table from 3 × `MAX_PEERS` addresses, and + the proptest twin checks recovery after `PEER_IDLE_MS`); +- reassembly abuse (now partly covered: the adversarial mode sends + conflicting, repeated and out-of-order fragment streams); - a full disk (`LogStore` corruption in the middle, oversized records, concurrent openers, and restart and seq continuity are covered since `848721a` and `4718b6f`); -- a quiet host restarting without having flushed (F18); +- a quiet host restarting without having flushed (F18). `tines-store` + now tests flushing through the trait, opening three times, and clamping + (`06a0577`), but no host-level test covers a long quiet run; +- forged peer time beyond overflow and range checks: nothing tests a + wrong-but-plausible estimate spreading through `observed_at` (F19); - most of the hosts: IPC abuse, ROS2 injection and self-echo (F11, F12). The service now has a two-instance exchange test (`d98895b`), and tines-bt has mock-radio tests for framing, binding and contact events (`7df9f41`). -The 1.0 fuzz harness (spec §7) should take an *adversarial peer* mode, not -just random bytes: structured frames with chosen ids and identities. +The 1.0 fuzz harness (spec §7) has an *adversarial peer* mode since +`37ca04d`, as well as random bytes: structured frames with chosen ids and +identities, checked against the §7 and §10 properties. It runs nightly in CI +(`.github/workflows/fuzz.yml`), with a proptest twin in the PR gate. The +agreed fuzz budget hasn't been run and recorded yet. ## 10. Where each finding is being fixed @@ -605,17 +704,18 @@ just random bytes: structured frames with chosen ids and identities. |---|---|---|---| | F1, F2, F5, F17 | Fleet milestone (M3) | `docs/superpowers/specs/2026-10-19-tines-fleet-fixes.md` | Fixed (§1) | | F3, F4 | Fleet milestone (M3) | same | Partly fixed; the rest has no spec yet | -| F18 | Host work, with or before host hardening | none yet | Open | +| F18 | Store side: tines-core 1.0 and tines-store; timed and shutdown flushes: host work, with or before host hardening | 1.0 spec §4, §9; none yet for the host part | Partly fixed (§1) | | F10–F12 | Host hardening, before a robot leaves the lab | `docs/superpowers/specs/2026-10-19-tines-host-hardening.md` | Open | -| F6, F8, F9, F13 | tines-core 1.0 | `docs/superpowers/specs/2026-10-18-tines-1-0-design.md` §10 | Open | +| F6, F8, F9, F13 | tines-core 1.0 | `docs/superpowers/specs/2026-10-18-tines-1-0-design.md` §10 | Fixed (§1); F13's Reassembler displacement accepted | +| F19 | Hosts and tines-store (decision J, plausibility bounds); authentication in 1.1 | 1.0 spec §9, decision J; §2a | Partly mitigated (§1) | | F7, F14, §8 | 1.1 (threat model, link authentication, signing) | 1.0 spec §2a, §2b | Open | | F15, F16 | Accepted for now | — | — | Mark each finding fixed here, with its commit, when it lands. The remaining F3 work (one configured peer) and F18 are host-only. Neither -blocks the tines-core 1.0 work, but both block a fleet of more than two -robots. +blocks tines-core 1.0, but both block a fleet of more than two robots. F19's +remainder waits for 1.1. ## 11. Recommendations, in order @@ -629,22 +729,27 @@ robots. locking). - F17 (monotonic ms clock). - Each with a regression test. -2. **Host follow-ups:** the F3 multi-peer allowlist and F18 scheduled - flush. Small, and needed before more than two robots share a fleet. +2. **Host follow-ups:** the F3 multi-peer allowlist and the rest of F18 + (timed and shutdown flushes in both hosts, and trim in tines-ros2). + Small, and needed before more than two robots share a fleet. The F18 + store side and the flush after first trusted time are done. 3. **Host hardening before any robot leaves the lab:** - F10 (trim, bounded channels, compaction). - F11 (socket permissions, peer creds, timeouts). - F12 (ignore own publications, SROS2 guidance, rate limits). -4. **Core hardening (rides with 1.0):** - - F8 (idle eviction). - - F9 (summary rate limit, no re-queue). - - F13 (event drop-oldest, purge acks on peer loss, reject trailing - bytes). +4. **Core hardening (rides with 1.0):** done on `feat/tines-core-1-0` + (2026-10-21), publish pending. + - F8 (idle eviction, including silent and contact-only peers). + - F9 (summary rate limit with slack; the rebuild skips what the peer's + filter holds). + - F13 (event drop-oldest, purge only the lost peer's acks, reject + trailing bytes; displacement accepted). - F6 (refuse NodeId change within a contact). 5. **1.1:** - Write the full threat model from §4/§8. - Choose link authentication. - - Add signing per spec §2a (F14, F7). + - Add signing per spec §2a (F14, F7), and authenticate peer time + (F19). - Add "replace if better signed" dedupe. 6. **Later:** confidentiality and rotating rendezvous identifiers (F15), only if the deployment needs them. diff --git a/tines-bt/README.md b/tines-bt/README.md index 759db2dd..6f104add 100644 --- a/tines-bt/README.md +++ b/tines-bt/README.md @@ -37,7 +37,7 @@ the peer's advertised [`NodeId`] instead: hard-coded one. A frame's declared sender is then checked against the bound [`NodeId`]: the -transport peeks the `from` field with [`wire::frame_sender`] and drops any frame +transport peeks the `from` field with [`frame_sender`] and drops any frame whose declared sender differs from the bound peer (an address-spoofing violation). @@ -49,7 +49,8 @@ violation). ## Multi-peer -Up to [`MAX_PEERS`] connections may be live at once, one per bound [`NodeId`]; +Up to `MAX_PEERS` (8, tines-bt's own cap, equal to tines-core's peer table) +connections may be live at once, one per bound [`NodeId`]; a second connection to the same [`NodeId`] replaces the first. **Current limit:** `BluerRadio::new` and `BtTransport::new` still take a single @@ -75,12 +76,11 @@ BlueZ version here, along with whether SeqPacket worked or the length-prefix fallback was needed. [`Transport`]: https://docs.rs/tines-core/latest/tines_core/transport/trait.Transport.html -[`NodeId`]: https://docs.rs/tines-core/latest/tines_core/observation/struct.NodeId.html +[`NodeId`]: https://docs.rs/tines-core/latest/tines_core/observation/type.NodeId.html [`Radio`]: crate::Radio [`BluerRadio`]: crate::BluerRadio [`adv::decode_ad_node_id`]: crate::adv::decode_ad_node_id -[`wire::frame_sender`]: https://docs.rs/tines-core/latest/tines_core/wire/fn.frame_sender.html +[`frame_sender`]: https://docs.rs/tines-core/latest/tines_core/fn.frame_sender.html [`MAX_FRAME`]: https://docs.rs/tines-core/latest/tines_core/consts/constant.MAX_FRAME.html -[`MAX_PEERS`]: https://docs.rs/tines-core/latest/tines_core/consts/constant.MAX_PEERS.html [`l2cap::SeqPacket`]: https://docs.rs/bluer/latest/bluer/l2cap/struct.SeqPacket.html [`SeqPacketListener`]: https://docs.rs/bluer/latest/bluer/l2cap/struct.SeqPacketListener.html