Files
tines/docs/superpowers/specs/2026-10-18-tines-1-0-design.md
T
kyleandClaude Opus 5.5 aedb8026d5
CI / ci (push) Failing after 33s
docs(spec): 1.0 decisions F (flush) and G (age across downtime)
- F: Store::flush(now) with a no-op default; Node::flush(now) replaces
  Node::store_mut. The caller's now anchors ages (fixes LogStore's stale
  clock marker, F18).
- G: age_ms is documented as a lower bound. Robots are assumed to have a
  trusted wall clock (RTC or GPS); LogStore adds power-off time on open
  from a wall-clock flush marker. Unknown downtime expires held addressed
  messages and floors held broadcasts' age at unknown_downtime_age_ms.
  Hosts set observed_at.
- AGENTS.md: specs and plans change only to record decisions Kyle has
  discussed and agreed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 01:14:37 -07:00

815 lines
43 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# tines-core 1.0 — design
Date: 2026-10-18
Status: approved by Kyle (2026-10-20). Kyle decided D1–D5 and Q1–Q3
(2026-10-18) and the review decisions R1–R3 and the `observed_at`
definition (2026-10-19); review fixes applied. v6 added §10 (core hardening
from `THREAT_MODEL.md`) and the adversarial fuzz mode; Kyle reviewed and
approved both (2026-10-20). Decisions F (`flush`) and G (age across downtime) added 2026-10-20.
Builds on: `2026-10-16-tines-design.md`, `2026-10-18-tines-addressed-delivery.md`,
`2026-10-18-tines-fleet-deployment-design.md` (assumed fully implemented)
## Intent
Make `tines-core` something a fleet can depend on for years:
- a wire format that says which version it is,
- a public API small enough to keep stable,
- a `Store` trait that third parties can implement without being broken by
the next release,
- evidence (fuzzing, pinned vectors, semver checks) that the untrusted-input
and compatibility promises hold.
1.0 is a promise, not a feature release. Nothing here adds behaviour to the
exchange; it removes, hides, pins, documents, and reserves space on the
wire for signatures, which ship in 1.1.
## Why
The fleet milestone puts tines on real robots that are mostly offline. A
robot that misses an update keeps running old code for weeks, so two problems
that don't matter in the simulator start to matter:
- **Wire drift.** Today a frame has no version. A node running a different
encoding either drops every frame as `Malformed` with no indication why, or
(worse) decodes a frame it misunderstands. Fleets need to fail loudly and
safely.
- **API drift.** 0.3.0 had to break every out-of-tree `Store` impl to add
`release`. Each such break forces every host crate (`tines-store`,
`tines-bt`, `tines-ros2`, `tines`) to move in lockstep. A frozen,
extensible surface stops that.
No robot runs tines yet, and none will be deployed with a pre-1.0
`tines-core`. 1.0 is the first version that ships on a robot, so there is no
installed base to migrate (D5).
## Goals
1. **Versioned wire format.** Every frame starts with a version byte. A node
drops frames of any other version, and malformed frames, and reports
both (D3).
2. **A stable, minimal public API** for `tines-core`: what an application, a
`Store` backend and a `Transport` adapter need, and nothing else.
3. **A settled `Store` trait** whose future additions can always be
non-breaking.
4. **A reserved signature slot** on every observation, with a defined
`Signature` type, so 1.1 can add signing and verification without
changing the wire format (D1).
5. **Evidence:** fuzzing of the untrusted-input path, pinned wire and
identity vectors, an MSRV checked in CI, and `cargo-semver-checks` gating
releases after 1.0.
6. **Complete docs:** every public item documented (`#![deny(missing_docs)]`),
plus a written compatibility policy.
## Non-goals
- New protocol behaviour. Custody, receipts, summaries and fragmentation stay
exactly as the fleet milestone leaves them.
- **Signing and verification: 1.1.** 1.0 defines the `Signature` type and
carries it on the wire, untouched. It never creates, checks or acts on a
signature. 1.1 adds signing, verification and key handling as a
non-breaking minor release with significant new features (§2a). 1.1
starts with a written threat model covering, among other things, whether
`age_ms` must be authenticated (§2b).
- Version negotiation or multi-version nodes. A 1.x node speaks wire v1
only; see "Compatibility policy".
- 1.0 for the other crates. `tines-sim`, `tines-store`, `tines-bt`,
`tines-ros2` and `tines` stay 0.x and version independently (D4).
- Migrating pre-1.0 stores. Pre-1.0 `tines-store` logs are wiped, not
migrated (D5).
- Topic schema/version tags. The core spec's open question stays a naming
convention in the topic string, owned by applications.
- LoRa efficiency (`FRAG_OVERHEAD`) — still a `tines-lora` concern.
## Anti-goals
- A 1.0 that has to go to 2.0 the first time a robot gets a new radio. Local
capacities (`MAX_PEERS`, queue caps) must be changeable in a minor release.
- A 1.1 that has to break the wire to add signatures. That's why the slot
is reserved now.
- Freezing internals. `Exchange`, `Peer`, `Bloom`, `Reassembler` and `Frame`
layout are implementation, not API.
- A version byte that lets old and new nodes misread each other's frames.
## Constraints
- All existing core invariants hold: `#![no_std]`, `alloc` optional, builds
for `thumbv7em-none-eabihf` without default features, no async/threads/I/O,
no panic on untrusted input, `Addr` never mapped to `NodeId` by core.
- The identity hash (`observer, seq, topic, payload, to`) does **not**
change. Its pinned vector carries over from 0.x unchanged. The signature
is excluded from the id, as the core spec planned.
- 0.x and 1.0 nodes can't talk (1.0 frames carry a version byte 0.x
doesn't expect). That doesn't matter: no robot runs 0.x (D5).
- `rust-toolchain.toml` stays pinned for CI; MSRV is a separate, lower floor.
## The how
### 1. Wire envelope, version byte, and drop reporting
- Every frame on the wire is `[WIRE_VERSION: u8] ‖ postcard(Frame)`.
`WIRE_VERSION` is a public constant.
- Decode: a frame whose first byte isn't `WIRE_VERSION` is dropped before
postcard sees it. Empty frames are dropped.
- **The v1 byte value is chosen so pre-1.0 nodes reject it.** 0.x decodes
the first byte as postcard's `Frame` variant tag (0–3 after the fleet
milestone). Any value ≥ 4 is an unknown variant and decodes as
`Malformed`. Conversely, a 0.x frame starts with 0–3, which a 1.0 node
rejects as a version mismatch. No robot runs 0.x, but test rigs and
simulators might, so this costs nothing to get right. Avoid 0–3 (valid
0.x tags). Prefer a value in 4–0x7F: postcard reads it as a single-byte
unknown tag. A value ≥ 0x80 also ends up `Malformed`, but only because
postcard treats it as a varint continuation and reads a second byte,
which is a needless subtlety.
- **Every drop is reported (D3).** A frame dropped at the envelope or in
decoding is reported against the transport `Addr` it arrived from, with a
`#[non_exhaustive]` `DropReason`:
- `VersionMismatch { version: u8 }`: an un-updated or foreign peer.
- `IdentityChanged`: a frame's `from` differs from the NodeId already
pinned for this contact (§10).
- `Malformed`: the version was right but the body didn't decode, or
decoded to something core must reject (e.g. a `Signature` whose length
doesn't fit its algorithm, §2a).
An operator then sees "met an un-updated robot" or "a peer is sending
garbage" instead of a silent non-exchange. `Malformed` from a v1 peer
means one of three things:
- a bug,
- an attack,
- a *newer* 1.x peer sending a `Frame` variant this node predates. That
case is expected and safe (see "What a version bump means" below), so
tooling shouldn't treat every `Malformed` as an alarm.
- **Reports carry the `Addr`, not a `NodeId` (R1).** The sender's NodeId is
inside the frame and can't be trusted before decode. The transport `Addr`
is local truth that doesn't depend on the frame, so it's what makes a
drop attributable. The report is only as good as the transport's
binding: on BLE, `Addr` must be the NodeId bound to the connection (see
`THREAT_MODEL.md` F3).
- **Drops are counted, not queued (R2).** Core keeps a small fixed table of
counters keyed by `(Addr, DropReason)`, so that `VersionMismatch` with
different versions counts separately. Each new drop increments its
counter instead of pushing an event. `drain_events` reports each nonzero
counter once as `NodeEvent::Dropped { addr, reason, count }` and resets
it.
- A flood of bad frames therefore costs one table slot, not the event
queue. `Delivered` receipts can never be evicted by drops.
- The counting is itself the rate limit; no separate limiter is needed.
- When the table is full, a new `(Addr, reason)` pair goes into one
overflow counter, reported with no address. Losing attribution under a
flood is acceptable; losing the count isn't.
- The table size is a local constant (§5).
- Expired addressed observations are *not* drops. They're well-formed and
are discarded by policy; reporting them would be noise.
- `SUMMARY_OVERHEAD` (private, `exchange.rs`) and `FRAG_OVERHEAD` absorb
the extra byte. Every frame
still fits `mtu()` (Review Focus #6).
- **What a version bump means:** any change to the encoding of an existing
variant, the meaning of an existing field, or a wire constant (§5) is a new
`WIRE_VERSION` and a semver-major release. Appending a *new* `Frame`
variant at the end is **not** a bump. Older 1.x nodes decode an unknown
tag as `Malformed`, drop it and report it, which is the same safe
degradation `Ack` relied on. New variants must therefore be optional to
the protocol: a peer that drops them still interoperates correctly.
### 2. Observation: fixed shape, invariant enforced by type
Today `to: Option<NodeId>` and `lifetime_ms: Option<u64>` are separate
public fields, and `reject()` has to drop the two malformed combinations.
1.0 makes the invalid states unrepresentable (D2):
```rust
#[non_exhaustive]
pub struct Observation {
pub observer: NodeId,
pub seq: u64,
pub age_ms: u64,
pub observed_at: Option<u64>,
pub topic: Topic,
pub payload: Payload,
pub dest: Option<Dest>, // replaces to + lifetime_ms
pub sig: Option<Signature>, // reserved; signing is 1.1 (§2a)
}
#[non_exhaustive]
pub struct Dest { pub to: NodeId, pub lifetime_ms: u64 }
```
- Fields stay readable (`Store` backends and belief policies need them), but
`#[non_exhaustive]` stops outside crates from building the struct
literally, so fields can be added in 1.x. Construction goes through
`Observation::broadcast(observer, seq, topic, payload)` and
`Observation::addressed(observer, seq, topic, payload, to, lifetime_ms)`,
plus `with_observed_at`. Applications normally never build one —
`Node::observe`/`observe_to` do. 1.0 has no public way to set `sig`; 1.1
adds it.
- **The observation's wire encoding changes in v1; only its identity is
preserved.** `dest` replaces `to` + `lifetime_ms`, and `sig` is appended,
so postcard encodes an `Observation` differently from 0.x. That's fine
under the version byte (§1). The id hash reads `dest.map(|d| d.to)` and
is byte-for-byte the same as 0.x; the pinned vector proves it. `sig` is
excluded from the id.
- **`observed_at` gets a defined meaning (Kyle confirmed, 2026-10-19).**
Today the field has no docs and no pinned units, but §2a signs it and
§2b cross-checks against it. Proposed definition:
- **Units:** the observer's wall-clock time when it made the
observation, in milliseconds since the Unix epoch (UTC).
- **`None`:** the observer had no clock it trusts. A robot without an
RTC or time sync must leave it `None` rather than guess.
- **Never changed:** set once by the observer; relays never change it.
- **Not a local tick:** it's unrelated to `Tick`, which stays local
monotonic milliseconds and never crosses the wire.
The rustdoc on the field says all of this.
- **`age_ms` is a lower bound on the true age (decision G).** It counts the
time the observation spent on running clocks, and not counted are:
- time a holder was powered off, since `Tick` restarts at 0 on every boot;
- time between a holder's last `flush` and a crash (§4);
- time in flight, since a frame carries the age it had when sent.
So errors only ever make an observation look younger, which is the
dangerous direction:
- a stale fact wins `LatestWins`;
- a young age spreads, and first-copy-wins dedupe keeps it;
- an addressed message outlives its `lifetime_ms`;
- `trim` by `max_age_ms` keeps what it should drop.
1.0 handles each gap:
- **Power-off time** is recovered by the store, from a wall clock, when it
opens (§9). The fleet assumes every robot has a clock it can trust
across a sleep, an RTC or GPS (decision G). It's cheap, and robots sleep
through charge cycles.
- **Crash time** is bounded by how often the host flushes (§4; host work,
`THREAT_MODEL.md` F18).
- **In-flight time** is negligible on the links tines targets, and is
accepted.
The rustdoc on `age_ms` says it's a lower bound and points applications
that need absolute time to `observed_at`. With a trusted clock assumed,
hosts set `observed_at` on every observation they make (R4). Today both
hosts pass `None`.
- `reject()` keeps only the expiry check; the malformed-pair branch and its
tests go away (the pairing is now enforced by the type, including on
decode).
### 2a. The signature slot (D1)
1.0 reserves the slot and fixes the type; **1.1 adds signing and
verification.** That's a non-breaking minor release with significant new
features. The rustdoc for `Signature`, `SigAlgo` and `Observation::sig`
must say so in those words, so nobody mistakes the 1.0 slot for a security
feature.
```rust
#[non_exhaustive]
pub struct Signature {
pub version: u8, // signature format version
pub algo: SigAlgo, // u8 on the wire
pub bytes: heapless::Vec<u8, MAX_SIG>, // the signature itself
}
#[non_exhaustive]
#[repr(u8)]
pub enum SigAlgo {
Ed25519 = 1,
// 0 is reserved (never valid). PQC algorithms get new ids later.
}
```
- **Coverage rule (normative):** a signature covers the whole observation,
and MUST also cover its own `version` and `algo` bytes. The second part
is what prevents fallback attacks: an attacker can't relabel a signature
as an older format version or a weaker algorithm without invalidating
it. The signed message is
`version ‖ algo ‖ observer ‖ seq ‖ observed_at ‖ topic ‖ payload ‖ dest`
(`dest` including `lifetime_ms`), in a canonical, length-prefixed
encoding that 1.1 pins with a test vector.
- **One field is necessarily unsigned: `age_ms`.** Every relay advances
it, so no observer signature can cover it. A relay can therefore make an
observation look older or younger than it is. 1.1's docs must say so,
and belief policies that care should weigh `observed_at` (signed) where
the observer has a clock. Whether age needs authenticating some other
way is an open 1.1 question with its own threat model (§2b).
- `sig` itself is excluded (a signature can't cover itself).
- The signed bytes are *not* the 16-byte id. A 128-bit truncated hash
gives only 64-bit collision resistance, which is too weak to sign, and
the id also leaves out `observed_at` and `lifetime_ms`.
- **The wire carries `algo` as a raw `u8`, not as a postcard enum tag.** A
postcard enum would make a 1.0 node reject any observation carrying an
algorithm id it doesn't know, so it would drop an observation it could
otherwise store and forward. Instead `SigAlgo` decodes through a `u8`
with a catch-all (`SigAlgo::Unknown(u8)`, or an equivalent the builder
picks). Unknown algorithms are carried and forwarded untouched. Only
`algo = 0` and a length that's wrong for a *known* algorithm (Ed25519
must be exactly 64 bytes) are `Malformed`.
- **Unknown versions are carried too.** 1.0 checks `version` only for
`Malformed`-level sanity (the builder pins the rule; e.g. `version = 0` is
invalid). It never rejects a well-formed signature it can't interpret,
for the same reason as unknown algorithms.
- Core never creates, checks or strips a signature in 1.0. `insert` stores
`sig` with the observation, and every send carries it. Relays don't
re-sign.
- **Which key verifies it** is 1.1's problem. The natural binding is
`observer` → public key through a fleet key registry the host supplies.
1.0 doesn't need a key id on the wire because the observer `NodeId` is
already there.
- **Known gap for 1.1:** dedupe is first-copy-wins, so an unsigned or
forged-signature copy that arrives first would shadow the real one. 1.1
needs "replace if better signed" semantics in `Store` (e.g. a defaulted
`replace_sig`, allowed under the §4 extension rule because the safe
default — keep the first copy — loses only the upgrade).
### 2b. Age, authenticity, and the 1.1 threat model (direction, not 1.0 work)
None of this changes 1.0. It records what 1.1 has to settle before it
designs signing, so the 1.0 slot doesn't get mistaken for the whole answer.
**1.1 needs a written threat model before choosing a mechanism.** In
particular it must decide whether `age_ms` needs to be authenticated, by
whom, and against which attackers. Two attackers are worth separating:
- **Outsider:** no fleet key. Can listen, inject frames, replay frames,
and change frames in transit on the radio.
- **Insider:** a fleet robot with a valid key that is compromised or buggy.
It can already drop, withhold or delay anything it carries; custody
already accepts carrier failure.
**What the observer's signature can't protect.** `age_ms` is advanced by
every holder, so the observer's §2a signature can't cover it. If it stays
unauthenticated, here is what a forged age breaks today:
| Forged age | Where core uses it | Effect |
|---|---|---|
| Lower (looks fresher) | `LatestWins` picks the smallest age | Stale world state beats a newer observation; robots plan on old facts. |
| Lower | `reject`: addressed expiry is `age_ms > lifetime_ms` | An expired directive (a task) lives again and is delivered after its lifetime. |
| Lower | `TrimPolicy::max_age_ms`, `keep_per_topic` keep the youngest | Stale data survives trimming and pushes out fresher data. |
| Higher (looks staler) | `LatestWins` | A fresh observation loses to an older one, which suppresses it. |
| Higher | addressed expiry | A live directive is rejected on arrival. For an insider that's the same as dropping it (accepted carrier failure); for an outsider it's a new way to kill custody traffic. |
| Higher | trim | Fresh data is trimmed early. |
| Either | dedupe keeps the first copy's age lineage (`insert` returns `Ok(false)` and leaves the stored age) | The first forged copy fixes the age at that receiver for good. A later honest copy can't correct it. |
The threat model should weigh these against what each attacker can
already do. An insider can already drop and delay, so age forgery adds
mainly the *freshness* attacks: stale facts looking new, and expired
directives coming back. An outsider can do all of the above unless frames
are authenticated hop by hop.
**Candidate mechanisms** (for 1.1 to choose between, after the threat
model):
- **Per-hop age statement.** The sending relay signs
`(observation id, age_ms, relay NodeId)`. The receiver checks it, then
discards it, and signs its own when it forwards. Size doesn't grow per
hop: about 80 bytes per transfer (relay id + 64-byte signature). It
costs one signature per send and one check per receive.
- Stops outsiders.
- Makes insider lies attributable to a specific robot, but doesn't
prevent them.
- Doesn't need the `Observation` slot: 1.1 can add it as a new frame
variant appended to `Frame`, which §1 allows in a minor release.
- **Authenticated link (likely preferred).** A short authenticated
handshake at contact start, then a short authentication tag on every
frame. It protects ages, `Ack`s (today guarded only by "accept from the
address we sent to"), summaries and fragments, for one signature
exchange per contact.
- Stops outsiders.
- Says nothing about insiders.
- Also a frame-level addition, so it fits 1.x.
- **Absolute-time cross-check.** `observed_at` is covered by the observer's
signature (§2a). A receiver whose clock roughly agrees with the
observer's can flag an `age_ms` that disagrees with
`now − observed_at` by more than an allowed skew. It costs nothing on the
wire, but needs clocks, which v1 robots may not have.
The observer's signature (§2a) is needed under every option: it's the only
thing that keeps the *content* checkable after any number of hops.
### 3. Public API surface
Public in 1.0 (everything else becomes `pub(crate)`):
| Item | Why it's public |
|---|---|
| `Node`, `ObserveError` | the facade applications use |
| `NodeEvent<A>` (non-exhaustive enum, generic over the transport `Addr`; replaces `ConfidenceEvent`), `DropReason` | `Delivered { peer, id }`, `Dropped { addr: Option<A>, reason, count }` (`addr` is `None` only for the overflow counter, §1); room for more |
| `Observation`, `Dest`, `Signature`, `SigAlgo`, `NodeId`, `ObservationId`, `Tick`, `Topic`, `Payload` | the data model |
| `Store`, `StoreError`, `TrimPolicy`, `MemStore` (`alloc`) | backends and their reference impl |
| `Transport`, `Event`, `SendError`, `FrameBuf` | transport adapters |
| `BeliefPolicy`, `LatestWins`, `SelfFirst` | the memory layer |
| `consts` (wire constants + local caps, §5), `WIRE_VERSION` | sizing |
| `frame_sender(bytes) -> Option<NodeId>` | lets a transport check a frame's claimed sender against the identity it bound to the connection without core's private `Frame` layout (fleet fixes spec §3; `THREAT_MODEL.md` F3) |
Made private: `Exchange`, `Peer`, `InFlight`, `Bloom` (including the
public `salt` field), `Reassembler`, `fragment`, `Frame`, `WireError`,
`FRAG_OVERHEAD`. Nothing outside `tines-core` uses them; the only mention
is a doc-comment link in `tines-sim/src/world.rs`, which is updated.
- **All public enums are `#[non_exhaustive]`**: `ObserveError`,
`StoreError`, `SendError`, `Event`, `NodeEvent`, `DropReason`,
`SigAlgo`. Adding an error case, an event, a drop reason or an algorithm
becomes a minor release. (Transports can still *construct* `Event`
variants; only exhaustive matches outside core need a wildcard.)
- **Public structs with public fields are `#[non_exhaustive]`** where they
may grow: `Observation`, `Dest`, `Signature`, `TrimPolicy`
(`TrimPolicy::default()` + field assignment still works).
- `Node` keeps its current methods: `new`, `id`, `next_seq`, `observe`,
`observe_to`, `poll`, `drain_events`, `belief`, `trim`, `store`,
`transport_mut`.
- It gains `flush(now) -> Result<(), StoreError>`, which delegates to
`Store::flush` (§4; decision F).
- `store_mut`, added by the fleet fixes so the service could reach
`LogStore::flush`, is removed. Read access through `store()` stays. `drain_events` returns `NodeEvent<T::Addr>`s: queued
`Delivered` events, then one `Dropped` per nonzero drop counter (§1).
- Every remaining public item is documented; `#![deny(missing_docs)]` goes in
`tines-core/src/lib.rs` (26 items are missing docs as of the fleet
milestone; most of them are the fields of the now-private types).
### 4. The `Store` trait, settled
The trait after the fleet milestone (`insert`, `contains`, `get`, `len`,
`for_each_id`, `for_each_broadcast_id`, `for_each_addressed_id`, `release`,
`is_released`, `for_each_topic`, `trim`) is the 1.0 trait, with these
changes:
- **`is_empty()`** gains a default (`len() == 0`), dropping the
`len_without_is_empty` allow.
- **`flush(&mut self, now: Tick) -> Result<(), StoreError>`** is added,
with a default that does nothing and returns `Ok(())` (decision F). It's
the store's durability point, and it does two things:
- everything inserted, released or trimmed so far is durable;
- `now` becomes the time anchor for ages after a restart.
More of the contract:
- Flushing twice is harmless.
- A `now` earlier than the last anchor is clamped to it.
- A failure returns `Err(StoreError::Backend)` and leaves the store
usable.
- The application decides when to flush; core never calls `flush`
itself. On flash, every flush costs wear, so how often is the
application's call, like `poll` and `trim`.
- `MemStore` keeps the default. `LogStore` implements it, and its clock
marker uses the caller's `now`; today it uses the tick of the last new
insert (`THREAT_MODEL.md` F18).
- **Release semantics are documented as best-effort.** `release` stays
infallible (`()`): if a persistent backend fails to record a release, the
worst case after reboot is a resumed copy, i.e. a duplicate custodian.
That's safe, never a lost message, so a `Result` would only add an error
path nobody can act on.
- **Stores keep `sig` verbatim.** `get` returns the signature it was given;
a backend that drops or alters it is non-conforming.
- **The exchange never trusts a backend's filtering.** Whether a backend's
`for_each_addressed_id` skips released or broadcast ids is an
optimisation only; core re-checks `dest` and `is_released` on every
candidate (already true after the fleet milestone; 1.0 pins it with a test
that uses a store which overrides nothing).
- **Rule for 1.x:** any method added to `Store`, `Transport` or
`BeliefPolicy` after 1.0 must have a default whose behaviour is *safe*: it
can lose performance or telemetry but never data or custody. A method that
can't have such a default is a 2.0 change. This is the rule `release`
violated in 0.3.0.
- A shared conformance test suite, `tines_core::store_conformance` (behind
a `conformance` feature, `std`-free), lets backends prove the contract:
- dedupe,
- age lineage,
- saturating ticks,
- release surviving `get`,
- `sig` round-tripping,
- trim order,
- no tombstones,
- `flush` being idempotent and leaving the store usable.
`MemStore` and `tines-store` both run it. This is new public API, called
out per AGENTS.md: it's what makes a third-party `Store` trustworthy.
### 5. Constants: wire vs local
`consts.rs` splits into two documented groups. The rule: **a constant is
a wire constant if changing it changes how a v1 peer decodes or interprets
a frame.** That covers capacity bounds a decoder enforces, and constants
both sides must agree on to read a field. Everything else is local.
- **Wire constants**, frozen for wire v1: changing any is a `WIRE_VERSION`
bump. The capacity bounds are `MAX_TOPIC`, `MAX_PAYLOAD`, `MAX_FRAME`,
`SUMMARY_BYTES`, `ACK_IDS_CAP` and `MAX_SIG` (= 64, Ed25519; Q1), plus the private
`MAX_FRAGMENTS`. `BLOOM_K` is a wire constant because both sides must
agree on it to read a summary.
- Post-quantum signatures don't fit in 64 bytes, so adding one is a wire
v2 / tines-core 2.0 change.
- **`MAX_FRAME = 1500` is a known 2.0 trigger (R3).** `Frag.data` is an
inline `Vec<u8, MAX_FRAME>`, so a peer can't decode a bigger frame. A
future radio that needs frames over 1500 bytes means wire v2. BLE (L2CAP
CoC) and LoRa are comfortably under it, and a bigger radio can still run
v1 with 1500-byte frames.
- **Local capacities and timings**, which can change in a minor release
because they never cross the wire: `MAX_PEERS`, `SEND_QUEUE_CAP`,
`ACK_BUFFER_CAP`, `BCAST_SENT_CAP`, `EVENT_QUEUE_CAP`,
`SUMMARY_INTERVAL_MS`, `REASSEMBLY_TIMEOUT_MS`, and the size of the
drop-counter table (§1). `EVENT_QUEUE_CAP` is local, even though
`ACK_IDS_CAP` is wire: the event queue never leaves the node, while an
`Ack` frame's id list is decoded by the peer.
AGENTS.md's constant list is updated to show the split.
A signed observation is up to 67 bytes bigger (`version` + `algo` + length +
64). Fragmentation already handles observations that don't fit one frame,
so nothing else changes. The fuzz and MTU tests must include
maximum-size, signed observations.
### 6. Bounce-table overflow, pinned
The bounce table (`SEND_QUEUE_CAP` entries, shared across peers, drop-oldest)
can overflow during a large sync. The 1.0 spec pins what overflow costs:
- A lost **Sent** record makes the peer's ack get rejected, so the
custodian keeps its copy. The result is a duplicate custodian, which is
safe.
- A lost **Received** record may let us offer an id back to the peer that
gave it to us. That peer either released it (and doesn't ack, per the
released-duplicate rule) or still holds custody (and acks, and we
release). Either way at least one holder remains. The cost is airtime.
- Requirement: a sim property drives the table past capacity across
`MAX_PEERS` peers and checks the "always ≥ 1 holder" invariant and that
every candidate still gets a turn (the custody cursor). Whether to give
each peer a fair share of the table instead of one shared FIFO is the
builder's call; it's a local change, not a wire one.
### 7. Evidence
- **Fuzzing.** A `fuzz/` crate (cargo-fuzz, outside the workspace, nightly)
with one main target that drives a real `Node` + `MemStore` through a
scripted `Transport`. The fuzz input is decoded into a sequence of
`Contact`/`Lost`/`Frame(bytes)`/`Tick(delta, may go backwards)`/`observe`
steps. After each step it checks:
- no panic,
- every sent frame is no larger than `mtu()` (randomised per run),
- no addressed observation is stored past its lifetime,
- every rejected frame is counted: the drained `Dropped` counts for an
address add up to the frames rejected from it (the overflow counter
included),
- stored signatures round-trip unchanged.
**Adversarial-peer mode.** Random bytes rarely get past the decoder, so
a second input mode builds *well-formed* frames from the fuzz input:
- Summaries with chosen bits, including all-zero, all-one and repeated.
- Obs and Frag frames with chosen ids, observers (including the node's
own NodeId), seqs, ages, lifetimes and addressing.
- Acks for ids that were or weren't sent to that address.
- Fragments that conflict, repeat or arrive out of order.
- `from` values that change within a contact.
These are driven from many addresses, beyond `MAX_PEERS`. On top of the
checks above, this mode checks the §10 properties: peer-table recovery,
bounded output per received summary, no NodeId change within a contact,
and drop-oldest event behaviour. It also checks the custody invariant
under the fleet fixes' F1 rule: an ack is sent only for a stored
observation.
Fuzzing through `Node` keeps every internal private. It runs as a
scheduled CI job (not a PR gate, since the toolchain is pinned stable),
with the corpus checked in.
- **Stable twin.** The same harness runs as a proptest in the normal test
suite with a bounded case count, so the PR gate still exercises it on the
pinned toolchain.
- **Pinned vectors** in `tines-core/tests/vectors.rs`:
- the identity hash (existing vector, unchanged),
- one golden encoding per `Frame` variant including the version byte,
- an observation carrying an Ed25519-shaped signature, and one carrying
an unknown algorithm id, both of which must decode,
- a 0.x-shaped frame that must be rejected as a version mismatch.
A change to any of these vectors is a wire-format change by definition.
- **MSRV.** `rust-version` is set in `[workspace.package]` to the lowest
version that builds `tines-core` (found with `cargo-msrv`, not guessed).
CI builds and tests `tines-core` on that version as well as the pinned
one. Raising MSRV is a minor release, called out in the CHANGELOG.
- **Semver gate.** After 1.0.0 is published, `cargo-semver-checks` against
the last published `tines-core` runs in `scripts/ci.sh`.
### 8. Compatibility policy (goes in README and the crate docs)
- **Wire:** every 1.x release speaks wire v1. Only new optional `Frame`
variants may be added, plus new `SigAlgo` ids and signature versions
within the 64-byte `MAX_SIG` budget. A wire-v2 change, including any
post-quantum signature or a `MAX_FRAME` above 1500, ships as
tines-core 2.0.
- **Identity:** `Observation::id` never changes within a major version, and
should not change across majors unless the hash itself is broken.
- **API:** standard semver, enforced by `cargo-semver-checks`; the
extension rule for traits in §4; MSRV bumps are minor.
- **Local behaviour** (capacities, timings, scheduling order, which
candidate goes first) can change in a minor release as long as the
protocol properties hold:
- custody: always ≥ 1 holder,
- receipts never shorten world state,
- every frame fits `mtu()`.
- **Signatures:** 1.0 carries them; 1.1 signs and verifies them, without
breaking the wire or the API.
### 9. tines-store and the upgrade (D5)
`tines-store` (0.x) moves to `tines-core = "1"` and its log format follows
the new `Observation` encoding. There is no migration:
- The log already has a versioned header and per-record CRC by then: the
fleet fixes (`2026-10-19-tines-fleet-fixes.md` §5) introduce both for
integrity reasons. 1.0 bumps that header's `format_version` for the new
`Observation` encoding. Opening a log with an older version is an error
that names the fix (delete the log). Silently misparsing an old log is
not allowed.
- **Ages across downtime (decision G).** The same format bump adds a
wall-clock reading to the clock marker `flush` writes: the host's trusted
wall time in milliseconds since the Unix epoch, or none.
- `LogStore::open` takes the host's current trusted wall time,
`Option<u64>`. The host passes `None` if its clock isn't trusted, for
example a dead RTC battery with no GPS fix yet.
- **Downtime known:** both readings are present and now ≥ the marker.
`open` adds `now − marker` to every held observation's age,
saturating, on top of the tick fold it already does.
- **Downtime unknown:** either reading is missing, or the clock went
backwards. `open` treats every held **addressed** observation as
expired: its age is set past `lifetime_ms`, so core never delivers or
forwards it, and `open` reports the count once. A custodian that can't
tell how long it slept must not deliver a directive that may have
expired. Losing a possibly valid one is the lesser harm, because
sharing is opportunistic and the originator can re-issue.
- **Broadcasts when downtime is unknown:** they're kept, but their age
isn't trusted. `open` raises every held broadcast's age to at least
`unknown_downtime_age_ms`, an open option the host configures.
- They lose `LatestWins` to any fresher fact and age out with the rest.
- They stay available as a last resort for topics nobody else has
reported.
- The floor must be below the host's trim `max_age_ms`; otherwise the
next trim drops them, and that's option 2 (drop) by another name.
The default is left to the builder (host hardening, F10, sets the
trim defaults).
- Every `open` that runs a fold reports the downtime it applied, or that
it was unknown, once (stderr plus a counter, like the skipped-record
count).
- Hosts wipe pre-1.0 logs on upgrade. That's acceptable because no robot
runs tines yet, and none will be deployed with a pre-1.0 `tines-core`.
- The header is what makes a later migration possible if one is ever
needed. Adding it is cheap now and expensive later.
### 10. Core hardening (from `THREAT_MODEL.md`)
The security assessment at `3b1a644` found core behaviours that a peer can
abuse without any key. All of them are local behaviour (§8), so 1.x could
change them. They're fixed before 1.0 anyway: they're cheap, and 1.0 is
when core's behaviour is written down as a promise. None of them adds
authentication. They bound what an unauthenticated peer can do; 1.1
decides who is allowed to be a peer.
1. **F6 — a peer's NodeId is fixed for the contact.**
- Today `on_frame` overwrites `peer.id` with each frame's `from`
(exchange.rs:723), so a peer can claim another identity mid-contact
and be handed that identity's addressed mail.
- New rule: the first decoded frame on a contact sets `peer.id`. A
later frame with a different `from` is dropped and counted as
`DropReason::IdentityChanged` (§1).
- The pin resets on `Contact`/`Lost`. On transports that bind identities
(tines-bt after the fleet fixes), the two always agree.
2. **F8 — idle peers are evicted.** A peer with no inbound frame for
`PEER_IDLE_MS` (local constant, default 30 s) is removed as if `Lost`.
Today peers leave only on `Lost`, so eight spoofed addresses can fill
the table until restart. With the fleet fixes' `Lost` events this
rarely triggers for honest peers, but it bounds the lockout.
3. **F9 — a summary can't be replayed for amplification.**
- At most one received Summary per peer is acted on per
`SUMMARY_INTERVAL_MS`. Extra ones are ignored; that's a policy, not a
drop.
- The send-queue rebuild skips ids already sent to that peer this
contact, using the existing per-peer sent record. That record is
best-effort under its cap, and a miss only costs a resend.
- The worst-case output per received summary stays bounded by
`SEND_QUEUE_CAP`, but repeating the summary no longer re-sends.
4. **F13 — smaller defects:**
- **Event queue drops the oldest** when full, as its docs and the fleet
spec say (today `push` silently drops the newest, exchange.rs:777).
This applies to `Delivered` events; `Dropped` uses counters (§1).
- **Pending acks for a removed peer are purged** on `Lost` or eviction,
so retries to a dead peer can't starve data frames.
- **Frames must be canonical.** Decode rejects trailing bytes after a
complete `Frame` (`postcard::take_from_bytes` and an empty remainder),
counted as `Malformed`. This is wire format, so it can't wait for 1.x,
and §7's vectors pin it with a valid frame plus one trailing byte.
5. **Considered and left out:**
- **A per-peer quota on new inserts** (threat model F10). A big honest
sync is indistinguishable from a flood without authentication, and a
quota would slow convergence for everyone. Storage bounds belong to
the host's trim policy and store cap (host hardening spec §1).
Revisit after 1.1.
- **Refusing inbound observations that claim this node as observer**
(threat model F14). It would stop forged "self" observations beating
`SelfFirst`, but also stop a robot recovering its own history from
peers after losing its log. Signatures in 1.1 fix the real problem.
New local constants, `PEER_IDLE_MS` and the drop-counter table size, join
§5's local group. `DropReason` gains `IdentityChanged`.
## Decisions (Kyle, 2026-10-18)
- **D1. Reserve the signature slot now: yes.** `Signature { version: u8,
algo: SigAlgo (u8), bytes }`; it must cover `version` and `algo`. Ed25519
is the only algorithm defined; PQC ids come later. Signing and
verification are 1.1, and the docs say so (§2a).
- **D2. `Dest` instead of `to` + `lifetime_ms`: yes, as recommended** (§2).
- **D3. Drop and report: yes**, and malformed frames are reported the same
way as version mismatches (§1).
- **D4. Only `tines-core` goes 1.0: yes.**
- **D5. No `tines-store` migration.** No robot runs tines yet, and none will
ship with a pre-1.0 `tines-core`; wiping logs is accepted (§9).
## Follow-up decisions (Kyle, 2026-10-18)
- **Q1. `MAX_SIG = 64`: agreed.** Ed25519 fits. Every observation
reserves `MAX_SIG` bytes inline, including on the MCU, and sizing for
post-quantum signatures (Falcon-512 is ~666 bytes, ML-DSA-44 is 2,420)
would cost that RAM for an algorithm nobody uses yet. A post-quantum
algorithm is therefore a wire v2 / tines-core 2.0 change. Smaller new
algorithms can still be added in 1.x through new `algo` ids.
- **Q2. Coverage: the whole observation, plus `version` and `algo`.** The
`version`/`algo` requirement exists to stop fallback attacks (§2a).
`age_ms` is the one field that can't be signed, because relays change it
(§2a).
- **Q3. Malformed frames are reported the same way as version
mismatches**, through `NodeEvent::Dropped { reason }` (§1).
## Review decisions (Kyle, 2026-10-19)
- **R1. Drop reports carry the transport `Addr`**, not a NodeId (§1).
- **R2. No second queue: drops are counted by `(Addr, reason)`** and
reported as `Dropped { addr, reason, count }` when drained. `Delivered`
receipts can't be evicted by drops, and the counting doubles as the rate
limit (§1).
- **R3. `MAX_FRAME` stays a wire constant at 1500**, and the spec names it
as a known 2.0 trigger (§5, §8).
The review's other fixes are applied too:
- `Malformed` can also mean a newer peer's optional variant (§1).
- The rule for wire vs local constants is written down (§5).
- The `Observation` encoding change is stated outright (§2).
- The `tines-sim` claim is corrected (§3).
- The version-byte guidance is corrected (§1).
- **R4. `observed_at` is defined** as the observer's wall-clock time in
milliseconds since the Unix epoch (UTC), or `None` if the observer has no
trusted clock. Relays never change it, and it's unrelated to `Tick` (§2).
## Decisions (Kyle, 2026-10-20)
- **F. `flush` goes on the `Store` trait, and `Node::flush(now)` replaces
`Node::store_mut`** (§3, §4).
- Every persistent backend needs a durability point and a time anchor,
so it belongs in the trait contract, not in one backend's own method.
- The default does nothing, so core stays I/O-free and `MemStore`
doesn't change.
- The application controls when flushes happen.
- Removing `store_mut` stops applications from changing the store behind
the exchange's back.
- Taking `now` also fixes `LogStore`'s stale clock marker (F18).
- The hosts' timed and shutdown flushes stay host work (F18).
- **G. Age across downtime** (§2, §9).
- `age_ms` is documented as a lower bound on the true age.
- Robots are assumed to have a trusted wall clock (RTC or GPS), cheap
for robots that sleep through charge cycles.
- Stores add power-off time on open, using a wall-clock reading in the
flush marker.
- When downtime can't be known, held addressed messages are expired:
possibly delivering an expired directive is worse than possibly
dropping a valid one, since sharing is opportunistic.
- Hosts set `observed_at`.
- When downtime can't be known, held broadcasts are kept but their age
is floored at `unknown_downtime_age_ms` (Kyle, 2026-10-20): assume the
worst about freshness without discarding data.
## Left to the builder
- The exact `WIRE_VERSION` value, within the rules in §1.
- The size of the drop-counter table (§1).
- The `SigAlgo` catch-all representation and the `version` sanity rule
(§2a).
- Fair-share vs shared FIFO for the bounce table (§6).
- Shape of the fuzz input decoder and the conformance suite's API.
- Order of work. Suggested:
1. vectors (they pin the starting point),
2. the envelope and drop reports,
3. `Dest`, then `Signature`,
4. the API narrowing,
5. core hardening (§10),
6. docs, fuzzing (including adversarial mode) and MSRV,
7. the 1.0.0 release, with `cargo-semver-checks` switched on.
## Done means
- `sh scripts/ci.sh` is green with `#![deny(missing_docs)]`, an MSRV job and
the vector tests.
- The fuzz target has run for an agreed budget (e.g. 24 CPU-hours) in both
input modes with no findings, and the corpus is committed.
- `THREAT_MODEL.md` marks F6, F8, F9 and F13 fixed.
- README has a "Stability" section with the §8 policy, including the 1.1
signature note; CHANGELOG has a 1.0.0 entry listing every break from 0.3.
- AGENTS.md lists this spec, the wire/local constant split and the trait
extension rule.
- `tines-store` refuses pre-1.0 logs with a clear error (§9).
- `tines-core` 1.0.0 is published, and the host crates depend on `"1"`.