Add the ledger: the audit writer and session state behind one lock

Implemented-By: OpenCode session (model recorded in docs/implementer-log.md)

crates/brokerd/src/ledger.rs

@@ -0,0 +1,499 @@
+//! The ledger: the audit writer and every session's state behind one lock, with the three steps
+//! that hold it. Without the one lock, two results for one session could each read `private` and
+//! the second write would put the taint back down; after any failed append or panic while the lock was held, the ledger refuses every later step, and only the next start's check puts a partial line on disk right.
+
+use std::sync::{Mutex, MutexGuard};
+
+use proto::{
+    ApprovalAnswer, AuditEvent, CallId, DataClass, DecisionRecord, DenyReason, GrantProblem, Mode,
+    SessionId, Timestamp, ToolRequest, ToolResponse,
+};
+
+use crate::approvals::Verdict;
+use crate::audit::{AuditError, Writer};
+use crate::grants::GrantSet;
+use crate::policy::{Ask, Denial, Label, SessionState};
+use crate::state::StateStore;
+
+/// The message `finish` returns when the raised taint and the record are not both on disk.
+pub const NOT_RECORDED: &str = "the result could not be recorded";
+/// After a panic while the lock was held: every call is denied until brokerd is restarted.
+pub const POISONED: &str = "brokerd: a thread panicked while holding the ledger; every call is denied until brokerd is restarted\nsee docs/runbook.md#audit-unavailable";
+/// After a failed append: every call is denied until brokerd is restarted.
+pub const STOPPED: &str = "brokerd: an earlier audit write failed; every call is denied until brokerd is restarted\nsee docs/runbook.md#audit-unavailable";
+
+/// What `grants::load` returns: a valid set, or the list of problems that made it invalid.
+pub type Grants = Result<GrantSet, Vec<GrantProblem>>;
+
+/// The audit writer behind one lock, so a step that fails to record refuses every later step.
+pub trait AuditSink: Send {
+    fn append(&mut self, time: Timestamp, event: AuditEvent) -> Result<u64, AuditError>;
+}
+
+impl AuditSink for Writer {
+    fn append(&mut self, time: Timestamp, event: AuditEvent) -> Result<u64, AuditError> {
+        Writer::append(self, time, event)
+    }
+}
+
+/// A call the broker is finishing: the session, the call, the decision it was approved under, and
+/// the label the result carries.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct Call {
+    pub session: SessionId,
+    pub call: CallId,
+    pub decision: u64,
+    pub label: Label,
+}
+
+impl Call {
+    /// The call and label of an approved decision, with the seq it was approved under.
+    pub fn of(decision: &crate::policy::Decision, seq: u64) -> Call {
+        let request = decision.request();
+        Call {
+            session: request.session.clone(),
+            call: request.call,
+            decision: seq,
+            label: decision.label(),
+        }
+    }
+}
+
+/// How a call was decided.
+#[derive(Debug)]
+pub enum Decided {
+    Allowed {
+        decision: crate::policy::Decision,
+        seq: u64,
+    },
+    Ask {
+        ask: Ask,
+        seq: u64,
+        state: SessionState,
+    },
+    Denied(DenyReason),
+}
+
+/// The value a decided call carries past the record, or the denial reason when there is none.
+enum RecordPayload {
+    Allowed(crate::policy::Decision),
+    Ask(Ask),
+    Denial(Denial),
+    Simple(DenyReason),
+}
+
+/// An owner's answer to a pending call.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum Answer {
+    Approved {
+        by: Option<String>,
+    },
+    Refused {
+        by: Option<String>,
+        reason: Option<String>,
+    },
+    Expired,
+}
+
+/// What an approval leaves: the verdict (run or deny) and the record written.
+#[derive(Debug)]
+pub struct Answered {
+    pub verdict: Verdict,
+    pub outcome: DecisionRecord,
+}
+
+struct Inner {
+    audit: Box<dyn AuditSink>,
+    state: StateStore,
+    stopped: bool,
+}
+
+/// The audit writer and every session's state behind one lock.
+pub struct Ledger {
+    inner: Mutex<Inner>,
+    log: Box<dyn Fn(&str) + Send + Sync>,
+}
+
+impl Ledger {
+    /// A ledger over the audit writer, the session store, and a sink for its notices.
+    pub fn new(
+        audit: Box<dyn AuditSink>,
+        state: StateStore,
+        log: Box<dyn Fn(&str) + Send + Sync>,
+    ) -> Ledger {
+        Ledger {
+            inner: Mutex::new(Inner {
+                audit,
+                state,
+                stopped: false,
+            }),
+            log,
+        }
+    }
+
+    /// The lock, `None` when the step cannot proceed: poisoned → `POISONED`, stopped → `STOPPED`.
+    fn lock(&self) -> Option<MutexGuard<'_, Inner>> {
+        let guard = match self.inner.lock() {
+            Ok(guard) => guard,
+            Err(_) => {
+                (self.log)(POISONED);
+                return None;
+            }
+        };
+        if guard.stopped {
+            (self.log)(STOPPED);
+            return None;
+        }
+        Some(guard)
+    }
+
+    /// Read a session's state, logging and returning "unknown" when it cannot be read.
+    fn state_read(
+        &self,
+        guard: &MutexGuard<Inner>,
+        session: &SessionId,
+    ) -> Result<SessionState, ()> {
+        match guard.state.read(session) {
+            Ok(state) => Ok(state),
+            Err(e) => {
+                (self.log)(&format!("brokerd: {e}"));
+                Err(())
+            }
+        }
+    }
+
+    /// Append a record, stopping the ledger on failure and logging why. `Ok(seq)` or `Err`.
+    fn append(
+        &self,
+        guard: &mut MutexGuard<Inner>,
+        now: Timestamp,
+        event: AuditEvent,
+    ) -> Result<u64, ()> {
+        match guard.audit.append(now, event) {
+            Ok(seq) => Ok(seq),
+            Err(e) => {
+                guard.stopped = true;
+                (self.log)(&format!("brokerd: {e}"));
+                Err(())
+            }
+        }
+    }
+
+    /// The refusal `finish` returns when the result could not be recorded.
+    fn not_recorded() -> ToolResponse {
+        ToolResponse::Failed {
+            message: NOT_RECORDED.to_string(),
+        }
+    }
+
+    /// The answer when the audit is unavailable: deny the verdict and record the denial.
+    fn audit_unavailable() -> Answered {
+        Answered {
+            verdict: Verdict::Denied(DenyReason::AuditUnavailable),
+            outcome: DecisionRecord::Denied {
+                reason: DenyReason::AuditUnavailable,
+            },
+        }
+    }
+
+    /// A denial verdict and its matching record, for one reason.
+    fn denied(reason: DenyReason) -> (Verdict, DecisionRecord) {
+        (Verdict::Denied(reason), DecisionRecord::Denied { reason })
+    }
+
+    /// The outcome of an approved re-decision: the named grant's mode, else allowed.
+    fn outcome_for(&self, decision: &crate::policy::Decision, grants: &Grants) -> DecisionRecord {
+        let mode = match grants {
+            Ok(set) => set
+                .grants()
+                .iter()
+                .find(|g| g.id == decision.grant())
+                .map(|g| g.grant.mode),
+            Err(_) => None,
+        };
+        match mode {
+            Some(Mode::Ask) => DecisionRecord::Ask {},
+            _ => DecisionRecord::Allowed {},
+        }
+    }
+
+    /// Decide and record a call.
+    pub fn decide(&self, request: ToolRequest, grants: &Grants, now: Timestamp) -> Decided {
+        let session = request.session.clone();
+        let call = request.call;
+        let tool = request.tool.clone();
+        let arguments = request.arguments.clone();
+
+        // 1. Lock fails → denied, nothing written.
+        let mut guard = match self.lock() {
+            Some(guard) => guard,
+            None => return Decided::Denied(DenyReason::AuditUnavailable),
+        };
+
+        // 2. Read the state, keeping the Result. The recorded state is "unknown" when unreadable.
+        let state = self.state_read(&guard, &session);
+        let recorded = match &state {
+            Ok(state) => *state,
+            Err(()) => SessionState {
+                taint: DataClass::Secret,
+                untrusted: true,
+            },
+        };
+
+        // 3. The outcome, first that applies.
+        let (outcome, grant, grant_sha256, payload) = match grants {
+            Err(_) => (
+                DecisionRecord::Denied {
+                    reason: DenyReason::GrantsInvalid,
+                },
+                None,
+                None,
+                RecordPayload::Simple(DenyReason::GrantsInvalid),
+            ),
+            Ok(set) => match state {
+                Err(()) => (
+                    DecisionRecord::Denied {
+                        reason: DenyReason::StateUnreadable,
+                    },
+                    None,
+                    None,
+                    RecordPayload::Simple(DenyReason::StateUnreadable),
+                ),
+                Ok(state) => match crate::policy::decide(request, set, state, now) {
+                    crate::policy::Outcome::Allowed(decision) => (
+                        DecisionRecord::Allowed {},
+                        Some(decision.grant().to_string()),
+                        Some(decision.grant_sha256()),
+                        RecordPayload::Allowed(decision),
+                    ),
+                    crate::policy::Outcome::Ask(ask) => (
+                        DecisionRecord::Ask {},
+                        Some(ask.grant().to_string()),
+                        Some(ask.grant_sha256()),
+                        RecordPayload::Ask(ask),
+                    ),
+                    crate::policy::Outcome::Denied(denial) => (
+                        DecisionRecord::Denied {
+                            reason: denial.reason,
+                        },
+                        denial.grant.clone(),
+                        denial.grant_sha256,
+                        RecordPayload::Denial(denial),
+                    ),
+                },
+            },
+        };
+
+        // 4. Write the record, then return the verdict.
+        let event = AuditEvent::Decision {
+            session,
+            call,
+            tool,
+            arguments,
+            outcome,
+            grant,
+            grant_sha256,
+            taint: recorded.taint,
+            untrusted: recorded.untrusted,
+        };
+        match self.append(&mut guard, now, event) {
+            Ok(seq) => match payload {
+                RecordPayload::Simple(reason) => Decided::Denied(reason),
+                RecordPayload::Allowed(decision) => Decided::Allowed { decision, seq },
+                RecordPayload::Ask(ask) => Decided::Ask {
+                    ask,
+                    seq,
+                    state: recorded,
+                },
+                RecordPayload::Denial(denial) => Decided::Denied(denial.reason),
+            },
+            Err(()) => Decided::Denied(DenyReason::AuditUnavailable),
+        }
+    }
+
+    /// Re-decide an approval and record it.
+    pub fn answer(
+        &self,
+        ask: Ask,
+        decision: u64,
+        answer: Answer,
+        grants: &Grants,
+        now: Timestamp,
+    ) -> Answered {
+        // 1. Lock fails → step fails.
+        let mut guard = match self.lock() {
+            Some(guard) => guard,
+            None => return Self::audit_unavailable(),
+        };
+
+        // 2. Read the state of the ask's session.
+        let session = ask.request().session.clone();
+        let call = ask.request().call;
+        let state = self.state_read(&guard, &session);
+        let recorded = match &state {
+            Ok(state) => *state,
+            Err(()) => SessionState {
+                taint: DataClass::Secret,
+                untrusted: true,
+            },
+        };
+
+        // 3. The answer and its fields.
+        let (answer, by, reason) = match &answer {
+            Answer::Approved { by } => (ApprovalAnswer::Approved, by.clone(), None),
+            Answer::Refused { by, reason } => (ApprovalAnswer::Refused, by.clone(), reason.clone()),
+            Answer::Expired => (ApprovalAnswer::Expired, None, None),
+        };
+
+        // 4. The re-decision (approved only); every other answer is a denial.
+        let redecide = match (answer, grants) {
+            (ApprovalAnswer::Approved, Ok(set)) => match &state {
+                Ok(state) => crate::policy::redecide(ask, set, *state, now),
+                Err(()) => Err(Denial::new(DenyReason::StateUnreadable)),
+            },
+            _ => Err(Denial::new(DenyReason::GrantsInvalid)),
+        };
+
+        // 5. The grant fields, then the verdict and outcome together.
+        let (grant, grant_sha256) = match &redecide {
+            Ok(decision) => (
+                Some(decision.grant().to_string()),
+                Some(decision.grant_sha256()),
+            ),
+            Err(_) => (None, None),
+        };
+        let (verdict, outcome) = match (answer, redecide) {
+            (ApprovalAnswer::Approved, Ok(decision)) => {
+                let outcome = self.outcome_for(&decision, grants);
+                (Verdict::Run(Box::new(decision)), outcome)
+            }
+            (ApprovalAnswer::Approved, Err(denial)) => Self::denied(denial.reason),
+            (ApprovalAnswer::Refused, _) => Self::denied(DenyReason::ApprovalRefused),
+            (ApprovalAnswer::Expired, _) => Self::denied(DenyReason::ApprovalExpired),
+        };
+
+        // 6. Record the approval, or step fails.
+        match self.append(
+            &mut guard,
+            now,
+            AuditEvent::Approval {
+                session,
+                call,
+                decision,
+                answer,
+                by,
+                post: None,
+                reason,
+                outcome: outcome.clone(),
+                grant,
+                grant_sha256,
+                taint: recorded.taint,
+                untrusted: recorded.untrusted,
+            },
+        ) {
+            Ok(_) => Answered { verdict, outcome },
+            Err(()) => Self::audit_unavailable(),
+        }
+    }
+
+    /// Raise the state, hash the response, and record a result, or stop the ledger on failure.
+    fn record_result(
+        &self,
+        guard: &mut MutexGuard<'_, Inner>,
+        call: &Call,
+        now: Timestamp,
+        status: proto::ResultStatus,
+        taint_after: DataClass,
+        response: &ToolResponse,
+    ) -> ToolResponse {
+        let (sha256, bytes) = match response {
+            ToolResponse::Result { content, .. } => match proto::sha256(content.as_bytes()) {
+                Ok(sha) => (sha, u64::try_from(content.len()).unwrap_or(u64::MAX)),
+                Err(_) => return Self::not_recorded(),
+            },
+            ToolResponse::Failed { message } => match proto::sha256(message.as_bytes()) {
+                Ok(sha) => (sha, u64::try_from(message.len()).unwrap_or(u64::MAX)),
+                Err(_) => return Self::not_recorded(),
+            },
+            _ => return Self::not_recorded(),
+        };
+        let (class, untrusted, truncated) = match response {
+            ToolResponse::Result {
+                class,
+                untrusted,
+                truncated,
+                ..
+            } => (*class, *untrusted, *truncated),
+            ToolResponse::Failed { .. } => (call.label.class, call.label.untrusted, false),
+            _ => return Self::not_recorded(),
+        };
+        let event = AuditEvent::Result {
+            session: call.session.clone(),
+            call: call.call,
+            decision: call.decision,
+            status,
+            class,
+            untrusted,
+            truncated,
+            bytes,
+            sha256,
+            taint_after,
+        };
+        match self.append(guard, now, event) {
+            Ok(_) => response.clone(),
+            Err(()) => Self::not_recorded(),
+        }
+    }
+
+    /// Raise the state and record a result.
+    pub fn finish(&self, call: &Call, response: ToolResponse, now: Timestamp) -> ToolResponse {
+        // 1. Lock fails → cannot record.
+        let mut guard = match self.lock() {
+            Some(guard) => guard,
+            None => return Self::not_recorded(),
+        };
+
+        match &response {
+            // 3. A result raises the state, then is recorded.
+            ToolResponse::Result { .. } => {
+                let raised = match self.state_read(&guard, &call.session) {
+                    Ok(state) => match guard.state.raise(&call.session, state, call.label) {
+                        Ok(raised) => raised,
+                        Err(e) => {
+                            (self.log)(&format!("brokerd: {e}"));
+                            return Self::not_recorded();
+                        }
+                    },
+                    Err(()) => return Self::not_recorded(),
+                };
+                self.record_result(
+                    &mut guard,
+                    call,
+                    now,
+                    proto::ResultStatus::Result,
+                    raised.taint,
+                    &response,
+                )
+            }
+            // 4. A failure changes no state, recorded by its message.
+            ToolResponse::Failed { .. } => {
+                let taint_after = match self.state_read(&guard, &call.session) {
+                    Ok(state) => state.taint,
+                    Err(()) => DataClass::Secret,
+                };
+                self.record_result(
+                    &mut guard,
+                    call,
+                    now,
+                    proto::ResultStatus::Failed,
+                    taint_after,
+                    &response,
+                )
+            }
+            // 5. The runner never returns these; nothing to record.
+            ToolResponse::Denied { .. } | ToolResponse::PendingApproval { .. } => {
+                Self::not_recorded()
+            }
+        }
+    }
+}

crates/brokerd/src/lib.rs

@@ -5,6 +5,7 @@ pub mod args;
 pub mod audit;
 pub mod config;
 pub mod grants;
+pub mod ledger;
 pub mod policy;
 pub mod runner;
 pub mod state;

crates/brokerd/tests/ledger.rs

@@ -0,0 +1,383 @@
+//! The ledger's first and third steps: decide and record; raise the state and record the result.
+//! And what happens after a failed append or a panic. Do not edit.
+
+#[path = "support/build.rs"]
+mod build;
+#[path = "support/rig.rs"]
+mod rig;
+#[path = "support/sink.rs"]
+mod sink;
+#[path = "support/tmp.rs"]
+mod tmp;
+
+use std::os::unix::fs::PermissionsExt;
+use std::panic::AssertUnwindSafe;
+
+use brokerd::ledger::{Call, Decided, Grants, NOT_RECORDED, POISONED, STOPPED};
+use brokerd::policy::{Label, SessionState};
+use build::{grant, now, read, set};
+use proto::{
+    AuditEvent, CallId, DataClass, DecisionRecord, DenyReason, GrantProblem, Mode, ResultStatus,
+    SessionId, ToolResponse,
+};
+use rig::Rig;
+
+fn one(grants: Vec<build::Build>) -> Grants {
+    Ok(set(grants))
+}
+
+fn notes(mode: Mode) -> Grants {
+    one(vec![grant("n", "read_file", mode).paths(&["/n"])])
+}
+
+fn call(label: Label) -> Call {
+    Call {
+        session: SessionId::new("s1").unwrap(),
+        call: CallId(1),
+        decision: 0,
+        label,
+    }
+}
+
+fn secret() -> Label {
+    Label {
+        class: DataClass::Secret,
+        untrusted: true,
+    }
+}
+
+fn result(content: &str, label: Label) -> ToolResponse {
+    ToolResponse::Result {
+        content: content.to_string(),
+        class: label.class,
+        untrusted: label.untrusted,
+        truncated: false,
+    }
+}
+
+#[test]
+fn an_allowed_call_is_recorded_in_full() {
+    let rig = Rig::new("ledger-allowed");
+    let ledger = rig.ledger();
+    let request = read("/n/a");
+    let arguments = request.arguments.clone();
+    match ledger.decide(request, &notes(Mode::Auto), now()) {
+        Decided::Allowed { decision, seq } => {
+            assert_eq!(seq, 0);
+            assert_eq!(decision.grant(), "n");
+        }
+        other => panic!("{other:?}"),
+    }
+    let records = rig.records();
+    assert_eq!(records.len(), 1);
+    assert_eq!(records[0].time, now());
+    assert_eq!(
+        records[0].event,
+        AuditEvent::Decision {
+            session: SessionId::new("s1").unwrap(),
+            call: CallId(1),
+            tool: "read_file".to_string(),
+            arguments,
+            outcome: DecisionRecord::Allowed {},
+            grant: Some("n".to_string()),
+            grant_sha256: Some(proto::sha256(b"n").unwrap()),
+            taint: DataClass::Private,
+            untrusted: false,
+        }
+    );
+}
+
+#[test]
+fn an_ask_is_recorded_as_ask_with_the_state_it_was_decided_at() {
+    let rig = Rig::new("ledger-ask");
+    let ledger = rig.ledger();
+    match ledger.decide(read("/n/a"), &notes(Mode::Ask), now()) {
+        Decided::Ask { ask, seq, state } => {
+            assert_eq!((seq, ask.grant()), (0, "n"));
+            assert_eq!(state, SessionState::default());
+        }
+        other => panic!("{other:?}"),
+    }
+    match &rig.events()[0] {
+        AuditEvent::Decision { outcome, grant, .. } => {
+            assert_eq!(*outcome, DecisionRecord::Ask {});
+            assert_eq!(grant.as_deref(), Some("n"));
+        }
+        other => panic!("{other:?}"),
+    }
+}
+
+#[test]
+fn a_denial_names_a_grant_only_when_a_grant_denied() {
+    let rig = Rig::new("ledger-denied");
+    let ledger = rig.ledger();
+    let none = ledger.decide(read("/n/a"), &one(Vec::new()), now());
+    assert!(
+        matches!(none, Decided::Denied(DenyReason::NoGrant)),
+        "{none:?}"
+    );
+    let deny = one(vec![grant("d", "read_file", Mode::Deny).paths(&["/n"])]);
+    let by = ledger.decide(read("/n/a"), &deny, now());
+    assert!(
+        matches!(by, Decided::Denied(DenyReason::DeniedByGrant)),
+        "{by:?}"
+    );
+    let events = rig.events();
+    let named: Vec<(DecisionRecord, Option<String>)> = events
+        .iter()
+        .map(|e| match e {
+            AuditEvent::Decision { outcome, grant, .. } => (outcome.clone(), grant.clone()),
+            other => panic!("{other:?}"),
+        })
+        .collect();
+    assert_eq!(
+        named,
+        [
+            (
+                DecisionRecord::Denied {
+                    reason: DenyReason::NoGrant
+                },
+                None
+            ),
+            (
+                DecisionRecord::Denied {
+                    reason: DenyReason::DeniedByGrant
+                },
+                Some("d".to_string())
+            ),
+        ]
+    );
+}
+
+#[test]
+fn invalid_grants_come_first_then_an_unreadable_state() {
+    let rig = Rig::new("ledger-order");
+    let ledger = rig.ledger();
+    std::fs::create_dir_all(rig.cfg.state_dir()).unwrap();
+    std::fs::write(rig.state_file("s1"), "not json").unwrap();
+    let invalid: Grants = Err(vec![GrantProblem {
+        file: "x.toml".to_string(),
+        line: None,
+        problem: "bad".to_string(),
+    }]);
+    let first = ledger.decide(read("/n/a"), &invalid, now());
+    assert!(
+        matches!(first, Decided::Denied(DenyReason::GrantsInvalid)),
+        "{first:?}"
+    );
+    let second = ledger.decide(read("/n/a"), &notes(Mode::Auto), now());
+    assert!(
+        matches!(second, Decided::Denied(DenyReason::StateUnreadable)),
+        "{second:?}"
+    );
+    // A state brokerd cannot read is recorded as the most sensitive one.
+    for event in rig.events() {
+        match event {
+            AuditEvent::Decision {
+                taint, untrusted, ..
+            } => assert_eq!((taint, untrusted), (DataClass::Secret, true)),
+            other => panic!("{other:?}"),
+        }
+    }
+    let damaged = rig.lines.with("see docs/runbook.md#broker-state-damaged");
+    assert!(!damaged.is_empty(), "{:?}", rig.lines.all());
+}
+
+#[test]
+fn a_failed_append_denies_this_call_and_every_later_one() {
+    let rig = Rig::new("ledger-stop");
+    let ledger = rig.ledger();
+    rig.switch.fail(true);
+    let first = ledger.decide(read("/n/a"), &notes(Mode::Auto), now());
+    assert!(
+        matches!(first, Decided::Denied(DenyReason::AuditUnavailable)),
+        "{first:?}"
+    );
+    assert_eq!(rig.switch.attempts(), 1);
+    // The sink would work again; the ledger does not try it.
+    rig.switch.fail(false);
+    let later = ledger.decide(read("/n/a"), &notes(Mode::Auto), now());
+    assert!(
+        matches!(later, Decided::Denied(DenyReason::AuditUnavailable)),
+        "{later:?}"
+    );
+    let finished = ledger.finish(&call(secret()), result("x", secret()), now());
+    assert_eq!(
+        finished,
+        ToolResponse::Failed {
+            message: NOT_RECORDED.to_string()
+        }
+    );
+    assert_eq!(rig.switch.attempts(), 1, "nothing more was written");
+    assert!(rig.records().is_empty());
+    assert!(!rig.state_file("s1").exists(), "the state was not raised");
+    assert!(!rig.lines.with(STOPPED).is_empty());
+    assert!(
+        rig.lines
+            .all()
+            .iter()
+            .all(|l| !l.contains("runbook") || l.ends_with("see docs/runbook.md#audit-unavailable")),
+        "{:?}",
+        rig.lines.all()
+    );
+}
+
+#[test]
+fn a_panic_while_holding_the_ledger_denies_every_later_call() {
+    let rig = Rig::new("ledger-poison");
+    let ledger = rig.ledger();
+    rig.switch.panic_next();
+    let panicked = std::panic::catch_unwind(AssertUnwindSafe(|| {
+        ledger.decide(read("/n/a"), &notes(Mode::Auto), now())
+    }));
+    assert!(panicked.is_err());
+    let later = ledger.decide(read("/n/a"), &notes(Mode::Auto), now());
+    assert!(
+        matches!(later, Decided::Denied(DenyReason::AuditUnavailable)),
+        "{later:?}"
+    );
+    assert_eq!(rig.switch.attempts(), 1);
+    assert!(
+        !rig.lines.with(POISONED).is_empty(),
+        "{:?}",
+        rig.lines.all()
+    );
+    assert!(POISONED.ends_with("see docs/runbook.md#audit-unavailable"));
+}
+
+#[test]
+fn a_result_raises_the_state_then_is_recorded() {
+    let rig = Rig::new("ledger-result");
+    let ledger = rig.ledger();
+    let answer = ledger.finish(&call(secret()), result("key", secret()), now());
+    assert_eq!(answer, result("key", secret()), "passed on unchanged");
+    let state = rig.state().read(&SessionId::new("s1").unwrap()).unwrap();
+    assert_eq!(
+        state,
+        SessionState {
+            taint: DataClass::Secret,
+            untrusted: true
+        }
+    );
+    assert_eq!(
+        rig.events(),
+        [AuditEvent::Result {
+            session: SessionId::new("s1").unwrap(),
+            call: CallId(1),
+            decision: 0,
+            status: ResultStatus::Result,
+            class: DataClass::Secret,
+            untrusted: true,
+            truncated: false,
+            bytes: 3,
+            sha256: proto::sha256(b"key").unwrap(),
+            taint_after: DataClass::Secret,
+        }]
+    );
+}
+
+#[test]
+fn taint_never_goes_down() {
+    let rig = Rig::new("ledger-down");
+    let ledger = rig.ledger();
+    let public = Label {
+        class: DataClass::Public,
+        untrusted: false,
+    };
+    ledger.finish(&call(secret()), result("a", secret()), now());
+    ledger.finish(&call(public), result("b", public), now());
+    let after: Vec<DataClass> = rig
+        .events()
+        .iter()
+        .map(|e| match e {
+            AuditEvent::Result { taint_after, .. } => *taint_after,
+            other => panic!("{other:?}"),
+        })
+        .collect();
+    assert_eq!(after, [DataClass::Secret, DataClass::Secret]);
+    let state = rig.state().read(&SessionId::new("s1").unwrap()).unwrap();
+    assert_eq!((state.taint, state.untrusted), (DataClass::Secret, true));
+}
+
+#[test]
+fn a_failure_changes_no_state_and_is_recorded_by_its_message() {
+    let rig = Rig::new("ledger-failed");
+    let ledger = rig.ledger();
+    let failed = ToolResponse::Failed {
+        message: "the tool timed out".to_string(),
+    };
+    assert_eq!(
+        ledger.finish(&call(secret()), failed.clone(), now()),
+        failed
+    );
+    assert!(!rig.state_file("s1").exists());
+    match &rig.events()[0] {
+        AuditEvent::Result {
+            status,
+            class,
+            bytes,
+            sha256,
+            taint_after,
+            ..
+        } => {
+            assert_eq!(*status, ResultStatus::Failed);
+            assert_eq!(
+                *class,
+                DataClass::Secret,
+                "the label the result would have had"
+            );
+            assert_eq!(*bytes, 18);
+            assert_eq!(*sha256, proto::sha256(b"the tool timed out").unwrap());
+            assert_eq!(*taint_after, DataClass::Private, "the taint before");
+        }
+        other => panic!("{other:?}"),
+    }
+}
+
+#[test]
+fn a_state_that_cannot_be_written_withholds_the_content() {
+    if tmp::running_as_root("a_state_that_cannot_be_written_withholds_the_content") {
+        return;
+    }
+    let rig = Rig::new("ledger-ro");
+    let ledger = rig.ledger();
+    let dir = rig.cfg.state_dir();
+    std::fs::create_dir_all(&dir).unwrap();
+    std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o500)).unwrap();
+    let answer = ledger.finish(&call(secret()), result("key", secret()), now());
+    std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o700)).unwrap();
+    assert_eq!(
+        answer,
+        ToolResponse::Failed {
+            message: NOT_RECORDED.to_string()
+        }
+    );
+    assert!(
+        rig.records().is_empty(),
+        "no Result for a state not on disk"
+    );
+    assert!(
+        !rig.lines
+            .with("see docs/runbook.md#broker-state-damaged")
+            .is_empty()
+    );
+}
+
+#[test]
+fn a_result_record_that_cannot_be_written_withholds_the_content() {
+    let rig = Rig::new("ledger-norecord");
+    let ledger = rig.ledger();
+    rig.switch.fail(true);
+    let answer = ledger.finish(&call(secret()), result("key", secret()), now());
+    assert_eq!(
+        answer,
+        ToolResponse::Failed {
+            message: NOT_RECORDED.to_string()
+        }
+    );
+    assert!(
+        !rig.lines
+            .with("see docs/runbook.md#audit-unavailable")
+            .is_empty()
+    );
+}

crates/brokerd/tests/ledger_answer.rs

@@ -0,0 +1,262 @@
+//! The ledger's second step: an approval decides again and is recorded; a refusal and an expiry
+//! are recorded as denials. Do not edit.
+
+#[path = "support/build.rs"]
+mod build;
+#[path = "support/rig.rs"]
+mod rig;
+#[path = "support/sink.rs"]
+mod sink;
+#[path = "support/tmp.rs"]
+mod tmp;
+
+use brokerd::approvals::Verdict;
+use brokerd::ledger::{Answer, Answered, Decided, Grants, Ledger};
+use brokerd::policy::{Ask, Label};
+use build::{grant, now, read, set};
+use proto::{
+    ApprovalAnswer, AuditEvent, CallId, DataClass, DecisionRecord, DenyReason, Mode, SessionId,
+};
+use rig::Rig;
+
+fn asking() -> Grants {
+    Ok(set(vec![grant("n", "read_file", Mode::Ask).paths(&["/n"])]))
+}
+
+fn pending(ledger: &Ledger) -> Ask {
+    match ledger.decide(read("/n/deep/a"), &asking(), now()) {
+        Decided::Ask { ask, seq: 0, .. } => ask,
+        other => panic!("{other:?}"),
+    }
+}
+
+fn bxctl() -> Answer {
+    Answer::Approved {
+        by: Some("bxctl".to_string()),
+    }
+}
+
+fn denied(reason: DenyReason) -> DecisionRecord {
+    DecisionRecord::Denied { reason }
+}
+
+fn verdict_reason(answered: &Answered) -> Option<DenyReason> {
+    match &answered.verdict {
+        Verdict::Run(_) => None,
+        Verdict::Denied(reason) => Some(*reason),
+    }
+}
+
+/// The one `Approval` record, as (answer, by, reason, outcome, grant).
+type Row = (
+    ApprovalAnswer,
+    Option<String>,
+    Option<String>,
+    DecisionRecord,
+    Option<String>,
+);
+
+fn approval(rig: &Rig) -> Row {
+    let events = rig.events();
+    assert_eq!(events.len(), 2, "a decision and one approval: {events:?}");
+    match &events[1] {
+        AuditEvent::Approval {
+            session,
+            call,
+            decision,
+            answer,
+            by,
+            post,
+            reason,
+            outcome,
+            grant,
+            ..
+        } => {
+            assert_eq!(session, &SessionId::new("s1").unwrap());
+            assert_eq!((*call, *decision, post), (CallId(1), 0, &None));
+            (
+                *answer,
+                by.clone(),
+                reason.clone(),
+                outcome.clone(),
+                grant.clone(),
+            )
+        }
+        other => panic!("{other:?}"),
+    }
+}
+
+#[test]
+fn an_approval_that_still_asks_lets_the_call_run() {
+    let rig = Rig::new("answer-ask");
+    let ledger = rig.ledger();
+    let ask = pending(&ledger);
+    let answered = ledger.answer(ask, 0, bxctl(), &asking(), now());
+    match &answered.verdict {
+        Verdict::Run(decision) => assert_eq!(decision.grant(), "n"),
+        other => panic!("{other:?}"),
+    }
+    assert_eq!(answered.outcome, DecisionRecord::Ask {});
+    let row = approval(&rig);
+    assert_eq!(
+        row,
+        (
+            ApprovalAnswer::Approved,
+            Some("bxctl".to_string()),
+            None,
+            DecisionRecord::Ask {},
+            Some("n".to_string())
+        )
+    );
+}
+
+#[test]
+fn an_approval_under_a_grant_that_is_now_auto_records_allowed() {
+    let rig = Rig::new("answer-auto");
+    let ledger = rig.ledger();
+    let ask = pending(&ledger);
+    let now_auto = Ok(set(vec![
+        grant("n", "read_file", Mode::Auto).paths(&["/n"]),
+    ]));
+    let answered = ledger.answer(ask, 0, bxctl(), &now_auto, now());
+    assert!(matches!(answered.verdict, Verdict::Run(_)));
+    assert_eq!(answered.outcome, DecisionRecord::Allowed {});
+}
+
+#[test]
+fn the_approval_names_the_grant_matched_now() {
+    let rig = Rig::new("answer-grant");
+    let ledger = rig.ledger();
+    let ask = pending(&ledger);
+    let wider = Ok(set(vec![
+        grant("n", "read_file", Mode::Ask).paths(&["/n"]),
+        grant("z-deep", "read_file", Mode::Ask).paths(&["/n/deep"]),
+    ]));
+    let answered = ledger.answer(ask, 0, bxctl(), &wider, now());
+    match &answered.verdict {
+        Verdict::Run(decision) => assert_eq!(decision.grant(), "z-deep"),
+        other => panic!("{other:?}"),
+    }
+    assert_eq!(approval(&rig).4.as_deref(), Some("z-deep"));
+}
+
+#[test]
+fn an_approval_after_the_grant_is_gone_is_denied() {
+    let rig = Rig::new("answer-gone");
+    let ledger = rig.ledger();
+    let ask = pending(&ledger);
+    let answered = ledger.answer(ask, 0, bxctl(), &Ok(set(Vec::new())), now());
+    assert_eq!(verdict_reason(&answered), Some(DenyReason::NoGrant));
+    assert_eq!(answered.outcome, denied(DenyReason::NoGrant));
+    let row = approval(&rig);
+    assert_eq!(
+        (row.0, row.3, row.4),
+        (ApprovalAnswer::Approved, denied(DenyReason::NoGrant), None)
+    );
+}
+
+#[test]
+fn an_approval_after_the_taint_rose_is_denied() {
+    let rig = Rig::new("answer-taint");
+    let ledger = rig.ledger();
+    let low = || {
+        Ok(set(vec![
+            grant("n", "read_file", Mode::Ask)
+                .paths(&["/n"])
+                .max_taint(DataClass::Private),
+        ]))
+    };
+    let ask = match ledger.decide(read("/n/a"), &low(), now()) {
+        Decided::Ask { ask, .. } => ask,
+        other => panic!("{other:?}"),
+    };
+    // Another call of the session read a secret while this one waited.
+    let s1 = SessionId::new("s1").unwrap();
+    let label = Label {
+        class: DataClass::Secret,
+        untrusted: false,
+    };
+    rig.state()
+        .raise(&s1, rig.state().read(&s1).unwrap(), label)
+        .unwrap();
+    let answered = ledger.answer(ask, 0, bxctl(), &low(), now());
+    assert_eq!(verdict_reason(&answered), Some(DenyReason::TaintTooHigh));
+    match &rig.events()[1] {
+        AuditEvent::Approval { taint, outcome, .. } => {
+            assert_eq!(*taint, DataClass::Secret, "the state at the re-decision");
+            assert_eq!(*outcome, denied(DenyReason::TaintTooHigh));
+        }
+        other => panic!("{other:?}"),
+    }
+}
+
+#[test]
+fn an_approval_with_invalid_grants_is_denied() {
+    let rig = Rig::new("answer-invalid");
+    let ledger = rig.ledger();
+    let ask = pending(&ledger);
+    let answered = ledger.answer(ask, 0, bxctl(), &Err(Vec::new()), now());
+    assert_eq!(verdict_reason(&answered), Some(DenyReason::GrantsInvalid));
+}
+
+#[test]
+fn a_refusal_is_recorded_with_the_owners_reason() {
+    let rig = Rig::new("answer-refuse");
+    let ledger = rig.ledger();
+    let ask = pending(&ledger);
+    let refused = Answer::Refused {
+        by: Some("bxctl".to_string()),
+        reason: Some("not now".to_string()),
+    };
+    let answered = ledger.answer(ask, 0, refused, &asking(), now());
+    assert_eq!(verdict_reason(&answered), Some(DenyReason::ApprovalRefused));
+    assert_eq!(
+        approval(&rig),
+        (
+            ApprovalAnswer::Refused,
+            Some("bxctl".to_string()),
+            Some("not now".to_string()),
+            denied(DenyReason::ApprovalRefused),
+            None
+        )
+    );
+}
+
+#[test]
+fn an_expiry_is_recorded_by_nobody() {
+    let rig = Rig::new("answer-expire");
+    let ledger = rig.ledger();
+    let ask = pending(&ledger);
+    let answered = ledger.answer(ask, 0, Answer::Expired, &asking(), now());
+    assert_eq!(verdict_reason(&answered), Some(DenyReason::ApprovalExpired));
+    assert_eq!(
+        approval(&rig),
+        (
+            ApprovalAnswer::Expired,
+            None,
+            None,
+            denied(DenyReason::ApprovalExpired),
+            None
+        )
+    );
+}
+
+#[test]
+fn an_approval_that_cannot_be_recorded_does_not_run() {
+    let rig = Rig::new("answer-norecord");
+    let ledger = rig.ledger();
+    let ask = pending(&ledger);
+    rig.switch.fail(true);
+    let answered = ledger.answer(ask, 0, bxctl(), &asking(), now());
+    assert_eq!(
+        verdict_reason(&answered),
+        Some(DenyReason::AuditUnavailable)
+    );
+    assert_eq!(answered.outcome, denied(DenyReason::AuditUnavailable));
+    assert_eq!(rig.events().len(), 1, "only the decision");
+    assert!(
+        !rig.lines
+            .with("see docs/runbook.md#audit-unavailable")
+            .is_empty()
+    );
+}

crates/brokerd/tests/support/rig.rs

@@ -0,0 +1,118 @@
+//! A home for ledger and broker tests: grants, audit log, session state, a flaky audit sink and
+//! a log to read. Do not edit.
+//!
+//! Included with `#[path = "support/rig.rs"] mod rig;`, beside `tmp` and `sink`. The broker
+//! tests add `client`.
+
+#![allow(dead_code)] // each test file uses a different part of this module
+
+use std::path::PathBuf;
+
+use brokerd::audit::Writer;
+use brokerd::config::{Approvals, Config, Paths, Sockets};
+use brokerd::ledger::Ledger;
+use brokerd::state::StateStore;
+use proto::{AuditEvent, AuditRecord, CallId, SessionId, ToolRequest};
+
+use crate::sink::{Flaky, Lines, Switch};
+use crate::tmp::TempDir;
+
+pub struct Rig {
+    pub dir: TempDir,
+    pub cfg: Config,
+    pub switch: Switch,
+    pub lines: Lines,
+}
+
+impl Rig {
+    pub fn new(tag: &str) -> Rig {
+        Rig::with_ttl(tag, 900_000)
+    }
+
+    pub fn with_ttl(tag: &str, ttl_ms: u64) -> Rig {
+        let dir = TempDir::new(tag);
+        let grants = dir.path().join("grants");
+        std::fs::create_dir_all(&grants).unwrap();
+        let cfg = Config {
+            paths: Paths {
+                home: dir.path().to_path_buf(),
+                grants,
+            },
+            sockets: Sockets::default(),
+            approvals: Approvals { ttl_ms },
+        };
+        Rig {
+            dir,
+            cfg,
+            switch: Switch::default(),
+            lines: Lines::default(),
+        }
+    }
+
+    pub fn state(&self) -> StateStore {
+        StateStore::new(&self.cfg.state_dir())
+    }
+
+    /// Opens the audit log (once: the writer holds its lock) behind the flaky sink.
+    pub fn ledger(&self) -> Ledger {
+        let opened = Writer::open(&self.cfg.audit_dir(), false).unwrap();
+        let sink = Flaky {
+            writer: opened.writer,
+            switch: self.switch.clone(),
+        };
+        Ledger::new(Box::new(sink), self.state(), self.lines.sink())
+    }
+
+    /// Writes `grants/<id>.toml`.
+    pub fn grant(&self, id: &str, text: &str) {
+        std::fs::write(self.cfg.paths.grants.join(format!("{id}.toml")), text).unwrap();
+    }
+
+    pub fn remove_grant(&self, id: &str) {
+        std::fs::remove_file(self.cfg.paths.grants.join(format!("{id}.toml"))).unwrap();
+    }
+
+    pub fn state_file(&self, session: &str) -> PathBuf {
+        self.cfg.state_dir().join(format!("{session}.json"))
+    }
+
+    /// Every record in the audit log, in order.
+    pub fn records(&self) -> Vec<AuditRecord> {
+        let dir = self.cfg.audit_dir();
+        let mut names: Vec<String> = std::fs::read_dir(&dir)
+            .unwrap()
+            .map(|e| e.unwrap().file_name().into_string().unwrap())
+            .filter(|n| n.ends_with(".jsonl"))
+            .collect();
+        names.sort();
+        let mut out = Vec::new();
+        for name in names {
+            let text = std::fs::read_to_string(dir.join(name)).unwrap();
+            for line in text.lines() {
+                out.push(serde_json::from_str(line).unwrap());
+            }
+        }
+        out
+    }
+
+    pub fn events(&self) -> Vec<AuditEvent> {
+        self.records().into_iter().map(|r| r.event).collect()
+    }
+}
+
+/// A grant file's text. `extra` goes before `[constraints]`, `constraints` after it.
+pub fn grant_text(tool: &str, mode: &str, extra: &str, constraints: &str) -> String {
+    format!(
+        "tool = \"{tool}\"\nmode = \"{mode}\"\nmax_taint = \"secret\"\nresult_class = \"private\"\n\
+         untrusted = false\n{extra}\n[constraints]\n{constraints}\n"
+    )
+}
+
+pub fn request(session: &str, call: u64, tool: &str, arguments: &str) -> ToolRequest {
+    ToolRequest {
+        session: SessionId::new(session).unwrap(),
+        call: CallId(call),
+        tool: tool.to_string(),
+        arguments: arguments.to_string(),
+    }
+}

crates/brokerd/tests/support/sink.rs

@@ -0,0 +1,78 @@
+//! An audit sink that fails on demand, and a log that tests can read. Do not edit.
+//!
+//! Included with `#[path = "support/sink.rs"] mod sink;`.
+
+#![allow(dead_code)] // each test file uses a different part of this module
+
+use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
+use std::sync::{Arc, Mutex};
+
+use brokerd::audit::{AuditError, Writer};
+use brokerd::ledger::AuditSink;
+use proto::{AuditEvent, Timestamp};
+
+/// Switches shared between a test and its `Flaky` sink.
+#[derive(Clone, Default)]
+pub struct Switch {
+    fail: Arc<AtomicBool>,
+    panic: Arc<AtomicBool>,
+    attempts: Arc<AtomicUsize>,
+}
+
+impl Switch {
+    /// Every append from now on fails, without writing anything.
+    pub fn fail(&self, on: bool) {
+        self.fail.store(on, Ordering::SeqCst);
+    }
+    /// The next append panics, as a bug part-way through a write would.
+    pub fn panic_next(&self) {
+        self.panic.store(true, Ordering::SeqCst);
+    }
+    /// How many appends the ledger has asked for.
+    pub fn attempts(&self) -> usize {
+        self.attempts.load(Ordering::SeqCst)
+    }
+}
+
+/// A real `Writer` behind a switch.
+pub struct Flaky {
+    pub writer: Writer,
+    pub switch: Switch,
+}
+
+impl AuditSink for Flaky {
+    fn append(&mut self, time: Timestamp, event: AuditEvent) -> Result<u64, AuditError> {
+        self.switch.attempts.fetch_add(1, Ordering::SeqCst);
+        if self.switch.panic.swap(false, Ordering::SeqCst) {
+            panic!("a bug part-way through a write");
+        }
+        if self.switch.fail.load(Ordering::SeqCst) {
+            return Err(AuditError::Io {
+                what: "cannot write to the test log".to_string(),
+                source: std::io::Error::other("the disk is full"),
+            });
+        }
+        self.writer.append(time, event)
+    }
+}
+
+/// Collects every line a ledger or broker prints.
+#[derive(Clone, Default)]
+pub struct Lines(Arc<Mutex<Vec<String>>>);
+
+impl Lines {
+    pub fn sink(&self) -> Box<dyn Fn(&str) + Send + Sync> {
+        let lines = Arc::clone(&self.0);
+        Box::new(move |line| lines.lock().unwrap().push(line.to_string()))
+    }
+    pub fn all(&self) -> Vec<String> {
+        self.0.lock().unwrap().clone()
+    }
+    /// The lines that hold `text`.
+    pub fn with(&self, text: &str) -> Vec<String> {
+        self.all()
+            .into_iter()
+            .filter(|l| l.contains(text))
+            .collect()
+    }
+}

docs/implementer-log.md

@@ -6,6 +6,7 @@ reviewer adds findings under "Reviews" once per milestone.
 
 | Task | Date | Status | Gate runs | First gate | Deviations | Notes | Model |
 |---|---|---|---|---|---|---|---|
+| M3a/12-brokerd-ledger | 2026-09-20 | done | 2 | fail | none | Wrote crates/brokerd/src/ledger.rs (499 lines): Ledger + Inner { audit, state, stopped } behind one Mutex, and the three steps that hold it. decide copies the request out, reads state then policy::decide, and records the outcome (allowed/ask/denied, grant fields set together) as AuditEvent::Decision; answer re-decides an approval (approved only) and records AuditEvent::Approval with the answer/by/reason; finish raises the state for a Result and records AuditEvent::Result by its message otherwise, returning response unchanged only once the raised taint and the record are both on disk. Helpers not_recorded/audit_unavailable/denied; every append Err sets stopped through the one append method, and finish logs the raise error "brokerd: {e}" before stopping. Step 5 verified: each numbered exit points at a line and every append Err goes through the one stopped place. Trimmed 588 to 499 by compressing the module doc; one clippy fix (needless `return` in the answer append match, which is the tail expression). 11 + 9 tests pass; `make gate` prints `gate: ok`. | ? |
 | M3a/08-brokerd-state | 2026-09-18 | done | 1 | pass | none | Wrote crates/brokerd/src/state.rs: RUNBOOK, StateError (Unreadable/Write with hand-written Display ending in RUNBOOK and std::error::Error), StateStore (new does not touch disk, path joins <dir>/<id>.json, read, raise) and the private StateFile with deny_unknown_fields. read has exactly one default path (ErrorKind::NotFound); Public taint is Unreadable; raise computes max(taint,label,Private) and ORs untrusted, always writes atomically in six steps mapping any error to Write(path, err). `cargo fmt` put `state` after `runner` in lib.rs. 9 tests pass; `make gate` prints `gate: ok`. | Laguna S 2.1 |
 | M1/01-workspace-and-gate | 2026-09-17 | done | 1 | pass | none | Crate skeletons, Cargo files and the given Makefile/deny.toml/test-gate-scripts.sh were already present untracked from a prior attempt; I verified them against the plan and created only the missing gate scripts, dependencies.md, egress.md and this log row. | Laguna S 2.1 |
 | M1/02-proto-values | 2026-09-17 | done | 1 | pass | none | Implemented ValueError, SessionId, Epoch, CallId, Hash32 and Timestamp in crates/proto/src/ids.rs and DataClass in class.rs, using serde try_from/into for string-backed JSON validation, a hand-written hex encoder and humantime for RFC 3339 parsing with canonical re-serialization. | Laguna S 2.1 |