31 Commits
Author SHA1 Message Date
kyleandClaude Opus 5.5 7e3783faae Specify M4a (conversations over Mattermost); propose P15 on secrets
M4 is split into M4a and M4b. The M4a spec rests on facts checked against
the owner's server and Mattermost's source at v11.11.0: the REST and
WebSocket shapes, and that clients will not post a message starting with
'/'. Records the design decisions, proposes P15 (secrets from a systemd
credential, the environment or an owner-only file), and adds the run-time
rows egress.md was missing since M2a and M3b.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 17:30:31 -07:00
kyleandClaude Opus 5.5 f0ec0efe9d CLAUDE.md: M3a finding 14 was closed by M3b task 02
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 16:43:07 -07:00
kyleandClaude Opus 5.5 8bcdb25ff1 Merge m3b: the runner and the tools (M3b)
Tasks 01 to 17 by Ornith, the review, and the design model's fixes to the
plan. Tool containers run from a Nix-built image named by digest; checked on
straylight.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 16:42:22 -07:00
kyleandClaude Opus 5.5 a43362edb4 Review M3b tasks 14 to 17: accept; M3b done
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 16:42:22 -07:00
kyle 9662966b45 toolkit: no thread panic in http_fetch, no casts, exact egress-proxy form
Implemented-By: OpenCode session (model recorded in docs/implementer-log.md)
2026-09-23 16:13:43 -07:00
kyle e69ba632e6 brokerd: escape container errors in the log; prefix and quote two messages
Implemented-By: OpenCode session (model recorded in docs/implementer-log.md)
2026-09-23 16:10:54 -07:00
kyle 93539dbead brokerd: start pipe threads safely and collect output within a grace period
Implemented-By: OpenCode session (model recorded in docs/implementer-log.md)
2026-09-23 16:07:15 -07:00
kyle c87aff0793 brokerd: move the container's pipe handling into pipes.rs 2026-09-23 15:59:44 -07:00
kyleandClaude Opus 5.5 a57a1305e7 M3b plan: follow-up tasks 14 to 17 for the review's lower findings
14 moves the pipe handling out of container.rs (a pure move, replayed on its
own); 15 starts threads with Builder and bounds output collection with a 2 s
grace period; 16 escapes container errors in the log and fixes two texts; 17
fixes toolkit's thread start, casts and the egress-proxy form. Each checked
against a reference, which is not kept. Tips T24 to T26 from this run.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 15:33:23 -07:00
kyleandClaude Opus 5.5 c3aaecdae2 M3b review fixes by the design model: curl --globoff, podman --pull=never
Review findings 1 and 2, both plan defects. curl gains --globoff and a
leading --disable; both podman runs gain --pull=never. The given fetch.rs and
the six golden files change with them. Checked on straylight with a rebuilt
image: a glob URL is one request, a missing image fails at once.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 15:25:51 -07:00
kyleandClaude Opus 5.5 75c2f1adbf Review M3b: accept with follow-ups; add the Nix expression for the image
On straylight, with real containers from deploy/tools-image.nix, every claim
held: no network without a grant, the limits, the file tools, http_fetch's
host checks including a redirect and a tailnet name, and no leftovers. Two
plan defects found there (curl globbing, podman pulling a missing image),
five lower findings.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 11:48:31 -07:00
kyle feb50abc88 brokerd serve: run tools in containers when [runner] is set
Implemented-By: OpenCode session (model recorded in docs/implementer-log.md)
2026-09-23 09:35:46 -07:00
kyle e6081a2177 brokerd: start and remove the egress proxy for http_fetch
Implemented-By: OpenCode session (model recorded in docs/implementer-log.md)
2026-09-23 09:22:24 -07:00
kyle cfe13787b0 brokerd: the Podman runtime
Implemented-By: OpenCode session (model recorded in docs/implementer-log.md)
2026-09-23 08:51:44 -07:00
kyleandClaude Opus 5.5 4a1c6fa0a5 M3b plan: task 11's skeleton writes run as glue over small helpers
The third attempt filled five functions, then ran out of room planning all of
run in one turn. run and run_container are now given; spawn, Io::start,
Io::finish and answer are small todo!()s. Checked fillable: 11 of 11 passed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 08:36:00 -07:00
kyleandClaude Opus 5.5 62d1da44d6 M3b plan: task 11 starts from a compiling skeleton of container.rs
Two sessions ended with nothing written, each out of room while planning the
whole file in one turn. The skeleton has the signatures, the constants and
the steps as comments; the task says to fill one function at a time.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 07:34:14 -07:00
kyleandClaude Opus 5.5 4458369d64 M3b plan: task 11 gives Podman::egress_dir(), so the field is read
Without it the field was never read in task 11, clippy's dead_code failed the
gate, and #[allow] is forbidden; the implementer stopped on the dilemma.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 07:16:56 -07:00
kyle 49ac8d72d2 brokerd: the podman argument lists
Implemented-By: OpenCode session (model recorded in docs/implementer-log.md)
2026-09-23 01:13:25 -07:00
kyle 81ce5a3345 brokerd: the [runner] section
Implemented-By: OpenCode session (model recorded in docs/implementer-log.md)
2026-09-23 01:07:21 -07:00
kyle 95872d94b8 toolkit: the egress proxy
Implemented-By: OpenCode session (model recorded in docs/implementer-log.md)
2026-09-23 01:01:13 -07:00
kyleandClaude Opus 5.5 e87d876f26 M3b plan: task 08 says not to join handler threads and to skip argv[0]
The first attempt at task 08 did both and stopped without a commit. Also asks
the implementer to debug inside the repository, since OpenCode refuses /tmp.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 00:46:23 -07:00
kyle 74da0c9d96 toolkit: is_public, the addresses the egress proxy may reach
Implemented-By: OpenCode session (model recorded in docs/implementer-log.md)
2026-09-23 00:19:18 -07:00
kyle adf6713866 toolkit: http_fetch through curl
Implemented-By: OpenCode session (model recorded in docs/implementer-log.md)
2026-09-23 00:14:16 -07:00
kyle ac1ecacadc toolkit: shell
Implemented-By: OpenCode session (model recorded in docs/implementer-log.md)
2026-09-23 00:06:44 -07:00
kyleandClaude Opus 5.5 5e55fe4c66 M3b plan: every task's git add includes Cargo.lock
Task 04 added dependencies to toolkit and its git add line left out the lock
file, so the driver stopped on an unclean tree. The lock change is folded into
task 04's commit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 00:00:12 -07:00
kyle 0d444dd5bf toolkit: the tool program, read_file and write_file
Implemented-By: OpenCode session (model recorded in docs/implementer-log.md)
2026-09-23 00:00:03 -07:00
kyle 644fda14da Refuse grant paths that cannot be mounted
Implemented-By: OpenCode session (model recorded in docs/implementer-log.md)
2026-09-22 23:06:09 -07:00
kyle bf9e79ac21 Seal the fetch target: one value holds the URL and its host
Implemented-By: OpenCode session (model recorded in docs/implementer-log.md)
2026-09-22 23:03:49 -07:00
kyle f095cca1ee Move the tools' arguments and the host rules to proto
Implemented-By: OpenCode session (model recorded in docs/implementer-log.md)
2026-09-22 22:59:40 -07:00
kyleandClaude Opus 5.5 655683c9e0 Apply P13: tool containers run from one Nix-built image named by digest
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-22 22:30:04 -07:00
kyleandClaude Opus 5.5 b426ca1958 Specify and plan M3b: the runner and the tools
A draft spec for the owner's review and 13 offline tasks with their given
tests: shared tool arguments and host rules in proto, the sealed fetch
target (M3a finding 14), the toolkit tools and SOCKS5 egress proxy, and
brokerd's [runner], podman argument lists, runtime and proxy lifecycle. Each
task's tests were run against a reference at that task's end state (560 to
638 tests, clippy clean); the reference is not in the repository. Adds the
runner-unavailable runbook entry and tip T23 (ETXTBSY in script tests).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-22 22:29:27 -07:00
118 changed files with 10479 additions and 131 deletions
+5 -2
View File
@@ -9,8 +9,11 @@ time (M0 to M7, table in `docs/milestones.md`). M0 (measurements), M1 (workspace
`Decision`), M2a (the inference path: `inferproxy`, `loopd`'s HTTP, SSE and llama client, the `Decision`), M2a (the inference path: `inferproxy`, `loopd`'s HTTP, SSE and llama client, the
startup self-test) and M2b (sessions, the turn loop, `loopd serve`, `bxctl chat`) are done. M3 is split: startup self-test) and M2b (sessions, the turn loop, `loopd serve`, `bxctl chat`) are done. M3 is split:
M3a (`brokerd`'s decision path, spec `docs/specs/2026-09-18-m3a-decision-path.md`) is done and M3a (`brokerd`'s decision path, spec `docs/specs/2026-09-18-m3a-decision-path.md`) is done and
merged, reviewed in `docs/implementer-log.md`; its open findings (17, 18, 20, and 14, which M3b's merged, reviewed in `docs/implementer-log.md`; its open findings (17, 18, 20) are listed there; 14 was
first task must take) are listed there. M3b (the container runner and tools) is next to plan. closed by M3b task 02. M3b (the container runner and the four tools, `docs/specs/2026-09-22-m3b-runner.md`) is done and
merged: tool containers run from the image `deploy/tools-image.nix` builds, checked on straylight.
M4 is split: M4a (`gatewayd`, conversations over Mattermost) is specified in
`docs/specs/2026-09-23-m4a-gateway.md`, a draft for review; M4b (approvals over Mattermost) follows.
Straylight now serves Ornith as four slots over one 262,144-token pool; see Straylight now serves Ornith as four slots over one 262,144-token pool; see
`docs/inference-contract.md`, "Deployment change, 2026-09-20", before relying on cache behaviour. `docs/inference-contract.md`, "Deployment change, 2026-09-20", before relying on cache behaviour.
`docs/runbook.md` has an entry for every `docs/runbook.md` has an entry for every
Generated
+2
View File
@@ -226,6 +226,8 @@ name = "toolkit"
version = "0.1.0" version = "0.1.0"
dependencies = [ dependencies = [
"proto", "proto",
"serde",
"serde_json",
] ]
[[package]] [[package]]
+43 -96
View File
@@ -4,7 +4,7 @@
//! path, a host or a URL is well formed. The module is pure: no I/O, no clock. Everything it reads //! path, a host or a URL is well formed. The module is pure: no I/O, no clock. Everything it reads
//! was written by the model, so it is treated as hostile. //! was written by the model, so it is treated as hostile.
use serde::{Deserialize, Serialize}; use proto::tools::{HttpFetchArgs, ReadFileArgs, ShellArgs, WriteFileArgs};
use std::fmt; use std::fmt;
/// Maximum length, in bytes, of a path. /// Maximum length, in bytes, of a path.
@@ -66,10 +66,39 @@ pub enum ToolArgs {
command: String, command: String,
cwd: Option<String>, cwd: Option<String>,
}, },
HttpFetch { HttpFetch(FetchUrl),
url: String, }
host: String,
}, /// A URL that passed the checks, and its host. Only `parse` makes one, so the host policy matched
/// is always the host of the URL the tool fetches.
///
/// ```compile_fail
/// let _ = brokerd::args::FetchUrl {
/// url: "https://evil.example/".to_string(),
/// host: "example.com".to_string(),
/// };
/// ```
///
/// ```
/// let args = brokerd::args::parse(brokerd::args::ToolName::HttpFetch, r#"{"url":"https://example.com/a"}"#);
/// let Ok(brokerd::args::ToolArgs::HttpFetch(target)) = args else { panic!("{args:?}") };
/// assert_eq!((target.url(), target.host()), ("https://example.com/a", "example.com"));
/// ```
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct FetchUrl {
url: String, // private
host: String, // private
}
impl FetchUrl {
/// The URL that passed the checks.
pub fn url(&self) -> &str {
&self.url
}
/// The host of that URL, as `parse` derived it.
pub fn host(&self) -> &str {
&self.host
}
} }
impl ToolArgs { impl ToolArgs {
@@ -79,7 +108,7 @@ impl ToolArgs {
ToolArgs::ReadFile { .. } => ToolName::ReadFile, ToolArgs::ReadFile { .. } => ToolName::ReadFile,
ToolArgs::WriteFile { .. } => ToolName::WriteFile, ToolArgs::WriteFile { .. } => ToolName::WriteFile,
ToolArgs::Shell { .. } => ToolName::Shell, ToolArgs::Shell { .. } => ToolName::Shell,
ToolArgs::HttpFetch { .. } => ToolName::HttpFetch, ToolArgs::HttpFetch(_) => ToolName::HttpFetch,
} }
} }
@@ -107,8 +136,10 @@ impl ToolArgs {
}; };
serde_json::to_string(&value).unwrap_or_else(|_| "{}".to_string()) serde_json::to_string(&value).unwrap_or_else(|_| "{}".to_string())
} }
ToolArgs::HttpFetch { url, .. } => { ToolArgs::HttpFetch(target) => {
let value = HttpFetchArgs { url: url.clone() }; let value = HttpFetchArgs {
url: target.url.clone(),
};
serde_json::to_string(&value).unwrap_or_else(|_| "{}".to_string()) serde_json::to_string(&value).unwrap_or_else(|_| "{}".to_string())
} }
} }
@@ -142,34 +173,6 @@ impl std::error::Error for ArgsError {
} }
} }
/// The argument shape for one tool, decoded then re-serialised.
#[derive(Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
struct ReadFileArgs {
path: String,
}
#[derive(Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
struct WriteFileArgs {
path: String,
content: String,
}
#[derive(Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
struct ShellArgs {
command: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
cwd: Option<String>,
}
#[derive(Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
struct HttpFetchArgs {
url: String,
}
/// Decode `arguments` for `tool` into a typed value. /// Decode `arguments` for `tool` into a typed value.
pub fn parse(tool: ToolName, arguments: &str) -> Result<ToolArgs, ArgsError> { pub fn parse(tool: ToolName, arguments: &str) -> Result<ToolArgs, ArgsError> {
match tool { match tool {
@@ -220,10 +223,10 @@ pub fn parse(tool: ToolName, arguments: &str) -> Result<ToolArgs, ArgsError> {
Some(host) => host.to_string(), Some(host) => host.to_string(),
None => return Err(ArgsError::Url(value.url)), None => return Err(ArgsError::Url(value.url)),
}; };
Ok(ToolArgs::HttpFetch { Ok(ToolArgs::HttpFetch(FetchUrl {
url: value.url, url: value.url,
host, host,
}) }))
} }
} }
} }
@@ -257,64 +260,8 @@ pub fn inside(grant_path: &str, path: &str) -> bool {
} }
} }
/// A host label is 1 to 63 bytes of `a-z`, `0-9` or `-`, and neither starts nor ends with `-`. /// Moved to `proto::hosts` in M3b, so `toolkit` checks hosts with the same rules.
fn valid_label(label: &str) -> bool { pub use proto::hosts::{host_matches, valid_host, valid_host_pattern};
if !(1..=63).contains(&label.len()) {
return false;
}
let bytes = label.as_bytes();
if bytes.iter().next() == Some(&b'-') || bytes.iter().last() == Some(&b'-') {
return false;
}
bytes
.iter()
.all(|&byte| matches!(byte, b'a'..=b'z' | b'0'..=b'9' | b'-'))
}
/// A host is 1 to 253 bytes of dot-separated labels, each a whole non-dot name, and the last label
/// starts with a letter. That last rule keeps out every spelling of an IPv4 address.
pub fn valid_host(host: &str) -> bool {
if !(1..=253).contains(&host.len()) {
return false;
}
let labels: Vec<&str> = host.split('.').collect();
if labels.len() < 2 {
return false;
}
for label in &labels {
if !valid_label(label) {
return false;
}
}
match labels.iter().last() {
Some(last) => matches!(last.bytes().next(), Some(byte) if byte.is_ascii_lowercase()),
None => false,
}
}
/// A host, or `*.` followed by a host. Nothing else.
pub fn valid_host_pattern(pattern: &str) -> bool {
match pattern.strip_prefix("*.") {
Some(base) => valid_host(base),
None => valid_host(pattern),
}
}
/// Does `pattern` match `host`? Without `*.` the strings must be equal; with `*.base` the host must
/// end in `.base` with something before the dot.
pub fn host_matches(pattern: &str, host: &str) -> bool {
let base = match pattern.strip_prefix("*.") {
Some(base) => base,
None => return pattern == host,
};
match host.strip_suffix(base) {
Some(before) => match before.strip_suffix('.') {
Some(prefix) => !prefix.is_empty(),
None => false,
},
None => false,
}
}
/// A character that may appear in a host name. /// A character that may appear in a host name.
fn host_char(c: char) -> bool { fn host_char(c: char) -> bool {
+150
View File
@@ -45,6 +45,79 @@ impl Default for Approvals {
} }
} }
fn default_podman() -> PathBuf {
PathBuf::from("podman")
}
fn default_egress_network() -> String {
"pasta".to_string()
}
fn default_output_cap() -> u64 {
262_144
}
fn default_memory() -> String {
"512m".to_string()
}
fn default_pids() -> u32 {
128
}
fn default_read_file_ms() -> u64 {
30_000
}
fn default_write_file_ms() -> u64 {
30_000
}
fn default_shell_ms() -> u64 {
100_000
}
fn default_http_fetch_ms() -> u64 {
60_000
}
#[derive(Debug, Clone, PartialEq, Eq, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct Runner {
#[serde(default = "default_podman")]
pub podman: PathBuf,
pub image: String,
#[serde(default = "default_egress_network")]
pub egress_network: String,
#[serde(default = "default_output_cap")]
pub output_cap: u64,
#[serde(default = "default_memory")]
pub memory: String,
#[serde(default = "default_pids")]
pub pids: u32,
#[serde(default = "default_read_file_ms")]
pub read_file_ms: u64,
#[serde(default = "default_write_file_ms")]
pub write_file_ms: u64,
#[serde(default = "default_shell_ms")]
pub shell_ms: u64,
#[serde(default = "default_http_fetch_ms")]
pub http_fetch_ms: u64,
}
impl Runner {
/// The time limit for one call of `tool`: the matching `_ms` field.
pub fn time_limit(&self, tool: crate::args::ToolName) -> std::time::Duration {
let ms = match tool {
crate::args::ToolName::ReadFile => self.read_file_ms,
crate::args::ToolName::WriteFile => self.write_file_ms,
crate::args::ToolName::Shell => self.shell_ms,
crate::args::ToolName::HttpFetch => self.http_fetch_ms,
};
std::time::Duration::from_millis(ms)
}
}
/// `memory` is well formed if it is one or more digits followed by `b`, `k`, `m` or `g`.
fn valid_memory(memory: &str) -> bool {
let bytes = memory.as_bytes();
bytes.len() >= 2
&& matches!(bytes[bytes.len() - 1], b'b' | b'k' | b'm' | b'g')
&& bytes[..bytes.len() - 1].iter().all(|&c| c.is_ascii_digit())
}
#[derive(Debug, Clone, PartialEq, Eq, Deserialize, Default)] #[derive(Debug, Clone, PartialEq, Eq, Deserialize, Default)]
#[serde(deny_unknown_fields)] #[serde(deny_unknown_fields)]
pub struct Config { pub struct Config {
@@ -54,6 +127,8 @@ pub struct Config {
pub sockets: Sockets, pub sockets: Sockets,
#[serde(default)] #[serde(default)]
pub approvals: Approvals, pub approvals: Approvals,
#[serde(default)]
pub runner: Option<Runner>,
} }
/// The longest an approval may wait: a day, the longest `loopd` waits after a pending frame. /// The longest an approval may wait: a day, the longest `loopd` waits after a pending frame.
@@ -99,6 +174,78 @@ impl Config {
), ),
)); ));
} }
if let Some(runner) = &config.runner {
let where_image = format!(
"[runner] image is {:?}; it must be named by digest: <name>@sha256:<64 hex digits>",
runner.image
);
let (name, hex) = match runner.image.rsplit_once("@sha256:") {
Some(pair) => pair,
None => {
return Err(ConfigError::Invalid(
path.to_path_buf(),
where_image.clone(),
));
}
};
if name.is_empty()
|| hex.len() != 64
|| !hex.chars().all(|c| matches!(c, '0'..='9' | 'a'..='f'))
{
return Err(ConfigError::Invalid(path.to_path_buf(), where_image));
}
if !valid_memory(&runner.memory) {
return Err(ConfigError::Invalid(
path.to_path_buf(),
format!(
"[runner] memory is {:?}; it must be a number and one of b, k, m, g",
runner.memory
),
));
}
if runner.egress_network.is_empty() || runner.podman.as_os_str().is_empty() {
return Err(ConfigError::Invalid(
path.to_path_buf(),
"[runner] podman and egress_network must not be empty".to_string(),
));
}
if runner.output_cap == 0 {
return Err(ConfigError::Invalid(
path.to_path_buf(),
"[runner] output_cap must be at least 1".to_string(),
));
}
if runner.pids == 0 {
return Err(ConfigError::Invalid(
path.to_path_buf(),
"[runner] pids must be at least 1".to_string(),
));
}
if runner.read_file_ms == 0 {
return Err(ConfigError::Invalid(
path.to_path_buf(),
"[runner] read_file_ms must be at least 1".to_string(),
));
}
if runner.write_file_ms == 0 {
return Err(ConfigError::Invalid(
path.to_path_buf(),
"[runner] write_file_ms must be at least 1".to_string(),
));
}
if runner.shell_ms == 0 {
return Err(ConfigError::Invalid(
path.to_path_buf(),
"[runner] shell_ms must be at least 1".to_string(),
));
}
if runner.http_fetch_ms == 0 {
return Err(ConfigError::Invalid(
path.to_path_buf(),
"[runner] http_fetch_ms must be at least 1".to_string(),
));
}
}
Ok(config) Ok(config)
} }
pub fn broker_socket(&self) -> PathBuf { pub fn broker_socket(&self) -> PathBuf {
@@ -121,4 +268,7 @@ impl Config {
pub fn state_dir(&self) -> PathBuf { pub fn state_dir(&self) -> PathBuf {
self.paths.home.join("broker/sessions") self.paths.home.join("broker/sessions")
} }
pub fn egress_dir(&self) -> PathBuf {
self.paths.home.join("run/egress")
}
} }
+374
View File
@@ -0,0 +1,374 @@
//! The Podman runtime: one fresh container per call, with the limits of `[runner]`. Whatever the
//! tool prints is the result's content; every failure is a fixed sentence, and what Podman itself
//! said goes only to `brokerd`'s log. M3b spec, section 6.
use std::ffi::OsString;
use std::os::unix::fs::{DirBuilderExt, PermissionsExt};
use std::path::{Path, PathBuf};
use std::process::{Child, Command, ExitStatus, Stdio};
use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::{Duration, Instant};
use crate::config::Runner;
use crate::pipes::{GRACE, Io};
use crate::podman;
use crate::runner::{RunError, RunOutput, RunSpec, Runtime};
pub const RUNBOOK: &str = "see docs/runbook.md#runner-unavailable";
pub const COULD_NOT_RUN: &str = "the tool could not run";
pub const CANNOT_START: &str = "the tool runner could not start the container";
pub const KILLED: &str = "the tool was stopped: it ran out of memory or was killed";
pub const TIMED_OUT: &str = "the tool ran past its time limit";
pub const OUTPUT_OPEN: &str = "the tool left its output open";
pub const UNEXPECTED: &str = "the tool failed with an unexpected status";
/// How often a running container is checked.
pub const POLL: Duration = Duration::from_millis(50);
/// How much of Podman's standard error is kept for the log.
pub const STDERR_KEPT: usize = 4096;
/// How long `run` waits for the egress proxy to make its socket.
pub const EGRESS_WAIT: Duration = Duration::from_secs(5);
/// How often the egress proxy's socket is checked while waiting.
pub const EGRESS_POLL: Duration = Duration::from_millis(20);
/// A log sink. Call it as `(self.log)("a line")`.
pub type Log = Arc<dyn Fn(&str) + Send + Sync>;
pub struct Podman {
runner: Runner,
egress_dir: PathBuf,
egress_wait: Duration,
log: Log,
next: AtomicU64,
}
impl Podman {
/// `egress_dir` is `Config::egress_dir()`; task 12 uses it.
pub fn new(runner: Runner, egress_dir: PathBuf, log: Log) -> Podman {
Podman {
runner,
egress_dir,
egress_wait: EGRESS_WAIT,
log,
next: AtomicU64::new(0),
}
}
/// The same runtime with another wait for the proxy's socket, for tests.
pub fn with_egress_wait(self, egress_wait: Duration) -> Podman {
Podman {
egress_wait,
..self
}
}
/// Where `http_fetch` calls get their directories. Public, so the field counts as read.
pub fn egress_dir(&self) -> &Path {
&self.egress_dir
}
/// Run one short `podman` command (`kill`, `rm -f`) whose result only matters for the log.
pub(crate) fn podman(&self, args: &[&str]) {
let what = args.join(" ");
let podman = &self.runner.podman;
match Command::new(podman)
.args(args)
.stdin(Stdio::null())
.stdout(Stdio::null())
.stderr(Stdio::null())
.status()
{
Ok(status) if status.success() => {}
Ok(status) => (self.log)(&format!("brokerd: podman {what} exited {status}")),
Err(e) => (self.log)(&format!("brokerd: podman {what} failed: {e}")),
}
}
/// The log for a `podman` that cannot be launched at all (task 11 step 3).
fn cannot_launch(&self, podman: &Path, e: std::io::Error) {
(self.log)(&format!(
"brokerd: cannot start {}: {}\n{}",
podman.display(),
e,
RUNBOOK
));
}
/// The log for a directory that cannot be made (task 12 step 1).
fn cannot_make(&self, dir: &Path, e: std::io::Error) -> RunError {
(self.log)(&format!(
"brokerd: cannot make {}: {e}\n{}",
dir.display(),
RUNBOOK
));
RunError::Unavailable(CANNOT_START.to_string())
}
/// Wait for `child` until `limit` has passed since `started`. `Some(status)` if it ended;
/// `None` if it ran too long, after `podman kill <name>`, `podman rm -f <name>`,
/// `child.kill()` and `child.wait()` (step 5).
fn wait(
&self,
child: &mut Child,
name: &str,
started: Instant,
limit: Duration,
) -> Option<ExitStatus> {
loop {
if let Ok(Some(status)) = child.try_wait() {
return Some(status);
}
if started.elapsed() >= limit {
self.podman(&["kill", name]);
self.podman(&["rm", "-f", name]);
let _ = child.kill();
let _ = child.wait();
return None;
}
std::thread::sleep(POLL);
}
}
}
impl Podman {
/// Steps 3 to 7 for one container: spawn, feed and read it, wait, answer. Task 12 calls it too.
/// Already written: it only joins the helpers below.
fn run_container(
&self,
name: &str,
args: Vec<OsString>,
input: String,
limit: Duration,
) -> Result<RunOutput, RunError> {
let Some(mut child) = self.spawn(args) else {
return Err(RunError::Unavailable(CANNOT_START.to_string()));
};
let started = Instant::now();
let cap = usize::try_from(self.runner.output_cap).unwrap_or(usize::MAX);
let io = match Io::start(&mut child, input, cap, STDERR_KEPT) {
Ok(io) => io,
Err(e) => {
// The container may be running: stop it before answering.
self.podman(&["kill", name]);
self.podman(&["rm", "-f", name]);
let _ = child.kill();
let _ = child.wait();
(self.log)(&format!(
"brokerd: cannot start a thread for {name}: {e}\n{RUNBOOK}"
));
return Err(RunError::Unavailable(CANNOT_START.to_string()));
}
};
let status = self.wait(&mut child, name, started, limit);
let done = io.finish(GRACE);
if done.open && status.is_some() {
(self.log)(&format!(
"brokerd: {name} ended but something still holds its output; it was abandoned"
));
return Err(RunError::Failed(OUTPUT_OPEN.to_string()));
}
self.answer(name, status, done.out, done.truncated, &done.err)
}
/// Step 3: spawn `podman` with `args` and all three standard streams piped. On failure, log
/// `brokerd: cannot start {podman path}: {e}` + "\n" + RUNBOOK and return `None`.
fn spawn(&self, args: Vec<OsString>) -> Option<Child> {
let podman = &self.runner.podman;
match Command::new(podman)
.args(args)
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
{
Ok(child) => Some(child),
Err(e) => {
self.cannot_launch(podman, e);
None
}
}
}
/// Step 7: the answer, by the table in the task. `status` is `None` when the time limit was
/// passed. `out` is the kept standard output, `err` the kept standard error (for the log only:
/// it never goes into a `RunError`).
fn answer(
&self,
name: &str,
status: Option<ExitStatus>,
out: Vec<u8>,
truncated: bool,
err: &str,
) -> Result<RunOutput, RunError> {
match status {
None => {
(self.log)(&format!(
"brokerd: stopped container {}: it ran past its time limit",
name
));
Err(RunError::Failed(TIMED_OUT.to_string()))
}
Some(status) => match status.code() {
Some(0) | Some(1) => Ok(RunOutput {
content: String::from_utf8_lossy(&out).into_owned(),
truncated,
}),
Some(2) => {
(self.log)(&format!("brokerd: {name}: the tool could not run: {err:?}"));
Err(RunError::Failed(COULD_NOT_RUN.to_string()))
}
Some(125..=127) => {
(self.log)(&format!(
"brokerd: podman could not start {name}: {err:?}\n{RUNBOOK}"
));
Err(RunError::Unavailable(CANNOT_START.to_string()))
}
Some(137) => Err(RunError::Failed(KILLED.to_string())),
_ => {
(self.log)(&format!(
"brokerd: container {name} exited {status}: {err:?}"
));
Err(RunError::Failed(UNEXPECTED.to_string()))
}
},
}
}
}
impl Runtime for Podman {
/// Step 1 (the container number), then step 2's choice: no egress runs the tool as before,
/// an egress starts the proxy first and gives the tool the directory. `run_container` is step 3
/// to 7 for either. Already written.
fn run(&self, spec: &RunSpec) -> Result<RunOutput, RunError> {
let n = self.next.fetch_add(1, Ordering::SeqCst);
let name = podman::container_name(spec.session(), spec.call(), n);
let input = spec.arguments().canonical_json();
let limit = self.runner.time_limit(spec.tool());
match spec.egress() {
None => {
let args = podman::tool_args(spec, &self.runner, &name, None);
self.run_container(&name, args, input, limit)
}
Some(hosts) => {
let dir = self.egress_dir.join(&name);
let _guard = self.start_egress(&name, &self.egress_dir, &dir, hosts)?;
let args = podman::tool_args(spec, &self.runner, &name, Some(&dir));
self.run_container(&name, args, input, limit)
}
}
}
}
impl Podman {
/// Steps 1 to 4 of the egress proxy's start (task 12). The guard it returns cleans up the
/// proxy and directory on every path; `run` keeps it alive until the tool's container ends.
fn start_egress(
&self,
name: &str,
egress_dir: &Path,
dir: &Path,
hosts: &[String],
) -> Result<EgressGuard<'_>, RunError> {
// The guard is created before anything can fail, so every return below
// cleans up the proxy and directory by dropping it.
let _guard = EgressGuard::new(self, name, dir.to_path_buf());
// 1. Make the egress directory (and its parents) 0700, then replace the
// call's directory if a crash left one behind.
std::fs::DirBuilder::new()
.recursive(true)
.mode(0o700)
.create(egress_dir)
.map_err(|e| self.cannot_make(dir, e))?;
let _ = std::fs::set_permissions(egress_dir, std::fs::Permissions::from_mode(0o700));
if dir.exists() {
match std::fs::remove_dir_all(dir) {
Ok(()) => {}
Err(ref e) if e.kind() == std::io::ErrorKind::NotFound => {}
Err(e) => return Err(self.cannot_make(dir, e)),
}
}
std::fs::DirBuilder::new()
.mode(0o700)
.create(dir)
.map_err(|e| self.cannot_make(dir, e))?;
// 2. Start the proxy, detached. A run that exits non-zero is logged and
// unavailable; one that cannot be launched at all uses task 11's log.
let args = podman::egress_args(&self.runner, name, dir, hosts);
match Command::new(&self.runner.podman)
.args(args)
.stdin(Stdio::null())
.stdout(Stdio::null())
.stderr(Stdio::piped())
.output()
{
Ok(output) if output.status.success() => {}
Ok(output) => {
let stderr = String::from_utf8_lossy(&output.stderr).into_owned();
(self.log)(&format!(
"brokerd: podman could not start {name}-egress: {stderr:?}\n{RUNBOOK}"
));
return Err(RunError::Unavailable(CANNOT_START.to_string()));
}
Err(e) => {
self.cannot_launch(&self.runner.podman, e);
return Err(RunError::Unavailable(CANNOT_START.to_string()));
}
}
// 3. Wait for the proxy to make its socket, or give up.
let socket = dir.join("egress.sock");
let deadline = Instant::now() + self.egress_wait;
loop {
if socket.exists() {
break;
}
if Instant::now() >= deadline {
let ms = self.egress_wait.as_millis();
(self.log)(&format!(
"brokerd: {name}-egress did not make its socket within {ms} ms\n{RUNBOOK}"
));
return Err(RunError::Unavailable(CANNOT_START.to_string()));
}
std::thread::sleep(EGRESS_POLL);
}
// 4. The tool's container may now run; `run` holds the guard until it
// returns, then drops it to remove the proxy and directory.
Ok(_guard)
}
}
/// Removes the egress proxy's container and its directory when `run` returns, on every path.
///
/// `run` creates it before anything can fail and holds it until the tool's container has ended, so
/// the proxy and directory never survive the call that started it.
struct EgressGuard<'a> {
podman: &'a Podman,
name: String,
dir: PathBuf,
}
impl<'a> Drop for EgressGuard<'a> {
fn drop(&mut self) {
self.podman.podman(&["rm", "-f", self.name.as_str()]);
match std::fs::remove_dir_all(&self.dir) {
Ok(()) => {}
Err(ref e) if e.kind() == std::io::ErrorKind::NotFound => {}
Err(e) => (self.podman.log)(&format!(
"brokerd: cannot remove {}: {e}",
self.dir.display()
)),
}
}
}
impl<'a> EgressGuard<'a> {
fn new(podman: &'a Podman, name: &str, dir: PathBuf) -> EgressGuard<'a> {
EgressGuard {
podman,
name: format!("{name}-egress"),
dir,
}
}
}
+8
View File
@@ -260,6 +260,14 @@ fn check_grant(grant: &LoadedGrant, problems: &mut Vec<GrantProblem>) {
None, None,
format!("{:?} is not a valid absolute path", path), format!("{:?} is not a valid absolute path", path),
); );
} else if path.contains([':', ',']) {
// Mounted as `--volume=<path>:<path>:ro`, where both are separators.
push(
problems,
file.clone(),
None,
format!("{:?} cannot be mounted: it contains ':' or ','", path),
);
} }
} }
// Rule 8: every host must be a valid host pattern. // Rule 8: every host must be a valid host pattern.
+3
View File
@@ -6,8 +6,11 @@ pub mod args;
pub mod audit; pub mod audit;
pub mod broker; pub mod broker;
pub mod config; pub mod config;
pub mod container;
pub mod grants; pub mod grants;
pub mod ledger; pub mod ledger;
pub mod pipes;
pub mod podman;
pub mod policy; pub mod policy;
pub mod runner; pub mod runner;
pub mod serve; pub mod serve;
+25 -7
View File
@@ -7,7 +7,8 @@ use std::sync::Arc;
use brokerd::audit::{AuditError, RECOVERED_NOTICE}; use brokerd::audit::{AuditError, RECOVERED_NOTICE};
use brokerd::config::Config; use brokerd::config::Config;
use brokerd::runner::Refusing; use brokerd::container::Podman;
use brokerd::runner::{Refusing, Runtime};
use brokerd::serve::{self, ServeError}; use brokerd::serve::{self, ServeError};
const USAGE: &str = "usage: brokerd serve --config <path> [--accept-break]"; const USAGE: &str = "usage: brokerd serve --config <path> [--accept-break]";
@@ -44,12 +45,28 @@ fn main() -> ExitCode {
let broker_path = cfg.broker_socket(); let broker_path = cfg.broker_socket();
let admin_path = cfg.admin_socket(); let admin_path = cfg.admin_socket();
let started = match serve::start( // The runtime and the line that says which. A `[runner]` runs calls in containers; without it,
cfg, // every call is refused, as in M3a.
accept_break, let log: Arc<dyn Fn(&str) + Send + Sync> = Arc::new(|line: &str| eprintln!("{line}"));
Box::new(Refusing), let runtime: Box<dyn Runtime>;
Arc::new(|line: &str| eprintln!("{line}")), let runtime_notice: String;
) { match &cfg.runner {
Some(runner) => {
let image = runner.image.clone();
runtime = Box::new(Podman::new(
runner.clone(),
cfg.egress_dir(),
Arc::clone(&log),
));
runtime_notice = format!("tools run in containers from {image}");
}
None => {
runtime = Box::new(Refusing);
runtime_notice = "no [runner] section: every tool call is refused".to_string();
}
}
let started = match serve::start(cfg, accept_break, runtime, log) {
Ok(started) => started, Ok(started) => started,
Err(ServeError::Audit(e @ AuditError::NothingToAccept)) => { Err(ServeError::Audit(e @ AuditError::NothingToAccept)) => {
eprintln!("brokerd: {e}"); eprintln!("brokerd: {e}");
@@ -80,6 +97,7 @@ fn main() -> ExitCode {
broker_path.display(), broker_path.display(),
admin_path.display() admin_path.display()
); );
eprintln!("brokerd: {runtime_notice}");
match started.run() { match started.run() {
Ok(()) => ExitCode::SUCCESS, Ok(()) => ExitCode::SUCCESS,
+128
View File
@@ -0,0 +1,128 @@
//! The three pipes of one container: its arguments go in on standard input, its output and errors
//! come back, each on its own thread so no pipe can block another. The threads start with
//! `Builder`, which returns an error instead of panicking, and the output is collected within a
//! grace period so a pipe held open by something outside the container cannot hold `brokerd` (M3b
//! review findings 3 and 4).
use std::io::{Read, Write};
use std::process::Child;
use std::sync::mpsc::{self, Receiver, RecvTimeoutError};
use std::thread::Builder;
use std::time::{Duration, Instant};
/// How long, after the container has ended, its output may take to reach its end.
pub const GRACE: Duration = Duration::from_secs(2);
/// What came back.
pub(crate) struct Finished {
pub out: Vec<u8>,
pub truncated: bool,
pub err: String,
/// A pipe had not ended by the grace deadline: something outside the container holds it.
pub open: bool,
}
pub(crate) struct Io {
stdout: Option<Receiver<(Vec<u8>, bool)>>,
stderr: Option<Receiver<(Vec<u8>, bool)>>,
}
impl Io {
/// Take `child`'s three pipes (`child.stdin.take()` and so on) and start one thread for each:
/// write `input` to standard input and then drop it (its handle is not kept, it is never
/// joined); `read_capped` standard output with `cap`; `read_capped` standard error with
/// `err_cap`. The reader threads are detached, keeping only the channel receiver. A pipe that
/// is `None` gets no thread; a thread that cannot be started is the returned error.
pub(crate) fn start(
child: &mut Child,
input: String,
cap: usize,
err_cap: usize,
) -> std::io::Result<Io> {
if let Some(mut stdin) = child.stdin.take() {
Builder::new().spawn(move || {
let _ = stdin.write_all(input.as_bytes());
})?;
}
let stdout = if let Some(stdout) = child.stdout.take() {
Some(reader(stdout, cap)?)
} else {
None
};
let stderr = if let Some(stderr) = child.stderr.take() {
Some(reader(stderr, err_cap)?)
} else {
None
};
Ok(Io { stdout, stderr })
}
/// Collect the output within `grace`: one deadline for both readers, after which a pipe that has
/// not ended counts as open (something outside the container holds it) and is abandoned.
pub(crate) fn finish(self, grace: Duration) -> Finished {
let until = Instant::now() + grace;
let (out, truncated, stdout_open) = collect(self.stdout, until);
let (err_bytes, _, stderr_open) = collect(self.stderr, until);
let err = String::from_utf8_lossy(&err_bytes).into_owned();
Finished {
out,
truncated,
err,
open: stdout_open || stderr_open,
}
}
}
/// One detached reader: `read_capped` `pipe` and hand the result back on a channel. The handle is
/// kept only long enough to start the thread; a spawn error is returned, a send error ignored.
fn reader(
pipe: impl Read + Send + 'static,
cap: usize,
) -> std::io::Result<Receiver<(Vec<u8>, bool)>> {
let (tx, rx) = mpsc::channel();
Builder::new().spawn(move || {
let _ = tx.send(read_capped(pipe, cap));
})?;
Ok(rx)
}
/// Read from `rx` until `until`, classifying the outcome: `None` (no pipe) is empty and closed, a
/// timeout is empty and open, and a disconnected channel (the reader panicked) is empty and closed.
fn collect(rx: Option<Receiver<(Vec<u8>, bool)>>, until: Instant) -> (Vec<u8>, bool, bool) {
match rx {
None => (Vec::new(), false, false),
Some(rx) => match rx.recv_timeout(until.saturating_duration_since(Instant::now())) {
Ok((bytes, truncated)) => (bytes, truncated, false),
Err(RecvTimeoutError::Timeout) => (Vec::new(), false, true),
Err(RecvTimeoutError::Disconnected) => (Vec::new(), false, false),
},
}
}
/// Everything `from` gives, keeping the first `cap` bytes; `true` if there was more. Reads on past
/// the cap, so the writer is never blocked or broken by a closed pipe. No indexing and no
/// subtraction that could wrap.
pub(crate) fn read_capped(from: impl Read, cap: usize) -> (Vec<u8>, bool) {
let mut from = from;
let mut kept: Vec<u8> = Vec::new();
let mut truncated = false;
let mut chunk = [0u8; 8192];
loop {
let n = match from.read(&mut chunk) {
Ok(0) => break,
Ok(n) => n,
Err(_) => break,
};
let remaining = cap.saturating_sub(kept.len());
if remaining == 0 {
truncated = true;
} else {
let take = n.min(remaining);
kept.extend_from_slice(chunk.get(..take).unwrap_or_default());
if take < n {
truncated = true;
}
}
}
(kept, truncated)
}
+112
View File
@@ -0,0 +1,112 @@
//! The `podman` argument lists for one call's container and for the `http_fetch` egress proxy.
//!
//! Nothing is passed through a shell: every argument is its own `OsString`, and the tool's
//! arguments go on standard input, never on the command line. Built without running Podman, so a
//! runtime that only builds the list can be tested as a golden file. Spec section 6.
use crate::config::Runner;
use crate::runner::RunSpec;
use proto::{CallId, SessionId};
use std::ffi::OsString;
use std::path::Path;
pub const EGRESS_MOUNT: &str = "/run/egress";
pub const EGRESS_SOCKET: &str = "/run/egress/egress.sock";
pub const TOOLKIT: &str = "/bin/toolkit";
/// The container's name: `boxmaker-<session>-<call>-<n>`, so it says whose call it is.
pub fn container_name(session: &SessionId, call: CallId, n: u64) -> String {
format!("boxmaker-{}-{}-{}", session.as_str(), call.0, n)
}
/// The six hardening flags, common to the tool and the proxy. `pids` and `memory` differ: the tool
/// takes the runner's, the proxy its fixed limits.
fn hardening(pids: u32, memory: &str) -> Vec<OsString> {
[
// A missing image is an error at once, never a pull: a pull is egress, and what runs must
// be exactly the image built for it.
"--pull=never",
"--read-only",
"--cap-drop=all",
"--security-opt=no-new-privileges",
"--userns=keep-id",
]
.map(OsString::from)
.into_iter()
.chain([
OsString::from(format!("--pids-limit={pids}")),
OsString::from(format!("--memory={memory}")),
])
.collect()
}
/// A `--volume=<host>:<container>:<mode>` argument, built with `push` so a directory need not be
/// UTF-8.
fn volume(host: &Path, container: &Path, mode: &str) -> OsString {
let mut arg = OsString::new();
arg.push("--volume=");
arg.push(host);
arg.push(":");
arg.push(container);
arg.push(":");
arg.push(mode);
arg
}
/// The tool's container. `egress` is the call's egress directory, for `http_fetch` only.
pub fn tool_args(
spec: &RunSpec,
runner: &Runner,
name: &str,
egress: Option<&Path>,
) -> Vec<OsString> {
let mut args: Vec<OsString> = [
"run",
"--rm",
"-i",
&format!("--name={name}"),
"--label=boxmaker=tool",
"--network=none",
]
.map(OsString::from)
.into_iter()
.collect();
args.extend(hardening(runner.pids, &runner.memory));
args.push(OsString::from("--tmpfs=/tmp:rw,size=64m,mode=1777"));
for mount in spec.mounts() {
let mode = if mount.writable { "rw" } else { "ro" };
args.push(volume(Path::new(&mount.path), Path::new(&mount.path), mode));
}
if let Some(dir) = egress {
args.push(volume(dir, Path::new(EGRESS_MOUNT), "rw"));
}
args.push(OsString::from(runner.image.as_str()));
args.push(OsString::from(TOOLKIT));
args.push(OsString::from(spec.tool().as_str()));
args
}
/// The egress proxy's container.
pub fn egress_args(runner: &Runner, name: &str, dir: &Path, hosts: &[String]) -> Vec<OsString> {
let mut args: Vec<OsString> = [
"run",
"-d",
"--rm",
&format!("--name={name}-egress"),
"--label=boxmaker=egress",
&format!("--network={}", runner.egress_network),
]
.map(OsString::from)
.into_iter()
.collect();
args.extend(hardening(64, "128m"));
args.push(volume(dir, Path::new(EGRESS_MOUNT), "rw"));
args.push(OsString::from(runner.image.as_str()));
args.push(OsString::from(TOOLKIT));
args.push(OsString::from("egress-proxy"));
args.push(OsString::from("--socket"));
args.push(OsString::from(EGRESS_SOCKET));
args.push(OsString::from("--allow"));
args.push(OsString::from(hosts.join(",")));
args
}
+2 -2
View File
@@ -404,12 +404,12 @@ fn covers(args: &ToolArgs, grant: &Grant) -> Option<Option<String>> {
} }
} }
ToolArgs::Shell { cwd: Some(cwd), .. } => best_path(grant, cwd, true), ToolArgs::Shell { cwd: Some(cwd), .. } => best_path(grant, cwd, true),
ToolArgs::HttpFetch { host, .. } => { ToolArgs::HttpFetch(target) => {
if grant if grant
.constraints .constraints
.hosts .hosts
.iter() .iter()
.any(|pattern| host_matches(pattern, host)) .any(|pattern| host_matches(pattern, target.host()))
{ {
Some(None) Some(None)
} else { } else {
+17 -4
View File
@@ -5,6 +5,8 @@
//! //!
//! ```compile_fail //! ```compile_fail
//! let _ = brokerd::runner::RunSpec { //! let _ = brokerd::runner::RunSpec {
//! session: proto::SessionId::new("s1").unwrap(),
//! call: proto::CallId(1),
//! tool: brokerd::args::ToolName::Shell, //! tool: brokerd::args::ToolName::Shell,
//! arguments: todo!(), //! arguments: todo!(),
//! mounts: Vec::new(), //! mounts: Vec::new(),
@@ -20,7 +22,7 @@
use crate::args::{ToolArgs, ToolName}; use crate::args::{ToolArgs, ToolName};
use crate::policy::Decision; use crate::policy::Decision;
use proto::ToolResponse; use proto::{CallId, SessionId, ToolResponse};
/// A directory mounted for one call: a path and whether the runtime may write to it. /// A directory mounted for one call: a path and whether the runtime may write to it.
#[derive(Debug, Clone, PartialEq, Eq)] #[derive(Debug, Clone, PartialEq, Eq)]
@@ -29,10 +31,13 @@ pub struct Mount {
pub writable: bool, pub writable: bool,
} }
/// What one call was turned into before it reached a `Runtime`: the tool, its arguments, the /// What one call was turned into before it reached a `Runtime`: the session and call it is, the
/// directories mounted and the hosts it may reach. Built only here, from a `Decision`. /// tool, its arguments, the directories mounted and the hosts it may reach. Built only here, from a
/// `Decision`.
#[derive(Debug)] #[derive(Debug)]
pub struct RunSpec { pub struct RunSpec {
session: SessionId,
call: CallId,
tool: ToolName, tool: ToolName,
arguments: ToolArgs, arguments: ToolArgs,
mounts: Vec<Mount>, mounts: Vec<Mount>,
@@ -40,6 +45,12 @@ pub struct RunSpec {
} }
impl RunSpec { impl RunSpec {
pub fn session(&self) -> &SessionId {
&self.session
}
pub fn call(&self) -> CallId {
self.call
}
pub fn tool(&self) -> ToolName { pub fn tool(&self) -> ToolName {
self.tool self.tool
} }
@@ -105,9 +116,11 @@ pub fn run(decision: Decision, runtime: &dyn Runtime) -> proto::ToolResponse {
.collect(), .collect(),
None, None,
), ),
ToolArgs::HttpFetch { .. } => (Vec::new(), Some(decision.hosts().to_vec())), ToolArgs::HttpFetch(_) => (Vec::new(), Some(decision.hosts().to_vec())),
}; };
let spec = RunSpec { let spec = RunSpec {
session: decision.request().session.clone(),
call: decision.request().call,
tool: args.tool(), tool: args.tool(),
arguments: args.clone(), arguments: args.clone(),
mounts, mounts,
+9 -9
View File
@@ -289,16 +289,16 @@ fn each_tool_parses_its_own_arguments() {
cwd: Some("/home/kyle".to_string()) cwd: Some("/home/kyle".to_string())
}) })
); );
assert_eq!( // A `FetchUrl` cannot be built outside `args`, so the parsed value is read through its getters.
parse( let fetched = parse(
ToolName::HttpFetch, ToolName::HttpFetch,
r#"{"url":"https://www.example.com/a"}"# r#"{"url":"https://www.example.com/a"}"#,
),
Ok(ToolArgs::HttpFetch {
url: "https://www.example.com/a".to_string(),
host: "www.example.com".to_string()
})
); );
let Ok(ToolArgs::HttpFetch(target)) = fetched else {
panic!("{fetched:?}")
};
assert_eq!(target.url(), "https://www.example.com/a");
assert_eq!(target.host(), "www.example.com");
// Field order and white space in the request do not matter. // Field order and white space in the request do not matter.
assert_eq!( assert_eq!(
parse( parse(
+140
View File
@@ -0,0 +1,140 @@
//! `[runner]` in `brokerd.toml` (M3b spec, section 6). Do not edit.
use std::path::{Path, PathBuf};
use std::time::Duration;
use brokerd::args::ToolName;
use brokerd::config::{Config, ConfigError};
const DIGEST: &str = "localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef";
fn fixture(name: &str) -> PathBuf {
Path::new(env!("CARGO_MANIFEST_DIR"))
.join("tests/fixtures/config")
.join(name)
}
fn invalid(name: &str) -> String {
match Config::load(&fixture(name)) {
Err(ConfigError::Invalid(_, why)) => why,
other => panic!("{name}: expected Invalid, got {other:?}"),
}
}
#[test]
fn without_a_runner_section_there_is_no_runner() {
assert_eq!(Config::load(&fixture("empty.toml")).unwrap().runner, None);
}
#[test]
fn a_runner_with_only_an_image_gets_every_default() {
let r = Config::load(&fixture("runner_minimal.toml"))
.unwrap()
.runner
.unwrap();
assert_eq!(r.podman, PathBuf::from("podman"));
assert_eq!(r.image, DIGEST);
assert_eq!(r.egress_network, "pasta");
assert_eq!(r.output_cap, 262_144);
assert_eq!(r.memory, "512m");
assert_eq!(r.pids, 128);
assert_eq!(
r.time_limit(ToolName::ReadFile),
Duration::from_millis(30_000)
);
assert_eq!(
r.time_limit(ToolName::WriteFile),
Duration::from_millis(30_000)
);
assert_eq!(
r.time_limit(ToolName::Shell),
Duration::from_millis(100_000)
);
assert_eq!(
r.time_limit(ToolName::HttpFetch),
Duration::from_millis(60_000)
);
}
#[test]
fn every_runner_value_can_be_set() {
let r = Config::load(&fixture("runner_full.toml"))
.unwrap()
.runner
.unwrap();
assert_eq!(r.podman, PathBuf::from("/run/current-system/sw/bin/podman"));
assert_eq!(r.egress_network, "slirp4netns");
assert_eq!((r.output_cap, r.memory.as_str(), r.pids), (1000, "1g", 64));
let limits: Vec<Duration> = ToolName::ALL.iter().map(|t| r.time_limit(*t)).collect();
assert_eq!(limits, [1, 2, 3, 4].map(Duration::from_millis));
}
#[test]
fn a_runner_needs_an_image_named_by_digest() {
assert!(matches!(
Config::load(&fixture("runner_no_image.toml")),
Err(ConfigError::Parse(..))
));
assert!(invalid("runner_tag.toml").contains("by digest"));
}
#[test]
fn unknown_runner_keys_are_errors() {
assert!(matches!(
Config::load(&fixture("runner_unknown_key.toml")),
Err(ConfigError::Parse(..))
));
}
#[test]
fn bad_values_are_errors_that_name_them() {
let dir = std::env::temp_dir().join(format!("bx-runner-cfg-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
let cases = [
(
format!("image = \"x@sha256:{}\"", "0".repeat(63)),
"by digest",
),
(
format!("image = \"x@sha256:{}\"", "A".repeat(64)),
"by digest",
),
(
format!("image = \"@sha256:{}\"", "0".repeat(64)),
"by digest",
),
(format!("image = \"{DIGEST}\"\nmemory = \"lots\""), "memory"),
(format!("image = \"{DIGEST}\"\nmemory = \"512\""), "memory"),
(format!("image = \"{DIGEST}\"\nmemory = \"m\""), "memory"),
(
format!("image = \"{DIGEST}\"\noutput_cap = 0"),
"output_cap",
),
(format!("image = \"{DIGEST}\"\npids = 0"), "pids"),
(format!("image = \"{DIGEST}\"\nshell_ms = 0"), "shell_ms"),
(
format!("image = \"{DIGEST}\"\nhttp_fetch_ms = 0"),
"http_fetch_ms",
),
(
format!("image = \"{DIGEST}\"\negress_network = \"\""),
"egress_network",
),
(format!("image = \"{DIGEST}\"\npodman = \"\""), "podman"),
];
for (n, (body, word)) in cases.iter().enumerate() {
let path = dir.join(format!("c{n}.toml"));
std::fs::write(&path, format!("[runner]\n{body}\n")).unwrap();
match Config::load(&path) {
Err(ConfigError::Invalid(_, why)) => assert!(why.contains(word), "{body}: {why}"),
other => panic!("{body}: expected Invalid, got {other:?}"),
}
}
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn the_egress_directory_is_under_home() {
let c = Config::parse("[paths]\nhome = \"/h\"\n").unwrap();
assert_eq!(c.egress_dir(), PathBuf::from("/h/run/egress"));
}
+271
View File
@@ -0,0 +1,271 @@
//! The Podman runtime against a fake `podman`: what it is given, and what each way a container can
//! end becomes (M3b spec, section 6). Every call goes through `runner::run`, as in `brokerd`.
//! Do not edit.
#[path = "support/build.rs"]
mod build;
#[path = "support/fake_podman.rs"]
mod fake_podman;
use std::time::{Duration, Instant};
use brokerd::container::{
CANNOT_START, COULD_NOT_RUN, KILLED, Podman, RUNBOOK, TIMED_OUT, UNEXPECTED,
};
use brokerd::policy::{Outcome, SessionState, decide};
use brokerd::runner::run;
use build::{grant, now, read, request, set};
use fake_podman::{Fake, Lines, serial};
use proto::{DataClass, Mode, ToolRequest, ToolResponse};
fn call(podman: &Podman, req: ToolRequest, grants: Vec<build::Build>) -> ToolResponse {
let decision = match decide(req, &set(grants), SessionState::default(), now()) {
Outcome::Allowed(d) => d,
other => panic!("not allowed: {other:?}"),
};
run(decision, podman)
}
fn notes() -> Vec<build::Build> {
vec![grant("notes", "read_file", Mode::Auto).paths(&["/n"])]
}
fn podman(fake: &Fake, extra: &str, log: &Lines) -> Podman {
Podman::new(fake.runner(extra), fake.dir.join("egress"), log.sink())
}
fn failed(message: &str) -> ToolResponse {
ToolResponse::Failed {
message: message.to_string(),
}
}
#[test]
fn a_tool_that_succeeds_is_a_result_labelled_by_its_grant() {
let _s = serial();
let fake = Fake::new("ok", r#"cat > "$D/stdin"; printf 'the file text'; exit 0"#);
let log = Lines::default();
let got = call(&podman(&fake, "", &log), read("/n/a.md"), notes());
assert_eq!(
got,
ToolResponse::Result {
content: "the file text".to_string(),
class: DataClass::Private,
untrusted: true,
truncated: false,
}
);
assert_eq!(
fake.stdin(),
r#"{"path":"/n/a.md"}"#,
"the arguments go on standard input"
);
}
#[test]
fn podman_is_given_the_tool_argument_list_and_the_first_container_is_numbered_0() {
let _s = serial();
let fake = Fake::new("args", r#"cat > /dev/null; exit 0"#);
let log = Lines::default();
let p = podman(&fake, "", &log);
call(&p, read("/n/a.md"), notes());
call(&p, read("/n/b.md"), notes());
let calls = fake.calls();
assert_eq!(
calls.len(),
2,
"one podman run per call and nothing else: {calls:?}"
);
assert_eq!(calls[0][3], "--name=boxmaker-s1-1-0");
assert_eq!(calls[1][3], "--name=boxmaker-s1-1-1");
// The whole list is `podman::tool_args`, tested as golden files in podman_args.rs.
assert_eq!(calls[0].first().map(String::as_str), Some("run"));
assert_eq!(calls[0].last().map(String::as_str), Some("read_file"));
assert!(calls[0].contains(&"--volume=/n:/n:ro".to_string()));
}
#[test]
fn exit_1_is_the_tools_own_error_and_still_a_result() {
let _s = serial();
let fake = Fake::new(
"e1",
"cat > /dev/null; printf 'read_file: /n/x: no such file'; exit 1",
);
let log = Lines::default();
let got = call(&podman(&fake, "", &log), read("/n/x"), notes());
assert!(
matches!(&got, ToolResponse::Result { content, .. } if content == "read_file: /n/x: no such file"),
"{got:?}"
);
}
#[test]
fn every_other_ending_is_a_fixed_sentence() {
let cases = [
("2", COULD_NOT_RUN),
("125", CANNOT_START),
("126", CANNOT_START),
("127", CANNOT_START),
("137", KILLED),
("3", UNEXPECTED),
("124", UNEXPECTED),
];
for (code, sentence) in cases {
let _s = serial();
let body = format!(
"cat > /dev/null; printf 'secret tool output'; echo 'podman said this' >&2; exit {code}"
);
let fake = Fake::new("codes", &body);
let log = Lines::default();
let got = call(&podman(&fake, "", &log), read("/n/a"), notes());
assert_eq!(got, failed(sentence), "exit {code}");
}
}
#[test]
fn a_podman_failure_is_logged_with_the_runbook_pointer_and_its_stderr() {
let _s = serial();
let fake = Fake::new(
"125",
"cat > /dev/null; echo 'Error: image not known' >&2; exit 125",
);
let log = Lines::default();
call(&podman(&fake, "", &log), read("/n/a"), notes());
let text = log.all();
assert!(text.contains("Error: image not known"), "{text}");
assert!(text.contains(RUNBOOK), "{text}");
}
#[test]
fn a_podman_that_cannot_be_started_is_unavailable_and_logged() {
let _s = serial();
let fake = Fake::new("missing", "exit 0");
let mut runner = fake.runner("");
runner.podman = fake.dir.join("no-such-podman");
let log = Lines::default();
let p = Podman::new(runner, fake.dir.join("egress"), log.sink());
assert_eq!(call(&p, read("/n/a"), notes()), failed(CANNOT_START));
assert!(log.all().contains(RUNBOOK), "{}", log.all());
}
#[test]
fn output_past_the_cap_is_cut_and_marked() {
let _s = serial();
let fake = Fake::new(
"cap",
"cat > /dev/null; head -c 1000 /dev/zero | tr '\\0' x; exit 0",
);
let log = Lines::default();
let got = call(
&podman(&fake, "output_cap = 100", &log),
read("/n/a"),
notes(),
);
assert!(
matches!(&got, ToolResponse::Result { content, truncated: true, .. } if *content == "x".repeat(100)),
"{got:?}"
);
let exact = Fake::new(
"cap-exact",
"cat > /dev/null; head -c 100 /dev/zero | tr '\\0' x; exit 0",
);
let got = call(
&podman(&exact, "output_cap = 100", &log),
read("/n/a"),
notes(),
);
assert!(
matches!(
&got,
ToolResponse::Result {
truncated: false,
..
}
),
"{got:?}"
);
}
#[test]
fn output_that_is_not_utf8_is_replaced() {
let _s = serial();
let fake = Fake::new("utf8", "cat > /dev/null; printf 'a\\377b'; exit 0");
let log = Lines::default();
let got = call(&podman(&fake, "", &log), read("/n/a"), notes());
assert!(
matches!(&got, ToolResponse::Result { content, .. } if content == "a\u{fffd}b"),
"{got:?}"
);
}
#[test]
fn a_tool_past_its_time_limit_is_killed_removed_and_failed() {
let _s = serial();
// `exec`, so killing the process kills the sleep and nothing holds the pipes open.
let fake = Fake::new("slow", "cat > /dev/null; exec sleep 30");
let log = Lines::default();
let started = Instant::now();
let got = call(
&podman(&fake, "read_file_ms = 300", &log),
read("/n/a"),
notes(),
);
let took = started.elapsed();
assert_eq!(got, failed(TIMED_OUT));
assert!(took >= Duration::from_millis(300), "{took:?}");
assert!(took < Duration::from_secs(5), "{took:?}");
let calls = fake.calls();
assert_eq!(calls.len(), 3, "{calls:?}");
assert_eq!(calls[1], ["kill", "boxmaker-s1-1-0"]);
assert_eq!(calls[2], ["rm", "-f", "boxmaker-s1-1-0"]);
}
#[test]
fn a_large_argument_is_written_whole_while_the_tool_reads_it() {
let _s = serial();
let fake = Fake::new("big", r#"cat > "$D/stdin"; printf done; exit 0"#);
let log = Lines::default();
let content = "y".repeat(900_000);
let req = request(
"write_file",
&format!(r#"{{"path":"/w/big.txt","content":"{content}"}}"#),
);
let got = call(
&podman(&fake, "", &log),
req,
vec![grant("w", "write_file", Mode::Auto).paths(&["/w"])],
);
assert!(
matches!(&got, ToolResponse::Result { content, .. } if content == "done"),
"{got:?}"
);
assert_eq!(
fake.stdin().len(),
content.len() + r#"{"path":"/w/big.txt","content":""}"#.len()
);
}
#[test]
fn a_tool_that_never_reads_its_input_still_ends() {
let _s = serial();
let fake = Fake::new("noread", "printf ignored; exit 0");
let log = Lines::default();
let req = request(
"write_file",
&format!(
r#"{{"path":"/w/big.txt","content":"{}"}}"#,
"z".repeat(900_000)
),
);
let started = Instant::now();
let got = call(
&podman(&fake, "write_file_ms = 5000", &log),
req,
vec![grant("w", "write_file", Mode::Auto).paths(&["/w"])],
);
assert!(
matches!(&got, ToolResponse::Result { content, .. } if content == "ignored"),
"{got:?}"
);
assert!(started.elapsed() < Duration::from_secs(4));
}
+213
View File
@@ -0,0 +1,213 @@
//! `http_fetch` through the Podman runtime: the proxy is started first, its socket awaited, the tool
//! run with the directory mounted, and the proxy and directory removed afterwards on every path
//! (M3b spec, section 6, "http_fetch"). Against a fake `podman`. Do not edit.
#[path = "support/build.rs"]
mod build;
#[path = "support/fake_podman.rs"]
mod fake_podman;
use std::os::unix::fs::PermissionsExt;
use std::time::{Duration, Instant};
use brokerd::container::{CANNOT_START, Podman, RUNBOOK, TIMED_OUT};
use brokerd::policy::{Outcome, SessionState, decide};
use brokerd::runner::run;
use build::{fetch, grant, now, set};
use fake_podman::{Fake, Lines, path, serial};
use proto::{Mode, ToolResponse};
/// For `run -d` (the proxy): make the socket file in the mounted directory, as the proxy does.
const PROXY_OK: &str = r#"for a in "$@"; do case "$a" in --volume=*:/run/egress:rw) v=${a#--volume=}; v=${v%:/run/egress:rw};; esac; done"#;
fn body(proxy: &str, tool: &str) -> String {
format!("{PROXY_OK}\nif [ \"$2\" = -d ]; then\n{proxy}\nfi\n{tool}")
}
fn call(podman: &Podman) -> ToolResponse {
let grants = set(vec![
grant("web", "http_fetch", Mode::Auto).hosts(&["example.com", "*.example.org"]),
]);
let decision = match decide(
fetch("https://example.com/a"),
&grants,
SessionState::default(),
now(),
) {
Outcome::Allowed(d) => d,
other => panic!("not allowed: {other:?}"),
};
run(decision, podman)
}
fn failed(message: &str) -> ToolResponse {
ToolResponse::Failed {
message: message.to_string(),
}
}
#[test]
fn the_proxy_starts_first_the_tool_gets_its_socket_and_both_are_cleaned_up() {
let _s = serial();
let fake = Fake::new(
"eg-ok",
&body(
r#": > "$v/egress.sock"; exit 0"#,
r#"cat > "$D/stdin"; printf 'body\n[http 200]'; exit 0"#,
),
);
let log = Lines::default();
let egress = fake.dir.join("egress");
let got = call(&Podman::new(fake.runner(""), egress.clone(), log.sink()));
assert!(
matches!(&got, ToolResponse::Result { content, .. } if content == "body\n[http 200]"),
"{got:?} {}",
log.all()
);
let calls = fake.calls();
assert_eq!(calls.len(), 3, "{calls:?}");
let dir = egress.join("boxmaker-s1-1-0");
// 1. The proxy, detached, with the call's hosts.
assert_eq!(&calls[0][..2], ["run", "-d"]);
assert!(calls[0].contains(&"--name=boxmaker-s1-1-0-egress".to_string()));
assert!(calls[0].contains(&format!("--volume={}:/run/egress:rw", path(&dir))));
assert_eq!(calls[0].last().unwrap(), "example.com,*.example.org");
// 2. The tool, with the same directory and no network.
assert_eq!(&calls[1][..3], ["run", "--rm", "-i"]);
assert!(calls[1].contains(&format!("--volume={}:/run/egress:rw", path(&dir))));
assert!(calls[1].contains(&"--network=none".to_string()));
assert_eq!(fake.stdin(), r#"{"url":"https://example.com/a"}"#);
// 3. The proxy removed, and its directory with it.
assert_eq!(calls[2], ["rm", "-f", "boxmaker-s1-1-0-egress"]);
assert!(!dir.exists(), "the call's directory is removed");
let mode = std::fs::metadata(&egress).unwrap().permissions().mode() & 0o777;
assert_eq!(mode, 0o700);
}
#[test]
fn a_proxy_that_podman_cannot_start_means_no_tool_and_nothing_left() {
let _s = serial();
let fake = Fake::new(
"eg-fail",
&body(
"echo 'Error: network pasta not found' >&2; exit 125",
"cat > /dev/null; exit 0",
),
);
let log = Lines::default();
let egress = fake.dir.join("egress");
let got = call(&Podman::new(fake.runner(""), egress.clone(), log.sink()));
assert_eq!(got, failed(CANNOT_START));
let calls = fake.calls();
assert!(
calls
.iter()
.all(|c| c.get(2).map(String::as_str) != Some("-i")),
"no tool ran: {calls:?}"
);
assert_eq!(
calls.last().unwrap(),
&["rm", "-f", "boxmaker-s1-1-0-egress"]
);
assert!(!egress.join("boxmaker-s1-1-0").exists());
assert!(
log.all().contains("network pasta not found"),
"{}",
log.all()
);
assert!(log.all().contains(RUNBOOK));
}
#[test]
fn a_proxy_that_makes_no_socket_in_time_means_no_tool_and_nothing_left() {
let _s = serial();
let fake = Fake::new("eg-nosock", &body("exit 0", "cat > /dev/null; exit 0"));
let log = Lines::default();
let egress = fake.dir.join("egress");
let podman = Podman::new(fake.runner(""), egress.clone(), log.sink())
.with_egress_wait(Duration::from_millis(200));
let started = Instant::now();
assert_eq!(call(&podman), failed(CANNOT_START));
assert!(started.elapsed() < Duration::from_secs(3));
let calls = fake.calls();
assert_eq!(calls.len(), 2, "{calls:?}");
assert_eq!(calls[1], ["rm", "-f", "boxmaker-s1-1-0-egress"]);
assert!(!egress.join("boxmaker-s1-1-0").exists());
assert!(log.all().contains(RUNBOOK));
}
#[test]
fn a_tool_that_runs_too_long_still_leaves_nothing_behind() {
let _s = serial();
let fake = Fake::new(
"eg-slow",
&body(
r#": > "$v/egress.sock"; exit 0"#,
"cat > /dev/null; exec sleep 30",
),
);
let log = Lines::default();
let egress = fake.dir.join("egress");
let got = call(&Podman::new(
fake.runner("http_fetch_ms = 300"),
egress.clone(),
log.sink(),
));
assert_eq!(got, failed(TIMED_OUT));
let calls = fake.calls();
let tail: Vec<Vec<String>> = calls[2..].to_vec();
assert_eq!(
tail,
[
vec!["kill", "boxmaker-s1-1-0"],
vec!["rm", "-f", "boxmaker-s1-1-0"],
vec!["rm", "-f", "boxmaker-s1-1-0-egress"],
]
.map(|c| c.into_iter().map(String::from).collect::<Vec<_>>())
);
assert!(!egress.join("boxmaker-s1-1-0").exists());
}
#[test]
fn a_tool_that_podman_cannot_start_still_leaves_nothing_behind() {
let _s = serial();
let fake = Fake::new(
"eg-toolfail",
&body(
r#": > "$v/egress.sock"; exit 0"#,
"cat > /dev/null; exit 125",
),
);
let log = Lines::default();
let egress = fake.dir.join("egress");
assert_eq!(
call(&Podman::new(fake.runner(""), egress.clone(), log.sink())),
failed(CANNOT_START)
);
assert_eq!(
fake.calls().last().unwrap(),
&["rm", "-f", "boxmaker-s1-1-0-egress"]
);
assert!(!egress.join("boxmaker-s1-1-0").exists());
}
#[test]
fn a_directory_left_by_a_crash_is_replaced() {
let _s = serial();
let fake = Fake::new(
"eg-stale",
&body(
r#"[ -e "$v/old.sock" ] && exit 9; : > "$v/egress.sock"; exit 0"#,
"cat > /dev/null; printf ok; exit 0",
),
);
let log = Lines::default();
let egress = fake.dir.join("egress");
std::fs::create_dir_all(egress.join("boxmaker-s1-1-0")).unwrap();
std::fs::write(egress.join("boxmaker-s1-1-0/old.sock"), "").unwrap();
let got = call(&Podman::new(fake.runner(""), egress.clone(), log.sink()));
assert!(
matches!(&got, ToolResponse::Result { content, .. } if content == "ok"),
"{got:?}"
);
}
+86
View File
@@ -0,0 +1,86 @@
//! After a container ends, its output is collected within a grace period, never waited on for
//! ever: something outside the container that still holds a pipe must not hold `brokerd` (M3b
//! review finding 4). Against a fake `podman` whose shell leaves a background `sleep` holding the
//! pipes, which real Podman does not do. Do not edit.
#[path = "support/build.rs"]
mod build;
#[path = "support/fake_podman.rs"]
mod fake_podman;
use std::time::{Duration, Instant};
use brokerd::container::{OUTPUT_OPEN, Podman, TIMED_OUT};
use brokerd::pipes::GRACE;
use brokerd::policy::{Outcome, SessionState, decide};
use brokerd::runner::run;
use build::{grant, now, read, set};
use fake_podman::{Fake, Lines, serial};
use proto::{Mode, ToolResponse};
fn call(fake: &Fake, extra: &str, log: &Lines) -> (ToolResponse, Duration) {
let podman = Podman::new(fake.runner(extra), fake.dir.join("egress"), log.sink());
let grants = set(vec![grant("notes", "read_file", Mode::Auto).paths(&["/n"])]);
let decision = match decide(read("/n/a"), &grants, SessionState::default(), now()) {
Outcome::Allowed(d) => d,
other => panic!("not allowed: {other:?}"),
};
let started = Instant::now();
let got = run(decision, &podman);
(got, started.elapsed())
}
#[test]
fn the_grace_period_is_two_seconds() {
assert_eq!(GRACE, Duration::from_secs(2));
}
#[test]
fn output_held_open_after_the_container_ended_is_abandoned_after_the_grace_period() {
let _s = serial();
// The shell exits at once; the background sleep keeps standard output and error open.
let fake = Fake::new("grace-open", "cat > /dev/null; printf ok; sleep 6 & exit 0");
let log = Lines::default();
let (got, took) = call(&fake, "", &log);
assert_eq!(
got,
ToolResponse::Failed {
message: OUTPUT_OPEN.to_string()
}
);
assert!(took >= GRACE, "{took:?}");
assert!(took < GRACE + Duration::from_secs(2), "{took:?}");
assert!(log.all().contains("abandoned"), "{}", log.all());
}
#[test]
fn a_tool_past_its_limit_is_answered_within_the_grace_period_even_if_its_pipes_stay_open() {
let _s = serial();
// No `exec`: killing the shell leaves the sleep holding the pipes.
let fake = Fake::new("grace-slow", "cat > /dev/null; sleep 6");
let log = Lines::default();
let (got, took) = call(&fake, "read_file_ms = 300", &log);
assert_eq!(
got,
ToolResponse::Failed {
message: TIMED_OUT.to_string()
}
);
assert!(
took < Duration::from_millis(300) + GRACE + Duration::from_secs(2),
"{took:?}"
);
}
#[test]
fn a_tool_that_ends_normally_is_not_slowed_by_the_grace_period() {
let _s = serial();
let fake = Fake::new("grace-ok", "cat > /dev/null; printf done; exit 0");
let log = Lines::default();
let (got, took) = call(&fake, "", &log);
assert!(
matches!(&got, ToolResponse::Result { content, .. } if content == "done"),
"{got:?}"
);
assert!(took < Duration::from_secs(1), "{took:?}");
}
+128
View File
@@ -0,0 +1,128 @@
//! What a tool or Podman writes on standard error reaches `brokerd`'s log escaped, one entry per
//! event: it cannot start a line of its own or forge a runbook pointer (M3b review finding 5).
//! Do not edit.
#[path = "support/build.rs"]
mod build;
#[path = "support/fake_podman.rs"]
mod fake_podman;
use brokerd::container::{CANNOT_START, COULD_NOT_RUN, Podman, UNEXPECTED};
use brokerd::policy::{Outcome, SessionState, decide};
use brokerd::runner::run;
use build::{fetch, grant, now, read, set};
use fake_podman::{Fake, Lines, serial};
use proto::{Mode, ToolRequest, ToolResponse};
const FORGED: &str = "real line\nbrokerd: forged\nsee docs/runbook.md#grants-invalid";
fn call(fake: &Fake, req: ToolRequest, grants: Vec<build::Build>, log: &Lines) -> ToolResponse {
let podman = Podman::new(fake.runner(""), fake.dir.join("egress"), log.sink());
let decision = match decide(req, &set(grants), SessionState::default(), now()) {
Outcome::Allowed(d) => d,
other => panic!("not allowed: {other:?}"),
};
run(decision, &podman)
}
/// No entry holds the forged text as lines of its own; the one that carries it has it escaped.
fn escaped(log: &Lines) {
let entries = log.0.lock().unwrap().clone();
for entry in &entries {
assert!(!entry.contains("\nbrokerd: forged"), "raw: {entry:?}");
assert!(
!entry.contains("\nsee docs/runbook.md#grants-invalid"),
"raw: {entry:?}"
);
}
assert!(
entries
.iter()
.any(|e| e.contains(r"real line\nbrokerd: forged")),
"the error is still logged, escaped: {entries:?}"
);
}
fn notes() -> Vec<build::Build> {
vec![grant("notes", "read_file", Mode::Auto).paths(&["/n"])]
}
#[test]
fn a_tool_that_could_not_run() {
let _s = serial();
let fake = Fake::new(
"log-2",
&format!("cat > /dev/null; printf '{FORGED}' >&2; exit 2"),
);
let log = Lines::default();
assert_eq!(
call(&fake, read("/n/a"), notes(), &log),
ToolResponse::Failed {
message: COULD_NOT_RUN.to_string()
}
);
escaped(&log);
}
#[test]
fn a_container_podman_could_not_start_keeps_its_one_real_pointer() {
let _s = serial();
let fake = Fake::new(
"log-125",
&format!("cat > /dev/null; printf '{FORGED}' >&2; exit 125"),
);
let log = Lines::default();
assert_eq!(
call(&fake, read("/n/a"), notes(), &log),
ToolResponse::Failed {
message: CANNOT_START.to_string()
}
);
escaped(&log);
let entries = log.0.lock().unwrap().clone();
assert!(
entries
.iter()
.any(|e| e.ends_with("\nsee docs/runbook.md#runner-unavailable")),
"{entries:?}"
);
}
#[test]
fn an_unexpected_ending() {
let _s = serial();
let fake = Fake::new(
"log-3",
&format!("cat > /dev/null; printf '{FORGED}' >&2; exit 3"),
);
let log = Lines::default();
assert_eq!(
call(&fake, read("/n/a"), notes(), &log),
ToolResponse::Failed {
message: UNEXPECTED.to_string()
}
);
escaped(&log);
}
#[test]
fn a_proxy_podman_could_not_start() {
let _s = serial();
let body =
format!("if [ \"$2\" = -d ]; then printf '{FORGED}' >&2; exit 125; fi; cat > /dev/null");
let fake = Fake::new("log-egress", &body);
let log = Lines::default();
let got = call(
&fake,
fetch("https://example.com/"),
vec![grant("web", "http_fetch", Mode::Auto).hosts(&["example.com"])],
&log,
);
assert_eq!(
got,
ToolResponse::Failed {
message: CANNOT_START.to_string()
}
);
escaped(&log);
}
+11
View File
@@ -0,0 +1,11 @@
[runner]
podman = "/run/current-system/sw/bin/podman"
image = "localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
egress_network = "slirp4netns"
output_cap = 1000
memory = "1g"
pids = 64
read_file_ms = 1
write_file_ms = 2
shell_ms = 3
http_fetch_ms = 4
@@ -0,0 +1,2 @@
[runner]
image = "localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
@@ -0,0 +1,2 @@
[runner]
podman = "podman"
+2
View File
@@ -0,0 +1,2 @@
[runner]
image = "localhost/boxmaker-tools:latest"
@@ -0,0 +1,3 @@
[runner]
image = "localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
network = "host"
+21
View File
@@ -0,0 +1,21 @@
run
-d
--rm
--name=boxmaker-s1-1-7-egress
--label=boxmaker=egress
--network=pasta
--pull=never
--read-only
--cap-drop=all
--security-opt=no-new-privileges
--userns=keep-id
--pids-limit=64
--memory=128m
--volume=/h/run/egress/boxmaker-s1-1-7:/run/egress:rw
localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
/bin/toolkit
egress-proxy
--socket
/run/egress/egress.sock
--allow
example.com,*.example.org
+18
View File
@@ -0,0 +1,18 @@
run
--rm
-i
--name=boxmaker-s1-1-7
--label=boxmaker=tool
--network=none
--pull=never
--read-only
--cap-drop=all
--security-opt=no-new-privileges
--userns=keep-id
--pids-limit=128
--memory=512m
--tmpfs=/tmp:rw,size=64m,mode=1777
--volume=/h/run/egress/boxmaker-s1-1-7:/run/egress:rw
localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
/bin/toolkit
http_fetch
+18
View File
@@ -0,0 +1,18 @@
run
--rm
-i
--name=boxmaker-s1-1-7
--label=boxmaker=tool
--network=none
--pull=never
--read-only
--cap-drop=all
--security-opt=no-new-privileges
--userns=keep-id
--pids-limit=128
--memory=512m
--tmpfs=/tmp:rw,size=64m,mode=1777
--volume=/home/kyle/notes:/home/kyle/notes:ro
localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
/bin/toolkit
read_file
+19
View File
@@ -0,0 +1,19 @@
run
--rm
-i
--name=boxmaker-s1-1-7
--label=boxmaker=tool
--network=none
--pull=never
--read-only
--cap-drop=all
--security-opt=no-new-privileges
--userns=keep-id
--pids-limit=128
--memory=512m
--tmpfs=/tmp:rw,size=64m,mode=1777
--volume=/srv/a:/srv/a:rw
--volume=/srv/b:/srv/b:rw
localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
/bin/toolkit
shell
@@ -0,0 +1,17 @@
run
--rm
-i
--name=boxmaker-s1-1-7
--label=boxmaker=tool
--network=none
--pull=never
--read-only
--cap-drop=all
--security-opt=no-new-privileges
--userns=keep-id
--pids-limit=128
--memory=512m
--tmpfs=/tmp:rw,size=64m,mode=1777
localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
/bin/toolkit
shell
+18
View File
@@ -0,0 +1,18 @@
run
--rm
-i
--name=boxmaker-s1-1-7
--label=boxmaker=tool
--network=none
--pull=never
--read-only
--cap-drop=all
--security-opt=no-new-privileges
--userns=keep-id
--pids-limit=128
--memory=512m
--tmpfs=/tmp:rw,size=64m,mode=1777
--volume=/home/kyle/out:/home/kyle/out:rw
localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
/bin/toolkit
write_file
+45
View File
@@ -0,0 +1,45 @@
//! A grant path is mounted into the tool container as `--volume=<path>:<path>:ro`, so a path with
//! `:` or `,` in it cannot be granted: the set is invalid, as for any other bad grant (M3b spec,
//! section 3). Do not edit.
#[path = "support/tmp.rs"]
mod tmp;
use brokerd::grants::load;
use tmp::TempDir;
fn grant_with_path(path: &str) -> String {
format!(
"tool = \"read_file\"\nmode = \"auto\"\nmax_taint = \"secret\"\n[constraints]\npaths = [{path:?}]\n"
)
}
#[test]
fn a_path_with_a_colon_or_a_comma_makes_the_set_invalid() {
for path in ["/home/kyle/a:b", "/home/kyle/a,b", "/x:/y", "/n,ro"] {
let dir = TempDir::new("mount-bad");
dir.write("notes.toml", &grant_with_path(path));
let problems = load(dir.path()).expect_err(path);
assert_eq!(problems.len(), 1, "{path}: {problems:?}");
assert_eq!(problems[0].file, "notes.toml");
assert!(
problems[0].problem.contains("cannot be mounted"),
"{path}: {}",
problems[0].problem
);
}
}
#[test]
fn other_punctuation_is_still_fine() {
for path in [
"/home/kyle/a b",
"/home/kyle/a;b",
"/home/kyle/a=b",
"/home/kyle/a.b-c_d",
] {
let dir = TempDir::new("mount-ok");
dir.write("notes.toml", &grant_with_path(path));
assert!(load(dir.path()).is_ok(), "{path}");
}
}
+80
View File
@@ -0,0 +1,80 @@
//! Two small texts from the M3b review: every line `brokerd serve` prints about its runtime starts
//! with `brokerd:`, and a bad `[runner]` value is quoted in its error. Do not edit.
#[path = "support/fake_podman.rs"]
mod fake_podman;
use std::io::Read;
use std::os::unix::net::UnixStream;
use std::process::{Command, Stdio};
use std::time::{Duration, Instant};
use brokerd::config::{Config, ConfigError};
use fake_podman::{Fake, IMAGE, serial};
#[test]
fn the_runtime_notice_starts_with_brokerd() {
let _s = serial();
let fake = Fake::new("notice", "exit 0");
let home = fake.dir.join("home");
std::fs::create_dir_all(home.join("grants")).unwrap();
let config = fake.dir.join("brokerd.toml");
std::fs::write(
&config,
format!(
"[paths]\nhome = \"{0}\"\ngrants = \"{0}/grants\"\n[runner]\npodman = \"{1}\"\nimage = \"{IMAGE}\"\n",
home.display(),
fake.script.display()
),
)
.unwrap();
let mut child = Command::new(env!("CARGO_BIN_EXE_brokerd"))
.args(["serve", "--config"])
.arg(&config)
.stderr(Stdio::piped())
.spawn()
.unwrap();
let until = Instant::now() + Duration::from_secs(10);
while UnixStream::connect(home.join("run/loop-broker/broker.sock")).is_err() {
assert!(Instant::now() < until, "brokerd never listened");
std::thread::sleep(Duration::from_millis(20));
}
std::thread::sleep(Duration::from_millis(100));
child.kill().unwrap();
child.wait().unwrap();
let mut printed = String::new();
child
.stderr
.take()
.unwrap()
.read_to_string(&mut printed)
.unwrap();
assert!(
printed
.lines()
.any(|l| l == format!("brokerd: tools run in containers from {IMAGE}")),
"{printed}"
);
}
#[test]
fn a_bad_runner_value_is_quoted() {
let dir = std::env::temp_dir().join(format!("bx-notice-cfg-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
let cases = [
("image = \"not by digest\"", "\"not by digest\""),
(
&*format!("image = \"{IMAGE}\"\nmemory = \"lots\""),
"\"lots\"",
),
];
for (n, (body, quoted)) in cases.iter().enumerate() {
let path = dir.join(format!("q{n}.toml"));
std::fs::write(&path, format!("[runner]\n{body}\n")).unwrap();
match Config::load(&path) {
Err(ConfigError::Invalid(_, why)) => assert!(why.contains(quoted), "{why}"),
other => panic!("{body}: {other:?}"),
}
}
let _ = std::fs::remove_dir_all(&dir);
}
+155
View File
@@ -0,0 +1,155 @@
//! The `podman` argument lists, as golden files: one argument per line, compared exactly. A
//! runtime that only builds the list stands in for Podman, so the lists are built from real
//! `RunSpec`s, which only `runner::run` can make. Do not edit.
#[path = "support/build.rs"]
mod build;
use std::ffi::OsString;
use std::path::{Path, PathBuf};
use std::sync::Mutex;
use brokerd::config::Runner;
use brokerd::podman::{container_name, egress_args, tool_args};
use brokerd::policy::{Outcome, SessionState, decide};
use brokerd::runner::{RunError, RunOutput, RunSpec, Runtime, run};
use build::{fetch, grant, now, read, request, set, shell, write};
use proto::{CallId, Mode, SessionId, ToolRequest};
const IMAGE: &str = "localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef";
fn runner() -> Runner {
let text = format!("[runner]\nimage = \"{IMAGE}\"\n");
brokerd::config::Config::parse(&text)
.unwrap()
.runner
.unwrap()
}
/// Builds the tool's argument list inside `run`, as the real runtime will.
struct Lists {
egress: Option<PathBuf>,
got: Mutex<Vec<Vec<OsString>>>,
}
impl Runtime for Lists {
fn run(&self, spec: &RunSpec) -> Result<RunOutput, RunError> {
let name = container_name(spec.session(), spec.call(), 7);
let args = tool_args(spec, &runner(), &name, self.egress.as_deref());
self.got.lock().unwrap().push(args);
Ok(RunOutput {
content: String::new(),
truncated: false,
})
}
}
fn list_for(grants: Vec<build::Build>, req: ToolRequest, egress: Option<&Path>) -> Vec<String> {
let decision = match decide(req, &set(grants), SessionState::default(), now()) {
Outcome::Allowed(d) => d,
other => panic!("not allowed: {other:?}"),
};
let lists = Lists {
egress: egress.map(Path::to_path_buf),
got: Mutex::new(Vec::new()),
};
run(decision, &lists);
let got = lists.got.into_inner().unwrap();
assert_eq!(got.len(), 1);
got[0]
.iter()
.map(|a| a.to_str().unwrap().to_string())
.collect()
}
fn golden(name: &str) -> Vec<String> {
let path = Path::new(env!("CARGO_MANIFEST_DIR"))
.join("tests/fixtures/podman")
.join(name);
let text = std::fs::read_to_string(&path).unwrap_or_else(|e| panic!("{}: {e}", path.display()));
text.lines().map(str::to_string).collect()
}
#[test]
fn the_container_name_says_whose_call_it_is() {
let s = SessionId::new("chat-17").unwrap();
assert_eq!(container_name(&s, CallId(42), 3), "boxmaker-chat-17-42-3");
}
#[test]
fn read_file_gets_its_one_directory_read_only_and_no_network() {
let got = list_for(
vec![grant("notes", "read_file", Mode::Auto).paths(&["/home/kyle/notes"])],
read("/home/kyle/notes/a.md"),
None,
);
assert_eq!(got, golden("read_file.args"));
}
#[test]
fn write_file_gets_its_one_directory_writable() {
let got = list_for(
vec![grant("out", "write_file", Mode::Auto).paths(&["/home/kyle/out"])],
write("/home/kyle/out/b.md"),
None,
);
assert_eq!(got, golden("write_file.args"));
}
#[test]
fn shell_gets_every_grant_directory_writable_in_order() {
let got = list_for(
vec![grant("sh", "shell", Mode::Auto).paths(&["/srv/a", "/srv/b"])],
shell(Some("/srv/b")),
None,
);
assert_eq!(got, golden("shell.args"));
let bare = list_for(
vec![grant("sh", "shell", Mode::Auto)],
request("shell", r#"{"command":"ls"}"#),
None,
);
assert_eq!(bare, golden("shell_no_paths.args"));
}
#[test]
fn http_fetch_gets_the_egress_directory_and_still_no_network() {
let got = list_for(
vec![grant("web", "http_fetch", Mode::Auto).hosts(&["example.com"])],
fetch("https://example.com/a"),
Some(Path::new("/h/run/egress/boxmaker-s1-1-7")),
);
assert_eq!(got, golden("http_fetch.args"));
assert!(got.contains(&"--network=none".to_string()));
}
#[test]
fn the_proxy_gets_a_network_the_socket_and_the_hosts() {
let got: Vec<String> = egress_args(
&runner(),
"boxmaker-s1-1-7",
Path::new("/h/run/egress/boxmaker-s1-1-7"),
&["example.com".to_string(), "*.example.org".to_string()],
)
.iter()
.map(|a| a.to_str().unwrap().to_string())
.collect();
assert_eq!(got, golden("egress.args"));
}
#[test]
fn no_argument_holds_a_shell_string_or_a_second_network() {
let got = list_for(
vec![grant("sh", "shell", Mode::Auto).paths(&["/srv/a"])],
request(
"shell",
r#"{"command":"curl evil.test; rm -rf /","cwd":"/srv/a"}"#,
),
None,
);
assert!(
got.iter().all(|a| !a.contains("evil.test")),
"the command goes on standard input"
);
assert_eq!(got.iter().filter(|a| a.starts_with("--network")).count(), 1);
}
+143
View File
@@ -0,0 +1,143 @@
//! `brokerd serve` with a `[runner]` section runs allowed calls through Podman (here a fake), and
//! says which runtime it uses. Without the section it refuses every call, as in M3a. Do not edit.
#[path = "support/fake_podman.rs"]
mod fake_podman;
use std::io::Read;
use std::os::unix::net::UnixStream;
use std::path::Path;
use std::process::{Child, Command, Stdio};
use std::time::{Duration, Instant};
use fake_podman::{Fake, IMAGE, serial};
use proto::{
CallId, DataClass, Envelope, Message, PROTOCOL_VERSION, SessionId, ToolRequest, ToolResponse,
read_frame, write_frame,
};
struct Running(Child);
impl Drop for Running {
fn drop(&mut self) {
let _ = self.0.kill();
let _ = self.0.wait();
}
}
impl Running {
fn stop(mut self) -> String {
let _ = self.0.kill();
let _ = self.0.wait();
let mut err = String::new();
if let Some(mut stderr) = self.0.stderr.take() {
let _ = stderr.read_to_string(&mut err);
}
err
}
}
fn start(home: &Path, runner: &str) -> Running {
std::fs::create_dir_all(home.join("grants")).unwrap();
std::fs::write(
home.join("grants/notes.toml"),
"tool = \"read_file\"\nmode = \"auto\"\nmax_taint = \"secret\"\n[constraints]\npaths = [\"/n\"]\n",
)
.unwrap();
let config = home.join("brokerd.toml");
std::fs::write(
&config,
format!(
"[paths]\nhome = \"{0}\"\ngrants = \"{0}/grants\"\n{runner}\n",
home.display()
),
)
.unwrap();
let child = Command::new(env!("CARGO_BIN_EXE_brokerd"))
.args(["serve", "--config"])
.arg(&config)
.stdout(Stdio::null())
.stderr(Stdio::piped())
.spawn()
.unwrap();
let running = Running(child);
let until = Instant::now() + Duration::from_secs(10);
while UnixStream::connect(home.join("run/loop-broker/broker.sock")).is_err() {
assert!(Instant::now() < until, "brokerd never listened");
std::thread::sleep(Duration::from_millis(20));
}
running
}
fn read_call(home: &Path) -> ToolResponse {
let mut stream = UnixStream::connect(home.join("run/loop-broker/broker.sock")).unwrap();
stream
.set_read_timeout(Some(Duration::from_secs(10)))
.unwrap();
let request = ToolRequest {
session: SessionId::new("s1").unwrap(),
call: CallId(1),
tool: "read_file".to_string(),
arguments: r#"{"path":"/n/a.md"}"#.to_string(),
};
let envelope = Envelope {
v: PROTOCOL_VERSION,
id: 1,
r#final: true,
msg: Message::ToolRequest(request),
};
write_frame(&mut stream, &envelope).unwrap();
match read_frame(&mut stream).unwrap().msg {
Message::ToolResponse(response) => response,
other => panic!("{other:?}"),
}
}
#[test]
fn with_a_runner_an_allowed_call_runs_in_a_container() {
let _s = serial();
let fake = Fake::new(
"serve",
"cat > /dev/null; printf 'from the container'; exit 0",
);
let home = fake.dir.join("home");
let running = start(
&home,
&format!(
"[runner]\npodman = \"{}\"\nimage = \"{IMAGE}\"\n",
fake.script.display()
),
);
assert_eq!(
read_call(&home),
ToolResponse::Result {
content: "from the container".to_string(),
class: DataClass::Private,
untrusted: true,
truncated: false,
}
);
let printed = running.stop();
assert!(
printed.contains(&format!("tools run in containers from {IMAGE}")),
"{printed}"
);
assert_eq!(fake.calls().len(), 1);
}
#[test]
fn without_a_runner_every_call_is_refused_and_it_says_so() {
let _s = serial();
let fake = Fake::new("serve-none", "exit 0");
let home = fake.dir.join("home");
let running = start(&home, "");
assert_eq!(
read_call(&home),
ToolResponse::Failed {
message: brokerd::runner::REFUSING.to_string()
}
);
let printed = running.stop();
assert!(printed.contains("no [runner] section"), "{printed}");
assert!(fake.calls().is_empty());
}
+102
View File
@@ -0,0 +1,102 @@
//! A fake `podman` for the runtime tests: a shell script that records every call's arguments and,
//! for `run`, does what the test says. Do not edit.
//!
//! Included with `#[path = "support/fake_podman.rs"] mod fake_podman;`.
#![allow(dead_code)] // each test file uses a different part of this module
use std::os::unix::fs::PermissionsExt;
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicU32, Ordering};
use std::sync::{Arc, Mutex, MutexGuard};
use brokerd::config::{Config, Runner};
static NEXT: AtomicU32 = AtomicU32::new(0);
static SERIAL: Mutex<()> = Mutex::new(());
/// Tests that write a script and run it take turns. Otherwise another test's fork can hold the
/// script open for writing at the moment it is run, and running it fails with "text file busy"
/// (ETXTBSY), which has nothing to do with the code under test.
pub fn serial() -> MutexGuard<'static, ()> {
SERIAL.lock().unwrap_or_else(|p| p.into_inner())
}
pub const IMAGE: &str = "localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef";
pub struct Fake {
pub dir: PathBuf,
pub script: PathBuf,
}
impl Drop for Fake {
fn drop(&mut self) {
let _ = std::fs::remove_dir_all(&self.dir);
}
}
impl Fake {
/// A fake whose `run` does `run_body` (a shell fragment; `$D` is the fake's directory). Every
/// other command (`kill`, `rm`) is recorded and succeeds.
pub fn new(tag: &str, run_body: &str) -> Fake {
let n = NEXT.fetch_add(1, Ordering::SeqCst);
let dir = std::env::temp_dir().join(format!("bx-fp-{tag}-{}-{n}", std::process::id()));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).unwrap();
let script = dir.join("podman");
let text = format!(
"#!/bin/sh\nD='{}'\nfor a in \"$@\"; do printf '%s\\n' \"$a\"; done >> \"$D/calls\"\necho --- >> \"$D/calls\"\n[ \"$1\" = run ] || exit 0\n{run_body}\n",
dir.display()
);
std::fs::write(&script, text).unwrap();
std::fs::set_permissions(&script, std::fs::Permissions::from_mode(0o755)).unwrap();
Fake { dir, script }
}
/// Every call so far, each as its arguments.
pub fn calls(&self) -> Vec<Vec<String>> {
let text = std::fs::read_to_string(self.dir.join("calls")).unwrap_or_default();
let mut calls = Vec::new();
let mut current = Vec::new();
for line in text.lines() {
if line == "---" {
calls.push(std::mem::take(&mut current));
} else {
current.push(line.to_string());
}
}
calls
}
/// What the last `run` read on standard input, if the body saved it to `$D/stdin`.
pub fn stdin(&self) -> String {
std::fs::read_to_string(self.dir.join("stdin")).unwrap_or_default()
}
/// A `[runner]` using this fake, with `extra` lines added.
pub fn runner(&self, extra: &str) -> Runner {
let text = format!(
"[runner]\npodman = \"{}\"\nimage = \"{IMAGE}\"\n{extra}\n",
self.script.display()
);
Config::parse(&text).unwrap().runner.unwrap()
}
}
/// A log that keeps its lines.
#[derive(Clone, Default)]
pub struct Lines(pub Arc<Mutex<Vec<String>>>);
impl Lines {
pub fn sink(&self) -> Arc<dyn Fn(&str) + Send + Sync> {
let lines = Arc::clone(&self.0);
Arc::new(move |l: &str| lines.lock().unwrap().push(l.to_string()))
}
pub fn all(&self) -> String {
self.0.lock().unwrap().join("\n")
}
}
pub fn path(p: &Path) -> String {
p.to_str().unwrap().to_string()
}
+1
View File
@@ -40,6 +40,7 @@ impl Rig {
}, },
sockets: Sockets::default(), sockets: Sockets::default(),
approvals: Approvals { ttl_ms }, approvals: Approvals { ttl_ms },
runner: None,
}; };
Rig { Rig {
dir, dir,
+61
View File
@@ -0,0 +1,61 @@
//! Host names and patterns, shared by `brokerd` and the egress proxy so both check hosts with the
//! same rules.
/// A host label is 1 to 63 bytes of `a-z`, `0-9` or `-`, and neither starts nor ends with `-`.
fn valid_label(label: &str) -> bool {
if !(1..=63).contains(&label.len()) {
return false;
}
let bytes = label.as_bytes();
if bytes.iter().next() == Some(&b'-') || bytes.iter().last() == Some(&b'-') {
return false;
}
bytes
.iter()
.all(|&byte| matches!(byte, b'a'..=b'z' | b'0'..=b'9' | b'-'))
}
/// A host is 1 to 253 bytes of dot-separated labels, each a whole non-dot name, and the last label
/// starts with a letter. That last rule keeps out every spelling of an IPv4 address.
pub fn valid_host(host: &str) -> bool {
if !(1..=253).contains(&host.len()) {
return false;
}
let labels: Vec<&str> = host.split('.').collect();
if labels.len() < 2 {
return false;
}
for label in &labels {
if !valid_label(label) {
return false;
}
}
match labels.iter().last() {
Some(last) => matches!(last.bytes().next(), Some(byte) if byte.is_ascii_lowercase()),
None => false,
}
}
/// A host, or `*.` followed by a host. Nothing else.
pub fn valid_host_pattern(pattern: &str) -> bool {
match pattern.strip_prefix("*.") {
Some(base) => valid_host(base),
None => valid_host(pattern),
}
}
/// Does `pattern` match `host`? Without `*.` the strings must be equal; with `*.base` the host must
/// end in `.base` with something before the dot.
pub fn host_matches(pattern: &str, host: &str) -> bool {
let base = match pattern.strip_prefix("*.") {
Some(base) => base,
None => return pattern == host,
};
match host.strip_suffix(base) {
Some(before) => match before.strip_suffix('.') {
Some(prefix) => !prefix.is_empty(),
None => false,
},
None => false,
}
}
+2
View File
@@ -6,8 +6,10 @@ pub mod class;
pub mod frame; pub mod frame;
pub mod grant; pub mod grant;
pub mod hash; pub mod hash;
pub mod hosts;
pub mod ids; pub mod ids;
pub mod log; pub mod log;
pub mod tools;
pub mod wire; pub mod wire;
pub use audit::{ pub use audit::{
+36
View File
@@ -0,0 +1,36 @@
//! The four tools' arguments, decoded from the model's JSON then re-serialised. Shared by
//! `brokerd` (which validates them) and `toolkit` (inside the container, which runs them), so the
//! two must agree on one definition.
use serde::{Deserialize, Serialize};
/// The arguments for `read_file`.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct ReadFileArgs {
pub path: String,
}
/// The arguments for `write_file`.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct WriteFileArgs {
pub path: String,
pub content: String,
}
/// The arguments for `shell`.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct ShellArgs {
pub command: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub cwd: Option<String>,
}
/// The arguments for `http_fetch`.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct HttpFetchArgs {
pub url: String,
}
+50
View File
@@ -0,0 +1,50 @@
//! Host names and patterns, shared by `brokerd` and `toolkit`'s egress proxy. The full tables are
//! in `crates/brokerd/tests/args.rs`, which reaches these through `brokerd::args`. Do not edit.
use proto::hosts::{host_matches, valid_host, valid_host_pattern};
#[test]
fn hosts() {
for good in ["example.com", "a.b.example.com", "x-1.example.org", "a.b"] {
assert!(valid_host(good), "{good}");
}
for bad in [
"",
"example",
"Example.com",
"-a.com",
"a-.com",
"a..com",
".a.com",
"a.com.",
"127.0.0.1",
"127.1",
"1.2.3.4x",
"[::1]",
"a_b.com",
"a.com:443",
"*.a.com",
] {
assert!(!valid_host(bad), "{bad}");
}
}
#[test]
fn patterns() {
assert!(valid_host_pattern("example.com"));
assert!(valid_host_pattern("*.example.com"));
for bad in ["*", "*.", "*.*.a.com", "a.*.com", "**.a.com", "*a.com"] {
assert!(!valid_host_pattern(bad), "{bad}");
}
}
#[test]
fn matching() {
assert!(host_matches("example.com", "example.com"));
assert!(!host_matches("example.com", "www.example.com"));
assert!(host_matches("*.example.com", "www.example.com"));
assert!(host_matches("*.example.com", "a.b.example.com"));
assert!(!host_matches("*.example.com", "example.com"));
assert!(!host_matches("*.example.com", "badexample.com"));
assert!(!host_matches("*.example.com", ".example.com"));
}
+60
View File
@@ -0,0 +1,60 @@
//! The four tools' arguments: `brokerd` writes them to a container's standard input, `toolkit`
//! reads them back, and both use these types. Do not edit.
use proto::tools::{HttpFetchArgs, ReadFileArgs, ShellArgs, WriteFileArgs};
#[test]
fn each_type_round_trips_in_field_order() {
let read = ReadFileArgs {
path: "/n/a.md".to_string(),
};
assert_eq!(
serde_json::to_string(&read).unwrap(),
r#"{"path":"/n/a.md"}"#
);
let write = WriteFileArgs {
path: "/n/a.md".to_string(),
content: "hi\n".to_string(),
};
let text = serde_json::to_string(&write).unwrap();
assert_eq!(text, r#"{"path":"/n/a.md","content":"hi\n"}"#);
assert_eq!(serde_json::from_str::<WriteFileArgs>(&text).unwrap(), write);
let fetch = HttpFetchArgs {
url: "https://example.com/".to_string(),
};
assert_eq!(
serde_json::to_string(&fetch).unwrap(),
r#"{"url":"https://example.com/"}"#
);
}
#[test]
fn a_shell_cwd_is_left_out_when_absent_and_may_be_null_or_missing() {
let bare = ShellArgs {
command: "ls".to_string(),
cwd: None,
};
assert_eq!(serde_json::to_string(&bare).unwrap(), r#"{"command":"ls"}"#);
for text in [r#"{"command":"ls"}"#, r#"{"command":"ls","cwd":null}"#] {
assert_eq!(serde_json::from_str::<ShellArgs>(text).unwrap(), bare);
}
let with = ShellArgs {
command: "ls".to_string(),
cwd: Some("/n".to_string()),
};
assert_eq!(
serde_json::to_string(&with).unwrap(),
r#"{"command":"ls","cwd":"/n"}"#
);
}
#[test]
fn unknown_and_missing_fields_are_refused() {
assert!(serde_json::from_str::<ReadFileArgs>(r#"{"path":"/a","mode":"x"}"#).is_err());
assert!(serde_json::from_str::<ReadFileArgs>(r#"{}"#).is_err());
assert!(serde_json::from_str::<WriteFileArgs>(r#"{"path":"/a"}"#).is_err());
assert!(serde_json::from_str::<ShellArgs>(r#"{"command":"ls","env":{}}"#).is_err());
assert!(
serde_json::from_str::<HttpFetchArgs>(r#"{"url":"https://a.b/","method":"POST"}"#).is_err()
);
}
+2
View File
@@ -10,3 +10,5 @@ workspace = true
[dependencies] [dependencies]
proto.workspace = true proto.workspace = true
serde.workspace = true
serde_json.workspace = true
+94
View File
@@ -0,0 +1,94 @@
//! `is_public`: the egress proxy connects only to public addresses. Everything in the M3b spec,
//! section 5, "Refused ranges" tables is not public; everything else is.
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
/// True if `ip` is a public unicast address.
pub fn is_public(ip: IpAddr) -> bool {
match ip {
IpAddr::V4(v4) => is_public_v4(v4),
IpAddr::V6(v6) => is_public_v6(v6),
}
}
fn is_public_v4(ip: Ipv4Addr) -> bool {
let [a, b, c, _] = ip.octets();
if a == 0 {
return false;
}
if a == 10 {
return false;
}
if a == 100 && (64..=127).contains(&b) {
return false;
}
if a == 127 {
return false;
}
if a == 169 && b == 254 {
return false;
}
if a == 172 && (16..=31).contains(&b) {
return false;
}
if a == 192 && b == 0 && c == 0 {
return false;
}
if a == 192 && b == 0 && c == 2 {
return false;
}
if a == 192 && b == 168 {
return false;
}
if a == 198 && (b == 18 || b == 19) {
return false;
}
if a == 198 && b == 51 && c == 100 {
return false;
}
if a == 203 && b == 0 && c == 113 {
return false;
}
if a >= 224 {
return false;
}
true
}
fn is_public_v6(ip: Ipv6Addr) -> bool {
let s = ip.segments();
if is_ipv4_mapped(s) {
let last32 = (u32::from(s[6]) << 16) | u32::from(s[7]);
return is_public_v4(Ipv4Addr::from(last32));
}
if is_nat64(s) {
let last32 = (u32::from(s[6]) << 16) | u32::from(s[7]);
return is_public_v4(Ipv4Addr::from(last32));
}
if s[0..6].iter().all(|&x| x == 0) {
return false;
}
if s[0] & 0xfe00 == 0xfc00 {
return false;
}
if s[0] & 0xffc0 == 0xfe80 {
return false;
}
if s[0] & 0xff00 == 0xff00 {
return false;
}
if s[0] == 0x2001 && s[1] == 0x0db8 {
return false;
}
true
}
/// `::ffff:0:0/96`: the high 80 bits are zero, then the `ffff` marker.
fn is_ipv4_mapped(s: [u16; 8]) -> bool {
s[0..5].iter().all(|&x| x == 0) && s[5] == 0xffff
}
/// `64:ff9b::/96`: the NAT64 prefix, then 32 zero bits, then the embedded IPv4 address.
fn is_nat64(s: [u16; 8]) -> bool {
s[0] == 0x64 && s[1] == 0xff9b && s[2] == 0 && s[3] == 0 && s[4] == 0 && s[5] == 0
}
+279
View File
@@ -0,0 +1,279 @@
//! The egress proxy: the only way `http_fetch`'s container reaches the network. It speaks the part
//! of SOCKS5 (RFC 1928) that `curl --proxy socks5h://` uses, allows only the call's hosts, only
//! port 443, only public addresses, and after the handshake copies bytes both ways without looking
//! at them. Everything read during the handshake comes from the untrusted side: read exactly what
//! the protocol says, never index or allocate by a number that has not been checked. Spec section 5.
use std::io::{Read, Write};
use std::net::{SocketAddr, TcpStream, ToSocketAddrs};
use std::os::unix::net::{UnixListener, UnixStream};
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::thread::JoinHandle;
use std::time::{Duration, Instant};
pub const HANDSHAKE_TIMEOUT: Duration = Duration::from_secs(10);
pub const CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
pub const MAX_CONNECTIONS: usize = 8;
// SOCKS5 reply codes the proxy sends:
pub const NOT_ALLOWED: u8 = 2;
pub const HOST_UNREACHABLE: u8 = 4;
pub const CONNECTION_REFUSED: u8 = 5;
pub const COMMAND_NOT_SUPPORTED: u8 = 7;
pub const ADDRESS_TYPE_NOT_SUPPORTED: u8 = 8;
/// The call's hosts: one or more patterns separated by ','. An empty piece is an error.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Allow {
patterns: Vec<String>,
}
impl Allow {
/// One or more host patterns separated by ','. Every piece must pass
/// `proto::hosts::valid_host_pattern`; an empty piece (",", "a.com,", "") is an error.
pub fn parse(list: &str) -> Result<Allow, String> {
let mut patterns = Vec::new();
for piece in list.split(',') {
if !proto::hosts::valid_host_pattern(piece) {
return Err(format!("'{piece}' is not a host or a *.host"));
}
patterns.push(piece.to_string());
}
Ok(Allow { patterns })
}
/// `valid_host(host)` and some pattern `host_matches` it.
pub fn permits(&self, host: &str) -> bool {
proto::hosts::valid_host(host)
&& self
.patterns
.iter()
.any(|p| proto::hosts::host_matches(p, host))
}
}
/// Name resolution and connecting, so tests need no network.
pub trait Dial: Send + Sync {
fn resolve(&self, host: &str, port: u16) -> std::io::Result<Vec<SocketAddr>>;
fn connect(&self, addr: SocketAddr, timeout: Duration) -> std::io::Result<TcpStream>;
}
/// The system's: `(host, port).to_socket_addrs()` and `TcpStream::connect_timeout`.
pub struct SystemDial;
impl Dial for SystemDial {
fn resolve(&self, host: &str, port: u16) -> std::io::Result<Vec<SocketAddr>> {
let addrs = format!("{}:{}", host, port).to_socket_addrs()?;
Ok(addrs.collect())
}
fn connect(&self, addr: SocketAddr, timeout: Duration) -> std::io::Result<TcpStream> {
TcpStream::connect_timeout(&addr, timeout)
}
}
pub struct Proxy {
allow: Allow,
dial: Arc<dyn Dial>,
handshake: Duration,
}
impl Proxy {
pub fn new(allow: Allow, dial: Arc<dyn Dial>) -> Proxy {
Proxy {
allow,
dial,
handshake: HANDSHAKE_TIMEOUT,
}
}
pub fn with_handshake_timeout(self, handshake: Duration) -> Proxy {
Proxy { handshake, ..self }
}
/// Accept for ever; at most MAX_CONNECTIONS handled at once, more are closed at once.
pub fn serve(self: Arc<Self>, listener: UnixListener) -> std::io::Result<()> {
let count = Arc::new(AtomicUsize::new(0));
for stream in listener.incoming() {
let stream = stream?;
if count.load(Ordering::SeqCst) >= MAX_CONNECTIONS {
drop(stream);
continue;
}
count.fetch_add(1, Ordering::SeqCst);
let proxy = Arc::clone(&self);
let closer = Arc::clone(&count);
let decrement = Arc::clone(&count);
let worker = match stream.try_clone() {
Ok(w) => w,
Err(_) => {
decrement.fetch_sub(1, Ordering::SeqCst);
drop(stream);
continue;
}
};
match start(move || {
proxy.handle(worker);
closer.fetch_sub(1, Ordering::SeqCst);
}) {
Ok(_) => {}
Err(_) => {
decrement.fetch_sub(1, Ordering::SeqCst);
drop(stream);
}
}
}
Ok(())
}
/// One connection, from greeting to the end of the copy.
pub fn handle(&self, client: UnixStream) {
let mut client = client;
let deadline = Instant::now() + self.handshake;
// 1. version, count. Not SOCKS5 -> stop, no reply.
let greeting = match read_n(&mut client, 2, deadline) {
Some(b) if b[0] == 5 => b,
_ => return,
};
// 2. methods. No auth method 0 (unauthenticated) -> refuse, else accept.
let methods = match read_n(&mut client, greeting[1] as usize, deadline) {
Some(b) => b,
None => return,
};
if methods.contains(&0) {
let _ = client.write_all(&[5, 0]);
} else {
let _ = client.write_all(&[5, 0xff]);
return;
}
// 3. version, command, reserved, kind.
let req = match read_n(&mut client, 4, deadline) {
Some(b) => b,
None => return,
};
if req[0] != 5 || req[2] != 0 {
return;
}
// 4. only connect.
if req[1] != 1 {
return refuse(&mut client, COMMAND_NOT_SUPPORTED);
}
// 5. only domain names; do not read the address.
if req[3] != 3 {
return refuse(&mut client, ADDRESS_TYPE_NOT_SUPPORTED);
}
// 6. name length; zero is not allowed.
let length = match read_n(&mut client, 1, deadline) {
Some(b) => b[0],
None => return,
} as usize;
if length == 0 {
return refuse(&mut client, NOT_ALLOWED);
}
// 7. name, then port.
let name = match read_n(&mut client, length, deadline) {
Some(b) => b,
None => return,
};
let port = match read_n(&mut client, 2, deadline) {
Some(b) => b,
None => return,
};
let port = u16::from_be_bytes([port[0], port[1]]);
// 8. UTF-8, port 443, host allowed.
let name = match std::str::from_utf8(&name) {
Ok(s) => s,
Err(_) => return refuse(&mut client, NOT_ALLOWED),
};
if port != 443 || !self.allow.permits(name) {
return refuse(&mut client, NOT_ALLOWED);
}
// 9. resolve; first public address, others skipped never tried.
let addrs = match self.dial.resolve(name, port) {
Ok(a) => a,
Err(_) => return refuse(&mut client, HOST_UNREACHABLE),
};
let addr = match addrs
.iter()
.find(|a| crate::addr::is_public(a.ip()))
.copied()
{
Some(a) => a,
None => return refuse(&mut client, HOST_UNREACHABLE),
};
// 10. connect.
let server = match self.dial.connect(addr, CONNECT_TIMEOUT) {
Ok(s) => s,
Err(_) => return refuse(&mut client, CONNECTION_REFUSED),
};
// 11. success reply.
if client.write_all(&[5, 0, 0, 1, 0, 0, 0, 0, 0, 0]).is_err() {
return;
}
copy_both_ways(client, server);
}
}
/// Start a thread that never panics on input: `std::thread::Builder`, which returns an error
/// instead of unwinding, unlike `spawn`.
fn start<F>(f: F) -> std::io::Result<JoinHandle<()>>
where
F: FnOnce() + Send + 'static,
{
std::thread::Builder::new()
.name("egress-handle".into())
.spawn(f)
}
/// Refuse: write the SOCKS5 reply and return, which closes the connection.
fn refuse(client: &mut UnixStream, code: u8) {
let _ = client.write_all(&[5, code, 0, 1, 0, 0, 0, 0, 0, 0]);
}
/// Read exactly `n` bytes before `deadline`, never more. The whole handshake shares one deadline:
/// before every read, set the read timeout to the time left; if none is left, or a read times out,
/// fails, or reads 0 bytes (the peer closed), stop: return `None`.
fn read_n(stream: &mut UnixStream, n: usize, deadline: Instant) -> Option<Vec<u8>> {
let mut out = Vec::with_capacity(n.min(255));
while out.len() < n {
let remaining = deadline.checked_duration_since(Instant::now())?;
if stream.set_read_timeout(Some(remaining)).is_err() {
return None;
}
let mut byte = [0u8; 1];
match stream.read(&mut byte) {
Ok(0) | Err(_) => return None,
Ok(_) => out.push(byte[0]),
}
}
Some(out)
}
/// After the handshake: copy bytes both ways, passing a half-close on. The client's read timeout
/// is cleared so the copy can run to its natural end.
fn copy_both_ways(mut client: UnixStream, mut server: TcpStream) {
let _ = client.set_read_timeout(None);
let mut server_write = match server.try_clone() {
Ok(s) => s,
Err(_) => return,
};
let mut client_read = match client.try_clone() {
Ok(s) => s,
Err(_) => return,
};
// Second thread: client -> server, then half-close the server's write side.
let upstream = match std::thread::Builder::new()
.name("egress-upstream".into())
.spawn(move || {
let _ = std::io::copy(&mut client_read, &mut server_write);
let _ = server_write.shutdown(std::net::Shutdown::Write);
}) {
Ok(handle) => handle,
Err(_) => return,
};
// Own thread: server -> client, then half-close the client's write side, then join.
let _ = std::io::copy(&mut server, &mut client);
let _ = client.shutdown(std::net::Shutdown::Write);
let _ = upstream.join();
}
+159
View File
@@ -0,0 +1,159 @@
//! The fourth tool, `http_fetch`: run `/bin/curl` with a fixed argument list against `args.url`,
//! through the egress proxy's socket `brokerd` mounts for this call, and report what curl printed
//! and how it ended. toolkit writes no HTTP or TLS client of its own; curl does that inside the
//! container. Spec sections 4 and 6.
use std::io::{self, Read};
use std::path::Path;
use std::process::{Command, Stdio};
use proto::tools::HttpFetchArgs;
use crate::Outcome;
pub const CURL: &str = "/bin/curl";
pub const PROXY: &str = "socks5h://localhost/run/egress/egress.sock";
pub const CA_BUNDLE: &str = "/etc/ssl/certs/ca-certificates.crt";
/// The most to keep from curl's standard error. curl can still run past it; the rest is drained
/// and discarded, never shown.
const MAX_STDERR: usize = 64 * 1024;
/// `curl`'s arguments for `url`, in order, without the program name.
pub fn curl_args(url: &str) -> Vec<String> {
vec![
// First, or it has no effect: never read a `.curlrc`.
"--disable".to_string(),
"--silent".to_string(),
"--show-error".to_string(),
// `[1-99999999]` in a URL is text, not millions of requests to the allowed host.
"--globoff".to_string(),
"--proto".to_string(),
"=https".to_string(),
"--proto-redir".to_string(),
"=https".to_string(),
"--location".to_string(),
"--max-redirs".to_string(),
"5".to_string(),
"--max-time".to_string(),
"50".to_string(),
"--max-filesize".to_string(),
"8388608".to_string(),
"--cacert".to_string(),
CA_BUNDLE.to_string(),
"--proxy".to_string(),
PROXY.to_string(),
"--write-out".to_string(),
"\n[http %{response_code}]".to_string(),
"--url".to_string(),
url.to_string(),
]
}
pub fn fetch(args: &HttpFetchArgs) -> Outcome {
fetch_with(Path::new(CURL), args)
}
pub fn fetch_with(curl: &Path, args: &HttpFetchArgs) -> Outcome {
let url = &args.url;
let args = curl_args(url);
let mut command = Command::new(curl);
command
.args(args)
.stdin(Stdio::null())
.stdout(Stdio::piped())
.stderr(Stdio::piped());
let mut child = match command.spawn() {
Ok(c) => c,
Err(e) => {
return Outcome::tool_error(format!(
"http_fetch: cannot start {}: {e}",
curl.display()
));
}
};
// Read standard error on its own thread while the main thread reads standard output to the
// end: two full pipes block each other, and the given test fills both.
let stderr_reader = match child.stderr.take() {
Some(r) => r,
None => return Outcome::tool_error("http_fetch: curl has no standard error".to_string()),
};
// `Builder`, not `spawn`, which panics when the system refuses a thread.
let stderr_handle = match std::thread::Builder::new().spawn(move || read_capped(stderr_reader))
{
Ok(handle) => handle,
Err(e) => {
let _ = child.kill();
let _ = child.wait();
return Outcome::tool_error(format!("http_fetch: cannot start a thread: {e}"));
}
};
let mut body = Vec::new();
let mut stdout = match child.stdout.take() {
Some(r) => r,
None => return Outcome::tool_error("http_fetch: curl has no standard output".to_string()),
};
if let Err(e) = stdout.read_to_end(&mut body) {
return Outcome::tool_error(format!("http_fetch: cannot read curl output: {e}"));
}
let status = match child.wait() {
Ok(s) => s,
Err(e) => return Outcome::tool_error(format!("http_fetch: cannot wait for curl: {e}")),
};
// A panicked reader lost its data; the body and status are still what curl gave us.
let stderr_bytes = stderr_handle.join().unwrap_or_default();
if status.success() {
return Outcome::done(String::from_utf8_lossy(&body).into_owned());
}
let why = match first_non_blank_line(&stderr_bytes) {
Some(line) => line,
None => match status.code() {
Some(code) => format!("curl exited {code}"),
None => "curl was killed".to_string(),
},
};
Outcome::tool_error(format!("http_fetch: {url}: {why}"))
}
/// Read `reader` to the end, keeping at most `MAX_STDERR` bytes; past the cap it keeps draining so
/// curl never blocks on a full pipe, but the rest is discarded.
fn read_capped(mut reader: impl Read) -> Vec<u8> {
let mut buf = [0u8; 8 * 1024];
let mut out: Vec<u8> = Vec::new();
loop {
let n = match reader.read(&mut buf) {
Ok(0) => break,
Ok(n) => n,
Err(e) if e.kind() == io::ErrorKind::Interrupted => continue,
Err(_) => break,
};
if out.len() >= MAX_STDERR {
continue;
}
let take = n.min(MAX_STDERR - out.len());
match buf.get(..take) {
Some(chunk) => out.extend_from_slice(chunk),
None => break,
}
}
out
}
/// The first line of `bytes` that is not blank after trimming, or `None` if every line is blank or
/// the bytes are not UTF-8 text.
fn first_non_blank_line(bytes: &[u8]) -> Option<String> {
let text = String::from_utf8_lossy(bytes);
for line in text.lines() {
let trimmed = line.trim();
if !trimmed.is_empty() {
return Some(trimmed.to_string());
}
}
None
}
+71
View File
@@ -0,0 +1,71 @@
//! The first two tools, `read_file` and `write_file`. `shell` and `http_fetch` come later; until
//! then `run` never reaches them, so they stay unknown tools.
use std::fs;
use std::io::Read;
use std::path::Path;
use proto::tools::{ReadFileArgs, WriteFileArgs};
use crate::Outcome;
pub const MAX_READ: usize = 1024 * 1024;
fn tool_err(tool: &str, path: &str, why: &str) -> Outcome {
Outcome::tool_error(format!("{tool}: {path}: {why}"))
}
pub fn read_file(args: &ReadFileArgs) -> Outcome {
let path = args.path.as_str();
let is_dir = match fs::metadata(path) {
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
return tool_err("read_file", path, "no such file");
}
Err(e) => return tool_err("read_file", path, &e.to_string()),
Ok(meta) => meta.is_dir(),
};
if is_dir {
return tool_err("read_file", path, "is a directory");
}
let file = match fs::File::open(path) {
Err(e) => return tool_err("read_file", path, &e.to_string()),
Ok(f) => f,
};
let mut buf = Vec::new();
let n = match file
.take(u64::try_from(MAX_READ).map_or(u64::MAX, |n| n.saturating_add(1)))
.read_to_end(&mut buf)
{
Err(e) => return tool_err("read_file", path, &e.to_string()),
Ok(n) => n,
};
if n > MAX_READ {
return tool_err("read_file", path, &format!("larger than {MAX_READ} bytes"));
}
match String::from_utf8(buf) {
Ok(text) => Outcome::done(text),
Err(_) => tool_err("read_file", path, "not UTF-8 text"),
}
}
pub fn write_file(args: &WriteFileArgs) -> Outcome {
let path = args.path.as_str();
let content = args.content.as_str();
match Path::new(path).parent() {
Some(p) if p.is_dir() => {}
_ => return tool_err("write_file", path, "the directory does not exist"),
}
if Path::new(path).is_dir() {
return tool_err("write_file", path, "is a directory");
}
if let Err(e) = fs::write(path, content) {
return tool_err("write_file", path, &e.to_string());
}
Outcome::done(format!("wrote {} bytes to {path}", content.len()))
}
+50
View File
@@ -0,0 +1,50 @@
//! Reading and parsing the arguments `brokerd` puts on `toolkit`'s standard input.
use std::io::Read;
pub const MAX_INPUT: usize = 2 * 1024 * 1024;
#[derive(Debug)]
pub enum InputError {
TooLarge,
NotUtf8,
Io(std::io::Error),
}
impl std::fmt::Display for InputError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
InputError::TooLarge => write!(f, "the arguments are larger than {MAX_INPUT} bytes"),
InputError::NotUtf8 => write!(f, "the arguments are not UTF-8"),
InputError::Io(e) => write!(f, "cannot read the arguments: {e}"),
}
}
}
impl std::error::Error for InputError {}
impl From<std::io::Error> for InputError {
fn from(e: std::io::Error) -> Self {
InputError::Io(e)
}
}
/// All of `stdin`, at most MAX_INPUT bytes, as UTF-8. Read at most MAX_INPUT + 1 bytes
/// (`Read::take`); more than MAX_INPUT is TooLarge.
pub fn read_input(stdin: &mut dyn std::io::Read) -> Result<String, InputError> {
let mut buf = Vec::new();
let n = stdin
.take(u64::try_from(MAX_INPUT).map_or(u64::MAX, |n| n.saturating_add(1)))
.read_to_end(&mut buf)?;
if n > MAX_INPUT {
return Err(InputError::TooLarge);
}
String::from_utf8(buf).map_err(|_| InputError::NotUtf8)
}
/// Parse `text` as the arguments of tool `name`; the error is the whole line for stderr:
/// "toolkit: {name}: the arguments do not parse: {serde's error}".
pub fn parse<T: serde::de::DeserializeOwned>(name: &str, text: &str) -> Result<T, String> {
serde_json::from_str(text)
.map_err(|e| format!("toolkit: {name}: the arguments do not parse: {e}"))
}
+100 -1
View File
@@ -1 +1,100 @@
//! Entry points that run inside tool containers. //! The programs that run inside tool containers.
use proto::tools::{HttpFetchArgs, ReadFileArgs, ShellArgs, WriteFileArgs};
pub mod addr;
pub mod egress;
pub mod fetch;
pub mod files;
pub mod input;
pub mod shell;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Exit {
Done,
ToolError,
Misuse,
}
impl Exit {
/// 0, 1, 2. No `as` cast: a `match`.
pub fn code(self) -> u8 {
match self {
Exit::Done => 0,
Exit::ToolError => 1,
Exit::Misuse => 2,
}
}
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Outcome {
pub exit: Exit,
pub stdout: String,
pub stderr: String,
}
impl Outcome {
pub fn done(stdout: String) -> Outcome {
Outcome {
exit: Exit::Done,
stdout,
stderr: String::new(),
}
}
pub fn tool_error(stdout: String) -> Outcome {
Outcome {
exit: Exit::ToolError,
stdout,
stderr: String::new(),
}
}
pub fn misuse(stderr: String) -> Outcome {
Outcome {
exit: Exit::Misuse,
stdout: String::new(),
stderr,
}
}
}
/// Run tool `name` with the arguments on `stdin`.
pub fn run(name: &str, stdin: &mut dyn std::io::Read) -> Outcome {
let text = match input::read_input(stdin) {
Ok(t) => t,
Err(e) => return Outcome::misuse(format!("toolkit: {e}")),
};
match name {
"read_file" => {
let args = match input::parse::<ReadFileArgs>(name, &text) {
Ok(a) => a,
Err(e) => return Outcome::misuse(e),
};
files::read_file(&args)
}
"write_file" => {
let args = match input::parse::<WriteFileArgs>(name, &text) {
Ok(a) => a,
Err(e) => return Outcome::misuse(e),
};
files::write_file(&args)
}
"shell" => {
let args = match input::parse::<ShellArgs>(name, &text) {
Ok(a) => a,
Err(e) => return Outcome::misuse(e),
};
shell::shell(&args)
}
"http_fetch" => {
let args = match input::parse::<HttpFetchArgs>(name, &text) {
Ok(a) => a,
Err(e) => return Outcome::misuse(e),
};
fetch::fetch(&args)
}
_ => Outcome::misuse(format!("toolkit: unknown tool {name:?}")),
}
}
+83 -3
View File
@@ -1,4 +1,84 @@
fn main() { use std::ffi::OsString;
eprintln!("toolkit: not implemented until M3"); use std::io::Write;
std::process::exit(2); use std::os::unix::ffi::OsStrExt;
use std::os::unix::net::UnixListener;
use std::process::ExitCode;
use std::sync::Arc;
use toolkit::egress::{Allow, Proxy, SystemDial};
fn main() -> ExitCode {
let args: Vec<OsString> = std::env::args_os().skip(1).collect();
match parse_egress_proxy(&args) {
Some(code) => code,
None => run_tool(&args),
}
}
/// The `egress-proxy --socket <path> --allow <list>` form. Anything else, including a program name
/// that is not `egress-proxy`, returns `None` so the tool form handles it.
fn parse_egress_proxy(args: &[OsString]) -> Option<ExitCode> {
// Exactly five words: a longer list is not this form, and goes to the tool form (exit 2).
if args.len() != 5 || args.first()?.as_bytes() != b"egress-proxy" {
return None;
}
// The form is `egress-proxy --socket <path> --allow <list>`, in that order.
let socket = match (args.get(1), args.get(2)) {
(Some(flag), Some(path)) if flag.as_bytes() == b"--socket" => path,
_ => return None,
};
let list = match (args.get(3), args.get(4)) {
(Some(flag), Some(value)) if flag.as_bytes() == b"--allow" => value,
_ => return None,
};
let list = match list.to_str() {
Some(s) => s,
None => {
eprint!("toolkit: egress-proxy: --allow: not valid UTF-8");
return Some(ExitCode::from(2));
}
};
let allow = match Allow::parse(list) {
Ok(a) => a,
Err(e) => {
eprint!("toolkit: egress-proxy: --allow: {e}");
return Some(ExitCode::from(2));
}
};
// The directory is fresh for each call, so a file already at the path is a mistake; do not
// remove it first.
let listener = match UnixListener::bind(socket) {
Ok(l) => l,
Err(e) => {
eprint!(
"toolkit: egress-proxy: cannot listen on {path}: {e}",
path = display(socket)
);
return Some(ExitCode::from(2));
}
};
let proxy = Arc::new(Proxy::new(allow, Arc::new(SystemDial)));
if let Err(e) = proxy.serve(listener) {
eprint!("toolkit: egress-proxy: {e}");
return Some(ExitCode::from(2));
}
Some(ExitCode::SUCCESS)
}
fn run_tool(args: &[OsString]) -> ExitCode {
let mut rest = args.iter();
let name = match (rest.next(), rest.next()) {
(Some(arg), None) => arg.to_str().unwrap_or("").to_string(),
_ => String::new(),
};
let outcome = toolkit::run(&name, &mut std::io::stdin().lock());
let _ = std::io::stdout().write_all(outcome.stdout.as_bytes());
let _ = std::io::stderr().write_all(outcome.stderr.as_bytes());
ExitCode::from(outcome.exit.code())
}
fn display(o: &OsString) -> String {
String::from_utf8_lossy(o.as_bytes()).into_owned()
} }
+93
View File
@@ -0,0 +1,93 @@
//! The third tool, `shell`: run one command under `/bin/sh -c` in a directory and report its
//! output (stdout and stderr together, in order) followed by how the command ended. The command's
//! own exit status is part of the result, not `toolkit`'s.
use std::io::{self, Read};
use std::os::unix::process::ExitStatusExt;
use std::path::Path;
use std::process::{Command, Stdio};
use proto::tools::ShellArgs;
use crate::Outcome;
pub const SHELL: &str = "/bin/sh";
pub const DEFAULT_CWD: &str = "/tmp";
pub const MAX_OUTPUT: usize = 1024 * 1024;
pub fn shell(args: &ShellArgs) -> Outcome {
let cwd = match args.cwd.as_deref() {
Some(c) => c,
None => DEFAULT_CWD,
};
if !Path::new(cwd).is_dir() {
return Outcome::tool_error(format!("shell: {cwd}: no such directory"));
}
let (mut reader, writer) = match io::pipe() {
Ok(p) => p,
Err(e) => return Outcome::tool_error(format!("shell: cannot make a pipe: {e}")),
};
let writer2 = match writer.try_clone() {
Ok(w) => w,
Err(e) => return Outcome::tool_error(format!("shell: cannot make a pipe: {e}")),
};
let mut command = Command::new(SHELL);
command.arg("-c").arg(&args.command).current_dir(cwd);
command.stdin(Stdio::null());
command.stdout(writer);
command.stderr(writer2);
let mut child = match command.spawn() {
Ok(c) => c,
Err(e) => return Outcome::tool_error(format!("shell: cannot start {SHELL}: {e}")),
};
// The Command still owns the write ends; while they are open reading never reaches EOF.
drop(command);
let mut buf = [0u8; 8 * 1024];
let mut out: Vec<u8> = Vec::new();
let mut dropped = false;
loop {
let n = match reader.read(&mut buf) {
Ok(0) => break,
Ok(n) => n,
Err(e) if e.kind() == io::ErrorKind::Interrupted => continue,
Err(e) => return Outcome::tool_error(format!("shell: cannot read the command: {e}")),
};
if out.len() >= MAX_OUTPUT {
// Past the limit: keep draining so the command never blocks on a full pipe.
dropped = true;
continue;
}
let chunk = match buf.get(..n) {
Some(c) => c,
None => continue,
};
let room = MAX_OUTPUT - out.len();
if chunk.len() <= room {
out.extend_from_slice(chunk);
} else {
out.extend_from_slice(chunk.get(..room).unwrap_or_default());
dropped = true;
}
}
let status = match child.wait() {
Ok(s) => s,
Err(e) => return Outcome::tool_error(format!("shell: cannot wait for the command: {e}")),
};
let mut text = String::from_utf8_lossy(&out).into_owned();
if dropped {
text.push_str(&format!("\n[output after {MAX_OUTPUT} bytes dropped]"));
}
match status.code() {
Some(code) => text.push_str(&format!("\n[exit {code}]")),
None => match status.signal() {
Some(sig) => text.push_str(&format!("\n[killed by signal {sig}]")),
None => text.push_str("\n[ended without a status]"),
},
}
Outcome::done(text)
}
+116
View File
@@ -0,0 +1,116 @@
//! `is_public`: the egress proxy connects only to public addresses. Every range in the M3b spec,
//! section 5, has a case at each end, and a public neighbour just outside it. Do not edit.
use std::net::IpAddr;
use toolkit::addr::is_public;
fn ip(s: &str) -> IpAddr {
s.parse().unwrap()
}
#[test]
fn refused_ipv4() {
for s in [
"0.0.0.0",
"0.255.255.255",
"10.0.0.0",
"10.255.255.255",
"100.64.0.0",
"100.100.100.100",
"100.127.255.255",
"127.0.0.1",
"127.255.255.255",
"169.254.0.1",
"169.254.255.255",
"172.16.0.0",
"172.31.255.255",
"192.0.0.0",
"192.0.0.255",
"192.0.2.1",
"192.168.0.1",
"192.168.255.255",
"198.18.0.0",
"198.19.255.255",
"198.51.100.7",
"203.0.113.9",
"224.0.0.1",
"239.255.255.255",
"240.0.0.0",
"255.255.255.255",
] {
assert!(!is_public(ip(s)), "{s} must be refused");
}
}
#[test]
fn public_ipv4() {
for s in [
"1.1.1.1",
"8.8.8.8",
"9.255.255.255",
"11.0.0.0",
"100.63.255.255",
"100.128.0.0",
"126.255.255.255",
"128.0.0.0",
"169.253.255.255",
"172.15.255.255",
"172.32.0.0",
"192.0.1.0",
"192.0.3.0",
"192.167.255.255",
"192.169.0.0",
"198.17.255.255",
"198.20.0.0",
"198.51.99.255",
"203.0.112.255",
"223.255.255.255",
"93.184.216.34",
] {
assert!(is_public(ip(s)), "{s} is public");
}
}
#[test]
fn refused_ipv6() {
for s in [
"::",
"::1",
"fc00::1",
"fdff:ffff::1",
"fd7a:115c:a1e0::1",
"fe80::1",
"febf::1",
"ff02::1",
"ff00::",
"2001:db8::1",
"2001:db8:ffff::1",
"::ffff:127.0.0.1",
"::ffff:10.1.2.3",
"::ffff:100.100.100.100",
"64:ff9b::7f00:1",
"64:ff9b::a01:203",
"::ffff:0.0.0.0",
"::127.0.0.1",
"::1.1.1.1",
"::ffff",
] {
assert!(!is_public(ip(s)), "{s} must be refused");
}
}
#[test]
fn public_ipv6() {
for s in [
"2606:4700:4700::1111",
"2a00:1450::1",
"fbff::1",
"fec0::1",
"2001:db9::1",
"::ffff:1.1.1.1",
"64:ff9b::101:101",
] {
assert!(is_public(ip(s)), "{s} is public");
}
}
+451
View File
@@ -0,0 +1,451 @@
//! The egress proxy against a fake resolver and a local echo server: every reply code, the host,
//! port and address checks, the byte copy both ways with half-close, the handshake deadline, the
//! connection limit, and `toolkit egress-proxy` as a program. No test needs a network. Do not edit.
mod support;
use std::collections::HashMap;
use std::io::{Read, Write};
use std::net::{Shutdown, SocketAddr, TcpListener, TcpStream};
use std::os::unix::net::{UnixListener, UnixStream};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use support::TempDir;
use toolkit::egress::{
ADDRESS_TYPE_NOT_SUPPORTED, Allow, COMMAND_NOT_SUPPORTED, CONNECTION_REFUSED, Dial,
HOST_UNREACHABLE, MAX_CONNECTIONS, NOT_ALLOWED, Proxy,
};
/// Names resolve from a table; every connection goes to one local echo server, except to
/// addresses listed as refusing. Records every address it was asked to connect to.
struct FakeDial {
names: HashMap<String, Vec<SocketAddr>>,
refusing: Vec<SocketAddr>,
echo: SocketAddr,
connected: Mutex<Vec<SocketAddr>>,
}
impl Dial for FakeDial {
fn resolve(&self, host: &str, _port: u16) -> std::io::Result<Vec<SocketAddr>> {
self.names
.get(host)
.cloned()
.ok_or_else(|| std::io::Error::other("no such name"))
}
fn connect(&self, addr: SocketAddr, _timeout: Duration) -> std::io::Result<TcpStream> {
self.connected.lock().unwrap().push(addr);
if self.refusing.contains(&addr) {
return Err(std::io::Error::from(std::io::ErrorKind::ConnectionRefused));
}
TcpStream::connect(self.echo)
}
}
fn sa(s: &str) -> SocketAddr {
s.parse().unwrap()
}
/// An echo server: copies back what it reads, and closes its side after reading the end.
fn echo_server() -> SocketAddr {
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let addr = listener.local_addr().unwrap();
std::thread::spawn(move || {
for stream in listener.incoming() {
let Ok(mut stream) = stream else { continue };
std::thread::spawn(move || {
let mut back = stream.try_clone().unwrap();
let _ = std::io::copy(&mut stream, &mut back);
let _ = back.shutdown(Shutdown::Write);
});
}
});
addr
}
fn dial() -> Arc<FakeDial> {
let mut names = HashMap::new();
names.insert("example.com".to_string(), vec![sa("93.184.216.34:443")]);
names.insert(
"www.example.org".to_string(),
vec![sa("[2606:2800::1]:443")],
);
names.insert(
"mixed.example.com".to_string(),
vec![
sa("10.0.0.1:443"),
sa("100.100.100.100:443"),
sa("1.1.1.1:443"),
],
);
names.insert(
"inside.example.com".to_string(),
vec![
sa("100.101.102.103:443"),
sa("127.0.0.1:443"),
sa("[::1]:443"),
],
);
names.insert("refusing.example.com".to_string(), vec![sa("8.8.8.8:443")]);
Arc::new(FakeDial {
names,
refusing: vec![sa("8.8.8.8:443")],
echo: echo_server(),
connected: Mutex::new(Vec::new()),
})
}
fn allow() -> Allow {
Allow::parse(
"example.com,*.example.org,mixed.example.com,inside.example.com,refusing.example.com,nowhere.example.com",
)
.unwrap()
}
/// A proxy on one end of a socket pair, handling that one connection on its own thread.
fn start(dial: Arc<FakeDial>) -> (UnixStream, std::thread::JoinHandle<()>) {
let (client, server) = UnixStream::pair().unwrap();
client
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
let proxy = Proxy::new(allow(), dial).with_handshake_timeout(Duration::from_millis(300));
let handle = std::thread::spawn(move || proxy.handle(server));
(client, handle)
}
fn request(host: &str, port: u16) -> Vec<u8> {
let mut bytes = vec![5, 1, 0, 3, u8::try_from(host.len()).unwrap()];
bytes.extend_from_slice(host.as_bytes());
bytes.extend_from_slice(&port.to_be_bytes());
bytes
}
fn read_exactly(client: &mut UnixStream, n: usize) -> Vec<u8> {
let mut buf = vec![0u8; n];
client.read_exact(&mut buf).unwrap();
buf
}
/// Read until the proxy closes; the bytes read. A close that leaves some of our bytes unread by
/// the proxy arrives as "connection reset" rather than as the end: both mean closed. A timeout
/// does not: the proxy did not close.
fn read_to_close(client: &mut UnixStream) -> Vec<u8> {
let mut rest = Vec::new();
let mut buf = [0u8; 4096];
loop {
match client.read(&mut buf) {
Ok(0) => return rest,
Ok(n) => rest.extend_from_slice(&buf[..n]),
Err(e) if e.kind() == std::io::ErrorKind::ConnectionReset => return rest,
Err(e) => panic!("the proxy did not close: {e}"),
}
}
}
/// Greeting, then `req`; the reply's code, and that the proxy then closed (for failures).
fn reply_code(req: &[u8]) -> u8 {
let (mut client, handle) = start(dial());
client.write_all(&[5, 1, 0]).unwrap();
assert_eq!(read_exactly(&mut client, 2), [5, 0]);
client.write_all(req).unwrap();
let reply = read_exactly(&mut client, 10);
assert_eq!(reply[0], 5);
assert_eq!(&reply[2..], &[0, 1, 0, 0, 0, 0, 0, 0]);
if reply[1] != 0 {
assert_eq!(
read_to_close(&mut client),
b"",
"a refusal is followed by the close"
);
handle.join().unwrap();
}
reply[1]
}
#[test]
fn an_allowed_host_is_connected_and_bytes_flow_both_ways_until_both_ends_close() {
let dial = dial();
let (mut client, handle) = start(Arc::clone(&dial));
client.write_all(&[5, 2, 2, 0]).unwrap();
assert_eq!(
read_exactly(&mut client, 2),
[5, 0],
"method 0 among others"
);
client.write_all(&request("example.com", 443)).unwrap();
assert_eq!(
read_exactly(&mut client, 10),
[5, 0, 0, 1, 0, 0, 0, 0, 0, 0]
);
client.write_all(b"hello through the tunnel").unwrap();
assert_eq!(read_exactly(&mut client, 24), b"hello through the tunnel");
client.shutdown(Shutdown::Write).unwrap();
assert_eq!(
read_to_close(&mut client),
b"",
"the half-close reached the server and back"
);
handle.join().unwrap();
assert_eq!(
*dial.connected.lock().unwrap(),
vec![sa("93.184.216.34:443")]
);
}
#[test]
fn a_wildcard_pattern_allows_a_name_under_it() {
assert_eq!(reply_code(&request("www.example.org", 443)), 0);
}
#[test]
fn a_greeting_without_method_zero_is_answered_ff_and_closed() {
let (mut client, handle) = start(dial());
client.write_all(&[5, 2, 1, 2]).unwrap();
assert_eq!(read_exactly(&mut client, 2), [5, 0xff]);
assert_eq!(read_to_close(&mut client), b"");
handle.join().unwrap();
}
#[test]
fn another_version_is_closed_without_a_word() {
let (mut client, handle) = start(dial());
client.write_all(&[4, 1, 0]).unwrap();
assert_eq!(read_to_close(&mut client), b"");
handle.join().unwrap();
}
#[test]
fn only_connect_by_name_to_an_allowed_host_on_443() {
let mut bind = request("example.com", 443);
bind[1] = 2;
assert_eq!(reply_code(&bind), COMMAND_NOT_SUPPORTED);
let mut udp = request("example.com", 443);
udp[1] = 3;
assert_eq!(reply_code(&udp), COMMAND_NOT_SUPPORTED);
assert_eq!(
reply_code(&[5, 1, 0, 1, 93, 184, 216, 34, 1, 187]),
ADDRESS_TYPE_NOT_SUPPORTED
);
let mut v6 = vec![5, 1, 0, 4];
v6.extend_from_slice(&[0; 16]);
v6.extend_from_slice(&443u16.to_be_bytes());
assert_eq!(reply_code(&v6), ADDRESS_TYPE_NOT_SUPPORTED);
for (host, port) in [
("example.com", 80),
("example.com", 8443),
("evil.test", 443),
("example.org", 443), // *.example.org does not cover example.org
("Example.com", 443), // not a valid host name
("example.com.", 443),
("127.0.0.1", 443), // an IP literal sent as a name
("wwwexample.com", 443),
] {
assert_eq!(
reply_code(&request(host, port)),
NOT_ALLOWED,
"{host}:{port}"
);
}
}
#[test]
fn an_empty_or_non_utf8_name_is_not_allowed() {
assert_eq!(reply_code(&[5, 1, 0, 3, 0, 1, 187]), NOT_ALLOWED);
assert_eq!(
reply_code(&[5, 1, 0, 3, 2, 0xff, 0xfe, 1, 187]),
NOT_ALLOWED
);
}
#[test]
fn only_public_addresses_are_used() {
let dial = dial();
let (mut client, handle) = start(Arc::clone(&dial));
client.write_all(&[5, 1, 0]).unwrap();
read_exactly(&mut client, 2);
client
.write_all(&request("mixed.example.com", 443))
.unwrap();
assert_eq!(read_exactly(&mut client, 10)[1], 0);
drop(client);
handle.join().unwrap();
assert_eq!(
*dial.connected.lock().unwrap(),
vec![sa("1.1.1.1:443")],
"the private and tailnet addresses were skipped, not tried"
);
}
#[test]
fn a_name_with_no_public_address_or_no_address_is_unreachable() {
assert_eq!(
reply_code(&request("inside.example.com", 443)),
HOST_UNREACHABLE
);
assert_eq!(
reply_code(&request("nowhere.example.com", 443)),
HOST_UNREACHABLE
);
}
#[test]
fn a_refused_connection_is_reply_5() {
assert_eq!(
reply_code(&request("refusing.example.com", 443)),
CONNECTION_REFUSED
);
}
#[test]
fn a_stalled_handshake_is_closed_at_the_deadline() {
let (mut client, handle) = start(dial());
client.write_all(&[5, 1, 0]).unwrap();
assert_eq!(read_exactly(&mut client, 2), [5, 0]);
client.write_all(&[5, 1]).unwrap(); // half a request, then nothing
let started = Instant::now();
assert_eq!(read_to_close(&mut client), b"");
let took = started.elapsed();
handle.join().unwrap();
assert!(
took < Duration::from_secs(2),
"one deadline for the handshake: {took:?}"
);
}
#[test]
fn a_handshake_that_trickles_still_ends_at_the_deadline() {
let (mut client, handle) = start(dial());
let started = Instant::now();
// One byte every 100 ms: each read is quick, but the handshake as a whole is not.
for byte in [5u8, 1, 0, 5, 1, 0, 3, 11] {
if client.write_all(&[byte]).is_err() {
break;
}
std::thread::sleep(Duration::from_millis(100));
}
let _ = read_to_close(&mut client);
handle.join().unwrap();
assert!(started.elapsed() < Duration::from_secs(2));
}
#[test]
fn allow_lists_parse_strictly() {
assert!(Allow::parse("example.com").is_ok());
assert!(Allow::parse("example.com,*.example.org").is_ok());
for bad in [
"",
",",
"example.com,",
"a.com,,b.com",
"Example.com",
"*",
"10.0.0.1",
] {
assert!(Allow::parse(bad).is_err(), "{bad:?}");
}
let allow = Allow::parse("*.example.org").unwrap();
assert!(allow.permits("a.example.org"));
assert!(!allow.permits("example.org"));
assert!(!allow.permits("A.example.org"));
}
#[test]
fn more_than_the_limit_of_connections_are_closed_at_once() {
let dir = TempDir::new("egress-limit");
let path = dir.path().join("egress.sock");
let listener = UnixListener::bind(&path).unwrap();
let proxy =
Arc::new(Proxy::new(allow(), dial()).with_handshake_timeout(Duration::from_secs(5)));
std::thread::spawn(move || {
let _ = proxy.serve(listener);
});
// MAX_CONNECTIONS clients that greet and then wait, holding their places.
let mut held = Vec::new();
for _ in 0..MAX_CONNECTIONS {
let mut c = UnixStream::connect(&path).unwrap();
c.set_read_timeout(Some(Duration::from_secs(5))).unwrap();
c.write_all(&[5, 1, 0]).unwrap();
assert_eq!(read_exactly(&mut c, 2), [5, 0]);
held.push(c);
}
let mut extra = UnixStream::connect(&path).unwrap();
extra
.set_read_timeout(Some(Duration::from_secs(2)))
.unwrap();
let _ = extra.write_all(&[5, 1, 0]);
let mut buf = [0u8; 2];
assert!(
matches!(extra.read(&mut buf), Ok(0) | Err(_)),
"the connection over the limit gets no answer"
);
// When one place is freed, a new connection is served again.
drop(held.pop());
std::thread::sleep(Duration::from_millis(200));
let mut again = UnixStream::connect(&path).unwrap();
again
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
again.write_all(&[5, 1, 0]).unwrap();
assert_eq!(read_exactly(&mut again, 2), [5, 0]);
}
#[test]
fn the_program_listens_where_it_is_told_and_refuses_what_is_not_allowed() {
let dir = TempDir::new("egress-prog");
let path = dir.path().join("egress.sock");
let mut child = std::process::Command::new(env!("CARGO_BIN_EXE_toolkit"))
.args([
"egress-proxy",
"--socket",
path.to_str().unwrap(),
"--allow",
"example.com",
])
.spawn()
.unwrap();
let until = Instant::now() + Duration::from_secs(5);
let mut client = loop {
if let Ok(c) = UnixStream::connect(&path) {
break c;
}
assert!(Instant::now() < until, "the proxy never listened");
std::thread::sleep(Duration::from_millis(20));
};
client
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
client.write_all(&[5, 1, 0]).unwrap();
assert_eq!(read_exactly(&mut client, 2), [5, 0]);
client.write_all(&request("evil.test", 443)).unwrap();
assert_eq!(read_exactly(&mut client, 10)[1], NOT_ALLOWED);
child.kill().unwrap();
child.wait().unwrap();
}
#[test]
fn the_program_refuses_bad_arguments() {
let dir = TempDir::new("egress-args");
let path = dir.path().join("egress.sock");
let sock = path.to_str().unwrap();
std::fs::write(dir.path().join("taken"), "").unwrap();
let taken = dir.path().join("taken");
for args in [
vec!["egress-proxy", "--socket", sock, "--allow", "Example.com"],
vec!["egress-proxy", "--socket", sock, "--allow", ""],
vec![
"egress-proxy",
"--socket",
taken.to_str().unwrap(),
"--allow",
"example.com",
],
vec!["egress-proxy", "--socket", sock],
vec!["egress-proxy", "--allow", "example.com", "--socket", sock],
] {
let out = std::process::Command::new(env!("CARGO_BIN_EXE_toolkit"))
.args(&args)
.output()
.unwrap();
assert_eq!(out.status.code(), Some(2), "{args:?}");
}
}
+42
View File
@@ -0,0 +1,42 @@
//! `toolkit egress-proxy` takes exactly `--socket <path> --allow <list>`: with anything after
//! them it is not the proxy, and exits 2 at once rather than listening (M3b review). Do not edit.
use std::process::{Command, Stdio};
use std::time::{Duration, Instant};
#[test]
fn trailing_arguments_are_not_the_proxy_form() {
let dir = std::env::temp_dir().join(format!("tk-egress-form-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).unwrap();
let socket = dir.join("egress.sock");
let mut child = Command::new(env!("CARGO_BIN_EXE_toolkit"))
.args([
"egress-proxy",
"--socket",
socket.to_str().unwrap(),
"--allow",
"example.com",
"extra",
])
.stdin(Stdio::null())
.stdout(Stdio::null())
.stderr(Stdio::null())
.spawn()
.unwrap();
let until = Instant::now() + Duration::from_secs(3);
let status = loop {
if let Some(status) = child.try_wait().unwrap() {
break status;
}
if Instant::now() > until {
let _ = child.kill();
let _ = child.wait();
panic!("it is listening: trailing arguments were accepted");
}
std::thread::sleep(Duration::from_millis(20));
};
assert_eq!(status.code(), Some(2));
assert!(!socket.exists(), "no socket was made");
let _ = std::fs::remove_dir_all(&dir);
}
+148
View File
@@ -0,0 +1,148 @@
//! `http_fetch`: the fixed `curl` arguments, and what `toolkit` makes of `curl`'s answer, against
//! fake `curl` scripts. Do not edit.
mod support;
use std::os::unix::fs::PermissionsExt;
use std::path::PathBuf;
use std::sync::{Mutex, MutexGuard};
use proto::tools::HttpFetchArgs;
use support::TempDir;
use toolkit::Exit;
use toolkit::fetch::{curl_args, fetch_with};
const URL: &str = "https://example.com/a?b=c";
/// Every test that starts a process takes its turn. Otherwise another test's fork can hold
/// the script open for writing at the moment it is run, and running it fails with "text file
/// busy" (ETXTBSY), which has nothing to do with the code under test.
static SERIAL: Mutex<()> = Mutex::new(());
fn serial() -> MutexGuard<'static, ()> {
SERIAL.lock().unwrap_or_else(|p| p.into_inner())
}
fn fake_curl(dir: &TempDir, body: &str) -> PathBuf {
let path = dir.path().join("curl");
std::fs::write(&path, format!("#!/bin/sh\n{body}\n")).unwrap();
std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o755)).unwrap();
path
}
fn args() -> HttpFetchArgs {
HttpFetchArgs {
url: URL.to_string(),
}
}
#[test]
fn the_argument_list_is_fixed_and_ends_with_the_url() {
let expected: Vec<&str> = vec![
"--disable",
"--silent",
"--show-error",
"--globoff",
"--proto",
"=https",
"--proto-redir",
"=https",
"--location",
"--max-redirs",
"5",
"--max-time",
"50",
"--max-filesize",
"8388608",
"--cacert",
"/etc/ssl/certs/ca-certificates.crt",
"--proxy",
"socks5h://localhost/run/egress/egress.sock",
"--write-out",
"\n[http %{response_code}]",
"--url",
URL,
];
assert_eq!(curl_args(URL), expected);
}
#[test]
fn curl_is_given_exactly_those_arguments() {
let _serial = serial();
let dir = TempDir::new("fetch-args");
let curl = fake_curl(&dir, r#"for a in "$@"; do printf '%s|' "$a"; done"#);
let got = fetch_with(&curl, &args());
assert_eq!(got.exit, Exit::Done);
let expected: String = curl_args(URL).iter().map(|a| format!("{a}|")).collect();
assert_eq!(got.stdout, expected);
}
#[test]
fn a_successful_fetch_is_the_body_and_the_status_line() {
let _serial = serial();
let dir = TempDir::new("fetch-ok");
let curl = fake_curl(&dir, r#"printf 'hello\n\n[http 404]'"#);
let got = fetch_with(&curl, &args());
assert_eq!(
(got.exit, got.stdout.as_str()),
(Exit::Done, "hello\n\n[http 404]")
);
}
#[test]
fn a_failed_fetch_is_exit_1_with_curls_first_error_line() {
let _serial = serial();
let dir = TempDir::new("fetch-fail");
let curl = fake_curl(
&dir,
"printf 'partial'; printf '\\ncurl: (97) cannot complete SOCKS5 connection to evil.test. (2)\\nmore\\n' >&2; exit 97",
);
let got = fetch_with(&curl, &args());
assert_eq!(got.exit, Exit::ToolError);
assert_eq!(
got.stdout,
format!(
"http_fetch: {URL}: curl: (97) cannot complete SOCKS5 connection to evil.test. (2)"
)
);
}
#[test]
fn a_failure_without_a_message_names_the_exit() {
let _serial = serial();
let dir = TempDir::new("fetch-quiet");
let curl = fake_curl(&dir, "exit 28");
let got = fetch_with(&curl, &args());
assert_eq!(
(got.exit, got.stdout.as_str()),
(
Exit::ToolError,
"http_fetch: https://example.com/a?b=c: curl exited 28"
)
);
}
#[test]
fn a_curl_that_cannot_start_is_exit_1() {
let _serial = serial();
let got = fetch_with(&PathBuf::from("/no/such/curl"), &args());
assert_eq!(got.exit, Exit::ToolError);
assert!(
got.stdout
.starts_with("http_fetch: cannot start /no/such/curl: "),
"{}",
got.stdout
);
}
#[test]
fn much_output_on_both_streams_does_not_stop_curl() {
let _serial = serial();
let dir = TempDir::new("fetch-both");
let curl = fake_curl(
&dir,
"head -c 300000 /dev/zero | tr '\\0' e >&2; head -c 300000 /dev/zero | tr '\\0' o; exit 0",
);
let got = fetch_with(&curl, &args());
assert_eq!((got.exit, got.stdout.len()), (Exit::Done, 300000));
}
+159
View File
@@ -0,0 +1,159 @@
//! `toolkit read_file` and `toolkit write_file`, run as `brokerd` runs them. Do not edit.
mod support;
use std::os::unix::fs::PermissionsExt;
use proto::tools::{ReadFileArgs, WriteFileArgs};
use support::{TempDir, json, toolkit};
use toolkit::files::MAX_READ;
fn read(path: &str) -> support::Ran {
toolkit(
&["read_file"],
&json(&ReadFileArgs {
path: path.to_string(),
}),
)
}
fn write(path: &str, content: &str) -> support::Ran {
let args = WriteFileArgs {
path: path.to_string(),
content: content.to_string(),
};
toolkit(&["write_file"], &json(&args))
}
#[test]
fn a_file_is_read_exactly() {
let dir = TempDir::new("read");
let text = "line one\nline two, no newline at the end: ✓";
std::fs::write(dir.at("a.md"), text).unwrap();
let ran = read(&dir.at("a.md"));
assert_eq!((ran.code, ran.stdout.as_str()), (0, text));
assert_eq!(ran.stderr, "");
}
#[test]
fn an_empty_file_is_empty_output() {
let dir = TempDir::new("read-empty");
std::fs::write(dir.at("e"), "").unwrap();
let ran = read(&dir.at("e"));
assert_eq!((ran.code, ran.stdout.as_str()), (0, ""));
}
#[test]
fn what_cannot_be_read_is_exit_1_with_one_line_for_the_model() {
let dir = TempDir::new("read-bad");
std::fs::write(dir.at("bin"), [0xff, 0xfe, 0x00]).unwrap();
std::fs::write(dir.at("big"), vec![b'a'; MAX_READ + 1]).unwrap();
std::fs::write(dir.at("exact"), vec![b'a'; MAX_READ]).unwrap();
let cases = [
(dir.at("missing"), "no such file"),
(dir.at(""), "is a directory"),
(dir.at("bin"), "not UTF-8 text"),
(dir.at("big"), "larger than 1048576 bytes"),
];
for (path, why) in cases {
let ran = read(&path);
assert_eq!(ran.code, 1, "{path}");
assert_eq!(ran.stdout, format!("read_file: {path}: {why}"));
}
let ran = read(&dir.at("exact"));
assert_eq!(
(ran.code, ran.stdout.len()),
(0, MAX_READ),
"exactly the limit is fine"
);
}
#[test]
fn an_unreadable_file_names_the_error() {
if is_root() {
return; // root reads anything
}
let dir = TempDir::new("read-perm");
std::fs::write(dir.at("secret"), "x").unwrap();
std::fs::set_permissions(dir.at("secret"), std::fs::Permissions::from_mode(0o000)).unwrap();
let ran = read(&dir.at("secret"));
assert_eq!(ran.code, 1);
assert!(
ran.stdout
.starts_with(&format!("read_file: {}: ", dir.at("secret"))),
"{}",
ran.stdout
);
assert!(ran.stdout.contains("ermission denied"), "{}", ran.stdout);
}
#[test]
fn a_file_is_written_created_or_replaced() {
let dir = TempDir::new("write");
let ran = write(&dir.at("new.md"), "hello ✓\n");
assert_eq!(ran.code, 0);
assert_eq!(
ran.stdout,
format!("wrote 10 bytes to {}", dir.at("new.md"))
);
assert_eq!(
std::fs::read_to_string(dir.at("new.md")).unwrap(),
"hello ✓\n"
);
let ran = write(&dir.at("new.md"), "");
assert_eq!(ran.stdout, format!("wrote 0 bytes to {}", dir.at("new.md")));
assert_eq!(std::fs::read_to_string(dir.at("new.md")).unwrap(), "");
}
#[test]
fn what_cannot_be_written_is_exit_1() {
let dir = TempDir::new("write-bad");
std::fs::create_dir(dir.at("sub")).unwrap();
let cases = [
(dir.at("nope/a.md"), "the directory does not exist"),
(dir.at("sub"), "is a directory"),
];
for (path, why) in cases {
let ran = write(&path, "x");
assert_eq!(ran.code, 1, "{path}");
assert_eq!(ran.stdout, format!("write_file: {path}: {why}"));
}
assert!(!dir.path().join("nope").exists(), "no directory is created");
}
#[test]
fn misuse_is_exit_2_with_nothing_on_standard_output() {
let long = vec![b' '; toolkit::input::MAX_INPUT + 1];
let cases: [(&[&str], &[u8], &str); 7] = [
(
&["read_file"],
br#"{"path":"/a","mode":1}"#,
"the arguments do not parse",
),
(&["read_file"], b"", "the arguments do not parse"),
(
&["write_file"],
br#"{"path":"/a"}"#,
"the arguments do not parse",
),
(&["read_file"], &[0xff, 0xfe], "not UTF-8"),
(&["read_file"], &long, "larger than 2097152 bytes"),
(&["format_disk"], b"{}", "unknown tool"),
(&[], b"{}", "unknown tool"),
];
for (args, input, why) in cases {
let ran = toolkit(args, input);
assert_eq!(ran.code, 2, "{args:?}");
assert_eq!(
ran.stdout, "",
"{args:?}: the model sees nothing of a misuse"
);
assert!(ran.stderr.contains(why), "{args:?}: {}", ran.stderr);
}
}
fn is_root() -> bool {
std::fs::read_to_string("/proc/self/status")
.map(|s| s.lines().any(|l| l.starts_with("Uid:\t0\t")))
.unwrap_or(false)
}
+89
View File
@@ -0,0 +1,89 @@
//! `toolkit shell`, run as `brokerd` runs it, with the host's `/bin/sh`. Do not edit.
mod support;
use proto::tools::ShellArgs;
use support::{TempDir, json, toolkit};
use toolkit::shell::MAX_OUTPUT;
fn sh(command: &str, cwd: Option<&str>) -> support::Ran {
let args = ShellArgs {
command: command.to_string(),
cwd: cwd.map(str::to_string),
};
toolkit(&["shell"], &json(&args))
}
#[test]
fn output_and_errors_come_back_in_order_then_the_exit_status() {
let ran = sh("echo one; echo two >&2; echo three", None);
assert_eq!(ran.code, 0);
assert_eq!(ran.stdout, "one\ntwo\nthree\n\n[exit 0]");
}
#[test]
fn a_failing_command_is_still_a_result_with_its_status() {
let ran = sh("echo nope; exit 3", None);
assert_eq!((ran.code, ran.stdout.as_str()), (0, "nope\n\n[exit 3]"));
let ran = sh("true", None);
assert_eq!(ran.stdout, "\n[exit 0]");
}
#[test]
fn a_command_killed_by_a_signal_says_so() {
let ran = sh("kill -9 $$", None);
assert_eq!(
(ran.code, ran.stdout.as_str()),
(0, "\n[killed by signal 9]")
);
}
#[test]
fn the_command_runs_in_cwd_or_in_tmp() {
let dir = TempDir::new("cwd");
let here = dir.path().to_str().unwrap();
let ran = sh("pwd", Some(here));
assert_eq!(ran.stdout, format!("{here}\n\n[exit 0]"));
let ran = sh("pwd", None);
assert_eq!(ran.stdout, "/tmp\n\n[exit 0]");
}
#[test]
fn a_missing_cwd_is_exit_1() {
let ran = sh("pwd", Some("/no/such/dir"));
assert_eq!(
(ran.code, ran.stdout.as_str()),
(1, "shell: /no/such/dir: no such directory")
);
}
#[test]
fn standard_input_is_empty() {
let ran = sh("cat; echo done", None);
assert_eq!(ran.stdout, "done\n\n[exit 0]");
}
#[test]
fn output_past_the_limit_is_dropped_and_the_command_still_finishes() {
let ran = sh(
&format!(
"head -c {} /dev/zero | tr '\\0' a; echo; echo end >&2; exit 4",
MAX_OUTPUT + 5000
),
None,
);
assert_eq!(ran.code, 0);
let expected_tail = format!("\n[output after {MAX_OUTPUT} bytes dropped]\n[exit 4]");
assert!(
ran.stdout.ends_with(&expected_tail),
"{}",
&ran.stdout[ran.stdout.len() - 80..]
);
assert_eq!(ran.stdout.len(), MAX_OUTPUT + expected_tail.len());
}
#[test]
fn output_that_is_not_utf8_is_replaced_not_refused() {
let ran = sh("printf 'a\\377b'", None);
assert_eq!(ran.stdout, "a\u{fffd}b\n[exit 0]");
}
+68
View File
@@ -0,0 +1,68 @@
//! Running the `toolkit` binary as `brokerd` does: arguments on standard input. Do not edit.
#![allow(dead_code)] // each test file uses its own part
use std::io::Write;
use std::path::{Path, PathBuf};
use std::process::{Command, Stdio};
use std::sync::atomic::{AtomicU32, Ordering};
static NEXT: AtomicU32 = AtomicU32::new(0);
/// A temporary directory, removed when dropped.
pub struct TempDir(pub PathBuf);
impl TempDir {
pub fn new(tag: &str) -> TempDir {
let n = NEXT.fetch_add(1, Ordering::SeqCst);
let path = std::env::temp_dir().join(format!("tk-{tag}-{}-{n}", std::process::id()));
let _ = std::fs::remove_dir_all(&path);
std::fs::create_dir_all(&path).unwrap();
TempDir(path)
}
pub fn path(&self) -> &Path {
&self.0
}
/// The path of `name` inside, as a string (the tools take strings).
pub fn at(&self, name: &str) -> String {
self.0.join(name).to_str().unwrap().to_string()
}
}
impl Drop for TempDir {
fn drop(&mut self) {
let _ = std::fs::remove_dir_all(&self.0);
}
}
pub struct Ran {
pub code: i32,
pub stdout: String,
pub stderr: String,
}
/// Run `toolkit <args…>` with `input` on standard input.
pub fn toolkit(args: &[&str], input: &[u8]) -> Ran {
let mut child = Command::new(env!("CARGO_BIN_EXE_toolkit"))
.args(args)
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
.unwrap();
let mut stdin = child.stdin.take().unwrap();
// A broken pipe here only means toolkit stopped reading, which some tests expect.
let _ = stdin.write_all(input);
drop(stdin);
let out = child.wait_with_output().unwrap();
Ran {
code: out.status.code().unwrap_or(-1),
stdout: String::from_utf8(out.stdout).unwrap(),
stderr: String::from_utf8_lossy(&out.stderr).into_owned(),
}
}
/// The JSON for one argument struct.
pub fn json<T: serde::Serialize>(value: &T) -> Vec<u8> {
serde_json::to_vec(value).unwrap()
}
+46
View File
@@ -0,0 +1,46 @@
# The image every tool container runs from (M3b spec, section 7; the brief's authority contract 4).
# Built from this repository's source by Nix on the host that runs brokerd, loaded with
# `podman load`, and named by digest in brokerd.toml's `[runner] image`. Nothing is pulled at call
# time.
#
# nix-build deploy/tools-image.nix # from the repository root; result is an image tarball
# podman load < result
# podman image inspect --format '{{.Digest}}' localhost/boxmaker-tools:latest
#
# Contents: /bin/toolkit (static, musl), busybox with every applet (so /bin/sh is busybox's),
# /bin/curl (static), the CA bundle. No package manager, no compiler, nothing else.
{ pkgs ? import <nixpkgs> { } }:
let
static = pkgs.pkgsStatic;
src = pkgs.lib.cleanSourceWith {
src = ../.;
# Only what the build reads: no target/, no .state/, no docs.
filter = path: type:
let rel = pkgs.lib.removePrefix (toString ../. + "/") (toString path);
in rel == "Cargo.toml" || rel == "Cargo.lock"
|| rel == "crates" || pkgs.lib.hasPrefix "crates/" rel;
};
toolkit = static.rustPlatform.buildRustPackage {
pname = "boxmaker-toolkit";
version = "0.1.0";
inherit src;
cargoLock.lockFile = ../Cargo.lock;
cargoBuildFlags = [ "-p" "toolkit" ];
doCheck = false; # the tests run in `make gate`, on the development machine
};
root = pkgs.runCommand "boxmaker-tools-root" { } ''
mkdir -p $out/bin $out/etc/ssl/certs $out/tmp $out/run/egress
# busybox first: `cp -a` keeps the store's read-only modes, so make the copies writable after.
cp -a ${static.busybox}/bin/. $out/bin/
chmod -R u+w $out/bin
cp ${toolkit}/bin/toolkit $out/bin/toolkit
cp ${static.curl.bin}/bin/curl $out/bin/curl
cp ${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt $out/etc/ssl/certs/ca-certificates.crt
'';
in
pkgs.dockerTools.buildLayeredImage {
name = "boxmaker-tools";
tag = "latest";
contents = [ root ];
config.WorkingDir = "/tmp";
}
+10 -2
View File
@@ -6,6 +6,13 @@ Newest first. A decision that changes `docs/design.md` lands in the same commit
| Date | Decision | Reason | | Date | Decision | Reason |
|---|---|---| |---|---|---|
| 2026-09-23 | M4 is split: M4a is conversations over Mattermost (`gatewayd`, the secret store), M4b is approvals over Mattermost. Spec `docs/specs/2026-09-23-m4a-gateway.md`. | As with M2 and M3: the authority-bearing half is reviewed separately, and the open question about long approvals lands in M4b. |
| 2026-09-23 | `gatewayd` reaches Mattermost over TLS, by default at the tailnet name `https://straylight.scylla-hammerhead.ts.net`; plain HTTP stays possible by URL. TLS is `rustls` 0.23 with `ring` and `rustls-native-certs` (the host's trust store, plus an optional CA file); `deny.toml` gains ISC and BSD-3-Clause. | Owner: portability, and a bigger picture that favours the tailnet name over host loopback. The host's store keeps a privately installed CA trusted; `webpki-roots` would not, and `aws-lc-rs` is a larger C build for nothing needed. |
| 2026-09-23 | Our own blocking WebSocket client (RFC 6455), adversarially tested; SHA-1 for its handshake check lives in `proto`. | `tungstenite` would add about ten crates for a few hundred lines of protocol. |
| 2026-09-23 | Secrets: one `SecretStore`, three backends chosen per secret: a systemd credential (encrypted at rest to the TPM and host key), an environment variable, or an owner-only file. Proposed as brief change P15. | Owner: a generic backend for portability. systemd 260 on straylight encrypts user-scoped credentials to its TPM (checked). |
| 2026-09-23 | In Mattermost: a thread is a session, and a top-level message starts a new one. Messages that arrive while a turn runs are sent together as the next turn. No thinking and no status in Mattermost: Boxmaker shows as typing while it works, then posts the answer. A message starting with `!` is a command to Boxmaker and never reaches the model (`!!` escapes it); M4's commands are `!approve <id>` and `!deny <id> [reason]`. | Owner's choices. `/` cannot be used: Mattermost's clients treat it as a slash command and do not post it (checked in the web client's source, v11.11.0), and a real slash command would need a listening port. |
| 2026-09-23 | M4b, agreed ahead of its spec: one `gateway.sock` connection per approval, open while it is pending; the approval posted in the asking thread or the owner's direct messages; answered only by an allowlisted ✅/❌ reaction on the approval post or `!approve`/`!deny` in its thread, after the post and while pending; long approvals in up to 4 parts with only the last answerable, else `bxctl` only; `bxctl deny` as another name for `refuse`. | Keeps `gatewayd` away from `admin.sock`; a closed connection withdraws the post; a hidden tail of an approval is where injected text would hide. |
| 2026-09-22 | M3b spec (`docs/specs/2026-09-22-m3b-runner.md`), draft for the owner's review, written for an overnight run. New: the tools' argument structs and the host rules move to `proto`, shared by `brokerd` and `toolkit`; `toolkit` exits 0/1/2 (done / the tool's own error, shown to the model / misuse, a fixed sentence); the egress proxy refuses any allowed name that resolves to a non-public address (including the tailnet's `100.64/10` and `::/96`); grant paths may not contain `:` or `,`; `[runner]` names the image by digest; tool time limits sit under `loopd`'s 120 s broker timeout, so `shell` defaults to 100 s. | A grant allows a host on the internet, not what its name points at; two programs reading the same data need one definition; `--volume` uses `:` and `,` as separators. The shell limit follows from the M3a deadline in `BrokerPort`. |
| 2026-09-22 | M3a review fixes. `loopd` waits at most a day after a pending frame and `brokerd` refuses `[approvals] ttl_ms` over a day. A socket whose directory is `/` or a symbolic link is refused at start. A listener out of file descriptors or memory pauses and retries instead of stopping `brokerd`. `brokerd` and `bxctl` share one rule for audit log file names (`proto::is_audit_log_name`, real months and days only). The fixes were made by the design model, and the fix commits were reviewed by a separate agent before the merge. | A far `expires` parked a turn for ever; `chmod` through a link changed its target; an idle-connection flood could stop the daemon; the two components disagreed about which files were the log. Ornith was under heavy contention. | | 2026-09-22 | M3a review fixes. `loopd` waits at most a day after a pending frame and `brokerd` refuses `[approvals] ttl_ms` over a day. A socket whose directory is `/` or a symbolic link is refused at start. A listener out of file descriptors or memory pauses and retries instead of stopping `brokerd`. `brokerd` and `bxctl` share one rule for audit log file names (`proto::is_audit_log_name`, real months and days only). The fixes were made by the design model, and the fix commits were reviewed by a separate agent before the merge. | A far `expires` parked a turn for ever; `chmod` through a link changed its target; an idle-connection flood could stop the daemon; the two components disagreed about which files were the log. Ornith was under heavy contention. |
| 2026-09-18 | M3a plan checks, `brokerd`. Every tool request gets a `Decision` record, including those denied `grants_invalid` or `state_unreadable`; forbidden kinds get none. An unreadable session state is recorded as `secret`, untrusted. A refusal that cannot be recorded is `error internal`, not `ok`. The re-decision's outcome is the matched grant's mode. A pending frame that cannot be sent is handled like a lost connection. Expiry lives in `admin`. A request frame without a read timeout is accepted for M3a. | Found while writing the reference for tasks 10 to 15: the spec left each case open, and `bxctl refuse` would have reported success for a refusal that was not on disk. | | 2026-09-18 | M3a plan checks, `brokerd`. Every tool request gets a `Decision` record, including those denied `grants_invalid` or `state_unreadable`; forbidden kinds get none. An unreadable session state is recorded as `secret`, untrusted. A refusal that cannot be recorded is `error internal`, not `ok`. The re-decision's outcome is the matched grant's mode. A pending frame that cannot be sent is handled like a lost connection. Expiry lives in `admin`. A request frame without a read timeout is accepted for M3a. | Found while writing the reference for tasks 10 to 15: the spec left each case open, and `bxctl refuse` would have reported success for a refusal that was not on disk. |
| 2026-09-18 | M3a plan checks, audit. `DecisionRecord::Allowed {}` and `Ask {}` are empty struct variants. The resumed verifier accepts a break naming an older file inside a failed region too. When the previous file's last line does not parse, an ordinary start verifies the whole log. A `Recovery` that describes no line is a failure. `abandoned` and `unfinished` both name the decision's `seq`. The report carries what a `Recovery` or `AcceptedBreak` must hold. `--accept-break` with nothing to accept writes nothing. | serde ignores `deny_unknown_fields` on unit variants of an internally tagged enum, so `{"outcome":"allowed","x":1}` decoded; found by `strict.rs` in two areas, and the struct-variant fix was kept over a hand-written `try_from` as the smaller one that also refuses `"reason":null`. As specified, a correctly accepted break could stop every later start while `--accept-break` said "nothing to accept". | | 2026-09-18 | M3a plan checks, audit. `DecisionRecord::Allowed {}` and `Ask {}` are empty struct variants. The resumed verifier accepts a break naming an older file inside a failed region too. When the previous file's last line does not parse, an ordinary start verifies the whole log. A `Recovery` that describes no line is a failure. `abandoned` and `unfinished` both name the decision's `seq`. The report carries what a `Recovery` or `AcceptedBreak` must hold. `--accept-break` with nothing to accept writes nothing. | serde ignores `deny_unknown_fields` on unit variants of an internally tagged enum, so `{"outcome":"allowed","x":1}` decoded; found by `strict.rs` in two areas, and the struct-variant fix was kept over a hand-written `try_from` as the smaller one that also refuses `"reason":null`. As specified, a correctly accepted break could stop every later start while `--accept-break` said "nothing to accept". |
@@ -52,7 +59,7 @@ Newest first. A decision that changes `docs/design.md` lands in the same commit
| Repository licence. Crates are `publish = false` until it is chosen. | before publishing | | Repository licence. Crates are `publish = false` until it is chosen. | before publishing |
| Audit integrity beyond a self-contained hash chain (for example anchoring the chain head outside `brokerd`). Deferred by the owner on 2026-09-17 to a later revision. | after v0 | | Audit integrity beyond a self-contained hash chain (for example anchoring the chain head outside `brokerd`). Deferred by the owner on 2026-09-17 to a later revision. | after v0 |
| How an approval whose arguments do not fit one Mattermost post is shown. Settled on 2026-09-18: what is being approved is never truncated, in any channel, because the unseen tail is where injected text would put its payload; `bxctl` prints arguments in full. Leading option for Mattermost: a thread of numbered chunks split on character boundaries of the escaped text, where only a reply or reaction on the last post counts and that post repeats the id, the chunk count and the hash of the whole; above some number of chunks, the post points to `bxctl approvals` and approval is local only. The post size limit and rate limits must be checked against the Mattermost API reference first. | M4 | | How an approval whose arguments do not fit one Mattermost post is shown. Settled on 2026-09-18: what is being approved is never truncated, in any channel, because the unseen tail is where injected text would put its payload; `bxctl` prints arguments in full. Leading option for Mattermost: a thread of numbered chunks split on character boundaries of the escaped text, where only a reply or reaction on the last post counts and that post repeats the id, the chunk count and the hash of the whole; above some number of chunks, the post points to `bxctl approvals` and approval is local only. The post size limit and rate limits must be checked against the Mattermost API reference first. | M4 |
| Secret store backend, and where the v0 file's key lives. straylight has no secrets manager today. Moved from M3 on 2026-09-18. | M4 | | Secret store: settled 2026-09-23 (systemd credentials, environment, owner-only file; P15 proposed). Where `brokerd` reads tool secrets from, and whether `SecretStore` moves out of `gatewayd` into a shared place, is decided when a tool first needs a secret. | when a tool needs a secret |
| Whether cloud-led sessions are ever allowed, and for which data classes. | M6 | | Whether cloud-led sessions are ever allowed, and for which data classes. | M6 |
| Where the M5b embedding model runs. The brief says a separate `llama-server` instance, but under the shared router it would be a third model against `--models-max 2` and could unload Ornith or Laguna. Owner's note, 2026-09-17: more compute is likely by then (possibly one to three DGX Spark machines), so do not design around today's memory limit. | M5b | | Where the M5b embedding model runs. The brief says a separate `llama-server` instance, but under the shared router it would be a third model against `--models-max 2` and could unload Ornith or Laguna. Owner's note, 2026-09-17: more compute is likely by then (possibly one to three DGX Spark machines), so do not design around today's memory limit. | M5b |
@@ -81,9 +88,10 @@ Later changes to the brief:
|---|---|---| |---|---|---|
| 2026-09-17 | The roles table gained a one-sentence Purpose column, and the crates that are not roles got a purpose table under Code constraints. No behaviour changed. | Owner's request: a reader, human or agent, should learn what each piece is for before what it holds. | | 2026-09-17 | The roles table gained a one-sentence Purpose column, and the crates that are not roles got a purpose table under Code constraints. No behaviour changed. | Owner's request: a reader, human or agent, should learn what each piece is for before what it holds. |
| 2026-09-18 | P14, State: the list gained `broker/sessions/<id>.json`, each session's taint and untrusted flag, written only by `brokerd`. | The M3a spec keeps this state in files the list did not name. It can be rebuilt from the audit log's `result` records, so the audit log stays the record of truth. Owner: a plain file also makes debugging easier. | | 2026-09-18 | P14, State: the list gained `broker/sessions/<id>.json`, each session's taint and untrusted flag, written only by `brokerd`. | The M3a spec keeps this state in files the list did not name. It can be rebuilt from the audit log's `result` records, so the audit log stays the record of truth. Owner: a plain file also makes debugging easier. |
| 2026-09-22 | P13, Authority contract 4: tool containers run from one image built from source by Nix and named by digest; nothing is pulled at call time. | Approved with the M3 design (2026-09-18); applied when the M3b spec came to rely on it. A pull at call time would be unlisted egress. |
## Proposed changes to the design brief ## Proposed changes to the design brief
| # | Change | Reason | | # | Change | Reason |
|---|---|---| |---|---|---|
| P13 | Authority contract 4: add "Tool containers run from one image built from source by Nix and named by digest; nothing is pulled at call time." | The brief does not say where tool images come from; a pull at call time would be unlisted egress. Decided 2026-09-18 above. | | P15 | State, the secrets line, becomes: "Secrets: behind a `SecretStore` trait with three backends, chosen per secret: a systemd credential (encrypted at rest; the key is the host's TPM and host key, not stored beside it), an environment variable, or a file that only the owner can read. The credential backend is the default; the other two put the secret in plaintext, in the process environment or on disk, and are for portability. None in config, none in the repo." | Owner's decision of 2026-09-23. The brief's "No plaintext secrets on disk" does not allow the file backend. |
+2 -2
View File
@@ -4,8 +4,8 @@ Every external crate has a row here. `scripts/check-dep-docs.sh` enforces it.
| Crate | Version | Used by | Why | | Crate | Version | Used by | Why |
|---|---|---|---| |---|---|---|---|
| `serde` | 1.0.229 | `proto`, `brokerd` | Derives serialization for every shared type. MIT OR Apache-2.0. | | `serde` | 1.0.229 | `proto`, `brokerd`, `toolkit` | Derives serialization for every shared type. MIT OR Apache-2.0. |
| `serde_json` | 1.0.151 | `proto`, `brokerd` | JSON for frames and log files. MIT OR Apache-2.0. | | `serde_json` | 1.0.151 | `proto`, `brokerd`, `toolkit` | JSON for frames and log files. MIT OR Apache-2.0. |
| `humantime` | 2.4.0 | `proto` | RFC 3339 timestamps. No dependencies of its own. MIT OR Apache-2.0. | | `humantime` | 2.4.0 | `proto` | RFC 3339 timestamps. No dependencies of its own. MIT OR Apache-2.0. |
| `toml` | 1.1.6 | `proto` tests; `brokerd` and `bxctl` from M3 | Parses grant files. MIT OR Apache-2.0. | | `toml` | 1.1.6 | `proto` tests; `brokerd` and `bxctl` from M3 | Parses grant files. MIT OR Apache-2.0. |
| `emsha` | 1.0.4 | `proto` | SHA-256. Written by the owner; no dependencies, no `unsafe`. Checked against `sha256sum` on 3,204 inputs. | | `emsha` | 1.0.4 | `proto` | SHA-256. Written by the owner; no dependencies, no `unsafe`. Checked against `sha256sum` on 3,204 inputs. |
+2 -1
View File
@@ -145,7 +145,8 @@ a multi-turn tool conversation and tool parsing had no failures in 20 trials.
3. No matching grant means deny. `ask` routes an approval request to the owner through `gatewayd` 3. No matching grant means deny. `ask` routes an approval request to the owner through `gatewayd`
and the turn suspends until answered or expired. and the turn suspends until answered or expired.
4. Each approved call runs in a fresh rootless container: no network unless granted, only granted 4. Each approved call runs in a fresh rootless container: no network unless granted, only granted
paths mounted, only the granted secret injected. paths mounted, only the granted secret injected. Tool containers run from one image built from
source by Nix and named by digest; nothing is pulled at call time.
5. **Audit log**: append-only JSONL, hash-chained, one record per decision including denials. 5. **Audit log**: append-only JSONL, hash-chained, one record per decision including denials.
6. **Data classes**: every tool result is labelled `public`, `private`, or `secret`. A session's 6. **Data classes**: every tool result is labelled `public`, `private`, or `secret`. A session's
taint is the union of labels it has seen. The cloud-consult tool is refused for tainted sessions taint is the union of labels it has seen. The cloud-consult tool is refused for tainted sessions
+7
View File
@@ -8,3 +8,10 @@ allowed.
| Development | `cargo` | crates.io | Downloading the crates listed in `docs/dependencies.md` | | Development | `cargo` | crates.io | Downloading the crates listed in `docs/dependencies.md` |
| Development | `make verify-device`, `tools/check-m3a-device.sh` | straylight's `llama-server` | Checks against the real model through a private `inferproxy`; the script also reads `/slots` with `curl` first | | Development | `make verify-device`, `tools/check-m3a-device.sh` | straylight's `llama-server` | Checks against the real model through a private `inferproxy`; the script also reads `/slots` with `curl` first |
| Development | `make audit` | github.com/rustsec/advisory-db | The RustSec advisory database, fetched by `cargo deny check advisories` | | Development | `make audit` | github.com/rustsec/advisory-db | The RustSec advisory database, fetched by `cargo deny check advisories` |
| Run time (M2a) | `inferproxy` | the `llama-server` router on the host (`straylight:11434`) | Every inference request `loopd` makes, forwarded byte for byte |
| Run time (M3b) | `brokerd`'s egress proxy container, for one `http_fetch` call | only the hosts the call's grant names, on port 443, at public addresses | The tool's HTTPS request, end to end; the proxy sees only the host name |
| Run time (M4a) | `gatewayd` | the owner's Mattermost at `https://straylight.scylla-hammerhead.ts.net` (tailnet) | REST calls and one WebSocket: the owner's messages in, Boxmaker's answers out. Mattermost stores messages unencrypted in its database; the server is the owner's |
The two run-time rows for M2a and M3b were missing until 2026-09-23; they were added with the M4a
spec.
+4
View File
@@ -57,6 +57,10 @@ How it is used:
| T20 | The gate runs on two platforms, Talos (Linux) and the Mac (macOS), and they differ where the tests touch the OS: sockets, file sync, `/tmp`. Accept a task only when the gate passes on both. Code that sets a socket option after the peer may have closed breaks on macOS only. | M3a: `BrokerPort`, the HTTP client and the brokerd test client passed on Talos and failed on the Mac from task 13 on; it was found only when the gate was first run on the Mac. | | T20 | The gate runs on two platforms, Talos (Linux) and the Mac (macOS), and they differ where the tests touch the OS: sockets, file sync, `/tmp`. Accept a task only when the gate passes on both. Code that sets a socket option after the peer may have closed breaks on macOS only. | M3a: `BrokerPort`, the HTTP client and the brokerd test client passed on Talos and failed on the Mac from task 13 on; it was found only when the gate was first run on the Mac. |
| T21 | When two components must agree on a set (which files are the log, which names are ids), give the tests one case that walks both. Each half was tested alone and they still disagreed. | M3a finding 2: `brokerd` accepted `2026-0x-18.jsonl` as a log file and `bxctl audit verify` ignored it, calling the log `ok` while leaving out half its records. | | T21 | When two components must agree on a set (which files are the log, which names are ids), give the tests one case that walks both. Each half was tested alone and they still disagreed. | M3a finding 2: `brokerd` accepted `2026-0x-18.jsonl` as a log file and `bxctl audit verify` ignored it, calling the log `ok` while leaving out half its records. |
| T22 | List the fail-closed states a task creates, with their runbook anchors, in the task itself. A script can check that a pointer names an existing entry; nothing can check for a pointer that was never written. | M3a finding 3: four startup failures in `serve` and `main` print no pointer, and the spec's own list of pointers omitted them. | | T22 | List the fail-closed states a task creates, with their runbook anchors, in the task itself. A script can check that a pointer names an existing entry; nothing can check for a pointer that was never written. | M3a finding 3: four startup failures in `serve` and `main` print no pointer, and the spec's own list of pointers omitted them. |
| T23 | A test that writes a script and then runs it must hold a lock shared by every test in its binary that starts a process. Otherwise another test's fork can hold the new script open for writing at that moment, and running it fails with "Text file busy" (ETXTBSY), about once in seven runs. Give the lock in the support file and say why. | M3b plan checks: the fake `curl` tests failed 8 times in 40 until every forking test took `serial()`; then 0 in 60. |
| T24 | A task that changes a `Cargo.toml` must stage `Cargo.lock` in its `git add` line. Better, put `Cargo.lock` in every task's `git add`; it is a no-op when unchanged. | M3b task 04: committed correctly, left the lock behind, and the driver stopped on an unclean tree. |
| T25 | Size a task by the largest function the model must hold in one turn, not by the task. Ornith writes one function with a few branches well; a function with half a dozen branches and threads (M3b's `run`) it plans in its head until the turn runs out, with nothing written. Give such a task a compiling skeleton with the big function already written as glue over small `todo!()` helpers, and say to fill one at a time with `cargo check` between. | M3b task 11: four sessions. Two wrote nothing; a whole-file skeleton got five of six functions; the finer skeleton finished it in ten minutes, and tasks 12 and 13 followed without a stop. |
| T26 | Replay each task's end state on its own, **and** read the task file against the reference for anything the reference has that the task does not ask for. The replay proves the tests can pass; only the reading finds a field the reference reads through a getter the task never mentions. | M3b task 11: the task stored `egress_dir` without the reference's getter, so the field was never read and clippy failed; the replay passed because the reference had the getter. |
## What worked and should be kept ## What worked and should be kept
+86
View File
@@ -6,6 +6,22 @@ reviewer adds findings under "Reviews" once per milestone.
| Task | Date | Status | Gate runs | First gate | Deviations | Notes | Model | | Task | Date | Status | Gate runs | First gate | Deviations | Notes | Model |
|---|---|---|---|---|---|---|---| |---|---|---|---|---|---|---|---|
| M3b/17-toolkit-nits | 2026-09-23 | done | 1 | pass | none | Three small fixes. `fetch.rs`: replaced `std::thread::spawn` with a `Builder::new().spawn` match that kills and waits on a spawn error and returns `Outcome::tool_error("http_fetch: cannot start a thread: {e}")`. `input.rs` and `files.rs`: replaced `MAX_INPUT as u64 + 1` / `MAX_READ as u64 + 1` with `u64::try_from(MAX_*).map_or(u64::MAX, |n| n.saturating_add(1))`. `main.rs` `parse_egress_proxy`: the first check is now `args.len() != 5 || args.first()?.as_bytes() != b"egress-proxy"` so a longer list goes to the tool form (exit 2). Copied `tests/egress_form.rs`; the one test failed before the fix and passed in 0.02s after. `grep "thread::spawn\| as u64" crates/toolkit/src/` prints nothing. `make gate` prints `gate: ok` first run. | ? |
| M3b/16-brokerd-log-escaping | 2026-09-23 | done | 1 | pass | none | Escaped the container's standard error before logging it (M3b review finding 5) and prefixed/quoted three small texts (finding 7). `container.rs` `answer`: exit 2 logs `brokerd: {name}: the tool could not run: {err:?}` instead of the raw stderr; exit 125..=127 logs `brokerd: podman could not start {name}: {err:?}\n{RUNBOOK}` (was `{err}\n{RUNBOOK}`); the `_` arm logs `brokerd: container {name} exited {status}: {err:?}` (the trailing `\n{err}` moved inside the debug format). `start_egress`: the proxy's `podman run -d` failure now logs `{stderr:?}`. `main.rs`: the runtime notice prints `brokerd: {runtime_notice}`. `config.rs`: the image and memory `[runner]` errors use `{:?}` so the bad value is quoted. Copied `tests/container_log.rs` and `tests/notices.rs`; 4, 2, 2 and 7 passed; `make gate` prints `gate: ok` first run. | ? |
| M3b/15-brokerd-pipes-grace | 2026-09-23 | done | 1 | pass | none | Fixed the two M3b review findings in `pipes.rs`. Added `pub const GRACE: Duration = Duration::from_secs(2)`; replaced the `JoinHandle`-holding `Io` with one holding `Option<Receiver<(Vec<u8>, bool)>>` for stdout/stderr and added the `Finished { out, truncated, err, open }` struct. `Io::start` now returns `std::io::Result<Io>`: the stdin writer is started with `Builder::new().spawn(...)` and its handle dropped (never joined, a spawn error returned with `?`); a new private `reader(pipe, cap) -> io::Result<Receiver<...>>` starts one detached reader per pipe and returns a spawn error, used for both stdout (`cap`) and stderr (`err_cap`). `Io::finish(grace)` sets one `until = Instant::now() + grace` and calls a private `collect(rx, until)` per receiver: `recv_timeout(until.saturating_duration_since(now))``Ok` keeps bytes, `Timeout` sets `open`, `Disconnected` (panicked reader) counts closed; `open` is `stdout_open || stderr_open`. `read_capped` unchanged in behavior, now `cap.saturating_sub(kept.len())` and `chunk.get(..take).unwrap_or_default()` (no indexing, no wrapping subtraction). `container.rs`: added `pub const OUTPUT_OPEN` after `TIMED_OUT`, `use crate::pipes::{GRACE, Io}`, and replaced the `run_container` tail with the task's `Io::start` match (on spawn error, `podman kill`/`rm -f`, `child.kill()`/`wait()`, log + `Unavailable(CANNOT_START)`), then `io.finish(GRACE)` and the `done.open && status.is_some()` check returning `Failed(OUTPUT_OPEN)` (a timed-out call, `status` None, still answers `TIMED_OUT`). All `std::thread::spawn` gone from `crates/brokerd/src/`. Copied `tests/container_grace.rs`; 4 passed in ~4.4s, container 11, container_egress 6, all five runs; `make gate` prints `gate: ok` first run. | ? |
| M3b/14-brokerd-pipes-module | 2026-09-23 | done | 1 | pass | none | Pure move: cut `struct Io`, `impl Io { start, finish }`, and `fn read_capped` from `container.rs` and pasted them into `crates/brokerd/src/pipes.rs` with bodies unchanged and `pub(crate)` visibility; `Io::start` now calls `crate::container::STDERR_KEPT` (the constant stays in `container.rs` since the tests import it). Added the module doc comment and the `use std::io::{Read, Write}`, `use std::process::Child`, `use std::thread::JoinHandle` lines. `lib.rs` gained `pub mod pipes;` between `ledger` and `podman`. `container.rs` gained `use crate::pipes::Io;` and lost the `Read, Write` and `JoinHandle` imports the compiler reported unused; nothing else changed. `cargo fmt --all` reflowed the `STDERR_KEPT` call line in `start` to wrap. `cargo check`/clippy clean; container 11, container_egress 6, serve_runner 2 pass; `grep "struct Io\|fn read_capped" container.rs` prints nothing and `container.rs` is 350 lines. `make gate` prints `gate: ok` on the first run. | ? |
| M3b/13-brokerd-serve-runner | 2026-09-23 | done | 1 | pass | none | The `Podman` runtime was already written in task 12 (`crates/brokerd/src/container.rs`), so this task only wired it into `serve`. In `main.rs`: after the config loads, added `let log: Arc<dyn Fn(&str) + Send + Sync> = Arc::new(|line| eprintln!("{line}"))`; chose the runtime and its notice with a match on `&cfg.runner``Some(runner)` captures `runner.image` first then `Box::new(Podman::new(runner.clone(), cfg.egress_dir(), Arc::clone(&log)))` with notice `tools run in containers from {image}`; `None``Box::new(Refusing)` with notice `no [runner] section: every tool call is refused`. Passed `runtime` and `log` to `serve::start` (its signature needed no change — it already took `Box<dyn Runtime>` and `Arc<dyn Fn(&str)+Send+Sync>`), and printed the notice with `eprintln!` after the `serving tools on … and approvals on …` line. Added `use brokerd::container::Podman` and `Runtime` to the imports. Copied `tests/serve_runner.rs`; the 2 new tests plus the 9 serve tests pass; `make gate` prints `gate: ok` on the first run. | ? |
| M3b/12-brokerd-egress | 2026-09-23 | done | 1 | pass | none | Wrote `crates/brokerd/src/container.rs` (432 lines). Added consts `EGRESS_WAIT` (5s) and `EGRESS_POLL` (20ms); `Podman` gained private `egress_wait` (set to `EGRESS_WAIT` by `new`) and the public `with_egress_wait`. Added private `cannot_launch(podman, e)` (the task-11 step 3 log `brokerd: cannot start {path}: {e}\n{RUNBOOK}`), which `spawn` now calls, and `cannot_make(dir, e)` returning `Err(Unavailable(CANNOT_START))` with `brokerd: cannot make {dir}: {e}\n{RUNBOOK}`. `Runtime::run` now branches on `spec.egress()`: `None` runs `tool_args(spec, &runner, &name, None)` as before; `Some(hosts)` computes `dir = egress_dir.join(&name)`, calls `start_egress`, then `tool_args(spec, &runner, &name, Some(&dir))`. `start_egress` creates the `EgressGuard` first (so every return cleans up), then step 1: `DirBuilder::new().recursive(true).mode(0o700).create(egress_dir)`, `set_permissions(egress_dir, 0o700)` anyway, `remove_dir_all(dir)` if it exists (NotFound ok), `DirBuilder::new().mode(0o700).create(dir)` non-recursively, each failure via `cannot_make`; step 2: `Command::new(&runner.podman).args(egress_args).stdin(null).stdout(null).stderr(piped()).output()`, non-success logs `brokerd: podman could not start {name}-egress: {stderr}\n{RUNBOOK}` and returns Unavailable, launch error uses `cannot_launch`; step 3: wait for `dir.join("egress.sock")` every 20 ms until `egress_wait`, then log `brokerd: {name}-egress did not make its socket within {ms} ms\n{RUNBOOK}` and return Unavailable; step 4 `Ok(_guard)`. `EgressGuard<'a>` holds `&'a Podman`, the container name `<name>-egress` and the directory; its `Drop` runs `podman rm -f <name>-egress` (the task-11 helper) then `remove_dir_all(dir)` (NotFound ok, else log `brokerd: cannot remove {dir}: {e}`). Added `use std::os::unix::fs::{DirBuilderExt, PermissionsExt};`. `cargo fmt` put the new `impl Podman` block after `impl Runtime`. 6 egress + 11 container tests pass ten runs in a row. First gate failed on `start_egress` being inside `impl Runtime` (not a trait member) and missing `DirBuilderExt`/`PermissionsExt`; then on the guard being created at the end instead of the start (the two "nothing left" tests need the `rm -f <name>-egress` call on a failed proxy), fixed by moving `EgressGuard::new` to the top and returning it; then on `unused_variable` for the drop guard and `mismatched_lifetime_syntaxes` on the return type (fixed to `EgressGuard<'_>`), the guard renamed `_guard` (1.98 still lints drop-only bindings). `make gate` prints `gate: ok`. | ? |
| M3b/11-brokerd-container | 2026-09-23 | done | 2 | fail | none | Wrote `crates/brokerd/src/container.rs`: the fixed-sentence constants (`COULD_NOT_RUN`, `CANNOT_START`, `KILLED`, `TIMED_OUT`, `UNEXPECTED`), `RUNBOOK`, `POLL`, `STDERR_KEPT` and the `Log` type; `Podman { runner, egress_dir, log, next: AtomicU64 }` with `new` (`next` starts at 0) and the public `egress_dir`. `Runtime::run` = `next.fetch_add` for the container number, `podman::container_name`, `podman::tool_args(spec, &runner, &name, None)`, `spec.arguments().canonical_json()`, `runner.time_limit(tool)`, then `run_container` (already written). `spawn` = `Command::new(&runner.podman)` with all three streams piped, on failure log `brokerd: cannot start {path}: {e}\n{RUNBOOK}` and return None. `Io::start` takes the three pipes and starts one thread each: write `input` then drop stdin, `read_capped` stdout with `cap`, `read_capped` stderr with `STDERR_KEPT`; `finish` joins all three (a missing or panicked thread counts as empty via `join().ok()`/`unwrap_or_default()`). `read_capped` reads past the cap with an 8 KiB buffer, keeping the first `cap` bytes and setting `truncated`, taking `remaining.min(n)` so it never indexes past what it kept. `wait` loops `child.try_wait()`, and once `limit` has elapsed runs `podman kill <name>` then `podman rm -f <name>` (each via the private `podman` helper — `.status()` with the three streams null, logging a line on non-success), then `child.kill()`/`child.wait()`, returning None. `podman` helper uses `&self.runner.podman`, not a `podman` field. `answer` follows the table: None→TIMED_OUT (log a line naming the container); exit 0/1→Ok with content `from_utf8_lossy(out)`; exit 2→COULD_NOT_RUN (log stderr); 125/126/127→CANNOT_START (log stderr then `\n{RUNBOOK}`); 137→KILLED; anything else incl. signal→UNEXPECTED (log status and stderr). The tool's output never enters a `RunError`; only the six constants do. Deleted the skeleton paragraph. Added `pub mod container;` to lib.rs after `config`. Copied `tests/support/fake_podman.rs` and `tests/container.rs`. `container` 11 passed ten runs; first gate failed on clippy `manual_range_patterns` (`125|126|127``125..=127`), fixed; `make gate` prints `gate: ok`. | ? |
| M3b/10-brokerd-podman-args | 2026-09-23 | done | 2 | fail | none | Wrote `crates/brokerd/src/podman.rs`: the `EGRESS_MOUNT`/`EGRESS_SOCKET`/`TOOLKIT` constants; `container_name` = `boxmaker-<session>-<call>-<n>`; the private `hardening(pids, memory)` emitting the six shared flags (`--read-only` to `--memory=…`) once; the private `volume(host, container, mode)` built with `push` so a directory need not be UTF-8; `tool_args` = `run --rm -i --name=<name> --label=boxmaker=tool --network=none`, then hardening with `runner.pids`/`runner.memory`, `--tmpfs=/tmp:rw,size=64m,mode=1777`, one `--volume=<path>:<path>:ro|rw` per `spec.mounts()` in order, the egress volume when `egress` is Some, then `<image> /bin/toolkit <tool>`; `egress_args` = `run -d --rm --name=<name>-egress --label=boxmaker=egress --network=<egress_network>`, hardening 64/128m, the egress volume, then `<image> /bin/toolkit egress-proxy --socket /run/egress/egress.sock --allow <hosts joined with ','>`. `RunSpec` gained private `session: SessionId` and `call: CallId` as its first two fields with `session()`/`call()` getters; `run` fills them from `decision.request()`, and the `compile_fail` doctest struct literal gains the two fields first. Copied `tests/podman_args.rs` and the six `podman/*.args` golden files. `podman_args` 7 passed, `runner` 8 passed, all doctests pass. First gate failed on clippy `redundant_closure` (`.map(|f| OsString::from(f))``.map(OsString::from)`). `make gate` prints `gate: ok`. | ? |
| M3b/09-brokerd-runner-config | 2026-09-23 | done | 2 | fail | none | Wrote `crates/brokerd/src/config.rs`: the `Runner` struct (`podman`, `image`, `egress_network`, `output_cap`, `memory`, `pids`, `read_file_ms`, `write_file_ms`, `shell_ms`, `http_fetch_ms`) with `#[serde(deny_unknown_fields)]` and one private `default_…()` per defaulted field; `image` is required with no default. `Config` gained `runner: Option<Runner>` with `#[serde(default)]`. `Runner::time_limit` maps each `ToolName` to its matching `_ms` field as a `Duration`. `Config::egress_dir()` joins `<home>/run/egress`. `load` runs, after the `ttl_ms` check and only when `runner` is `Some`, the four checks in order (first problem wins): image must be `<name>@sha256:<64 lowercase hex>` via `rsplit_once("@sha256:")` with a non-empty name and exactly 64 `0-9a-f`, memory must be digits then one of b/k/m/g (`valid_memory`), `egress_network`/`podman` non-empty, and the six non-negative fields checked for zero in order — each returns `ConfigError::Invalid`. `Config::parse` runs none of them. Copied `tests/config_runner.rs`, the five `runner_*.toml` fixtures, and the new `support/rig.rs`, which builds `Config` with `runner: None`. All brokerd tests pass; `config` and `config_runner` each 7 passed. First gate failed on clippy `incompatible_msrv`: `PathBuf::is_empty()` is stable since 1.98 but the MSRV is 1.95, fixed with `as_os_str().is_empty()` (the pattern the file already used for the socket paths). `make gate` prints `gate: ok`. | ? |
| M3b/08-toolkit-egress-proxy | 2026-09-23 | done | 2 | fail | none | Copied `docs/plans/M3b/files/crates/toolkit/tests/egress.rs` to `crates/toolkit/tests/`. Wrote `crates/toolkit/src/egress.rs`: the reply-code and timeout constants; `Allow { patterns: Vec<String> }` with `parse` splitting on ',' and rejecting any piece that fails `proto::hosts::valid_host_pattern` (an empty piece like `","`/`"x,"`/`""` errors) and `permits` = `valid_host(host) && any host_matches`; the `Dial` trait and `SystemDial` (`to_socket_addrs()?.collect()` and `connect_timeout`); `Proxy { allow, dial: Arc<dyn Dial>, handshake }` with `new`/`with_handshake_timeout`/`serve`/`handle`. `handle` shares one deadline across the whole handshake: `read_n` reads exactly the byte count the protocol gives, setting the read timeout to `deadline.checked_duration_since(now)` before each read and returning `None` (stop, no reply) on no time left, a timeout, a failed read, or 0 bytes — so a client trickling one byte per 100 ms is still cut at the deadline. The 11 exits are in order (egress.rs:133 version, :144 methods 0, :154 version/reserved, :160 command, :164 kind without reading the address, :172 zero length, :187 non-UTF-8, :190 port/host, :195 resolve / :203 no public addr, :208 connect, :211 success reply); step 9 takes the first address where `crate::addr::is_public` is true via `.find`, never trying the skipped non-public ones. `serve` is a single-threaded accept loop with an `AtomicUsize` count: over the limit is dropped at once, otherwise a `std::thread::Builder` thread handles it and uncounts itself on return, and `serve` never joins (so the second client is not starved). After the handshake the read timeout is cleared and two `try_clone`'d threads copy both ways with a half-close each way, then join. `main.rs` gained the `egress-proxy --socket <path> --allow <list>` form before the tool form via `std::env::args_os().skip(1)` (without `skip(1)` the first element is the program path and the form never matches); `Allow::parse`/bind (no removal first — a pre-existing file is a mistake)/`serve` errors print to stderr and exit 2. `lib.rs` gained `pub mod egress;`. First gate failed on clippy `question_mark` (accept loop → `stream?`) and `manual_contains` (`methods.contains(&0)`); fixed both and `make gate` prints `gate: ok`. 15 egress tests pass ten runs in a row. | ? |
| M3b/07-toolkit-addr | 2026-09-23 | done | 1 | pass | none | Wrote `crates/toolkit/src/addr.rs`: `is_public(ip)` matches on `IpAddr` and dispatches to `is_public_v4`/`is_public_v6`. The IPv4 function checks the 13 refused ranges in table order with early returns (lines 16-52) then returns `true`. The IPv6 function checks the two "judge as IPv4" rows first — `is_ipv4_mapped` for `::ffff:0:0/96` (lines 60-63) and `is_nat64` for `64:ff9b::/96` (lines 64-67) — reconstructing the last 32 bits as an `Ipv4Addr` via `(u32::from(s[6]) << 16) | u32::from(s[7])` with no `as` casts, then judging it through `is_public_v4`; the remaining rows `::/96` (line 68), `fc00::/7` (71), `fe80::/10` (74), `ff00::/8` (77) and `2001:db8::/32` (80) follow. `cargo fmt --all` first. Added `pub mod addr;` to `lib.rs` in alphabetical position (before `fetch`). All 4 addr tests pass; `make gate` prints `gate: ok` on the first run. | ? |
| M3b/06-toolkit-fetch | 2026-09-23 | done | 3 | fail | none | Wrote `crates/toolkit/src/fetch.rs`: `CURL=/bin/curl`, `PROXY=socks5h://localhost/run/egress/egress.sock`, `CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt`, `MAX_STDERR=65536`. `curl_args(url)` returns the 21 fixed strings in order ending in `--url <url>`. `fetch` calls `fetch_with(Path::new(CURL), args)`. `fetch_with`: spawn curl with `curl_args(&args.url)`, null stdin, stdout/stderr piped (spawn failure -> `"http_fetch: cannot start {}: {e}"`); read stderr on its own thread via the `read_capped` helper (8 KiB buffer, keeps the first 64 KiB and drains/discards the rest so curl never blocks on a full pipe), main thread reads stdout to the end; `wait()` then join the thread (a panicked reader falls back to empty stderr); exit 0 -> `done(from_utf8_lossy(body))` with the `--write-out` status line already in it, else the first non-blank trimmed stderr line as `why` or `"curl exited {code}"` or `"curl was killed"` when there is no code, wrapped as `"http_fetch: {url}: {why}"`, and a wait failure -> `"http_fetch: cannot wait for curl: {e}"`. `lib.rs` gained `pub mod fetch` (between `files` and `input`) and an `"http_fetch"` arm parsing `HttpFetchArgs` like the other three tools. First gate run failed on a rustfmt line-wrap of the spawn-error return; second on clippy `manual_unwrap_or_default` for the stderr-join match, switched to `unwrap_or_default()`. 7 fetch tests pass ten runs; `make gate` prints `gate: ok`. | ? |
| M3b/05-toolkit-shell | 2026-09-23 | done | 2 | fail | none | Wrote `crates/toolkit/src/shell.rs` (`SHELL=/bin/sh`, `DEFAULT_CWD=/tmp`, `MAX_OUTPUT=1048576`). `shell()`: `cwd` defaults to `/tmp`, `Path::is_dir` check returns `"shell: {cwd}: no such directory"`; one `io::pipe` with a `try_clone`'d second write end (stdout gets `writer`, stderr gets `writer2`); `Command::new(SHELL).arg("-c").arg(command).current_dir(cwd)` with null stdin, `spawn`, then `drop(command)` so the parent holds no write end and reading reaches EOF (the hang fix); an 8 KiB-buffer read loop keeping the first `MAX_OUTPUT` bytes and draining+discarding the rest with a `dropped` flag, retrying `Interrupted`, decoding with `from_utf8_lossy`; `wait()` then appending the dropped line and the exit/signal/none status in that order via a `match` on `code()`/`signal()``Outcome::done`. `lib.rs` gained `pub mod shell` (after `input`) and a `"shell"` arm parsing `ShellArgs` like the other two tools. One logic bug caught by the limit test: when the buffer fills `MAX_OUTPUT` exactly the drain branch never set `dropped`, fixed by marking `dropped` in the drain branch. First gate failed on an unused `Write` import (removed); 8 shell tests pass five runs, `make gate` prints `gate: ok`. | ? |
| M3b/04-toolkit-files | 2026-09-22 | done | 1 | pass | none | Wrote `crates/toolkit/src/input.rs` (`MAX_INPUT`, `InputError` with hand-written `Display` + `std::error::Error` + `From<io::Error>`, `read_input` reading at most `MAX_INPUT+1` bytes via `Read::take` and flagging `TooLarge`, `parse` returning the whole stderr line) and `crates/toolkit/src/files.rs` (`MAX_READ`, `read_file` and `write_file`). `read_file` walks the exits in order: metadata NotFound→"no such file", other metadata error→Display text, directory→"is a directory", open failure→text, `take(MAX_READ+1)` read failure→text, over-limit→"larger than {MAX_READ} bytes", non-UTF-8→"not UTF-8 text", else `done`. `write_file` checks the parent is an existing directory before writing so no directory is ever created, then the path-is-directory case, then `fs::write`. Replaced the `lib.rs`/`main.rs` stubs: `Exit` via a `match` (no `as`), `Outcome` with `done`/`tool_error`/`misuse` (misuse stdout empty), `run` reading input before the name lookup for every name. `Cargo.toml` gained `serde` and `serde_json` workspace deps; `dependencies.md` lists `toolkit` under both. Copied the two given test files. `an_unreadable_file_names_the_error` ran for real (UID 1000, not root). 7 `files.rs` tests pass; `make gate` prints `gate: ok` on the first run. | ? |
| M3b/02-brokerd-fetch-url | 2026-09-22 | done | 1 | pass | none | Sealed the fetch target: replaced the two-field `ToolArgs::HttpFetch { url, host }` variant with a tuple variant `HttpFetch(FetchUrl)` holding a new `FetchUrl { url, host }` struct whose fields are private and exposed only through `url()`/`host()`; `parse` is the only constructor. Added the two doctests word for word (a `compile_fail` proving the struct cannot be built outside `args`, and a passing one reading the parsed value through getters). Changed the four match sites that built or read the old variant: `tool` (`HttpFetch(_)`), `canonical_json` (`HttpFetch(target)` serialising `HttpFetchArgs { url: target.url.clone() }`, still only the URL), `parse` (`HttpFetch(FetchUrl { url: value.url, host })`), and `policy::covers` (`HttpFetch(target)` matching with `host_matches(pattern, target.host())`); `runner.rs` `HttpFetch { .. }` became `HttpFetch(_)`. Copied the new `tests/args.rs` (one test now reads the parsed value through getters). `grep "FetchUrl {"` shows the struct def, impl opener, the `parse` line and the doctest only; `grep "HttpFetch {"` shows nothing. All brokerd tests and doctests pass; `make gate` prints `gate: ok` on the first run. | ? |
| M3b/01-proto-tools-hosts | 2026-09-22 | done | 1 | pass | none | Copied `tools.rs` and `hosts.rs` from the plan's `files/` into `crates/proto/tests/`; they fail to compile until the modules exist. Created `crates/proto/src/tools.rs` with the four public arg structs (ReadFile/Write/Shell/HttpFetch), each `#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]` and `#[serde(deny_unknown_fields)]`, `ShellArgs` keeping its `#[serde(default, skip_serializing_if = "Option::is_none")]` cwd; and `crates/proto/src/hosts.rs` with private `valid_label` plus `valid_host`, `valid_host_pattern`, `host_matches`, bodies and doc comments unchanged. Added `pub mod hosts;` and `pub mod tools;` to `proto/src/lib.rs` in alphabetical order (no re-exports). Deleted the private structs from `brokerd/src/args.rs`, added `use proto::tools::{...}` at the top and the `pub use proto::hosts::{...}` re-export where the functions were, and dropped the now-unused `use serde::{Deserialize, Serialize}`; `url_host`, `valid_path`, `inside` and `MAX_URL` stay. New suites 3 and 3 pass; brokerd args/grants/policy pass unchanged. `grep "fn valid_host\|struct ShellArgs"` shows one extra line, `brokerd/tests/args.rs:91`, the pre-existing test `valid_hosts_and_patterns` (substring match, not a duplicate definition). Gate passed on the first run. | ? |
| M3a/23-review-fixes | 2026-09-22 | done | 1 | pass | none | Done by the design model, not an implementer run: Ornith was under heavy contention. Findings 1 to 13 and 16 fixed. Tests that failed without their fix cover 1, 2, 3, 4, 7, 8, 9, 10 and 16; two more (a torn first record, an empty later file) passed before the fix and are regression guards; 6, 11 and 13 were checked by reading. 14 moves to M3b's first task; of 15, the log-name rule, `MAX_PATH`'s doc and the unused push are fixed, and the two that change the baseline wait for an epoch change. The server expectations in the device tests were updated to the new layout. | Claude Opus 5.5 | | M3a/23-review-fixes | 2026-09-22 | done | 1 | pass | none | Done by the design model, not an implementer run: Ornith was under heavy contention. Findings 1 to 13 and 16 fixed. Tests that failed without their fix cover 1, 2, 3, 4, 7, 8, 9, 10 and 16; two more (a torn first record, an empty later file) passed before the fix and are regression guards; 6, 11 and 13 were checked by reading. 14 moves to M3b's first task; of 15, the log-name rule, `MAX_PATH`'s doc and the unused push are fixed, and the two that change the baseline wait for an epoch change. The server expectations in the device tests were updated to the new layout. | Claude Opus 5.5 |
| M3a/22-end-to-end | 2026-09-22 | done | 1 | pass | none | Copied `end_to_end.rs`, `Makefile` (workspace build then ignored e2e with `BOXMAKER_BROKERD`) and `tools/check-m3a-device.sh`. Without the variable: 0 passed, 1 ignored. With it: 1 passed five runs. `--ignored` without the variable fails with "set BOXMAKER_BROKERD". `make gate` ran the e2e line (`1 passed`) and prints `gate: ok`. Did not run the straylight script. | Grok 4.6 | | M3a/22-end-to-end | 2026-09-22 | done | 1 | pass | none | Copied `end_to_end.rs`, `Makefile` (workspace build then ignored e2e with `BOXMAKER_BROKERD`) and `tools/check-m3a-device.sh`. Without the variable: 0 passed, 1 ignored. With it: 1 passed five runs. `--ignored` without the variable fails with "set BOXMAKER_BROKERD". `make gate` ran the e2e line (`1 passed`) and prints `gate: ok`. Did not run the straylight script. | Grok 4.6 |
| M3a/21-runbook-check | 2026-09-22 | done | 1 | pass | none | Wrote `scripts/check-runbook.sh`: find `*.rs` under crates (prune `target/`), awk out every `docs/runbook.md#` pointer, empty anchors fail, each remaining anchor must match a whole `## <anchor>` line, every missing one is reported with its files, then one exit. Step 5: dropping `-x` from grep failed with "the entry is the whole line, at level two"; `exit 1` at a missing-anchor report failed with "both missing entries and their files are reported". Real tree exits 0. `make gate` prints `gate: ok`. | Grok 4.6 | | M3a/21-runbook-check | 2026-09-22 | done | 1 | pass | none | Wrote `scripts/check-runbook.sh`: find `*.rs` under crates (prune `target/`), awk out every `docs/runbook.md#` pointer, empty anchors fail, each remaining anchor must match a whole `## <anchor>` line, every missing one is reported with its files, then one exit. Step 5: dropping `-x` from grep failed with "the entry is the whole line, at level two"; `exit 1` at a missing-anchor report failed with "both missing entries and their files are reported". Real tree exits 0. `make gate` prints `gate: ok`. | Grok 4.6 |
@@ -417,4 +433,74 @@ each task's commit. Three rows were malformed by the first run's orchestrator: t
six cells, and its notes had been pasted into task 05's row; task 10's notes into task 11's; and six cells, and its notes had been pasted into task 05's row; task 10's notes into task 11's; and
task 01's stopped row carried the notes of M2b task 11. Each is moved back or removed, with a task 01's stopped row carried the notes of M2b task 11. Each is moved back or removed, with a
bracketed mark, and pipes inside code are escaped so every row has its eight cells. bracketed mark, and pipes inside code are escaped so every row has its eight cells.
| M3b/03-brokerd-grant-mount-rule | 2026-09-22 | done | 1 | pass | none | Copied `tests/grants_mount.rs` from the plan's `files/`. Added a third arm to the path loop in `check_grant` (grants.rs:263), an `else if path.contains([':', ','])` checked only when the first two arms did not apply, reporting `"{:?} cannot be mounted: it contains ':' or ','"`. The `else if` chain means a path already reported as invalid is not reported twice. `cargo fmt --all` kept the `push` multi-line (the single-line form in the task exceeds 100 columns); the wording matches the task verbatim. Both suites pass (17 grants, 2 mount); `make gate` prints `gate: ok` on the first run. | ? |
### M3b, tasks 01 to 13 — reviewed 2026-09-23 by the design model (Claude)
Accepted, with follow-ups. All work by Ornith-1.5-35B-A3B through `tools/run-plan.sh`. The code
does what the spec says, and on straylight, with real containers from the Nix-built image, every
claim of the milestone held.
| Check | Result |
|---|---|
| 13 task commits, each with the trailer; 6 plan commits by the design model during the run | pass |
| All 28 given test and fixture files identical to the plan | pass (`container.rs` differs from its skeleton, as intended) |
| `make gate` on Talos | `gate: ok`, 638 tests, the same count as the reference |
| `toolkit` and `brokerd` suites ten times in a row | no failure |
| Banned constructs in new library code | `thread::spawn` four times, `as u64` on two constants (findings 3 and 7) |
| Independent review by a separate agent, given only the code, the spec and the plan | no serious defect; its points are below |
**On straylight** (2026-09-23; image `localhost/boxmaker-tools@sha256:04459bec…`, 16 MB, built by
`deploy/tools-image.nix`; `brokerd` from this branch with `[runner]`):
| Claim | Seen |
|---|---|
| A granted file is read; a symlink in the granted directory to `~/.ssh/id_ed25519` is not | `hello from the notes`; `read_file: …/notes/key: no such file` |
| `write_file` writes as the owner | the file is owned by uid 1000 |
| No network without a grant | from `shell`: `100.100.100.100` unreachable, `1.1.1.1` unreachable, no DNS, only `lo` |
| Hardening | no capabilities, read-only root, writable `/tmp` |
| Limits | 8 s limit stopped `sleep 60` at 8 s; 64 processes stopped a fork loop; 256m killed a memory hog |
| `http_fetch` reaches only allowed hosts | `example.com` 200; redirect `google.com``www.google.com` refused at the proxy (reply 2); an allowed name resolving to the tailnet (`100.88.197.9`) refused (reply 4); a host with no grant denied before any container |
| No container outlives its call | `podman ps -a --filter label=boxmaker` empty after every call; egress directories removed |
| # | Severity | Owner | Finding | Fix |
|---|---|---|---|---|
| 1 | medium | plan (task 06, spec 6) | `curl` expands globs in the URL (documented behaviour: `https://example.com/[1-3]` is three requests; the review first wrote "seen on straylight", but only the last response was seen, not counted), so `[1-99999999]` would hammer an allowed host and buffer every body. After the fix, the same URL is one request, counted on straylight. The fixed argument list lacks `--globoff`. | Fixed by the design model (`--globoff`, and `--disable` first) |
| 2 | low | plan (task 10, spec 6) | `podman run` has no `--pull=never`: with an image that is not loaded, Podman tries to pull it (seen on straylight). Here the name starts `localhost/`, so the pull fails, but a pull is unlisted egress and the call should fail at once. | Fixed by the design model (`--pull=never`); a missing image now fails in 46 ms, no pull |
| 3 | low | implementer (11), plan | `std::thread::spawn` in `container.rs` (three) and `toolkit/src/fetch.rs` panics if a thread cannot be made; after the spawn of the container, a panic drops the `Child` without `podman kill`, so the container runs on without its limit. The reference had the same; the task did not say. | Task 15 (brokerd), task 17 (toolkit) |
| 4 | low | implementer (11) | The time limit bounds the wait, not the joins after it: if another process held the pipes, `run` would block until it let go (shown with a fake `podman` without `exec`: 6 s for a 0.3 s limit). Real Podman released them at the kill (8 s limit, 8 s seen). | Task 15 |
| 5 | low | implementer (11) | Podman's standard error, which the tool can write to, goes into `brokerd`'s log unescaped, so a tool can forge log lines (a fake runbook pointer). It never reaches a `RunError`. | Task 16 |
| 6 | low | spec (section 5) | `is_public` passes local-use NAT64 `64:ff9b:1::/48` and 6to4 `2002::/16` with a private IPv4 inside. Neither is in use on straylight. A host with its own public address would be reachable by an allowed name that points at it; straylight has none (its addresses are LAN, tailnet and Tailscale's ULA, all refused). | Spec, when next touched |
| 7 | nit | implementer | The runtime notice lacks its `brokerd:` prefix; two config messages use `{}` where the task gave `{:?}`; `chunk[..take]` where the skeleton said `get`; `egress-proxy` accepts trailing arguments; `as u64` on two constants. | Tasks 16 and 17 (all but the indexing, gone with task 15) |
What was good: the proxy handshake reads exactly what the protocol gives against one deadline, has
no panic path, and tries only public addresses (the independent review probed it and found it
sound); the egress guard is created before anything can fail and cleans up on every path, a panic
included; nothing from the model reaches `podman`'s command line.
**The run.** 11 of 13 tasks committed on the first attempt. Task 04 committed but left `Cargo.lock`
out (the plan's `git add` line). Task 08 needed two attempts (the first did not skip `argv[0]` and
joined each handler thread). Task 11 needed four: the task could not be written as given (a field
nothing read, and `#[allow]` forbidden), then two sessions ran out of room planning the whole file
in one turn; a skeleton, and then a finer one with `run` as glue over small helpers, got it done.
Every stop was a task-writing problem or a turn-size problem, not a wrong implementation.
### M3b, tasks 14 to 17 — reviewed 2026-09-23 by the design model (Claude)
Accepted; M3b is done. All four by Ornith, each committed on its first attempt: task 14 a pure
move, tasks 15 and 16 with the glue given exactly, task 17 small. The given tests are unchanged;
`make gate` prints `gate: ok` with 649 tests, the reference's count; `thread::spawn` and `as u64`
are gone from `brokerd` and `toolkit`; `container.rs` is 374 lines and `pipes.rs` 128.
On straylight with the final code (image `sha256:08dfabf0…`), every row of the first review's
table held again, and the glob URL made one request. One behaviour to know: a shell command that
leaves a background process (`sleep 30 &`) runs to the time limit and is then killed and removed,
because `toolkit shell` reads its output to the end (task 05) and the container has not ended; the
grace period of task 15 covers only a container that has.
Open, low: finding 6 (`is_public` does not refuse local-use NAT64 `64:ff9b:1::/48` or 6to4
`2002::/16` with a private IPv4 inside; neither is in use on straylight, and it has no public
address of its own), and M3a findings 17, 18 and 20. Owner foot-guns noted by the independent review
and not fixed: a grant path that overlaps a container path (`/bin`, `/tmp`, `/run/egress`), and a
`home` containing `:`.
+2 -1
View File
@@ -11,7 +11,8 @@ One session per milestone. The design brief is `docs/design.md`.
| M2 | `loopd` core + `bxctl chat` | Append-only loop, baseline ≤ 3k tokens, self-test, runaway control, with one fake in-process tool | Prefix-extension property test; turn-2 cache hit on straylight | | M2 | `loopd` core + `bxctl chat` | Append-only loop, baseline ≤ 3k tokens, self-test, runaway control, with one fake in-process tool | Prefix-extension property test; turn-2 cache hit on straylight |
| M3a | `brokerd` decisions | Grants, deny by default, taint, hash-chained audit, `ask` approvals through `bxctl`, two sockets, `loopd`'s broker port; tools do not run yet | Tests for every deny path; audit tampering suite; a denial reaches Ornith on straylight as a tool result | | M3a | `brokerd` decisions | Grants, deny by default, taint, hash-chained audit, `ask` approvals through `bxctl`, two sockets, `loopd`'s broker port; tools do not run yet | Tests for every deny path; audit tampering suite; a denial reaches Ornith on straylight as a tool result |
| M3b | `brokerd` runner | Rootless container runner, four tools (read file, write file, sandboxed shell, HTTP fetch with host allowlist), egress proxy | A container with no network grant cannot reach the network; fetch refused for hosts outside the grant | | M3b | `brokerd` runner | Rootless container runner, four tools (read file, write file, sandboxed shell, HTTP fetch with host allowlist), egress proxy | A container with no network grant cannot reach the network; fetch refused for hosts outside the grant |
| M4 | `gatewayd` + Mattermost | Bot connection, user-ID allowlist, threads as sessions, async delivery, approvals by reply or reaction, no listening port | End-to-end direct message on the real server; a non-allowlisted user gets no response | | M4a | `gatewayd` + Mattermost conversations | Bot connection over TLS, user-ID allowlist, threads as sessions, bursts as one turn, typing while working, catching up after downtime, the secret store, no listening port | A direct message answered on the real server; a non-allowlisted user gets no response |
| M4b | Approvals over Mattermost | `gateway.sock`, approval posts, answers by reaction or `!approve`/`!deny`, long approvals in parts | An `ask` call approved and refused from Mattermost; the audit log names the post |
| M5 | Scheduler + memory | Heartbeat and cron on their own slot in isolated sessions; core block, `recall` and `remember` with provenance, FTS5 index, flush-then-compact at idle | Scheduled run leaves the main slot's cache intact; a fact from epoch 1 is recalled in epoch 2; a note written after reading untrusted content is marked on recall | | M5 | Scheduler + memory | Heartbeat and cron on their own slot in isolated sessions; core block, `recall` and `remember` with provenance, FTS5 index, flush-then-compact at idle | Scheduled run leaves the main slot's cache intact; a fact from epoch 1 is recalled in epoch 2; a note written after reading untrusted content is marked on recall |
| M5b | Embedding index (when justified) | Local embedding server, hybrid retrieval behind the `Retriever` trait, rebuild on model change | A paraphrased query finds a note that lexical search misses; no memory text leaves the host | | M5b | Embedding index (when justified) | Local embedding server, hybrid retrieval behind the `Retriever` trait, rebuild on model change | A paraphrased query finds a note that lexical search misses; no memory text leaves the host |
| M6 | Subagents + cloud consult | Subagents on their slot returning summaries; consult tool gated by data-class grants with full payload log | A tainted session is refused; payload log matches what was sent | | M6 | Subagents + cloud consult | Subagents on their slot returning summaries; consult tool gated by data-class grants with full payload log | A tainted session is refused; payload log matches what was sent |
+93
View File
@@ -0,0 +1,93 @@
# M3b task 01: the tools' arguments and the host rules move to `proto`
**Branch:** `m3b` (create it: `git switch master && git switch -c m3b`; `git status --short` must be
empty first, otherwise stop)
**Commit subject:** `Move the tools' arguments and the host rules to proto`
## Goal
Two programs will now read the same things: `brokerd` parses the model's tool arguments and checks
hosts, and `toolkit` (inside the container) reads the same arguments and checks the same hosts.
Each must have **one** definition. Move them into `proto`; `brokerd` uses them from there. Nothing
changes in behaviour: every existing test passes unchanged.
## Files
- Copy: `crates/proto/tests/tools.rs`, `crates/proto/tests/hosts.rs`
- Create: `crates/proto/src/tools.rs`, `crates/proto/src/hosts.rs`
- Modify: `crates/proto/src/lib.rs`, `crates/brokerd/src/args.rs`, `docs/implementer-log.md`
## Interfaces
`crates/proto/src/tools.rs` (module doc: the four tools' arguments, shared by `brokerd` and
`toolkit`):
```rust
use serde::{Deserialize, Serialize};
// Each: #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)]
pub struct ReadFileArgs { pub path: String }
pub struct WriteFileArgs { pub path: String, pub content: String }
pub struct ShellArgs {
pub command: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub cwd: Option<String>,
}
pub struct HttpFetchArgs { pub url: String }
```
These are the four private structs at the bottom of `brokerd/src/args.rs` today (`ReadFileArgs`,
`WriteFileArgs`, `ShellArgs`, `HttpFetchArgs`), made public, with `Debug, Clone, PartialEq, Eq`
added. **Move** them: delete them from `args.rs` and `use proto::tools::{…}` there instead.
`crates/proto/src/hosts.rs` (module doc: host names and patterns, shared by `brokerd` and the
egress proxy):
```rust
pub fn valid_host(host: &str) -> bool;
pub fn valid_host_pattern(pattern: &str) -> bool;
pub fn host_matches(pattern: &str, host: &str) -> bool;
fn valid_label(label: &str) -> bool; // private, as it is now
```
**Move** these four functions from `brokerd/src/args.rs`, bodies and doc comments unchanged. In
`args.rs`, where they were, put:
```rust
/// Moved to `proto::hosts` in M3b, so `toolkit` checks hosts with the same rules.
pub use proto::hosts::{host_matches, valid_host, valid_host_pattern};
```
so that `brokerd::args::valid_host` and the others still exist for `brokerd`'s code and tests.
`url_host`, `valid_path`, `inside` and `MAX_URL` stay in `args.rs`.
In `crates/proto/src/lib.rs`: add `pub mod hosts;` and `pub mod tools;` in alphabetical order with
the other `pub mod` lines. No `pub use` for them: callers write `proto::tools::ShellArgs` and
`proto::hosts::host_matches`.
`args.rs` no longer needs `use serde::{Deserialize, Serialize};` once the structs are gone; remove
it if the compiler says it is unused.
## Steps
- [ ] **1. Branch and copy.** `git switch master && git switch -c m3b`, then
`cp docs/plans/M3b/files/crates/proto/tests/tools.rs docs/plans/M3b/files/crates/proto/tests/hosts.rs crates/proto/tests/`
- [ ] **2. See them fail.** `cargo test -p proto --test tools --test hosts`. Expected: they do not
compile (`proto::tools` and `proto::hosts` do not exist).
- [ ] **3. Move the code** as above. Run `cargo fmt --all`.
- [ ] **4. See them pass.** `cargo test -p proto --test tools --test hosts`. Expected: 3 and 3
passed. Then `cargo test -p brokerd --test args --test grants --test policy`: all pass, unchanged.
- [ ] **5. Check nothing is defined twice.** `grep -rn "fn valid_host\|struct ShellArgs" crates/`
must show only `crates/proto/src/`.
- [ ] **6. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **7. Log and commit.** Add your row to `docs/implementer-log.md`, then
`git add crates/proto crates/brokerd docs/implementer-log.md Cargo.lock && git commit`
## Done when
- Both new suites pass, every existing suite passes, `make gate` prints `gate: ok`.
## Stop and report if
- A given test needs a change to pass.
- An existing test fails after the move.
+104
View File
@@ -0,0 +1,104 @@
# M3b task 02: seal the fetch target
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `Seal the fetch target: one value holds the URL and its host`
## Goal
Today `ToolArgs::HttpFetch { url, host }` has two public fields. Policy matches `host` against the
grant; from M3b on, the runtime fetches `url`. Any code could build one where the two disagree
(`url` of `evil.example`, `host` of `example.com`), and policy would allow a fetch of the wrong
host. Make that impossible: one sealed value, built only by `args::parse` after the URL checks.
This is M3a review finding 14.
## Files
- Copy: `crates/brokerd/tests/args.rs` (replaces the old one; one test now reads the parsed value
through getters, because it can no longer build one)
- Modify: `crates/brokerd/src/args.rs`, `crates/brokerd/src/policy.rs`,
`crates/brokerd/src/runner.rs`, `docs/implementer-log.md`
## Interfaces
In `args.rs`, the variant becomes a tuple variant holding a new struct:
```rust
pub enum ToolArgs {
ReadFile { path: String },
WriteFile { path: String, content: String },
Shell { command: String, cwd: Option<String> },
HttpFetch(FetchUrl),
}
/// A URL that passed the checks, and its host. Only `parse` makes one, so the host policy matched
/// is always the host of the URL the tool fetches.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct FetchUrl {
url: String, // private
host: String, // private
}
impl FetchUrl {
pub fn url(&self) -> &str;
pub fn host(&self) -> &str;
}
```
No `pub fn new`, no `Default`, no `Deserialize`, no public field: `parse` (in the same module) is
the only code that writes `FetchUrl { url, host }`.
Put these two doctests in `FetchUrl`'s doc comment, word for word. The first proves the struct
cannot be built outside `args`; the second proves the first fails for that reason and not a typo:
````rust
/// ```compile_fail
/// let _ = brokerd::args::FetchUrl {
/// url: "https://evil.example/".to_string(),
/// host: "example.com".to_string(),
/// };
/// ```
///
/// ```
/// let args = brokerd::args::parse(brokerd::args::ToolName::HttpFetch, r#"{"url":"https://example.com/a"}"#);
/// let Ok(brokerd::args::ToolArgs::HttpFetch(target)) = args else { panic!("{args:?}") };
/// assert_eq!((target.url(), target.host()), ("https://example.com/a", "example.com"));
/// ```
````
Every place that matched the old variant changes. There are four:
1. `ToolArgs::tool`: `ToolArgs::HttpFetch(_) => ToolName::HttpFetch`.
2. `ToolArgs::canonical_json`: `ToolArgs::HttpFetch(target) =>` serialise
`HttpFetchArgs { url: target.url.clone() }` (still only the URL: the host is not an argument).
3. `parse`'s `ToolName::HttpFetch` arm: `Ok(ToolArgs::HttpFetch(FetchUrl { url: value.url, host }))`.
4. `policy.rs`, in `covers`: `ToolArgs::HttpFetch(target) =>` and match the grant's patterns with
`host_matches(pattern, target.host())`.
In `runner.rs`: `ToolArgs::HttpFetch { .. }` becomes `ToolArgs::HttpFetch(_)`.
Search for any other: `grep -rn "HttpFetch {" crates/` must show nothing when you are done.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/brokerd/tests/args.rs crates/brokerd/tests/`
- [ ] **2. See it fail.** `cargo test -p brokerd --test args`. Expected: it does not compile
(`ToolArgs::HttpFetch(target)` is not a tuple variant yet).
- [ ] **3. Make the change.** Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p brokerd --test args --test policy --test runner`.
Expected: `args` 13 passed; the others pass unchanged. `cargo test -p brokerd --doc`: every
doctest passes, including your two new ones (`compile_fail` counts as passing when it fails to
compile).
- [ ] **5. Check the seal.** `grep -rn "FetchUrl {" crates/brokerd/src/` shows the struct
definition, the `parse` line, and the doctest only.
- [ ] **6. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **7. Log and commit.** `git add crates/brokerd docs/implementer-log.md Cargo.lock && git commit`
## Done when
- `cargo test -p brokerd` and `cargo test -p brokerd --doc` pass; `make gate` prints `gate: ok`.
## Stop and report if
- Some code outside `args.rs` needs to build a `FetchUrl`: that would break the seal.
- A test other than `args.rs` needs a change.
@@ -0,0 +1,53 @@
# M3b task 03: a grant path that cannot be mounted is invalid
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `Refuse grant paths that cannot be mounted`
## Goal
From M3b, a grant's paths are mounted into the tool container as `--volume=<path>:<path>:ro`. In
that argument `:` and `,` are separators, so a path containing either would be read as something
else. A grant with such a path is **invalid**, which (as for every invalid grant) makes the whole
set invalid and denies every call until it is fixed.
## Files
- Copy: `crates/brokerd/tests/grants_mount.rs`
- Modify: `crates/brokerd/src/grants.rs`, `docs/implementer-log.md`
## The rule
In `check_grant`, the loop over `inner.constraints.paths` has two cases today: the path is `/`, or
it is not a valid path. Add a third, checked **only when the first two did not apply** (a path that
is already reported as invalid is not reported twice):
```rust
} else if path.contains([':', ',']) {
// Mounted as `--volume=<path>:<path>:ro`, where both are separators.
push(problems, file.clone(), None,
format!("{:?} cannot be mounted: it contains ':' or ','", path));
}
```
So a grant file with `paths = ["/home/kyle/a:b"]` gives exactly one problem, for that file, whose
text contains `cannot be mounted`. Other punctuation (space, `;`, `=`, `.`, `-`, `_`) stays fine.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/brokerd/tests/grants_mount.rs crates/brokerd/tests/`
- [ ] **2. See it fail.** `cargo test -p brokerd --test grants_mount`. Expected: 1 of 2 fails
(`a_path_with_a_colon_or_a_comma_makes_the_set_invalid`).
- [ ] **3. Add the rule.** Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p brokerd --test grants_mount --test grants`. Expected: 2 and
17 passed.
- [ ] **5. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/brokerd docs/implementer-log.md Cargo.lock && git commit`
## Done when
- Both suites pass; `make gate` prints `gate: ok`.
## Stop and report if
- A given test wants a path with `:` or `,` to be accepted, or wants two problems for one path.
+151
View File
@@ -0,0 +1,151 @@
# M3b task 04: the `toolkit` program, `read_file` and `write_file`
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `toolkit: the tool program, read_file and write_file`
## Goal
`toolkit` is the one binary inside a tool container. `brokerd` will run `toolkit <tool>` with the
tool's arguments as JSON on standard input (the `proto::tools` structs from task 01). `toolkit`
prints what the model should see on standard output and exits 0 (done), 1 (the tool could not do
it, for a reason the model should read) or 2 (it was run wrongly). It decides nothing: whether the
call may run was decided before the container existed. Spec section 4.
This task makes the program and its first two tools. `shell` and `http_fetch` come in tasks 05 and
06; until then they are unknown tools (exit 2).
## Files
- Copy: `crates/toolkit/tests/support/mod.rs`, `crates/toolkit/tests/files.rs`
- Create: `crates/toolkit/src/input.rs`, `crates/toolkit/src/files.rs`
- Replace: `crates/toolkit/src/lib.rs`, `crates/toolkit/src/main.rs` (both are stubs today)
- Modify: `crates/toolkit/Cargo.toml`, `docs/dependencies.md`, `docs/implementer-log.md`
`Cargo.toml`: under `[dependencies]`, after `proto.workspace = true`, add
`serde.workspace = true` and `serde_json.workspace = true`, each on its own line.
`docs/dependencies.md`: in the `serde` and `serde_json` rows, the "Used by" column becomes
`` `proto`, `brokerd`, `toolkit` ``.
## Interfaces
`lib.rs` (module doc: the programs that run inside tool containers):
```rust
pub mod files;
pub mod input;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Exit { Done, ToolError, Misuse }
impl Exit {
/// 0, 1, 2. No `as` cast: a `match`.
pub fn code(self) -> u8;
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Outcome { pub exit: Exit, pub stdout: String, pub stderr: String }
impl Outcome {
pub fn done(stdout: String) -> Outcome; // Done, stderr empty
pub fn tool_error(stdout: String) -> Outcome; // ToolError, stderr empty
pub fn misuse(stderr: String) -> Outcome; // Misuse, stdout EMPTY: the model sees nothing
}
/// Run tool `name` with the arguments on `stdin`.
pub fn run(name: &str, stdin: &mut dyn std::io::Read) -> Outcome;
```
`run`:
1. `input::read_input(stdin)`; an error `e` → `Outcome::misuse(format!("toolkit: {e}"))`.
2. By `name`: `"read_file"` → `input::parse::<ReadFileArgs>(name, &text)` then
`files::read_file(&args)`; `"write_file"` → the same with `WriteFileArgs` and
`files::write_file`. A parse error `e` (already a whole line) → `Outcome::misuse(e)`.
3. Any other name → `Outcome::misuse(format!("toolkit: unknown tool {name:?}"))`.
The input is read **before** the name is looked at, for every name.
`input.rs`:
```rust
pub const MAX_INPUT: usize = 2 * 1024 * 1024;
#[derive(Debug)]
pub enum InputError { TooLarge, NotUtf8, Io(std::io::Error) }
// Display, by hand:
// TooLarge → "the arguments are larger than 2097152 bytes" (write it with {MAX_INPUT})
// NotUtf8 → "the arguments are not UTF-8"
// Io(e) → "cannot read the arguments: {e}"
/// All of `stdin`, at most MAX_INPUT bytes, as UTF-8. Read at most MAX_INPUT + 1 bytes
/// (`Read::take`); more than MAX_INPUT is TooLarge.
pub fn read_input(stdin: &mut dyn std::io::Read) -> Result<String, InputError>;
/// Parse `text` as the arguments of tool `name`; the error is the whole line for stderr:
/// "toolkit: {name}: the arguments do not parse: {serde's error}".
pub fn parse<T: serde::de::DeserializeOwned>(name: &str, text: &str) -> Result<T, String>;
```
`files.rs`:
```rust
pub const MAX_READ: usize = 1024 * 1024;
pub fn read_file(args: &ReadFileArgs) -> Outcome;
pub fn write_file(args: &WriteFileArgs) -> Outcome;
```
## `read_file`: every exit
Every failure is `Outcome::tool_error(format!("read_file: {path}: {why}"))`, one line, no newline
at the end.
1. `std::fs::metadata(path)` fails with `ErrorKind::NotFound` → why = `no such file`.
2. It fails any other way → why = the error's `Display` text.
3. It is a directory → why = `is a directory`.
4. Opening fails → why = the error's text.
5. Read at most `MAX_READ + 1` bytes (`take`). Reading fails → the error's text. More than
`MAX_READ` bytes → why = `larger than 1048576 bytes` (write it with `{MAX_READ}`). Exactly
`MAX_READ` is fine.
6. Not UTF-8 → why = `not UTF-8 text`.
7. Otherwise `Outcome::done(text)`: the content exactly, nothing added.
## `write_file`: every exit
Every failure is `Outcome::tool_error(format!("write_file: {path}: {why}"))`.
1. The path's parent is not an existing directory (`Path::parent` is `None`, or not `is_dir()`) →
why = `the directory does not exist`. **No directory is ever created.**
2. The path is a directory → why = `is a directory`.
3. `std::fs::write(path, content)` fails → why = the error's text.
4. Otherwise `Outcome::done(format!("wrote {} bytes to {path}", content.len()))` (bytes, not
characters).
## `main.rs`
Read the arguments with `std::env::args_os()` (never `args()`, which panics on non-UTF-8). Skip the
program name. If there is exactly one argument, it is the tool name (`to_str()`, or `""` if it is
not UTF-8); otherwise the name is `""`, which `run` answers as an unknown tool. Call
`toolkit::run(name, &mut std::io::stdin().lock())`, write `stdout` to standard output and `stderr`
to standard error with `write_all` (ignore their errors), and return
`ExitCode::from(outcome.exit.code())`.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then `mkdir -p crates/toolkit/tests/support` and
`cp docs/plans/M3b/files/crates/toolkit/tests/support/mod.rs crates/toolkit/tests/support/` and
`cp docs/plans/M3b/files/crates/toolkit/tests/files.rs crates/toolkit/tests/`
- [ ] **2. See it fail.** `cargo test -p toolkit --test files`. Expected: it does not compile.
- [ ] **3. Write the code** and the two manifest and doc changes. Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p toolkit --test files`. Expected: 7 passed. One of them
returns early and passes if you are root; you should not be.
- [ ] **5. Walk the exits.** Point at the line of your code for each numbered exit above.
- [ ] **6. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **7. Log and commit.** `git add crates/toolkit docs/dependencies.md docs/implementer-log.md Cargo.lock && git commit`
## Done when
- `cargo test -p toolkit --test files` reports 7 passed; `make gate` prints `gate: ok`.
## Stop and report if
- A test wants a misuse to print anything on standard output.
- A test wants `write_file` to create a directory.
+80
View File
@@ -0,0 +1,80 @@
# M3b task 05: `toolkit shell`
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `toolkit: shell`
## Goal
`toolkit shell` runs one command under `/bin/sh -c`, in the call's `cwd` or in `/tmp`, and prints
its output (standard output and standard error together, in the order written) followed by how it
ended. The command's own exit status is part of the result, not `toolkit`'s: `toolkit` exits 0
whenever the command ran. Spec section 4.
## Files
- Copy: `crates/toolkit/tests/shell.rs`
- Create: `crates/toolkit/src/shell.rs`
- Modify: `crates/toolkit/src/lib.rs`, `docs/implementer-log.md`
## Interfaces
```rust
pub const SHELL: &str = "/bin/sh";
pub const DEFAULT_CWD: &str = "/tmp";
pub const MAX_OUTPUT: usize = 1024 * 1024;
pub fn shell(args: &proto::tools::ShellArgs) -> crate::Outcome;
```
In `lib.rs`: add `pub mod shell;` and, in `run`, a `"shell"` arm before the unknown-tool arm,
parsing `ShellArgs` exactly as the other two tools do and calling `shell::shell(&args)`.
## `shell`: every step and exit
1. `cwd` is `args.cwd`, or `DEFAULT_CWD` when it is `None`. If `Path::new(cwd).is_dir()` is false →
`Outcome::tool_error(format!("shell: {cwd}: no such directory"))`.
2. Make **one** pipe for both output streams: `let (mut reader, writer) = std::io::pipe()?`, and a
second write end with `writer.try_clone()`. (`std::io::pipe` is stable since Rust 1.87.) A
failure here → `tool_error(format!("shell: cannot make a pipe: {e}"))`.
3. `Command::new(SHELL).arg("-c").arg(&args.command).current_dir(cwd)`, standard input
`Stdio::null()`, standard output the first write end, standard error the second. Spawn it. A
failure → `tool_error(format!("shell: cannot start {SHELL}: {e}"))`.
4. **Drop the `Command`** right after spawning (`drop(command)`): it still holds the write ends, and
while any write end is open, reading never reaches the end.
5. Read `reader` to the end in a loop with an 8 KiB buffer. Keep the first `MAX_OUTPUT` bytes;
**keep reading after that and throw the rest away**, remembering that something was dropped. Do
not stop reading: the command would block on a full pipe. Retry `ErrorKind::Interrupted`; stop
on any other error. No indexing that can panic: use `buf.get(..n)` and the like.
6. `child.wait()`. A failure → `tool_error(format!("shell: cannot wait for the command: {e}"))`.
7. The text is the kept bytes decoded with `String::from_utf8_lossy` (invalid bytes become
U+FFFD, never an error). Then append, in this order:
- if something was dropped: `format!("\n[output after {MAX_OUTPUT} bytes dropped]")`;
- `format!("\n[exit {code}]")` if `status.code()` is `Some`, else
`format!("\n[killed by signal {n}]")` if `status.signal()` is `Some`
(`std::os::unix::process::ExitStatusExt`), else `"\n[ended without a status]"`.
8. `Outcome::done(text)`.
A command that leaves a background process holding the pipe (`sleep 100 &`) makes step 5 wait for
it; that is expected, and `brokerd`'s time limit ends it.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/toolkit/tests/shell.rs crates/toolkit/tests/`
- [ ] **2. See it fail.** `cargo test -p toolkit --test shell`. Expected: it does not compile.
- [ ] **3. Write `shell.rs`** and the `lib.rs` changes. Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p toolkit --test shell`. Expected: 8 passed. Run it five
times; it must pass every time.
- [ ] **5. Walk the steps.** Point at the line for steps 4 and 5 in particular.
- [ ] **6. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **7. Log and commit.** `git add crates/toolkit docs/implementer-log.md Cargo.lock && git commit`
## Done when
- `cargo test -p toolkit --test shell` reports 8 passed five times running; `make gate` prints
`gate: ok`.
## Stop and report if
- A test hangs: that is almost always step 4 (a write end still open). Report it if dropping the
`Command` does not fix it.
+91
View File
@@ -0,0 +1,91 @@
# M3b task 06: `toolkit http_fetch`
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `toolkit: http_fetch through curl`
## Goal
`toolkit http_fetch` runs `/bin/curl` with a **fixed** argument list and the URL, through the egress
proxy's socket that `brokerd` mounts at `/run/egress/egress.sock` for this call. It prints the body
and a status line, or one line saying why the fetch failed. We do not write an HTTP or TLS client:
`curl` does that, inside the container. Spec sections 4 and 6.
## Files
- Copy: `crates/toolkit/tests/fetch.rs`
- Create: `crates/toolkit/src/fetch.rs`
- Modify: `crates/toolkit/src/lib.rs`, `docs/implementer-log.md`
## Interfaces
```rust
pub const CURL: &str = "/bin/curl";
pub const PROXY: &str = "socks5h://localhost/run/egress/egress.sock";
pub const CA_BUNDLE: &str = "/etc/ssl/certs/ca-certificates.crt";
/// `curl`'s arguments for `url`, in order, without the program name.
pub fn curl_args(url: &str) -> Vec<String>;
pub fn fetch(args: &proto::tools::HttpFetchArgs) -> crate::Outcome; // fetch_with(Path::new(CURL), args)
/// `fetch` with another `curl`, for tests.
pub fn fetch_with(curl: &std::path::Path, args: &proto::tools::HttpFetchArgs) -> crate::Outcome;
```
In `lib.rs`: `pub mod fetch;`, and an `"http_fetch"` arm in `run` like the others, calling
`fetch::fetch(&args)`.
## `curl_args`
Exactly these 21 strings, in this order (the test compares them one by one):
```
--silent --show-error --proto =https --proto-redir =https --location --max-redirs 5
--max-time 50 --max-filesize 8388608 --cacert <CA_BUNDLE> --proxy <PROXY>
--write-out "\n[http %{response_code}]" --url <url>
```
In Rust, `--write-out`'s value is the literal `"\n[http %{response_code}]"`: a real newline
character, then `[http %{response_code}]` (curl fills in the status). `--url <url>` (not a bare
URL) keeps a URL from ever being read as an option.
## `fetch_with`: every exit
1. Spawn `curl` with `curl_args(&args.url)`, standard input `Stdio::null()`, standard output and
standard error piped. A failure →
`Outcome::tool_error(format!("http_fetch: cannot start {}: {e}", curl.display()))`.
2. Read standard error **on its own thread**, keeping at most 64 KiB, while the main thread reads
standard output to the end. (Reading one and then the other can deadlock when both are full:
the given test writes 300,000 bytes to each.)
3. Wait for `curl`, then join the thread.
4. `curl` succeeded → `Outcome::done(body)`, the body decoded with `from_utf8_lossy`. The status
line is already in it, from `--write-out`.
5. It failed → `Outcome::tool_error(format!("http_fetch: {url}: {why}"))`, where `why` is the first
line of standard error that is not blank, trimmed; if there is none, `curl exited {code}`, or
`curl was killed` when there is no code. The body, if any, is not shown.
6. Waiting failed → `tool_error(format!("http_fetch: cannot wait for curl: {e}"))`.
## About the given tests
`fetch.rs` writes small shell scripts that stand in for `curl` and runs them. Every test that
starts a process takes a lock first (`let _serial = serial();`). Without it, another test's fork
can hold a just-written script open, and running it fails with "Text file busy" (ETXTBSY) about
once in seven runs. If you add a test that writes and runs a script, take the lock too.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/toolkit/tests/fetch.rs crates/toolkit/tests/`
- [ ] **2. See it fail.** `cargo test -p toolkit --test fetch`. Expected: it does not compile.
- [ ] **3. Write `fetch.rs`** and the `lib.rs` changes. Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p toolkit --test fetch`. Expected: 7 passed. Run it ten
times; it must pass every time.
- [ ] **5. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/toolkit docs/implementer-log.md Cargo.lock && git commit`
## Done when
- `cargo test -p toolkit --test fetch` reports 7 passed ten times running; `make gate` prints
`gate: ok`.
## Stop and report if
- A test fails with "Text file busy" even with the lock: report the run and the test.
+82
View File
@@ -0,0 +1,82 @@
# M3b task 07: which addresses are public
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `toolkit: is_public, the addresses the egress proxy may reach`
## Goal
The egress proxy (task 08) connects only to **public** addresses. A grant allows a host on the
internet; if its name resolves into the tailnet (`100.64.0.0/10`), the host (`127.0.0.1`) or a
private network, the connection is refused even though the name is allowed. This task is the one
pure function that decides it. Spec section 5, "Public".
## Files
- Copy: `crates/toolkit/tests/addr.rs`
- Create: `crates/toolkit/src/addr.rs`
- Modify: `crates/toolkit/src/lib.rs` (`pub mod addr;`), `docs/implementer-log.md`
## Interface
```rust
/// True if `ip` is a public unicast address.
pub fn is_public(ip: std::net::IpAddr) -> bool;
```
## Refused ranges: every one of these is **not** public
IPv4 (with `[a, b, c, _] = ip.octets()`):
| Range | Test |
|---|---|
| `0.0.0.0/8` | `a == 0` |
| `10.0.0.0/8` | `a == 10` |
| `100.64.0.0/10` (the tailnet) | `a == 100 && (64..=127).contains(&b)` |
| `127.0.0.0/8` | `a == 127` |
| `169.254.0.0/16` | `a == 169 && b == 254` |
| `172.16.0.0/12` | `a == 172 && (16..=31).contains(&b)` |
| `192.0.0.0/24` | `a == 192 && b == 0 && c == 0` |
| `192.0.2.0/24` | `a == 192 && b == 0 && c == 2` |
| `192.168.0.0/16` | `a == 192 && b == 168` |
| `198.18.0.0/15` | `a == 198 && (b == 18 \|\| b == 19)` |
| `198.51.100.0/24` | `a == 198 && b == 51 && c == 100` |
| `203.0.113.0/24` | `a == 203 && b == 0 && c == 113` |
| `224.0.0.0/4` and `240.0.0.0/4` | `a >= 224` |
IPv6 (with `s = ip.segments()`, eight `u16`s):
| Range | Test |
|---|---|
| `::/96` (holds `::`, `::1`, and the old IPv4-compatible form) | the first six segments are all 0 |
| IPv4-mapped `::ffff:0:0/96` | `s[0..5]` all 0 and `s[5] == 0xffff`: judge the last 32 bits as IPv4 |
| NAT64 `64:ff9b::/96` | `s[0..6] == [0x64, 0xff9b, 0, 0, 0, 0]`: judge the last 32 bits as IPv4 |
| `fc00::/7` | `s[0] & 0xfe00 == 0xfc00` |
| `fe80::/10` | `s[0] & 0xffc0 == 0xfe80` |
| `ff00::/8` | `s[0] & 0xff00 == 0xff00` |
| `2001:db8::/32` | `s[0] == 0x2001 && s[1] == 0x0db8` |
Check the three "judge as IPv4" rows **before** the others. The last 32 bits as IPv4 are
`Ipv4Addr::from(((s[6] as u32) << 16) | s[7] as u32)` in spirit, but **no `as` casts**: use
`u32::from(s[6])` and `u32::from(s[7])`, or `s[6].to_be_bytes()` and `s[7].to_be_bytes()`.
Everything else is public. Do not use the standard library's `is_global` (unstable) or add ranges
that are not in these tables: the test checks public neighbours just outside each range too.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/toolkit/tests/addr.rs crates/toolkit/tests/`
- [ ] **2. See it fail.** `cargo test -p toolkit --test addr`. Expected: it does not compile.
- [ ] **3. Write `addr.rs`.** Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p toolkit --test addr`. Expected: 4 passed.
- [ ] **5. Walk the tables.** Point at the line for each row of both tables.
- [ ] **6. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **7. Log and commit.** `git add crates/toolkit docs/implementer-log.md Cargo.lock && git commit`
## Done when
- `cargo test -p toolkit --test addr` reports 4 passed; `make gate` prints `gate: ok`.
## Stop and report if
- A test's expectation disagrees with these tables.
+176
View File
@@ -0,0 +1,176 @@
# M3b task 08: the egress proxy
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `toolkit: the egress proxy`
## Goal
`http_fetch`'s container has no network. Its only way out is a Unix socket to this proxy, which
runs in a second container that does have a network. The proxy speaks the part of SOCKS5
(RFC 1928) that `curl --proxy socks5h://` uses, allows only the call's hosts, only port 443, only
public addresses (task 07), and after the handshake copies bytes both ways without looking at them
(TLS runs end to end between `curl` and the server). **Everything it reads during the handshake
comes from the untrusted side**: read exactly what the protocol says and nothing more, and never
index or allocate by a number you have not checked. Spec section 5.
## Files
- Copy: `crates/toolkit/tests/egress.rs`
- Create: `crates/toolkit/src/egress.rs`
- Modify: `crates/toolkit/src/lib.rs` (`pub mod egress;`), `crates/toolkit/src/main.rs`,
`docs/implementer-log.md`
## Interfaces
```rust
pub const HANDSHAKE_TIMEOUT: Duration = Duration::from_secs(10);
pub const CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
pub const MAX_CONNECTIONS: usize = 8;
// SOCKS5 reply codes the proxy sends:
pub const NOT_ALLOWED: u8 = 2;
pub const HOST_UNREACHABLE: u8 = 4;
pub const CONNECTION_REFUSED: u8 = 5;
pub const COMMAND_NOT_SUPPORTED: u8 = 7;
pub const ADDRESS_TYPE_NOT_SUPPORTED: u8 = 8;
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Allow { /* patterns: Vec<String>, private */ }
impl Allow {
/// One or more host patterns separated by ','. Every piece must pass
/// `proto::hosts::valid_host_pattern`; an empty piece (",", "a.com,", "") is an error.
pub fn parse(list: &str) -> Result<Allow, String>;
/// `valid_host(host)` and some pattern `host_matches` it.
pub fn permits(&self, host: &str) -> bool;
}
/// Name resolution and connecting, so tests need no network.
pub trait Dial: Send + Sync {
fn resolve(&self, host: &str, port: u16) -> std::io::Result<Vec<SocketAddr>>;
fn connect(&self, addr: SocketAddr, timeout: Duration) -> std::io::Result<TcpStream>;
}
/// The system's: `(host, port).to_socket_addrs()` and `TcpStream::connect_timeout`.
pub struct SystemDial;
impl Dial for SystemDial { }
pub struct Proxy { /* allow: Allow, dial: Arc<dyn Dial>, handshake: Duration — private */ }
impl Proxy {
pub fn new(allow: Allow, dial: Arc<dyn Dial>) -> Proxy; // handshake = HANDSHAKE_TIMEOUT
pub fn with_handshake_timeout(self, handshake: Duration) -> Proxy; // for tests
/// Accept for ever; at most MAX_CONNECTIONS handled at once, more are closed at once.
pub fn serve(self: Arc<Self>, listener: UnixListener) -> std::io::Result<()>;
/// One connection, from greeting to the end of the copy.
pub fn handle(&self, client: UnixStream);
}
```
## `handle`: the handshake, every exit
The whole handshake shares **one deadline**: `now + handshake`. Before every read, set the read
timeout to the time left (`deadline.checked_duration_since(Instant::now())`); if none is left, or a
read times out, fails, or reads 0 bytes (the peer closed), stop: return without a reply. A client
that sends one byte every 100 ms must still be cut off at the deadline (a test does exactly that).
Read with a loop that fills a buffer of the exact length the protocol gives, never more.
"Refuse with `code`" means: write `[5, code, 0, 1, 0, 0, 0, 0, 0, 0]` (ignore a write error) and
return, which closes the connection.
1. Read 2 bytes: `version`, `count`. `version != 5` → return, no reply.
2. Read `count` bytes of methods. If none of them is 0: write `[5, 0xff]`, return. Else write
`[5, 0]`.
3. Read 4 bytes: `version`, `command`, `reserved`, `kind`. `version != 5` or `reserved != 0`
return, no reply.
4. `command != 1` → refuse with `COMMAND_NOT_SUPPORTED`.
5. `kind != 3` (not a domain name; 1 is IPv4, 4 is IPv6) → refuse with
`ADDRESS_TYPE_NOT_SUPPORTED`. Do not read the address.
6. Read 1 byte: `length`. `length == 0` → refuse with `NOT_ALLOWED`.
7. Read `length` bytes of name, then 2 bytes of port (`u16::from_be_bytes`).
8. The name is not UTF-8, or `port != 443`, or `!allow.permits(&name)` → refuse with
`NOT_ALLOWED`.
9. `dial.resolve(&name, port)`. An error → refuse with `HOST_UNREACHABLE`. Take the **first**
address for which `crate::addr::is_public(addr.ip())` is true; none → refuse with
`HOST_UNREACHABLE`. Non-public addresses are skipped, **never tried**.
10. `dial.connect(addr, CONNECT_TIMEOUT)`. An error → refuse with `CONNECTION_REFUSED`.
11. Write the success reply `[5, 0, 0, 1, 0, 0, 0, 0, 0, 0]`; if that write fails, return.
## After the handshake: the copy
Clear the client's read timeout (`set_read_timeout(None)`). Copy both ways until both directions
are done, passing a half-close on:
- A second thread copies client → server (`std::io::copy`), then `shutdown(Shutdown::Write)` on
the server side.
- The handler's own thread copies server → client, then `shutdown(Shutdown::Write)` on the
client side, then joins the second thread.
Use `try_clone` for the second handle of each stream. If a `try_clone` fails, return.
## `serve`
For each accepted stream: if the number of connections being handled is already
`MAX_CONNECTIONS`, drop the stream at once (the client reads the end). Otherwise count it, handle
it on its own thread (`std::thread::Builder`, not `spawn`, which panics; if the thread cannot be
started, uncount it), and uncount it **inside that thread** when `handle` returns. An `accept`
error ends `serve` with that error. Use an `AtomicUsize` for the count.
**Never wait for (`join`) a handler thread in `serve`.** Start it, drop its `JoinHandle`, and go
straight back to `accept`. Joining makes the proxy serve one connection at a time: the second
client never gets its greeting, and `more_than_the_limit_of_connections_are_closed_at_once` fails
with a read timeout (`WouldBlock`). (A first attempt at this task did exactly that.)
## `main.rs`
Add one form before the tool form: the arguments **after the program name** are exactly
`egress-proxy --socket <path> --allow <list>`, in that order, all UTF-8. Take them with
`std::env::args_os().skip(1)`: without `skip(1)`, the first element is the program's own path, the
form never matches, and the proxy never listens (the given test then fails with "the proxy never
listened", and standard error shows `toolkit: unknown tool ""`; a first attempt did exactly that).
Then:
1. `Allow::parse(list)`; an error `e` → print `toolkit: egress-proxy: --allow: {e}` to standard
error, exit 2.
2. `UnixListener::bind(path)`. **Do not remove anything first**: the directory is fresh for each
call, so a file already there is a mistake. An error → print
`toolkit: egress-proxy: cannot listen on {path}: {e}`, exit 2.
3. `Arc::new(Proxy::new(allow, Arc::new(SystemDial))).serve(listener)`. If it returns an error →
print `toolkit: egress-proxy: {e}`, exit 2.
Any other argument list that starts with `egress-proxy` is not this form, and goes to the tool form
as before, which answers it as an unknown tool (exit 2).
## About the given tests
`egress.rs` handles one connection per test over `UnixStream::pair()`, with a fake `Dial` whose
names resolve from a table and whose connections all go to a local echo server. A proxy that closes
with some of the client's bytes unread makes the client's next read fail with "connection reset"
rather than read the end; the tests treat both as closed. That is normal and needs nothing from
you.
## Debugging
Work only inside the repository. Do not use `/tmp` or other directories outside it: OpenCode
refuses them, and a refused command ends nothing but wastes the turn. To try the program by hand,
put the socket under `target/`, for example
`target/debug/toolkit egress-proxy --socket target/egress-try.sock --allow example.com`.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/toolkit/tests/egress.rs crates/toolkit/tests/`
- [ ] **2. See it fail.** `cargo test -p toolkit --test egress`. Expected: it does not compile.
- [ ] **3. Write `egress.rs`** and the `lib.rs` and `main.rs` changes. Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p toolkit --test egress`. Expected: 15 passed. Run it ten
times; it must pass every time.
- [ ] **5. Walk the exits.** Point at the line of your code for each of the 11 handshake steps.
- [ ] **6. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **7. Log and commit.** `git add crates/toolkit docs/implementer-log.md Cargo.lock && git commit`
## Done when
- `cargo test -p toolkit --test egress` reports 15 passed ten times running; `make gate` prints
`gate: ok`.
## Stop and report if
- A test wants the proxy to try a non-public address, or to answer an IP-literal request with
anything but `ADDRESS_TYPE_NOT_SUPPORTED`.
- A test hangs.
+100
View File
@@ -0,0 +1,100 @@
# M3b task 09: `[runner]` in `brokerd.toml`
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `brokerd: the [runner] section`
## Goal
`[runner]` says how tools run: which `podman`, which image, and the limits. Without the section,
`brokerd` keeps M3a's runtime, which refuses every call (task 13 does that choice). The image must
be named **by digest**, so what runs is exactly what was built. Spec section 6, "Configuration".
## Files
- Copy: `crates/brokerd/tests/config_runner.rs`, the five files
`crates/brokerd/tests/fixtures/config/runner_*.toml`, and `crates/brokerd/tests/support/rig.rs`
(replaces the old one: it builds a `Config` with a struct literal, which now needs
`runner: None`)
- Modify: `crates/brokerd/src/config.rs`, `docs/implementer-log.md`
## Interfaces
In `config.rs`:
```rust
#[derive(Debug, Clone, PartialEq, Eq, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct Runner {
#[serde(default = "default_podman")] pub podman: PathBuf, // "podman"
pub image: String, // required, no default
#[serde(default = "default_egress_network")] pub egress_network: String, // "pasta"
#[serde(default = "default_output_cap")] pub output_cap: u64, // 262_144
#[serde(default = "default_memory")] pub memory: String, // "512m"
#[serde(default = "default_pids")] pub pids: u32, // 128
#[serde(default = "default_read_file_ms")] pub read_file_ms: u64, // 30_000
#[serde(default = "default_write_file_ms")] pub write_file_ms: u64, // 30_000
#[serde(default = "default_shell_ms")] pub shell_ms: u64, // 100_000
#[serde(default = "default_http_fetch_ms")] pub http_fetch_ms: u64, // 60_000
}
impl Runner {
/// The time limit for one call of `tool`: the matching `_ms` field.
pub fn time_limit(&self, tool: crate::args::ToolName) -> std::time::Duration;
}
pub struct Config {
// paths, sockets, approvals as now, then:
#[serde(default)]
pub runner: Option<Runner>,
}
impl Config {
// … as now, plus:
/// `<home>/run/egress`: where each `http_fetch` call gets a directory for the proxy's socket.
pub fn egress_dir(&self) -> PathBuf;
}
```
Write one small private `fn default_…() -> …` per defaulted field (serde needs a function path).
Put the struct attributes on separate lines as usual; the table above only saves space.
## Checks in `Config::load`
After the `ttl_ms` check, if `runner` is `Some`, check it and return the first problem as
`ConfigError::Invalid(path, text)`. In this order:
1. `image` is not `<name>@sha256:<hex>` where `<name>` is not empty and `<hex>` is exactly 64 of
`0-9a-f` (lowercase). Split with `rsplit_once("@sha256:")`. Text:
`[runner] image is {image:?}; it must be named by digest: <name>@sha256:<64 hex digits>`.
2. `memory` is not digits followed by one of `b`, `k`, `m`, `g` (at least one digit). Text:
`[runner] memory is {memory:?}; it must be a number and one of b, k, m, g`.
3. `egress_network` or `podman` is empty. Text:
`[runner] podman and egress_network must not be empty`.
4. Any of `output_cap`, `pids`, `read_file_ms`, `write_file_ms`, `shell_ms`, `http_fetch_ms` is 0,
checked in that order. Text: `[runner] <name> must be at least 1`, with the field's name.
A missing `image` or an unknown key is already a `ConfigError::Parse` from serde; do not add
checks for those. `Config::parse` (the string version the tests use for plain values) does not run
these checks; only `load` does, as for `ttl_ms` today.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/brokerd/tests/config_runner.rs crates/brokerd/tests/`,
`cp docs/plans/M3b/files/crates/brokerd/tests/fixtures/config/runner_*.toml crates/brokerd/tests/fixtures/config/`,
`cp docs/plans/M3b/files/crates/brokerd/tests/support/rig.rs crates/brokerd/tests/support/`
- [ ] **2. See it fail.** `cargo test -p brokerd --test config_runner`. Expected: it does not
compile.
- [ ] **3. Write the code.** Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p brokerd --test config_runner --test config`. Expected: 7
and 7 passed. Then `cargo test -p brokerd`: everything passes.
- [ ] **5. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/brokerd docs/implementer-log.md Cargo.lock && git commit`
## Done when
- `cargo test -p brokerd` passes; `make gate` prints `gate: ok`.
## Stop and report if
- A test wants an image named by tag (`:latest`) to be accepted.
+107
View File
@@ -0,0 +1,107 @@
# M3b task 10: the `podman` argument lists
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `brokerd: the podman argument lists`
## Goal
Build, without running anything, the exact `podman` arguments for one call's container, and for
`http_fetch`'s egress proxy. Every argument is its own `OsString`: nothing is ever passed through a
shell, and the tool's arguments go on standard input, never on the command line. The lists are
tested as golden files, one argument per line. Spec section 6.
The container's name says whose call it is, so `RunSpec` gains the session and the call.
## Files
- Copy: `crates/brokerd/tests/podman_args.rs` and the six files
`crates/brokerd/tests/fixtures/podman/*.args`
- Create: `crates/brokerd/src/podman.rs`
- Modify: `crates/brokerd/src/runner.rs`, `crates/brokerd/src/lib.rs` (`pub mod podman;`),
`docs/implementer-log.md`
## `RunSpec` gains two fields
In `runner.rs`, `RunSpec` gets `session: SessionId` and `call: CallId` as its **first** two
fields (private, like the others), with getters:
```rust
pub fn session(&self) -> &SessionId;
pub fn call(&self) -> CallId;
```
`run` fills them from `decision.request().session.clone()` and `decision.request().call`. The
`compile_fail` doctest at the top of `runner.rs` builds a `RunSpec` with a struct literal: add the
two fields to it, first, so it still fails **only** because the fields are private:
```rust
//! session: proto::SessionId::new("s1").unwrap(),
//! call: proto::CallId(1),
```
## `podman.rs`
```rust
pub const EGRESS_MOUNT: &str = "/run/egress";
pub const EGRESS_SOCKET: &str = "/run/egress/egress.sock";
pub const TOOLKIT: &str = "/bin/toolkit";
/// "boxmaker-<session>-<call>-<n>"
pub fn container_name(session: &SessionId, call: CallId, n: u64) -> String;
/// The tool's container. `egress` is the call's egress directory, for `http_fetch` only.
pub fn tool_args(spec: &RunSpec, runner: &Runner, name: &str, egress: Option<&Path>) -> Vec<OsString>;
/// The egress proxy's container.
pub fn egress_args(runner: &Runner, name: &str, dir: &Path, hosts: &[String]) -> Vec<OsString>;
```
`tool_args`, in this order:
```
run --rm -i --name=<name> --label=boxmaker=tool --network=none
--read-only --cap-drop=all --security-opt=no-new-privileges --userns=keep-id
--pids-limit=<runner.pids> --memory=<runner.memory>
--tmpfs=/tmp:rw,size=64m,mode=1777
--volume=<path>:<path>:ro (or :rw when writable) one per spec.mounts(), in order
--volume=<egress dir>:/run/egress:rw only when `egress` is Some
<runner.image> /bin/toolkit <spec.tool().as_str()>
```
`egress_args`, in this order:
```
run -d --rm --name=<name>-egress --label=boxmaker=egress --network=<runner.egress_network>
--read-only --cap-drop=all --security-opt=no-new-privileges --userns=keep-id
--pids-limit=64 --memory=128m
--volume=<dir>:/run/egress:rw
<runner.image> /bin/toolkit egress-proxy --socket /run/egress/egress.sock --allow <hosts joined with ",">
```
The six hardening flags (`--read-only` to `--memory=…`) are the same in both, so write them once in
a private function. Build the `--volume=<dir>:…` argument as an `OsString` with `push`, so a
directory need not be UTF-8 (`dir.as_os_str()`), not with `format!` on `display()`.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/brokerd/tests/podman_args.rs crates/brokerd/tests/`,
`mkdir -p crates/brokerd/tests/fixtures/podman`,
`cp docs/plans/M3b/files/crates/brokerd/tests/fixtures/podman/*.args crates/brokerd/tests/fixtures/podman/`
- [ ] **2. See it fail.** `cargo test -p brokerd --test podman_args`. Expected: it does not
compile.
- [ ] **3. Write the code.** Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p brokerd --test podman_args --test runner`. Expected: 7
and 8 passed. `cargo test -p brokerd --doc`: every doctest passes.
- [ ] **5. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/brokerd docs/implementer-log.md Cargo.lock && git commit`
## Done when
- `cargo test -p brokerd` and `cargo test -p brokerd --doc` pass; `make gate` prints `gate: ok`.
## Stop and report if
- A golden file disagrees with the lists above: report both, do not edit the golden file.
- Anything would put the tool's arguments (the command, the path, the URL) on `podman`'s command
line.
+135
View File
@@ -0,0 +1,135 @@
# M3b task 11: the Podman runtime
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `brokerd: the Podman runtime`
## Goal
`Podman` implements `Runtime`: each call runs in a fresh container, its arguments go on standard
input, its output is kept up to a cap, and it is stopped at its time limit. Whatever the tool
prints is the result's content, labelled by the grant; every failure is a **fixed sentence**, and
what Podman itself said goes only to `brokerd`'s log. Spec section 6, "One call". `http_fetch`'s
proxy is task 12; in this task every call runs its container with no egress directory.
## Files
- Copy: `crates/brokerd/tests/support/fake_podman.rs`, `crates/brokerd/tests/container.rs`, and
the **skeleton** `crates/brokerd/src/container.rs` (every signature, the constants, and `todo!()`
bodies with the steps as comments)
- Fill in: `crates/brokerd/src/container.rs`
- Modify: `crates/brokerd/src/lib.rs` (`pub mod container;`), `docs/implementer-log.md`
## Interfaces
```rust
pub const RUNBOOK: &str = "see docs/runbook.md#runner-unavailable";
pub const COULD_NOT_RUN: &str = "the tool could not run";
pub const CANNOT_START: &str = "the tool runner could not start the container";
pub const KILLED: &str = "the tool was stopped: it ran out of memory or was killed";
pub const TIMED_OUT: &str = "the tool ran past its time limit";
pub const UNEXPECTED: &str = "the tool failed with an unexpected status";
pub const POLL: Duration = Duration::from_millis(50);
pub const STDERR_KEPT: usize = 4096;
pub type Log = Arc<dyn Fn(&str) + Send + Sync>;
pub struct Podman { /* runner: Runner, egress_dir: PathBuf, log: Log, next: AtomicU64 — private */ }
impl Podman {
/// `egress_dir` is `Config::egress_dir()`; task 12 uses it.
pub fn new(runner: Runner, egress_dir: PathBuf, log: Log) -> Podman;
/// Where `http_fetch` calls get their directories. Nothing in this task calls it; it is public,
/// so the field counts as read and clippy's `dead_code` does not fire. Do not add `#[allow]`.
pub fn egress_dir(&self) -> &Path;
}
impl Runtime for Podman {
fn run(&self, spec: &RunSpec) -> Result<RunOutput, RunError>;
}
```
## `run`: every step and exit
1. `n = next.fetch_add(1, SeqCst)`; `name = podman::container_name(spec.session(), spec.call(), n)`.
The first call of a `Podman` is number 0.
2. `args = podman::tool_args(spec, &runner, &name, None)`;
`input = spec.arguments().canonical_json()`; `limit = runner.time_limit(spec.tool())`.
3. Spawn `Command::new(&runner.podman).args(args)` with standard input, output and error all
piped. A failure → log `brokerd: cannot start {podman path}: {e}` + `"\n"` + `RUNBOOK`, and
return `Err(RunError::Unavailable(CANNOT_START))`.
4. Three threads, so no pipe can block another: one **writes** `input` to standard input and then
drops it (closing it); one **reads** standard output, keeping the first `output_cap` bytes and
**reading on past the cap**, throwing the rest away, and remembering that there was more; one
reads standard error the same way, keeping `STDERR_KEPT` bytes. (Stopping to read would leave
the tool blocked on a full pipe; closing the pipe would make Podman fail. Both give the wrong
answer.) Write the reading loop once, as a private function returning `(Vec<u8>, bool)`.
5. Wait with a loop: `child.try_wait()`; if it has ended, go to 6. If `limit` has passed since the
spawn, **stop the container**: run `podman kill <name>`, then `podman rm -f <name>` (each with
`.status()`, standard streams null; log a line if one does not succeed, and go on), then
`child.kill()` and `child.wait()`. Otherwise sleep `POLL` and try again.
6. Join the three threads (a thread that panicked counts as empty output).
7. The answer:
| How it ended | Answer | Log |
|---|---|---|
| past the time limit (step 5) | `Err(RunError::Failed(TIMED_OUT))` | a line naming the container |
| exit 0 or 1 | `Ok(RunOutput { content, truncated })`, content = the kept output decoded with `from_utf8_lossy` | — |
| exit 2 | `Err(RunError::Failed(COULD_NOT_RUN))` | the kept standard error |
| exit 125, 126 or 127 | `Err(RunError::Unavailable(CANNOT_START))` | the kept standard error, then `"\n"` + `RUNBOOK` |
| exit 137 | `Err(RunError::Failed(KILLED))` | — |
| anything else, including killed by a signal | `Err(RunError::Failed(UNEXPECTED))` | the status and the kept standard error |
**The tool's output never goes into a `RunError`**: only the six constants do.
Also add `pub(crate) fn podman(&self, args: &[&str])`, the helper step 5 uses to run a short
`podman` command and log if it fails; task 12 uses it too.
## About the given tests
The tests run `brokerd`'s real `runner::run` against a fake `podman`: a shell script that records
every call's arguments in a file and then does what the test says. As in task 06, every test takes
the `serial()` lock first, because writing a script and running it at once races with other tests'
forks ("Text file busy"). The time-limit test's fake ends with `exec sleep 30`, so killing the
process kills the sleep and nothing keeps the pipes open.
## How to work (read this first)
Three earlier sessions on this task ended without a commit: each tried to plan the whole file
(the third, the whole of `run`) in one turn and ran out of room while still deciding. So:
- **Start from the skeleton.** It compiles. `run` and `run_container` are **already written**: they
only call the helpers. Replace the `todo!()`s **one function at a time**, in this order:
`read_capped`, `new`, `egress_dir`, `podman`, `wait`, `spawn` (step 3), `Io::start` (step 4),
`Io::finish` (step 6), `answer` (step 7, the table). After each, run `cargo check -p brokerd`
and fix what it says before going on. Each helper is a few lines; none needs the others'
details beyond its signature.
- **Let the compiler answer API questions.** If you are unsure whether something compiles (how to
call the log, which trait a type has), write it and run `cargo check`. The log is called as
`(self.log)(&line)`.
- Keep each turn short: write, check, next. Do not restate the task to yourself.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/brokerd/tests/support/fake_podman.rs crates/brokerd/tests/support/`
and `cp docs/plans/M3b/files/crates/brokerd/tests/container.rs crates/brokerd/tests/`
and `cp docs/plans/M3b/files/crates/brokerd/src/container.rs crates/brokerd/src/`, then add
`pub mod container;` to `crates/brokerd/src/lib.rs` after `pub mod config;`.
- [ ] **2. See it fail.** `cargo test -p brokerd --test container`. Expected: it compiles (with
warnings about unused variables) and all 11 fail on `todo!()`.
- [ ] **3. Fill in `container.rs`** as in "How to work". Delete the skeleton paragraph from the
module doc. Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p brokerd --test container`. Expected: 11 passed. Run it ten
times; it must pass every time.
- [ ] **5. Walk the table.** Point at the line of your code for each row, and check that no row
puts output into a `RunError`.
- [ ] **6. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **7. Log and commit.** `git add crates/brokerd docs/implementer-log.md Cargo.lock && git commit`
## Done when
- `cargo test -p brokerd --test container` reports 11 passed ten times running; `make gate` prints
`gate: ok`.
## Stop and report if
- A test hangs, or fails only sometimes: report which, and how often, rather than add sleeps.
- A test wants tool output inside a failure message.
+93
View File
@@ -0,0 +1,93 @@
# M3b task 12: the egress proxy's lifecycle
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `brokerd: start and remove the egress proxy for http_fetch`
## Goal
For an `http_fetch` call, the tool's container still has no network. Before it runs, `brokerd`
makes a directory for this call, starts the egress proxy (task 08) in its own container with a
network and that directory mounted, and waits for the proxy's socket. The tool's container gets the
same directory. Afterwards, **on every path**, the proxy's container and the directory are removed.
Spec section 6, "`http_fetch`".
## Files
- Copy: `crates/brokerd/tests/container_egress.rs`
- Modify: `crates/brokerd/src/container.rs`, `docs/implementer-log.md`
## Interfaces (added to `container.rs`)
```rust
pub const EGRESS_WAIT: Duration = Duration::from_secs(5);
impl Podman {
/// The same runtime with another wait for the proxy's socket, for tests.
pub fn with_egress_wait(self, egress_wait: Duration) -> Podman;
}
```
`Podman` gains a private field `egress_wait`, set to `EGRESS_WAIT` by `new`.
## `run`, now
After step 2 of task 11 (name, input, limit):
- `spec.egress()` is `None` → exactly as before: `tool_args(…, None)` and `run_container`.
- `spec.egress()` is `Some(hosts)` → start the proxy (below), and on success call
`run_container` with `tool_args(spec, &runner, &name, Some(&dir))`.
`run_container` (from task 11) does steps 3 to 7 for any argument list, so the only change to
`run` is this choice. Put the proxy's start in its own function (`start_egress`), returning the
guard or the error.
### Starting the proxy: every step and exit
Use a **guard**: a private struct holding the proxy's container name (`<name>-egress`) and the
directory, whose `Drop` runs `podman rm -f <name>-egress` (the helper from task 11) and then
`remove_dir_all(dir)` (log an error other than `NotFound`). Create the guard **first**, before
anything can fail, so that every return below cleans up by dropping it, and keep it alive until
the tool's container has finished.
1. `dir = egress_dir.join(&name)`. Make `egress_dir` (and its parents) with mode 0700
(`DirBuilder::new().recursive(true).mode(0o700)`), then set its mode to 0700 anyway (it may
have existed). If `dir` already exists, it was left by a crash: `remove_dir_all` it (a
`NotFound` is fine). Then create `dir` with mode 0700, not recursively. Any failure → log
`brokerd: cannot make {dir}: {e}` + `"\n"` + `RUNBOOK`, return
`Err(RunError::Unavailable(CANNOT_START))`.
2. Run `podman` with `podman::egress_args(&runner, &name, &dir, hosts)`, standard input and output
null, standard error piped, with `.output()`. It does not succeed → log
`brokerd: podman could not start {name}-egress: {stderr}` + `"\n"` + `RUNBOOK`, return
`Unavailable(CANNOT_START)`. It cannot be started at all → the same log as task 11 step 3, and
the same answer.
3. Wait until `dir.join("egress.sock")` exists, checking every 20 ms, for at most `egress_wait`.
It does not appear → log `brokerd: {name}-egress did not make its socket within {ms} ms` +
`"\n"` + `RUNBOOK`, return `Unavailable(CANNOT_START)`.
4. Return the guard. Run the tool's container as in task 11. Whatever it answers, the guard is
dropped after it, removing the proxy and the directory.
So for a successful call the fake sees exactly three `podman` calls: `run -d …` (the proxy),
`run --rm -i …` (the tool), `rm -f <name>-egress`. For a tool past its time limit it sees the
proxy's run, the tool's run, `kill <name>`, `rm -f <name>`, then `rm -f <name>-egress`.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/brokerd/tests/container_egress.rs crates/brokerd/tests/`
- [ ] **2. See it fail.** `cargo test -p brokerd --test container_egress`. Expected: it does not
compile (`with_egress_wait` does not exist).
- [ ] **3. Write the code.** Run `cargo fmt --all`. `container.rs` must stay under 500 lines.
- [ ] **4. See it pass.** `cargo test -p brokerd --test container_egress --test container`.
Expected: 6 and 11 passed. Run them ten times; they must pass every time.
- [ ] **5. Walk the exits.** For each of the four steps, say how the guard cleans up after it.
- [ ] **6. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **7. Log and commit.** `git add crates/brokerd docs/implementer-log.md Cargo.lock && git commit`
## Done when
- Both suites pass ten times running; `make gate` prints `gate: ok`.
## Stop and report if
- A path leaves the proxy's container or the directory behind, and a guard does not fix it.
- The tool's container would get a network.
+56
View File
@@ -0,0 +1,56 @@
# M3b task 13: `brokerd serve` uses the runtime
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `brokerd serve: run tools in containers when [runner] is set`
## Goal
With a `[runner]` section, `brokerd serve` runs allowed calls through `Podman`; without it, it keeps
M3a's `Refusing` runtime. It prints one line saying which, so the owner can see it at a glance.
## Files
- Copy: `crates/brokerd/tests/serve_runner.rs`
- Modify: `crates/brokerd/src/main.rs`, `docs/implementer-log.md`
## The change
In `main.rs`, after the config is loaded and before `serve::start`:
```rust
let log: Arc<dyn Fn(&str) + Send + Sync> = Arc::new(|line: &str| eprintln!("{line}"));
```
and choose the runtime and its notice:
- `cfg.runner` is `Some(runner)``Box::new(Podman::new(runner, cfg.egress_dir(), Arc::clone(&log)))`,
notice `brokerd: tools run in containers from {runner.image}` (take the image before `runner`
moves).
- `None``Box::new(Refusing)`, notice `brokerd: no [runner] section: every tool call is refused`.
Pass the runtime and `log` to `serve::start` (instead of `Box::new(Refusing)` and the closure made
there today). Print the notice with `eprintln!` right after the existing
`brokerd: serving tools on … and approvals on …` line. Nothing else changes.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/brokerd/tests/serve_runner.rs crates/brokerd/tests/`
- [ ] **2. See it fail.** `cargo test -p brokerd --test serve_runner`. Expected: 2 fail (the
notices are missing, and the call is refused).
- [ ] **3. Make the change.** Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p brokerd --test serve_runner --test serve`. Expected: 2
and 9 passed.
- [ ] **5. Run the gate.** `make gate`. Expected last line: `gate: ok`, with about 638 tests in
all.
- [ ] **6. Log and commit.** `git add crates/brokerd docs/implementer-log.md Cargo.lock && git commit`
This is the last task of M3b. Stop after the commit; the review comes next.
## Done when
- `cargo test -p brokerd` passes; `make gate` prints `gate: ok`.
## Stop and report if
- `serve::start`'s signature would have to change.
+63
View File
@@ -0,0 +1,63 @@
# M3b task 14: move the pipe handling into its own module
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `brokerd: move the container's pipe handling into pipes.rs`
## Goal
`container.rs` is 432 lines, and the next two tasks add to the pipe handling. Before they do, move
that code into its own file, **without changing what it does**. This is a pure move: every test
passes before and after, unchanged.
## Files
- Create: `crates/brokerd/src/pipes.rs`
- Modify: `crates/brokerd/src/container.rs`, `crates/brokerd/src/lib.rs`, `docs/implementer-log.md`
- No test files change.
## What moves
From `container.rs`, cut these three items, **bodies unchanged**, and paste them into `pipes.rs`:
1. `struct Io { … }` (the three `JoinHandle` fields) → `pub(crate) struct Io`.
2. `impl Io { fn start(…) … fn finish(…) … }` → the same, with `pub(crate) fn start` and
`pub(crate) fn finish`.
3. `fn read_capped(…)``pub(crate) fn read_capped`.
Their doc comments move with them. `pipes.rs` starts with a module doc comment:
```rust
//! The three pipes of one container: its arguments go in on standard input, its output and errors
//! come back, each on its own thread so no pipe can block another.
```
and the `use` lines those items need (`std::io::{Read, Write}`, `std::process::Child`,
`std::thread::JoinHandle`). `Io::start` uses `STDERR_KEPT`: leave that constant in `container.rs`
(the tests import it from there) and write `crate::container::STDERR_KEPT` in `pipes.rs`.
In `lib.rs`: `pub mod pipes;`, between `pub mod ledger;` and `pub mod podman;`.
In `container.rs`: add `use crate::pipes::Io;`, and remove the `use` lines the compiler then says
are unused. Nothing else in `container.rs` changes.
## Steps
- [ ] **1. Branch.** `git switch m3b`.
- [ ] **2. Move the three items** as above. Run `cargo fmt --all`.
- [ ] **3. Check it builds.** `cargo check -p brokerd`, then
`cargo clippy -p brokerd --all-targets -- -D warnings`. Fix only imports.
- [ ] **4. See every test pass, unchanged.**
`cargo test -p brokerd --test container --test container_egress --test serve_runner`. Expected:
11, 6 and 2 passed.
- [ ] **5. Check it moved.** `grep -n "struct Io\|fn read_capped" crates/brokerd/src/container.rs`
prints nothing; `wc -l crates/brokerd/src/container.rs` is under 380.
- [ ] **6. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **7. Log and commit.** `git add crates/brokerd docs/implementer-log.md Cargo.lock && git commit`
## Done when
- `container.rs` no longer holds `Io` or `read_capped`; `make gate` prints `gate: ok`.
## Stop and report if
- Any test fails after the move: a pure move cannot break one, so report instead of changing code.
+152
View File
@@ -0,0 +1,152 @@
# M3b task 15: no thread panics, and a grace period for the output
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `brokerd: start pipe threads safely and collect output within a grace period`
## Goal
Two defects from the M3b review, both in `pipes.rs` (task 14):
- **Finding 3.** `std::thread::spawn` panics when the system cannot make a thread. The panic drops
the `Child` without stopping the container, which then runs on without its time limit. Use
`std::thread::Builder`, which returns an error instead, and stop the container on that error.
- **Finding 4.** `Io::finish` joins the readers with no limit. Each reader stops only at its pipe's
end, so if any other process still holds a pipe, `brokerd` waits for it: the call's time limit
does not bound it. After the container has ended, give the output a **grace period** of 2 s,
then abandon what has not ended.
## Files
- Copy: `crates/brokerd/tests/container_grace.rs`
- Modify: `crates/brokerd/src/pipes.rs`, `crates/brokerd/src/container.rs`,
`docs/implementer-log.md`
## `pipes.rs`, new shape
```rust
use std::sync::mpsc::{self, Receiver, RecvTimeoutError};
use std::thread::Builder;
use std::time::{Duration, Instant};
/// How long, after the container has ended, its output may take to reach its end.
pub const GRACE: Duration = Duration::from_secs(2);
/// What came back.
pub(crate) struct Finished {
pub out: Vec<u8>,
pub truncated: bool,
pub err: String,
/// A pipe had not ended by the grace deadline: something outside the container holds it.
pub open: bool,
}
pub(crate) struct Io {
stdout: Option<Receiver<(Vec<u8>, bool)>>,
stderr: Option<Receiver<(Vec<u8>, bool)>>,
}
impl Io {
pub(crate) fn start(child: &mut Child, input: String, cap: usize, err_cap: usize)
-> std::io::Result<Io>;
pub(crate) fn finish(self, grace: Duration) -> Finished;
}
pub(crate) fn read_capped(from: impl Read, cap: usize) -> (Vec<u8>, bool); // as now
```
Write each function in turn, running `cargo check -p brokerd` after each.
### `Io::start`
1. If `child.stdin.take()` is `Some(stdin)`: start a thread with `Builder::new().spawn(…)` that
writes `input` with `write_all` (ignore its error) and then drops `stdin`. **Do not keep its
handle and never join it**: it ends when the input is written or the pipe breaks. A spawn error
→ return it (`?`).
2. For standard output with `cap`, and then standard error with `err_cap`: if the pipe is
`Some`, make a channel (`mpsc::channel()`), start a thread with `Builder::new().spawn(…)` that
runs `read_capped(pipe, cap)` and `send`s the result (ignore a send error), and keep the
`Receiver`. A spawn error → return it. Write a small private function
`fn reader(pipe: impl Read + Send + 'static, cap: usize) -> std::io::Result<Receiver<(Vec<u8>, bool)>>`
and use it for both.
3. `Ok(Io { stdout, stderr })`.
### `Io::finish(grace)`
One deadline for both readers: `let until = Instant::now() + grace;`. For each receiver (standard
output first), a small private function `collect(rx, until) -> (Vec<u8>, bool, bool)`:
- `None` (no pipe) → `(empty, false, false)`.
- `rx.recv_timeout(until.saturating_duration_since(Instant::now()))`:
- `Ok((bytes, truncated))``(bytes, truncated, false)`;
- `Err(RecvTimeoutError::Timeout)``(empty, false, true)`: **open**;
- `Err(RecvTimeoutError::Disconnected)` (the reader panicked) → `(empty, false, false)`.
Return `Finished { out, truncated, err: String::from_utf8_lossy(&err_bytes).into_owned(),
open: stdout_open || stderr_open }`.
### `read_capped`
Unchanged in what it does, but no indexing and no subtraction that could wrap: keep
`let take = n.min(cap.saturating_sub(kept.len()));`, copy with
`kept.extend_from_slice(chunk.get(..take).unwrap_or_default());`, and set `truncated` when
`take < n`. Retry `ErrorKind::Interrupted`; stop on any other error.
## `container.rs`
1. A new constant after `TIMED_OUT`:
`pub const OUTPUT_OPEN: &str = "the tool left its output open";`
2. `use crate::pipes::{GRACE, Io};`
3. In `run_container`, replace the lines from `let io = Io::start(…)` to the end of the function
with exactly this:
```rust
let io = match Io::start(&mut child, input, cap, STDERR_KEPT) {
Ok(io) => io,
Err(e) => {
// The container may be running: stop it before answering.
self.podman(&["kill", name]);
self.podman(&["rm", "-f", name]);
let _ = child.kill();
let _ = child.wait();
(self.log)(&format!("brokerd: cannot start a thread for {name}: {e}\n{RUNBOOK}"));
return Err(RunError::Unavailable(CANNOT_START.to_string()));
}
};
let status = self.wait(&mut child, name, started, limit);
let done = io.finish(GRACE);
if done.open && status.is_some() {
(self.log)(&format!(
"brokerd: {name} ended but something still holds its output; it was abandoned"
));
return Err(RunError::Failed(OUTPUT_OPEN.to_string()));
}
self.answer(name, status, done.out, done.truncated, &done.err)
```
When the time limit was passed (`status` is `None`), an open pipe changes nothing: the answer is
still `TIMED_OUT`, from `answer`.
Also: `grep -n "thread::spawn" crates/brokerd/src/` must print nothing when you are done.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/brokerd/tests/container_grace.rs crates/brokerd/tests/`
- [ ] **2. See it fail.** `cargo test -p brokerd --test container_grace`. Expected: it does not
compile (`OUTPUT_OPEN` and `pipes::GRACE` do not exist).
- [ ] **3. Write `pipes.rs`, one function at a time, then `container.rs`.** Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p brokerd --test container_grace`. Expected: 4 passed, in
about 4 s (two tests wait out the grace period on purpose). Then
`cargo test -p brokerd --test container --test container_egress`: 11 and 6 passed. Run all three
five times.
- [ ] **5. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/brokerd docs/implementer-log.md Cargo.lock && git commit`
## Done when
- The three suites pass five times running; no `thread::spawn` in `crates/brokerd/src/`; `make
gate` prints `gate: ok`.
## Stop and report if
- A test takes much longer than stated, or passes only sometimes.
+61
View File
@@ -0,0 +1,61 @@
# M3b task 16: escape what the container writes before logging it, and three small texts
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `brokerd: escape container errors in the log; prefix and quote two messages`
## Goal
**M3b review finding 5.** What a container writes on standard error (and Podman's own errors, which
the tool can influence) goes into `brokerd`'s log as it is, newlines included. A tool can therefore
write a line that looks like one of `brokerd`'s own, for example a fake
`see docs/runbook.md#…` pointer. Log it with Rust's debug formatting (`{err:?}`), which quotes it and
writes every newline and control character as an escape: one event stays one entry.
And three small texts from the review (finding 7): the runtime notice lacks its `brokerd:` prefix,
and two `[runner]` errors should quote the bad value.
## Files
- Copy: `crates/brokerd/tests/container_log.rs`, `crates/brokerd/tests/notices.rs`
- Modify: `crates/brokerd/src/container.rs`, `crates/brokerd/src/main.rs`,
`crates/brokerd/src/config.rs`, `docs/implementer-log.md`
## The changes, exactly
In `container.rs`, `answer`, three log calls become:
| Case | Log call |
|---|---|
| exit 2 | `(self.log)(&format!("brokerd: {name}: the tool could not run: {err:?}"));` |
| exit 125 to 127 | `(self.log)(&format!("brokerd: podman could not start {name}: {err:?}\n{RUNBOOK}"));` |
| any other ending | `(self.log)(&format!("brokerd: container {name} exited {status}: {err:?}"));` |
In `start_egress`, where the proxy's `podman run -d` does not succeed, the standard error in the log
call becomes `{stderr:?}` in the same way (the rest of that line, and its `\n{RUNBOOK}`, stay).
The only raw `\n` left in any of these is the one before `RUNBOOK`, which is ours.
In `main.rs`: the line that prints the runtime notice becomes `eprintln!("brokerd: {runtime_notice}");`.
In `config.rs`: in the two messages `[runner] image is {}; …` and `[runner] memory is {}; …`, the
`{}` becomes `{:?}`, so the value is shown in quotes.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/brokerd/tests/container_log.rs docs/plans/M3b/files/crates/brokerd/tests/notices.rs crates/brokerd/tests/`
- [ ] **2. See them fail.** `cargo test -p brokerd --no-fail-fast --test container_log --test notices`.
Expected: 4 and 2 fail.
- [ ] **3. Make the changes.** Run `cargo fmt --all`.
- [ ] **4. See them pass.** `cargo test -p brokerd --test container_log --test notices --test serve_runner --test config_runner`.
Expected: 4, 2, 2 and 7 passed.
- [ ] **5. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/brokerd docs/implementer-log.md Cargo.lock && git commit`
## Done when
- The four suites pass; `make gate` prints `gate: ok`.
## Stop and report if
- A test wants container output anywhere but the log.
+69
View File
@@ -0,0 +1,69 @@
# M3b task 17: three small fixes in `toolkit`
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `toolkit: no thread panic in http_fetch, no casts, exact egress-proxy form`
## Goal
Three findings from the M3b review (3 and 7), all small:
1. `fetch.rs` starts its standard-error reader with `std::thread::spawn`, which panics when the
system cannot make a thread. Use `std::thread::Builder`.
2. `input.rs` and `files.rs` compute a byte limit with an `as u64` cast, which AGENTS forbids.
3. `toolkit egress-proxy --socket <path> --allow <list> extra` is taken as the proxy form and
listens. The form is **exactly** five words; anything longer is not it.
## Files
- Copy: `crates/toolkit/tests/egress_form.rs`
- Modify: `crates/toolkit/src/fetch.rs`, `crates/toolkit/src/input.rs`,
`crates/toolkit/src/files.rs`, `crates/toolkit/src/main.rs`, `docs/implementer-log.md`
## The changes, exactly
**1. `fetch.rs`.** Replace
`let stderr_handle = std::thread::spawn(move || read_capped(stderr_reader));` with:
```rust
// `Builder`, not `spawn`, which panics when the system refuses a thread.
let stderr_handle = match std::thread::Builder::new().spawn(move || read_capped(stderr_reader)) {
Ok(handle) => handle,
Err(e) => {
let _ = child.kill();
let _ = child.wait();
return Outcome::tool_error(format!("http_fetch: cannot start a thread: {e}"));
}
};
```
**2. `input.rs` and `files.rs`.** Replace `.take(MAX_INPUT as u64 + 1)` with
`.take(u64::try_from(MAX_INPUT).map_or(u64::MAX, |n| n.saturating_add(1)))`, and the same for
`MAX_READ` in `files.rs`.
**3. `main.rs`.** In `parse_egress_proxy`, the first check becomes:
```rust
// Exactly five words: a longer list is not this form, and goes to the tool form (exit 2).
if args.len() != 5 || args.first()?.as_bytes() != b"egress-proxy" {
return None;
}
```
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/toolkit/tests/egress_form.rs crates/toolkit/tests/`
- [ ] **2. See it fail.** `cargo test -p toolkit --test egress_form`. Expected: 1 fails after about
3 s ("it is listening").
- [ ] **3. Make the three changes.** Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p toolkit`. Expected: every suite passes; `egress_form` 1
passed quickly.
- [ ] **5. Check.** `grep -n "thread::spawn\| as u64" crates/toolkit/src/` prints nothing.
- [ ] **6. Run the gate.** `make gate`. Expected last line: `gate: ok`, with about 649 tests.
- [ ] **7. Log and commit.** `git add crates/toolkit docs/implementer-log.md Cargo.lock && git commit`
This is the last follow-up task of M3b. Stop after the commit.
## Done when
- `cargo test -p toolkit` passes; the grep prints nothing; `make gate` prints `gate: ok`.
+101
View File
@@ -0,0 +1,101 @@
# M3b implementation plan: the runner and the tools
> **For the implementing model:** do not work from this file. The owner gives you one task file at
> a time (`01-…` to `13-…`). This file is the index for the owner and the reviewer.
**Goal:** an allowed tool call runs in a fresh rootless Podman container, with no network unless
it is `http_fetch`, which reaches the grant's hosts only through an egress proxy of our own. The
four tools and the proxy are programs in `toolkit`; `brokerd` builds the `podman` argument lists
and runs them.
**Architecture:** shared definitions move to `proto` first (01), the fetch target is sealed (02),
and grant paths that cannot be mounted are refused (03). `toolkit` gets its four tools (04 to 06),
the public-address check (07) and the proxy (08). `brokerd` gets `[runner]` (09), the argument
lists (10), the runtime (11), the proxy's lifecycle (12) and the wiring in `serve` (13).
**Spec:** `docs/specs/2026-09-22-m3b-runner.md`. Brief: `docs/design.md`. Every fail-closed
message ends with a pointer into `docs/runbook.md`; the `runner-unavailable` entry is already
there.
**No task needs Podman, Nix or a network.** Everything runs against fakes: fake `curl` and
`podman` scripts, a fake resolver, `UnixStream::pair()`. The image and the checks on straylight
are done by the design model afterwards.
## Global constraints
- Everything in `AGENTS.md`, including "Lessons from earlier reviews".
- No new dependency. `toolkit` gains `serde` and `serde_json`, both already vetted (task 04).
- Branch `m3b`. One task, one fresh OpenCode session, one commit. Run `cargo fmt --all` before the
gate. Review happens once, after task 13.
- Tests that write a script and run it take a lock (`serial()`): otherwise another test's fork
can make running it fail with "text file busy". Keep that pattern in anything you add.
## Tasks
The last column is how the given tests were checked before hand-over (decision of 2026-09-18, tip
T17). For M3b every task had a reference implementation: the tests of each task were run against
it at that task's end state, then the reference was deleted so it cannot be read (tip T18).
| # | File | Delivers | Tests | Check |
|---|---|---|---|---|
| 01 | `01-proto-tools-hosts.md` | `proto::tools`, `proto::hosts`; `brokerd::args` uses them | `proto/tests/tools.rs`, `hosts.rs` | reference; the given test pinned an accepted serde weakness, removed |
| 02 | `02-brokerd-fetch-url.md` | `ToolArgs::HttpFetch(FetchUrl)`, sealed (M3a finding 14) | `brokerd/tests/args.rs` (changed), a `compile_fail` doctest | reference |
| 03 | `03-brokerd-grant-mount-rule.md` | grant paths with `:` or `,` are invalid | `brokerd/tests/grants_mount.rs` | reference |
| 04 | `04-toolkit-files.md` | the `toolkit` program, `read_file`, `write_file` | `toolkit/tests/files.rs`, `support/mod.rs` | reference |
| 05 | `05-toolkit-shell.md` | `shell` | `toolkit/tests/shell.rs` | reference |
| 06 | `06-toolkit-fetch.md` | `http_fetch` through `curl` | `toolkit/tests/fetch.rs` | reference; found the ETXTBSY race (1 in 7 runs), fixed with `serial()` |
| 07 | `07-toolkit-addr.md` | `is_public` | `toolkit/tests/addr.rs` | reference; found `::/96` missing from the spec, added |
| 08 | `08-toolkit-egress-proxy.md` | the SOCKS5 egress proxy | `toolkit/tests/egress.rs` | reference; 20 runs clean |
| 09 | `09-brokerd-runner-config.md` | `[runner]` | `config_runner.rs`, 5 fixtures, `support/rig.rs` (changed) | reference |
| 10 | `10-brokerd-podman-args.md` | the `podman` argument lists; `RunSpec` gains session and call | `podman_args.rs`, 6 golden files | reference |
| 11 | `11-brokerd-container.md` | the Podman runtime | `container.rs`, `support/fake_podman.rs` | reference; 20 runs clean |
| 12 | `12-brokerd-egress.md` | the proxy's lifecycle for `http_fetch` | `container_egress.rs` | reference; 20 runs clean |
| 13 | `13-brokerd-serve-runner.md` | `brokerd serve` uses the runtime | `serve_runner.rs` | reference |
| 14 | `14-brokerd-pipes-module.md` | review follow-up: the pipe handling moves to `pipes.rs`, unchanged | none new | replayed on its own |
| 15 | `15-brokerd-pipes-grace.md` | review findings 3 and 4: `thread::Builder`; a 2 s grace period for the output | `container_grace.rs` | reference; 5 runs clean |
| 16 | `16-brokerd-log-escaping.md` | review findings 5 and 7: container errors escaped in the log; two texts | `container_log.rs`, `notices.rs` | reference; red without it |
| 17 | `17-toolkit-nits.md` | review findings 3 and 7 in `toolkit` | `egress_form.rs` | reference; red without it |
At the end of task 13: about 638 tests; at the end of task 17: about 649.
## Changes during the run
- 2026-09-23, after task 04: task 04 adds dependencies to `toolkit`, which changes `Cargo.lock`, but
its `git add` line left the lock out, so the driver stopped on an unclean tree. The design model
folded the lock into task 04's commit and added `Cargo.lock` to every task's `git add` (a no-op
when it has not changed). The owner resumes from task 05.
- 2026-09-23, task 08: the first attempt passed 13 of 15 tests and ended without a commit, after
OpenCode refused a command in `/tmp`. Its two defects: `main` did not skip the program name, so
`egress-proxy` never matched; and `serve` joined each handler thread, serving one connection at a
time. The attempt is saved in `.state/runs/M3b/08-first-attempt.diff`; the tree was reset, and the
task now says both things explicitly and to debug inside the repository. Resume from task 08.
- 2026-09-23, task 11: the session ended in deliberation, with nothing written. The task stored
`egress_dir` for task 12 but gave nothing that read it, so clippy's `dead_code` would fail the
gate and `#[allow]` is forbidden: a task that could not be written as described (tip T16). The
reference had a public `egress_dir()` getter; the task now gives it. Resume from task 11.
- 2026-09-23, task 11 again: the second session also ended with nothing written, cut off while
deliberating over API details the compiler would have settled. The task now hands over a
compiling skeleton of `container.rs` (signatures, constants, the steps as comments, `todo!()`
bodies), checked against the given tests (11 red), and says to fill one function at a time with
`cargo check` between. Resume from task 11.
- 2026-09-23, task 11, third attempt: it filled five of the six functions from the skeleton, with
`cargo check` between, then planned all of `run` in one turn and was cut off (two compile errors
left). Saved in `.state/runs/M3b/11-third-attempt.diff`. The skeleton now has `run` and
`run_container` written as glue, and the rest as small helpers (`spawn`, `Io::start`,
`Io::finish`, `answer`); the design model filled them in a scratch copy to check the split can
pass (11 of 11, five runs, clippy clean) and removed that. Task 12 now calls `run_container`.
Resume from task 11.
- 2026-09-23, after the review: two plan defects found on straylight were fixed by the design
model: `curl` gains `--globoff` (and a leading `--disable`), since `[1-3]` in a URL made three
requests; both `podman run`s gain `--pull=never`, since a missing image made Podman try to pull.
The given `fetch.rs` and the six golden files changed with them. Follow-up tasks 14 to 17 are for
the implementer; run them with `tools/run-plan.sh docs/plans/M3b 14`.
## Running it
```sh
BOXMAKER_MODEL=straylight/ornith-1.5-35b-a3b tools/run-plan.sh docs/plans/M3b
```
Keep the OpenCode TUI closed while it runs.
@@ -0,0 +1,148 @@
//! The Podman runtime: one fresh container per call, with the limits of `[runner]`. Whatever the
//! tool prints is the result's content; every failure is a fixed sentence, and what Podman itself
//! said goes only to `brokerd`'s log. M3b spec, section 6.
//!
//! SKELETON from task 11: replace every `todo!()`, one function at a time, in the order the task
//! gives, running `cargo check -p brokerd` after each. `run` and `run_container` are already
//! written. Then delete this paragraph.
use std::ffi::OsString;
use std::io::{Read, Write};
use std::path::{Path, PathBuf};
use std::process::{Child, Command, ExitStatus, Stdio};
use std::thread::JoinHandle;
use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::{Duration, Instant};
use crate::config::Runner;
use crate::podman;
use crate::runner::{RunError, RunOutput, RunSpec, Runtime};
pub const RUNBOOK: &str = "see docs/runbook.md#runner-unavailable";
pub const COULD_NOT_RUN: &str = "the tool could not run";
pub const CANNOT_START: &str = "the tool runner could not start the container";
pub const KILLED: &str = "the tool was stopped: it ran out of memory or was killed";
pub const TIMED_OUT: &str = "the tool ran past its time limit";
pub const UNEXPECTED: &str = "the tool failed with an unexpected status";
/// How often a running container is checked.
pub const POLL: Duration = Duration::from_millis(50);
/// How much of Podman's standard error is kept for the log.
pub const STDERR_KEPT: usize = 4096;
/// A log sink. Call it as `(self.log)("a line")`.
pub type Log = Arc<dyn Fn(&str) + Send + Sync>;
pub struct Podman {
runner: Runner,
egress_dir: PathBuf,
log: Log,
next: AtomicU64,
}
impl Podman {
/// `egress_dir` is `Config::egress_dir()`; task 12 uses it.
pub fn new(runner: Runner, egress_dir: PathBuf, log: Log) -> Podman {
todo!("store the four fields; `next` starts at 0")
}
/// Where `http_fetch` calls get their directories. Public, so the field counts as read.
pub fn egress_dir(&self) -> &Path {
todo!()
}
/// Run one short `podman` command (`kill`, `rm -f`) whose result only matters for the log.
pub(crate) fn podman(&self, args: &[&str]) {
todo!("`.status()` with the three standard streams null; log a line if it is not a success")
}
/// Wait for `child` until `limit` has passed since `started`. `Some(status)` if it ended;
/// `None` if it ran too long, after `podman kill <name>`, `podman rm -f <name>`,
/// `child.kill()` and `child.wait()` (step 5).
fn wait(&self, child: &mut Child, name: &str, started: Instant, limit: Duration) -> Option<ExitStatus> {
todo!()
}
}
impl Podman {
/// Steps 3 to 7 for one container: spawn, feed and read it, wait, answer. Task 12 calls it too.
/// Already written: it only joins the helpers below.
fn run_container(
&self,
name: &str,
args: Vec<OsString>,
input: String,
limit: Duration,
) -> Result<RunOutput, RunError> {
let Some(mut child) = self.spawn(args) else {
return Err(RunError::Unavailable(CANNOT_START.to_string()));
};
let started = Instant::now();
let cap = usize::try_from(self.runner.output_cap).unwrap_or(usize::MAX);
let io = Io::start(&mut child, input, cap);
let status = self.wait(&mut child, name, started, limit);
let (out, truncated, err) = io.finish();
self.answer(name, status, out, truncated, &err)
}
/// Step 3: spawn `podman` with `args` and all three standard streams piped. On failure, log
/// `brokerd: cannot start {podman path}: {e}` + "\n" + RUNBOOK and return `None`.
fn spawn(&self, args: Vec<OsString>) -> Option<Child> {
todo!()
}
/// Step 7: the answer, by the table in the task. `status` is `None` when the time limit was
/// passed. `out` is the kept standard output, `err` the kept standard error (for the log only:
/// it never goes into a `RunError`).
fn answer(
&self,
name: &str,
status: Option<ExitStatus>,
out: Vec<u8>,
truncated: bool,
err: &str,
) -> Result<RunOutput, RunError> {
todo!()
}
}
impl Runtime for Podman {
/// Steps 1 and 2, then `run_container`. Already written.
fn run(&self, spec: &RunSpec) -> Result<RunOutput, RunError> {
let n = self.next.fetch_add(1, Ordering::SeqCst);
let name = podman::container_name(spec.session(), spec.call(), n);
let args = podman::tool_args(spec, &self.runner, &name, None);
let input = spec.arguments().canonical_json();
let limit = self.runner.time_limit(spec.tool());
self.run_container(&name, args, input, limit)
}
}
/// Step 4: the three threads that feed and read one container, so no pipe can block another.
struct Io {
writer: Option<JoinHandle<()>>,
stdout: Option<JoinHandle<(Vec<u8>, bool)>>,
stderr: Option<JoinHandle<(Vec<u8>, bool)>>,
}
impl Io {
/// Take `child`'s three pipes (`child.stdin.take()` and so on) and start one thread for each:
/// write `input` to standard input and then drop it; `read_capped` standard output with `cap`;
/// `read_capped` standard error with `STDERR_KEPT`. A pipe that is `None` gets no thread.
fn start(child: &mut Child, input: String, cap: usize) -> Io {
todo!()
}
/// Step 6: join the three threads. A thread that is missing or panicked counts as empty
/// (`join().ok()`, `unwrap_or_default()`). Returns the kept standard output, whether there was
/// more, and the kept standard error decoded with `from_utf8_lossy`.
fn finish(self) -> (Vec<u8>, bool, String) {
todo!()
}
}
/// Everything `from` gives, keeping the first `cap` bytes; `true` if there was more. Reads on past
/// the cap, so the writer is never blocked or broken by a closed pipe. No indexing: `buf.get(..n)`.
fn read_capped(from: impl Read, cap: usize) -> (Vec<u8>, bool) {
todo!()
}
@@ -0,0 +1,428 @@
//! Tests for tool arguments: their shape, and the form of paths, hosts and URLs. Do not edit.
//!
//! Every row of the "Paths" and "Hosts" tables in the M3a spec is a case here.
use brokerd::args::{
ArgsError, MAX_PATH, MAX_URL, ToolArgs, ToolName, host_matches, inside, parse, url_host,
valid_host, valid_host_pattern, valid_path,
};
#[test]
fn the_four_tool_names() {
let names = ["read_file", "write_file", "shell", "http_fetch"];
for (tool, name) in ToolName::ALL.into_iter().zip(names) {
assert_eq!(tool.as_str(), name);
assert_eq!(ToolName::parse(name), Some(tool));
}
for other in [
"",
"echo",
"clock",
"call_tool",
"Read_File",
"read_file ",
"readfile",
] {
assert_eq!(ToolName::parse(other), None, "{other:?}");
}
}
#[test]
fn valid_paths() {
let longest = format!("/{}", "a".repeat(MAX_PATH - 1));
assert_eq!(longest.len(), MAX_PATH);
for path in [
"/",
"/etc",
"/home/kyle/notes/a.md",
"/home/kyle/notes",
"/with space/and\ttab",
"/dots.in.names/..hidden/...",
"/unicode/\u{e9}t\u{e9}",
longest.as_str(),
] {
assert!(valid_path(path), "{path:?} should be valid");
}
}
#[test]
fn invalid_paths() {
let too_long = format!("/{}", "a".repeat(MAX_PATH));
assert_eq!(too_long.len(), MAX_PATH + 1);
for path in [
"",
"notes/a.md",
"./notes",
"~/notes",
"/home/kyle/notes/../.ssh/id",
"/home/kyle//notes/./a.md",
"/home//kyle",
"/home/./kyle",
"/home/kyle/",
"/home/kyle/..",
"/..",
"/.",
"//",
"/nul\0byte",
too_long.as_str(),
] {
assert!(!valid_path(path), "{path:?} should be invalid");
}
}
/// The table in the spec, row by row, for the rows about form and containment.
#[test]
fn inside_is_by_whole_components() {
let grant = "/home/kyle/notes";
assert!(inside(grant, "/home/kyle/notes/a.md"));
assert!(inside(grant, "/home/kyle/notes"));
assert!(inside(grant, "/home/kyle/notes/deep/er/b.md"));
assert!(!inside(grant, "/home/kyle/notes2/a.md"));
assert!(!inside(grant, "/home/kyle/note"));
assert!(!inside(grant, "/home/kyle"));
assert!(!inside(grant, "/"));
assert!(!inside(grant, "/other/home/kyle/notes/a.md"));
// A grant of the root is refused when grants are loaded, but the function is still right.
assert!(inside("/", "/etc/passwd"));
assert!(inside("/", "/"));
}
#[test]
fn valid_hosts_and_patterns() {
let label63 = "a".repeat(63);
let long = format!("{label63}.{label63}.{label63}.{}.com", "b".repeat(57));
assert_eq!(long.len(), 253);
for host in [
"example.com",
"www.example.com",
"a.b.example.com",
"xn--bcher-kva.example",
"1password.com",
"3.example.org",
"a-b.c-d.io",
long.as_str(),
] {
assert!(valid_host(host), "{host:?} should be a valid host");
assert!(
valid_host_pattern(host),
"{host:?} should be a valid pattern"
);
let wild = format!("*.{host}");
assert!(!valid_host(&wild), "{wild:?} is a pattern, not a host");
}
assert!(valid_host_pattern("*.example.com"));
assert!(valid_host_pattern("*.a.b.example.com"));
}
#[test]
fn invalid_hosts_and_patterns() {
let label64 = format!("{}.com", "a".repeat(64));
let too_long = format!("{}.com", ["a".repeat(63).as_str(); 4].join("."));
assert!(too_long.len() > 253);
for host in [
"",
"localhost",
"com",
"Example.com",
"example.COM",
"example.com.",
".example.com",
"example..com",
"-example.com",
"example-.com",
"exa_mple.com",
"example.com:443",
"example.com/path",
"user@example.com",
"exa mple.com",
"[::1]",
"::1",
// Every spelling of an IPv4 address: the last label does not start with a letter.
"127.0.0.1",
"127.1",
"10.0.0.0x1",
"1.2.3.4",
"example.123",
"b\u{fc}cher.example",
label64.as_str(),
too_long.as_str(),
] {
assert!(!valid_host(host), "{host:?} should not be a valid host");
assert!(
!valid_host_pattern(host),
"{host:?} should not be a valid pattern"
);
}
for pattern in [
"*",
"*.",
"*.com",
"*example.com",
"www.*.com",
"*.*.example.com",
"**.example.com",
"*.Example.com",
"*.127.0.0.1",
] {
assert!(!valid_host_pattern(pattern), "{pattern:?}");
}
}
/// The host table in the spec, row by row.
#[test]
fn host_matching() {
assert!(host_matches("example.com", "example.com"));
assert!(!host_matches("example.com", "www.example.com"));
assert!(host_matches("*.example.com", "www.example.com"));
assert!(host_matches("*.example.com", "a.b.example.com"));
assert!(!host_matches("*.example.com", "example.com"));
// A suffix is not enough: the match is by whole labels.
assert!(!host_matches("*.example.com", "badexample.com"));
assert!(!host_matches("*.example.com", "www.example.com.evil.org"));
assert!(!host_matches("example.com", "example.com.evil.org"));
assert!(!host_matches("*.example.com", ".example.com"));
}
#[test]
fn valid_urls_and_their_hosts() {
let base = "https://example.com/";
let longest = format!("{base}{}", "a".repeat(MAX_URL - base.len()));
assert_eq!(longest.len(), MAX_URL);
for (url, host) in [
("https://example.com", "example.com"),
("https://example.com/", "example.com"),
("https://example.com:443", "example.com"),
("https://example.com:443/", "example.com"),
("https://www.example.com/a/b.html", "www.example.com"),
("https://example.com/search?q=a+b&x=%20#frag", "example.com"),
("https://example.com/@user", "example.com"),
("https://example.com/a:8080/b", "example.com"),
("https://example.com/https://other.org/", "example.com"),
("https://example.com/back\\slash", "example.com"),
(longest.as_str(), "example.com"),
] {
assert_eq!(url_host(url), Some(host), "{url}");
}
}
#[test]
fn invalid_urls() {
let base = "https://example.com/";
let too_long = format!("{base}{}", "a".repeat(MAX_URL - base.len() + 1));
assert_eq!(too_long.len(), MAX_URL + 1);
for url in [
"",
"example.com",
"http://example.com/",
"HTTPS://example.com/",
"https:/example.com/",
"https://",
"https:///path",
"ftp://example.com/",
"file:///etc/passwd",
// userinfo
"https://user@example.com/",
"https://user:pw@example.com/",
"https://example.com@evil.org/",
// ports
"https://example.com:8443/",
"https://example.com:80/",
"https://example.com:/",
"https://example.com:443x/",
"https://example.com:4433/",
"https://example.com:443:443/",
// what follows the host must be the end, `:443` or `/`
"https://example.com?q=1",
"https://example.com#frag",
"https://example.com\\@evil.org/",
// hosts that are not host names
"https://localhost/",
"https://127.0.0.1/",
"https://127.1/",
"https://[::1]/",
"https://Example.com/",
"https://example.com./",
"https://b\u{fc}cher.example/",
// the rest must be printable ASCII with no space
"https://example.com/a b",
"https://example.com/a\tb",
"https://example.com/a\nb",
"https://example.com/caf\u{e9}",
"https://example.com/\u{7f}",
" https://example.com/",
"https://example.com/ ",
too_long.as_str(),
] {
assert_eq!(url_host(url), None, "{url:?} should be invalid");
}
}
#[test]
fn each_tool_parses_its_own_arguments() {
assert_eq!(
parse(ToolName::ReadFile, r#"{"path":"/home/kyle/notes/a.md"}"#),
Ok(ToolArgs::ReadFile {
path: "/home/kyle/notes/a.md".to_string()
})
);
assert_eq!(
parse(
ToolName::WriteFile,
r#"{"path":"/home/kyle/notes/a.md","content":"line\n"}"#
),
Ok(ToolArgs::WriteFile {
path: "/home/kyle/notes/a.md".to_string(),
content: "line\n".to_string()
})
);
assert_eq!(
parse(ToolName::Shell, r#"{"command":"ls -l"}"#),
Ok(ToolArgs::Shell {
command: "ls -l".to_string(),
cwd: None
})
);
assert_eq!(
parse(ToolName::Shell, r#"{"command":"ls","cwd":"/home/kyle"}"#),
Ok(ToolArgs::Shell {
command: "ls".to_string(),
cwd: Some("/home/kyle".to_string())
})
);
// A `FetchUrl` cannot be built outside `args`, so the parsed value is read through its getters.
let fetched = parse(
ToolName::HttpFetch,
r#"{"url":"https://www.example.com/a"}"#,
);
let Ok(ToolArgs::HttpFetch(target)) = fetched else {
panic!("{fetched:?}")
};
assert_eq!(target.url(), "https://www.example.com/a");
assert_eq!(target.host(), "www.example.com");
// Field order and white space in the request do not matter.
assert_eq!(
parse(
ToolName::WriteFile,
" { \"content\" : \"x\" , \"path\" : \"/a/b\" } "
),
Ok(ToolArgs::WriteFile {
path: "/a/b".to_string(),
content: "x".to_string()
})
);
// `command` and `content` are not inspected.
assert!(parse(ToolName::Shell, r#"{"command":""}"#).is_ok());
assert!(parse(ToolName::Shell, r#"{"command":"rm -rf / ; ../../x"}"#).is_ok());
assert!(parse(ToolName::WriteFile, r#"{"path":"/a/b","content":""}"#).is_ok());
}
#[test]
fn arguments_of_the_wrong_shape_are_refused() {
let cases: [(ToolName, &str); 17] = [
(ToolName::ReadFile, ""),
(ToolName::ReadFile, "null"),
(ToolName::ReadFile, "[]"),
(ToolName::ReadFile, r#""/etc/hosts""#),
(ToolName::ReadFile, "{}"),
(ToolName::ReadFile, r#"{"path":"/a","mode":"r"}"#),
(ToolName::ReadFile, r#"{"path":"/a","path":"/b"}"#),
(ToolName::ReadFile, r#"{"path":7}"#),
(ToolName::ReadFile, r#"{"path":null}"#),
(ToolName::ReadFile, r#"{"path":"/a"} trailing"#),
(ToolName::WriteFile, r#"{"path":"/a/b"}"#),
(ToolName::WriteFile, r#"{"content":"x"}"#),
(
ToolName::WriteFile,
r#"{"path":"/a/b","content":"x","append":true}"#,
),
(ToolName::Shell, r#"{"cwd":"/a"}"#),
(ToolName::Shell, r#"{"command":"ls","cwd":"/a","env":{}}"#),
(ToolName::Shell, r#"{"command":["ls"]}"#),
(
ToolName::HttpFetch,
r#"{"url":"https://example.com/","method":"POST"}"#,
),
];
for (tool, text) in cases {
match parse(tool, text) {
Err(ArgsError::Shape(_)) => {}
other => panic!("{tool:?} {text:?}: expected a shape error, got {other:?}"),
}
}
// One tool's arguments do not fit another tool.
assert!(parse(ToolName::Shell, r#"{"path":"/a"}"#).is_err());
assert!(parse(ToolName::ReadFile, r#"{"url":"https://example.com/"}"#).is_err());
}
#[test]
fn a_bad_path_or_url_is_refused_in_every_tool_that_takes_one() {
for bad in ["notes/a.md", "/home/kyle/notes/../.ssh/id", "/a//b", "/a/"] {
let quoted = serde_json::to_string(bad).unwrap();
let read = format!(r#"{{"path":{quoted}}}"#);
let write = format!(r#"{{"path":{quoted},"content":"x"}}"#);
let shell = format!(r#"{{"command":"ls","cwd":{quoted}}}"#);
assert_eq!(
parse(ToolName::ReadFile, &read),
Err(ArgsError::Path(bad.to_string()))
);
assert_eq!(
parse(ToolName::WriteFile, &write),
Err(ArgsError::Path(bad.to_string()))
);
assert_eq!(
parse(ToolName::Shell, &shell),
Err(ArgsError::Path(bad.to_string()))
);
}
assert_eq!(
parse(ToolName::HttpFetch, r#"{"url":"http://example.com/"}"#),
Err(ArgsError::Url("http://example.com/".to_string()))
);
// A NUL can only arrive as a JSON escape; it is refused once decoded.
let nul = format!("{{\"path\":\"/a{}u0000b\"}}", '\\');
assert!(matches!(
parse(ToolName::ReadFile, &nul),
Err(ArgsError::Path(_))
));
// `cwd: null` is the same as no `cwd`.
assert_eq!(
parse(ToolName::Shell, r#"{"command":"ls","cwd":null}"#),
Ok(ToolArgs::Shell {
command: "ls".to_string(),
cwd: None
})
);
}
/// What the owner is shown is the parsed value written out again, so two spellings of one path
/// look the same. The escape is built from pieces so that no tool rewrites it on the way here.
#[test]
fn canonical_json_shows_what_was_parsed() {
let escaped_slash = format!("{}u002f", '\\');
let sneaky = format!("{{ \"path\" : \"{escaped_slash}etc{escaped_slash}hosts\" }}");
assert!(sneaky.contains("u002fetc"));
let args = parse(ToolName::ReadFile, &sneaky).unwrap();
assert_eq!(
args,
ToolArgs::ReadFile {
path: "/etc/hosts".to_string()
}
);
assert_eq!(args.canonical_json(), r#"{"path":"/etc/hosts"}"#);
// Fields come out in the spec's order whatever order they came in.
let write = parse(ToolName::WriteFile, r#"{"content":"x\ny","path":"/a/b"}"#).unwrap();
assert_eq!(
write.canonical_json(),
r#"{"path":"/a/b","content":"x\ny"}"#
);
let shell = parse(ToolName::Shell, r#"{"cwd":"/a","command":"ls"}"#).unwrap();
assert_eq!(shell.canonical_json(), r#"{"command":"ls","cwd":"/a"}"#);
// An absent cwd is left out, and the host is never written: it is not an argument.
let bare = parse(ToolName::Shell, r#"{"command":"ls","cwd":null}"#).unwrap();
assert_eq!(bare.canonical_json(), r#"{"command":"ls"}"#);
let fetch = parse(ToolName::HttpFetch, r#"{"url":"https://example.com/a"}"#).unwrap();
assert_eq!(fetch.canonical_json(), r#"{"url":"https://example.com/a"}"#);
assert_eq!(fetch.tool(), ToolName::HttpFetch);
assert_eq!(write.tool(), ToolName::WriteFile);
}
@@ -0,0 +1,140 @@
//! `[runner]` in `brokerd.toml` (M3b spec, section 6). Do not edit.
use std::path::{Path, PathBuf};
use std::time::Duration;
use brokerd::args::ToolName;
use brokerd::config::{Config, ConfigError};
const DIGEST: &str = "localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef";
fn fixture(name: &str) -> PathBuf {
Path::new(env!("CARGO_MANIFEST_DIR"))
.join("tests/fixtures/config")
.join(name)
}
fn invalid(name: &str) -> String {
match Config::load(&fixture(name)) {
Err(ConfigError::Invalid(_, why)) => why,
other => panic!("{name}: expected Invalid, got {other:?}"),
}
}
#[test]
fn without_a_runner_section_there_is_no_runner() {
assert_eq!(Config::load(&fixture("empty.toml")).unwrap().runner, None);
}
#[test]
fn a_runner_with_only_an_image_gets_every_default() {
let r = Config::load(&fixture("runner_minimal.toml"))
.unwrap()
.runner
.unwrap();
assert_eq!(r.podman, PathBuf::from("podman"));
assert_eq!(r.image, DIGEST);
assert_eq!(r.egress_network, "pasta");
assert_eq!(r.output_cap, 262_144);
assert_eq!(r.memory, "512m");
assert_eq!(r.pids, 128);
assert_eq!(
r.time_limit(ToolName::ReadFile),
Duration::from_millis(30_000)
);
assert_eq!(
r.time_limit(ToolName::WriteFile),
Duration::from_millis(30_000)
);
assert_eq!(
r.time_limit(ToolName::Shell),
Duration::from_millis(100_000)
);
assert_eq!(
r.time_limit(ToolName::HttpFetch),
Duration::from_millis(60_000)
);
}
#[test]
fn every_runner_value_can_be_set() {
let r = Config::load(&fixture("runner_full.toml"))
.unwrap()
.runner
.unwrap();
assert_eq!(r.podman, PathBuf::from("/run/current-system/sw/bin/podman"));
assert_eq!(r.egress_network, "slirp4netns");
assert_eq!((r.output_cap, r.memory.as_str(), r.pids), (1000, "1g", 64));
let limits: Vec<Duration> = ToolName::ALL.iter().map(|t| r.time_limit(*t)).collect();
assert_eq!(limits, [1, 2, 3, 4].map(Duration::from_millis));
}
#[test]
fn a_runner_needs_an_image_named_by_digest() {
assert!(matches!(
Config::load(&fixture("runner_no_image.toml")),
Err(ConfigError::Parse(..))
));
assert!(invalid("runner_tag.toml").contains("by digest"));
}
#[test]
fn unknown_runner_keys_are_errors() {
assert!(matches!(
Config::load(&fixture("runner_unknown_key.toml")),
Err(ConfigError::Parse(..))
));
}
#[test]
fn bad_values_are_errors_that_name_them() {
let dir = std::env::temp_dir().join(format!("bx-runner-cfg-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
let cases = [
(
format!("image = \"x@sha256:{}\"", "0".repeat(63)),
"by digest",
),
(
format!("image = \"x@sha256:{}\"", "A".repeat(64)),
"by digest",
),
(
format!("image = \"@sha256:{}\"", "0".repeat(64)),
"by digest",
),
(format!("image = \"{DIGEST}\"\nmemory = \"lots\""), "memory"),
(format!("image = \"{DIGEST}\"\nmemory = \"512\""), "memory"),
(format!("image = \"{DIGEST}\"\nmemory = \"m\""), "memory"),
(
format!("image = \"{DIGEST}\"\noutput_cap = 0"),
"output_cap",
),
(format!("image = \"{DIGEST}\"\npids = 0"), "pids"),
(format!("image = \"{DIGEST}\"\nshell_ms = 0"), "shell_ms"),
(
format!("image = \"{DIGEST}\"\nhttp_fetch_ms = 0"),
"http_fetch_ms",
),
(
format!("image = \"{DIGEST}\"\negress_network = \"\""),
"egress_network",
),
(format!("image = \"{DIGEST}\"\npodman = \"\""), "podman"),
];
for (n, (body, word)) in cases.iter().enumerate() {
let path = dir.join(format!("c{n}.toml"));
std::fs::write(&path, format!("[runner]\n{body}\n")).unwrap();
match Config::load(&path) {
Err(ConfigError::Invalid(_, why)) => assert!(why.contains(word), "{body}: {why}"),
other => panic!("{body}: expected Invalid, got {other:?}"),
}
}
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn the_egress_directory_is_under_home() {
let c = Config::parse("[paths]\nhome = \"/h\"\n").unwrap();
assert_eq!(c.egress_dir(), PathBuf::from("/h/run/egress"));
}
@@ -0,0 +1,271 @@
//! The Podman runtime against a fake `podman`: what it is given, and what each way a container can
//! end becomes (M3b spec, section 6). Every call goes through `runner::run`, as in `brokerd`.
//! Do not edit.
#[path = "support/build.rs"]
mod build;
#[path = "support/fake_podman.rs"]
mod fake_podman;
use std::time::{Duration, Instant};
use brokerd::container::{
CANNOT_START, COULD_NOT_RUN, KILLED, Podman, RUNBOOK, TIMED_OUT, UNEXPECTED,
};
use brokerd::policy::{Outcome, SessionState, decide};
use brokerd::runner::run;
use build::{grant, now, read, request, set};
use fake_podman::{Fake, Lines, serial};
use proto::{DataClass, Mode, ToolRequest, ToolResponse};
fn call(podman: &Podman, req: ToolRequest, grants: Vec<build::Build>) -> ToolResponse {
let decision = match decide(req, &set(grants), SessionState::default(), now()) {
Outcome::Allowed(d) => d,
other => panic!("not allowed: {other:?}"),
};
run(decision, podman)
}
fn notes() -> Vec<build::Build> {
vec![grant("notes", "read_file", Mode::Auto).paths(&["/n"])]
}
fn podman(fake: &Fake, extra: &str, log: &Lines) -> Podman {
Podman::new(fake.runner(extra), fake.dir.join("egress"), log.sink())
}
fn failed(message: &str) -> ToolResponse {
ToolResponse::Failed {
message: message.to_string(),
}
}
#[test]
fn a_tool_that_succeeds_is_a_result_labelled_by_its_grant() {
let _s = serial();
let fake = Fake::new("ok", r#"cat > "$D/stdin"; printf 'the file text'; exit 0"#);
let log = Lines::default();
let got = call(&podman(&fake, "", &log), read("/n/a.md"), notes());
assert_eq!(
got,
ToolResponse::Result {
content: "the file text".to_string(),
class: DataClass::Private,
untrusted: true,
truncated: false,
}
);
assert_eq!(
fake.stdin(),
r#"{"path":"/n/a.md"}"#,
"the arguments go on standard input"
);
}
#[test]
fn podman_is_given_the_tool_argument_list_and_the_first_container_is_numbered_0() {
let _s = serial();
let fake = Fake::new("args", r#"cat > /dev/null; exit 0"#);
let log = Lines::default();
let p = podman(&fake, "", &log);
call(&p, read("/n/a.md"), notes());
call(&p, read("/n/b.md"), notes());
let calls = fake.calls();
assert_eq!(
calls.len(),
2,
"one podman run per call and nothing else: {calls:?}"
);
assert_eq!(calls[0][3], "--name=boxmaker-s1-1-0");
assert_eq!(calls[1][3], "--name=boxmaker-s1-1-1");
// The whole list is `podman::tool_args`, tested as golden files in podman_args.rs.
assert_eq!(calls[0].first().map(String::as_str), Some("run"));
assert_eq!(calls[0].last().map(String::as_str), Some("read_file"));
assert!(calls[0].contains(&"--volume=/n:/n:ro".to_string()));
}
#[test]
fn exit_1_is_the_tools_own_error_and_still_a_result() {
let _s = serial();
let fake = Fake::new(
"e1",
"cat > /dev/null; printf 'read_file: /n/x: no such file'; exit 1",
);
let log = Lines::default();
let got = call(&podman(&fake, "", &log), read("/n/x"), notes());
assert!(
matches!(&got, ToolResponse::Result { content, .. } if content == "read_file: /n/x: no such file"),
"{got:?}"
);
}
#[test]
fn every_other_ending_is_a_fixed_sentence() {
let cases = [
("2", COULD_NOT_RUN),
("125", CANNOT_START),
("126", CANNOT_START),
("127", CANNOT_START),
("137", KILLED),
("3", UNEXPECTED),
("124", UNEXPECTED),
];
for (code, sentence) in cases {
let _s = serial();
let body = format!(
"cat > /dev/null; printf 'secret tool output'; echo 'podman said this' >&2; exit {code}"
);
let fake = Fake::new("codes", &body);
let log = Lines::default();
let got = call(&podman(&fake, "", &log), read("/n/a"), notes());
assert_eq!(got, failed(sentence), "exit {code}");
}
}
#[test]
fn a_podman_failure_is_logged_with_the_runbook_pointer_and_its_stderr() {
let _s = serial();
let fake = Fake::new(
"125",
"cat > /dev/null; echo 'Error: image not known' >&2; exit 125",
);
let log = Lines::default();
call(&podman(&fake, "", &log), read("/n/a"), notes());
let text = log.all();
assert!(text.contains("Error: image not known"), "{text}");
assert!(text.contains(RUNBOOK), "{text}");
}
#[test]
fn a_podman_that_cannot_be_started_is_unavailable_and_logged() {
let _s = serial();
let fake = Fake::new("missing", "exit 0");
let mut runner = fake.runner("");
runner.podman = fake.dir.join("no-such-podman");
let log = Lines::default();
let p = Podman::new(runner, fake.dir.join("egress"), log.sink());
assert_eq!(call(&p, read("/n/a"), notes()), failed(CANNOT_START));
assert!(log.all().contains(RUNBOOK), "{}", log.all());
}
#[test]
fn output_past_the_cap_is_cut_and_marked() {
let _s = serial();
let fake = Fake::new(
"cap",
"cat > /dev/null; head -c 1000 /dev/zero | tr '\\0' x; exit 0",
);
let log = Lines::default();
let got = call(
&podman(&fake, "output_cap = 100", &log),
read("/n/a"),
notes(),
);
assert!(
matches!(&got, ToolResponse::Result { content, truncated: true, .. } if *content == "x".repeat(100)),
"{got:?}"
);
let exact = Fake::new(
"cap-exact",
"cat > /dev/null; head -c 100 /dev/zero | tr '\\0' x; exit 0",
);
let got = call(
&podman(&exact, "output_cap = 100", &log),
read("/n/a"),
notes(),
);
assert!(
matches!(
&got,
ToolResponse::Result {
truncated: false,
..
}
),
"{got:?}"
);
}
#[test]
fn output_that_is_not_utf8_is_replaced() {
let _s = serial();
let fake = Fake::new("utf8", "cat > /dev/null; printf 'a\\377b'; exit 0");
let log = Lines::default();
let got = call(&podman(&fake, "", &log), read("/n/a"), notes());
assert!(
matches!(&got, ToolResponse::Result { content, .. } if content == "a\u{fffd}b"),
"{got:?}"
);
}
#[test]
fn a_tool_past_its_time_limit_is_killed_removed_and_failed() {
let _s = serial();
// `exec`, so killing the process kills the sleep and nothing holds the pipes open.
let fake = Fake::new("slow", "cat > /dev/null; exec sleep 30");
let log = Lines::default();
let started = Instant::now();
let got = call(
&podman(&fake, "read_file_ms = 300", &log),
read("/n/a"),
notes(),
);
let took = started.elapsed();
assert_eq!(got, failed(TIMED_OUT));
assert!(took >= Duration::from_millis(300), "{took:?}");
assert!(took < Duration::from_secs(5), "{took:?}");
let calls = fake.calls();
assert_eq!(calls.len(), 3, "{calls:?}");
assert_eq!(calls[1], ["kill", "boxmaker-s1-1-0"]);
assert_eq!(calls[2], ["rm", "-f", "boxmaker-s1-1-0"]);
}
#[test]
fn a_large_argument_is_written_whole_while_the_tool_reads_it() {
let _s = serial();
let fake = Fake::new("big", r#"cat > "$D/stdin"; printf done; exit 0"#);
let log = Lines::default();
let content = "y".repeat(900_000);
let req = request(
"write_file",
&format!(r#"{{"path":"/w/big.txt","content":"{content}"}}"#),
);
let got = call(
&podman(&fake, "", &log),
req,
vec![grant("w", "write_file", Mode::Auto).paths(&["/w"])],
);
assert!(
matches!(&got, ToolResponse::Result { content, .. } if content == "done"),
"{got:?}"
);
assert_eq!(
fake.stdin().len(),
content.len() + r#"{"path":"/w/big.txt","content":""}"#.len()
);
}
#[test]
fn a_tool_that_never_reads_its_input_still_ends() {
let _s = serial();
let fake = Fake::new("noread", "printf ignored; exit 0");
let log = Lines::default();
let req = request(
"write_file",
&format!(
r#"{{"path":"/w/big.txt","content":"{}"}}"#,
"z".repeat(900_000)
),
);
let started = Instant::now();
let got = call(
&podman(&fake, "write_file_ms = 5000", &log),
req,
vec![grant("w", "write_file", Mode::Auto).paths(&["/w"])],
);
assert!(
matches!(&got, ToolResponse::Result { content, .. } if content == "ignored"),
"{got:?}"
);
assert!(started.elapsed() < Duration::from_secs(4));
}
@@ -0,0 +1,213 @@
//! `http_fetch` through the Podman runtime: the proxy is started first, its socket awaited, the tool
//! run with the directory mounted, and the proxy and directory removed afterwards on every path
//! (M3b spec, section 6, "http_fetch"). Against a fake `podman`. Do not edit.
#[path = "support/build.rs"]
mod build;
#[path = "support/fake_podman.rs"]
mod fake_podman;
use std::os::unix::fs::PermissionsExt;
use std::time::{Duration, Instant};
use brokerd::container::{CANNOT_START, Podman, RUNBOOK, TIMED_OUT};
use brokerd::policy::{Outcome, SessionState, decide};
use brokerd::runner::run;
use build::{fetch, grant, now, set};
use fake_podman::{Fake, Lines, path, serial};
use proto::{Mode, ToolResponse};
/// For `run -d` (the proxy): make the socket file in the mounted directory, as the proxy does.
const PROXY_OK: &str = r#"for a in "$@"; do case "$a" in --volume=*:/run/egress:rw) v=${a#--volume=}; v=${v%:/run/egress:rw};; esac; done"#;
fn body(proxy: &str, tool: &str) -> String {
format!("{PROXY_OK}\nif [ \"$2\" = -d ]; then\n{proxy}\nfi\n{tool}")
}
fn call(podman: &Podman) -> ToolResponse {
let grants = set(vec![
grant("web", "http_fetch", Mode::Auto).hosts(&["example.com", "*.example.org"]),
]);
let decision = match decide(
fetch("https://example.com/a"),
&grants,
SessionState::default(),
now(),
) {
Outcome::Allowed(d) => d,
other => panic!("not allowed: {other:?}"),
};
run(decision, podman)
}
fn failed(message: &str) -> ToolResponse {
ToolResponse::Failed {
message: message.to_string(),
}
}
#[test]
fn the_proxy_starts_first_the_tool_gets_its_socket_and_both_are_cleaned_up() {
let _s = serial();
let fake = Fake::new(
"eg-ok",
&body(
r#": > "$v/egress.sock"; exit 0"#,
r#"cat > "$D/stdin"; printf 'body\n[http 200]'; exit 0"#,
),
);
let log = Lines::default();
let egress = fake.dir.join("egress");
let got = call(&Podman::new(fake.runner(""), egress.clone(), log.sink()));
assert!(
matches!(&got, ToolResponse::Result { content, .. } if content == "body\n[http 200]"),
"{got:?} {}",
log.all()
);
let calls = fake.calls();
assert_eq!(calls.len(), 3, "{calls:?}");
let dir = egress.join("boxmaker-s1-1-0");
// 1. The proxy, detached, with the call's hosts.
assert_eq!(&calls[0][..2], ["run", "-d"]);
assert!(calls[0].contains(&"--name=boxmaker-s1-1-0-egress".to_string()));
assert!(calls[0].contains(&format!("--volume={}:/run/egress:rw", path(&dir))));
assert_eq!(calls[0].last().unwrap(), "example.com,*.example.org");
// 2. The tool, with the same directory and no network.
assert_eq!(&calls[1][..3], ["run", "--rm", "-i"]);
assert!(calls[1].contains(&format!("--volume={}:/run/egress:rw", path(&dir))));
assert!(calls[1].contains(&"--network=none".to_string()));
assert_eq!(fake.stdin(), r#"{"url":"https://example.com/a"}"#);
// 3. The proxy removed, and its directory with it.
assert_eq!(calls[2], ["rm", "-f", "boxmaker-s1-1-0-egress"]);
assert!(!dir.exists(), "the call's directory is removed");
let mode = std::fs::metadata(&egress).unwrap().permissions().mode() & 0o777;
assert_eq!(mode, 0o700);
}
#[test]
fn a_proxy_that_podman_cannot_start_means_no_tool_and_nothing_left() {
let _s = serial();
let fake = Fake::new(
"eg-fail",
&body(
"echo 'Error: network pasta not found' >&2; exit 125",
"cat > /dev/null; exit 0",
),
);
let log = Lines::default();
let egress = fake.dir.join("egress");
let got = call(&Podman::new(fake.runner(""), egress.clone(), log.sink()));
assert_eq!(got, failed(CANNOT_START));
let calls = fake.calls();
assert!(
calls
.iter()
.all(|c| c.get(2).map(String::as_str) != Some("-i")),
"no tool ran: {calls:?}"
);
assert_eq!(
calls.last().unwrap(),
&["rm", "-f", "boxmaker-s1-1-0-egress"]
);
assert!(!egress.join("boxmaker-s1-1-0").exists());
assert!(
log.all().contains("network pasta not found"),
"{}",
log.all()
);
assert!(log.all().contains(RUNBOOK));
}
#[test]
fn a_proxy_that_makes_no_socket_in_time_means_no_tool_and_nothing_left() {
let _s = serial();
let fake = Fake::new("eg-nosock", &body("exit 0", "cat > /dev/null; exit 0"));
let log = Lines::default();
let egress = fake.dir.join("egress");
let podman = Podman::new(fake.runner(""), egress.clone(), log.sink())
.with_egress_wait(Duration::from_millis(200));
let started = Instant::now();
assert_eq!(call(&podman), failed(CANNOT_START));
assert!(started.elapsed() < Duration::from_secs(3));
let calls = fake.calls();
assert_eq!(calls.len(), 2, "{calls:?}");
assert_eq!(calls[1], ["rm", "-f", "boxmaker-s1-1-0-egress"]);
assert!(!egress.join("boxmaker-s1-1-0").exists());
assert!(log.all().contains(RUNBOOK));
}
#[test]
fn a_tool_that_runs_too_long_still_leaves_nothing_behind() {
let _s = serial();
let fake = Fake::new(
"eg-slow",
&body(
r#": > "$v/egress.sock"; exit 0"#,
"cat > /dev/null; exec sleep 30",
),
);
let log = Lines::default();
let egress = fake.dir.join("egress");
let got = call(&Podman::new(
fake.runner("http_fetch_ms = 300"),
egress.clone(),
log.sink(),
));
assert_eq!(got, failed(TIMED_OUT));
let calls = fake.calls();
let tail: Vec<Vec<String>> = calls[2..].to_vec();
assert_eq!(
tail,
[
vec!["kill", "boxmaker-s1-1-0"],
vec!["rm", "-f", "boxmaker-s1-1-0"],
vec!["rm", "-f", "boxmaker-s1-1-0-egress"],
]
.map(|c| c.into_iter().map(String::from).collect::<Vec<_>>())
);
assert!(!egress.join("boxmaker-s1-1-0").exists());
}
#[test]
fn a_tool_that_podman_cannot_start_still_leaves_nothing_behind() {
let _s = serial();
let fake = Fake::new(
"eg-toolfail",
&body(
r#": > "$v/egress.sock"; exit 0"#,
"cat > /dev/null; exit 125",
),
);
let log = Lines::default();
let egress = fake.dir.join("egress");
assert_eq!(
call(&Podman::new(fake.runner(""), egress.clone(), log.sink())),
failed(CANNOT_START)
);
assert_eq!(
fake.calls().last().unwrap(),
&["rm", "-f", "boxmaker-s1-1-0-egress"]
);
assert!(!egress.join("boxmaker-s1-1-0").exists());
}
#[test]
fn a_directory_left_by_a_crash_is_replaced() {
let _s = serial();
let fake = Fake::new(
"eg-stale",
&body(
r#"[ -e "$v/old.sock" ] && exit 9; : > "$v/egress.sock"; exit 0"#,
"cat > /dev/null; printf ok; exit 0",
),
);
let log = Lines::default();
let egress = fake.dir.join("egress");
std::fs::create_dir_all(egress.join("boxmaker-s1-1-0")).unwrap();
std::fs::write(egress.join("boxmaker-s1-1-0/old.sock"), "").unwrap();
let got = call(&Podman::new(fake.runner(""), egress.clone(), log.sink()));
assert!(
matches!(&got, ToolResponse::Result { content, .. } if content == "ok"),
"{got:?}"
);
}
@@ -0,0 +1,86 @@
//! After a container ends, its output is collected within a grace period, never waited on for
//! ever: something outside the container that still holds a pipe must not hold `brokerd` (M3b
//! review finding 4). Against a fake `podman` whose shell leaves a background `sleep` holding the
//! pipes, which real Podman does not do. Do not edit.
#[path = "support/build.rs"]
mod build;
#[path = "support/fake_podman.rs"]
mod fake_podman;
use std::time::{Duration, Instant};
use brokerd::container::{OUTPUT_OPEN, Podman, TIMED_OUT};
use brokerd::pipes::GRACE;
use brokerd::policy::{Outcome, SessionState, decide};
use brokerd::runner::run;
use build::{grant, now, read, set};
use fake_podman::{Fake, Lines, serial};
use proto::{Mode, ToolResponse};
fn call(fake: &Fake, extra: &str, log: &Lines) -> (ToolResponse, Duration) {
let podman = Podman::new(fake.runner(extra), fake.dir.join("egress"), log.sink());
let grants = set(vec![grant("notes", "read_file", Mode::Auto).paths(&["/n"])]);
let decision = match decide(read("/n/a"), &grants, SessionState::default(), now()) {
Outcome::Allowed(d) => d,
other => panic!("not allowed: {other:?}"),
};
let started = Instant::now();
let got = run(decision, &podman);
(got, started.elapsed())
}
#[test]
fn the_grace_period_is_two_seconds() {
assert_eq!(GRACE, Duration::from_secs(2));
}
#[test]
fn output_held_open_after_the_container_ended_is_abandoned_after_the_grace_period() {
let _s = serial();
// The shell exits at once; the background sleep keeps standard output and error open.
let fake = Fake::new("grace-open", "cat > /dev/null; printf ok; sleep 6 & exit 0");
let log = Lines::default();
let (got, took) = call(&fake, "", &log);
assert_eq!(
got,
ToolResponse::Failed {
message: OUTPUT_OPEN.to_string()
}
);
assert!(took >= GRACE, "{took:?}");
assert!(took < GRACE + Duration::from_secs(2), "{took:?}");
assert!(log.all().contains("abandoned"), "{}", log.all());
}
#[test]
fn a_tool_past_its_limit_is_answered_within_the_grace_period_even_if_its_pipes_stay_open() {
let _s = serial();
// No `exec`: killing the shell leaves the sleep holding the pipes.
let fake = Fake::new("grace-slow", "cat > /dev/null; sleep 6");
let log = Lines::default();
let (got, took) = call(&fake, "read_file_ms = 300", &log);
assert_eq!(
got,
ToolResponse::Failed {
message: TIMED_OUT.to_string()
}
);
assert!(
took < Duration::from_millis(300) + GRACE + Duration::from_secs(2),
"{took:?}"
);
}
#[test]
fn a_tool_that_ends_normally_is_not_slowed_by_the_grace_period() {
let _s = serial();
let fake = Fake::new("grace-ok", "cat > /dev/null; printf done; exit 0");
let log = Lines::default();
let (got, took) = call(&fake, "", &log);
assert!(
matches!(&got, ToolResponse::Result { content, .. } if content == "done"),
"{got:?}"
);
assert!(took < Duration::from_secs(1), "{took:?}");
}
@@ -0,0 +1,128 @@
//! What a tool or Podman writes on standard error reaches `brokerd`'s log escaped, one entry per
//! event: it cannot start a line of its own or forge a runbook pointer (M3b review finding 5).
//! Do not edit.
#[path = "support/build.rs"]
mod build;
#[path = "support/fake_podman.rs"]
mod fake_podman;
use brokerd::container::{CANNOT_START, COULD_NOT_RUN, Podman, UNEXPECTED};
use brokerd::policy::{Outcome, SessionState, decide};
use brokerd::runner::run;
use build::{fetch, grant, now, read, set};
use fake_podman::{Fake, Lines, serial};
use proto::{Mode, ToolRequest, ToolResponse};
const FORGED: &str = "real line\nbrokerd: forged\nsee docs/runbook.md#grants-invalid";
fn call(fake: &Fake, req: ToolRequest, grants: Vec<build::Build>, log: &Lines) -> ToolResponse {
let podman = Podman::new(fake.runner(""), fake.dir.join("egress"), log.sink());
let decision = match decide(req, &set(grants), SessionState::default(), now()) {
Outcome::Allowed(d) => d,
other => panic!("not allowed: {other:?}"),
};
run(decision, &podman)
}
/// No entry holds the forged text as lines of its own; the one that carries it has it escaped.
fn escaped(log: &Lines) {
let entries = log.0.lock().unwrap().clone();
for entry in &entries {
assert!(!entry.contains("\nbrokerd: forged"), "raw: {entry:?}");
assert!(
!entry.contains("\nsee docs/runbook.md#grants-invalid"),
"raw: {entry:?}"
);
}
assert!(
entries
.iter()
.any(|e| e.contains(r"real line\nbrokerd: forged")),
"the error is still logged, escaped: {entries:?}"
);
}
fn notes() -> Vec<build::Build> {
vec![grant("notes", "read_file", Mode::Auto).paths(&["/n"])]
}
#[test]
fn a_tool_that_could_not_run() {
let _s = serial();
let fake = Fake::new(
"log-2",
&format!("cat > /dev/null; printf '{FORGED}' >&2; exit 2"),
);
let log = Lines::default();
assert_eq!(
call(&fake, read("/n/a"), notes(), &log),
ToolResponse::Failed {
message: COULD_NOT_RUN.to_string()
}
);
escaped(&log);
}
#[test]
fn a_container_podman_could_not_start_keeps_its_one_real_pointer() {
let _s = serial();
let fake = Fake::new(
"log-125",
&format!("cat > /dev/null; printf '{FORGED}' >&2; exit 125"),
);
let log = Lines::default();
assert_eq!(
call(&fake, read("/n/a"), notes(), &log),
ToolResponse::Failed {
message: CANNOT_START.to_string()
}
);
escaped(&log);
let entries = log.0.lock().unwrap().clone();
assert!(
entries
.iter()
.any(|e| e.ends_with("\nsee docs/runbook.md#runner-unavailable")),
"{entries:?}"
);
}
#[test]
fn an_unexpected_ending() {
let _s = serial();
let fake = Fake::new(
"log-3",
&format!("cat > /dev/null; printf '{FORGED}' >&2; exit 3"),
);
let log = Lines::default();
assert_eq!(
call(&fake, read("/n/a"), notes(), &log),
ToolResponse::Failed {
message: UNEXPECTED.to_string()
}
);
escaped(&log);
}
#[test]
fn a_proxy_podman_could_not_start() {
let _s = serial();
let body =
format!("if [ \"$2\" = -d ]; then printf '{FORGED}' >&2; exit 125; fi; cat > /dev/null");
let fake = Fake::new("log-egress", &body);
let log = Lines::default();
let got = call(
&fake,
fetch("https://example.com/"),
vec![grant("web", "http_fetch", Mode::Auto).hosts(&["example.com"])],
&log,
);
assert_eq!(
got,
ToolResponse::Failed {
message: CANNOT_START.to_string()
}
);
escaped(&log);
}
@@ -0,0 +1,11 @@
[runner]
podman = "/run/current-system/sw/bin/podman"
image = "localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
egress_network = "slirp4netns"
output_cap = 1000
memory = "1g"
pids = 64
read_file_ms = 1
write_file_ms = 2
shell_ms = 3
http_fetch_ms = 4
@@ -0,0 +1,2 @@
[runner]
image = "localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
@@ -0,0 +1,2 @@
[runner]
podman = "podman"
@@ -0,0 +1,2 @@
[runner]
image = "localhost/boxmaker-tools:latest"
@@ -0,0 +1,3 @@
[runner]
image = "localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
network = "host"
@@ -0,0 +1,21 @@
run
-d
--rm
--name=boxmaker-s1-1-7-egress
--label=boxmaker=egress
--network=pasta
--pull=never
--read-only
--cap-drop=all
--security-opt=no-new-privileges
--userns=keep-id
--pids-limit=64
--memory=128m
--volume=/h/run/egress/boxmaker-s1-1-7:/run/egress:rw
localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
/bin/toolkit
egress-proxy
--socket
/run/egress/egress.sock
--allow
example.com,*.example.org
@@ -0,0 +1,18 @@
run
--rm
-i
--name=boxmaker-s1-1-7
--label=boxmaker=tool
--network=none
--pull=never
--read-only
--cap-drop=all
--security-opt=no-new-privileges
--userns=keep-id
--pids-limit=128
--memory=512m
--tmpfs=/tmp:rw,size=64m,mode=1777
--volume=/h/run/egress/boxmaker-s1-1-7:/run/egress:rw
localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
/bin/toolkit
http_fetch
@@ -0,0 +1,18 @@
run
--rm
-i
--name=boxmaker-s1-1-7
--label=boxmaker=tool
--network=none
--pull=never
--read-only
--cap-drop=all
--security-opt=no-new-privileges
--userns=keep-id
--pids-limit=128
--memory=512m
--tmpfs=/tmp:rw,size=64m,mode=1777
--volume=/home/kyle/notes:/home/kyle/notes:ro
localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
/bin/toolkit
read_file
@@ -0,0 +1,19 @@
run
--rm
-i
--name=boxmaker-s1-1-7
--label=boxmaker=tool
--network=none
--pull=never
--read-only
--cap-drop=all
--security-opt=no-new-privileges
--userns=keep-id
--pids-limit=128
--memory=512m
--tmpfs=/tmp:rw,size=64m,mode=1777
--volume=/srv/a:/srv/a:rw
--volume=/srv/b:/srv/b:rw
localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
/bin/toolkit
shell
@@ -0,0 +1,17 @@
run
--rm
-i
--name=boxmaker-s1-1-7
--label=boxmaker=tool
--network=none
--pull=never
--read-only
--cap-drop=all
--security-opt=no-new-privileges
--userns=keep-id
--pids-limit=128
--memory=512m
--tmpfs=/tmp:rw,size=64m,mode=1777
localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
/bin/toolkit
shell
@@ -0,0 +1,18 @@
run
--rm
-i
--name=boxmaker-s1-1-7
--label=boxmaker=tool
--network=none
--pull=never
--read-only
--cap-drop=all
--security-opt=no-new-privileges
--userns=keep-id
--pids-limit=128
--memory=512m
--tmpfs=/tmp:rw,size=64m,mode=1777
--volume=/home/kyle/out:/home/kyle/out:rw
localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
/bin/toolkit
write_file

Some files were not shown because too many files have changed in this diff Show More