M5b plan: task 13, instance lock in brokerd, memoryd and gatewayd
Each locks <home>/run/<role>.lock in main (gatewayd after its secret, so nothing is made before the secret is read). brokerd's old second-instance test in serve.rs expected the audit lock's message, so serve.rs has a new version; task 01 now copies serve.rs-task01. Checks: given tests fail at ref 12 as the task says (2 each in brokerd, memoryd, gatewayd; serve.rs passes); the task's quoted edits applied to ref 12 reproduce ref 13's tree exactly; replay gate ok with 1118 tests (1113 - 1 + 6), /tmp unchanged; 14 mutations, all caught; the six instance test files stressed 25 runs under 16 busy loops and 20 under 32, no failure. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
@@ -53,7 +53,8 @@ This is the complete list. Each one is covered by an edit below or by a given fi
|
||||
- **new versions** of: `crates/bxcore/tests/wire.rs`, `crates/bxcore/tests/strict.rs`;
|
||||
`crates/brokerd/tests/admin.rs`, `broker.rs`, `broker_pending.rs`, `courier.rs`, `ledger.rs`,
|
||||
`serve.rs`; `crates/loopd/tests/broker_port.rs`, `tools.rs`, `turn_broker.rs`. Your
|
||||
`broker.rs` is the given `broker.rs-task01`: the given `broker.rs` is task 09's.
|
||||
`broker.rs` is the given `broker.rs-task01`: the given `broker.rs` is task 09's. Your
|
||||
`serve.rs` is the given `serve.rs-task01`: the given `serve.rs` is task 13's.
|
||||
- Modify: `crates/bxcore/src/wire.rs`, `crates/brokerd/src/args.rs`,
|
||||
`crates/brokerd/src/policy/mod.rs`, `crates/brokerd/src/ledger/mod.rs`,
|
||||
`crates/brokerd/src/broker.rs`, `crates/loopd/src/tools.rs`, `crates/loopd/src/turn.rs`,
|
||||
@@ -498,8 +499,8 @@ The event keeps only the reason: the owner's texts do not change.
|
||||
crates/brokerd/tests/
|
||||
cp docs/plans/M5b/files/crates/brokerd/tests/ledger.rs \
|
||||
crates/brokerd/tests/
|
||||
cp docs/plans/M5b/files/crates/brokerd/tests/serve.rs \
|
||||
crates/brokerd/tests/
|
||||
cp docs/plans/M5b/files/crates/brokerd/tests/serve.rs-task01 \
|
||||
crates/brokerd/tests/serve.rs
|
||||
cp docs/plans/M5b/files/crates/loopd/tests/broker_port.rs \
|
||||
crates/loopd/tests/
|
||||
cp docs/plans/M5b/files/crates/loopd/tests/tools.rs \
|
||||
|
||||
@@ -0,0 +1,280 @@
|
||||
# M5b task 13: `brokerd`, `memoryd` and `gatewayd` take the instance lock too
|
||||
|
||||
**Branch:** `m5b` (run `git switch m5b`; `git status --short` must be empty, otherwise stop)
|
||||
**Commit subject:** `brokerd, memoryd, gatewayd: a lock file keeps each to one instance`
|
||||
|
||||
## Goal
|
||||
|
||||
Task 12 wrote `bxcore::instance`, the lock that keeps a daemon to one instance (M5b spec,
|
||||
section 9a), and made `loopd` and `inferproxy` take it. This task makes the other three daemons
|
||||
take it the same way: before they touch their sockets, each locks `<home>/run/<role>.lock` and
|
||||
holds the lock for the life of the process. A second copy against the same home exits 1 with one
|
||||
line and touches nothing else:
|
||||
|
||||
```text
|
||||
brokerd: another brokerd is running (it holds /var/lib/boxmaker/run/brokerd.lock); see docs/runbook.md#already-running
|
||||
memoryd: another memoryd is running (it holds /var/lib/boxmaker/run/memoryd.lock); see docs/runbook.md#already-running
|
||||
gatewayd: another gatewayd is running (it holds /var/lib/boxmaker/run/gatewayd.lock); see docs/runbook.md#already-running
|
||||
```
|
||||
|
||||
Any other error on the lock file stops the daemon too, with the line
|
||||
`<role>: cannot take the lock <lock file>: <error>; see docs/runbook.md#already-running`. Both
|
||||
lines come from `LockError::line`; the runbook entry `already-running` is there since task 12, and
|
||||
covers all five daemons.
|
||||
|
||||
## Files
|
||||
|
||||
- Copy, from `docs/plans/M5b/files/` (the same path under it):
|
||||
- new tests: `crates/brokerd/tests/instance.rs`, `crates/memoryd/tests/instance.rs`,
|
||||
`crates/gatewayd/tests/instance.rs`
|
||||
- a **new version** of `crates/brokerd/tests/serve.rs` (it replaces the file in the tree). Do
|
||||
not copy `serve.rs-task01`: it is the older version.
|
||||
- Modify: `crates/brokerd/src/main.rs` (B1, B2), `crates/memoryd/src/main.rs` (D1, D2),
|
||||
`crates/gatewayd/src/main.rs` (G1 to G3), `docs/implementer-log.md`.
|
||||
- Do not touch: `crates/bxcore` (`instance` is done), any other source file, any `Cargo.toml`,
|
||||
any other test, `docs/runbook.md`.
|
||||
|
||||
## Facts you need
|
||||
|
||||
- **What `bxcore::instance` gives** (task 12; do not change it):
|
||||
|
||||
| Name | What it is |
|
||||
|---|---|
|
||||
| `instance::lock` | `pub fn lock(path: &Path) -> Result<InstanceLock, LockError>`: creates the file (0600) and a missing directory (0700), and tries the lock once |
|
||||
| `InstanceLock` | the held lock; the lock is released when it is dropped or the process ends |
|
||||
| `LockError` | `Held` (another instance holds it) or `Io(std::io::Error)` |
|
||||
| `LockError::line` | `e.line("brokerd", &lock_path) -> String`: the whole line to print, pointer included |
|
||||
|
||||
- **Keep the lock alive.** Each daemon binds the `InstanceLock` to a name, `_instance`, in `main`
|
||||
(or `serve` for `gatewayd`), the function that serves until the process ends. `let _ = …`, or a
|
||||
`match` arm `Ok(_) => {}`, would drop it, and the lock with it, at once. The edits show the form.
|
||||
- **Where each daemon takes it**, and why there:
|
||||
- `brokerd` (`main` in `crates/brokerd/src/main.rs`): right after the config loads, before
|
||||
`serve::start`, which opens the audit log and binds both sockets. The home is
|
||||
`cfg.paths.home`.
|
||||
- `memoryd` (`main` in `crates/memoryd/src/main.rs`): right after the config loads, before
|
||||
`serve::start`, which creates `memory/` and the index and binds three sockets. The home is
|
||||
`config.paths.home`.
|
||||
- `gatewayd` (`serve` in `crates/gatewayd/src/main.rs`): after the secret is loaded, before the
|
||||
state directory is made and before `run`, which binds `gateway.sock` and then connects to
|
||||
Mattermost. Not earlier: the test `a_missing_secret_stops_at_start` in
|
||||
`crates/gatewayd/tests/main.rs` checks that nothing under the home is made before the secret
|
||||
is read, and the lock creates `<home>/run`. The home is `config.paths.home`.
|
||||
- **`brokerd`'s audit lock stays.** `brokerd` has held a lock on `<home>/audit/.lock` since M3a,
|
||||
and a second `brokerd` used to stop there, with `brokerd is already running` and the pointer
|
||||
`brokerd-already-running`. It stays: it guards the log itself, and its own test
|
||||
(`a_second_writer_is_refused` in `crates/brokerd/tests/audit.rs`) still passes. But the instance
|
||||
lock now comes first, so a second `brokerd` stops with the new line instead. That is why
|
||||
`serve.rs` has a new version: its test `a_second_brokerd_on_the_same_home_refuses_to_start`
|
||||
expected the old message, and it is gone; `instance.rs` checks the second `brokerd` now. Nothing
|
||||
else in `serve.rs` changes.
|
||||
- **The given tests start each daemon as a process**, as `brokerd`'s `serve.rs` does. The
|
||||
`gatewayd` one runs it against the fake Mattermost of `tests/support/fake_mm.rs`, with its token
|
||||
in the environment (`GW_TEST_TOKEN`), and checks that it answers on `gateway.sock` by asking it
|
||||
to post an approval, as `brokerd` would. Every wait has a deadline and every process is killed by
|
||||
a guard. Before this task, the `memoryd` and `gatewayd` tests that expect a second instance to be
|
||||
refused wait out their deadline (the second copy starts and serves), so each of those files takes
|
||||
about 10 s to fail. That is this, not a hang.
|
||||
- The calls the edits make: `instance::lock` and `LockError::line` from the table above;
|
||||
`Path::join` for the lock file's path; `eprintln!` and `ExitCode::from(1)` as the lines around
|
||||
them already do.
|
||||
|
||||
## The edits, exactly
|
||||
|
||||
Nothing else in these files changes. Each quoted "before" block occurs exactly once in its file.
|
||||
|
||||
### `crates/brokerd/src/main.rs`
|
||||
|
||||
**B1.** The line
|
||||
|
||||
```rust
|
||||
use brokerd::serve::{self, ServeError};
|
||||
```
|
||||
|
||||
becomes
|
||||
|
||||
```rust
|
||||
use brokerd::serve::{self, ServeError};
|
||||
use bxcore::instance;
|
||||
```
|
||||
|
||||
**B2.** In `main`, the line
|
||||
|
||||
```rust
|
||||
// Keep the socket paths for the serving notice, before `cfg` is moved into `start`.
|
||||
```
|
||||
|
||||
becomes
|
||||
|
||||
```rust
|
||||
// One brokerd per home (M5b spec, section 9a). The lock comes before anything is made, the
|
||||
// audit log's own lock included, so a second copy never takes the sockets of the one that
|
||||
// runs. Held until the process ends.
|
||||
let lock_path = cfg.paths.home.join("run/brokerd.lock");
|
||||
let _instance = match instance::lock(&lock_path) {
|
||||
Ok(held) => held,
|
||||
Err(e) => {
|
||||
eprintln!("{}", e.line("brokerd", &lock_path));
|
||||
return ExitCode::from(1);
|
||||
}
|
||||
};
|
||||
|
||||
// Keep the socket paths for the serving notice, before `cfg` is moved into `start`.
|
||||
```
|
||||
|
||||
### `crates/memoryd/src/main.rs`
|
||||
|
||||
**D1.** The line
|
||||
|
||||
```rust
|
||||
use memoryd::config::Config;
|
||||
```
|
||||
|
||||
becomes
|
||||
|
||||
```rust
|
||||
use bxcore::instance;
|
||||
use memoryd::config::Config;
|
||||
```
|
||||
|
||||
**D2.** In `main`, the line
|
||||
|
||||
```rust
|
||||
let log: serve::Log = Arc::new(|line: &str| eprintln!("{line}"));
|
||||
```
|
||||
|
||||
becomes
|
||||
|
||||
```rust
|
||||
// One memoryd per home (M5b spec, section 9a). The lock comes before anything under the home
|
||||
// is made or bound, so a second copy never takes the sockets of the one that runs. Held until
|
||||
// the process ends.
|
||||
let lock_path = config.paths.home.join("run/memoryd.lock");
|
||||
let _instance = match instance::lock(&lock_path) {
|
||||
Ok(held) => held,
|
||||
Err(e) => {
|
||||
eprintln!("{}", e.line("memoryd", &lock_path));
|
||||
return ExitCode::from(1);
|
||||
}
|
||||
};
|
||||
let log: serve::Log = Arc::new(|line: &str| eprintln!("{line}"));
|
||||
```
|
||||
|
||||
### `crates/gatewayd/src/main.rs`
|
||||
|
||||
**G1.** The line
|
||||
|
||||
```rust
|
||||
use bxcore::secrets;
|
||||
```
|
||||
|
||||
becomes
|
||||
|
||||
```rust
|
||||
use bxcore::{instance, secrets};
|
||||
```
|
||||
|
||||
**G2.** In the comment at the top of `serve`, the line
|
||||
|
||||
```rust
|
||||
// Then `if let Some(warning) = &loaded.warning { eprintln!("{warning}"); }`.
|
||||
```
|
||||
|
||||
becomes
|
||||
|
||||
```rust
|
||||
// Then `if let Some(warning) = &loaded.warning { eprintln!("{warning}"); }`.
|
||||
// 3b. The instance lock, `<home>/run/gatewayd.lock` (M5b spec, section 9a): after the secret,
|
||||
// which is read before anything is made, and before the directory and the socket.
|
||||
```
|
||||
|
||||
**G3.** In `serve`, the lines
|
||||
|
||||
```rust
|
||||
if let Some(warning) = &loaded.warning {
|
||||
eprintln!("{warning}");
|
||||
}
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
if let Some(warning) = &loaded.warning {
|
||||
eprintln!("{warning}");
|
||||
}
|
||||
let lock_path = config.paths.home.join("run/gatewayd.lock");
|
||||
let _instance = match instance::lock(&lock_path) {
|
||||
Ok(held) => held,
|
||||
Err(e) => {
|
||||
eprintln!("{}", e.line("gatewayd", &lock_path));
|
||||
return ExitCode::from(1);
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1. Copy the given tests.** From the repository root, four commands:
|
||||
|
||||
```sh
|
||||
cp docs/plans/M5b/files/crates/brokerd/tests/instance.rs crates/brokerd/tests/
|
||||
cp docs/plans/M5b/files/crates/brokerd/tests/serve.rs crates/brokerd/tests/
|
||||
cp docs/plans/M5b/files/crates/memoryd/tests/instance.rs crates/memoryd/tests/
|
||||
cp docs/plans/M5b/files/crates/gatewayd/tests/instance.rs crates/gatewayd/tests/
|
||||
```
|
||||
|
||||
- [ ] **2. See them fail.** Three commands, each with a shell timeout of 600000 ms. Each file
|
||||
compiles; `serve.rs` passes.
|
||||
- `cargo test -p brokerd --no-fail-fast --test instance --test serve` (`--no-fail-fast`, or
|
||||
cargo stops at the first failing file): `serve` 8 passed. `instance`: 2 failed.
|
||||
`a_second_brokerd_on_the_same_home_is_refused_and_the_first_keeps_serving` fails on `the
|
||||
lock's line, and nothing else` (the second stopped at the audit lock, with `brokerd is already
|
||||
running`); `after_the_first_is_killed_a_new_brokerd_removes_the_stale_sockets_and_serves`
|
||||
fails on `the lock file, mode 0600`, `left: None`, `right: Some(384)`.
|
||||
- `cargo test -p memoryd --test instance`: 2 failed (about 10 s).
|
||||
`a_second_memoryd_on_the_same_home_is_refused_and_the_first_keeps_serving` fails on `the
|
||||
second memoryd: …` with `left: None`, `right: Some(1)` (it started and served);
|
||||
`after_the_first_is_killed_a_new_memoryd_removes_the_stale_sockets_and_serves` on `the lock
|
||||
file, mode 0600`, `left: None`.
|
||||
- `cargo test -p gatewayd --test instance`: 2 failed (about 10 s), the same two shapes:
|
||||
`the second gatewayd: …` with `left: None`, and `the lock file, mode 0600` with `left: None`.
|
||||
- [ ] **3. `brokerd`.** Make edits B1 and B2. `cargo check -p brokerd`: no error.
|
||||
`cargo test -p brokerd --test instance --test serve`: `instance` 2 passed, `serve` 8 passed.
|
||||
- [ ] **4. `memoryd`.** Make edits D1 and D2. `cargo check -p memoryd`: no error.
|
||||
`cargo test -p memoryd --test instance`: 2 passed.
|
||||
- [ ] **5. `gatewayd`.** Make edits G1 to G3. `cargo check -p gatewayd`: no error.
|
||||
`cargo test -p gatewayd --test instance --test main`: `instance` 2 passed, `main` 5 passed.
|
||||
- [ ] **6. Check.** `grep -rn 'instance::lock(' crates/*/src` prints exactly five lines, one in
|
||||
the `main.rs` of each of `brokerd`, `gatewayd`, `inferproxy`, `loopd` and `memoryd`.
|
||||
`grep -rn 'runbook.md#already-running' crates/*/src` still prints exactly one line, in
|
||||
`crates/bxcore/src/instance.rs`. `git diff --stat -- crates/bxcore '*Cargo.toml'` prints nothing.
|
||||
- [ ] **7. Run the gate.** `cargo fmt --all`, then `make gate`, with a shell timeout of 600000 ms.
|
||||
Expected last line: `gate: ok`, with 1118 tests: 1113 before this task, minus 1 in the new
|
||||
`crates/brokerd/tests/serve.rs` (its old second-instance test), plus 2 in each of the three new
|
||||
`instance.rs` files.
|
||||
- [ ] **8. Log and commit.** `git status --short` lists `crates/brokerd/src/main.rs`,
|
||||
`crates/memoryd/src/main.rs`, `crates/gatewayd/src/main.rs`, `crates/brokerd/tests/serve.rs`
|
||||
and your log as modified, and the three `instance.rs` test files as new. Stage with
|
||||
`git add crates/brokerd crates/memoryd crates/gatewayd docs/implementer-log.md Cargo.lock`,
|
||||
then commit as `AGENTS.md` says. `Cargo.lock` does not change in this task; the `git add` is
|
||||
harmless.
|
||||
|
||||
## Done when
|
||||
|
||||
- `make gate` prints `gate: ok` with 1118 tests.
|
||||
- `git diff HEAD~1 --stat` lists only the three `main.rs` files, `crates/brokerd/tests/serve.rs`,
|
||||
the three new `instance.rs` test files and `docs/implementer-log.md`.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- A "before" block of B1, B2, D1, D2 or G1 to G3 is not in its file exactly as shown, or is there
|
||||
more than once.
|
||||
- Step 2 shows a different result: other failing tests, other messages, or other counts.
|
||||
- `cargo check` reports an error after an edit made as shown. Quote it and stop.
|
||||
- After step 3, 4 or 5 a test fails although the edits are as shown. Do not change a test or
|
||||
`bxcore`: stop, and give the failing tests' names and their messages.
|
||||
- The fix seems to need a change anywhere else: `bxcore`, another file of a daemon, the test
|
||||
support files, a `Cargo.toml`, the runbook. It does not; stop and say why you think so.
|
||||
- A test outside these files fails in the gate: check once whether it fails without your change,
|
||||
then stop and report both results.
|
||||
- The gate passes with a count other than 1118.
|
||||
@@ -0,0 +1,218 @@
|
||||
//! One `brokerd` per home (M5b spec, section 9a): a second `brokerd serve` against the same home is
|
||||
//! refused with the lock's line, and the first keeps serving; once the first is killed, a new one
|
||||
//! removes the stale sockets and serves. Every wait has a deadline, and every process is killed
|
||||
//! and every path removed by a guard (tip T44). Do not edit.
|
||||
|
||||
#[path = "support/tmp.rs"]
|
||||
mod tmp;
|
||||
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
use std::os::unix::net::UnixStream;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::process::{Child, Command, Stdio};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use bxcore::{Empty, Envelope, Message, PROTOCOL_VERSION, read_frame, write_frame};
|
||||
use tmp::TempDir;
|
||||
|
||||
const POINTER: &str = "see docs/runbook.md#already-running";
|
||||
|
||||
struct Home {
|
||||
dir: TempDir,
|
||||
config: PathBuf,
|
||||
}
|
||||
|
||||
impl Home {
|
||||
fn new(tag: &str) -> Home {
|
||||
let dir = TempDir::new(tag);
|
||||
std::fs::create_dir_all(dir.path().join("grants")).unwrap();
|
||||
let text = format!(
|
||||
"[paths]\nhome = \"{home}\"\ngrants = \"{home}/grants\"\n",
|
||||
home = dir.path().display()
|
||||
);
|
||||
let config = dir.write("brokerd.toml", &text);
|
||||
Home { dir, config }
|
||||
}
|
||||
|
||||
fn path(&self, relative: &str) -> PathBuf {
|
||||
self.dir.path().join(relative)
|
||||
}
|
||||
|
||||
fn command(&self) -> Command {
|
||||
let mut command = Command::new(env!("CARGO_BIN_EXE_brokerd"));
|
||||
command
|
||||
.args(["serve", "--config"])
|
||||
.arg(&self.config)
|
||||
.stdin(Stdio::null())
|
||||
.stdout(Stdio::null());
|
||||
command
|
||||
}
|
||||
|
||||
/// `brokerd serve` with its stderr in `<home>/<name>.err`, for the messages of a failing test.
|
||||
fn start(&self, name: &str) -> Daemon {
|
||||
let err = std::fs::File::create(self.path(&format!("{name}.err"))).unwrap();
|
||||
Daemon(self.command().stderr(Stdio::from(err)).spawn().unwrap())
|
||||
}
|
||||
|
||||
fn stderr_of(&self, name: &str) -> String {
|
||||
std::fs::read_to_string(self.path(&format!("{name}.err"))).unwrap_or_default()
|
||||
}
|
||||
|
||||
/// Runs `brokerd serve` to its end, at most 10 s (then it is killed), and gives its exit code
|
||||
/// (`None` if it had to be killed) and its stderr.
|
||||
fn run_to_end(&self) -> (Option<i32>, String) {
|
||||
let mut child = Daemon(self.command().stderr(Stdio::piped()).spawn().unwrap());
|
||||
let deadline = Instant::now() + Duration::from_secs(10);
|
||||
let mut code = None;
|
||||
while Instant::now() < deadline {
|
||||
if let Ok(Some(status)) = child.0.try_wait() {
|
||||
code = status.code();
|
||||
break;
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(20));
|
||||
}
|
||||
let _ = child.0.kill();
|
||||
let _ = child.0.wait();
|
||||
let mut stderr = String::new();
|
||||
if let Some(mut pipe) = child.0.stderr.take() {
|
||||
let _ = std::io::Read::read_to_string(&mut pipe, &mut stderr);
|
||||
}
|
||||
(code, stderr)
|
||||
}
|
||||
|
||||
fn tools(&self) -> PathBuf {
|
||||
self.path("run/loop-broker/broker.sock")
|
||||
}
|
||||
|
||||
fn admin(&self) -> PathBuf {
|
||||
self.path("run/owner-broker/admin.sock")
|
||||
}
|
||||
}
|
||||
|
||||
/// A running `brokerd serve`, killed and waited for when dropped. Declared after the `Home`, so it
|
||||
/// drops first.
|
||||
struct Daemon(Child);
|
||||
|
||||
impl Drop for Daemon {
|
||||
fn drop(&mut self) {
|
||||
let _ = self.0.kill();
|
||||
let _ = self.0.wait();
|
||||
}
|
||||
}
|
||||
|
||||
impl Daemon {
|
||||
fn running(&mut self) -> bool {
|
||||
matches!(self.0.try_wait(), Ok(None))
|
||||
}
|
||||
}
|
||||
|
||||
/// True when `brokerd` answers on its owner's socket: `approvals` gets the list of approvals. Any
|
||||
/// other answer, or none within 5 s, is false.
|
||||
fn answers(admin: &Path) -> bool {
|
||||
let Ok(mut stream) = UnixStream::connect(admin) else {
|
||||
return false;
|
||||
};
|
||||
let limit = Some(Duration::from_secs(5));
|
||||
if stream.set_read_timeout(limit).is_err() || stream.set_write_timeout(limit).is_err() {
|
||||
return false;
|
||||
}
|
||||
let ask = Envelope {
|
||||
v: PROTOCOL_VERSION,
|
||||
id: 1,
|
||||
r#final: true,
|
||||
msg: Message::Approvals(Empty {}),
|
||||
};
|
||||
if write_frame(&mut stream, &ask).is_err() {
|
||||
return false;
|
||||
}
|
||||
matches!(
|
||||
read_frame(&mut stream),
|
||||
Ok(Envelope {
|
||||
msg: Message::ApprovalList(_),
|
||||
..
|
||||
})
|
||||
)
|
||||
}
|
||||
|
||||
/// Waits up to 10 s for `brokerd` to answer on `admin` and to listen on `tools`.
|
||||
fn wait_until_it_answers(home: &Home) -> bool {
|
||||
let deadline = Instant::now() + Duration::from_secs(10);
|
||||
while Instant::now() < deadline {
|
||||
if answers(&home.admin()) && UnixStream::connect(home.tools()).is_ok() {
|
||||
return true;
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(20));
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
fn inode(path: &Path) -> u64 {
|
||||
use std::os::unix::fs::MetadataExt;
|
||||
std::fs::symlink_metadata(path)
|
||||
.map(|m| m.ino())
|
||||
.unwrap_or(0)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_second_brokerd_on_the_same_home_is_refused_and_the_first_keeps_serving() {
|
||||
let home = Home::new("inst-twice");
|
||||
let mut first = home.start("first");
|
||||
assert!(
|
||||
wait_until_it_answers(&home),
|
||||
"the first brokerd never answered; stderr: {}",
|
||||
home.stderr_of("first")
|
||||
);
|
||||
let before = (inode(&home.tools()), inode(&home.admin()));
|
||||
|
||||
let (code, stderr) = home.run_to_end();
|
||||
let line = format!(
|
||||
"brokerd: another brokerd is running (it holds {}); {POINTER}",
|
||||
home.path("run/brokerd.lock").display()
|
||||
);
|
||||
assert_eq!(code, Some(1), "the second brokerd: {stderr}");
|
||||
assert_eq!(
|
||||
stderr,
|
||||
format!("{line}\n"),
|
||||
"the lock's line, and nothing else"
|
||||
);
|
||||
|
||||
assert!(first.running(), "the first brokerd is still running");
|
||||
assert_eq!(
|
||||
(inode(&home.tools()), inode(&home.admin())),
|
||||
before,
|
||||
"the first's sockets are still there"
|
||||
);
|
||||
assert!(
|
||||
answers(&home.admin()),
|
||||
"the first brokerd still answers; its stderr: {}",
|
||||
home.stderr_of("first")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn after_the_first_is_killed_a_new_brokerd_removes_the_stale_sockets_and_serves() {
|
||||
let home = Home::new("inst-restart");
|
||||
let mut first = home.start("first");
|
||||
assert!(
|
||||
wait_until_it_answers(&home),
|
||||
"the first brokerd never answered; stderr: {}",
|
||||
home.stderr_of("first")
|
||||
);
|
||||
let _ = first.0.kill();
|
||||
let _ = first.0.wait();
|
||||
assert!(
|
||||
home.tools().exists() && home.admin().exists(),
|
||||
"a killed brokerd leaves its sockets"
|
||||
);
|
||||
assert!(!answers(&home.admin()), "nobody serves the stale socket");
|
||||
|
||||
let _next = home.start("next");
|
||||
assert!(
|
||||
wait_until_it_answers(&home),
|
||||
"the new brokerd never answered; stderr: {}",
|
||||
home.stderr_of("next")
|
||||
);
|
||||
let lock = home.path("run/brokerd.lock");
|
||||
let mode = std::fs::metadata(&lock).map(|m| m.permissions().mode() & 0o777);
|
||||
assert_eq!(mode.ok(), Some(0o600), "the lock file, mode 0600");
|
||||
}
|
||||
@@ -216,24 +216,6 @@ fn a_stale_socket_is_replaced() {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_second_brokerd_on_the_same_home_refuses_to_start() {
|
||||
let home = Home::new("serve-twice", 900_000);
|
||||
let _running = home.serve();
|
||||
let second = home.run(&[]);
|
||||
assert_eq!(second.status.code(), Some(1));
|
||||
let text = stderr(&second);
|
||||
assert!(text.contains("brokerd is already running"), "{text}");
|
||||
assert!(
|
||||
text.trim_end()
|
||||
.ends_with("see docs/runbook.md#brokerd-already-running"),
|
||||
"{text}"
|
||||
);
|
||||
// The first still has its sockets.
|
||||
let frames = exchange(&home.admin(), 1, Message::Approvals(Empty {}));
|
||||
assert!(matches!(frames[0].msg, Message::ApprovalList(_)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn it_answers_each_socket_and_refuses_the_other_kinds() {
|
||||
let home = Home::new("serve-kinds", 900_000);
|
||||
|
||||
@@ -0,0 +1,411 @@
|
||||
//! `brokerd serve` as a process: its startup, both sockets, and the expiry thread. Do not edit.
|
||||
|
||||
#[path = "support/tmp.rs"]
|
||||
mod tmp;
|
||||
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
use std::os::unix::net::{UnixListener, UnixStream};
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::process::{Child, Command, Output, Stdio};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use brokerd::audit::RECOVERED_NOTICE;
|
||||
use brokerd::runner::REFUSING;
|
||||
use bxcore::{
|
||||
AuditEvent, CallId, DenyReason, Empty, Envelope, ErrorCode, Message, PROTOCOL_VERSION,
|
||||
ResultStatus, SessionId, ToolRequest, ToolResponse,
|
||||
};
|
||||
use tmp::TempDir;
|
||||
|
||||
struct Home {
|
||||
dir: TempDir,
|
||||
config: PathBuf,
|
||||
}
|
||||
|
||||
impl Home {
|
||||
fn new(tag: &str, ttl_ms: u64) -> Home {
|
||||
let dir = TempDir::new(tag);
|
||||
std::fs::create_dir_all(dir.path().join("grants")).unwrap();
|
||||
let text = format!(
|
||||
"[paths]\nhome = \"{home}\"\ngrants = \"{home}/grants\"\n[approvals]\nttl_ms = {ttl_ms}\n",
|
||||
home = dir.path().display()
|
||||
);
|
||||
let config = dir.write("brokerd.toml", &text);
|
||||
Home { dir, config }
|
||||
}
|
||||
|
||||
fn path(&self, relative: &str) -> PathBuf {
|
||||
self.dir.path().join(relative)
|
||||
}
|
||||
|
||||
fn tools(&self) -> PathBuf {
|
||||
self.path("run/loop-broker/broker.sock")
|
||||
}
|
||||
|
||||
fn admin(&self) -> PathBuf {
|
||||
self.path("run/owner-broker/admin.sock")
|
||||
}
|
||||
|
||||
fn grant(&self, id: &str, mode: &str) {
|
||||
let text = format!(
|
||||
"tool = \"read_file\"\nmode = \"{mode}\"\nmax_taint = \"secret\"\n\
|
||||
result_class = \"private\"\nuntrusted = false\n[constraints]\npaths = [\"/n\"]\n"
|
||||
);
|
||||
std::fs::write(self.path(&format!("grants/{id}.toml")), text).unwrap();
|
||||
}
|
||||
|
||||
fn command(&self, extra: &[&str]) -> Command {
|
||||
let mut command = Command::new(env!("CARGO_BIN_EXE_brokerd"));
|
||||
command
|
||||
.args(["serve", "--config"])
|
||||
.arg(&self.config)
|
||||
.args(extra);
|
||||
command
|
||||
}
|
||||
|
||||
/// Starts `brokerd serve` and waits until both sockets answer.
|
||||
fn serve(&self) -> Running {
|
||||
let child = self
|
||||
.command(&[])
|
||||
.stdout(Stdio::null())
|
||||
.stderr(Stdio::piped())
|
||||
.spawn()
|
||||
.unwrap();
|
||||
let running = Running(Some(child));
|
||||
let until = Instant::now() + Duration::from_secs(10);
|
||||
while UnixStream::connect(self.tools()).is_err()
|
||||
|| UnixStream::connect(self.admin()).is_err()
|
||||
{
|
||||
assert!(Instant::now() < until, "brokerd never listened");
|
||||
std::thread::sleep(Duration::from_millis(20));
|
||||
}
|
||||
running
|
||||
}
|
||||
|
||||
/// Runs `brokerd serve` expecting it to exit by itself.
|
||||
fn run(&self, extra: &[&str]) -> Output {
|
||||
self.command(extra).output().unwrap()
|
||||
}
|
||||
|
||||
fn events(&self) -> Vec<AuditEvent> {
|
||||
let dir = self.path("audit");
|
||||
let mut names: Vec<String> = std::fs::read_dir(&dir)
|
||||
.unwrap()
|
||||
.map(|e| e.unwrap().file_name().into_string().unwrap())
|
||||
.filter(|n| n.ends_with(".jsonl"))
|
||||
.collect();
|
||||
names.sort();
|
||||
names
|
||||
.iter()
|
||||
.flat_map(|n| {
|
||||
let text = std::fs::read_to_string(dir.join(n)).unwrap();
|
||||
text.lines()
|
||||
.map(|l| {
|
||||
serde_json::from_str::<bxcore::AuditRecord>(l)
|
||||
.unwrap()
|
||||
.event
|
||||
})
|
||||
.collect::<Vec<_>>()
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
}
|
||||
|
||||
/// Kills the daemon when dropped; `stop` returns what it printed.
|
||||
struct Running(Option<Child>);
|
||||
|
||||
impl Running {
|
||||
fn stop(mut self) -> String {
|
||||
let mut child = self.0.take().unwrap();
|
||||
child.kill().unwrap();
|
||||
let output = child.wait_with_output().unwrap();
|
||||
String::from_utf8_lossy(&output.stderr).to_string()
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for Running {
|
||||
fn drop(&mut self) {
|
||||
if let Some(child) = &mut self.0 {
|
||||
let _ = child.kill();
|
||||
let _ = child.wait();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn mode(path: &Path) -> u32 {
|
||||
std::fs::metadata(path).unwrap().permissions().mode() & 0o777
|
||||
}
|
||||
|
||||
fn exchange(socket: &Path, id: u64, msg: Message) -> Vec<Envelope> {
|
||||
let mut stream = UnixStream::connect(socket).unwrap();
|
||||
stream
|
||||
.set_read_timeout(Some(Duration::from_secs(10)))
|
||||
.unwrap();
|
||||
let env = Envelope {
|
||||
v: PROTOCOL_VERSION,
|
||||
id,
|
||||
r#final: true,
|
||||
msg,
|
||||
};
|
||||
bxcore::write_frame(&mut stream, &env).unwrap();
|
||||
let mut frames = Vec::new();
|
||||
loop {
|
||||
let env = bxcore::read_frame(&mut stream).unwrap();
|
||||
let last = env.r#final;
|
||||
frames.push(env);
|
||||
if last {
|
||||
return frames;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn read_notes(call: u64) -> Message {
|
||||
Message::ToolRequest(ToolRequest {
|
||||
session: SessionId::new("s1").unwrap(),
|
||||
call: CallId(call),
|
||||
tool: "read_file".to_string(),
|
||||
arguments: r#"{"path":"/n/a"}"#.to_string(),
|
||||
})
|
||||
}
|
||||
|
||||
fn last_response(frames: &[Envelope]) -> &ToolResponse {
|
||||
match &frames.last().unwrap().msg {
|
||||
Message::ToolResponse(r) => r,
|
||||
other => panic!("{other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
fn stderr(output: &Output) -> String {
|
||||
String::from_utf8_lossy(&output.stderr).to_string()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn it_makes_its_directories_0700_and_its_sockets_0600() {
|
||||
let home = Home::new("serve-modes", 900_000);
|
||||
// One directory found too open, one made.
|
||||
std::fs::create_dir_all(home.path("run/owner-broker")).unwrap();
|
||||
std::fs::set_permissions(
|
||||
home.path("run/owner-broker"),
|
||||
std::fs::Permissions::from_mode(0o755),
|
||||
)
|
||||
.unwrap();
|
||||
let running = home.serve();
|
||||
assert_eq!(mode(&home.path("run/loop-broker")), 0o700);
|
||||
assert_eq!(mode(&home.path("run/owner-broker")), 0o700);
|
||||
assert_eq!(mode(&home.tools()), 0o600);
|
||||
assert_eq!(mode(&home.admin()), 0o600);
|
||||
assert_eq!(mode(&home.path("audit")), 0o700);
|
||||
let printed = running.stop();
|
||||
assert!(printed.contains("serving"), "{printed}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_stale_socket_is_replaced() {
|
||||
let home = Home::new("serve-stale", 900_000);
|
||||
std::fs::create_dir_all(home.path("run/loop-broker")).unwrap();
|
||||
drop(UnixListener::bind(home.tools()).unwrap());
|
||||
assert!(home.tools().exists(), "the stale socket file is there");
|
||||
let _running = home.serve();
|
||||
let frames = exchange(&home.tools(), 3, read_notes(3));
|
||||
assert_eq!(
|
||||
last_response(&frames),
|
||||
&ToolResponse::Denied {
|
||||
reason: DenyReason::NoGrant,
|
||||
detail: None,
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_second_brokerd_on_the_same_home_refuses_to_start() {
|
||||
let home = Home::new("serve-twice", 900_000);
|
||||
let _running = home.serve();
|
||||
let second = home.run(&[]);
|
||||
assert_eq!(second.status.code(), Some(1));
|
||||
let text = stderr(&second);
|
||||
assert!(text.contains("brokerd is already running"), "{text}");
|
||||
assert!(
|
||||
text.trim_end()
|
||||
.ends_with("see docs/runbook.md#brokerd-already-running"),
|
||||
"{text}"
|
||||
);
|
||||
// The first still has its sockets.
|
||||
let frames = exchange(&home.admin(), 1, Message::Approvals(Empty {}));
|
||||
assert!(matches!(frames[0].msg, Message::ApprovalList(_)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn it_answers_each_socket_and_refuses_the_other_kinds() {
|
||||
let home = Home::new("serve-kinds", 900_000);
|
||||
home.grant("notes", "auto");
|
||||
let running = home.serve();
|
||||
let frames = exchange(&home.tools(), 7, read_notes(7));
|
||||
assert_eq!(frames[0].id, 7);
|
||||
// The production runtime runs nothing.
|
||||
assert_eq!(
|
||||
last_response(&frames),
|
||||
&ToolResponse::Failed {
|
||||
message: REFUSING.to_string()
|
||||
}
|
||||
);
|
||||
let wrong = exchange(&home.tools(), 8, Message::Approvals(Empty {}));
|
||||
assert!(matches!(&wrong[0].msg, Message::Error(e) if e.code == ErrorCode::Forbidden));
|
||||
let wrong = exchange(&home.admin(), 9, read_notes(9));
|
||||
assert!(matches!(&wrong[0].msg, Message::Error(e) if e.code == ErrorCode::Forbidden));
|
||||
let printed = running.stop();
|
||||
assert!(
|
||||
printed.contains("approvals on broker.sock\nsee docs/runbook.md#socket-forbidden"),
|
||||
"{printed}"
|
||||
);
|
||||
assert!(
|
||||
printed.contains("tool_request on admin.sock\nsee docs/runbook.md#socket-forbidden"),
|
||||
"{printed}"
|
||||
);
|
||||
assert!(matches!(
|
||||
home.events().as_slice(),
|
||||
[
|
||||
AuditEvent::Decision { .. },
|
||||
AuditEvent::Result {
|
||||
status: ResultStatus::Failed,
|
||||
..
|
||||
}
|
||||
]
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_approval_nobody_answers_expires() {
|
||||
let home = Home::new("serve-expire", 100);
|
||||
home.grant("notes", "ask");
|
||||
let _running = home.serve();
|
||||
let started = Instant::now();
|
||||
let frames = exchange(&home.tools(), 2, read_notes(2));
|
||||
assert_eq!(frames.len(), 2, "{frames:?}");
|
||||
assert!(matches!(
|
||||
&frames[0].msg,
|
||||
Message::ToolResponse(ToolResponse::PendingApproval { approval: 0, .. })
|
||||
));
|
||||
assert_eq!(
|
||||
last_response(&frames),
|
||||
&ToolResponse::Denied {
|
||||
reason: DenyReason::ApprovalExpired,
|
||||
detail: None,
|
||||
}
|
||||
);
|
||||
// The expiry thread looks every second.
|
||||
assert!(
|
||||
started.elapsed() < Duration::from_secs(5),
|
||||
"{:?}",
|
||||
started.elapsed()
|
||||
);
|
||||
}
|
||||
|
||||
fn copy_case(home: &Home, case: &str, only: &[&str]) {
|
||||
let from = format!(
|
||||
"{}/../bxcore/tests/fixtures/audit/{case}",
|
||||
env!("CARGO_MANIFEST_DIR")
|
||||
);
|
||||
std::fs::create_dir_all(home.path("audit")).unwrap();
|
||||
for name in only {
|
||||
std::fs::copy(
|
||||
format!("{from}/{name}"),
|
||||
home.path(&format!("audit/{name}")),
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
}
|
||||
|
||||
fn snapshot(dir: &Path) -> Vec<(String, Vec<u8>)> {
|
||||
let mut all: Vec<(String, Vec<u8>)> = std::fs::read_dir(dir)
|
||||
.unwrap()
|
||||
.map(|e| e.unwrap())
|
||||
.filter(|e| e.file_name().to_string_lossy().ends_with(".jsonl"))
|
||||
.map(|e| {
|
||||
(
|
||||
e.file_name().into_string().unwrap(),
|
||||
std::fs::read(e.path()).unwrap(),
|
||||
)
|
||||
})
|
||||
.collect();
|
||||
all.sort();
|
||||
all
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_broken_chain_stops_it_before_any_socket_and_nothing_is_written() {
|
||||
let home = Home::new("serve-broken", 900_000);
|
||||
copy_case(&home, "changed-byte", &["2026-09-17.jsonl"]);
|
||||
let before = snapshot(&home.path("audit"));
|
||||
let output = home.run(&[]);
|
||||
assert_eq!(output.status.code(), Some(1));
|
||||
let text = stderr(&output);
|
||||
assert!(text.contains("2026-09-17.jsonl:4: "), "{text}");
|
||||
assert!(
|
||||
text.trim_end()
|
||||
.ends_with("see docs/runbook.md#audit-chain-broken"),
|
||||
"{text}"
|
||||
);
|
||||
assert_eq!(snapshot(&home.path("audit")), before);
|
||||
assert!(!home.tools().exists() && !home.admin().exists());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_torn_tail_is_recovered_and_it_serves() {
|
||||
let home = Home::new("serve-torn", 900_000);
|
||||
copy_case(
|
||||
&home,
|
||||
"torn-tail",
|
||||
&["2026-09-17.jsonl", "2026-09-18.jsonl"],
|
||||
);
|
||||
let running = home.serve();
|
||||
let printed = running.stop();
|
||||
assert!(printed.contains(RECOVERED_NOTICE), "{printed}");
|
||||
assert!(RECOVERED_NOTICE.ends_with("see docs/runbook.md#audit-recovered"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn accept_break_with_nothing_to_accept_exits_2() {
|
||||
let home = Home::new("serve-nothing", 900_000);
|
||||
let output = home.run(&["--accept-break"]);
|
||||
assert_eq!(output.status.code(), Some(2));
|
||||
assert!(
|
||||
stderr(&output).contains("nothing to accept"),
|
||||
"{}",
|
||||
stderr(&output)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bad_arguments_and_bad_configs_do_not_start() {
|
||||
let brokerd = env!("CARGO_BIN_EXE_brokerd");
|
||||
for args in [
|
||||
&[][..],
|
||||
&["serve"][..],
|
||||
&["serve", "--config"][..],
|
||||
&["serve", "--config", "a", "--config", "b"][..],
|
||||
&["serve", "--config", "a", "--loud"][..],
|
||||
&["run", "--config", "a"][..],
|
||||
] {
|
||||
let output = Command::new(brokerd).args(args).output().unwrap();
|
||||
assert_eq!(output.status.code(), Some(2), "{args:?}");
|
||||
assert!(
|
||||
stderr(&output).starts_with("usage: brokerd serve"),
|
||||
"{args:?}"
|
||||
);
|
||||
}
|
||||
let home = Home::new("serve-config", 900_000);
|
||||
std::fs::write(&home.config, "[paths]\nhoem = \"/x\"\n").unwrap();
|
||||
let output = home.run(&[]);
|
||||
assert_eq!(output.status.code(), Some(1));
|
||||
assert!(
|
||||
stderr(&output).contains("brokerd.toml"),
|
||||
"{}",
|
||||
stderr(&output)
|
||||
);
|
||||
std::fs::remove_file(&home.config).unwrap();
|
||||
assert_eq!(home.run(&[]).status.code(), Some(1));
|
||||
assert!(
|
||||
!home.path("audit").exists(),
|
||||
"nothing made before the config is read"
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,218 @@
|
||||
//! One `gatewayd` per home (M5b spec, section 9a): a second `gatewayd serve` against the same home
|
||||
//! is refused with the lock's line, and the first keeps serving `gateway.sock`; once the first is
|
||||
//! killed, a new one removes the stale socket and serves. Each runs against the fake Mattermost.
|
||||
//! Every wait has a deadline, and every process is killed and every path removed by a guard (tip
|
||||
//! T44). Do not edit.
|
||||
|
||||
#[path = "support/brokerd_client.rs"]
|
||||
mod brokerd_client;
|
||||
#[path = "support/fake_mm.rs"]
|
||||
mod fake_mm;
|
||||
#[path = "support/tmp.rs"]
|
||||
mod tmp;
|
||||
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::process::{Child, Command, Stdio};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use brokerd_client::{ask, pending};
|
||||
use fake_mm::{FakeMm, KYLE};
|
||||
use tmp::TempDir;
|
||||
|
||||
const POINTER: &str = "see docs/runbook.md#already-running";
|
||||
const TOKEN: &str = "tok-instance-test";
|
||||
const WAIT: Duration = Duration::from_secs(10);
|
||||
|
||||
struct Home {
|
||||
dir: TempDir,
|
||||
config: PathBuf,
|
||||
}
|
||||
|
||||
impl Home {
|
||||
fn new(tag: &str, mm: &FakeMm) -> Home {
|
||||
let dir = TempDir::new(tag);
|
||||
let text = format!(
|
||||
"[mattermost]\nurl = \"{}\"\n[secrets.mattermost_token]\nenv = \"GW_TEST_TOKEN\"\n\
|
||||
[allow]\nusers = [\"{KYLE}\"]\n[paths]\nhome = \"{}\"\n",
|
||||
mm.url(),
|
||||
dir.path().display()
|
||||
);
|
||||
let config = dir.write("gatewayd.toml", &text);
|
||||
Home { dir, config }
|
||||
}
|
||||
|
||||
fn path(&self, relative: &str) -> PathBuf {
|
||||
self.dir.path().join(relative)
|
||||
}
|
||||
|
||||
fn socket(&self) -> PathBuf {
|
||||
self.path("run/gateway/gateway.sock")
|
||||
}
|
||||
|
||||
fn command(&self) -> Command {
|
||||
let mut command = Command::new(env!("CARGO_BIN_EXE_gatewayd"));
|
||||
command
|
||||
.args(["serve", "--config"])
|
||||
.arg(&self.config)
|
||||
.env_remove("CREDENTIALS_DIRECTORY")
|
||||
.env("GW_TEST_TOKEN", TOKEN)
|
||||
.stdin(Stdio::null())
|
||||
.stdout(Stdio::null());
|
||||
command
|
||||
}
|
||||
|
||||
/// `gatewayd serve` with its stderr in `<home>/<name>.err`, for the messages of a failing test.
|
||||
fn start(&self, name: &str) -> Daemon {
|
||||
let err = std::fs::File::create(self.path(&format!("{name}.err"))).unwrap();
|
||||
Daemon(self.command().stderr(Stdio::from(err)).spawn().unwrap())
|
||||
}
|
||||
|
||||
fn stderr_of(&self, name: &str) -> String {
|
||||
std::fs::read_to_string(self.path(&format!("{name}.err"))).unwrap_or_default()
|
||||
}
|
||||
|
||||
/// Runs `gatewayd serve` to its end, at most 10 s (then it is killed), and gives its exit code
|
||||
/// (`None` if it had to be killed) and its stderr.
|
||||
fn run_to_end(&self) -> (Option<i32>, String) {
|
||||
let mut child = Daemon(self.command().stderr(Stdio::piped()).spawn().unwrap());
|
||||
let deadline = Instant::now() + WAIT;
|
||||
let mut code = None;
|
||||
while Instant::now() < deadline {
|
||||
if let Ok(Some(status)) = child.0.try_wait() {
|
||||
code = status.code();
|
||||
break;
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(20));
|
||||
}
|
||||
let _ = child.0.kill();
|
||||
let _ = child.0.wait();
|
||||
let mut stderr = String::new();
|
||||
if let Some(mut pipe) = child.0.stderr.take() {
|
||||
let _ = std::io::Read::read_to_string(&mut pipe, &mut stderr);
|
||||
}
|
||||
(code, stderr)
|
||||
}
|
||||
}
|
||||
|
||||
/// A running `gatewayd serve`, killed and waited for when dropped. Declared after the `Home` and
|
||||
/// the `FakeMm`, so it drops first.
|
||||
struct Daemon(Child);
|
||||
|
||||
impl Drop for Daemon {
|
||||
fn drop(&mut self) {
|
||||
let _ = self.0.kill();
|
||||
let _ = self.0.wait();
|
||||
}
|
||||
}
|
||||
|
||||
impl Daemon {
|
||||
fn running(&mut self) -> bool {
|
||||
matches!(self.0.try_wait(), Ok(None))
|
||||
}
|
||||
}
|
||||
|
||||
/// True when `gatewayd` answers on `gateway.sock` as it does for `brokerd`: approval `approval`,
|
||||
/// of a `bxctl chat` session, is posted to the first approver and the posts come back. The
|
||||
/// connection is then closed, which ends the approval.
|
||||
fn answers(socket: &Path, approval: u64) -> bool {
|
||||
match ask(socket, pending(approval, "chat-1789700000-1", "{}")) {
|
||||
Ok(asking) => {
|
||||
let answered = !asking.posts.is_empty();
|
||||
asking.close();
|
||||
answered
|
||||
}
|
||||
Err(_) => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// Waits up to 10 s for `gatewayd` to answer on `socket`.
|
||||
fn wait_until_it_answers(socket: &Path, approval: u64) -> bool {
|
||||
let deadline = Instant::now() + WAIT;
|
||||
while Instant::now() < deadline {
|
||||
if answers(socket, approval) {
|
||||
return true;
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(50));
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
fn inode(path: &Path) -> u64 {
|
||||
use std::os::unix::fs::MetadataExt;
|
||||
std::fs::symlink_metadata(path)
|
||||
.map(|m| m.ino())
|
||||
.unwrap_or(0)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_second_gatewayd_on_the_same_home_is_refused_and_the_first_keeps_serving() {
|
||||
let mm = FakeMm::start();
|
||||
let home = Home::new("inst-twice", &mm);
|
||||
let mut first = home.start("first");
|
||||
let _ws = mm.next_ws(WAIT);
|
||||
assert!(
|
||||
wait_until_it_answers(&home.socket(), 1),
|
||||
"the first gatewayd never answered; stderr: {}",
|
||||
home.stderr_of("first")
|
||||
);
|
||||
let before = inode(&home.socket());
|
||||
|
||||
let (code, stderr) = home.run_to_end();
|
||||
let line = format!(
|
||||
"gatewayd: another gatewayd is running (it holds {}); {POINTER}",
|
||||
home.path("run/gatewayd.lock").display()
|
||||
);
|
||||
assert_eq!(code, Some(1), "the second gatewayd: {stderr}");
|
||||
assert_eq!(
|
||||
stderr,
|
||||
format!("{line}\n"),
|
||||
"the lock's line, and nothing else"
|
||||
);
|
||||
|
||||
assert!(first.running(), "the first gatewayd is still running");
|
||||
assert_eq!(
|
||||
inode(&home.socket()),
|
||||
before,
|
||||
"the first's socket is still there"
|
||||
);
|
||||
assert!(
|
||||
answers(&home.socket(), 2),
|
||||
"the first gatewayd still answers; its stderr: {}",
|
||||
home.stderr_of("first")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn after_the_first_is_killed_a_new_gatewayd_removes_the_stale_socket_and_serves() {
|
||||
let mm = FakeMm::start();
|
||||
let home = Home::new("inst-restart", &mm);
|
||||
let mut first = home.start("first");
|
||||
let _ws = mm.next_ws(WAIT);
|
||||
assert!(
|
||||
wait_until_it_answers(&home.socket(), 1),
|
||||
"the first gatewayd never answered; stderr: {}",
|
||||
home.stderr_of("first")
|
||||
);
|
||||
let _ = first.0.kill();
|
||||
let _ = first.0.wait();
|
||||
assert!(
|
||||
home.socket().exists(),
|
||||
"a killed gatewayd leaves its socket"
|
||||
);
|
||||
assert!(
|
||||
!answers(&home.socket(), 2),
|
||||
"nobody serves the stale socket"
|
||||
);
|
||||
|
||||
let _next = home.start("next");
|
||||
let _ws = mm.next_ws(WAIT);
|
||||
assert!(
|
||||
wait_until_it_answers(&home.socket(), 3),
|
||||
"the new gatewayd never answered; stderr: {}",
|
||||
home.stderr_of("next")
|
||||
);
|
||||
let lock = home.path("run/gatewayd.lock");
|
||||
let mode = std::fs::metadata(&lock).map(|m| m.permissions().mode() & 0o777);
|
||||
assert_eq!(mode.ok(), Some(0o600), "the lock file, mode 0600");
|
||||
}
|
||||
@@ -0,0 +1,223 @@
|
||||
//! One `memoryd` per home (M5b spec, section 9a): a second `memoryd serve` against the same home is
|
||||
//! refused with the lock's line, and the first keeps serving; once the first is killed, a new one
|
||||
//! removes the stale sockets and serves. Every wait has a deadline, and every process is killed
|
||||
//! and every path removed by a guard (tip T44). Do not edit.
|
||||
|
||||
#[path = "support/tmp.rs"]
|
||||
mod tmp;
|
||||
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
use std::os::unix::net::UnixStream;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::process::{Child, Command, Stdio};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use bxcore::{Empty, Envelope, Message, PROTOCOL_VERSION, read_frame, write_frame};
|
||||
use tmp::TempDir;
|
||||
|
||||
const POINTER: &str = "see docs/runbook.md#already-running";
|
||||
const SOCKETS: [&str; 3] = [
|
||||
"run/loop-memory/memory.sock",
|
||||
"run/broker-memory/memory.sock",
|
||||
"run/owner-memory/admin.sock",
|
||||
];
|
||||
|
||||
struct Home {
|
||||
dir: TempDir,
|
||||
config: PathBuf,
|
||||
}
|
||||
|
||||
impl Home {
|
||||
fn new(tag: &str) -> Home {
|
||||
let dir = TempDir::new(tag);
|
||||
let soul = dir.write("etc/soul.md", "You are Boxmaker.\n");
|
||||
let text = format!(
|
||||
"[paths]\nhome = \"{}\"\n[memory]\nsoul_starter = \"{}\"\n",
|
||||
dir.path().display(),
|
||||
soul.display()
|
||||
);
|
||||
let config = dir.write("memoryd.toml", &text);
|
||||
Home { dir, config }
|
||||
}
|
||||
|
||||
fn path(&self, relative: &str) -> PathBuf {
|
||||
self.dir.path().join(relative)
|
||||
}
|
||||
|
||||
fn command(&self) -> Command {
|
||||
let mut command = Command::new(env!("CARGO_BIN_EXE_memoryd"));
|
||||
command
|
||||
.args(["serve", "--config"])
|
||||
.arg(&self.config)
|
||||
.stdin(Stdio::null())
|
||||
.stdout(Stdio::null());
|
||||
command
|
||||
}
|
||||
|
||||
/// `memoryd serve` with its stderr in `<home>/<name>.err`, for the messages of a failing test.
|
||||
fn start(&self, name: &str) -> Daemon {
|
||||
let err = std::fs::File::create(self.path(&format!("{name}.err"))).unwrap();
|
||||
Daemon(self.command().stderr(Stdio::from(err)).spawn().unwrap())
|
||||
}
|
||||
|
||||
fn stderr_of(&self, name: &str) -> String {
|
||||
std::fs::read_to_string(self.path(&format!("{name}.err"))).unwrap_or_default()
|
||||
}
|
||||
|
||||
/// Runs `memoryd serve` to its end, at most 10 s (then it is killed), and gives its exit code
|
||||
/// (`None` if it had to be killed) and its stderr.
|
||||
fn run_to_end(&self) -> (Option<i32>, String) {
|
||||
let mut child = Daemon(self.command().stderr(Stdio::piped()).spawn().unwrap());
|
||||
let deadline = Instant::now() + Duration::from_secs(10);
|
||||
let mut code = None;
|
||||
while Instant::now() < deadline {
|
||||
if let Ok(Some(status)) = child.0.try_wait() {
|
||||
code = status.code();
|
||||
break;
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(20));
|
||||
}
|
||||
let _ = child.0.kill();
|
||||
let _ = child.0.wait();
|
||||
let mut stderr = String::new();
|
||||
if let Some(mut pipe) = child.0.stderr.take() {
|
||||
let _ = std::io::Read::read_to_string(&mut pipe, &mut stderr);
|
||||
}
|
||||
(code, stderr)
|
||||
}
|
||||
|
||||
fn inodes(&self) -> Vec<u64> {
|
||||
SOCKETS.iter().map(|s| inode(&self.path(s))).collect()
|
||||
}
|
||||
}
|
||||
|
||||
/// A running `memoryd serve`, killed and waited for when dropped. Declared after the `Home`, so it
|
||||
/// drops first.
|
||||
struct Daemon(Child);
|
||||
|
||||
impl Drop for Daemon {
|
||||
fn drop(&mut self) {
|
||||
let _ = self.0.kill();
|
||||
let _ = self.0.wait();
|
||||
}
|
||||
}
|
||||
|
||||
impl Daemon {
|
||||
fn running(&mut self) -> bool {
|
||||
matches!(self.0.try_wait(), Ok(None))
|
||||
}
|
||||
}
|
||||
|
||||
/// True when `memoryd` answers on `socket`, `loopd`'s: `memory_block` gets the block. Any other
|
||||
/// answer, or none within 5 s, is false.
|
||||
fn answers(socket: &Path) -> bool {
|
||||
let Ok(mut stream) = UnixStream::connect(socket) else {
|
||||
return false;
|
||||
};
|
||||
let limit = Some(Duration::from_secs(5));
|
||||
if stream.set_read_timeout(limit).is_err() || stream.set_write_timeout(limit).is_err() {
|
||||
return false;
|
||||
}
|
||||
let ask = Envelope {
|
||||
v: PROTOCOL_VERSION,
|
||||
id: 1,
|
||||
r#final: true,
|
||||
msg: Message::MemoryBlock(Empty {}),
|
||||
};
|
||||
if write_frame(&mut stream, &ask).is_err() {
|
||||
return false;
|
||||
}
|
||||
matches!(
|
||||
read_frame(&mut stream),
|
||||
Ok(Envelope {
|
||||
msg: Message::MemoryBlockReply(_),
|
||||
..
|
||||
})
|
||||
)
|
||||
}
|
||||
|
||||
/// Waits up to 10 s for `memoryd` to answer on `loopd`'s socket and to listen on the other two.
|
||||
fn wait_until_it_answers(home: &Home) -> bool {
|
||||
let deadline = Instant::now() + Duration::from_secs(10);
|
||||
while Instant::now() < deadline {
|
||||
if answers(&home.path(SOCKETS[0]))
|
||||
&& SOCKETS[1..]
|
||||
.iter()
|
||||
.all(|s| UnixStream::connect(home.path(s)).is_ok())
|
||||
{
|
||||
return true;
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(20));
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
fn inode(path: &Path) -> u64 {
|
||||
use std::os::unix::fs::MetadataExt;
|
||||
std::fs::symlink_metadata(path)
|
||||
.map(|m| m.ino())
|
||||
.unwrap_or(0)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_second_memoryd_on_the_same_home_is_refused_and_the_first_keeps_serving() {
|
||||
let home = Home::new("inst-twice");
|
||||
let mut first = home.start("first");
|
||||
assert!(
|
||||
wait_until_it_answers(&home),
|
||||
"the first memoryd never answered; stderr: {}",
|
||||
home.stderr_of("first")
|
||||
);
|
||||
let before = home.inodes();
|
||||
|
||||
let (code, stderr) = home.run_to_end();
|
||||
let line = format!(
|
||||
"memoryd: another memoryd is running (it holds {}); {POINTER}",
|
||||
home.path("run/memoryd.lock").display()
|
||||
);
|
||||
assert_eq!(code, Some(1), "the second memoryd: {stderr}");
|
||||
assert_eq!(
|
||||
stderr,
|
||||
format!("{line}\n"),
|
||||
"the lock's line, and nothing else"
|
||||
);
|
||||
|
||||
assert!(first.running(), "the first memoryd is still running");
|
||||
assert_eq!(home.inodes(), before, "the first's sockets are still there");
|
||||
assert!(
|
||||
answers(&home.path(SOCKETS[0])),
|
||||
"the first memoryd still answers; its stderr: {}",
|
||||
home.stderr_of("first")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn after_the_first_is_killed_a_new_memoryd_removes_the_stale_sockets_and_serves() {
|
||||
let home = Home::new("inst-restart");
|
||||
let mut first = home.start("first");
|
||||
assert!(
|
||||
wait_until_it_answers(&home),
|
||||
"the first memoryd never answered; stderr: {}",
|
||||
home.stderr_of("first")
|
||||
);
|
||||
let _ = first.0.kill();
|
||||
let _ = first.0.wait();
|
||||
assert!(
|
||||
SOCKETS.iter().all(|s| home.path(s).exists()),
|
||||
"a killed memoryd leaves its sockets"
|
||||
);
|
||||
assert!(
|
||||
!answers(&home.path(SOCKETS[0])),
|
||||
"nobody serves the stale socket"
|
||||
);
|
||||
|
||||
let _next = home.start("next");
|
||||
assert!(
|
||||
wait_until_it_answers(&home),
|
||||
"the new memoryd never answered; stderr: {}",
|
||||
home.stderr_of("next")
|
||||
);
|
||||
let lock = home.path("run/memoryd.lock");
|
||||
let mode = std::fs::metadata(&lock).map(|m| m.permissions().mode() & 0o777);
|
||||
assert_eq!(mode.ok(), Some(0o600), "the lock file, mode 0600");
|
||||
}
|
||||
Reference in New Issue
Block a user