boxmaker/docs/plans/M5b/13-instance-lock-rest.md

# M5b task 13: `brokerd`, `memoryd` and `gatewayd` take the instance lock too

**Branch:** `m5b` (run `git switch m5b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `brokerd, memoryd, gatewayd: a lock file keeps each to one instance`

## Goal

Task 12 wrote `bxcore::instance`, the lock that keeps a daemon to one instance (M5b spec,
section 9a), and made `loopd` and `inferproxy` take it. This task makes the other three daemons
take it the same way: before they touch their sockets, each locks `<home>/run/<role>.lock` and
holds the lock for the life of the process. A second copy against the same home exits 1 with one
line and touches nothing else:

```text
brokerd: another brokerd is running (it holds /var/lib/boxmaker/run/brokerd.lock); see docs/runbook.md#already-running
memoryd: another memoryd is running (it holds /var/lib/boxmaker/run/memoryd.lock); see docs/runbook.md#already-running
gatewayd: another gatewayd is running (it holds /var/lib/boxmaker/run/gatewayd.lock); see docs/runbook.md#already-running
```

Any other error on the lock file stops the daemon too, with the line
`<role>: cannot take the lock <lock file>: <error>; see docs/runbook.md#already-running`. Both
lines come from `LockError::line`; the runbook entry `already-running` is there since task 12, and
covers all five daemons.

## Files

- Copy, from `docs/plans/M5b/files/` (the same path under it):
  - new tests: `crates/brokerd/tests/instance.rs`, `crates/memoryd/tests/instance.rs`,
    `crates/gatewayd/tests/instance.rs`
  - a **new version** of `crates/brokerd/tests/serve.rs` (it replaces the file in the tree). Do
    not copy `serve.rs-task01`: it is the older version.
- Modify: `crates/brokerd/src/main.rs` (B1, B2), `crates/memoryd/src/main.rs` (D1, D2),
  `crates/gatewayd/src/main.rs` (G1 to G3), `docs/implementer-log.md`.
- Do not touch: `crates/bxcore` (`instance` is done), any other source file, any `Cargo.toml`,
  any other test, `docs/runbook.md`.

## Facts you need

- **What `bxcore::instance` gives** (task 12; do not change it):

  | Name | What it is |
  |---|---|
  | `instance::lock` | `pub fn lock(path: &Path) -> Result<InstanceLock, LockError>`: creates the file (0600) and a missing directory (0700), and tries the lock once |
  | `InstanceLock` | the held lock; the lock is released when it is dropped or the process ends |
  | `LockError` | `Held` (another instance holds it) or `Io(std::io::Error)` |
  | `LockError::line` | `e.line("brokerd", &lock_path) -> String`: the whole line to print, pointer included |

- **Keep the lock alive.** Each daemon binds the `InstanceLock` to a name, `_instance`, in `main`
  (or `serve` for `gatewayd`), the function that serves until the process ends. `let _ = …`, or a
  `match` arm `Ok(_) => {}`, would drop it, and the lock with it, at once. The edits show the form.
- **Where each daemon takes it**, and why there:
  - `brokerd` (`main` in `crates/brokerd/src/main.rs`): right after the config loads, before
    `serve::start`, which opens the audit log and binds both sockets. The home is
    `cfg.paths.home`.
  - `memoryd` (`main` in `crates/memoryd/src/main.rs`): right after the config loads, before
    `serve::start`, which creates `memory/` and the index and binds three sockets. The home is
    `config.paths.home`.
  - `gatewayd` (`serve` in `crates/gatewayd/src/main.rs`): after the secret is loaded, before the
    state directory is made and before `run`, which binds `gateway.sock` and then connects to
    Mattermost. Not earlier: the test `a_missing_secret_stops_at_start` in
    `crates/gatewayd/tests/main.rs` checks that nothing under the home is made before the secret
    is read, and the lock creates `<home>/run`. The home is `config.paths.home`.
- **`brokerd`'s audit lock stays.** `brokerd` has held a lock on `<home>/audit/.lock` since M3a,
  and a second `brokerd` used to stop there, with `brokerd is already running` and the pointer
  `brokerd-already-running`. It stays: it guards the log itself, and its own test
  (`a_second_writer_is_refused` in `crates/brokerd/tests/audit.rs`) still passes. But the instance
  lock now comes first, so a second `brokerd` stops with the new line instead. That is why
  `serve.rs` has a new version: its test `a_second_brokerd_on_the_same_home_refuses_to_start`
  expected the old message, and it is gone; `instance.rs` checks the second `brokerd` now. Nothing
  else in `serve.rs` changes.
- **The given tests start each daemon as a process**, as `brokerd`'s `serve.rs` does. The
  `gatewayd` one runs it against the fake Mattermost of `tests/support/fake_mm.rs`, with its token
  in the environment (`GW_TEST_TOKEN`), and checks that it answers on `gateway.sock` by asking it
  to post an approval, as `brokerd` would. Every wait has a deadline and every process is killed by
  a guard. Before this task, the `memoryd` and `gatewayd` tests that expect a second instance to be
  refused wait out their deadline (the second copy starts and serves), so each of those files takes
  about 10 s to fail. That is this, not a hang.
- The calls the edits make: `instance::lock` and `LockError::line` from the table above;
  `Path::join` for the lock file's path; `eprintln!` and `ExitCode::from(1)` as the lines around
  them already do.

## The edits, exactly

Nothing else in these files changes. Each quoted "before" block occurs exactly once in its file.

### `crates/brokerd/src/main.rs`

**B1.** The line

```rust
use brokerd::serve::{self, ServeError};
```

becomes

```rust
use brokerd::serve::{self, ServeError};
use bxcore::instance;
```

**B2.** In `main`, the line

```rust
    // Keep the socket paths for the serving notice, before `cfg` is moved into `start`.
```

becomes

```rust
    // One brokerd per home (M5b spec, section 9a). The lock comes before anything is made, the
    // audit log's own lock included, so a second copy never takes the sockets of the one that
    // runs. Held until the process ends.
    let lock_path = cfg.paths.home.join("run/brokerd.lock");
    let _instance = match instance::lock(&lock_path) {
        Ok(held) => held,
        Err(e) => {
            eprintln!("{}", e.line("brokerd", &lock_path));
            return ExitCode::from(1);
        }
    };

    // Keep the socket paths for the serving notice, before `cfg` is moved into `start`.
```

### `crates/memoryd/src/main.rs`

**D1.** The line

```rust
use memoryd::config::Config;
```

becomes

```rust
use bxcore::instance;
use memoryd::config::Config;
```

**D2.** In `main`, the line

```rust
    let log: serve::Log = Arc::new(|line: &str| eprintln!("{line}"));
```

becomes

```rust
    // One memoryd per home (M5b spec, section 9a). The lock comes before anything under the home
    // is made or bound, so a second copy never takes the sockets of the one that runs. Held until
    // the process ends.
    let lock_path = config.paths.home.join("run/memoryd.lock");
    let _instance = match instance::lock(&lock_path) {
        Ok(held) => held,
        Err(e) => {
            eprintln!("{}", e.line("memoryd", &lock_path));
            return ExitCode::from(1);
        }
    };
    let log: serve::Log = Arc::new(|line: &str| eprintln!("{line}"));
```

### `crates/gatewayd/src/main.rs`

**G1.** The line

```rust
use bxcore::secrets;
```

becomes

```rust
use bxcore::{instance, secrets};
```

**G2.** In the comment at the top of `serve`, the line

```rust
    //    Then `if let Some(warning) = &loaded.warning { eprintln!("{warning}"); }`.
```

becomes

```rust
    //    Then `if let Some(warning) = &loaded.warning { eprintln!("{warning}"); }`.
    // 3b. The instance lock, `<home>/run/gatewayd.lock` (M5b spec, section 9a): after the secret,
    //    which is read before anything is made, and before the directory and the socket.
```

**G3.** In `serve`, the lines

```rust
    if let Some(warning) = &loaded.warning {
        eprintln!("{warning}");
    }
```

become

```rust
    if let Some(warning) = &loaded.warning {
        eprintln!("{warning}");
    }
    let lock_path = config.paths.home.join("run/gatewayd.lock");
    let _instance = match instance::lock(&lock_path) {
        Ok(held) => held,
        Err(e) => {
            eprintln!("{}", e.line("gatewayd", &lock_path));
            return ExitCode::from(1);
        }
    };
```

## Steps

- [ ] **1. Copy the given tests.** From the repository root, four commands:

  ```sh
  cp docs/plans/M5b/files/crates/brokerd/tests/instance.rs crates/brokerd/tests/
  cp docs/plans/M5b/files/crates/brokerd/tests/serve.rs crates/brokerd/tests/
  cp docs/plans/M5b/files/crates/memoryd/tests/instance.rs crates/memoryd/tests/
  cp docs/plans/M5b/files/crates/gatewayd/tests/instance.rs crates/gatewayd/tests/
  ```

- [ ] **2. See them fail.** Three commands, each with a shell timeout of 600000 ms. Each file
  compiles; `serve.rs` passes.
  - `cargo test -p brokerd --no-fail-fast --test instance --test serve` (`--no-fail-fast`, or
    cargo stops at the first failing file): `serve` 8 passed. `instance`: 2 failed.
    `a_second_brokerd_on_the_same_home_is_refused_and_the_first_keeps_serving` fails on `the
    lock's line, and nothing else` (the second stopped at the audit lock, with `brokerd is already
    running`); `after_the_first_is_killed_a_new_brokerd_removes_the_stale_sockets_and_serves`
    fails on `the lock file, mode 0600`, `left: None`, `right: Some(384)`.
  - `cargo test -p memoryd --test instance`: 2 failed (about 10 s).
    `a_second_memoryd_on_the_same_home_is_refused_and_the_first_keeps_serving` fails on `the
    second memoryd: …` with `left: None`, `right: Some(1)` (it started and served);
    `after_the_first_is_killed_a_new_memoryd_removes_the_stale_sockets_and_serves` on `the lock
    file, mode 0600`, `left: None`.
  - `cargo test -p gatewayd --test instance`: 2 failed (about 10 s), the same two shapes:
    `the second gatewayd: …` with `left: None`, and `the lock file, mode 0600` with `left: None`.
- [ ] **3. `brokerd`.** Make edits B1 and B2. `cargo check -p brokerd`: no error.
  `cargo test -p brokerd --test instance --test serve`: `instance` 2 passed, `serve` 8 passed.
- [ ] **4. `memoryd`.** Make edits D1 and D2. `cargo check -p memoryd`: no error.
  `cargo test -p memoryd --test instance`: 2 passed.
- [ ] **5. `gatewayd`.** Make edits G1 to G3. `cargo check -p gatewayd`: no error.
  `cargo test -p gatewayd --test instance --test main`: `instance` 2 passed, `main` 5 passed.
- [ ] **6. Check.** `grep -rn 'instance::lock(' crates/*/src` prints exactly five lines, one in
  the `main.rs` of each of `brokerd`, `gatewayd`, `inferproxy`, `loopd` and `memoryd`.
  `grep -rn 'runbook.md#already-running' crates/*/src` still prints exactly one line, in
  `crates/bxcore/src/instance.rs`. `git diff --stat -- crates/bxcore '*Cargo.toml'` prints nothing.
- [ ] **7. Run the gate.** `cargo fmt --all`, then `make gate`, with a shell timeout of 600000 ms.
  Expected last line: `gate: ok`, with 1118 tests: 1113 before this task, minus 1 in the new
  `crates/brokerd/tests/serve.rs` (its old second-instance test), plus 2 in each of the three new
  `instance.rs` files.
- [ ] **8. Log and commit.** `git status --short` lists `crates/brokerd/src/main.rs`,
  `crates/memoryd/src/main.rs`, `crates/gatewayd/src/main.rs`, `crates/brokerd/tests/serve.rs`
  and your log as modified, and the three `instance.rs` test files as new. Stage with
  `git add crates/brokerd crates/memoryd crates/gatewayd docs/implementer-log.md Cargo.lock`,
  then commit as `AGENTS.md` says. `Cargo.lock` does not change in this task; the `git add` is
  harmless.

## Done when

- `make gate` prints `gate: ok` with 1118 tests.
- `git diff HEAD~1 --stat` lists only the three `main.rs` files, `crates/brokerd/tests/serve.rs`,
  the three new `instance.rs` test files and `docs/implementer-log.md`.

## Stop and report if

- A "before" block of B1, B2, D1, D2 or G1 to G3 is not in its file exactly as shown, or is there
  more than once.
- Step 2 shows a different result: other failing tests, other messages, or other counts.
- `cargo check` reports an error after an edit made as shown. Quote it and stop.
- After step 3, 4 or 5 a test fails although the edits are as shown. Do not change a test or
  `bxcore`: stop, and give the failing tests' names and their messages.
- The fix seems to need a change anywhere else: `bxcore`, another file of a daemon, the test
  support files, a `Cargo.toml`, the runbook. It does not; stop and say why you think so.
- A test outside these files fails in the gate: check once whether it fails without your change,
  then stop and report both results.
- The gate passes with a count other than 1118.