boxmaker/docs/plans/M5b/13-instance-lock-rest.md

M5b task 13: brokerd, memoryd and gatewayd take the instance lock too

Branch: m5b (run git switch m5b; git status --short must be empty, otherwise stop) Commit subject: brokerd, memoryd, gatewayd: a lock file keeps each to one instance

Goal

Task 12 wrote bxcore::instance, the lock that keeps a daemon to one instance (M5b spec, section 9a), and made loopd and inferproxy take it. This task makes the other three daemons take it the same way: before they touch their sockets, each locks <home>/run/<role>.lock and holds the lock for the life of the process. A second copy against the same home exits 1 with one line and touches nothing else:

brokerd: another brokerd is running (it holds /var/lib/boxmaker/run/brokerd.lock); see docs/runbook.md#already-running
memoryd: another memoryd is running (it holds /var/lib/boxmaker/run/memoryd.lock); see docs/runbook.md#already-running
gatewayd: another gatewayd is running (it holds /var/lib/boxmaker/run/gatewayd.lock); see docs/runbook.md#already-running

Any other error on the lock file stops the daemon too, with the line <role>: cannot take the lock <lock file>: <error>; see docs/runbook.md#already-running. Both lines come from LockError::line; the runbook entry already-running is there since task 12, and covers all five daemons.

Files

• Copy, from docs/plans/M5b/files/ (the same path under it):
  • new tests: crates/brokerd/tests/instance.rs, crates/memoryd/tests/instance.rs, crates/gatewayd/tests/instance.rs
  • a new version of crates/brokerd/tests/serve.rs (it replaces the file in the tree). Do not copy serve.rs-task01: it is the older version.
• Modify: crates/brokerd/src/main.rs (B1, B2), crates/memoryd/src/main.rs (D1, D2), crates/gatewayd/src/main.rs (G1 to G3), docs/implementer-log.md.
• Do not touch: crates/bxcore (instance is done), any other source file, any Cargo.toml, any other test, docs/runbook.md.

Facts you need

• What bxcore::instance gives (task 12; do not change it):

  Name	What it is
  instance::lock	pub fn lock(path: &Path) -> Result<InstanceLock, LockError>: creates the file (0600) and a missing directory (0700), and tries the lock once
  InstanceLock	the held lock; the lock is released when it is dropped or the process ends
  LockError	Held (another instance holds it) or Io(std::io::Error)
  LockError::line	e.line("brokerd", &lock_path) -> String: the whole line to print, pointer included

• Keep the lock alive. Each daemon binds the InstanceLock to a name, _instance, in main (or serve for gatewayd), the function that serves until the process ends. let _ = …, or a match arm Ok(_) => {}, would drop it, and the lock with it, at once. The edits show the form.

• Where each daemon takes it, and why there:

  • brokerd (main in crates/brokerd/src/main.rs): right after the config loads, before serve::start, which opens the audit log and binds both sockets. The home is cfg.paths.home.
  • memoryd (main in crates/memoryd/src/main.rs): right after the config loads, before serve::start, which creates memory/ and the index and binds three sockets. The home is config.paths.home.
  • gatewayd (serve in crates/gatewayd/src/main.rs): after the secret is loaded, before the state directory is made and before run, which binds gateway.sock and then connects to Mattermost. Not earlier: the test a_missing_secret_stops_at_start in crates/gatewayd/tests/main.rs checks that nothing under the home is made before the secret is read, and the lock creates <home>/run. The home is config.paths.home.

• brokerd's audit lock stays. brokerd has held a lock on <home>/audit/.lock since M3a, and a second brokerd used to stop there, with brokerd is already running and the pointer brokerd-already-running. It stays: it guards the log itself, and its own test (a_second_writer_is_refused in crates/brokerd/tests/audit.rs) still passes. But the instance lock now comes first, so a second brokerd stops with the new line instead. That is why serve.rs has a new version: its test a_second_brokerd_on_the_same_home_refuses_to_start expected the old message, and it is gone; instance.rs checks the second brokerd now. Nothing else in serve.rs changes.

• The given tests start each daemon as a process, as brokerd's serve.rs does. The gatewayd one runs it against the fake Mattermost of tests/support/fake_mm.rs, with its token in the environment (GW_TEST_TOKEN), and checks that it answers on gateway.sock by asking it to post an approval, as brokerd would. Every wait has a deadline and every process is killed by a guard. Before this task, the memoryd and gatewayd tests that expect a second instance to be refused wait out their deadline (the second copy starts and serves), so each of those files takes about 10 s to fail. That is this, not a hang.

• The calls the edits make: instance::lock and LockError::line from the table above; Path::join for the lock file's path; eprintln! and ExitCode::from(1) as the lines around them already do.

The edits, exactly

Nothing else in these files changes. Each quoted "before" block occurs exactly once in its file.

crates/brokerd/src/main.rs

B1. The line

use brokerd::serve::{self, ServeError};

becomes

use brokerd::serve::{self, ServeError};
use bxcore::instance;

B2. In main, the line

    // Keep the socket paths for the serving notice, before `cfg` is moved into `start`.

becomes

    // One brokerd per home (M5b spec, section 9a). The lock comes before anything is made, the
    // audit log's own lock included, so a second copy never takes the sockets of the one that
    // runs. Held until the process ends.
    let lock_path = cfg.paths.home.join("run/brokerd.lock");
    let _instance = match instance::lock(&lock_path) {
        Ok(held) => held,
        Err(e) => {
            eprintln!("{}", e.line("brokerd", &lock_path));
            return ExitCode::from(1);
        }
    };

    // Keep the socket paths for the serving notice, before `cfg` is moved into `start`.

crates/memoryd/src/main.rs

D1. The line

use memoryd::config::Config;

becomes

use bxcore::instance;
use memoryd::config::Config;

D2. In main, the line

    let log: serve::Log = Arc::new(|line: &str| eprintln!("{line}"));

becomes

    // One memoryd per home (M5b spec, section 9a). The lock comes before anything under the home
    // is made or bound, so a second copy never takes the sockets of the one that runs. Held until
    // the process ends.
    let lock_path = config.paths.home.join("run/memoryd.lock");
    let _instance = match instance::lock(&lock_path) {
        Ok(held) => held,
        Err(e) => {
            eprintln!("{}", e.line("memoryd", &lock_path));
            return ExitCode::from(1);
        }
    };
    let log: serve::Log = Arc::new(|line: &str| eprintln!("{line}"));

crates/gatewayd/src/main.rs

G1. The line

use bxcore::secrets;

becomes

use bxcore::{instance, secrets};

G2. In the comment at the top of serve, the line

    //    Then `if let Some(warning) = &loaded.warning { eprintln!("{warning}"); }`.

becomes

    //    Then `if let Some(warning) = &loaded.warning { eprintln!("{warning}"); }`.
    // 3b. The instance lock, `<home>/run/gatewayd.lock` (M5b spec, section 9a): after the secret,
    //    which is read before anything is made, and before the directory and the socket.

G3. In serve, the lines

    if let Some(warning) = &loaded.warning {
        eprintln!("{warning}");
    }

become

    if let Some(warning) = &loaded.warning {
        eprintln!("{warning}");
    }
    let lock_path = config.paths.home.join("run/gatewayd.lock");
    let _instance = match instance::lock(&lock_path) {
        Ok(held) => held,
        Err(e) => {
            eprintln!("{}", e.line("gatewayd", &lock_path));
            return ExitCode::from(1);
        }
    };

Steps

• 1. Copy the given tests. From the repository root, four commands:

  cp docs/plans/M5b/files/crates/brokerd/tests/instance.rs crates/brokerd/tests/
  cp docs/plans/M5b/files/crates/brokerd/tests/serve.rs crates/brokerd/tests/
  cp docs/plans/M5b/files/crates/memoryd/tests/instance.rs crates/memoryd/tests/
  cp docs/plans/M5b/files/crates/gatewayd/tests/instance.rs crates/gatewayd/tests/
  

• 2. See them fail. Three commands, each with a shell timeout of 600000 ms. Each file compiles; serve.rs passes.

  • cargo test -p brokerd --no-fail-fast --test instance --test serve (--no-fail-fast, or cargo stops at the first failing file): serve 8 passed. instance: 2 failed. a_second_brokerd_on_the_same_home_is_refused_and_the_first_keeps_serving fails on the lock's line, and nothing else (the second stopped at the audit lock, with brokerd is already running); after_the_first_is_killed_a_new_brokerd_removes_the_stale_sockets_and_serves fails on the lock file, mode 0600, left: None, right: Some(384).
  • cargo test -p memoryd --test instance: 2 failed (about 10 s). a_second_memoryd_on_the_same_home_is_refused_and_the_first_keeps_serving fails on the second memoryd: … with left: None, right: Some(1) (it started and served); after_the_first_is_killed_a_new_memoryd_removes_the_stale_sockets_and_serves on the lock file, mode 0600, left: None.
  • cargo test -p gatewayd --test instance: 2 failed (about 10 s), the same two shapes: the second gatewayd: … with left: None, and the lock file, mode 0600 with left: None.

• 3. brokerd. Make edits B1 and B2. cargo check -p brokerd: no error. cargo test -p brokerd --test instance --test serve: instance 2 passed, serve 8 passed.

• 4. memoryd. Make edits D1 and D2. cargo check -p memoryd: no error. cargo test -p memoryd --test instance: 2 passed.

• 5. gatewayd. Make edits G1 to G3. cargo check -p gatewayd: no error. cargo test -p gatewayd --test instance --test main: instance 2 passed, main 5 passed.

• 6. Check. grep -rn 'instance::lock(' crates/*/src prints exactly five lines, one in the main.rs of each of brokerd, gatewayd, inferproxy, loopd and memoryd. grep -rn 'runbook.md#already-running' crates/*/src still prints exactly one line, in crates/bxcore/src/instance.rs. git diff --stat -- crates/bxcore '*Cargo.toml' prints nothing.

• 7. Run the gate. cargo fmt --all, then make gate, with a shell timeout of 600000 ms. Expected last line: gate: ok, with 1118 tests: 1113 before this task, minus 1 in the new crates/brokerd/tests/serve.rs (its old second-instance test), plus 2 in each of the three new instance.rs files.

• 8. Log and commit. git status --short lists crates/brokerd/src/main.rs, crates/memoryd/src/main.rs, crates/gatewayd/src/main.rs, crates/brokerd/tests/serve.rs and your log as modified, and the three instance.rs test files as new. Stage with git add crates/brokerd crates/memoryd crates/gatewayd docs/implementer-log.md Cargo.lock, then commit as AGENTS.md says. Cargo.lock does not change in this task; the git add is harmless.

Done when

• make gate prints gate: ok with 1118 tests.
• git diff HEAD~1 --stat lists only the three main.rs files, crates/brokerd/tests/serve.rs, the three new instance.rs test files and docs/implementer-log.md.

Stop and report if

• A "before" block of B1, B2, D1, D2 or G1 to G3 is not in its file exactly as shown, or is there more than once.
• Step 2 shows a different result: other failing tests, other messages, or other counts.
• cargo check reports an error after an edit made as shown. Quote it and stop.
• After step 3, 4 or 5 a test fails although the edits are as shown. Do not change a test or bxcore: stop, and give the failing tests' names and their messages.
• The fix seems to need a change anywhere else: bxcore, another file of a daemon, the test support files, a Cargo.toml, the runbook. It does not; stop and say why you think so.
• A test outside these files fails in the gate: check once whether it fails without your change, then stop and report both results.
• The gate passes with a count other than 1118.