boxmaker/docs/plans/M5b/12-instance-lock.md

M5b task 12: a lock file keeps loopd and inferproxy to one instance each

Branch: m5b (run git switch m5b; git status --short must be empty, otherwise stop) Commit subject: bxcore, loopd, inferproxy: a lock file keeps each to one instance

Goal

Every daemon removes a socket it finds at its path before it binds, so a socket left by a daemon that was killed never blocks the next start. But the removal is unconditional: a second copy of a daemon, started by mistake, takes the path from the one that runs, which keeps running on a socket nobody can reach, and nothing says so. From this task on, a daemon refuses to start instead (M5b spec, section 9a):

• Before it touches its sockets, each daemon takes an exclusive lock on a lock file with the standard library's File::try_lock, and holds it for the life of the process. The system releases it when the process ends, however it ends, so a killed daemon never leaves a lock behind. The file itself stays where it is.

• The lock file is <home>/run/<role>.lock (here loopd's: <home>/run/loopd.lock), and for inferproxy, which has no home, <socket>.lock beside its --listen socket.

• Another instance holds it: the daemon exits 1 with one line, and touches nothing else:

  loopd: another loopd is running (it holds /var/lib/boxmaker/run/loopd.lock); see docs/runbook.md#already-running
  

• Any other error on the lock file (a run/ it cannot write, a read-only socket directory): it exits 1, naming the file and the error:

  inferproxy: cannot take the lock /run/bx/infer.sock.lock: Permission denied (os error 13); see docs/runbook.md#already-running
  

This task writes the lock itself, bxcore::instance, and makes loopd and inferproxy take it. Task 13 does the same for brokerd, memoryd and gatewayd. The runbook entry the pointer names, already-running, is given as a fragment (it already covers all five daemons).

Name	What it is
bxcore::instance	new module (given as a skeleton; you fill two functions)
instance::lock	pub fn lock(path: &Path) -> Result<InstanceLock, LockError>
instance::InstanceLock	the held lock: the open file. Given.
instance::LockError	Held, or Io(std::io::Error), with Display and Error. Given.
LockError::line	pub fn line(&self, role: &str, path: &Path) -> String: the line the daemon prints
instance::RUNBOOK	"see docs/runbook.md#already-running". Given.

Files

• Copy, from docs/plans/M5b/files/ (the same path under it):
  • the skeleton crates/bxcore/src/instance.rs (you fill 2 functions in it)
  • tests: crates/bxcore/tests/instance.rs, crates/loopd/tests/instance.rs, crates/inferproxy/tests/instance.rs (all three new)
• Append, from docs/plans/M5b/files/, to the end of docs/runbook.md, with one blank line before it: runbook-already-running.md (step 3 gives the command).
• Modify: crates/bxcore/src/lib.rs (L1, L2), crates/loopd/src/main.rs (M1, M2), crates/inferproxy/src/main.rs (P1, P2), docs/implementer-log.md.
• Do not touch: any other source file, any Cargo.toml, any other test, and in the skeleton anything but the 2 todo!() bodies.

Facts you need

• File::try_lock (standard library, stable since Rust 1.89; the workspace needs 1.95, so it is there): pub fn try_lock(&self) -> Result<(), TryLockError>. It tries once and never waits. On Unix it is flock with LOCK_EX | LOCK_NB. From its documentation: it returns Err(TryLockError::WouldBlock) "if a different lock is already held on this file (via another handle/descriptor)", and "the lock will be released when this file (along with any other file descriptors/handles duplicated or inherited from it) is closed". So:
  • Two separate opens of the same file are two holders, in one process as in two. The bxcore tests rely on that: they call lock twice in one test.
  • The lock lives as long as the File does. InstanceLock holds the File, and each daemon keeps its InstanceLock in a local variable of the function that serves, so it lives until the process ends. let _ = lock(..) or a match arm Ok(_) => {} would drop it at once, and the lock with it; the edits bind it to a name, _instance.
  • Rust opens files with close-on-exec, so a program a daemon starts does not inherit the lock.
• std::fs::TryLockError (stable since 1.89): pub enum TryLockError { Error(io::Error), WouldBlock }. Error never holds an error of kind WouldBlock.
• The modes. OpenOptionsExt::mode(0o600) and DirBuilderExt::mode(0o700) apply only when the file or directory is created, and the process's umask can only remove bits. An existing lock file is opened as it is, and never truncated (truncate(false)): it holds nothing, and a killed daemon's file must be taken again unchanged.
• The directory. lock creates the lock file's directory when it is missing, with DirBuilder::new().recursive(true).mode(0o700), as brokerd, memoryd and gatewayd already make their socket directories (and so <home>/run on a first start). An existing directory is fine, and an empty parent (a lock file named without a directory) is the current directory, also fine. A directory that cannot be written is not made writable: the error is LockError::Io, and the daemon stops with the second line of the Goal.
• Where the lock is taken.
  • loopd serve (run_serve in crates/loopd/src/main.rs): right after the config loads, before the stale socket is removed, and so before the self-test. A second loopd then never sends the server a prompt either, which would take slot 0's cache from the one that runs. The home is cfg.paths.home ([paths] home, or $BOXMAKER_HOME). loopd selftest takes no lock: it binds no socket.
  • inferproxy (main in crates/inferproxy/src/main.rs): right after the arguments are parsed, before the stale socket is removed. The lock file is the --listen path with .lock added. Since lock makes a missing directory, inferproxy now also starts when its socket's directory does not exist yet; before, bind failed there.
• The given tests start the daemons as processes. Every wait has a deadline and every process is killed by a guard. Before this task, the tests that expect a second instance to be refused wait out that deadline (the second copy starts and serves), so the loopd file takes about 20 s to fail and the inferproxy file about 10 s. A test that "hangs" for that long is this, not a fault. The fake inference server in crates/loopd/tests/support/mod.rs does not change.
• The calls the bodies and edits make:

  Call	Signature, and what it gives
  DirBuilder	DirBuilder::new().recursive(true).mode(0o700).create(dir) -> io::Result<()>
  OpenOptions	OpenOptions::new().write(true).create(true).truncate(false).mode(0o600).open(path) -> io::Result<File>
  File::try_lock	file.try_lock() -> Result<(), TryLockError>
  Result::map_err	.map_err(LockError::Io)?: an io::Error becomes LockError::Io
  Path::parent	path.parent() -> Option<&Path>
  format!, Path::display	format!("… {} …", path.display()); {RUNBOOK} and {e} inline
  instance::lock	instance::lock(&lock_path) -> Result<InstanceLock, LockError>
  LockError::line	e.line("loopd", &lock_path) -> String
  OsString::push	let mut lock_path = path.clone().into_os_string(); lock_path.push(".lock");

The edits, exactly

Nothing else in these files changes. Each quoted "before" block occurs exactly once in its file.

crates/bxcore/src/lib.rs

L1. The lines

//! Shared data types and the frame codec for Boxmaker. No policy and no I/O beyond frames, except
//! the secret store (`secrets`), behind the feature of that name.

become

//! Shared data types and the frame codec for Boxmaker. No policy and no I/O beyond frames, except
//! the secret store (`secrets`), behind the feature of that name, and the lock that keeps a daemon
//! to one instance (`instance`).

L2. The line

pub mod ids;

becomes

pub mod ids;
pub mod instance;

instance is not behind a feature: loopd uses it.

crates/loopd/src/main.rs

M1. The lines

use std::time::Duration;

use loopd::broker_port::{BrokerPort, NoBroker, not_configured_line};

become

use std::time::Duration;

use bxcore::instance;
use loopd::broker_port::{BrokerPort, NoBroker, not_configured_line};

M2. In run_serve, the lines

    let socket = cfg.channel_socket();
    if socket.exists()

become

    // One loopd per home (M5b spec, section 9a). The lock comes before the socket is touched and
    // before the self-test, so a second copy neither takes the socket of the one that runs nor
    // sends its slot a prompt. Held until the process ends.
    let lock_path = cfg.paths.home.join("run/loopd.lock");
    let _instance = match instance::lock(&lock_path) {
        Ok(held) => held,
        Err(e) => {
            eprintln!("{}", e.line("loopd", &lock_path));
            return ExitCode::from(1);
        }
    };

    let socket = cfg.channel_socket();
    if socket.exists()

crates/inferproxy/src/main.rs

P1. The line

use inferproxy::{Limits, serve};

becomes

use bxcore::instance;
use inferproxy::{Limits, serve};

inferproxy depends on bxcore already; its Cargo.toml does not change.

P2. In main, the lines

    let path: PathBuf = listen.into();
    match fs::remove_file(&path) {

become

    let path: PathBuf = listen.into();
    // One inferproxy per socket (M5b spec, section 9a): the lock beside the socket comes before
    // the stale socket is removed, so a second copy never takes the socket of the one that runs.
    // Held until the process ends; `process::exit` leaves it to the system to release.
    let mut lock_path = path.clone().into_os_string();
    lock_path.push(".lock");
    let lock_path = PathBuf::from(lock_path);
    let _instance = match instance::lock(&lock_path) {
        Ok(held) => held,
        Err(e) => {
            eprintln!("{}", e.line("inferproxy", &lock_path));
            process::exit(1);
        }
    };
    match fs::remove_file(&path) {

Steps

• 1. Copy the given tests. From the repository root, three commands:

  cp docs/plans/M5b/files/crates/bxcore/tests/instance.rs crates/bxcore/tests/
  cp docs/plans/M5b/files/crates/loopd/tests/instance.rs crates/loopd/tests/
  cp docs/plans/M5b/files/crates/inferproxy/tests/instance.rs crates/inferproxy/tests/
  

• 2. See them fail. Three commands, one per file, each with a shell timeout of 600000 ms:

  • cargo test -p bxcore --test instance does not compile: one E0432, unresolved import bxcore::instance.

  • cargo test -p loopd --test instance compiles, and its 3 tests fail (the run takes about 20 s):

    Test	Before this task
    a_second_loopd_on_the_same_home_is_refused_and_the_first_keeps_serving	the second loopd: …, left: None, right: Some(1): the second copy started and served
    after_the_first_is_killed_a_new_loopd_removes_the_stale_socket_and_serves	the lock file, mode 0600, left: None, right: Some(384)
    a_run_directory_that_cannot_be_written_stops_loopd_naming_the_error	wanted the line …: loopd ran its self-test and failed later, on the socket directory

  • cargo test -p inferproxy --test instance compiles, and its 3 tests fail (about 10 s): the same three shapes, the second inferproxy: … with left: None, the lock file, beside the socket with left: None, and wanted the line for the read-only directory.

• 3. The runbook. From the repository root:

  { echo; cat docs/plans/M5b/files/runbook-already-running.md; } >> docs/runbook.md
  

  Then grep -c '^## already-running$' docs/runbook.md prints 1.

• 4. The module. Copy the skeleton, then make edits L1 and L2:

  cp docs/plans/M5b/files/crates/bxcore/src/instance.rs crates/bxcore/src/
  

  cargo check -p bxcore: no error, and 5 warnings, all from the unfilled bodies: two lines of unused imports (DirBuilder, OpenOptions, TryLockError; DirBuilderExt, OpenOptionsExt) and three unused variables (path twice, role). They go as you fill the bodies. cargo test -p bxcore --test instance now compiles: 1 test passes (the_error_displays_and_has_a_source: Display and Error are given), and 7 fail with not yet implemented.

• 5. Fill the two functions of instance.rs, one at a time, in this order, each from its comment, and run cargo check -p bxcore after each: LockError::line, then lock. Change nothing else in the file. Then cargo test -p bxcore --test instance: 8 passed.

• 6. loopd. Make edits M1 and M2. cargo check -p loopd: no error. cargo test -p loopd --test instance: 3 passed.

• 7. inferproxy. Make edits P1 and P2. cargo check -p inferproxy: no error. cargo test -p inferproxy --test instance: 3 passed.

• 8. Check. grep -rn 'runbook.md#already-running' crates/*/src prints exactly one line, the RUNBOOK constant in crates/bxcore/src/instance.rs: both daemons print the pointer through line, never a copy of it. git diff --stat -- '*Cargo.toml' prints nothing.

• 9. Run the gate. cargo fmt --all, then make gate, with a shell timeout of 600000 ms. Expected last line: gate: ok, with 1113 tests: 1099 before this task, plus 8 in crates/bxcore/tests/instance.rs, 3 in crates/loopd/tests/instance.rs and 3 in crates/inferproxy/tests/instance.rs.

• 10. Log and commit. git status --short lists crates/bxcore/src/lib.rs, crates/loopd/src/main.rs, crates/inferproxy/src/main.rs, docs/runbook.md and your log as modified, and crates/bxcore/src/instance.rs and the three test files as new. Stage with git add crates/bxcore crates/loopd crates/inferproxy docs/runbook.md docs/implementer-log.md Cargo.lock, then commit as AGENTS.md says. Cargo.lock does not change in this task; the git add is harmless.

Done when

• make gate prints gate: ok with 1113 tests.
• git diff HEAD~1 --stat lists only crates/bxcore/src/lib.rs, crates/bxcore/src/instance.rs, crates/loopd/src/main.rs, crates/inferproxy/src/main.rs, the three new test files, docs/runbook.md and docs/implementer-log.md.

Stop and report if

• A "before" block of L1, L2, M1, M2, P1 or P2 is not in its file exactly as shown, or is there more than once.
• Step 2, 3 or 4 shows a different result: other errors, other failing tests, or other counts.
• cargo check reports an error after a body you filled or an edit you made, and the comment or the edit does not tell you how to fix it. Quote the error, and stop.
• A comment in the skeleton does not fit the code under it. Quote it and stop.
• After step 5, 6 or 7 a test fails although each function follows its comment and each edit is as shown. Do not change a test or the skeleton's given code: stop, and give the failing tests' names and their messages.
• The fix seems to need a change anywhere else: another file of loopd or inferproxy, the test support files, a Cargo.toml. It does not; stop and say why you think so.
• A test outside these three files fails in the gate: check once whether it fails without your change, then stop and report both results.
• The gate passes with a count other than 1113.