bxcore, loopd, inferproxy: a lock file keeps each to one instance

Co-Authored-By: ornith-1.5-35b-a3b <agent@wntrmute.dev>

docs/runbook.md

@@ -782,3 +782,51 @@ For an `open!` tag the content is `<%s>`, for a `close!` tag `</%s>`. The first
   change: record it in `docs/decisions.md` and do not run `loopd` until it is settled.
 
 **Check.** `loopd selftest --config <path>` prints `selftest: ok`.
+
+## already-running
+
+**What you see.** A daemon exits 1 at start, before it touches its sockets, with one of these
+lines:
+
+- `<role>: another <role> is running (it holds <lock file>); see docs/runbook.md#already-running`
+- `<role>: cannot take the lock <lock file>: <error>; see docs/runbook.md#already-running`
+
+`<role>` is `loopd`, `brokerd`, `memoryd`, `gatewayd` or `inferproxy`. The lock file is
+`$BOXMAKER_HOME/run/<role>.lock`; for `inferproxy` it is its `--listen` socket's path with
+`.lock` added (`infer.sock.lock`, beside `infer.sock`).
+
+**Why.** Every daemon removes a socket it finds at its path before it binds, so that a socket
+left by a daemon that was killed never blocks the next start. A second copy started by mistake
+would therefore take the sockets of the one that runs, which would keep running, unreachable,
+and nothing would say so. So each daemon first takes an exclusive lock on its lock file, and
+holds it for as long as it runs (M5b spec, section 9a). The system releases the lock when the
+process ends, however it ends, so a killed daemon never leaves a lock behind. The file stays in
+place and holds nothing.
+
+**Confirm.** For `another <role> is running`, find the copy that runs, with its command line
+(the `--config` path says which home it serves):
+
+```sh
+pgrep -a -x <role>
+ls -l "$BOXMAKER_HOME/run/<role>.lock"
+```
+
+For `cannot take the lock`, the error says why: `Permission denied` means the daemon's user
+cannot create or open the file, or create `run/` (`ls -ld "$BOXMAKER_HOME" "$BOXMAKER_HOME/run"`
+and `ls -l` on the lock file show who owns them); `Not a directory` means a path above it is a
+file; `Read-only file system` means what it says.
+
+**Fix.** Two copies of a role against the same home are always a mistake: use the one that runs.
+To restart it, stop it (`pkill -x <role>`, or Ctrl-C in its terminal), wait until `pgrep -x
+<role>` prints nothing, and start it again. Two copies with different homes are fine (each has
+its own lock), as long as their configs do not name the same socket path. For a lock that cannot
+be taken, give the daemon's user the directory (`chown`), or point its home somewhere it can
+write.
+
+Never delete a lock file while its daemon runs: the lock is held on the open file, so a new copy
+would create a new file, lock that, and take the sockets. Deleting it while the daemon is stopped
+does no harm, and does not help either.
+
+**Check.** `pgrep -x <role>` prints one process id, and the daemon prints its start line
+(`loopd: serving on …`, `inferproxy: <socket> -> <upstream>`, `brokerd: serving tools on …`,
+`memoryd: serving`, `gatewayd: connected to …`).