kyle
2f9d3d97e6
Dixie is capped (max_inflight 2, queue_depth 8, queue_wait 20s) and never queued on llama-server. Streaming chat, thinking off, 45s fallback to clean. Digest and chunk-embedding caches are content-addressed SQLite (Honcho context is not cached). Recover uses the guarded Fetcher. guardian-star is refused in code.
113 lines
9.4 KiB
Markdown
113 lines
9.4 KiB
Markdown
# CLAUDE.md
|
||
|
||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||
|
||
## What this is
|
||
|
||
`modernc.org/libc` is a partial reimplementation of C libc in pure Go. It is the **runtime** for C programs translated to Go by the `modernc.org/ccgo` transpiler (notably `modernc.org/sqlite`). It is not a standalone library for general use; its API tracks the needs of ccgo-generated code and may change incompatibly between versions.
|
||
|
||
Compatibility note: callers of this package (e.g. `modernc.org/sqlite`) must use the libc version that matches their generated code (see the consumer's `go.mod`). Do not bump the libc version of a downstream consumer in isolation without re-translating its C sources.
|
||
|
||
## Architectural split: musl-derived vs hand-written
|
||
|
||
This is the most important thing to understand before editing any file. There are two completely separate code paths selected by build constraint:
|
||
|
||
1. **musl-derived (linux/amd64, arm64, loong64, ppc64le, s390x, riscv64, 386, arm)** — the libc implementation is generated by transpiling musl-libc's C source to Go via `ccgo/v4`. The canonical build tag is:
|
||
```
|
||
//go:build linux && (amd64 || arm64 || loong64 || ppc64le || s390x || riscv64 || 386 || arm)
|
||
```
|
||
Generated files for this path: `ccgo_linux_*.go` (~4 MB each, the bulk of the musl translation), the per-arch `musl_*.go` files written by older tooling, and on amd64 the assembly stubs in `abi0_linux_amd64.{go,s}` (generated by `qbecc --abi0wrap`). The hand-written glue lives in `libc_musl.go`, `native_musl.go` (native Go replacements for the hot musl routines memcpy/memmove/memset/memcmp/fabs/strcspn on all musl targets; the transpiled originals survive in the generated files as `___musl_*`, renamed by the generator), `etc_musl.go`, `mem_brk_musl.go`, `mem_musl.go`, `memgrind_musl.go`, `pthread_musl.go`, `syscall_musl.go`, `aliases.go`, `atomic.go`, `atomic{32,64}.go`, `builtin{32,64}.go`, `libc_linux_statfs.go`.
|
||
|
||
2. **Hand-written (darwin, freebsd, netbsd, openbsd, illumos, windows, linux/mips64le)** — the libc surface is implemented by hand in Go, with the negated build constraint:
|
||
```
|
||
//go:build !(linux && (amd64 || arm64 || loong64 || ppc64le || s390x || riscv64 || 386 || arm))
|
||
```
|
||
Main files: `libc.go`, `libc_unix.go`, `libc_unix{1,2,3}.go`, `libc_<goos>.go`, `libc_<goos>_<goarch>.go`, `libc_windows*.go`, `pthread.go`, `mem.go`, `memgrind.go`, `mem_brk.go`. These ports also pull in an older set of ccgo/v3-generated `musl_<goos>_<goarch>.go` files for the platform-independent pieces (strtod, math, network, etc.).
|
||
|
||
When you make a change you almost always need to make it in both code paths. Look at the existing build tag of the file you're editing — if it has the long `linux && (amd64 || ...)` form, you're in the musl path, and there's a sibling file with the negated tag that needs the parallel change.
|
||
|
||
## Common commands
|
||
|
||
Build / sanity check:
|
||
- `go build ./...` — current platform only
|
||
- `make editor` — fast iteration: compile tests to /dev/null, build generator, lint generated asm
|
||
- `make build_all_targets` — full cross-compile sweep across all supported GOOS/GOARCH × build tags (slow; required before submitting per `CONTRIBUTING.md`)
|
||
- `make vet` — vet, filtering out known abi0 false positives
|
||
|
||
Tests:
|
||
- `make short-test` — `go test -short` plus log-grep summary
|
||
- `make test` — full test, 24h timeout
|
||
- `make xtest` — full test with `-tags=ccgo.dmesg,ccgo.assert` (verbose ccgo debugging)
|
||
- `make libc-test` — runs only `TestLibc`, which transpiles and runs the musl libc-test suite under `testdata/nsz.repo.hu/libc-test`
|
||
- `make xlibc` — `TestLibc` with debug tags
|
||
- `make membrk-test` — full test with `-tags=libc.membrk` (the sbrk-style debug allocator)
|
||
- Run one test: `go test -v -run TestStrtod -count=1`
|
||
- Filter `TestLibc` subtests: `go test -v -run TestLibc -re '<regexp>'` (the `-re` flag is defined in `all_musl_test.go`)
|
||
|
||
Regenerate musl-derived code:
|
||
- `make generate` — downloads the pinned musl tarball if needed, re-translates it via `go run generator.go` into `$DIR` (default `/tmp/libc`), then rebuilds and short-tests. **Only works on linux.** Output files (`ccgo_linux_*.go`, etc.) are committed to the repo and should not be hand-edited.
|
||
- `make dev` — same as `generate` but with `GO_GENERATE_DEV=1` and `-tags=ccgo.dmesg,ccgo.assert` for ccgo debugging
|
||
- Cross-arch generation: set `GO_GENERATE_GOARCH=arm64` etc. The `builder.json` `autogen` regex lists which arches CI auto-regenerates; recent commits like `pi32 auto generate` are these.
|
||
- Pinned musl version: `internal/archive/archive.go` (currently `musl-7ada6dde6f9dc6a2836c3d92c2f762d35fd229e0`).
|
||
|
||
## Build tags
|
||
|
||
Functional tags (passed via `-tags=`):
|
||
- `libc.membrk` — replaces malloc with a fixed-size sbrk-style allocator (1 GiB on 64-bit), `free` becomes a no-op, heap pre-filled with PRNG bytes. Used to surface use-after-free in downstream code.
|
||
- `libc.memgrind` — wraps the allocator with an audit table; `MemAuditStart` / `MemAuditReport` detect leaks. Adds overhead.
|
||
- `libc.dmesg` — appends a per-pid log to `/tmp/libc.log` of major libc events.
|
||
- `libc.strace` — flips `__ccgo_strace` to `true` so every `Xfoo` entry prints a trace line. Heavy.
|
||
- `libc.memexpvar` — exposes allocator stats through `expvar`.
|
||
- `ccgo.dmesg`, `ccgo.assert` — consumed by ccgo during generation, not at runtime.
|
||
|
||
`libc.membrk` and `libc.memgrind` are mutually exclusive (the file build tags enforce it).
|
||
|
||
## Symbol naming conventions
|
||
|
||
ccgo encodes C identifiers with prefixes; the same scheme is used in hand-written files for consistency:
|
||
- `Xfoo` — externally visible C function `foo` (e.g. `Xmalloc`, `Xputchar`)
|
||
- `Yfoo` — abi0-wrapped entry to `Xfoo` (linux/amd64 only, in `abi0_linux_amd64.go`)
|
||
- `Tfoo_t` — typedef'd C type `foo_t` (e.g. `Tsize_t`, `Tssize_t`)
|
||
- `Sfoo` — struct tag `foo`, `Ufoo` — union tag, `Ffield` — struct field
|
||
- `X__foo` / `X___foo` — C identifiers starting with `_` get folded; both `__errno_location` and `___errno_location` exist
|
||
- Lowercase `xfoo`, `_foo` — internal helpers
|
||
|
||
### Ignore `capi_<goos>_<goarch>.go`
|
||
|
||
They are vestigial. Very old ccgo versions consulted them years ago; they survive only so that existing code in the wild does not break, and that is their entire purpose. **Current ccgo discovers this libc's API by parsing the libc sources**, so the manifests are not consulted and not authoritative. Adding a new `Xfoo` needs no `capi_*.go` change.
|
||
|
||
Treat them as carrying **zero** information about what this libc exports today. Most were last touched in 2023 (`capi_windows_amd64.go` 2023-02-22, `capi_linux_amd64.go` 2023-06-08), so they are years stale and will disagree with reality.
|
||
|
||
In particular, do not use them as evidence when diagnosing a symbol-resolution problem — grep the actual `func X...` definitions instead. Because they are stale *uniformly*, they compare equal across libc versions that genuinely differ, so "the capi files are identical, therefore libc is not the variable" is a false conclusion. That reasoning has already sent one real investigation down the wrong path.
|
||
|
||
## TLS — thread-local state
|
||
|
||
A `*TLS` represents a C "thread". Every `Xfoo` takes one as the first argument. `TLS` is **not safe for concurrent use** — one goroutine per `TLS`.
|
||
|
||
- Inside an `Xmain`-style entry point: `Start(main)` creates the main TLS, locks the goroutine to the OS thread, and runs `main`.
|
||
- Library callers (e.g. sqlite): call `NewTLS()` per goroutine, `defer tls.Close()`. The first `NewTLS()` is the "main" thread and is *not* OS-thread-locked.
|
||
- Pthread support: `Xpthread_create` spawns a goroutine with its own TLS. The musl pthread path (`pthread_musl.go`) and the hand-written path (`pthread.go`) implement this separately.
|
||
- `tls_linux_amd64.{go,s}` — assembly stubs that let abi0-translated code reach `tls.Alloc` / `tls.Free` without going through Go calling conventions.
|
||
|
||
## Subpackages
|
||
|
||
The repo contains a constellation of subpackages under `errno/`, `fcntl/`, `pthread/`, `signal/`, `stdio/`, `stdlib/`, `sys/{types,socket,stat,vfs,random}`, `time/`, `unistd/`, `pwd/`, `grp/`, `netdb/`, `netinet/`, etc. They are thin: mostly typedefs and integer constants for use by ccgo-generated callers. They are not meant to be imported by general Go code.
|
||
|
||
`internal/archive` — declares the pinned musl version; `internal/autogen` and `internal/overlay` — supporting files for the generator.
|
||
|
||
## What is generated vs hand-written
|
||
|
||
Code generated (do not hand-edit; regenerate via `make generate`):
|
||
- `ccgo_linux_*.go` — the bulk of the musl translation, one per Linux arch
|
||
- `musl_<goos>_<goarch>.go` — older ccgo/v3 generation for non-Linux ports and some helpers
|
||
- `abi0_linux_amd64.{go,s}` — qbecc-generated abi0 wrappers
|
||
- `capi_*.go` — vestigial exported-symbol manifests. Not regenerated, not consulted by current ccgo, years stale; see "Ignore `capi_<goos>_<goarch>.go`" above. Kept only for old code in the wild.
|
||
|
||
Anything else (`libc.go`, `libc_musl.go`, `libc_<goos>.go`, `libc_unix*.go`, `mem*.go`, `pthread*.go`, `aliases.go`, etc.) is hand-written.
|
||
|
||
`generate.go` and `genheaders.go` are old (`//go:build ignore`) ccgo/v3 helpers, kept around but mostly superseded by `generator.go` (ccgo/v4).
|
||
|
||
## Debugging artifacts in the tree
|
||
|
||
Several large files in the repo root are debug logs / surface dumps (`log`, `log-test`, `log-generate`, `surface.old`, `surface.new`). They are tracked but stale; do not depend on their contents.
|