p2: digest with in-sift FIFO, SQLite caches, Honcho context, recover
Dixie is capped (max_inflight 2, queue_depth 8, queue_wait 20s) and never queued on llama-server. Streaming chat, thinking off, 45s fallback to clean. Digest and chunk-embedding caches are content-addressed SQLite (Honcho context is not cached). Recover uses the guarded Fetcher. guardian-star is refused in code.
This commit is contained in:
+15
@@ -0,0 +1,15 @@
|
||||
# This file lists authors for copyright purposes. This file is distinct from
|
||||
# the CONTRIBUTORS files. See the latter for an explanation.
|
||||
#
|
||||
# Names should be added to this file as:
|
||||
# Name or Organization <email address>
|
||||
#
|
||||
# The email address is not required for organizations.
|
||||
#
|
||||
# Please keep the list sorted.
|
||||
|
||||
Chris Waldon <christopher(dot)waldon(dot)dev@gmail.com>
|
||||
Gleb Sakhnov <gleb.sakhnov@gmail.com>
|
||||
Jan Mercl <0xjnml@gmail.com>
|
||||
Scot C Bontrager <scot@indievisible.org>
|
||||
Steffen Butzer <steffen(dot)butzer@outlook.com>
|
||||
+72
@@ -0,0 +1,72 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Overview
|
||||
|
||||
`modernc.org/memory` is a single-package, dependency-light Go library implementing a C-style `malloc`/`free` allocator on top of raw OS mappings (`mmap` on unix, `VirtualAlloc` on Windows). It exists to serve the modernc C-to-Go stack (`modernc.org/libc` and everything above it), which needs memory the Go GC neither moves nor scans, and addresses that stay valid when held only as `uintptr`.
|
||||
|
||||
The repo is flat: ten Go files, one test file. `internal/autogen/` is an empty placeholder for the modernc builder infrastructure described by `builder.json`.
|
||||
|
||||
## Commands
|
||||
|
||||
```sh
|
||||
go test # full suite, ~2m15s (each test churns a 128 MiB quota)
|
||||
go test -tags=memory.counters # same, with the accounting assertions actually live — see Build tags
|
||||
go test -run TestUMalloc # a single test
|
||||
go test -run @ -bench . -benchmem # benchmarks only ("@" matches no test name)
|
||||
make editor # gofmt -l -s -w *.go — the pre-commit step
|
||||
make build_all_targets # go build + staticcheck across every supported GOOS/GOARCH
|
||||
make cpu | make mem # bench under a profiler, then open pprof
|
||||
make todo # grep TODO/BUG/println/unused-assignment markers
|
||||
```
|
||||
|
||||
`make all` is the author's personal pipeline: it needs `misspell`, `unconvert`, `maligned` and `golint` installed and greps a file named `log` that it does not create (the convention is `go test ... |& tee log`). It usually fails locally — reach for the individual targets instead.
|
||||
|
||||
**`go vet`**: the standalone command reports dozens of `possible misuse of unsafe.Pointer` across `memory.go` and `all_test.go` and exits 1. These are inherent to the design — the allocator deliberately round-trips addresses through `uintptr` into memory the GC does not manage — and must not be "fixed". For a meaningful signal use `go vet -unsafeptr=false ./...` or `staticcheck`; both are clean.
|
||||
|
||||
## Build tags
|
||||
|
||||
- **`memory.counters`** (`counters.go` / `nocounters.go`) — enables the `Allocator.Allocs/Bytes/Mmaps` bookkeeping. Without it those fields stay 0, which silently makes the `alloc.Allocs != 0 || alloc.Mmaps != 0 || alloc.Bytes != 0` assertion ending nearly every test vacuous; only the `len(alloc.regs) != 0` leak check survives. **Any change to mmap/unmap accounting must be tested with `-tags=memory.counters`.**
|
||||
- **`memory.trace`** (`trace_enabled.go` / `trace_disabled.go`) — logs every Malloc/Free/Realloc/UsableSize call to stderr.
|
||||
|
||||
Both constants are compile-time `false` by default, so the guarded code costs nothing in normal builds.
|
||||
|
||||
## Architecture
|
||||
|
||||
Everything hangs off one invariant: **every mapping is `pageSize`-aligned (64 KiB, `pageSizeLog = 16`), so any allocated address masks down to its page header with `p &^ pageMask`.** There are no per-allocation headers and no address→metadata lookup on the hot path — `Free` and `UsableSize` recover everything from that single AND.
|
||||
|
||||
`page` (`memory.go:99`) is the four-word header at the start of every mapping: `brk` (bump index), `log` (size class), `size` (bytes actually mapped), `used` (live slots). `init()` asserts `sizeof(page) % mallocAllign == 0`.
|
||||
|
||||
Two allocation paths, discriminated by `page.log`:
|
||||
|
||||
- **`log != 0` — shared page**, for `size <= maxSlotSize` (`1 << (pageSizeLog-2)` = 16 KiB). The request rounds up to a power-of-two slot no smaller than `mallocAllign` (`2*sizeof(uintptr)`); `log` is that exponent. A page holds `cap[log] = pageAvail / (1<<log)` slots handed out by bumping `brk`; `a.pages[log]` is the current carve target and is zeroed once it fills. Freed slots go on `a.lists[log]`, an intrusive doubly-linked list whose `node{prev,next}` is written *inside the free slot* — hence the two-word minimum slot size.
|
||||
- **`log == 0` — dedicated page**, for anything larger. One allocation per mapping at `pg + headerSize`, released on `Free` (see Page retention). `log == 0` is a safe sentinel precisely because `mallocAllign` rounding makes the smallest real class `log` 4 on 64-bit, 3 on 32-bit.
|
||||
|
||||
`a.regs` is the set of live mappings. It is touched only by `mmap`/`unmap`/`Close` (and by tests as a leak check), never on the allocation fast path.
|
||||
|
||||
**Page draining** (`UintptrFree`, `memory.go:340-356`): when a shared page's `used` reaches 0, all `brk` of its slots are unlinked from the global `a.lists[log]` before the page is retained or unmapped. Skipping that walk would leave free-list nodes pointing into memory that is no longer on the list — or no longer mapped at all.
|
||||
|
||||
**Alignment costs syscalls** (`mmap_unix.go:34`): to get a 64 KiB-aligned result, `mmap` asks for `size + pageSize` and unmaps the misaligned head and the surplus tail — up to three syscalls per page acquisition. `Allocator.mmap` amortizes this for 64 KiB regions by acquiring `slabBatch` of them in one mapping and pooling the rest (unix only — `canCarve`). If the *tail* unmap is rejected by the kernel the code keeps the enlarged mapping rather than failing (fixes bigsort on linux/s390x, cznic/sqlite#207), so `page.size` may legitimately exceed what was requested. The `TODO` at `memory.go:206` is about reusing that surplus, which requires moving `cap` out of `Allocator` and into `page` so capacity becomes per-page.
|
||||
|
||||
Windows gets alignment for free — `VirtualAlloc` is 64 KiB-granular — so `mmap_windows.go` just rounds up and returns.
|
||||
|
||||
**Platform and width splits**: `pageSizeLog` is declared per platform in `mmap_unix.go` and `mmap_windows.go` (16 in both since `e95c668`). `rawmem`, the huge array type used to fabricate slices over raw memory, is declared per pointer width in `memory32.go` / `memory64.go` — its build-tag list is what needs editing when adding an architecture.
|
||||
|
||||
## API surface
|
||||
|
||||
Four families over one core, all in `memory.go`: `Malloc`/`Calloc`/`Realloc`/`Free` returning `[]byte`, the `Unsafe*` variants returning `unsafe.Pointer`, the `Uintptr*` variants, and `UsableSize`/`UnsafeUsableSize`/`UintptrUsableSize`. **The `Uintptr*` methods are the implementation; the other two families are thin wrappers** — algorithm changes belong in `UintptrMalloc`/`UintptrFree`/`UintptrRealloc` and nowhere else.
|
||||
|
||||
The zero `Allocator` is ready to use and **is not goroutine-safe** — there is no locking anywhere; callers serialize. Slice-returning calls set cap to `usableSize(p)`, which is why `Free` and `Realloc` reslice to `b[:cap(b)]` to recover the block address.
|
||||
|
||||
## Tests
|
||||
|
||||
`all_test.go` is the entire suite: randomized allocate / verify / free cycles over a 128 MiB quota, in `Small` (≤ 2× OS page, shared pages only) and `Big` (≤ 2× 64 KiB, forces the dedicated-page path) variants, across all three access styles. Sizes and contents come from `mathutil.NewFC32`, a full-cycle PRNG the tests `Seek(0)` to replay — expected values are regenerated rather than stored, and failures reproduce deterministically. Every allocation is filled with its pattern and re-verified, so slot overlap or a wrong `usableSize` surfaces as "corrupted heap". Tests read unexported state (`alloc.regs`, `page.log`), so they must stay in-package.
|
||||
|
||||
## Page retention
|
||||
|
||||
`UintptrFree` does **not** return an empty region to the OS eagerly. A drained shared page is kept as its size class's carve target; every other empty region — a drained page whose class already has a carve target, or a dedicated page — goes via `release` into `a.freed`, a pool keyed by region size and bounded by `maxFreedSize` (4 MiB plus the high-water mark of the live mapping). `Allocator.mmap` serves requests from the pool before mapping fresh, with request sizes collapsed into region classes (`mmapSize`) so pool hits are likely. `Trim` releases the carve targets and the pool; `Close` releases everything.
|
||||
|
||||
This is why every test and benchmark calls `alloc.Trim()` before asserting `Allocs`/`Mmaps`/`Bytes`/`regs` are zero. Asserting after `Close` instead would be vacuous — `Close` ends with `*a = Allocator{}`, so every field reads zero regardless.
|
||||
|
||||
`HANDOFF.md` records the diagnosis, the measurements, and the design options that were weighed before picking this policy. Its "Status: open" header is historical; note also that it undercounts the affected test sites as 8 (there are 16 — 10 using `t`, 6 using `b`).
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
# This file lists people who contributed code to this repository. The AUTHORS
|
||||
# file lists the copyright holders; this file lists people.
|
||||
#
|
||||
# Names should be added to this file like so:
|
||||
# Name <email address>
|
||||
#
|
||||
# Please keep the list sorted.
|
||||
|
||||
Anup Kodlekere <anup.kodlekere@ibm.com>
|
||||
Chris Waldon <christopher(dot)waldon(dot)dev@gmail.com>
|
||||
Gleb Sakhnov <gleb.sakhnov@gmail.com>
|
||||
Jan Mercl <0xjnml@gmail.com>
|
||||
Scot C Bontrager <scot@indievisible.org>
|
||||
Steffen Butzer <steffen(dot)butzer@outlook.com>
|
||||
ZHU Zijia <piggynl@outlook.com>
|
||||
+206
@@ -0,0 +1,206 @@
|
||||
# HANDOFF: no page-retention hysteresis in `Allocator.UintptrFree`
|
||||
|
||||
**Status:** resolved 2026-08-06. Retention is now unconditional and bounded to
|
||||
one empty page per size class — design option 1 below — combined with the
|
||||
explicit flush of option 3, `(*Allocator).Trim`. The tests keep their full
|
||||
`Allocs`/`Mmaps`/`Bytes`/`regs` assertions by calling `Trim` first; moving them
|
||||
after `Close` would have made them vacuous, since `Close` ends with
|
||||
`*a = Allocator{}`. Two corrections to what follows: the invariant is asserted
|
||||
at 16 sites, not 8 (`TestFree` and `TestMalloc` were missed, and 6 of the 16 are
|
||||
benchmarks using `b`), and retention is per size class, since a global pool
|
||||
would need the per-page capacity that the `memory.go` TODO still tracks.
|
||||
|
||||
Everything below is the original diagnosis, kept for its measurements and
|
||||
rationale. It describes the code as of `0a6f754`.
|
||||
|
||||
## The ask
|
||||
|
||||
`UintptrFree` returns a shared page to the OS the moment its last live slot is
|
||||
freed. There is no hysteresis, so an allocation pattern whose live count for a
|
||||
size class repeatedly crosses zero pays a full `mmap`/`munmap` round trip per
|
||||
turnover — in the degenerate case, **one `mmap` + three `munmap`s per single
|
||||
alloc/free pair**. Decide whether this allocator should retain empty pages, and
|
||||
under what policy.
|
||||
|
||||
## Root cause
|
||||
|
||||
`memory.go:222-241`, the tail of `UintptrFree`:
|
||||
|
||||
```go
|
||||
(*page)(unsafe.Pointer(pg)).used--
|
||||
if (*page)(unsafe.Pointer(pg)).used != 0 {
|
||||
return nil
|
||||
}
|
||||
... unlink every node of this page from a.lists[log] ...
|
||||
if a.pages[log] == pg {
|
||||
a.pages[log] = 0
|
||||
}
|
||||
...
|
||||
return a.unmap(pg) // <-- unconditional, immediate
|
||||
```
|
||||
|
||||
`used == 0` means "no live slots", which is treated as "give the page back".
|
||||
`a.pages[log]` (the current carve target) is cleared too, so the next allocation
|
||||
of that size class must `newSharedPage` → `mmap` again.
|
||||
|
||||
Two amplifiers make each turnover cost more than one syscall pair:
|
||||
|
||||
- `mmap(size)` (`mmap_unix.go:29`) asks the kernel for `size + pageSize` so it
|
||||
can align the result, then unmaps the misaligned head and the tail — up to
|
||||
**two trim `munmap`s per `mmap`**, on top of the one that releases the page.
|
||||
A 64 KiB page therefore costs a 128 KiB kernel request. Measured totals below
|
||||
show all three `munmap`s in the synthetic case and an average of two under the
|
||||
real workload.
|
||||
- With `pageSizeLog = 16` and `maxSlotSizeLog = pageSizeLog - 2`, every size
|
||||
class up to **16 KiB** uses shared pages (`memory.go:262`), so this covers the
|
||||
whole small/medium range, not just tiny allocations.
|
||||
|
||||
## Why it has not bitten before
|
||||
|
||||
Long-lived consumers (sqlite et al.) keep a working set alive, so `used` rarely
|
||||
reaches 0 for a hot size class — the page stays mapped and the free list does its
|
||||
job. The pathology needs a consumer that repeatedly *drains* a size class.
|
||||
|
||||
## Standalone reproducer
|
||||
|
||||
No other module needed. Allocate one slot and free it, in a loop:
|
||||
|
||||
```go
|
||||
a := &memory.Allocator{}
|
||||
defer a.Close()
|
||||
for i := 0; i < n; i++ {
|
||||
p, _ := a.UintptrMalloc(size)
|
||||
a.UintptrFree(p)
|
||||
}
|
||||
```
|
||||
|
||||
`n` = 200000, `strace -c -f -e trace=mmap,munmap`, AMD Ryzen 9 3900X, go1.26.5:
|
||||
|
||||
| size | build | `mmap` | `munmap` | syscall time | end state |
|
||||
|---|---|---|---|---|---|
|
||||
| 64 B | v1.11.0 | 200,022 | 600,000 | 4.33 s | `Mmaps=0 Bytes=0` |
|
||||
| 64 B | + retention | 25 | 3 | 0.0002 s | `Mmaps=1 Bytes=65536` |
|
||||
| 4096 B | v1.11.0 | 200,024 | 600,000 | 4.49 s | `Mmaps=0 Bytes=0` |
|
||||
| 4096 B | + retention | 24 | 2 | ~0 | `Mmaps=1 Bytes=65536` |
|
||||
|
||||
That is ~21.6 µs of kernel time per alloc/free pair, and it is size-independent
|
||||
across the shared-page range.
|
||||
|
||||
## The consumer that hit it
|
||||
|
||||
`modernc.org/libquickjs` v0.13.1, which tracks QuickJS release 2026-06-04
|
||||
(`bellard/quickjs` `3d5e064`). That release added a small-block arena allocator:
|
||||
31 size classes from 16 to 512 bytes, carved out of `JS_MALLOC_ARENA_SIZE`
|
||||
= **4096**-byte arenas, and it **frees an arena as soon as its last block is
|
||||
freed**. QuickJS's free-on-empty policy sits directly on top of this package's
|
||||
free-on-empty policy and the two resonate: 4096 → `log` 12 → shared page path,
|
||||
15 arenas per 64 KiB page, cycling constantly.
|
||||
|
||||
V8 `v8-v7` benchmark scores (throughput, higher is better), medians of 3
|
||||
interleaved rounds on a quiet host, go1.26.5, libc v1.74.4:
|
||||
|
||||
| bench | libquickjs v0.12.10<br>(QuickJS 2025-09-13) | libquickjs v0.13.1<br>(QuickJS 2026-06-04) | v0.13.1 + retention |
|
||||
|---|---|---|---|
|
||||
| Richards | 175.5 | 243.5 | 243 |
|
||||
| DeltaBlue | 208.5 | **16** | 265.5 |
|
||||
| Crypto | 118.5 | 157.5 | 166 |
|
||||
| RayTrace | 285.5 | **195.5** | 464.5 |
|
||||
| EarleyBoyer | 465.5 | 618.5 | 622.5 |
|
||||
| RegExp | 99.5 | 189 | 190 |
|
||||
| Splay | 911.5 | **340.5** | 1149 |
|
||||
| NavierStokes | 208.5 | 322 | 323 |
|
||||
| **Score** | **240** | **187.5** | **348.5** |
|
||||
|
||||
Retention moves *only* the three allocation-bound benchmarks and leaves the other
|
||||
five within noise (243.5→243, 189→190, 322→323), which is what makes the
|
||||
attribution airtight.
|
||||
|
||||
Syscall counts and profile for DeltaBlue alone:
|
||||
|
||||
| build | `mmap` | `munmap` | syscall time |
|
||||
|---|---|---|---|
|
||||
| v0.12.10 (no arena allocator) | 302 | 532 | 0.0001 s |
|
||||
| v0.13.1 | **1,345,702** | **2,691,328** | **21.1 s** |
|
||||
| v0.13.1 + retention | 72 | 62 | 0.0008 s |
|
||||
|
||||
```
|
||||
10.43s 71.73% internal/runtime/syscall/linux.Syscall6
|
||||
2.22s 15.27% modernc.org/memory.(*Allocator).mmap (62.79% cum)
|
||||
0.03s 0.21% golang.org/x/sys/unix.munmap (58.94% cum)
|
||||
```
|
||||
|
||||
72% of a JavaScript benchmark's runtime in `mmap`/`munmap`. Note the third
|
||||
column of the score table: with retention the *new* QuickJS overtakes the old one
|
||||
everywhere, which is the expected outcome — the arena allocator is a win once it
|
||||
stops thrashing.
|
||||
|
||||
## The experiment (proof of diagnosis, NOT a shippable patch)
|
||||
|
||||
Retain one empty page per size class by recycling it as the carve target instead
|
||||
of unmapping it:
|
||||
|
||||
```diff
|
||||
if a.pages[log] == pg {
|
||||
a.pages[log] = 0
|
||||
}
|
||||
+ if a.pages[log] == 0 {
|
||||
+ (*page)(unsafe.Pointer(pg)).brk = 0
|
||||
+ (*page)(unsafe.Pointer(pg)).used = 0
|
||||
+ a.pages[log] = pg
|
||||
+ return nil
|
||||
+ }
|
||||
if counters {
|
||||
a.Bytes -= (*page)(unsafe.Pointer(pg)).size
|
||||
}
|
||||
return a.unmap(pg)
|
||||
```
|
||||
|
||||
That is the whole change behind every "+ retention" number above. Its bound is
|
||||
one 64 KiB page per size class per allocator (≤ 64 classes → ≤ 4 MiB worst case,
|
||||
in practice only for classes actually used).
|
||||
|
||||
### Why it cannot be merged as written
|
||||
|
||||
`all_test.go` asserts, after freeing everything, at 8 sites (`:137`, `:203`,
|
||||
`:285`, `:307`, `:331`, `:401`, `:466`, `:544`):
|
||||
|
||||
```go
|
||||
if alloc.Allocs != 0 || alloc.Mmaps != 0 || alloc.Bytes != 0 || len(alloc.regs) != 0 {
|
||||
```
|
||||
|
||||
Retention breaks `Mmaps`, `Bytes` and `len(regs)` by construction — visible in
|
||||
the reproducer's end-state column, and `go test ./...` fails with the patch
|
||||
applied. **This is a policy conflict, not a leak**: `Close()` still walks
|
||||
`a.regs` and unmaps everything, so nothing escapes the allocator's lifetime. But
|
||||
the invariant "every byte is back with the OS after the last `Free`" is real,
|
||||
is asserted by those tests, and consumers may rely on it. Changing it is your
|
||||
call.
|
||||
|
||||
## Design options
|
||||
|
||||
1. **Unconditional bounded retention** (the diff above). Simplest, helps every
|
||||
consumer automatically, no API change. Costs the invariant; the tests would
|
||||
need to assert after `Close`, or a flush would need exposing.
|
||||
2. **Opt-in.** An `Allocator` field or constructor knob, default off, leaving
|
||||
existing behaviour and tests untouched so `libquickjs`/`libc` can turn it on.
|
||||
Safest, but each consumer has to know to ask.
|
||||
3. **Explicit flush.** Always retain; add `(*Allocator).Trim()` and let callers
|
||||
decide when to give pages back.
|
||||
4. **Bounded hysteresis, documented.** As (1), but with the retention count an
|
||||
explicit, documented parameter rather than incidentally "one".
|
||||
|
||||
Open questions worth settling at the same time:
|
||||
|
||||
- Per size class (as prototyped) or one global pool of empty pages? The latter
|
||||
caps total retention regardless of how many classes are hot.
|
||||
- Is the align-by-overallocating in `mmap` worth revisiting? It doubles the
|
||||
syscall cost of every page acquisition, so it amplifies exactly this problem.
|
||||
|
||||
## Provenance
|
||||
|
||||
Found while investigating why `modernc.org/libquickjs` did not show the +42% that
|
||||
QuickJS 2026-06-04 advertises. Native C on the same host gains +60.8% on v8-v7
|
||||
(+43.5% with `DIRECT_DISPATCH 0`, the configuration the ccgo build corresponds
|
||||
to); the Go build instead *regressed* 21.7%. This allocator interaction was the
|
||||
sole cause: with retention the Go build gains +45.2%, matching the native +43.5%,
|
||||
and five of the eight benchmarks had been gaining 32–91% all along.
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
Copyright (c) 2017 The Memory Authors. All rights reserved.
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions are
|
||||
met:
|
||||
|
||||
* Redistributions of source code must retain the above copyright
|
||||
notice, this list of conditions and the following disclaimer.
|
||||
* Redistributions in binary form must reproduce the above
|
||||
copyright notice, this list of conditions and the following disclaimer
|
||||
in the documentation and/or other materials provided with the
|
||||
distribution.
|
||||
* Neither the names of the authors nor the names of the
|
||||
contributors may be used to endorse or promote products derived from
|
||||
this software without specific prior written permission.
|
||||
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
||||
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
||||
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
||||
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
||||
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
||||
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
||||
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
||||
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
||||
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
||||
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
Copyright (c) 2009 The Go Authors. All rights reserved.
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions are
|
||||
met:
|
||||
|
||||
* Redistributions of source code must retain the above copyright
|
||||
notice, this list of conditions and the following disclaimer.
|
||||
* Redistributions in binary form must reproduce the above
|
||||
copyright notice, this list of conditions and the following disclaimer
|
||||
in the documentation and/or other materials provided with the
|
||||
distribution.
|
||||
* Neither the name of Google Inc. nor the names of its
|
||||
contributors may be used to endorse or promote products derived from
|
||||
this software without specific prior written permission.
|
||||
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
||||
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
||||
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
||||
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
||||
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
||||
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
||||
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
||||
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
||||
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
||||
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
+1
@@ -0,0 +1 @@
|
||||
https://commons.wikimedia.org/wiki/File:Memory_infra_logo.png
|
||||
+25
@@ -0,0 +1,25 @@
|
||||
Copyright (c) 2011, Evan Shaw <edsrzf@gmail.com>
|
||||
All rights reserved.
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions are met:
|
||||
* Redistributions of source code must retain the above copyright
|
||||
notice, this list of conditions and the following disclaimer.
|
||||
* Redistributions in binary form must reproduce the above copyright
|
||||
notice, this list of conditions and the following disclaimer in the
|
||||
documentation and/or other materials provided with the distribution.
|
||||
* Neither the name of the copyright holder nor the
|
||||
names of its contributors may be used to endorse or promote products
|
||||
derived from this software without specific prior written permission.
|
||||
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
|
||||
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
|
||||
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
||||
DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY
|
||||
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
|
||||
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
|
||||
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
|
||||
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
||||
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
|
||||
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
|
||||
+112
@@ -0,0 +1,112 @@
|
||||
# Copyright 2017 The Memory Authors. All rights reserved.
|
||||
# Use of this source code is governed by a BSD-style
|
||||
# license that can be found in the LICENSE file.
|
||||
|
||||
.PHONY: all clean cover cpu editor internalError later mem nuke todo edit build_all_targets
|
||||
|
||||
grep=--include=*.go --include=*.l --include=*.y --include=*.yy
|
||||
ngrep='TODOOK\|parser\.go\|scanner\.go\|.*_string\.go'
|
||||
|
||||
all: editor
|
||||
go vet 2>&1 | grep -v $(ngrep) || true
|
||||
golint 2>&1 | grep -v $(ngrep) || true
|
||||
make todo
|
||||
misspell *.go
|
||||
maligned || true
|
||||
unconvert -apply
|
||||
staticcheck | grep -v 'lexer\.go' || true
|
||||
grep -n 'FAIL\|PASS' log
|
||||
date
|
||||
|
||||
clean:
|
||||
go clean
|
||||
rm -f *~ *.test *.out
|
||||
|
||||
cover:
|
||||
t=$(shell tempfile) ; go test -coverprofile $$t && go tool cover -html $$t && unlink $$t
|
||||
|
||||
cpu: clean
|
||||
go test -run @ -bench . -cpuprofile cpu.out
|
||||
go tool pprof -lines *.test cpu.out
|
||||
|
||||
edit:
|
||||
@ 1>/dev/null 2>/dev/null gvim -p Makefile go.mod builder.json *.go &
|
||||
|
||||
editor:
|
||||
gofmt -l -s -w *.go
|
||||
|
||||
build_all_targets:
|
||||
GOOS=darwin GOARCH=amd64 go build
|
||||
GOOS=darwin GOARCH=amd64 staticcheck
|
||||
GOOS=darwin GOARCH=arm64 go build
|
||||
GOOS=darwin GOARCH=arm64 staticcheck
|
||||
GOOS=freebsd GOARCH=386 go build
|
||||
GOOS=freebsd GOARCH=386 staticcheck
|
||||
GOOS=freebsd GOARCH=amd64 go build
|
||||
GOOS=freebsd GOARCH=amd64 staticcheck
|
||||
GOOS=freebsd GOARCH=arm go build
|
||||
GOOS=freebsd GOARCH=arm staticcheck
|
||||
GOOS=freebsd GOARCH=arm64 go build
|
||||
GOOS=freebsd GOARCH=arm64 staticcheck
|
||||
GOOS=illumos GOARCH=amd64 go build
|
||||
GOOS=illumos GOARCH=amd64 staticcheck
|
||||
GOOS=linux GOARCH=386 go build
|
||||
GOOS=linux GOARCH=386 staticcheck
|
||||
GOOS=linux GOARCH=amd64 go build
|
||||
GOOS=linux GOARCH=amd64 staticcheck
|
||||
GOOS=linux GOARCH=arm go build
|
||||
GOOS=linux GOARCH=arm staticcheck
|
||||
GOOS=linux GOARCH=arm64 go build
|
||||
GOOS=linux GOARCH=arm64 staticcheck
|
||||
GOOS=linux GOARCH=loong64 go build
|
||||
GOOS=linux GOARCH=loong64 staticcheck
|
||||
GOOS=linux GOARCH=mips go build
|
||||
GOOS=linux GOARCH=mips staticcheck
|
||||
GOOS=linux GOARCH=mips64le go build
|
||||
GOOS=linux GOARCH=mips64le staticcheck
|
||||
GOOS=linux GOARCH=mipsle go build
|
||||
GOOS=linux GOARCH=mipsle staticcheck
|
||||
GOOS=linux GOARCH=ppc64le go build
|
||||
GOOS=linux GOARCH=ppc64le staticcheck
|
||||
GOOS=linux GOARCH=riscv64 go build
|
||||
GOOS=linux GOARCH=riscv64 staticcheck
|
||||
GOOS=linux GOARCH=s390x go build
|
||||
GOOS=linux GOARCH=s390x staticcheck
|
||||
GOOS=netbsd GOARCH=386 go build
|
||||
GOOS=netbsd GOARCH=386 staticcheck
|
||||
GOOS=netbsd GOARCH=amd64 go build
|
||||
GOOS=netbsd GOARCH=amd64 staticcheck
|
||||
GOOS=netbsd GOARCH=arm go build
|
||||
GOOS=netbsd GOARCH=arm staticcheck
|
||||
GOOS=openbsd GOARCH=386 go build
|
||||
GOOS=openbsd GOARCH=386 staticcheck
|
||||
GOOS=openbsd GOARCH=amd64 go build
|
||||
GOOS=openbsd GOARCH=amd64 staticcheck
|
||||
GOOS=openbsd GOARCH=arm64 go build
|
||||
GOOS=openbsd GOARCH=arm64 staticcheck
|
||||
GOOS=windows GOARCH=386 go build
|
||||
GOOS=windows GOARCH=386 staticcheck
|
||||
GOOS=windows GOARCH=amd64 go build
|
||||
GOOS=windows GOARCH=amd64 staticcheck
|
||||
GOOS=windows GOARCH=arm64 go build
|
||||
GOOS=windows GOARCH=arm64 staticcheck
|
||||
|
||||
internalError:
|
||||
egrep -ho '"internal error.*"' *.go | sort | cat -n
|
||||
|
||||
later:
|
||||
@grep -n $(grep) LATER * || true
|
||||
@grep -n $(grep) MAYBE * || true
|
||||
|
||||
mem: clean
|
||||
go test -run @ -bench . -memprofile mem.out -memprofilerate 1 -timeout 24h
|
||||
go tool pprof -lines -web -alloc_space *.test mem.out
|
||||
|
||||
nuke: clean
|
||||
go clean -i
|
||||
|
||||
todo:
|
||||
@grep -nr $(grep) ^[[:space:]]*_[[:space:]]*=[[:space:]][[:alpha:]][[:alnum:]]* * | grep -v $(ngrep) || true
|
||||
@grep -nr $(grep) TODO * | grep -v $(ngrep) || true
|
||||
@grep -nr $(grep) BUG * | grep -v $(ngrep) || true
|
||||
@grep -nr $(grep) [^[:alpha:]]println * | grep -v $(ngrep) || true
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
|
||||
|
||||
Package memory implements a memory allocator.
|
||||
|
||||
## Build status
|
||||
|
||||
available at https://modern-c.appspot.com/-/builder/?importpath=modernc.org%2fmemory
|
||||
|
||||
Installation
|
||||
|
||||
$ go get modernc.org/memory
|
||||
|
||||
[](https://pkg.go.dev/modernc.org/memory)
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"autogen": "<none>",
|
||||
"autoupdate": ".",
|
||||
"autotag": ".",
|
||||
"test": "."
|
||||
}
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
// Copyright 2017 The Memory Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
//go:build memory.counters
|
||||
// +build memory.counters
|
||||
|
||||
package memory // import "modernc.org/memory"
|
||||
|
||||
const counters = true
|
||||
+53
@@ -0,0 +1,53 @@
|
||||
// Copyright 2026 The Memory Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
//go:build unix
|
||||
|
||||
package memory // import "modernc.org/memory"
|
||||
|
||||
import (
|
||||
"unsafe"
|
||||
|
||||
"golang.org/x/sys/unix"
|
||||
)
|
||||
|
||||
// canDecommit gates the decommit policy: pooled regions past the hot window
|
||||
// hand their pages back to the OS.
|
||||
const canDecommit = true
|
||||
|
||||
// madviseAdvice is the advice passed for a region falling out of the hot
|
||||
// window.
|
||||
//
|
||||
// MADV_DONTNEED rather than MADV_FREE: MADV_FREE reclaims lazily, so the
|
||||
// resident set does not actually drop until the machine is under pressure
|
||||
// - measured, a 2 GiB pool still read as 2 GiB of RSS. That is the same
|
||||
// reason the Go runtime defaults to MADV_DONTNEED.
|
||||
const madviseAdvice = unix.MADV_DONTNEED
|
||||
|
||||
// decommit asks the kernel to reclaim the physical pages backing the empty
|
||||
// region at p without unmapping it. The mapping stays, and with it the address,
|
||||
// the VMA and the absence of a future mmap - that is the whole point: retention
|
||||
// keeps its syscall and map-count savings while handing the resident set back.
|
||||
//
|
||||
// MADV_DONTNEED does not change vm_flags, so this does not split the VMA.
|
||||
//
|
||||
// The first OS page is left alone because it carries the page header, so
|
||||
// nothing has to be reconstructed when the region comes back out of the pool.
|
||||
// That holds only while osPageSize < pageSize; on a kernel configured with
|
||||
// 64 KiB base pages the two are equal and decommit turns itself off rather than
|
||||
// zeroing a live header. Such hosts would instead need page.size restored from
|
||||
// the pool's map key in the reuse path.
|
||||
func decommit(p uintptr, size int) {
|
||||
if size <= osPageSize {
|
||||
return
|
||||
}
|
||||
|
||||
n := size - osPageSize
|
||||
unix.Madvise((*rawmem)(unsafe.Pointer(p + uintptr(osPageSize)))[:n:n], madviseAdvice)
|
||||
}
|
||||
|
||||
// recommit is what makes a region from the pool's decommitted prefix usable
|
||||
// again before it is handed out. On unix there is nothing to do: pages
|
||||
// discarded by MADV_DONTNEED soft-fault back zero-filled on first touch.
|
||||
func recommit(uintptr, int) error { return nil }
|
||||
+51
@@ -0,0 +1,51 @@
|
||||
// Copyright 2026 The Memory Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package memory // import "modernc.org/memory"
|
||||
|
||||
// canDecommit gates the decommit policy: pooled regions past the hot window
|
||||
// hand their pages back to the OS.
|
||||
const canDecommit = true
|
||||
|
||||
// decommit returns the physical pages and the commit charge backing the empty
|
||||
// region at p to the OS without releasing the address reservation - the
|
||||
// windows analogue of the unix madvise(MADV_DONTNEED) path. Returning the
|
||||
// commit charge matters more here than the working set: Windows does not
|
||||
// overcommit, so a drained pool that stayed committed would hold
|
||||
// pagefile-backed commit against the machine-global limit for as long as it
|
||||
// is retained.
|
||||
//
|
||||
// The first OS page stays committed because it carries the page header.
|
||||
// Unlike MADV_DONTNEED'ed pages, MEM_DECOMMIT'ed pages fault on touch instead
|
||||
// of soft-faulting zeroes, so the reuse path recommits them via recommit
|
||||
// before the region is handed out; the pool's cold watermark is what knows
|
||||
// when that is needed.
|
||||
//
|
||||
// A failed VirtualFree leaves the pages committed, which is safe: the region
|
||||
// is merely counted as decommitted, and recommit of already-committed pages
|
||||
// succeeds and preserves them.
|
||||
func decommit(p uintptr, size int) {
|
||||
if size <= osPageSize {
|
||||
return
|
||||
}
|
||||
|
||||
procVirtualFree.Call(p+uintptr(osPageSize), uintptr(size-osPageSize), _MEM_DECOMMIT)
|
||||
}
|
||||
|
||||
// recommit commits, zero-filled, the pages decommit returned, making a region
|
||||
// popped from the pool's decommitted prefix usable again. It can fail when
|
||||
// the machine is out of commit; the caller then leaves the pool untouched and
|
||||
// reports the failure.
|
||||
func recommit(p uintptr, size int) error {
|
||||
if size <= osPageSize {
|
||||
return nil
|
||||
}
|
||||
|
||||
addr, _, err := procVirtualAlloc.Call(p+uintptr(osPageSize), uintptr(size-osPageSize), _MEM_COMMIT, _PAGE_READWRITE)
|
||||
if addr == 0 {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
BIN
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
+685
@@ -0,0 +1,685 @@
|
||||
// Copyright 2017 The Memory Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
// Package memory implements a memory allocator.
|
||||
//
|
||||
// # Build status
|
||||
//
|
||||
// available at https://modern-c.appspot.com/-/builder/?importpath=modernc.org%2fmemory
|
||||
//
|
||||
// # Changelog
|
||||
//
|
||||
// 2017-10-03 Added alternative, unsafe.Pointer-based API.
|
||||
//
|
||||
// Package memory implements a memory allocator.
|
||||
//
|
||||
// # Changelog
|
||||
//
|
||||
// 2017-10-03 Added alternative, unsafe.Pointer-based API.
|
||||
//
|
||||
// # Page retention
|
||||
//
|
||||
// Memory is acquired from the OS in 64 KiB units or, for larger
|
||||
// allocations, in units rounded up to a power of two on 64-bit hosts.
|
||||
// An Allocator retains empty regions for reuse instead of unmapping
|
||||
// them as soon as their last slot is freed: each size class keeps its
|
||||
// current carve-target page, and beyond that, retired regions of any
|
||||
// size go into a pool bounded by 4 MiB plus the high-water mark of the
|
||||
// live mapping. An allocation pattern that repeatedly drains a size
|
||||
// class, or repeatedly frees and reallocates large buffers, no longer
|
||||
// pays a mmap/munmap round trip per turnover. Use Allocator.Trim to
|
||||
// hand the retained regions back.
|
||||
//
|
||||
// # Benchmarks
|
||||
//
|
||||
// jnml@3900x:~/src/modernc.org/memory$ date ; go version ; go test -run @ -bench . -benchmem |& tee log
|
||||
// Mon Sep 25 16:02:02 CEST 2023
|
||||
// go version go1.21.1 linux/amd64
|
||||
// goos: linux
|
||||
// goarch: amd64
|
||||
// pkg: modernc.org/memory
|
||||
// cpu: AMD Ryzen 9 3900X 12-Core Processor
|
||||
// BenchmarkFree16-24 123506772 9.802 ns/op 0 B/op 0 allocs/op
|
||||
// BenchmarkFree32-24 73853230 15.08 ns/op 0 B/op 0 allocs/op
|
||||
// BenchmarkFree64-24 43070334 25.15 ns/op 0 B/op 0 allocs/op
|
||||
// BenchmarkCalloc16-24 59353304 18.92 ns/op 0 B/op 0 allocs/op
|
||||
// BenchmarkCalloc32-24 39415004 29.00 ns/op 0 B/op 0 allocs/op
|
||||
// BenchmarkCalloc64-24 35825725 32.02 ns/op 0 B/op 0 allocs/op
|
||||
// BenchmarkGoCalloc16-24 38274313 26.99 ns/op 16 B/op 1 allocs/op
|
||||
// BenchmarkGoCalloc32-24 44590477 33.06 ns/op 32 B/op 1 allocs/op
|
||||
// BenchmarkGoCalloc64-24 44233016 37.20 ns/op 64 B/op 1 allocs/op
|
||||
// BenchmarkMalloc16-24 145736911 7.720 ns/op 0 B/op 0 allocs/op
|
||||
// BenchmarkMalloc32-24 128898334 7.887 ns/op 0 B/op 0 allocs/op
|
||||
// BenchmarkMalloc64-24 149569483 7.994 ns/op 0 B/op 0 allocs/op
|
||||
// BenchmarkUintptrFree16-24 117043012 9.205 ns/op 0 B/op 0 allocs/op
|
||||
// BenchmarkUintptrFree32-24 77399617 14.20 ns/op 0 B/op 0 allocs/op
|
||||
// BenchmarkUintptrFree64-24 48770785 25.04 ns/op 0 B/op 0 allocs/op
|
||||
// BenchmarkUintptrCalloc16-24 79257636 15.44 ns/op 0 B/op 0 allocs/op
|
||||
// BenchmarkUintptrCalloc32-24 49644562 23.62 ns/op 0 B/op 0 allocs/op
|
||||
// BenchmarkUintptrCalloc64-24 39854710 28.22 ns/op 0 B/op 0 allocs/op
|
||||
// BenchmarkUintptrMalloc16-24 252987727 4.525 ns/op 0 B/op 0 allocs/op
|
||||
// BenchmarkUintptrMalloc32-24 241423840 4.433 ns/op 0 B/op 0 allocs/op
|
||||
// BenchmarkUintptrMalloc64-24 256450324 4.669 ns/op 0 B/op 0 allocs/op
|
||||
// PASS
|
||||
// ok modernc.org/memory 93.178s
|
||||
// jnml@3900x:~/src/modernc.org/memory$
|
||||
package memory // import "modernc.org/memory"
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math/bits"
|
||||
"os"
|
||||
"unsafe"
|
||||
)
|
||||
|
||||
const (
|
||||
headerSize = unsafe.Sizeof(page{})
|
||||
mallocAllign = 2 * unsafe.Sizeof(uintptr(0))
|
||||
maxSlotSize = 1 << maxSlotSizeLog
|
||||
maxSlotSizeLog = pageSizeLog - 2
|
||||
pageAvail = pageSize - headerSize
|
||||
pageMask = pageSize - 1
|
||||
pageSize = 1 << pageSizeLog
|
||||
)
|
||||
|
||||
func init() {
|
||||
if unsafe.Sizeof(page{})%mallocAllign != 0 {
|
||||
panic("internal error")
|
||||
}
|
||||
}
|
||||
|
||||
// if n%m != 0 { n += m-n%m }. m must be a power of 2.
|
||||
func roundup(n, m int) int { return (n + m - 1) &^ (m - 1) }
|
||||
|
||||
type node struct {
|
||||
prev, next uintptr // *node
|
||||
}
|
||||
|
||||
type page struct {
|
||||
brk int
|
||||
log uint
|
||||
size int
|
||||
used int
|
||||
}
|
||||
|
||||
// Allocator allocates and frees memory. Its zero value is ready for use. The
|
||||
// exported counters are updated only when build tag memory.counters is
|
||||
// present.
|
||||
//
|
||||
// An Allocator retains empty regions for reuse, see Trim.
|
||||
type Allocator struct {
|
||||
Allocs int // # of allocs.
|
||||
Bytes int // Asked from OS.
|
||||
cap [64]int
|
||||
lists [64]uintptr // *node
|
||||
Mmaps int // Asked from OS.
|
||||
pages [64]uintptr // *page
|
||||
regs map[uintptr]struct{} // map[*page]struct{}
|
||||
freed map[int][]uintptr // empty regions retained for reuse, keyed by region size
|
||||
freedSize int // total bytes in freed
|
||||
cold map[int]int // the bottom cold[size] regions of freed[size] are decommitted
|
||||
live int // bytes mapped and not in freed
|
||||
hiLive int // high-water mark of live
|
||||
}
|
||||
|
||||
// maxFreedSize returns the bound on the total bytes retained in
|
||||
// a.freed: a fixed floor plus the high-water mark of the live mapping.
|
||||
// The high-water mark rather than the current live size matters for
|
||||
// large transient buffers: when one is freed, live has already shrunk
|
||||
// by the buffer, and it is exactly the region most worth retaining.
|
||||
func (a *Allocator) maxFreedSize() int { return 4<<20 + a.hiLive }
|
||||
|
||||
// addLive adds n bytes to the live mapping size, tracking its
|
||||
// high-water mark.
|
||||
func (a *Allocator) addLive(n int) {
|
||||
a.live += n
|
||||
if a.live > a.hiLive {
|
||||
a.hiLive = a.live
|
||||
}
|
||||
}
|
||||
|
||||
// canRetain reports whether a.freed has room to retain size more bytes.
|
||||
func (a *Allocator) canRetain(size int) bool { return a.freedSize+size <= a.maxFreedSize() }
|
||||
|
||||
// mmapSize returns the region size class that backs a request for size
|
||||
// bytes: requests up to pageSize share the single pageSize class, and
|
||||
// on 64-bit larger requests are rounded up to a power of two. The
|
||||
// extra address space is never touched and so costs nothing, and
|
||||
// collapsing sizes into classes makes reuse via a.freed much more
|
||||
// likely.
|
||||
//
|
||||
// Classes are multiples of mmapGranularity - the granularity the platform
|
||||
// mmap actually maps at - so the class of a request agrees with the
|
||||
// page.size the mapping comes back with, which is what retain later keys
|
||||
// the pool by. Rounding to anything finer describes a mapping the platform
|
||||
// cannot make: on windows/386 a 1 MiB+1 request rounded to OS pages misses
|
||||
// forever the 1 MiB+64 KiB region its own mapping put into the pool.
|
||||
func mmapSize(size int) int {
|
||||
switch {
|
||||
case size <= pageSize && pageSize%osPageSize == 0:
|
||||
return pageSize
|
||||
case size > pageSize && bits.UintSize == 64:
|
||||
return roundup(1<<bits.Len(uint(size-1)), mmapGranularity)
|
||||
default:
|
||||
return roundup(size, mmapGranularity)
|
||||
}
|
||||
}
|
||||
|
||||
// slabBatch is how many pageSize regions a single mapping acquires
|
||||
// when a pageSize region is needed and the pool is empty.
|
||||
const slabBatch = 16
|
||||
|
||||
func (a *Allocator) mmap(size int) (uintptr /* *page */, error) {
|
||||
size = mmapSize(size)
|
||||
if s := a.freed[size]; len(s) != 0 {
|
||||
p := s[len(s)-1]
|
||||
if canDecommit {
|
||||
if c := a.cold[size]; c >= len(s) { // popping from the decommitted prefix
|
||||
if err := recommit(p, size); err != nil {
|
||||
return 0, err // pool left untouched
|
||||
}
|
||||
|
||||
a.cold[size] = len(s) - 1
|
||||
}
|
||||
}
|
||||
a.freed[size] = s[:len(s)-1]
|
||||
a.freedSize -= size
|
||||
a.addLive(size)
|
||||
pg := (*page)(unsafe.Pointer(p))
|
||||
pg.brk = 0
|
||||
pg.used = 0
|
||||
return p, nil
|
||||
}
|
||||
|
||||
// Aligning a fresh mapping to a pageSize boundary costs up to two
|
||||
// munmap calls for the trimmed-off ends, so when a single pageSize
|
||||
// region is needed, acquire a batch in one mapping and put the
|
||||
// rest into the pool: three syscalls amortized over slabBatch
|
||||
// regions instead of per region.
|
||||
if canCarve && size == pageSize {
|
||||
if p, n, err := mmap(slabBatch * pageSize); err == nil {
|
||||
if n%pageSize == 0 && a.canRetain(n-pageSize) {
|
||||
for off := pageSize; off < n; off += pageSize {
|
||||
q := p + uintptr(off)
|
||||
a.reg(q, pageSize)
|
||||
a.retain(q, pageSize)
|
||||
}
|
||||
return a.reg(p, pageSize), nil
|
||||
}
|
||||
|
||||
if err := unmap(p, n); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
}
|
||||
// Fall through and map a single region.
|
||||
}
|
||||
|
||||
p, size, err := mmap(size)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
//TODO(jnml) The returned size may now be nearly as twice as large as we asked
|
||||
//for. Use that extra capacity. For that we need to move the respective
|
||||
//Allocator.cap item into the page struct so the page cap becomes dynamic.
|
||||
//
|
||||
// Related: This is a consequence of fixing the bigsort.test failures on
|
||||
// linux/s390x, see: https://gitlab.com/cznic/sqlite/-/issues/207
|
||||
return a.reg(p, size), nil
|
||||
}
|
||||
|
||||
// reg registers the freshly mapped size-byte region at p.
|
||||
func (a *Allocator) reg(p uintptr, size int) uintptr {
|
||||
a.addLive(size)
|
||||
if counters {
|
||||
a.Mmaps++
|
||||
a.Bytes += size
|
||||
}
|
||||
if a.regs == nil {
|
||||
a.regs = map[uintptr]struct{}{}
|
||||
}
|
||||
(*page)(unsafe.Pointer(p)).size = size
|
||||
a.regs[p] = struct{}{}
|
||||
return p
|
||||
}
|
||||
|
||||
func (a *Allocator) newPage(size int) (uintptr /* *page */, error) {
|
||||
size += int(headerSize)
|
||||
p, err := a.mmap(size)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
(*page)(unsafe.Pointer(p)).log = 0
|
||||
return p, nil
|
||||
}
|
||||
|
||||
func (a *Allocator) newSharedPage(log uint) (uintptr /* *page */, error) {
|
||||
if a.cap[log] == 0 {
|
||||
a.cap[log] = int(pageAvail) / (1 << log)
|
||||
}
|
||||
size := int(headerSize) + a.cap[log]<<log
|
||||
p, err := a.mmap(size)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
a.pages[log] = p
|
||||
(*page)(unsafe.Pointer(p)).log = log
|
||||
return p, nil
|
||||
}
|
||||
|
||||
func (a *Allocator) unmap(p uintptr /* *page */) error {
|
||||
delete(a.regs, p)
|
||||
size := (*page)(unsafe.Pointer(p)).size
|
||||
if counters {
|
||||
a.Mmaps--
|
||||
a.Bytes -= size
|
||||
}
|
||||
return unmap(p, size)
|
||||
}
|
||||
|
||||
// release retires the empty region at p: it is retained in a.freed for
|
||||
// reuse, unless the retained total is at its bound, in which case the
|
||||
// region is returned to the OS. Retained regions stay in a.regs and in
|
||||
// the counters; Trim and Close return them to the OS.
|
||||
func (a *Allocator) release(p uintptr /* *page */) error {
|
||||
size := (*page)(unsafe.Pointer(p)).size
|
||||
if a.canRetain(size) {
|
||||
a.retain(p, size)
|
||||
return nil
|
||||
}
|
||||
|
||||
a.live -= size
|
||||
return a.unmap(p)
|
||||
}
|
||||
|
||||
// hotBytes is how much of the retained pool stays committed per size class.
|
||||
// 4 MiB costs nothing measurable against not decommitting at all (speedtest1
|
||||
// and production simulations) while returning the hoard above it.
|
||||
const hotBytes = 4 << 20
|
||||
|
||||
// retain puts the empty size-byte region at p into a.freed.
|
||||
func (a *Allocator) retain(p uintptr, size int) {
|
||||
if a.freed == nil {
|
||||
a.freed = map[int][]uintptr{}
|
||||
}
|
||||
a.freed[size] = append(a.freed[size], p)
|
||||
a.freedSize += size
|
||||
a.live -= size
|
||||
if !canDecommit {
|
||||
return
|
||||
}
|
||||
|
||||
// Keep the newest hotBytes worth of this class committed: those are the
|
||||
// regions a churning workload takes straight back, where decommitting only
|
||||
// buys a page fault. What falls out of that window is hoard - it is holding
|
||||
// a resident set the workload has stopped using - so hand its pages back
|
||||
// while keeping the mapping.
|
||||
//
|
||||
// a.cold counts the stack's already-decommitted bottom, so only a region
|
||||
// newly falling out of the hot window pays a syscall: a pool oscillating
|
||||
// around the boundary would otherwise re-madvise the same region on every
|
||||
// retain.
|
||||
hot := hotBytes / size
|
||||
if hot < 1 {
|
||||
hot = 1
|
||||
}
|
||||
if s := a.freed[size]; len(s)-a.cold[size] > hot {
|
||||
if a.cold == nil {
|
||||
a.cold = map[int]int{}
|
||||
}
|
||||
decommit(s[a.cold[size]], size)
|
||||
a.cold[size]++
|
||||
}
|
||||
}
|
||||
|
||||
// UintptrCalloc is like Calloc except it returns an uintptr.
|
||||
func (a *Allocator) UintptrCalloc(size int) (r uintptr, err error) {
|
||||
if trace {
|
||||
defer func() {
|
||||
fmt.Fprintf(os.Stderr, "Calloc(%#x) %#x, %v\n", size, r, err)
|
||||
}()
|
||||
}
|
||||
if r, err = a.UintptrMalloc(size); r == 0 || err != nil {
|
||||
return 0, err
|
||||
}
|
||||
b := ((*rawmem)(unsafe.Pointer(r)))[:size:size]
|
||||
for i := range b {
|
||||
b[i] = 0
|
||||
}
|
||||
return r, nil
|
||||
}
|
||||
|
||||
// UintptrFree is like Free except its argument is an uintptr, which must have
|
||||
// been acquired from UintptrCalloc or UintptrMalloc or UintptrRealloc.
|
||||
func (a *Allocator) UintptrFree(p uintptr) (err error) {
|
||||
if trace {
|
||||
defer func() {
|
||||
fmt.Fprintf(os.Stderr, "Free(%#x) %v\n", p, err)
|
||||
}()
|
||||
}
|
||||
if p == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
if counters {
|
||||
a.Allocs--
|
||||
}
|
||||
pg := p &^ uintptr(pageMask)
|
||||
log := (*page)(unsafe.Pointer(pg)).log
|
||||
if log == 0 {
|
||||
return a.release(pg)
|
||||
}
|
||||
|
||||
(*node)(unsafe.Pointer(p)).prev = 0
|
||||
(*node)(unsafe.Pointer(p)).next = a.lists[log]
|
||||
if next := (*node)(unsafe.Pointer(p)).next; next != 0 {
|
||||
(*node)(unsafe.Pointer(next)).prev = p
|
||||
}
|
||||
a.lists[log] = p
|
||||
(*page)(unsafe.Pointer(pg)).used--
|
||||
if (*page)(unsafe.Pointer(pg)).used != 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
for i := 0; i < (*page)(unsafe.Pointer(pg)).brk; i++ {
|
||||
n := pg + headerSize + uintptr(i)<<log
|
||||
next := (*node)(unsafe.Pointer(n)).next
|
||||
prev := (*node)(unsafe.Pointer(n)).prev
|
||||
switch {
|
||||
case prev == 0:
|
||||
a.lists[log] = next
|
||||
if next != 0 {
|
||||
(*node)(unsafe.Pointer(next)).prev = 0
|
||||
}
|
||||
case next == 0:
|
||||
(*node)(unsafe.Pointer(prev)).next = 0
|
||||
default:
|
||||
(*node)(unsafe.Pointer(prev)).next = next
|
||||
(*node)(unsafe.Pointer(next)).prev = prev
|
||||
}
|
||||
}
|
||||
|
||||
// The page is now empty. Retain it as the carve target of its size class
|
||||
// rather than returning it to the OS, unless the class already has one.
|
||||
// Otherwise a pattern that repeatedly drains the class pays a mmap/munmap
|
||||
// round trip per turnover. Trim releases the pages retained this way.
|
||||
if a.pages[log] == 0 || a.pages[log] == pg {
|
||||
(*page)(unsafe.Pointer(pg)).brk = 0
|
||||
a.pages[log] = pg
|
||||
return nil
|
||||
}
|
||||
|
||||
return a.release(pg)
|
||||
}
|
||||
|
||||
// UintptrMalloc is like Malloc except it returns an uinptr.
|
||||
func (a *Allocator) UintptrMalloc(size int) (r uintptr, err error) {
|
||||
if trace {
|
||||
defer func() {
|
||||
fmt.Fprintf(os.Stderr, "Malloc(%#x) %#x, %v\n", size, r, err)
|
||||
}()
|
||||
}
|
||||
if size < 0 {
|
||||
panic("invalid malloc size")
|
||||
}
|
||||
|
||||
if size == 0 {
|
||||
return 0, nil
|
||||
}
|
||||
if size > int(^uint(0)>>1)-int(headerSize) {
|
||||
return 0, fmt.Errorf("memory: allocation size %d is too large", size)
|
||||
}
|
||||
|
||||
if counters {
|
||||
a.Allocs++
|
||||
}
|
||||
log := uint(bits.Len(uint((size+int(mallocAllign)-1)&^int(mallocAllign-1) - 1)))
|
||||
if log > maxSlotSizeLog {
|
||||
p, err := a.newPage(size)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
return p + headerSize, nil
|
||||
}
|
||||
|
||||
if a.lists[log] == 0 && a.pages[log] == 0 {
|
||||
if _, err := a.newSharedPage(log); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
}
|
||||
|
||||
if p := a.pages[log]; p != 0 {
|
||||
(*page)(unsafe.Pointer(p)).used++
|
||||
(*page)(unsafe.Pointer(p)).brk++
|
||||
if (*page)(unsafe.Pointer(p)).brk == a.cap[log] {
|
||||
a.pages[log] = 0
|
||||
}
|
||||
return p + headerSize + uintptr((*page)(unsafe.Pointer(p)).brk-1)<<log, nil
|
||||
}
|
||||
|
||||
n := a.lists[log]
|
||||
p := n &^ uintptr(pageMask)
|
||||
a.lists[log] = (*node)(unsafe.Pointer(n)).next
|
||||
if next := (*node)(unsafe.Pointer(n)).next; next != 0 {
|
||||
(*node)(unsafe.Pointer(next)).prev = 0
|
||||
}
|
||||
(*page)(unsafe.Pointer(p)).used++
|
||||
return n, nil
|
||||
}
|
||||
|
||||
// UintptrRealloc is like Realloc except its first argument is an uintptr,
|
||||
// which must have been returned from UintptrCalloc, UintptrMalloc or
|
||||
// UintptrRealloc.
|
||||
func (a *Allocator) UintptrRealloc(p uintptr, size int) (r uintptr, err error) {
|
||||
if trace {
|
||||
defer func() {
|
||||
fmt.Fprintf(os.Stderr, "UnsafeRealloc(%#x, %#x) %#x, %v\n", p, size, r, err)
|
||||
}()
|
||||
}
|
||||
switch {
|
||||
case p == 0:
|
||||
return a.UintptrMalloc(size)
|
||||
case size == 0 && p != 0:
|
||||
return 0, a.UintptrFree(p)
|
||||
}
|
||||
|
||||
us := UintptrUsableSize(p)
|
||||
if us >= size {
|
||||
return p, nil
|
||||
}
|
||||
|
||||
if r, err = a.UintptrMalloc(size); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
if us < size {
|
||||
size = us
|
||||
}
|
||||
copy((*rawmem)(unsafe.Pointer(r))[:size:size], (*rawmem)(unsafe.Pointer(p))[:size:size])
|
||||
return r, a.UintptrFree(p)
|
||||
}
|
||||
|
||||
// UintptrUsableSize is like UsableSize except its argument is an uintptr,
|
||||
// which must have been returned from UintptrCalloc, UintptrMalloc or
|
||||
// UintptrRealloc.
|
||||
func UintptrUsableSize(p uintptr) (r int) {
|
||||
if trace {
|
||||
defer func() {
|
||||
fmt.Fprintf(os.Stderr, "UsableSize(%#x) %#x\n", p, r)
|
||||
}()
|
||||
}
|
||||
if p == 0 {
|
||||
return 0
|
||||
}
|
||||
|
||||
return usableSize(p)
|
||||
}
|
||||
|
||||
func usableSize(p uintptr) (r int) {
|
||||
pg := p &^ uintptr(pageMask)
|
||||
if log := (*page)(unsafe.Pointer(pg)).log; log != 0 {
|
||||
return 1 << log
|
||||
}
|
||||
|
||||
return (*page)(unsafe.Pointer(pg)).size - int(headerSize)
|
||||
}
|
||||
|
||||
// Calloc is like Malloc except the allocated memory is zeroed.
|
||||
func (a *Allocator) Calloc(size int) (r []byte, err error) {
|
||||
p, err := a.UintptrCalloc(size)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return (*rawmem)(unsafe.Pointer(p))[:size:usableSize(p)], nil
|
||||
}
|
||||
|
||||
// Close releases all OS resources used by a and sets it to its zero value.
|
||||
//
|
||||
// It's not necessary to Close the Allocator when exiting a process.
|
||||
func (a *Allocator) Close() (err error) {
|
||||
for p := range a.regs {
|
||||
if e := a.unmap(p); e != nil && err == nil {
|
||||
err = e
|
||||
}
|
||||
}
|
||||
*a = Allocator{}
|
||||
return err
|
||||
}
|
||||
|
||||
// Free deallocates memory (as in C.free). The argument of Free must have been
|
||||
// acquired from Calloc or Malloc or Realloc.
|
||||
func (a *Allocator) Free(b []byte) (err error) {
|
||||
if b = b[:cap(b)]; len(b) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
return a.UintptrFree(uintptr(unsafe.Pointer(&b[0])))
|
||||
}
|
||||
|
||||
// Malloc allocates size bytes and returns a byte slice of the allocated
|
||||
// memory. The memory is not initialized. Malloc panics for size < 0 and
|
||||
// returns (nil, nil) for zero size.
|
||||
//
|
||||
// It's ok to reslice the returned slice but the result of appending to it
|
||||
// cannot be passed to Free or Realloc as it may refer to a different backing
|
||||
// array afterwards.
|
||||
func (a *Allocator) Malloc(size int) (r []byte, err error) {
|
||||
p, err := a.UintptrMalloc(size)
|
||||
if p == 0 || err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return (*rawmem)(unsafe.Pointer(p))[:size:usableSize(p)], nil
|
||||
}
|
||||
|
||||
// Realloc changes the size of the backing array of b to size bytes or returns
|
||||
// an error, if any. The contents will be unchanged in the range from the
|
||||
// start of the region up to the minimum of the old and new sizes. If the
|
||||
// new size is larger than the old size, the added memory will not be
|
||||
// initialized. If b's backing array is of zero size, then the call is
|
||||
// equivalent to Malloc(size), for all values of size; if size is equal to
|
||||
// zero, and b's backing array is not of zero size, then the call is equivalent
|
||||
// to Free(b). Unless b's backing array is of zero size, it must have been
|
||||
// returned by an earlier call to Malloc, Calloc or Realloc. If the area
|
||||
// pointed to was moved, a Free(b) is done.
|
||||
func (a *Allocator) Realloc(b []byte, size int) (r []byte, err error) {
|
||||
var p uintptr
|
||||
if b = b[:cap(b)]; len(b) != 0 {
|
||||
p = uintptr(unsafe.Pointer(&b[0]))
|
||||
}
|
||||
if p, err = a.UintptrRealloc(p, size); p == 0 || err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return (*rawmem)(unsafe.Pointer(p))[:size:usableSize(p)], nil
|
||||
}
|
||||
|
||||
// Trim returns to the OS the empty regions a retains for reuse: each
|
||||
// size class's empty carve-target page and the bounded pool of retired
|
||||
// regions. Pages still having live allocations are not affected and a
|
||||
// stays ready for use.
|
||||
//
|
||||
// Trim is never necessary for correctness. It trades a smaller resident set
|
||||
// now for the cost of mapping those pages again later.
|
||||
func (a *Allocator) Trim() (err error) {
|
||||
if trace {
|
||||
defer func() {
|
||||
fmt.Fprintf(os.Stderr, "Trim() %v\n", err)
|
||||
}()
|
||||
}
|
||||
for log := range a.pages {
|
||||
pg := a.pages[log]
|
||||
if pg == 0 || (*page)(unsafe.Pointer(pg)).used != 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
a.pages[log] = 0
|
||||
a.live -= (*page)(unsafe.Pointer(pg)).size
|
||||
if e := a.unmap(pg); e != nil && err == nil {
|
||||
err = e
|
||||
}
|
||||
}
|
||||
for _, s := range a.freed {
|
||||
for _, pg := range s {
|
||||
if e := a.unmap(pg); e != nil && err == nil {
|
||||
err = e
|
||||
}
|
||||
}
|
||||
}
|
||||
a.freed = nil
|
||||
a.freedSize = 0
|
||||
a.cold = nil
|
||||
return err
|
||||
}
|
||||
|
||||
// UsableSize reports the size of the memory block allocated at p, which must
|
||||
// point to the first byte of a slice returned from Calloc, Malloc or Realloc.
|
||||
// The allocated memory block size can be larger than the size originally
|
||||
// requested from Calloc, Malloc or Realloc.
|
||||
func UsableSize(p *byte) (r int) { return UintptrUsableSize(uintptr(unsafe.Pointer(p))) }
|
||||
|
||||
// UnsafeCalloc is like Calloc except it returns an unsafe.Pointer.
|
||||
func (a *Allocator) UnsafeCalloc(size int) (r unsafe.Pointer, err error) {
|
||||
p, err := a.UintptrCalloc(size)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return unsafe.Pointer(p), nil
|
||||
}
|
||||
|
||||
// UnsafeFree is like Free except its argument is an unsafe.Pointer, which must
|
||||
// have been acquired from UnsafeCalloc or UnsafeMalloc or UnsafeRealloc.
|
||||
func (a *Allocator) UnsafeFree(p unsafe.Pointer) (err error) { return a.UintptrFree(uintptr(p)) }
|
||||
|
||||
// UnsafeMalloc is like Malloc except it returns an unsafe.Pointer.
|
||||
func (a *Allocator) UnsafeMalloc(size int) (r unsafe.Pointer, err error) {
|
||||
p, err := a.UintptrMalloc(size)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return unsafe.Pointer(p), nil
|
||||
}
|
||||
|
||||
// UnsafeRealloc is like Realloc except its first argument is an
|
||||
// unsafe.Pointer, which must have been returned from UnsafeCalloc,
|
||||
// UnsafeMalloc or UnsafeRealloc.
|
||||
func (a *Allocator) UnsafeRealloc(p unsafe.Pointer, size int) (r unsafe.Pointer, err error) {
|
||||
q, err := a.UintptrRealloc(uintptr(p), size)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return unsafe.Pointer(q), nil
|
||||
}
|
||||
|
||||
// UnsafeUsableSize is like UsableSize except its argument is an
|
||||
// unsafe.Pointer, which must have been returned from UnsafeCalloc,
|
||||
// UnsafeMalloc or UnsafeRealloc.
|
||||
func UnsafeUsableSize(p unsafe.Pointer) (r int) { return UintptrUsableSize(uintptr(p)) }
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
// Copyright 2018 The Memory Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
//go:build 386 || arm || armbe || mips || mipsle || ppc || s390 || s390x || sparc
|
||||
// +build 386 arm armbe mips mipsle ppc s390 s390x sparc
|
||||
|
||||
package memory // import "modernc.org/memory"
|
||||
|
||||
type rawmem [1<<31 - 2]byte
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
// Copyright 2018 The Memory Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
//go:build amd64 || amd64p32 || arm64 || arm64be || mips64 || mips64le || mips64p32 || mips64p32le || ppc64 || ppc64le || sparc64 || riscv64 || loong64
|
||||
// +build amd64 amd64p32 arm64 arm64be mips64 mips64le mips64p32 mips64p32le ppc64 ppc64le sparc64 riscv64 loong64
|
||||
|
||||
package memory // import "modernc.org/memory"
|
||||
|
||||
type rawmem [1<<50 - 1]byte
|
||||
+79
@@ -0,0 +1,79 @@
|
||||
// Copyright 2011 Evan Shaw. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE-MMAP-GO file.
|
||||
|
||||
//go:build unix
|
||||
|
||||
// Modifications (c) 2017 The Memory Authors.
|
||||
|
||||
package memory // import "modernc.org/memory"
|
||||
|
||||
import (
|
||||
"golang.org/x/sys/unix"
|
||||
"os"
|
||||
"unsafe"
|
||||
)
|
||||
|
||||
const pageSizeLog = 16
|
||||
|
||||
// canCarve reports that unmap can release any page-aligned sub-range
|
||||
// of a mapping, so a large mapping can be carved into independently
|
||||
// releasable pageSize regions.
|
||||
const canCarve = true
|
||||
|
||||
var (
|
||||
osPageMask = osPageSize - 1
|
||||
osPageSize = os.Getpagesize()
|
||||
|
||||
// mmapGranularity is the granularity mmap in this file actually maps
|
||||
// at: exactly what was asked, rounded to OS pages.
|
||||
mmapGranularity = osPageSize
|
||||
)
|
||||
|
||||
func unmap(addr uintptr, size int) error {
|
||||
return unix.MunmapPtr(unsafe.Pointer(addr), uintptr(size))
|
||||
}
|
||||
|
||||
// pageSize aligned.
|
||||
func mmap(size int) (uintptr, int, error) {
|
||||
size = roundup(size, osPageSize)
|
||||
// Ask for more so we can align the result at a pageSize boundary
|
||||
n := size + pageSize
|
||||
up, err := unix.MmapPtr(-1, 0, nil, uintptr(n), unix.PROT_READ|unix.PROT_WRITE, unix.MAP_PRIVATE|unix.MAP_ANON)
|
||||
if err != nil {
|
||||
return 0, 0, err
|
||||
}
|
||||
|
||||
p := uintptr(up)
|
||||
if p&uintptr(osPageMask) != 0 {
|
||||
panic("internal error")
|
||||
}
|
||||
|
||||
mod := int(p) & pageMask
|
||||
if mod != 0 { // Return the extra part before pageSize aligned block
|
||||
m := pageSize - mod
|
||||
if err := unmap(p, m); err != nil {
|
||||
unmap(p, n) // Do not leak the first mmap
|
||||
return 0, 0, err
|
||||
}
|
||||
|
||||
n -= m
|
||||
p += uintptr(m)
|
||||
}
|
||||
|
||||
if p&uintptr(pageMask) != 0 {
|
||||
panic("internal error")
|
||||
}
|
||||
|
||||
if n > size { // Return the extra part after pageSize aligned block
|
||||
if err := unmap(p+uintptr(size), n-size); err != nil {
|
||||
// Do not error when the kernel rejects the extra part after, just return the
|
||||
// unexpectedly enlarged size.
|
||||
//
|
||||
// Fixes the bigsort.test failures on linux/s390x, see: https://gitlab.com/cznic/sqlite/-/issues/207
|
||||
size = n
|
||||
}
|
||||
}
|
||||
|
||||
return p, size, nil
|
||||
}
|
||||
+58
@@ -0,0 +1,58 @@
|
||||
// Copyright 2017 The Memory Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package memory // import "modernc.org/memory"
|
||||
|
||||
import (
|
||||
syscall "golang.org/x/sys/windows"
|
||||
"os"
|
||||
)
|
||||
|
||||
const (
|
||||
_MEM_COMMIT = 0x1000
|
||||
_MEM_RESERVE = 0x2000
|
||||
_MEM_DECOMMIT = 0x4000
|
||||
_MEM_RELEASE = 0x8000
|
||||
|
||||
_PAGE_READWRITE = 0x0004
|
||||
_PAGE_NOACCESS = 0x0001
|
||||
)
|
||||
|
||||
const pageSizeLog = 16
|
||||
|
||||
// canCarve is false on Windows: VirtualFree with MEM_RELEASE frees
|
||||
// only whole VirtualAlloc allocations, never a sub-range.
|
||||
const canCarve = false
|
||||
|
||||
// mmapGranularity is the granularity mmap in this file actually maps at:
|
||||
// VirtualAlloc reserves at 64 KiB granularity, so mmap rounds every request
|
||||
// up to a pageSize multiple and page.size records that.
|
||||
const mmapGranularity = pageSize
|
||||
|
||||
var (
|
||||
modkernel32 = syscall.NewLazySystemDLL("kernel32.dll")
|
||||
osPageMask = osPageSize - 1
|
||||
osPageSize = os.Getpagesize()
|
||||
procVirtualAlloc = modkernel32.NewProc("VirtualAlloc")
|
||||
procVirtualFree = modkernel32.NewProc("VirtualFree")
|
||||
)
|
||||
|
||||
// pageSize aligned.
|
||||
func mmap(size int) (uintptr, int, error) {
|
||||
size = roundup(size, pageSize)
|
||||
addr, _, err := procVirtualAlloc.Call(0, uintptr(size), _MEM_COMMIT|_MEM_RESERVE, _PAGE_READWRITE)
|
||||
if err.(syscall.Errno) != 0 || addr == 0 {
|
||||
return addr, size, err
|
||||
}
|
||||
return addr, size, nil
|
||||
}
|
||||
|
||||
func unmap(addr uintptr, size int) error {
|
||||
r, _, err := procVirtualFree.Call(addr, 0, _MEM_RELEASE)
|
||||
if r == 0 {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
// Copyright 2017 The Memory Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
//go:build !memory.counters
|
||||
// +build !memory.counters
|
||||
|
||||
package memory // import "modernc.org/memory"
|
||||
|
||||
const counters = false
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
// Copyright 2017 The Memory Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
//go:build !memory.trace
|
||||
// +build !memory.trace
|
||||
|
||||
package memory // import "modernc.org/memory"
|
||||
|
||||
const trace = false
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
// Copyright 2017 The Memory Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
//go:build memory.trace
|
||||
// +build memory.trace
|
||||
|
||||
package memory // import "modernc.org/memory"
|
||||
|
||||
const trace = true
|
||||
Reference in New Issue
Block a user