From 0ce3bfc641acb88306db50b5c15b6aec34e444b1 Mon Sep 17 00:00:00 2001 From: "K. Isom" Date: Thu, 24 Sep 2026 13:51:06 -0700 Subject: [PATCH] Research: prompt-injection defences for persisted memory; the M5a check refuses all default-ignorable code points The research feeds M5b's sanitizer. One finding is a bug in the M5a draft: category Cf misses variation selectors, the combining grapheme joiner, and the Hangul and Braille fillers. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../2026-09-24-prompt-injection-memory.md | 168 ++++++++++++++++++ docs/specs/2026-09-24-m5a-local-memory.md | 8 +- 2 files changed, 174 insertions(+), 2 deletions(-) create mode 100644 docs/research/2026-09-24-prompt-injection-memory.md diff --git a/docs/research/2026-09-24-prompt-injection-memory.md b/docs/research/2026-09-24-prompt-injection-memory.md new file mode 100644 index 0000000..9f5d133 --- /dev/null +++ b/docs/research/2026-09-24-prompt-injection-memory.md @@ -0,0 +1,168 @@ +# Prompt-injection defences for persisted memory + +Research of 2026-09-24, for the sanitizer module planned as M5b hardening (`docs/decisions.md`, +row of that date). Sources were read that day; hermes-agent was read at commit e8acdc5 and tirith +at c18753a. Every claim below has its source. What could not be checked is listed at the end. + +## 1. Attacks on persisted memory + +- **Stored instructions.** Injected text gets the agent to save a standing instruction. + - SpAIware wrote one into ChatGPT's memory. It leaked every later chat through rendered markdown + images ([embracethered, 2024](https://embracethered.com/blog/posts/2024/chatgpt-macos-app-persistent-data-exfiltration/)). + - The Gemini attack delays the write: a hidden instruction waits until the user says "yes" or + "sure" ([embracethered, 2025](https://embracethered.com/blog/posts/2025/gemini-memory-persistence-prompt-injection/)). + - OWASP names the class ASI06, Memory & Context Poisoning + ([OWASP agentic top 10](https://genai.owasp.org/2025/12/09/owasp-top-10-for-agentic-applications-the-benchmark-for-agentic-security-in-the-age-of-autonomous-ai/)). +- **Injection through queries alone.** MINJA reached 98.2% injection success by chatting with the + agent and nothing else ([arXiv 2503.03704](https://arxiv.org/abs/2503.03704)). The rate is + lower when the agent already holds realistic amounts of legitimate memory + ([arXiv 2601.05504](https://arxiv.org/abs/2601.05504)). +- **Retrieval backdoors.** AgentPoison: over 80% success at under 0.1% poisoning + ([arXiv 2407.12784](https://arxiv.org/abs/2407.12784)). This matters less for identity files, + which are always in the prompt rather than retrieved. It matters for `recall`. +- **Poisoned procedures and reasoning.** + - MemoryGraft plants fake "successful experiences" + ([arXiv 2512.16962](https://arxiv.org/abs/2512.16962)). + - FARMA forges reasoning in wording that passes keyword checks, with up to 100% success + ([arXiv 2607.05029](https://arxiv.org/abs/2607.05029)). +- **Plain false facts.** A write-time screen that caught 83.2% of injections rejected none of 360 + poisoned facts ([arXiv 2608.21230](https://arxiv.org/abs/2608.21230)). Existing injection + defences do not cover memory poisoning (MPBench, + [arXiv 2606.04329](https://arxiv.org/abs/2606.04329)). +- **Invisible payloads.** + - Unicode tag characters (the U+E0000 block) carry ASCII text that models read + ([embracethered](https://embracethered.com/blog/posts/2024/hiding-and-finding-text-with-unicode-tags/)). + - Variation selectors can hide bytes behind an emoji + ([FireTail](https://www.firetail.ai/blog/peek-a-boo-emoji-smuggling-and-modern-llms)). + +Text in the system prompt has the highest privilege under an instruction hierarchy +([arXiv 2404.13208](https://arxiv.org/abs/2404.13208)), and memory placed there inherits it. That +is why identity files must be written only by clean sessions. + +## 2. Checks on the text itself + +- **M5a's category `Cf` rule has gaps.** Checked with Unicode 16 data, none of these are `Cf`: + - variation selectors U+FE00–FE0F and U+E0100–E01EF (`Mn`); + - the combining grapheme joiner U+034F (`Mn`); + - Hangul fillers U+3164 and U+115F (`Lo`); + - Braille blank U+2800 (`So`); + - U+2028 (`Zl`). + + The fix is to refuse the `Default_Ignorable_Code_Point` property, plus an explicit list: those + fillers, `Zl`/`Zp`, private use, noncharacters and unassigned code points. +- **Character tricks defeat classifiers.** They evaded six classifier guardrails, including + Prompt Guard and Azure Prompt Shield, up to 100% of the time + ([arXiv 2504.11168](https://arxiv.org/abs/2504.11168)). Refusing those characters before any + other check closes that route. +- **Scan several normalised forms.** tirith scans the raw text and forms derived from it: + - invisible characters stripped; + - NFKC; + - a confusable skeleton (UTS #39); + - spaced-out letters collapsed; + - a bounded leetspeak fold; + - short base64 and hex decoded. + + Source: tirith, `crates/tirith-core/src/deobfuscate.rs`. tirith is AGPL-3.0 or commercial + ([repository](https://github.com/sheeki03/tirith)): reimplement the ideas, do not copy the code. + NFKC does not fold Cyrillic look-alikes. The skeleton does; it comes from the `unicode-security` + crate, MIT/Apache-2.0 ([docs.rs](https://docs.rs/unicode-security/latest/unicode_security/)). +- **The most reusable pattern set is hermes-agent's `tools/threat_patterns.py` (MIT).** It is + applied to NFKC text in three scopes. Its "strict" scope, used for memory, adds: + - SSH and agent-config persistence; + - sending data to URLs; + - dumping the conversation; + - hardcoded secrets. + + Its rule is to match attack behaviour, not bossy English, to keep false positives down + ([source](https://github.com/NousResearch/hermes-agent/blob/main/tools/threat_patterns.py)). +- **Phrase lists miss paraphrase (FARMA) and false facts (2608.21230).** They are a tripwire, not + a boundary. + +## 3. Classifier models + +| Detector | Size, licence | Measured | Weakness | +|---|---|---|---| +| [Llama Prompt Guard 2](https://huggingface.co/meta-llama/Llama-Prompt-Guard-2-86M) | 22M/86M, Llama 4 Community, 512-token window | 97.5% (86M) and 88.7% (22M) recall at 1% false positives, Meta's own evaluation | adaptive attacks succeed over 90% ([arXiv 2510.09023](https://arxiv.org/html/2510.09023)) | +| [ProtectAI deberta-v3-base v2](https://huggingface.co/protectai/deberta-v3-base-prompt-injection-v2) | 0.2B, Apache-2.0, archived | 95.25% accuracy | English only; false positives on system-prompt-like text; adaptive over 90% | +| [PIGuard](https://github.com/leolee99/PIGuard) ([arXiv 2410.22770](https://arxiv.org/abs/2410.22770)) | MIT | 30.8% better than prior guards on NotInject (benign text with trigger words) | adaptive 71% | +| [DataSentinel](https://arxiv.org/abs/2504.11358) | Mistral-7B fine-tune | false positives near 0 | adaptive about 80% | +| Known-answer check with a general model | any | high error both ways | false negatives up to 0.93 under adaptive attack | + +Memory entries are often imperative preferences ("always reply in UTC"): exactly the benign text +that trips classifiers. Nothing measures a 9B general model, such as ornith-1.5-9b, as a judge. + +## 4. Design-level defences + +- **Taint and information-flow control have the strongest evidence.** + - FIDES tracks integrity labels dynamically ([arXiv 2505.23643](https://arxiv.org/abs/2505.23643)). + - CaMeL solves 77% of AgentDojo tasks with provable security, against 84% undefended + ([arXiv 2503.18813](https://arxiv.org/abs/2503.18813)). + - A catalogue of design patterns is in [arXiv 2506.08837](https://arxiv.org/abs/2506.08837). + + Boxmaker's session taint is a coarse form of these. +- **Provenance and lineage.** + - MemLineage blocks actions whose justification traces to external sources, reaching 0% attack + success on its workloads ([arXiv 2605.14421](https://arxiv.org/abs/2605.14421)). + - HMAC-signed writes stop unsigned injection ([arXiv 2606.12703](https://arxiv.org/abs/2606.12703)). + - Ranking recall by provenance alone did not help (p=0.80; 2608.21230). +- **Spotlighting (marking where text came from).** Attack success fell from over 50% to under 2% + ([arXiv 2403.14720](https://arxiv.org/abs/2403.14720)), but adaptive attacks exceed 95%. It is + cheap framing, not a boundary. +- **Models trained against injection.** StruQ, SecAlign and Meta SecAlign + ([arXiv 2507.02735](https://arxiv.org/abs/2507.02735)) change the model itself, which is outside + the harness's control. Meta SecAlign still showed 96% adaptive success. +- **Consensus voting.** A-MemGuard ([arXiv 2510.02373](https://arxiv.org/abs/2510.02373)) is + defeated by FARMA. + +## 5. Recommendations for the sanitizer, ranked + +1. **Taint as the main control.** + - It must stay sticky for the whole session, including across compaction (M5c). A write + triggered later in a session that read a document (the Gemini pattern) must still become an + untrusted note. + - The journal keeps each entry's provenance. + + Boxmaker's taint is already sticky within a session. M5c must carry it across epochs. +2. **Refuse invisible and unusual code points.** The set is: + - `Default_Ignorable_Code_Point`; + - control characters other than `\n` and `\t`; + - `Zl` and `Zp`, private use, unassigned code points, noncharacters; + - all variation selectors; + - the Hangul and Braille fillers. + + Refuse rather than strip, so the owner and the model see the same bytes. The cost is emoji + joined sequences and flags, which is acceptable for memory. +3. **Scan normalised forms** (raw text, NFKC with casefold, the UTS #39 skeleton, whitespace + collapsed, leet folded) against the patterns. Refuse single words that mix scripts. Store the + original text in NFC. +4. **A wider markup check.** + - Take the special tokens from Ornith's own tokenizer at startup, rather than a hand list. + - Add the generic forms: `<|…|>`, `[INST]`, `<>`, ``, and lines starting + `system:`, `assistant:` or `user:`. + - Also check the skeleton form. +5. **Adopt hermes-agent's strict patterns** (MIT, with attribution), keeping its rule of matching + attack behaviour, not bossy English. +6. **Limits on structure.** + - Refuse markdown images and links, raw HTML, code fences and headings. SpAIware leaked + through a markdown image, and Mattermost renders markdown. + - Refuse base64 or hex runs of 32 characters or more, and values shaped like secrets. +7. **Show the owner what was written.** A periodic digest or diff is the only defence in the + evidence against false facts and paraphrase. This is at odds with the decision that writes are + journalled and not announced; it is a question for the owner, not a build item. +8. **A classifier only as an advisory flag, later.** The costs: + - an ONNX runtime to embed; + - Prompt Guard's licence; + - false positives on preferences; + - over 70% adaptive bypass. + + Do not gate writes on a 9B model as a judge; nothing measures it. + +## Not verified + +- The full text of OWASP ASI06's mitigations. +- Per-model NotInject numbers. +- PIGuard's size. +- Whether hermes-agent scans memory again when loading it. +- Ornith's special-token list. +- The per-defence adaptive figures come from a summary of arXiv 2510.09023's HTML page, not its + tables. diff --git a/docs/specs/2026-09-24-m5a-local-memory.md b/docs/specs/2026-09-24-m5a-local-memory.md index e97d971..9a38f55 100644 --- a/docs/specs/2026-09-24-m5a-local-memory.md +++ b/docs/specs/2026-09-24-m5a-local-memory.md @@ -211,8 +211,12 @@ the file fails safely: every `old` must match an entry of the file as it is now, if it contains: 1. a control character other than newline and tab (C0, DEL, C1); -2. an invisible or direction-changing character (Unicode general category `Cf`: zero-width - spaces and joiners, bidirectional controls, and the like); +2. an invisible or direction-changing character: any code point with the Unicode property + `Default_Ignorable_Code_Point` (zero-width spaces and joiners, bidirectional controls, + variation selectors, tag characters U+E0000 to U+E007F, and the like), and U+034F, U+115F, + U+3164, U+2800, U+2028 and U+2029, which are invisible without being in that property or in + category `Cf` (`docs/research/2026-09-24-prompt-injection-memory.md`, section 2). The table + is fixed in the code, from Unicode 16 data; 3. chat-template markup: `<|` or `|>`, or `