Research: prompt-injection defences for persisted memory; the M5a check refuses all default-ignorable code points

The research feeds M5b's sanitizer. One finding is a bug in the M5a draft: category Cf misses variation selectors, the combining grapheme joiner, and the Hangul and Braille fillers. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-24 13:51:06 -07:00
parent 64a5931354
commit 0ce3bfc641
2 changed files with 174 additions and 2 deletions
@@ -0,0 +1,168 @@
 # Prompt-injection defences for persisted memory
 Research of 2026-09-24, for the sanitizer module planned as M5b hardening (`docs/decisions.md`,
 row of that date). Sources were read that day; hermes-agent was read at commit e8acdc5 and tirith
 at c18753a. Every claim below has its source. What could not be checked is listed at the end.
 ## 1. Attacks on persisted memory
 - **Stored instructions.** Injected text gets the agent to save a standing instruction.
  - SpAIware wrote one into ChatGPT's memory. It leaked every later chat through rendered markdown
    images ([embracethered, 2024](https://embracethered.com/blog/posts/2024/chatgpt-macos-app-persistent-data-exfiltration/)).
  - The Gemini attack delays the write: a hidden instruction waits until the user says "yes" or
    "sure" ([embracethered, 2025](https://embracethered.com/blog/posts/2025/gemini-memory-persistence-prompt-injection/)).
  - OWASP names the class ASI06, Memory & Context Poisoning
    ([OWASP agentic top 10](https://genai.owasp.org/2025/12/09/owasp-top-10-for-agentic-applications-the-benchmark-for-agentic-security-in-the-age-of-autonomous-ai/)).
 - **Injection through queries alone.** MINJA reached 98.2% injection success by chatting with the
  agent and nothing else ([arXiv 2503.03704](https://arxiv.org/abs/2503.03704)). The rate is
  lower when the agent already holds realistic amounts of legitimate memory
  ([arXiv 2601.05504](https://arxiv.org/abs/2601.05504)).
 - **Retrieval backdoors.** AgentPoison: over 80% success at under 0.1% poisoning
  ([arXiv 2407.12784](https://arxiv.org/abs/2407.12784)). This matters less for identity files,
  which are always in the prompt rather than retrieved. It matters for `recall`.
 - **Poisoned procedures and reasoning.**
  - MemoryGraft plants fake "successful experiences"
    ([arXiv 2512.16962](https://arxiv.org/abs/2512.16962)).
  - FARMA forges reasoning in wording that passes keyword checks, with up to 100% success
    ([arXiv 2607.05029](https://arxiv.org/abs/2607.05029)).
 - **Plain false facts.** A write-time screen that caught 83.2% of injections rejected none of 360
  poisoned facts ([arXiv 2608.21230](https://arxiv.org/abs/2608.21230)). Existing injection
  defences do not cover memory poisoning (MPBench,
  [arXiv 2606.04329](https://arxiv.org/abs/2606.04329)).
 - **Invisible payloads.**
  - Unicode tag characters (the U+E0000 block) carry ASCII text that models read
    ([embracethered](https://embracethered.com/blog/posts/2024/hiding-and-finding-text-with-unicode-tags/)).
  - Variation selectors can hide bytes behind an emoji
    ([FireTail](https://www.firetail.ai/blog/peek-a-boo-emoji-smuggling-and-modern-llms)).
 Text in the system prompt has the highest privilege under an instruction hierarchy
 ([arXiv 2404.13208](https://arxiv.org/abs/2404.13208)), and memory placed there inherits it. That
 is why identity files must be written only by clean sessions.
 ## 2. Checks on the text itself
 - **M5a's category `Cf` rule has gaps.** Checked with Unicode 16 data, none of these are `Cf`:
  - variation selectors U+FE00–FE0F and U+E0100–E01EF (`Mn`);
  - the combining grapheme joiner U+034F (`Mn`);
  - Hangul fillers U+3164 and U+115F (`Lo`);
  - Braille blank U+2800 (`So`);
  - U+2028 (`Zl`).
  The fix is to refuse the `Default_Ignorable_Code_Point` property, plus an explicit list: those
  fillers, `Zl`/`Zp`, private use, noncharacters and unassigned code points.
 - **Character tricks defeat classifiers.** They evaded six classifier guardrails, including
  Prompt Guard and Azure Prompt Shield, up to 100% of the time
  ([arXiv 2504.11168](https://arxiv.org/abs/2504.11168)). Refusing those characters before any
  other check closes that route.
 - **Scan several normalised forms.** tirith scans the raw text and forms derived from it:
  - invisible characters stripped;
  - NFKC;
  - a confusable skeleton (UTS #39);
  - spaced-out letters collapsed;
  - a bounded leetspeak fold;
  - short base64 and hex decoded.
  Source: tirith, `crates/tirith-core/src/deobfuscate.rs`. tirith is AGPL-3.0 or commercial
  ([repository](https://github.com/sheeki03/tirith)): reimplement the ideas, do not copy the code.
  NFKC does not fold Cyrillic look-alikes. The skeleton does; it comes from the `unicode-security`
  crate, MIT/Apache-2.0 ([docs.rs](https://docs.rs/unicode-security/latest/unicode_security/)).
 - **The most reusable pattern set is hermes-agent's `tools/threat_patterns.py` (MIT).** It is
  applied to NFKC text in three scopes. Its "strict" scope, used for memory, adds:
  - SSH and agent-config persistence;
  - sending data to URLs;
  - dumping the conversation;
  - hardcoded secrets.
  Its rule is to match attack behaviour, not bossy English, to keep false positives down
  ([source](https://github.com/NousResearch/hermes-agent/blob/main/tools/threat_patterns.py)).
 - **Phrase lists miss paraphrase (FARMA) and false facts (2608.21230).** They are a tripwire, not
  a boundary.
 ## 3. Classifier models
 | Detector | Size, licence | Measured | Weakness |
 |---|---|---|---|
 | [Llama Prompt Guard 2](https://huggingface.co/meta-llama/Llama-Prompt-Guard-2-86M) | 22M/86M, Llama 4 Community, 512-token window | 97.5% (86M) and 88.7% (22M) recall at 1% false positives, Meta's own evaluation | adaptive attacks succeed over 90% ([arXiv 2510.09023](https://arxiv.org/html/2510.09023)) |
 | [ProtectAI deberta-v3-base v2](https://huggingface.co/protectai/deberta-v3-base-prompt-injection-v2) | 0.2B, Apache-2.0, archived | 95.25% accuracy | English only; false positives on system-prompt-like text; adaptive over 90% |
 | [PIGuard](https://github.com/leolee99/PIGuard) ([arXiv 2410.22770](https://arxiv.org/abs/2410.22770)) | MIT | 30.8% better than prior guards on NotInject (benign text with trigger words) | adaptive 71% |
 | [DataSentinel](https://arxiv.org/abs/2504.11358) | Mistral-7B fine-tune | false positives near 0 | adaptive about 80% |
 | Known-answer check with a general model | any | high error both ways | false negatives up to 0.93 under adaptive attack |
 Memory entries are often imperative preferences ("always reply in UTC"): exactly the benign text
 that trips classifiers. Nothing measures a 9B general model, such as ornith-1.5-9b, as a judge.
 ## 4. Design-level defences
 - **Taint and information-flow control have the strongest evidence.**
  - FIDES tracks integrity labels dynamically ([arXiv 2505.23643](https://arxiv.org/abs/2505.23643)).
  - CaMeL solves 77% of AgentDojo tasks with provable security, against 84% undefended
    ([arXiv 2503.18813](https://arxiv.org/abs/2503.18813)).
  - A catalogue of design patterns is in [arXiv 2506.08837](https://arxiv.org/abs/2506.08837).
  Boxmaker's session taint is a coarse form of these.
 - **Provenance and lineage.**
  - MemLineage blocks actions whose justification traces to external sources, reaching 0% attack
    success on its workloads ([arXiv 2605.14421](https://arxiv.org/abs/2605.14421)).
  - HMAC-signed writes stop unsigned injection ([arXiv 2606.12703](https://arxiv.org/abs/2606.12703)).
  - Ranking recall by provenance alone did not help (p=0.80; 2608.21230).
 - **Spotlighting (marking where text came from).** Attack success fell from over 50% to under 2%
  ([arXiv 2403.14720](https://arxiv.org/abs/2403.14720)), but adaptive attacks exceed 95%. It is
  cheap framing, not a boundary.
 - **Models trained against injection.** StruQ, SecAlign and Meta SecAlign
  ([arXiv 2507.02735](https://arxiv.org/abs/2507.02735)) change the model itself, which is outside
  the harness's control. Meta SecAlign still showed 96% adaptive success.
 - **Consensus voting.** A-MemGuard ([arXiv 2510.02373](https://arxiv.org/abs/2510.02373)) is
  defeated by FARMA.
 ## 5. Recommendations for the sanitizer, ranked
 1. **Taint as the main control.**
   - It must stay sticky for the whole session, including across compaction (M5c). A write
     triggered later in a session that read a document (the Gemini pattern) must still become an
     untrusted note.
   - The journal keeps each entry's provenance.
   Boxmaker's taint is already sticky within a session. M5c must carry it across epochs.
 2. **Refuse invisible and unusual code points.** The set is:
   - `Default_Ignorable_Code_Point`;
   - control characters other than `\n` and `\t`;
   - `Zl` and `Zp`, private use, unassigned code points, noncharacters;
   - all variation selectors;
   - the Hangul and Braille fillers.
   Refuse rather than strip, so the owner and the model see the same bytes. The cost is emoji
   joined sequences and flags, which is acceptable for memory.
 3. **Scan normalised forms** (raw text, NFKC with casefold, the UTS #39 skeleton, whitespace
   collapsed, leet folded) against the patterns. Refuse single words that mix scripts. Store the
   original text in NFC.
 4. **A wider markup check.**
   - Take the special tokens from Ornith's own tokenizer at startup, rather than a hand list.
   - Add the generic forms: `<|…|>`, `[INST]`, `<<SYS>>`, `<start_of_turn>`, and lines starting
     `system:`, `assistant:` or `user:`.
   - Also check the skeleton form.
 5. **Adopt hermes-agent's strict patterns** (MIT, with attribution), keeping its rule of matching
   attack behaviour, not bossy English.
 6. **Limits on structure.**
   - Refuse markdown images and links, raw HTML, code fences and headings. SpAIware leaked
     through a markdown image, and Mattermost renders markdown.
   - Refuse base64 or hex runs of 32 characters or more, and values shaped like secrets.
 7. **Show the owner what was written.** A periodic digest or diff is the only defence in the
   evidence against false facts and paraphrase. This is at odds with the decision that writes are
   journalled and not announced; it is a question for the owner, not a build item.
 8. **A classifier only as an advisory flag, later.** The costs:
   - an ONNX runtime to embed;
   - Prompt Guard's licence;
   - false positives on preferences;
   - over 70% adaptive bypass.
   Do not gate writes on a 9B model as a judge; nothing measures it.
 ## Not verified
 - The full text of OWASP ASI06's mitigations.
 - Per-model NotInject numbers.
 - PIGuard's size.
 - Whether hermes-agent scans memory again when loading it.
 - Ornith's special-token list.
 - The per-defence adaptive figures come from a summary of arXiv 2510.09023's HTML page, not its
  tables.
@@ -211,8 +211,12 @@ the file fails safely: every `old` must match an entry of the file as it is now,
 if it contains:
 1. a control character other than newline and tab (C0, DEL, C1);
-2. an invisible or direction-changing character (Unicode general category `Cf`: zero-width
+2. an invisible or direction-changing character: any code point with the Unicode property
-   spaces and joiners, bidirectional controls, and the like);
+   `Default_Ignorable_Code_Point` (zero-width spaces and joiners, bidirectional controls,
   variation selectors, tag characters U+E0000 to U+E007F, and the like), and U+034F, U+115F,
   U+3164, U+2800, U+2028 and U+2029, which are invisible without being in that property or in
   category `Cf` (`docs/research/2026-09-24-prompt-injection-memory.md`, section 2). The table
   is fixed in the code, from Unicode 16 data;
 3. chat-template markup: `<|` or `|>`, or `<think`, `</think`, `<tool_call`, `</tool_call`;
 4. one of these phrases, ignoring case and runs of spaces: `ignore previous instructions`,
   `ignore all previous`, `disregard previous instructions`, `disregard all previous`,