boxmaker/docs/research/2026-09-24-prompt-injection-memory.md

Prompt-injection defences for persisted memory

Research of 2026-09-24, for the sanitizer module planned as M5b hardening (docs/decisions.md, row of that date). Sources were read that day; hermes-agent was read at commit e8acdc5 and tirith at c18753a. Every claim below has its source. What could not be checked is listed at the end.

1. Attacks on persisted memory

• Stored instructions. Injected text gets the agent to save a standing instruction.
  • SpAIware wrote one into ChatGPT's memory. It leaked every later chat through rendered markdown images (embracethered, 2024).
  • The Gemini attack delays the write: a hidden instruction waits until the user says "yes" or "sure" (embracethered, 2025).
  • OWASP names the class ASI06, Memory & Context Poisoning (OWASP agentic top 10).
• Injection through queries alone. MINJA reached 98.2% injection success by chatting with the agent and nothing else (arXiv 2503.03704). The rate is lower when the agent already holds realistic amounts of legitimate memory (arXiv 2601.05504).
• Retrieval backdoors. AgentPoison: over 80% success at under 0.1% poisoning (arXiv 2407.12784). This matters less for identity files, which are always in the prompt rather than retrieved. It matters for recall.
• Poisoned procedures and reasoning.
  • MemoryGraft plants fake "successful experiences" (arXiv 2512.16962).
  • FARMA forges reasoning in wording that passes keyword checks, with up to 100% success (arXiv 2607.05029).
• Plain false facts. A write-time screen that caught 83.2% of injections rejected none of 360 poisoned facts (arXiv 2608.21230). Existing injection defences do not cover memory poisoning (MPBench, arXiv 2606.04329).
• Invisible payloads.
  • Unicode tag characters (the U+E0000 block) carry ASCII text that models read (embracethered).
  • Variation selectors can hide bytes behind an emoji (FireTail).

Text in the system prompt has the highest privilege under an instruction hierarchy (arXiv 2404.13208), and memory placed there inherits it. That is why identity files must be written only by clean sessions.

2. Checks on the text itself

• M5a's category Cf rule has gaps. Checked with Unicode 16 data, none of these are Cf:

  • variation selectors U+FE00–FE0F and U+E0100–E01EF (Mn);
  • the combining grapheme joiner U+034F (Mn);
  • Hangul fillers U+3164 and U+115F (Lo);
  • Braille blank U+2800 (So);
  • U+2028 (Zl).

  The fix is to refuse the Default_Ignorable_Code_Point property, plus an explicit list: those fillers, Zl/Zp, private use, noncharacters and unassigned code points.

• Character tricks defeat classifiers. They evaded six classifier guardrails, including Prompt Guard and Azure Prompt Shield, up to 100% of the time (arXiv 2504.11168). Refusing those characters before any other check closes that route.

• Scan several normalised forms. tirith scans the raw text and forms derived from it:

  • invisible characters stripped;
  • NFKC;
  • a confusable skeleton (UTS #39);
  • spaced-out letters collapsed;
  • a bounded leetspeak fold;
  • short base64 and hex decoded.

  Source: tirith, crates/tirith-core/src/deobfuscate.rs. tirith is AGPL-3.0 or commercial (repository): reimplement the ideas, do not copy the code. NFKC does not fold Cyrillic look-alikes. The skeleton does; it comes from the unicode-security crate, MIT/Apache-2.0 (docs.rs).

• The most reusable pattern set is hermes-agent's tools/threat_patterns.py (MIT). It is applied to NFKC text in three scopes. Its "strict" scope, used for memory, adds:

  • SSH and agent-config persistence;
  • sending data to URLs;
  • dumping the conversation;
  • hardcoded secrets.

  Its rule is to match attack behaviour, not bossy English, to keep false positives down (source).

• Phrase lists miss paraphrase (FARMA) and false facts (2608.21230). They are a tripwire, not a boundary.

3. Classifier models

Detector	Size, licence	Measured	Weakness
Llama Prompt Guard 2	22M/86M, Llama 4 Community, 512-token window	97.5% (86M) and 88.7% (22M) recall at 1% false positives, Meta's own evaluation	adaptive attacks succeed over 90% (arXiv 2510.09023)
ProtectAI deberta-v3-base v2	0.2B, Apache-2.0, archived	95.25% accuracy	English only; false positives on system-prompt-like text; adaptive over 90%
PIGuard (arXiv 2410.22770)	MIT	30.8% better than prior guards on NotInject (benign text with trigger words)	adaptive 71%
DataSentinel	Mistral-7B fine-tune	false positives near 0	adaptive about 80%
Known-answer check with a general model	any	high error both ways	false negatives up to 0.93 under adaptive attack

Memory entries are often imperative preferences ("always reply in UTC"): exactly the benign text that trips classifiers. Nothing measures a 9B general model, such as ornith-1.5-9b, as a judge.

4. Design-level defences

• Taint and information-flow control have the strongest evidence.

  • FIDES tracks integrity labels dynamically (arXiv 2505.23643).
  • CaMeL solves 77% of AgentDojo tasks with provable security, against 84% undefended (arXiv 2503.18813).
  • A catalogue of design patterns is in arXiv 2506.08837.

  Boxmaker's session taint is a coarse form of these.

• Provenance and lineage.

  • MemLineage blocks actions whose justification traces to external sources, reaching 0% attack success on its workloads (arXiv 2605.14421).
  • HMAC-signed writes stop unsigned injection (arXiv 2606.12703).
  • Ranking recall by provenance alone did not help (p=0.80; 2608.21230).

• Spotlighting (marking where text came from). Attack success fell from over 50% to under 2% (arXiv 2403.14720), but adaptive attacks exceed 95%. It is cheap framing, not a boundary.

• Models trained against injection. StruQ, SecAlign and Meta SecAlign (arXiv 2507.02735) change the model itself, which is outside the harness's control. Meta SecAlign still showed 96% adaptive success.

• Consensus voting. A-MemGuard (arXiv 2510.02373) is defeated by FARMA.

5. Recommendations for the sanitizer, ranked

1. Taint as the main control.

   • It must stay sticky for the whole session, including across compaction (M5c). A write triggered later in a session that read a document (the Gemini pattern) must still become an untrusted note.
   • The journal keeps each entry's provenance.

   Boxmaker's taint is already sticky within a session. M5c must carry it across epochs.

2. Refuse invisible and unusual code points. The set is:

   • Default_Ignorable_Code_Point;
   • control characters other than \n and \t;
   • Zl and Zp, private use, unassigned code points, noncharacters;
   • all variation selectors;
   • the Hangul and Braille fillers.

   Refuse rather than strip, so the owner and the model see the same bytes. The cost is emoji joined sequences and flags, which is acceptable for memory.

3. Scan normalised forms (raw text, NFKC with casefold, the UTS #39 skeleton, whitespace collapsed, leet folded) against the patterns. Refuse single words that mix scripts. Store the original text in NFC.

4. A wider markup check.

   • Take the special tokens from Ornith's own tokenizer at startup, rather than a hand list.
   • Add the generic forms: <|…|>, [INST], <<SYS>>, <start_of_turn>, and lines starting system:, assistant: or user:.
   • Also check the skeleton form.

5. Adopt hermes-agent's strict patterns (MIT, with attribution), keeping its rule of matching attack behaviour, not bossy English.

6. Limits on structure.

   • Refuse markdown images and links, raw HTML, code fences and headings. SpAIware leaked through a markdown image, and Mattermost renders markdown.
   • Refuse base64 or hex runs of 32 characters or more, and values shaped like secrets.

7. Show the owner what was written. A periodic digest or diff is the only defence in the evidence against false facts and paraphrase. This is at odds with the decision that writes are journalled and not announced; it is a question for the owner, not a build item.

8. A classifier only as an advisory flag, later. The costs:

   • an ONNX runtime to embed;
   • Prompt Guard's licence;
   • false positives on preferences;
   • over 70% adaptive bypass.

   Do not gate writes on a 9B model as a judge; nothing measures it.

Not verified

• The full text of OWASP ASI06's mitigations.
• Per-model NotInject numbers.
• PIGuard's size.
• Whether hermes-agent scans memory again when loading it.
• Ornith's special-token list.
• The per-defence adaptive figures come from a summary of arXiv 2510.09023's HTML page, not its tables.