Replay back-test, 2026-10-12 (development-set numbers)

Compiled 2026-10-12. This is Test 1 from the plan, section 2: would the source set have shown each of the 37 Containment Index incidents within 24 hours of its first public trace? Read the caveats first. Code: agent/replay/ (backtest.py, archives.py, report.py, README); results and cached archive responses: data/replay/.

Caveats (why this is not evidence)

• Development set. The eight patterns and the trace terms were derived from these same 37 incidents, so this score cannot be evidence of anything. The live test set starts today, adjudicated by Byron and the outside reviewer.
• It measures archives, not Ria. Four dated public archives (Hacker News via Algolia, arXiv, the AI Incident Database weekly snapshot, GDELT) stand in for the watch loop. No screening model ran. A hit means an item naming the incident existed in the archive inside the window.
• Trace dates are human-reviewed and mostly approximate. 29 of 37 dates are guesses to the day; 13 are exact. Windows start at 00:00 UTC on the trace date, so "24h" means the same UTC day and "48h" includes one day of slack for date grain. The second pass changed some dates and terms after looking at the misses, which is exactly the contamination the plan warns about.
• The match rule moved the number. A loose match (two terms anywhere in the item) gave 73% at 24h with visible false positives (Show HN posts, unrelated papers matching on "OpenAI" and "agents"). The strict rule (title plus URL slug, at least one non-generic term, short terms as whole words) gives the figures below. Quote the strict 24h number.
• GDELT failed on 13 incidents (rate limits, "phrase too short"), so its contribution is a floor. The AI Incident Database snapshot uses submission date, which flatters it, and it ends at 2026-10-04.
• Recall only. Precision, pages per day and latency on live traffic are not measured here.

Headline, strict match

Plan target: 80% at 24h on the live set after 90 days. The source set, as proxied by these four archives, is below that bar today. Hacker News does nearly all the work (first to hit on 25 of 37). The direct feeds that would raise it most are listed below (Reuters, X, TechCrunch, The Verge, arXiv, the lab blogs); several are already tier 1 in agent/watch/sources.json, so the live loop should outperform this proxy, which is a claim to test, not a result.

Three incidents never surfaced in any archive within 30 days: the agent-to-agent prompt-injection worm (RIA-010), the Alibaba ROME reverse-tunnel case disclosed only in a paper body (RIA-024), and the AI Village unsolicited-email case (RIA-036). Two more had a first public trace nobody wrote about for days: the Amazon Q wiper PR (22 days) and the Hugging Face breach post (5.8 days). Those are the cases where only a direct read of the primary channel (a GitHub PR feed, a lab blog, a paper body) could have helped.

---

Full report, as generated

Question: would Ria's watch loop have surfaced each known containment incident within 24 hours of its first public trace? This measures the source set (what existed in dated archives), not any screening model.

Run 2026-10-12T13:06:41Z. 37 incidents, 37 with a public trace date, 0 without. Match rule: >=2 distinct trace_terms in the item title (+URL for HN/AIID/GDELT), at least one not an operator/generic word. Windows start at 00:00Z on the trace date (day-grain), so 24h means the same UTC day; 48h adds one day of date-grain slack.

Headline

Target in PLAN.md: 80% by day 14, 90% live. The exact-only column drops incidents whose trace date is a guess.

Per-archive contribution

AIID caveat: its date_submitted is day-grain and is when someone submitted a report, not when editors published it, so AIID is flattered. Snapshot covers submissions up to 2026-10-04T00:00:00Z. Wayback is not counted as a discovery archive (it only shows when a known URL was first crawled); it is used below to sanity-check trace dates.

Not caught within 24h (18 of 37)

Ten sources that would raise the catch rate most (greedy set cover over matching hits)

Within 24h (incidents newly covered by each additional source):

Within 72h (incidents newly covered by each additional source):

By archive, Hacker News alone gives the first matching item for 25 incidents and covers 18 of 37 within 24h; the domain tables below are the sites HN/news/AIID linked to, which is the list of direct feeds worth adding.

Domain = where the matching HN link / news article / AIID report / arXiv paper lives, i.e. the feeds worth subscribing to directly. news.ycombinator.com means an Ask/Show HN text post, so watch HN itself.

Longest lag from first public trace to first matching item

Separately, time the incident stayed hidden before its first public trace (entry start date to trace date, approximate):

Trace-date sanity check (Wayback first capture of primary sources)

No primary source was archived more than 24h before its recorded trace date.

Failures and gaps

• aiid not covering: RIA-017
• gdelt errors on: RIA-002, RIA-003, RIA-006, RIA-008, RIA-014, RIA-017, RIA-021, RIA-023, RIA-024, RIA-030, RIA-031, RIA-035, RIA-036

Per-incident results