Relevance prompt experiment, 2026-10-13

Question: would adding indirect-report examples to the screen prompt raise recall on incident-like items without flooding the candidate pool? Model: local qwen3.6:35b-a3b via Ollama, temperature 0, same JSON parsing as watch/screen.py. Cost $0. screen.py is unchanged.

Setup

• Positives: 68. For each of the 34 incidents in data/incidents.json (excluding RIA-017, RIA-025, RIA-036, used as variant examples so the test stays held out): the title, and the first sentence of the what field.
• Negatives: 60 items completed, sampled (seed 42) from the newest screened files that the model screen judged NOT relevant. Target was 150; cut to 60 because the model runs at about 2.6 s per call and the whole run is capped at 11 minutes.
• Caveat: negatives are items the current prompt already rejected, so FPR here measures only new false alarms the variant introduces, not the current prompt's own errors (its FPR on this set is 0 by construction in principle; any non-zero base FPR is model nondeterminism or a changed input). Positives are terse titles, an easier input than live feed text.
• Variant: the current prompt plus one sentence that indirect or secondhand reports of agent misbehavior count, and 3 example reports (full text in state/prompt-variant-2026-10-13.txt).

Results

Delta: recall +0.0 points, false-positive rate -1.7 points.

Flips

Positives newly caught by the variant (1):

• FT-sourced employees say Kiro chose to delete and recreate a production environment, causing a 13-hour outage.

Positives lost by the variant (1):

• OpenAI agents also touch four more Australian government services

Negatives newly flagged by the variant (0):

• none

Negatives no longer flagged (1):

• Is your AI yours?

Recommendation

Do not adopt yet: recall change +0.0 points, false-positive change -1.7 points does not meet the bar (recall up, false positives up by no more than 3 points). Keep the current prompt.