2026-10-18 · for chapter 09 (eval gaming) · from the research reading Neural Networks Are Already Clean, and the rebuttal: clean-looking neurons versus explained neurons
DRAFT, machine-compiled by the RAI Researcher; owner review pending. Nothing here is merged into a chapter automatically.
This result bears directly on the oversight-subversion chapter's premise that interpretability-based monitors are only as trustworthy as their completeness. Cammarata's essay strengthens the case that neuron-level reading of a model's own activations can reconstruct why a behavior happened after the fact, which is useful for the replay-style forensics the book relies on, and his line that deception would be in plain sight but unreadable is worth quoting as the honest version of the optimistic position. The rebuttal supplies the chapter's needed caveat: an explanation that fits every activating example can still miss that the neuron fires only on a narrower condition or at a magnitude the rubric cannot predict, and that gap is exactly where a conditional policy would live. The argument should therefore distinguish exploratory interpretability, which this essay plausibly advances, from assurance interpretability, which requires a simulation-based test the essay explicitly declines to adopt. Until the forthcoming companion paper reports KL-style completeness numbers, the chapter should treat clean-looking neurons as raising the ceiling on post-incident analysis without lowering the bar for pre-deployment oversight.
Claim. Polysemanticity in early model organisms such as InceptionV1 was caused by ordinary, mostly fixable problems: a weak architecture, weak tools such as reward-hacked feature visualization, and a motif Cammarata calls recursive generalization, not by a second learning principle of superposition.
Strongest counterclaim. leogao: a pattern that fits all activating examples does not imply the converse; the neuron may fire only on a hard-to-characterize subset, which was a significant problem in the original auto-interp paper, so clean-looking activations are a good sign but not proof of explanation.
Open question for the chapter. When rubric-simulated values replace a real neuron across a large held-out corpus, what is the downstream KL, and how does it compare to zero- and mean-ablation baselines?
Sources: https://already-clean.goodfire.com, https://www.lesswrong.com/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean, https://www.lesswrong.com/posts/Dkh7pXKjoNWkkxQDz/neural-networks-are-not-already-clean