Dead Salmons of Interpretability
Interactive false-positive traps showing how feature attribution, probing, SAEs, and circuits 'find' structure in pure noise.
About this notebook
This notebook accompanies "The Dead Salmons of AI Interpretability" (Méloux, Dirupo, Portet, and Peyrard, 2025), a paper about false positives in AI interpretability research. It rebuilds the paper's failure modes as simulations with NumPy. The first demo repeats the dead-salmon fMRI experiment: it correlates thousands of noise-only voxels with a fake task, and about five percent pass p < 0.05. A second demo repeatedly splits the cars dataset from vega-datasets into two random groups and runs a t-test on every numeric feature, so every significant result is false by construction. Six case studies then apply the same trap to feature attribution, probing, causal discovery, sparse autoencoders, concept directions, and circuit search.
Sliders set the shape of each experiment, such as the number of voxels, the alpha level, and the number of tests, and marimo re-runs the simulation when you move one. A custom anywidget shows each result as a grid of tests next to a p-value histogram. Drag the red threshold line on the histogram, and the grid repaints the tests that count as significant. Two buttons snap the threshold to the naive alpha cutoff or the Bonferroni cutoff. Other sections use their own widgets, such as a side-by-side saliency comparison for a trained and a random model.
Each widget is an anywidget class with traitlets state that syncs between Python and the browser. JavaScript redraws the grid and histogram during the drag, and marimo re-runs the dependent cells when the threshold changes. The notebook is one self-contained Python file: even the salmon image at the top is an embedded base64 PNG.
