Neural Thickets

About this notebook

This notebook explains the paper "Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights" by Yulu Gan and Phillip Isola. The paper argues that many task-specific experts sit close to pretrained weights, which makes random local perturbations a plausible post-training strategy. The notebook rebuilds the paper's 1D toy experiment: a small PyTorch MLP that predicts the next value of 1D signals such as sinusoids, square waves, and mixed signals. Plotly draws every chart.

The notebook builds the idea in stages, and each stage is interactive. A minesweeper-style grid game shows why random guessing succeeds when targets are dense: controls set the grid size and the number of hidden diamonds, and each click is one guess. A Plotly loss landscape then contrasts a step-by-step SGD path with 50 Gaussian samples that arrive in one parallel step. In the 1D lab you configure the MLP, pretrain it, and sample random perturbations around its weights. You can select any perturbation and compare it with the base model on every signal category, and RandOpt controls sort the perturbations and show a top-k ensemble. marimo re-runs the dependent charts each time a control changes.

The 1D training, model, and dataset code comes from the paper authors' RandOpt repository, and everything runs on the CPU. The grid game is a self-contained HTML page inside an iframe, and the perturbation animation is a custom anywidget. The closing section, Controllable RandOpt, is the notebook's own extension, not a result from the paper. A 2D simulation and a compute-budget model test whether a shifted or stretched sampling distribution can find experts with fewer parallel guesses.