Seed of Thought
by marimo team
Reproduce the String Seed of Thought paper: showing LLMs struggle to sample uniformly and a trick to fix it.
About this notebook
This notebook is a partial reproduction of the paper String Seed of Thought (ICLR 2026). It tests whether a language model can pick options at random. The paper argues that models fail to sample from a target distribution and favour a few options. The notebook checks this claim on a small local Qwen model. It uses transformers and torch to run the model, pandas to hold the samples, numpy to compute entropy, and altair to draw the charts. The notebook creates its own data by asking the model the same question many times and counting the replies. The four tasks are a digit from 1 to 10, a letter from A to E, and a coin flip in two option orders.
You pick the model with a dropdown. The choices are Qwen2.5-0.5B, 1.5B, and 3B Instruct. You set the number of samples per task with a slider, from 100 to 5000. The notebook runs each task twice. The baseline asks the question directly. The Seed of Thought run first asks for a random 12-character string, then asks for the answer. The grouped bar charts show the percent of samples that matched each option. A dashed crimson rule marks the target percent for a uniform choice. A table below reports the entropy in bits against the maximum for each task. The order of the options may change the result.
The notebook reads each reply in three ways. The baseline uses a strict regex. The Seed of Thought text is parsed both with the same regex and by taking the last character or word. The tail parse catches replies where the model writes freely and picks the choice only at the end. The notebook selects the device at run time. It uses cuda with bfloat16 when a GPU is present, mps on Apple hardware, or cpu with float32. Sampling uses temperature 1.0 and batches all samples in one call to model.generate.
