2D Character Embeddings
by marimo team
Train a tiny next-character predictor and watch a 2D embedding form for each letter.
About this notebook
This notebook learns a two-dimensional embedding for each character by training a next-character predictor on news headlines. The vocabulary is the 26 letters and the space. The notebook lowercases each headline, replaces every other character with a space, and builds one pair of current character and next character for each position.
The model is a small neural network written by hand in numpy. It reads each character as a two-dimensional point. A hidden layer of 16 units with a tanh activation transforms the point, and a final layer produces a score for every possible next character. The notebook trains the network with manual backpropagation and the Adam optimizer for 1,000 steps, and it saves a copy of the embedding table every few steps. Because the embedding has only two dimensions, the embedding table is the plot and needs no further projection.
The notebook renders each saved embedding as a matplotlib frame. Each frame draws every letter at its point, colored by type: the space in purple, the vowels in red, and the consonants in slate. A loss curve below the map marks the current training step. A FramePlayer widget from the wigglystuff package replays the frames. Press play to watch the letters move as training proceeds, with the vowels grouping together and the space and rare letters staying apart.
