# /// script # requires-python = ">=3.11" # dependencies = [ # "marimo", # "matplotlib", # "numpy", # ] # /// import marimo __generated_with = "0.23.3" app = marimo.App(width="medium") @app.cell(hide_code=True) def _(mo): mo.Html(""" """) return @app.cell(hide_code=True) def _(mo): mo.md(r""" # From Features to Pixels: Scaling Snake with a ConvNet *An empirical investigation into spatial representation for RL: when does a Convolutional Neural Network outperform hand-crafted feature engineering on Snake?* [![Open in marimo](https://marimo.io/shield.svg)](https://molab.marimo.io/github/Saheb/rl-snake/blob/main/notebooks/convnet.py/wasm) ### The Problem: Features That Don't Scale Every RL agent in this series so far has perceived Snake through a 24-dimensional feature vector: hand-crafted danger flags, food directions, flood-fill estimates. A human expert decided which information mattered and how to represent it. This works well on a 10×10 board. But what happens on 16×16? On 20×20? The feature vector is **fixed in size regardless of board scale**. "Danger straight" is one bit whether the board is 10 cells or 100 cells wide. The spatial patterns that matter — a snake curling back on itself, a pocket forming in the corner, an escape route narrowing three moves ahead — cannot be expressed in 24 numbers on a large board. The engineer would have to re-engineer the features for every new board size. A ConvNet takes the raw grid as input and learns its own spatial features. Larger board → larger input → the same architecture handles it, with no additional engineering. **The central question:** Does the ConvNet actually match or outperform hand-crafted features? And at what board size does the advantage appear? """) return @app.cell(hide_code=True) def _(mo, wasm_iframe): _intro = mo.md(""" ### What the Feature Vector Sees The 24-dim feature vector reduces the entire Snake board to a fixed-size summary. Below is a side-by-side of the raw board and the 24 features it extracts. Each feature is a single number — spatial relationships that span the whole board are collapsed into boolean flags and flood-fill fractions. **Notice** what the feature vector *cannot* say: it has no representation of which cells are occupied in the snake's body beyond the immediate neighbours, no concept of a corridor narrowing 4 tiles ahead, no awareness of the snake's shape. """) _feature_html = """

Raw Board (10×10)

Step 0

24-Dim Feature Vector

""" mo.vstack([ _intro, wasm_iframe(_feature_html, height="310px"), mo.callout( mo.md("**The hand-engineering ceiling.** The 24 features encode everything a human decided mattered. On a 10×10 board the agent has all the information it needs to reach ~5 apples per game. On a 16×16 board, the same 24 features now represent a much sparser, coarser view of a much larger space — three binary danger flags cannot capture whether a corridor 6 cells ahead is a dead end."), kind="info" ) ]) return @app.cell(hide_code=True) def _(mo, wasm_iframe): _intro2 = mo.md(r""" ### What a ConvNet Sees: The 3-Channel Grid Encoding Instead of hand-crafted features, we pass the raw board directly to a Convolutional Neural Network. The board is encoded as **three binary channels**, each `board_size × board_size`: | Channel | What it encodes | |---------|-----------------| | **Ch 0 — Body** | 1 wherever the snake's body is (excluding the head) | | **Ch 1 — Head** | 1 only at the head position | | **Ch 2 — Food** | 1 at the food position | Separating head from body is critical — the ConvNet needs to know *where the snake came from* to infer its current direction. Without a separate head channel, the model would see an ambiguous blob of body cells. **Click through the animation** to see how the three channels update as the snake moves. """) _channel_html = """

Board View

As a human sees it

Ch 0 — Body

Snake body only (not head)

Ch 1 — Head

Head position only

Ch 2 — Food

Food position

Step 0

Purple = 1 (active) | Light = 0 (inactive)

""" mo.vstack([ _intro2, wasm_iframe(_channel_html, height="230px"), mo.callout( mo.md( "**Why separate head from body?** With a single channel, the ConvNet can't distinguish " "which end of the snake it's looking at. The head channel provides the " "anchor the network needs to infer direction from the body's shape — " "the same spatial reasoning that lets a human glance at a screenshot and instantly " "know which way the snake is moving." ), kind="neutral" ) ]) return @app.cell(hide_code=True) def _(mo): mo.md(r""" ### The Architecture: ConvNet + Dueling Head The ConvNet takes the 3-channel grid and extracts spatial features through three convolutional layers. Because we use `padding=1` with `kernel_size=3`, the spatial dimensions are preserved — a 10×10 board stays 10×10 through all conv layers. Only the number of channels grows. ``` Input: (batch, 3, board_size, board_size) │ ▼ Conv2d(3→32, 3×3, padding=1) + ReLU → (batch, 32, H, W) │ ▼ Conv2d(32→64, 3×3, padding=1) + ReLU → (batch, 64, H, W) │ ▼ Conv2d(64→64, 3×3, padding=1) + ReLU → (batch, 64, H, W) │ ▼ Flatten → (batch, 64 × H × W) │ ├─▶ Value stream: Linear(flat, 256) → Linear(256, 1) └─▶ Advantage stream: Linear(flat, 256) → Linear(256, 4) │ ▼ Dueling combination: Q = V + (A − mean(A)) Output: (batch, 4) ← one Q-value per absolute action ``` The same three conv layers work on any board size. The only thing that changes is the `flat` dimension entering the FC head: | Board | Conv output | FC input (flat) | Parameters | |-------|------------|-----------------|------------| | 10×10 | 64×10×10 | 6,400 | ~3.3 M | | 16×16 | 64×16×16 | 16,384 | ~8.4 M | | 20×20 | 64×20×20 | 25,600 | ~13.2 M | This is the same architectural pattern as the original Atari DQN (Mnih et al., 2015), scaled down to Snake's grid dimensions. ### The Action Space: 4 Absolute Directions The feature-vector DQN uses **3 relative actions** (straight / turn-left / turn-right). This is possible because the feature vector explicitly encodes the current direction — the network can always tell "which way am I facing?" The ConvNet uses **4 absolute actions** (up / right / down / left). Direction is implicit in the body shape and head channel — the network must *infer* it. The trade-off: | | Feature DQN | ConvNet DQN | |---|---|---| | Actions | 3 (relative) | 4 (absolute) | | Reversal | Impossible by design | Possible, causes immediate death | | Direction knowledge | Explicit in features | Learned from grid | | Feature engineering | Required | None | The ConvNet must learn from experience that reversing direction kills it. Early in training it does this sometimes, generating strong negative feedback. This is not a weakness — it is the model discovering a rule the feature engineer hand-coded for free. """) return @app.cell(hide_code=True) def _(mo): mo.md(r""" --- ### Experiment 1: 10×10 — Can the ConvNet Match Hand-Crafted Features? The feature-vector DQN was already well-tuned for 10×10 Snake across our previous experiments (curiosity.py): Dueling network, 4-step returns, Prioritized Experience Replay, 3 seeds, 5,000 games. We ran the same setup with ConvNet DQN — same PER, same n-step, same Dueling head — only the state representation and action space change. **One important calibration note.** The two agents differ in training *intensity*, not just architecture: | | Feature DQN | ConvNet DQN | |---|---|---| | Gradient updates/game | ~3,000 (3 × batch 1,000 + per-step) | 128 (1 × batch 128) | | Reason | FC backward pass is cheap | Conv backward pass is ~10× heavier | This is a ~23× compute gap per game. A fair comparison by *wall-clock time* or *total gradient steps* would look different. What we're comparing here is performance under a fixed game budget (5,000 games), which shows how quickly each representation lets the agent learn. **Hypothesis:** On 10×10, the feature vector should be ahead — it provides free supervision (danger flags, flood-fill estimates) that the ConvNet must rediscover from reward signal alone. Given more games, the ConvNet should close the gap. """) return @app.cell(hide_code=True) def _(mo, np, plt): mo.stop( plt is None or np is None, mo.callout(mo.md("_(matplotlib/numpy unavailable in this runtime)_"), kind="warn") ) _GAMES_100 = list(range(100, 10001, 100)) # Feature DQN 10×10 dense — 10k games, 3 seeds (matched game count to ConvNet v3) _feat_10_mean = [0.117, 0.11, 0.113, 0.1, 0.1, 0.103, 0.107, 0.11, 0.113, 0.113, 0.123, 0.127, 0.13, 0.137, 0.14, 0.147, 0.157, 0.17, 0.18, 0.2, 0.213, 0.233, 0.25, 0.273, 0.283, 0.303, 0.32, 0.343, 0.373, 0.403, 0.43, 0.457, 0.483, 0.513, 0.537, 0.567, 0.6, 0.633, 0.67, 0.707, 0.763, 0.81, 0.863, 0.923, 0.98, 1.037, 1.107, 1.19, 1.283, 1.413, 1.567, 1.697, 1.83, 1.937, 2.073, 2.197, 2.33, 2.433, 2.543, 2.65, 2.753, 2.873, 2.967, 3.08, 3.173, 3.267, 3.36, 3.443, 3.537, 3.613, 3.703, 3.79, 3.87, 3.947, 4.007, 4.08, 4.14, 4.217, 4.297, 4.36, 4.423, 4.49, 4.557, 4.623, 4.69, 4.747, 4.793, 4.847, 4.907, 4.963, 5.017, 5.073, 5.14, 5.21, 5.26, 5.307, 5.363, 5.413, 5.463, 5.5] _feat_10_std = [0.021, 0.017, 0.015, 0.01, 0.017, 0.015, 0.015, 0.01, 0.015, 0.015, 0.015, 0.021, 0.017, 0.021, 0.02, 0.015, 0.015, 0.01, 0.01, 0.02, 0.015, 0.015, 0.02, 0.015, 0.015, 0.015, 0.01, 0.015, 0.015, 0.015, 0.017, 0.021, 0.025, 0.025, 0.025, 0.025, 0.02, 0.025, 0.017, 0.006, 0.012, 0.01, 0.015, 0.015, 0.0, 0.012, 0.012, 0.017, 0.012, 0.021, 0.025, 0.023, 0.017, 0.015, 0.055, 0.059, 0.056, 0.06, 0.064, 0.062, 0.057, 0.046, 0.04, 0.026, 0.006, 0.006, 0.017, 0.015, 0.023, 0.021, 0.032, 0.017, 0.026, 0.031, 0.04, 0.036, 0.044, 0.047, 0.042, 0.036, 0.021, 0.01, 0.015, 0.023, 0.026, 0.04, 0.035, 0.025, 0.04, 0.047, 0.051, 0.065, 0.075, 0.082, 0.089, 0.091, 0.086, 0.096, 0.106, 0.111] # ConvNet DQN v3 10×10 dense — sweep-tuned hyperparameters, 10k games, 3 seeds _conv_10_mean = [0.19, 0.207, 0.21, 0.227, 0.28, 0.317, 0.36, 0.403, 0.453, 0.517, 0.573, 0.627, 0.687, 0.747, 0.81, 0.863, 0.917, 0.963, 1.007, 1.053, 1.1, 1.14, 1.18, 1.223, 1.27, 1.313, 1.363, 1.41, 1.463, 1.527, 1.603, 1.687, 1.893, 2.17, 2.433, 2.73, 3.02, 3.323, 3.61, 3.883, 4.15, 4.41, 4.64, 4.877, 5.097, 5.303, 5.51, 5.71, 5.89, 6.08, 6.267, 6.443, 6.62, 6.797, 6.96, 7.127, 7.3, 7.47, 7.633, 7.787, 7.927, 8.077, 8.233, 8.377, 8.527, 8.663, 8.813, 8.943, 9.08, 9.207, 9.333, 9.47, 9.597, 9.717, 9.827, 9.94, 10.06, 10.18, 10.297, 10.41, 10.527, 10.637, 10.737, 10.853, 10.947, 11.04, 11.133, 11.223, 11.32, 11.427, 11.527, 11.62, 11.713, 11.81, 11.897, 11.98, 12.073, 12.163, 12.243, 12.323] _conv_10_std = [0.066, 0.035, 0.01, 0.006, 0.026, 0.025, 0.036, 0.031, 0.04, 0.035, 0.035, 0.031, 0.029, 0.029, 0.026, 0.025, 0.025, 0.025, 0.031, 0.025, 0.026, 0.026, 0.026, 0.023, 0.026, 0.032, 0.032, 0.035, 0.04, 0.046, 0.049, 0.072, 0.127, 0.148, 0.155, 0.176, 0.185, 0.191, 0.212, 0.217, 0.202, 0.201, 0.217, 0.22, 0.237, 0.24, 0.26, 0.278, 0.296, 0.305, 0.325, 0.329, 0.316, 0.31, 0.324, 0.326, 0.291, 0.284, 0.289, 0.295, 0.295, 0.284, 0.295, 0.309, 0.298, 0.319, 0.301, 0.318, 0.303, 0.315, 0.326, 0.32, 0.332, 0.323, 0.323, 0.321, 0.321, 0.33, 0.325, 0.333, 0.335, 0.327, 0.329, 0.325, 0.316, 0.315, 0.32, 0.323, 0.334, 0.338, 0.342, 0.347, 0.343, 0.337, 0.335, 0.334, 0.333, 0.341, 0.341, 0.34] gs = np.array(_GAMES_100) feat_mu = np.array(_feat_10_mean) feat_sd = np.array(_feat_10_std) conv_mu = np.array(_conv_10_mean) conv_sd = np.array(_conv_10_std) fig10, ax10 = plt.subplots(figsize=(9, 4)) ax10.plot(gs, feat_mu, color='tab:blue', lw=2, label='Feature DQN (n=3) — 5.50 final') ax10.fill_between(gs, feat_mu - feat_sd, feat_mu + feat_sd, color='tab:blue', alpha=0.15) ax10.plot(gs, conv_mu, color='tab:red', lw=2, ls='--', label='ConvNet DQN v3 (n=3) — 12.32 final') ax10.fill_between(gs, conv_mu - conv_sd, conv_mu + conv_sd, color='tab:red', alpha=0.15) ax10.set_xlabel('Training Game') ax10.set_ylabel('Mean Score (cumulative)') ax10.set_title('10×10 Dense: Feature DQN vs ConvNet DQN v3 (10,000 games, 3 seeds)') ax10.grid(True, alpha=0.3) ax10.legend(loc='upper left') fig10.tight_layout() fig10 return ax10, conv_mu, conv_sd, feat_mu, feat_sd, fig10, gs @app.cell(hide_code=True) def _(mo): mo.md(r""" --- ### Experiment 2: 16×16 — The Hypothesis That Didn't Hold On a 16×16 board (256 cells vs 100), the spatial complexity grows substantially. The original hypothesis was that the 24-dim feature vector would struggle — that spatial patterns unrepresentable in 24 numbers would become the limiting factor. **The original hypothesis was wrong** — and then corrected by better experiments. The v1 ConvNet (wrong hyperparameters) scored 0.20 on 16×16, which looked like an architectural failure. But after a 30-trial W&B hyperparameter sweep, ConvNet v3 reaches **10.83** — nearly double the Feature DQN's 5.92. The key insight: the v1 ConvNet failed because of *computational* choices, not architecture. It had only 128 gradient updates/game vs the Feature DQN's ~3,000, and epsilon floored too early before the network had learned anything useful. Fix those, and the ConvNet wins decisively at both scales. **What scales better:** The ConvNet's advantage grows with board size. On 10×10 it's 2.24× better; on 16×16 it's 1.83× better (but from a higher absolute ceiling). The Feature DQN plateaus around 5.5–6 on both boards — its hand-crafted features are scale-invariant, but the policy hits a ceiling the features cannot express. The ConvNet keeps climbing because it learns spatial representations the engineer didn't encode. """) return @app.cell(hide_code=True) def _(mo, np, plt): mo.stop( plt is None or np is None, mo.callout(mo.md("_(matplotlib/numpy unavailable in this runtime)_"), kind="warn") ) _GAMES_160 = list(range(100, 16001, 100)) gs16 = np.array(_GAMES_160) # Feature DQN 16×16 dense — 16k games, 3 seeds (matched game count to ConvNet v3) _feat_16_mean = [0.08, 0.067, 0.07, 0.067, 0.067, 0.067, 0.07, 0.067, 0.063, 0.067, 0.067, 0.07, 0.067, 0.07, 0.073, 0.08, 0.08, 0.08, 0.08, 0.077, 0.08, 0.083, 0.087, 0.083, 0.09, 0.093, 0.1, 0.1, 0.103, 0.11, 0.113, 0.117, 0.123, 0.133, 0.14, 0.147, 0.16, 0.167, 0.17, 0.187, 0.2, 0.213, 0.223, 0.24, 0.253, 0.27, 0.29, 0.31, 0.33, 0.347, 0.36, 0.383, 0.403, 0.423, 0.437, 0.46, 0.483, 0.503, 0.52, 0.547, 0.58, 0.603, 0.64, 0.66, 0.69, 0.723, 0.76, 0.797, 0.83, 0.867, 0.917, 0.963, 1.013, 1.06, 1.107, 1.163, 1.233, 1.313, 1.4, 1.497, 1.58, 1.663, 1.757, 1.837, 1.927, 1.997, 2.077, 2.16, 2.243, 2.307, 2.38, 2.453, 2.513, 2.573, 2.643, 2.707, 2.77, 2.833, 2.91, 2.983, 3.04, 3.11, 3.173, 3.237, 3.31, 3.373, 3.437, 3.493, 3.557, 3.61, 3.673, 3.727, 3.78, 3.817, 3.87, 3.923, 3.983, 4.023, 4.073, 4.117, 4.163, 4.227, 4.273, 4.337, 4.387, 4.443, 4.5, 4.547, 4.597, 4.643, 4.69, 4.73, 4.777, 4.827, 4.86, 4.91, 4.957, 4.993, 5.033, 5.073, 5.107, 5.157, 5.197, 5.24, 5.283, 5.33, 5.377, 5.42, 5.467, 5.503, 5.543, 5.583, 5.623, 5.667, 5.71, 5.757, 5.787, 5.83, 5.877, 5.92] _feat_16_std = [0.069, 0.031, 0.026, 0.031, 0.021, 0.021, 0.017, 0.021, 0.015, 0.015, 0.015, 0.01, 0.006, 0.01, 0.012, 0.01, 0.01, 0.01, 0.01, 0.015, 0.01, 0.015, 0.015, 0.015, 0.02, 0.015, 0.017, 0.017, 0.023, 0.026, 0.023, 0.021, 0.025, 0.015, 0.02, 0.025, 0.02, 0.025, 0.02, 0.015, 0.02, 0.021, 0.015, 0.02, 0.021, 0.017, 0.01, 0.017, 0.017, 0.023, 0.026, 0.031, 0.04, 0.04, 0.045, 0.056, 0.05, 0.047, 0.046, 0.045, 0.036, 0.04, 0.04, 0.036, 0.036, 0.042, 0.036, 0.038, 0.026, 0.038, 0.04, 0.042, 0.042, 0.052, 0.057, 0.067, 0.068, 0.065, 0.066, 0.07, 0.066, 0.059, 0.055, 0.055, 0.064, 0.064, 0.072, 0.072, 0.075, 0.08, 0.085, 0.085, 0.085, 0.075, 0.075, 0.071, 0.066, 0.06, 0.053, 0.055, 0.052, 0.053, 0.064, 0.067, 0.062, 0.064, 0.076, 0.072, 0.078, 0.07, 0.064, 0.058, 0.052, 0.042, 0.046, 0.031, 0.045, 0.045, 0.065, 0.075, 0.075, 0.09, 0.095, 0.08, 0.075, 0.07, 0.07, 0.065, 0.065, 0.07, 0.075, 0.066, 0.06, 0.076, 0.082, 0.072, 0.083, 0.076, 0.085, 0.085, 0.086, 0.09, 0.09, 0.075, 0.07, 0.075, 0.078, 0.089, 0.076, 0.072, 0.081, 0.09, 0.09, 0.087, 0.098, 0.107, 0.101, 0.1, 0.1, 0.1] # ConvNet DQN v3 16×16 dense — sweep-tuned hyperparameters, 16k games, seed 0 _conv_16_mean = [0.21, 0.21, 0.22, 0.22, 0.22, 0.23, 0.25, 0.27, 0.29, 0.32, 0.34, 0.37, 0.4, 0.43, 0.47, 0.5, 0.53, 0.56, 0.6, 0.63, 0.66, 0.69, 0.72, 0.75, 0.78, 0.81, 0.83, 0.86, 0.88, 0.91, 0.93, 0.95, 0.97, 1.0, 1.02, 1.04, 1.06, 1.08, 1.1, 1.12, 1.14, 1.15, 1.17, 1.19, 1.21, 1.22, 1.24, 1.26, 1.27, 1.29, 1.3, 1.32, 1.34, 1.35, 1.37, 1.38, 1.4, 1.42, 1.43, 1.45, 1.46, 1.48, 1.5, 1.52, 1.54, 1.55, 1.58, 1.6, 1.62, 1.65, 1.67, 1.7, 1.73, 1.76, 1.81, 1.91, 2.02, 2.15, 2.27, 2.39, 2.52, 2.65, 2.78, 2.9, 3.04, 3.18, 3.34, 3.49, 3.62, 3.74, 3.87, 3.99, 4.13, 4.26, 4.38, 4.53, 4.67, 4.81, 4.95, 5.07, 5.19, 5.31, 5.43, 5.57, 5.7, 5.81, 5.93, 6.05, 6.18, 6.29, 6.42, 6.53, 6.64, 6.76, 6.86, 6.98, 7.09, 7.2, 7.31, 7.41, 7.52, 7.62, 7.73, 7.83, 7.93, 8.03, 8.11, 8.2, 8.31, 8.39, 8.49, 8.59, 8.68, 8.78, 8.85, 8.94, 9.02, 9.1, 9.18, 9.27, 9.35, 9.42, 9.5, 9.57, 9.66, 9.73, 9.8, 9.89, 9.99, 10.07, 10.16, 10.24, 10.31, 10.39, 10.47, 10.54, 10.61, 10.69, 10.75, 10.83] f16_mu = np.array(_feat_16_mean) f16_sd = np.array(_feat_16_std) c16_mu = np.array(_conv_16_mean) fig16, ax16 = plt.subplots(figsize=(9, 4)) ax16.plot(gs16, f16_mu, color='tab:blue', lw=2, label='Feature DQN (n=3) — 5.92 final') ax16.fill_between(gs16, f16_mu - f16_sd, f16_mu + f16_sd, color='tab:blue', alpha=0.15) ax16.plot(gs16, c16_mu, color='tab:red', lw=2, ls='--', label='ConvNet DQN v3 (n=1) — 10.83 final') ax16.axvline(x=8000, color='tab:red', ls=':', lw=1.2, alpha=0.5) ax16.text(8100, 0.5, 'ε floors\n(game 8,000)', fontsize=9, color='tab:red', alpha=0.7) ax16.axvline(x=8000, color='tab:blue', ls=':', lw=1.2, alpha=0.5) ax16.set_xlabel('Training Game') ax16.set_ylabel('Mean Score (cumulative)') ax16.set_title('16×16 Dense: Feature DQN vs ConvNet DQN v3 (16,000 games)') ax16.grid(True, alpha=0.3) ax16.legend(loc='upper left') fig16.tight_layout() fig16 return ax16, c16_mu, f16_mu, f16_sd, fig16, gs16 @app.cell(hide_code=True) def _(mo): mo.md(r""" --- ### What Did the ConvNet v3 Actually Learn? After a 30-trial W&B hyperparameter sweep (sweep ID: igzvteil), the key findings were: - **`n_steps=1` universally better**: multi-step returns hurt when combined with PER — every top sweep config used single-step returns - **Larger network wins**: ch1=64, ch2=128 consistently outperformed ch1=32, ch2=64 - **`epsilon_end=0.01`**: exploiting harder (vs 0.05) improved final scores across all top configs - **`gamma=0.995`**: very patient credit assignment, values distant food - **`train_calls=10`**: more gradient updates per episode = better ConvNet v3 on 10×10 reaches **12.32 mean score** at 10k games. The curve is still rising at game 10,000, suggesting further improvement is possible. On 16×16, it reaches **10.83** — nearly double Feature DQN's 5.92. The filters the ConvNet learns are analogous to the hand-crafted features, but discovered from data: - **Local edge detectors**: snake body boundaries — the learned version of danger flags - **Head proximity filters**: head adjacent to body — the learned version of "danger straight" - **Food co-occurrence filters**: head near food channel — learned proximity signal Given the Atari DQN's training budget (10M+ frames), these would sharpen into richer, more generalizable spatial policies that the 24-dim feature vector simply cannot express. """) return @app.cell(hide_code=True) def _(mo): _summary_table = mo.md(r""" --- ### What We Actually Found | | Feature DQN | ConvNet DQN v1 | ConvNet DQN v3 | |---|---|---|---| | **10×10** (10k games) | **5.50** | 1.49 (5k) | **12.32** ✓ | | **16×16** (16k games) | **5.92** | 0.20 (5k) | **10.83** ✓ | | Gradient updates/game | ~3,000 | 128 | 10 × batch 512 | | State size | 24 floats (fixed) | 3 × H × W | 3 × H × W | | Feature engineering | Expert-designed | None | None | | Hyperparameter search | Manual | Manual | 30-trial W&B sweep | | Action space | 3 relative | 4 absolute | 4 absolute | **The original hypothesis was partially wrong.** The 24-dim feature vector does *not* break down at 16×16 — it was already scale-invariant by design. But it *does* plateau around 5.5–6 on both boards. The ConvNet, once properly tuned, blows past that ceiling. **ConvNet v1 failed by budget, not by design.** At 128 gradient updates/game vs 3,000, it got ~23× less training per game, and the epsilon schedule killed it on 16×16 before it learned anything. The v3 sweep fixed both. **The two genuine advantages of the ConvNet representation:** 1. *No domain knowledge required* — the feature vector needed a human who understood Snake. The ConvNet works on any grid game without modification. 2. *Higher ceiling* — Feature DQN plateaus at ~5.5–6. ConvNet v3 reaches 12.32 on 10×10 and is still rising. On larger boards, the gap will widen further as multi-step spatial patterns become unreachable by 24 fixed features. **What did not help:** RND (Random Network Distillation) exploration bonus showed zero improvement over ConvNet v3 — mean 11.96 vs 12.32. Curiosity-based exploration helps weak baselines with poor coverage but not well-tuned agents that already explore effectively. ### Lineage The 3-channel grid encoding used here is exactly how DreamerV3 (Hafner et al., 2023) represents grid worlds in its world model. The progression from hand-crafted features → ConvNet → learned world model is the central arc of modern RL representation learning. """) _lit = mo.ui.tabs({ "Atari DQN (2015)": mo.md(""" **Mnih et al., Nature 2015** — [arXiv:1312.5602](https://arxiv.org/abs/1312.5602) **Problem solved:** Learn to play 49 Atari games from raw pixels using a single architecture, with no game-specific feature engineering. **Key result:** Superhuman performance on 29 of 49 games using a 3-layer CNN + fully connected Q-network trained end-to-end from pixel inputs. **Relation:** The direct architectural ancestor of the ConvNet used here. Atari frames are 84×84×4 (4 stacked grayscale frames); Snake uses 10×10×3 (3 binary channels). Same pattern: CNN feature extractor → Q-value head. """), "Rainbow DQN (2017)": mo.md(""" **Hessel et al., AAAI 2018** — [arXiv:1710.02298](https://arxiv.org/abs/1710.02298) **Problem solved:** Which DQN improvements combine well and in what combination do they yield the best performance? **Key result:** Combining 6 extensions (Double DQN, Dueling, PER, N-step, Distributional C51, Noisy Networks) outperforms all individual variants. The ConvNet DQN here already uses 3 of these (Dueling, PER, N-step) — Rainbow adds the remaining three. **Relation:** Natural next step for this project — applying the full Rainbow stack on top of the ConvNet backbone. """), "DreamerV3 (2023)": mo.md(""" **Hafner et al., DeepMind 2023** — [arXiv:2301.04104](https://arxiv.org/abs/2301.04104) **Problem solved:** A single agent that achieves human-level performance across 150+ diverse tasks with fixed hyperparameters. **Key result:** Solved Minecraft diamond collection from scratch with no domain knowledge. Uses a world model learned from image observations to plan in latent space — no environment rollouts during policy training. **Relation:** Dreamer's grid-world representation is exactly our 3-channel encoding. The ConvNet here is the encoder in Dreamer's RSSM. The full Dreamer architecture adds a learned dynamics model, reward predictor, and policy trained entirely inside imagined rollouts. """), }) mo.vstack([_summary_table, _lit]) return @app.cell(hide_code=True) def _(): import base64 import marimo as mo try: with mo.capture_stderr(): import matplotlib.pyplot as plt except ImportError: plt = None try: import numpy as np except ImportError: np = None def wasm_iframe(html: str, *, width: str = "100%", height: str = "400px"): encoded = base64.b64encode(html.encode("utf-8")).decode("ascii") return mo.Html( f'' ) return base64, mo, np, plt, wasm_iframe if __name__ == "__main__": app.run()