Can LLMs Plan?
LLM-BabyBench: a benchmark evaluating LLMs' grounded planning and reasoning in simulated grid-world environments.
About this notebook
This notebook tests whether large language models can plan inside small grid worlds. It follows the question from the LLM-BabyBench paper (Choukrani et al., 2025). The worlds use the wanderland library, which builds BabyAI and MiniGrid style scenes. A mascot named Mo must collect every gem, avoid lava, and reach the goal tile. The notebook runs open-weight models through the transformers and featherlm libraries on a GPU. It uses torch for tensors and pandas for the tile grid.
You steer Mo by hand first. You type commands or tap buttons in a code editor, then press Run to move him through a 3D scene. Next you build worlds. A spreadsheet editor sets one tile per cell, and sliders control width, height, seed, gem count, and obstacles. Each generated world passes a breadth-first search oracle, so it stays solvable. Then you pick a model from a dropdown and press a button. The model reads the same world as text and returns a plan. The notebook replays the plan in the scene and compares its length to the oracle.
The models load in-kernel on the GPU only when you select them. The menu lists open-weight models from 4B to about 31B parameters, including Qwen3, Phi-4, and Gemma. The notebook streams each answer and shows the reasoning text and the committed plan. A seed pins the sampling, so a run repeats the same output. A later act gives the model a run_python tool. The model may then write its own search program instead of planning each step by hand.
