Prompt Repetition
by marimo team
Reproduce research on prompt repetition with a notebook that can run experiments properly.
About this notebook
This notebook tests a prompt repetition technique on a multiple-choice task. It repeats a question two or three times inside one prompt. The idea comes from a paper that reports gains for non-reasoning models. The data comes from the ARC-Easy split of the allenai/ai2_arc dataset, loaded with the datasets library. The notebook sends each question to an open-source model through the openai AsyncOpenAI client. It defines the run parameters with a pydantic model and can also run as a command line script.
The reader edits three widgets before the run. A text field sets the base URL. A second text field sets the model names. A slider sets the concurrency. An EnvConfig widget from wigglystuff lets the reader paste a Weights and Biases API key. The reader then clicks a "Run evaluation" button to start the work. After the run the notebook shows a Markdown table with accuracy per model and repeat count. It also shows the raw results in an interactive table.
The notebook scores each question at repeat count one and at the chosen repeat count. It runs the requests concurrently with an asyncio semaphore. It uses temperature zero and a short token limit. A local diskcache store keeps each result, so repeated runs may skip finished calls. The code logs a results table and accuracy summaries to Weights and Biases. It also tracks the calls with weave.
