Viraj Tipnis - Founder, Acto
In a demonstration task from Acto, an AI agent receives a straightforward instruction: update a financial forecast using supplied assumptions while preserving the workbook’s structure and logic.
The agent returns a workbook with the expected FY27 earnings before interest, taxes, depreciation, and amortization (EBITDA) of $47.2 million. The automated check passes. But an expert inspecting the workbook finds that the agent replaced a formula with a hard-coded value. The answer is correct, but the workbook no longer calculates it as intended.
For Viraj Tipnis, founder of Acto, this example illustrates a central challenge in evaluating agents that perform real work. Automated checks can test specific requirements, but domain experts help determine whether those requirements capture what makes the work useful and trustworthy.
Acto uses marimo to connect those assessments with the evidence behind them. Researchers build review applications in Python, experts inspect the agent’s work, and their findings inform tests of the automated evaluators.
#Researchers needed expert feedback connected to the evidence
Acto is a reinforcement-learning research lab studying agents performing tasks in workplace software, including spreadsheets, documents, and email. Its implementation uses Harbor, the Terminal-Bench team’s open-source evaluation framework, to run tasks, collect outputs, and apply automated evaluators called verifiers.
Before marimo, researchers reconstructed cases from scattered files, screenshots, and logs. Expert feedback often arrived in a separate note or message, detached from the exact run it described. When a verifier changed, researchers struggled to compare its result with an expert’s earlier assessment.
Acto’s internal dashboard served research operations rather than focused expert review. Spreadsheets could not accommodate the different outputs and execution evidence. Traditional notebooks could combine code and evidence, but exposed unfamiliar tools to reviewers who did not regularly use Python. A custom application would become another product to maintain.
The team needed a way for researchers and experts to examine the same work and preserve the reasoning behind each judgment.
#Experts can inspect the task, output, and supporting evidence together
Acto’s research team built Result Review, a marimo application that places the agent’s task beside its output and scoring controls. The verifier result, reference solution, run configuration, and environment are available as supporting evidence.
Result Review presents the task and acceptance criteria, with controls for selecting the run and output to inspect. This example concerns a processor-performance model.
For the forecast workbook, the expert can assess whether the agent followed the instruction to preserve its logic, rather than relying on the passing score. The expert records a judgment and rationale against the specific run.
That assessment remains separate from the automated score. Acto saves it as an auditable review record linked to the run, leaving the original output and verifier result unchanged. Researchers can then investigate the disagreement without losing either assessment.
“marimo gave us a shared research surface: Python-native enough for researchers, accessible enough for domain review, and flexible enough to work directly with application state and task-specific evaluators,” says Viraj.
When the output alone does not explain a failure, reviewers can inspect the recorded sequence of agent actions and observations. Result Review displays supported trajectories alongside the output, with screenshots when they were captured during execution. These are event-level traces rather than video replays. Experts can also compare two runs side by side and record which produced stronger work.
The same approach extends beyond spreadsheets. In a processor-modeling task, an expert assesses whether a model ranks promising CPU designs well enough to guide engineering decisions, alongside its numerical accuracy.
#An expert’s finding becomes a test of the evaluator
The workbook example makes the gap in the automated check concrete: checking that a file opens and returns the expected total does not establish that its formulas remain intact.
Acto uses another marimo application, Task Studio, to help researchers turn expert-defined acceptance criteria into verifiers and test them against known examples. A revised check can inspect the workbook’s formulas, rather than only its displayed result.
Researchers can apply that check to preserved outputs without asking the agent to repeat its work. They test both known failures and expert-approved examples to determine whether the revision catches the problem without rejecting valid work.
marimo’s reactive execution updates related views when the author changes the task or acceptance checks. Running the verifier remains deliberate: the author selects “Test these examples” and examines the results. These tests do not overwrite the original run’s score.
In Task Studio, researchers specify examples that should pass or fail, then select “Test these examples” to check the verifier. This screenshot uses a scientific demonstration task.
“That lets us ask a better question than whether aggregate reward increased: is the evaluator aligned with the target behavior, and does it retain enough resolution to distinguish meaningful capability gains from noise or reward hacking?” says Viraj.
Expert involvement also begins before execution. In Task Review, experts independently assess the task, its expected outputs, and its acceptance criteria before an agent attempts it. This helps researchers examine both the instructions they give agents and the standards used to judge their work.
The applications support evaluation and review, not automatic model training. Revised verifiers determine how subsequent evaluation runs are scored. Review records and verifier results can also supply signals to a separate downstream training system that updates model weights.
#marimo connects Python inspection tools with an interface experts can use
Acto needs researchers to write task-specific code and domain experts to assess the evidence that code exposes. marimo supports both within the same application framework.
Researchers can use Python to read a workbook, inspect a model, or load execution records directly. App mode presents the relevant evidence and controls to experts without requiring them to navigate notebook code. Reactive execution keeps related views consistent as inputs change.
Experts assess individual criteria and record their verdict, supporting evidence, rationale, and confidence in the review interface.
The interface can also change with the task. Reviewing a workbook requires different evidence from reviewing a model, and researchers need to adjust how much they show.
“The interface has to surface the minimum sufficient evidence for an expert to make a calibrated judgment,” says Viraj.
With marimo, the research team can adapt review interfaces to these kinds of tasks without maintaining a separate custom review product. Acto’s public harbor-marimo repository shares its evaluation and review infrastructure with other developers. Other frameworks can support a similar review interface by providing equivalent run data, outputs, and scores.
#Expert findings remain useful after the review ends
With marimo, Acto’s researchers no longer have to reconstruct each case by hand, and an expert’s finding does not end as a detached note. It remains linked to the task, the agent’s output, and the execution evidence. Researchers can return to that assessment, test revised checks against the preserved work, and use what they learn in subsequent evaluations.
Research context: Acto’s research includes task types represented by GDPval and Terminal-Bench. It builds on computer-use research including OSWorld, CUA-Gym, and OpenComputer (repository).
marimo is the world’s best programming environment for working with data, and it’s all free and open source. Just pip install marimo or uv add marimo to get started.
