How the Sloan Digital Sky Survey uses marimo to share petabyte-scale astronomy with the world
By Kayvon Sharifi, Andy Casey, and Sarah Tsai
“What used to take weeks now takes minutes. marimo fundamentally changed how we can share data within SDSS, and how quickly we can go from problem identified to solution, which hasn’t happened before. It’s a tremendous change for research acceleration.” — Professor Andy Casey, Research Scientist, Flatiron Institute / Monash University
#Mapping the universe
The Sloan Digital Sky Survey (SDSS) is one of the most influential astronomical projects of the last quarter century. Since first light at Apache Point Observatory in 2000, SDSS has accumulated tens of petabytes of imaging and spectroscopic data across 20 public data releases, observing everything from nearby stars to quasars near the edge of the observable universe. Its data releases are foundational infrastructure for modern astronomy: hundreds of peer-reviewed papers cite SDSS each year, and its optical and infrared spectroscopic instruments have produced the most detailed three-dimensional map of the universe ever made.
Kayvon Sharifi is a fourth-year PhD student at Georgia State University studying the gravitational unraveling of nearby star clusters, using kinematic data from SDSS to trace how clusters evaporate under the pull of the Milky Way. Andy Casey is a research professor in astrophysics jointly appointed at Monash University in Australia and the Flatiron Institute in New York. His work centers on stellar spectroscopy, decomposing starlight to infer temperature, chemical composition, and the conditions that shape stellar evolution and planet formation, with a particular fascination for “oddball” stars. Both are active SDSS collaborators, and when they discovered marimo, they built interactive tools in days that the collaboration had wanted for years, developing a new way for the hundreds of astronomers in SDSS to get to their data.
#Challenge
SDSS produces hundreds of terabytes of imaging and spectroscopic data, served through a network of databases, file servers, and cloud-hosted archives built up over decades by the SDSS collaboration. Making that data usable by hundreds of collaborators worldwide has always required considerable infrastructure investment.
Even with that infrastructure in place, parts of the day-to-day workflow around tutorial notebooks and data access remained manual:
- Notebooks that worked on a maintainer’s laptop sometimes broke for first-time users navigating SDSS’s data model, making debugging harder to do remotely
- Exploring how a result changes with a parameter often meant editing code and manually re-running cells, since standard notebooks don’t recompute reactively
- New members joining SDSS had to spend time reading documentation to understand which data products held the spectra or catalogs they needed, before they could start exploring
marimo is gaining popularity within SDSS as a way to close these gaps. By letting users adjust a parameter and see plots and tables update automatically, without re-running cells by hand, it’s made it faster for astronomers to explore SDSS data live and for new collaborators to get oriented. Built on top of the infrastructure the collaboration has spent decades developing, marimo has become a valuable layer for streamlining how researchers actually get hands-on with the data — turning static notebooks into a live, explorable interface.
#Bringing in marimo
Kayvon discovered marimo as an alternative to Streamlit while building a web app to support an introductory physics class. He watched Vincent Warmerdam’s video on interactively selecting points in matplotlib charts and had an immediate “aha” moment. For an astronomer, the ability to click on a point in a scatter plot and instantly retrieve the underlying spectrum was a very exciting use case, and within a couple of days, Kayvon had a working prototype. The core mechanics of reactive execution and interactive widgets were “super quick and easy to learn.”
The SDSS team adopted marimo as a notebook interface on top of their existing data infrastructure. The data still lives where it always has (in SDSS’s databases, in cloud-hosted FITS archives, in CasJobs), but marimo now provides an optimized layer where astronomers can interact with that data: query it, visualize it, and quickly share their findings.
marimo uniquely provides a solution to the several challenges SDSS has been facing:
#Reproducibility, by construction
marimo notebooks are stored as pure Python files, which means SDSS can version-control them in Git like any other code artifact. There’s no hidden state, because marimo’s reactive execution model keeps every cell in sync with its dependencies automatically.
marimo also tracks package dependencies in the notebook file itself. A student opening an SDSS tutorial in 2030 will get the same environment as the one the maintainer used to write it.
#Real interactivity, no widget plumbing
Kayvon used marimo’s built-in UI elements to build tools the SDSS collaboration has wanted for years. Researchers can upload a list of their stars’ IDs, and the marimo notebook immediately retrieves the associated data, e.g. spectra and outputs of SDSS pipelines. The data are automatically shown in a customizable and interactive scatter plot, where researchers can make click-and-drag selections and immediately view the corresponding spectra, with marimo handling the index matching in the background, work that previously required careful manual bookkeeping. Researchers can then download and pull the data for their stars without writing a single query. In the marimo notebook shown below, collaboration members can select different star clusters and similarly display, interact with, and export the data.
The MWM DR20 BOSS Star Cluster Explorer: choose a cluster, make a box selection on the color–magnitude diagram, and view and export the spectra for the selected stars.
#Sharing without a server
marimo notebooks can be published to the web via Pyodide, which runs the notebook’s Python entirely in the reader’s browser. For SDSS, this means tutorials that previously required a running BinderHub session can now be embedded directly in documentation pages, blog posts, or course material, with no backend to keep alive.
#Notebooks as apps
SDSS researchers also build internal tools, such as dashboards for data-release validation, quick-look apps for new observations, and scratchpads that quietly graduate into long-lived utilities. marimo notebooks can be served as web apps with one command, which means the same notebook a researcher prototypes in becomes the app the rest of the team uses.
“For years we’ve been saying it would be great if we had some interactive tools that could help debug strange issues as they appear,” Andy says. “And now a graduate student can just make a bespoke interactive tool in minutes.”
#Results
marimo is becoming an important tool for sharing data and analysis in the Sloan Digital Sky Survey. Kayvon and Andy are daily active users, alongside a growing share of the survey’s hundreds of astronomers who reach SDSS data through the marimo apps Kayvon has built and deployed on a BinderHub instance maintained by the Flatiron Institute. Adopting the tool has driven critical results for the collaboration including:
- Speed of insight: problem-to-solution cycles that used to take weeks now happen in minutes. Instead of debating on a call how it would be great to have a tool that does X and parking it indefinitely, a graduate student walks along and shows it before the meeting is over. As Andy put it, “instead of having to figure that out in weeks or whatever, minutes later, somebody has just got it open and we’re already looking and figured out the issue.”
- Years-long wishlists shared internally in days. Tools the collaboration “had been saying it would be great to have for years” have now been deployed.
- Compounding adoption from notebooks-as-apps. Interactive notebooks are immediately available as deployable apps, so once any one collaborator ships a tool the rest of the survey can use it the same day, which has triggered organic adoption across the collaboration.
#Looking ahead
The work SDSS does to make the universe queryable by anyone has always depended on the gap between observation and accessibility being as small as possible. The next step is bringing marimo into the public data release itself: the tools Kayvon has built for collaborators could welcome any astronomer into SDSS data in future releases.
As Kayvon put it about the experience of using marimo more broadly: “Everything just worked smoothly. I was like, that should be a functionality…and then I looked it up, and it just was there.”
marimo is the world’s best programming environment for working with data, and it’s all free and open source. Just pip install marimo or uv add marimo to get started.
