Curating Data with Hugging Face and dltHub
by dltHub
Load, validate, filter, and publish ML datasets using dlt pipelines with Hugging Face and LanceDB.
About this notebook
This notebook loads the OpenVid video dataset from Hugging Face and curates a training-ready subset. The source is the lance-format/openvid-lance parquet dataset. The pipeline uses dlt from dltHub as the loader and LanceDB as the destination. DuckDB reads the remote parquet files, and Ibis runs the analytics. The dlthub.data_quality module runs the validation checks.
You enter a Hugging Face token through an EnvConfig widget. A slider sets how many rows to fetch and load into LanceDB. The notebook shows the loaded schema as a Mermaid diagram and previews the videos table. A group of sliders sets minimum thresholds for aesthetic score, motion, temporal consistency, FPS, and duration. When you move a slider, the notebook filters the data and reports how many videos remain.
DuckDB reads the parquet files through the hf:// adapter. The resource removes heavy columns like video_blob and embedding and keeps only metadata. It streams rows in batches of 1000 into LanceDB. The quality checks validate uniqueness, nullability, score ranges, and camera-motion categories. The final cells show example code to write the filtered dataset back to Hugging Face through the dlt filesystem destination.
