datasets

RAG & Retrieval

A lightweight Python library for one‑line loading and efficient preprocessing of public or local datasets across many modalities (text, audio, image, video, etc.)

Track releases GitHub Website

Python Latest 5.0.0 · 1mo ago Security brief →

Features

One‑line dataset loading from the Hugging Face Hub or local files with `load_dataset()`
Native support for CSV, JSON, Parquet, Arrow, audio/video/PDF/NIfTI and many other formats
Multi‑modal handling (text, image, video, audio, PDF, 3D medical data) in a unified API
Streaming mode for on‑the‑fly iteration without full download
Smart caching and multi‑processing for fast, reproducible preprocessing

Recent releases

View all 19 releases →

No immediate action

5.0.0 Breaking risk 1mo

Agent traces + shuffling + new formats

Open

4.8.5 Bug fix 3mo

Fixed JSON decoding before DataFrame.to_json and related conversions.

Full changelog

Main bug fixes

fix: decode Json() values before calling DataFrame.to_json() (#8116) by @Brianzhengca in https://github.com/huggingface/datasets/pull/8122
Fix: decode JSON type before to_list or to_dict is called by @ItsTania in https://github.com/huggingface/datasets/pull/8137
Fix batching for table-formatted datasets by @bluehyena in https://github.com/huggingface/datasets/pull/8126
Fix iterable map resume state by @Brianzhengca in https://github.com/huggingface/datasets/pull/8147
don't embed remote files in download_and_prepare to parquet by @lhoestq in https://github.com/huggingface/datasets/pull/8150

Other improvements and bug fixes

Parse agent traces by @lhoestq in https://github.com/huggingface/datasets/pull/8113
🔒 Pin GitHub Actions to commit SHAs by @paulinebm in https://github.com/huggingface/datasets/pull/8114
chore: bump doc-builder SHA for PR upload workflow by @rtrompier in https://github.com/huggingface/datasets/pull/8134
Remove print statement in JSON processing by @lhoestq in https://github.com/huggingface/datasets/pull/8136
Don't include files list DatasetInfo (and remove old stuff) by @lhoestq in https://github.com/huggingface/datasets/pull/8128
update ci uer by @lhoestq in https://github.com/huggingface/datasets/pull/8139
fix warning in ci by @lhoestq in https://github.com/huggingface/datasets/pull/8140
fix mask in embed_storage for remote files by @lhoestq in https://github.com/huggingface/datasets/pull/8151
fix original_files missing in ci json test by @lhoestq in https://github.com/huggingface/datasets/pull/8152
Fix null in embed storage by @lhoestq in https://github.com/huggingface/datasets/pull/8154
Fix base_path in integration tests by @lhoestq in https://github.com/huggingface/datasets/pull/8155

New Contributors

@paulinebm made their first contribution in https://github.com/huggingface/datasets/pull/8114
@Brianzhengca made their first contribution in https://github.com/huggingface/datasets/pull/8122
@bluehyena made their first contribution in https://github.com/huggingface/datasets/pull/8126
@rtrompier made their first contribution in https://github.com/huggingface/datasets/pull/8134
@ItsTania made their first contribution in https://github.com/huggingface/datasets/pull/8137

Full Changelog: https://github.com/huggingface/datasets/compare/4.8.4...4.8.5

View release on GitHub

4.8.4 Bug fix 4mo

Notable features

Support for the latest torchvision version

Full changelog

What's Changed

Support latest torchvision by @lhoestq in https://github.com/huggingface/datasets/pull/8087
fix regression when loading JSON with one file = one object by @lhoestq in https://github.com/huggingface/datasets/pull/8086

Full Changelog: https://github.com/huggingface/datasets/compare/4.8.3...4.8.4

View release on GitHub

4.8.3 Bug fix 4mo

Fixed the split_dataset_by_node step and corrected the Json.cast_storage docstring.

Full changelog

What's Changed

Fix split_dataset_by_node step by @lhoestq in https://github.com/huggingface/datasets/pull/8081
Fix docstring of Json.cast_storage by @albertvillanova in https://github.com/huggingface/datasets/pull/8080

Full Changelog: https://github.com/huggingface/datasets/compare/4.8.2...4.8.3

View release on GitHub

4.8.1 Bug fix 4mo

Fixed formatted iter arrow function yielding twice.

Full changelog