datasets

Main bug fixes

• fix: decode Json() values before calling DataFrame.to_json() (#8116) by @Brianzhengca in https://github.com/huggingface/datasets/pull/8122
• Fix: decode JSON type before to_list or to_dict is called by @ItsTania in https://github.com/huggingface/datasets/pull/8137
• Fix batching for table-formatted datasets by @bluehyena in https://github.com/huggingface/datasets/pull/8126
• Fix iterable map resume state by @Brianzhengca in https://github.com/huggingface/datasets/pull/8147
• don't embed remote files in download_and_prepare to parquet by @lhoestq in https://github.com/huggingface/datasets/pull/8150

Other improvements and bug fixes

• Parse agent traces by @lhoestq in https://github.com/huggingface/datasets/pull/8113
• 🔒 Pin GitHub Actions to commit SHAs by @paulinebm in https://github.com/huggingface/datasets/pull/8114
• chore: bump doc-builder SHA for PR upload workflow by @rtrompier in https://github.com/huggingface/datasets/pull/8134
• Remove print statement in JSON processing by @lhoestq in https://github.com/huggingface/datasets/pull/8136
• Don't include files list DatasetInfo (and remove old stuff) by @lhoestq in https://github.com/huggingface/datasets/pull/8128
• update ci uer by @lhoestq in https://github.com/huggingface/datasets/pull/8139
• fix warning in ci by @lhoestq in https://github.com/huggingface/datasets/pull/8140
• fix mask in embed_storage for remote files by @lhoestq in https://github.com/huggingface/datasets/pull/8151
• fix original_files missing in ci json test by @lhoestq in https://github.com/huggingface/datasets/pull/8152
• Fix null in embed storage by @lhoestq in https://github.com/huggingface/datasets/pull/8154
• Fix base_path in integration tests by @lhoestq in https://github.com/huggingface/datasets/pull/8155

New Contributors

• @paulinebm made their first contribution in https://github.com/huggingface/datasets/pull/8114
• @Brianzhengca made their first contribution in https://github.com/huggingface/datasets/pull/8122
• @bluehyena made their first contribution in https://github.com/huggingface/datasets/pull/8126
• @rtrompier made their first contribution in https://github.com/huggingface/datasets/pull/8134
• @ItsTania made their first contribution in https://github.com/huggingface/datasets/pull/8137

Full Changelog: https://github.com/huggingface/datasets/compare/4.8.4...4.8.5

What's Changed

• Support latest torchvision by @lhoestq in https://github.com/huggingface/datasets/pull/8087
• fix regression when loading JSON with one file = one object by @lhoestq in https://github.com/huggingface/datasets/pull/8086

Full Changelog: https://github.com/huggingface/datasets/compare/4.8.3...4.8.4

What's Changed

• Fix split_dataset_by_node step by @lhoestq in https://github.com/huggingface/datasets/pull/8081
• Fix docstring of Json.cast_storage by @albertvillanova in https://github.com/huggingface/datasets/pull/8080

Full Changelog: https://github.com/huggingface/datasets/compare/4.8.2...4.8.3

What's Changed

• Fix formatted iter arrow double yield by @HaukurPall in https://github.com/huggingface/datasets/pull/8063

Full Changelog: https://github.com/huggingface/datasets/compare/4.8.0...4.8.1

Dataset Features

• Read (and write) from HF Storage Buckets: load raw data, process and save to Dataset Repos by @lhoestq in https://github.com/huggingface/datasets/pull/8064

  from datasets import load_dataset
  # load raw data from a Storage Bucket on HF
  ds = load_dataset("buckets/username/data-bucket", data_files=["*.jsonl"])
  # or manually, using hf:// paths
  ds = load_dataset("json", data_files=["hf://buckets/username/data-bucket/*.jsonl"])
  # process, filter
  ds = ds.map(...).filter(...)
  # publish the AI-ready dataset
  ds.push_to_hub("username/my-dataset-ready-for-training")
  

  This also fixes multiprocessed push_to_hub on macos that was causing segfault (now it uses spawn instead of fork).
  And it bumps dill and multiprocess versions to support python 3.14

• Datasets streaming iterable packaged improvements and fixes by @Michael-RDev in https://github.com/huggingface/datasets/pull/8068

  • added max_shard_size to IterableDataset.push_to_hub (but requires iterating twice to know the full dataset twice - improvements are welcome)
  • more arrow-native iterable operations for IterableDataset
  • better support of glob patterns in archives, e.g. zip://*.jsonl::hf://datasets/username/dataset-name/data.zip
  • fixes for to_pandas, videofolder, load_dataset_builder kwargs

What's Changed

• fix reshard_data_sources by @lhoestq in https://github.com/huggingface/datasets/pull/8061
• Improve error message for invalid data_files pattern format by @kushalkkb in https://github.com/huggingface/datasets/pull/8060
• fix null filling in missing jsonl columns by @lhoestq in https://github.com/huggingface/datasets/pull/8069

New Contributors

• @kushalkkb made their first contribution in https://github.com/huggingface/datasets/pull/8060
• @Michael-RDev made their first contribution in https://github.com/huggingface/datasets/pull/8068

Full Changelog: https://github.com/huggingface/datasets/compare/4.7.0...4.8.0