Add polars compatibility

Hi ! thanks for adding polars support :)

You added from_polars in arrow_dataset.py but not to_polars, is this on purpose ?

Also no need to touch table.py imo, which is for arrow-only logic (tables are just wrappers of pyarrow.Table with the exact same methods + optimization to existing methods + separation between in-memory and memory-mapped)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6531). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @lhoestq, thanks for pointing out the missing `to_polars` method.

I see your point about `table.py` so I removed them.

I also added tests in `test_arrow_dataset.py`, `test_dataset_dict.py`, and `test_formatting.py`. Let me know if I am missing any.

duckdb index files were deleted yesterday in dataset_with_script@ref/convert/parquet so I changed the hash to reflect the new SHA.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004993 / 0.011353 (-0.006360) | 0.003658 / 0.011008 (-0.007350) | 0.063868 / 0.038508 (0.025360) | 0.030022 / 0.023109 (0.006912) | 0.246359 / 0.275898 (-0.029539) | 0.273409 / 0.323480 (-0.050070) | 0.003091 / 0.007986 (-0.004894) | 0.003383 / 0.004328 (-0.000945) | 0.050666 / 0.004250 (0.046415) | 0.040609 / 0.037052 (0.003557) | 0.267250 / 0.258489 (0.008761) | 0.289823 / 0.293841 (-0.004018) | 0.027635 / 0.128546 (-0.100911) | 0.010786 / 0.075646 (-0.064860) | 0.208442 / 0.419271 (-0.210830) | 0.036627 / 0.043533 (-0.006906) | 0.254116 / 0.255139 (-0.001023) | 0.274368 / 0.283200 (-0.008832) | 0.018222 / 0.141683 (-0.123460) | 1.184472 / 1.452155 (-0.267683) | 1.194309 / 1.492716 (-0.298407) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092861 / 0.018006 (0.074855) | 0.304736 / 0.000490 (0.304246) | 0.000219 / 0.000200 (0.000019) | 0.000175 / 0.000054 (0.000121) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019378 / 0.037411 (-0.018034) | 0.062342 / 0.014526 (0.047817) | 0.074107 / 0.176557 (-0.102450) | 0.121746 / 0.737135 (-0.615390) | 0.075657 / 0.296338 (-0.220681) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.286474 / 0.215209 (0.071265) | 2.832043 / 2.077655 (0.754389) | 1.453520 / 1.504120 (-0.050600) | 1.324714 / 1.541195 (-0.216480) | 1.335439 / 1.468490 (-0.133051) | 0.571753 / 4.584777 (-4.013024) | 2.427361 / 3.745712 (-1.318352) | 2.899838 / 5.269862 (-2.370024) | 1.775754 / 4.565676 (-2.789922) | 0.064177 / 0.424275 (-0.360098) | 0.004978 / 0.007607 (-0.002629) | 0.343585 / 0.226044 (0.117541) | 3.368494 / 2.268929 (1.099565) | 1.819825 / 55.444624 (-53.624800) | 1.502633 / 6.876477 (-5.373844) | 1.549182 / 2.142072 (-0.592891) | 0.658245 / 4.805227 (-4.146983) | 0.120052 / 6.500664 (-6.380612) | 0.043051 / 0.075469 (-0.032419) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.977055 / 1.841788 (-0.864733) | 11.595567 / 8.074308 (3.521259) | 9.450951 / 10.191392 (-0.740441) | 0.141060 / 0.680424 (-0.539364) | 0.014359 / 0.534201 (-0.519842) | 0.289938 / 0.579283 (-0.289345) | 0.266035 / 0.434364 (-0.168329) | 0.326802 / 0.540337 (-0.213536) | 0.431913 / 1.386936 (-0.955023) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005391 / 0.011353 (-0.005961) | 0.003724 / 0.011008 (-0.007284) | 0.050432 / 0.038508 (0.011924) | 0.029904 / 0.023109 (0.006794) | 0.270870 / 0.275898 (-0.005028) | 0.296773 / 0.323480 (-0.026706) | 0.004265 / 0.007986 (-0.003721) | 0.002751 / 0.004328 (-0.001577) | 0.050366 / 0.004250 (0.046116) | 0.046415 / 0.037052 (0.009363) | 0.283272 / 0.258489 (0.024783) | 0.320188 / 0.293841 (0.026347) | 0.029827 / 0.128546 (-0.098719) | 0.010736 / 0.075646 (-0.064910) | 0.059541 / 0.419271 (-0.359731) | 0.057080 / 0.043533 (0.013548) | 0.270653 / 0.255139 (0.015514) | 0.291235 / 0.283200 (0.008035) | 0.018590 / 0.141683 (-0.123093) | 1.129402 / 1.452155 (-0.322752) | 1.194499 / 1.492716 (-0.298217) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.102220 / 0.018006 (0.084214) | 0.302176 / 0.000490 (0.301686) | 0.000229 / 0.000200 (0.000029) | 0.000056 / 0.000054 (0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022809 / 0.037411 (-0.014602) | 0.076054 / 0.014526 (0.061528) | 0.087466 / 0.176557 (-0.089091) | 0.128495 / 0.737135 (-0.608640) | 0.089933 / 0.296338 (-0.206406) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.296546 / 0.215209 (0.081337) | 2.898693 / 2.077655 (0.821039) | 1.605002 / 1.504120 (0.100883) | 1.468370 / 1.541195 (-0.072825) | 1.503541 / 1.468490 (0.035051) | 0.577233 / 4.584777 (-4.007544) | 2.460154 / 3.745712 (-1.285558) | 2.755651 / 5.269862 (-2.514211) | 1.777711 / 4.565676 (-2.787966) | 0.063137 / 0.424275 (-0.361138) | 0.005056 / 0.007607 (-0.002551) | 0.350189 / 0.226044 (0.124145) | 3.485473 / 2.268929 (1.216545) | 1.952553 / 55.444624 (-53.492072) | 1.669108 / 6.876477 (-5.207369) | 1.788504 / 2.142072 (-0.353569) | 0.672869 / 4.805227 (-4.132359) | 0.117717 / 6.500664 (-6.382948) | 0.040499 / 0.075469 (-0.034970) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.048187 / 1.841788 (-0.793601) | 12.663229 / 8.074308 (4.588921) | 10.316487 / 10.191392 (0.125095) | 0.142537 / 0.680424 (-0.537887) | 0.016024 / 0.534201 (-0.518177) | 0.292735 / 0.579283 (-0.286548) | 0.273294 / 0.434364 (-0.161069) | 0.327636 / 0.540337 (-0.212701) | 0.443062 / 1.386936 (-0.943874) |

![](https://cml.dev/watermark.png#90b896193a0a315789022580eb4c80305d168d4d "CML watermark")

I'm so excited I tweeted about it: https://x.com/qlhoest/status/1766135995513082086?s=20 I hope it's fine !

Thanks @lhoestq for the support and totally fine with the share! Happy to see people excited for this 😃

Hey there,

I've just finished adding support to convert and format to `polars.DataFrame`. This was in response to the open issue about integrating Polars [#3334](https://github.com/huggingface/datasets/issues/3334). Datasets can be switched to Polars format via `Dataset.set_format("polars")`. I've also included `to_polars` and `from_polars`. All polars functions are checked via config.POLARS_AVAILABLE.

A few notes:
This only supports `DataFrames` and not `LazyFrames`. This probably could be integrated fairly easily via `is_lazy` args in `set_format`, and `to_polars`.

Let me know your feedbacks.


Impossible to save a mapped dataset to disk

I solved it with `train_dataset.with_format(None)`
But then faced some more issues (which i later solved too).

Huggingface does not seem to care, so I do. Here is an updated training script which saves a pre-processed (mapped) dataset to your local directory if you specify `--save_precomputed_data_dir=DIR_NAME`. Then use `--train_precomputed_data_dir` with the same dir to load it instead of `--dataset_name`.

[Proper SDXL trainer code](https://github.com/kopyl/diffusers/blob/main/examples/text_to_image/train_text_to_image_sdxl.py)
[Notebook for pre-computing a dataset and saving locally](https://colab.research.google.com/drive/17Yo08hePx-NlHs99RecdeiNc8CQg4O7l?usp=sharing)

Example:

1st run (nothing is pre-computed yet):
```
accelerate launch train_text_to_image_sdxl.py \
    --pretrained_model_name_or_path=stabilityai/stable-diffusion-xl-base-1.0 \
    --pretrained_vae_model_name_or_path=madebyollin/sdxl-vae-fp16-fix \
    --dataset_name=lambdalabs/pokemon-blip-captions \
    --save_precomputed_data_dir="test-5"
```

2nd run (the pre-computed dataset is saved to `test-5` directory):
```
accelerate launch train_text_to_image_sdxl.py \
    --pretrained_model_name_or_path=stabilityai/stable-diffusion-xl-base-1.0 \
    --pretrained_vae_model_name_or_path=madebyollin/sdxl-vae-fp16-fix \
    --train_precomputed_data_dir test-5
```

This way when you're gonna be using your pre-computed dataset you don't need to worry about re-mapping your dataset when you change an argument for your trainer script

### Describe the bug

I want to play around with different hyperparameters when training but don't want to re-map my dataset with 3 million samples each time for tens of hours when I [fully fine-tune SDXL](https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_sdxl.py).

After I do the mapping like this:
```
train_dataset = train_dataset.map(compute_embeddings_fn, batched=True)
train_dataset = train_dataset.map(
    compute_vae_encodings_fn,
    batched=True,
    batch_size=16,
)
```
and try to save it like this:
`train_dataset.save_to_disk("test")`
i get this error ([full traceback](https://pastebin.com/kq3vt739)):
```
TypeError: Object of type function is not JSON serializable
The format kwargs must be JSON serializable, but key 'transform' isn't.
```

But what is interesting is that pushing to hub works like that:
`train_dataset.push_to_hub("kopyl/mapped-833-icons-sdxl-1024-dataset", token=True)`
Here is the link of the pushed dataset: https://huggingface.co/datasets/kopyl/mapped-833-icons-sdxl-1024-dataset

### Steps to reproduce the bug

Here is the self-contained notebook:

https://colab.research.google.com/drive/1RtCsEMVcwWcMwlWURk_cj_9xUBHz065M?usp=sharing

### Expected behavior

It should be easily saved to disk

### Environment info

NVIDIA A100, Linux (NC24ads A100 v4 from Azure), CUDA 12.2.

[pip freeze](https://pastebin.com/QTNb6iru)


Impossible to only download a test split

The only way right now is to load with streaming=True

This feature has been proposed for a long time. I'm looking forward to the implementation. On clusters `streaming=True` is not an option since we do not have Internet on compute nodes. See: https://github.com/huggingface/datasets/discussions/1896#discussioncomment-2359593

I've spent a significant amount of time trying to locate the split object inside my _split_generators() custom function.
Then after diving [in the code](https://github.com/huggingface/datasets/blob/5ff3670c18ed34fa8ddfa70a9aa403ae6cc9ad54/src/datasets/load.py#L2558) I realized that `download_and_prepare` is executed before! split is passed to the dataset builder in `as_dataset`.

If I'm not missing something, this seems like bad design, for the following use case:

> Imagine there is a huge dataset that has an evaluation test set and you want to just download and run just to compare your method.

Is there a current workaround that can help me achieve the same result?

Thank you,


set dev version

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6528). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004875 / 0.011353 (-0.006478) | 0.003501 / 0.011008 (-0.007507) | 0.062604 / 0.038508 (0.024096) | 0.031916 / 0.023109 (0.008806) | 0.256138 / 0.275898 (-0.019760) | 0.278514 / 0.323480 (-0.044966) | 0.002917 / 0.007986 (-0.005069) | 0.002636 / 0.004328 (-0.001693) | 0.049154 / 0.004250 (0.044904) | 0.041985 / 0.037052 (0.004933) | 0.256857 / 0.258489 (-0.001632) | 0.282628 / 0.293841 (-0.011213) | 0.027506 / 0.128546 (-0.101041) | 0.010736 / 0.075646 (-0.064910) | 0.207268 / 0.419271 (-0.212003) | 0.035312 / 0.043533 (-0.008221) | 0.259274 / 0.255139 (0.004135) | 0.281463 / 0.283200 (-0.001737) | 0.019905 / 0.141683 (-0.121778) | 1.108719 / 1.452155 (-0.343435) | 1.177871 / 1.492716 (-0.314845) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.004435 / 0.018006 (-0.013571) | 0.310643 / 0.000490 (0.310153) | 0.000243 / 0.000200 (0.000043) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018013 / 0.037411 (-0.019398) | 0.060702 / 0.014526 (0.046176) | 0.073243 / 0.176557 (-0.103314) | 0.119523 / 0.737135 (-0.617613) | 0.074204 / 0.296338 (-0.222134) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.281075 / 0.215209 (0.065866) | 2.722154 / 2.077655 (0.644499) | 1.441052 / 1.504120 (-0.063068) | 1.305940 / 1.541195 (-0.235255) | 1.356752 / 1.468490 (-0.111738) | 0.570399 / 4.584777 (-4.014378) | 2.329158 / 3.745712 (-1.416554) | 2.749093 / 5.269862 (-2.520768) | 1.717752 / 4.565676 (-2.847925) | 0.063228 / 0.424275 (-0.361047) | 0.004981 / 0.007607 (-0.002626) | 0.330601 / 0.226044 (0.104557) | 3.300987 / 2.268929 (1.032059) | 1.778673 / 55.444624 (-53.665951) | 1.507841 / 6.876477 (-5.368636) | 1.520454 / 2.142072 (-0.621619) | 0.650816 / 4.805227 (-4.154412) | 0.118606 / 6.500664 (-6.382058) | 0.042199 / 0.075469 (-0.033271) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.919668 / 1.841788 (-0.922119) | 11.293437 / 8.074308 (3.219129) | 9.928525 / 10.191392 (-0.262867) | 0.127142 / 0.680424 (-0.553282) | 0.013470 / 0.534201 (-0.520731) | 0.284648 / 0.579283 (-0.294636) | 0.264942 / 0.434364 (-0.169422) | 0.321866 / 0.540337 (-0.218471) | 0.414513 / 1.386936 (-0.972423) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005052 / 0.011353 (-0.006301) | 0.003204 / 0.011008 (-0.007804) | 0.051102 / 0.038508 (0.012594) | 0.032105 / 0.023109 (0.008996) | 0.273923 / 0.275898 (-0.001976) | 0.297031 / 0.323480 (-0.026449) | 0.004002 / 0.007986 (-0.003984) | 0.002636 / 0.004328 (-0.001693) | 0.047696 / 0.004250 (0.043445) | 0.044086 / 0.037052 (0.007034) | 0.277779 / 0.258489 (0.019289) | 0.306678 / 0.293841 (0.012837) | 0.028557 / 0.128546 (-0.099989) | 0.010631 / 0.075646 (-0.065015) | 0.056419 / 0.419271 (-0.362852) | 0.054285 / 0.043533 (0.010752) | 0.276506 / 0.255139 (0.021367) | 0.296315 / 0.283200 (0.013116) | 0.018642 / 0.141683 (-0.123040) | 1.146926 / 1.452155 (-0.305229) | 1.257625 / 1.492716 (-0.235092) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094231 / 0.018006 (0.076225) | 0.302805 / 0.000490 (0.302315) | 0.000229 / 0.000200 (0.000029) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022510 / 0.037411 (-0.014901) | 0.076092 / 0.014526 (0.061566) | 0.090642 / 0.176557 (-0.085915) | 0.127016 / 0.737135 (-0.610120) | 0.089169 / 0.296338 (-0.207169) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.290812 / 0.215209 (0.075603) | 2.858528 / 2.077655 (0.780873) | 1.577555 / 1.504120 (0.073436) | 1.447810 / 1.541195 (-0.093384) | 1.447546 / 1.468490 (-0.020944) | 0.559118 / 4.584777 (-4.025659) | 2.408930 / 3.745712 (-1.336782) | 2.733761 / 5.269862 (-2.536101) | 1.700107 / 4.565676 (-2.865570) | 0.062447 / 0.424275 (-0.361828) | 0.004999 / 0.007607 (-0.002608) | 0.340207 / 0.226044 (0.114162) | 3.344131 / 2.268929 (1.075203) | 1.902289 / 55.444624 (-53.542335) | 1.628226 / 6.876477 (-5.248251) | 1.629435 / 2.142072 (-0.512637) | 0.625011 / 4.805227 (-4.180216) | 0.119929 / 6.500664 (-6.380735) | 0.041097 / 0.075469 (-0.034372) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.977461 / 1.841788 (-0.864327) | 12.303189 / 8.074308 (4.228881) | 11.008743 / 10.191392 (0.817351) | 0.128578 / 0.680424 (-0.551845) | 0.015305 / 0.534201 (-0.518896) | 0.286468 / 0.579283 (-0.292816) | 0.275824 / 0.434364 (-0.158540) | 0.321487 / 0.540337 (-0.218851) | 0.420591 / 1.386936 (-0.966345) |

![](https://cml.dev/watermark.png#5ff3670c18ed34fa8ddfa70a9aa403ae6cc9ad54 "CML watermark")


Release: 2.16.0

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6527). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004870 / 0.011353 (-0.006483) | 0.003606 / 0.011008 (-0.007402) | 0.062719 / 0.038508 (0.024211) | 0.031785 / 0.023109 (0.008676) | 0.238809 / 0.275898 (-0.037089) | 0.263000 / 0.323480 (-0.060480) | 0.002844 / 0.007986 (-0.005142) | 0.002698 / 0.004328 (-0.001631) | 0.048070 / 0.004250 (0.043819) | 0.042333 / 0.037052 (0.005280) | 0.243032 / 0.258489 (-0.015457) | 0.273197 / 0.293841 (-0.020644) | 0.027498 / 0.128546 (-0.101048) | 0.010592 / 0.075646 (-0.065055) | 0.204770 / 0.419271 (-0.214502) | 0.034837 / 0.043533 (-0.008696) | 0.242518 / 0.255139 (-0.012621) | 0.267461 / 0.283200 (-0.015739) | 0.018479 / 0.141683 (-0.123204) | 1.105444 / 1.452155 (-0.346710) | 1.163659 / 1.492716 (-0.329057) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.004717 / 0.018006 (-0.013289) | 0.303338 / 0.000490 (0.302849) | 0.000221 / 0.000200 (0.000021) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018298 / 0.037411 (-0.019113) | 0.061225 / 0.014526 (0.046699) | 0.073514 / 0.176557 (-0.103043) | 0.120230 / 0.737135 (-0.616905) | 0.076195 / 0.296338 (-0.220144) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.284731 / 0.215209 (0.069522) | 2.773463 / 2.077655 (0.695809) | 1.498239 / 1.504120 (-0.005881) | 1.372143 / 1.541195 (-0.169052) | 1.448949 / 1.468490 (-0.019542) | 0.572516 / 4.584777 (-4.012261) | 2.404041 / 3.745712 (-1.341671) | 2.763047 / 5.269862 (-2.506814) | 1.722419 / 4.565676 (-2.843257) | 0.063104 / 0.424275 (-0.361172) | 0.004989 / 0.007607 (-0.002618) | 0.341864 / 0.226044 (0.115820) | 3.391635 / 2.268929 (1.122707) | 1.872694 / 55.444624 (-53.571931) | 1.594490 / 6.876477 (-5.281987) | 1.596940 / 2.142072 (-0.545132) | 0.645265 / 4.805227 (-4.159962) | 0.117408 / 6.500664 (-6.383256) | 0.042405 / 0.075469 (-0.033064) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.963207 / 1.841788 (-0.878580) | 11.676551 / 8.074308 (3.602243) | 10.194287 / 10.191392 (0.002895) | 0.130329 / 0.680424 (-0.550094) | 0.015381 / 0.534201 (-0.518820) | 0.288848 / 0.579283 (-0.290435) | 0.264781 / 0.434364 (-0.169583) | 0.321212 / 0.540337 (-0.219126) | 0.418308 / 1.386936 (-0.968628) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005533 / 0.011353 (-0.005819) | 0.003733 / 0.011008 (-0.007276) | 0.048877 / 0.038508 (0.010369) | 0.030263 / 0.023109 (0.007154) | 0.281161 / 0.275898 (0.005263) | 0.302971 / 0.323480 (-0.020509) | 0.003943 / 0.007986 (-0.004043) | 0.002717 / 0.004328 (-0.001612) | 0.047845 / 0.004250 (0.043594) | 0.045809 / 0.037052 (0.008757) | 0.283337 / 0.258489 (0.024848) | 0.312914 / 0.293841 (0.019073) | 0.029074 / 0.128546 (-0.099472) | 0.010775 / 0.075646 (-0.064871) | 0.057461 / 0.419271 (-0.361810) | 0.053756 / 0.043533 (0.010223) | 0.281809 / 0.255139 (0.026670) | 0.298339 / 0.283200 (0.015139) | 0.019270 / 0.141683 (-0.122413) | 1.117575 / 1.452155 (-0.334580) | 1.191703 / 1.492716 (-0.301013) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093513 / 0.018006 (0.075507) | 0.301267 / 0.000490 (0.300777) | 0.000211 / 0.000200 (0.000012) | 0.000045 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022278 / 0.037411 (-0.015133) | 0.076805 / 0.014526 (0.062279) | 0.088820 / 0.176557 (-0.087736) | 0.127903 / 0.737135 (-0.609233) | 0.092988 / 0.296338 (-0.203350) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.297787 / 0.215209 (0.082578) | 2.899652 / 2.077655 (0.821997) | 1.598830 / 1.504120 (0.094710) | 1.469398 / 1.541195 (-0.071797) | 1.511099 / 1.468490 (0.042609) | 0.559785 / 4.584777 (-4.024992) | 2.426448 / 3.745712 (-1.319264) | 2.798811 / 5.269862 (-2.471051) | 1.737790 / 4.565676 (-2.827887) | 0.062219 / 0.424275 (-0.362056) | 0.005120 / 0.007607 (-0.002487) | 0.351051 / 0.226044 (0.125007) | 3.492063 / 2.268929 (1.223134) | 1.965674 / 55.444624 (-53.478950) | 1.672874 / 6.876477 (-5.203603) | 1.709700 / 2.142072 (-0.432373) | 0.639347 / 4.805227 (-4.165880) | 0.126383 / 6.500664 (-6.374281) | 0.042731 / 0.075469 (-0.032738) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.968619 / 1.841788 (-0.873168) | 12.671030 / 8.074308 (4.596722) | 11.125347 / 10.191392 (0.933955) | 0.142983 / 0.680424 (-0.537441) | 0.015726 / 0.534201 (-0.518475) | 0.288610 / 0.579283 (-0.290673) | 0.276473 / 0.434364 (-0.157891) | 0.326590 / 0.540337 (-0.213748) | 0.423832 / 1.386936 (-0.963104) |

![](https://cml.dev/watermark.png#a85fb52fc8ddb41307e61cbf6a5189f3bba27829 "CML watermark")


Preserve order of configs and splits when using Parquet exports

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6526). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005101 / 0.011353 (-0.006252) | 0.003471 / 0.011008 (-0.007537) | 0.062293 / 0.038508 (0.023785) | 0.032650 / 0.023109 (0.009541) | 0.249241 / 0.275898 (-0.026657) | 0.277079 / 0.323480 (-0.046400) | 0.002971 / 0.007986 (-0.005015) | 0.002637 / 0.004328 (-0.001691) | 0.048415 / 0.004250 (0.044165) | 0.042832 / 0.037052 (0.005779) | 0.247840 / 0.258489 (-0.010649) | 0.283994 / 0.293841 (-0.009847) | 0.027764 / 0.128546 (-0.100782) | 0.010544 / 0.075646 (-0.065102) | 0.208810 / 0.419271 (-0.210462) | 0.035744 / 0.043533 (-0.007789) | 0.252811 / 0.255139 (-0.002328) | 0.276163 / 0.283200 (-0.007036) | 0.018581 / 0.141683 (-0.123102) | 1.130043 / 1.452155 (-0.322112) | 1.194298 / 1.492716 (-0.298418) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.004488 / 0.018006 (-0.013518) | 0.302072 / 0.000490 (0.301582) | 0.000211 / 0.000200 (0.000012) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.017799 / 0.037411 (-0.019613) | 0.061146 / 0.014526 (0.046620) | 0.081796 / 0.176557 (-0.094761) | 0.120407 / 0.737135 (-0.616729) | 0.075211 / 0.296338 (-0.221127) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295349 / 0.215209 (0.080140) | 2.953511 / 2.077655 (0.875857) | 1.495332 / 1.504120 (-0.008788) | 1.364144 / 1.541195 (-0.177051) | 1.429562 / 1.468490 (-0.038928) | 0.574325 / 4.584777 (-4.010452) | 2.384352 / 3.745712 (-1.361360) | 2.843625 / 5.269862 (-2.426236) | 1.806802 / 4.565676 (-2.758875) | 0.065076 / 0.424275 (-0.359199) | 0.004970 / 0.007607 (-0.002638) | 0.339935 / 0.226044 (0.113891) | 3.375103 / 2.268929 (1.106175) | 1.822921 / 55.444624 (-53.621703) | 1.546126 / 6.876477 (-5.330350) | 1.573630 / 2.142072 (-0.568442) | 0.655081 / 4.805227 (-4.150146) | 0.122446 / 6.500664 (-6.378218) | 0.042220 / 0.075469 (-0.033249) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.942127 / 1.841788 (-0.899661) | 11.470401 / 8.074308 (3.396093) | 10.025961 / 10.191392 (-0.165431) | 0.129087 / 0.680424 (-0.551337) | 0.014141 / 0.534201 (-0.520060) | 0.285470 / 0.579283 (-0.293813) | 0.266755 / 0.434364 (-0.167608) | 0.323391 / 0.540337 (-0.216947) | 0.427645 / 1.386936 (-0.959291) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005578 / 0.011353 (-0.005775) | 0.003734 / 0.011008 (-0.007274) | 0.049200 / 0.038508 (0.010692) | 0.030981 / 0.023109 (0.007872) | 0.281195 / 0.275898 (0.005297) | 0.309950 / 0.323480 (-0.013530) | 0.004046 / 0.007986 (-0.003939) | 0.002709 / 0.004328 (-0.001620) | 0.048505 / 0.004250 (0.044254) | 0.046245 / 0.037052 (0.009193) | 0.280130 / 0.258489 (0.021641) | 0.313739 / 0.293841 (0.019898) | 0.029828 / 0.128546 (-0.098718) | 0.011152 / 0.075646 (-0.064495) | 0.057753 / 0.419271 (-0.361518) | 0.055112 / 0.043533 (0.011580) | 0.281861 / 0.255139 (0.026722) | 0.304402 / 0.283200 (0.021203) | 0.019931 / 0.141683 (-0.121752) | 1.150585 / 1.452155 (-0.301570) | 1.217850 / 1.492716 (-0.274866) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091552 / 0.018006 (0.073546) | 0.301772 / 0.000490 (0.301282) | 0.000225 / 0.000200 (0.000025) | 0.000046 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023189 / 0.037411 (-0.014223) | 0.078741 / 0.014526 (0.064216) | 0.092320 / 0.176557 (-0.084236) | 0.129636 / 0.737135 (-0.607500) | 0.091673 / 0.296338 (-0.204665) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.298542 / 0.215209 (0.083333) | 2.899358 / 2.077655 (0.821703) | 1.673896 / 1.504120 (0.169776) | 1.489518 / 1.541195 (-0.051677) | 1.542853 / 1.468490 (0.074363) | 0.559843 / 4.584777 (-4.024934) | 2.422101 / 3.745712 (-1.323611) | 2.844592 / 5.269862 (-2.425270) | 1.794527 / 4.565676 (-2.771150) | 0.064615 / 0.424275 (-0.359660) | 0.005078 / 0.007607 (-0.002530) | 0.355112 / 0.226044 (0.129068) | 3.462129 / 2.268929 (1.193200) | 1.975393 / 55.444624 (-53.469231) | 1.706513 / 6.876477 (-5.169963) | 1.716954 / 2.142072 (-0.425118) | 0.642094 / 4.805227 (-4.163133) | 0.119215 / 6.500664 (-6.381449) | 0.041941 / 0.075469 (-0.033528) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.986774 / 1.841788 (-0.855014) | 12.702049 / 8.074308 (4.627741) | 11.727663 / 10.191392 (1.536271) | 0.135008 / 0.680424 (-0.545416) | 0.016055 / 0.534201 (-0.518146) | 0.293564 / 0.579283 (-0.285719) | 0.284884 / 0.434364 (-0.149480) | 0.332524 / 0.540337 (-0.207814) | 0.425392 / 1.386936 (-0.961544) |

![](https://cml.dev/watermark.png#7b5fc585fcaf77b92839e82d0ce2c2fbf0d9ea95 "CML watermark")

Preserve order of configs and splits, as defined in dataset infos.

Fix #6521.


BBox type

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6525). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

closing in favor of other ideas that would not involve any typing

see [internal discussion](https://huggingface.slack.com/archives/C02EK7C3SHW/p1703097195609209)

Draft to get some feedback on a possible `BBox` feature type that can be used to get object detection bounding boxes data in one format or another.

```python
>>> from datasets import load_dataset, BBox
>>> ds = load_dataset("svhn", "full_numbers", split="train")
>>> ds[0]
{
  'image': <PIL.PngImagePlugin.PngImageFile image mode=RGB size=107x46 at 0x126409BE0>,
  'digits': {'bbox': [[38, 1, 21, 40], [57, 3, 16, 40]], 'label': [4, 6]}
}
>>> ds = ds.rename_column("digits", "annotations").cast_column("annotations", BBox(format="coco"))
>>> ds[0]
{
  'image': <PIL.PngImagePlugin.PngImageFile image mode=RGB size=107x46 at 0x147730070>,
  'annotations': [{'bbox': [38, 1, 21, 40], 'category_id': 4}, {'bbox': [57, 3, 16, 40], 'category_id': 6}]
}
```

note that it's a type for a list of bounding boxes, not just one - which would be needed to switch from a format to another using type casting.


Streaming the Pile: Missing Files

Hello @FelixLabelle,

As you can see in the Community tab of the corresponding dataset, it is a known issue: https://huggingface.co/datasets/EleutherAI/pile/discussions/15

The data has been taken down due to reported copyright infringement.

Feel free to continue the discussion there.

### Describe the bug

The pile does not stream, a "File not Found error" is returned. It looks like the Pile's files have been moved.

### Steps to reproduce the bug

To reproduce run the following code:

```
from datasets import load_dataset

dataset = load_dataset('EleutherAI/pile', 'en', split='train', streaming=True)
next(iter(dataset))
```

I get the following error:
`FileNotFoundError: https://the-eye.eu/public/AI/pile/train/00.jsonl.zst`

### Expected behavior

Return the data in a stream.

### Environment info

- `datasets` version: 2.12.0
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.11.5
- Huggingface_hub version: 0.15.1
- PyArrow version: 11.0.0
- Pandas version: 2.0.3


fix tests

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6523). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005160 / 0.011353 (-0.006192) | 0.003962 / 0.011008 (-0.007046) | 0.064952 / 0.038508 (0.026444) | 0.053291 / 0.023109 (0.030182) | 0.237182 / 0.275898 (-0.038716) | 0.263855 / 0.323480 (-0.059625) | 0.004157 / 0.007986 (-0.003829) | 0.002901 / 0.004328 (-0.001428) | 0.050679 / 0.004250 (0.046428) | 0.044885 / 0.037052 (0.007832) | 0.243806 / 0.258489 (-0.014683) | 0.273828 / 0.293841 (-0.020013) | 0.028681 / 0.128546 (-0.099866) | 0.011086 / 0.075646 (-0.064560) | 0.211987 / 0.419271 (-0.207285) | 0.035881 / 0.043533 (-0.007652) | 0.249618 / 0.255139 (-0.005521) | 0.262880 / 0.283200 (-0.020319) | 0.017788 / 0.141683 (-0.123895) | 1.209060 / 1.452155 (-0.243094) | 1.272143 / 1.492716 (-0.220574) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.004594 / 0.018006 (-0.013412) | 0.305188 / 0.000490 (0.304698) | 0.000213 / 0.000200 (0.000013) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019526 / 0.037411 (-0.017886) | 0.062280 / 0.014526 (0.047754) | 0.074983 / 0.176557 (-0.101573) | 0.123466 / 0.737135 (-0.613670) | 0.076240 / 0.296338 (-0.220099) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.276001 / 0.215209 (0.060792) | 2.689614 / 2.077655 (0.611959) | 1.441092 / 1.504120 (-0.063028) | 1.319775 / 1.541195 (-0.221419) | 1.386904 / 1.468490 (-0.081587) | 0.561388 / 4.584777 (-4.023389) | 2.386718 / 3.745712 (-1.358994) | 2.813959 / 5.269862 (-2.455903) | 1.727447 / 4.565676 (-2.838230) | 0.061965 / 0.424275 (-0.362310) | 0.004977 / 0.007607 (-0.002630) | 0.335077 / 0.226044 (0.109032) | 3.313860 / 2.268929 (1.044932) | 1.814018 / 55.444624 (-53.630606) | 1.542840 / 6.876477 (-5.333637) | 1.586887 / 2.142072 (-0.555185) | 0.643225 / 4.805227 (-4.162002) | 0.117834 / 6.500664 (-6.382830) | 0.044024 / 0.075469 (-0.031445) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.952804 / 1.841788 (-0.888984) | 12.447378 / 8.074308 (4.373070) | 11.281734 / 10.191392 (1.090342) | 0.143407 / 0.680424 (-0.537017) | 0.014749 / 0.534201 (-0.519452) | 0.289298 / 0.579283 (-0.289985) | 0.268217 / 0.434364 (-0.166146) | 0.327995 / 0.540337 (-0.212343) | 0.430302 / 1.386936 (-0.956634) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005683 / 0.011353 (-0.005670) | 0.003813 / 0.011008 (-0.007195) | 0.048943 / 0.038508 (0.010435) | 0.060730 / 0.023109 (0.037621) | 0.266925 / 0.275898 (-0.008973) | 0.292553 / 0.323480 (-0.030927) | 0.004236 / 0.007986 (-0.003750) | 0.002790 / 0.004328 (-0.001538) | 0.048962 / 0.004250 (0.044711) | 0.040354 / 0.037052 (0.003302) | 0.266353 / 0.258489 (0.007864) | 0.298397 / 0.293841 (0.004556) | 0.029977 / 0.128546 (-0.098570) | 0.010788 / 0.075646 (-0.064858) | 0.057529 / 0.419271 (-0.361743) | 0.032896 / 0.043533 (-0.010636) | 0.266696 / 0.255139 (0.011557) | 0.283422 / 0.283200 (0.000223) | 0.020939 / 0.141683 (-0.120744) | 1.169867 / 1.452155 (-0.282287) | 1.213586 / 1.492716 (-0.279130) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.097035 / 0.018006 (0.079029) | 0.306968 / 0.000490 (0.306478) | 0.000234 / 0.000200 (0.000034) | 0.000046 / 0.000054 (-0.000008) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023343 / 0.037411 (-0.014068) | 0.078238 / 0.014526 (0.063712) | 0.091083 / 0.176557 (-0.085474) | 0.131487 / 0.737135 (-0.605649) | 0.092614 / 0.296338 (-0.203724) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.294454 / 0.215209 (0.079245) | 2.881053 / 2.077655 (0.803398) | 1.623934 / 1.504120 (0.119814) | 1.509001 / 1.541195 (-0.032194) | 1.567541 / 1.468490 (0.099051) | 0.574326 / 4.584777 (-4.010451) | 2.476826 / 3.745712 (-1.268886) | 2.826183 / 5.269862 (-2.443678) | 1.771949 / 4.565676 (-2.793727) | 0.063663 / 0.424275 (-0.360613) | 0.005039 / 0.007607 (-0.002568) | 0.354861 / 0.226044 (0.128816) | 3.397655 / 2.268929 (1.128727) | 1.961958 / 55.444624 (-53.482666) | 1.694795 / 6.876477 (-5.181682) | 1.719459 / 2.142072 (-0.422614) | 0.654512 / 4.805227 (-4.150715) | 0.119285 / 6.500664 (-6.381379) | 0.042146 / 0.075469 (-0.033323) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.982187 / 1.841788 (-0.859601) | 12.944627 / 8.074308 (4.870319) | 11.370381 / 10.191392 (1.178989) | 0.142759 / 0.680424 (-0.537665) | 0.016319 / 0.534201 (-0.517882) | 0.291339 / 0.579283 (-0.287944) | 0.276842 / 0.434364 (-0.157522) | 0.324285 / 0.540337 (-0.216052) | 0.426234 / 1.386936 (-0.960702) |

![](https://cml.dev/watermark.png#e1b82eaa75d2c610e59b463a67d685ec858c0838 "CML watermark")


Loading HF Hub Dataset (private org repo) fails to load all features

### Describe the bug

When pushing a `Dataset` with multiple `Features` (`input`, `output`, `tags`) to Huggingface Hub (private org repo), and later downloading the `Dataset`, only `input` and `output` load - I believe the expected behavior is for all `Features` to be loaded by default?

### Steps to reproduce the bug

Pushing the data. `data_concat` is a `list` of `dict`s.
```python
for datum in data_concat:
    datum_tags = {d["key"]: d["value"] for d in datum["tags"]}
    split_fraction = # some logic that generates a train/test split number
    if split_faction < test_fraction:
        data_test.append(datum)
    else:
        data_train.append(datum)

dataset = DatasetDict(
    {
        "train": Dataset.from_list(data_train),
        "test": Dataset.from_list(data_test),
        "full": Dataset.from_list(data_concat),
    },
)

dataset_shuffled = dataset.shuffle(seed=shuffle_seed)

dataset_shuffled.push_to_hub(
    repo_id=hf_repo_id,
    private=True,
    config_name=m,
    revision=revision,
    token=hf_token,
)
```

Loading it later:
```python
        dataset = datasets.load_dataset(
            path=hf_repo_id,
            name=name,
            token=hf_token,
        )
```

Produces:
```
DatasetDict({
    train: Dataset({
        features: ['input', 'output'],
        num_rows: <obfuscated>
    })
    test: Dataset({
        features: ['input', 'output'],
        num_rows: <obfuscated>
    })
    full: Dataset({
        features: ['input', 'output'],
        num_rows: <obfuscated>
    })
})
```

### Expected behavior

The expected result is below:
```
DatasetDict({
    train: Dataset({
        features: ['input', 'output', 'tags'],
        num_rows: <obfuscated>
    })
    test: Dataset({
        features: ['input', 'output', 'tags'],
        num_rows: <obfuscated>
    })
    full: Dataset({
        features: ['input', 'output', 'tags'],
        num_rows: <obfuscated>
    })
})
```

My workaround is as follows:
```python
dsinfo = datasets.get_dataset_config_info(
    path=data_files,
    config_name=data_config,
    token=hf_token,
)

allfeatures = dsinfo.features.copy()
if "tags" not in allfeatures:
    allfeatures["tags"] = [{"key": Value(dtype="string", id=None), "value": Value(dtype="string", id=None)}]

dataset = datasets.load_dataset(
    path=data_files,
    name=data_config,
    features=allfeatures,
    token=hf_token,
)
```

Interestingly enough (and perhaps a related bug?), if I don't add the `tags` to `allfeatures` above (i.e. only loading `input` and `output`), it throws an error when executing `load_dataset`:

```
ValueError: Couldn't cast
tags: list<element: struct<key: string, value: string>>
  child 0, element: struct<key: string, value: string>
      child 0, key: string
      child 1, value: string
input: <obfuscated>
output: <obfuscated>
-- schema metadata --
huggingface: '{"info": {"features": {"tags": [{"key": {"dtype": "string",' + 532
to
{'input': <obfuscated>, 'output': <obfuscated>
because column names don't match
```

Traceback for this:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/bt/github/core/.venv/lib/python3.11/site-packages/datasets/load.py", line 2152, in load_dataset
    builder_instance.download_and_prepare(
  File "/Users/bt/github/core/.venv/lib/python3.11/site-packages/datasets/builder.py", line 948, in download_and_prepare
    self._download_and_prepare(
  File "/Users/bt/github/core/.venv/lib/python3.11/site-packages/datasets/builder.py", line 1043, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/Users/bt/github/core/.venv/lib/python3.11/site-packages/datasets/builder.py", line 1805, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/Users/bt/github/core/.venv/lib/python3.11/site-packages/datasets/builder.py", line 1950, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
```

### Environment info

- `datasets` version: 2.15.0
- Platform: macOS-14.0-arm64-arm-64bit
- Python version: 3.11.5
- `huggingface_hub` version: 0.19.4
- PyArrow version: 14.0.1
- Pandas version: 2.1.4
- `fsspec` version: 2023.10.0


The order of the splits is not preserved

bug

Something isn't working

After investigation, I think the issue was introduced by the use of the Parquet export:
- #6448

I am proposing a fix.

CC: @lhoestq

We had a regression and the order of the splits is not preserved. They are alphabetically sorted, instead of preserving original "train", "validation", "test" order.

Check: In branch "main"
```python
In [9]: dataset = load_dataset("adversarial_qa", '"adversarialQA")

In [10]: dataset
Out[10]:
DatasetDict({
    test: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers', 'metadata'],
        num_rows: 3000
    })
    train: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers', 'metadata'],
        num_rows: 30000
    })
    validation: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers', 'metadata'],
        num_rows: 3000
    })
})
```

Before (2.15.0) it was:
```python
DatasetDict({
    train: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers', 'metadata'],
        num_rows: 30000
    })
    validation: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers', 'metadata'],
        num_rows: 3000
    })
    test: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers', 'metadata'],
        num_rows: 3000
    })
})
```

See issues:
- https://huggingface.co/datasets/adversarial_qa/discussions/3
- https://huggingface.co/datasets/beans/discussions/4

This is a regression because it was previously fixed. See:
- #6196
- #5728


Support commit_description parameter in push_to_hub

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6520). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005484 / 0.011353 (-0.005869) | 0.003537 / 0.011008 (-0.007471) | 0.062631 / 0.038508 (0.024123) | 0.048037 / 0.023109 (0.024927) | 0.240342 / 0.275898 (-0.035556) | 0.268103 / 0.323480 (-0.055377) | 0.002927 / 0.007986 (-0.005059) | 0.002609 / 0.004328 (-0.001719) | 0.048112 / 0.004250 (0.043862) | 0.046111 / 0.037052 (0.009058) | 0.249249 / 0.258489 (-0.009240) | 0.277723 / 0.293841 (-0.016118) | 0.028374 / 0.128546 (-0.100172) | 0.010900 / 0.075646 (-0.064746) | 0.206252 / 0.419271 (-0.213019) | 0.035262 / 0.043533 (-0.008271) | 0.247438 / 0.255139 (-0.007701) | 0.270003 / 0.283200 (-0.013197) | 0.019157 / 0.141683 (-0.122526) | 1.116833 / 1.452155 (-0.335322) | 1.174495 / 1.492716 (-0.318221) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092490 / 0.018006 (0.074484) | 0.302794 / 0.000490 (0.302304) | 0.000213 / 0.000200 (0.000013) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018669 / 0.037411 (-0.018743) | 0.061902 / 0.014526 (0.047376) | 0.073612 / 0.176557 (-0.102945) | 0.121196 / 0.737135 (-0.615940) | 0.075960 / 0.296338 (-0.220378) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.286983 / 0.215209 (0.071774) | 2.836819 / 2.077655 (0.759165) | 1.506635 / 1.504120 (0.002515) | 1.387134 / 1.541195 (-0.154061) | 1.442310 / 1.468490 (-0.026180) | 0.571281 / 4.584777 (-4.013496) | 2.440220 / 3.745712 (-1.305492) | 2.775306 / 5.269862 (-2.494555) | 1.727047 / 4.565676 (-2.838630) | 0.064955 / 0.424275 (-0.359320) | 0.004982 / 0.007607 (-0.002625) | 0.343153 / 0.226044 (0.117108) | 3.388745 / 2.268929 (1.119817) | 1.878983 / 55.444624 (-53.565641) | 1.592642 / 6.876477 (-5.283835) | 1.601037 / 2.142072 (-0.541035) | 0.636882 / 4.805227 (-4.168345) | 0.117804 / 6.500664 (-6.382861) | 0.042467 / 0.075469 (-0.033002) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.941534 / 1.841788 (-0.900254) | 12.093230 / 8.074308 (4.018922) | 10.590854 / 10.191392 (0.399462) | 0.136636 / 0.680424 (-0.543788) | 0.015244 / 0.534201 (-0.518957) | 0.300216 / 0.579283 (-0.279067) | 0.267622 / 0.434364 (-0.166742) | 0.337526 / 0.540337 (-0.202811) | 0.426856 / 1.386936 (-0.960080) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005282 / 0.011353 (-0.006071) | 0.003595 / 0.011008 (-0.007413) | 0.049237 / 0.038508 (0.010729) | 0.054057 / 0.023109 (0.030948) | 0.269781 / 0.275898 (-0.006117) | 0.293544 / 0.323480 (-0.029936) | 0.003991 / 0.007986 (-0.003995) | 0.002705 / 0.004328 (-0.001623) | 0.048755 / 0.004250 (0.044505) | 0.040425 / 0.037052 (0.003373) | 0.264753 / 0.258489 (0.006264) | 0.312773 / 0.293841 (0.018932) | 0.030011 / 0.128546 (-0.098535) | 0.010707 / 0.075646 (-0.064939) | 0.058164 / 0.419271 (-0.361107) | 0.033365 / 0.043533 (-0.010168) | 0.268854 / 0.255139 (0.013715) | 0.283618 / 0.283200 (0.000418) | 0.019571 / 0.141683 (-0.122111) | 1.114738 / 1.452155 (-0.337417) | 1.178990 / 1.492716 (-0.313726) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092183 / 0.018006 (0.074177) | 0.303797 / 0.000490 (0.303307) | 0.000218 / 0.000200 (0.000018) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023088 / 0.037411 (-0.014323) | 0.079813 / 0.014526 (0.065287) | 0.089593 / 0.176557 (-0.086964) | 0.128127 / 0.737135 (-0.609008) | 0.091578 / 0.296338 (-0.204761) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.300153 / 0.215209 (0.084944) | 2.919532 / 2.077655 (0.841877) | 1.587870 / 1.504120 (0.083750) | 1.459031 / 1.541195 (-0.082164) | 1.483305 / 1.468490 (0.014815) | 0.555865 / 4.584777 (-4.028912) | 2.388350 / 3.745712 (-1.357362) | 2.817947 / 5.269862 (-2.451914) | 1.764446 / 4.565676 (-2.801230) | 0.067142 / 0.424275 (-0.357133) | 0.005148 / 0.007607 (-0.002460) | 0.347998 / 0.226044 (0.121953) | 3.431208 / 2.268929 (1.162280) | 1.942175 / 55.444624 (-53.502450) | 1.676606 / 6.876477 (-5.199871) | 1.692431 / 2.142072 (-0.449641) | 0.645974 / 4.805227 (-4.159253) | 0.117729 / 6.500664 (-6.382935) | 0.041670 / 0.075469 (-0.033799) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.981554 / 1.841788 (-0.860234) | 12.671959 / 8.074308 (4.597650) | 11.230694 / 10.191392 (1.039302) | 0.132694 / 0.680424 (-0.547730) | 0.015694 / 0.534201 (-0.518507) | 0.290271 / 0.579283 (-0.289013) | 0.279358 / 0.434364 (-0.155006) | 0.326515 / 0.540337 (-0.213823) | 0.421755 / 1.386936 (-0.965181) |

![](https://cml.dev/watermark.png#0b2147ac644596b66886f398012351641672ee54 "CML watermark")

Support `commit_description` parameter in `push_to_hub`.

CC: @Wauplin


Support push_to_hub canonical datasets

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6519). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@huggingface/datasets this PR is ready for review.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005306 / 0.011353 (-0.006047) | 0.003454 / 0.011008 (-0.007555) | 0.062157 / 0.038508 (0.023649) | 0.051945 / 0.023109 (0.028835) | 0.241834 / 0.275898 (-0.034064) | 0.265590 / 0.323480 (-0.057890) | 0.003149 / 0.007986 (-0.004837) | 0.002695 / 0.004328 (-0.001633) | 0.049197 / 0.004250 (0.044947) | 0.045576 / 0.037052 (0.008524) | 0.242866 / 0.258489 (-0.015623) | 0.280963 / 0.293841 (-0.012878) | 0.028466 / 0.128546 (-0.100080) | 0.010670 / 0.075646 (-0.064976) | 0.206501 / 0.419271 (-0.212771) | 0.035314 / 0.043533 (-0.008219) | 0.240893 / 0.255139 (-0.014246) | 0.264762 / 0.283200 (-0.018438) | 0.019988 / 0.141683 (-0.121695) | 1.095222 / 1.452155 (-0.356933) | 1.144051 / 1.492716 (-0.348666) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.098034 / 0.018006 (0.080028) | 0.308541 / 0.000490 (0.308051) | 0.000261 / 0.000200 (0.000061) | 0.000059 / 0.000054 (0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018646 / 0.037411 (-0.018766) | 0.062881 / 0.014526 (0.048355) | 0.074062 / 0.176557 (-0.102494) | 0.120860 / 0.737135 (-0.616276) | 0.075388 / 0.296338 (-0.220951) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282974 / 0.215209 (0.067765) | 2.755589 / 2.077655 (0.677934) | 1.459536 / 1.504120 (-0.044584) | 1.364543 / 1.541195 (-0.176652) | 1.429860 / 1.468490 (-0.038630) | 0.573277 / 4.584777 (-4.011500) | 2.422983 / 3.745712 (-1.322730) | 3.257258 / 5.269862 (-2.012603) | 1.930053 / 4.565676 (-2.635623) | 0.067476 / 0.424275 (-0.356799) | 0.005612 / 0.007607 (-0.001995) | 0.351538 / 0.226044 (0.125494) | 3.380356 / 2.268929 (1.111427) | 1.837887 / 55.444624 (-53.606738) | 1.537994 / 6.876477 (-5.338483) | 1.623630 / 2.142072 (-0.518442) | 0.662652 / 4.805227 (-4.142576) | 0.127074 / 6.500664 (-6.373590) | 0.049311 / 0.075469 (-0.026158) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.151273 / 1.841788 (-0.690515) | 12.766622 / 8.074308 (4.692314) | 10.967610 / 10.191392 (0.776218) | 0.131305 / 0.680424 (-0.549119) | 0.014227 / 0.534201 (-0.519974) | 0.292054 / 0.579283 (-0.287229) | 0.262737 / 0.434364 (-0.171627) | 0.334360 / 0.540337 (-0.205978) | 0.446711 / 1.386936 (-0.940225) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005194 / 0.011353 (-0.006159) | 0.003508 / 0.011008 (-0.007500) | 0.049287 / 0.038508 (0.010779) | 0.052109 / 0.023109 (0.029000) | 0.271501 / 0.275898 (-0.004397) | 0.290959 / 0.323480 (-0.032521) | 0.004347 / 0.007986 (-0.003638) | 0.002659 / 0.004328 (-0.001669) | 0.048769 / 0.004250 (0.044518) | 0.039388 / 0.037052 (0.002336) | 0.272811 / 0.258489 (0.014322) | 0.305632 / 0.293841 (0.011791) | 0.028419 / 0.128546 (-0.100127) | 0.010617 / 0.075646 (-0.065029) | 0.057433 / 0.419271 (-0.361838) | 0.032383 / 0.043533 (-0.011149) | 0.266566 / 0.255139 (0.011427) | 0.290993 / 0.283200 (0.007794) | 0.019939 / 0.141683 (-0.121743) | 1.157623 / 1.452155 (-0.294532) | 1.183298 / 1.492716 (-0.309419) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.099074 / 0.018006 (0.081068) | 0.315282 / 0.000490 (0.314792) | 0.000235 / 0.000200 (0.000035) | 0.000057 / 0.000054 (0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022692 / 0.037411 (-0.014719) | 0.076455 / 0.014526 (0.061929) | 0.089094 / 0.176557 (-0.087462) | 0.126407 / 0.737135 (-0.610728) | 0.089588 / 0.296338 (-0.206750) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.338853 / 0.215209 (0.123644) | 2.809843 / 2.077655 (0.732188) | 1.538262 / 1.504120 (0.034143) | 1.418290 / 1.541195 (-0.122905) | 1.435145 / 1.468490 (-0.033345) | 0.565763 / 4.584777 (-4.019014) | 2.491525 / 3.745712 (-1.254187) | 2.944879 / 5.269862 (-2.324983) | 1.835840 / 4.565676 (-2.729837) | 0.065101 / 0.424275 (-0.359174) | 0.005196 / 0.007607 (-0.002412) | 0.345291 / 0.226044 (0.119247) | 3.399658 / 2.268929 (1.130729) | 1.892321 / 55.444624 (-53.552303) | 1.608293 / 6.876477 (-5.268184) | 1.651188 / 2.142072 (-0.490884) | 0.647806 / 4.805227 (-4.157421) | 0.119318 / 6.500664 (-6.381346) | 0.043058 / 0.075469 (-0.032412) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.983956 / 1.841788 (-0.857831) | 13.516125 / 8.074308 (5.441817) | 11.712571 / 10.191392 (1.521179) | 0.134253 / 0.680424 (-0.546171) | 0.015844 / 0.534201 (-0.518357) | 0.292444 / 0.579283 (-0.286839) | 0.282182 / 0.434364 (-0.152182) | 0.329327 / 0.540337 (-0.211010) | 0.419960 / 1.386936 (-0.966976) |

![](https://cml.dev/watermark.png#a887ee78835573f5d80f9e414e8443b4caff3541 "CML watermark")

Support `push_to_hub` canonical datasets.

This is necessary in the Space to convert script-datasets to Parquet: https://huggingface.co/spaces/albertvillanova/convert-dataset-to-parquet

Note that before this PR, the `repo_id` "dataset_name" was transformed to "user/dataset_name". This behavior was introduced by:
- #6269


fix get_metadata_patterns function args error

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6518). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

hello!
@albertvillanova
Thank you very much for your recognition。
When can this PR be merged?

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005205 / 0.011353 (-0.006148) | 0.003730 / 0.011008 (-0.007278) | 0.063195 / 0.038508 (0.024687) | 0.052329 / 0.023109 (0.029219) | 0.247299 / 0.275898 (-0.028599) | 0.269600 / 0.323480 (-0.053880) | 0.004801 / 0.007986 (-0.003185) | 0.002728 / 0.004328 (-0.001600) | 0.049195 / 0.004250 (0.044944) | 0.044859 / 0.037052 (0.007807) | 0.253047 / 0.258489 (-0.005442) | 0.277253 / 0.293841 (-0.016588) | 0.028370 / 0.128546 (-0.100176) | 0.011095 / 0.075646 (-0.064551) | 0.211090 / 0.419271 (-0.208182) | 0.035944 / 0.043533 (-0.007589) | 0.252755 / 0.255139 (-0.002384) | 0.269466 / 0.283200 (-0.013733) | 0.017514 / 0.141683 (-0.124169) | 1.107815 / 1.452155 (-0.344339) | 1.154989 / 1.492716 (-0.337728) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093925 / 0.018006 (0.075919) | 0.300923 / 0.000490 (0.300433) | 0.000219 / 0.000200 (0.000019) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018268 / 0.037411 (-0.019143) | 0.060508 / 0.014526 (0.045983) | 0.074564 / 0.176557 (-0.101992) | 0.121523 / 0.737135 (-0.615612) | 0.077394 / 0.296338 (-0.218945) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.275859 / 0.215209 (0.060650) | 2.707593 / 2.077655 (0.629938) | 1.419178 / 1.504120 (-0.084942) | 1.286737 / 1.541195 (-0.254458) | 1.350504 / 1.468490 (-0.117986) | 0.570461 / 4.584777 (-4.014316) | 2.400795 / 3.745712 (-1.344917) | 2.840876 / 5.269862 (-2.428986) | 1.724044 / 4.565676 (-2.841633) | 0.063819 / 0.424275 (-0.360456) | 0.004961 / 0.007607 (-0.002647) | 0.342537 / 0.226044 (0.116492) | 3.370942 / 2.268929 (1.102013) | 1.788659 / 55.444624 (-53.655966) | 1.501921 / 6.876477 (-5.374556) | 1.535352 / 2.142072 (-0.606721) | 0.651838 / 4.805227 (-4.153390) | 0.118979 / 6.500664 (-6.381685) | 0.047796 / 0.075469 (-0.027673) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.949850 / 1.841788 (-0.891937) | 11.581988 / 8.074308 (3.507680) | 10.462837 / 10.191392 (0.271445) | 0.133298 / 0.680424 (-0.547125) | 0.015008 / 0.534201 (-0.519193) | 0.299265 / 0.579283 (-0.280018) | 0.268864 / 0.434364 (-0.165500) | 0.332888 / 0.540337 (-0.207450) | 0.420423 / 1.386936 (-0.966513) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005309 / 0.011353 (-0.006044) | 0.003628 / 0.011008 (-0.007380) | 0.049545 / 0.038508 (0.011036) | 0.054095 / 0.023109 (0.030985) | 0.270679 / 0.275898 (-0.005219) | 0.295744 / 0.323480 (-0.027736) | 0.004131 / 0.007986 (-0.003855) | 0.002732 / 0.004328 (-0.001596) | 0.048714 / 0.004250 (0.044464) | 0.039916 / 0.037052 (0.002863) | 0.272354 / 0.258489 (0.013865) | 0.310553 / 0.293841 (0.016712) | 0.029525 / 0.128546 (-0.099021) | 0.011322 / 0.075646 (-0.064324) | 0.058007 / 0.419271 (-0.361265) | 0.032883 / 0.043533 (-0.010650) | 0.273609 / 0.255139 (0.018470) | 0.291780 / 0.283200 (0.008581) | 0.020538 / 0.141683 (-0.121145) | 1.118031 / 1.452155 (-0.334123) | 1.160777 / 1.492716 (-0.331940) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092966 / 0.018006 (0.074959) | 0.301432 / 0.000490 (0.300943) | 0.000225 / 0.000200 (0.000025) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022736 / 0.037411 (-0.014676) | 0.077655 / 0.014526 (0.063129) | 0.093386 / 0.176557 (-0.083171) | 0.129694 / 0.737135 (-0.607441) | 0.092790 / 0.296338 (-0.203548) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.299161 / 0.215209 (0.083952) | 2.923300 / 2.077655 (0.845645) | 1.629661 / 1.504120 (0.125541) | 1.510797 / 1.541195 (-0.030398) | 1.507269 / 1.468490 (0.038778) | 0.574346 / 4.584777 (-4.010431) | 2.454396 / 3.745712 (-1.291316) | 2.843402 / 5.269862 (-2.426460) | 1.774815 / 4.565676 (-2.790861) | 0.063601 / 0.424275 (-0.360674) | 0.004977 / 0.007607 (-0.002630) | 0.347693 / 0.226044 (0.121649) | 3.430054 / 2.268929 (1.161126) | 1.987308 / 55.444624 (-53.457316) | 1.682756 / 6.876477 (-5.193721) | 1.688463 / 2.142072 (-0.453609) | 0.646449 / 4.805227 (-4.158778) | 0.117860 / 6.500664 (-6.382804) | 0.041305 / 0.075469 (-0.034164) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.987355 / 1.841788 (-0.854433) | 12.398721 / 8.074308 (4.324412) | 11.070442 / 10.191392 (0.879050) | 0.134946 / 0.680424 (-0.545477) | 0.016172 / 0.534201 (-0.518029) | 0.293359 / 0.579283 (-0.285924) | 0.282271 / 0.434364 (-0.152093) | 0.331919 / 0.540337 (-0.208418) | 0.432137 / 1.386936 (-0.954799) |

![](https://cml.dev/watermark.png#2246d3187222ef939aa8e69cd1aa476cf9526945 "CML watermark")

Bug get_metadata_patterns arg error https://github.com/huggingface/datasets/issues/6517


Bug get_metadata_patterns arg error

https://github.com/huggingface/datasets/blob/3f149204a2a5948287adcade5e90707aa5207a92/src/datasets/load.py#L1240C1-L1240C69
metadata_patterns = get_metadata_patterns(base_path, download_config=self.download_config)