The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6945). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005725 / 0.011353 (-0.005627) | 0.003788 / 0.011008 (-0.007220) | 0.063059 / 0.038508 (0.024551) | 0.031364 / 0.023109 (0.008255) | 0.259209 / 0.275898 (-0.016689) | 0.278805 / 0.323480 (-0.044675) | 0.003032 / 0.007986 (-0.004953) | 0.002633 / 0.004328 (-0.001696) | 0.049804 / 0.004250 (0.045554) | 0.046717 / 0.037052 (0.009665) | 0.267246 / 0.258489 (0.008757) | 0.299271 / 0.293841 (0.005430) | 0.027687 / 0.128546 (-0.100860) | 0.010524 / 0.075646 (-0.065123) | 0.201736 / 0.419271 (-0.217536) | 0.036192 / 0.043533 (-0.007341) | 0.264492 / 0.255139 (0.009353) | 0.280809 / 0.283200 (-0.002391) | 0.018187 / 0.141683 (-0.123496) | 1.170751 / 1.452155 (-0.281404) | 1.223450 / 1.492716 (-0.269266) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096610 / 0.018006 (0.078604) | 0.297122 / 0.000490 (0.296632) | 0.000211 / 0.000200 (0.000011) | 0.000046 / 0.000054 (-0.000008) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018380 / 0.037411 (-0.019031) | 0.062214 / 0.014526 (0.047688) | 0.075833 / 0.176557 (-0.100723) | 0.121825 / 0.737135 (-0.615310) | 0.075475 / 0.296338 (-0.220864) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.275601 / 0.215209 (0.060392) | 2.698014 / 2.077655 (0.620359) | 1.434043 / 1.504120 (-0.070077) | 1.313217 / 1.541195 (-0.227978) | 1.339014 / 1.468490 (-0.129476) | 0.566703 / 4.584777 (-4.018074) | 2.367794 / 3.745712 (-1.377918) | 2.660787 / 5.269862 (-2.609074) | 1.738503 / 4.565676 (-2.827174) | 0.061693 / 0.424275 (-0.362582) | 0.004978 / 0.007607 (-0.002629) | 0.334719 / 0.226044 (0.108675) | 3.300889 / 2.268929 (1.031960) | 1.764493 / 55.444624 (-53.680131) | 1.475956 / 6.876477 (-5.400521) | 1.635988 / 2.142072 (-0.506084) | 0.643906 / 4.805227 (-4.161321) | 0.118002 / 6.500664 (-6.382662) | 0.042593 / 0.075469 (-0.032876) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.953511 / 1.841788 (-0.888276) | 11.489727 / 8.074308 (3.415419) | 9.775017 / 10.191392 (-0.416375) | 0.139864 / 0.680424 (-0.540560) | 0.014219 / 0.534201 (-0.519982) | 0.284389 / 0.579283 (-0.294894) | 0.264250 / 0.434364 (-0.170113) | 0.323471 / 0.540337 (-0.216866) | 0.415189 / 1.386936 (-0.971747) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005437 / 0.011353 (-0.005916) | 0.003710 / 0.011008 (-0.007298) | 0.049940 / 0.038508 (0.011432) | 0.032565 / 0.023109 (0.009456) | 0.266374 / 0.275898 (-0.009524) | 0.288069 / 0.323480 (-0.035411) | 0.004140 / 0.007986 (-0.003845) | 0.002669 / 0.004328 (-0.001660) | 0.049646 / 0.004250 (0.045395) | 0.040926 / 0.037052 (0.003874) | 0.278805 / 0.258489 (0.020316) | 0.311396 / 0.293841 (0.017555) | 0.029363 / 0.128546 (-0.099183) | 0.010260 / 0.075646 (-0.065386) | 0.058222 / 0.419271 (-0.361049) | 0.033063 / 0.043533 (-0.010470) | 0.266798 / 0.255139 (0.011659) | 0.283091 / 0.283200 (-0.000109) | 0.017904 / 0.141683 (-0.123779) | 1.139531 / 1.452155 (-0.312624) | 1.163909 / 1.492716 (-0.328808) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.089063 / 0.018006 (0.071057) | 0.296757 / 0.000490 (0.296268) | 0.000202 / 0.000200 (0.000002) | 0.000054 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022843 / 0.037411 (-0.014568) | 0.076032 / 0.014526 (0.061507) | 0.087545 / 0.176557 (-0.089012) | 0.128870 / 0.737135 (-0.608266) | 0.089359 / 0.296338 (-0.206980) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.285213 / 0.215209 (0.070004) | 2.854950 / 2.077655 (0.777295) | 1.539311 / 1.504120 (0.035191) | 1.413753 / 1.541195 (-0.127442) | 1.440819 / 1.468490 (-0.027671) | 0.564734 / 4.584777 (-4.020043) | 0.944924 / 3.745712 (-2.800788) | 2.703612 / 5.269862 (-2.566249) | 1.749429 / 4.565676 (-2.816247) | 0.063239 / 0.424275 (-0.361036) | 0.005024 / 0.007607 (-0.002583) | 0.340866 / 0.226044 (0.114821) | 3.359511 / 2.268929 (1.090582) | 1.895794 / 55.444624 (-53.548831) | 1.606613 / 6.876477 (-5.269864) | 1.756539 / 2.142072 (-0.385533) | 0.646553 / 4.805227 (-4.158675) | 0.121278 / 6.500664 (-6.379386) | 0.041066 / 0.075469 (-0.034403) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.005548 / 1.841788 (-0.836240) | 12.080103 / 8.074308 (4.005794) | 10.444822 / 10.191392 (0.253430) | 0.145024 / 0.680424 (-0.535400) | 0.015287 / 0.534201 (-0.518914) | 0.288567 / 0.579283 (-0.290716) | 0.118034 / 0.434364 (-0.316330) | 0.333474 / 0.540337 (-0.206864) | 0.421716 / 1.386936 (-0.965220) |

![](https://cml.dev/watermark.png#3d95159dbd918009e1ff710dba0cd15d96d4264e "CML watermark")

@albertvillanova could I ask why we should use latest `requests` here? we are using `docker` and `datasets` in the same time. However, docker requires requests<2.32.0.

Hi @pingsutw,

We updated the minimum required `requests` version for security reasons: https://www.cve.org/CVERecord?id=CVE-2024-35195
- affected versions < 2.32.0

Latest version of `docker` should normally support `requests` >= 2.32.0: https://github.com/docker/docker-py/releases/tag/7.1.0
> Fixed an issue due to an update in the [requests](https://github.com/psf/requests) package breaking docker-py by applying the https://github.com/psf/requests/pull/6710
- https://github.com/docker/docker-py/pull/3257

I guess you need to update your `docker` library as well:
```
pip install -U docker
```

> I guess you need to update your docker library as well:

Thank you! it works for me 👍

Update yanked version of minimum requests requirement.

Version 2.32.1 was yanked: https://pypi.org/project/requests/2.32.1/


Set dev version

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6944). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005150 / 0.011353 (-0.006203) | 0.003663 / 0.011008 (-0.007346) | 0.062832 / 0.038508 (0.024324) | 0.031928 / 0.023109 (0.008819) | 0.246455 / 0.275898 (-0.029443) | 0.272121 / 0.323480 (-0.051359) | 0.004220 / 0.007986 (-0.003765) | 0.002756 / 0.004328 (-0.001573) | 0.050071 / 0.004250 (0.045821) | 0.046074 / 0.037052 (0.009022) | 0.259676 / 0.258489 (0.001187) | 0.290674 / 0.293841 (-0.003167) | 0.027822 / 0.128546 (-0.100724) | 0.010791 / 0.075646 (-0.064855) | 0.202827 / 0.419271 (-0.216445) | 0.037057 / 0.043533 (-0.006476) | 0.256128 / 0.255139 (0.000989) | 0.269422 / 0.283200 (-0.013777) | 0.017395 / 0.141683 (-0.124288) | 1.125919 / 1.452155 (-0.326236) | 1.177708 / 1.492716 (-0.315008) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.098466 / 0.018006 (0.080460) | 0.305508 / 0.000490 (0.305018) | 0.000232 / 0.000200 (0.000032) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018866 / 0.037411 (-0.018545) | 0.062079 / 0.014526 (0.047553) | 0.074670 / 0.176557 (-0.101886) | 0.121025 / 0.737135 (-0.616111) | 0.075883 / 0.296338 (-0.220455) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.291880 / 0.215209 (0.076671) | 2.874064 / 2.077655 (0.796409) | 1.477040 / 1.504120 (-0.027080) | 1.356198 / 1.541195 (-0.184997) | 1.354676 / 1.468490 (-0.113814) | 0.559731 / 4.584777 (-4.025046) | 2.362746 / 3.745712 (-1.382966) | 2.678838 / 5.269862 (-2.591024) | 1.752633 / 4.565676 (-2.813044) | 0.064023 / 0.424275 (-0.360252) | 0.005035 / 0.007607 (-0.002572) | 0.354807 / 0.226044 (0.128762) | 3.424463 / 2.268929 (1.155534) | 1.810476 / 55.444624 (-53.634149) | 1.519031 / 6.876477 (-5.357446) | 1.693957 / 2.142072 (-0.448116) | 0.647987 / 4.805227 (-4.157240) | 0.118993 / 6.500664 (-6.381671) | 0.042186 / 0.075469 (-0.033283) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.982565 / 1.841788 (-0.859223) | 11.645075 / 8.074308 (3.570767) | 9.588360 / 10.191392 (-0.603032) | 0.142369 / 0.680424 (-0.538055) | 0.014025 / 0.534201 (-0.520176) | 0.285668 / 0.579283 (-0.293616) | 0.265825 / 0.434364 (-0.168539) | 0.323371 / 0.540337 (-0.216966) | 0.421227 / 1.386936 (-0.965709) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005587 / 0.011353 (-0.005766) | 0.003664 / 0.011008 (-0.007345) | 0.050411 / 0.038508 (0.011903) | 0.033268 / 0.023109 (0.010159) | 0.266631 / 0.275898 (-0.009267) | 0.291135 / 0.323480 (-0.032345) | 0.004275 / 0.007986 (-0.003710) | 0.002822 / 0.004328 (-0.001506) | 0.049349 / 0.004250 (0.045099) | 0.040653 / 0.037052 (0.003601) | 0.282641 / 0.258489 (0.024152) | 0.315460 / 0.293841 (0.021619) | 0.029343 / 0.128546 (-0.099203) | 0.010606 / 0.075646 (-0.065040) | 0.058783 / 0.419271 (-0.360489) | 0.033205 / 0.043533 (-0.010327) | 0.266805 / 0.255139 (0.011666) | 0.288907 / 0.283200 (0.005707) | 0.017817 / 0.141683 (-0.123866) | 1.128132 / 1.452155 (-0.324023) | 1.175120 / 1.492716 (-0.317597) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095653 / 0.018006 (0.077647) | 0.304825 / 0.000490 (0.304335) | 0.000212 / 0.000200 (0.000012) | 0.000045 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022766 / 0.037411 (-0.014645) | 0.076598 / 0.014526 (0.062072) | 0.088314 / 0.176557 (-0.088242) | 0.127888 / 0.737135 (-0.609247) | 0.090391 / 0.296338 (-0.205947) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.293384 / 0.215209 (0.078175) | 2.883742 / 2.077655 (0.806087) | 1.533868 / 1.504120 (0.029748) | 1.391964 / 1.541195 (-0.149231) | 1.423732 / 1.468490 (-0.044759) | 0.575457 / 4.584777 (-4.009320) | 0.970860 / 3.745712 (-2.774852) | 2.711405 / 5.269862 (-2.558457) | 1.774468 / 4.565676 (-2.791208) | 0.064611 / 0.424275 (-0.359664) | 0.005120 / 0.007607 (-0.002487) | 0.343892 / 0.226044 (0.117847) | 3.362579 / 2.268929 (1.093650) | 1.880200 / 55.444624 (-53.564424) | 1.587435 / 6.876477 (-5.289042) | 1.756464 / 2.142072 (-0.385609) | 0.661469 / 4.805227 (-4.143759) | 0.119030 / 6.500664 (-6.381634) | 0.041704 / 0.075469 (-0.033765) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.025008 / 1.841788 (-0.816780) | 12.146244 / 8.074308 (4.071936) | 10.397267 / 10.191392 (0.205875) | 0.145917 / 0.680424 (-0.534507) | 0.015779 / 0.534201 (-0.518422) | 0.287122 / 0.579283 (-0.292161) | 0.125464 / 0.434364 (-0.308900) | 0.323315 / 0.540337 (-0.217023) | 0.416761 / 1.386936 (-0.970175) |

![](https://cml.dev/watermark.png#e2d15a6b1871f3998986853298e4338d72891491 "CML watermark")


Release 2.19.2

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6943). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


Import sorting is disabled by flake8 noqa directive after switching to ruff linter

maintenance

Maintenance tasks

When we switched to `ruff` linter in PR:
- #5519

import sorting was disabled in all files containing the `# flake8: noqa` directive
- https://github.com/astral-sh/ruff/issues/11679

We should re-enable import sorting on those files.


Supporting FFCV: Fast Forward Computer Vision

enhancement

New feature or request

### Feature request

Supporting FFCV, https://github.com/libffcv/ffcv

### Motivation

According to the benchmark, FFCV seems to be fastest image loading method.

### Your contribution

no


Enable Sharding to Equal Sized Shards

New feature or request

### Feature request

Add an option when sharding a dataset to have all shards the same size. Will be good to provide both an option of duplication, and by truncation.

### Motivation

Currently the behavior of sharding is "If n % i == l, then the first l shards will have length (n // i) + 1, and the remaining shards will have length (n // i).". However, when using FSDP we want the shards to have the same size. This requires the user to manually handle this situation, but it will be nice if we had an option to shard the dataset into equally sized shards.

### Your contribution

For now just a PR. I can also add code that does what is needed, but probably not efficient.
Shard to equal size by duplication:
```
remainder = len(dataset) % num_shards
num_missing_examples = num_shards - remainder
duplicated = dataset.select(list(range(num_missing_examples)))
dataset = concatenate_datasets([dataset, duplicated])
shard = dataset.shard(num_shards, shard_idx)
```
Or by truncation:
```
shard = dataset.shard(num_shards, shard_idx)
num_examples_per_shard = len(dataset) // num_shards
shard = shard.select(list(range(num_examples_per_shard)))
```


ExpectedMoreSplits error when using data_dir

bug

Something isn't working

As reported by @regisss, an `ExpectedMoreSplits` error is raised when passing `data_dir`:
```python
from datasets import load_dataset

dataset = load_dataset(
    "lvwerra/stack-exchange-paired",
    split="train",
    cache_dir=None,
    data_dir="data/rl",
)
```
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2609, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1027, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1140, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/usr/local/lib/python3.10/dist-packages/datasets/utils/info_utils.py", line 92, in verify_splits
    raise ExpectedMoreSplits(str(set(expected_splits) - set(recorded_splits)))
datasets.utils.info_utils.ExpectedMoreSplits: {'test'}
```


Fix expected splits when passing data_files or dir

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6938). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

fix is included in https://github.com/huggingface/datasets/pull/6925

reported on slack:

The following code snippet gives an error with v2.19 but not with v2.18:
from datasets import load_dataset
```
dataset = load_dataset(
    "lvwerra/stack-exchange-paired",
    split="train",
    cache_dir=None,
    data_dir="data/rl",
)
```
and the error is:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2609, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1027, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1140, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/usr/local/lib/python3.10/dist-packages/datasets/utils/info_utils.py", line 92, in verify_splits
    raise ExpectedMoreSplits(str(set(expected_splits) - set(recorded_splits)))
datasets.utils.info_utils.ExpectedMoreSplits: {'test'}
```


Something isn't working

The JSON loader implicitly coerces floats to integers.

The column values `[0.0, 1.0, 2.0]` are coerced to `[0, 1, 2]`.
See CI error in dataset-viewer: https://github.com/huggingface/dataset-viewer/actions/runs/9290164936/job/25576926446
```
 =================================== FAILURES ===================================
___________________________ test_statistics_endpoint ___________________________

normal_user_public_json_dataset = 'DVUser/tmp-dataset-17170199043860'

    def test_statistics_endpoint(normal_user_public_json_dataset: str) -> None:
        dataset = normal_user_public_json_dataset
        config, split = get_default_config_split()
        statistics_response = poll_until_ready_and_assert(
            relative_url=f"/statistics?dataset={dataset}&config={config}&split={split}",
            check_x_revision=True,
            dataset=dataset,
        )

        content = statistics_response.json()
        assert len(content) == 3
        assert sorted(content) == ["num_examples", "partial", "statistics"], statistics_response
        statistics = content["statistics"]
        num_examples = content["num_examples"]
        partial = content["partial"]

        assert isinstance(statistics, list), statistics
        assert len(statistics) == 6
        assert num_examples == 4
        assert partial is False

        string_label_column = statistics[0]
        assert "column_name" in string_label_column
        assert "column_statistics" in string_label_column
        assert "column_type" in string_label_column
        assert string_label_column["column_name"] == "col_1"
        assert string_label_column["column_type"] == "string_label"  # 4 unique values -> label
        assert isinstance(string_label_column["column_statistics"], dict)
        assert string_label_column["column_statistics"] == {
            "nan_count": 0,
            "nan_proportion": 0.0,
            "no_label_count": 0,
            "no_label_proportion": 0.0,
            "n_unique": 4,
            "frequencies": {
                "There goes another one.": 1,
                "Vader turns round and round in circles as his ship spins into space.": 1,
                "We count thirty Rebel ships, Lord Vader.": 1,
                "The wingman spots the pirateship coming at him and warns the Dark Lord": 1,
            },
        }

        int_column = statistics[1]
        assert "column_name" in int_column
        assert "column_statistics" in int_column
        assert "column_type" in int_column
        assert int_column["column_name"] == "col_2"
        assert int_column["column_type"] == "int"
        assert isinstance(int_column["column_statistics"], dict)
        assert int_column["column_statistics"] == {
            "histogram": {"bin_edges": [0, 1, 2, 3, 3], "hist": [1, 1, 1, 1]},
            "max": 3,
            "mean": 1.5,
            "median": 1.5,
            "min": 0,
            "nan_count": 0,
            "nan_proportion": 0.0,
            "std": 1.29099,
        }

        float_column = statistics[2]
        assert "column_name" in float_column
        assert "column_statistics" in float_column
        assert "column_type" in float_column
        assert float_column["column_name"] == "col_3"
>       assert float_column["column_type"] == "float"
E       AssertionError: assert 'int' == 'float'
E         - float
E         + int

tests/test_14_statistics.py:72: AssertionError

=========================== short test summary info ============================
FAILED tests/test_14_statistics.py::test_statistics_endpoint - AssertionError: assert 'int' == 'float'
  - float
  + int
```

This bug was introduced after:
- #6914

We have reported the issue to pandas:
- https://github.com/pandas-dev/pandas/issues/58866


save_to_disk() freezes when saving on s3 bucket with multiprocessing

### Describe the bug

I'm trying to save a `Dataset` using the `save_to_disk()` function with:
- `num_proc > 1`
- `dataset_path` being a s3 bucket path e.g. "s3://{bucket_name}/{dataset_folder}/"

The hf progress bar shows up but the saving does not seem to start.
When using one processor only (`num_proc=1`), everything works fine.
When saving the dataset on local disk (as opposed to s3 bucket) with `num_proc > 1`, everything works fine.

Thank you for your help! :)

### Steps to reproduce the bug

I tried without any storage options:

```
from datasets import load_dataset

sandbox_ds = load_dataset("openai_humaneval")
sandbox_ds["test"].save_to_disk(
    "s3://bucket-name/test_multiprocessing_saving/",
    num_proc=4,
)
```

and with the specific s3fs storage options:
```
from datasets import load_dataset
from s3fs import S3FileSystem

def get_s3fs():
    return S3FileSystem()

sandbox_ds = load_dataset("openai_humaneval")
sandbox_ds["test"].save_to_disk(
    "s3://bucket-name/test_multiprocessing_saving/",
    num_proc=4,
    storage_options=get_s3fs().storage_options, # also tried: storage_options=S3FileSystem().storage_options
)
```

I'm guessing I might use `storage_options` parameter wrongly, but I didn't find anything online that made it work.

**NB**: Behavior is the same when trying to save the whole `DatasetDict`.

### Expected behavior

Progress bar fills in and saving is carried out.

### Environment info

`datasets==2.18.0`


Support for pathlib.Path in datasets 2.19.0

### Describe the bug

After the recent update of `datasets`, Dataset.save_to_disk does not accept a pathlib.Path anymore. It was supported in 2.18.0 and previous versions. Is this intentional? Was it supported before only because of a Python dusk-typing miracle?

### Steps to reproduce the bug

```
from datasets import Dataset
import pathlib

path = pathlib.Path("./my_out_path")
Dataset.from_dict(
    {"text": ["hello world"], "label": [777], "split": ["train"]}
.save_to_disk(path)
```
This results in an error when using datasets 2.19:
```
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/Users/jb/scratch/venv/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 1515, in save_to_disk
    fs, _ = url_to_fs(dataset_path, **(storage_options or {}))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jb/scratch/venv/lib/python3.11/site-packages/fsspec/core.py", line 383, in url_to_fs
    chain = _un_chain(url, kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jb/scratch/venv/lib/python3.11/site-packages/fsspec/core.py", line 323, in _un_chain
    if "::" in path
       ^^^^^^^^^^^^
TypeError: argument of type 'PosixPath' is not iterable
```

Converting to str works, however.
```
Dataset.from_dict(
     {"text": ["hello world"], "label": [777], "split": ["train"]}
).save_to_disk(str(path))
```

### Expected behavior

My dataset gets saved to disk without an error.

### Environment info

aiohttp==3.9.5
aiosignal==1.3.1
attrs==23.2.0
certifi==2024.2.2
charset-normalizer==3.3.2
datasets==2.19.0
dill==0.3.8
filelock==3.14.0
frozenlist==1.4.1
fsspec==2024.3.1
huggingface-hub==0.23.2
idna==3.7
multidict==6.0.5
multiprocess==0.70.16
numpy==1.26.4
packaging==24.0
pandas==2.2.2
pyarrow==16.1.0
pyarrow-hotfix==0.6
python-dateutil==2.9.0.post0
pytz==2024.1
PyYAML==6.0.1
requests==2.32.3
six==1.16.0
tqdm==4.66.4
typing_extensions==4.12.0
tzdata==2024.1
urllib3==2.2.1
xxhash==3.4.1
yarl==1.9.4


Revert ci user

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6934). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005218 / 0.011353 (-0.006135) | 0.003313 / 0.011008 (-0.007695) | 0.062992 / 0.038508 (0.024484) | 0.029621 / 0.023109 (0.006512) | 0.244421 / 0.275898 (-0.031477) | 0.267178 / 0.323480 (-0.056302) | 0.002986 / 0.007986 (-0.005000) | 0.002607 / 0.004328 (-0.001721) | 0.049149 / 0.004250 (0.044898) | 0.045362 / 0.037052 (0.008310) | 0.252862 / 0.258489 (-0.005627) | 0.286326 / 0.293841 (-0.007515) | 0.027888 / 0.128546 (-0.100658) | 0.010295 / 0.075646 (-0.065352) | 0.205525 / 0.419271 (-0.213746) | 0.036696 / 0.043533 (-0.006837) | 0.248716 / 0.255139 (-0.006423) | 0.263803 / 0.283200 (-0.019397) | 0.016926 / 0.141683 (-0.124757) | 1.123093 / 1.452155 (-0.329062) | 1.155434 / 1.492716 (-0.337282) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092349 / 0.018006 (0.074343) | 0.298154 / 0.000490 (0.297664) | 0.000213 / 0.000200 (0.000013) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018496 / 0.037411 (-0.018915) | 0.061983 / 0.014526 (0.047457) | 0.075043 / 0.176557 (-0.101514) | 0.120678 / 0.737135 (-0.616457) | 0.074917 / 0.296338 (-0.221422) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.290558 / 0.215209 (0.075349) | 2.842635 / 2.077655 (0.764981) | 1.485761 / 1.504120 (-0.018359) | 1.346948 / 1.541195 (-0.194247) | 1.352424 / 1.468490 (-0.116066) | 0.564567 / 4.584777 (-4.020210) | 2.393583 / 3.745712 (-1.352129) | 2.654061 / 5.269862 (-2.615800) | 1.729154 / 4.565676 (-2.836523) | 0.064652 / 0.424275 (-0.359623) | 0.004973 / 0.007607 (-0.002634) | 0.334924 / 0.226044 (0.108879) | 3.330518 / 2.268929 (1.061590) | 1.773848 / 55.444624 (-53.670776) | 1.513796 / 6.876477 (-5.362681) | 1.676492 / 2.142072 (-0.465580) | 0.650551 / 4.805227 (-4.154677) | 0.118423 / 6.500664 (-6.382241) | 0.042700 / 0.075469 (-0.032769) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.943394 / 1.841788 (-0.898394) | 11.235766 / 8.074308 (3.161458) | 9.896586 / 10.191392 (-0.294806) | 0.130174 / 0.680424 (-0.550249) | 0.014148 / 0.534201 (-0.520053) | 0.284002 / 0.579283 (-0.295281) | 0.261354 / 0.434364 (-0.173010) | 0.320839 / 0.540337 (-0.219499) | 0.422399 / 1.386936 (-0.964537) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005496 / 0.011353 (-0.005857) | 0.003603 / 0.011008 (-0.007406) | 0.050104 / 0.038508 (0.011596) | 0.032939 / 0.023109 (0.009830) | 0.265643 / 0.275898 (-0.010255) | 0.291819 / 0.323480 (-0.031661) | 0.004273 / 0.007986 (-0.003713) | 0.002715 / 0.004328 (-0.001613) | 0.049191 / 0.004250 (0.044941) | 0.040782 / 0.037052 (0.003730) | 0.276562 / 0.258489 (0.018072) | 0.314307 / 0.293841 (0.020466) | 0.029878 / 0.128546 (-0.098669) | 0.010134 / 0.075646 (-0.065513) | 0.058686 / 0.419271 (-0.360585) | 0.033562 / 0.043533 (-0.009971) | 0.265961 / 0.255139 (0.010822) | 0.282009 / 0.283200 (-0.001191) | 0.018956 / 0.141683 (-0.122727) | 1.149668 / 1.452155 (-0.302487) | 1.192242 / 1.492716 (-0.300474) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.089449 / 0.018006 (0.071443) | 0.300346 / 0.000490 (0.299856) | 0.000198 / 0.000200 (-0.000001) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022094 / 0.037411 (-0.015317) | 0.075987 / 0.014526 (0.061461) | 0.088191 / 0.176557 (-0.088365) | 0.127698 / 0.737135 (-0.609437) | 0.089642 / 0.296338 (-0.206696) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.299127 / 0.215209 (0.083918) | 2.961219 / 2.077655 (0.883565) | 1.589108 / 1.504120 (0.084988) | 1.464060 / 1.541195 (-0.077135) | 1.475249 / 1.468490 (0.006759) | 0.569041 / 4.584777 (-4.015736) | 0.966965 / 3.745712 (-2.778747) | 2.653049 / 5.269862 (-2.616813) | 1.733650 / 4.565676 (-2.832026) | 0.062537 / 0.424275 (-0.361738) | 0.005003 / 0.007607 (-0.002605) | 0.353345 / 0.226044 (0.127301) | 3.432888 / 2.268929 (1.163960) | 1.953217 / 55.444624 (-53.491407) | 1.651995 / 6.876477 (-5.224482) | 1.764549 / 2.142072 (-0.377523) | 0.647255 / 4.805227 (-4.157973) | 0.116827 / 6.500664 (-6.383837) | 0.040765 / 0.075469 (-0.034704) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.985490 / 1.841788 (-0.856298) | 11.965147 / 8.074308 (3.890839) | 10.488286 / 10.191392 (0.296894) | 0.142134 / 0.680424 (-0.538290) | 0.015415 / 0.534201 (-0.518786) | 0.289864 / 0.579283 (-0.289419) | 0.122778 / 0.434364 (-0.311586) | 0.328691 / 0.540337 (-0.211647) | 0.422677 / 1.386936 (-0.964259) |

![](https://cml.dev/watermark.png#456f790d2c2e9181bc305ab3d54fe2ca58742b9b "CML watermark")

There was an incident in hub-ci that invalidated our token. It's been fixed so I reverted this change


update ci user

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6933). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004937 / 0.011353 (-0.006416) | 0.003706 / 0.011008 (-0.007302) | 0.062627 / 0.038508 (0.024119) | 0.031372 / 0.023109 (0.008263) | 0.246616 / 0.275898 (-0.029282) | 0.272196 / 0.323480 (-0.051284) | 0.004129 / 0.007986 (-0.003856) | 0.002766 / 0.004328 (-0.001562) | 0.049975 / 0.004250 (0.045725) | 0.045098 / 0.037052 (0.008046) | 0.261802 / 0.258489 (0.003313) | 0.290088 / 0.293841 (-0.003753) | 0.027082 / 0.128546 (-0.101465) | 0.010442 / 0.075646 (-0.065205) | 0.201795 / 0.419271 (-0.217477) | 0.037081 / 0.043533 (-0.006452) | 0.249500 / 0.255139 (-0.005639) | 0.268800 / 0.283200 (-0.014399) | 0.017556 / 0.141683 (-0.124127) | 1.137201 / 1.452155 (-0.314953) | 1.186993 / 1.492716 (-0.305723) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.097426 / 0.018006 (0.079419) | 0.303653 / 0.000490 (0.303163) | 0.000235 / 0.000200 (0.000035) | 0.000049 / 0.000054 (-0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.020206 / 0.037411 (-0.017206) | 0.063673 / 0.014526 (0.049147) | 0.076173 / 0.176557 (-0.100383) | 0.122459 / 0.737135 (-0.614676) | 0.076958 / 0.296338 (-0.219380) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282146 / 0.215209 (0.066937) | 2.785682 / 2.077655 (0.708027) | 1.468847 / 1.504120 (-0.035273) | 1.346731 / 1.541195 (-0.194464) | 1.378459 / 1.468490 (-0.090031) | 0.564961 / 4.584777 (-4.019816) | 2.400095 / 3.745712 (-1.345617) | 2.658285 / 5.269862 (-2.611577) | 1.747873 / 4.565676 (-2.817803) | 0.063763 / 0.424275 (-0.360512) | 0.004969 / 0.007607 (-0.002638) | 0.337764 / 0.226044 (0.111720) | 3.309568 / 2.268929 (1.040639) | 1.812516 / 55.444624 (-53.632109) | 1.521519 / 6.876477 (-5.354957) | 1.690091 / 2.142072 (-0.451982) | 0.640922 / 4.805227 (-4.164305) | 0.119291 / 6.500664 (-6.381373) | 0.042195 / 0.075469 (-0.033274) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.965327 / 1.841788 (-0.876461) | 11.538832 / 8.074308 (3.464523) | 9.594644 / 10.191392 (-0.596748) | 0.144687 / 0.680424 (-0.535737) | 0.014049 / 0.534201 (-0.520152) | 0.296873 / 0.579283 (-0.282410) | 0.269281 / 0.434364 (-0.165083) | 0.325091 / 0.540337 (-0.215246) | 0.420917 / 1.386936 (-0.966019) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005239 / 0.011353 (-0.006114) | 0.003168 / 0.011008 (-0.007840) | 0.049301 / 0.038508 (0.010793) | 0.032248 / 0.023109 (0.009139) | 0.266463 / 0.275898 (-0.009435) | 0.293311 / 0.323480 (-0.030168) | 0.004185 / 0.007986 (-0.003800) | 0.002681 / 0.004328 (-0.001647) | 0.048644 / 0.004250 (0.044393) | 0.040366 / 0.037052 (0.003314) | 0.280345 / 0.258489 (0.021856) | 0.312745 / 0.293841 (0.018904) | 0.029616 / 0.128546 (-0.098930) | 0.010001 / 0.075646 (-0.065646) | 0.057365 / 0.419271 (-0.361906) | 0.033189 / 0.043533 (-0.010344) | 0.267601 / 0.255139 (0.012462) | 0.285647 / 0.283200 (0.002448) | 0.017119 / 0.141683 (-0.124564) | 1.139776 / 1.452155 (-0.312378) | 1.172451 / 1.492716 (-0.320266) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095462 / 0.018006 (0.077455) | 0.303009 / 0.000490 (0.302519) | 0.000227 / 0.000200 (0.000027) | 0.000055 / 0.000054 (0.000000) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023026 / 0.037411 (-0.014385) | 0.077905 / 0.014526 (0.063380) | 0.087275 / 0.176557 (-0.089282) | 0.127355 / 0.737135 (-0.609780) | 0.088940 / 0.296338 (-0.207399) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.298267 / 0.215209 (0.083058) | 2.894679 / 2.077655 (0.817024) | 1.568663 / 1.504120 (0.064543) | 1.438342 / 1.541195 (-0.102853) | 1.456110 / 1.468490 (-0.012380) | 0.556337 / 4.584777 (-4.028440) | 0.969795 / 3.745712 (-2.775917) | 2.667348 / 5.269862 (-2.602513) | 1.767169 / 4.565676 (-2.798507) | 0.060969 / 0.424275 (-0.363306) | 0.005009 / 0.007607 (-0.002598) | 0.343299 / 0.226044 (0.117255) | 3.396529 / 2.268929 (1.127601) | 1.889816 / 55.444624 (-53.554808) | 1.635077 / 6.876477 (-5.241400) | 1.795238 / 2.142072 (-0.346835) | 0.631876 / 4.805227 (-4.173352) | 0.115483 / 6.500664 (-6.385181) | 0.041772 / 0.075469 (-0.033697) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.008423 / 1.841788 (-0.833364) | 12.432488 / 8.074308 (4.358180) | 10.418002 / 10.191392 (0.226610) | 0.142395 / 0.680424 (-0.538029) | 0.015718 / 0.534201 (-0.518483) | 0.281917 / 0.579283 (-0.297366) | 0.132619 / 0.434364 (-0.301745) | 0.318500 / 0.540337 (-0.221838) | 0.410798 / 1.386936 (-0.976138) |

![](https://cml.dev/watermark.png#3d6cd158d2e3bb9030fea7c5a9580b9d34d721ac "CML watermark")

token is ok to be public since it's only for the hub-ci


Update dataset_dict.py

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005050 / 0.011353 (-0.006303) | 0.003786 / 0.011008 (-0.007222) | 0.062406 / 0.038508 (0.023898) | 0.029459 / 0.023109 (0.006349) | 0.262388 / 0.275898 (-0.013510) | 0.274119 / 0.323480 (-0.049361) | 0.004085 / 0.007986 (-0.003901) | 0.002754 / 0.004328 (-0.001574) | 0.048779 / 0.004250 (0.044529) | 0.046187 / 0.037052 (0.009135) | 0.263513 / 0.258489 (0.005024) | 0.294260 / 0.293841 (0.000419) | 0.027391 / 0.128546 (-0.101155) | 0.010567 / 0.075646 (-0.065080) | 0.200225 / 0.419271 (-0.219046) | 0.036165 / 0.043533 (-0.007367) | 0.251757 / 0.255139 (-0.003382) | 0.268271 / 0.283200 (-0.014928) | 0.018446 / 0.141683 (-0.123237) | 1.125787 / 1.452155 (-0.326368) | 1.163172 / 1.492716 (-0.329544) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.004428 / 0.018006 (-0.013578) | 0.301730 / 0.000490 (0.301241) | 0.000215 / 0.000200 (0.000015) | 0.000045 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019424 / 0.037411 (-0.017987) | 0.062269 / 0.014526 (0.047743) | 0.074289 / 0.176557 (-0.102268) | 0.121069 / 0.737135 (-0.616067) | 0.076485 / 0.296338 (-0.219853) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.277315 / 0.215209 (0.062106) | 2.742027 / 2.077655 (0.664372) | 1.472970 / 1.504120 (-0.031150) | 1.350065 / 1.541195 (-0.191130) | 1.378806 / 1.468490 (-0.089684) | 0.567742 / 4.584777 (-4.017035) | 2.376752 / 3.745712 (-1.368960) | 2.662459 / 5.269862 (-2.607402) | 1.750396 / 4.565676 (-2.815280) | 0.063589 / 0.424275 (-0.360686) | 0.004987 / 0.007607 (-0.002620) | 0.326441 / 0.226044 (0.100397) | 3.224125 / 2.268929 (0.955197) | 1.801623 / 55.444624 (-53.643001) | 1.534712 / 6.876477 (-5.341765) | 1.652365 / 2.142072 (-0.489708) | 0.647624 / 4.805227 (-4.157603) | 0.117161 / 6.500664 (-6.383504) | 0.041908 / 0.075469 (-0.033561) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.954879 / 1.841788 (-0.886909) | 11.571875 / 8.074308 (3.497567) | 9.489146 / 10.191392 (-0.702246) | 0.141630 / 0.680424 (-0.538794) | 0.014764 / 0.534201 (-0.519437) | 0.285003 / 0.579283 (-0.294280) | 0.266138 / 0.434364 (-0.168226) | 0.323527 / 0.540337 (-0.216810) | 0.419658 / 1.386936 (-0.967278) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005359 / 0.011353 (-0.005994) | 0.003615 / 0.011008 (-0.007393) | 0.050692 / 0.038508 (0.012184) | 0.033632 / 0.023109 (0.010522) | 0.273614 / 0.275898 (-0.002284) | 0.303780 / 0.323480 (-0.019700) | 0.004171 / 0.007986 (-0.003814) | 0.002687 / 0.004328 (-0.001642) | 0.050002 / 0.004250 (0.045751) | 0.040824 / 0.037052 (0.003772) | 0.287759 / 0.258489 (0.029270) | 0.324144 / 0.293841 (0.030303) | 0.029101 / 0.128546 (-0.099445) | 0.010244 / 0.075646 (-0.065402) | 0.059599 / 0.419271 (-0.359672) | 0.033146 / 0.043533 (-0.010387) | 0.276592 / 0.255139 (0.021453) | 0.293670 / 0.283200 (0.010470) | 0.018270 / 0.141683 (-0.123413) | 1.126216 / 1.452155 (-0.325939) | 1.155658 / 1.492716 (-0.337058) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093537 / 0.018006 (0.075530) | 0.302706 / 0.000490 (0.302216) | 0.000216 / 0.000200 (0.000016) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023118 / 0.037411 (-0.014293) | 0.076995 / 0.014526 (0.062469) | 0.089476 / 0.176557 (-0.087080) | 0.130705 / 0.737135 (-0.606430) | 0.090258 / 0.296338 (-0.206081) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.285920 / 0.215209 (0.070710) | 2.830581 / 2.077655 (0.752927) | 1.561695 / 1.504120 (0.057575) | 1.522791 / 1.541195 (-0.018403) | 1.429875 / 1.468490 (-0.038615) | 0.566683 / 4.584777 (-4.018094) | 0.957157 / 3.745712 (-2.788555) | 2.663718 / 5.269862 (-2.606143) | 1.748885 / 4.565676 (-2.816791) | 0.063697 / 0.424275 (-0.360578) | 0.004996 / 0.007607 (-0.002611) | 0.340042 / 0.226044 (0.113998) | 3.352792 / 2.268929 (1.083863) | 1.907189 / 55.444624 (-53.537435) | 1.608177 / 6.876477 (-5.268300) | 1.775438 / 2.142072 (-0.366634) | 0.645264 / 4.805227 (-4.159963) | 0.116441 / 6.500664 (-6.384223) | 0.040671 / 0.075469 (-0.034798) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.005050 / 1.841788 (-0.836738) | 12.040057 / 8.074308 (3.965749) | 10.213560 / 10.191392 (0.022168) | 0.138383 / 0.680424 (-0.542041) | 0.015409 / 0.534201 (-0.518792) | 0.283509 / 0.579283 (-0.295774) | 0.125501 / 0.434364 (-0.308863) | 0.318816 / 0.540337 (-0.221521) | 0.415454 / 1.386936 (-0.971482) |

![](https://cml.dev/watermark.png#cbb29cea0e21dc0eb8f7de01d0c6ed5718d6ce4e "CML watermark")

shape returns (number of rows, number of columns)


[WebDataset] Support compressed files

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6931). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005362 / 0.011353 (-0.005991) | 0.003969 / 0.011008 (-0.007039) | 0.063390 / 0.038508 (0.024882) | 0.030814 / 0.023109 (0.007705) | 0.246891 / 0.275898 (-0.029007) | 0.271047 / 0.323480 (-0.052432) | 0.004036 / 0.007986 (-0.003950) | 0.002732 / 0.004328 (-0.001597) | 0.049466 / 0.004250 (0.045216) | 0.047227 / 0.037052 (0.010175) | 0.255978 / 0.258489 (-0.002511) | 0.297956 / 0.293841 (0.004115) | 0.028641 / 0.128546 (-0.099905) | 0.010510 / 0.075646 (-0.065136) | 0.204268 / 0.419271 (-0.215004) | 0.037093 / 0.043533 (-0.006440) | 0.247287 / 0.255139 (-0.007852) | 0.263830 / 0.283200 (-0.019370) | 0.018335 / 0.141683 (-0.123348) | 1.116074 / 1.452155 (-0.336081) | 1.182589 / 1.492716 (-0.310128) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094435 / 0.018006 (0.076429) | 0.310422 / 0.000490 (0.309932) | 0.000215 / 0.000200 (0.000015) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019220 / 0.037411 (-0.018192) | 0.062090 / 0.014526 (0.047564) | 0.074511 / 0.176557 (-0.102046) | 0.121825 / 0.737135 (-0.615310) | 0.075406 / 0.296338 (-0.220933) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.281185 / 0.215209 (0.065976) | 2.770157 / 2.077655 (0.692502) | 1.472095 / 1.504120 (-0.032025) | 1.339342 / 1.541195 (-0.201853) | 1.374621 / 1.468490 (-0.093869) | 0.566607 / 4.584777 (-4.018170) | 2.357642 / 3.745712 (-1.388070) | 2.735034 / 5.269862 (-2.534827) | 1.782779 / 4.565676 (-2.782897) | 0.063046 / 0.424275 (-0.361229) | 0.005015 / 0.007607 (-0.002592) | 0.336690 / 0.226044 (0.110646) | 3.360955 / 2.268929 (1.092027) | 1.804424 / 55.444624 (-53.640200) | 1.517334 / 6.876477 (-5.359143) | 1.665254 / 2.142072 (-0.476818) | 0.627185 / 4.805227 (-4.178042) | 0.114388 / 6.500664 (-6.386276) | 0.041788 / 0.075469 (-0.033681) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.975270 / 1.841788 (-0.866517) | 11.647633 / 8.074308 (3.573325) | 9.872873 / 10.191392 (-0.318519) | 0.141744 / 0.680424 (-0.538680) | 0.014524 / 0.534201 (-0.519677) | 0.286697 / 0.579283 (-0.292586) | 0.266837 / 0.434364 (-0.167527) | 0.328513 / 0.540337 (-0.211825) | 0.424676 / 1.386936 (-0.962260) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005654 / 0.011353 (-0.005699) | 0.004058 / 0.011008 (-0.006950) | 0.051030 / 0.038508 (0.012522) | 0.033085 / 0.023109 (0.009976) | 0.307532 / 0.275898 (0.031634) | 0.335672 / 0.323480 (0.012192) | 0.004244 / 0.007986 (-0.003742) | 0.002842 / 0.004328 (-0.001487) | 0.050131 / 0.004250 (0.045880) | 0.040709 / 0.037052 (0.003656) | 0.319514 / 0.258489 (0.061025) | 0.357153 / 0.293841 (0.063312) | 0.029014 / 0.128546 (-0.099532) | 0.010999 / 0.075646 (-0.064648) | 0.058789 / 0.419271 (-0.360482) | 0.033284 / 0.043533 (-0.010249) | 0.310783 / 0.255139 (0.055644) | 0.331466 / 0.283200 (0.048266) | 0.018998 / 0.141683 (-0.122685) | 1.138822 / 1.452155 (-0.313332) | 1.180731 / 1.492716 (-0.311985) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095725 / 0.018006 (0.077719) | 0.302788 / 0.000490 (0.302298) | 0.000206 / 0.000200 (0.000006) | 0.000056 / 0.000054 (0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023247 / 0.037411 (-0.014164) | 0.077619 / 0.014526 (0.063093) | 0.090489 / 0.176557 (-0.086067) | 0.132033 / 0.737135 (-0.605102) | 0.090964 / 0.296338 (-0.205374) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.297912 / 0.215209 (0.082703) | 2.954107 / 2.077655 (0.876452) | 1.591155 / 1.504120 (0.087035) | 1.469217 / 1.541195 (-0.071978) | 1.513315 / 1.468490 (0.044825) | 0.562728 / 4.584777 (-4.022049) | 0.960093 / 3.745712 (-2.785620) | 2.852106 / 5.269862 (-2.417756) | 1.861668 / 4.565676 (-2.704009) | 0.063530 / 0.424275 (-0.360745) | 0.005194 / 0.007607 (-0.002413) | 0.351116 / 0.226044 (0.125072) | 3.498787 / 2.268929 (1.229859) | 1.952223 / 55.444624 (-53.492401) | 1.696208 / 6.876477 (-5.180269) | 1.861650 / 2.142072 (-0.280422) | 0.653494 / 4.805227 (-4.151733) | 0.123797 / 6.500664 (-6.376868) | 0.042696 / 0.075469 (-0.032773) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.006657 / 1.841788 (-0.835131) | 12.659771 / 8.074308 (4.585463) | 10.672140 / 10.191392 (0.480748) | 0.143726 / 0.680424 (-0.536698) | 0.015895 / 0.534201 (-0.518306) | 0.285952 / 0.579283 (-0.293331) | 0.126078 / 0.434364 (-0.308286) | 0.325943 / 0.540337 (-0.214395) | 0.410774 / 1.386936 (-0.976162) |

![](https://cml.dev/watermark.png#88d53d1ae762bec6736fffb000e6540e52bf1998 "CML watermark")


### Describe the bug

When I run the code en = load_dataset("allenai/c4", "en", streaming=True), I encounter an error: raise ValueError(f"Couldn't infer the same data file format for all splits. Got {split_modules}") ValueError: Couldn't infer the same data file format for all splits. Got {'train': ('json', {}), 'validation': (None, {})}.
However, running dataset = load_dataset('allenai/c4', streaming=True, data_files={'validation': 'en/c4-validation.00003-of-00008.json.gz'}, split='validation') works fine. What is the issue here?

### Steps to reproduce the bug

run code：
import os
os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'
from datasets import load_dataset

en = load_dataset("allenai/c4", "en", streaming=True)

### Expected behavior

Successfully loaded the dataset.

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-6.5.0-28-generic-x86_64-with-glibc2.17
- Python version: 3.8.19
- `huggingface_hub` version: 0.22.2
- PyArrow version: 15.0.2
- Pandas version: 2.0.3
- `fsspec` version: 2024.2.0


Avoid downloading the whole dataset when only README.me has been touched on hub.

enhancement

New feature or request

you're right, we're tackling this here: https://github.com/huggingface/dataset-viewer/issues/2757

### Feature request

`datasets.load_dataset()` triggers a new download of the **whole dataset** when the README.md file has been touched on huggingface hub, even if data files / parquet files are the exact same.

I think the current behaviour of the load_dataset function is triggered whenever a change of the hash of latest commit on huggingface hub, but  is there a clever way to only download again the dataset **if and only if** data is modified ?

### Motivation

The current behaviour is a waste of network bandwidth / disk space / research time.

### Your contribution

I don't have time to submit a PR, but I hope a simple solution will emerge from this issue !


Update process.mdx: Code Listings Fixes

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005062 / 0.011353 (-0.006291) | 0.003410 / 0.011008 (-0.007598) | 0.062241 / 0.038508 (0.023733) | 0.030294 / 0.023109 (0.007185) | 0.249249 / 0.275898 (-0.026649) | 0.267718 / 0.323480 (-0.055761) | 0.003047 / 0.007986 (-0.004938) | 0.002661 / 0.004328 (-0.001668) | 0.049142 / 0.004250 (0.044892) | 0.047929 / 0.037052 (0.010877) | 0.255262 / 0.258489 (-0.003227) | 0.286241 / 0.293841 (-0.007600) | 0.027064 / 0.128546 (-0.101482) | 0.010374 / 0.075646 (-0.065273) | 0.201454 / 0.419271 (-0.217818) | 0.036586 / 0.043533 (-0.006947) | 0.255200 / 0.255139 (0.000061) | 0.267660 / 0.283200 (-0.015539) | 0.018621 / 0.141683 (-0.123062) | 1.159821 / 1.452155 (-0.292334) | 1.171597 / 1.492716 (-0.321120) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.004752 / 0.018006 (-0.013254) | 0.295427 / 0.000490 (0.294937) | 0.000225 / 0.000200 (0.000025) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018914 / 0.037411 (-0.018497) | 0.061180 / 0.014526 (0.046654) | 0.073649 / 0.176557 (-0.102907) | 0.120142 / 0.737135 (-0.616993) | 0.074754 / 0.296338 (-0.221585) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.286637 / 0.215209 (0.071428) | 2.807941 / 2.077655 (0.730287) | 1.473577 / 1.504120 (-0.030542) | 1.353112 / 1.541195 (-0.188083) | 1.363020 / 1.468490 (-0.105470) | 0.567745 / 4.584777 (-4.017032) | 2.384887 / 3.745712 (-1.360826) | 2.685132 / 5.269862 (-2.584730) | 1.755922 / 4.565676 (-2.809755) | 0.062296 / 0.424275 (-0.361979) | 0.004941 / 0.007607 (-0.002666) | 0.346752 / 0.226044 (0.120707) | 3.378623 / 2.268929 (1.109694) | 1.809070 / 55.444624 (-53.635555) | 1.531490 / 6.876477 (-5.344986) | 1.687954 / 2.142072 (-0.454119) | 0.639917 / 4.805227 (-4.165310) | 0.118455 / 6.500664 (-6.382209) | 0.043072 / 0.075469 (-0.032397) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.977154 / 1.841788 (-0.864634) | 11.380127 / 8.074308 (3.305819) | 9.621632 / 10.191392 (-0.569760) | 0.141768 / 0.680424 (-0.538655) | 0.014120 / 0.534201 (-0.520081) | 0.285073 / 0.579283 (-0.294210) | 0.264801 / 0.434364 (-0.169563) | 0.322357 / 0.540337 (-0.217981) | 0.431192 / 1.386936 (-0.955744) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005162 / 0.011353 (-0.006191) | 0.003499 / 0.011008 (-0.007509) | 0.049667 / 0.038508 (0.011159) | 0.032473 / 0.023109 (0.009363) | 0.259988 / 0.275898 (-0.015910) | 0.285723 / 0.323480 (-0.037757) | 0.004197 / 0.007986 (-0.003789) | 0.002710 / 0.004328 (-0.001618) | 0.049235 / 0.004250 (0.044984) | 0.040440 / 0.037052 (0.003387) | 0.276791 / 0.258489 (0.018302) | 0.311990 / 0.293841 (0.018149) | 0.029217 / 0.128546 (-0.099329) | 0.010217 / 0.075646 (-0.065429) | 0.057844 / 0.419271 (-0.361427) | 0.032799 / 0.043533 (-0.010734) | 0.260705 / 0.255139 (0.005566) | 0.280439 / 0.283200 (-0.002761) | 0.018682 / 0.141683 (-0.123001) | 1.135946 / 1.452155 (-0.316208) | 1.163144 / 1.492716 (-0.329572) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.097968 / 0.018006 (0.079961) | 0.309276 / 0.000490 (0.308786) | 0.000214 / 0.000200 (0.000014) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022623 / 0.037411 (-0.014788) | 0.075471 / 0.014526 (0.060945) | 0.087928 / 0.176557 (-0.088629) | 0.129537 / 0.737135 (-0.607599) | 0.089376 / 0.296338 (-0.206963) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.298223 / 0.215209 (0.083014) | 2.940462 / 2.077655 (0.862807) | 1.586024 / 1.504120 (0.081904) | 1.451161 / 1.541195 (-0.090034) | 1.457707 / 1.468490 (-0.010783) | 0.571172 / 4.584777 (-4.013604) | 0.961591 / 3.745712 (-2.784121) | 2.661258 / 5.269862 (-2.608604) | 1.755172 / 4.565676 (-2.810504) | 0.063430 / 0.424275 (-0.360845) | 0.005034 / 0.007607 (-0.002573) | 0.352356 / 0.226044 (0.126312) | 3.454986 / 2.268929 (1.186057) | 1.967375 / 55.444624 (-53.477249) | 1.638465 / 6.876477 (-5.238012) | 1.774098 / 2.142072 (-0.367975) | 0.650094 / 4.805227 (-4.155134) | 0.117377 / 6.500664 (-6.383287) | 0.041229 / 0.075469 (-0.034240) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.014356 / 1.841788 (-0.827432) | 12.175823 / 8.074308 (4.101515) | 10.657486 / 10.191392 (0.466094) | 0.145080 / 0.680424 (-0.535344) | 0.015563 / 0.534201 (-0.518638) | 0.287093 / 0.579283 (-0.292190) | 0.127164 / 0.434364 (-0.307200) | 0.318518 / 0.540337 (-0.221820) | 0.415333 / 1.386936 (-0.971603) |

![](https://cml.dev/watermark.png#372078f617d9210c7f073c22f5f6f4fbee52c67f "CML watermark")


Update process.mdx: Minor Code Listings Updates and Fixes


Update process.mdx: Fix code listing in Shard section


Fix NonMatchingSplitsSizesError/ExpectedMoreSplits when passing data_dir/data_files in no-code Hub datasets

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6925). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Do you think this is worth making a patch release for?
CC: @huggingface/datasets

I will add some regression tests before merging.

And I will make a patch release afterwards.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004959 / 0.011353 (-0.006394) | 0.003654 / 0.011008 (-0.007354) | 0.064087 / 0.038508 (0.025579) | 0.031942 / 0.023109 (0.008833) | 0.236830 / 0.275898 (-0.039068) | 0.265359 / 0.323480 (-0.058121) | 0.003108 / 0.007986 (-0.004878) | 0.002824 / 0.004328 (-0.001504) | 0.049102 / 0.004250 (0.044852) | 0.046070 / 0.037052 (0.009017) | 0.248830 / 0.258489 (-0.009659) | 0.283900 / 0.293841 (-0.009941) | 0.027799 / 0.128546 (-0.100747) | 0.010572 / 0.075646 (-0.065074) | 0.223595 / 0.419271 (-0.195677) | 0.036951 / 0.043533 (-0.006582) | 0.238813 / 0.255139 (-0.016326) | 0.253841 / 0.283200 (-0.029359) | 0.018471 / 0.141683 (-0.123212) | 1.131969 / 1.452155 (-0.320186) | 1.173763 / 1.492716 (-0.318954) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095504 / 0.018006 (0.077498) | 0.301469 / 0.000490 (0.300979) | 0.000212 / 0.000200 (0.000012) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019194 / 0.037411 (-0.018217) | 0.062313 / 0.014526 (0.047787) | 0.075852 / 0.176557 (-0.100704) | 0.121996 / 0.737135 (-0.615140) | 0.076416 / 0.296338 (-0.219923) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.292465 / 0.215209 (0.077256) | 2.910234 / 2.077655 (0.832579) | 1.479672 / 1.504120 (-0.024448) | 1.332281 / 1.541195 (-0.208913) | 1.354095 / 1.468490 (-0.114395) | 0.573438 / 4.584777 (-4.011339) | 2.382406 / 3.745712 (-1.363307) | 2.708289 / 5.269862 (-2.561572) | 1.739665 / 4.565676 (-2.826011) | 0.063514 / 0.424275 (-0.360761) | 0.005008 / 0.007607 (-0.002599) | 0.350070 / 0.226044 (0.124025) | 3.475837 / 2.268929 (1.206909) | 1.804639 / 55.444624 (-53.639985) | 1.520472 / 6.876477 (-5.356005) | 1.658061 / 2.142072 (-0.484011) | 0.648495 / 4.805227 (-4.156732) | 0.118394 / 6.500664 (-6.382270) | 0.042557 / 0.075469 (-0.032912) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.960772 / 1.841788 (-0.881016) | 11.451629 / 8.074308 (3.377321) | 9.613331 / 10.191392 (-0.578061) | 0.130259 / 0.680424 (-0.550164) | 0.015828 / 0.534201 (-0.518373) | 0.287581 / 0.579283 (-0.291702) | 0.266517 / 0.434364 (-0.167847) | 0.327334 / 0.540337 (-0.213003) | 0.427881 / 1.386936 (-0.959055) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005364 / 0.011353 (-0.005989) | 0.003723 / 0.011008 (-0.007285) | 0.049990 / 0.038508 (0.011482) | 0.032023 / 0.023109 (0.008913) | 0.258609 / 0.275898 (-0.017289) | 0.281250 / 0.323480 (-0.042230) | 0.004222 / 0.007986 (-0.003764) | 0.002799 / 0.004328 (-0.001529) | 0.049546 / 0.004250 (0.045296) | 0.040298 / 0.037052 (0.003246) | 0.273552 / 0.258489 (0.015063) | 0.304042 / 0.293841 (0.010201) | 0.030116 / 0.128546 (-0.098430) | 0.010792 / 0.075646 (-0.064855) | 0.058427 / 0.419271 (-0.360845) | 0.033415 / 0.043533 (-0.010118) | 0.258794 / 0.255139 (0.003655) | 0.275304 / 0.283200 (-0.007896) | 0.017944 / 0.141683 (-0.123739) | 1.109291 / 1.452155 (-0.342864) | 1.156627 / 1.492716 (-0.336090) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096700 / 0.018006 (0.078693) | 0.301108 / 0.000490 (0.300618) | 0.000208 / 0.000200 (0.000008) | 0.000054 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022632 / 0.037411 (-0.014779) | 0.075813 / 0.014526 (0.061287) | 0.090302 / 0.176557 (-0.086254) | 0.130375 / 0.737135 (-0.606760) | 0.089710 / 0.296338 (-0.206629) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.297091 / 0.215209 (0.081882) | 2.910379 / 2.077655 (0.832725) | 1.570460 / 1.504120 (0.066340) | 1.441619 / 1.541195 (-0.099576) | 1.442417 / 1.468490 (-0.026073) | 0.570034 / 4.584777 (-4.014743) | 0.952613 / 3.745712 (-2.793099) | 2.659274 / 5.269862 (-2.610588) | 1.751013 / 4.565676 (-2.814663) | 0.064639 / 0.424275 (-0.359636) | 0.005145 / 0.007607 (-0.002462) | 0.347478 / 0.226044 (0.121434) | 3.443862 / 2.268929 (1.174933) | 1.897246 / 55.444624 (-53.547379) | 1.609267 / 6.876477 (-5.267210) | 1.755116 / 2.142072 (-0.386956) | 0.658982 / 4.805227 (-4.146245) | 0.117000 / 6.500664 (-6.383664) | 0.041453 / 0.075469 (-0.034016) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.005843 / 1.841788 (-0.835944) | 12.101306 / 8.074308 (4.026998) | 10.370706 / 10.191392 (0.179314) | 0.139374 / 0.680424 (-0.541050) | 0.015605 / 0.534201 (-0.518596) | 0.286978 / 0.579283 (-0.292305) | 0.122951 / 0.434364 (-0.311413) | 0.331729 / 0.540337 (-0.208609) | 0.422088 / 1.386936 (-0.964848) |

![](https://cml.dev/watermark.png#157585f964b1c7f675860af0d21712555b34aabc "CML watermark")

Fix `NonMatchingSplitsSizesError` or `ExpectedMoreSplits` error for no-code Hub datasets if the user passes:
- `data_dir`
- `data_files`

The proposed solution is to avoid using exported dataset info (from Parquet exports) in these cases.
Additionally, also if the user passes `revision` other than "main" (so that no network requests are made).

This PR fixes a bug introduced by:
- #6714

Fix #6918, fix #6939.


Caching map result of DatasetDict.

Hi!

I'm currenty using the map function to tokenize a somewhat large dataset, so I need to use the cache to save ~25 mins.

Changing num_proc incduces the recomputation of the map, I'm not sure why and if this is excepted behavior?

here it says, that cached files are loaded sequentially:

https://github.com/huggingface/datasets/blob/bb2664cf540d5ce4b066365e7c8b26e7f1ca4743/src/datasets/arrow_dataset.py#L3005-L3006

it seems like I can pass in a fingerprint, and load it directly:

https://github.com/huggingface/datasets/blob/bb2664cf540d5ce4b066365e7c8b26e7f1ca4743/src/datasets/arrow_dataset.py#L3108-L3125

**Environment Setup:**

- Python 3.11.9
- datasets  2.19.1 conda-forge
- Linux 6.1.83-1.el9.elrepo.x86_64

**MRE**
```python
fixed raw_datasets
fixed tokenize_function

tokenized_datasets = raw_datasets.map(
                        tokenize_function,
                        batched=True,
                        num_proc=9,
                        remove_columns=['text'],
                        load_from_cache_file= True,
                        desc="Running tokenizer on dataset line_by_line",
                    )

tokenized_datasets = raw_datasets.map(
                        tokenize_function,
                        batched=True,
                        num_proc=5,
                        remove_columns=['text'],
                        load_from_cache_file= True,
                        desc="Running tokenizer on dataset line_by_line",
                    )
```


Export Parquet Tablet Audio-Set is null bytes in Arrow

### Describe the bug

Exporting the processed audio inside the table with the dataset.to_parquet function, the object pyarrow {bytes: null, path: "Some/Path"}
At the same time, the same dataset uploaded to the hub has bit arrays

![Screenshot from 2024-05-27 19-14-49](https://github.com/huggingface/datasets/assets/140120605/ddfba089-426f-4659-9df4-7a634c948b9e)
![Screenshot from 2024-05-27 19-12-51](https://github.com/huggingface/datasets/assets/140120605/4cf8c0a1-650e-491b-86c8-b475c284a021)

### Steps to reproduce the bug

1.Get dataset from audio and cast it
2.Export and push dataset
3.It’s scary to be indignant at the difference in the uploaded dataset and the fact that it was saved locally
```py
from datasets import Dataset, Audio
df = Dataset.from_csv("./datasets.csv")
df = df.cast_column("audio", Audio(16000))
df.to_parquet("./datasets.parquet")
df.push_to_hub(repo_id="************", token="**********************")
```

You can use "try replicate case" for this
[replicate_packet.zip](https://github.com/huggingface/datasets/files/15457114/replicate_packet.zip)

### Expected behavior

Two parquet tables identical in content. It is obvious?

### Environment info

Python 3.11+ (I try did it in 3.12 and got same result )