Update process.mdx: Code Listings Fixes

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005062 / 0.011353 (-0.006291) | 0.003410 / 0.011008 (-0.007598) | 0.062241 / 0.038508 (0.023733) | 0.030294 / 0.023109 (0.007185) | 0.249249 / 0.275898 (-0.026649) | 0.267718 / 0.323480 (-0.055761) | 0.003047 / 0.007986 (-0.004938) | 0.002661 / 0.004328 (-0.001668) | 0.049142 / 0.004250 (0.044892) | 0.047929 / 0.037052 (0.010877) | 0.255262 / 0.258489 (-0.003227) | 0.286241 / 0.293841 (-0.007600) | 0.027064 / 0.128546 (-0.101482) | 0.010374 / 0.075646 (-0.065273) | 0.201454 / 0.419271 (-0.217818) | 0.036586 / 0.043533 (-0.006947) | 0.255200 / 0.255139 (0.000061) | 0.267660 / 0.283200 (-0.015539) | 0.018621 / 0.141683 (-0.123062) | 1.159821 / 1.452155 (-0.292334) | 1.171597 / 1.492716 (-0.321120) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.004752 / 0.018006 (-0.013254) | 0.295427 / 0.000490 (0.294937) | 0.000225 / 0.000200 (0.000025) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018914 / 0.037411 (-0.018497) | 0.061180 / 0.014526 (0.046654) | 0.073649 / 0.176557 (-0.102907) | 0.120142 / 0.737135 (-0.616993) | 0.074754 / 0.296338 (-0.221585) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.286637 / 0.215209 (0.071428) | 2.807941 / 2.077655 (0.730287) | 1.473577 / 1.504120 (-0.030542) | 1.353112 / 1.541195 (-0.188083) | 1.363020 / 1.468490 (-0.105470) | 0.567745 / 4.584777 (-4.017032) | 2.384887 / 3.745712 (-1.360826) | 2.685132 / 5.269862 (-2.584730) | 1.755922 / 4.565676 (-2.809755) | 0.062296 / 0.424275 (-0.361979) | 0.004941 / 0.007607 (-0.002666) | 0.346752 / 0.226044 (0.120707) | 3.378623 / 2.268929 (1.109694) | 1.809070 / 55.444624 (-53.635555) | 1.531490 / 6.876477 (-5.344986) | 1.687954 / 2.142072 (-0.454119) | 0.639917 / 4.805227 (-4.165310) | 0.118455 / 6.500664 (-6.382209) | 0.043072 / 0.075469 (-0.032397) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.977154 / 1.841788 (-0.864634) | 11.380127 / 8.074308 (3.305819) | 9.621632 / 10.191392 (-0.569760) | 0.141768 / 0.680424 (-0.538655) | 0.014120 / 0.534201 (-0.520081) | 0.285073 / 0.579283 (-0.294210) | 0.264801 / 0.434364 (-0.169563) | 0.322357 / 0.540337 (-0.217981) | 0.431192 / 1.386936 (-0.955744) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005162 / 0.011353 (-0.006191) | 0.003499 / 0.011008 (-0.007509) | 0.049667 / 0.038508 (0.011159) | 0.032473 / 0.023109 (0.009363) | 0.259988 / 0.275898 (-0.015910) | 0.285723 / 0.323480 (-0.037757) | 0.004197 / 0.007986 (-0.003789) | 0.002710 / 0.004328 (-0.001618) | 0.049235 / 0.004250 (0.044984) | 0.040440 / 0.037052 (0.003387) | 0.276791 / 0.258489 (0.018302) | 0.311990 / 0.293841 (0.018149) | 0.029217 / 0.128546 (-0.099329) | 0.010217 / 0.075646 (-0.065429) | 0.057844 / 0.419271 (-0.361427) | 0.032799 / 0.043533 (-0.010734) | 0.260705 / 0.255139 (0.005566) | 0.280439 / 0.283200 (-0.002761) | 0.018682 / 0.141683 (-0.123001) | 1.135946 / 1.452155 (-0.316208) | 1.163144 / 1.492716 (-0.329572) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.097968 / 0.018006 (0.079961) | 0.309276 / 0.000490 (0.308786) | 0.000214 / 0.000200 (0.000014) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022623 / 0.037411 (-0.014788) | 0.075471 / 0.014526 (0.060945) | 0.087928 / 0.176557 (-0.088629) | 0.129537 / 0.737135 (-0.607599) | 0.089376 / 0.296338 (-0.206963) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.298223 / 0.215209 (0.083014) | 2.940462 / 2.077655 (0.862807) | 1.586024 / 1.504120 (0.081904) | 1.451161 / 1.541195 (-0.090034) | 1.457707 / 1.468490 (-0.010783) | 0.571172 / 4.584777 (-4.013604) | 0.961591 / 3.745712 (-2.784121) | 2.661258 / 5.269862 (-2.608604) | 1.755172 / 4.565676 (-2.810504) | 0.063430 / 0.424275 (-0.360845) | 0.005034 / 0.007607 (-0.002573) | 0.352356 / 0.226044 (0.126312) | 3.454986 / 2.268929 (1.186057) | 1.967375 / 55.444624 (-53.477249) | 1.638465 / 6.876477 (-5.238012) | 1.774098 / 2.142072 (-0.367975) | 0.650094 / 4.805227 (-4.155134) | 0.117377 / 6.500664 (-6.383287) | 0.041229 / 0.075469 (-0.034240) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.014356 / 1.841788 (-0.827432) | 12.175823 / 8.074308 (4.101515) | 10.657486 / 10.191392 (0.466094) | 0.145080 / 0.680424 (-0.535344) | 0.015563 / 0.534201 (-0.518638) | 0.287093 / 0.579283 (-0.292190) | 0.127164 / 0.434364 (-0.307200) | 0.318518 / 0.540337 (-0.221820) | 0.415333 / 1.386936 (-0.971603) |

![](https://cml.dev/watermark.png#372078f617d9210c7f073c22f5f6f4fbee52c67f "CML watermark")


Update process.mdx: Minor Code Listings Updates and Fixes


Update process.mdx: Fix code listing in Shard section


Fix NonMatchingSplitsSizesError/ExpectedMoreSplits when passing data_dir/data_files in no-code Hub datasets

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6925). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Do you think this is worth making a patch release for?
CC: @huggingface/datasets

I will add some regression tests before merging.

And I will make a patch release afterwards.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004959 / 0.011353 (-0.006394) | 0.003654 / 0.011008 (-0.007354) | 0.064087 / 0.038508 (0.025579) | 0.031942 / 0.023109 (0.008833) | 0.236830 / 0.275898 (-0.039068) | 0.265359 / 0.323480 (-0.058121) | 0.003108 / 0.007986 (-0.004878) | 0.002824 / 0.004328 (-0.001504) | 0.049102 / 0.004250 (0.044852) | 0.046070 / 0.037052 (0.009017) | 0.248830 / 0.258489 (-0.009659) | 0.283900 / 0.293841 (-0.009941) | 0.027799 / 0.128546 (-0.100747) | 0.010572 / 0.075646 (-0.065074) | 0.223595 / 0.419271 (-0.195677) | 0.036951 / 0.043533 (-0.006582) | 0.238813 / 0.255139 (-0.016326) | 0.253841 / 0.283200 (-0.029359) | 0.018471 / 0.141683 (-0.123212) | 1.131969 / 1.452155 (-0.320186) | 1.173763 / 1.492716 (-0.318954) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095504 / 0.018006 (0.077498) | 0.301469 / 0.000490 (0.300979) | 0.000212 / 0.000200 (0.000012) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019194 / 0.037411 (-0.018217) | 0.062313 / 0.014526 (0.047787) | 0.075852 / 0.176557 (-0.100704) | 0.121996 / 0.737135 (-0.615140) | 0.076416 / 0.296338 (-0.219923) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.292465 / 0.215209 (0.077256) | 2.910234 / 2.077655 (0.832579) | 1.479672 / 1.504120 (-0.024448) | 1.332281 / 1.541195 (-0.208913) | 1.354095 / 1.468490 (-0.114395) | 0.573438 / 4.584777 (-4.011339) | 2.382406 / 3.745712 (-1.363307) | 2.708289 / 5.269862 (-2.561572) | 1.739665 / 4.565676 (-2.826011) | 0.063514 / 0.424275 (-0.360761) | 0.005008 / 0.007607 (-0.002599) | 0.350070 / 0.226044 (0.124025) | 3.475837 / 2.268929 (1.206909) | 1.804639 / 55.444624 (-53.639985) | 1.520472 / 6.876477 (-5.356005) | 1.658061 / 2.142072 (-0.484011) | 0.648495 / 4.805227 (-4.156732) | 0.118394 / 6.500664 (-6.382270) | 0.042557 / 0.075469 (-0.032912) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.960772 / 1.841788 (-0.881016) | 11.451629 / 8.074308 (3.377321) | 9.613331 / 10.191392 (-0.578061) | 0.130259 / 0.680424 (-0.550164) | 0.015828 / 0.534201 (-0.518373) | 0.287581 / 0.579283 (-0.291702) | 0.266517 / 0.434364 (-0.167847) | 0.327334 / 0.540337 (-0.213003) | 0.427881 / 1.386936 (-0.959055) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005364 / 0.011353 (-0.005989) | 0.003723 / 0.011008 (-0.007285) | 0.049990 / 0.038508 (0.011482) | 0.032023 / 0.023109 (0.008913) | 0.258609 / 0.275898 (-0.017289) | 0.281250 / 0.323480 (-0.042230) | 0.004222 / 0.007986 (-0.003764) | 0.002799 / 0.004328 (-0.001529) | 0.049546 / 0.004250 (0.045296) | 0.040298 / 0.037052 (0.003246) | 0.273552 / 0.258489 (0.015063) | 0.304042 / 0.293841 (0.010201) | 0.030116 / 0.128546 (-0.098430) | 0.010792 / 0.075646 (-0.064855) | 0.058427 / 0.419271 (-0.360845) | 0.033415 / 0.043533 (-0.010118) | 0.258794 / 0.255139 (0.003655) | 0.275304 / 0.283200 (-0.007896) | 0.017944 / 0.141683 (-0.123739) | 1.109291 / 1.452155 (-0.342864) | 1.156627 / 1.492716 (-0.336090) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096700 / 0.018006 (0.078693) | 0.301108 / 0.000490 (0.300618) | 0.000208 / 0.000200 (0.000008) | 0.000054 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022632 / 0.037411 (-0.014779) | 0.075813 / 0.014526 (0.061287) | 0.090302 / 0.176557 (-0.086254) | 0.130375 / 0.737135 (-0.606760) | 0.089710 / 0.296338 (-0.206629) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.297091 / 0.215209 (0.081882) | 2.910379 / 2.077655 (0.832725) | 1.570460 / 1.504120 (0.066340) | 1.441619 / 1.541195 (-0.099576) | 1.442417 / 1.468490 (-0.026073) | 0.570034 / 4.584777 (-4.014743) | 0.952613 / 3.745712 (-2.793099) | 2.659274 / 5.269862 (-2.610588) | 1.751013 / 4.565676 (-2.814663) | 0.064639 / 0.424275 (-0.359636) | 0.005145 / 0.007607 (-0.002462) | 0.347478 / 0.226044 (0.121434) | 3.443862 / 2.268929 (1.174933) | 1.897246 / 55.444624 (-53.547379) | 1.609267 / 6.876477 (-5.267210) | 1.755116 / 2.142072 (-0.386956) | 0.658982 / 4.805227 (-4.146245) | 0.117000 / 6.500664 (-6.383664) | 0.041453 / 0.075469 (-0.034016) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.005843 / 1.841788 (-0.835944) | 12.101306 / 8.074308 (4.026998) | 10.370706 / 10.191392 (0.179314) | 0.139374 / 0.680424 (-0.541050) | 0.015605 / 0.534201 (-0.518596) | 0.286978 / 0.579283 (-0.292305) | 0.122951 / 0.434364 (-0.311413) | 0.331729 / 0.540337 (-0.208609) | 0.422088 / 1.386936 (-0.964848) |

![](https://cml.dev/watermark.png#157585f964b1c7f675860af0d21712555b34aabc "CML watermark")

Fix `NonMatchingSplitsSizesError` or `ExpectedMoreSplits` error for no-code Hub datasets if the user passes:
- `data_dir`
- `data_files`

The proposed solution is to avoid using exported dataset info (from Parquet exports) in these cases.
Additionally, also if the user passes `revision` other than "main" (so that no network requests are made).

This PR fixes a bug introduced by:
- #6714

Fix #6918, fix #6939.


Caching map result of DatasetDict.

Hi!

I'm currenty using the map function to tokenize a somewhat large dataset, so I need to use the cache to save ~25 mins.

Changing num_proc incduces the recomputation of the map, I'm not sure why and if this is excepted behavior?

here it says, that cached files are loaded sequentially:

https://github.com/huggingface/datasets/blob/bb2664cf540d5ce4b066365e7c8b26e7f1ca4743/src/datasets/arrow_dataset.py#L3005-L3006

it seems like I can pass in a fingerprint, and load it directly:

https://github.com/huggingface/datasets/blob/bb2664cf540d5ce4b066365e7c8b26e7f1ca4743/src/datasets/arrow_dataset.py#L3108-L3125

**Environment Setup:**

- Python 3.11.9
- datasets  2.19.1 conda-forge
- Linux 6.1.83-1.el9.elrepo.x86_64

**MRE**
```python
fixed raw_datasets
fixed tokenize_function

tokenized_datasets = raw_datasets.map(
                        tokenize_function,
                        batched=True,
                        num_proc=9,
                        remove_columns=['text'],
                        load_from_cache_file= True,
                        desc="Running tokenizer on dataset line_by_line",
                    )

tokenized_datasets = raw_datasets.map(
                        tokenize_function,
                        batched=True,
                        num_proc=5,
                        remove_columns=['text'],
                        load_from_cache_file= True,
                        desc="Running tokenizer on dataset line_by_line",
                    )
```


Export Parquet Tablet Audio-Set is null bytes in Arrow

### Describe the bug

Exporting the processed audio inside the table with the dataset.to_parquet function, the object pyarrow {bytes: null, path: "Some/Path"}
At the same time, the same dataset uploaded to the hub has bit arrays

![Screenshot from 2024-05-27 19-14-49](https://github.com/huggingface/datasets/assets/140120605/ddfba089-426f-4659-9df4-7a634c948b9e)
![Screenshot from 2024-05-27 19-12-51](https://github.com/huggingface/datasets/assets/140120605/4cf8c0a1-650e-491b-86c8-b475c284a021)

### Steps to reproduce the bug

1.Get dataset from audio and cast it
2.Export and push dataset
3.It’s scary to be indignant at the difference in the uploaded dataset and the fact that it was saved locally
```py
from datasets import Dataset, Audio
df = Dataset.from_csv("./datasets.csv")
df = df.cast_column("audio", Audio(16000))
df.to_parquet("./datasets.parquet")
df.push_to_hub(repo_id="************", token="**********************")
```

You can use "try replicate case" for this
[replicate_packet.zip](https://github.com/huggingface/datasets/files/15457114/replicate_packet.zip)

### Expected behavior

Two parquet tables identical in content. It is obvious?

### Environment info

Python 3.11+ (I try did it in 3.12 and got same result )


Remove torchaudio remnants from code

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6922). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005525 / 0.011353 (-0.005828) | 0.004013 / 0.011008 (-0.006996) | 0.063931 / 0.038508 (0.025423) | 0.033857 / 0.023109 (0.010748) | 0.250910 / 0.275898 (-0.024988) | 0.278289 / 0.323480 (-0.045191) | 0.004289 / 0.007986 (-0.003697) | 0.002800 / 0.004328 (-0.001529) | 0.050127 / 0.004250 (0.045877) | 0.048901 / 0.037052 (0.011848) | 0.260628 / 0.258489 (0.002139) | 0.293904 / 0.293841 (0.000063) | 0.028339 / 0.128546 (-0.100207) | 0.010879 / 0.075646 (-0.064767) | 0.203618 / 0.419271 (-0.215654) | 0.036241 / 0.043533 (-0.007292) | 0.250481 / 0.255139 (-0.004657) | 0.274274 / 0.283200 (-0.008926) | 0.018912 / 0.141683 (-0.122771) | 1.146785 / 1.452155 (-0.305370) | 1.199795 / 1.492716 (-0.292921) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095571 / 0.018006 (0.077564) | 0.302961 / 0.000490 (0.302471) | 0.000217 / 0.000200 (0.000017) | 0.000109 / 0.000054 (0.000055) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.020121 / 0.037411 (-0.017290) | 0.063231 / 0.014526 (0.048705) | 0.075434 / 0.176557 (-0.101122) | 0.123994 / 0.737135 (-0.613141) | 0.076479 / 0.296338 (-0.219860) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.277816 / 0.215209 (0.062607) | 2.775481 / 2.077655 (0.697826) | 1.454881 / 1.504120 (-0.049239) | 1.339055 / 1.541195 (-0.202140) | 1.347810 / 1.468490 (-0.120681) | 0.572802 / 4.584777 (-4.011975) | 2.357490 / 3.745712 (-1.388222) | 2.822548 / 5.269862 (-2.447313) | 1.746538 / 4.565676 (-2.819138) | 0.066159 / 0.424275 (-0.358116) | 0.005037 / 0.007607 (-0.002570) | 0.329256 / 0.226044 (0.103212) | 3.277511 / 2.268929 (1.008582) | 1.807855 / 55.444624 (-53.636769) | 1.505507 / 6.876477 (-5.370970) | 1.634237 / 2.142072 (-0.507835) | 0.643999 / 4.805227 (-4.161229) | 0.117494 / 6.500664 (-6.383170) | 0.042634 / 0.075469 (-0.032835) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.977689 / 1.841788 (-0.864098) | 12.261836 / 8.074308 (4.187528) | 9.871541 / 10.191392 (-0.319851) | 0.147293 / 0.680424 (-0.533130) | 0.015134 / 0.534201 (-0.519067) | 0.287677 / 0.579283 (-0.291606) | 0.264622 / 0.434364 (-0.169742) | 0.330511 / 0.540337 (-0.209826) | 0.467618 / 1.386936 (-0.919318) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005690 / 0.011353 (-0.005663) | 0.003801 / 0.011008 (-0.007207) | 0.051817 / 0.038508 (0.013309) | 0.033355 / 0.023109 (0.010246) | 0.264416 / 0.275898 (-0.011482) | 0.288494 / 0.323480 (-0.034986) | 0.004246 / 0.007986 (-0.003740) | 0.002814 / 0.004328 (-0.001515) | 0.050547 / 0.004250 (0.046297) | 0.042977 / 0.037052 (0.005925) | 0.276884 / 0.258489 (0.018395) | 0.303758 / 0.293841 (0.009917) | 0.029412 / 0.128546 (-0.099134) | 0.010697 / 0.075646 (-0.064949) | 0.059497 / 0.419271 (-0.359775) | 0.033670 / 0.043533 (-0.009862) | 0.261311 / 0.255139 (0.006172) | 0.286478 / 0.283200 (0.003278) | 0.019386 / 0.141683 (-0.122297) | 1.155943 / 1.452155 (-0.296211) | 1.198512 / 1.492716 (-0.294205) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092954 / 0.018006 (0.074948) | 0.294144 / 0.000490 (0.293655) | 0.000213 / 0.000200 (0.000013) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023013 / 0.037411 (-0.014398) | 0.077161 / 0.014526 (0.062635) | 0.089957 / 0.176557 (-0.086600) | 0.129305 / 0.737135 (-0.607831) | 0.091006 / 0.296338 (-0.205333) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.294091 / 0.215209 (0.078882) | 2.885395 / 2.077655 (0.807741) | 1.555658 / 1.504120 (0.051538) | 1.423276 / 1.541195 (-0.117919) | 1.476485 / 1.468490 (0.007995) | 0.569507 / 4.584777 (-4.015270) | 0.979221 / 3.745712 (-2.766491) | 2.818503 / 5.269862 (-2.451358) | 1.871938 / 4.565676 (-2.693739) | 0.064342 / 0.424275 (-0.359933) | 0.005495 / 0.007607 (-0.002112) | 0.351451 / 0.226044 (0.125407) | 3.516078 / 2.268929 (1.247149) | 1.928351 / 55.444624 (-53.516273) | 1.625362 / 6.876477 (-5.251115) | 1.813756 / 2.142072 (-0.328317) | 0.657642 / 4.805227 (-4.147585) | 0.117893 / 6.500664 (-6.382771) | 0.042009 / 0.075469 (-0.033460) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.032893 / 1.841788 (-0.808894) | 12.983400 / 8.074308 (4.909092) | 10.747204 / 10.191392 (0.555812) | 0.133163 / 0.680424 (-0.547261) | 0.015875 / 0.534201 (-0.518326) | 0.312592 / 0.579283 (-0.266691) | 0.124780 / 0.434364 (-0.309584) | 0.350735 / 0.540337 (-0.189603) | 0.447130 / 1.386936 (-0.939806) |

![](https://cml.dev/watermark.png#048c789607af0370c1f2337248897956f7a91617 "CML watermark")

Remove torchaudio remnants from code.

Follow-up on:
- #5573


Support fsspec 2024.5.0

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6921). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005252 / 0.011353 (-0.006100) | 0.003752 / 0.011008 (-0.007257) | 0.064034 / 0.038508 (0.025526) | 0.031205 / 0.023109 (0.008096) | 0.248903 / 0.275898 (-0.026995) | 0.275808 / 0.323480 (-0.047671) | 0.003135 / 0.007986 (-0.004851) | 0.002635 / 0.004328 (-0.001693) | 0.049869 / 0.004250 (0.045619) | 0.047602 / 0.037052 (0.010549) | 0.259738 / 0.258489 (0.001249) | 0.296131 / 0.293841 (0.002290) | 0.027467 / 0.128546 (-0.101080) | 0.010449 / 0.075646 (-0.065197) | 0.201369 / 0.419271 (-0.217903) | 0.036317 / 0.043533 (-0.007216) | 0.244347 / 0.255139 (-0.010792) | 0.267597 / 0.283200 (-0.015602) | 0.019930 / 0.141683 (-0.121753) | 1.149012 / 1.452155 (-0.303143) | 1.188083 / 1.492716 (-0.304633) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095190 / 0.018006 (0.077184) | 0.300705 / 0.000490 (0.300215) | 0.000222 / 0.000200 (0.000022) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019297 / 0.037411 (-0.018115) | 0.063183 / 0.014526 (0.048657) | 0.075094 / 0.176557 (-0.101463) | 0.123556 / 0.737135 (-0.613579) | 0.076721 / 0.296338 (-0.219618) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.284136 / 0.215209 (0.068927) | 2.814041 / 2.077655 (0.736387) | 1.471038 / 1.504120 (-0.033082) | 1.344002 / 1.541195 (-0.197193) | 1.353875 / 1.468490 (-0.114615) | 0.599495 / 4.584777 (-3.985282) | 2.394491 / 3.745712 (-1.351221) | 2.781734 / 5.269862 (-2.488128) | 1.729829 / 4.565676 (-2.835848) | 0.064194 / 0.424275 (-0.360081) | 0.005022 / 0.007607 (-0.002585) | 0.343384 / 0.226044 (0.117340) | 3.357067 / 2.268929 (1.088139) | 1.816323 / 55.444624 (-53.628301) | 1.549405 / 6.876477 (-5.327072) | 1.594394 / 2.142072 (-0.547679) | 0.660650 / 4.805227 (-4.144578) | 0.120271 / 6.500664 (-6.380393) | 0.042422 / 0.075469 (-0.033047) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.975776 / 1.841788 (-0.866011) | 11.828093 / 8.074308 (3.753784) | 9.384164 / 10.191392 (-0.807228) | 0.140761 / 0.680424 (-0.539663) | 0.014038 / 0.534201 (-0.520163) | 0.284904 / 0.579283 (-0.294379) | 0.263430 / 0.434364 (-0.170934) | 0.320856 / 0.540337 (-0.219482) | 0.419199 / 1.386936 (-0.967737) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005672 / 0.011353 (-0.005681) | 0.003667 / 0.011008 (-0.007341) | 0.049989 / 0.038508 (0.011481) | 0.033115 / 0.023109 (0.010006) | 0.269808 / 0.275898 (-0.006090) | 0.293286 / 0.323480 (-0.030193) | 0.004238 / 0.007986 (-0.003748) | 0.002722 / 0.004328 (-0.001606) | 0.049516 / 0.004250 (0.045265) | 0.042076 / 0.037052 (0.005024) | 0.282182 / 0.258489 (0.023693) | 0.310817 / 0.293841 (0.016976) | 0.029824 / 0.128546 (-0.098722) | 0.010516 / 0.075646 (-0.065130) | 0.058223 / 0.419271 (-0.361049) | 0.033263 / 0.043533 (-0.010270) | 0.268769 / 0.255139 (0.013630) | 0.288308 / 0.283200 (0.005108) | 0.018531 / 0.141683 (-0.123151) | 1.136806 / 1.452155 (-0.315349) | 1.192636 / 1.492716 (-0.300080) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096583 / 0.018006 (0.078577) | 0.303678 / 0.000490 (0.303188) | 0.000211 / 0.000200 (0.000011) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022741 / 0.037411 (-0.014670) | 0.075799 / 0.014526 (0.061273) | 0.089930 / 0.176557 (-0.086626) | 0.129093 / 0.737135 (-0.608042) | 0.089672 / 0.296338 (-0.206666) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.292789 / 0.215209 (0.077580) | 2.860137 / 2.077655 (0.782483) | 1.566678 / 1.504120 (0.062558) | 1.437756 / 1.541195 (-0.103439) | 1.472347 / 1.468490 (0.003857) | 0.566814 / 4.584777 (-4.017963) | 0.963918 / 3.745712 (-2.781794) | 2.717199 / 5.269862 (-2.552663) | 1.763612 / 4.565676 (-2.802064) | 0.063601 / 0.424275 (-0.360674) | 0.005308 / 0.007607 (-0.002299) | 0.363111 / 0.226044 (0.137066) | 3.458222 / 2.268929 (1.189293) | 1.939185 / 55.444624 (-53.505440) | 1.659552 / 6.876477 (-5.216925) | 1.801006 / 2.142072 (-0.341067) | 0.648884 / 4.805227 (-4.156343) | 0.116259 / 6.500664 (-6.384405) | 0.041384 / 0.075469 (-0.034085) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.001594 / 1.841788 (-0.840194) | 12.371125 / 8.074308 (4.296817) | 10.489763 / 10.191392 (0.298371) | 0.132500 / 0.680424 (-0.547924) | 0.014742 / 0.534201 (-0.519459) | 0.282258 / 0.579283 (-0.297026) | 0.122755 / 0.434364 (-0.311608) | 0.346068 / 0.540337 (-0.194269) | 0.424943 / 1.386936 (-0.961994) |

![](https://cml.dev/watermark.png#df445c20346a34c08e7e039e4ec1a302eef3a69c "CML watermark")

Support fsspec 2024.5.0.


[WebDataset] Add `.pth` support for torch tensors

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6920). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005643 / 0.011353 (-0.005710) | 0.003810 / 0.011008 (-0.007198) | 0.065896 / 0.038508 (0.027388) | 0.031692 / 0.023109 (0.008583) | 0.258297 / 0.275898 (-0.017601) | 0.294555 / 0.323480 (-0.028925) | 0.004403 / 0.007986 (-0.003583) | 0.002857 / 0.004328 (-0.001472) | 0.049848 / 0.004250 (0.045597) | 0.049719 / 0.037052 (0.012666) | 0.266393 / 0.258489 (0.007904) | 0.306214 / 0.293841 (0.012373) | 0.028283 / 0.128546 (-0.100264) | 0.010450 / 0.075646 (-0.065196) | 0.203064 / 0.419271 (-0.216208) | 0.036535 / 0.043533 (-0.006998) | 0.247839 / 0.255139 (-0.007300) | 0.270538 / 0.283200 (-0.012661) | 0.018748 / 0.141683 (-0.122935) | 1.117478 / 1.452155 (-0.334677) | 1.162575 / 1.492716 (-0.330141) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.101074 / 0.018006 (0.083068) | 0.304321 / 0.000490 (0.303831) | 0.000270 / 0.000200 (0.000070) | 0.000045 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019036 / 0.037411 (-0.018376) | 0.064496 / 0.014526 (0.049970) | 0.076848 / 0.176557 (-0.099709) | 0.122979 / 0.737135 (-0.614156) | 0.078008 / 0.296338 (-0.218330) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.287009 / 0.215209 (0.071800) | 2.839084 / 2.077655 (0.761429) | 1.495977 / 1.504120 (-0.008143) | 1.379147 / 1.541195 (-0.162047) | 1.413170 / 1.468490 (-0.055320) | 0.616408 / 4.584777 (-3.968369) | 2.419183 / 3.745712 (-1.326529) | 2.905720 / 5.269862 (-2.364142) | 1.801634 / 4.565676 (-2.764043) | 0.064034 / 0.424275 (-0.360241) | 0.005098 / 0.007607 (-0.002509) | 0.341732 / 0.226044 (0.115688) | 3.365262 / 2.268929 (1.096334) | 1.844335 / 55.444624 (-53.600289) | 1.561450 / 6.876477 (-5.315027) | 1.646254 / 2.142072 (-0.495819) | 0.654993 / 4.805227 (-4.150234) | 0.119837 / 6.500664 (-6.380827) | 0.043375 / 0.075469 (-0.032094) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.000352 / 1.841788 (-0.841435) | 12.765122 / 8.074308 (4.690813) | 9.818879 / 10.191392 (-0.372513) | 0.133986 / 0.680424 (-0.546438) | 0.014065 / 0.534201 (-0.520136) | 0.295859 / 0.579283 (-0.283424) | 0.268497 / 0.434364 (-0.165867) | 0.330909 / 0.540337 (-0.209429) | 0.449218 / 1.386936 (-0.937718) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005646 / 0.011353 (-0.005707) | 0.003926 / 0.011008 (-0.007082) | 0.050437 / 0.038508 (0.011929) | 0.031828 / 0.023109 (0.008719) | 0.268218 / 0.275898 (-0.007680) | 0.292987 / 0.323480 (-0.030493) | 0.004353 / 0.007986 (-0.003633) | 0.002933 / 0.004328 (-0.001395) | 0.050357 / 0.004250 (0.046107) | 0.042988 / 0.037052 (0.005935) | 0.281627 / 0.258489 (0.023138) | 0.305664 / 0.293841 (0.011824) | 0.030162 / 0.128546 (-0.098385) | 0.010856 / 0.075646 (-0.064790) | 0.059528 / 0.419271 (-0.359744) | 0.033800 / 0.043533 (-0.009733) | 0.268200 / 0.255139 (0.013061) | 0.284982 / 0.283200 (0.001782) | 0.019105 / 0.141683 (-0.122578) | 1.171714 / 1.452155 (-0.280441) | 1.205690 / 1.492716 (-0.287026) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.100979 / 0.018006 (0.082973) | 0.314691 / 0.000490 (0.314201) | 0.000217 / 0.000200 (0.000017) | 0.000050 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023816 / 0.037411 (-0.013596) | 0.081749 / 0.014526 (0.067223) | 0.090118 / 0.176557 (-0.086438) | 0.131615 / 0.737135 (-0.605520) | 0.091821 / 0.296338 (-0.204517) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.301222 / 0.215209 (0.086013) | 2.835310 / 2.077655 (0.757655) | 1.562396 / 1.504120 (0.058276) | 1.432365 / 1.541195 (-0.108830) | 1.468358 / 1.468490 (-0.000132) | 0.561300 / 4.584777 (-4.023477) | 0.962294 / 3.745712 (-2.783419) | 2.799705 / 5.269862 (-2.470157) | 1.803035 / 4.565676 (-2.762642) | 0.064104 / 0.424275 (-0.360171) | 0.005480 / 0.007607 (-0.002127) | 0.342519 / 0.226044 (0.116475) | 3.406286 / 2.268929 (1.137357) | 1.966962 / 55.444624 (-53.477663) | 1.654664 / 6.876477 (-5.221813) | 1.829303 / 2.142072 (-0.312769) | 0.650932 / 4.805227 (-4.154295) | 0.119211 / 6.500664 (-6.381453) | 0.043739 / 0.075469 (-0.031730) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.006657 / 1.841788 (-0.835130) | 12.915348 / 8.074308 (4.841040) | 10.808156 / 10.191392 (0.616764) | 0.132664 / 0.680424 (-0.547760) | 0.015574 / 0.534201 (-0.518627) | 0.284525 / 0.579283 (-0.294758) | 0.122322 / 0.434364 (-0.312042) | 0.326826 / 0.540337 (-0.213511) | 0.416593 / 1.386936 (-0.970343) |

![](https://cml.dev/watermark.png#15ffefe5be194790a50af88ae1236a51b0ac95e6 "CML watermark")

In this PR I add support for `.pth` but with `weights_only=True` to disallow the use of pickle


Invalid YAML in README.md: unknown tag !<tag:yaml.org,2002:python/tuple>

### Describe the bug

I wrote a notebook to load an existing dataset, process it, and upload as a private dataset using `dataset.push_to_hub(...)` at the end. The push to hub is failing with:

```
ValueError: Invalid metadata in README.md.
- Invalid YAML in README.md: unknown tag !<tag:yaml.org,2002:python[/tuple](http://192.168.1.128:8888/tuple)> (50:11)

 47 |             - 4
 48 |             - 4
 49 |             - 8
 50 |           - !!binary |
----------------^
 51 |             TwAAAA==
 52 |           '1': !!python[/object/apply](http://192.168.1.128:8888/object/apply):nump ...
```

My dataset has a `train` and `validation` dataset.  These are the features:

```
{'c1': Value(dtype='string', id=None),
 'c2': Value(dtype='string', id=None),
 'c3': [{'value': Value(dtype='string', id=None),
   'start': Value(dtype='int64', id=None),
   'end': Value(dtype='int64', id=None),
   'label': Value(dtype='string', id=None)}],
 'c4': Value(dtype='string', id=None),
 'c5': Value(dtype='string', id=None),
 'c6': Value(dtype='string', id=None),
 'c7': Value(dtype='string', id=None),
 'c8': Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None),
 'c9': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),
 'c10': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),
 'labels': Sequence(feature=ClassLabel(names=['O', 'B-ABC', 'I-ABC', ...], id=None), length=-1, id=None),
 'c12': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}
```

This used to work until I decided to cast the `labels` feature to a `Sequence(ClassLabel(...))` type with:

```
ds['train'] = ds['train'].cast_column("labels", Sequence(ClassLabel(names=list(labels))))
ds['validation'] = ds['validation'].cast_column("labels", Sequence(ClassLabel(names=list(labels))))
```

### Steps to reproduce the bug

1. Start with any token classification dataset.
2. Add a `labels` column with data such as `[0,0,0,12,13,13,13,0,0]`.
3. Cast the label column from `Sequence` to `Sequence(ClassLabel))` with:

```
labels = ['O', 'B-TEST', 'I-TEST']
ds = ds.cast_column("labels", Sequence(ClassLabel(names=labels)))
```

4. Push to hub with `ds.push_to_hub("me/awesome-stuff-dataset")`

### Expected behavior

I expected `push_to_hub` to successfully push my dataset to the hub without error.

### Environment info

Python 3.11.9

datasets==2.19.1
transformers==4.41.1
PyYAML==6.0.1


NonMatchingSplitsSizesError when using data_dir

bug

Something isn't working

Thanks for reporting, @srehaag.

We are investigating this issue.

I confirm there is a bug for data-based Hub datasets when the user passes `data_dir`, which was introduced by PR:
- #6714

### Describe the bug

Loading a dataset from with a data_dir argument generates a NonMatchingSplitsSizesError if there are multiple directories in the dataset.

This appears to happen because the expected split is calculated based on the data in all the directories whereas the recorded split is calculated based on the data in the directory specified using the data_dir argument.

This is recent behavior. Until the past few weeks loading using the data_dir argument worked without any issue.

### Steps to reproduce the bug

Simple test dataset available here: https://huggingface.co/datasets/srehaag/hf-bug-temp

The dataset contains two directories "data1" and "data2", each with a file called "train.parquet" with a 2 x 5 table.

from datasets import load_dataset
dataset = load_dataset("srehaag/hf-bug-temp", data_dir = "data1")

Generates:

---------------------------------------------------------------------------
NonMatchingSplitsSizesError               Traceback (most recent call last)
Cell In[3], line 2
      1 from datasets import load_dataset
----> 2 dataset = load_dataset("srehaag/hf-bug-temp", data_dir = "data1")

File ~/.python/current/lib/python3.10/site-packages/datasets/load.py:2609, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, ignore_verifications, keep_in_memory, save_infos, revision, token, use_auth_token, task, streaming, num_proc, storage_options, trust_remote_code, **config_kwargs)
   2606     return builder_instance.as_streaming_dataset(split=split)
   2608 # Download and prepare data
-> 2609 builder_instance.download_and_prepare(
   2610     download_config=download_config,
   2611     download_mode=download_mode,
   2612     verification_mode=verification_mode,
   2613     num_proc=num_proc,
   2614     storage_options=storage_options,
   2615 )
   2617 # Build dataset for splits
   2618 keep_in_memory = (
   2619     keep_in_memory if keep_in_memory is not None else is_small_dataset(builder_instance.info.dataset_size)
   2620 )

File ~/.python/current/lib/python3.10/site-packages/datasets/builder.py:1027, in DatasetBuilder.download_and_prepare(self, output_dir, download_config, download_mode, verification_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, file_format, max_shard_size, num_proc, storage_options, **download_and_prepare_kwargs)
   1025     if num_proc is not None:
   1026         prepare_split_kwargs["num_proc"] = num_proc
-> 1027     self._download_and_prepare(
   1028         dl_manager=dl_manager,
   1029         verification_mode=verification_mode,
   1030         **prepare_split_kwargs,
   1031         **download_and_prepare_kwargs,
   1032     )
   1033 # Sync info
   1034 self.info.dataset_size = sum(split.num_bytes for split in self.info.splits.values())

File ~/.python/current/lib/python3.10/site-packages/datasets/builder.py:1140, in DatasetBuilder._download_and_prepare(self, dl_manager, verification_mode, **prepare_split_kwargs)
   1137     dl_manager.manage_extracted_files()
   1139 if verification_mode == VerificationMode.BASIC_CHECKS or verification_mode == VerificationMode.ALL_CHECKS:
-> 1140     verify_splits(self.info.splits, split_dict)
   1142 # Update the info object with the splits.
   1143 self.info.splits = split_dict

File ~/.python/current/lib/python3.10/site-packages/datasets/utils/info_utils.py:101, in verify_splits(expected_splits, recorded_splits)
     95 bad_splits = [
     96     {"expected": expected_splits[name], "recorded": recorded_splits[name]}
     97     for name in expected_splits
     98     if expected_splits[name].num_examples != recorded_splits[name].num_examples
     99 ]
    100 if len(bad_splits) > 0:
--> 101     raise NonMatchingSplitsSizesError(str(bad_splits))
    102 logger.info("All the splits matched successfully.")

NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=212, num_examples=10, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='train', num_bytes=106, num_examples=5, shard_lengths=None, dataset_name='hf-bug-temp')}]

__________

By contrast, this loads the data from both data1/train.parquet and data2/train.parquet without any error message:

from datasets import load_dataset
dataset = load_dataset("srehaag/hf-bug-temp")

### Expected behavior

Should load the 5 x 2 table from data1/train.parquet without error message.

### Environment info

Used Codespaces to simplify environment (see details below), but bug is present across various configurations.

- `datasets` version: 2.19.1
- Platform: Linux-6.5.0-1021-azure-x86_64-with-glibc2.31
- Python version: 3.10.13
- `huggingface_hub` version: 0.23.1
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.3.1


WinError 32 The process cannot access the file during load_dataset

### Describe the bug

When I try to load the opus_book from hugging face (following the [guide on the website](https://huggingface.co/docs/transformers/main/en/tasks/translation))

```python
from datasets import load_dataset, Dataset

dataset = load_dataset("Helsinki-NLP/opus_books", "en-fr", features=["id", "translation"])
```

I get an error:
`PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:/Users/Me/.cache/huggingface/datasets/Helsinki-NLP___parquet/ca-de-a39f1ef185b9b73b/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec.incomplete\\parquet-train-00000-00000-of-NNNNN.arrow'
`

Full stacktrace

```python

AttributeError                            Traceback (most recent call last)
File c:\Users\Me\.conda\envs\ia\lib\site-packages\datasets\builder.py:1858, in ArrowBasedBuilder._prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
   [1857](file:///C:/Users/Me/.conda/envs/ia/lib/site-packages/datasets/builder.py:1857) _time = time.time()
-> [1858](file:///C:/Users/Me/.conda/envs/ia/lib/site-packages/datasets/builder.py:1858) for _, table in generator:
   [1859](file:///C:/Users/Me/.conda/envs/ia/lib/site-packages/datasets/builder.py:1859)     if max_shard_size is not None and writer._num_bytes > max_shard_size:

File c:\Users\Me\.conda\envs\ia\lib\site-packages\datasets\packaged_modules\parquet\parquet.py:59, in Parquet._generate_tables(self, files)
     [58](file:///C:/Users/Me/.conda/envs/ia/lib/site-packages/datasets/packaged_modules/parquet/parquet.py:58) def _generate_tables(self, files):
---> [59](file:///C:/Users/Me/.conda/envs/ia/lib/site-packages/datasets/packaged_modules/parquet/parquet.py:59)     schema = self.config.features.arrow_schema if self.config.features is not None else None
     [60](file:///C:/Users/Me/.conda/envs/ia/lib/site-packages/datasets/packaged_modules/parquet/parquet.py:60)     if self.config.features is not None and self.config.columns is not None:

AttributeError: 'list' object has no attribute 'arrow_schema'

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
File c:\Users\Me\.conda\envs\ia\lib\site-packages\datasets\builder.py:1882, in ArrowBasedBuilder._prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
   [1881](file:///C:/Users/Me/.conda/envs/ia/lib/site-packages/datasets/builder.py:1881) num_shards = shard_id + 1
-> [1882](file:///C:/Users/Me/.conda/envs/ia/lib/site-packages/datasets/builder.py:1882) num_examples, num_bytes = writer.finalize()
   [1883](file:///C:/Users/Me/.conda/envs/ia/lib/site-packages/datasets/builder.py:1883) writer.close()

File c:\Users\Me\.conda\envs\ia\lib\site-packages\datasets\arrow_writer.py:584, in ArrowWriter.finalize(self, close_stream)
    [583](file:///C:/Users/Me/.conda/envs/ia/lib/site-packages/datasets/arrow_writer.py:583) # If schema is known, infer features even if no examples were written
--> [584](file:///C:/Users/Me/.conda/envs/ia/lib/site-packages/datasets/arrow_writer.py:584) if self.pa_writer is None and self.schema:
...
--> [627](file:///C:/Users/Me/.conda/envs/ia/lib/shutil.py:627)         os.unlink(fullname)
    [628](file:///C:/Users/Me/.conda/envs/ia/lib/shutil.py:628)     except OSError:
    [629](file:///C:/Users/Me/.conda/envs/ia/lib/shutil.py:629)         onerror(os.unlink, fullname, sys.exc_info())

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:/Users/Me/.cache/huggingface/datasets/Helsinki-NLP___parquet/ca-de-a39f1ef185b9b73b/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec.incomplete\\parquet-train-00000-00000-of-NNNNN.arrow'
```

### Steps to reproduce the bug

Steps to reproduce:

Just execute these lines

```python
from datasets import load_dataset, Dataset

dataset = load_dataset("Helsinki-NLP/opus_books", "en-fr", features=["id", "translation"])
```

### Expected behavior

I expect the dataset to be loaded without any errors.

### Environment info

| Package| Version|
|--------|--------|
| transformers| 4.37.2|
| python| 3.9.19|
| pytorch| 2.3.0|
| datasets|2.12.0 |
| arrow  |  1.2.3|

I am using Conda on Windows 11.


```push_to_hub()``` - Prevent Automatic Generation of Splits

### Describe the bug

I currently have a dataset which has not been splited. When pushing the dataset to my hugging face dataset repository, it is split into a testing and training set. How can I prevent the split from happening?

### Steps to reproduce the bug

1. Have a unsplit dataset

```python
Dataset({ features: ['input', 'output', 'Attack', '__index_level_0__'], num_rows: 944685 })
```

2. Push it to huggingface

```python
dataset.push_to_hub(dataset_name)
```

3. On the hugging face dataset repo, the dataset then appears to be splited:

![image](https://github.com/huggingface/datasets/assets/29337128/b4fbc141-42b0-4f49-98df-dd479648fe09)

4. Indeed, when loading the dataset from this repo, the dataset is split in two testing and training set.

```python
from datasets import load_dataset, Dataset

dataset = load_dataset("Jetlime/NF-CSE-CIC-IDS2018-v2", streaming=True)
dataset
```
output:

```
IterableDatasetDict({
    train: IterableDataset({
        features: ['input', 'output', 'Attack', '__index_level_0__'],
        n_shards: 2
    })
    test: IterableDataset({
        features: ['input', 'output', 'Attack', '__index_level_0__'],
        n_shards: 1
    })
```

### Expected behavior

The dataset shall not be splited, as not requested.

### Environment info

- `datasets` version: 2.19.1
- Platform: Linux-6.2.0-35-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.23.0
- PyArrow version: 15.0.2
- Pandas version: 2.2.2
- `fsspec` version: 2024.3.1


Validate config name and data_files in packaged modules

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6915). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I pushed a change that fixes 2.15 cache reloading (I fixed the packaged module hash), feel free to merge if this change is fine for you

Something weird happened in GitHub: I just merged this PR to main, See: https://github.com/huggingface/datasets/commit/5bbbf1b19766e31a6905f3e82bf3aa3f9f84a982

However this PR still appears as Open...

If I retry to merge this PR, an error appears: "Merge attempt failed: Merge already in progress"
![Screenshot from 2024-06-06 06-29-22](https://github.com/huggingface/datasets/assets/8515462/5fe87442-cc5d-4e9b-b60e-fdfbab830c81)

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005543 / 0.011353 (-0.005810) | 0.004059 / 0.011008 (-0.006949) | 0.064678 / 0.038508 (0.026170) | 0.032615 / 0.023109 (0.009506) | 0.245883 / 0.275898 (-0.030015) | 0.273545 / 0.323480 (-0.049935) | 0.004268 / 0.007986 (-0.003718) | 0.003160 / 0.004328 (-0.001168) | 0.051982 / 0.004250 (0.047731) | 0.051186 / 0.037052 (0.014134) | 0.254009 / 0.258489 (-0.004480) | 0.289594 / 0.293841 (-0.004247) | 0.028459 / 0.128546 (-0.100087) | 0.011061 / 0.075646 (-0.064585) | 0.203571 / 0.419271 (-0.215700) | 0.038049 / 0.043533 (-0.005484) | 0.243700 / 0.255139 (-0.011439) | 0.264816 / 0.283200 (-0.018383) | 0.019556 / 0.141683 (-0.122127) | 1.114395 / 1.452155 (-0.337759) | 1.168915 / 1.492716 (-0.323802) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.098814 / 0.018006 (0.080808) | 0.308218 / 0.000490 (0.307728) | 0.000221 / 0.000200 (0.000022) | 0.000047 / 0.000054 (-0.000008) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019660 / 0.037411 (-0.017752) | 0.070542 / 0.014526 (0.056017) | 0.078906 / 0.176557 (-0.097650) | 0.126658 / 0.737135 (-0.610477) | 0.080427 / 0.296338 (-0.215911) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.280686 / 0.215209 (0.065477) | 2.767480 / 2.077655 (0.689825) | 1.455325 / 1.504120 (-0.048795) | 1.336677 / 1.541195 (-0.204518) | 1.380359 / 1.468490 (-0.088131) | 0.576310 / 4.584777 (-4.008467) | 2.431829 / 3.745712 (-1.313883) | 2.815266 / 5.269862 (-2.454595) | 1.908962 / 4.565676 (-2.656714) | 0.065306 / 0.424275 (-0.358969) | 0.005229 / 0.007607 (-0.002378) | 0.336018 / 0.226044 (0.109973) | 3.349283 / 2.268929 (1.080355) | 1.814696 / 55.444624 (-53.629929) | 1.520969 / 6.876477 (-5.355508) | 1.735322 / 2.142072 (-0.406751) | 0.661513 / 4.805227 (-4.143714) | 0.121465 / 6.500664 (-6.379199) | 0.044505 / 0.075469 (-0.030964) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.989204 / 1.841788 (-0.852584) | 12.608414 / 8.074308 (4.534106) | 10.133358 / 10.191392 (-0.058034) | 0.133986 / 0.680424 (-0.546438) | 0.014332 / 0.534201 (-0.519869) | 0.293207 / 0.579283 (-0.286076) | 0.265657 / 0.434364 (-0.168707) | 0.325972 / 0.540337 (-0.214365) | 0.478103 / 1.386936 (-0.908833) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006070 / 0.011353 (-0.005283) | 0.004122 / 0.011008 (-0.006886) | 0.050572 / 0.038508 (0.012064) | 0.033732 / 0.023109 (0.010623) | 0.271282 / 0.275898 (-0.004616) | 0.296247 / 0.323480 (-0.027233) | 0.004400 / 0.007986 (-0.003585) | 0.002914 / 0.004328 (-0.001415) | 0.049332 / 0.004250 (0.045082) | 0.042213 / 0.037052 (0.005161) | 0.281230 / 0.258489 (0.022741) | 0.315514 / 0.293841 (0.021673) | 0.030864 / 0.128546 (-0.097682) | 0.011185 / 0.075646 (-0.064461) | 0.059227 / 0.419271 (-0.360044) | 0.034006 / 0.043533 (-0.009527) | 0.270059 / 0.255139 (0.014920) | 0.284014 / 0.283200 (0.000814) | 0.019502 / 0.141683 (-0.122181) | 1.143650 / 1.452155 (-0.308505) | 1.190968 / 1.492716 (-0.301749) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.100502 / 0.018006 (0.082496) | 0.307863 / 0.000490 (0.307373) | 0.000212 / 0.000200 (0.000012) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023442 / 0.037411 (-0.013969) | 0.080185 / 0.014526 (0.065659) | 0.089372 / 0.176557 (-0.087185) | 0.131030 / 0.737135 (-0.606105) | 0.091174 / 0.296338 (-0.205165) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.304187 / 0.215209 (0.088978) | 3.043055 / 2.077655 (0.965400) | 1.629578 / 1.504120 (0.125459) | 1.533762 / 1.541195 (-0.007432) | 1.546134 / 1.468490 (0.077643) | 0.577739 / 4.584777 (-4.007038) | 0.986310 / 3.745712 (-2.759402) | 2.791650 / 5.269862 (-2.478212) | 1.841190 / 4.565676 (-2.724487) | 0.064943 / 0.424275 (-0.359333) | 0.005251 / 0.007607 (-0.002356) | 0.355009 / 0.226044 (0.128965) | 3.560935 / 2.268929 (1.292007) | 1.991995 / 55.444624 (-53.452629) | 1.708796 / 6.876477 (-5.167681) | 1.917721 / 2.142072 (-0.224351) | 0.667667 / 4.805227 (-4.137561) | 0.119956 / 6.500664 (-6.380708) | 0.042069 / 0.075469 (-0.033400) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.006242 / 1.841788 (-0.835546) | 13.321644 / 8.074308 (5.247336) | 10.712409 / 10.191392 (0.521017) | 0.134036 / 0.680424 (-0.546388) | 0.017645 / 0.534201 (-0.516555) | 0.289077 / 0.579283 (-0.290206) | 0.131356 / 0.434364 (-0.303007) | 0.333062 / 0.540337 (-0.207275) | 0.425327 / 1.386936 (-0.961609) |

![](https://cml.dev/watermark.png#09ebf5190afbd017f3ca24ef444be2d933411eed "CML watermark")

Indeed, the merge commit is: https://github.com/huggingface/datasets/commit/5bbbf1b19766e31a6905f3e82bf3aa3f9f84a982

The following commit is just empty: https://github.com/huggingface/datasets/commit/09ebf5190afbd017f3ca24ef444be2d933411eed

Validate the config attributes `name` and `data_files` in packaged modules by making the derived classes call their parent `__post_init__` method.

Note that their parent `BuilderConfig` validates its attributes `name` and `data_files` in its `__post_init__` method: https://github.com/huggingface/datasets/blob/60d21efbc01e15d0b596ac1072750cbecd91548a/src/datasets/builder.py#L128-L137

This PR makes the derived config classes call their parent `__post_init__` method to validate their `name` and `data_files` attributes.


Preserve JSON column order and support list of strings field

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6914). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005492 / 0.011353 (-0.005861) | 0.004087 / 0.011008 (-0.006921) | 0.065334 / 0.038508 (0.026826) | 0.032282 / 0.023109 (0.009173) | 0.246441 / 0.275898 (-0.029457) | 0.278807 / 0.323480 (-0.044673) | 0.003245 / 0.007986 (-0.004741) | 0.003795 / 0.004328 (-0.000534) | 0.050082 / 0.004250 (0.045832) | 0.050613 / 0.037052 (0.013561) | 0.258885 / 0.258489 (0.000396) | 0.297257 / 0.293841 (0.003416) | 0.028847 / 0.128546 (-0.099699) | 0.011377 / 0.075646 (-0.064270) | 0.206089 / 0.419271 (-0.213182) | 0.037354 / 0.043533 (-0.006178) | 0.257319 / 0.255139 (0.002180) | 0.275134 / 0.283200 (-0.008066) | 0.018064 / 0.141683 (-0.123619) | 1.112371 / 1.452155 (-0.339783) | 1.160909 / 1.492716 (-0.331807) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.101893 / 0.018006 (0.083887) | 0.311084 / 0.000490 (0.310594) | 0.000208 / 0.000200 (0.000008) | 0.000042 / 0.000054 (-0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019548 / 0.037411 (-0.017863) | 0.064396 / 0.014526 (0.049870) | 0.074900 / 0.176557 (-0.101656) | 0.122750 / 0.737135 (-0.614385) | 0.076693 / 0.296338 (-0.219646) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.288609 / 0.215209 (0.073400) | 2.831354 / 2.077655 (0.753699) | 1.453961 / 1.504120 (-0.050159) | 1.327702 / 1.541195 (-0.213493) | 1.382140 / 1.468490 (-0.086351) | 0.568465 / 4.584777 (-4.016312) | 2.427199 / 3.745712 (-1.318513) | 2.810586 / 5.269862 (-2.459275) | 1.839227 / 4.565676 (-2.726449) | 0.063219 / 0.424275 (-0.361056) | 0.005111 / 0.007607 (-0.002496) | 0.341447 / 0.226044 (0.115403) | 3.357429 / 2.268929 (1.088501) | 1.806501 / 55.444624 (-53.638123) | 1.541696 / 6.876477 (-5.334781) | 1.755400 / 2.142072 (-0.386673) | 0.661442 / 4.805227 (-4.143785) | 0.120203 / 6.500664 (-6.380461) | 0.044429 / 0.075469 (-0.031040) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.987810 / 1.841788 (-0.853978) | 12.765467 / 8.074308 (4.691159) | 10.497788 / 10.191392 (0.306396) | 0.132723 / 0.680424 (-0.547701) | 0.014484 / 0.534201 (-0.519717) | 0.285763 / 0.579283 (-0.293520) | 0.264377 / 0.434364 (-0.169987) | 0.326971 / 0.540337 (-0.213367) | 0.429432 / 1.386936 (-0.957504) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005996 / 0.011353 (-0.005357) | 0.004092 / 0.011008 (-0.006916) | 0.051660 / 0.038508 (0.013152) | 0.036661 / 0.023109 (0.013552) | 0.271133 / 0.275898 (-0.004765) | 0.295728 / 0.323480 (-0.027752) | 0.004452 / 0.007986 (-0.003534) | 0.002915 / 0.004328 (-0.001413) | 0.050669 / 0.004250 (0.046418) | 0.044431 / 0.037052 (0.007378) | 0.284683 / 0.258489 (0.026194) | 0.318799 / 0.293841 (0.024958) | 0.031094 / 0.128546 (-0.097452) | 0.010810 / 0.075646 (-0.064836) | 0.059740 / 0.419271 (-0.359531) | 0.034912 / 0.043533 (-0.008621) | 0.268779 / 0.255139 (0.013640) | 0.291294 / 0.283200 (0.008095) | 0.019769 / 0.141683 (-0.121914) | 1.124833 / 1.452155 (-0.327322) | 1.168301 / 1.492716 (-0.324416) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.097080 / 0.018006 (0.079074) | 0.304636 / 0.000490 (0.304146) | 0.000232 / 0.000200 (0.000032) | 0.000060 / 0.000054 (0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023186 / 0.037411 (-0.014225) | 0.082232 / 0.014526 (0.067706) | 0.089427 / 0.176557 (-0.087130) | 0.132715 / 0.737135 (-0.604421) | 0.092820 / 0.296338 (-0.203518) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.300672 / 0.215209 (0.085463) | 2.969603 / 2.077655 (0.891948) | 1.577827 / 1.504120 (0.073707) | 1.440768 / 1.541195 (-0.100427) | 1.494526 / 1.468490 (0.026035) | 0.574599 / 4.584777 (-4.010178) | 0.963300 / 3.745712 (-2.782412) | 2.847854 / 5.269862 (-2.422008) | 1.841248 / 4.565676 (-2.724428) | 0.062321 / 0.424275 (-0.361954) | 0.005389 / 0.007607 (-0.002218) | 0.350853 / 0.226044 (0.124808) | 3.463514 / 2.268929 (1.194586) | 1.937661 / 55.444624 (-53.506964) | 1.665320 / 6.876477 (-5.211157) | 1.849028 / 2.142072 (-0.293044) | 0.655333 / 4.805227 (-4.149894) | 0.119062 / 6.500664 (-6.381602) | 0.043387 / 0.075469 (-0.032082) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.004118 / 1.841788 (-0.837670) | 13.350894 / 8.074308 (5.276585) | 11.179363 / 10.191392 (0.987971) | 0.135169 / 0.680424 (-0.545255) | 0.016298 / 0.534201 (-0.517903) | 0.288467 / 0.579283 (-0.290816) | 0.132712 / 0.434364 (-0.301651) | 0.325436 / 0.540337 (-0.214901) | 0.413406 / 1.386936 (-0.973530) |

![](https://cml.dev/watermark.png#670e1cf31606f397ae0f858b568b1b4ed50c1843 "CML watermark")

Preserve column order when loading from a JSON file with a list of dict (or with a field containing a list of dicts).

Additionally, support JSON file with a list of strings field.

Fix #6913.


Column order is nondeterministic when loading from JSON

Something isn't working

As reported by @meg-huggingface, the order of the JSON object keys is not preserved while loading a dataset from a JSON file with a list of objects.

For example, when loading a JSON files with a list of objects, each with the following ordered keys:
- [ID, Language, Topic],

the resulting dataset may have columns:
- [ID, Topic, Language], or
- [Topic, Language, ID], or
- [Topic, ID, Language],...

This issue is caused by the use of a Python set (which does not preserve the order):
https://github.com/huggingface/datasets/blob/60d21efbc01e15d0b596ac1072750cbecd91548a/src/datasets/packaged_modules/json/json.py#L168
introduced in
- #5772


Add MedImg for streaming

dataset request

Requesting to add a new dataset

@mariosasko, @lhoestq, @albertvillanova
Hello! Can anyone help? or can you guys suggest who can help with this?

Hi ! Feel free to download the dataset and create a `Dataset` object with it.

Then your'll be able to use `push_to_hub()` to upload the dataset to HF in Parquet format and make it streamable :)

> Hi ! Feel free to download the dataset and create a `Dataset` object with it.
>
> Then your'll be able to use `push_to_hub()` to upload the dataset to HF in Parquet format and make it streamable :)

The dataset is several TB in total, which I do not have the resources to handle.

### Feature request

Host the MedImg dataset (similar to Imagenet but for biomedical images).

### Motivation

There is a clear need for biomedical image foundation models and large scale biomedical datasets that are easily streamable. This would be an excellent tool for the biomedical community.

### Your contribution

MedImg can be found [here](https://www.cuilab.cn/medimg/#).


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6911). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005136 / 0.011353 (-0.006217) | 0.003136 / 0.011008 (-0.007872) | 0.063752 / 0.038508 (0.025244) | 0.031060 / 0.023109 (0.007950) | 0.249848 / 0.275898 (-0.026050) | 0.275918 / 0.323480 (-0.047561) | 0.004047 / 0.007986 (-0.003938) | 0.002696 / 0.004328 (-0.001632) | 0.049884 / 0.004250 (0.045634) | 0.044646 / 0.037052 (0.007593) | 0.264769 / 0.258489 (0.006280) | 0.299874 / 0.293841 (0.006033) | 0.027530 / 0.128546 (-0.101016) | 0.010026 / 0.075646 (-0.065620) | 0.204007 / 0.419271 (-0.215265) | 0.035982 / 0.043533 (-0.007550) | 0.253560 / 0.255139 (-0.001579) | 0.276206 / 0.283200 (-0.006993) | 0.017770 / 0.141683 (-0.123913) | 1.156008 / 1.452155 (-0.296146) | 1.197265 / 1.492716 (-0.295451) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092960 / 0.018006 (0.074954) | 0.302876 / 0.000490 (0.302386) | 0.000214 / 0.000200 (0.000014) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019060 / 0.037411 (-0.018351) | 0.062262 / 0.014526 (0.047737) | 0.073836 / 0.176557 (-0.102721) | 0.122327 / 0.737135 (-0.614809) | 0.076050 / 0.296338 (-0.220289) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282489 / 0.215209 (0.067280) | 2.745084 / 2.077655 (0.667429) | 1.453044 / 1.504120 (-0.051076) | 1.339065 / 1.541195 (-0.202130) | 1.341395 / 1.468490 (-0.127095) | 0.586497 / 4.584777 (-3.998280) | 2.342198 / 3.745712 (-1.403514) | 2.684984 / 5.269862 (-2.584878) | 1.703738 / 4.565676 (-2.861939) | 0.062489 / 0.424275 (-0.361786) | 0.004906 / 0.007607 (-0.002701) | 0.332325 / 0.226044 (0.106280) | 3.255381 / 2.268929 (0.986452) | 1.797045 / 55.444624 (-53.647579) | 1.515197 / 6.876477 (-5.361280) | 1.508317 / 2.142072 (-0.633756) | 0.635973 / 4.805227 (-4.169254) | 0.117292 / 6.500664 (-6.383372) | 0.041456 / 0.075469 (-0.034013) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.973934 / 1.841788 (-0.867853) | 11.288665 / 8.074308 (3.214356) | 9.269404 / 10.191392 (-0.921988) | 0.143190 / 0.680424 (-0.537234) | 0.014366 / 0.534201 (-0.519835) | 0.285936 / 0.579283 (-0.293347) | 0.261632 / 0.434364 (-0.172732) | 0.327191 / 0.540337 (-0.213146) | 0.418900 / 1.386936 (-0.968036) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005131 / 0.011353 (-0.006222) | 0.003181 / 0.011008 (-0.007827) | 0.049697 / 0.038508 (0.011189) | 0.032754 / 0.023109 (0.009645) | 0.263954 / 0.275898 (-0.011944) | 0.285110 / 0.323480 (-0.038370) | 0.004133 / 0.007986 (-0.003852) | 0.002713 / 0.004328 (-0.001615) | 0.051684 / 0.004250 (0.047433) | 0.040607 / 0.037052 (0.003554) | 0.277919 / 0.258489 (0.019429) | 0.304773 / 0.293841 (0.010932) | 0.029530 / 0.128546 (-0.099016) | 0.010176 / 0.075646 (-0.065470) | 0.058501 / 0.419271 (-0.360771) | 0.033436 / 0.043533 (-0.010097) | 0.269899 / 0.255139 (0.014760) | 0.284490 / 0.283200 (0.001290) | 0.017092 / 0.141683 (-0.124591) | 1.132399 / 1.452155 (-0.319756) | 1.167290 / 1.492716 (-0.325427) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094460 / 0.018006 (0.076454) | 0.301462 / 0.000490 (0.300972) | 0.000202 / 0.000200 (0.000002) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022767 / 0.037411 (-0.014645) | 0.075993 / 0.014526 (0.061467) | 0.087729 / 0.176557 (-0.088827) | 0.127599 / 0.737135 (-0.609536) | 0.088873 / 0.296338 (-0.207465) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.286420 / 0.215209 (0.071211) | 2.811376 / 2.077655 (0.733722) | 1.558645 / 1.504120 (0.054525) | 1.426371 / 1.541195 (-0.114824) | 1.422347 / 1.468490 (-0.046143) | 0.567181 / 4.584777 (-4.017596) | 0.936731 / 3.745712 (-2.808982) | 2.643566 / 5.269862 (-2.626296) | 1.727843 / 4.565676 (-2.837834) | 0.062748 / 0.424275 (-0.361527) | 0.005033 / 0.007607 (-0.002574) | 0.339708 / 0.226044 (0.113663) | 3.354119 / 2.268929 (1.085190) | 1.877594 / 55.444624 (-53.567030) | 1.589202 / 6.876477 (-5.287274) | 1.707780 / 2.142072 (-0.434292) | 0.644520 / 4.805227 (-4.160708) | 0.115226 / 6.500664 (-6.385438) | 0.040004 / 0.075469 (-0.035465) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.002774 / 1.841788 (-0.839014) | 11.812647 / 8.074308 (3.738339) | 10.384198 / 10.191392 (0.192806) | 0.131120 / 0.680424 (-0.549304) | 0.014862 / 0.534201 (-0.519339) | 0.282873 / 0.579283 (-0.296410) | 0.120415 / 0.434364 (-0.313949) | 0.321995 / 0.540337 (-0.218343) | 0.441987 / 1.386936 (-0.944949) |

![](https://cml.dev/watermark.png#b12a2c5016499cc1d110798c6815f0245f61010e "CML watermark")

Remove dead code for non-dict data_files from packaged modules.

Since the merge of this PR:
- #2986

the builders' variable self.config.data_files is always a dict, which makes the condition on (str, list, tuple) dead code.