Allow concatenation of datasets with mixed structs

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6587). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005403 / 0.011353 (-0.005950) | 0.003807 / 0.011008 (-0.007201) | 0.063850 / 0.038508 (0.025342) | 0.028242 / 0.023109 (0.005132) | 0.242866 / 0.275898 (-0.033032) | 0.266015 / 0.323480 (-0.057464) | 0.004111 / 0.007986 (-0.003875) | 0.002816 / 0.004328 (-0.001513) | 0.048862 / 0.004250 (0.044611) | 0.043036 / 0.037052 (0.005984) | 0.255149 / 0.258489 (-0.003340) | 0.280105 / 0.293841 (-0.013736) | 0.028182 / 0.128546 (-0.100365) | 0.010997 / 0.075646 (-0.064649) | 0.208131 / 0.419271 (-0.211141) | 0.036030 / 0.043533 (-0.007502) | 0.241551 / 0.255139 (-0.013588) | 0.260741 / 0.283200 (-0.022459) | 0.018045 / 0.141683 (-0.123638) | 1.175308 / 1.452155 (-0.276847) | 1.192160 / 1.492716 (-0.300556) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094579 / 0.018006 (0.076573) | 0.309850 / 0.000490 (0.309360) | 0.000232 / 0.000200 (0.000032) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019519 / 0.037411 (-0.017892) | 0.062201 / 0.014526 (0.047675) | 0.074017 / 0.176557 (-0.102539) | 0.121987 / 0.737135 (-0.615148) | 0.078958 / 0.296338 (-0.217380) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.286306 / 0.215209 (0.071097) | 2.777004 / 2.077655 (0.699350) | 1.481445 / 1.504120 (-0.022675) | 1.348643 / 1.541195 (-0.192552) | 1.382257 / 1.468490 (-0.086234) | 0.571436 / 4.584777 (-4.013341) | 2.373279 / 3.745712 (-1.372433) | 2.749366 / 5.269862 (-2.520496) | 1.724937 / 4.565676 (-2.840739) | 0.062233 / 0.424275 (-0.362042) | 0.005013 / 0.007607 (-0.002594) | 0.339623 / 0.226044 (0.113579) | 3.385770 / 2.268929 (1.116842) | 1.832023 / 55.444624 (-53.612601) | 1.556172 / 6.876477 (-5.320305) | 1.573301 / 2.142072 (-0.568772) | 0.648866 / 4.805227 (-4.156361) | 0.121228 / 6.500664 (-6.379436) | 0.041684 / 0.075469 (-0.033786) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.974595 / 1.841788 (-0.867192) | 11.519692 / 8.074308 (3.445383) | 9.773075 / 10.191392 (-0.418317) | 0.138149 / 0.680424 (-0.542274) | 0.014068 / 0.534201 (-0.520133) | 0.288161 / 0.579283 (-0.291122) | 0.272832 / 0.434364 (-0.161532) | 0.324476 / 0.540337 (-0.215862) | 0.419962 / 1.386936 (-0.966974) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005668 / 0.011353 (-0.005685) | 0.003637 / 0.011008 (-0.007371) | 0.049582 / 0.038508 (0.011074) | 0.030982 / 0.023109 (0.007872) | 0.273036 / 0.275898 (-0.002862) | 0.297562 / 0.323480 (-0.025918) | 0.004382 / 0.007986 (-0.003603) | 0.002763 / 0.004328 (-0.001566) | 0.050807 / 0.004250 (0.046556) | 0.046914 / 0.037052 (0.009862) | 0.287443 / 0.258489 (0.028954) | 0.319694 / 0.293841 (0.025853) | 0.051110 / 0.128546 (-0.077436) | 0.010650 / 0.075646 (-0.064997) | 0.058254 / 0.419271 (-0.361018) | 0.033419 / 0.043533 (-0.010114) | 0.275634 / 0.255139 (0.020495) | 0.288618 / 0.283200 (0.005419) | 0.018004 / 0.141683 (-0.123678) | 1.134166 / 1.452155 (-0.317989) | 1.192533 / 1.492716 (-0.300183) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.098573 / 0.018006 (0.080566) | 0.308152 / 0.000490 (0.307662) | 0.000249 / 0.000200 (0.000049) | 0.000045 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022443 / 0.037411 (-0.014968) | 0.075628 / 0.014526 (0.061103) | 0.088807 / 0.176557 (-0.087750) | 0.127519 / 0.737135 (-0.609617) | 0.090156 / 0.296338 (-0.206182) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.294493 / 0.215209 (0.079284) | 2.862084 / 2.077655 (0.784429) | 1.585962 / 1.504120 (0.081842) | 1.466366 / 1.541195 (-0.074829) | 1.503306 / 1.468490 (0.034816) | 0.581524 / 4.584777 (-4.003253) | 2.475593 / 3.745712 (-1.270120) | 2.852014 / 5.269862 (-2.417847) | 1.834047 / 4.565676 (-2.731630) | 0.064009 / 0.424275 (-0.360266) | 0.005094 / 0.007607 (-0.002514) | 0.355960 / 0.226044 (0.129916) | 3.428849 / 2.268929 (1.159920) | 1.958501 / 55.444624 (-53.486124) | 1.675448 / 6.876477 (-5.201029) | 1.719960 / 2.142072 (-0.422113) | 0.659609 / 4.805227 (-4.145618) | 0.119036 / 6.500664 (-6.381628) | 0.041800 / 0.075469 (-0.033669) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.025955 / 1.841788 (-0.815833) | 12.432417 / 8.074308 (4.358108) | 10.444854 / 10.191392 (0.253462) | 0.130106 / 0.680424 (-0.550318) | 0.015655 / 0.534201 (-0.518546) | 0.288184 / 0.579283 (-0.291099) | 0.285023 / 0.434364 (-0.149340) | 0.329244 / 0.540337 (-0.211093) | 0.415484 / 1.386936 (-0.971452) |

![](https://cml.dev/watermark.png#b262b060525efd973cac3f2073ba3944f3ddd7e3 "CML watermark")

Fixes #6466

The idea is to do a recursive check for structs. PyArrow handles it well enough.

For a demo you can do:

```python
from datasets import Dataset, concatenate_datasets

ds = Dataset.from_dict({'speaker': [{'name': 'Ben', 'email': None}]})
ds2 = Dataset.from_dict({'speaker': [{'name': 'Fred', 'email': 'abc@aol.com'}]})
print(concatenate_datasets([ds, ds2]).features)
print(concatenate_datasets([ds, ds2]).to_dict())
```

Now both the features and the rows are fixed.

I note that Sequence suffers from the same problem, so I can fix that in a future PR once this one is merged.


keep more info in DatasetInfo.from_merge #6585

> @JochenSiegWork fyi, that seems to also affect the `trainer.push_to_hub()` method, which I guess also needs to parse that DatasetInfo from the `kwargs` used by `push_to_hub`. There is short discussion about it [here](https://github.com/huggingface/blog/issues/1623). Would be great if you can check if your PR would also fix that!

Hi @thiagobarbosa, it might be related but I didn't worked with `push_to_hub` yet. I don't see a minimal example reproducing the specific error in your link. However, if you have a running version producing the error locally you can test it by pulling this PR and run your specific example locally.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6586). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004729 / 0.011353 (-0.006624) | 0.002983 / 0.011008 (-0.008025) | 0.062482 / 0.038508 (0.023974) | 0.028406 / 0.023109 (0.005297) | 0.255896 / 0.275898 (-0.020002) | 0.276423 / 0.323480 (-0.047057) | 0.003828 / 0.007986 (-0.004157) | 0.002601 / 0.004328 (-0.001728) | 0.048954 / 0.004250 (0.044704) | 0.040661 / 0.037052 (0.003609) | 0.277710 / 0.258489 (0.019221) | 0.290360 / 0.293841 (-0.003481) | 0.027105 / 0.128546 (-0.101441) | 0.010168 / 0.075646 (-0.065478) | 0.206835 / 0.419271 (-0.212436) | 0.035226 / 0.043533 (-0.008306) | 0.262567 / 0.255139 (0.007428) | 0.273979 / 0.283200 (-0.009221) | 0.017576 / 0.141683 (-0.124106) | 1.125588 / 1.452155 (-0.326566) | 1.185018 / 1.492716 (-0.307698) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092192 / 0.018006 (0.074186) | 0.298350 / 0.000490 (0.297861) | 0.000217 / 0.000200 (0.000017) | 0.000051 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.017925 / 0.037411 (-0.019486) | 0.060285 / 0.014526 (0.045759) | 0.076579 / 0.176557 (-0.099978) | 0.118830 / 0.737135 (-0.618305) | 0.073017 / 0.296338 (-0.223322) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.288149 / 0.215209 (0.072940) | 2.840004 / 2.077655 (0.762349) | 1.495758 / 1.504120 (-0.008361) | 1.362338 / 1.541195 (-0.178857) | 1.389746 / 1.468490 (-0.078744) | 0.576891 / 4.584777 (-4.007886) | 2.375724 / 3.745712 (-1.369988) | 2.707405 / 5.269862 (-2.562457) | 1.719850 / 4.565676 (-2.845826) | 0.067055 / 0.424275 (-0.357220) | 0.005039 / 0.007607 (-0.002568) | 0.346626 / 0.226044 (0.120581) | 3.468346 / 2.268929 (1.199418) | 1.860686 / 55.444624 (-53.583938) | 1.582929 / 6.876477 (-5.293548) | 1.613131 / 2.142072 (-0.528941) | 0.659022 / 4.805227 (-4.146206) | 0.118477 / 6.500664 (-6.382187) | 0.041614 / 0.075469 (-0.033855) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.005062 / 1.841788 (-0.836726) | 11.203210 / 8.074308 (3.128902) | 10.320764 / 10.191392 (0.129372) | 0.128541 / 0.680424 (-0.551883) | 0.014646 / 0.534201 (-0.519555) | 0.285280 / 0.579283 (-0.294003) | 0.263613 / 0.434364 (-0.170751) | 0.321161 / 0.540337 (-0.219177) | 0.420565 / 1.386936 (-0.966371) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005288 / 0.011353 (-0.006065) | 0.003048 / 0.011008 (-0.007960) | 0.049196 / 0.038508 (0.010688) | 0.032104 / 0.023109 (0.008994) | 0.279345 / 0.275898 (0.003447) | 0.300194 / 0.323480 (-0.023286) | 0.004045 / 0.007986 (-0.003941) | 0.002594 / 0.004328 (-0.001735) | 0.047680 / 0.004250 (0.043430) | 0.044294 / 0.037052 (0.007241) | 0.292330 / 0.258489 (0.033841) | 0.318610 / 0.293841 (0.024769) | 0.050417 / 0.128546 (-0.078129) | 0.010326 / 0.075646 (-0.065320) | 0.057372 / 0.419271 (-0.361899) | 0.032985 / 0.043533 (-0.010548) | 0.277717 / 0.255139 (0.022579) | 0.295692 / 0.283200 (0.012493) | 0.017756 / 0.141683 (-0.123927) | 1.166277 / 1.452155 (-0.285877) | 1.213337 / 1.492716 (-0.279380) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091365 / 0.018006 (0.073359) | 0.296261 / 0.000490 (0.295772) | 0.000225 / 0.000200 (0.000025) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021973 / 0.037411 (-0.015438) | 0.074631 / 0.014526 (0.060106) | 0.085645 / 0.176557 (-0.090911) | 0.125181 / 0.737135 (-0.611955) | 0.086893 / 0.296338 (-0.209445) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.294110 / 0.215209 (0.078901) | 2.855531 / 2.077655 (0.777876) | 1.583204 / 1.504120 (0.079084) | 1.453911 / 1.541195 (-0.087284) | 1.467031 / 1.468490 (-0.001460) | 0.581214 / 4.584777 (-4.003562) | 2.423626 / 3.745712 (-1.322086) | 2.736665 / 5.269862 (-2.533197) | 1.707000 / 4.565676 (-2.858676) | 0.061171 / 0.424275 (-0.363104) | 0.004789 / 0.007607 (-0.002818) | 0.344546 / 0.226044 (0.118502) | 3.530955 / 2.268929 (1.262027) | 1.962532 / 55.444624 (-53.482092) | 1.670207 / 6.876477 (-5.206270) | 1.669041 / 2.142072 (-0.473031) | 0.642298 / 4.805227 (-4.162929) | 0.115503 / 6.500664 (-6.385161) | 0.040729 / 0.075469 (-0.034740) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.973101 / 1.841788 (-0.868687) | 11.823894 / 8.074308 (3.749586) | 10.664592 / 10.191392 (0.473200) | 0.139848 / 0.680424 (-0.540576) | 0.015728 / 0.534201 (-0.518473) | 0.289135 / 0.579283 (-0.290148) | 0.271325 / 0.434364 (-0.163039) | 0.332253 / 0.540337 (-0.208085) | 0.416982 / 1.386936 (-0.969954) |

![](https://cml.dev/watermark.png#ca76ca1152fce82bfeaab9f9a33849d4d7f9dd63 "CML watermark")

* try not to merge DatasetInfos if they're equal

* fixes losing DatasetInfo during parallel Dataset.map


losing DatasetInfo in Dataset.map when num_proc > 1

Hi ! This issue comes from the fact that `map()` with `num_proc>1` shards the dataset in multiple chunks to be processed (one per process) and merges them. The DatasetInfos of each chunk are then merged together, but for some fields like `dataset_name` it's not been implemented and default to None.

The DatasetInfo merge is defined here, in case you'd like to contribute an improvement:

https://github.com/huggingface/datasets/blob/d2e0034122a788015c0834a72e6c6279e7ecbac5/src/datasets/info.py#L269-L270

### Describe the bug

Hello and thanks for developing this package!

When I process a Dataset with the map function using multiple processors some set attributes of the DatasetInfo get lost and are None in the resulting Dataset.

### Steps to reproduce the bug

```python
from datasets import Dataset, DatasetInfo

def run_map(num_proc):
    dataset = Dataset.from_dict(
        {"col1": [0, 1], "col2": [3, 4]},
        info=DatasetInfo(
            dataset_name="my_dataset",
        ),
    )

    ds = dataset.map(lambda x: x, num_proc=num_proc)
    print(ds.info.dataset_name)

run_map(1)
run_map(2)
```

This puts out:

```bash
Map: 100%|██████████| 2/2 [00:00<00:00, 724.66 examples/s]
my_dataset
Map (num_proc=2): 100%|██████████| 2/2 [00:00<00:00, 18.25 examples/s]
None
```

### Expected behavior

I expect the DatasetInfo to be kept as it was and there should be no difference in the output of running map with num_proc=1 and num_proc=2.

Expected output:
```bash
Map: 100%|██████████| 2/2 [00:00<00:00, 724.66 examples/s]
my_dataset
Map (num_proc=2): 100%|██████████| 2/2 [00:00<00:00, 18.25 examples/s]
my_dataset
```

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.17
- Python version: 3.8.18
- `huggingface_hub` version: 0.20.2
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
- `fsspec` version: 2023.9.2


np.fromfile not supported

@lhoestq
Can you provide me with some ideas？

@lhoestq
```
Traceback (most recent call last):
  File "/home/dongzf/miniconda3/envs/dataset_ai/lib/python3.11/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dongzf/miniconda3/envs/dataset_ai/lib/python3.11/runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "/home/dongzf/.vscode/extensions/ms-python.python-2023.22.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
    cli.main()
  File "/home/dongzf/.vscode/extensions/ms-python.python-2023.22.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "/home/dongzf/.vscode/extensions/ms-python.python-2023.22.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
    runpy.run_path(target, run_name="__main__")
  File "/home/dongzf/.vscode/extensions/ms-python.python-2023.22.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
    return _run_module_code(code, init_globals, run_name,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dongzf/.vscode/extensions/ms-python.python-2023.22.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/home/dongzf/.vscode/extensions/ms-python.python-2023.22.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "/mnt/sda/code/dataset_ai/dataset_ai/example/test.py", line 83, in <module>
    data = xnumpy_fromfile(current_dir, download_config=config,dtype=numpy.float32,)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/sda/code/dataset_ai/dataset_ai/src/datasets/download/streaming_download_manager.py", line 765, in xnumpy_fromfile
    return np.fromfile(xopen(filepath_or_buffer, "rb", download_config=download_config).read(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: embedded null byte
```

    not add read()
the error is

return np.fromfile(xopen(filepath_or_buffer, "rb", download_config=download_config), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
io.UnsupportedOperation: fileno

xopen return obj do not have fileno function
I don't know why?

I used this method to read point cloud data in the script

```python
with open(velodyne_filepath,"rb") as obj:
    velodyne_data = numpy.frombuffer(obj.read(), dtype=numpy.float32).reshape([-1, 4])
```

How to do np.fromfile  to use it like np.load

```python
def xnumpy_fromfile(filepath_or_buffer, *args, download_config: Optional[DownloadConfig] = None, **kwargs):
    import numpy as np

    if hasattr(filepath_or_buffer, "read"):
        return np.fromfile(filepath_or_buffer, *args, **kwargs)
    else:
        filepath_or_buffer = str(filepath_or_buffer)
        return np.fromfile(xopen(filepath_or_buffer, "rb", download_config=download_config).read(), *args, **kwargs)
```
this is not work


remove eli5 test

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6583). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005024 / 0.011353 (-0.006329) | 0.003172 / 0.011008 (-0.007836) | 0.062934 / 0.038508 (0.024426) | 0.031737 / 0.023109 (0.008628) | 0.249251 / 0.275898 (-0.026647) | 0.273084 / 0.323480 (-0.050396) | 0.002958 / 0.007986 (-0.005027) | 0.002726 / 0.004328 (-0.001603) | 0.048519 / 0.004250 (0.044269) | 0.043608 / 0.037052 (0.006556) | 0.253648 / 0.258489 (-0.004841) | 0.280095 / 0.293841 (-0.013746) | 0.027500 / 0.128546 (-0.101046) | 0.010545 / 0.075646 (-0.065101) | 0.206781 / 0.419271 (-0.212490) | 0.035515 / 0.043533 (-0.008018) | 0.259449 / 0.255139 (0.004310) | 0.271488 / 0.283200 (-0.011712) | 0.019352 / 0.141683 (-0.122331) | 1.152002 / 1.452155 (-0.300153) | 1.190325 / 1.492716 (-0.302391) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093253 / 0.018006 (0.075247) | 0.302182 / 0.000490 (0.301692) | 0.000216 / 0.000200 (0.000016) | 0.000047 / 0.000054 (-0.000008) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.017889 / 0.037411 (-0.019523) | 0.060292 / 0.014526 (0.045766) | 0.072640 / 0.176557 (-0.103917) | 0.121320 / 0.737135 (-0.615815) | 0.073866 / 0.296338 (-0.222472) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282910 / 0.215209 (0.067701) | 2.779815 / 2.077655 (0.702160) | 1.537929 / 1.504120 (0.033809) | 1.405990 / 1.541195 (-0.135205) | 1.407911 / 1.468490 (-0.060579) | 0.561551 / 4.584777 (-4.023226) | 2.368053 / 3.745712 (-1.377659) | 2.732608 / 5.269862 (-2.537254) | 1.710274 / 4.565676 (-2.855402) | 0.061925 / 0.424275 (-0.362350) | 0.004975 / 0.007607 (-0.002632) | 0.338843 / 0.226044 (0.112799) | 3.328579 / 2.268929 (1.059650) | 1.865994 / 55.444624 (-53.578631) | 1.603145 / 6.876477 (-5.273332) | 1.615440 / 2.142072 (-0.526633) | 0.635646 / 4.805227 (-4.169581) | 0.116185 / 6.500664 (-6.384479) | 0.041964 / 0.075469 (-0.033505) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.956977 / 1.841788 (-0.884811) | 11.539802 / 8.074308 (3.465494) | 10.048855 / 10.191392 (-0.142537) | 0.128758 / 0.680424 (-0.551666) | 0.013491 / 0.534201 (-0.520710) | 0.287330 / 0.579283 (-0.291953) | 0.262416 / 0.434364 (-0.171947) | 0.327327 / 0.540337 (-0.213011) | 0.418423 / 1.386936 (-0.968513) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004963 / 0.011353 (-0.006390) | 0.003335 / 0.011008 (-0.007673) | 0.052082 / 0.038508 (0.013574) | 0.029302 / 0.023109 (0.006192) | 0.284986 / 0.275898 (0.009088) | 0.304082 / 0.323480 (-0.019398) | 0.004065 / 0.007986 (-0.003921) | 0.002643 / 0.004328 (-0.001685) | 0.049504 / 0.004250 (0.045253) | 0.044514 / 0.037052 (0.007461) | 0.287064 / 0.258489 (0.028575) | 0.312921 / 0.293841 (0.019080) | 0.029195 / 0.128546 (-0.099351) | 0.010471 / 0.075646 (-0.065175) | 0.057620 / 0.419271 (-0.361651) | 0.050221 / 0.043533 (0.006689) | 0.285392 / 0.255139 (0.030253) | 0.302111 / 0.283200 (0.018912) | 0.018690 / 0.141683 (-0.122993) | 1.165637 / 1.452155 (-0.286518) | 1.203757 / 1.492716 (-0.288959) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095035 / 0.018006 (0.077028) | 0.304447 / 0.000490 (0.303957) | 0.000231 / 0.000200 (0.000031) | 0.000051 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022345 / 0.037411 (-0.015066) | 0.077195 / 0.014526 (0.062669) | 0.089564 / 0.176557 (-0.086992) | 0.129248 / 0.737135 (-0.607887) | 0.091974 / 0.296338 (-0.204365) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.300641 / 0.215209 (0.085432) | 2.936669 / 2.077655 (0.859014) | 1.649100 / 1.504120 (0.144980) | 1.510693 / 1.541195 (-0.030502) | 1.517011 / 1.468490 (0.048521) | 0.572511 / 4.584777 (-4.012266) | 2.442704 / 3.745712 (-1.303009) | 2.833089 / 5.269862 (-2.436772) | 1.762668 / 4.565676 (-2.803008) | 0.063754 / 0.424275 (-0.360521) | 0.005034 / 0.007607 (-0.002573) | 0.401631 / 0.226044 (0.175586) | 3.418986 / 2.268929 (1.150057) | 1.989639 / 55.444624 (-53.454986) | 1.695776 / 6.876477 (-5.180701) | 1.712822 / 2.142072 (-0.429250) | 0.654029 / 4.805227 (-4.151198) | 0.117624 / 6.500664 (-6.383040) | 0.041058 / 0.075469 (-0.034411) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.986008 / 1.841788 (-0.855779) | 12.146838 / 8.074308 (4.072530) | 11.105900 / 10.191392 (0.914508) | 0.139938 / 0.680424 (-0.540486) | 0.015117 / 0.534201 (-0.519084) | 0.286151 / 0.579283 (-0.293132) | 0.272960 / 0.434364 (-0.161404) | 0.323370 / 0.540337 (-0.216967) | 0.427379 / 1.386936 (-0.959557) |

![](https://cml.dev/watermark.png#91888ea888fec1f2c96d8316a569439e64eb508e "CML watermark")

since the dataset is defunct


Fix for Incorrect ex_iterable used with multi num_worker

A toy example to reveal the bug.

```python
"""
DATASETS_VERBOSITY=debug torchrun --nproc-per-node 2 main.py
"""
import torch.utils.data
import torch.distributed
import datasets.distributed
import datasets

# num shards = 4
shards = [(0, 100), (100, 200), (200, 300), (300, 400)]

def gen(shards):
    for st, ed in shards:
        yield from range(st, ed)

torch.distributed.init_process_group()

# want to create total worker = world_size * 8
ds = datasets.IterableDataset.from_generator(gen, gen_kwargs={'shards': shards})
ds = datasets.distributed.split_dataset_by_node(
    ds,
    rank=torch.distributed.get_rank(),
    world_size=torch.distributed.get_world_size(),
)
dl = torch.utils.data.DataLoader(ds, batch_size=10, num_workers=8)

for x in dl:
    print(f"RANK={torch.distributed.get_rank()} {x}")
```

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005401 / 0.011353 (-0.005952) | 0.004023 / 0.011008 (-0.006985) | 0.064601 / 0.038508 (0.026093) | 0.028567 / 0.023109 (0.005457) | 0.245476 / 0.275898 (-0.030422) | 0.292727 / 0.323480 (-0.030752) | 0.003080 / 0.007986 (-0.004905) | 0.002779 / 0.004328 (-0.001549) | 0.050046 / 0.004250 (0.045796) | 0.043906 / 0.037052 (0.006854) | 0.273896 / 0.258489 (0.015407) | 0.308430 / 0.293841 (0.014589) | 0.028442 / 0.128546 (-0.100104) | 0.010694 / 0.075646 (-0.064953) | 0.209048 / 0.419271 (-0.210223) | 0.036062 / 0.043533 (-0.007471) | 0.242689 / 0.255139 (-0.012450) | 0.261695 / 0.283200 (-0.021504) | 0.018519 / 0.141683 (-0.123163) | 1.122735 / 1.452155 (-0.329420) | 1.172680 / 1.492716 (-0.320036) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093827 / 0.018006 (0.075820) | 0.302650 / 0.000490 (0.302161) | 0.000218 / 0.000200 (0.000018) | 0.000045 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018778 / 0.037411 (-0.018633) | 0.067516 / 0.014526 (0.052990) | 0.079693 / 0.176557 (-0.096864) | 0.125907 / 0.737135 (-0.611228) | 0.081771 / 0.296338 (-0.214568) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.281809 / 0.215209 (0.066600) | 2.773937 / 2.077655 (0.696283) | 1.443622 / 1.504120 (-0.060497) | 1.334359 / 1.541195 (-0.206836) | 1.364813 / 1.468490 (-0.103677) | 0.561670 / 4.584777 (-4.023107) | 2.338292 / 3.745712 (-1.407420) | 2.807595 / 5.269862 (-2.462267) | 1.734162 / 4.565676 (-2.831514) | 0.063681 / 0.424275 (-0.360594) | 0.004934 / 0.007607 (-0.002673) | 0.336781 / 0.226044 (0.110737) | 3.311744 / 2.268929 (1.042815) | 1.826802 / 55.444624 (-53.617822) | 1.579604 / 6.876477 (-5.296872) | 1.620526 / 2.142072 (-0.521546) | 0.647061 / 4.805227 (-4.158166) | 0.117729 / 6.500664 (-6.382935) | 0.042216 / 0.075469 (-0.033253) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.994289 / 1.841788 (-0.847499) | 12.266185 / 8.074308 (4.191877) | 9.634035 / 10.191392 (-0.557357) | 0.144521 / 0.680424 (-0.535902) | 0.013787 / 0.534201 (-0.520414) | 0.288353 / 0.579283 (-0.290930) | 0.262183 / 0.434364 (-0.172181) | 0.336960 / 0.540337 (-0.203378) | 0.441142 / 1.386936 (-0.945794) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005678 / 0.011353 (-0.005675) | 0.004011 / 0.011008 (-0.006998) | 0.049319 / 0.038508 (0.010811) | 0.032543 / 0.023109 (0.009434) | 0.276389 / 0.275898 (0.000491) | 0.298495 / 0.323480 (-0.024985) | 0.004192 / 0.007986 (-0.003794) | 0.002765 / 0.004328 (-0.001563) | 0.048739 / 0.004250 (0.044489) | 0.046212 / 0.037052 (0.009160) | 0.286614 / 0.258489 (0.028125) | 0.315949 / 0.293841 (0.022108) | 0.029833 / 0.128546 (-0.098714) | 0.010762 / 0.075646 (-0.064884) | 0.058489 / 0.419271 (-0.360783) | 0.052258 / 0.043533 (0.008725) | 0.275873 / 0.255139 (0.020734) | 0.288668 / 0.283200 (0.005468) | 0.018828 / 0.141683 (-0.122855) | 1.140196 / 1.452155 (-0.311959) | 1.229500 / 1.492716 (-0.263217) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094161 / 0.018006 (0.076155) | 0.303519 / 0.000490 (0.303030) | 0.000219 / 0.000200 (0.000019) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022088 / 0.037411 (-0.015324) | 0.076376 / 0.014526 (0.061850) | 0.088705 / 0.176557 (-0.087851) | 0.127602 / 0.737135 (-0.609533) | 0.088689 / 0.296338 (-0.207649) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.292363 / 0.215209 (0.077154) | 2.859215 / 2.077655 (0.781561) | 1.566389 / 1.504120 (0.062270) | 1.439195 / 1.541195 (-0.102000) | 1.463805 / 1.468490 (-0.004685) | 0.551660 / 4.584777 (-4.033116) | 2.427462 / 3.745712 (-1.318250) | 2.712372 / 5.269862 (-2.557490) | 1.811331 / 4.565676 (-2.754346) | 0.061539 / 0.424275 (-0.362736) | 0.005062 / 0.007607 (-0.002545) | 0.341984 / 0.226044 (0.115940) | 3.352171 / 2.268929 (1.083242) | 1.917550 / 55.444624 (-53.527074) | 1.642668 / 6.876477 (-5.233809) | 1.817204 / 2.142072 (-0.324868) | 0.630849 / 4.805227 (-4.174379) | 0.115788 / 6.500664 (-6.384876) | 0.041041 / 0.075469 (-0.034428) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.017725 / 1.841788 (-0.824062) | 12.976994 / 8.074308 (4.902686) | 10.307414 / 10.191392 (0.116022) | 0.141090 / 0.680424 (-0.539334) | 0.015548 / 0.534201 (-0.518653) | 0.288184 / 0.579283 (-0.291099) | 0.276409 / 0.434364 (-0.157955) | 0.328289 / 0.540337 (-0.212048) | 0.429138 / 1.386936 (-0.957798) |

![](https://cml.dev/watermark.png#31ae21ff806c3d1fc19a48ce41178c82d2f69368 "CML watermark")

Corrects an issue where `self._ex_iterable` was erroneously used instead of `ex_iterable`, when both Distributed Data Parallel (DDP) and multi num_worker are used concurrently. This improper usage led to the generation of incorrect `shards_indices`, subsequently causing issues with the control flow responsible for worker creation. The fix ensures the appropriate iterable is used, thus providing a more accurate determination of whether a new worker should be instantiated or not.


fix os.listdir return name is empty string

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6581). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq

hello,
Can you help me check if there are any issues with this PR? Why hasn't anyone merged?

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004968 / 0.011353 (-0.006385) | 0.003516 / 0.011008 (-0.007492) | 0.063787 / 0.038508 (0.025279) | 0.031695 / 0.023109 (0.008586) | 0.240081 / 0.275898 (-0.035817) | 0.260984 / 0.323480 (-0.062496) | 0.003832 / 0.007986 (-0.004153) | 0.002680 / 0.004328 (-0.001648) | 0.049199 / 0.004250 (0.044948) | 0.044720 / 0.037052 (0.007668) | 0.255812 / 0.258489 (-0.002677) | 0.275923 / 0.293841 (-0.017918) | 0.026849 / 0.128546 (-0.101697) | 0.010473 / 0.075646 (-0.065174) | 0.209069 / 0.419271 (-0.210202) | 0.035731 / 0.043533 (-0.007802) | 0.246596 / 0.255139 (-0.008543) | 0.265889 / 0.283200 (-0.017311) | 0.017607 / 0.141683 (-0.124075) | 1.128648 / 1.452155 (-0.323507) | 1.174379 / 1.492716 (-0.318338) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.098214 / 0.018006 (0.080207) | 0.311969 / 0.000490 (0.311480) | 0.000266 / 0.000200 (0.000066) | 0.000056 / 0.000054 (0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018401 / 0.037411 (-0.019010) | 0.061347 / 0.014526 (0.046821) | 0.073628 / 0.176557 (-0.102928) | 0.121359 / 0.737135 (-0.615776) | 0.075148 / 0.296338 (-0.221190) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.274098 / 0.215209 (0.058889) | 2.707633 / 2.077655 (0.629978) | 1.453615 / 1.504120 (-0.050504) | 1.311942 / 1.541195 (-0.229253) | 1.332394 / 1.468490 (-0.136096) | 0.566947 / 4.584777 (-4.017830) | 2.383291 / 3.745712 (-1.362421) | 2.754779 / 5.269862 (-2.515083) | 1.725164 / 4.565676 (-2.840512) | 0.062124 / 0.424275 (-0.362152) | 0.005111 / 0.007607 (-0.002496) | 0.334217 / 0.226044 (0.108173) | 3.271619 / 2.268929 (1.002690) | 1.776906 / 55.444624 (-53.667718) | 1.519238 / 6.876477 (-5.357239) | 1.534722 / 2.142072 (-0.607351) | 0.646143 / 4.805227 (-4.159084) | 0.117015 / 6.500664 (-6.383649) | 0.042578 / 0.075469 (-0.032891) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.948488 / 1.841788 (-0.893299) | 11.598027 / 8.074308 (3.523719) | 10.269199 / 10.191392 (0.077807) | 0.144887 / 0.680424 (-0.535537) | 0.014745 / 0.534201 (-0.519456) | 0.289185 / 0.579283 (-0.290099) | 0.275243 / 0.434364 (-0.159120) | 0.328088 / 0.540337 (-0.212250) | 0.430161 / 1.386936 (-0.956775) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005020 / 0.011353 (-0.006333) | 0.003246 / 0.011008 (-0.007762) | 0.049810 / 0.038508 (0.011302) | 0.032215 / 0.023109 (0.009105) | 0.271033 / 0.275898 (-0.004866) | 0.294957 / 0.323480 (-0.028523) | 0.004192 / 0.007986 (-0.003793) | 0.002652 / 0.004328 (-0.001677) | 0.049132 / 0.004250 (0.044881) | 0.047818 / 0.037052 (0.010766) | 0.292370 / 0.258489 (0.033881) | 0.316142 / 0.293841 (0.022301) | 0.049539 / 0.128546 (-0.079007) | 0.010533 / 0.075646 (-0.065113) | 0.058131 / 0.419271 (-0.361141) | 0.033807 / 0.043533 (-0.009725) | 0.277623 / 0.255139 (0.022484) | 0.292294 / 0.283200 (0.009094) | 0.021110 / 0.141683 (-0.120573) | 1.160997 / 1.452155 (-0.291157) | 1.213553 / 1.492716 (-0.279163) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.098220 / 0.018006 (0.080214) | 0.312342 / 0.000490 (0.311852) | 0.000231 / 0.000200 (0.000031) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022893 / 0.037411 (-0.014519) | 0.075572 / 0.014526 (0.061046) | 0.088357 / 0.176557 (-0.088199) | 0.126354 / 0.737135 (-0.610782) | 0.089763 / 0.296338 (-0.206575) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.284368 / 0.215209 (0.069159) | 2.785497 / 2.077655 (0.707842) | 1.499364 / 1.504120 (-0.004756) | 1.376020 / 1.541195 (-0.165175) | 1.394270 / 1.468490 (-0.074220) | 0.571945 / 4.584777 (-4.012832) | 2.419148 / 3.745712 (-1.326564) | 2.796974 / 5.269862 (-2.472887) | 1.749531 / 4.565676 (-2.816145) | 0.064088 / 0.424275 (-0.360187) | 0.005294 / 0.007607 (-0.002313) | 0.336250 / 0.226044 (0.110206) | 3.315933 / 2.268929 (1.047004) | 1.877165 / 55.444624 (-53.567459) | 1.592336 / 6.876477 (-5.284140) | 1.599979 / 2.142072 (-0.542093) | 0.655617 / 4.805227 (-4.149610) | 0.117636 / 6.500664 (-6.383028) | 0.040813 / 0.075469 (-0.034656) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.976887 / 1.841788 (-0.864901) | 12.668753 / 8.074308 (4.594445) | 11.081253 / 10.191392 (0.889861) | 0.134494 / 0.680424 (-0.545930) | 0.016053 / 0.534201 (-0.518148) | 0.291607 / 0.579283 (-0.287676) | 0.287726 / 0.434364 (-0.146638) | 0.328108 / 0.540337 (-0.212229) | 0.425194 / 1.386936 (-0.961742) |

![](https://cml.dev/watermark.png#32672349e3e5abe21505fdbda122dd3426f8920f "CML watermark")

fix #6588

xlistdir return name is empty string

for example：
`
from datasets.download.streaming_download_manager import xjoin
from datasets.download.streaming_download_manager import xlistdir
config = DownloadConfig(storage_options=options)
manger = StreamingDownloadManager("ILSVRC2012",download_config=config)
input_path = "lakefs://datalab/main/imagenet/ILSVRC2012.zip"

download_files = manger.download_and_extract(input_path)

current_dir = xjoin(download_files,"ILSVRC2012/Images/ILSVRC2012_img_train")
folder_list = xlistdir(current_dir)

`


dataset cache only stores one config of the dataset in parquet dir, and uses that for all other configs resulting in showing same data in all configs.

### Describe the bug

ds = load_dataset("ai2_arc", "ARC-Easy"), i have tried to force redownload, delete cache and changing the cache dir.

### Steps to reproduce the bug

dataset = []
dataset_name = "ai2_arc"
possible_configs = [
    'ARC-Challenge',
    'ARC-Easy'
    ]
for config in possible_configs:
    dataset_slice = load_dataset(dataset_name, config,ignore_verifications=True,cache_dir='ai2_arc_files')
    dataset.append(dataset_slice)

### Expected behavior

all configs should get saved in cache with their respective names.

### Environment info

ai2_arc


Hi @haok1402, I have created an issue in the Discussion tab of the corresponding dataset: https://huggingface.co/datasets/eli5/discussions/7
Let's continue the discussion there!

### Describe the bug

Unable to load `eli5` dataset with streaming.

### Steps to reproduce the bug

This fails with FileNotFoundError: https://files.pushshift.io/reddit/submissions
```
from datasets import load_dataset
load_dataset("eli5", streaming=True)
```

This works correctly.
```
from datasets import load_dataset
load_dataset("eli5")
```

### Expected behavior

- Loading `eli5` dataset should not raise an error under the streaming mode.
- Or at the very least, show a warning that streaming mode is not supported with `eli5` dataset.

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-6.2.0-39-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.19.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
- `fsspec` version: 2023.6.0


Faster webdataset streaming

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6578). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I added faster streaming support using streaming Requests instances in `huggingface_hub` and will be available in 0.21.

This PR can be used with https://github.com/huggingface/huggingface_hub/pull/1967 to get fast WebDataset streaming

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004941 / 0.011353 (-0.006412) | 0.003431 / 0.011008 (-0.007577) | 0.062768 / 0.038508 (0.024260) | 0.029212 / 0.023109 (0.006103) | 0.253053 / 0.275898 (-0.022845) | 0.273061 / 0.323480 (-0.050419) | 0.004114 / 0.007986 (-0.003871) | 0.002713 / 0.004328 (-0.001616) | 0.048481 / 0.004250 (0.044231) | 0.040001 / 0.037052 (0.002949) | 0.268461 / 0.258489 (0.009971) | 0.287767 / 0.293841 (-0.006074) | 0.027885 / 0.128546 (-0.100661) | 0.010474 / 0.075646 (-0.065172) | 0.207989 / 0.419271 (-0.211282) | 0.035893 / 0.043533 (-0.007640) | 0.256833 / 0.255139 (0.001694) | 0.274197 / 0.283200 (-0.009003) | 0.017283 / 0.141683 (-0.124400) | 1.133597 / 1.452155 (-0.318558) | 1.206661 / 1.492716 (-0.286055) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.089610 / 0.018006 (0.071604) | 0.306051 / 0.000490 (0.305562) | 0.000217 / 0.000200 (0.000017) | 0.000042 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018686 / 0.037411 (-0.018725) | 0.061253 / 0.014526 (0.046727) | 0.073654 / 0.176557 (-0.102903) | 0.120499 / 0.737135 (-0.616637) | 0.074827 / 0.296338 (-0.221511) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.293756 / 0.215209 (0.078547) | 2.897755 / 2.077655 (0.820100) | 1.558146 / 1.504120 (0.054026) | 1.458020 / 1.541195 (-0.083174) | 1.453489 / 1.468490 (-0.015001) | 0.576666 / 4.584777 (-4.008111) | 2.423441 / 3.745712 (-1.322271) | 2.727760 / 5.269862 (-2.542102) | 1.750287 / 4.565676 (-2.815390) | 0.062094 / 0.424275 (-0.362181) | 0.004940 / 0.007607 (-0.002667) | 0.338815 / 0.226044 (0.112770) | 3.342677 / 2.268929 (1.073748) | 1.928335 / 55.444624 (-53.516290) | 1.629965 / 6.876477 (-5.246511) | 1.651836 / 2.142072 (-0.490236) | 0.644354 / 4.805227 (-4.160874) | 0.117890 / 6.500664 (-6.382774) | 0.041907 / 0.075469 (-0.033562) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.984399 / 1.841788 (-0.857389) | 11.516572 / 8.074308 (3.442264) | 10.326922 / 10.191392 (0.135530) | 0.130821 / 0.680424 (-0.549603) | 0.014084 / 0.534201 (-0.520117) | 0.287078 / 0.579283 (-0.292205) | 0.263466 / 0.434364 (-0.170898) | 0.326867 / 0.540337 (-0.213470) | 0.425313 / 1.386936 (-0.961623) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005305 / 0.011353 (-0.006048) | 0.003646 / 0.011008 (-0.007362) | 0.049402 / 0.038508 (0.010894) | 0.031719 / 0.023109 (0.008610) | 0.272579 / 0.275898 (-0.003319) | 0.295241 / 0.323480 (-0.028239) | 0.004309 / 0.007986 (-0.003677) | 0.002781 / 0.004328 (-0.001548) | 0.048134 / 0.004250 (0.043883) | 0.044702 / 0.037052 (0.007650) | 0.288201 / 0.258489 (0.029712) | 0.320351 / 0.293841 (0.026510) | 0.051327 / 0.128546 (-0.077219) | 0.011019 / 0.075646 (-0.064628) | 0.057983 / 0.419271 (-0.361288) | 0.034211 / 0.043533 (-0.009322) | 0.272856 / 0.255139 (0.017717) | 0.290007 / 0.283200 (0.006807) | 0.018656 / 0.141683 (-0.123027) | 1.135017 / 1.452155 (-0.317138) | 1.183904 / 1.492716 (-0.308813) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090854 / 0.018006 (0.072847) | 0.299654 / 0.000490 (0.299165) | 0.000224 / 0.000200 (0.000024) | 0.000063 / 0.000054 (0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021882 / 0.037411 (-0.015529) | 0.075297 / 0.014526 (0.060771) | 0.086620 / 0.176557 (-0.089937) | 0.127125 / 0.737135 (-0.610011) | 0.088622 / 0.296338 (-0.207717) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.287104 / 0.215209 (0.071895) | 2.802723 / 2.077655 (0.725068) | 1.570137 / 1.504120 (0.066017) | 1.452234 / 1.541195 (-0.088961) | 1.465457 / 1.468490 (-0.003033) | 0.564965 / 4.584777 (-4.019812) | 2.416724 / 3.745712 (-1.328988) | 2.645057 / 5.269862 (-2.624805) | 1.727599 / 4.565676 (-2.838078) | 0.063338 / 0.424275 (-0.360937) | 0.005018 / 0.007607 (-0.002589) | 0.345280 / 0.226044 (0.119235) | 3.384323 / 2.268929 (1.115395) | 1.957227 / 55.444624 (-53.487397) | 1.667620 / 6.876477 (-5.208856) | 1.795339 / 2.142072 (-0.346733) | 0.642049 / 4.805227 (-4.163178) | 0.114853 / 6.500664 (-6.385811) | 0.040459 / 0.075469 (-0.035010) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.023640 / 1.841788 (-0.818147) | 11.998130 / 8.074308 (3.923822) | 10.858137 / 10.191392 (0.666744) | 0.130235 / 0.680424 (-0.550189) | 0.016201 / 0.534201 (-0.518000) | 0.289743 / 0.579283 (-0.289540) | 0.275100 / 0.434364 (-0.159264) | 0.329299 / 0.540337 (-0.211039) | 0.418632 / 1.386936 (-0.968304) |

![](https://cml.dev/watermark.png#98495237883c5ed5a36fac125e68cad97598916f "CML watermark")

requests.get(..., streaming=True) is faster than using HTTP range requests when streaming large TAR files

it can be enabled using block_size=0 in fsspec

cc @rwightman


502 Server Errors when streaming large dataset

Hi! We should be able to avoid this error by retrying to read the data when it happens. I'll open a PR in `huggingface_hub` to address this.

Thanks for the fix @mariosasko! Just wondering whether "500 error" should also be excluded? I got these errors overnight:

```
huggingface_hub.utils._errors.HfHubHTTPError: 500 Server Error: Internal Server Error for url: https://huggingface.co/da
tasets/sanchit-gandhi/concatenated-train-set-label-length-256/resolve/91e6a0cd0356605b021384ded813cfcf356a221c/train/tra
in-02618-of-04012.parquet (Request ID: Root=1-65b18b81-627f2c2943bbb8ab68d19ee2;129537bd-1934-4257-a4d8-1cb774f8e1f8)

Internal Error - We're working hard to fix this as soon as possible!
```

Gently pining @mariosasko and @Wauplin - when trying to stream this large dataset from the HF Hub, I'm running into `500 Internal Server Errors` as described above. I'd love to be able to use the Hub exclusively to stream data when training, but this error pops up a few times a week, terminating training runs and causing me to have to rewind to the last saved checkpoint. Do we reckon there's a way we can protect Datasets' streaming against these errors? The same reproducer as the [original comment](https://github.com/huggingface/datasets/issues/6577#issue-2074790848) can be used, but it's somewhat random whether we hit a 500 error. Leaving the full traceback below:

```
Traceback (most recent call last):
  File "/home/sanchitgandhi/hf/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loo
p
    data = fetcher.fetch(index)
  File "/home/sanchitgandhi/hf/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 32, in fetch
    data.append(next(self.dataset_iter))
  File "/home/sanchitgandhi/datasets/src/datasets/iterable_dataset.py", line 1367, in __iter__
    yield from self._iter_pytorch()
  File "/home/sanchitgandhi/datasets/src/datasets/iterable_dataset.py", line 1302, in _iter_pytorch
    for key, example in ex_iterable:
  File "/home/sanchitgandhi/datasets/src/datasets/iterable_dataset.py", line 987, in __iter__
    for x in self.ex_iterable:
  File "/home/sanchitgandhi/datasets/src/datasets/iterable_dataset.py", line 867, in __iter__
    yield from self._iter()
  File "/home/sanchitgandhi/datasets/src/datasets/iterable_dataset.py", line 904, in _iter
    for key, example in iterator:
  File "/home/sanchitgandhi/datasets/src/datasets/iterable_dataset.py", line 679, in __iter__
    yield from self._iter()
  File "/home/sanchitgandhi/datasets/src/datasets/iterable_dataset.py", line 741, in _iter                    [235/1892]
    for key, example in iterator:
  File "/home/sanchitgandhi/datasets/src/datasets/iterable_dataset.py", line 1119, in __iter__
    for key, example in self.ex_iterable:
  File "/home/sanchitgandhi/datasets/src/datasets/iterable_dataset.py", line 282, in __iter__
    for key, pa_table in self.generate_tables_fn(**self.kwargs):
  File "/home/sanchitgandhi/datasets/src/datasets/packaged_modules/parquet/parquet.py", line 87, in _generate_tables
    for batch_idx, record_batch in enumerate(
  File "pyarrow/_parquet.pyx", line 1587, in iter_batches
  File "pyarrow/types.pxi", line 88, in pyarrow.lib._datatype_to_pep3118
  File "/home/sanchitgandhi/datasets/src/datasets/download/streaming_download_manager.py", line 342, in read_with_retrie
s
    out = read(*args, **kwargs)
  File "/home/sanchitgandhi/hf/lib/python3.10/site-packages/fsspec/spec.py", line 1856, in read
    out = self.cache._fetch(self.loc, self.loc + length)
  File "/home/sanchitgandhi/hf/lib/python3.10/site-packages/fsspec/caching.py", line 189, in _fetch
    self.cache = self.fetcher(start, end)  # new block replaces old
  File "/home/sanchitgandhi/hf/lib/python3.10/site-packages/huggingface_hub/hf_file_system.py", line 629, in _fetch_rang
e
    hf_raise_for_status(r)
  File "/home/sanchitgandhi/hf/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 362, in hf_raise_for
_status
    raise HfHubHTTPError(str(e), response=response) from e
huggingface_hub.utils._errors.HfHubHTTPError: 500 Server Error: Internal Server Error for url: https://huggingface.co/da
tasets/sanchit-gandhi/concatenated-train-set-label-length-256-conditioned/resolve/3c3c0cce51df9f9d2e75968bb2a1851894f504
0d/train/train-03515-of-04010.parquet (Request ID: Root=1-65c7c4c4-153fe71401558c8c2d272c8a;fec3ec68-4a0a-4bfd-95ba-b0a0
5684d612)

Internal Error - We're working hard to fix this as soon as possible!

@sanchit-gandhi thanks for the feedback. I've opened https://github.com/huggingface/huggingface_hub/pull/2026 to make the download process more robust. I believe that you've witness this problem on Saturday due to the Hub outage. Hope the PR will make your life easier though :)

Awesome, thanks @Wauplin! Makes sense re the Hub outage

### Describe the bug

When streaming a [large ASR dataset](https://huggingface.co/datasets/sanchit-gandhi/concatenated-train-set) from the Hug (~3TB) I often encounter 502 Server Errors seemingly randomly during streaming:
```
huggingface_hub.utils._errors.HfHubHTTPError: 502 Server Error: Bad Gateway for url: https://huggingface.co/datasets/sanchit-gandhi/concatenated-train-set/resolve/7d2acc5c59de848e456e951a76e805304d6fb350/train/train-00288-of-07135.parquet
```

This is despite the parquet file definitely existing on the Hub: https://huggingface.co/datasets/sanchit-gandhi/concatenated-train-set/blob/main/train/train-00228-of-07135.parquet
And having the correct commit id: [7d2acc5c59de848e456e951a76e805304d6fb350](https://huggingface.co/datasets/sanchit-gandhi/concatenated-train-set/commits/main/train)

I’m wondering whether this is coming from datasets? Or from the Hub side?

### Steps to reproduce the bug

Reproducer:
```python
from datasets import load_dataset
from torch.utils.data import DataLoader
from tqdm import tqdm

NUM_EPOCHS = 20

dataset = load_dataset("sanchit-gandhi/concatenated-train-set", "train", streaming=True)
dataset = dataset.with_format("torch")
dataloader = DataLoader(dataset["train"], batch_size=256, drop_last=True, pin_memory=True, num_workers=16)

for epoch in tqdm(range(NUM_EPOCHS), desc="Epoch", position=0):
    for batch in tqdm(dataloader, desc="Batch", position=1):
        continue
```

Running the above script tends to fail within about 2 hours with a traceback like the following:

<details>

<summary> Traceback: </summary>

```python
1029     for batch in train_loader:
1030   File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
1031     data = self._next_data()
1032   File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1325, in _next_data
1033     return self._process_data(data)
1034   File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
1035     data.reraise()
1036   File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/torch/_utils.py", line 694, in reraise
1037     raise exception
1038 huggingface_hub.utils._errors.HfHubHTTPError: Caught HfHubHTTPError in DataLoader worker process 10.
1039 Original Traceback (most recent call last):
1040   File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/huggingface_hub/utils/_errors.py", line 286, in hf_raise_for_status
1041     response.raise_for_status()
1042   File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/requests/models.py", line 1021, in raise_for_status
1043     raise HTTPError(http_error_msg, response=self)
1044 requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://huggingface.co/datasets/sanchit-gandhi/concatenated-train-set/resolve/7d2acc5c59de848e456e951a76e805304d6fb350/train/train-00288-of-07135.parquet
1045 The above exception was the direct cause of the following exception:
1046 Traceback (most recent call last):
1047   File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
1048     data = fetcher.fetch(index)
1049   File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 32, in fetch
1050     data.append(next(self.dataset_iter))
1051   File "/home/sanchitgandhi/datasets/src/datasets/iterable_dataset.py", line 1363, in __iter__
1052     yield from self._iter_pytorch()
1053   File "/home/sanchitgandhi/datasets/src/datasets/iterable_dataset.py", line 1298, in _iter_pytorch
1054     for key, example in ex_iterable:
1055   File "/home/sanchitgandhi/datasets/src/datasets/iterable_dataset.py", line 983, in __iter__
1056     for x in self.ex_iterable:
1057   File "/home/sanchitgandhi/datasets/src/datasets/iterable_dataset.py", line 863, in __iter__
1058     yield from self._iter()
1059   File "/home/sanchitgandhi/datasets/src/datasets/iterable_dataset.py", line 900, in _iter
1060     for key, example in iterator:
1061   File "/home/sanchitgandhi/datasets/src/datasets/iterable_dataset.py", line 679, in __iter__
1062     yield from self._iter()
1063   File "/home/sanchitgandhi/datasets/src/datasets/iterable_dataset.py", line 741, in _iter
1064     for key, example in iterator:
1065   File "/home/sanchitgandhi/datasets/src/datasets/iterable_dataset.py", line 863, in __iter__
1066     yield from self._iter()
1067   File "/home/sanchitgandhi/datasets/src/datasets/iterable_dataset.py", line 900, in _iter
1068     for key, example in iterator:
1069   File "/home/sanchitgandhi/datasets/src/datasets/iterable_dataset.py", line 1115, in __iter__
1070     for key, example in self.ex_iterable:
1071   File "/home/sanchitgandhi/datasets/src/datasets/iterable_dataset.py", line 679, in __iter__
1072     yield from self._iter()
1073   File "/home/sanchitgandhi/datasets/src/datasets/iterable_dataset.py", line 741, in _iter
1074     for key, example in iterator:
1075   File "/home/sanchitgandhi/datasets/src/datasets/iterable_dataset.py", line 1115, in __iter__
1076     for key, example in self.ex_iterable:
1077   File "/home/sanchitgandhi/datasets/src/datasets/iterable_dataset.py", line 282, in __iter__
1078     for key, pa_table in self.generate_tables_fn(**self.kwargs):
1079   File "/home/sanchitgandhi/datasets/src/datasets/packaged_modules/parquet/parquet.py", line 87, in _generate_tables
1080     for batch_idx, record_batch in enumerate(
1081   File "pyarrow/_parquet.pyx", line 1367, in iter_batches
1082   File "pyarrow/types.pxi", line 88, in pyarrow.lib._datatype_to_pep3118
1083   File "/home/sanchitgandhi/datasets/src/datasets/download/streaming_download_manager.py", line 341, in read_with_retries
1084     out = read(*args, **kwargs)
1085   File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/fsspec/spec.py", line 1856, in read
1086     out = self.cache._fetch(self.loc, self.loc + length)
1087   File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/fsspec/caching.py", line 189, in _fetch
1088     self.cache = self.fetcher(start, end)  # new block replaces old
1089   File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/huggingface_hub/hf_file_system.py", line 626, in _fetch_range
1090     hf_raise_for_status(r)
1091   File "/home/sanchitgandhi/hf/lib/python3.8/site-packages/huggingface_hub/utils/_errors.py", line 333, in hf_raise_for_status
1092     raise HfHubHTTPError(str(e), response=response) from e
1093 huggingface_hub.utils._errors.HfHubHTTPError: 502 Server Error: Bad Gateway for url: https://huggingface.co/datasets/sanchit-gandhi/concatenated-train-set/resolve/7d2acc5c59de848e456e951a76e805304d6fb350/train/train-00288-of-07135.parquet
```

</details>

### Expected behavior

Should be able to stream the dataset without any 502 error.

### Environment info

- `datasets` version: 2.16.2.dev0
- Platform: Linux-5.13.0-1023-gcp-x86_64-with-glibc2.29
- Python version: 3.8.10
- `huggingface_hub` version: 0.20.1
- PyArrow version: 14.0.2
- Pandas version: 2.0.3
- `fsspec` version: 2023.10.0


document page 404 not found after redirection

Thanks for reporting! I've opened a PR with a fix.

### Describe the bug

The redirected page encountered 404 not found.

### Steps to reproduce the bug

1. In this tutorial: https://huggingface.co/learn/nlp-course/chapter5/4?fw=pt
original md: https://github.com/huggingface/course/blob/2c733c2246b8b7e0e6f19a9e5d15bb12df43b2a3/chapters/en/chapter5/4.mdx#L49

```
By default, 🤗 Datasets will decompress the files needed to load a dataset. If you want to preserve hard drive space, you can pass `DownloadConfig(delete_extracted=True)` to the `download_config` argument of `load_dataset()`. See the [documentation](https://huggingface.co/docs/datasets/package_reference/builder_classes.html?#datasets.utils.DownloadConfig) for more details.
```
The documentation points to `https://huggingface.co/docs/datasets/package_reference/builder_classes.html?#datasets.utils.DownloadConfig`

it shows `The documentation page PACKAGE_REFERENCE/BUILDER_CLASSES.HTML doesn’t exist in v2.16.1, but exists on the main version. Click [here](https://huggingface.co/docs/datasets/main/en/package_reference/builder_classes.html) to redirect to the main version of the documentation.`

But the redirected website `https://huggingface.co/docs/datasets/main/en/package_reference/builder_classes.html` is 404 not found.

### Expected behavior

I Guess the redirected webisite should be
`https://huggingface.co/docs/datasets/main/en/package_reference/builder_classes`    (without `.html`)
or `https://huggingface.co/docs/datasets/main/en/package_reference/builder_classes#datasets.DownloadConfig`.

### Environment info

Datasets main


[IterableDataset] Fix `drop_last_batch`in map after shuffling or sharding

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6575). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005095 / 0.011353 (-0.006257) | 0.003531 / 0.011008 (-0.007478) | 0.063634 / 0.038508 (0.025126) | 0.031187 / 0.023109 (0.008078) | 0.246375 / 0.275898 (-0.029523) | 0.261204 / 0.323480 (-0.062276) | 0.002898 / 0.007986 (-0.005088) | 0.003280 / 0.004328 (-0.001049) | 0.050739 / 0.004250 (0.046488) | 0.042905 / 0.037052 (0.005852) | 0.244506 / 0.258489 (-0.013983) | 0.269403 / 0.293841 (-0.024438) | 0.027588 / 0.128546 (-0.100959) | 0.010860 / 0.075646 (-0.064787) | 0.208332 / 0.419271 (-0.210939) | 0.035762 / 0.043533 (-0.007771) | 0.244448 / 0.255139 (-0.010691) | 0.278464 / 0.283200 (-0.004735) | 0.019839 / 0.141683 (-0.121844) | 1.145340 / 1.452155 (-0.306815) | 1.173240 / 1.492716 (-0.319476) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090472 / 0.018006 (0.072466) | 0.300883 / 0.000490 (0.300394) | 0.000202 / 0.000200 (0.000003) | 0.000049 / 0.000054 (-0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.017884 / 0.037411 (-0.019527) | 0.060629 / 0.014526 (0.046103) | 0.073157 / 0.176557 (-0.103400) | 0.120065 / 0.737135 (-0.617070) | 0.074519 / 0.296338 (-0.221820) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.289586 / 0.215209 (0.074377) | 2.821042 / 2.077655 (0.743387) | 1.515515 / 1.504120 (0.011395) | 1.390569 / 1.541195 (-0.150625) | 1.433238 / 1.468490 (-0.035252) | 0.567357 / 4.584777 (-4.017420) | 2.345483 / 3.745712 (-1.400229) | 2.803964 / 5.269862 (-2.465898) | 1.775343 / 4.565676 (-2.790334) | 0.063186 / 0.424275 (-0.361089) | 0.005013 / 0.007607 (-0.002594) | 0.335607 / 0.226044 (0.109562) | 3.307071 / 2.268929 (1.038143) | 1.875228 / 55.444624 (-53.569396) | 1.618286 / 6.876477 (-5.258191) | 1.615963 / 2.142072 (-0.526109) | 0.642633 / 4.805227 (-4.162594) | 0.117222 / 6.500664 (-6.383443) | 0.042590 / 0.075469 (-0.032879) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.960724 / 1.841788 (-0.881064) | 11.652978 / 8.074308 (3.578670) | 10.069318 / 10.191392 (-0.122074) | 0.128161 / 0.680424 (-0.552263) | 0.014095 / 0.534201 (-0.520106) | 0.288386 / 0.579283 (-0.290897) | 0.260373 / 0.434364 (-0.173991) | 0.327443 / 0.540337 (-0.212894) | 0.419020 / 1.386936 (-0.967916) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005018 / 0.011353 (-0.006335) | 0.003503 / 0.011008 (-0.007505) | 0.049718 / 0.038508 (0.011210) | 0.029311 / 0.023109 (0.006202) | 0.271097 / 0.275898 (-0.004801) | 0.297370 / 0.323480 (-0.026110) | 0.004230 / 0.007986 (-0.003755) | 0.002741 / 0.004328 (-0.001587) | 0.049686 / 0.004250 (0.045435) | 0.044171 / 0.037052 (0.007119) | 0.274851 / 0.258489 (0.016362) | 0.309554 / 0.293841 (0.015714) | 0.029488 / 0.128546 (-0.099058) | 0.010767 / 0.075646 (-0.064880) | 0.057739 / 0.419271 (-0.361532) | 0.053319 / 0.043533 (0.009786) | 0.277739 / 0.255139 (0.022600) | 0.291341 / 0.283200 (0.008142) | 0.019587 / 0.141683 (-0.122096) | 1.113823 / 1.452155 (-0.338332) | 1.169409 / 1.492716 (-0.323307) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091889 / 0.018006 (0.073883) | 0.309162 / 0.000490 (0.308672) | 0.000222 / 0.000200 (0.000022) | 0.000055 / 0.000054 (0.000000) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022202 / 0.037411 (-0.015209) | 0.076113 / 0.014526 (0.061587) | 0.088416 / 0.176557 (-0.088141) | 0.126822 / 0.737135 (-0.610314) | 0.089540 / 0.296338 (-0.206798) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.293697 / 0.215209 (0.078487) | 2.880680 / 2.077655 (0.803026) | 1.580122 / 1.504120 (0.076002) | 1.449492 / 1.541195 (-0.091703) | 1.478900 / 1.468490 (0.010410) | 0.563402 / 4.584777 (-4.021375) | 2.408692 / 3.745712 (-1.337020) | 2.794108 / 5.269862 (-2.475754) | 1.728549 / 4.565676 (-2.837128) | 0.063152 / 0.424275 (-0.361123) | 0.004985 / 0.007607 (-0.002622) | 0.343340 / 0.226044 (0.117295) | 3.426454 / 2.268929 (1.157525) | 1.932918 / 55.444624 (-53.511706) | 1.649533 / 6.876477 (-5.226944) | 1.673416 / 2.142072 (-0.468656) | 0.640000 / 4.805227 (-4.165227) | 0.115501 / 6.500664 (-6.385163) | 0.040756 / 0.075469 (-0.034713) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.992468 / 1.841788 (-0.849319) | 12.392072 / 8.074308 (4.317764) | 11.025362 / 10.191392 (0.833970) | 0.130788 / 0.680424 (-0.549635) | 0.015647 / 0.534201 (-0.518554) | 0.285914 / 0.579283 (-0.293369) | 0.277208 / 0.434364 (-0.157156) | 0.322917 / 0.540337 (-0.217420) | 0.427308 / 1.386936 (-0.959628) |

![](https://cml.dev/watermark.png#999790bcf52f883de1b5233c5632ae73395021cf "CML watermark")

It was not taken into account e.g. when passing to a DataLoader with num_workers>0

Fix https://github.com/huggingface/datasets/issues/6565