Added functionality: sort datasets by multiple keys

> Thanks! I've left some comments.
>
> We should also add some tests, mainly to make sure `reverse` behaves as expected. Let me know if you need help with that.

Thanks for the offer! I couldn't find any guidelines on how huggingface goes about testing, so it would indeed be great to get a few pointers on that. I assume I should expand on the `test_sort` function in `test_arrow_dataset.py` but since I am not very familiar with the `datasets` package, it isn't immediately for which cases I should test (i.e., expand on).

@MichlF

Resolving a comment means that the comment has been addressed with the code change, so since this is not the case here, can you please "unresolve" the comments and address them adequately?

> I assume I should expand on the `test_sort` function in `test_arrow_dataset.py`

Yes, that's correct. I think one test to check sorting on multiple keys and another one to check if an error is raised when `len(reverse)!=len(column_names)` should be enough.

> Yes, that's correct. I think one test to check sorting on multiple keys and another one to check if an error is raised when `len(reverse)!=len(column_names)` should be enough.

I have added the tests in https://github.com/huggingface/datasets/pull/5502/commits/0efa259732e822e94d67b96a70031a3daccedfc1 by keeping them in the same format of the tests of the old `sort` function. Let me know if they can be improved.

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010170 / 0.011353 (-0.001183) | 0.005891 / 0.011008 (-0.005117) | 0.100416 / 0.038508 (0.061908) | 0.041309 / 0.023109 (0.018200) | 0.300813 / 0.275898 (0.024915) | 0.376679 / 0.323480 (0.053199) | 0.008806 / 0.007986 (0.000821) | 0.005964 / 0.004328 (0.001636) | 0.075862 / 0.004250 (0.071611) | 0.050370 / 0.037052 (0.013318) | 0.313365 / 0.258489 (0.054876) | 0.351184 / 0.293841 (0.057343) | 0.039556 / 0.128546 (-0.088991) | 0.012462 / 0.075646 (-0.063185) | 0.337141 / 0.419271 (-0.082130) | 0.049678 / 0.043533 (0.006145) | 0.298547 / 0.255139 (0.043408) | 0.317547 / 0.283200 (0.034347) | 0.113595 / 0.141683 (-0.028088) | 1.448467 / 1.452155 (-0.003688) | 1.501303 / 1.492716 (0.008587) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.011005 / 0.018006 (-0.007002) | 0.527430 / 0.000490 (0.526940) | 0.005073 / 0.000200 (0.004873) | 0.000100 / 0.000054 (0.000045) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030377 / 0.037411 (-0.007034) | 0.116932 / 0.014526 (0.102406) | 0.124047 / 0.176557 (-0.052509) | 0.192358 / 0.737135 (-0.544777) | 0.130528 / 0.296338 (-0.165811) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.401158 / 0.215209 (0.185949) | 4.005854 / 2.077655 (1.928200) | 1.810365 / 1.504120 (0.306245) | 1.626490 / 1.541195 (0.085295) | 1.752591 / 1.468490 (0.284101) | 0.709065 / 4.584777 (-3.875712) | 3.893356 / 3.745712 (0.147643) | 3.655180 / 5.269862 (-1.614682) | 1.873660 / 4.565676 (-2.692017) | 0.085860 / 0.424275 (-0.338415) | 0.012671 / 0.007607 (0.005063) | 0.512804 / 0.226044 (0.286759) | 5.103426 / 2.268929 (2.834497) | 2.336148 / 55.444624 (-53.108477) | 2.000140 / 6.876477 (-4.876336) | 2.095155 / 2.142072 (-0.046918) | 0.848612 / 4.805227 (-3.956615) | 0.171840 / 6.500664 (-6.328824) | 0.064144 / 0.075469 (-0.011325) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.222106 / 1.841788 (-0.619682) | 15.828559 / 8.074308 (7.754251) | 14.995298 / 10.191392 (4.803906) | 0.172783 / 0.680424 (-0.507641) | 0.029296 / 0.534201 (-0.504905) | 0.447469 / 0.579283 (-0.131814) | 0.658615 / 0.434364 (0.224251) | 1.527607 / 0.540337 (0.987270) | 1.830018 / 1.386936 (0.443082) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007922 / 0.011353 (-0.003431) | 0.005369 / 0.011008 (-0.005639) | 0.076580 / 0.038508 (0.038071) | 0.038770 / 0.023109 (0.015661) | 0.338995 / 0.275898 (0.063097) | 0.380865 / 0.323480 (0.057385) | 0.006489 / 0.007986 (-0.001497) | 0.004421 / 0.004328 (0.000093) | 0.074143 / 0.004250 (0.069893) | 0.054224 / 0.037052 (0.017171) | 0.348887 / 0.258489 (0.090397) | 0.395044 / 0.293841 (0.101203) | 0.037040 / 0.128546 (-0.091507) | 0.012547 / 0.075646 (-0.063099) | 0.087521 / 0.419271 (-0.331751) | 0.049918 / 0.043533 (0.006385) | 0.342428 / 0.255139 (0.087289) | 0.362216 / 0.283200 (0.079016) | 0.107204 / 0.141683 (-0.034479) | 1.509206 / 1.452155 (0.057052) | 1.596010 / 1.492716 (0.103293) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.246795 / 0.018006 (0.228788) | 0.505998 / 0.000490 (0.505509) | 0.000446 / 0.000200 (0.000246) | 0.000064 / 0.000054 (0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031591 / 0.037411 (-0.005821) | 0.117595 / 0.014526 (0.103069) | 0.132500 / 0.176557 (-0.044056) | 0.202244 / 0.737135 (-0.534891) | 0.136624 / 0.296338 (-0.159715) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.428235 / 0.215209 (0.213026) | 4.262691 / 2.077655 (2.185036) | 2.057348 / 1.504120 (0.553228) | 1.928559 / 1.541195 (0.387364) | 2.120838 / 1.468490 (0.652347) | 0.706300 / 4.584777 (-3.878477) | 3.951828 / 3.745712 (0.206115) | 2.144218 / 5.269862 (-3.125644) | 1.359500 / 4.565676 (-3.206177) | 0.085404 / 0.424275 (-0.338872) | 0.012363 / 0.007607 (0.004756) | 0.529985 / 0.226044 (0.303941) | 5.295831 / 2.268929 (3.026903) | 2.522602 / 55.444624 (-52.922022) | 2.182850 / 6.876477 (-4.693627) | 2.270187 / 2.142072 (0.128114) | 0.841676 / 4.805227 (-3.963551) | 0.168366 / 6.500664 (-6.332298) | 0.065371 / 0.075469 (-0.010098) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.261464 / 1.841788 (-0.580324) | 17.010125 / 8.074308 (8.935817) | 14.304453 / 10.191392 (4.113061) | 0.177782 / 0.680424 (-0.502642) | 0.017762 / 0.534201 (-0.516439) | 0.427283 / 0.579283 (-0.152000) | 0.455176 / 0.434364 (0.020812) | 0.525962 / 0.540337 (-0.014375) | 0.625583 / 1.386936 (-0.761353) |

![](https://cml.dev/watermark.png#3b2aba6637dc61f145acda40e4e7b028c3947d72 "CML watermark")

Added functionality implementation: sort datasets by multiple keys/columns as discussed in https://github.com/huggingface/datasets/issues/5425.


Increase chunk size for speeding up file downloads

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5501). All of your documentation changes will be reflected on that endpoint.

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008407 / 0.011353 (-0.002946) | 0.004651 / 0.011008 (-0.006357) | 0.100367 / 0.038508 (0.061859) | 0.029107 / 0.023109 (0.005998) | 0.302798 / 0.275898 (0.026900) | 0.354379 / 0.323480 (0.030899) | 0.006985 / 0.007986 (-0.001001) | 0.003365 / 0.004328 (-0.000963) | 0.078312 / 0.004250 (0.074062) | 0.034205 / 0.037052 (-0.002847) | 0.310431 / 0.258489 (0.051941) | 0.346239 / 0.293841 (0.052398) | 0.033800 / 0.128546 (-0.094747) | 0.011515 / 0.075646 (-0.064131) | 0.323588 / 0.419271 (-0.095684) | 0.040766 / 0.043533 (-0.002767) | 0.300914 / 0.255139 (0.045775) | 0.332983 / 0.283200 (0.049784) | 0.087500 / 0.141683 (-0.054182) | 1.469505 / 1.452155 (0.017350) | 1.505119 / 1.492716 (0.012403) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.187319 / 0.018006 (0.169313) | 0.405498 / 0.000490 (0.405008) | 0.001000 / 0.000200 (0.000800) | 0.000069 / 0.000054 (0.000015) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022583 / 0.037411 (-0.014828) | 0.098096 / 0.014526 (0.083570) | 0.104272 / 0.176557 (-0.072284) | 0.142801 / 0.737135 (-0.594335) | 0.109749 / 0.296338 (-0.186590) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.423343 / 0.215209 (0.208134) | 4.215116 / 2.077655 (2.137461) | 1.899714 / 1.504120 (0.395594) | 1.689579 / 1.541195 (0.148384) | 1.710292 / 1.468490 (0.241801) | 0.690976 / 4.584777 (-3.893801) | 3.432501 / 3.745712 (-0.313212) | 1.899600 / 5.269862 (-3.370261) | 1.279801 / 4.565676 (-3.285876) | 0.082763 / 0.424275 (-0.341512) | 0.012545 / 0.007607 (0.004938) | 0.531381 / 0.226044 (0.305336) | 5.320077 / 2.268929 (3.051148) | 2.370705 / 55.444624 (-53.073919) | 2.007089 / 6.876477 (-4.869388) | 2.062412 / 2.142072 (-0.079661) | 0.814998 / 4.805227 (-3.990229) | 0.149822 / 6.500664 (-6.350842) | 0.064399 / 0.075469 (-0.011070) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.226196 / 1.841788 (-0.615591) | 13.823443 / 8.074308 (5.749134) | 13.813667 / 10.191392 (3.622275) | 0.161289 / 0.680424 (-0.519135) | 0.028569 / 0.534201 (-0.505632) | 0.390360 / 0.579283 (-0.188923) | 0.396217 / 0.434364 (-0.038147) | 0.483120 / 0.540337 (-0.057217) | 0.570041 / 1.386936 (-0.816895) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006422 / 0.011353 (-0.004931) | 0.004528 / 0.011008 (-0.006481) | 0.076043 / 0.038508 (0.037535) | 0.027631 / 0.023109 (0.004522) | 0.340622 / 0.275898 (0.064724) | 0.376694 / 0.323480 (0.053214) | 0.004993 / 0.007986 (-0.002992) | 0.003403 / 0.004328 (-0.000926) | 0.074521 / 0.004250 (0.070270) | 0.037568 / 0.037052 (0.000516) | 0.343423 / 0.258489 (0.084934) | 0.387729 / 0.293841 (0.093888) | 0.031790 / 0.128546 (-0.096757) | 0.011767 / 0.075646 (-0.063879) | 0.085182 / 0.419271 (-0.334090) | 0.042867 / 0.043533 (-0.000666) | 0.341269 / 0.255139 (0.086130) | 0.368460 / 0.283200 (0.085261) | 0.090153 / 0.141683 (-0.051530) | 1.536490 / 1.452155 (0.084335) | 1.596403 / 1.492716 (0.103686) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.222373 / 0.018006 (0.204367) | 0.396145 / 0.000490 (0.395655) | 0.000384 / 0.000200 (0.000184) | 0.000062 / 0.000054 (0.000008) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024801 / 0.037411 (-0.012610) | 0.099711 / 0.014526 (0.085185) | 0.106094 / 0.176557 (-0.070463) | 0.147819 / 0.737135 (-0.589316) | 0.110065 / 0.296338 (-0.186274) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.442863 / 0.215209 (0.227654) | 4.420043 / 2.077655 (2.342388) | 2.070136 / 1.504120 (0.566016) | 1.862363 / 1.541195 (0.321168) | 1.910890 / 1.468490 (0.442400) | 0.702570 / 4.584777 (-3.882207) | 3.435855 / 3.745712 (-0.309857) | 1.871290 / 5.269862 (-3.398572) | 1.169321 / 4.565676 (-3.396355) | 0.083674 / 0.424275 (-0.340601) | 0.012823 / 0.007607 (0.005216) | 0.539330 / 0.226044 (0.313285) | 5.403317 / 2.268929 (3.134389) | 2.536508 / 55.444624 (-52.908117) | 2.179629 / 6.876477 (-4.696847) | 2.207586 / 2.142072 (0.065514) | 0.812256 / 4.805227 (-3.992972) | 0.152915 / 6.500664 (-6.347749) | 0.068431 / 0.075469 (-0.007038) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.294982 / 1.841788 (-0.546806) | 13.912811 / 8.074308 (5.838503) | 13.415658 / 10.191392 (3.224266) | 0.149531 / 0.680424 (-0.530893) | 0.016785 / 0.534201 (-0.517416) | 0.381055 / 0.579283 (-0.198228) | 0.392084 / 0.434364 (-0.042280) | 0.472614 / 0.540337 (-0.067724) | 0.559799 / 1.386936 (-0.827137) |

![](https://cml.dev/watermark.png#6ef20f9b71acbb387caab2d297d8c22ba3db3633 "CML watermark")

We simply do GET requests to hf.co to download files from the Hub right now. We may switch to hfh when we update how we do caching

You can try on any dataset hosted on the hub like `imagenet-1k`

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010931 / 0.011353 (-0.000422) | 0.005730 / 0.011008 (-0.005278) | 0.116653 / 0.038508 (0.078145) | 0.041439 / 0.023109 (0.018330) | 0.359559 / 0.275898 (0.083661) | 0.408398 / 0.323480 (0.084918) | 0.009193 / 0.007986 (0.001208) | 0.006024 / 0.004328 (0.001695) | 0.087743 / 0.004250 (0.083492) | 0.048636 / 0.037052 (0.011584) | 0.363133 / 0.258489 (0.104643) | 0.407144 / 0.293841 (0.113303) | 0.044610 / 0.128546 (-0.083936) | 0.014075 / 0.075646 (-0.061571) | 0.396506 / 0.419271 (-0.022766) | 0.057014 / 0.043533 (0.013482) | 0.358254 / 0.255139 (0.103115) | 0.399887 / 0.283200 (0.116687) | 0.115337 / 0.141683 (-0.026346) | 1.731655 / 1.452155 (0.279500) | 1.813276 / 1.492716 (0.320560) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.210197 / 0.018006 (0.192191) | 0.475887 / 0.000490 (0.475397) | 0.003323 / 0.000200 (0.003123) | 0.000100 / 0.000054 (0.000045) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031686 / 0.037411 (-0.005725) | 0.131167 / 0.014526 (0.116641) | 0.137919 / 0.176557 (-0.038637) | 0.184843 / 0.737135 (-0.552293) | 0.144998 / 0.296338 (-0.151340) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.471371 / 0.215209 (0.256162) | 4.693739 / 2.077655 (2.616084) | 2.251567 / 1.504120 (0.747447) | 1.993653 / 1.541195 (0.452458) | 2.053236 / 1.468490 (0.584746) | 0.809226 / 4.584777 (-3.775551) | 4.494120 / 3.745712 (0.748408) | 2.436921 / 5.269862 (-2.832940) | 1.541973 / 4.565676 (-3.023704) | 0.098401 / 0.424275 (-0.325874) | 0.014329 / 0.007607 (0.006722) | 0.597813 / 0.226044 (0.371769) | 5.964035 / 2.268929 (3.695107) | 2.709283 / 55.444624 (-52.735341) | 2.323537 / 6.876477 (-4.552940) | 2.401707 / 2.142072 (0.259635) | 0.976379 / 4.805227 (-3.828848) | 0.194638 / 6.500664 (-6.306026) | 0.076904 / 0.075469 (0.001435) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.516877 / 1.841788 (-0.324911) | 18.228010 / 8.074308 (10.153702) | 16.631750 / 10.191392 (6.440358) | 0.176030 / 0.680424 (-0.504394) | 0.033769 / 0.534201 (-0.500432) | 0.520511 / 0.579283 (-0.058773) | 0.531764 / 0.434364 (0.097400) | 0.648658 / 0.540337 (0.108321) | 0.779124 / 1.386936 (-0.607812) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008635 / 0.011353 (-0.002718) | 0.005785 / 0.011008 (-0.005223) | 0.087042 / 0.038508 (0.048534) | 0.039632 / 0.023109 (0.016523) | 0.419719 / 0.275898 (0.143821) | 0.463860 / 0.323480 (0.140380) | 0.006621 / 0.007986 (-0.001364) | 0.004655 / 0.004328 (0.000327) | 0.087003 / 0.004250 (0.082753) | 0.057122 / 0.037052 (0.020069) | 0.417820 / 0.258489 (0.159331) | 0.485981 / 0.293841 (0.192140) | 0.042606 / 0.128546 (-0.085940) | 0.014369 / 0.075646 (-0.061278) | 0.101939 / 0.419271 (-0.317333) | 0.058303 / 0.043533 (0.014770) | 0.415053 / 0.255139 (0.159914) | 0.439914 / 0.283200 (0.156714) | 0.134628 / 0.141683 (-0.007055) | 1.765464 / 1.452155 (0.313309) | 1.843963 / 1.492716 (0.351247) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.307156 / 0.018006 (0.289150) | 0.476657 / 0.000490 (0.476167) | 0.019718 / 0.000200 (0.019518) | 0.000160 / 0.000054 (0.000105) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035286 / 0.037411 (-0.002125) | 0.138094 / 0.014526 (0.123568) | 0.144768 / 0.176557 (-0.031789) | 0.191386 / 0.737135 (-0.545750) | 0.151988 / 0.296338 (-0.144350) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.504733 / 0.215209 (0.289523) | 5.027048 / 2.077655 (2.949394) | 2.441571 / 1.504120 (0.937451) | 2.198242 / 1.541195 (0.657047) | 2.298473 / 1.468490 (0.829983) | 0.848048 / 4.584777 (-3.736729) | 4.613102 / 3.745712 (0.867390) | 2.522824 / 5.269862 (-2.747037) | 1.610159 / 4.565676 (-2.955517) | 0.105197 / 0.424275 (-0.319078) | 0.015195 / 0.007607 (0.007588) | 0.626976 / 0.226044 (0.400932) | 6.268459 / 2.268929 (3.999530) | 3.014387 / 55.444624 (-52.430237) | 2.554102 / 6.876477 (-4.322375) | 2.656051 / 2.142072 (0.513979) | 1.027978 / 4.805227 (-3.777249) | 0.200686 / 6.500664 (-6.299978) | 0.077104 / 0.075469 (0.001635) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.485228 / 1.841788 (-0.356560) | 18.319949 / 8.074308 (10.245641) | 15.855739 / 10.191392 (5.664347) | 0.204365 / 0.680424 (-0.476059) | 0.023824 / 0.534201 (-0.510377) | 0.505000 / 0.579283 (-0.074283) | 0.502866 / 0.434364 (0.068502) | 0.629574 / 0.540337 (0.089237) | 0.746602 / 1.386936 (-0.640334) |

![](https://cml.dev/watermark.png#900d429d3601657f766737b8670f855033078d57 "CML watermark")

Original fix: https://github.com/huggingface/huggingface_hub/pull/1267
Not sure this function is actually still called though.

I haven't done benches on this. Is there a dataset where files are hosted on the hub through cloudfront so we can have the same setup as in `hf_hub` ?


WMT19 custom download checksum error

I update the `datatsets` version and it works.

### Describe the bug

I use the following scripts to download data from WMT19:
```python
import datasets
from datasets import inspect_dataset, load_dataset_builder
from wmt19.wmt_utils import _TRAIN_SUBSETS,_DEV_SUBSETS

## this is a must due to: https://discuss.huggingface.co/t/load-dataset-hangs-with-local-files/28034/3
if __name__ == '__main__':
    dev_subsets,train_subsets = [],[]
    for subset in _TRAIN_SUBSETS:
        if subset.target=='en' and 'de' in subset.sources:
            train_subsets.append(subset.name)
    for subset in _DEV_SUBSETS:
        if subset.target=='en' and 'de' in subset.sources:
            dev_subsets.append(subset.name)

    inspect_dataset("wmt19", "./wmt19")
    builder = load_dataset_builder(
        "./wmt19/wmt_utils.py",
        language_pair=("de", "en"),
        subsets={
            datasets.Split.TRAIN: train_subsets,
            datasets.Split.VALIDATION: dev_subsets,
        },

    )

    builder.download_and_prepare()
    ds = builder.as_dataset()
    ds.to_json("../data/wmt19/ende/data.json")
```
And I got the following error:
```
Traceback (most recent call last):                                                                                | 0/2 [00:00<?, ?obj/s]
  File "draft.py", line 26, in <module>
    builder.download_and_prepare()                                                                                | 0/1 [00:00<?, ?obj/s]
  File "/Users/hannibal046/anaconda3/lib/python3.8/site-packages/datasets/builder.py", line 605, in download_and_prepare
    self._download_and_prepare(%|                                                                                | 0/1 [00:00<?, ?obj/s]
  File "/Users/hannibal046/anaconda3/lib/python3.8/site-packages/datasets/builder.py", line 1104, in _download_and_prepare
    super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)                          | 0/1 [00:00<?, ?obj/s]
  File "/Users/hannibal046/anaconda3/lib/python3.8/site-packages/datasets/builder.py", line 676, in _download_and_prepare
    verify_checksums(s #13:   0%|                                                                                | 0/1 [00:00<?, ?obj/s]
  File "/Users/hannibal046/anaconda3/lib/python3.8/site-packages/datasets/utils/info_utils.py", line 35, in verify_checksums
    raise UnexpectedDownloadedFile(str(set(recorded_checksums) - set(expected_checksums)))                              | 0/1 [00:00<?, ?obj/s]
datasets.utils.info_utils.UnexpectedDownloadedFile: {'https://s3.amazonaws.com/web-language-models/paracrawl/release1/paracrawl-release1.en-de.zipporah0-dedup-clean.tgz', 'https://huggingface.co/datasets/wmt/wmt13/resolve/main-zip/training-parallel-europarl-v7.zip', 'https://huggingface.co/datasets/wmt/wmt18/resolve/main-zip/translation-task/rapid2016.zip', 'https://huggingface.co/datasets/wmt/wmt18/resolve/main-zip/translation-task/training-parallel-nc-v13.zip', 'https://huggingface.co/datasets/wmt/wmt17/resolve/main-zip/translation-task/training-parallel-nc-v12.zip', 'https://huggingface.co/datasets/wmt/wmt14/resolve/main-zip/training-parallel-nc-v9.zip', 'https://huggingface.co/datasets/wmt/wmt15/resolve/main-zip/training-parallel-nc-v10.zip', 'https://huggingface.co/datasets/wmt/wmt16/resolve/main-zip/translation-task/training-parallel-nc-v11.zip'}

```

### Steps to reproduce the bug

see above

### Expected behavior

download data successfully

### Environment info

datasets==2.1.0
python==3.8


`load_dataset` has ~4 seconds of overhead for cached data

enhancement

New feature or request

Hi ! To skip the verification step that checks if newer data exist, you can enable offline mode with `HF_DATASETS_OFFLINE=1`.

Although I agree this step should be much faster for datasets hosted on the HF Hub - we could just compare the commit hash from the local data and the remote git repository. We're not been leveraging the git commit hashes, since the library was built before we even had git repositories for each dataset on HF.

Thanks @lhoestq, for memory when I recorded those times I had `HF_DATASETS_OFFLINE` set.

### Feature request

When loading a dataset that has been cached locally, the `load_dataset` function takes a lot longer than it should take to fetch the dataset from disk (or memory).
This is particularly noticeable for smaller datasets. For example, wikitext-2, comparing `load_data` (once cached) and `load_from_disk`, the `load_dataset` method takes 40 times longer.

⏱    4.84s ⮜ load_dataset
⏱    119ms ⮜ load_from_disk

### Motivation

I assume this is doing something like checking for a newer version.

If so, that's an age old problem: do you make the user wait _every single time they load from cache_ or do you do something like load from cache always, _then_ check for a newer version and alert if they have stale data. The decision usually revolves around what percentage of the time the data will have been updated, and how dangerous old data is.

For most datasets it's extremely unlikely that there will be a newer version on any given run, so 99% of the time this is just wasted time.

Maybe you don't want to make that decision for all users, but at least having the _option_ to not wait for checks would be an improvement.

### Your contribution

.


TypeError: 'bool' object is not iterable when filtering a datasets.arrow_dataset.Dataset

Hi! Instead of a single boolean, your filter function should return an iterable (of booleans) in the batched mode like so:
```python
train_dataset = train_dataset.filter(
            function=lambda batch: [image is not None for image in batch["image"]],
            batched=True,
            batch_size=10)
```

PS: You can make this operation much faster by operating directly on the arrow data to skip the decoding part:
```python
train_dataset = train_dataset.with_format("arrow")
train_dataset = train_dataset.filter(
            function=lambda table: table["image"].is_valid().to_pylist(),
            batched=True,
            batch_size=100)
train_dataset = train_dataset.with_format(None)
```

I hit the same issue and the error message isn't really clear on what's going wrong. It might be helpful to update the docs with a batched example.

### Describe the bug

Hi,

Thanks for the amazing work on the library!

**Describe the bug**

I think I might have noticed a small bug in the filter method.
Having loaded a dataset using `load_dataset`,  when I try to filter out empty entries with `batched=True`, I get a TypeError.

### Steps to reproduce the bug

```
train_dataset = train_dataset.filter(
            function=lambda example: example["image"] is not None,
            batched=True,
            batch_size=10)
```
Error message:

```
File .../lib/python3.9/site-packages/datasets/fingerprint.py:480, in fingerprint_transform.<locals>._fingerprint.<locals>.wrapper(*args, **kwargs)
    476             validate_fingerprint(kwargs[fingerprint_name])
    478 # Call actual function
--> 480 out = func(self, *args, **kwargs)
...
-> 5666 indices_array = [i for i, to_keep in zip(indices, mask) if to_keep]
   5667 if indices_mapping is not None:
   5668     indices_array = pa.array(indices_array, type=pa.uint64())

TypeError: 'bool' object is not iterable
```

**Removing batched=True allows to bypass the issue.**

### Expected behavior

According to the doc, "[batch_size corresponds to the] number of examples per batch provided to function if batched = True", so we shouldn't need to remove the batchd=True arg?

source: https://huggingface.co/docs/datasets/v2.9.0/en/package_reference/main_classes#datasets.Dataset.filter

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-5.4.0-122-generic-x86_64-with-glibc2.31
- Python version: 3.9.10
- PyArrow version: 10.0.1
- Pandas version: 1.5.3


Improved error message for gated/private repos

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009491 / 0.011353 (-0.001862) | 0.004690 / 0.011008 (-0.006319) | 0.111904 / 0.038508 (0.073396) | 0.030781 / 0.023109 (0.007671) | 0.309442 / 0.275898 (0.033544) | 0.389511 / 0.323480 (0.066031) | 0.007277 / 0.007986 (-0.000709) | 0.004364 / 0.004328 (0.000036) | 0.074501 / 0.004250 (0.070250) | 0.036799 / 0.037052 (-0.000254) | 0.320279 / 0.258489 (0.061790) | 0.353887 / 0.293841 (0.060046) | 0.047969 / 0.128546 (-0.080577) | 0.017281 / 0.075646 (-0.058366) | 0.339655 / 0.419271 (-0.079617) | 0.049317 / 0.043533 (0.005784) | 0.321221 / 0.255139 (0.066082) | 0.354743 / 0.283200 (0.071544) | 0.098634 / 0.141683 (-0.043049) | 1.408640 / 1.452155 (-0.043515) | 1.488361 / 1.492716 (-0.004356) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.233677 / 0.018006 (0.215671) | 0.604424 / 0.000490 (0.603934) | 0.003834 / 0.000200 (0.003634) | 0.000103 / 0.000054 (0.000049) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022682 / 0.037411 (-0.014729) | 0.103800 / 0.014526 (0.089274) | 0.113868 / 0.176557 (-0.062689) | 0.155111 / 0.737135 (-0.582025) | 0.111862 / 0.296338 (-0.184476) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.474992 / 0.215209 (0.259783) | 4.755325 / 2.077655 (2.677670) | 1.889754 / 1.504120 (0.385634) | 1.597009 / 1.541195 (0.055814) | 1.639570 / 1.468490 (0.171080) | 0.970681 / 4.584777 (-3.614096) | 4.782567 / 3.745712 (1.036855) | 4.350465 / 5.269862 (-0.919397) | 2.413533 / 4.565676 (-2.152144) | 0.115510 / 0.424275 (-0.308765) | 0.011663 / 0.007607 (0.004055) | 0.626450 / 0.226044 (0.400406) | 6.238147 / 2.268929 (3.969218) | 2.603070 / 55.444624 (-52.841555) | 2.030378 / 6.876477 (-4.846099) | 1.996883 / 2.142072 (-0.145190) | 1.206436 / 4.805227 (-3.598792) | 0.203018 / 6.500664 (-6.297646) | 0.060550 / 0.075469 (-0.014919) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.259850 / 1.841788 (-0.581937) | 14.079936 / 8.074308 (6.005628) | 16.036329 / 10.191392 (5.844937) | 0.221546 / 0.680424 (-0.458878) | 0.042416 / 0.534201 (-0.491785) | 0.438851 / 0.579283 (-0.140432) | 0.507053 / 0.434364 (0.072689) | 0.518672 / 0.540337 (-0.021665) | 0.585278 / 1.386936 (-0.801659) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010718 / 0.011353 (-0.000635) | 0.005469 / 0.011008 (-0.005539) | 0.075624 / 0.038508 (0.037116) | 0.029103 / 0.023109 (0.005994) | 0.353294 / 0.275898 (0.077395) | 0.353674 / 0.323480 (0.030194) | 0.005678 / 0.007986 (-0.002308) | 0.004610 / 0.004328 (0.000282) | 0.075213 / 0.004250 (0.070963) | 0.040032 / 0.037052 (0.002980) | 0.344363 / 0.258489 (0.085874) | 0.376861 / 0.293841 (0.083020) | 0.043718 / 0.128546 (-0.084828) | 0.016057 / 0.075646 (-0.059589) | 0.087746 / 0.419271 (-0.331526) | 0.051380 / 0.043533 (0.007848) | 0.336904 / 0.255139 (0.081765) | 0.357636 / 0.283200 (0.074436) | 0.089425 / 0.141683 (-0.052258) | 1.377462 / 1.452155 (-0.074692) | 1.448844 / 1.492716 (-0.043872) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.259038 / 0.018006 (0.241031) | 0.512284 / 0.000490 (0.511794) | 0.005666 / 0.000200 (0.005466) | 0.000123 / 0.000054 (0.000068) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023669 / 0.037411 (-0.013742) | 0.097979 / 0.014526 (0.083453) | 0.117947 / 0.176557 (-0.058610) | 0.140764 / 0.737135 (-0.596372) | 0.114700 / 0.296338 (-0.181638) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.528844 / 0.215209 (0.313635) | 5.073828 / 2.077655 (2.996173) | 2.088738 / 1.504120 (0.584618) | 1.855820 / 1.541195 (0.314626) | 1.838639 / 1.468490 (0.370149) | 0.968228 / 4.584777 (-3.616549) | 4.589792 / 3.745712 (0.844079) | 2.586149 / 5.269862 (-2.683712) | 1.714241 / 4.565676 (-2.851435) | 0.124502 / 0.424275 (-0.299774) | 0.012115 / 0.007607 (0.004507) | 0.679539 / 0.226044 (0.453494) | 6.541335 / 2.268929 (4.272407) | 2.749153 / 55.444624 (-52.695471) | 2.124164 / 6.876477 (-4.752313) | 2.181249 / 2.142072 (0.039177) | 1.196846 / 4.805227 (-3.608381) | 0.213352 / 6.500664 (-6.287312) | 0.075021 / 0.075469 (-0.000448) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.254301 / 1.841788 (-0.587487) | 14.494254 / 8.074308 (6.419946) | 16.619679 / 10.191392 (6.428287) | 0.205158 / 0.680424 (-0.475266) | 0.022181 / 0.534201 (-0.512019) | 0.422928 / 0.579283 (-0.156355) | 0.539825 / 0.434364 (0.105461) | 0.523165 / 0.540337 (-0.017173) | 0.615014 / 1.386936 (-0.771922) |

![](https://cml.dev/watermark.png#e4d8a3d43569d61e73f7ab12ff3a6b48466afa8d "CML watermark")

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.011522 / 0.011353 (0.000169) | 0.006906 / 0.011008 (-0.004102) | 0.114692 / 0.038508 (0.076184) | 0.037686 / 0.023109 (0.014577) | 0.393662 / 0.275898 (0.117764) | 0.377730 / 0.323480 (0.054250) | 0.008212 / 0.007986 (0.000226) | 0.005470 / 0.004328 (0.001142) | 0.086962 / 0.004250 (0.082712) | 0.039085 / 0.037052 (0.002033) | 0.357565 / 0.258489 (0.099076) | 0.404384 / 0.293841 (0.110543) | 0.055523 / 0.128546 (-0.073023) | 0.018277 / 0.075646 (-0.057369) | 0.389812 / 0.419271 (-0.029459) | 0.058706 / 0.043533 (0.015173) | 0.344735 / 0.255139 (0.089597) | 0.395734 / 0.283200 (0.112535) | 0.096098 / 0.141683 (-0.045584) | 1.546654 / 1.452155 (0.094499) | 1.665314 / 1.492716 (0.172597) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.255893 / 0.018006 (0.237887) | 0.589563 / 0.000490 (0.589074) | 0.005890 / 0.000200 (0.005690) | 0.000123 / 0.000054 (0.000069) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029167 / 0.037411 (-0.008245) | 0.113561 / 0.014526 (0.099036) | 0.125361 / 0.176557 (-0.051195) | 0.182225 / 0.737135 (-0.554910) | 0.125147 / 0.296338 (-0.171192) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.596859 / 0.215209 (0.381650) | 5.797725 / 2.077655 (3.720071) | 2.238420 / 1.504120 (0.734300) | 1.933177 / 1.541195 (0.391982) | 2.030750 / 1.468490 (0.562260) | 1.122655 / 4.584777 (-3.462122) | 5.247913 / 3.745712 (1.502201) | 2.792742 / 5.269862 (-2.477120) | 1.861487 / 4.565676 (-2.704190) | 0.133009 / 0.424275 (-0.291266) | 0.013219 / 0.007607 (0.005612) | 0.696905 / 0.226044 (0.470861) | 6.961298 / 2.268929 (4.692369) | 2.895352 / 55.444624 (-52.549273) | 2.353677 / 6.876477 (-4.522799) | 2.458804 / 2.142072 (0.316731) | 1.271905 / 4.805227 (-3.533322) | 0.224850 / 6.500664 (-6.275814) | 0.083773 / 0.075469 (0.008304) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.502425 / 1.841788 (-0.339363) | 16.959241 / 8.074308 (8.884933) | 19.865569 / 10.191392 (9.674177) | 0.228608 / 0.680424 (-0.451816) | 0.044035 / 0.534201 (-0.490166) | 0.545172 / 0.579283 (-0.034112) | 0.677193 / 0.434364 (0.242829) | 0.608988 / 0.540337 (0.068650) | 0.719210 / 1.386936 (-0.667726) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008297 / 0.011353 (-0.003056) | 0.005729 / 0.011008 (-0.005280) | 0.084762 / 0.038508 (0.046254) | 0.030622 / 0.023109 (0.007512) | 0.408017 / 0.275898 (0.132119) | 0.432114 / 0.323480 (0.108634) | 0.006965 / 0.007986 (-0.001021) | 0.004830 / 0.004328 (0.000502) | 0.087375 / 0.004250 (0.083124) | 0.048110 / 0.037052 (0.011058) | 0.414978 / 0.258489 (0.156489) | 0.446136 / 0.293841 (0.152295) | 0.064351 / 0.128546 (-0.064195) | 0.018273 / 0.075646 (-0.057374) | 0.114853 / 0.419271 (-0.304418) | 0.056962 / 0.043533 (0.013429) | 0.427791 / 0.255139 (0.172652) | 0.428829 / 0.283200 (0.145629) | 0.108004 / 0.141683 (-0.033679) | 1.639285 / 1.452155 (0.187130) | 1.652106 / 1.492716 (0.159390) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.359744 / 0.018006 (0.341738) | 0.596060 / 0.000490 (0.595570) | 0.025448 / 0.000200 (0.025248) | 0.000158 / 0.000054 (0.000104) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026348 / 0.037411 (-0.011064) | 0.119153 / 0.014526 (0.104628) | 0.129304 / 0.176557 (-0.047253) | 0.195670 / 0.737135 (-0.541465) | 0.135559 / 0.296338 (-0.160780) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.588963 / 0.215209 (0.373754) | 5.682957 / 2.077655 (3.605302) | 2.380178 / 1.504120 (0.876059) | 2.131299 / 1.541195 (0.590104) | 2.167839 / 1.468490 (0.699349) | 1.126418 / 4.584777 (-3.458359) | 5.289104 / 3.745712 (1.543392) | 2.952128 / 5.269862 (-2.317734) | 1.922974 / 4.565676 (-2.642702) | 0.143874 / 0.424275 (-0.280401) | 0.015399 / 0.007607 (0.007792) | 0.815675 / 0.226044 (0.589631) | 7.320146 / 2.268929 (5.051217) | 3.453670 / 55.444624 (-51.990954) | 2.579133 / 6.876477 (-4.297344) | 2.532331 / 2.142072 (0.390258) | 1.345881 / 4.805227 (-3.459347) | 0.242448 / 6.500664 (-6.258216) | 0.070007 / 0.075469 (-0.005462) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.433173 / 1.841788 (-0.408614) | 17.127287 / 8.074308 (9.052979) | 17.953878 / 10.191392 (7.762486) | 0.220035 / 0.680424 (-0.460389) | 0.028660 / 0.534201 (-0.505541) | 0.496233 / 0.579283 (-0.083050) | 0.591587 / 0.434364 (0.157223) | 0.635204 / 0.540337 (0.094867) | 0.702143 / 1.386936 (-0.684793) |

![](https://cml.dev/watermark.png#7cfac43b980ab9e4a69c2328f085770996323005 "CML watermark")

Using `use_auth_token=True` is not needed anymore. If a user logged in, the token will be automatically retrieved. Also include a mention for gated repos

See https://github.com/huggingface/huggingface_hub/pull/1064


Add a `reduce` method

New feature or request

Hi! Sure, feel free to open a PR, so we can see the API you have in mind.

I would like to give it a go! #self-assign

Closing as `Dataset.map` can be used instead (see https://github.com/huggingface/datasets/pull/5533#issuecomment-1440571658 and https://github.com/huggingface/datasets/pull/5533#issuecomment-1446403263)

### Feature request

Right now the `Dataset` class implements `map()` and `filter()`, but leaves out the third functional idiom popular among Python users: `reduce`.

### Motivation

A `reduce` method is often useful when calculating dataset statistics, for example, the occurrence of a particular n-gram or the average line length of a code dataset.

### Your contribution

I haven't contributed to `datasets` before, but I don't expect this will be too difficult, since the implementation will closely follow that of `map` and `filter`. I could have a crack over the weekend.


to_tf_dataset fails with datetime UTC columns even if not included in columns argument

bug

Something isn't working

good first issue

Good for newcomers

Hi! This is indeed a bug in our zero-copy logic.

To fix it, instead of the line:
https://github.com/huggingface/datasets/blob/7cfac43b980ab9e4a69c2328f085770996323005/src/datasets/features/features.py#L702

we should have:
```python
return pa.types.is_primitive(pa_type) and not (pa.types.is_boolean(pa_type) or pa.types.is_temporal(pa_type))
```

@mariosasko submitted a small PR [here](https://github.com/huggingface/datasets/pull/5504)

### Describe the bug

There appears to be some eager behavior in `to_tf_dataset` that runs against every column in a dataset even if they aren't included in the columns argument. This is problematic with datetime UTC columns due to them not working with zero copy. If I don't have UTC information in my datetime column, then everything works as expected.

### Steps to reproduce the bug

```python
import numpy as np
import pandas as pd
from datasets import Dataset

df = pd.DataFrame(np.random.rand(2, 1), columns=["x"])
# df["dt"] = pd.to_datetime(["2023-01-01", "2023-01-01"])  # works fine
df["dt"] = pd.to_datetime(["2023-01-01 00:00:00.00000+00:00", "2023-01-01 00:00:00.00000+00:00"])
df.to_parquet("test.pq")

ds = Dataset.from_parquet("test.pq")
tf_ds = ds.to_tf_dataset(columns=["x"], batch_size=2, shuffle=True)
```

```
ArrowInvalid                              Traceback (most recent call last)
Cell In[1], line 12
      8 df.to_parquet("test.pq")
     11 ds = Dataset.from_parquet("test.pq")
---> 12 tf_ds = ds.to_tf_dataset(columns=["r"], batch_size=2, shuffle=True)

File ~/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py:411, in TensorflowDatasetMixin.to_tf_dataset(self, batch_size, columns, shuffle, collate_fn, drop_remainder, collate_fn_args, label_cols, prefetch, num_workers)
    407     dataset = self
    409 # TODO(Matt, QL): deprecate the retention of label_ids and label
--> 411 output_signature, columns_to_np_types = dataset._get_output_signature(
    412     dataset,
    413     collate_fn=collate_fn,
    414     collate_fn_args=collate_fn_args,
    415     cols_to_retain=cols_to_retain,
    416     batch_size=batch_size if drop_remainder else None,
    417 )
    419 if "labels" in output_signature:
    420     if ("label_ids" in columns or "label" in columns) and "labels" not in columns:

File ~/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py:254, in TensorflowDatasetMixin._get_output_signature(dataset, collate_fn, collate_fn_args, cols_to_retain, batch_size, num_test_batches)
    252 for _ in range(num_test_batches):
    253     indices = sample(range(len(dataset)), test_batch_size)
--> 254     test_batch = dataset[indices]
    255     if cols_to_retain is not None:
    256         test_batch = {key: value for key, value in test_batch.items() if key in cols_to_retain}

File ~/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py:2590, in Dataset.__getitem__(self, key)
   2588 def __getitem__(self, key):  # noqa: F811
   2589     """Can be used to index columns (by string names) or rows (by integer index or iterable of indices or bools)."""
-> 2590     return self._getitem(
   2591         key,
   2592     )

File ~/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py:2575, in Dataset._getitem(self, key, **kwargs)
   2573 formatter = get_formatter(format_type, features=self.features, **format_kwargs)
   2574 pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
-> 2575 formatted_output = format_table(
   2576     pa_subtable, key, formatter=formatter, format_columns=format_columns, output_all_columns=output_all_columns
   2577 )
   2578 return formatted_output

File ~/venv/lib/python3.8/site-packages/datasets/formatting/formatting.py:634, in format_table(table, key, formatter, format_columns, output_all_columns)
    632 python_formatter = PythonFormatter(features=None)
    633 if format_columns is None:
--> 634     return formatter(pa_table, query_type=query_type)
    635 elif query_type == "column":
    636     if key in format_columns:

File ~/venv/lib/python3.8/site-packages/datasets/formatting/formatting.py:410, in Formatter.__call__(self, pa_table, query_type)
    408     return self.format_column(pa_table)
    409 elif query_type == "batch":
--> 410     return self.format_batch(pa_table)

File ~/venv/lib/python3.8/site-packages/datasets/formatting/np_formatter.py:78, in NumpyFormatter.format_batch(self, pa_table)
     77 def format_batch(self, pa_table: pa.Table) -> Mapping:
---> 78     batch = self.numpy_arrow_extractor().extract_batch(pa_table)
     79     batch = self.python_features_decoder.decode_batch(batch)
     80     batch = self.recursive_tensorize(batch)

File ~/venv/lib/python3.8/site-packages/datasets/formatting/formatting.py:164, in NumpyArrowExtractor.extract_batch(self, pa_table)
    163 def extract_batch(self, pa_table: pa.Table) -> dict:
--> 164     return {col: self._arrow_array_to_numpy(pa_table[col]) for col in pa_table.column_names}

File ~/venv/lib/python3.8/site-packages/datasets/formatting/formatting.py:164, in <dictcomp>(.0)
    163 def extract_batch(self, pa_table: pa.Table) -> dict:
--> 164     return {col: self._arrow_array_to_numpy(pa_table[col]) for col in pa_table.column_names}

File ~/venv/lib/python3.8/site-packages/datasets/formatting/formatting.py:185, in NumpyArrowExtractor._arrow_array_to_numpy(self, pa_array)
    181     else:
    182         zero_copy_only = _is_zero_copy_only(pa_array.type) and all(
    183             not _is_array_with_nulls(chunk) for chunk in pa_array.chunks
    184         )
--> 185         array: List = [
    186             row for chunk in pa_array.chunks for row in chunk.to_numpy(zero_copy_only=zero_copy_only)
    187         ]
    188 else:
    189     if isinstance(pa_array.type, _ArrayXDExtensionType):
    190         # don't call to_pylist() to preserve dtype of the fixed-size array

File ~/venv/lib/python3.8/site-packages/datasets/formatting/formatting.py:186, in <listcomp>(.0)
    181     else:
    182         zero_copy_only = _is_zero_copy_only(pa_array.type) and all(
    183             not _is_array_with_nulls(chunk) for chunk in pa_array.chunks
    184         )
    185         array: List = [
--> 186             row for chunk in pa_array.chunks for row in chunk.to_numpy(zero_copy_only=zero_copy_only)
    187         ]
    188 else:
    189     if isinstance(pa_array.type, _ArrayXDExtensionType):
    190         # don't call to_pylist() to preserve dtype of the fixed-size array

File ~/venv/lib/python3.8/site-packages/pyarrow/array.pxi:1475, in pyarrow.lib.Array.to_numpy()

File ~/venv/lib/python3.8/site-packages/pyarrow/error.pxi:100, in pyarrow.lib.check_status()

ArrowInvalid: Needed to copy 1 chunks with 0 nulls, but zero_copy_only was True
```

### Expected behavior

I think there are two potential issues/fixes
1. Proper handling of datetime UTC columns (perhaps there is something incorrect with zero copy handling here)
2. Not eagerly running against every column in a dataset when the columns argument of `to_tf_dataset` specifies a subset of columns (although I'm not sure if this is unavoidable)

### Environment info

- `datasets` version: 2.9.0
- Platform: macOS-13.2-x86_64-i386-64bit
- Python version: 3.8.12
- PyArrow version: 11.0.0
- Pandas version: 1.5.3


Update audio installation doc page

documentation

Improvements or additions to documentation

Totally agree, the docs should be in sync with our code.

Indeed to avoid confusing users, I think we should have updated the docs at the same time as this PR:
- #5167

@albertvillanova yeah sure I should have, but I forgot back then, sorry for that 😶

No, @polinaeterna, nothing to be sorry about.

My comment was for all of us datasets team, as a reminder: when making a PR, but also when reviewing some other's PR, we should not forget to update the corresponding docstring and doc pages. It is something we can improve if we help each other in reminding about it... :hugs:

@polinaeterna I think we can close this issue now as we no longer use `torchaudio` for decoding.

Our [installation documentation page](https://huggingface.co/docs/datasets/installation#audio) says that one can use Datasets for mp3 only with `torchaudio<0.12`. `torchaudio>0.12` is actually supported too but requires a specific version of ffmpeg which is not easily installed on all linux versions but there is a custom ubuntu repo for it, we have insctructions in the code: https://github.com/huggingface/datasets/blob/main/src/datasets/features/audio.py#L327
So we should update the doc page. But first investigate [this issue](5488).


Remove unused `load_from_cache_file` arg from `Dataset.shard()` docstring

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5493). All of your documentation changes will be reflected on that endpoint.

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008956 / 0.011353 (-0.002397) | 0.004590 / 0.011008 (-0.006418) | 0.101305 / 0.038508 (0.062797) | 0.030347 / 0.023109 (0.007237) | 0.302492 / 0.275898 (0.026594) | 0.335986 / 0.323480 (0.012506) | 0.007272 / 0.007986 (-0.000714) | 0.004303 / 0.004328 (-0.000025) | 0.078592 / 0.004250 (0.074341) | 0.035545 / 0.037052 (-0.001507) | 0.316052 / 0.258489 (0.057563) | 0.342523 / 0.293841 (0.048682) | 0.034128 / 0.128546 (-0.094419) | 0.011475 / 0.075646 (-0.064171) | 0.325272 / 0.419271 (-0.093999) | 0.041815 / 0.043533 (-0.001717) | 0.303093 / 0.255139 (0.047955) | 0.331987 / 0.283200 (0.048788) | 0.087264 / 0.141683 (-0.054419) | 1.476284 / 1.452155 (0.024129) | 1.562034 / 1.492716 (0.069318) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.206502 / 0.018006 (0.188496) | 0.409893 / 0.000490 (0.409404) | 0.002479 / 0.000200 (0.002279) | 0.000073 / 0.000054 (0.000018) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022891 / 0.037411 (-0.014520) | 0.100209 / 0.014526 (0.085683) | 0.105576 / 0.176557 (-0.070981) | 0.141035 / 0.737135 (-0.596100) | 0.109733 / 0.296338 (-0.186606) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.413791 / 0.215209 (0.198582) | 4.125890 / 2.077655 (2.048235) | 1.833023 / 1.504120 (0.328903) | 1.631325 / 1.541195 (0.090130) | 1.708406 / 1.468490 (0.239916) | 0.690100 / 4.584777 (-3.894677) | 3.379058 / 3.745712 (-0.366654) | 2.019044 / 5.269862 (-3.250818) | 1.323332 / 4.565676 (-3.242344) | 0.082709 / 0.424275 (-0.341566) | 0.012434 / 0.007607 (0.004827) | 0.527139 / 0.226044 (0.301095) | 5.271529 / 2.268929 (3.002601) | 2.297311 / 55.444624 (-53.147314) | 1.949021 / 6.876477 (-4.927456) | 2.001098 / 2.142072 (-0.140975) | 0.811591 / 4.805227 (-3.993636) | 0.149028 / 6.500664 (-6.351637) | 0.066233 / 0.075469 (-0.009236) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.254276 / 1.841788 (-0.587512) | 13.638485 / 8.074308 (5.564177) | 13.943274 / 10.191392 (3.751882) | 0.147426 / 0.680424 (-0.532997) | 0.028602 / 0.534201 (-0.505599) | 0.398080 / 0.579283 (-0.181203) | 0.402178 / 0.434364 (-0.032186) | 0.477045 / 0.540337 (-0.063292) | 0.567731 / 1.386936 (-0.819205) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006936 / 0.011353 (-0.004417) | 0.004614 / 0.011008 (-0.006394) | 0.079779 / 0.038508 (0.041271) | 0.027941 / 0.023109 (0.004832) | 0.347224 / 0.275898 (0.071326) | 0.378183 / 0.323480 (0.054703) | 0.005249 / 0.007986 (-0.002737) | 0.004907 / 0.004328 (0.000579) | 0.078678 / 0.004250 (0.074428) | 0.041912 / 0.037052 (0.004860) | 0.347838 / 0.258489 (0.089349) | 0.386760 / 0.293841 (0.092919) | 0.032680 / 0.128546 (-0.095867) | 0.014321 / 0.075646 (-0.061325) | 0.087924 / 0.419271 (-0.331347) | 0.045060 / 0.043533 (0.001527) | 0.340986 / 0.255139 (0.085847) | 0.368689 / 0.283200 (0.085489) | 0.093274 / 0.141683 (-0.048409) | 1.474435 / 1.452155 (0.022281) | 1.569753 / 1.492716 (0.077037) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.206789 / 0.018006 (0.188783) | 0.416518 / 0.000490 (0.416028) | 0.000404 / 0.000200 (0.000204) | 0.000059 / 0.000054 (0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026207 / 0.037411 (-0.011205) | 0.101914 / 0.014526 (0.087388) | 0.108585 / 0.176557 (-0.067972) | 0.150438 / 0.737135 (-0.586697) | 0.110744 / 0.296338 (-0.185594) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.443571 / 0.215209 (0.228362) | 4.433139 / 2.077655 (2.355485) | 2.109525 / 1.504120 (0.605405) | 1.901484 / 1.541195 (0.360290) | 1.968812 / 1.468490 (0.500322) | 0.704334 / 4.584777 (-3.880443) | 3.392028 / 3.745712 (-0.353684) | 3.072693 / 5.269862 (-2.197168) | 1.552227 / 4.565676 (-3.013449) | 0.083741 / 0.424275 (-0.340534) | 0.012627 / 0.007607 (0.005020) | 0.544706 / 0.226044 (0.318662) | 5.462743 / 2.268929 (3.193815) | 2.551265 / 55.444624 (-52.893360) | 2.208075 / 6.876477 (-4.668401) | 2.259092 / 2.142072 (0.117020) | 0.810687 / 4.805227 (-3.994540) | 0.152347 / 6.500664 (-6.348317) | 0.068346 / 0.075469 (-0.007123) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.269716 / 1.841788 (-0.572072) | 14.215698 / 8.074308 (6.141390) | 13.691773 / 10.191392 (3.500381) | 0.152620 / 0.680424 (-0.527804) | 0.017219 / 0.534201 (-0.516982) | 0.382533 / 0.579283 (-0.196750) | 0.388994 / 0.434364 (-0.045370) | 0.479400 / 0.540337 (-0.060938) | 0.572699 / 1.386936 (-0.814237) |

![](https://cml.dev/watermark.png#f2d90f14cd6e756abeb27045940a6756104cc2d6 "CML watermark")


Push_to_hub in a pull request

enhancement

New feature or request

good first issue

Good for newcomers

Assigned to myself and will get to it in the next week, but if someone finds this issue annoying and wants to submit a PR before I do, just ping me here and I'll reassign :).

I would like to be assigned to this issue, @nateraw . #self-assign

Right now `ds.push_to_hub()` can push a dataset on `main` or on a new branch with `branch=`, but there is no way to open a pull request. Even passing `branch=refs/pr/x` doesn't seem to work: it tries to create a branch with that name

cc @nateraw

It should be possible to tweak the use of `huggingface_hub` in `push_to_hub` to make it open a PR or push to an existing PR


[MINOR] Typo

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008726 / 0.011353 (-0.002627) | 0.004589 / 0.011008 (-0.006419) | 0.101078 / 0.038508 (0.062570) | 0.029732 / 0.023109 (0.006622) | 0.298309 / 0.275898 (0.022411) | 0.367800 / 0.323480 (0.044320) | 0.007025 / 0.007986 (-0.000961) | 0.003513 / 0.004328 (-0.000815) | 0.079531 / 0.004250 (0.075281) | 0.035588 / 0.037052 (-0.001465) | 0.307850 / 0.258489 (0.049361) | 0.351603 / 0.293841 (0.057762) | 0.033593 / 0.128546 (-0.094954) | 0.011669 / 0.075646 (-0.063977) | 0.323025 / 0.419271 (-0.096246) | 0.042047 / 0.043533 (-0.001486) | 0.300565 / 0.255139 (0.045426) | 0.329362 / 0.283200 (0.046163) | 0.089001 / 0.141683 (-0.052682) | 1.472799 / 1.452155 (0.020644) | 1.488902 / 1.492716 (-0.003814) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.012491 / 0.018006 (-0.005515) | 0.408245 / 0.000490 (0.407755) | 0.003878 / 0.000200 (0.003678) | 0.000078 / 0.000054 (0.000023) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023698 / 0.037411 (-0.013713) | 0.100442 / 0.014526 (0.085916) | 0.108233 / 0.176557 (-0.068323) | 0.145308 / 0.737135 (-0.591827) | 0.113121 / 0.296338 (-0.183218) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.420490 / 0.215209 (0.205281) | 4.179838 / 2.077655 (2.102183) | 2.156007 / 1.504120 (0.651887) | 1.911358 / 1.541195 (0.370163) | 1.867961 / 1.468490 (0.399471) | 0.685254 / 4.584777 (-3.899523) | 3.382386 / 3.745712 (-0.363326) | 3.285657 / 5.269862 (-1.984205) | 1.693878 / 4.565676 (-2.871798) | 0.081680 / 0.424275 (-0.342595) | 0.012182 / 0.007607 (0.004575) | 0.526021 / 0.226044 (0.299977) | 5.276217 / 2.268929 (3.007289) | 2.541518 / 55.444624 (-52.903106) | 2.313452 / 6.876477 (-4.563025) | 2.340000 / 2.142072 (0.197928) | 0.807099 / 4.805227 (-3.998128) | 0.147587 / 6.500664 (-6.353077) | 0.064280 / 0.075469 (-0.011189) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.223466 / 1.841788 (-0.618321) | 13.911365 / 8.074308 (5.837057) | 14.261550 / 10.191392 (4.070158) | 0.135922 / 0.680424 (-0.544502) | 0.028832 / 0.534201 (-0.505368) | 0.393142 / 0.579283 (-0.186141) | 0.400507 / 0.434364 (-0.033857) | 0.471792 / 0.540337 (-0.068546) | 0.558278 / 1.386936 (-0.828658) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006644 / 0.011353 (-0.004709) | 0.004531 / 0.011008 (-0.006478) | 0.076285 / 0.038508 (0.037777) | 0.027249 / 0.023109 (0.004140) | 0.343137 / 0.275898 (0.067239) | 0.378498 / 0.323480 (0.055018) | 0.004950 / 0.007986 (-0.003036) | 0.003422 / 0.004328 (-0.000907) | 0.075662 / 0.004250 (0.071412) | 0.039692 / 0.037052 (0.002640) | 0.343402 / 0.258489 (0.084913) | 0.385067 / 0.293841 (0.091226) | 0.032382 / 0.128546 (-0.096164) | 0.011577 / 0.075646 (-0.064069) | 0.085534 / 0.419271 (-0.333738) | 0.052139 / 0.043533 (0.008606) | 0.342176 / 0.255139 (0.087037) | 0.367298 / 0.283200 (0.084098) | 0.096088 / 0.141683 (-0.045595) | 1.470770 / 1.452155 (0.018615) | 1.567316 / 1.492716 (0.074600) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.217664 / 0.018006 (0.199657) | 0.397807 / 0.000490 (0.397317) | 0.006864 / 0.000200 (0.006664) | 0.000099 / 0.000054 (0.000044) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025064 / 0.037411 (-0.012348) | 0.100906 / 0.014526 (0.086380) | 0.107444 / 0.176557 (-0.069113) | 0.143679 / 0.737135 (-0.593457) | 0.112460 / 0.296338 (-0.183879) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.442634 / 0.215209 (0.227425) | 4.410687 / 2.077655 (2.333032) | 2.067445 / 1.504120 (0.563325) | 1.860569 / 1.541195 (0.319374) | 1.943523 / 1.468490 (0.475033) | 0.694585 / 4.584777 (-3.890192) | 3.375906 / 3.745712 (-0.369806) | 3.483334 / 5.269862 (-1.786528) | 1.437700 / 4.565676 (-3.127977) | 0.083138 / 0.424275 (-0.341137) | 0.012979 / 0.007607 (0.005372) | 0.536414 / 0.226044 (0.310370) | 5.379872 / 2.268929 (3.110943) | 2.517907 / 55.444624 (-52.926717) | 2.164772 / 6.876477 (-4.711705) | 2.212839 / 2.142072 (0.070767) | 0.799675 / 4.805227 (-4.005553) | 0.150253 / 6.500664 (-6.350411) | 0.067033 / 0.075469 (-0.008436) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.295592 / 1.841788 (-0.546196) | 14.372932 / 8.074308 (6.298623) | 13.618423 / 10.191392 (3.427031) | 0.141212 / 0.680424 (-0.539212) | 0.016933 / 0.534201 (-0.517268) | 0.385664 / 0.579283 (-0.193619) | 0.386919 / 0.434364 (-0.047445) | 0.477022 / 0.540337 (-0.063315) | 0.565158 / 1.386936 (-0.821778) |

![](https://cml.dev/watermark.png#38c715cc787a81d0fd894205b4b24aca2f45f84b "CML watermark")


Do not add index column by default when exporting to CSV

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008581 / 0.011353 (-0.002772) | 0.004519 / 0.011008 (-0.006490) | 0.099721 / 0.038508 (0.061213) | 0.029217 / 0.023109 (0.006107) | 0.298229 / 0.275898 (0.022331) | 0.332605 / 0.323480 (0.009125) | 0.006880 / 0.007986 (-0.001106) | 0.003324 / 0.004328 (-0.001005) | 0.078143 / 0.004250 (0.073892) | 0.034262 / 0.037052 (-0.002790) | 0.304162 / 0.258489 (0.045673) | 0.342351 / 0.293841 (0.048510) | 0.033387 / 0.128546 (-0.095159) | 0.011397 / 0.075646 (-0.064249) | 0.321527 / 0.419271 (-0.097744) | 0.040886 / 0.043533 (-0.002647) | 0.299968 / 0.255139 (0.044829) | 0.322484 / 0.283200 (0.039285) | 0.083832 / 0.141683 (-0.057851) | 1.482241 / 1.452155 (0.030086) | 1.548438 / 1.492716 (0.055721) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.191002 / 0.018006 (0.172996) | 0.403423 / 0.000490 (0.402933) | 0.002493 / 0.000200 (0.002293) | 0.000074 / 0.000054 (0.000020) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023720 / 0.037411 (-0.013691) | 0.100806 / 0.014526 (0.086281) | 0.105314 / 0.176557 (-0.071242) | 0.141490 / 0.737135 (-0.595645) | 0.108695 / 0.296338 (-0.187644) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.412250 / 0.215209 (0.197041) | 4.124830 / 2.077655 (2.047175) | 1.851948 / 1.504120 (0.347828) | 1.651597 / 1.541195 (0.110403) | 1.712486 / 1.468490 (0.243996) | 0.696634 / 4.584777 (-3.888143) | 3.304220 / 3.745712 (-0.441492) | 1.862776 / 5.269862 (-3.407086) | 1.159452 / 4.565676 (-3.406224) | 0.082930 / 0.424275 (-0.341345) | 0.012586 / 0.007607 (0.004979) | 0.524499 / 0.226044 (0.298455) | 5.249235 / 2.268929 (2.980307) | 2.293187 / 55.444624 (-53.151437) | 1.950101 / 6.876477 (-4.926376) | 2.008274 / 2.142072 (-0.133799) | 0.811641 / 4.805227 (-3.993586) | 0.148785 / 6.500664 (-6.351879) | 0.064461 / 0.075469 (-0.011008) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.232227 / 1.841788 (-0.609561) | 13.235896 / 8.074308 (5.161588) | 13.837420 / 10.191392 (3.646028) | 0.135586 / 0.680424 (-0.544838) | 0.028935 / 0.534201 (-0.505266) | 0.397064 / 0.579283 (-0.182220) | 0.393814 / 0.434364 (-0.040549) | 0.480450 / 0.540337 (-0.059887) | 0.561159 / 1.386936 (-0.825777) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006696 / 0.011353 (-0.004657) | 0.004528 / 0.011008 (-0.006480) | 0.077335 / 0.038508 (0.038827) | 0.027181 / 0.023109 (0.004072) | 0.345379 / 0.275898 (0.069481) | 0.372544 / 0.323480 (0.049064) | 0.006808 / 0.007986 (-0.001178) | 0.003284 / 0.004328 (-0.001045) | 0.077379 / 0.004250 (0.073129) | 0.039954 / 0.037052 (0.002901) | 0.348094 / 0.258489 (0.089605) | 0.382315 / 0.293841 (0.088474) | 0.031694 / 0.128546 (-0.096852) | 0.011714 / 0.075646 (-0.063933) | 0.086425 / 0.419271 (-0.332846) | 0.041778 / 0.043533 (-0.001754) | 0.342161 / 0.255139 (0.087022) | 0.363798 / 0.283200 (0.080599) | 0.091315 / 0.141683 (-0.050368) | 1.462066 / 1.452155 (0.009912) | 1.541417 / 1.492716 (0.048700) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.235840 / 0.018006 (0.217834) | 0.397096 / 0.000490 (0.396606) | 0.004597 / 0.000200 (0.004397) | 0.000079 / 0.000054 (0.000025) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024296 / 0.037411 (-0.013115) | 0.099167 / 0.014526 (0.084641) | 0.108257 / 0.176557 (-0.068299) | 0.143434 / 0.737135 (-0.593701) | 0.111933 / 0.296338 (-0.184406) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.440306 / 0.215209 (0.225096) | 4.374065 / 2.077655 (2.296410) | 2.072653 / 1.504120 (0.568533) | 1.864829 / 1.541195 (0.323635) | 1.927970 / 1.468490 (0.459479) | 0.710118 / 4.584777 (-3.874659) | 3.391216 / 3.745712 (-0.354496) | 1.888847 / 5.269862 (-3.381015) | 1.178740 / 4.565676 (-3.386936) | 0.083950 / 0.424275 (-0.340325) | 0.012567 / 0.007607 (0.004960) | 0.540557 / 0.226044 (0.314513) | 5.437621 / 2.268929 (3.168692) | 2.531165 / 55.444624 (-52.913460) | 2.181450 / 6.876477 (-4.695027) | 2.209108 / 2.142072 (0.067035) | 0.814236 / 4.805227 (-3.990991) | 0.153000 / 6.500664 (-6.347664) | 0.066769 / 0.075469 (-0.008700) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.301057 / 1.841788 (-0.540731) | 14.066786 / 8.074308 (5.992478) | 13.641455 / 10.191392 (3.450063) | 0.138838 / 0.680424 (-0.541586) | 0.016733 / 0.534201 (-0.517468) | 0.391823 / 0.579283 (-0.187460) | 0.390817 / 0.434364 (-0.043547) | 0.487682 / 0.540337 (-0.052656) | 0.581134 / 1.386936 (-0.805802) |

![](https://cml.dev/watermark.png#b065547654efa0ec633cf373ac1512884c68b2e1 "CML watermark")

As pointed out by @merveenoyan, default behavior of `Dataset.to_csv` adds the index as an additional column without name.

This PR changes the default behavior, so that now the index column is not written.

To add the index column, now you need to pass `index=True` and also `index_label=<name of the index colum>` to name that column.

CC: @merveenoyan