Don't expand_info in HF glob

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6469). All of your documentation changes will be reflected on that endpoint.

Merging this one for now, but lmk if you had other optimizations in mind for the next version of `huggingface_hub`

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004998 / 0.011353 (-0.006355) | 0.003523 / 0.011008 (-0.007486) | 0.064932 / 0.038508 (0.026424) | 0.050107 / 0.023109 (0.026998) | 0.253715 / 0.275898 (-0.022183) | 0.275364 / 0.323480 (-0.048116) | 0.003902 / 0.007986 (-0.004084) | 0.002716 / 0.004328 (-0.001612) | 0.048458 / 0.004250 (0.044208) | 0.037802 / 0.037052 (0.000750) | 0.262328 / 0.258489 (0.003839) | 0.285911 / 0.293841 (-0.007930) | 0.027112 / 0.128546 (-0.101435) | 0.010780 / 0.075646 (-0.064867) | 0.206447 / 0.419271 (-0.212824) | 0.035771 / 0.043533 (-0.007761) | 0.255031 / 0.255139 (-0.000108) | 0.270530 / 0.283200 (-0.012670) | 0.017152 / 0.141683 (-0.124530) | 1.094734 / 1.452155 (-0.357421) | 1.163480 / 1.492716 (-0.329237) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092944 / 0.018006 (0.074938) | 0.301042 / 0.000490 (0.300553) | 0.000238 / 0.000200 (0.000038) | 0.000049 / 0.000054 (-0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019090 / 0.037411 (-0.018321) | 0.061046 / 0.014526 (0.046520) | 0.073330 / 0.176557 (-0.103227) | 0.121124 / 0.737135 (-0.616012) | 0.080544 / 0.296338 (-0.215795) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.323866 / 0.215209 (0.108657) | 2.797727 / 2.077655 (0.720072) | 1.502994 / 1.504120 (-0.001126) | 1.376177 / 1.541195 (-0.165018) | 1.422741 / 1.468490 (-0.045749) | 0.562990 / 4.584777 (-4.021786) | 2.431781 / 3.745712 (-1.313931) | 2.783226 / 5.269862 (-2.486635) | 1.788055 / 4.565676 (-2.777621) | 0.064206 / 0.424275 (-0.360069) | 0.004989 / 0.007607 (-0.002618) | 0.338282 / 0.226044 (0.112237) | 3.356226 / 2.268929 (1.087297) | 1.855644 / 55.444624 (-53.588980) | 1.580876 / 6.876477 (-5.295601) | 1.617418 / 2.142072 (-0.524655) | 0.636816 / 4.805227 (-4.168411) | 0.117680 / 6.500664 (-6.382985) | 0.042560 / 0.075469 (-0.032909) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.956410 / 1.841788 (-0.885377) | 11.764886 / 8.074308 (3.690578) | 10.535801 / 10.191392 (0.344409) | 0.137797 / 0.680424 (-0.542627) | 0.014368 / 0.534201 (-0.519833) | 0.286213 / 0.579283 (-0.293070) | 0.267093 / 0.434364 (-0.167271) | 0.334802 / 0.540337 (-0.205535) | 0.441866 / 1.386936 (-0.945070) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005348 / 0.011353 (-0.006005) | 0.003551 / 0.011008 (-0.007458) | 0.049226 / 0.038508 (0.010718) | 0.052072 / 0.023109 (0.028963) | 0.268025 / 0.275898 (-0.007873) | 0.289968 / 0.323480 (-0.033512) | 0.004034 / 0.007986 (-0.003952) | 0.002675 / 0.004328 (-0.001653) | 0.048099 / 0.004250 (0.043848) | 0.040141 / 0.037052 (0.003089) | 0.272974 / 0.258489 (0.014485) | 0.296097 / 0.293841 (0.002256) | 0.028972 / 0.128546 (-0.099575) | 0.010689 / 0.075646 (-0.064957) | 0.057853 / 0.419271 (-0.361418) | 0.032488 / 0.043533 (-0.011045) | 0.272018 / 0.255139 (0.016879) | 0.287179 / 0.283200 (0.003980) | 0.018446 / 0.141683 (-0.123237) | 1.140346 / 1.452155 (-0.311809) | 1.247743 / 1.492716 (-0.244974) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091987 / 0.018006 (0.073980) | 0.300527 / 0.000490 (0.300037) | 0.000224 / 0.000200 (0.000024) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021390 / 0.037411 (-0.016021) | 0.068768 / 0.014526 (0.054242) | 0.080798 / 0.176557 (-0.095759) | 0.119081 / 0.737135 (-0.618054) | 0.082461 / 0.296338 (-0.213878) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.286631 / 0.215209 (0.071422) | 2.804633 / 2.077655 (0.726978) | 1.574122 / 1.504120 (0.070002) | 1.459994 / 1.541195 (-0.081201) | 1.499739 / 1.468490 (0.031249) | 0.579595 / 4.584777 (-4.005182) | 2.426407 / 3.745712 (-1.319306) | 2.917994 / 5.269862 (-2.351868) | 1.846439 / 4.565676 (-2.719238) | 0.063274 / 0.424275 (-0.361001) | 0.005028 / 0.007607 (-0.002579) | 0.341114 / 0.226044 (0.115070) | 3.402677 / 2.268929 (1.133748) | 1.940980 / 55.444624 (-53.503645) | 1.651902 / 6.876477 (-5.224575) | 1.677037 / 2.142072 (-0.465036) | 0.651576 / 4.805227 (-4.153651) | 0.116398 / 6.500664 (-6.384266) | 0.041060 / 0.075469 (-0.034409) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.973278 / 1.841788 (-0.868509) | 12.248332 / 8.074308 (4.174024) | 10.830627 / 10.191392 (0.639235) | 0.143146 / 0.680424 (-0.537278) | 0.016249 / 0.534201 (-0.517952) | 0.298563 / 0.579283 (-0.280720) | 0.278643 / 0.434364 (-0.155721) | 0.338206 / 0.540337 (-0.202132) | 0.589485 / 1.386936 (-0.797451) |

![](https://cml.dev/watermark.png#da29ac32c57e079199c173e4404342cc105ed774 "CML watermark")

Finally fix https://github.com/huggingface/datasets/issues/5537


Use auth to get parquet export

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6468). All of your documentation changes will be reflected on that endpoint.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005076 / 0.011353 (-0.006277) | 0.003510 / 0.011008 (-0.007499) | 0.062939 / 0.038508 (0.024431) | 0.049191 / 0.023109 (0.026082) | 0.259088 / 0.275898 (-0.016810) | 0.273523 / 0.323480 (-0.049957) | 0.003902 / 0.007986 (-0.004083) | 0.002699 / 0.004328 (-0.001630) | 0.049077 / 0.004250 (0.044827) | 0.037174 / 0.037052 (0.000121) | 0.256467 / 0.258489 (-0.002022) | 0.291235 / 0.293841 (-0.002606) | 0.028119 / 0.128546 (-0.100427) | 0.010404 / 0.075646 (-0.065243) | 0.205825 / 0.419271 (-0.213446) | 0.035741 / 0.043533 (-0.007792) | 0.253219 / 0.255139 (-0.001920) | 0.274986 / 0.283200 (-0.008214) | 0.018379 / 0.141683 (-0.123304) | 1.131139 / 1.452155 (-0.321016) | 1.175875 / 1.492716 (-0.316841) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090717 / 0.018006 (0.072710) | 0.299285 / 0.000490 (0.298796) | 0.000217 / 0.000200 (0.000017) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018678 / 0.037411 (-0.018733) | 0.060558 / 0.014526 (0.046032) | 0.073828 / 0.176557 (-0.102728) | 0.119302 / 0.737135 (-0.617833) | 0.075261 / 0.296338 (-0.221078) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.277018 / 0.215209 (0.061809) | 2.713255 / 2.077655 (0.635601) | 1.427512 / 1.504120 (-0.076608) | 1.311374 / 1.541195 (-0.229821) | 1.348756 / 1.468490 (-0.119734) | 0.561777 / 4.584777 (-4.023000) | 2.393578 / 3.745712 (-1.352134) | 2.798109 / 5.269862 (-2.471753) | 1.754808 / 4.565676 (-2.810869) | 0.062302 / 0.424275 (-0.361973) | 0.004948 / 0.007607 (-0.002659) | 0.328468 / 0.226044 (0.102423) | 3.246558 / 2.268929 (0.977629) | 1.786816 / 55.444624 (-53.657808) | 1.482937 / 6.876477 (-5.393540) | 1.516109 / 2.142072 (-0.625963) | 0.634457 / 4.805227 (-4.170770) | 0.116505 / 6.500664 (-6.384159) | 0.042162 / 0.075469 (-0.033308) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.935312 / 1.841788 (-0.906476) | 11.540599 / 8.074308 (3.466291) | 10.512593 / 10.191392 (0.321201) | 0.129638 / 0.680424 (-0.550786) | 0.013994 / 0.534201 (-0.520207) | 0.291490 / 0.579283 (-0.287793) | 0.263641 / 0.434364 (-0.170722) | 0.328718 / 0.540337 (-0.211619) | 0.437598 / 1.386936 (-0.949338) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005192 / 0.011353 (-0.006161) | 0.003454 / 0.011008 (-0.007554) | 0.049448 / 0.038508 (0.010940) | 0.050968 / 0.023109 (0.027859) | 0.273702 / 0.275898 (-0.002196) | 0.296934 / 0.323480 (-0.026545) | 0.004066 / 0.007986 (-0.003920) | 0.002611 / 0.004328 (-0.001718) | 0.048284 / 0.004250 (0.044034) | 0.041399 / 0.037052 (0.004346) | 0.283000 / 0.258489 (0.024511) | 0.302553 / 0.293841 (0.008712) | 0.029086 / 0.128546 (-0.099460) | 0.010510 / 0.075646 (-0.065137) | 0.058097 / 0.419271 (-0.361175) | 0.032992 / 0.043533 (-0.010541) | 0.271752 / 0.255139 (0.016613) | 0.293535 / 0.283200 (0.010335) | 0.016958 / 0.141683 (-0.124725) | 1.130126 / 1.452155 (-0.322028) | 1.187228 / 1.492716 (-0.305488) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092321 / 0.018006 (0.074315) | 0.302599 / 0.000490 (0.302109) | 0.000215 / 0.000200 (0.000015) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021837 / 0.037411 (-0.015574) | 0.071148 / 0.014526 (0.056622) | 0.082448 / 0.176557 (-0.094108) | 0.128083 / 0.737135 (-0.609053) | 0.090864 / 0.296338 (-0.205474) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.296248 / 0.215209 (0.081039) | 2.881130 / 2.077655 (0.803476) | 1.580360 / 1.504120 (0.076240) | 1.454642 / 1.541195 (-0.086553) | 1.461453 / 1.468490 (-0.007037) | 0.567500 / 4.584777 (-4.017277) | 2.493708 / 3.745712 (-1.252004) | 2.756623 / 5.269862 (-2.513239) | 1.771319 / 4.565676 (-2.794358) | 0.062287 / 0.424275 (-0.361988) | 0.004917 / 0.007607 (-0.002691) | 0.348034 / 0.226044 (0.121990) | 3.426938 / 2.268929 (1.158010) | 1.954190 / 55.444624 (-53.490435) | 1.660870 / 6.876477 (-5.215607) | 1.675118 / 2.142072 (-0.466955) | 0.636843 / 4.805227 (-4.168384) | 0.115028 / 6.500664 (-6.385636) | 0.040702 / 0.075469 (-0.034767) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.988076 / 1.841788 (-0.853711) | 11.890867 / 8.074308 (3.816559) | 10.621169 / 10.191392 (0.429777) | 0.131568 / 0.680424 (-0.548856) | 0.014994 / 0.534201 (-0.519207) | 0.288900 / 0.579283 (-0.290384) | 0.272092 / 0.434364 (-0.162272) | 0.329397 / 0.540337 (-0.210940) | 0.569337 / 1.386936 (-0.817599) |

![](https://cml.dev/watermark.png#ae3b4a2268adc2f21568ff63891e9a83530c7e29 "CML watermark")

added `token` to the `_datasets_server` functions


New version release request

enhancement

New feature or request

We will publish it soon (we usually do it in intervals of 1-2 months, so probably next week)

### Feature request

Hi!
I am using `datasets` in library `xtuner` and am highly interested in the features introduced since v2.15.0.

To avoid installation from source in our pypi wheels, we are eagerly waiting for the new release. So, Does your team have a new release plan for v2.15.1 and could you please share it with us?

Thanks very much!

### Motivation

.

### Your contribution

.


Can't align optional features of struct

Friendly bump, I would be happy to work on this issue once I get the go-ahead from the dev team.

Thanks for the PR!

I'm struggling with this as well and would love to see this PR merged. My case is slightly different, with keys completely missing rather than being `None`:

```
ds = Dataset.from_dict({'speaker': [{'name': 'Ben'}]})
ds2 = Dataset.from_dict({'speaker': [{'name': 'Fred', 'email': 'abc@aol.com'}]})
print(concatenate_datasets([ds, ds2]).features)
print(concatenate_datasets([ds, ds2]).to_dict())
```

I would expect this to work as well because other Dataset functions already handle this situation well. For example, this works just as expected:

```
ds = Dataset.from_dict({'n': [1,2]})
ds_mapped = ds.map(lambda x: {
    'speaker': {'name': 'Ben'} if x['n'] == 1 else {'name': 'Fred', 'email': 'abc@aol.com'}
})
print(ds_mapped)
```

@vova-cyberhaven can you check with the new release if it fixes your issue?

### Describe the bug

Hello!

I'm currently experiencing an issue where I can't concatenate datasets if an inner field of a Feature is Optional.

I have a column named `speaker`, and this holds some information about a speaker.

```python
@dataclass
class Speaker:
   name: str
   email: Optional[str]
```

If I have two datasets, one happens to have `email` always None, then I get `The features can't be aligned because the key email of features`

### Steps to reproduce the bug

You can run the following script:

```python

ds = Dataset.from_dict({'speaker': [{'name': 'Ben', 'email': None}]})
ds2 = Dataset.from_dict({'speaker': [{'name': 'Fred', 'email': 'abc@aol.com'}]})
concatenate_datasets([ds, ds2])

>>>The features can't be aligned because the key speaker of features {'speaker': {'email': Value(dtype='string', id=None), 'name': Value(dtype='string', id=None)}} has unexpected type - {'email': Value(dtype='string', id=None), 'name': Value(dtype='string', id=None)} (expected either {'email': Value(dtype='null', id=None), 'name': Value(dtype='string', id=None)} or Value("null").

```

### Expected behavior

I think this should work; if two top-level columns were in the same situation it would properly cast to `string`.

```python
ds = Dataset.from_dict({'email': [None, None]})
ds2 = Dataset.from_dict({'email': ['abc@aol.com', 'one@yahoo.com']})
concatenate_datasets([ds, ds2])
>>> # Works!
```

### Environment info

- `datasets` version: 2.15.1.dev0
- Platform: Linux-5.15.0-89-generic-x86_64-with-glibc2.35
- Python version: 3.9.13
- `huggingface_hub` version: 0.19.4
- PyArrow version: 9.0.0
- Pandas version: 1.4.4
- `fsspec` version: 2023.6.0

I would be happy to fix this issue.


`load_dataset` uses out-of-date cache instead of re-downloading a changed dataset

Hi, thanks for reporting! https://github.com/huggingface/datasets/pull/6459 will fix this.

### Describe the bug

When a dataset is updated on the hub, using `load_dataset` will load the locally cached dataset instead of re-downloading the updated dataset

### Steps to reproduce the bug

Here is a minimal example script to
1. create an initial dataset and upload
2. download it so it is stored in cache
3. change the dataset and re-upload
4. redownload

```python
import time

from datasets import Dataset, DatasetDict, DownloadMode, load_dataset

username = "YOUR_USERNAME_HERE"

initial = Dataset.from_dict({"foo": [1, 2, 3]})
print(f"Intial {initial['foo']}")
initial_ds = DatasetDict({"train": initial})
initial_ds.push_to_hub("test")

time.sleep(1)

download = load_dataset(f"{username}/test", split="train")
changed = download.map(lambda x: {"foo": x["foo"] + 1})
print(f"Changed {changed['foo']}")
changed.push_to_hub("test")

time.sleep(1)

download_again = load_dataset(f"{username}/test", split="train")
print(f"Download Changed {download_again['foo']}")
# >>> gives the out-dated [1,2,3] when it should be changed [2,3,4]
```

The redownloaded dataset should be the changed dataset but it is actually the cached, initial dataset. Force-redownloading gives the correct dataset

```python
download_again_force = load_dataset(f"{username}/test", split="train", download_mode=DownloadMode.FORCE_REDOWNLOAD)
print(f"Force Download Changed {download_again_force['foo']}")
# >>> [2,3,4]
```

### Expected behavior

I assumed there should be some sort of hashing that should check for changes in the dataset and re-download if the hashes don't match

### Environment info

- `datasets` version: 2.15.0                                                   │
- Platform: Linux-5.15.0-1028-nvidia-x86_64-with-glibc2.17                     │
- Python version: 3.8.17                                                       │
- `huggingface_hub` version: 0.19.4                                            │
- PyArrow version: 13.0.0                                                      │
- Pandas version: 2.0.3                                                        │
- `fsspec` version: 2023.6.0


Add concurrent loading of shards to datasets.load_from_disk

If we use multithreading no need to ask for `num_proc`. And maybe we the same numbers of threads as tqdm by default (IIRC it's `max(32, cpu_count() + 4)`) - you can even use `tqdm.contrib.concurrent.thread_map` directly to simplify the code

Also you can ignore the `IN_MEMORY_MAX_SIZE` config for this. This parameter is kinda legacy.

Have you been able to run the benchmark on a fresh node ? The speed up doesn't seem that big in your first report

I got some fresh nodes with the 32 threads I'm loading the dataset with around 315 seconds (without any preloading). Sequentially, it used to take around 1865 seconds.
Ok I'll roll back the changes and switch to `tqdm.contrib.concurrent.thread_map` without the `num_proc` parameter.

I switched to `tqdm.contrib.concurrent.thread_map` the code looks much simpler now.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6464). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the update ! Btw you should tell Jack Morris that you added this :) see https://x.com/jxmnop/status/1749812573984461145?s=20

The CI fail is unrelated to this PR - I'm trying to fix it on `main` right now

> Thanks for the update ! Btw you should tell Jack Morris that you added this :) see https://x.com/jxmnop/status/1749812573984461145?s=20
>
> The CI fail is unrelated to this PR - I'm trying to fix it on `main` right now

Thank you! I'll let him know :)

great work guys! letting you know here too

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005268 / 0.011353 (-0.006085) | 0.003520 / 0.011008 (-0.007488) | 0.063247 / 0.038508 (0.024739) | 0.032337 / 0.023109 (0.009228) | 0.243251 / 0.275898 (-0.032647) | 0.265816 / 0.323480 (-0.057664) | 0.002960 / 0.007986 (-0.005025) | 0.002733 / 0.004328 (-0.001595) | 0.048965 / 0.004250 (0.044715) | 0.044341 / 0.037052 (0.007289) | 0.260352 / 0.258489 (0.001863) | 0.288546 / 0.293841 (-0.005295) | 0.027903 / 0.128546 (-0.100643) | 0.010897 / 0.075646 (-0.064749) | 0.210852 / 0.419271 (-0.208419) | 0.036302 / 0.043533 (-0.007231) | 0.247440 / 0.255139 (-0.007699) | 0.263024 / 0.283200 (-0.020176) | 0.017732 / 0.141683 (-0.123951) | 1.144206 / 1.452155 (-0.307949) | 1.206135 / 1.492716 (-0.286581) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.098404 / 0.018006 (0.080398) | 0.310268 / 0.000490 (0.309778) | 0.000231 / 0.000200 (0.000031) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018342 / 0.037411 (-0.019070) | 0.060620 / 0.014526 (0.046094) | 0.074248 / 0.176557 (-0.102308) | 0.121025 / 0.737135 (-0.616110) | 0.075331 / 0.296338 (-0.221008) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.293721 / 0.215209 (0.078512) | 2.854259 / 2.077655 (0.776605) | 1.520735 / 1.504120 (0.016615) | 1.393490 / 1.541195 (-0.147705) | 1.494905 / 1.468490 (0.026415) | 0.573812 / 4.584777 (-4.010965) | 2.418383 / 3.745712 (-1.327329) | 2.803916 / 5.269862 (-2.465945) | 1.741646 / 4.565676 (-2.824030) | 0.063341 / 0.424275 (-0.360934) | 0.004950 / 0.007607 (-0.002658) | 0.341758 / 0.226044 (0.115714) | 3.392918 / 2.268929 (1.123989) | 1.867037 / 55.444624 (-53.577587) | 1.571381 / 6.876477 (-5.305096) | 1.582883 / 2.142072 (-0.559190) | 0.663660 / 4.805227 (-4.141567) | 0.119587 / 6.500664 (-6.381077) | 0.042071 / 0.075469 (-0.033398) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.940976 / 1.841788 (-0.900811) | 11.841958 / 8.074308 (3.767650) | 10.510954 / 10.191392 (0.319562) | 0.131927 / 0.680424 (-0.548497) | 0.015373 / 0.534201 (-0.518828) | 0.294245 / 0.579283 (-0.285038) | 0.269355 / 0.434364 (-0.165009) | 0.330173 / 0.540337 (-0.210165) | 0.436809 / 1.386936 (-0.950127) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005609 / 0.011353 (-0.005744) | 0.003800 / 0.011008 (-0.007208) | 0.055693 / 0.038508 (0.017185) | 0.032606 / 0.023109 (0.009497) | 0.302372 / 0.275898 (0.026474) | 0.370530 / 0.323480 (0.047050) | 0.004291 / 0.007986 (-0.003694) | 0.002783 / 0.004328 (-0.001546) | 0.049351 / 0.004250 (0.045101) | 0.048186 / 0.037052 (0.011133) | 0.290022 / 0.258489 (0.031533) | 0.323358 / 0.293841 (0.029517) | 0.053929 / 0.128546 (-0.074617) | 0.011251 / 0.075646 (-0.064395) | 0.058885 / 0.419271 (-0.360387) | 0.033833 / 0.043533 (-0.009699) | 0.283546 / 0.255139 (0.028407) | 0.292416 / 0.283200 (0.009216) | 0.017682 / 0.141683 (-0.124001) | 1.141791 / 1.452155 (-0.310364) | 1.202540 / 1.492716 (-0.290177) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.101240 / 0.018006 (0.083233) | 0.313274 / 0.000490 (0.312784) | 0.000255 / 0.000200 (0.000055) | 0.000054 / 0.000054 (-0.000000) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023144 / 0.037411 (-0.014268) | 0.078418 / 0.014526 (0.063892) | 0.089716 / 0.176557 (-0.086840) | 0.129065 / 0.737135 (-0.608070) | 0.090976 / 0.296338 (-0.205362) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.294585 / 0.215209 (0.079376) | 2.921350 / 2.077655 (0.843695) | 1.600977 / 1.504120 (0.096857) | 1.483218 / 1.541195 (-0.057977) | 1.533599 / 1.468490 (0.065109) | 0.580064 / 4.584777 (-4.004712) | 2.463501 / 3.745712 (-1.282211) | 2.905853 / 5.269862 (-2.364009) | 1.799701 / 4.565676 (-2.765975) | 0.065057 / 0.424275 (-0.359218) | 0.005080 / 0.007607 (-0.002527) | 0.352292 / 0.226044 (0.126248) | 3.429664 / 2.268929 (1.160735) | 1.970752 / 55.444624 (-53.473872) | 1.697151 / 6.876477 (-5.179326) | 1.751678 / 2.142072 (-0.390394) | 0.679264 / 4.805227 (-4.125963) | 0.118197 / 6.500664 (-6.382467) | 0.041834 / 0.075469 (-0.033635) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.985756 / 1.841788 (-0.856032) | 13.335160 / 8.074308 (5.260852) | 11.524807 / 10.191392 (1.333415) | 0.134892 / 0.680424 (-0.545532) | 0.016855 / 0.534201 (-0.517346) | 0.294599 / 0.579283 (-0.284685) | 0.285988 / 0.434364 (-0.148376) | 0.331423 / 0.540337 (-0.208914) | 0.418765 / 1.386936 (-0.968171) |

![](https://cml.dev/watermark.png#65434e449b6bb6c57121d9518d92abe9a97e0bb0 "CML watermark")

In some file systems (like luster), memory mapping arrow files takes time. This can be accelerated by performing the mmap in parallel on processes or threads.

- Threads seem to be faster than processes when gathering the list of tables from the workers (see https://github.com/huggingface/datasets/issues/2252).
- I'm not sure if using threads would respect the `IN_MEMORY_MAX_SIZE` config.
- I'm not sure if we need to expose num_proc from `BaseReader.read` to `DatasetBuilder.as_dataset`. Since ` DatasetBuilder.as_dataset` is used in many places beside `load_dataset`.

### Tests on luster file system (on a shared partial node):
Loading 1231 shards of ~2GBs.
The files were pre-loaded in another process before the script runs (couldn't get a fresh node).
```python
import logging
from time import perf_counter

import datasets
logger = datasets.logging.get_logger(__name__)
datasets.logging.set_verbosity_info()
logging.basicConfig(level=logging.DEBUG, format="%(message)s")

class catchtime:
    # context to measure loading time: https://stackoverflow.com/questions/33987060/python-context-manager-that-measures-time
    def __init__(self, debug_print="Time", logger=logger):
        self.debug_print = debug_print
        self.logger = logger

    def __enter__(self):
        self.start = perf_counter()
        return self

    def __exit__(self, type, value, traceback):
        self.time = perf_counter() - self.start
        readout = f"{self.debug_print}: {self.time:.3f} seconds"
        self.logger.info(readout)

dataset_path=""

# warmup
with catchtime("Loading in parallel", logger=logger):
   ds = datasets.load_from_disk(dataset_path,num_proc=16)

# num_proc=16
with catchtime("Loading in parallel", logger=logger):
   ds = datasets.load_from_disk(dataset_path,num_proc=16)
# num_proc=32
with catchtime("Loading in parallel", logger=logger):
   ds = datasets.load_from_disk(dataset_path,num_proc=32)
# num_proc=1
with catchtime("Loading in conseq", logger=logger):
   ds = datasets.load_from_disk(dataset_path,num_proc=1)

```

#### Run 1
```
open file: .../dataset_dict.json
Loading the dataset from disk using 16 threads: 100%|██████████| 1231/1231 [01:28<00:00, 13.96shards/s]
Loading in parallel: 88.690 seconds
open file: .../dataset_dict.json
Loading the dataset from disk using 16 threads: 100%|██████████| 1231/1231 [01:48<00:00, 11.31shards/s]
Loading in parallel: 109.339 seconds
open file: .../dataset_dict.json
Loading the dataset from disk using 32 threads: 100%|██████████| 1231/1231 [01:06<00:00, 18.56shards/s]
Loading in parallel: 66.931 seconds
open file: .../dataset_dict.json
Loading the dataset from disk: 100%|██████████| 1231/1231 [05:09<00:00,  3.98shards/s]
Loading in conseq: 309.792 seconds
```
#### Run 2

```
open file: .../dataset_dict.json
Loading the dataset from disk using 16 threads: 100%|██████████| 1231/1231 [01:38<00:00, 12.53shards/s]
Loading in parallel: 98.831 seconds
open file: .../dataset_dict.json
Loading the dataset from disk using 16 threads: 100%|██████████| 1231/1231 [02:01<00:00, 10.16shards/s]
Loading in parallel: 121.669 seconds
open file: .../dataset_dict.json
Loading the dataset from disk using 32 threads: 100%|██████████| 1231/1231 [01:07<00:00, 18.18shards/s]
Loading in parallel: 68.192 seconds
open file: .../dataset_dict.json
Loading the dataset from disk: 100%|██████████| 1231/1231 [05:19<00:00,  3.86shards/s]
Loading in conseq: 319.759 seconds
```
#### Run 3
```
open file: .../dataset_dict.json
Loading the dataset from disk using 16 threads: 100%|██████████| 1231/1231 [01:36<00:00, 12.74shards/s]
Loading in parallel: 96.936 seconds
open file: .../dataset_dict.json
Loading the dataset from disk using 16 threads: 100%|██████████| 1231/1231 [02:00<00:00, 10.24shards/s]
Loading in parallel: 120.761 seconds
open file: .../dataset_dict.json
Loading the dataset from disk using 32 threads: 100%|██████████| 1231/1231 [01:08<00:00, 18.04shards/s]
Loading in parallel: 68.666 seconds
open file: .../dataset_dict.json
Loading the dataset from disk: 100%|██████████| 1231/1231 [05:35<00:00,  3.67shards/s]
Loading in conseq: 335.777 seconds
```

fix #2252


Disable benchmarks in PRs

It's a way to detect regressions in performance sensitive methods like map, and find the commit that lead to the regression

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005357 / 0.011353 (-0.005996) | 0.003295 / 0.011008 (-0.007713) | 0.062354 / 0.038508 (0.023846) | 0.054207 / 0.023109 (0.031098) | 0.240030 / 0.275898 (-0.035869) | 0.267863 / 0.323480 (-0.055617) | 0.002925 / 0.007986 (-0.005061) | 0.002634 / 0.004328 (-0.001695) | 0.047952 / 0.004250 (0.043702) | 0.038424 / 0.037052 (0.001372) | 0.248059 / 0.258489 (-0.010430) | 0.271923 / 0.293841 (-0.021918) | 0.027513 / 0.128546 (-0.101034) | 0.010344 / 0.075646 (-0.065302) | 0.210864 / 0.419271 (-0.208407) | 0.035911 / 0.043533 (-0.007622) | 0.245166 / 0.255139 (-0.009973) | 0.260914 / 0.283200 (-0.022285) | 0.016709 / 0.141683 (-0.124974) | 1.098324 / 1.452155 (-0.353830) | 1.162638 / 1.492716 (-0.330079) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094419 / 0.018006 (0.076413) | 0.303209 / 0.000490 (0.302719) | 0.000214 / 0.000200 (0.000014) | 0.000053 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018350 / 0.037411 (-0.019061) | 0.060625 / 0.014526 (0.046099) | 0.072545 / 0.176557 (-0.104012) | 0.120905 / 0.737135 (-0.616231) | 0.073858 / 0.296338 (-0.222480) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282011 / 0.215209 (0.066802) | 2.758741 / 2.077655 (0.681086) | 1.431691 / 1.504120 (-0.072429) | 1.315883 / 1.541195 (-0.225312) | 1.344235 / 1.468490 (-0.124255) | 0.562117 / 4.584777 (-4.022660) | 2.385641 / 3.745712 (-1.360071) | 2.785402 / 5.269862 (-2.484460) | 1.753912 / 4.565676 (-2.811764) | 0.064054 / 0.424275 (-0.360221) | 0.005050 / 0.007607 (-0.002557) | 0.336452 / 0.226044 (0.110407) | 3.302481 / 2.268929 (1.033553) | 1.794105 / 55.444624 (-53.650519) | 1.519346 / 6.876477 (-5.357131) | 1.514911 / 2.142072 (-0.627161) | 0.655779 / 4.805227 (-4.149449) | 0.117913 / 6.500664 (-6.382751) | 0.042229 / 0.075469 (-0.033240) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.935196 / 1.841788 (-0.906591) | 11.490113 / 8.074308 (3.415805) | 10.542446 / 10.191392 (0.351054) | 0.129614 / 0.680424 (-0.550810) | 0.014919 / 0.534201 (-0.519282) | 0.288448 / 0.579283 (-0.290835) | 0.266929 / 0.434364 (-0.167435) | 0.328830 / 0.540337 (-0.211507) | 0.475510 / 1.386936 (-0.911426) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005469 / 0.011353 (-0.005884) | 0.003798 / 0.011008 (-0.007210) | 0.049129 / 0.038508 (0.010621) | 0.055490 / 0.023109 (0.032380) | 0.265828 / 0.275898 (-0.010070) | 0.286031 / 0.323480 (-0.037448) | 0.004075 / 0.007986 (-0.003910) | 0.002668 / 0.004328 (-0.001660) | 0.047823 / 0.004250 (0.043573) | 0.041946 / 0.037052 (0.004894) | 0.270359 / 0.258489 (0.011869) | 0.294287 / 0.293841 (0.000446) | 0.029643 / 0.128546 (-0.098903) | 0.010523 / 0.075646 (-0.065123) | 0.057370 / 0.419271 (-0.361902) | 0.033149 / 0.043533 (-0.010384) | 0.264408 / 0.255139 (0.009269) | 0.280413 / 0.283200 (-0.002787) | 0.018313 / 0.141683 (-0.123370) | 1.105982 / 1.452155 (-0.346173) | 1.182486 / 1.492716 (-0.310230) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092643 / 0.018006 (0.074637) | 0.301320 / 0.000490 (0.300831) | 0.000221 / 0.000200 (0.000021) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021253 / 0.037411 (-0.016158) | 0.068052 / 0.014526 (0.053527) | 0.080821 / 0.176557 (-0.095736) | 0.119320 / 0.737135 (-0.617816) | 0.081952 / 0.296338 (-0.214387) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.288536 / 0.215209 (0.073327) | 2.819900 / 2.077655 (0.742245) | 1.545210 / 1.504120 (0.041090) | 1.422047 / 1.541195 (-0.119147) | 1.439158 / 1.468490 (-0.029332) | 0.564910 / 4.584777 (-4.019867) | 2.430474 / 3.745712 (-1.315238) | 2.763979 / 5.269862 (-2.505882) | 1.732203 / 4.565676 (-2.833474) | 0.062692 / 0.424275 (-0.361583) | 0.004936 / 0.007607 (-0.002671) | 0.341626 / 0.226044 (0.115582) | 3.366623 / 2.268929 (1.097694) | 1.917198 / 55.444624 (-53.527426) | 1.637635 / 6.876477 (-5.238842) | 1.625953 / 2.142072 (-0.516119) | 0.634936 / 4.805227 (-4.170291) | 0.115336 / 6.500664 (-6.385328) | 0.040946 / 0.075469 (-0.034524) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.964865 / 1.841788 (-0.876922) | 12.077233 / 8.074308 (4.002925) | 10.664120 / 10.191392 (0.472728) | 0.132084 / 0.680424 (-0.548340) | 0.015931 / 0.534201 (-0.518270) | 0.289181 / 0.579283 (-0.290102) | 0.276943 / 0.434364 (-0.157420) | 0.324884 / 0.540337 (-0.215453) | 0.552570 / 1.386936 (-0.834366) |

![](https://cml.dev/watermark.png#4ac3f2b3f6d867673e41a0253f9e1ad48db68a8e "CML watermark")

In order to keep PR pages less spammy / more readable.
Having the benchmarks on commits on `main` is enough imo


Missing DatasetNotFoundError

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005594 / 0.011353 (-0.005759) | 0.003672 / 0.011008 (-0.007337) | 0.062796 / 0.038508 (0.024288) | 0.059432 / 0.023109 (0.036323) | 0.253976 / 0.275898 (-0.021922) | 0.281155 / 0.323480 (-0.042325) | 0.003023 / 0.007986 (-0.004962) | 0.003320 / 0.004328 (-0.001008) | 0.049059 / 0.004250 (0.044809) | 0.040252 / 0.037052 (0.003200) | 0.259526 / 0.258489 (0.001037) | 0.318798 / 0.293841 (0.024957) | 0.027883 / 0.128546 (-0.100663) | 0.010883 / 0.075646 (-0.064763) | 0.206948 / 0.419271 (-0.212323) | 0.036335 / 0.043533 (-0.007198) | 0.253209 / 0.255139 (-0.001930) | 0.275173 / 0.283200 (-0.008026) | 0.020365 / 0.141683 (-0.121318) | 1.121630 / 1.452155 (-0.330524) | 1.174680 / 1.492716 (-0.318036) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.098372 / 0.018006 (0.080366) | 0.309949 / 0.000490 (0.309460) | 0.000225 / 0.000200 (0.000025) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019495 / 0.037411 (-0.017916) | 0.062321 / 0.014526 (0.047795) | 0.074525 / 0.176557 (-0.102031) | 0.121832 / 0.737135 (-0.615303) | 0.077612 / 0.296338 (-0.218727) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.288156 / 0.215209 (0.072947) | 2.816411 / 2.077655 (0.738756) | 1.497926 / 1.504120 (-0.006193) | 1.378137 / 1.541195 (-0.163058) | 1.446466 / 1.468490 (-0.022024) | 0.566195 / 4.584777 (-4.018582) | 2.391933 / 3.745712 (-1.353780) | 2.929290 / 5.269862 (-2.340572) | 1.828215 / 4.565676 (-2.737462) | 0.063312 / 0.424275 (-0.360963) | 0.005199 / 0.007607 (-0.002408) | 0.342883 / 0.226044 (0.116838) | 3.378388 / 2.268929 (1.109459) | 1.865710 / 55.444624 (-53.578915) | 1.573442 / 6.876477 (-5.303035) | 1.631228 / 2.142072 (-0.510845) | 0.651614 / 4.805227 (-4.153613) | 0.118177 / 6.500664 (-6.382487) | 0.043303 / 0.075469 (-0.032166) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.950694 / 1.841788 (-0.891094) | 12.559851 / 8.074308 (4.485543) | 10.751123 / 10.191392 (0.559731) | 0.143107 / 0.680424 (-0.537317) | 0.014469 / 0.534201 (-0.519732) | 0.289531 / 0.579283 (-0.289752) | 0.267316 / 0.434364 (-0.167047) | 0.327748 / 0.540337 (-0.212590) | 0.437758 / 1.386936 (-0.949178) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005669 / 0.011353 (-0.005684) | 0.003831 / 0.011008 (-0.007177) | 0.049096 / 0.038508 (0.010588) | 0.061408 / 0.023109 (0.038299) | 0.274571 / 0.275898 (-0.001327) | 0.299978 / 0.323480 (-0.023501) | 0.004216 / 0.007986 (-0.003769) | 0.002848 / 0.004328 (-0.001480) | 0.048755 / 0.004250 (0.044504) | 0.042576 / 0.037052 (0.005524) | 0.276781 / 0.258489 (0.018292) | 0.300903 / 0.293841 (0.007062) | 0.030243 / 0.128546 (-0.098303) | 0.010967 / 0.075646 (-0.064679) | 0.057879 / 0.419271 (-0.361392) | 0.033206 / 0.043533 (-0.010327) | 0.277620 / 0.255139 (0.022481) | 0.296263 / 0.283200 (0.013064) | 0.019022 / 0.141683 (-0.122660) | 1.125615 / 1.452155 (-0.326539) | 1.278016 / 1.492716 (-0.214700) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096836 / 0.018006 (0.078830) | 0.307491 / 0.000490 (0.307001) | 0.000230 / 0.000200 (0.000030) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021552 / 0.037411 (-0.015859) | 0.071099 / 0.014526 (0.056573) | 0.082432 / 0.176557 (-0.094124) | 0.121826 / 0.737135 (-0.615310) | 0.084902 / 0.296338 (-0.211437) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.328113 / 0.215209 (0.112904) | 2.989613 / 2.077655 (0.911959) | 1.604904 / 1.504120 (0.100784) | 1.485459 / 1.541195 (-0.055735) | 1.524829 / 1.468490 (0.056339) | 0.580589 / 4.584777 (-4.004188) | 2.440087 / 3.745712 (-1.305625) | 2.944697 / 5.269862 (-2.325164) | 1.832728 / 4.565676 (-2.732949) | 0.064423 / 0.424275 (-0.359852) | 0.004991 / 0.007607 (-0.002616) | 0.357878 / 0.226044 (0.131834) | 3.515415 / 2.268929 (1.246487) | 1.964492 / 55.444624 (-53.480132) | 1.684058 / 6.876477 (-5.192418) | 1.730294 / 2.142072 (-0.411778) | 0.661228 / 4.805227 (-4.143999) | 0.122894 / 6.500664 (-6.377770) | 0.041776 / 0.075469 (-0.033693) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.969849 / 1.841788 (-0.871939) | 12.897067 / 8.074308 (4.822758) | 10.908200 / 10.191392 (0.716808) | 0.141139 / 0.680424 (-0.539285) | 0.015377 / 0.534201 (-0.518824) | 0.288625 / 0.579283 (-0.290658) | 0.279020 / 0.434364 (-0.155344) | 0.328386 / 0.540337 (-0.211951) | 0.590833 / 1.386936 (-0.796103) |

![](https://cml.dev/watermark.png#39ea60eaabb05d8ee38c072f375816cf87fce1a9 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004986 / 0.011353 (-0.006367) | 0.003070 / 0.011008 (-0.007938) | 0.062433 / 0.038508 (0.023925) | 0.050639 / 0.023109 (0.027530) | 0.241807 / 0.275898 (-0.034091) | 0.262517 / 0.323480 (-0.060963) | 0.003826 / 0.007986 (-0.004160) | 0.002602 / 0.004328 (-0.001727) | 0.048508 / 0.004250 (0.044257) | 0.037276 / 0.037052 (0.000224) | 0.245757 / 0.258489 (-0.012732) | 0.272969 / 0.293841 (-0.020871) | 0.027139 / 0.128546 (-0.101407) | 0.010265 / 0.075646 (-0.065381) | 0.207279 / 0.419271 (-0.211992) | 0.035312 / 0.043533 (-0.008221) | 0.247535 / 0.255139 (-0.007604) | 0.260668 / 0.283200 (-0.022532) | 0.016496 / 0.141683 (-0.125187) | 1.137510 / 1.452155 (-0.314645) | 1.167870 / 1.492716 (-0.324847) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091743 / 0.018006 (0.073736) | 0.298649 / 0.000490 (0.298159) | 0.000208 / 0.000200 (0.000009) | 0.000053 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019053 / 0.037411 (-0.018359) | 0.060300 / 0.014526 (0.045774) | 0.072154 / 0.176557 (-0.104402) | 0.120293 / 0.737135 (-0.616842) | 0.073923 / 0.296338 (-0.222415) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.283058 / 0.215209 (0.067849) | 2.769503 / 2.077655 (0.691849) | 1.457016 / 1.504120 (-0.047104) | 1.335753 / 1.541195 (-0.205441) | 1.325986 / 1.468490 (-0.142504) | 0.562553 / 4.584777 (-4.022224) | 2.406144 / 3.745712 (-1.339568) | 2.778063 / 5.269862 (-2.491799) | 1.782199 / 4.565676 (-2.783477) | 0.062490 / 0.424275 (-0.361785) | 0.004912 / 0.007607 (-0.002695) | 0.338500 / 0.226044 (0.112456) | 3.309746 / 2.268929 (1.040818) | 1.819693 / 55.444624 (-53.624931) | 1.510295 / 6.876477 (-5.366182) | 1.578402 / 2.142072 (-0.563671) | 0.637517 / 4.805227 (-4.167710) | 0.117018 / 6.500664 (-6.383647) | 0.048149 / 0.075469 (-0.027320) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.939424 / 1.841788 (-0.902364) | 11.494891 / 8.074308 (3.420583) | 10.115194 / 10.191392 (-0.076198) | 0.126751 / 0.680424 (-0.553673) | 0.013567 / 0.534201 (-0.520634) | 0.282501 / 0.579283 (-0.296782) | 0.260594 / 0.434364 (-0.173770) | 0.325940 / 0.540337 (-0.214397) | 0.426186 / 1.386936 (-0.960750) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005405 / 0.011353 (-0.005948) | 0.003557 / 0.011008 (-0.007451) | 0.051139 / 0.038508 (0.012631) | 0.053446 / 0.023109 (0.030337) | 0.268051 / 0.275898 (-0.007847) | 0.292343 / 0.323480 (-0.031136) | 0.004716 / 0.007986 (-0.003269) | 0.002677 / 0.004328 (-0.001651) | 0.047634 / 0.004250 (0.043384) | 0.041062 / 0.037052 (0.004009) | 0.269225 / 0.258489 (0.010736) | 0.297462 / 0.293841 (0.003621) | 0.029292 / 0.128546 (-0.099254) | 0.010947 / 0.075646 (-0.064699) | 0.057845 / 0.419271 (-0.361426) | 0.032793 / 0.043533 (-0.010740) | 0.265308 / 0.255139 (0.010169) | 0.288242 / 0.283200 (0.005043) | 0.018311 / 0.141683 (-0.123372) | 1.140957 / 1.452155 (-0.311197) | 1.204883 / 1.492716 (-0.287833) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091375 / 0.018006 (0.073368) | 0.285922 / 0.000490 (0.285432) | 0.000238 / 0.000200 (0.000038) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021277 / 0.037411 (-0.016134) | 0.068853 / 0.014526 (0.054328) | 0.081002 / 0.176557 (-0.095555) | 0.120998 / 0.737135 (-0.616138) | 0.082741 / 0.296338 (-0.213598) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.299398 / 0.215209 (0.084189) | 2.909622 / 2.077655 (0.831967) | 1.624381 / 1.504120 (0.120261) | 1.501683 / 1.541195 (-0.039512) | 1.523045 / 1.468490 (0.054555) | 0.548960 / 4.584777 (-4.035817) | 2.413297 / 3.745712 (-1.332415) | 2.817852 / 5.269862 (-2.452010) | 1.754407 / 4.565676 (-2.811270) | 0.061912 / 0.424275 (-0.362363) | 0.004880 / 0.007607 (-0.002727) | 0.353989 / 0.226044 (0.127944) | 3.496147 / 2.268929 (1.227219) | 2.003026 / 55.444624 (-53.441598) | 1.702013 / 6.876477 (-5.174463) | 1.680935 / 2.142072 (-0.461137) | 0.630183 / 4.805227 (-4.175044) | 0.113786 / 6.500664 (-6.386878) | 0.040061 / 0.075469 (-0.035408) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.957218 / 1.841788 (-0.884569) | 11.914469 / 8.074308 (3.840160) | 10.488896 / 10.191392 (0.297504) | 0.129292 / 0.680424 (-0.551132) | 0.016603 / 0.534201 (-0.517598) | 0.287367 / 0.579283 (-0.291916) | 0.271332 / 0.434364 (-0.163032) | 0.325577 / 0.540337 (-0.214761) | 0.560553 / 1.386936 (-0.826383) |

![](https://cml.dev/watermark.png#2d31e434bbeafdf6a70cb80539342d8fe5f5fd27 "CML watermark")

continuation of https://github.com/huggingface/datasets/pull/6431

this should fix the CI in https://github.com/huggingface/datasets/pull/6458 too


Fix shard retry mechanism in `push_to_hub`

@Wauplin Maybe `504` should be added to the `retry_on_status_codes` tuple [here](https://github.com/huggingface/huggingface_hub/blob/5eefebee2c150a2df950ab710db350e96c711433/src/huggingface_hub/lfs.py#L300) to guard against https://github.com/huggingface/datasets/issues/3872

We could but I'm not sure to have witness a 504 on S3 before. The issue reported in https://github.com/huggingface/datasets/issues/3872 is a 504 on the `/upload` endpoint on the Hub and this is not an endpoint that is retried on [this line](https://github.com/huggingface/huggingface_hub/blob/5eefebee2c150a2df950ab710db350e96c711433/src/huggingface_hub/lfs.py#L300).

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005110 / 0.011353 (-0.006243) | 0.003307 / 0.011008 (-0.007701) | 0.062601 / 0.038508 (0.024093) | 0.049644 / 0.023109 (0.026534) | 0.243195 / 0.275898 (-0.032703) | 0.273543 / 0.323480 (-0.049936) | 0.003862 / 0.007986 (-0.004123) | 0.002624 / 0.004328 (-0.001705) | 0.048273 / 0.004250 (0.044023) | 0.037820 / 0.037052 (0.000768) | 0.249134 / 0.258489 (-0.009355) | 0.319359 / 0.293841 (0.025518) | 0.027816 / 0.128546 (-0.100730) | 0.010422 / 0.075646 (-0.065225) | 0.206607 / 0.419271 (-0.212665) | 0.035719 / 0.043533 (-0.007814) | 0.250300 / 0.255139 (-0.004839) | 0.290377 / 0.283200 (0.007177) | 0.018459 / 0.141683 (-0.123224) | 1.114664 / 1.452155 (-0.337490) | 1.171429 / 1.492716 (-0.321288) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091483 / 0.018006 (0.073477) | 0.302770 / 0.000490 (0.302281) | 0.000203 / 0.000200 (0.000003) | 0.000047 / 0.000054 (-0.000007) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018870 / 0.037411 (-0.018541) | 0.062692 / 0.014526 (0.048166) | 0.075381 / 0.176557 (-0.101176) | 0.122338 / 0.737135 (-0.614797) | 0.075608 / 0.296338 (-0.220730) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.288115 / 0.215209 (0.072906) | 2.816183 / 2.077655 (0.738528) | 1.535601 / 1.504120 (0.031481) | 1.409546 / 1.541195 (-0.131648) | 1.438569 / 1.468490 (-0.029921) | 0.561797 / 4.584777 (-4.022980) | 2.373921 / 3.745712 (-1.371791) | 2.739437 / 5.269862 (-2.530424) | 1.750921 / 4.565676 (-2.814755) | 0.062114 / 0.424275 (-0.362161) | 0.004965 / 0.007607 (-0.002642) | 0.348614 / 0.226044 (0.122569) | 3.519631 / 2.268929 (1.250703) | 1.910797 / 55.444624 (-53.533827) | 1.610541 / 6.876477 (-5.265936) | 1.617972 / 2.142072 (-0.524100) | 0.639421 / 4.805227 (-4.165806) | 0.117371 / 6.500664 (-6.383293) | 0.041851 / 0.075469 (-0.033618) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.945563 / 1.841788 (-0.896224) | 11.362399 / 8.074308 (3.288090) | 10.468468 / 10.191392 (0.277075) | 0.128925 / 0.680424 (-0.551499) | 0.013892 / 0.534201 (-0.520309) | 0.285487 / 0.579283 (-0.293796) | 0.269295 / 0.434364 (-0.165069) | 0.324843 / 0.540337 (-0.215495) | 0.438452 / 1.386936 (-0.948484) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005303 / 0.011353 (-0.006050) | 0.003162 / 0.011008 (-0.007846) | 0.048177 / 0.038508 (0.009669) | 0.048708 / 0.023109 (0.025599) | 0.271663 / 0.275898 (-0.004235) | 0.289948 / 0.323480 (-0.033532) | 0.003955 / 0.007986 (-0.004030) | 0.002616 / 0.004328 (-0.001713) | 0.047510 / 0.004250 (0.043260) | 0.039938 / 0.037052 (0.002886) | 0.277449 / 0.258489 (0.018960) | 0.300315 / 0.293841 (0.006474) | 0.029263 / 0.128546 (-0.099283) | 0.010403 / 0.075646 (-0.065244) | 0.056682 / 0.419271 (-0.362590) | 0.032757 / 0.043533 (-0.010776) | 0.273291 / 0.255139 (0.018152) | 0.289023 / 0.283200 (0.005824) | 0.017843 / 0.141683 (-0.123840) | 1.124762 / 1.452155 (-0.327393) | 1.176646 / 1.492716 (-0.316070) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.004568 / 0.018006 (-0.013438) | 0.300715 / 0.000490 (0.300225) | 0.000212 / 0.000200 (0.000012) | 0.000049 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021528 / 0.037411 (-0.015883) | 0.068317 / 0.014526 (0.053792) | 0.081358 / 0.176557 (-0.095199) | 0.119297 / 0.737135 (-0.617838) | 0.082445 / 0.296338 (-0.213893) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.289681 / 0.215209 (0.074472) | 2.843862 / 2.077655 (0.766208) | 1.574257 / 1.504120 (0.070137) | 1.454026 / 1.541195 (-0.087169) | 1.478379 / 1.468490 (0.009889) | 0.558259 / 4.584777 (-4.026518) | 2.513261 / 3.745712 (-1.232451) | 2.759751 / 5.269862 (-2.510111) | 1.730335 / 4.565676 (-2.835341) | 0.063805 / 0.424275 (-0.360470) | 0.004991 / 0.007607 (-0.002616) | 0.346586 / 0.226044 (0.120542) | 3.369163 / 2.268929 (1.100234) | 1.934734 / 55.444624 (-53.509890) | 1.658864 / 6.876477 (-5.217613) | 1.645621 / 2.142072 (-0.496452) | 0.636633 / 4.805227 (-4.168594) | 0.116839 / 6.500664 (-6.383825) | 0.040863 / 0.075469 (-0.034606) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.960925 / 1.841788 (-0.880863) | 11.769189 / 8.074308 (3.694881) | 10.713662 / 10.191392 (0.522270) | 0.140510 / 0.680424 (-0.539914) | 0.015424 / 0.534201 (-0.518777) | 0.288039 / 0.579283 (-0.291244) | 0.277623 / 0.434364 (-0.156741) | 0.322622 / 0.540337 (-0.217716) | 0.539805 / 1.386936 (-0.847131) |

![](https://cml.dev/watermark.png#07ad81c15bd3b954defe779fc37ba5f432f5ff2a "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005501 / 0.011353 (-0.005852) | 0.003754 / 0.011008 (-0.007254) | 0.062628 / 0.038508 (0.024120) | 0.059951 / 0.023109 (0.036842) | 0.254851 / 0.275898 (-0.021047) | 0.272133 / 0.323480 (-0.051347) | 0.003962 / 0.007986 (-0.004024) | 0.002759 / 0.004328 (-0.001569) | 0.048412 / 0.004250 (0.044161) | 0.039349 / 0.037052 (0.002297) | 0.253093 / 0.258489 (-0.005397) | 0.287048 / 0.293841 (-0.006793) | 0.027197 / 0.128546 (-0.101349) | 0.010828 / 0.075646 (-0.064819) | 0.206371 / 0.419271 (-0.212901) | 0.035881 / 0.043533 (-0.007652) | 0.254905 / 0.255139 (-0.000234) | 0.273819 / 0.283200 (-0.009381) | 0.018041 / 0.141683 (-0.123642) | 1.103970 / 1.452155 (-0.348185) | 1.166340 / 1.492716 (-0.326377) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093196 / 0.018006 (0.075190) | 0.302690 / 0.000490 (0.302200) | 0.000219 / 0.000200 (0.000019) | 0.000045 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019552 / 0.037411 (-0.017860) | 0.062337 / 0.014526 (0.047811) | 0.074070 / 0.176557 (-0.102486) | 0.120998 / 0.737135 (-0.616137) | 0.076265 / 0.296338 (-0.220074) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.272637 / 0.215209 (0.057427) | 2.693350 / 2.077655 (0.615696) | 1.398020 / 1.504120 (-0.106100) | 1.285706 / 1.541195 (-0.255488) | 1.342810 / 1.468490 (-0.125680) | 0.565378 / 4.584777 (-4.019399) | 2.390131 / 3.745712 (-1.355581) | 2.892137 / 5.269862 (-2.377725) | 1.819840 / 4.565676 (-2.745836) | 0.062789 / 0.424275 (-0.361486) | 0.004920 / 0.007607 (-0.002687) | 0.329281 / 0.226044 (0.103237) | 3.261664 / 2.268929 (0.992735) | 1.775102 / 55.444624 (-53.669523) | 1.514341 / 6.876477 (-5.362136) | 1.530805 / 2.142072 (-0.611267) | 0.641009 / 4.805227 (-4.164218) | 0.118626 / 6.500664 (-6.382038) | 0.042732 / 0.075469 (-0.032737) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.933179 / 1.841788 (-0.908609) | 12.085247 / 8.074308 (4.010939) | 10.541596 / 10.191392 (0.350204) | 0.140141 / 0.680424 (-0.540283) | 0.014646 / 0.534201 (-0.519555) | 0.289640 / 0.579283 (-0.289643) | 0.281042 / 0.434364 (-0.153322) | 0.326462 / 0.540337 (-0.213876) | 0.441981 / 1.386936 (-0.944955) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005259 / 0.011353 (-0.006094) | 0.003766 / 0.011008 (-0.007242) | 0.048782 / 0.038508 (0.010273) | 0.064946 / 0.023109 (0.041836) | 0.264529 / 0.275898 (-0.011369) | 0.289675 / 0.323480 (-0.033805) | 0.004057 / 0.007986 (-0.003928) | 0.002805 / 0.004328 (-0.001523) | 0.047709 / 0.004250 (0.043459) | 0.041149 / 0.037052 (0.004096) | 0.271254 / 0.258489 (0.012765) | 0.296685 / 0.293841 (0.002844) | 0.029486 / 0.128546 (-0.099060) | 0.010608 / 0.075646 (-0.065038) | 0.056392 / 0.419271 (-0.362879) | 0.033181 / 0.043533 (-0.010352) | 0.267029 / 0.255139 (0.011890) | 0.284987 / 0.283200 (0.001787) | 0.018045 / 0.141683 (-0.123637) | 1.137358 / 1.452155 (-0.314796) | 1.184007 / 1.492716 (-0.308709) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.004603 / 0.018006 (-0.013403) | 0.303901 / 0.000490 (0.303411) | 0.000225 / 0.000200 (0.000025) | 0.000055 / 0.000054 (0.000000) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021957 / 0.037411 (-0.015454) | 0.069427 / 0.014526 (0.054901) | 0.082394 / 0.176557 (-0.094163) | 0.120745 / 0.737135 (-0.616390) | 0.084571 / 0.296338 (-0.211767) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.292832 / 0.215209 (0.077623) | 2.824295 / 2.077655 (0.746640) | 1.563273 / 1.504120 (0.059153) | 1.440202 / 1.541195 (-0.100992) | 1.489810 / 1.468490 (0.021320) | 0.561120 / 4.584777 (-4.023657) | 2.439045 / 3.745712 (-1.306667) | 2.867139 / 5.269862 (-2.402722) | 1.793812 / 4.565676 (-2.771865) | 0.062797 / 0.424275 (-0.361478) | 0.005033 / 0.007607 (-0.002574) | 0.343648 / 0.226044 (0.117604) | 3.432285 / 2.268929 (1.163357) | 1.918175 / 55.444624 (-53.526449) | 1.637245 / 6.876477 (-5.239232) | 1.709246 / 2.142072 (-0.432826) | 0.634744 / 4.805227 (-4.170483) | 0.115782 / 6.500664 (-6.384882) | 0.041228 / 0.075469 (-0.034241) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.962369 / 1.841788 (-0.879418) | 12.750819 / 8.074308 (4.676511) | 10.927356 / 10.191392 (0.735964) | 0.143454 / 0.680424 (-0.536970) | 0.015348 / 0.534201 (-0.518853) | 0.291207 / 0.579283 (-0.288076) | 0.276924 / 0.434364 (-0.157440) | 0.327287 / 0.540337 (-0.213050) | 0.577439 / 1.386936 (-0.809497) |

![](https://cml.dev/watermark.png#544ad95f6b6da7fee44a2bc838e15a5e0156c946 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005070 / 0.011353 (-0.006283) | 0.003475 / 0.011008 (-0.007533) | 0.061985 / 0.038508 (0.023477) | 0.048539 / 0.023109 (0.025430) | 0.229935 / 0.275898 (-0.045963) | 0.255247 / 0.323480 (-0.068233) | 0.003919 / 0.007986 (-0.004066) | 0.002664 / 0.004328 (-0.001664) | 0.048892 / 0.004250 (0.044642) | 0.037381 / 0.037052 (0.000328) | 0.238517 / 0.258489 (-0.019972) | 0.284069 / 0.293841 (-0.009772) | 0.027513 / 0.128546 (-0.101033) | 0.010778 / 0.075646 (-0.064868) | 0.205004 / 0.419271 (-0.214268) | 0.035553 / 0.043533 (-0.007980) | 0.230117 / 0.255139 (-0.025022) | 0.251150 / 0.283200 (-0.032050) | 0.017951 / 0.141683 (-0.123732) | 1.145548 / 1.452155 (-0.306607) | 1.191659 / 1.492716 (-0.301057) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092335 / 0.018006 (0.074329) | 0.300264 / 0.000490 (0.299774) | 0.000206 / 0.000200 (0.000006) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018608 / 0.037411 (-0.018804) | 0.060376 / 0.014526 (0.045850) | 0.073551 / 0.176557 (-0.103006) | 0.118840 / 0.737135 (-0.618295) | 0.074447 / 0.296338 (-0.221892) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.287033 / 0.215209 (0.071824) | 2.770958 / 2.077655 (0.693303) | 1.443986 / 1.504120 (-0.060134) | 1.314627 / 1.541195 (-0.226567) | 1.342287 / 1.468490 (-0.126203) | 0.559607 / 4.584777 (-4.025170) | 2.409678 / 3.745712 (-1.336034) | 2.772566 / 5.269862 (-2.497295) | 1.743511 / 4.565676 (-2.822165) | 0.062277 / 0.424275 (-0.361998) | 0.004952 / 0.007607 (-0.002655) | 0.330581 / 0.226044 (0.104537) | 3.280385 / 2.268929 (1.011456) | 1.809599 / 55.444624 (-53.635025) | 1.532186 / 6.876477 (-5.344290) | 1.529689 / 2.142072 (-0.612383) | 0.645213 / 4.805227 (-4.160014) | 0.117564 / 6.500664 (-6.383100) | 0.041657 / 0.075469 (-0.033812) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.943912 / 1.841788 (-0.897876) | 11.414317 / 8.074308 (3.340009) | 10.394915 / 10.191392 (0.203523) | 0.129271 / 0.680424 (-0.551153) | 0.013934 / 0.534201 (-0.520267) | 0.288217 / 0.579283 (-0.291066) | 0.267171 / 0.434364 (-0.167193) | 0.327112 / 0.540337 (-0.213225) | 0.446680 / 1.386936 (-0.940256) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005200 / 0.011353 (-0.006152) | 0.003453 / 0.011008 (-0.007555) | 0.048736 / 0.038508 (0.010228) | 0.051073 / 0.023109 (0.027964) | 0.276591 / 0.275898 (0.000693) | 0.294495 / 0.323480 (-0.028985) | 0.004069 / 0.007986 (-0.003917) | 0.002945 / 0.004328 (-0.001383) | 0.047090 / 0.004250 (0.042839) | 0.040445 / 0.037052 (0.003393) | 0.278464 / 0.258489 (0.019975) | 0.304020 / 0.293841 (0.010179) | 0.028811 / 0.128546 (-0.099736) | 0.010388 / 0.075646 (-0.065259) | 0.057214 / 0.419271 (-0.362057) | 0.032588 / 0.043533 (-0.010945) | 0.277694 / 0.255139 (0.022555) | 0.294979 / 0.283200 (0.011779) | 0.018384 / 0.141683 (-0.123299) | 1.162332 / 1.452155 (-0.289822) | 1.188355 / 1.492716 (-0.304361) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090501 / 0.018006 (0.072495) | 0.303122 / 0.000490 (0.302632) | 0.000222 / 0.000200 (0.000022) | 0.000053 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022536 / 0.037411 (-0.014876) | 0.068452 / 0.014526 (0.053926) | 0.080932 / 0.176557 (-0.095625) | 0.119185 / 0.737135 (-0.617950) | 0.081513 / 0.296338 (-0.214825) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.291522 / 0.215209 (0.076313) | 2.849467 / 2.077655 (0.771812) | 1.597395 / 1.504120 (0.093275) | 1.512872 / 1.541195 (-0.028323) | 1.488144 / 1.468490 (0.019654) | 0.572436 / 4.584777 (-4.012341) | 2.440129 / 3.745712 (-1.305583) | 2.788045 / 5.269862 (-2.481817) | 1.754246 / 4.565676 (-2.811430) | 0.066706 / 0.424275 (-0.357569) | 0.005035 / 0.007607 (-0.002573) | 0.336621 / 0.226044 (0.110576) | 3.322820 / 2.268929 (1.053891) | 1.940494 / 55.444624 (-53.504130) | 1.670022 / 6.876477 (-5.206454) | 1.666353 / 2.142072 (-0.475720) | 0.646180 / 4.805227 (-4.159047) | 0.116676 / 6.500664 (-6.383988) | 0.040559 / 0.075469 (-0.034910) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.971396 / 1.841788 (-0.870392) | 11.782426 / 8.074308 (3.708118) | 10.672034 / 10.191392 (0.480642) | 0.137658 / 0.680424 (-0.542766) | 0.016210 / 0.534201 (-0.517991) | 0.288302 / 0.579283 (-0.290981) | 0.280775 / 0.434364 (-0.153589) | 0.326962 / 0.540337 (-0.213375) | 0.558511 / 1.386936 (-0.828425) |

![](https://cml.dev/watermark.png#76020180407d7ea9a0b535758d8d1b241fd19d8c "CML watermark")

When it fails, `preupload_lfs_files` throws a [`RuntimeError`](https://github.com/huggingface/huggingface_hub/blob/5eefebee2c150a2df950ab710db350e96c711433/src/huggingface_hub/_commit_api.py#L402) error and chains the original HTTP error. This PR modifies the retry mechanism's error handling to account for that.

Fix https://github.com/huggingface/datasets/issues/6392