Set dev version

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6117). All of your documentation changes will be reflected on that endpoint.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.012516 / 0.011353 (0.001163) | 0.004725 / 0.011008 (-0.006283) | 0.112245 / 0.038508 (0.073736) | 0.079146 / 0.023109 (0.056037) | 0.386415 / 0.275898 (0.110517) | 0.420441 / 0.323480 (0.096961) | 0.005682 / 0.007986 (-0.002304) | 0.004169 / 0.004328 (-0.000160) | 0.077847 / 0.004250 (0.073597) | 0.055763 / 0.037052 (0.018711) | 0.385529 / 0.258489 (0.127040) | 0.422711 / 0.293841 (0.128870) | 0.047212 / 0.128546 (-0.081334) | 0.013711 / 0.075646 (-0.061935) | 0.342856 / 0.419271 (-0.076416) | 0.066788 / 0.043533 (0.023255) | 0.380728 / 0.255139 (0.125589) | 0.416241 / 0.283200 (0.133041) | 0.034676 / 0.141683 (-0.107007) | 1.679661 / 1.452155 (0.227506) | 1.838014 / 1.492716 (0.345297) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.219556 / 0.018006 (0.201550) | 0.524728 / 0.000490 (0.524238) | 0.005045 / 0.000200 (0.004845) | 0.000124 / 0.000054 (0.000069) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025475 / 0.037411 (-0.011936) | 0.085937 / 0.014526 (0.071412) | 0.099245 / 0.176557 (-0.077311) | 0.158995 / 0.737135 (-0.578141) | 0.101504 / 0.296338 (-0.194835) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.582200 / 0.215209 (0.366991) | 5.794340 / 2.077655 (3.716685) | 2.473635 / 1.504120 (0.969515) | 2.168135 / 1.541195 (0.626941) | 2.215886 / 1.468490 (0.747396) | 0.855599 / 4.584777 (-3.729178) | 5.003067 / 3.745712 (1.257354) | 4.503566 / 5.269862 (-0.766295) | 2.912248 / 4.565676 (-1.653428) | 0.103267 / 0.424275 (-0.321008) | 0.012114 / 0.007607 (0.004507) | 0.712240 / 0.226044 (0.486196) | 7.131946 / 2.268929 (4.863017) | 3.280052 / 55.444624 (-52.164573) | 2.583472 / 6.876477 (-4.293004) | 2.820758 / 2.142072 (0.678686) | 1.132097 / 4.805227 (-3.673131) | 0.232191 / 6.500664 (-6.268473) | 0.082966 / 0.075469 (0.007497) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.581125 / 1.841788 (-0.260662) | 22.723878 / 8.074308 (14.649570) | 19.969347 / 10.191392 (9.777955) | 0.234365 / 0.680424 (-0.446059) | 0.030245 / 0.534201 (-0.503956) | 0.470843 / 0.579283 (-0.108440) | 0.558069 / 0.434364 (0.123705) | 0.534878 / 0.540337 (-0.005460) | 0.801025 / 1.386936 (-0.585911) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008524 / 0.011353 (-0.002829) | 0.005083 / 0.011008 (-0.005925) | 0.078054 / 0.038508 (0.039546) | 0.082025 / 0.023109 (0.058915) | 0.458027 / 0.275898 (0.182129) | 0.498232 / 0.323480 (0.174752) | 0.005938 / 0.007986 (-0.002048) | 0.003776 / 0.004328 (-0.000553) | 0.080413 / 0.004250 (0.076163) | 0.060485 / 0.037052 (0.023433) | 0.462816 / 0.258489 (0.204327) | 0.513970 / 0.293841 (0.220129) | 0.047574 / 0.128546 (-0.080973) | 0.013424 / 0.075646 (-0.062222) | 0.087707 / 0.419271 (-0.331565) | 0.065007 / 0.043533 (0.021474) | 0.465844 / 0.255139 (0.210705) | 0.498474 / 0.283200 (0.215274) | 0.033518 / 0.141683 (-0.108164) | 1.737507 / 1.452155 (0.285352) | 1.848291 / 1.492716 (0.355574) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.316710 / 0.018006 (0.298703) | 0.504415 / 0.000490 (0.503925) | 0.042128 / 0.000200 (0.041928) | 0.000171 / 0.000054 (0.000117) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032097 / 0.037411 (-0.005314) | 0.099371 / 0.014526 (0.084845) | 0.109311 / 0.176557 (-0.067246) | 0.177373 / 0.737135 (-0.559762) | 0.110753 / 0.296338 (-0.185585) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.688060 / 0.215209 (0.472851) | 6.255219 / 2.077655 (4.177564) | 2.696845 / 1.504120 (1.192725) | 2.395424 / 1.541195 (0.854230) | 2.414870 / 1.468490 (0.946380) | 0.865704 / 4.584777 (-3.719073) | 5.086828 / 3.745712 (1.341116) | 4.648107 / 5.269862 (-0.621754) | 3.091119 / 4.565676 (-1.474558) | 0.101787 / 0.424275 (-0.322489) | 0.008829 / 0.007607 (0.001222) | 0.772398 / 0.226044 (0.546354) | 7.700366 / 2.268929 (5.431438) | 3.608632 / 55.444624 (-51.835992) | 2.923309 / 6.876477 (-3.953168) | 2.952141 / 2.142072 (0.810069) | 1.093006 / 4.805227 (-3.712221) | 0.224363 / 6.500664 (-6.276301) | 0.074927 / 0.075469 (-0.000542) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.638414 / 1.841788 (-0.203374) | 23.486781 / 8.074308 (15.412473) | 21.129104 / 10.191392 (10.937712) | 0.259955 / 0.680424 (-0.420469) | 0.027305 / 0.534201 (-0.506895) | 0.464448 / 0.579283 (-0.114835) | 0.553737 / 0.434364 (0.119373) | 0.571318 / 0.540337 (0.030981) | 0.772917 / 1.386936 (-0.614019) |

![](https://cml.dev/watermark.png#3ec5ee9e78b464364796651d995823c7ecb0f951 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009093 / 0.011353 (-0.002260) | 0.005283 / 0.011008 (-0.005725) | 0.112299 / 0.038508 (0.073791) | 0.081341 / 0.023109 (0.058232) | 0.363799 / 0.275898 (0.087901) | 0.409261 / 0.323480 (0.085781) | 0.006400 / 0.007986 (-0.001586) | 0.003965 / 0.004328 (-0.000363) | 0.074389 / 0.004250 (0.070139) | 0.060654 / 0.037052 (0.023602) | 0.391046 / 0.258489 (0.132557) | 0.430514 / 0.293841 (0.136673) | 0.054900 / 0.128546 (-0.073646) | 0.017972 / 0.075646 (-0.057675) | 0.410875 / 0.419271 (-0.008396) | 0.067405 / 0.043533 (0.023873) | 0.371468 / 0.255139 (0.116329) | 0.435061 / 0.283200 (0.151861) | 0.038063 / 0.141683 (-0.103620) | 1.733509 / 1.452155 (0.281354) | 1.833899 / 1.492716 (0.341182) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.243230 / 0.018006 (0.225224) | 0.605636 / 0.000490 (0.605146) | 0.004890 / 0.000200 (0.004690) | 0.000098 / 0.000054 (0.000043) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027624 / 0.037411 (-0.009787) | 0.084799 / 0.014526 (0.070273) | 0.104405 / 0.176557 (-0.072152) | 0.165383 / 0.737135 (-0.571752) | 0.102083 / 0.296338 (-0.194255) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.578334 / 0.215209 (0.363125) | 5.369520 / 2.077655 (3.291866) | 2.294174 / 1.504120 (0.790055) | 2.054195 / 1.541195 (0.513000) | 2.007304 / 1.468490 (0.538814) | 0.839283 / 4.584777 (-3.745494) | 5.262288 / 3.745712 (1.516576) | 4.363346 / 5.269862 (-0.906516) | 2.854903 / 4.565676 (-1.710773) | 0.096975 / 0.424275 (-0.327300) | 0.008237 / 0.007607 (0.000630) | 0.646746 / 0.226044 (0.420702) | 6.250621 / 2.268929 (3.981693) | 2.900377 / 55.444624 (-52.544247) | 2.283238 / 6.876477 (-4.593239) | 2.443785 / 2.142072 (0.301713) | 0.991719 / 4.805227 (-3.813508) | 0.189755 / 6.500664 (-6.310909) | 0.067906 / 0.075469 (-0.007563) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.515563 / 1.841788 (-0.326225) | 21.956499 / 8.074308 (13.882191) | 19.161750 / 10.191392 (8.970358) | 0.238199 / 0.680424 (-0.442225) | 0.026771 / 0.534201 (-0.507430) | 0.450195 / 0.579283 (-0.129088) | 0.585168 / 0.434364 (0.150804) | 0.522945 / 0.540337 (-0.017393) | 0.776244 / 1.386936 (-0.610693) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007997 / 0.011353 (-0.003356) | 0.005021 / 0.011008 (-0.005988) | 0.087308 / 0.038508 (0.048800) | 0.077760 / 0.023109 (0.054650) | 0.425313 / 0.275898 (0.149415) | 0.451470 / 0.323480 (0.127990) | 0.006848 / 0.007986 (-0.001137) | 0.004812 / 0.004328 (0.000484) | 0.071198 / 0.004250 (0.066947) | 0.058325 / 0.037052 (0.021273) | 0.427411 / 0.258489 (0.168922) | 0.466069 / 0.293841 (0.172228) | 0.048686 / 0.128546 (-0.079861) | 0.011841 / 0.075646 (-0.063806) | 0.086225 / 0.419271 (-0.333047) | 0.060500 / 0.043533 (0.016967) | 0.435580 / 0.255139 (0.180441) | 0.456919 / 0.283200 (0.173719) | 0.035094 / 0.141683 (-0.106588) | 1.582805 / 1.452155 (0.130650) | 1.717838 / 1.492716 (0.225122) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.283967 / 0.018006 (0.265960) | 0.517496 / 0.000490 (0.517006) | 0.014747 / 0.000200 (0.014547) | 0.000099 / 0.000054 (0.000045) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027870 / 0.037411 (-0.009541) | 0.083835 / 0.014526 (0.069309) | 0.099157 / 0.176557 (-0.077400) | 0.173210 / 0.737135 (-0.563925) | 0.094212 / 0.296338 (-0.202127) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.535720 / 0.215209 (0.320511) | 5.273730 / 2.077655 (3.196075) | 2.422560 / 1.504120 (0.918440) | 2.131416 / 1.541195 (0.590222) | 2.192000 / 1.468490 (0.723510) | 0.708469 / 4.584777 (-3.876308) | 4.758092 / 3.745712 (1.012380) | 3.940729 / 5.269862 (-1.329133) | 2.553093 / 4.565676 (-2.012583) | 0.084895 / 0.424275 (-0.339380) | 0.008730 / 0.007607 (0.001123) | 0.646975 / 0.226044 (0.420930) | 6.294811 / 2.268929 (4.025883) | 3.293964 / 55.444624 (-52.150660) | 2.568985 / 6.876477 (-4.307492) | 2.743786 / 2.142072 (0.601713) | 0.899733 / 4.805227 (-3.905494) | 0.193484 / 6.500664 (-6.307181) | 0.070012 / 0.075469 (-0.005457) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.502255 / 1.841788 (-0.339532) | 20.690234 / 8.074308 (12.615926) | 18.375791 / 10.191392 (8.184399) | 0.200135 / 0.680424 (-0.480289) | 0.029434 / 0.534201 (-0.504767) | 0.477267 / 0.579283 (-0.102016) | 0.566869 / 0.434364 (0.132505) | 0.543756 / 0.540337 (0.003418) | 0.700476 / 1.386936 (-0.686460) |

![](https://cml.dev/watermark.png#ef17d9fd6c648bb41d43ba301c3de4d7b6f833d8 "CML watermark")


[Docs] The "Process" how-to guide lacks description of `select_columns` function

enhancement

New feature or request

### Feature request

The [how to process dataset guide](https://huggingface.co/docs/datasets/main/en/process) currently does not mention the [`select_columns`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.select_columns) function. It would be nice to include it in the guide.

### Motivation

This function is a commonly requested feature (see this [forum thread](https://discuss.huggingface.co/t/how-to-create-a-new-dataset-from-another-dataset-and-select-specific-columns-and-the-data-along-with-the-column/15120) and #5468 #5474). However, it has not been included in the guide since its implementation by PR #5480.

Mentioning it in the guide would help future users discover this added feature.

### Your contribution

I could submit a PR to add a brief description of the function to said guide.


Release: 2.14.3

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007578 / 0.011353 (-0.003775) | 0.004271 / 0.011008 (-0.006738) | 0.086607 / 0.038508 (0.048098) | 0.063209 / 0.023109 (0.040099) | 0.351724 / 0.275898 (0.075826) | 0.399261 / 0.323480 (0.075781) | 0.004767 / 0.007986 (-0.003219) | 0.003487 / 0.004328 (-0.000842) | 0.071483 / 0.004250 (0.067233) | 0.051281 / 0.037052 (0.014229) | 0.387726 / 0.258489 (0.129237) | 0.408446 / 0.293841 (0.114605) | 0.041189 / 0.128546 (-0.087357) | 0.012446 / 0.075646 (-0.063200) | 0.331147 / 0.419271 (-0.088124) | 0.056721 / 0.043533 (0.013188) | 0.361306 / 0.255139 (0.106167) | 0.409651 / 0.283200 (0.126451) | 0.035485 / 0.141683 (-0.106198) | 1.461391 / 1.452155 (0.009236) | 1.554820 / 1.492716 (0.062104) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.237119 / 0.018006 (0.219113) | 0.518731 / 0.000490 (0.518241) | 0.004192 / 0.000200 (0.003992) | 0.000114 / 0.000054 (0.000059) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024912 / 0.037411 (-0.012499) | 0.089420 / 0.014526 (0.074894) | 0.091209 / 0.176557 (-0.085347) | 0.152580 / 0.737135 (-0.584555) | 0.089660 / 0.296338 (-0.206678) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.515223 / 0.215209 (0.300014) | 5.328359 / 2.077655 (3.250705) | 1.974326 / 1.504120 (0.470206) | 1.665216 / 1.541195 (0.124021) | 1.736040 / 1.468490 (0.267550) | 0.734746 / 4.584777 (-3.850031) | 4.186613 / 3.745712 (0.440901) | 3.535760 / 5.269862 (-1.734102) | 2.333247 / 4.565676 (-2.232429) | 0.071845 / 0.424275 (-0.352430) | 0.006147 / 0.007607 (-0.001460) | 0.546649 / 0.226044 (0.320605) | 5.452281 / 2.268929 (3.183353) | 2.512984 / 55.444624 (-52.931640) | 2.104210 / 6.876477 (-4.772267) | 2.409251 / 2.142072 (0.267178) | 0.822797 / 4.805227 (-3.982430) | 0.166648 / 6.500664 (-6.334016) | 0.056350 / 0.075469 (-0.019119) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.397798 / 1.841788 (-0.443989) | 20.549399 / 8.074308 (12.475091) | 19.118168 / 10.191392 (8.926776) | 0.216361 / 0.680424 (-0.464063) | 0.027064 / 0.534201 (-0.507136) | 0.410762 / 0.579283 (-0.168521) | 0.559225 / 0.434364 (0.124861) | 0.468028 / 0.540337 (-0.072309) | 0.691520 / 1.386936 (-0.695416) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006463 / 0.011353 (-0.004890) | 0.003879 / 0.011008 (-0.007130) | 0.058723 / 0.038508 (0.020215) | 0.057202 / 0.023109 (0.034092) | 0.344397 / 0.275898 (0.068499) | 0.360388 / 0.323480 (0.036908) | 0.005502 / 0.007986 (-0.002483) | 0.004101 / 0.004328 (-0.000227) | 0.058168 / 0.004250 (0.053917) | 0.059112 / 0.037052 (0.022060) | 0.362206 / 0.258489 (0.103717) | 0.386444 / 0.293841 (0.092603) | 0.036613 / 0.128546 (-0.091934) | 0.010482 / 0.075646 (-0.065165) | 0.065850 / 0.419271 (-0.353421) | 0.046528 / 0.043533 (0.002995) | 0.349568 / 0.255139 (0.094429) | 0.360181 / 0.283200 (0.076981) | 0.029030 / 0.141683 (-0.112653) | 1.314569 / 1.452155 (-0.137586) | 1.422393 / 1.492716 (-0.070324) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.281554 / 0.018006 (0.263548) | 0.608018 / 0.000490 (0.607528) | 0.004568 / 0.000200 (0.004368) | 0.000182 / 0.000054 (0.000127) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023515 / 0.037411 (-0.013896) | 0.072994 / 0.014526 (0.058468) | 0.080688 / 0.176557 (-0.095868) | 0.125904 / 0.737135 (-0.611232) | 0.085457 / 0.296338 (-0.210882) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.471530 / 0.215209 (0.256321) | 4.796197 / 2.077655 (2.718542) | 2.189181 / 1.504120 (0.685061) | 1.886649 / 1.541195 (0.345454) | 1.871067 / 1.468490 (0.402577) | 0.661043 / 4.584777 (-3.923734) | 4.344027 / 3.745712 (0.598315) | 3.656967 / 5.269862 (-1.612895) | 2.286033 / 4.565676 (-2.279644) | 0.079146 / 0.424275 (-0.345129) | 0.006840 / 0.007607 (-0.000767) | 0.588750 / 0.226044 (0.362706) | 6.301286 / 2.268929 (4.032357) | 3.074702 / 55.444624 (-52.369923) | 2.398739 / 6.876477 (-4.477738) | 2.555057 / 2.142072 (0.412985) | 0.874189 / 4.805227 (-3.931038) | 0.191423 / 6.500664 (-6.309241) | 0.061227 / 0.075469 (-0.014242) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.472763 / 1.841788 (-0.369024) | 19.441304 / 8.074308 (11.366996) | 15.974276 / 10.191392 (5.782884) | 0.172503 / 0.680424 (-0.507921) | 0.027016 / 0.534201 (-0.507185) | 0.356085 / 0.579283 (-0.223198) | 0.473251 / 0.434364 (0.038887) | 0.427949 / 0.540337 (-0.112388) | 0.588924 / 1.386936 (-0.798013) |

![](https://cml.dev/watermark.png#0973da6e60ac7c1d24229ba6aa6881747b21858a "CML watermark")

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006166 / 0.011353 (-0.005187) | 0.003558 / 0.011008 (-0.007450) | 0.080576 / 0.038508 (0.042068) | 0.066542 / 0.023109 (0.043432) | 0.323997 / 0.275898 (0.048099) | 0.369828 / 0.323480 (0.046348) | 0.004896 / 0.007986 (-0.003090) | 0.002909 / 0.004328 (-0.001419) | 0.062553 / 0.004250 (0.058302) | 0.049795 / 0.037052 (0.012742) | 0.321369 / 0.258489 (0.062880) | 0.422860 / 0.293841 (0.129019) | 0.027394 / 0.128546 (-0.101152) | 0.007954 / 0.075646 (-0.067693) | 0.264122 / 0.419271 (-0.155149) | 0.044881 / 0.043533 (0.001349) | 0.316702 / 0.255139 (0.061563) | 0.374718 / 0.283200 (0.091518) | 0.021728 / 0.141683 (-0.119955) | 1.394456 / 1.452155 (-0.057699) | 1.474936 / 1.492716 (-0.017780) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.191902 / 0.018006 (0.173896) | 0.430468 / 0.000490 (0.429979) | 0.003790 / 0.000200 (0.003590) | 0.000069 / 0.000054 (0.000015) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024974 / 0.037411 (-0.012438) | 0.073053 / 0.014526 (0.058527) | 0.083801 / 0.176557 (-0.092756) | 0.143457 / 0.737135 (-0.593678) | 0.085099 / 0.296338 (-0.211240) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.428411 / 0.215209 (0.213202) | 4.278077 / 2.077655 (2.200422) | 2.230039 / 1.504120 (0.725919) | 2.057191 / 1.541195 (0.515996) | 2.120109 / 1.468490 (0.651619) | 0.495242 / 4.584777 (-4.089535) | 3.031299 / 3.745712 (-0.714413) | 2.802685 / 5.269862 (-2.467176) | 1.839828 / 4.565676 (-2.725849) | 0.056875 / 0.424275 (-0.367401) | 0.006446 / 0.007607 (-0.001161) | 0.498958 / 0.226044 (0.272913) | 4.980440 / 2.268929 (2.711511) | 2.659659 / 55.444624 (-52.784965) | 2.315174 / 6.876477 (-4.561303) | 2.475920 / 2.142072 (0.333848) | 0.586946 / 4.805227 (-4.218282) | 0.124291 / 6.500664 (-6.376373) | 0.060701 / 0.075469 (-0.014768) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.245062 / 1.841788 (-0.596725) | 18.201444 / 8.074308 (10.127136) | 13.723271 / 10.191392 (3.531879) | 0.130203 / 0.680424 (-0.550221) | 0.016773 / 0.534201 (-0.517428) | 0.332909 / 0.579283 (-0.246374) | 0.347469 / 0.434364 (-0.086895) | 0.381364 / 0.540337 (-0.158973) | 0.541723 / 1.386936 (-0.845213) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005934 / 0.011353 (-0.005419) | 0.003573 / 0.011008 (-0.007435) | 0.062195 / 0.038508 (0.023687) | 0.059026 / 0.023109 (0.035917) | 0.413993 / 0.275898 (0.138095) | 0.459552 / 0.323480 (0.136072) | 0.004610 / 0.007986 (-0.003376) | 0.002907 / 0.004328 (-0.001421) | 0.062983 / 0.004250 (0.058733) | 0.047797 / 0.037052 (0.010745) | 0.415461 / 0.258489 (0.156972) | 0.417424 / 0.293841 (0.123583) | 0.027098 / 0.128546 (-0.101449) | 0.008106 / 0.075646 (-0.067540) | 0.067600 / 0.419271 (-0.351672) | 0.041432 / 0.043533 (-0.002101) | 0.407861 / 0.255139 (0.152722) | 0.430774 / 0.283200 (0.147575) | 0.020738 / 0.141683 (-0.120945) | 1.435127 / 1.452155 (-0.017028) | 1.486961 / 1.492716 (-0.005755) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.231174 / 0.018006 (0.213168) | 0.421208 / 0.000490 (0.420718) | 0.005411 / 0.000200 (0.005211) | 0.000078 / 0.000054 (0.000023) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025362 / 0.037411 (-0.012049) | 0.078534 / 0.014526 (0.064008) | 0.085304 / 0.176557 (-0.091252) | 0.139048 / 0.737135 (-0.598087) | 0.087015 / 0.296338 (-0.209323) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.448506 / 0.215209 (0.233297) | 4.486694 / 2.077655 (2.409039) | 2.488022 / 1.504120 (0.983902) | 2.325321 / 1.541195 (0.784126) | 2.381311 / 1.468490 (0.912821) | 0.502102 / 4.584777 (-4.082675) | 3.018326 / 3.745712 (-0.727386) | 2.824922 / 5.269862 (-2.444940) | 1.857414 / 4.565676 (-2.708263) | 0.057514 / 0.424275 (-0.366761) | 0.006829 / 0.007607 (-0.000779) | 0.521939 / 0.226044 (0.295895) | 5.224393 / 2.268929 (2.955465) | 2.933132 / 55.444624 (-52.511492) | 2.661187 / 6.876477 (-4.215290) | 2.781950 / 2.142072 (0.639878) | 0.592927 / 4.805227 (-4.212300) | 0.126685 / 6.500664 (-6.373979) | 0.064188 / 0.075469 (-0.011281) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.351107 / 1.841788 (-0.490681) | 18.344453 / 8.074308 (10.270145) | 13.838788 / 10.191392 (3.647396) | 0.157881 / 0.680424 (-0.522543) | 0.016636 / 0.534201 (-0.517565) | 0.331597 / 0.579283 (-0.247686) | 0.345573 / 0.434364 (-0.088791) | 0.397361 / 0.540337 (-0.142976) | 0.534289 / 1.386936 (-0.852647) |

![](https://cml.dev/watermark.png#582e722a76534904c0f3038d32ebb8db88ce9128 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006399 / 0.011353 (-0.004954) | 0.003872 / 0.011008 (-0.007136) | 0.083722 / 0.038508 (0.045214) | 0.068845 / 0.023109 (0.045736) | 0.329112 / 0.275898 (0.053214) | 0.343295 / 0.323480 (0.019815) | 0.005137 / 0.007986 (-0.002849) | 0.003303 / 0.004328 (-0.001026) | 0.064495 / 0.004250 (0.060245) | 0.051448 / 0.037052 (0.014395) | 0.322554 / 0.258489 (0.064065) | 0.361934 / 0.293841 (0.068093) | 0.030821 / 0.128546 (-0.097726) | 0.008482 / 0.075646 (-0.067164) | 0.288136 / 0.419271 (-0.131135) | 0.051935 / 0.043533 (0.008402) | 0.308283 / 0.255139 (0.053144) | 0.343421 / 0.283200 (0.060221) | 0.023639 / 0.141683 (-0.118044) | 1.485442 / 1.452155 (0.033288) | 1.533282 / 1.492716 (0.040565) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.218163 / 0.018006 (0.200157) | 0.464473 / 0.000490 (0.463983) | 0.003097 / 0.000200 (0.002897) | 0.000081 / 0.000054 (0.000026) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028650 / 0.037411 (-0.008761) | 0.083295 / 0.014526 (0.068769) | 0.096468 / 0.176557 (-0.080088) | 0.152086 / 0.737135 (-0.585050) | 0.102586 / 0.296338 (-0.193752) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.393038 / 0.215209 (0.177829) | 3.925514 / 2.077655 (1.847859) | 1.938419 / 1.504120 (0.434300) | 1.760265 / 1.541195 (0.219071) | 1.810024 / 1.468490 (0.341534) | 0.486232 / 4.584777 (-4.098545) | 3.618747 / 3.745712 (-0.126965) | 3.206950 / 5.269862 (-2.062912) | 1.999240 / 4.565676 (-2.566436) | 0.056986 / 0.424275 (-0.367289) | 0.007193 / 0.007607 (-0.000415) | 0.469313 / 0.226044 (0.243269) | 4.688670 / 2.268929 (2.419741) | 2.400332 / 55.444624 (-53.044292) | 2.074197 / 6.876477 (-4.802279) | 2.290823 / 2.142072 (0.148751) | 0.582339 / 4.805227 (-4.222888) | 0.134127 / 6.500664 (-6.366537) | 0.061061 / 0.075469 (-0.014408) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.272782 / 1.841788 (-0.569006) | 19.463375 / 8.074308 (11.389067) | 14.306819 / 10.191392 (4.115427) | 0.164608 / 0.680424 (-0.515816) | 0.018626 / 0.534201 (-0.515575) | 0.395225 / 0.579283 (-0.184058) | 0.408984 / 0.434364 (-0.025380) | 0.463364 / 0.540337 (-0.076974) | 0.630425 / 1.386936 (-0.756511) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006465 / 0.011353 (-0.004888) | 0.003975 / 0.011008 (-0.007033) | 0.063643 / 0.038508 (0.025134) | 0.075214 / 0.023109 (0.052105) | 0.361734 / 0.275898 (0.085836) | 0.396664 / 0.323480 (0.073184) | 0.005251 / 0.007986 (-0.002735) | 0.003249 / 0.004328 (-0.001080) | 0.063841 / 0.004250 (0.059591) | 0.054504 / 0.037052 (0.017451) | 0.374791 / 0.258489 (0.116302) | 0.399205 / 0.293841 (0.105364) | 0.031355 / 0.128546 (-0.097192) | 0.008483 / 0.075646 (-0.067163) | 0.070234 / 0.419271 (-0.349037) | 0.048336 / 0.043533 (0.004803) | 0.373484 / 0.255139 (0.118345) | 0.382174 / 0.283200 (0.098974) | 0.022560 / 0.141683 (-0.119123) | 1.449799 / 1.452155 (-0.002355) | 1.525255 / 1.492716 (0.032539) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.228350 / 0.018006 (0.210343) | 0.444344 / 0.000490 (0.443855) | 0.003699 / 0.000200 (0.003499) | 0.000079 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030681 / 0.037411 (-0.006731) | 0.087340 / 0.014526 (0.072814) | 0.098636 / 0.176557 (-0.077920) | 0.151665 / 0.737135 (-0.585471) | 0.100840 / 0.296338 (-0.195498) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.417857 / 0.215209 (0.202648) | 4.168407 / 2.077655 (2.090752) | 2.201758 / 1.504120 (0.697638) | 1.997834 / 1.541195 (0.456639) | 2.127693 / 1.468490 (0.659202) | 0.486429 / 4.584777 (-4.098348) | 3.676335 / 3.745712 (-0.069378) | 3.226268 / 5.269862 (-2.043594) | 2.027255 / 4.565676 (-2.538422) | 0.056759 / 0.424275 (-0.367516) | 0.007628 / 0.007607 (0.000021) | 0.500482 / 0.226044 (0.274438) | 4.996236 / 2.268929 (2.727307) | 2.628884 / 55.444624 (-52.815740) | 2.347611 / 6.876477 (-4.528866) | 2.551328 / 2.142072 (0.409255) | 0.582449 / 4.805227 (-4.222778) | 0.132844 / 6.500664 (-6.367821) | 0.061791 / 0.075469 (-0.013678) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.373718 / 1.841788 (-0.468070) | 19.921217 / 8.074308 (11.846909) | 14.209642 / 10.191392 (4.018250) | 0.185334 / 0.680424 (-0.495090) | 0.018228 / 0.534201 (-0.515973) | 0.395549 / 0.579283 (-0.183734) | 0.404446 / 0.434364 (-0.029918) | 0.472456 / 0.540337 (-0.067882) | 0.622739 / 1.386936 (-0.764197) |

![](https://cml.dev/watermark.png#33f736eafa0f77de03aa6894ea4a6c923702e5d1 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006007 / 0.011353 (-0.005346) | 0.003588 / 0.011008 (-0.007420) | 0.080334 / 0.038508 (0.041826) | 0.058932 / 0.023109 (0.035823) | 0.404613 / 0.275898 (0.128715) | 0.438377 / 0.323480 (0.114897) | 0.003468 / 0.007986 (-0.004518) | 0.003702 / 0.004328 (-0.000627) | 0.062936 / 0.004250 (0.058686) | 0.047987 / 0.037052 (0.010934) | 0.411409 / 0.258489 (0.152920) | 0.450244 / 0.293841 (0.156403) | 0.027007 / 0.128546 (-0.101539) | 0.007932 / 0.075646 (-0.067714) | 0.261390 / 0.419271 (-0.157882) | 0.044992 / 0.043533 (0.001459) | 0.409730 / 0.255139 (0.154591) | 0.433331 / 0.283200 (0.150131) | 0.020446 / 0.141683 (-0.121237) | 1.425418 / 1.452155 (-0.026736) | 1.479242 / 1.492716 (-0.013475) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.187375 / 0.018006 (0.169368) | 0.428532 / 0.000490 (0.428043) | 0.003406 / 0.000200 (0.003206) | 0.000072 / 0.000054 (0.000018) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024390 / 0.037411 (-0.013022) | 0.072571 / 0.014526 (0.058045) | 0.083513 / 0.176557 (-0.093044) | 0.144395 / 0.737135 (-0.592741) | 0.084813 / 0.296338 (-0.211526) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.409176 / 0.215209 (0.193967) | 4.078082 / 2.077655 (2.000428) | 1.913596 / 1.504120 (0.409476) | 1.718470 / 1.541195 (0.177275) | 1.753106 / 1.468490 (0.284616) | 0.494167 / 4.584777 (-4.090610) | 3.029531 / 3.745712 (-0.716181) | 2.807331 / 5.269862 (-2.462531) | 1.839471 / 4.565676 (-2.726206) | 0.057169 / 0.424275 (-0.367106) | 0.006433 / 0.007607 (-0.001175) | 0.482666 / 0.226044 (0.256621) | 4.817601 / 2.268929 (2.548673) | 2.449967 / 55.444624 (-52.994658) | 2.113891 / 6.876477 (-4.762586) | 2.399293 / 2.142072 (0.257221) | 0.578903 / 4.805227 (-4.226324) | 0.124306 / 6.500664 (-6.376358) | 0.061572 / 0.075469 (-0.013897) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.254692 / 1.841788 (-0.587096) | 18.414049 / 8.074308 (10.339741) | 13.992059 / 10.191392 (3.800667) | 0.146671 / 0.680424 (-0.533753) | 0.016925 / 0.534201 (-0.517275) | 0.333124 / 0.579283 (-0.246159) | 0.348007 / 0.434364 (-0.086357) | 0.378519 / 0.540337 (-0.161819) | 0.532540 / 1.386936 (-0.854396) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006050 / 0.011353 (-0.005303) | 0.003614 / 0.011008 (-0.007394) | 0.061707 / 0.038508 (0.023199) | 0.062874 / 0.023109 (0.039765) | 0.364760 / 0.275898 (0.088862) | 0.398136 / 0.323480 (0.074656) | 0.005598 / 0.007986 (-0.002388) | 0.002836 / 0.004328 (-0.001493) | 0.061880 / 0.004250 (0.057630) | 0.048165 / 0.037052 (0.011113) | 0.372656 / 0.258489 (0.114167) | 0.403967 / 0.293841 (0.110126) | 0.027046 / 0.128546 (-0.101501) | 0.008091 / 0.075646 (-0.067555) | 0.066783 / 0.419271 (-0.352489) | 0.041186 / 0.043533 (-0.002347) | 0.376009 / 0.255139 (0.120870) | 0.391769 / 0.283200 (0.108569) | 0.021020 / 0.141683 (-0.120663) | 1.514593 / 1.452155 (0.062438) | 1.548506 / 1.492716 (0.055790) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.237610 / 0.018006 (0.219604) | 0.434274 / 0.000490 (0.433784) | 0.009720 / 0.000200 (0.009520) | 0.000098 / 0.000054 (0.000043) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025605 / 0.037411 (-0.011807) | 0.078971 / 0.014526 (0.064445) | 0.088154 / 0.176557 (-0.088403) | 0.139112 / 0.737135 (-0.598023) | 0.088890 / 0.296338 (-0.207449) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.420027 / 0.215209 (0.204818) | 4.189493 / 2.077655 (2.111838) | 2.143907 / 1.504120 (0.639787) | 1.967032 / 1.541195 (0.425837) | 2.011845 / 1.468490 (0.543355) | 0.496692 / 4.584777 (-4.088085) | 3.025456 / 3.745712 (-0.720256) | 2.828436 / 5.269862 (-2.441426) | 1.860673 / 4.565676 (-2.705003) | 0.057199 / 0.424275 (-0.367076) | 0.006770 / 0.007607 (-0.000838) | 0.491281 / 0.226044 (0.265236) | 4.918065 / 2.268929 (2.649136) | 2.593172 / 55.444624 (-52.851452) | 2.250750 / 6.876477 (-4.625727) | 2.406235 / 2.142072 (0.264162) | 0.588648 / 4.805227 (-4.216579) | 0.125635 / 6.500664 (-6.375029) | 0.061697 / 0.075469 (-0.013773) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.374065 / 1.841788 (-0.467722) | 18.439315 / 8.074308 (10.365007) | 14.031660 / 10.191392 (3.840268) | 0.153665 / 0.680424 (-0.526759) | 0.016980 / 0.534201 (-0.517221) | 0.331799 / 0.579283 (-0.247484) | 0.343201 / 0.434364 (-0.091163) | 0.392445 / 0.540337 (-0.147892) | 0.530387 / 1.386936 (-0.856549) |

![](https://cml.dev/watermark.png#33f736eafa0f77de03aa6894ea4a6c923702e5d1 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008189 / 0.011353 (-0.003164) | 0.004598 / 0.011008 (-0.006410) | 0.102199 / 0.038508 (0.063691) | 0.077961 / 0.023109 (0.054852) | 0.364936 / 0.275898 (0.089038) | 0.402606 / 0.323480 (0.079126) | 0.005522 / 0.007986 (-0.002464) | 0.004007 / 0.004328 (-0.000322) | 0.071560 / 0.004250 (0.067310) | 0.055818 / 0.037052 (0.018765) | 0.378394 / 0.258489 (0.119905) | 0.428990 / 0.293841 (0.135149) | 0.043142 / 0.128546 (-0.085404) | 0.013254 / 0.075646 (-0.062392) | 0.331102 / 0.419271 (-0.088170) | 0.061407 / 0.043533 (0.017875) | 0.387397 / 0.255139 (0.132258) | 0.416062 / 0.283200 (0.132862) | 0.036330 / 0.141683 (-0.105353) | 1.735352 / 1.452155 (0.283198) | 1.773329 / 1.492716 (0.280613) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.188587 / 0.018006 (0.170581) | 0.519506 / 0.000490 (0.519016) | 0.004702 / 0.000200 (0.004502) | 0.000097 / 0.000054 (0.000043) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027152 / 0.037411 (-0.010260) | 0.094296 / 0.014526 (0.079770) | 0.098155 / 0.176557 (-0.078402) | 0.162541 / 0.737135 (-0.574595) | 0.112092 / 0.296338 (-0.184246) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.537555 / 0.215209 (0.322346) | 5.486821 / 2.077655 (3.409166) | 2.377127 / 1.504120 (0.873008) | 2.073205 / 1.541195 (0.532011) | 2.075130 / 1.468490 (0.606640) | 0.783779 / 4.584777 (-3.800998) | 5.029524 / 3.745712 (1.283812) | 4.382724 / 5.269862 (-0.887138) | 2.836180 / 4.565676 (-1.729496) | 0.108840 / 0.424275 (-0.315435) | 0.008123 / 0.007607 (0.000516) | 0.673460 / 0.226044 (0.447416) | 6.674030 / 2.268929 (4.405102) | 3.208922 / 55.444624 (-52.235702) | 2.464908 / 6.876477 (-4.411568) | 2.661929 / 2.142072 (0.519856) | 0.962529 / 4.805227 (-3.842698) | 0.197974 / 6.500664 (-6.302690) | 0.066656 / 0.075469 (-0.008813) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.430373 / 1.841788 (-0.411415) | 21.180540 / 8.074308 (13.106232) | 19.027491 / 10.191392 (8.836099) | 0.217520 / 0.680424 (-0.462904) | 0.028038 / 0.534201 (-0.506163) | 0.435266 / 0.579283 (-0.144017) | 0.529510 / 0.434364 (0.095147) | 0.511011 / 0.540337 (-0.029327) | 0.728940 / 1.386936 (-0.657996) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007883 / 0.011353 (-0.003470) | 0.004448 / 0.011008 (-0.006560) | 0.071350 / 0.038508 (0.032842) | 0.075269 / 0.023109 (0.052160) | 0.396705 / 0.275898 (0.120807) | 0.457809 / 0.323480 (0.134329) | 0.005193 / 0.007986 (-0.002792) | 0.003695 / 0.004328 (-0.000633) | 0.078087 / 0.004250 (0.073836) | 0.054276 / 0.037052 (0.017224) | 0.412184 / 0.258489 (0.153695) | 0.452400 / 0.293841 (0.158559) | 0.049762 / 0.128546 (-0.078784) | 0.013206 / 0.075646 (-0.062440) | 0.085985 / 0.419271 (-0.333287) | 0.058837 / 0.043533 (0.015304) | 0.432481 / 0.255139 (0.177342) | 0.433260 / 0.283200 (0.150060) | 0.031190 / 0.141683 (-0.110493) | 1.582707 / 1.452155 (0.130552) | 1.664457 / 1.492716 (0.171741) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.223639 / 0.018006 (0.205633) | 0.524388 / 0.000490 (0.523899) | 0.005489 / 0.000200 (0.005289) | 0.000099 / 0.000054 (0.000045) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030182 / 0.037411 (-0.007230) | 0.089309 / 0.014526 (0.074783) | 0.103306 / 0.176557 (-0.073250) | 0.162624 / 0.737135 (-0.574511) | 0.108957 / 0.296338 (-0.187381) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.577423 / 0.215209 (0.362214) | 5.900154 / 2.077655 (3.822500) | 2.687369 / 1.504120 (1.183249) | 2.513061 / 1.541195 (0.971866) | 2.506453 / 1.468490 (1.037963) | 0.830838 / 4.584777 (-3.753939) | 5.032195 / 3.745712 (1.286483) | 4.396827 / 5.269862 (-0.873035) | 2.884230 / 4.565676 (-1.681447) | 0.102239 / 0.424275 (-0.322036) | 0.008178 / 0.007607 (0.000571) | 0.710027 / 0.226044 (0.483983) | 7.149626 / 2.268929 (4.880698) | 3.403605 / 55.444624 (-52.041019) | 2.661970 / 6.876477 (-4.214506) | 2.760227 / 2.142072 (0.618154) | 1.043981 / 4.805227 (-3.761246) | 0.195028 / 6.500664 (-6.305636) | 0.065211 / 0.075469 (-0.010258) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.581265 / 1.841788 (-0.260522) | 21.640230 / 8.074308 (13.565922) | 19.031860 / 10.191392 (8.840468) | 0.196903 / 0.680424 (-0.483520) | 0.027061 / 0.534201 (-0.507140) | 0.444995 / 0.579283 (-0.134288) | 0.528195 / 0.434364 (0.093831) | 0.521540 / 0.540337 (-0.018797) | 0.730204 / 1.386936 (-0.656732) |

![](https://cml.dev/watermark.png#33f736eafa0f77de03aa6894ea4a6c923702e5d1 "CML watermark")


Cache not being used when loading commonvoice 8.0.0

You can avoid this by using the `revision` parameter in `load_dataset` to always force downloading a specific commit (if not specified it defaults to HEAD, hence the redownload).

Thanks @mariosasko this works well, looks like I should have read the documentation a bit more carefully.

It is still a bit confusing which hash I should provide:  passing `revision = c8fd66e85f086e3abb11eeee55b1737a3d1e8487` from https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0/commits/main caused the cached version at `~/.cache/huggingface/datasets/mozilla-foundation___common_voice_8_0/en/8.0.0/b2f8b72f8f30b2e98c41ccf855954d9e35a5fa498c43332df198534ff9797a4a` to be loaded, so I had to know that it was the previous commit unless I've missed something else.

### Describe the bug

I have commonvoice 8.0.0 downloaded in `~/.cache/huggingface/datasets/mozilla-foundation___common_voice_8_0/en/8.0.0/b2f8b72f8f30b2e98c41ccf855954d9e35a5fa498c43332df198534ff9797a4a`.  The folder contains all the arrow files etc, and was used as the cached version last time I touched the ec2 instance I'm working on.  Now, with the same command that downloaded it initially:

```
dataset = load_dataset("mozilla-foundation/common_voice_8_0", "en", use_auth_token="<mytoken>")
```

it tries to redownload the dataset to `~/.cache/huggingface/datasets/mozilla-foundation___common_voice_8_0/en/8.0.0/05bdc7940b0a336ceeaeef13470c89522c29a8e4494cbeece64fb472a87acb32`

### Steps to reproduce the bug

Steps to reproduce the behavior:

1.  ```dataset = load_dataset("mozilla-foundation/common_voice_8_0", "en", use_auth_token="<mytoken>")```
2. dataset is updated by maintainers
3. ```dataset = load_dataset("mozilla-foundation/common_voice_8_0", "en", use_auth_token="<mytoken>")```

### Expected behavior

I expect that it uses the already downloaded data in `~/.cache/huggingface/datasets/mozilla-foundation___common_voice_8_0/en/8.0.0/b2f8b72f8f30b2e98c41ccf855954d9e35a5fa498c43332df198534ff9797a4a`.

Not sure what's happening in 2. but if, say it's an issue with the dataset referenced by "mozilla-foundation/common_voice_8_0" being modified by the maintainers, how would I force datasets to point to the original version I downloaded?

EDIT:  It was indeed that the maintainers had updated the dataset (v 8.0.0).  However I still cant load the dataset from disk instead of redownloading, with for example:

```
load_dataset(".cache/huggingface/datasets/downloads/extracted/<hash>/cv-corpus-8.0-2022-01-19/en/", "en")

> ...
> File [~/miniconda3/envs/aa_torch2/lib/python3.10/site-packages/datasets/table.py:1938](.../ python3.10/site-packages/datasets/table.py:1938), in cast_array_to_feature(array, feature, allow_number_to_str)
   1937 elif not isinstance(feature, (Sequence, dict, list, tuple)):
-> 1938     return array_cast(array, feature(), allow_number_to_str=allow_number_to_str)
...
   1794         e = e.__context__
-> 1795     raise DatasetGenerationError("An error occurred while generating the dataset") from e
   1797 yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_shards, shard_lengths)

DatasetGenerationError: An error occurred while generating the dataset
```

### Environment info

datasets==2.7.0
python==3.10.8
OS:  AWS Linux


load_dataset() fails with streamlit caching inside docker

Hi! This should be fixed in the latest (patch) release (run `pip install -U datasets` to install it). This behavior was due to a bug in our authentication logic.

### Describe the bug

When calling `load_dataset` in a streamlit application running within a docker container, get a failure with the error message:

EmptyDatasetError: The directory at hf://datasets/fetch-rewards/inc-rings-2000@bea27cf60842b3641eae418f38864a2ec4cde684 doesn't contain any data files
Traceback:
File "/opt/conda/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 552, in _run_script
    exec(code, module.__dict__)
File "/home/user/app/app.py", line 62, in <module>
    dashboard()
File "/home/user/app/app.py", line 47, in dashboard
    feat_dict, path_gml = load_data(hf_repo, model_gml_dict[selected_model], hf_token)
File "/opt/conda/lib/python3.10/site-packages/streamlit/runtime/caching/cache_utils.py", line 211, in wrapper
    return cached_func(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/streamlit/runtime/caching/cache_utils.py", line 240, in __call__
    return self._get_or_create_cached_value(args, kwargs)
File "/opt/conda/lib/python3.10/site-packages/streamlit/runtime/caching/cache_utils.py", line 266, in _get_or_create_cached_value
    return self._handle_cache_miss(cache, value_key, func_args, func_kwargs)
File "/opt/conda/lib/python3.10/site-packages/streamlit/runtime/caching/cache_utils.py", line 320, in _handle_cache_miss
    computed_value = self._info.func(*func_args, **func_kwargs)
File "/home/user/app/hf_interface.py", line 16, in load_data
    hf_dataset = load_dataset(repo_id, use_auth_token=hf_token)
File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 2109, in load_dataset
    builder_instance = load_dataset_builder(
File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 1795, in load_dataset_builder
    dataset_module = dataset_module_factory(
File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 1486, in dataset_module_factory
    raise e1 from None
File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 1476, in dataset_module_factory
    ).get_module()
File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 1032, in get_module
    else get_data_patterns(base_path, download_config=self.download_config)
File "/opt/conda/lib/python3.10/site-packages/datasets/data_files.py", line 458, in get_data_patterns
    raise EmptyDatasetError(f"The directory at {base_path} doesn't contain any data files") from None

### Steps to reproduce the bug

```python
@st.cache_resource
def load_data(repo_id: str, hf_token=None):
    """Load data from HuggingFace Hub
    """

    hf_dataset = load_dataset(repo_id, use_auth_token=hf_token)
    hf_dataset = hf_dataset.map(lambda x: json.loads(x["ground_truth"]), remove_columns=["ground_truth"])

    return hf_dataset
```

### Expected behavior

Expect to load.
Note: works fine with datasets==2.13.1

### Environment info

datasets==2.14.2,
Ubuntu bionic-based Docker container.


yaml error using push_to_hub with generated README.md

Thanks for reporting! This is a bug in converting the `ArrayXD` types to YAML. It will be fixed soon.

### Describe the bug

When I construct a dataset with the following features:
```
features = Features(
    {
        "pixel_values": Array3D(dtype="float64", shape=(3, 224, 224)),
        "input_ids": Sequence(feature=Value(dtype="int64")),
        "attention_mask": Sequence(Value(dtype="int64")),
        "tokens": Sequence(Value(dtype="string")),
        "bbox": Array2D(dtype="int64", shape=(512, 4)),
    }
)
```
and run `push_to_hub`, the individual `*.parquet` files are pushed, but when trying to upload the auto-generated README, I run into the following error:
```
Traceback (most recent call last):
  File "/Users/kevintee/.pyenv/versions/dev2/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 261, in hf_raise_for_status
    response.raise_for_status()
  File "/Users/kevintee/.pyenv/versions/dev2/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://huggingface.co/api/datasets/looppayments/multitask_document_classification_dataset/commit/main

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/kevintee/loop-payments/ml/src/ml/data_scripts/build_document_classification_training_data.py", line 297, in <module>
    build_dataset()
  File "/Users/kevintee/loop-payments/ml/src/ml/data_scripts/build_document_classification_training_data.py", line 290, in build_dataset
    push_to_hub(dataset, "multitask_document_classification_dataset")
  File "/Users/kevintee/loop-payments/ml/src/ml/data_scripts/build_document_classification_training_data.py", line 135, in push_to_hub
    dataset.push_to_hub(f"looppayments/{dataset_name}", private=True)
  File "/Users/kevintee/.pyenv/versions/dev2/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 5577, in push_to_hub
    HfApi(endpoint=config.HF_ENDPOINT).upload_file(
  File "/Users/kevintee/.pyenv/versions/dev2/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "/Users/kevintee/.pyenv/versions/dev2/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 828, in _inner
    return fn(self, *args, **kwargs)
  File "/Users/kevintee/.pyenv/versions/dev2/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 3221, in upload_file
    commit_info = self.create_commit(
  File "/Users/kevintee/.pyenv/versions/dev2/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "/Users/kevintee/.pyenv/versions/dev2/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 828, in _inner
    return fn(self, *args, **kwargs)
  File "/Users/kevintee/.pyenv/versions/dev2/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 2728, in create_commit
    hf_raise_for_status(commit_resp, endpoint_name="commit")
  File "/Users/kevintee/.pyenv/versions/dev2/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 299, in hf_raise_for_status
    raise BadRequestError(message, response=response) from e
huggingface_hub.utils._errors.BadRequestError:  (Request ID: Root=1-64ca9c3d-2d2bbef354e102482a9a168e;bc00371c-8549-4859-9f41-43ff140ad36e)

Bad request for commit endpoint:
Invalid YAML in README.md: unknown tag !<tag:yaml.org,2002:python/tuple> (10:9)

  7 |         - 3
  8 |         - 224
  9 |         - 224
 10 |         dtype: float64
--------------^
 11 |   - name: input_ids
 12 |     sequence: int64
```

My guess is that the auto-generated yaml is unable to be parsed for some reason.

### Steps to reproduce the bug

The description contains most of what's needed to reproduce the issue, but I've added a shortened code snippet:

```
from datasets import Array2D, Array3D, ClassLabel, Dataset, Features, Sequence, Value
from PIL import Image
from transformers import AutoProcessor

features = Features(
    {
        "pixel_values": Array3D(dtype="float64", shape=(3, 224, 224)),
        "input_ids": Sequence(feature=Value(dtype="int64")),
        "attention_mask": Sequence(Value(dtype="int64")),
        "tokens": Sequence(Value(dtype="string")),
        "bbox": Array2D(dtype="int64", shape=(512, 4)),
    }
)

processor = AutoProcessor.from_pretrained("microsoft/layoutlmv3-base", apply_ocr=False)

def preprocess_dataset(rows):
    # Get images
    images = [
        Image.open(png_filename).convert("RGB") for png_filename in rows["png_filename"]
    ]

    encoding = processor(
        images,
        rows["tokens"],
        boxes=rows["bbox"],
        truncation=True,
        padding="max_length",
    )
    encoding["tokens"] = rows["tokens"]
    return encoding

dataset = dataset.map(
    preprocess_dataset,
    batched=True,
    batch_size=5,
    features=features,
)
```

### Expected behavior

Using datasets==2.11.0, I'm able to succesfully push_to_hub, no issues, but with datasets==2.14.2, I run into the above error.

### Environment info

- `datasets` version: 2.14.2
- Platform: macOS-12.5-arm64-arm-64bit
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.3


raise FileNotFoundError("Directory {dataset_path} is neither a `Dataset` directory nor a `DatasetDict` directory." )

This should work: `load_dataset("path/to/downloaded_repo")`

`load_from_disk` is intended to be used on directories created with `Dataset.save_to_disk` or `DatasetDict.save_to_disk`

> This should work: `load_dataset("path/to/downloaded_repo")`
>
> `load_from_disk` is intended to be used on directories created with `Dataset.save_to_disk` or `DatasetDict.save_to_disk`

Thanks for your help. This works.

### Describe the bug

For researchers in some countries or regions, it is usually the case that the download ability of `load_dataset` is disabled due to the complex network environment. People in these regions often prefer to use git clone or other programming tricks to manually download the files to the disk (for example, [How to elegantly download hf models, zhihu zhuanlan](https://zhuanlan.zhihu.com/p/475260268) proposed a crawlder based solution,  and [Is there any mirror for hf_hub, zhihu answer](https://www.zhihu.com/question/371644077) provided some cloud based solutions, and [How to avoid pitfalls on Hugging face downloading, zhihu zhuanlan] gave some useful suggestions), and then use `load_from_disk` to get the dataset object.
However, when one finally has the local files on the disk, it is still buggy when trying to load the files into objects.

### Steps to reproduce the bug

Steps to reproduce the bug:
1. Found CIFAR dataset in hugging face: https://huggingface.co/datasets/cifar100/tree/main
2. Click ":" button to show "Clone repository" option, and then follow the prompts on the box:
 ```bash
  cd my_directory_absolute
  git lfs install
  git clone https://huggingface.co/datasets/cifar100
  ls  my_directory_absolute/cifar100 # confirm that the directory exists and it is OK.
  ```
3. Write A python file to try to load the dataset
```python
from datasets import load_dataset, load_from_disk
dataset = load_from_disk("my_directory_absolute/cifar100")
```
Notice that according to issue #3700 , it is wrong to use load_dataset("my_directory_absolute/cifar100"), so we must use load_from_disk instead.

4. Then you will see the error reported:
```log
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[5], line 9
      1 from datasets import load_dataset, load_from_disk
----> 9 dataset = load_from_disk("my_directory_absolute/cifar100")

File [~/miniconda3/envs/ai/lib/python3.10/site-packages/datasets/load.py:2232), in load_from_disk(dataset_path, fs, keep_in_memory, storage_options)
   2230     return DatasetDict.load_from_disk(dataset_path, keep_in_memory=keep_in_memory, storage_options=storage_options)
   2231 else:
-> 2232     raise FileNotFoundError(
   2233         f"Directory {dataset_path} is neither a `Dataset` directory nor a `DatasetDict` directory."
   2234     )

FileNotFoundError: Directory my_directory_absolute/cifar100 is neither a `Dataset` directory nor a `DatasetDict` directory.
```

### Expected behavior

The dataset should be load successfully.

### Environment info

```bash
datasets-cli env
```
-> results:
```txt

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 2.14.2
- Platform: Linux-4.18.0-372.32.1.el8_6.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
```


[BUG] Dataset initialized from in-memory data does not create cache.

This is expected behavior. You must provide `cache_file_name` when performing `.map` on an in-memory dataset for the result to be cached.

### Describe the bug

`Dataset` initialized from in-memory data (dictionary in my case, haven't tested with other types) does not create cache when processed with the `map` method, unlike `Dataset` initialized by other methods such as `load_dataset`.

### Steps to reproduce the bug

```python
# below code was run the second time so the map function can be loaded from cache if exists

from datasets import load_dataset, Dataset

dataset = load_dataset("tatsu-lab/alpaca")['train']

dataset = dataset.map(lambda x: {'input': x['input'] + 'hi'}) # some random map
print(len(dataset.cache_files))
# 1

# copy the exact same data but initialize from a dictionary
memory_dataset = Dataset.from_dict({
    'instruction': dataset['instruction'],
    'input': dataset['input'],
    'output': dataset['output'],
    'text': dataset['text']})

memory_dataset = memory_dataset.map(lambda x: {'input': x['input'] + 'hi'}) # exact same map
print(len(memory_dataset.cache_files))
# Map: 100%|██████████| 52002[/52002]
# 0
```

### Expected behavior

The `map` function should create cache regardless of the method the `Dataset` was created.

### Environment info

- `datasets` version: 2.14.2
- Platform: Linux-5.15.0-41-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.14.1
- PyArrow version: 11.0.0
- Pandas version: 1.5.3


Problems in downloading Amazon reviews from HF

Thanks for reporting, @610v4nn1.

Indeed, the source data files are no longer available. We have contacted the authors of the dataset and they report that Amazon has decided to stop distributing the multilingual reviews dataset.

We are adding a notification about this issue to the dataset card.

See: https://huggingface.co/datasets/amazon_reviews_multi/discussions/4#64c3898db63057f1fd3ce1a0

The dataset can be accessed from https://www.kaggle.com/datasets/mexwell/amazon-reviews-multi.

For those willing to transform the csv files from Kaggle into Huggingface datasets for their NLP course (exercise on summarisation), you can use this code on Google Collab:

`from datasets import load_dataset

import pandas as pd
from datasets import Dataset, DatasetDict

# Load your CSV previously downloaded files from Kaggle on Google Collab
train_csv_path = "/content/train.csv"
validation_csv_path = "/content/validation.csv"
test_csv_path = '/content/test.csv'

# Read CSV files into pandas DataFrames
train_df = pd.read_csv(train_csv_path, engine='python')
validation_df = pd.read_csv(validation_csv_path, engine='python')
test_df = pd.read_csv(test_csv_path, engine='python')

# Filter by language ('es' for Spanish and 'en' for English)
spanish_train_df = train_df[train_df['language'] == 'es']
spanish_validation_df = validation_df[validation_df['language'] == 'es']
spanish_test_df = test_df[test_df['language'] == 'es']

english_train_df = train_df[train_df['language'] == 'en']
english_validation_df = validation_df[validation_df['language'] == 'en']
english_test_df = test_df[test_df['language'] == 'en']

# Create Hugging Face datasets
spanish_dataset = DatasetDict({
    'train': Dataset.from_pandas(spanish_train_df),
    'validation': Dataset.from_pandas(spanish_validation_df),
    'test': Dataset.from_pandas(spanish_test_df)
})

english_dataset = DatasetDict({
    'train': Dataset.from_pandas(english_train_df),
    'validation': Dataset.from_pandas(english_validation_df),
    'test': Dataset.from_pandas(english_test_df)
})
english_dataset = english_dataset.remove_columns(['Unnamed: 0', '__index_level_0__'])
spanish_dataset = spanish_dataset.remove_columns(['Unnamed: 0', '__index_level_0__'])`

### Describe the bug

I have a script downloading `amazon_reviews_multi`.

When the download starts, I get

```
Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]
Downloading data: 243B [00:00, 1.43MB/s]
Downloading data files: 100%|██████████| 1/1 [00:01<00:00,  1.54s/it]
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 842.40it/s]
Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]
Downloading data: 243B [00:00, 928kB/s]
Downloading data files: 100%|██████████| 1/1 [00:01<00:00,  1.42s/it]
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 832.70it/s]
Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]
Downloading data: 243B [00:00, 1.81MB/s]
Downloading data files: 100%|██████████| 1/1 [00:01<00:00,  1.40s/it]
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 1294.14it/s]
Generating train split:   0%|          | 0/200000 [00:00<?, ? examples/s]
```

the file is clearly too small to contain the requested dataset, in fact it contains en error message:

```
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>AGJWSY3ZADT2QVWE</RequestId><HostId>Gx1O2KXnxtQFqvzDLxyVSTq3+TTJuTnuVFnJL3SP89Yp8UzvYLPTVwd1PpniE4EvQzT3tCaqEJw=</HostId></Error>
```

obviously the script fails:

```
>           raise DatasetGenerationError("An error occurred while generating the dataset") from e
E           datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
```

### Steps to reproduce the bug

1. load_dataset("amazon_reviews_multi", name="en", split="train", cache_dir="ADDYOURPATHHERE")

### Expected behavior

I would expect the dataset to be downloaded and processed

### Environment info

* The problem is present with both datasets 2.12.0 and 2.14.2
* python version 3.10.12


Loading local datasets got strangely stuck

Yesterday I waited for more than 12 hours to make sure it was really **stuck** instead of proceeding too slow.

I've had similar weird issues with `load_dataset` as well. Not multiple files, but dataset is quite big, about 50G.

We use a generic multiprocessing code, so there is little we can do about this - unfortunately, turning off multiprocessing seems to be the only solution. Multithreading would make our code easier to maintain and (most likely) avoid issues such as this one, but we cannot use it until the GIL is dropped (no-GIL Python should be released in 2024, so we can start exploring this then)

The problem seems to be the `Generating train split`. Is it possible to avoid that? I have a dataset saved, just want to load it but somehow running into issues with that again.

Hey guys, recently I ran into this problem again and I spent one whole day trying to locate the problem. Finally I found the problem seems to be with `pyarrow`'s json parser, and it seems a long-existing problem. Similar issue can be found in #2181. Anyway, my solution is to adjust the `load_dataset`'s parameter `chunksize`. You can inspect the parameter set in `datasets/packaged_modules/json/json.py`, now the actual chunksize should be very small, and you can increase the value. For me, `chunksize=10<<23` could solve the stuck problem. But I also find that too big `chunksize`, like `10 << 30`, would also cause a stuck, which is rather weird. I think I may explore this when I am free. And hope this can help those who also encounter the same problem.

Experiencing the same issue with the `kaist-ai/Feedback-Collection` dataset, which is comparatively small i.e. 100k rows.
Code to reproduce

```
from datasets import load_dataset
dataset = load_dataset("kaist-ai/Feedback-Collection")
```

I have tried setting `num_proc=1` as well as `chunksize=1024, 64` but problem persists. Any pointers?

### Describe the bug

I try to use `load_dataset()` to load several local `.jsonl` files as a dataset. Every line of these files is a json structure only containing one key `text` (yeah it is a dataset for NLP model). The code snippet is as:
```python
ds = load_dataset("json", data_files=LIST_OF_FILE_PATHS, num_proc=16)['train']
```
However, I found that the loading process can get stuck -- the progress bar `Generating train split` no more proceed. When I was trying to find the cause and solution, I found a really strange behavior. If I load the dataset in this way:
```python
dlist = list()
for _ in LIST_OF_FILE_PATHS:
    dlist.append(load_dataset("json", data_files=_)['train'])
ds = concatenate_datasets(dlist)
```
I can actually successfully load all the files despite its slow speed. But if I load them in batch like above, things go wrong. I did try to use Control-C to trace the stuck point but the program cannot be terminated in this way when `num_proc` is set to `None`. The only thing I can do is use Control-Z to hang it up then kill it. If I use more than 2 cpus, a Control-C would simply cause the following error:
```bash
^C
Process ForkPoolWorker-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/multiprocess/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.10/dist-packages/multiprocess/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.10/dist-packages/multiprocess/pool.py", line 114, in worker
    task = get()
  File "/usr/local/lib/python3.10/dist-packages/multiprocess/queues.py", line 368, in get
    res = self._reader.recv_bytes()
  File "/usr/local/lib/python3.10/dist-packages/multiprocess/connection.py", line 224, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/local/lib/python3.10/dist-packages/multiprocess/connection.py", line 422, in _recv_bytes
    buf = self._recv(4)
  File "/usr/local/lib/python3.10/dist-packages/multiprocess/connection.py", line 387, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
Generating train split: 92431 examples [01:23, 1104.25 examples/s]
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py", line 1373, in iflatmap_unordered
    yield queue.get(timeout=0.05)
  File "<string>", line 2, in get
  File "/usr/local/lib/python3.10/dist-packages/multiprocess/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "/usr/local/lib/python3.10/dist-packages/multiprocess/connection.py", line 258, in recv
    buf = self._recv_bytes()
  File "/usr/local/lib/python3.10/dist-packages/multiprocess/connection.py", line 422, in _recv_bytes
    buf = self._recv(4)
  File "/usr/local/lib/python3.10/dist-packages/multiprocess/connection.py", line 387, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/data/liyongyuan/source/batch_load.py", line 11, in <module>
    a = load_dataset(
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2133, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 954, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1049, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1842, in _prepare_split
    for job_id, done, content in iflatmap_unordered(
  File "/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py", line 1387, in iflatmap_unordered
    [async_result.get(timeout=0.05) for async_result in async_results]
  File "/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py", line 1387, in <listcomp>
    [async_result.get(timeout=0.05) for async_result in async_results]
  File "/usr/local/lib/python3.10/dist-packages/multiprocess/pool.py", line 770, in get
    raise TimeoutError
multiprocess.context.TimeoutError
```
I have validated the basic correctness of these `.jsonl` files. They are correctly formatted (or they cannot be loaded singly by `load_dataset`) though some of the json may contain too long text (more than 1e7 characters). I do not know if this could be the problem. And there should not be any bottleneck in system's resource. The whole dataset is ~300GB, and I am using a cloud server with plenty of storage and 1TB ram.
Thanks for your efforts and patience! Any suggestion or help would be appreciated.

### Steps to reproduce the bug

1. use load_dataset() with `data_files = LIST_OF_FILES`

### Expected behavior

All the files should be smoothly loaded.

### Environment info

- Datasets: A private dataset. ~2500 `.jsonl` files. ~300GB in total. Each json structure only contains one key: `text`. Format checked.
- `datasets` version: 2.14.2
- Platform: Linux-4.19.91-014.kangaroo.alios7.x86_64-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.15.1
- PyArrow version: 10.0.1.dev0+ga6eabc2b.d20230609
- Pandas version: 1.5.2