Fix authentication issues

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006103 / 0.011353 (-0.005250) | 0.003588 / 0.011008 (-0.007420) | 0.080335 / 0.038508 (0.041827) | 0.059634 / 0.023109 (0.036525) | 0.356093 / 0.275898 (0.080195) | 0.407376 / 0.323480 (0.083896) | 0.005343 / 0.007986 (-0.002643) | 0.002928 / 0.004328 (-0.001400) | 0.062580 / 0.004250 (0.058330) | 0.047544 / 0.037052 (0.010491) | 0.364305 / 0.258489 (0.105816) | 0.421463 / 0.293841 (0.127623) | 0.027249 / 0.128546 (-0.101298) | 0.008010 / 0.075646 (-0.067636) | 0.262543 / 0.419271 (-0.156728) | 0.044978 / 0.043533 (0.001445) | 0.339344 / 0.255139 (0.084205) | 0.395288 / 0.283200 (0.112088) | 0.021425 / 0.141683 (-0.120258) | 1.439767 / 1.452155 (-0.012387) | 1.498081 / 1.492716 (0.005365) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.196976 / 0.018006 (0.178970) | 0.435383 / 0.000490 (0.434893) | 0.004559 / 0.000200 (0.004359) | 0.000071 / 0.000054 (0.000016) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023653 / 0.037411 (-0.013759) | 0.072944 / 0.014526 (0.058418) | 0.083651 / 0.176557 (-0.092906) | 0.144590 / 0.737135 (-0.592545) | 0.084844 / 0.296338 (-0.211494) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.398752 / 0.215209 (0.183543) | 3.959539 / 2.077655 (1.881884) | 1.935277 / 1.504120 (0.431157) | 1.751994 / 1.541195 (0.210799) | 1.828386 / 1.468490 (0.359896) | 0.500492 / 4.584777 (-4.084284) | 3.086630 / 3.745712 (-0.659082) | 2.851664 / 5.269862 (-2.418198) | 1.869792 / 4.565676 (-2.695885) | 0.058509 / 0.424275 (-0.365766) | 0.006500 / 0.007607 (-0.001107) | 0.467468 / 0.226044 (0.241424) | 4.686168 / 2.268929 (2.417240) | 2.427632 / 55.444624 (-53.016993) | 2.193194 / 6.876477 (-4.683283) | 2.408574 / 2.142072 (0.266501) | 0.592173 / 4.805227 (-4.213054) | 0.125381 / 6.500664 (-6.375283) | 0.060679 / 0.075469 (-0.014790) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.236066 / 1.841788 (-0.605722) | 18.591689 / 8.074308 (10.517381) | 14.138774 / 10.191392 (3.947382) | 0.147455 / 0.680424 (-0.532968) | 0.016921 / 0.534201 (-0.517280) | 0.328129 / 0.579283 (-0.251154) | 0.348872 / 0.434364 (-0.085491) | 0.380311 / 0.540337 (-0.160026) | 0.532901 / 1.386936 (-0.854035) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005914 / 0.011353 (-0.005438) | 0.003614 / 0.011008 (-0.007394) | 0.062857 / 0.038508 (0.024349) | 0.060633 / 0.023109 (0.037524) | 0.419684 / 0.275898 (0.143786) | 0.449025 / 0.323480 (0.125546) | 0.004595 / 0.007986 (-0.003391) | 0.002861 / 0.004328 (-0.001467) | 0.063253 / 0.004250 (0.059003) | 0.048770 / 0.037052 (0.011718) | 0.419838 / 0.258489 (0.161349) | 0.465183 / 0.293841 (0.171342) | 0.027350 / 0.128546 (-0.101196) | 0.008065 / 0.075646 (-0.067582) | 0.068321 / 0.419271 (-0.350950) | 0.041083 / 0.043533 (-0.002449) | 0.400831 / 0.255139 (0.145692) | 0.449286 / 0.283200 (0.166086) | 0.020472 / 0.141683 (-0.121210) | 1.437215 / 1.452155 (-0.014940) | 1.503679 / 1.492716 (0.010963) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.230764 / 0.018006 (0.212758) | 0.420774 / 0.000490 (0.420285) | 0.004012 / 0.000200 (0.003812) | 0.000069 / 0.000054 (0.000014) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026009 / 0.037411 (-0.011402) | 0.077943 / 0.014526 (0.063417) | 0.087281 / 0.176557 (-0.089276) | 0.139422 / 0.737135 (-0.597713) | 0.089090 / 0.296338 (-0.207248) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.417298 / 0.215209 (0.202088) | 4.152303 / 2.077655 (2.074648) | 2.179996 / 1.504120 (0.675877) | 2.020619 / 1.541195 (0.479424) | 2.085241 / 1.468490 (0.616751) | 0.501111 / 4.584777 (-4.083666) | 3.079849 / 3.745712 (-0.665863) | 2.820607 / 5.269862 (-2.449255) | 1.863988 / 4.565676 (-2.701688) | 0.057662 / 0.424275 (-0.366613) | 0.006778 / 0.007607 (-0.000830) | 0.498661 / 0.226044 (0.272616) | 4.986503 / 2.268929 (2.717574) | 2.620676 / 55.444624 (-52.823949) | 2.297546 / 6.876477 (-4.578931) | 2.458148 / 2.142072 (0.316075) | 0.599490 / 4.805227 (-4.205738) | 0.125102 / 6.500664 (-6.375562) | 0.061411 / 0.075469 (-0.014059) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.323816 / 1.841788 (-0.517971) | 18.462614 / 8.074308 (10.388306) | 13.845826 / 10.191392 (3.654434) | 0.146115 / 0.680424 (-0.534309) | 0.016862 / 0.534201 (-0.517339) | 0.335449 / 0.579283 (-0.243834) | 0.343792 / 0.434364 (-0.090572) | 0.394068 / 0.540337 (-0.146269) | 0.536378 / 1.386936 (-0.850558) |

![](https://cml.dev/watermark.png#de3f00368c9236e9410821f5fddb95d6069883c1 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006825 / 0.011353 (-0.004527) | 0.004005 / 0.011008 (-0.007003) | 0.085504 / 0.038508 (0.046996) | 0.077252 / 0.023109 (0.054143) | 0.351891 / 0.275898 (0.075993) | 0.383404 / 0.323480 (0.059924) | 0.004153 / 0.007986 (-0.003833) | 0.003344 / 0.004328 (-0.000985) | 0.064936 / 0.004250 (0.060685) | 0.057653 / 0.037052 (0.020601) | 0.368155 / 0.258489 (0.109666) | 0.406122 / 0.293841 (0.112282) | 0.032049 / 0.128546 (-0.096497) | 0.008698 / 0.075646 (-0.066949) | 0.292394 / 0.419271 (-0.126878) | 0.053634 / 0.043533 (0.010101) | 0.358273 / 0.255139 (0.103134) | 0.378441 / 0.283200 (0.095242) | 0.026928 / 0.141683 (-0.114755) | 1.458718 / 1.452155 (0.006563) | 1.536231 / 1.492716 (0.043515) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.213956 / 0.018006 (0.195950) | 0.458620 / 0.000490 (0.458130) | 0.002718 / 0.000200 (0.002519) | 0.000078 / 0.000054 (0.000023) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027870 / 0.037411 (-0.009541) | 0.083922 / 0.014526 (0.069396) | 0.152056 / 0.176557 (-0.024501) | 0.151584 / 0.737135 (-0.585552) | 0.095698 / 0.296338 (-0.200641) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.407762 / 0.215209 (0.192553) | 4.074324 / 2.077655 (1.996669) | 2.089929 / 1.504120 (0.585809) | 1.920024 / 1.541195 (0.378829) | 2.013410 / 1.468490 (0.544920) | 0.486056 / 4.584777 (-4.098721) | 3.656869 / 3.745712 (-0.088843) | 3.304008 / 5.269862 (-1.965854) | 2.074363 / 4.565676 (-2.491313) | 0.057293 / 0.424275 (-0.366982) | 0.007240 / 0.007607 (-0.000367) | 0.482696 / 0.226044 (0.256652) | 4.833251 / 2.268929 (2.564322) | 2.570391 / 55.444624 (-52.874233) | 2.220619 / 6.876477 (-4.655857) | 2.426316 / 2.142072 (0.284243) | 0.584811 / 4.805227 (-4.220416) | 0.134907 / 6.500664 (-6.365757) | 0.061115 / 0.075469 (-0.014354) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.251969 / 1.841788 (-0.589818) | 19.601611 / 8.074308 (11.527303) | 14.190217 / 10.191392 (3.998825) | 0.166296 / 0.680424 (-0.514128) | 0.018334 / 0.534201 (-0.515867) | 0.395172 / 0.579283 (-0.184111) | 0.410440 / 0.434364 (-0.023924) | 0.462263 / 0.540337 (-0.078074) | 0.645504 / 1.386936 (-0.741432) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006991 / 0.011353 (-0.004362) | 0.004084 / 0.011008 (-0.006924) | 0.065208 / 0.038508 (0.026700) | 0.077809 / 0.023109 (0.054699) | 0.386472 / 0.275898 (0.110574) | 0.418686 / 0.323480 (0.095206) | 0.005346 / 0.007986 (-0.002640) | 0.003416 / 0.004328 (-0.000912) | 0.066209 / 0.004250 (0.061958) | 0.057517 / 0.037052 (0.020465) | 0.407684 / 0.258489 (0.149195) | 0.425438 / 0.293841 (0.131597) | 0.032166 / 0.128546 (-0.096380) | 0.008662 / 0.075646 (-0.066985) | 0.071712 / 0.419271 (-0.347560) | 0.049764 / 0.043533 (0.006231) | 0.394882 / 0.255139 (0.139743) | 0.403589 / 0.283200 (0.120389) | 0.023688 / 0.141683 (-0.117995) | 1.468488 / 1.452155 (0.016334) | 1.533118 / 1.492716 (0.040401) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.252949 / 0.018006 (0.234943) | 0.447355 / 0.000490 (0.446865) | 0.011721 / 0.000200 (0.011521) | 0.000107 / 0.000054 (0.000052) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031444 / 0.037411 (-0.005968) | 0.089390 / 0.014526 (0.074864) | 0.100103 / 0.176557 (-0.076454) | 0.153301 / 0.737135 (-0.583835) | 0.101336 / 0.296338 (-0.195003) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.408574 / 0.215209 (0.193365) | 4.073135 / 2.077655 (1.995480) | 2.086550 / 1.504120 (0.582430) | 1.930651 / 1.541195 (0.389457) | 2.013548 / 1.468490 (0.545058) | 0.477235 / 4.584777 (-4.107542) | 3.547545 / 3.745712 (-0.198167) | 3.321957 / 5.269862 (-1.947905) | 2.057705 / 4.565676 (-2.507971) | 0.056730 / 0.424275 (-0.367545) | 0.007882 / 0.007607 (0.000275) | 0.487297 / 0.226044 (0.261253) | 4.874184 / 2.268929 (2.605255) | 2.631129 / 55.444624 (-52.813496) | 2.235755 / 6.876477 (-4.640722) | 2.463329 / 2.142072 (0.321257) | 0.578308 / 4.805227 (-4.226919) | 0.132726 / 6.500664 (-6.367938) | 0.064883 / 0.075469 (-0.010586) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.347564 / 1.841788 (-0.494223) | 20.192973 / 8.074308 (12.118665) | 14.563553 / 10.191392 (4.372161) | 0.168244 / 0.680424 (-0.512180) | 0.018638 / 0.534201 (-0.515563) | 0.394789 / 0.579283 (-0.184494) | 0.419677 / 0.434364 (-0.014687) | 0.480274 / 0.540337 (-0.060063) | 0.641204 / 1.386936 (-0.745732) |

![](https://cml.dev/watermark.png#9c7a0d56b60bf700d6a491fa30eaf66500969315 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005939 / 0.011353 (-0.005413) | 0.003457 / 0.011008 (-0.007551) | 0.079985 / 0.038508 (0.041477) | 0.056492 / 0.023109 (0.033383) | 0.312356 / 0.275898 (0.036458) | 0.354038 / 0.323480 (0.030558) | 0.004551 / 0.007986 (-0.003435) | 0.002828 / 0.004328 (-0.001501) | 0.062369 / 0.004250 (0.058119) | 0.044712 / 0.037052 (0.007660) | 0.318244 / 0.258489 (0.059755) | 0.361977 / 0.293841 (0.068136) | 0.026460 / 0.128546 (-0.102086) | 0.007928 / 0.075646 (-0.067719) | 0.261378 / 0.419271 (-0.157894) | 0.044209 / 0.043533 (0.000676) | 0.313931 / 0.255139 (0.058792) | 0.339553 / 0.283200 (0.056354) | 0.019776 / 0.141683 (-0.121907) | 1.443126 / 1.452155 (-0.009029) | 1.508149 / 1.492716 (0.015432) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.183801 / 0.018006 (0.165795) | 0.427967 / 0.000490 (0.427477) | 0.002028 / 0.000200 (0.001828) | 0.000062 / 0.000054 (0.000007) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023697 / 0.037411 (-0.013715) | 0.072128 / 0.014526 (0.057602) | 0.083701 / 0.176557 (-0.092855) | 0.142821 / 0.737135 (-0.594315) | 0.082276 / 0.296338 (-0.214063) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.434427 / 0.215209 (0.219218) | 4.325962 / 2.077655 (2.248308) | 2.277115 / 1.504120 (0.772995) | 2.093736 / 1.541195 (0.552541) | 2.127984 / 1.468490 (0.659494) | 0.502336 / 4.584777 (-4.082441) | 3.023243 / 3.745712 (-0.722469) | 2.805154 / 5.269862 (-2.464708) | 1.821273 / 4.565676 (-2.744403) | 0.057480 / 0.424275 (-0.366795) | 0.006365 / 0.007607 (-0.001242) | 0.508258 / 0.226044 (0.282213) | 5.087950 / 2.268929 (2.819022) | 2.705029 / 55.444624 (-52.739596) | 2.378392 / 6.876477 (-4.498085) | 2.515380 / 2.142072 (0.373307) | 0.589283 / 4.805227 (-4.215944) | 0.125719 / 6.500664 (-6.374945) | 0.061074 / 0.075469 (-0.014395) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.221895 / 1.841788 (-0.619893) | 18.025917 / 8.074308 (9.951609) | 13.556901 / 10.191392 (3.365509) | 0.142614 / 0.680424 (-0.537809) | 0.016731 / 0.534201 (-0.517469) | 0.328374 / 0.579283 (-0.250910) | 0.342553 / 0.434364 (-0.091811) | 0.374502 / 0.540337 (-0.165836) | 0.534173 / 1.386936 (-0.852763) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005817 / 0.011353 (-0.005536) | 0.003500 / 0.011008 (-0.007509) | 0.062240 / 0.038508 (0.023732) | 0.058128 / 0.023109 (0.035019) | 0.424014 / 0.275898 (0.148116) | 0.468453 / 0.323480 (0.144973) | 0.004641 / 0.007986 (-0.003345) | 0.002821 / 0.004328 (-0.001508) | 0.062180 / 0.004250 (0.057930) | 0.047578 / 0.037052 (0.010526) | 0.427367 / 0.258489 (0.168878) | 0.467889 / 0.293841 (0.174048) | 0.027144 / 0.128546 (-0.101403) | 0.007969 / 0.075646 (-0.067678) | 0.067764 / 0.419271 (-0.351508) | 0.040719 / 0.043533 (-0.002814) | 0.423663 / 0.255139 (0.168524) | 0.458556 / 0.283200 (0.175356) | 0.019196 / 0.141683 (-0.122487) | 1.471546 / 1.452155 (0.019392) | 1.547541 / 1.492716 (0.054825) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.228777 / 0.018006 (0.210770) | 0.406663 / 0.000490 (0.406173) | 0.003688 / 0.000200 (0.003488) | 0.000075 / 0.000054 (0.000021) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025494 / 0.037411 (-0.011917) | 0.076339 / 0.014526 (0.061814) | 0.084233 / 0.176557 (-0.092324) | 0.136995 / 0.737135 (-0.600140) | 0.085443 / 0.296338 (-0.210895) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.420441 / 0.215209 (0.205232) | 4.187018 / 2.077655 (2.109363) | 2.142139 / 1.504120 (0.638019) | 1.974530 / 1.541195 (0.433335) | 2.027321 / 1.468490 (0.558831) | 0.498116 / 4.584777 (-4.086661) | 2.988514 / 3.745712 (-0.757198) | 2.782046 / 5.269862 (-2.487816) | 1.821725 / 4.565676 (-2.743951) | 0.057711 / 0.424275 (-0.366564) | 0.006664 / 0.007607 (-0.000944) | 0.491015 / 0.226044 (0.264971) | 4.921037 / 2.268929 (2.652108) | 2.574964 / 55.444624 (-52.869661) | 2.251703 / 6.876477 (-4.624774) | 2.361154 / 2.142072 (0.219082) | 0.593362 / 4.805227 (-4.211865) | 0.126107 / 6.500664 (-6.374557) | 0.061840 / 0.075469 (-0.013630) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.327459 / 1.841788 (-0.514328) | 18.062960 / 8.074308 (9.988652) | 13.669253 / 10.191392 (3.477861) | 0.130719 / 0.680424 (-0.549705) | 0.016564 / 0.534201 (-0.517637) | 0.335821 / 0.579283 (-0.243462) | 0.341691 / 0.434364 (-0.092673) | 0.392651 / 0.540337 (-0.147686) | 0.529650 / 1.386936 (-0.857286) |

![](https://cml.dev/watermark.png#c65806b0542996e56825ab46a3ce8f9c07ab0df3 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009625 / 0.011353 (-0.001728) | 0.005354 / 0.011008 (-0.005654) | 0.114350 / 0.038508 (0.075842) | 0.086637 / 0.023109 (0.063528) | 0.465381 / 0.275898 (0.189483) | 0.490411 / 0.323480 (0.166931) | 0.006575 / 0.007986 (-0.001411) | 0.004287 / 0.004328 (-0.000041) | 0.093134 / 0.004250 (0.088884) | 0.060209 / 0.037052 (0.023156) | 0.459570 / 0.258489 (0.201080) | 0.523320 / 0.293841 (0.229479) | 0.047943 / 0.128546 (-0.080603) | 0.014764 / 0.075646 (-0.060882) | 0.383887 / 0.419271 (-0.035384) | 0.069864 / 0.043533 (0.026331) | 0.469122 / 0.255139 (0.213983) | 0.509953 / 0.283200 (0.226753) | 0.037800 / 0.141683 (-0.103883) | 1.877589 / 1.452155 (0.425434) | 2.014913 / 1.492716 (0.522197) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.309146 / 0.018006 (0.291140) | 0.644390 / 0.000490 (0.643900) | 0.005017 / 0.000200 (0.004817) | 0.000102 / 0.000054 (0.000048) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032964 / 0.037411 (-0.004447) | 0.103236 / 0.014526 (0.088711) | 0.119950 / 0.176557 (-0.056607) | 0.207674 / 0.737135 (-0.529461) | 0.117278 / 0.296338 (-0.179060) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.605464 / 0.215209 (0.390255) | 6.027805 / 2.077655 (3.950150) | 2.719725 / 1.504120 (1.215605) | 2.262752 / 1.541195 (0.721558) | 2.330310 / 1.468490 (0.861820) | 0.862537 / 4.584777 (-3.722240) | 5.347080 / 3.745712 (1.601368) | 4.792170 / 5.269862 (-0.477691) | 3.103694 / 4.565676 (-1.461983) | 0.103646 / 0.424275 (-0.320629) | 0.009411 / 0.007607 (0.001804) | 0.743052 / 0.226044 (0.517008) | 7.289684 / 2.268929 (5.020755) | 3.436530 / 55.444624 (-52.008094) | 2.722440 / 6.876477 (-4.154036) | 2.952380 / 2.142072 (0.810308) | 1.047688 / 4.805227 (-3.757539) | 0.212724 / 6.500664 (-6.287940) | 0.081473 / 0.075469 (0.006004) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.714437 / 1.841788 (-0.127351) | 24.384330 / 8.074308 (16.310022) | 22.444162 / 10.191392 (12.252770) | 0.226264 / 0.680424 (-0.454160) | 0.030530 / 0.534201 (-0.503671) | 0.473999 / 0.579283 (-0.105284) | 0.575005 / 0.434364 (0.140641) | 0.542789 / 0.540337 (0.002451) | 0.776079 / 1.386936 (-0.610857) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009366 / 0.011353 (-0.001987) | 0.005239 / 0.011008 (-0.005769) | 0.085116 / 0.038508 (0.046608) | 0.089600 / 0.023109 (0.066491) | 0.485778 / 0.275898 (0.209880) | 0.540054 / 0.323480 (0.216574) | 0.006290 / 0.007986 (-0.001695) | 0.004054 / 0.004328 (-0.000274) | 0.083535 / 0.004250 (0.079284) | 0.067200 / 0.037052 (0.030148) | 0.519520 / 0.258489 (0.261031) | 0.544049 / 0.293841 (0.250208) | 0.054300 / 0.128546 (-0.074246) | 0.013650 / 0.075646 (-0.061996) | 0.102515 / 0.419271 (-0.316757) | 0.063054 / 0.043533 (0.019522) | 0.491724 / 0.255139 (0.236585) | 0.547498 / 0.283200 (0.264298) | 0.039266 / 0.141683 (-0.102416) | 1.801226 / 1.452155 (0.349071) | 1.861778 / 1.492716 (0.369061) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.313009 / 0.018006 (0.295003) | 0.587695 / 0.000490 (0.587205) | 0.004972 / 0.000200 (0.004772) | 0.000110 / 0.000054 (0.000055) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029230 / 0.037411 (-0.008181) | 0.091154 / 0.014526 (0.076628) | 0.110505 / 0.176557 (-0.066052) | 0.164204 / 0.737135 (-0.572932) | 0.107812 / 0.296338 (-0.188526) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.610535 / 0.215209 (0.395326) | 6.162517 / 2.077655 (4.084862) | 2.866718 / 1.504120 (1.362598) | 2.542412 / 1.541195 (1.001218) | 2.584136 / 1.468490 (1.115645) | 0.874319 / 4.584777 (-3.710458) | 5.257184 / 3.745712 (1.511472) | 4.705840 / 5.269862 (-0.564022) | 2.971708 / 4.565676 (-1.593969) | 0.099026 / 0.424275 (-0.325249) | 0.009142 / 0.007607 (0.001535) | 0.728660 / 0.226044 (0.502615) | 7.560922 / 2.268929 (5.291994) | 3.439521 / 55.444624 (-52.005103) | 2.854730 / 6.876477 (-4.021746) | 3.088951 / 2.142072 (0.946879) | 0.973621 / 4.805227 (-3.831606) | 0.209792 / 6.500664 (-6.290872) | 0.081107 / 0.075469 (0.005638) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.716809 / 1.841788 (-0.124978) | 24.386927 / 8.074308 (16.312619) | 20.715524 / 10.191392 (10.524131) | 0.260831 / 0.680424 (-0.419592) | 0.030701 / 0.534201 (-0.503500) | 0.490018 / 0.579283 (-0.089265) | 0.590424 / 0.434364 (0.156060) | 0.589942 / 0.540337 (0.049604) | 0.798094 / 1.386936 (-0.588842) |

![](https://cml.dev/watermark.png#c0a77dc943de68a17f23f141517028c734c78623 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006592 / 0.011353 (-0.004761) | 0.003880 / 0.011008 (-0.007128) | 0.083761 / 0.038508 (0.045253) | 0.075966 / 0.023109 (0.052857) | 0.315291 / 0.275898 (0.039393) | 0.355920 / 0.323480 (0.032440) | 0.004972 / 0.007986 (-0.003014) | 0.003053 / 0.004328 (-0.001275) | 0.063553 / 0.004250 (0.059302) | 0.050794 / 0.037052 (0.013742) | 0.317681 / 0.258489 (0.059192) | 0.361991 / 0.293841 (0.068150) | 0.028119 / 0.128546 (-0.100427) | 0.008203 / 0.075646 (-0.067443) | 0.271756 / 0.419271 (-0.147516) | 0.046701 / 0.043533 (0.003168) | 0.316520 / 0.255139 (0.061381) | 0.350499 / 0.283200 (0.067300) | 0.022399 / 0.141683 (-0.119284) | 1.416017 / 1.452155 (-0.036138) | 1.503087 / 1.492716 (0.010371) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.208250 / 0.018006 (0.190244) | 0.470345 / 0.000490 (0.469856) | 0.003687 / 0.000200 (0.003487) | 0.000073 / 0.000054 (0.000019) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026163 / 0.037411 (-0.011248) | 0.083315 / 0.014526 (0.068789) | 0.088541 / 0.176557 (-0.088015) | 0.150078 / 0.737135 (-0.587057) | 0.088862 / 0.296338 (-0.207476) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.404911 / 0.215209 (0.189702) | 4.059257 / 2.077655 (1.981602) | 1.890987 / 1.504120 (0.386867) | 1.726608 / 1.541195 (0.185413) | 1.767479 / 1.468490 (0.298989) | 0.518826 / 4.584777 (-4.065951) | 3.212145 / 3.745712 (-0.533567) | 3.029933 / 5.269862 (-2.239929) | 2.000203 / 4.565676 (-2.565474) | 0.059631 / 0.424275 (-0.364644) | 0.006707 / 0.007607 (-0.000900) | 0.485741 / 0.226044 (0.259697) | 4.871938 / 2.268929 (2.603010) | 2.418856 / 55.444624 (-53.025769) | 2.084847 / 6.876477 (-4.791630) | 2.207992 / 2.142072 (0.065920) | 0.614354 / 4.805227 (-4.190873) | 0.128932 / 6.500664 (-6.371732) | 0.062342 / 0.075469 (-0.013127) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.325792 / 1.841788 (-0.515995) | 19.718995 / 8.074308 (11.644687) | 15.278535 / 10.191392 (5.087143) | 0.146719 / 0.680424 (-0.533705) | 0.017718 / 0.534201 (-0.516483) | 0.335709 / 0.579283 (-0.243574) | 0.378060 / 0.434364 (-0.056304) | 0.391135 / 0.540337 (-0.149202) | 0.548045 / 1.386936 (-0.838891) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006504 / 0.011353 (-0.004849) | 0.003742 / 0.011008 (-0.007266) | 0.064405 / 0.038508 (0.025897) | 0.077618 / 0.023109 (0.054509) | 0.365325 / 0.275898 (0.089427) | 0.408109 / 0.323480 (0.084629) | 0.004909 / 0.007986 (-0.003076) | 0.002972 / 0.004328 (-0.001356) | 0.063933 / 0.004250 (0.059682) | 0.052916 / 0.037052 (0.015863) | 0.370891 / 0.258489 (0.112402) | 0.412134 / 0.293841 (0.118293) | 0.028171 / 0.128546 (-0.100375) | 0.008150 / 0.075646 (-0.067497) | 0.069248 / 0.419271 (-0.350024) | 0.042353 / 0.043533 (-0.001180) | 0.368117 / 0.255139 (0.112978) | 0.397548 / 0.283200 (0.114348) | 0.022967 / 0.141683 (-0.118716) | 1.472740 / 1.452155 (0.020586) | 1.524028 / 1.492716 (0.031311) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.256854 / 0.018006 (0.238848) | 0.471499 / 0.000490 (0.471009) | 0.009609 / 0.000200 (0.009409) | 0.000109 / 0.000054 (0.000054) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027978 / 0.037411 (-0.009433) | 0.086741 / 0.014526 (0.072215) | 0.091189 / 0.176557 (-0.085368) | 0.146117 / 0.737135 (-0.591018) | 0.092358 / 0.296338 (-0.203980) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.426356 / 0.215209 (0.211147) | 4.263782 / 2.077655 (2.186127) | 2.178198 / 1.504120 (0.674078) | 2.015405 / 1.541195 (0.474211) | 2.055966 / 1.468490 (0.587476) | 0.507531 / 4.584777 (-4.077246) | 3.175967 / 3.745712 (-0.569745) | 3.055697 / 5.269862 (-2.214165) | 1.987663 / 4.565676 (-2.578014) | 0.058452 / 0.424275 (-0.365823) | 0.006944 / 0.007607 (-0.000663) | 0.502534 / 0.226044 (0.276489) | 5.024693 / 2.268929 (2.755765) | 2.754971 / 55.444624 (-52.689653) | 2.470845 / 6.876477 (-4.405632) | 2.698675 / 2.142072 (0.556602) | 0.602357 / 4.805227 (-4.202871) | 0.129490 / 6.500664 (-6.371174) | 0.065127 / 0.075469 (-0.010342) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.398487 / 1.841788 (-0.443301) | 19.692279 / 8.074308 (11.617971) | 15.124064 / 10.191392 (4.932672) | 0.148938 / 0.680424 (-0.531486) | 0.017418 / 0.534201 (-0.516783) | 0.340480 / 0.579283 (-0.238803) | 0.377223 / 0.434364 (-0.057141) | 0.405303 / 0.540337 (-0.135034) | 0.548923 / 1.386936 (-0.838013) |

![](https://cml.dev/watermark.png#58e62af004b6b8b84dcfd897a4bc71637cfa6c3f "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006433 / 0.011353 (-0.004920) | 0.004002 / 0.011008 (-0.007006) | 0.084130 / 0.038508 (0.045622) | 0.070628 / 0.023109 (0.047519) | 0.312372 / 0.275898 (0.036474) | 0.343993 / 0.323480 (0.020513) | 0.003936 / 0.007986 (-0.004050) | 0.003336 / 0.004328 (-0.000993) | 0.064715 / 0.004250 (0.060465) | 0.052511 / 0.037052 (0.015458) | 0.314092 / 0.258489 (0.055603) | 0.363152 / 0.293841 (0.069311) | 0.030898 / 0.128546 (-0.097648) | 0.008396 / 0.075646 (-0.067250) | 0.288083 / 0.419271 (-0.131188) | 0.051654 / 0.043533 (0.008122) | 0.315252 / 0.255139 (0.060113) | 0.346756 / 0.283200 (0.063556) | 0.025167 / 0.141683 (-0.116515) | 1.487265 / 1.452155 (0.035110) | 1.557528 / 1.492716 (0.064812) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.206517 / 0.018006 (0.188510) | 0.458359 / 0.000490 (0.457869) | 0.003719 / 0.000200 (0.003519) | 0.000070 / 0.000054 (0.000016) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029631 / 0.037411 (-0.007780) | 0.083856 / 0.014526 (0.069330) | 0.340431 / 0.176557 (0.163875) | 0.153864 / 0.737135 (-0.583271) | 0.095951 / 0.296338 (-0.200388) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.379182 / 0.215209 (0.163973) | 3.783396 / 2.077655 (1.705741) | 1.835932 / 1.504120 (0.331813) | 1.667563 / 1.541195 (0.126369) | 1.739309 / 1.468490 (0.270818) | 0.478957 / 4.584777 (-4.105820) | 3.521974 / 3.745712 (-0.223738) | 3.237635 / 5.269862 (-2.032227) | 2.000300 / 4.565676 (-2.565377) | 0.056389 / 0.424275 (-0.367887) | 0.007242 / 0.007607 (-0.000365) | 0.452642 / 0.226044 (0.226598) | 4.524339 / 2.268929 (2.255411) | 2.346210 / 55.444624 (-53.098414) | 1.957196 / 6.876477 (-4.919281) | 2.180051 / 2.142072 (0.037979) | 0.570205 / 4.805227 (-4.235022) | 0.131346 / 6.500664 (-6.369318) | 0.059327 / 0.075469 (-0.016142) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.244709 / 1.841788 (-0.597079) | 19.566277 / 8.074308 (11.491969) | 14.172598 / 10.191392 (3.981206) | 0.166493 / 0.680424 (-0.513931) | 0.018281 / 0.534201 (-0.515920) | 0.391608 / 0.579283 (-0.187675) | 0.402642 / 0.434364 (-0.031722) | 0.464974 / 0.540337 (-0.075364) | 0.637565 / 1.386936 (-0.749371) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006929 / 0.011353 (-0.004424) | 0.004114 / 0.011008 (-0.006894) | 0.064589 / 0.038508 (0.026081) | 0.083334 / 0.023109 (0.060225) | 0.391280 / 0.275898 (0.115382) | 0.426157 / 0.323480 (0.102678) | 0.005336 / 0.007986 (-0.002650) | 0.003395 / 0.004328 (-0.000934) | 0.064560 / 0.004250 (0.060310) | 0.057094 / 0.037052 (0.020042) | 0.398959 / 0.258489 (0.140470) | 0.432470 / 0.293841 (0.138629) | 0.031412 / 0.128546 (-0.097134) | 0.008670 / 0.075646 (-0.066976) | 0.071249 / 0.419271 (-0.348022) | 0.048934 / 0.043533 (0.005401) | 0.384207 / 0.255139 (0.129068) | 0.407992 / 0.283200 (0.124792) | 0.024492 / 0.141683 (-0.117191) | 1.467788 / 1.452155 (0.015634) | 1.541011 / 1.492716 (0.048295) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.279607 / 0.018006 (0.261600) | 0.448899 / 0.000490 (0.448410) | 0.020990 / 0.000200 (0.020790) | 0.000132 / 0.000054 (0.000078) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030313 / 0.037411 (-0.007099) | 0.089209 / 0.014526 (0.074684) | 0.101024 / 0.176557 (-0.075532) | 0.153468 / 0.737135 (-0.583667) | 0.103219 / 0.296338 (-0.193120) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.429176 / 0.215209 (0.213967) | 4.302234 / 2.077655 (2.224580) | 2.291103 / 1.504120 (0.786983) | 2.126257 / 1.541195 (0.585062) | 2.207090 / 1.468490 (0.738600) | 0.484643 / 4.584777 (-4.100134) | 3.557429 / 3.745712 (-0.188283) | 3.253804 / 5.269862 (-2.016058) | 2.026087 / 4.565676 (-2.539589) | 0.057793 / 0.424275 (-0.366482) | 0.007761 / 0.007607 (0.000154) | 0.504819 / 0.226044 (0.278775) | 5.046868 / 2.268929 (2.777940) | 2.773149 / 55.444624 (-52.671475) | 2.398036 / 6.876477 (-4.478440) | 2.608094 / 2.142072 (0.466021) | 0.630499 / 4.805227 (-4.174729) | 0.135496 / 6.500664 (-6.365168) | 0.061329 / 0.075469 (-0.014140) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.327124 / 1.841788 (-0.514664) | 19.889796 / 8.074308 (11.815488) | 14.196100 / 10.191392 (4.004708) | 0.161963 / 0.680424 (-0.518461) | 0.018529 / 0.534201 (-0.515672) | 0.392325 / 0.579283 (-0.186958) | 0.404836 / 0.434364 (-0.029528) | 0.475898 / 0.540337 (-0.064439) | 0.633563 / 1.386936 (-0.753373) |

![](https://cml.dev/watermark.png#e4684fc1032321abf0d494b0c130ea7c82ebda80 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006390 / 0.011353 (-0.004963) | 0.003683 / 0.011008 (-0.007325) | 0.081274 / 0.038508 (0.042766) | 0.062193 / 0.023109 (0.039083) | 0.355360 / 0.275898 (0.079462) | 0.396471 / 0.323480 (0.072992) | 0.003569 / 0.007986 (-0.004416) | 0.003928 / 0.004328 (-0.000400) | 0.062292 / 0.004250 (0.058041) | 0.049700 / 0.037052 (0.012648) | 0.354604 / 0.258489 (0.096115) | 0.419436 / 0.293841 (0.125595) | 0.027151 / 0.128546 (-0.101395) | 0.007954 / 0.075646 (-0.067692) | 0.262231 / 0.419271 (-0.157041) | 0.045483 / 0.043533 (0.001950) | 0.354285 / 0.255139 (0.099146) | 0.385178 / 0.283200 (0.101978) | 0.021183 / 0.141683 (-0.120500) | 1.420785 / 1.452155 (-0.031370) | 1.531545 / 1.492716 (0.038829) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.202298 / 0.018006 (0.184292) | 0.442172 / 0.000490 (0.441683) | 0.003565 / 0.000200 (0.003366) | 0.000074 / 0.000054 (0.000020) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024229 / 0.037411 (-0.013183) | 0.074352 / 0.014526 (0.059826) | 0.087530 / 0.176557 (-0.089026) | 0.146478 / 0.737135 (-0.590658) | 0.085145 / 0.296338 (-0.211194) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.388395 / 0.215209 (0.173186) | 3.877623 / 2.077655 (1.799968) | 1.882444 / 1.504120 (0.378324) | 1.707871 / 1.541195 (0.166676) | 1.772132 / 1.468490 (0.303642) | 0.491937 / 4.584777 (-4.092840) | 3.057947 / 3.745712 (-0.687765) | 2.822390 / 5.269862 (-2.447471) | 1.879719 / 4.565676 (-2.685957) | 0.056830 / 0.424275 (-0.367445) | 0.006415 / 0.007607 (-0.001192) | 0.458945 / 0.226044 (0.232900) | 4.594502 / 2.268929 (2.325574) | 2.339677 / 55.444624 (-53.104948) | 1.983750 / 6.876477 (-4.892727) | 2.173792 / 2.142072 (0.031719) | 0.580390 / 4.805227 (-4.224838) | 0.124568 / 6.500664 (-6.376096) | 0.061694 / 0.075469 (-0.013775) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.265108 / 1.841788 (-0.576680) | 18.415254 / 8.074308 (10.340946) | 13.963829 / 10.191392 (3.772437) | 0.148926 / 0.680424 (-0.531498) | 0.016919 / 0.534201 (-0.517282) | 0.331082 / 0.579283 (-0.248201) | 0.345777 / 0.434364 (-0.088587) | 0.381123 / 0.540337 (-0.159214) | 0.543297 / 1.386936 (-0.843639) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006121 / 0.011353 (-0.005232) | 0.003717 / 0.011008 (-0.007291) | 0.063653 / 0.038508 (0.025144) | 0.063723 / 0.023109 (0.040613) | 0.360233 / 0.275898 (0.084335) | 0.398353 / 0.323480 (0.074873) | 0.004696 / 0.007986 (-0.003290) | 0.002876 / 0.004328 (-0.001452) | 0.063057 / 0.004250 (0.058806) | 0.050258 / 0.037052 (0.013206) | 0.362946 / 0.258489 (0.104457) | 0.403260 / 0.293841 (0.109419) | 0.027738 / 0.128546 (-0.100809) | 0.008025 / 0.075646 (-0.067621) | 0.068781 / 0.419271 (-0.350491) | 0.042114 / 0.043533 (-0.001419) | 0.363546 / 0.255139 (0.108407) | 0.385640 / 0.283200 (0.102440) | 0.021757 / 0.141683 (-0.119926) | 1.482364 / 1.452155 (0.030209) | 1.571859 / 1.492716 (0.079143) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.235628 / 0.018006 (0.217622) | 0.439909 / 0.000490 (0.439419) | 0.003070 / 0.000200 (0.002870) | 0.000075 / 0.000054 (0.000020) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027045 / 0.037411 (-0.010366) | 0.080413 / 0.014526 (0.065887) | 0.088953 / 0.176557 (-0.087603) | 0.141907 / 0.737135 (-0.595228) | 0.090604 / 0.296338 (-0.205735) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.423250 / 0.215209 (0.208041) | 4.216510 / 2.077655 (2.138855) | 2.162946 / 1.504120 (0.658826) | 2.014561 / 1.541195 (0.473366) | 2.086347 / 1.468490 (0.617857) | 0.496591 / 4.584777 (-4.088186) | 3.089594 / 3.745712 (-0.656118) | 2.853640 / 5.269862 (-2.416221) | 1.878149 / 4.565676 (-2.687527) | 0.056914 / 0.424275 (-0.367361) | 0.006762 / 0.007607 (-0.000845) | 0.493470 / 0.226044 (0.267426) | 4.929966 / 2.268929 (2.661037) | 2.640885 / 55.444624 (-52.803739) | 2.335950 / 6.876477 (-4.540527) | 2.565866 / 2.142072 (0.423793) | 0.585433 / 4.805227 (-4.219794) | 0.124969 / 6.500664 (-6.375695) | 0.062361 / 0.075469 (-0.013108) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.369144 / 1.841788 (-0.472644) | 19.037582 / 8.074308 (10.963274) | 14.069141 / 10.191392 (3.877749) | 0.146469 / 0.680424 (-0.533954) | 0.016911 / 0.534201 (-0.517290) | 0.336802 / 0.579283 (-0.242482) | 0.336411 / 0.434364 (-0.097953) | 0.392360 / 0.540337 (-0.147977) | 0.536078 / 1.386936 (-0.850858) |

![](https://cml.dev/watermark.png#12cfc1196e62847e2e8239fbd727a02cbc86ddec "CML watermark")

This PR fixes 3 authentication issues:
- Fix authentication when passing `token`.
- Fix authentication in `Audio.decode_example` and `Image.decode_example`.
- Fix authentication to resolve `data_files` in repositories without script.

This PR also fixes our CI so that we properly test when passing `token` and we do not use the token stored in `HfFolder`.

Fix #6126.

## Details

### Fix authentication when passing `token`

See c0a77dc943de68a17f23f141517028c734c78623

The root issue was caused when the `token` was set in an already instantiated `DownloadConfig` and thus not propagated to `self._storage_options`:

  ```python
  download_config.token = token
  ```

As this usage pattern is very common, the fix consists in overriding `DownloadConfig.__setattr__`.

This fixes authentication issues in the following functions:
- `load_dataset` and `load_dataset_builder`
- `Dataset.push_to_hub` and `Dataset.push_to_hub`
- `inspect.get_dataset_config_info`, `inspect.get_dataset_infos` and `inspect.get_dataset_split_names`

### Fix authentication in `Audio.decode_example` and `Image.decode_example`.

See: 58e62af004b6b8b84dcfd897a4bc71637cfa6c3f

The `token` was not set because the `repo_id` was wrongly tried to be parsed from an HTTP URL (`"http://..."`), instead of an HFFileSystem URL (`"hf://"`)

### Fix authentication to resolve `data_files` in repositories without script

See: e4684fc1032321abf0d494b0c130ea7c82ebda80

This is fixed by passing `download_config` to the function `create_builder_configs_from_metadata_configs`


Private datasets do not load when passing token

bug

Something isn't working

Our CI did not catch this issue because with current implementation, stored token in `HfFolder` (which always exists) is used by default.

I can confirm this and have the same problem (and just went almost crazy because I couldn't figure out the source of this problem because on another computer everything worked well even with `DownloadMode.FORCE_REDOWNLOAD`).

We are planning to do a patch release today, after the merge of the fix:
- #6127

In the meantime, the problem can be circumvented by passing `download_config` instead:
```python
from datasets import DownloadConfig, load_dataset

load_dataset("<DATASET-NAME>", split="train", download_config=DownloadConfig(token="<TOKEN>"))
```

> We are planning to do a patch release today, after the merge of the fix:
>
>     * [Fix authentication issues #6127](https://github.com/huggingface/datasets/pull/6127)
>
>
> In the meantime, the problem can be circumvented by passing `download_config` instead:
>
> ```python
> from datasets import DownloadConfig, load_dataset
>
> load_dataset("<DATASET-NAME>", split="train", download_config=DownloadConfig(token="<TOKEN>"))
> ```

This did not work for me (there was some other error with the split being an unexpected size 0). Downgrading to 2.13 fixed it....

### Describe the bug

Since the release of `datasets` 2.14, private/gated datasets do not load when passing `token`: they raise  `EmptyDatasetError`.

This is a non-planned  backward incompatible breaking change.

Note that private datasets do load if instead `download_config` is passed:
```python
from datasets import DownloadConfig, load_dataset

ds = load_dataset("albertvillanova/tmp-private", split="train", download_config=DownloadConfig(token="<MY-TOKEN>"))
ds
```
gives
```
Dataset({
    features: ['text'],
    num_rows: 4
})
```

### Steps to reproduce the bug

```python
from datasets import load_dataset

ds = load_dataset("albertvillanova/tmp-private", split="train", token="<MY-TOKEN>")
```
gives
```
---------------------------------------------------------------------------
EmptyDatasetError                         Traceback (most recent call last)
[<ipython-input-2-25b48732107a>](https://localhost:8080/#) in <cell line: 3>()
      1 from datasets import load_dataset
      2
----> 3 ds = load_dataset("albertvillanova/tmp-private", split="train", token="<MY-TOKEN>")

5 frames
[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://localhost:8080/#) in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, ignore_verifications, keep_in_memory, save_infos, revision, token, use_auth_token, task, streaming, num_proc, storage_options, **config_kwargs)
   2107
   2108     # Create a dataset builder
-> 2109     builder_instance = load_dataset_builder(
   2110         path=path,
   2111         name=name,

[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://localhost:8080/#) in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, token, use_auth_token, storage_options, **config_kwargs)
   1793         download_config = download_config.copy() if download_config else DownloadConfig()
   1794         download_config.storage_options.update(storage_options)
-> 1795     dataset_module = dataset_module_factory(
   1796         path,
   1797         revision=revision,

[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://localhost:8080/#) in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1484                     raise ConnectionError(f"Couldn't reach the Hugging Face Hub for dataset '{path}': {e1}") from None
   1485                 if isinstance(e1, EmptyDatasetError):
-> 1486                     raise e1 from None
   1487                 if isinstance(e1, FileNotFoundError):
   1488                     raise FileNotFoundError(

[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://localhost:8080/#) in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1474                     download_config=download_config,
   1475                     download_mode=download_mode,
-> 1476                 ).get_module()
   1477         except (
   1478             Exception

[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://localhost:8080/#) in get_module(self)
   1030             sanitize_patterns(self.data_files)
   1031             if self.data_files is not None
-> 1032             else get_data_patterns(base_path, download_config=self.download_config)
   1033         )
   1034         data_files = DataFilesDict.from_patterns(

[/usr/local/lib/python3.10/dist-packages/datasets/data_files.py](https://localhost:8080/#) in get_data_patterns(base_path, download_config)
    457         return _get_data_files_patterns(resolver)
    458     except FileNotFoundError:
--> 459         raise EmptyDatasetError(f"The directory at {base_path} doesn't contain any data files") from None
    460
    461

EmptyDatasetError: The directory at hf://datasets/albertvillanova/tmp-private@79b9e4fe79670a9a050d6ebc385464891915a71d doesn't contain any data files
```

### Expected behavior

The dataset should load.

### Environment info

- `datasets` version: 2.14.3
- Platform: Linux-5.15.109+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- PyArrow version: 9.0.0
- Pandas version: 1.5.3


Reinforcement Learning and Robotics are not task categories in HF datasets metadata

### Describe the bug

In https://huggingface.co/models there are task categories for RL and robotics but none in https://huggingface.co/datasets

Our lab is currently moving our datasets over to hugging face and would like to be able to add those 2 tags

Moreover we see some older datasets that do have that tag, but we can't seem to add it ourselves.

### Steps to reproduce the bug

1. Create a new dataset on Hugging face
2. Try to type reinforcemement-learning or robotics into the tasks categories, it does not allow you to commit

### Expected behavior

Expected to be able to add RL and robotics as task categories as some previous datasets have these tags

### Environment info

N/A


Datasets crashing runs due to KeyError

i once had the same error and I could fix that by pushing a fake or a dummy commit on my hugging face dataset repo

> Hi! We need a reproducer to fix this. Can you provide a link to the dataset (if it's public)?

Hi Mario,

Unfortunately, the dataset in question is currently private until the model is trained and released.

This is not happening with one dataset but numerous hosted private datasets.

I am only loading the dataset and doing nothing else currently. It seems to happen completely sporadically.

Thank you,

Enrico

Hi,

I have the same error in the dataset viewer with my dataset
https://huggingface.co/datasets/elsaEU/ELSA10M_track1

Has anyone solved this issue?

Edit: After a dummy commit the error changed in ConfigNamesError

@rs9000 The problem seems to be the (large) number of commits, as explained in https://huggingface.co/docs/hub/repositories-recommendations. This can be fixed by running:
```python
import huggingface_hub
huggingface_hub.super_squash_history(repo_id="elsaEU/ELSA10M_track1")
```

The issue stems from `push_to_hub` creating one commit per shard -  https://github.com/huggingface/datasets/pull/6269 should fix this issue (will create one commit per 50 uploaded shards by default). The linked PR will be included in the next `datasets` release.

cc @lhoestq @severo for visibility

#6269 has been merged, so I'm closing this issue

### Describe the bug

Hi all,

I have been running into a pretty persistent issue recently when trying to load datasets.
```python
    train_dataset = load_dataset(
        'llama-2-7b-tokenized',
        split = 'train'
    )
```
I receive a KeyError which crashes the runs.
```
Traceback (most recent call last):
    main()

    train_dataset = load_dataset(
                    ^^^^^^^^^^^^^
    builder_instance = load_dataset_builder(
                       ^^^^^^^^^^^^^^^^^^^^^
    dataset_module = dataset_module_factory(
                     ^^^^^^^^^^^^^^^^^^^^^^^
    raise e1 from None

    ).get_module()
      ^^^^^^^^^^^^
    else get_data_patterns(base_path, download_config=self.download_config)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    return _get_data_files_patterns(resolver)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    data_files = pattern_resolver(pattern)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
    fs, _, _ = get_fs_token_paths(pattern, storage_options=storage_options)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    paths = [f for f in sorted(fs.glob(paths)) if not fs.isdir(f)]
                               ^^^^^^^^^^^^^^

    allpaths = self.find(root, maxdepth=depth, withdirs=True, detail=True, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    for _, dirs, files in self.walk(path, maxdepth, detail=True, **kwargs):

    listing = self.ls(path, detail=True, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    "last_modified": parse_datetime(tree_item["lastCommit"]["date"]),
                                    ~~~~~~~~~^^^^^^^^^^^^^^
KeyError: 'lastCommit'
```
Any help would be greatly appreciated.

Thank you,

Enrico

### Steps to reproduce the bug

Load the dataset from the Huggingface hub.
```python
    train_dataset = load_dataset(
        'llama-2-7b-tokenized',
        split = 'train'
    )
```

### Expected behavior

Loads the dataset.

### Environment info

datasets-2.14.3
CUDA 11.8
Python 3.11


Inaccurate Bounding Boxes in "wildreceipt" Dataset

Hi! Thanks for the investigation, but we are not the authors of these datasets, so please report this on the Hub instead so that the actual authors can fix it.

### Describe the bug

I would like to bring to your attention an issue related to the accuracy of bounding boxes within the "wildreceipt" dataset, which is made available through the Hugging Face API. Specifically, I have identified a discrepancy between the bounding boxes generated by the dataset loading commands, namely `load_dataset("Theivaprakasham/wildreceipt")` and `load_dataset("jinhybr/WildReceipt")`, and the actual labels and corresponding bounding boxes present in the dataset.

To illustrate this divergence, I've provided two examples in the form of screenshots. These screenshots highlight the contrasting outcomes between my personal implementation of the dataloader and the implementation offered by Hugging Face:

**Example 1:**

![image](https://github.com/huggingface/datasets/assets/50714796/7a6604d2-899d-4102-a008-1a28c90698f1)
![image](https://github.com/huggingface/datasets/assets/50714796/eba458c7-d3af-4868-a520-8b683aa96f66)
![image](https://github.com/huggingface/datasets/assets/50714796/9f394891-5f5b-46f7-8e52-071b724aedab)

**Example 2:**

![image](https://github.com/huggingface/datasets/assets/50714796/a2b2a8d3-124e-4990-b64a-5133cf4be2fe)
![image](https://github.com/huggingface/datasets/assets/50714796/6ee25642-35aa-40ad-ac1e-899d33be90df)
![image](https://github.com/huggingface/datasets/assets/50714796/5e42ff91-9fc4-4520-8803-0e225656f96c)

It's important to note that my dataloader implementation is based on the same dataset files as utilized in the Hugging Face implementation. For your reference, you can access the dataset files through this link: [wildreceipt dataset files](https://download.openmmlab.com/mmocr/data/wildreceipt.tar).

This inconsistency in bounding box accuracy warrants investigation and rectification for maintaining the integrity of the "wildreceipt" dataset. Your attention and assistance in addressing this matter would be greatly appreciated.

### Steps to reproduce the bug

```python
import matplotlib.pyplot as plt
from datasets import load_dataset

# Define functions to convert bounding box formats
def convert_format1(box):
    x, y, w, h = box
    x2, y2 = x + w, y + h
    return [x, y, x2, y2]

def convert_format2(box):
    x1, y1, x2, y2 = box
    return [x1, y1, x2, y2]

def plot_cropped_image(image, box, title):
    cropped_image = image.crop(box)
    plt.imshow(cropped_image)
    plt.title(title)
    plt.axis('off')
    plt.savefig(title+'.png')
    plt.show()

doc_index = 1
word_index = 3
dataset = load_dataset("Theivaprakasham/wildreceipt")['train']

bbox_hugging_face = dataset[doc_index]['bboxes'][word_index]
text_unit_face = dataset[doc_index]['words'][word_index]

common_box_hugface_1 = convert_format1(bbox_hugging_face)
common_box_hugface_2 = convert_format2(bbox_hugging_face)

plot_cropped_image(image_hugging, common_box_hugface_1,
                   f'Hugging Face Bouding boxes (x,y,w,h format) \n its associated text unit: {text_unit_face}')
plot_cropped_image(image_hugging, common_box_hugface_2,
                   f'Hugging Face Bouding boxes (x1,y1,x2, y2 format) \n its associated text unit: {text_unit_face}')
```

### Expected behavior

The bounding boxes generated by the "wildreceipt" dataset in HuggingFace implementation loading commands should accurately match the actual labels and bounding boxes of the dataset.

### Environment info

- Python version: 3.8
- Hugging Face datasets version: 2.14.2
- Dataset file taken from this link: https://download.openmmlab.com/mmocr/data/wildreceipt.tar


Upload README via `push_to_hub`

enhancement

New feature or request

You can use `huggingface_hub`'s [Card API](https://huggingface.co/docs/huggingface_hub/package_reference/cards) to programmatically push a dataset card to the Hub.

### Feature request

`push_to_hub` now allows users to upload datasets programmatically. However, based on the latest doc, we still need to open the dataset page to add readme file manually.

However, I do discover snippets to intialize a README for every `push_to_hub`:
```
dataset_card = (
    DatasetCard(
        "---\n"
        + str(dataset_card_data)
        + "\n---\n"
        + f'# Dataset Card for "{repo_id.split("/")[-1]}"\n\n[More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)'
    )
    if dataset_card is None
    else dataset_card
)

HfApi(endpoint=config.HF_ENDPOINT).upload_file(
    path_or_fileobj=str(dataset_card).encode(),
    path_in_repo="README.md",
    repo_id=repo_id,
    token=token,
    repo_type="dataset",
    revision=branch,
)
```

So, if we can enable `push_to_hub` to upload a readme file by ourselves instead of using the auto generated ones, it can save ton of time, and will definitely alleviate the current "lack-of-dataset-card" situation.

### Motivation

as elabrated above.

### Your contribution

I might be able to make a pr.


Small typo in the code example of create imagefolder dataset

Hi,

I found a small typo in the code example of create imagefolder dataset. It confused me a little when I first saw it.

Best Regards.

Xin

Fix type of code example of load imagefolder dataset


Lookahead streaming support?

New feature or request

In which format is your dataset? We could expose the `pre_buffer` flag for Parquet to use PyArrow's background thread pool to speed up loading.

### Feature request

From what I understand, streaming dataset currently pulls the data, and process the data as it is requested.
This can introduce significant latency delays when data is loaded into the training process, needing to wait for each segment.

While the delays might be dataset specific (or even mapping instruction/tokenizer specific)

Is it possible to introduce a `streaming_lookahead` parameter, which is used for predictable workloads (even shuffled dataset with fixed seed). As we can predict in advance what the next few datasamples will be. And fetch them while the current set is being trained.

With enough CPU & bandwidth to keep up with the training process, and a sufficiently large lookahead, this will reduce the various latency involved while waiting for the dataset to be ready between batches.

### Motivation

Faster streaming performance, while training over extra large TB sized datasets

### Your contribution

I currently use HF dataset, with pytorch lightning trainer for RWKV project, and would be able to help test this feature if supported.


[Docs] Add description of `select_columns` to guide

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007755 / 0.011353 (-0.003598) | 0.004618 / 0.011008 (-0.006391) | 0.098132 / 0.038508 (0.059624) | 0.086759 / 0.023109 (0.063650) | 0.374668 / 0.275898 (0.098770) | 0.417131 / 0.323480 (0.093651) | 0.004604 / 0.007986 (-0.003382) | 0.005461 / 0.004328 (0.001132) | 0.077249 / 0.004250 (0.072999) | 0.063247 / 0.037052 (0.026195) | 0.391801 / 0.258489 (0.133312) | 0.432139 / 0.293841 (0.138298) | 0.036755 / 0.128546 (-0.091791) | 0.010011 / 0.075646 (-0.065636) | 0.346175 / 0.419271 (-0.073097) | 0.061503 / 0.043533 (0.017971) | 0.374063 / 0.255139 (0.118924) | 0.435873 / 0.283200 (0.152673) | 0.029476 / 0.141683 (-0.112207) | 1.786945 / 1.452155 (0.334790) | 1.857190 / 1.492716 (0.364474) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.253939 / 0.018006 (0.235933) | 0.506847 / 0.000490 (0.506358) | 0.007278 / 0.000200 (0.007079) | 0.000451 / 0.000054 (0.000397) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032938 / 0.037411 (-0.004474) | 0.097493 / 0.014526 (0.082967) | 0.112090 / 0.176557 (-0.064467) | 0.177986 / 0.737135 (-0.559149) | 0.112060 / 0.296338 (-0.184278) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.481858 / 0.215209 (0.266649) | 4.814894 / 2.077655 (2.737239) | 2.496428 / 1.504120 (0.992308) | 2.309965 / 1.541195 (0.768770) | 2.393819 / 1.468490 (0.925329) | 0.564670 / 4.584777 (-4.020107) | 4.151222 / 3.745712 (0.405510) | 3.676115 / 5.269862 (-1.593747) | 2.346165 / 4.565676 (-2.219512) | 0.066344 / 0.424275 (-0.357931) | 0.009006 / 0.007607 (0.001399) | 0.567699 / 0.226044 (0.341654) | 5.686799 / 2.268929 (3.417871) | 3.031044 / 55.444624 (-52.413580) | 2.606259 / 6.876477 (-4.270217) | 2.864876 / 2.142072 (0.722804) | 0.681730 / 4.805227 (-4.123498) | 0.155405 / 6.500664 (-6.345259) | 0.071492 / 0.075469 (-0.003977) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.514446 / 1.841788 (-0.327341) | 22.624912 / 8.074308 (14.550604) | 16.754145 / 10.191392 (6.562753) | 0.193113 / 0.680424 (-0.487311) | 0.021808 / 0.534201 (-0.512393) | 0.468241 / 0.579283 (-0.111042) | 0.499647 / 0.434364 (0.065283) | 0.539571 / 0.540337 (-0.000766) | 0.771268 / 1.386936 (-0.615668) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007562 / 0.011353 (-0.003791) | 0.004548 / 0.011008 (-0.006460) | 0.075998 / 0.038508 (0.037490) | 0.081648 / 0.023109 (0.058539) | 0.462876 / 0.275898 (0.186978) | 0.499366 / 0.323480 (0.175886) | 0.005839 / 0.007986 (-0.002147) | 0.003753 / 0.004328 (-0.000576) | 0.075918 / 0.004250 (0.071668) | 0.063233 / 0.037052 (0.026181) | 0.459024 / 0.258489 (0.200535) | 0.506388 / 0.293841 (0.212547) | 0.036179 / 0.128546 (-0.092367) | 0.009961 / 0.075646 (-0.065685) | 0.082061 / 0.419271 (-0.337211) | 0.056469 / 0.043533 (0.012936) | 0.459567 / 0.255139 (0.204428) | 0.482578 / 0.283200 (0.199378) | 0.026363 / 0.141683 (-0.115320) | 1.742247 / 1.452155 (0.290092) | 1.807166 / 1.492716 (0.314450) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.330526 / 0.018006 (0.312520) | 0.511674 / 0.000490 (0.511184) | 0.040969 / 0.000200 (0.040769) | 0.000176 / 0.000054 (0.000121) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035492 / 0.037411 (-0.001920) | 0.104338 / 0.014526 (0.089813) | 0.116973 / 0.176557 (-0.059583) | 0.180218 / 0.737135 (-0.556917) | 0.118801 / 0.296338 (-0.177538) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.492196 / 0.215209 (0.276987) | 4.910271 / 2.077655 (2.832616) | 2.542562 / 1.504120 (1.038442) | 2.333516 / 1.541195 (0.792321) | 2.439682 / 1.468490 (0.971192) | 0.571966 / 4.584777 (-4.012811) | 4.089801 / 3.745712 (0.344089) | 3.732129 / 5.269862 (-1.537733) | 2.375887 / 4.565676 (-2.189789) | 0.067376 / 0.424275 (-0.356900) | 0.008350 / 0.007607 (0.000743) | 0.583942 / 0.226044 (0.357897) | 5.840002 / 2.268929 (3.571074) | 3.062520 / 55.444624 (-52.382104) | 2.722512 / 6.876477 (-4.153965) | 2.938307 / 2.142072 (0.796234) | 0.689459 / 4.805227 (-4.115769) | 0.155632 / 6.500664 (-6.345032) | 0.072387 / 0.075469 (-0.003082) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.595587 / 1.841788 (-0.246201) | 23.035478 / 8.074308 (14.961170) | 16.457675 / 10.191392 (6.266283) | 0.170819 / 0.680424 (-0.509605) | 0.022042 / 0.534201 (-0.512159) | 0.466824 / 0.579283 (-0.112459) | 0.486350 / 0.434364 (0.051986) | 0.574330 / 0.540337 (0.033993) | 0.764913 / 1.386936 (-0.622023) |

![](https://cml.dev/watermark.png#664a1cb72ea1e6ef7c47e671e2686ca4a35e8d63 "CML watermark")

Closes #6116


IterableDataset.from_generator() fails with pickle error when provided a generator or iterator

Hi! `IterableDataset.from_generator` expects a generator function, not the object (to be consistent with `Dataset.from_generator`).

You can fix the above snippet as follows:
```python
train_dataset = IterableDataset.from_generator(line_generator, fn_kwargs={"files": model_training_files})
```

to anyone reaching this issue, the argument is `gen_kwargs`:
```py
train_dataset = IterableDataset.from_generator(line_generator, gen_kwargs={"files": model_training_files})
```

### Describe the bug

**Description**
Providing a generator in an instantiation of IterableDataset.from_generator() fails with `TypeError: cannot pickle 'generator' object` when the generator argument is supplied with a generator.

**Code example**
```
def line_generator(files: List[Path]):

    if isinstance(files, str):
        files = [Path(files)]

    for file in files:
        if isinstance(file, str):
            file = Path(file)
        yield from open(file,'r').readlines()

...
model_training_files = ['file1.txt', 'file2.txt', 'file3.txt']
train_dataset = IterableDataset.from_generator(generator=line_generator(model_training_files))
```

**Traceback**
Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/contextlib.py", line 135, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/d3p692/code/clem_bert/venv/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 691, in _no_cache_fields
    yield
  File "/Users/d3p692/code/clem_bert/venv/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 701, in dumps
    dump(obj, file)
  File "/Users/d3p692/code/clem_bert/venv/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 676, in dump
    Pickler(file, recurse=True).dump(obj)
  File "/Users/d3p692/code/clem_bert/venv/lib/python3.9/site-packages/dill/_dill.py", line 394, in dump
    StockPickler.dump(self, obj)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/pickle.py", line 487, in dump
    self.save(obj)
  File "/Users/d3p692/code/clem_bert/venv/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 666, in save
    dill.Pickler.save(self, obj, save_persistent_id=save_persistent_id)
  File "/Users/d3p692/code/clem_bert/venv/lib/python3.9/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/d3p692/code/clem_bert/venv/lib/python3.9/site-packages/dill/_dill.py", line 1186, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/Users/d3p692/code/clem_bert/venv/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 666, in save
    dill.Pickler.save(self, obj, save_persistent_id=save_persistent_id)
  File "/Users/d3p692/code/clem_bert/venv/lib/python3.9/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/pickle.py", line 578, in save
    rv = reduce(self.proto)
TypeError: cannot pickle 'generator' object

### Steps to reproduce the bug

1. Create a set of text files to iterate over.
2. Create a generator that returns the lines in each file until all files are exhausted.
3. Instantiate the dataset over the generator by instantiating an IterableDataset.from_generator().
4. Wait for the explosion.

### Expected behavior

I would expect that since the function claims to accept a generator that there would be no crash. Instead, I would expect the dataset to return all the lines in the files as queued up in the `line_generator()` function.

### Environment info

datasets.__version__ == '2.13.1'
Python 3.9.6
Platform: Darwin WE35261 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:22 PDT 2023; root:xnu-8796.121.3~7/RELEASE_X86_64 x86_64