Set dev version

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6428). All of your documentation changes will be reflected on that endpoint.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004839 / 0.011353 (-0.006514) | 0.002928 / 0.011008 (-0.008080) | 0.061730 / 0.038508 (0.023221) | 0.030523 / 0.023109 (0.007414) | 0.252679 / 0.275898 (-0.023219) | 0.281597 / 0.323480 (-0.041883) | 0.003025 / 0.007986 (-0.004961) | 0.002374 / 0.004328 (-0.001955) | 0.048134 / 0.004250 (0.043884) | 0.045843 / 0.037052 (0.008791) | 0.256274 / 0.258489 (-0.002215) | 0.288704 / 0.293841 (-0.005137) | 0.023486 / 0.128546 (-0.105060) | 0.007186 / 0.075646 (-0.068461) | 0.202519 / 0.419271 (-0.216753) | 0.058192 / 0.043533 (0.014659) | 0.256448 / 0.255139 (0.001309) | 0.279417 / 0.283200 (-0.003783) | 0.019942 / 0.141683 (-0.121740) | 1.100954 / 1.452155 (-0.351201) | 1.168183 / 1.492716 (-0.324533) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091314 / 0.018006 (0.073308) | 0.298614 / 0.000490 (0.298124) | 0.000232 / 0.000200 (0.000032) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018071 / 0.037411 (-0.019340) | 0.062265 / 0.014526 (0.047740) | 0.073228 / 0.176557 (-0.103328) | 0.119163 / 0.737135 (-0.617972) | 0.074717 / 0.296338 (-0.221622) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.273906 / 0.215209 (0.058697) | 2.683995 / 2.077655 (0.606340) | 1.418773 / 1.504120 (-0.085347) | 1.310473 / 1.541195 (-0.230722) | 1.303152 / 1.468490 (-0.165339) | 0.390846 / 4.584777 (-4.193931) | 2.346407 / 3.745712 (-1.399305) | 2.582945 / 5.269862 (-2.686916) | 1.569549 / 4.565676 (-2.996128) | 0.044893 / 0.424275 (-0.379383) | 0.004754 / 0.007607 (-0.002853) | 0.323491 / 0.226044 (0.097447) | 3.229736 / 2.268929 (0.960808) | 1.783551 / 55.444624 (-53.661074) | 1.499685 / 6.876477 (-5.376792) | 1.515826 / 2.142072 (-0.626246) | 0.475768 / 4.805227 (-4.329460) | 0.099579 / 6.500664 (-6.401085) | 0.042709 / 0.075469 (-0.032760) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.926120 / 1.841788 (-0.915667) | 11.597189 / 8.074308 (3.522881) | 10.327055 / 10.191392 (0.135663) | 0.127479 / 0.680424 (-0.552945) | 0.014844 / 0.534201 (-0.519357) | 0.261181 / 0.579283 (-0.318102) | 0.258407 / 0.434364 (-0.175957) | 0.303192 / 0.540337 (-0.237146) | 0.416665 / 1.386936 (-0.970271) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004759 / 0.011353 (-0.006594) | 0.002780 / 0.011008 (-0.008228) | 0.047991 / 0.038508 (0.009483) | 0.052263 / 0.023109 (0.029153) | 0.261228 / 0.275898 (-0.014670) | 0.287779 / 0.323480 (-0.035701) | 0.003961 / 0.007986 (-0.004024) | 0.002357 / 0.004328 (-0.001971) | 0.047755 / 0.004250 (0.043505) | 0.038066 / 0.037052 (0.001014) | 0.269502 / 0.258489 (0.011013) | 0.298348 / 0.293841 (0.004507) | 0.024398 / 0.128546 (-0.104149) | 0.007189 / 0.075646 (-0.068457) | 0.053356 / 0.419271 (-0.365915) | 0.032459 / 0.043533 (-0.011074) | 0.266389 / 0.255139 (0.011250) | 0.305367 / 0.283200 (0.022168) | 0.017629 / 0.141683 (-0.124054) | 1.145789 / 1.452155 (-0.306366) | 1.204778 / 1.492716 (-0.287938) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091347 / 0.018006 (0.073341) | 0.298671 / 0.000490 (0.298181) | 0.000229 / 0.000200 (0.000029) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021374 / 0.037411 (-0.016037) | 0.068869 / 0.014526 (0.054344) | 0.080443 / 0.176557 (-0.096113) | 0.118759 / 0.737135 (-0.618376) | 0.081646 / 0.296338 (-0.214692) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295274 / 0.215209 (0.080065) | 2.889349 / 2.077655 (0.811695) | 1.561020 / 1.504120 (0.056900) | 1.425025 / 1.541195 (-0.116170) | 1.495446 / 1.468490 (0.026956) | 0.403825 / 4.584777 (-4.180952) | 2.404905 / 3.745712 (-1.340807) | 2.590104 / 5.269862 (-2.679758) | 1.570559 / 4.565676 (-2.995118) | 0.046342 / 0.424275 (-0.377933) | 0.004799 / 0.007607 (-0.002809) | 0.349981 / 0.226044 (0.123937) | 3.437341 / 2.268929 (1.168412) | 1.948155 / 55.444624 (-53.496469) | 1.637765 / 6.876477 (-5.238711) | 1.671521 / 2.142072 (-0.470551) | 0.479500 / 4.805227 (-4.325727) | 0.098305 / 6.500664 (-6.402359) | 0.040864 / 0.075469 (-0.034605) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.979986 / 1.841788 (-0.861801) | 12.169722 / 8.074308 (4.095413) | 11.297345 / 10.191392 (1.105953) | 0.129123 / 0.680424 (-0.551301) | 0.015389 / 0.534201 (-0.518812) | 0.270964 / 0.579283 (-0.308319) | 0.269590 / 0.434364 (-0.164774) | 0.310662 / 0.540337 (-0.229675) | 0.406272 / 1.386936 (-0.980664) |

![](https://cml.dev/watermark.png#31873f1e9acbe013e6d396d1ed5492db8cd59dd3 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004620 / 0.011353 (-0.006733) | 0.002971 / 0.011008 (-0.008038) | 0.062864 / 0.038508 (0.024355) | 0.028743 / 0.023109 (0.005634) | 0.246729 / 0.275898 (-0.029169) | 0.271165 / 0.323480 (-0.052315) | 0.003930 / 0.007986 (-0.004056) | 0.002422 / 0.004328 (-0.001906) | 0.047430 / 0.004250 (0.043180) | 0.044895 / 0.037052 (0.007843) | 0.249128 / 0.258489 (-0.009361) | 0.283384 / 0.293841 (-0.010457) | 0.023288 / 0.128546 (-0.105259) | 0.007241 / 0.075646 (-0.068405) | 0.207551 / 0.419271 (-0.211720) | 0.055008 / 0.043533 (0.011475) | 0.252781 / 0.255139 (-0.002358) | 0.296924 / 0.283200 (0.013724) | 0.017860 / 0.141683 (-0.123822) | 1.094597 / 1.452155 (-0.357558) | 1.162314 / 1.492716 (-0.330402) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091423 / 0.018006 (0.073417) | 0.302833 / 0.000490 (0.302343) | 0.000242 / 0.000200 (0.000042) | 0.000042 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018143 / 0.037411 (-0.019268) | 0.066371 / 0.014526 (0.051845) | 0.072774 / 0.176557 (-0.103783) | 0.119062 / 0.737135 (-0.618073) | 0.102836 / 0.296338 (-0.193502) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.280117 / 0.215209 (0.064908) | 2.757955 / 2.077655 (0.680301) | 1.494994 / 1.504120 (-0.009126) | 1.375325 / 1.541195 (-0.165870) | 1.384179 / 1.468490 (-0.084311) | 0.399824 / 4.584777 (-4.184953) | 2.368575 / 3.745712 (-1.377137) | 2.574035 / 5.269862 (-2.695827) | 1.548738 / 4.565676 (-3.016939) | 0.045841 / 0.424275 (-0.378434) | 0.004799 / 0.007607 (-0.002808) | 0.331522 / 0.226044 (0.105478) | 3.324471 / 2.268929 (1.055543) | 1.838637 / 55.444624 (-53.605987) | 1.562854 / 6.876477 (-5.313623) | 1.581736 / 2.142072 (-0.560336) | 0.468832 / 4.805227 (-4.336396) | 0.099309 / 6.500664 (-6.401355) | 0.042078 / 0.075469 (-0.033391) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.928468 / 1.841788 (-0.913320) | 11.331143 / 8.074308 (3.256835) | 10.296213 / 10.191392 (0.104821) | 0.138912 / 0.680424 (-0.541511) | 0.014044 / 0.534201 (-0.520157) | 0.267293 / 0.579283 (-0.311991) | 0.267267 / 0.434364 (-0.167097) | 0.306560 / 0.540337 (-0.233778) | 0.423926 / 1.386936 (-0.963010) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004842 / 0.011353 (-0.006511) | 0.002917 / 0.011008 (-0.008091) | 0.048263 / 0.038508 (0.009755) | 0.051453 / 0.023109 (0.028344) | 0.278330 / 0.275898 (0.002432) | 0.298569 / 0.323480 (-0.024911) | 0.003936 / 0.007986 (-0.004049) | 0.002479 / 0.004328 (-0.001850) | 0.048281 / 0.004250 (0.044031) | 0.038925 / 0.037052 (0.001872) | 0.285258 / 0.258489 (0.026769) | 0.313701 / 0.293841 (0.019860) | 0.024916 / 0.128546 (-0.103630) | 0.007142 / 0.075646 (-0.068504) | 0.053634 / 0.419271 (-0.365638) | 0.032842 / 0.043533 (-0.010690) | 0.279373 / 0.255139 (0.024234) | 0.295844 / 0.283200 (0.012644) | 0.018142 / 0.141683 (-0.123541) | 1.136960 / 1.452155 (-0.315195) | 1.184438 / 1.492716 (-0.308278) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090271 / 0.018006 (0.072264) | 0.299940 / 0.000490 (0.299450) | 0.000234 / 0.000200 (0.000034) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021175 / 0.037411 (-0.016237) | 0.070924 / 0.014526 (0.056398) | 0.080584 / 0.176557 (-0.095972) | 0.119278 / 0.737135 (-0.617857) | 0.082361 / 0.296338 (-0.213977) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.298312 / 0.215209 (0.083103) | 2.895361 / 2.077655 (0.817706) | 1.616120 / 1.504120 (0.112001) | 1.484444 / 1.541195 (-0.056750) | 1.541893 / 1.468490 (0.073403) | 0.409968 / 4.584777 (-4.174809) | 2.423639 / 3.745712 (-1.322073) | 2.585122 / 5.269862 (-2.684740) | 1.540343 / 4.565676 (-3.025333) | 0.046604 / 0.424275 (-0.377671) | 0.004742 / 0.007607 (-0.002865) | 0.341659 / 0.226044 (0.115614) | 3.409259 / 2.268929 (1.140330) | 2.007068 / 55.444624 (-53.437556) | 1.681348 / 6.876477 (-5.195129) | 1.719253 / 2.142072 (-0.422819) | 0.482301 / 4.805227 (-4.322926) | 0.099619 / 6.500664 (-6.401045) | 0.041247 / 0.075469 (-0.034222) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.971783 / 1.841788 (-0.870004) | 12.208000 / 8.074308 (4.133692) | 10.948230 / 10.191392 (0.756838) | 0.131824 / 0.680424 (-0.548599) | 0.015696 / 0.534201 (-0.518505) | 0.272265 / 0.579283 (-0.307018) | 0.276093 / 0.434364 (-0.158270) | 0.305897 / 0.540337 (-0.234441) | 0.411632 / 1.386936 (-0.975304) |

![](https://cml.dev/watermark.png#2bf75fe522c6fedd16d00b4a928f613dee11f73c "CML watermark")


Release: 2.15.0

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004331 / 0.011353 (-0.007022) | 0.002573 / 0.011008 (-0.008435) | 0.061002 / 0.038508 (0.022494) | 0.029259 / 0.023109 (0.006149) | 0.242983 / 0.275898 (-0.032915) | 0.267629 / 0.323480 (-0.055851) | 0.003906 / 0.007986 (-0.004080) | 0.002383 / 0.004328 (-0.001946) | 0.047574 / 0.004250 (0.043323) | 0.042153 / 0.037052 (0.005101) | 0.245821 / 0.258489 (-0.012668) | 0.276479 / 0.293841 (-0.017362) | 0.022498 / 0.128546 (-0.106049) | 0.006775 / 0.075646 (-0.068871) | 0.201795 / 0.419271 (-0.217477) | 0.052443 / 0.043533 (0.008910) | 0.248320 / 0.255139 (-0.006819) | 0.261964 / 0.283200 (-0.021235) | 0.016764 / 0.141683 (-0.124919) | 1.118702 / 1.452155 (-0.333453) | 1.203079 / 1.492716 (-0.289638) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.088808 / 0.018006 (0.070801) | 0.296526 / 0.000490 (0.296037) | 0.000203 / 0.000200 (0.000003) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018816 / 0.037411 (-0.018595) | 0.062295 / 0.014526 (0.047769) | 0.075228 / 0.176557 (-0.101329) | 0.119916 / 0.737135 (-0.617219) | 0.077206 / 0.296338 (-0.219132) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.276723 / 0.215209 (0.061514) | 2.711431 / 2.077655 (0.633776) | 1.425590 / 1.504120 (-0.078530) | 1.301383 / 1.541195 (-0.239812) | 1.316314 / 1.468490 (-0.152176) | 0.402709 / 4.584777 (-4.182068) | 2.347229 / 3.745712 (-1.398483) | 2.596937 / 5.269862 (-2.672925) | 1.560658 / 4.565676 (-3.005018) | 0.046162 / 0.424275 (-0.378113) | 0.004760 / 0.007607 (-0.002848) | 0.330522 / 0.226044 (0.104478) | 3.244072 / 2.268929 (0.975143) | 1.747603 / 55.444624 (-53.697021) | 1.475534 / 6.876477 (-5.400943) | 1.485135 / 2.142072 (-0.656938) | 0.476794 / 4.805227 (-4.328433) | 0.098496 / 6.500664 (-6.402168) | 0.040740 / 0.075469 (-0.034729) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.939020 / 1.841788 (-0.902768) | 11.235187 / 8.074308 (3.160878) | 10.194975 / 10.191392 (0.003583) | 0.126241 / 0.680424 (-0.554182) | 0.013990 / 0.534201 (-0.520211) | 0.269149 / 0.579283 (-0.310134) | 0.256950 / 0.434364 (-0.177414) | 0.301282 / 0.540337 (-0.239056) | 0.421490 / 1.386936 (-0.965446) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004956 / 0.011353 (-0.006397) | 0.002478 / 0.011008 (-0.008530) | 0.047773 / 0.038508 (0.009265) | 0.050076 / 0.023109 (0.026967) | 0.261915 / 0.275898 (-0.013983) | 0.282553 / 0.323480 (-0.040927) | 0.003881 / 0.007986 (-0.004105) | 0.002329 / 0.004328 (-0.001999) | 0.048091 / 0.004250 (0.043841) | 0.038188 / 0.037052 (0.001135) | 0.265502 / 0.258489 (0.007013) | 0.292568 / 0.293841 (-0.001273) | 0.024172 / 0.128546 (-0.104374) | 0.006865 / 0.075646 (-0.068781) | 0.053199 / 0.419271 (-0.366072) | 0.032201 / 0.043533 (-0.011332) | 0.265774 / 0.255139 (0.010635) | 0.277954 / 0.283200 (-0.005245) | 0.017798 / 0.141683 (-0.123885) | 1.121503 / 1.452155 (-0.330652) | 1.176319 / 1.492716 (-0.316398) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.087027 / 0.018006 (0.069020) | 0.296182 / 0.000490 (0.295693) | 0.000216 / 0.000200 (0.000017) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.020990 / 0.037411 (-0.016421) | 0.069693 / 0.014526 (0.055168) | 0.081098 / 0.176557 (-0.095459) | 0.117760 / 0.737135 (-0.619375) | 0.081493 / 0.296338 (-0.214845) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295078 / 0.215209 (0.079869) | 2.876602 / 2.077655 (0.798947) | 1.558011 / 1.504120 (0.053891) | 1.426715 / 1.541195 (-0.114480) | 1.443785 / 1.468490 (-0.024705) | 0.400826 / 4.584777 (-4.183951) | 2.378903 / 3.745712 (-1.366810) | 2.473128 / 5.269862 (-2.796734) | 1.500785 / 4.565676 (-3.064891) | 0.045438 / 0.424275 (-0.378837) | 0.004953 / 0.007607 (-0.002654) | 0.348182 / 0.226044 (0.122137) | 3.427751 / 2.268929 (1.158822) | 1.925173 / 55.444624 (-53.519451) | 1.633354 / 6.876477 (-5.243123) | 1.651573 / 2.142072 (-0.490499) | 0.473260 / 4.805227 (-4.331968) | 0.097613 / 6.500664 (-6.403051) | 0.040196 / 0.075469 (-0.035273) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.951780 / 1.841788 (-0.890008) | 11.709342 / 8.074308 (3.635034) | 10.571831 / 10.191392 (0.380439) | 0.134344 / 0.680424 (-0.546079) | 0.022116 / 0.534201 (-0.512084) | 0.269651 / 0.579283 (-0.309632) | 0.272310 / 0.434364 (-0.162054) | 0.306434 / 0.540337 (-0.233903) | 0.408320 / 1.386936 (-0.978616) |

![](https://cml.dev/watermark.png#7ea64b77079cf76675421917472c05d06ace63fc "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004402 / 0.011353 (-0.006951) | 0.002732 / 0.011008 (-0.008277) | 0.062799 / 0.038508 (0.024291) | 0.029155 / 0.023109 (0.006046) | 0.241925 / 0.275898 (-0.033973) | 0.275694 / 0.323480 (-0.047786) | 0.003989 / 0.007986 (-0.003997) | 0.002528 / 0.004328 (-0.001801) | 0.048410 / 0.004250 (0.044160) | 0.043729 / 0.037052 (0.006677) | 0.248843 / 0.258489 (-0.009646) | 0.282980 / 0.293841 (-0.010860) | 0.023828 / 0.128546 (-0.104718) | 0.006972 / 0.075646 (-0.068675) | 0.213222 / 0.419271 (-0.206049) | 0.054883 / 0.043533 (0.011350) | 0.251353 / 0.255139 (-0.003786) | 0.269818 / 0.283200 (-0.013381) | 0.016906 / 0.141683 (-0.124777) | 1.114109 / 1.452155 (-0.338045) | 1.162942 / 1.492716 (-0.329774) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093724 / 0.018006 (0.075718) | 0.301989 / 0.000490 (0.301499) | 0.000213 / 0.000200 (0.000014) | 0.000049 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018245 / 0.037411 (-0.019166) | 0.062237 / 0.014526 (0.047712) | 0.075644 / 0.176557 (-0.100913) | 0.119655 / 0.737135 (-0.617480) | 0.074525 / 0.296338 (-0.221814) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.274534 / 0.215209 (0.059324) | 2.683678 / 2.077655 (0.606024) | 1.453306 / 1.504120 (-0.050814) | 1.347630 / 1.541195 (-0.193564) | 1.352875 / 1.468490 (-0.115615) | 0.398425 / 4.584777 (-4.186352) | 2.375738 / 3.745712 (-1.369974) | 2.591573 / 5.269862 (-2.678289) | 1.555527 / 4.565676 (-3.010150) | 0.045656 / 0.424275 (-0.378619) | 0.004898 / 0.007607 (-0.002709) | 0.330591 / 0.226044 (0.104547) | 3.247638 / 2.268929 (0.978710) | 1.816676 / 55.444624 (-53.627948) | 1.531754 / 6.876477 (-5.344723) | 1.543196 / 2.142072 (-0.598877) | 0.472489 / 4.805227 (-4.332739) | 0.099311 / 6.500664 (-6.401353) | 0.042139 / 0.075469 (-0.033330) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.945472 / 1.841788 (-0.896316) | 11.476550 / 8.074308 (3.402242) | 10.281157 / 10.191392 (0.089765) | 0.141062 / 0.680424 (-0.539362) | 0.013634 / 0.534201 (-0.520567) | 0.268778 / 0.579283 (-0.310505) | 0.263542 / 0.434364 (-0.170822) | 0.307918 / 0.540337 (-0.232420) | 0.421231 / 1.386936 (-0.965705) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005090 / 0.011353 (-0.006263) | 0.003135 / 0.011008 (-0.007873) | 0.048058 / 0.038508 (0.009550) | 0.052898 / 0.023109 (0.029789) | 0.273233 / 0.275898 (-0.002665) | 0.299516 / 0.323480 (-0.023964) | 0.004126 / 0.007986 (-0.003860) | 0.002331 / 0.004328 (-0.001997) | 0.047627 / 0.004250 (0.043376) | 0.039076 / 0.037052 (0.002023) | 0.276625 / 0.258489 (0.018136) | 0.308180 / 0.293841 (0.014340) | 0.024929 / 0.128546 (-0.103618) | 0.007396 / 0.075646 (-0.068251) | 0.053408 / 0.419271 (-0.365863) | 0.032896 / 0.043533 (-0.010637) | 0.275412 / 0.255139 (0.020273) | 0.292014 / 0.283200 (0.008814) | 0.018336 / 0.141683 (-0.123347) | 1.123565 / 1.452155 (-0.328589) | 1.175382 / 1.492716 (-0.317334) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093799 / 0.018006 (0.075793) | 0.304219 / 0.000490 (0.303729) | 0.000231 / 0.000200 (0.000031) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021034 / 0.037411 (-0.016377) | 0.069961 / 0.014526 (0.055435) | 0.080311 / 0.176557 (-0.096246) | 0.118603 / 0.737135 (-0.618532) | 0.084003 / 0.296338 (-0.212335) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.305610 / 0.215209 (0.090401) | 2.962027 / 2.077655 (0.884372) | 1.598604 / 1.504120 (0.094484) | 1.476227 / 1.541195 (-0.064967) | 1.528960 / 1.468490 (0.060470) | 0.404545 / 4.584777 (-4.180232) | 2.423147 / 3.745712 (-1.322565) | 2.516632 / 5.269862 (-2.753229) | 1.529000 / 4.565676 (-3.036677) | 0.045780 / 0.424275 (-0.378495) | 0.004784 / 0.007607 (-0.002823) | 0.358836 / 0.226044 (0.132792) | 3.508782 / 2.268929 (1.239853) | 1.954513 / 55.444624 (-53.490111) | 1.672824 / 6.876477 (-5.203653) | 1.683482 / 2.142072 (-0.458590) | 0.479014 / 4.805227 (-4.326213) | 0.098325 / 6.500664 (-6.402340) | 0.040934 / 0.075469 (-0.034536) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.974770 / 1.841788 (-0.867017) | 11.956137 / 8.074308 (3.881829) | 10.956458 / 10.191392 (0.765066) | 0.141800 / 0.680424 (-0.538624) | 0.015439 / 0.534201 (-0.518762) | 0.271783 / 0.579283 (-0.307500) | 0.278058 / 0.434364 (-0.156306) | 0.305823 / 0.540337 (-0.234514) | 0.415677 / 1.386936 (-0.971259) |

![](https://cml.dev/watermark.png#0caf91285116ec910f409e82cc6e1f4cff7496e3 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004483 / 0.011353 (-0.006870) | 0.002560 / 0.011008 (-0.008448) | 0.061428 / 0.038508 (0.022920) | 0.029460 / 0.023109 (0.006351) | 0.238971 / 0.275898 (-0.036927) | 0.271768 / 0.323480 (-0.051712) | 0.003970 / 0.007986 (-0.004016) | 0.002408 / 0.004328 (-0.001921) | 0.047755 / 0.004250 (0.043505) | 0.043358 / 0.037052 (0.006306) | 0.245543 / 0.258489 (-0.012946) | 0.278230 / 0.293841 (-0.015611) | 0.023819 / 0.128546 (-0.104727) | 0.006856 / 0.075646 (-0.068790) | 0.204603 / 0.419271 (-0.214668) | 0.054521 / 0.043533 (0.010989) | 0.246277 / 0.255139 (-0.008862) | 0.271230 / 0.283200 (-0.011969) | 0.017283 / 0.141683 (-0.124400) | 1.088955 / 1.452155 (-0.363200) | 1.245141 / 1.492716 (-0.247575) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091534 / 0.018006 (0.073528) | 0.299517 / 0.000490 (0.299027) | 0.000215 / 0.000200 (0.000015) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018105 / 0.037411 (-0.019306) | 0.061860 / 0.014526 (0.047334) | 0.074494 / 0.176557 (-0.102063) | 0.120107 / 0.737135 (-0.617029) | 0.073406 / 0.296338 (-0.222932) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.278140 / 0.215209 (0.062931) | 2.746208 / 2.077655 (0.668553) | 1.476264 / 1.504120 (-0.027856) | 1.356498 / 1.541195 (-0.184697) | 1.362998 / 1.468490 (-0.105492) | 0.401884 / 4.584777 (-4.182893) | 2.409836 / 3.745712 (-1.335877) | 2.579087 / 5.269862 (-2.690775) | 1.545021 / 4.565676 (-3.020656) | 0.046001 / 0.424275 (-0.378274) | 0.004812 / 0.007607 (-0.002795) | 0.339767 / 0.226044 (0.113722) | 3.341599 / 2.268929 (1.072670) | 1.821498 / 55.444624 (-53.623127) | 1.559311 / 6.876477 (-5.317166) | 1.570368 / 2.142072 (-0.571704) | 0.472688 / 4.805227 (-4.332539) | 0.099549 / 6.500664 (-6.401115) | 0.041644 / 0.075469 (-0.033825) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.951988 / 1.841788 (-0.889799) | 11.371459 / 8.074308 (3.297150) | 10.229446 / 10.191392 (0.038054) | 0.128105 / 0.680424 (-0.552319) | 0.014418 / 0.534201 (-0.519783) | 0.268615 / 0.579283 (-0.310668) | 0.263956 / 0.434364 (-0.170407) | 0.302213 / 0.540337 (-0.238125) | 0.427224 / 1.386936 (-0.959712) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005150 / 0.011353 (-0.006203) | 0.002557 / 0.011008 (-0.008451) | 0.048092 / 0.038508 (0.009584) | 0.050522 / 0.023109 (0.027413) | 0.272195 / 0.275898 (-0.003703) | 0.294191 / 0.323480 (-0.029289) | 0.004098 / 0.007986 (-0.003887) | 0.002350 / 0.004328 (-0.001978) | 0.048682 / 0.004250 (0.044432) | 0.038381 / 0.037052 (0.001328) | 0.275530 / 0.258489 (0.017041) | 0.303991 / 0.293841 (0.010150) | 0.024734 / 0.128546 (-0.103812) | 0.006926 / 0.075646 (-0.068720) | 0.053683 / 0.419271 (-0.365588) | 0.032675 / 0.043533 (-0.010858) | 0.272816 / 0.255139 (0.017677) | 0.291754 / 0.283200 (0.008554) | 0.018290 / 0.141683 (-0.123392) | 1.127696 / 1.452155 (-0.324459) | 1.187080 / 1.492716 (-0.305636) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091224 / 0.018006 (0.073218) | 0.288838 / 0.000490 (0.288348) | 0.000226 / 0.000200 (0.000026) | 0.000045 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021409 / 0.037411 (-0.016003) | 0.069846 / 0.014526 (0.055320) | 0.079962 / 0.176557 (-0.096594) | 0.118575 / 0.737135 (-0.618560) | 0.080223 / 0.296338 (-0.216115) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.290835 / 0.215209 (0.075626) | 2.831787 / 2.077655 (0.754133) | 1.587728 / 1.504120 (0.083608) | 1.461939 / 1.541195 (-0.079256) | 1.495257 / 1.468490 (0.026767) | 0.397653 / 4.584777 (-4.187124) | 2.399903 / 3.745712 (-1.345809) | 2.527615 / 5.269862 (-2.742247) | 1.501555 / 4.565676 (-3.064121) | 0.045742 / 0.424275 (-0.378533) | 0.004797 / 0.007607 (-0.002811) | 0.339139 / 0.226044 (0.113094) | 3.358340 / 2.268929 (1.089412) | 1.968955 / 55.444624 (-53.475670) | 1.663598 / 6.876477 (-5.212879) | 1.673995 / 2.142072 (-0.468078) | 0.463444 / 4.805227 (-4.341783) | 0.098008 / 6.500664 (-6.402656) | 0.040836 / 0.075469 (-0.034633) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.974033 / 1.841788 (-0.867755) | 11.863206 / 8.074308 (3.788897) | 10.892389 / 10.191392 (0.700997) | 0.128884 / 0.680424 (-0.551540) | 0.015319 / 0.534201 (-0.518882) | 0.268931 / 0.579283 (-0.310353) | 0.274148 / 0.434364 (-0.160216) | 0.305407 / 0.540337 (-0.234930) | 0.410574 / 1.386936 (-0.976362) |

![](https://cml.dev/watermark.png#0caf91285116ec910f409e82cc6e1f4cff7496e3 "CML watermark")


More robust temporary directory deletion

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6426). All of your documentation changes will be reflected on that endpoint.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004750 / 0.011353 (-0.006603) | 0.002928 / 0.011008 (-0.008080) | 0.061962 / 0.038508 (0.023454) | 0.029878 / 0.023109 (0.006768) | 0.233380 / 0.275898 (-0.042518) | 0.262221 / 0.323480 (-0.061259) | 0.002982 / 0.007986 (-0.005004) | 0.003698 / 0.004328 (-0.000630) | 0.048565 / 0.004250 (0.044314) | 0.046107 / 0.037052 (0.009055) | 0.240090 / 0.258489 (-0.018399) | 0.267294 / 0.293841 (-0.026547) | 0.023335 / 0.128546 (-0.105211) | 0.007221 / 0.075646 (-0.068425) | 0.200903 / 0.419271 (-0.218369) | 0.059237 / 0.043533 (0.015705) | 0.234929 / 0.255139 (-0.020210) | 0.256326 / 0.283200 (-0.026874) | 0.018549 / 0.141683 (-0.123134) | 1.103519 / 1.452155 (-0.348635) | 1.156573 / 1.492716 (-0.336143) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091205 / 0.018006 (0.073199) | 0.303533 / 0.000490 (0.303043) | 0.000204 / 0.000200 (0.000004) | 0.000042 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018572 / 0.037411 (-0.018839) | 0.062323 / 0.014526 (0.047797) | 0.074528 / 0.176557 (-0.102029) | 0.120295 / 0.737135 (-0.616841) | 0.076786 / 0.296338 (-0.219552) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.278814 / 0.215209 (0.063605) | 2.745483 / 2.077655 (0.667829) | 1.486073 / 1.504120 (-0.018047) | 1.385334 / 1.541195 (-0.155861) | 1.386351 / 1.468490 (-0.082139) | 0.395545 / 4.584777 (-4.189232) | 2.409468 / 3.745712 (-1.336244) | 2.670702 / 5.269862 (-2.599159) | 1.629245 / 4.565676 (-2.936432) | 0.045990 / 0.424275 (-0.378286) | 0.004782 / 0.007607 (-0.002825) | 0.332912 / 0.226044 (0.106867) | 3.249277 / 2.268929 (0.980349) | 1.888690 / 55.444624 (-53.555934) | 1.533462 / 6.876477 (-5.343015) | 1.576045 / 2.142072 (-0.566027) | 0.473090 / 4.805227 (-4.332138) | 0.099448 / 6.500664 (-6.401216) | 0.042613 / 0.075469 (-0.032857) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.944229 / 1.841788 (-0.897559) | 12.103621 / 8.074308 (4.029313) | 10.643471 / 10.191392 (0.452079) | 0.143004 / 0.680424 (-0.537420) | 0.013872 / 0.534201 (-0.520329) | 0.272026 / 0.579283 (-0.307257) | 0.298701 / 0.434364 (-0.135663) | 0.310299 / 0.540337 (-0.230038) | 0.420934 / 1.386936 (-0.966002) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004904 / 0.011353 (-0.006449) | 0.003064 / 0.011008 (-0.007945) | 0.047982 / 0.038508 (0.009474) | 0.056354 / 0.023109 (0.033245) | 0.292893 / 0.275898 (0.016995) | 0.348744 / 0.323480 (0.025264) | 0.003988 / 0.007986 (-0.003997) | 0.002431 / 0.004328 (-0.001898) | 0.049108 / 0.004250 (0.044857) | 0.039055 / 0.037052 (0.002002) | 0.278129 / 0.258489 (0.019640) | 0.318547 / 0.293841 (0.024706) | 0.025040 / 0.128546 (-0.103507) | 0.007166 / 0.075646 (-0.068480) | 0.053967 / 0.419271 (-0.365305) | 0.033128 / 0.043533 (-0.010405) | 0.272849 / 0.255139 (0.017710) | 0.312143 / 0.283200 (0.028943) | 0.017942 / 0.141683 (-0.123741) | 1.192297 / 1.452155 (-0.259857) | 1.328102 / 1.492716 (-0.164615) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090903 / 0.018006 (0.072896) | 0.301260 / 0.000490 (0.300770) | 0.000215 / 0.000200 (0.000015) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021112 / 0.037411 (-0.016300) | 0.070181 / 0.014526 (0.055656) | 0.082431 / 0.176557 (-0.094126) | 0.121973 / 0.737135 (-0.615163) | 0.083617 / 0.296338 (-0.212721) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.289587 / 0.215209 (0.074378) | 2.877895 / 2.077655 (0.800240) | 1.721417 / 1.504120 (0.217297) | 1.536023 / 1.541195 (-0.005171) | 1.550917 / 1.468490 (0.082427) | 0.402978 / 4.584777 (-4.181799) | 2.431767 / 3.745712 (-1.313946) | 2.544419 / 5.269862 (-2.725442) | 1.554562 / 4.565676 (-3.011115) | 0.046260 / 0.424275 (-0.378015) | 0.004923 / 0.007607 (-0.002684) | 0.341584 / 0.226044 (0.115540) | 3.362133 / 2.268929 (1.093205) | 1.928741 / 55.444624 (-53.515884) | 1.654798 / 6.876477 (-5.221679) | 1.715111 / 2.142072 (-0.426962) | 0.471029 / 4.805227 (-4.334198) | 0.098912 / 6.500664 (-6.401752) | 0.041018 / 0.075469 (-0.034451) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.992880 / 1.841788 (-0.848907) | 12.083890 / 8.074308 (4.009582) | 11.023833 / 10.191392 (0.832441) | 0.139217 / 0.680424 (-0.541207) | 0.015183 / 0.534201 (-0.519018) | 0.271637 / 0.579283 (-0.307646) | 0.278910 / 0.434364 (-0.155454) | 0.306891 / 0.540337 (-0.233447) | 0.424412 / 1.386936 (-0.962524) |

![](https://cml.dev/watermark.png#2d51f37eb9996d4c52250ee6e987ccce0d74f2f4 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004545 / 0.011353 (-0.006808) | 0.002955 / 0.011008 (-0.008054) | 0.062119 / 0.038508 (0.023611) | 0.029357 / 0.023109 (0.006248) | 0.240068 / 0.275898 (-0.035830) | 0.273376 / 0.323480 (-0.050104) | 0.003884 / 0.007986 (-0.004102) | 0.002390 / 0.004328 (-0.001938) | 0.048621 / 0.004250 (0.044371) | 0.043867 / 0.037052 (0.006815) | 0.247240 / 0.258489 (-0.011249) | 0.279187 / 0.293841 (-0.014654) | 0.023377 / 0.128546 (-0.105169) | 0.007261 / 0.075646 (-0.068385) | 0.201913 / 0.419271 (-0.217359) | 0.057063 / 0.043533 (0.013530) | 0.245698 / 0.255139 (-0.009441) | 0.265644 / 0.283200 (-0.017556) | 0.018077 / 0.141683 (-0.123606) | 1.133225 / 1.452155 (-0.318930) | 1.186380 / 1.492716 (-0.306336) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.089639 / 0.018006 (0.071632) | 0.298918 / 0.000490 (0.298428) | 0.000198 / 0.000200 (-0.000002) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019037 / 0.037411 (-0.018374) | 0.062580 / 0.014526 (0.048055) | 0.072974 / 0.176557 (-0.103582) | 0.119909 / 0.737135 (-0.617226) | 0.075021 / 0.296338 (-0.221317) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.276561 / 0.215209 (0.061352) | 2.697281 / 2.077655 (0.619626) | 1.419772 / 1.504120 (-0.084348) | 1.302079 / 1.541195 (-0.239115) | 1.329143 / 1.468490 (-0.139347) | 0.395528 / 4.584777 (-4.189249) | 2.365788 / 3.745712 (-1.379925) | 2.583802 / 5.269862 (-2.686059) | 1.561983 / 4.565676 (-3.003694) | 0.045269 / 0.424275 (-0.379006) | 0.004826 / 0.007607 (-0.002781) | 0.331041 / 0.226044 (0.104996) | 3.292523 / 2.268929 (1.023595) | 1.797865 / 55.444624 (-53.646759) | 1.509229 / 6.876477 (-5.367248) | 1.498884 / 2.142072 (-0.643188) | 0.458518 / 4.805227 (-4.346709) | 0.098076 / 6.500664 (-6.402588) | 0.042290 / 0.075469 (-0.033179) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.922331 / 1.841788 (-0.919457) | 11.605041 / 8.074308 (3.530732) | 10.471664 / 10.191392 (0.280272) | 0.130325 / 0.680424 (-0.550098) | 0.014084 / 0.534201 (-0.520117) | 0.278877 / 0.579283 (-0.300406) | 0.263104 / 0.434364 (-0.171259) | 0.306723 / 0.540337 (-0.233615) | 0.416238 / 1.386936 (-0.970698) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005094 / 0.011353 (-0.006259) | 0.002794 / 0.011008 (-0.008214) | 0.048189 / 0.038508 (0.009680) | 0.050409 / 0.023109 (0.027300) | 0.272618 / 0.275898 (-0.003280) | 0.293589 / 0.323480 (-0.029891) | 0.003995 / 0.007986 (-0.003991) | 0.002373 / 0.004328 (-0.001956) | 0.048269 / 0.004250 (0.044018) | 0.038751 / 0.037052 (0.001698) | 0.273495 / 0.258489 (0.015006) | 0.309244 / 0.293841 (0.015403) | 0.024681 / 0.128546 (-0.103866) | 0.007390 / 0.075646 (-0.068256) | 0.053844 / 0.419271 (-0.365427) | 0.032395 / 0.043533 (-0.011137) | 0.271963 / 0.255139 (0.016824) | 0.289557 / 0.283200 (0.006357) | 0.018659 / 0.141683 (-0.123024) | 1.154478 / 1.452155 (-0.297676) | 1.199772 / 1.492716 (-0.292944) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.089771 / 0.018006 (0.071764) | 0.299468 / 0.000490 (0.298978) | 0.000219 / 0.000200 (0.000020) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021854 / 0.037411 (-0.015558) | 0.070280 / 0.014526 (0.055754) | 0.080956 / 0.176557 (-0.095600) | 0.119430 / 0.737135 (-0.617705) | 0.082778 / 0.296338 (-0.213561) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.304273 / 0.215209 (0.089064) | 2.968264 / 2.077655 (0.890609) | 1.592363 / 1.504120 (0.088243) | 1.460795 / 1.541195 (-0.080400) | 1.501545 / 1.468490 (0.033055) | 0.411001 / 4.584777 (-4.173776) | 2.464273 / 3.745712 (-1.281439) | 2.524585 / 5.269862 (-2.745277) | 1.537443 / 4.565676 (-3.028234) | 0.046163 / 0.424275 (-0.378112) | 0.004783 / 0.007607 (-0.002824) | 0.354251 / 0.226044 (0.128206) | 3.512087 / 2.268929 (1.243158) | 1.968156 / 55.444624 (-53.476468) | 1.664966 / 6.876477 (-5.211510) | 1.685013 / 2.142072 (-0.457060) | 0.485793 / 4.805227 (-4.319435) | 0.099789 / 6.500664 (-6.400875) | 0.040705 / 0.075469 (-0.034764) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.966570 / 1.841788 (-0.875218) | 12.023188 / 8.074308 (3.948880) | 11.122602 / 10.191392 (0.931210) | 0.141002 / 0.680424 (-0.539422) | 0.015955 / 0.534201 (-0.518246) | 0.270293 / 0.579283 (-0.308990) | 0.281839 / 0.434364 (-0.152525) | 0.307279 / 0.540337 (-0.233058) | 0.434687 / 1.386936 (-0.952249) |

![](https://cml.dev/watermark.png#7eaad71464e85c7358eaa36494227a43257ffcd8 "CML watermark")

What would be the impact for non-windows users ?

Also I wonder if a gc.collect() after the `del` could help to remove the PermissionError ? Or register the dataset for deletion on copy/pickle maybe ?

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004973 / 0.011353 (-0.006380) | 0.002753 / 0.011008 (-0.008256) | 0.061489 / 0.038508 (0.022981) | 0.051122 / 0.023109 (0.028012) | 0.228783 / 0.275898 (-0.047115) | 0.256982 / 0.323480 (-0.066498) | 0.002873 / 0.007986 (-0.005112) | 0.003544 / 0.004328 (-0.000784) | 0.048721 / 0.004250 (0.044471) | 0.039137 / 0.037052 (0.002085) | 0.244988 / 0.258489 (-0.013501) | 0.275230 / 0.293841 (-0.018611) | 0.023034 / 0.128546 (-0.105513) | 0.006988 / 0.075646 (-0.068658) | 0.202780 / 0.419271 (-0.216492) | 0.035325 / 0.043533 (-0.008207) | 0.241722 / 0.255139 (-0.013417) | 0.259671 / 0.283200 (-0.023528) | 0.019875 / 0.141683 (-0.121808) | 1.098667 / 1.452155 (-0.353488) | 1.161444 / 1.492716 (-0.331272) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093591 / 0.018006 (0.075585) | 0.298703 / 0.000490 (0.298213) | 0.000219 / 0.000200 (0.000019) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018319 / 0.037411 (-0.019092) | 0.062993 / 0.014526 (0.048467) | 0.074313 / 0.176557 (-0.102244) | 0.123089 / 0.737135 (-0.614046) | 0.075177 / 0.296338 (-0.221162) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.268584 / 0.215209 (0.053375) | 2.633116 / 2.077655 (0.555461) | 1.390743 / 1.504120 (-0.113377) | 1.277385 / 1.541195 (-0.263810) | 1.287934 / 1.468490 (-0.180556) | 0.387934 / 4.584777 (-4.196843) | 2.345819 / 3.745712 (-1.399893) | 2.558169 / 5.269862 (-2.711693) | 1.569812 / 4.565676 (-2.995865) | 0.045297 / 0.424275 (-0.378978) | 0.005238 / 0.007607 (-0.002369) | 0.359704 / 0.226044 (0.133659) | 3.204688 / 2.268929 (0.935759) | 1.753321 / 55.444624 (-53.691303) | 1.492223 / 6.876477 (-5.384254) | 1.498207 / 2.142072 (-0.643865) | 0.459830 / 4.805227 (-4.345397) | 0.098194 / 6.500664 (-6.402470) | 0.042632 / 0.075469 (-0.032837) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.963020 / 1.841788 (-0.878768) | 11.500470 / 8.074308 (3.426161) | 10.451882 / 10.191392 (0.260490) | 0.127706 / 0.680424 (-0.552718) | 0.014084 / 0.534201 (-0.520117) | 0.269728 / 0.579283 (-0.309555) | 0.260283 / 0.434364 (-0.174080) | 0.303717 / 0.540337 (-0.236620) | 0.397028 / 1.386936 (-0.989908) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004823 / 0.011353 (-0.006529) | 0.002751 / 0.011008 (-0.008257) | 0.048719 / 0.038508 (0.010211) | 0.051409 / 0.023109 (0.028300) | 0.267139 / 0.275898 (-0.008759) | 0.287659 / 0.323480 (-0.035821) | 0.003959 / 0.007986 (-0.004027) | 0.002376 / 0.004328 (-0.001953) | 0.047942 / 0.004250 (0.043692) | 0.039742 / 0.037052 (0.002690) | 0.268348 / 0.258489 (0.009859) | 0.297201 / 0.293841 (0.003360) | 0.024226 / 0.128546 (-0.104320) | 0.007103 / 0.075646 (-0.068544) | 0.053310 / 0.419271 (-0.365961) | 0.032716 / 0.043533 (-0.010816) | 0.269469 / 0.255139 (0.014330) | 0.287752 / 0.283200 (0.004553) | 0.018191 / 0.141683 (-0.123492) | 1.114086 / 1.452155 (-0.338069) | 1.188054 / 1.492716 (-0.304662) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091072 / 0.018006 (0.073066) | 0.300367 / 0.000490 (0.299877) | 0.000218 / 0.000200 (0.000018) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.020970 / 0.037411 (-0.016441) | 0.070356 / 0.014526 (0.055830) | 0.081339 / 0.176557 (-0.095218) | 0.120741 / 0.737135 (-0.616394) | 0.081677 / 0.296338 (-0.214662) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.290405 / 0.215209 (0.075196) | 2.863877 / 2.077655 (0.786222) | 1.524603 / 1.504120 (0.020483) | 1.397917 / 1.541195 (-0.143278) | 1.402635 / 1.468490 (-0.065855) | 0.405525 / 4.584777 (-4.179252) | 2.432474 / 3.745712 (-1.313239) | 2.446277 / 5.269862 (-2.823585) | 1.550300 / 4.565676 (-3.015377) | 0.046545 / 0.424275 (-0.377730) | 0.004824 / 0.007607 (-0.002783) | 0.343578 / 0.226044 (0.117534) | 3.436850 / 2.268929 (1.167922) | 1.897200 / 55.444624 (-53.547425) | 1.625222 / 6.876477 (-5.251255) | 1.730488 / 2.142072 (-0.411585) | 0.482099 / 4.805227 (-4.323129) | 0.097828 / 6.500664 (-6.402836) | 0.040385 / 0.075469 (-0.035084) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.950975 / 1.841788 (-0.890812) | 11.875024 / 8.074308 (3.800715) | 10.430301 / 10.191392 (0.238909) | 0.130546 / 0.680424 (-0.549878) | 0.015423 / 0.534201 (-0.518778) | 0.269592 / 0.579283 (-0.309691) | 0.282505 / 0.434364 (-0.151859) | 0.305567 / 0.540337 (-0.234771) | 0.522142 / 1.386936 (-0.864794) |

![](https://cml.dev/watermark.png#c166692aa955528180dd4d55474a984f6044896d "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004983 / 0.011353 (-0.006369) | 0.003346 / 0.011008 (-0.007662) | 0.062233 / 0.038508 (0.023725) | 0.050246 / 0.023109 (0.027137) | 0.305738 / 0.275898 (0.029839) | 0.321863 / 0.323480 (-0.001617) | 0.003870 / 0.007986 (-0.004116) | 0.002610 / 0.004328 (-0.001718) | 0.047734 / 0.004250 (0.043483) | 0.037611 / 0.037052 (0.000559) | 0.299121 / 0.258489 (0.040632) | 0.327370 / 0.293841 (0.033529) | 0.027009 / 0.128546 (-0.101537) | 0.010816 / 0.075646 (-0.064830) | 0.204627 / 0.419271 (-0.214645) | 0.035708 / 0.043533 (-0.007825) | 0.291837 / 0.255139 (0.036698) | 0.313646 / 0.283200 (0.030447) | 0.017277 / 0.141683 (-0.124405) | 1.097907 / 1.452155 (-0.354248) | 1.163203 / 1.492716 (-0.329513) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091933 / 0.018006 (0.073926) | 0.298787 / 0.000490 (0.298297) | 0.000204 / 0.000200 (0.000004) | 0.000051 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018349 / 0.037411 (-0.019062) | 0.061520 / 0.014526 (0.046994) | 0.073159 / 0.176557 (-0.103397) | 0.118657 / 0.737135 (-0.618478) | 0.073601 / 0.296338 (-0.222737) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.276297 / 0.215209 (0.061088) | 2.725668 / 2.077655 (0.648013) | 1.458079 / 1.504120 (-0.046041) | 1.331236 / 1.541195 (-0.209959) | 1.347919 / 1.468490 (-0.120571) | 0.565954 / 4.584777 (-4.018823) | 2.380883 / 3.745712 (-1.364829) | 2.800533 / 5.269862 (-2.469329) | 1.740534 / 4.565676 (-2.825142) | 0.065617 / 0.424275 (-0.358658) | 0.004907 / 0.007607 (-0.002700) | 0.335973 / 0.226044 (0.109929) | 3.337405 / 2.268929 (1.068476) | 1.819852 / 55.444624 (-53.624772) | 1.542724 / 6.876477 (-5.333752) | 1.509508 / 2.142072 (-0.632565) | 0.648618 / 4.805227 (-4.156609) | 0.116812 / 6.500664 (-6.383852) | 0.041561 / 0.075469 (-0.033909) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.943488 / 1.841788 (-0.898299) | 11.184770 / 8.074308 (3.110462) | 10.406311 / 10.191392 (0.214919) | 0.129841 / 0.680424 (-0.550583) | 0.013736 / 0.534201 (-0.520465) | 0.287281 / 0.579283 (-0.292002) | 0.267403 / 0.434364 (-0.166961) | 0.325319 / 0.540337 (-0.215019) | 0.454207 / 1.386936 (-0.932729) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005169 / 0.011353 (-0.006183) | 0.003155 / 0.011008 (-0.007854) | 0.048101 / 0.038508 (0.009593) | 0.048726 / 0.023109 (0.025617) | 0.275768 / 0.275898 (-0.000130) | 0.291209 / 0.323480 (-0.032271) | 0.003984 / 0.007986 (-0.004001) | 0.002586 / 0.004328 (-0.001742) | 0.047751 / 0.004250 (0.043500) | 0.040176 / 0.037052 (0.003124) | 0.279161 / 0.258489 (0.020672) | 0.297371 / 0.293841 (0.003530) | 0.028502 / 0.128546 (-0.100044) | 0.010103 / 0.075646 (-0.065544) | 0.056920 / 0.419271 (-0.362351) | 0.032174 / 0.043533 (-0.011359) | 0.271925 / 0.255139 (0.016786) | 0.289572 / 0.283200 (0.006372) | 0.017981 / 0.141683 (-0.123702) | 1.192972 / 1.452155 (-0.259183) | 1.223231 / 1.492716 (-0.269485) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091363 / 0.018006 (0.073356) | 0.298106 / 0.000490 (0.297616) | 0.000216 / 0.000200 (0.000016) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021509 / 0.037411 (-0.015902) | 0.068377 / 0.014526 (0.053851) | 0.079798 / 0.176557 (-0.096759) | 0.120546 / 0.737135 (-0.616589) | 0.080602 / 0.296338 (-0.215737) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.300809 / 0.215209 (0.085600) | 2.921144 / 2.077655 (0.843489) | 1.621096 / 1.504120 (0.116976) | 1.504265 / 1.541195 (-0.036930) | 1.508050 / 1.468490 (0.039560) | 0.554291 / 4.584777 (-4.030486) | 2.418798 / 3.745712 (-1.326914) | 2.768088 / 5.269862 (-2.501773) | 1.728267 / 4.565676 (-2.837410) | 0.062943 / 0.424275 (-0.361332) | 0.004891 / 0.007607 (-0.002716) | 0.350298 / 0.226044 (0.124254) | 3.442782 / 2.268929 (1.173853) | 1.960163 / 55.444624 (-53.484461) | 1.682000 / 6.876477 (-5.194477) | 1.680311 / 2.142072 (-0.461761) | 0.631201 / 4.805227 (-4.174026) | 0.115211 / 6.500664 (-6.385453) | 0.041279 / 0.075469 (-0.034190) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.962478 / 1.841788 (-0.879310) | 11.671463 / 8.074308 (3.597155) | 10.640129 / 10.191392 (0.448737) | 0.130649 / 0.680424 (-0.549775) | 0.016169 / 0.534201 (-0.518032) | 0.286894 / 0.579283 (-0.292389) | 0.269319 / 0.434364 (-0.165045) | 0.324512 / 0.540337 (-0.215825) | 0.550874 / 1.386936 (-0.836062) |

![](https://cml.dev/watermark.png#69f135121beb1616f1d7c7584b317d4e41e21275 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005078 / 0.011353 (-0.006275) | 0.003950 / 0.011008 (-0.007058) | 0.063345 / 0.038508 (0.024837) | 0.054486 / 0.023109 (0.031377) | 0.243213 / 0.275898 (-0.032685) | 0.264079 / 0.323480 (-0.059401) | 0.003922 / 0.007986 (-0.004064) | 0.002631 / 0.004328 (-0.001698) | 0.048660 / 0.004250 (0.044409) | 0.037205 / 0.037052 (0.000153) | 0.244577 / 0.258489 (-0.013912) | 0.276025 / 0.293841 (-0.017816) | 0.027134 / 0.128546 (-0.101412) | 0.010921 / 0.075646 (-0.064726) | 0.209792 / 0.419271 (-0.209479) | 0.035999 / 0.043533 (-0.007534) | 0.245671 / 0.255139 (-0.009468) | 0.262807 / 0.283200 (-0.020393) | 0.018173 / 0.141683 (-0.123510) | 1.084417 / 1.452155 (-0.367738) | 1.148284 / 1.492716 (-0.344432) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093128 / 0.018006 (0.075122) | 0.301606 / 0.000490 (0.301117) | 0.000221 / 0.000200 (0.000021) | 0.000050 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018718 / 0.037411 (-0.018693) | 0.060819 / 0.014526 (0.046293) | 0.073050 / 0.176557 (-0.103507) | 0.120043 / 0.737135 (-0.617092) | 0.075374 / 0.296338 (-0.220965) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.291080 / 0.215209 (0.075871) | 2.808802 / 2.077655 (0.731148) | 1.485686 / 1.504120 (-0.018434) | 1.354356 / 1.541195 (-0.186839) | 1.347863 / 1.468490 (-0.120627) | 0.571501 / 4.584777 (-4.013276) | 2.377960 / 3.745712 (-1.367752) | 2.768023 / 5.269862 (-2.501839) | 1.754360 / 4.565676 (-2.811316) | 0.063115 / 0.424275 (-0.361160) | 0.004941 / 0.007607 (-0.002666) | 0.338281 / 0.226044 (0.112237) | 3.340587 / 2.268929 (1.071658) | 1.849479 / 55.444624 (-53.595145) | 1.551846 / 6.876477 (-5.324631) | 1.539090 / 2.142072 (-0.602983) | 0.644522 / 4.805227 (-4.160705) | 0.117398 / 6.500664 (-6.383266) | 0.042239 / 0.075469 (-0.033230) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.949496 / 1.841788 (-0.892291) | 11.548352 / 8.074308 (3.474044) | 10.478065 / 10.191392 (0.286673) | 0.129534 / 0.680424 (-0.550890) | 0.015378 / 0.534201 (-0.518822) | 0.287221 / 0.579283 (-0.292062) | 0.262944 / 0.434364 (-0.171419) | 0.321727 / 0.540337 (-0.218611) | 0.432354 / 1.386936 (-0.954582) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005256 / 0.011353 (-0.006097) | 0.003491 / 0.011008 (-0.007517) | 0.048647 / 0.038508 (0.010139) | 0.054011 / 0.023109 (0.030901) | 0.271786 / 0.275898 (-0.004112) | 0.291964 / 0.323480 (-0.031516) | 0.004035 / 0.007986 (-0.003950) | 0.002671 / 0.004328 (-0.001657) | 0.048108 / 0.004250 (0.043857) | 0.040421 / 0.037052 (0.003368) | 0.278594 / 0.258489 (0.020105) | 0.300707 / 0.293841 (0.006867) | 0.028924 / 0.128546 (-0.099623) | 0.010600 / 0.075646 (-0.065047) | 0.057649 / 0.419271 (-0.361623) | 0.034221 / 0.043533 (-0.009312) | 0.276692 / 0.255139 (0.021553) | 0.293545 / 0.283200 (0.010345) | 0.017908 / 0.141683 (-0.123775) | 1.135108 / 1.452155 (-0.317047) | 1.190823 / 1.492716 (-0.301893) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095243 / 0.018006 (0.077237) | 0.301885 / 0.000490 (0.301396) | 0.000235 / 0.000200 (0.000035) | 0.000056 / 0.000054 (0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021561 / 0.037411 (-0.015850) | 0.069054 / 0.014526 (0.054529) | 0.080466 / 0.176557 (-0.096091) | 0.121323 / 0.737135 (-0.615812) | 0.081891 / 0.296338 (-0.214448) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.293957 / 0.215209 (0.078748) | 2.869035 / 2.077655 (0.791380) | 1.608837 / 1.504120 (0.104717) | 1.440594 / 1.541195 (-0.100601) | 1.464775 / 1.468490 (-0.003715) | 0.565663 / 4.584777 (-4.019114) | 2.439456 / 3.745712 (-1.306256) | 2.794775 / 5.269862 (-2.475087) | 1.750026 / 4.565676 (-2.815651) | 0.063291 / 0.424275 (-0.360984) | 0.004930 / 0.007607 (-0.002677) | 0.347169 / 0.226044 (0.121125) | 3.408260 / 2.268929 (1.139331) | 1.920933 / 55.444624 (-53.523691) | 1.648821 / 6.876477 (-5.227656) | 1.639022 / 2.142072 (-0.503051) | 0.642870 / 4.805227 (-4.162357) | 0.117077 / 6.500664 (-6.383587) | 0.040784 / 0.075469 (-0.034685) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.993501 / 1.841788 (-0.848287) | 12.012423 / 8.074308 (3.938115) | 10.740932 / 10.191392 (0.549540) | 0.132409 / 0.680424 (-0.548015) | 0.015294 / 0.534201 (-0.518907) | 0.287902 / 0.579283 (-0.291381) | 0.281350 / 0.434364 (-0.153014) | 0.329201 / 0.540337 (-0.211137) | 0.553199 / 1.386936 (-0.833737) |

![](https://cml.dev/watermark.png#ecd3a22c5dec2133491a320515e12956512439eb "CML watermark")

While fixing the Windows errors in #6362, I noticed that `PermissionError` can still easily be thrown on the session exit by the temporary cache directory's finalizer (we would also have to keep track of intermediate datasets, copies, etc.). ~~Due to the low usage of `datasets` on Windows, this PR takes a simpler approach to the issue than https://github.com/huggingface/datasets/pull/2403 - it tries to delete the temporary cache directory, and if this fails, logs a warning message about using a `delete-temp-cache` CLI command to delete it manually. The problematic references are freed after the session exits, so the CLI command should then succeed.~~ This PR implements `Dataset.__setstate__` to register datasets with temporary cache files for deletion.