Release: 2.10.0

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009437 / 0.011353 (-0.001916) | 0.004999 / 0.011008 (-0.006010) | 0.098839 / 0.038508 (0.060331) | 0.035496 / 0.023109 (0.012386) | 0.300726 / 0.275898 (0.024828) | 0.359793 / 0.323480 (0.036313) | 0.007694 / 0.007986 (-0.000292) | 0.003980 / 0.004328 (-0.000348) | 0.075240 / 0.004250 (0.070989) | 0.041149 / 0.037052 (0.004097) | 0.313185 / 0.258489 (0.054696) | 0.344111 / 0.293841 (0.050270) | 0.037775 / 0.128546 (-0.090772) | 0.011901 / 0.075646 (-0.063745) | 0.332631 / 0.419271 (-0.086641) | 0.047194 / 0.043533 (0.003661) | 0.306902 / 0.255139 (0.051763) | 0.321725 / 0.283200 (0.038525) | 0.101031 / 0.141683 (-0.040652) | 1.458778 / 1.452155 (0.006623) | 1.530196 / 1.492716 (0.037480) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.203241 / 0.018006 (0.185235) | 0.447147 / 0.000490 (0.446657) | 0.004159 / 0.000200 (0.003959) | 0.000131 / 0.000054 (0.000076) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025845 / 0.037411 (-0.011566) | 0.106966 / 0.014526 (0.092440) | 0.115876 / 0.176557 (-0.060681) | 0.179052 / 0.737135 (-0.558084) | 0.123012 / 0.296338 (-0.173327) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.408766 / 0.215209 (0.193557) | 4.080400 / 2.077655 (2.002745) | 1.893747 / 1.504120 (0.389627) | 1.709389 / 1.541195 (0.168194) | 1.768071 / 1.468490 (0.299581) | 0.689717 / 4.584777 (-3.895059) | 3.760897 / 3.745712 (0.015185) | 2.017050 / 5.269862 (-3.252811) | 1.333027 / 4.565676 (-3.232650) | 0.083559 / 0.424275 (-0.340716) | 0.011951 / 0.007607 (0.004344) | 0.512313 / 0.226044 (0.286268) | 5.162696 / 2.268929 (2.893767) | 2.418559 / 55.444624 (-53.026065) | 2.110178 / 6.876477 (-4.766299) | 2.113635 / 2.142072 (-0.028437) | 0.835171 / 4.805227 (-3.970056) | 0.164222 / 6.500664 (-6.336442) | 0.061955 / 0.075469 (-0.013515) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.198336 / 1.841788 (-0.643452) | 14.531468 / 8.074308 (6.457160) | 13.882133 / 10.191392 (3.690741) | 0.154524 / 0.680424 (-0.525900) | 0.028782 / 0.534201 (-0.505419) | 0.441808 / 0.579283 (-0.137475) | 0.433096 / 0.434364 (-0.001268) | 0.518229 / 0.540337 (-0.022108) | 0.603201 / 1.386936 (-0.783735) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007385 / 0.011353 (-0.003967) | 0.005193 / 0.011008 (-0.005815) | 0.075517 / 0.038508 (0.037009) | 0.033192 / 0.023109 (0.010083) | 0.332299 / 0.275898 (0.056401) | 0.363043 / 0.323480 (0.039563) | 0.006368 / 0.007986 (-0.001617) | 0.004003 / 0.004328 (-0.000326) | 0.073710 / 0.004250 (0.069460) | 0.046916 / 0.037052 (0.009863) | 0.336307 / 0.258489 (0.077818) | 0.384910 / 0.293841 (0.091069) | 0.038132 / 0.128546 (-0.090414) | 0.012283 / 0.075646 (-0.063364) | 0.088036 / 0.419271 (-0.331235) | 0.049699 / 0.043533 (0.006166) | 0.333953 / 0.255139 (0.078814) | 0.352961 / 0.283200 (0.069762) | 0.101905 / 0.141683 (-0.039778) | 1.470480 / 1.452155 (0.018325) | 1.498212 / 1.492716 (0.005496) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.275067 / 0.018006 (0.257061) | 0.452589 / 0.000490 (0.452099) | 0.047067 / 0.000200 (0.046867) | 0.000983 / 0.000054 (0.000929) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028649 / 0.037411 (-0.008762) | 0.108385 / 0.014526 (0.093859) | 0.121213 / 0.176557 (-0.055343) | 0.192236 / 0.737135 (-0.544899) | 0.124620 / 0.296338 (-0.171719) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.428742 / 0.215209 (0.213533) | 4.264893 / 2.077655 (2.187238) | 2.061650 / 1.504120 (0.557530) | 1.873267 / 1.541195 (0.332072) | 1.961012 / 1.468490 (0.492522) | 0.708904 / 4.584777 (-3.875873) | 3.821289 / 3.745712 (0.075577) | 3.287231 / 5.269862 (-1.982631) | 1.903539 / 4.565676 (-2.662137) | 0.086474 / 0.424275 (-0.337801) | 0.012101 / 0.007607 (0.004494) | 0.531411 / 0.226044 (0.305367) | 5.216785 / 2.268929 (2.947857) | 2.575209 / 55.444624 (-52.869416) | 2.264902 / 6.876477 (-4.611574) | 2.291225 / 2.142072 (0.149153) | 0.853486 / 4.805227 (-3.951741) | 0.168550 / 6.500664 (-6.332114) | 0.064158 / 0.075469 (-0.011311) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.295830 / 1.841788 (-0.545958) | 14.419524 / 8.074308 (6.345216) | 13.397985 / 10.191392 (3.206593) | 0.181367 / 0.680424 (-0.499057) | 0.017666 / 0.534201 (-0.516535) | 0.420645 / 0.579283 (-0.158638) | 0.421025 / 0.434364 (-0.013339) | 0.527369 / 0.540337 (-0.012969) | 0.627175 / 1.386936 (-0.759761) |

![](https://cml.dev/watermark.png#589b49dfaffa729bc9997a38d4cedafb107ea2e4 "CML watermark")

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008717 / 0.011353 (-0.002635) | 0.004573 / 0.011008 (-0.006435) | 0.103660 / 0.038508 (0.065151) | 0.035274 / 0.023109 (0.012165) | 0.298563 / 0.275898 (0.022665) | 0.384397 / 0.323480 (0.060917) | 0.006932 / 0.007986 (-0.001053) | 0.003422 / 0.004328 (-0.000907) | 0.080193 / 0.004250 (0.075943) | 0.039767 / 0.037052 (0.002714) | 0.310296 / 0.258489 (0.051807) | 0.351361 / 0.293841 (0.057520) | 0.033532 / 0.128546 (-0.095014) | 0.011543 / 0.075646 (-0.064104) | 0.374816 / 0.419271 (-0.044456) | 0.046046 / 0.043533 (0.002513) | 0.306918 / 0.255139 (0.051779) | 0.382242 / 0.283200 (0.099042) | 0.098945 / 0.141683 (-0.042738) | 1.456929 / 1.452155 (0.004775) | 1.535763 / 1.492716 (0.043046) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.011759 / 0.018006 (-0.006247) | 0.405345 / 0.000490 (0.404855) | 0.002667 / 0.000200 (0.002467) | 0.000075 / 0.000054 (0.000020) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023924 / 0.037411 (-0.013487) | 0.095537 / 0.014526 (0.081011) | 0.106959 / 0.176557 (-0.069598) | 0.170782 / 0.737135 (-0.566353) | 0.109169 / 0.296338 (-0.187170) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.437521 / 0.215209 (0.222312) | 4.383556 / 2.077655 (2.305902) | 2.092055 / 1.504120 (0.587935) | 1.889316 / 1.541195 (0.348121) | 1.937436 / 1.468490 (0.468946) | 0.700175 / 4.584777 (-3.884602) | 3.358107 / 3.745712 (-0.387605) | 3.243226 / 5.269862 (-2.026636) | 1.620497 / 4.565676 (-2.945180) | 0.083063 / 0.424275 (-0.341212) | 0.012970 / 0.007607 (0.005363) | 0.544226 / 0.226044 (0.318181) | 5.483315 / 2.268929 (3.214386) | 2.555183 / 55.444624 (-52.889441) | 2.204230 / 6.876477 (-4.672247) | 2.230551 / 2.142072 (0.088478) | 0.816121 / 4.805227 (-3.989106) | 0.151356 / 6.500664 (-6.349308) | 0.068564 / 0.075469 (-0.006905) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.208420 / 1.841788 (-0.633367) | 13.652597 / 8.074308 (5.578289) | 14.096318 / 10.191392 (3.904926) | 0.154473 / 0.680424 (-0.525951) | 0.028436 / 0.534201 (-0.505765) | 0.399949 / 0.579283 (-0.179334) | 0.398961 / 0.434364 (-0.035403) | 0.488703 / 0.540337 (-0.051634) | 0.572640 / 1.386936 (-0.814296) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006373 / 0.011353 (-0.004979) | 0.004368 / 0.011008 (-0.006640) | 0.076410 / 0.038508 (0.037902) | 0.027055 / 0.023109 (0.003945) | 0.336969 / 0.275898 (0.061071) | 0.374533 / 0.323480 (0.051053) | 0.004781 / 0.007986 (-0.003204) | 0.003317 / 0.004328 (-0.001011) | 0.076099 / 0.004250 (0.071849) | 0.038414 / 0.037052 (0.001361) | 0.339578 / 0.258489 (0.081089) | 0.384138 / 0.293841 (0.090297) | 0.031581 / 0.128546 (-0.096965) | 0.011666 / 0.075646 (-0.063981) | 0.085690 / 0.419271 (-0.333582) | 0.042277 / 0.043533 (-0.001256) | 0.337931 / 0.255139 (0.082792) | 0.365827 / 0.283200 (0.082628) | 0.088713 / 0.141683 (-0.052970) | 1.519789 / 1.452155 (0.067635) | 1.583097 / 1.492716 (0.090381) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.223472 / 0.018006 (0.205466) | 0.392474 / 0.000490 (0.391984) | 0.002739 / 0.000200 (0.002539) | 0.000076 / 0.000054 (0.000021) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024373 / 0.037411 (-0.013038) | 0.099822 / 0.014526 (0.085296) | 0.106128 / 0.176557 (-0.070428) | 0.174688 / 0.737135 (-0.562447) | 0.112660 / 0.296338 (-0.183678) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.436317 / 0.215209 (0.221108) | 4.358277 / 2.077655 (2.280622) | 2.089746 / 1.504120 (0.585626) | 1.881040 / 1.541195 (0.339845) | 1.923653 / 1.468490 (0.455163) | 0.698176 / 4.584777 (-3.886601) | 3.346460 / 3.745712 (-0.399252) | 3.301429 / 5.269862 (-1.968433) | 1.391042 / 4.565676 (-3.174634) | 0.083025 / 0.424275 (-0.341250) | 0.012459 / 0.007607 (0.004851) | 0.533011 / 0.226044 (0.306967) | 5.334984 / 2.268929 (3.066056) | 2.534105 / 55.444624 (-52.910520) | 2.206295 / 6.876477 (-4.670181) | 2.231752 / 2.142072 (0.089680) | 0.798650 / 4.805227 (-4.006577) | 0.150070 / 6.500664 (-6.350594) | 0.066898 / 0.075469 (-0.008571) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.310527 / 1.841788 (-0.531261) | 13.920492 / 8.074308 (5.846184) | 13.359382 / 10.191392 (3.167990) | 0.154561 / 0.680424 (-0.525863) | 0.016387 / 0.534201 (-0.517814) | 0.379892 / 0.579283 (-0.199391) | 0.376746 / 0.434364 (-0.057618) | 0.462606 / 0.540337 (-0.077732) | 0.550895 / 1.386936 (-0.836041) |

![](https://cml.dev/watermark.png#cac733fdaef84cfee92856bd259ce024ec157c91 "CML watermark")

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009373 / 0.011353 (-0.001980) | 0.005212 / 0.011008 (-0.005797) | 0.099287 / 0.038508 (0.060779) | 0.035175 / 0.023109 (0.012066) | 0.307012 / 0.275898 (0.031114) | 0.335105 / 0.323480 (0.011625) | 0.008006 / 0.007986 (0.000020) | 0.004017 / 0.004328 (-0.000311) | 0.075519 / 0.004250 (0.071269) | 0.040276 / 0.037052 (0.003223) | 0.302615 / 0.258489 (0.044126) | 0.361742 / 0.293841 (0.067901) | 0.038773 / 0.128546 (-0.089773) | 0.011892 / 0.075646 (-0.063754) | 0.334199 / 0.419271 (-0.085073) | 0.048035 / 0.043533 (0.004503) | 0.301361 / 0.255139 (0.046222) | 0.321996 / 0.283200 (0.038796) | 0.101818 / 0.141683 (-0.039865) | 1.442601 / 1.452155 (-0.009554) | 1.530669 / 1.492716 (0.037953) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.201470 / 0.018006 (0.183464) | 0.496305 / 0.000490 (0.495815) | 0.003794 / 0.000200 (0.003594) | 0.000149 / 0.000054 (0.000094) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028401 / 0.037411 (-0.009010) | 0.107924 / 0.014526 (0.093398) | 0.121716 / 0.176557 (-0.054840) | 0.187407 / 0.737135 (-0.549728) | 0.124755 / 0.296338 (-0.171583) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.395667 / 0.215209 (0.180457) | 3.939079 / 2.077655 (1.861424) | 1.776308 / 1.504120 (0.272188) | 1.583487 / 1.541195 (0.042292) | 1.682957 / 1.468490 (0.214467) | 0.677322 / 4.584777 (-3.907455) | 3.796987 / 3.745712 (0.051275) | 3.406199 / 5.269862 (-1.863663) | 1.905467 / 4.565676 (-2.660210) | 0.083189 / 0.424275 (-0.341086) | 0.012156 / 0.007607 (0.004549) | 0.507078 / 0.226044 (0.281033) | 5.031293 / 2.268929 (2.762365) | 2.228403 / 55.444624 (-53.216221) | 1.885760 / 6.876477 (-4.990717) | 1.962340 / 2.142072 (-0.179732) | 0.824979 / 4.805227 (-3.980248) | 0.162107 / 6.500664 (-6.338557) | 0.062324 / 0.075469 (-0.013145) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.205104 / 1.841788 (-0.636683) | 15.368896 / 8.074308 (7.294588) | 14.757540 / 10.191392 (4.566148) | 0.177544 / 0.680424 (-0.502880) | 0.029097 / 0.534201 (-0.505104) | 0.445252 / 0.579283 (-0.134031) | 0.456521 / 0.434364 (0.022157) | 0.544166 / 0.540337 (0.003829) | 0.640675 / 1.386936 (-0.746261) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007438 / 0.011353 (-0.003914) | 0.005236 / 0.011008 (-0.005772) | 0.075379 / 0.038508 (0.036871) | 0.033274 / 0.023109 (0.010165) | 0.344584 / 0.275898 (0.068686) | 0.372161 / 0.323480 (0.048681) | 0.005914 / 0.007986 (-0.002071) | 0.004176 / 0.004328 (-0.000152) | 0.073311 / 0.004250 (0.069061) | 0.050845 / 0.037052 (0.013793) | 0.338978 / 0.258489 (0.080489) | 0.391563 / 0.293841 (0.097722) | 0.037559 / 0.128546 (-0.090987) | 0.012455 / 0.075646 (-0.063192) | 0.086224 / 0.419271 (-0.333047) | 0.052956 / 0.043533 (0.009423) | 0.338529 / 0.255139 (0.083390) | 0.356752 / 0.283200 (0.073553) | 0.105864 / 0.141683 (-0.035819) | 1.467727 / 1.452155 (0.015572) | 1.588727 / 1.492716 (0.096010) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.215959 / 0.018006 (0.197953) | 0.440619 / 0.000490 (0.440129) | 0.000397 / 0.000200 (0.000197) | 0.000058 / 0.000054 (0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028855 / 0.037411 (-0.008556) | 0.114239 / 0.014526 (0.099713) | 0.121726 / 0.176557 (-0.054830) | 0.190377 / 0.737135 (-0.546759) | 0.127858 / 0.296338 (-0.168480) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.415399 / 0.215209 (0.200190) | 4.159012 / 2.077655 (2.081357) | 1.987593 / 1.504120 (0.483474) | 1.794785 / 1.541195 (0.253591) | 1.924819 / 1.468490 (0.456329) | 0.696082 / 4.584777 (-3.888694) | 3.820461 / 3.745712 (0.074749) | 2.139236 / 5.269862 (-3.130626) | 1.348593 / 4.565676 (-3.217084) | 0.086536 / 0.424275 (-0.337739) | 0.012510 / 0.007607 (0.004902) | 0.518804 / 0.226044 (0.292760) | 5.188659 / 2.268929 (2.919730) | 2.501303 / 55.444624 (-52.943322) | 2.138831 / 6.876477 (-4.737646) | 2.220451 / 2.142072 (0.078378) | 0.836277 / 4.805227 (-3.968950) | 0.170940 / 6.500664 (-6.329724) | 0.067326 / 0.075469 (-0.008143) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.307848 / 1.841788 (-0.533940) | 15.995785 / 8.074308 (7.921477) | 13.646285 / 10.191392 (3.454893) | 0.181120 / 0.680424 (-0.499304) | 0.017500 / 0.534201 (-0.516701) | 0.426697 / 0.579283 (-0.152586) | 0.436702 / 0.434364 (0.002338) | 0.518060 / 0.540337 (-0.022278) | 0.632577 / 1.386936 (-0.754359) |

![](https://cml.dev/watermark.png#cac733fdaef84cfee92856bd259ce024ec157c91 "CML watermark")


Update csv.py

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008358 / 0.011353 (-0.002995) | 0.004555 / 0.011008 (-0.006453) | 0.100935 / 0.038508 (0.062427) | 0.029473 / 0.023109 (0.006364) | 0.336165 / 0.275898 (0.060266) | 0.420397 / 0.323480 (0.096917) | 0.006609 / 0.007986 (-0.001376) | 0.003338 / 0.004328 (-0.000991) | 0.078639 / 0.004250 (0.074388) | 0.034051 / 0.037052 (-0.003001) | 0.342820 / 0.258489 (0.084331) | 0.399392 / 0.293841 (0.105551) | 0.033935 / 0.128546 (-0.094611) | 0.011555 / 0.075646 (-0.064092) | 0.323467 / 0.419271 (-0.095804) | 0.040675 / 0.043533 (-0.002858) | 0.321247 / 0.255139 (0.066108) | 0.370967 / 0.283200 (0.087767) | 0.085766 / 0.141683 (-0.055917) | 1.461158 / 1.452155 (0.009003) | 1.504641 / 1.492716 (0.011925) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.180060 / 0.018006 (0.162053) | 0.403623 / 0.000490 (0.403134) | 0.002253 / 0.000200 (0.002053) | 0.000072 / 0.000054 (0.000017) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022793 / 0.037411 (-0.014618) | 0.098869 / 0.014526 (0.084343) | 0.104512 / 0.176557 (-0.072045) | 0.167721 / 0.737135 (-0.569414) | 0.107969 / 0.296338 (-0.188370) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.411179 / 0.215209 (0.195969) | 4.095345 / 2.077655 (2.017690) | 1.825992 / 1.504120 (0.321872) | 1.624386 / 1.541195 (0.083192) | 1.654903 / 1.468490 (0.186413) | 0.695041 / 4.584777 (-3.889736) | 3.319087 / 3.745712 (-0.426625) | 1.881945 / 5.269862 (-3.387917) | 1.250360 / 4.565676 (-3.315316) | 0.082405 / 0.424275 (-0.341870) | 0.012499 / 0.007607 (0.004892) | 0.522846 / 0.226044 (0.296801) | 5.241103 / 2.268929 (2.972175) | 2.293100 / 55.444624 (-53.151524) | 1.942937 / 6.876477 (-4.933540) | 1.957434 / 2.142072 (-0.184638) | 0.809782 / 4.805227 (-3.995445) | 0.148290 / 6.500664 (-6.352374) | 0.064157 / 0.075469 (-0.011312) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.185616 / 1.841788 (-0.656172) | 13.616791 / 8.074308 (5.542483) | 13.741806 / 10.191392 (3.550414) | 0.137396 / 0.680424 (-0.543028) | 0.028751 / 0.534201 (-0.505450) | 0.397636 / 0.579283 (-0.181647) | 0.403594 / 0.434364 (-0.030770) | 0.484039 / 0.540337 (-0.056299) | 0.568398 / 1.386936 (-0.818538) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006712 / 0.011353 (-0.004640) | 0.004511 / 0.011008 (-0.006497) | 0.076946 / 0.038508 (0.038438) | 0.027219 / 0.023109 (0.004110) | 0.350769 / 0.275898 (0.074871) | 0.408539 / 0.323480 (0.085059) | 0.005014 / 0.007986 (-0.002971) | 0.003361 / 0.004328 (-0.000968) | 0.077106 / 0.004250 (0.072856) | 0.040105 / 0.037052 (0.003053) | 0.342041 / 0.258489 (0.083552) | 0.426355 / 0.293841 (0.132514) | 0.031684 / 0.128546 (-0.096863) | 0.011575 / 0.075646 (-0.064072) | 0.085797 / 0.419271 (-0.333474) | 0.041575 / 0.043533 (-0.001958) | 0.340837 / 0.255139 (0.085698) | 0.390461 / 0.283200 (0.107262) | 0.089531 / 0.141683 (-0.052152) | 1.504600 / 1.452155 (0.052445) | 1.538712 / 1.492716 (0.045996) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.236679 / 0.018006 (0.218673) | 0.396258 / 0.000490 (0.395768) | 0.006479 / 0.000200 (0.006279) | 0.000081 / 0.000054 (0.000026) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024682 / 0.037411 (-0.012729) | 0.100167 / 0.014526 (0.085641) | 0.106627 / 0.176557 (-0.069929) | 0.174592 / 0.737135 (-0.562543) | 0.109499 / 0.296338 (-0.186839) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.444702 / 0.215209 (0.229493) | 4.462779 / 2.077655 (2.385125) | 2.087711 / 1.504120 (0.583591) | 1.874900 / 1.541195 (0.333705) | 1.918609 / 1.468490 (0.450119) | 0.705867 / 4.584777 (-3.878910) | 3.355483 / 3.745712 (-0.390229) | 2.808348 / 5.269862 (-2.461514) | 1.253319 / 4.565676 (-3.312358) | 0.083747 / 0.424275 (-0.340528) | 0.012491 / 0.007607 (0.004884) | 0.542885 / 0.226044 (0.316841) | 5.453921 / 2.268929 (3.184993) | 2.545688 / 55.444624 (-52.898937) | 2.185022 / 6.876477 (-4.691455) | 2.215351 / 2.142072 (0.073279) | 0.808201 / 4.805227 (-3.997027) | 0.151754 / 6.500664 (-6.348910) | 0.066886 / 0.075469 (-0.008583) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.298583 / 1.841788 (-0.543205) | 14.014276 / 8.074308 (5.939968) | 13.505338 / 10.191392 (3.313946) | 0.142033 / 0.680424 (-0.538391) | 0.016863 / 0.534201 (-0.517338) | 0.381195 / 0.579283 (-0.198088) | 0.384455 / 0.434364 (-0.049909) | 0.465765 / 0.540337 (-0.074572) | 0.552571 / 1.386936 (-0.834366) |

![](https://cml.dev/watermark.png#a29cca79ce64a5c64ad7047e57845b22154d7b8d "CML watermark")

Removed mangle_dup_cols=True from BuilderConfig.

It triggered following deprecation warning:
/usr/local/lib/python3.8/dist-packages/datasets/download/streaming_download_manager.py:776: FutureWarning: the 'mangle_dupe_cols' keyword is deprecated and will be removed in a future version. Please take steps to stop the use of 'mangle_dupe_cols'
  return pd.read_csv(xopen(filepath_or_buffer, "rb", use_auth_token=use_auth_token), **kwargs)

Further documentation of pandas: https://pandas.pydata.org/docs/whatsnew/v1.4.0.html#mangle-dupe-cols-in-read-csv-no-longer-renames-unique-columns-conflicting-with-target-names

At first sight it seems like this flag is resolved internally, it might need some more research.


Add pre-commit config yaml file to enable automatic code formatting

Better yet have someone enable pre-commit CI https://pre-commit.ci/ and it will apply the pre-commit fixes to the PR automatically as an additional commit.

@Skylion007 hi! I agree with @nateraw here, I'd better not force to use pre-commit so I'm not setting it up in the CI for now. And regarding end-of-file - currently it's being done by `black`.

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008704 / 0.011353 (-0.002649) | 0.004448 / 0.011008 (-0.006560) | 0.099530 / 0.038508 (0.061022) | 0.029739 / 0.023109 (0.006629) | 0.329267 / 0.275898 (0.053369) | 0.368805 / 0.323480 (0.045325) | 0.006852 / 0.007986 (-0.001133) | 0.004575 / 0.004328 (0.000246) | 0.076838 / 0.004250 (0.072588) | 0.033885 / 0.037052 (-0.003167) | 0.336340 / 0.258489 (0.077851) | 0.384880 / 0.293841 (0.091039) | 0.034051 / 0.128546 (-0.094495) | 0.011638 / 0.075646 (-0.064009) | 0.321650 / 0.419271 (-0.097622) | 0.041202 / 0.043533 (-0.002330) | 0.330841 / 0.255139 (0.075702) | 0.361329 / 0.283200 (0.078130) | 0.084864 / 0.141683 (-0.056819) | 1.454005 / 1.452155 (0.001850) | 1.542167 / 1.492716 (0.049451) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.196207 / 0.018006 (0.178200) | 0.400675 / 0.000490 (0.400185) | 0.000403 / 0.000200 (0.000203) | 0.000059 / 0.000054 (0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022694 / 0.037411 (-0.014717) | 0.095139 / 0.014526 (0.080613) | 0.104129 / 0.176557 (-0.072427) | 0.168688 / 0.737135 (-0.568447) | 0.109243 / 0.296338 (-0.187096) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.427520 / 0.215209 (0.212311) | 4.237726 / 2.077655 (2.160071) | 2.191887 / 1.504120 (0.687767) | 1.987750 / 1.541195 (0.446555) | 1.996540 / 1.468490 (0.528050) | 0.696416 / 4.584777 (-3.888361) | 3.454536 / 3.745712 (-0.291176) | 2.023600 / 5.269862 (-3.246261) | 1.336394 / 4.565676 (-3.229282) | 0.082933 / 0.424275 (-0.341342) | 0.012572 / 0.007607 (0.004965) | 0.534330 / 0.226044 (0.308285) | 5.347588 / 2.268929 (3.078659) | 2.640397 / 55.444624 (-52.804228) | 2.338266 / 6.876477 (-4.538211) | 2.431969 / 2.142072 (0.289897) | 0.821335 / 4.805227 (-3.983893) | 0.151905 / 6.500664 (-6.348759) | 0.067983 / 0.075469 (-0.007486) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.228841 / 1.841788 (-0.612947) | 13.660437 / 8.074308 (5.586128) | 13.729442 / 10.191392 (3.538050) | 0.165835 / 0.680424 (-0.514589) | 0.028753 / 0.534201 (-0.505448) | 0.400143 / 0.579283 (-0.179140) | 0.403714 / 0.434364 (-0.030650) | 0.492168 / 0.540337 (-0.048170) | 0.581151 / 1.386936 (-0.805785) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006289 / 0.011353 (-0.005064) | 0.004419 / 0.011008 (-0.006589) | 0.077220 / 0.038508 (0.038712) | 0.027170 / 0.023109 (0.004060) | 0.344988 / 0.275898 (0.069090) | 0.374150 / 0.323480 (0.050670) | 0.004842 / 0.007986 (-0.003144) | 0.003289 / 0.004328 (-0.001039) | 0.076200 / 0.004250 (0.071950) | 0.036287 / 0.037052 (-0.000766) | 0.345764 / 0.258489 (0.087275) | 0.387439 / 0.293841 (0.093599) | 0.031547 / 0.128546 (-0.096999) | 0.011586 / 0.075646 (-0.064060) | 0.086599 / 0.419271 (-0.332672) | 0.042338 / 0.043533 (-0.001195) | 0.355384 / 0.255139 (0.100246) | 0.369474 / 0.283200 (0.086275) | 0.090945 / 0.141683 (-0.050738) | 1.488632 / 1.452155 (0.036477) | 1.554606 / 1.492716 (0.061890) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.212962 / 0.018006 (0.194956) | 0.399647 / 0.000490 (0.399157) | 0.003055 / 0.000200 (0.002856) | 0.000083 / 0.000054 (0.000029) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024349 / 0.037411 (-0.013062) | 0.100342 / 0.014526 (0.085817) | 0.105657 / 0.176557 (-0.070899) | 0.175139 / 0.737135 (-0.561997) | 0.110014 / 0.296338 (-0.186324) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.434785 / 0.215209 (0.219575) | 4.346950 / 2.077655 (2.269295) | 2.045411 / 1.504120 (0.541291) | 1.844258 / 1.541195 (0.303064) | 1.889503 / 1.468490 (0.421013) | 0.704530 / 4.584777 (-3.880247) | 3.362435 / 3.745712 (-0.383277) | 2.797205 / 5.269862 (-2.472656) | 1.504431 / 4.565676 (-3.061245) | 0.083331 / 0.424275 (-0.340945) | 0.012274 / 0.007607 (0.004666) | 0.531123 / 0.226044 (0.305078) | 5.322588 / 2.268929 (3.053660) | 2.483875 / 55.444624 (-52.960750) | 2.147218 / 6.876477 (-4.729258) | 2.164024 / 2.142072 (0.021952) | 0.807191 / 4.805227 (-3.998036) | 0.151189 / 6.500664 (-6.349475) | 0.068027 / 0.075469 (-0.007442) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.316001 / 1.841788 (-0.525787) | 13.892785 / 8.074308 (5.818477) | 13.485982 / 10.191392 (3.294590) | 0.138904 / 0.680424 (-0.541520) | 0.016748 / 0.534201 (-0.517453) | 0.379840 / 0.579283 (-0.199443) | 0.384854 / 0.434364 (-0.049510) | 0.464275 / 0.540337 (-0.076063) | 0.553622 / 1.386936 (-0.833314) |

![](https://cml.dev/watermark.png#a940972a9a38543b2066129dc6e7987e08dca082 "CML watermark")

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009179 / 0.011353 (-0.002174) | 0.005080 / 0.011008 (-0.005929) | 0.099061 / 0.038508 (0.060553) | 0.035252 / 0.023109 (0.012143) | 0.293496 / 0.275898 (0.017598) | 0.360365 / 0.323480 (0.036886) | 0.007757 / 0.007986 (-0.000229) | 0.003985 / 0.004328 (-0.000343) | 0.076021 / 0.004250 (0.071771) | 0.042286 / 0.037052 (0.005233) | 0.316542 / 0.258489 (0.058053) | 0.341711 / 0.293841 (0.047870) | 0.037970 / 0.128546 (-0.090576) | 0.011977 / 0.075646 (-0.063670) | 0.333341 / 0.419271 (-0.085931) | 0.049211 / 0.043533 (0.005678) | 0.297401 / 0.255139 (0.042262) | 0.313424 / 0.283200 (0.030224) | 0.105719 / 0.141683 (-0.035964) | 1.487879 / 1.452155 (0.035724) | 1.529785 / 1.492716 (0.037068) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.201062 / 0.018006 (0.183056) | 0.438024 / 0.000490 (0.437534) | 0.002129 / 0.000200 (0.001929) | 0.000083 / 0.000054 (0.000028) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026422 / 0.037411 (-0.010989) | 0.104863 / 0.014526 (0.090337) | 0.114934 / 0.176557 (-0.061623) | 0.179173 / 0.737135 (-0.557962) | 0.119734 / 0.296338 (-0.176604) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.397195 / 0.215209 (0.181986) | 3.959945 / 2.077655 (1.882290) | 1.794059 / 1.504120 (0.289939) | 1.606814 / 1.541195 (0.065619) | 1.674681 / 1.468490 (0.206191) | 0.680130 / 4.584777 (-3.904646) | 3.742730 / 3.745712 (-0.002982) | 2.021793 / 5.269862 (-3.248069) | 1.322726 / 4.565676 (-3.242950) | 0.084519 / 0.424275 (-0.339756) | 0.012012 / 0.007607 (0.004405) | 0.510076 / 0.226044 (0.284032) | 5.084163 / 2.268929 (2.815234) | 2.241032 / 55.444624 (-53.203592) | 1.911936 / 6.876477 (-4.964540) | 1.947992 / 2.142072 (-0.194080) | 0.838779 / 4.805227 (-3.966448) | 0.165103 / 6.500664 (-6.335561) | 0.060722 / 0.075469 (-0.014747) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.180274 / 1.841788 (-0.661514) | 14.285364 / 8.074308 (6.211056) | 12.941205 / 10.191392 (2.749813) | 0.153815 / 0.680424 (-0.526609) | 0.028554 / 0.534201 (-0.505647) | 0.441551 / 0.579283 (-0.137732) | 0.434906 / 0.434364 (0.000542) | 0.516120 / 0.540337 (-0.024217) | 0.603062 / 1.386936 (-0.783874) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007287 / 0.011353 (-0.004066) | 0.004998 / 0.011008 (-0.006010) | 0.074997 / 0.038508 (0.036489) | 0.033209 / 0.023109 (0.010100) | 0.336836 / 0.275898 (0.060938) | 0.365562 / 0.323480 (0.042082) | 0.005739 / 0.007986 (-0.002246) | 0.003942 / 0.004328 (-0.000387) | 0.074681 / 0.004250 (0.070430) | 0.049530 / 0.037052 (0.012478) | 0.335642 / 0.258489 (0.077153) | 0.388874 / 0.293841 (0.095033) | 0.037198 / 0.128546 (-0.091349) | 0.011983 / 0.075646 (-0.063664) | 0.087601 / 0.419271 (-0.331671) | 0.053761 / 0.043533 (0.010228) | 0.334142 / 0.255139 (0.079003) | 0.351348 / 0.283200 (0.068148) | 0.107462 / 0.141683 (-0.034221) | 1.497015 / 1.452155 (0.044860) | 1.608287 / 1.492716 (0.115571) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.255395 / 0.018006 (0.237389) | 0.439141 / 0.000490 (0.438651) | 0.021391 / 0.000200 (0.021191) | 0.000230 / 0.000054 (0.000176) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028331 / 0.037411 (-0.009080) | 0.108744 / 0.014526 (0.094218) | 0.118201 / 0.176557 (-0.058355) | 0.189556 / 0.737135 (-0.547579) | 0.123112 / 0.296338 (-0.173226) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.431394 / 0.215209 (0.216185) | 4.296121 / 2.077655 (2.218466) | 2.126371 / 1.504120 (0.622251) | 1.978178 / 1.541195 (0.436983) | 2.082674 / 1.468490 (0.614184) | 0.701789 / 4.584777 (-3.882988) | 3.791495 / 3.745712 (0.045783) | 2.115267 / 5.269862 (-3.154594) | 1.342159 / 4.565676 (-3.223517) | 0.088132 / 0.424275 (-0.336143) | 0.011903 / 0.007607 (0.004295) | 0.528398 / 0.226044 (0.302354) | 5.270077 / 2.268929 (3.001148) | 2.498860 / 55.444624 (-52.945765) | 2.155515 / 6.876477 (-4.720962) | 2.192866 / 2.142072 (0.050793) | 0.859596 / 4.805227 (-3.945631) | 0.170544 / 6.500664 (-6.330120) | 0.063883 / 0.075469 (-0.011587) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.240679 / 1.841788 (-0.601109) | 14.497379 / 8.074308 (6.423071) | 12.881417 / 10.191392 (2.690025) | 0.147295 / 0.680424 (-0.533129) | 0.017465 / 0.534201 (-0.516736) | 0.424695 / 0.579283 (-0.154588) | 0.414929 / 0.434364 (-0.019435) | 0.536079 / 0.540337 (-0.004259) | 0.638245 / 1.386936 (-0.748691) |

![](https://cml.dev/watermark.png#a940972a9a38543b2066129dc6e7987e08dca082 "CML watermark")

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008806 / 0.011353 (-0.002547) | 0.004712 / 0.011008 (-0.006297) | 0.102383 / 0.038508 (0.063875) | 0.030260 / 0.023109 (0.007151) | 0.330175 / 0.275898 (0.054277) | 0.376816 / 0.323480 (0.053337) | 0.008065 / 0.007986 (0.000079) | 0.003534 / 0.004328 (-0.000794) | 0.078824 / 0.004250 (0.074573) | 0.036704 / 0.037052 (-0.000349) | 0.331848 / 0.258489 (0.073359) | 0.351031 / 0.293841 (0.057190) | 0.033406 / 0.128546 (-0.095140) | 0.011543 / 0.075646 (-0.064103) | 0.322114 / 0.419271 (-0.097157) | 0.041249 / 0.043533 (-0.002284) | 0.309413 / 0.255139 (0.054274) | 0.329156 / 0.283200 (0.045956) | 0.088636 / 0.141683 (-0.053047) | 1.508226 / 1.452155 (0.056071) | 1.557203 / 1.492716 (0.064487) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.196696 / 0.018006 (0.178690) | 0.426360 / 0.000490 (0.425870) | 0.001263 / 0.000200 (0.001064) | 0.000079 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023747 / 0.037411 (-0.013664) | 0.100756 / 0.014526 (0.086230) | 0.105817 / 0.176557 (-0.070739) | 0.172573 / 0.737135 (-0.564562) | 0.110705 / 0.296338 (-0.185634) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.436913 / 0.215209 (0.221704) | 4.365753 / 2.077655 (2.288099) | 2.201346 / 1.504120 (0.697226) | 1.978800 / 1.541195 (0.437605) | 1.951585 / 1.468490 (0.483094) | 0.699208 / 4.584777 (-3.885569) | 3.381492 / 3.745712 (-0.364220) | 2.966174 / 5.269862 (-2.303687) | 1.487521 / 4.565676 (-3.078156) | 0.082673 / 0.424275 (-0.341602) | 0.012436 / 0.007607 (0.004829) | 0.553276 / 0.226044 (0.327232) | 5.554081 / 2.268929 (3.285153) | 2.653286 / 55.444624 (-52.791339) | 2.404788 / 6.876477 (-4.471689) | 2.484610 / 2.142072 (0.342537) | 0.817073 / 4.805227 (-3.988154) | 0.151619 / 6.500664 (-6.349045) | 0.068259 / 0.075469 (-0.007210) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.273481 / 1.841788 (-0.568306) | 13.908825 / 8.074308 (5.834517) | 13.106695 / 10.191392 (2.915303) | 0.139609 / 0.680424 (-0.540815) | 0.028425 / 0.534201 (-0.505776) | 0.395626 / 0.579283 (-0.183657) | 0.405526 / 0.434364 (-0.028838) | 0.465628 / 0.540337 (-0.074709) | 0.542824 / 1.386936 (-0.844112) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006821 / 0.011353 (-0.004532) | 0.004570 / 0.011008 (-0.006438) | 0.076568 / 0.038508 (0.038060) | 0.028109 / 0.023109 (0.004999) | 0.342768 / 0.275898 (0.066870) | 0.390680 / 0.323480 (0.067200) | 0.005056 / 0.007986 (-0.002930) | 0.003359 / 0.004328 (-0.000970) | 0.075835 / 0.004250 (0.071584) | 0.038888 / 0.037052 (0.001836) | 0.343489 / 0.258489 (0.085000) | 0.400766 / 0.293841 (0.106925) | 0.031816 / 0.128546 (-0.096730) | 0.011637 / 0.075646 (-0.064009) | 0.085474 / 0.419271 (-0.333797) | 0.041740 / 0.043533 (-0.001793) | 0.342501 / 0.255139 (0.087362) | 0.377467 / 0.283200 (0.094267) | 0.091532 / 0.141683 (-0.050151) | 1.457368 / 1.452155 (0.005213) | 1.537187 / 1.492716 (0.044471) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.187507 / 0.018006 (0.169501) | 0.415706 / 0.000490 (0.415217) | 0.001816 / 0.000200 (0.001616) | 0.000072 / 0.000054 (0.000018) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026251 / 0.037411 (-0.011161) | 0.106609 / 0.014526 (0.092083) | 0.109822 / 0.176557 (-0.066735) | 0.180462 / 0.737135 (-0.556674) | 0.114647 / 0.296338 (-0.181691) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.438804 / 0.215209 (0.223595) | 4.387960 / 2.077655 (2.310306) | 2.056804 / 1.504120 (0.552684) | 1.848584 / 1.541195 (0.307389) | 1.939470 / 1.468490 (0.470980) | 0.702539 / 4.584777 (-3.882238) | 3.419535 / 3.745712 (-0.326177) | 1.933889 / 5.269862 (-3.335973) | 1.189631 / 4.565676 (-3.376045) | 0.084105 / 0.424275 (-0.340170) | 0.012520 / 0.007607 (0.004913) | 0.538125 / 0.226044 (0.312081) | 5.370000 / 2.268929 (3.101072) | 2.497487 / 55.444624 (-52.947137) | 2.156054 / 6.876477 (-4.720423) | 2.225909 / 2.142072 (0.083837) | 0.811456 / 4.805227 (-3.993771) | 0.151461 / 6.500664 (-6.349203) | 0.066940 / 0.075469 (-0.008530) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.301246 / 1.841788 (-0.540542) | 14.459755 / 8.074308 (6.385447) | 13.147151 / 10.191392 (2.955759) | 0.129236 / 0.680424 (-0.551188) | 0.016427 / 0.534201 (-0.517774) | 0.380047 / 0.579283 (-0.199236) | 0.392217 / 0.434364 (-0.042147) | 0.470338 / 0.540337 (-0.069999) | 0.559800 / 1.386936 (-0.827136) |

![](https://cml.dev/watermark.png#a940972a9a38543b2066129dc6e7987e08dca082 "CML watermark")

@huggingface/datasets do you think it would be useful? Motivation - sometimes PRs are like 30% "fix: style" commits :)

If so - I need to double check the config but for me locally it works as expected.