Drop Python 3.7 support

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006152 / 0.011353 (-0.005200) | 0.003916 / 0.011008 (-0.007092) | 0.097355 / 0.038508 (0.058847) | 0.037228 / 0.023109 (0.014119) | 0.315753 / 0.275898 (0.039855) | 0.387949 / 0.323480 (0.064470) | 0.004804 / 0.007986 (-0.003181) | 0.002975 / 0.004328 (-0.001353) | 0.076932 / 0.004250 (0.072682) | 0.053497 / 0.037052 (0.016445) | 0.331143 / 0.258489 (0.072654) | 0.388347 / 0.293841 (0.094506) | 0.027535 / 0.128546 (-0.101011) | 0.008509 / 0.075646 (-0.067137) | 0.312639 / 0.419271 (-0.106632) | 0.047212 / 0.043533 (0.003679) | 0.316875 / 0.255139 (0.061736) | 0.352191 / 0.283200 (0.068992) | 0.021380 / 0.141683 (-0.120303) | 1.541401 / 1.452155 (0.089247) | 1.519420 / 1.492716 (0.026704) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.206332 / 0.018006 (0.188326) | 0.412252 / 0.000490 (0.411762) | 0.005119 / 0.000200 (0.004919) | 0.000077 / 0.000054 (0.000022) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023856 / 0.037411 (-0.013556) | 0.098216 / 0.014526 (0.083691) | 0.106553 / 0.176557 (-0.070003) | 0.168767 / 0.737135 (-0.568369) | 0.109244 / 0.296338 (-0.187094) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.457580 / 0.215209 (0.242371) | 4.583246 / 2.077655 (2.505591) | 2.296356 / 1.504120 (0.792236) | 2.096216 / 1.541195 (0.555021) | 2.159086 / 1.468490 (0.690596) | 0.557905 / 4.584777 (-4.026872) | 3.345910 / 3.745712 (-0.399802) | 1.767436 / 5.269862 (-3.502426) | 1.021583 / 4.565676 (-3.544094) | 0.067265 / 0.424275 (-0.357011) | 0.011411 / 0.007607 (0.003804) | 0.559841 / 0.226044 (0.333797) | 5.586892 / 2.268929 (3.317963) | 2.735520 / 55.444624 (-52.709104) | 2.429393 / 6.876477 (-4.447084) | 2.544901 / 2.142072 (0.402829) | 0.667603 / 4.805227 (-4.137625) | 0.136244 / 6.500664 (-6.364421) | 0.066961 / 0.075469 (-0.008508) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.206529 / 1.841788 (-0.635259) | 13.988306 / 8.074308 (5.913998) | 13.481813 / 10.191392 (3.290421) | 0.161901 / 0.680424 (-0.518523) | 0.016850 / 0.534201 (-0.517351) | 0.367657 / 0.579283 (-0.211626) | 0.393343 / 0.434364 (-0.041021) | 0.465288 / 0.540337 (-0.075050) | 0.559888 / 1.386936 (-0.827048) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005956 / 0.011353 (-0.005397) | 0.003734 / 0.011008 (-0.007274) | 0.077841 / 0.038508 (0.039333) | 0.036532 / 0.023109 (0.013422) | 0.438923 / 0.275898 (0.163025) | 0.490133 / 0.323480 (0.166653) | 0.004651 / 0.007986 (-0.003335) | 0.002881 / 0.004328 (-0.001448) | 0.077868 / 0.004250 (0.073618) | 0.051700 / 0.037052 (0.014647) | 0.448018 / 0.258489 (0.189529) | 0.500304 / 0.293841 (0.206464) | 0.029051 / 0.128546 (-0.099496) | 0.008498 / 0.075646 (-0.067148) | 0.082932 / 0.419271 (-0.336339) | 0.043665 / 0.043533 (0.000132) | 0.431613 / 0.255139 (0.176474) | 0.458749 / 0.283200 (0.175549) | 0.021951 / 0.141683 (-0.119731) | 1.556043 / 1.452155 (0.103888) | 1.588391 / 1.492716 (0.095675) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.220674 / 0.018006 (0.202667) | 0.415408 / 0.000490 (0.414918) | 0.002613 / 0.000200 (0.002413) | 0.000075 / 0.000054 (0.000020) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025548 / 0.037411 (-0.011863) | 0.103633 / 0.014526 (0.089107) | 0.115193 / 0.176557 (-0.061364) | 0.163971 / 0.737135 (-0.573164) | 0.114754 / 0.296338 (-0.181585) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.456823 / 0.215209 (0.241614) | 4.569950 / 2.077655 (2.492296) | 2.196339 / 1.504120 (0.692219) | 1.985822 / 1.541195 (0.444628) | 2.044083 / 1.468490 (0.575593) | 0.567919 / 4.584777 (-4.016858) | 3.397515 / 3.745712 (-0.348197) | 1.741087 / 5.269862 (-3.528775) | 1.041237 / 4.565676 (-3.524440) | 0.068963 / 0.424275 (-0.355313) | 0.011677 / 0.007607 (0.004070) | 0.565010 / 0.226044 (0.338966) | 5.625886 / 2.268929 (3.356957) | 2.670658 / 55.444624 (-52.773967) | 2.300279 / 6.876477 (-4.576198) | 2.392178 / 2.142072 (0.250106) | 0.680226 / 4.805227 (-4.125001) | 0.139119 / 6.500664 (-6.361545) | 0.067953 / 0.075469 (-0.007516) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.303280 / 1.841788 (-0.538507) | 14.458686 / 8.074308 (6.384378) | 14.409369 / 10.191392 (4.217977) | 0.144581 / 0.680424 (-0.535843) | 0.016634 / 0.534201 (-0.517567) | 0.364607 / 0.579283 (-0.214676) | 0.394521 / 0.434364 (-0.039843) | 0.433417 / 0.540337 (-0.106921) | 0.527127 / 1.386936 (-0.859809) |

![](https://cml.dev/watermark.png#04a36f9546484dceadb84a133c1a460281d018f8 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006245 / 0.011353 (-0.005108) | 0.003871 / 0.011008 (-0.007138) | 0.098823 / 0.038508 (0.060315) | 0.039853 / 0.023109 (0.016744) | 0.314989 / 0.275898 (0.039091) | 0.376733 / 0.323480 (0.053254) | 0.004754 / 0.007986 (-0.003232) | 0.002971 / 0.004328 (-0.001357) | 0.078451 / 0.004250 (0.074201) | 0.053160 / 0.037052 (0.016107) | 0.324443 / 0.258489 (0.065954) | 0.361488 / 0.293841 (0.067647) | 0.027942 / 0.128546 (-0.100604) | 0.008535 / 0.075646 (-0.067111) | 0.315526 / 0.419271 (-0.103745) | 0.045706 / 0.043533 (0.002174) | 0.329614 / 0.255139 (0.074475) | 0.336339 / 0.283200 (0.053139) | 0.021278 / 0.141683 (-0.120405) | 1.529710 / 1.452155 (0.077555) | 1.566833 / 1.492716 (0.074116) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.215263 / 0.018006 (0.197257) | 0.440320 / 0.000490 (0.439830) | 0.002627 / 0.000200 (0.002427) | 0.000075 / 0.000054 (0.000021) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023971 / 0.037411 (-0.013441) | 0.100549 / 0.014526 (0.086023) | 0.106995 / 0.176557 (-0.069561) | 0.169630 / 0.737135 (-0.567505) | 0.111614 / 0.296338 (-0.184724) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.424911 / 0.215209 (0.209702) | 4.246920 / 2.077655 (2.169266) | 1.923321 / 1.504120 (0.419202) | 1.714795 / 1.541195 (0.173600) | 1.772906 / 1.468490 (0.304416) | 0.554676 / 4.584777 (-4.030101) | 3.478896 / 3.745712 (-0.266816) | 2.800494 / 5.269862 (-2.469368) | 1.382630 / 4.565676 (-3.183047) | 0.067271 / 0.424275 (-0.357004) | 0.010967 / 0.007607 (0.003360) | 0.526769 / 0.226044 (0.300725) | 5.288564 / 2.268929 (3.019636) | 2.337459 / 55.444624 (-53.107165) | 1.999975 / 6.876477 (-4.876502) | 2.102680 / 2.142072 (-0.039392) | 0.672181 / 4.805227 (-4.133046) | 0.135097 / 6.500664 (-6.365567) | 0.066950 / 0.075469 (-0.008519) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.264365 / 1.841788 (-0.577423) | 14.282440 / 8.074308 (6.208132) | 14.220200 / 10.191392 (4.028808) | 0.139055 / 0.680424 (-0.541369) | 0.016681 / 0.534201 (-0.517520) | 0.367936 / 0.579283 (-0.211348) | 0.393959 / 0.434364 (-0.040404) | 0.424438 / 0.540337 (-0.115900) | 0.508065 / 1.386936 (-0.878872) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006514 / 0.011353 (-0.004839) | 0.003890 / 0.011008 (-0.007118) | 0.078871 / 0.038508 (0.040363) | 0.038080 / 0.023109 (0.014971) | 0.358282 / 0.275898 (0.082384) | 0.430654 / 0.323480 (0.107174) | 0.005712 / 0.007986 (-0.002273) | 0.003030 / 0.004328 (-0.001299) | 0.078636 / 0.004250 (0.074386) | 0.057771 / 0.037052 (0.020719) | 0.368814 / 0.258489 (0.110325) | 0.437047 / 0.293841 (0.143206) | 0.029470 / 0.128546 (-0.099076) | 0.008523 / 0.075646 (-0.067124) | 0.083334 / 0.419271 (-0.335938) | 0.044505 / 0.043533 (0.000972) | 0.357484 / 0.255139 (0.102345) | 0.393839 / 0.283200 (0.110639) | 0.023340 / 0.141683 (-0.118343) | 1.561033 / 1.452155 (0.108878) | 1.595560 / 1.492716 (0.102844) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.204149 / 0.018006 (0.186143) | 0.442747 / 0.000490 (0.442257) | 0.003105 / 0.000200 (0.002905) | 0.000085 / 0.000054 (0.000030) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027002 / 0.037411 (-0.010409) | 0.105595 / 0.014526 (0.091070) | 0.108695 / 0.176557 (-0.067861) | 0.163182 / 0.737135 (-0.573953) | 0.114999 / 0.296338 (-0.181339) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.483713 / 0.215209 (0.268504) | 4.836063 / 2.077655 (2.758409) | 2.488072 / 1.504120 (0.983952) | 2.289556 / 1.541195 (0.748361) | 2.342912 / 1.468490 (0.874422) | 0.565937 / 4.584777 (-4.018840) | 3.479085 / 3.745712 (-0.266627) | 1.770922 / 5.269862 (-3.498940) | 1.046084 / 4.565676 (-3.519592) | 0.067857 / 0.424275 (-0.356418) | 0.011283 / 0.007607 (0.003676) | 0.592966 / 0.226044 (0.366921) | 5.932842 / 2.268929 (3.663914) | 2.956252 / 55.444624 (-52.488372) | 2.602704 / 6.876477 (-4.273772) | 2.715625 / 2.142072 (0.573552) | 0.674299 / 4.805227 (-4.130929) | 0.136039 / 6.500664 (-6.364625) | 0.067629 / 0.075469 (-0.007840) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.333734 / 1.841788 (-0.508054) | 14.561943 / 8.074308 (6.487634) | 14.455385 / 10.191392 (4.263993) | 0.132020 / 0.680424 (-0.548404) | 0.016893 / 0.534201 (-0.517308) | 0.367146 / 0.579283 (-0.212137) | 0.399623 / 0.434364 (-0.034741) | 0.432658 / 0.540337 (-0.107680) | 0.530475 / 1.386936 (-0.856461) |

![](https://cml.dev/watermark.png#18da5adb22b2b403b8d8ae673192746d2ed7e9f9 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006045 / 0.011353 (-0.005308) | 0.003906 / 0.011008 (-0.007103) | 0.097558 / 0.038508 (0.059050) | 0.038827 / 0.023109 (0.015718) | 0.393564 / 0.275898 (0.117666) | 0.442459 / 0.323480 (0.118980) | 0.004792 / 0.007986 (-0.003194) | 0.002984 / 0.004328 (-0.001345) | 0.076419 / 0.004250 (0.072169) | 0.053606 / 0.037052 (0.016554) | 0.409743 / 0.258489 (0.151254) | 0.445753 / 0.293841 (0.151912) | 0.027753 / 0.128546 (-0.100793) | 0.008428 / 0.075646 (-0.067219) | 0.310267 / 0.419271 (-0.109004) | 0.057582 / 0.043533 (0.014049) | 0.396624 / 0.255139 (0.141485) | 0.416288 / 0.283200 (0.133089) | 0.029048 / 0.141683 (-0.112635) | 1.495362 / 1.452155 (0.043207) | 1.546331 / 1.492716 (0.053615) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.203832 / 0.018006 (0.185826) | 0.423649 / 0.000490 (0.423160) | 0.004533 / 0.000200 (0.004333) | 0.000076 / 0.000054 (0.000022) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023084 / 0.037411 (-0.014328) | 0.100503 / 0.014526 (0.085977) | 0.105058 / 0.176557 (-0.071499) | 0.168506 / 0.737135 (-0.568629) | 0.112019 / 0.296338 (-0.184320) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.425877 / 0.215209 (0.210668) | 4.251278 / 2.077655 (2.173624) | 1.931339 / 1.504120 (0.427219) | 1.730578 / 1.541195 (0.189383) | 1.750637 / 1.468490 (0.282147) | 0.559307 / 4.584777 (-4.025470) | 3.461665 / 3.745712 (-0.284047) | 2.826959 / 5.269862 (-2.442903) | 1.418448 / 4.565676 (-3.147229) | 0.067881 / 0.424275 (-0.356394) | 0.011394 / 0.007607 (0.003787) | 0.533226 / 0.226044 (0.307181) | 5.341849 / 2.268929 (3.072921) | 2.367832 / 55.444624 (-53.076792) | 2.027240 / 6.876477 (-4.849236) | 2.095852 / 2.142072 (-0.046220) | 0.673790 / 4.805227 (-4.131437) | 0.136044 / 6.500664 (-6.364620) | 0.066350 / 0.075469 (-0.009119) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.203740 / 1.841788 (-0.638048) | 13.720879 / 8.074308 (5.646571) | 13.405939 / 10.191392 (3.214547) | 0.146792 / 0.680424 (-0.533632) | 0.016844 / 0.534201 (-0.517357) | 0.373455 / 0.579283 (-0.205828) | 0.394596 / 0.434364 (-0.039768) | 0.464715 / 0.540337 (-0.075623) | 0.558931 / 1.386936 (-0.828005) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006118 / 0.011353 (-0.005235) | 0.003817 / 0.011008 (-0.007191) | 0.077494 / 0.038508 (0.038985) | 0.037507 / 0.023109 (0.014398) | 0.387030 / 0.275898 (0.111132) | 0.437352 / 0.323480 (0.113872) | 0.004810 / 0.007986 (-0.003176) | 0.002935 / 0.004328 (-0.001394) | 0.077143 / 0.004250 (0.072892) | 0.053986 / 0.037052 (0.016933) | 0.393164 / 0.258489 (0.134675) | 0.449603 / 0.293841 (0.155762) | 0.029303 / 0.128546 (-0.099244) | 0.008481 / 0.075646 (-0.067165) | 0.083363 / 0.419271 (-0.335908) | 0.043877 / 0.043533 (0.000344) | 0.378175 / 0.255139 (0.123036) | 0.403996 / 0.283200 (0.120797) | 0.021688 / 0.141683 (-0.119995) | 1.541606 / 1.452155 (0.089452) | 1.552996 / 1.492716 (0.060280) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.236759 / 0.018006 (0.218752) | 0.416221 / 0.000490 (0.415732) | 0.000862 / 0.000200 (0.000662) | 0.000070 / 0.000054 (0.000016) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025543 / 0.037411 (-0.011868) | 0.101731 / 0.014526 (0.087206) | 0.108482 / 0.176557 (-0.068075) | 0.160290 / 0.737135 (-0.576845) | 0.111392 / 0.296338 (-0.184946) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.457767 / 0.215209 (0.242558) | 4.565976 / 2.077655 (2.488321) | 2.245413 / 1.504120 (0.741294) | 2.031458 / 1.541195 (0.490264) | 2.073193 / 1.468490 (0.604702) | 0.560461 / 4.584777 (-4.024316) | 3.422536 / 3.745712 (-0.323176) | 2.977017 / 5.269862 (-2.292845) | 1.377021 / 4.565676 (-3.188655) | 0.068444 / 0.424275 (-0.355831) | 0.011036 / 0.007607 (0.003429) | 0.571501 / 0.226044 (0.345456) | 5.702652 / 2.268929 (3.433723) | 2.727132 / 55.444624 (-52.717492) | 2.399269 / 6.876477 (-4.477208) | 2.574281 / 2.142072 (0.432208) | 0.682600 / 4.805227 (-4.122627) | 0.136943 / 6.500664 (-6.363722) | 0.067126 / 0.075469 (-0.008343) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.322196 / 1.841788 (-0.519592) | 14.239509 / 8.074308 (6.165201) | 14.235779 / 10.191392 (4.044387) | 0.148262 / 0.680424 (-0.532162) | 0.016566 / 0.534201 (-0.517635) | 0.364034 / 0.579283 (-0.215249) | 0.399157 / 0.434364 (-0.035207) | 0.426348 / 0.540337 (-0.113990) | 0.520804 / 1.386936 (-0.866132) |

![](https://cml.dev/watermark.png#8f57aae06bd325d76cb70cb774450f3a66f169cf "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007808 / 0.011353 (-0.003545) | 0.004706 / 0.011008 (-0.006303) | 0.100530 / 0.038508 (0.062022) | 0.052052 / 0.023109 (0.028943) | 0.419300 / 0.275898 (0.143402) | 0.488451 / 0.323480 (0.164971) | 0.006350 / 0.007986 (-0.001636) | 0.003875 / 0.004328 (-0.000453) | 0.076489 / 0.004250 (0.072238) | 0.077554 / 0.037052 (0.040502) | 0.435863 / 0.258489 (0.177373) | 0.483241 / 0.293841 (0.189400) | 0.037518 / 0.128546 (-0.091028) | 0.009857 / 0.075646 (-0.065789) | 0.340933 / 0.419271 (-0.078339) | 0.087046 / 0.043533 (0.043514) | 0.410721 / 0.255139 (0.155582) | 0.428995 / 0.283200 (0.145795) | 0.041701 / 0.141683 (-0.099982) | 1.821017 / 1.452155 (0.368862) | 1.837021 / 1.492716 (0.344305) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.228444 / 0.018006 (0.210438) | 0.480446 / 0.000490 (0.479956) | 0.004963 / 0.000200 (0.004763) | 0.000101 / 0.000054 (0.000046) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032485 / 0.037411 (-0.004926) | 0.096500 / 0.014526 (0.081974) | 0.111547 / 0.176557 (-0.065010) | 0.178842 / 0.737135 (-0.558294) | 0.111099 / 0.296338 (-0.185240) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.467159 / 0.215209 (0.251950) | 4.701676 / 2.077655 (2.624021) | 2.390560 / 1.504120 (0.886440) | 2.197722 / 1.541195 (0.656528) | 2.264705 / 1.468490 (0.796215) | 0.568667 / 4.584777 (-4.016110) | 4.200724 / 3.745712 (0.455012) | 3.777625 / 5.269862 (-1.492236) | 2.372451 / 4.565676 (-2.193225) | 0.067562 / 0.424275 (-0.356714) | 0.008947 / 0.007607 (0.001340) | 0.556910 / 0.226044 (0.330865) | 5.528927 / 2.268929 (3.259998) | 2.902780 / 55.444624 (-52.541844) | 2.507933 / 6.876477 (-4.368544) | 2.734627 / 2.142072 (0.592554) | 0.683305 / 4.805227 (-4.121922) | 0.158288 / 6.500664 (-6.342376) | 0.071252 / 0.075469 (-0.004217) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.487502 / 1.841788 (-0.354286) | 22.193341 / 8.074308 (14.119033) | 15.922607 / 10.191392 (5.731215) | 0.172189 / 0.680424 (-0.508235) | 0.021502 / 0.534201 (-0.512699) | 0.471198 / 0.579283 (-0.108085) | 0.475979 / 0.434364 (0.041615) | 0.544675 / 0.540337 (0.004338) | 0.756102 / 1.386936 (-0.630834) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007635 / 0.011353 (-0.003717) | 0.004614 / 0.011008 (-0.006394) | 0.075852 / 0.038508 (0.037344) | 0.049700 / 0.023109 (0.026591) | 0.425957 / 0.275898 (0.150059) | 0.512590 / 0.323480 (0.189110) | 0.006921 / 0.007986 (-0.001065) | 0.003714 / 0.004328 (-0.000615) | 0.075536 / 0.004250 (0.071286) | 0.070206 / 0.037052 (0.033153) | 0.455706 / 0.258489 (0.197217) | 0.512231 / 0.293841 (0.218390) | 0.036685 / 0.128546 (-0.091861) | 0.009793 / 0.075646 (-0.065853) | 0.084208 / 0.419271 (-0.335064) | 0.065262 / 0.043533 (0.021729) | 0.423761 / 0.255139 (0.168622) | 0.456791 / 0.283200 (0.173591) | 0.044539 / 0.141683 (-0.097144) | 1.797029 / 1.452155 (0.344874) | 1.864124 / 1.492716 (0.371408) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.366840 / 0.018006 (0.348834) | 0.479254 / 0.000490 (0.478765) | 0.070383 / 0.000200 (0.070183) | 0.000762 / 0.000054 (0.000707) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034233 / 0.037411 (-0.003178) | 0.103140 / 0.014526 (0.088614) | 0.117099 / 0.176557 (-0.059457) | 0.178532 / 0.737135 (-0.558603) | 0.120092 / 0.296338 (-0.176247) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.492993 / 0.215209 (0.277784) | 4.878776 / 2.077655 (2.801121) | 2.566666 / 1.504120 (1.062547) | 2.356383 / 1.541195 (0.815188) | 2.454723 / 1.468490 (0.986233) | 0.571432 / 4.584777 (-4.013345) | 4.240554 / 3.745712 (0.494842) | 7.509259 / 5.269862 (2.239398) | 4.040294 / 4.565676 (-0.525382) | 0.067409 / 0.424275 (-0.356866) | 0.008657 / 0.007607 (0.001050) | 0.585751 / 0.226044 (0.359707) | 5.967668 / 2.268929 (3.698739) | 3.195573 / 55.444624 (-52.249052) | 2.839772 / 6.876477 (-4.036704) | 2.806319 / 2.142072 (0.664246) | 0.681502 / 4.805227 (-4.123725) | 0.158673 / 6.500664 (-6.341991) | 0.073224 / 0.075469 (-0.002245) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.623335 / 1.841788 (-0.218453) | 22.490806 / 8.074308 (14.416498) | 16.762435 / 10.191392 (6.571043) | 0.180961 / 0.680424 (-0.499463) | 0.022716 / 0.534201 (-0.511485) | 0.472910 / 0.579283 (-0.106373) | 0.471616 / 0.434364 (0.037252) | 0.548192 / 0.540337 (0.007854) | 0.734357 / 1.386936 (-0.652579) |

![](https://cml.dev/watermark.png#c0498b47a00153d4730352b6595fc51ab054fb95 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005858 / 0.011353 (-0.005495) | 0.003512 / 0.011008 (-0.007497) | 0.079739 / 0.038508 (0.041231) | 0.057736 / 0.023109 (0.034627) | 0.317640 / 0.275898 (0.041742) | 0.354157 / 0.323480 (0.030677) | 0.004772 / 0.007986 (-0.003214) | 0.002824 / 0.004328 (-0.001504) | 0.063288 / 0.004250 (0.059037) | 0.049542 / 0.037052 (0.012489) | 0.323974 / 0.258489 (0.065485) | 0.372149 / 0.293841 (0.078308) | 0.026841 / 0.128546 (-0.101705) | 0.007846 / 0.075646 (-0.067800) | 0.262546 / 0.419271 (-0.156725) | 0.051952 / 0.043533 (0.008420) | 0.319439 / 0.255139 (0.064300) | 0.343862 / 0.283200 (0.060663) | 0.027021 / 0.141683 (-0.114662) | 1.445211 / 1.452155 (-0.006944) | 1.485006 / 1.492716 (-0.007711) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.183174 / 0.018006 (0.165167) | 0.422794 / 0.000490 (0.422304) | 0.004148 / 0.000200 (0.003948) | 0.000067 / 0.000054 (0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023037 / 0.037411 (-0.014374) | 0.071300 / 0.014526 (0.056775) | 0.083022 / 0.176557 (-0.093535) | 0.146215 / 0.737135 (-0.590920) | 0.082549 / 0.296338 (-0.213789) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.422846 / 0.215209 (0.207637) | 4.215280 / 2.077655 (2.137626) | 2.256802 / 1.504120 (0.752682) | 2.056867 / 1.541195 (0.515673) | 2.102478 / 1.468490 (0.633988) | 0.497552 / 4.584777 (-4.087225) | 3.049716 / 3.745712 (-0.695996) | 4.209227 / 5.269862 (-1.060635) | 2.599947 / 4.565676 (-1.965730) | 0.059131 / 0.424275 (-0.365144) | 0.006459 / 0.007607 (-0.001148) | 0.495047 / 0.226044 (0.269003) | 4.952332 / 2.268929 (2.683404) | 2.675260 / 55.444624 (-52.769365) | 2.333223 / 6.876477 (-4.543254) | 2.449573 / 2.142072 (0.307500) | 0.583420 / 4.805227 (-4.221807) | 0.125140 / 6.500664 (-6.375524) | 0.060209 / 0.075469 (-0.015260) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.215033 / 1.841788 (-0.626755) | 18.101107 / 8.074308 (10.026799) | 13.489222 / 10.191392 (3.297830) | 0.147122 / 0.680424 (-0.533302) | 0.016567 / 0.534201 (-0.517634) | 0.329909 / 0.579283 (-0.249374) | 0.340952 / 0.434364 (-0.093412) | 0.379166 / 0.540337 (-0.161172) | 0.510767 / 1.386936 (-0.876169) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005942 / 0.011353 (-0.005411) | 0.003628 / 0.011008 (-0.007380) | 0.061975 / 0.038508 (0.023467) | 0.058331 / 0.023109 (0.035221) | 0.393277 / 0.275898 (0.117379) | 0.410740 / 0.323480 (0.087261) | 0.004546 / 0.007986 (-0.003440) | 0.002826 / 0.004328 (-0.001503) | 0.062216 / 0.004250 (0.057966) | 0.049801 / 0.037052 (0.012748) | 0.394070 / 0.258489 (0.135581) | 0.414407 / 0.293841 (0.120566) | 0.027161 / 0.128546 (-0.101385) | 0.007901 / 0.075646 (-0.067746) | 0.066778 / 0.419271 (-0.352493) | 0.041354 / 0.043533 (-0.002179) | 0.379432 / 0.255139 (0.124293) | 0.402966 / 0.283200 (0.119766) | 0.020279 / 0.141683 (-0.121404) | 1.416986 / 1.452155 (-0.035169) | 1.474335 / 1.492716 (-0.018382) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.226147 / 0.018006 (0.208140) | 0.404361 / 0.000490 (0.403871) | 0.000358 / 0.000200 (0.000158) | 0.000054 / 0.000054 (-0.000000) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025105 / 0.037411 (-0.012306) | 0.075849 / 0.014526 (0.061323) | 0.084781 / 0.176557 (-0.091775) | 0.137415 / 0.737135 (-0.599720) | 0.086288 / 0.296338 (-0.210051) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.445925 / 0.215209 (0.230716) | 4.453478 / 2.077655 (2.375823) | 2.419048 / 1.504120 (0.914928) | 2.246363 / 1.541195 (0.705168) | 2.304022 / 1.468490 (0.835532) | 0.499132 / 4.584777 (-4.085645) | 3.001336 / 3.745712 (-0.744376) | 2.902593 / 5.269862 (-2.367269) | 1.819843 / 4.565676 (-2.745834) | 0.057210 / 0.424275 (-0.367065) | 0.006338 / 0.007607 (-0.001269) | 0.523280 / 0.226044 (0.297236) | 5.235969 / 2.268929 (2.967040) | 2.897585 / 55.444624 (-52.547039) | 2.541586 / 6.876477 (-4.334891) | 2.564233 / 2.142072 (0.422160) | 0.584714 / 4.805227 (-4.220513) | 0.124611 / 6.500664 (-6.376053) | 0.061774 / 0.075469 (-0.013695) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.349799 / 1.841788 (-0.491988) | 18.225076 / 8.074308 (10.150768) | 13.781518 / 10.191392 (3.590126) | 0.130562 / 0.680424 (-0.549862) | 0.016434 / 0.534201 (-0.517767) | 0.331607 / 0.579283 (-0.247676) | 0.343456 / 0.434364 (-0.090908) | 0.380437 / 0.540337 (-0.159900) | 0.522793 / 1.386936 (-0.864143) |

![](https://cml.dev/watermark.png#f0a3dbbd2e7ace162346d95ec27db674e80c1e23 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.013721 / 0.011353 (0.002368) | 0.005715 / 0.011008 (-0.005293) | 0.090116 / 0.038508 (0.051608) | 0.087185 / 0.023109 (0.064075) | 0.427813 / 0.275898 (0.151915) | 0.390614 / 0.323480 (0.067135) | 0.006976 / 0.007986 (-0.001009) | 0.004231 / 0.004328 (-0.000098) | 0.078320 / 0.004250 (0.074070) | 0.066235 / 0.037052 (0.029183) | 0.439904 / 0.258489 (0.181415) | 0.424119 / 0.293841 (0.130278) | 0.050362 / 0.128546 (-0.078184) | 0.014992 / 0.075646 (-0.060654) | 0.293519 / 0.419271 (-0.125753) | 0.066906 / 0.043533 (0.023373) | 0.449657 / 0.255139 (0.194518) | 0.393800 / 0.283200 (0.110600) | 0.032258 / 0.141683 (-0.109425) | 1.539534 / 1.452155 (0.087379) | 1.675292 / 1.492716 (0.182576) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.210515 / 0.018006 (0.192508) | 0.506817 / 0.000490 (0.506327) | 0.001938 / 0.000200 (0.001738) | 0.000118 / 0.000054 (0.000064) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026019 / 0.037411 (-0.011393) | 0.080635 / 0.014526 (0.066109) | 0.103050 / 0.176557 (-0.073507) | 0.160597 / 0.737135 (-0.576538) | 0.095844 / 0.296338 (-0.200495) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.506359 / 0.215209 (0.291150) | 5.041586 / 2.077655 (2.963931) | 2.198288 / 1.504120 (0.694168) | 1.987544 / 1.541195 (0.446349) | 1.866790 / 1.468490 (0.398300) | 0.681642 / 4.584777 (-3.903135) | 4.719306 / 3.745712 (0.973593) | 7.669869 / 5.269862 (2.400008) | 4.466082 / 4.565676 (-0.099595) | 0.092974 / 0.424275 (-0.331301) | 0.008196 / 0.007607 (0.000589) | 0.707656 / 0.226044 (0.481612) | 6.974507 / 2.268929 (4.705579) | 3.254206 / 55.444624 (-52.190418) | 2.499019 / 6.876477 (-4.377457) | 2.509089 / 2.142072 (0.367017) | 0.915952 / 4.805227 (-3.889276) | 0.192119 / 6.500664 (-6.308545) | 0.065473 / 0.075469 (-0.009996) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.309078 / 1.841788 (-0.532710) | 19.660348 / 8.074308 (11.586040) | 16.659582 / 10.191392 (6.468190) | 0.194315 / 0.680424 (-0.486109) | 0.027773 / 0.534201 (-0.506428) | 0.401241 / 0.579283 (-0.178042) | 0.515799 / 0.434364 (0.081435) | 0.488772 / 0.540337 (-0.051566) | 0.604790 / 1.386936 (-0.782146) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006823 / 0.011353 (-0.004530) | 0.003940 / 0.011008 (-0.007068) | 0.061533 / 0.038508 (0.023025) | 0.065241 / 0.023109 (0.042132) | 0.411790 / 0.275898 (0.135892) | 0.475720 / 0.323480 (0.152241) | 0.005376 / 0.007986 (-0.002609) | 0.003433 / 0.004328 (-0.000895) | 0.065703 / 0.004250 (0.061452) | 0.050736 / 0.037052 (0.013683) | 0.435890 / 0.258489 (0.177401) | 0.436698 / 0.293841 (0.142857) | 0.040357 / 0.128546 (-0.088189) | 0.011578 / 0.075646 (-0.064069) | 0.072831 / 0.419271 (-0.346440) | 0.055698 / 0.043533 (0.012165) | 0.408225 / 0.255139 (0.153086) | 0.439551 / 0.283200 (0.156352) | 0.030469 / 0.141683 (-0.111214) | 1.443866 / 1.452155 (-0.008289) | 1.502022 / 1.492716 (0.009306) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.290338 / 0.018006 (0.272332) | 0.540726 / 0.000490 (0.540236) | 0.003244 / 0.000200 (0.003044) | 0.000170 / 0.000054 (0.000116) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030865 / 0.037411 (-0.006547) | 0.090866 / 0.014526 (0.076340) | 0.106224 / 0.176557 (-0.070332) | 0.166583 / 0.737135 (-0.570553) | 0.104448 / 0.296338 (-0.191891) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.518025 / 0.215209 (0.302816) | 6.027065 / 2.077655 (3.949410) | 2.671840 / 1.504120 (1.167720) | 2.273949 / 1.541195 (0.732754) | 2.414892 / 1.468490 (0.946402) | 0.774318 / 4.584777 (-3.810459) | 5.020364 / 3.745712 (1.274652) | 4.146927 / 5.269862 (-1.122934) | 2.584598 / 4.565676 (-1.981078) | 0.089519 / 0.424275 (-0.334756) | 0.009181 / 0.007607 (0.001574) | 0.654467 / 0.226044 (0.428423) | 6.421595 / 2.268929 (4.152666) | 3.091589 / 55.444624 (-52.353036) | 2.554798 / 6.876477 (-4.321679) | 2.441354 / 2.142072 (0.299282) | 0.943386 / 4.805227 (-3.861841) | 0.173641 / 6.500664 (-6.327023) | 0.072209 / 0.075469 (-0.003260) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.557147 / 1.841788 (-0.284641) | 19.980747 / 8.074308 (11.906439) | 17.816813 / 10.191392 (7.625421) | 0.212078 / 0.680424 (-0.468346) | 0.025435 / 0.534201 (-0.508766) | 0.396200 / 0.579283 (-0.183084) | 0.546249 / 0.434364 (0.111885) | 0.459632 / 0.540337 (-0.080705) | 0.616548 / 1.386936 (-0.770388) |

![](https://cml.dev/watermark.png#535e972a70a3d4f8490a7e1a77ac43d5a4ab2655 "CML watermark")

`hfh` and `transformers` have dropped Python 3.7 support, so we should do the same :).

(Based on the stats, it seems less than 10% of the users use `datasets` with Python 3.7)


Misc improvements

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006897 / 0.011353 (-0.004456) | 0.004207 / 0.011008 (-0.006802) | 0.104828 / 0.038508 (0.066320) | 0.048054 / 0.023109 (0.024945) | 0.373991 / 0.275898 (0.098093) | 0.426740 / 0.323480 (0.103260) | 0.005540 / 0.007986 (-0.002446) | 0.003531 / 0.004328 (-0.000797) | 0.079304 / 0.004250 (0.075053) | 0.066996 / 0.037052 (0.029944) | 0.370675 / 0.258489 (0.112186) | 0.414154 / 0.293841 (0.120313) | 0.031567 / 0.128546 (-0.096979) | 0.008843 / 0.075646 (-0.066803) | 0.357426 / 0.419271 (-0.061845) | 0.067040 / 0.043533 (0.023508) | 0.362384 / 0.255139 (0.107245) | 0.376056 / 0.283200 (0.092856) | 0.032985 / 0.141683 (-0.108697) | 1.560603 / 1.452155 (0.108448) | 1.619024 / 1.492716 (0.126308) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.229059 / 0.018006 (0.211053) | 0.440513 / 0.000490 (0.440023) | 0.004647 / 0.000200 (0.004447) | 0.000085 / 0.000054 (0.000030) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029517 / 0.037411 (-0.007894) | 0.120974 / 0.014526 (0.106448) | 0.125070 / 0.176557 (-0.051486) | 0.184695 / 0.737135 (-0.552441) | 0.130244 / 0.296338 (-0.166095) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.436930 / 0.215209 (0.221721) | 4.356118 / 2.077655 (2.278463) | 2.049169 / 1.504120 (0.545049) | 1.842898 / 1.541195 (0.301703) | 1.918948 / 1.468490 (0.450458) | 0.553573 / 4.584777 (-4.031204) | 3.883195 / 3.745712 (0.137483) | 3.209780 / 5.269862 (-2.060081) | 1.551707 / 4.565676 (-3.013970) | 0.068181 / 0.424275 (-0.356094) | 0.012370 / 0.007607 (0.004762) | 0.539899 / 0.226044 (0.313854) | 5.380008 / 2.268929 (3.111079) | 2.518178 / 55.444624 (-52.926446) | 2.174190 / 6.876477 (-4.702286) | 2.317812 / 2.142072 (0.175740) | 0.674154 / 4.805227 (-4.131073) | 0.149313 / 6.500664 (-6.351351) | 0.068297 / 0.075469 (-0.007172) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.261426 / 1.841788 (-0.580362) | 15.316378 / 8.074308 (7.242070) | 13.573512 / 10.191392 (3.382120) | 0.190022 / 0.680424 (-0.490401) | 0.018697 / 0.534201 (-0.515504) | 0.448122 / 0.579283 (-0.131161) | 0.435044 / 0.434364 (0.000681) | 0.550065 / 0.540337 (0.009728) | 0.653547 / 1.386936 (-0.733389) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007116 / 0.011353 (-0.004237) | 0.004375 / 0.011008 (-0.006633) | 0.081793 / 0.038508 (0.043285) | 0.047980 / 0.023109 (0.024871) | 0.392185 / 0.275898 (0.116287) | 0.462263 / 0.323480 (0.138783) | 0.005574 / 0.007986 (-0.002412) | 0.003552 / 0.004328 (-0.000776) | 0.080413 / 0.004250 (0.076162) | 0.065539 / 0.037052 (0.028487) | 0.413137 / 0.258489 (0.154648) | 0.467377 / 0.293841 (0.173536) | 0.034386 / 0.128546 (-0.094160) | 0.009183 / 0.075646 (-0.066464) | 0.087542 / 0.419271 (-0.331730) | 0.053954 / 0.043533 (0.010421) | 0.385096 / 0.255139 (0.129957) | 0.404900 / 0.283200 (0.121701) | 0.025908 / 0.141683 (-0.115775) | 1.550159 / 1.452155 (0.098005) | 1.598794 / 1.492716 (0.106078) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.246222 / 0.018006 (0.228216) | 0.441095 / 0.000490 (0.440605) | 0.006863 / 0.000200 (0.006663) | 0.000109 / 0.000054 (0.000055) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032179 / 0.037411 (-0.005233) | 0.120112 / 0.014526 (0.105586) | 0.129326 / 0.176557 (-0.047230) | 0.184542 / 0.737135 (-0.552593) | 0.135038 / 0.296338 (-0.161300) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.459002 / 0.215209 (0.243793) | 4.580258 / 2.077655 (2.502604) | 2.296689 / 1.504120 (0.792569) | 2.104338 / 1.541195 (0.563143) | 2.182896 / 1.468490 (0.714406) | 0.546447 / 4.584777 (-4.038330) | 3.854047 / 3.745712 (0.108335) | 1.873829 / 5.269862 (-3.396032) | 1.116484 / 4.565676 (-3.449193) | 0.067158 / 0.424275 (-0.357117) | 0.012035 / 0.007607 (0.004428) | 0.556642 / 0.226044 (0.330597) | 5.574436 / 2.268929 (3.305508) | 2.828223 / 55.444624 (-52.616402) | 2.519851 / 6.876477 (-4.356626) | 2.668594 / 2.142072 (0.526521) | 0.675989 / 4.805227 (-4.129238) | 0.146075 / 6.500664 (-6.354589) | 0.067788 / 0.075469 (-0.007681) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.345958 / 1.841788 (-0.495830) | 15.672748 / 8.074308 (7.598440) | 14.937583 / 10.191392 (4.746191) | 0.163479 / 0.680424 (-0.516945) | 0.018364 / 0.534201 (-0.515837) | 0.433296 / 0.579283 (-0.145987) | 0.432463 / 0.434364 (-0.001901) | 0.512000 / 0.540337 (-0.028338) | 0.619397 / 1.386936 (-0.767539) |

![](https://cml.dev/watermark.png#0832d48a07ed00b406271f4b4439e6d54ae38ebf "CML watermark")

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010097 / 0.011353 (-0.001256) | 0.005070 / 0.011008 (-0.005939) | 0.118638 / 0.038508 (0.080130) | 0.043651 / 0.023109 (0.020542) | 0.356074 / 0.275898 (0.080176) | 0.414578 / 0.323480 (0.091098) | 0.005939 / 0.007986 (-0.002046) | 0.004927 / 0.004328 (0.000598) | 0.089545 / 0.004250 (0.085294) | 0.067533 / 0.037052 (0.030481) | 0.371550 / 0.258489 (0.113061) | 0.417808 / 0.293841 (0.123967) | 0.045186 / 0.128546 (-0.083361) | 0.015763 / 0.075646 (-0.059883) | 0.393304 / 0.419271 (-0.025967) | 0.065123 / 0.043533 (0.021591) | 0.345057 / 0.255139 (0.089918) | 0.378809 / 0.283200 (0.095610) | 0.033243 / 0.141683 (-0.108440) | 1.679956 / 1.452155 (0.227802) | 1.775456 / 1.492716 (0.282739) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.229723 / 0.018006 (0.211717) | 0.554630 / 0.000490 (0.554140) | 0.008729 / 0.000200 (0.008529) | 0.000183 / 0.000054 (0.000129) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027284 / 0.037411 (-0.010128) | 0.114741 / 0.014526 (0.100215) | 0.129188 / 0.176557 (-0.047369) | 0.189270 / 0.737135 (-0.547866) | 0.126000 / 0.296338 (-0.170339) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.580417 / 0.215209 (0.365208) | 5.829337 / 2.077655 (3.751683) | 2.421191 / 1.504120 (0.917071) | 2.063673 / 1.541195 (0.522479) | 2.133427 / 1.468490 (0.664937) | 0.830964 / 4.584777 (-3.753813) | 5.107139 / 3.745712 (1.361427) | 4.599451 / 5.269862 (-0.670410) | 2.406502 / 4.565676 (-2.159175) | 0.100422 / 0.424275 (-0.323853) | 0.011850 / 0.007607 (0.004243) | 0.741881 / 0.226044 (0.515836) | 7.425689 / 2.268929 (5.156760) | 3.068948 / 55.444624 (-52.375676) | 2.496292 / 6.876477 (-4.380184) | 2.566420 / 2.142072 (0.424348) | 1.093084 / 4.805227 (-3.712144) | 0.224106 / 6.500664 (-6.276558) | 0.084549 / 0.075469 (0.009080) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.416315 / 1.841788 (-0.425473) | 16.306901 / 8.074308 (8.232593) | 19.792419 / 10.191392 (9.601027) | 0.224223 / 0.680424 (-0.456201) | 0.026385 / 0.534201 (-0.507816) | 0.463460 / 0.579283 (-0.115823) | 0.598385 / 0.434364 (0.164021) | 0.543981 / 0.540337 (0.003644) | 0.647454 / 1.386936 (-0.739482) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009470 / 0.011353 (-0.001883) | 0.004800 / 0.011008 (-0.006208) | 0.094276 / 0.038508 (0.055768) | 0.045157 / 0.023109 (0.022048) | 0.397302 / 0.275898 (0.121404) | 0.474213 / 0.323480 (0.150733) | 0.005826 / 0.007986 (-0.002160) | 0.003724 / 0.004328 (-0.000605) | 0.090060 / 0.004250 (0.085809) | 0.066671 / 0.037052 (0.029618) | 0.439560 / 0.258489 (0.181071) | 0.468598 / 0.293841 (0.174757) | 0.044549 / 0.128546 (-0.083997) | 0.014000 / 0.075646 (-0.061646) | 0.110457 / 0.419271 (-0.308815) | 0.065898 / 0.043533 (0.022365) | 0.408101 / 0.255139 (0.152962) | 0.433473 / 0.283200 (0.150273) | 0.038438 / 0.141683 (-0.103245) | 1.767781 / 1.452155 (0.315626) | 1.791575 / 1.492716 (0.298859) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.230257 / 0.018006 (0.212251) | 0.492280 / 0.000490 (0.491790) | 0.005110 / 0.000200 (0.004910) | 0.000119 / 0.000054 (0.000065) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028854 / 0.037411 (-0.008557) | 0.111702 / 0.014526 (0.097176) | 0.122040 / 0.176557 (-0.054517) | 0.179103 / 0.737135 (-0.558032) | 0.128869 / 0.296338 (-0.167470) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.634795 / 0.215209 (0.419586) | 6.204760 / 2.077655 (4.127105) | 2.692479 / 1.504120 (1.188359) | 2.324260 / 1.541195 (0.783066) | 2.380640 / 1.468490 (0.912149) | 0.887827 / 4.584777 (-3.696950) | 5.251648 / 3.745712 (1.505935) | 2.632767 / 5.269862 (-2.637095) | 1.745721 / 4.565676 (-2.819955) | 0.108364 / 0.424275 (-0.315911) | 0.013409 / 0.007607 (0.005802) | 0.783427 / 0.226044 (0.557383) | 7.765144 / 2.268929 (5.496216) | 3.340686 / 55.444624 (-52.103938) | 2.715340 / 6.876477 (-4.161137) | 2.768604 / 2.142072 (0.626531) | 1.119746 / 4.805227 (-3.685481) | 0.210804 / 6.500664 (-6.289860) | 0.072600 / 0.075469 (-0.002869) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.517334 / 1.841788 (-0.324454) | 17.046837 / 8.074308 (8.972529) | 19.371090 / 10.191392 (9.179698) | 0.194275 / 0.680424 (-0.486148) | 0.026712 / 0.534201 (-0.507488) | 0.462731 / 0.579283 (-0.116552) | 0.568958 / 0.434364 (0.134595) | 0.555707 / 0.540337 (0.015370) | 0.663654 / 1.386936 (-0.723283) |

![](https://cml.dev/watermark.png#5d20476b1d4c8e11e0ffafc1570cbf4bd19011cf "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006423 / 0.011353 (-0.004930) | 0.003882 / 0.011008 (-0.007126) | 0.082976 / 0.038508 (0.044468) | 0.071281 / 0.023109 (0.048171) | 0.311367 / 0.275898 (0.035469) | 0.348228 / 0.323480 (0.024748) | 0.005315 / 0.007986 (-0.002671) | 0.003326 / 0.004328 (-0.001003) | 0.064641 / 0.004250 (0.060391) | 0.056134 / 0.037052 (0.019081) | 0.314071 / 0.258489 (0.055582) | 0.360534 / 0.293841 (0.066693) | 0.030642 / 0.128546 (-0.097904) | 0.008301 / 0.075646 (-0.067345) | 0.285820 / 0.419271 (-0.133451) | 0.069241 / 0.043533 (0.025708) | 0.313995 / 0.255139 (0.058856) | 0.336656 / 0.283200 (0.053457) | 0.031686 / 0.141683 (-0.109997) | 1.467627 / 1.452155 (0.015472) | 1.536493 / 1.492716 (0.043777) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.196518 / 0.018006 (0.178512) | 0.458235 / 0.000490 (0.457745) | 0.005599 / 0.000200 (0.005399) | 0.000088 / 0.000054 (0.000034) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027371 / 0.037411 (-0.010040) | 0.080986 / 0.014526 (0.066460) | 0.093296 / 0.176557 (-0.083260) | 0.150592 / 0.737135 (-0.586543) | 0.094150 / 0.296338 (-0.202188) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.379412 / 0.215209 (0.164202) | 3.797927 / 2.077655 (1.720272) | 1.830654 / 1.504120 (0.326534) | 1.669569 / 1.541195 (0.128374) | 1.746738 / 1.468490 (0.278248) | 0.479536 / 4.584777 (-4.105241) | 3.592867 / 3.745712 (-0.152845) | 5.468098 / 5.269862 (0.198237) | 3.268013 / 4.565676 (-1.297663) | 0.056635 / 0.424275 (-0.367640) | 0.007224 / 0.007607 (-0.000383) | 0.456681 / 0.226044 (0.230636) | 4.566736 / 2.268929 (2.297807) | 2.362831 / 55.444624 (-53.081793) | 1.965141 / 6.876477 (-4.911336) | 2.156905 / 2.142072 (0.014833) | 0.572543 / 4.805227 (-4.232684) | 0.132203 / 6.500664 (-6.368461) | 0.059254 / 0.075469 (-0.016215) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.256134 / 1.841788 (-0.585654) | 19.905438 / 8.074308 (11.831130) | 14.179556 / 10.191392 (3.988164) | 0.168043 / 0.680424 (-0.512381) | 0.018215 / 0.534201 (-0.515986) | 0.392740 / 0.579283 (-0.186543) | 0.398397 / 0.434364 (-0.035967) | 0.463806 / 0.540337 (-0.076531) | 0.616248 / 1.386936 (-0.770688) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006564 / 0.011353 (-0.004789) | 0.003923 / 0.011008 (-0.007085) | 0.063929 / 0.038508 (0.025421) | 0.073780 / 0.023109 (0.050671) | 0.360242 / 0.275898 (0.084344) | 0.395078 / 0.323480 (0.071598) | 0.005265 / 0.007986 (-0.002720) | 0.003229 / 0.004328 (-0.001100) | 0.064094 / 0.004250 (0.059843) | 0.057468 / 0.037052 (0.020416) | 0.369530 / 0.258489 (0.111041) | 0.411159 / 0.293841 (0.117318) | 0.031278 / 0.128546 (-0.097268) | 0.008424 / 0.075646 (-0.067222) | 0.070411 / 0.419271 (-0.348860) | 0.048714 / 0.043533 (0.005181) | 0.361280 / 0.255139 (0.106141) | 0.382468 / 0.283200 (0.099269) | 0.023059 / 0.141683 (-0.118624) | 1.452369 / 1.452155 (0.000215) | 1.519192 / 1.492716 (0.026475) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.223745 / 0.018006 (0.205739) | 0.442086 / 0.000490 (0.441596) | 0.000379 / 0.000200 (0.000179) | 0.000055 / 0.000054 (0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030919 / 0.037411 (-0.006493) | 0.088483 / 0.014526 (0.073958) | 0.101165 / 0.176557 (-0.075391) | 0.154332 / 0.737135 (-0.582804) | 0.103030 / 0.296338 (-0.193309) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.414520 / 0.215209 (0.199311) | 4.126754 / 2.077655 (2.049099) | 2.142677 / 1.504120 (0.638557) | 1.995300 / 1.541195 (0.454106) | 2.101678 / 1.468490 (0.633188) | 0.481099 / 4.584777 (-4.103678) | 3.562813 / 3.745712 (-0.182900) | 3.392463 / 5.269862 (-1.877399) | 1.983943 / 4.565676 (-2.581734) | 0.056594 / 0.424275 (-0.367681) | 0.007216 / 0.007607 (-0.000391) | 0.495085 / 0.226044 (0.269041) | 4.955640 / 2.268929 (2.686712) | 2.629434 / 55.444624 (-52.815191) | 2.269577 / 6.876477 (-4.606900) | 2.357708 / 2.142072 (0.215635) | 0.612370 / 4.805227 (-4.192857) | 0.131169 / 6.500664 (-6.369495) | 0.061029 / 0.075469 (-0.014440) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.339438 / 1.841788 (-0.502350) | 19.757611 / 8.074308 (11.683303) | 14.246254 / 10.191392 (4.054862) | 0.170750 / 0.680424 (-0.509674) | 0.018192 / 0.534201 (-0.516009) | 0.395693 / 0.579283 (-0.183590) | 0.411003 / 0.434364 (-0.023361) | 0.478531 / 0.540337 (-0.061806) | 0.650291 / 1.386936 (-0.736645) |

![](https://cml.dev/watermark.png#3e34d06d746688dd5d26e4c85517b7e1a2f361ca "CML watermark")

Contains the following improvements:

* fixes a "share dataset" link in README and modifies the "hosting" part in the disclaimer section
* updates `Makefile` to also run the style checks on `utils` and `setup.py`
* deletes a test for GH-hosted datasets (no longer supported)
* deletes `convert_dataset.sh` (outdated)
* aligns `utils/release.py` with `transformers` (the current version is outdated)


interleave_datasets & DataCollatorForLanguageModeling having a conflict ?

### Describe the bug

Hi everyone :)

I have two local & custom datasets (1 "sentence" per line) which I split along the 95/5 lines for pre-training a Bert model. I use a modified version of `run_mlm.py` in order to be able to make use of `interleave_dataset`:

- `tokenize()` runs fine
- `group_text()` runs fine

Everytime, on step 19, I get

```pytb
  File "env/lib/python3.9/site-packages/transformers/data/data_collator.py", line 779, in torch_mask_tokens
    inputs[indices_random] = random_words[indices_random]
RuntimeError: Index put requires the source and destination dtypes match, got Float for the destination and Long for the source.
```

I tried:
- training without interleave on dataset 1, it runs
- training without interleave on dataset 2, it runs
- training without `.to_iterable_dataset()`, it hangs then crash
- training without group_text() and padding to max_length seemed to fix the issue, but who knows if this was just because it was an issue that would come much later in terms of steps.

I might have coded something wrong, but I don't get what

### Steps to reproduce the bug

I have this function:

```py
def build_dataset(path: str, percent: str):
    dataset = load_dataset(
        "text",
        data_files={"train": [path]},
        split=f"train[{percent}]"
    )
    dataset = dataset.map(
        lambda examples: tokenize(examples["text"]),
        batched=True,
        num_proc=num_proc,
    )

    dataset = dataset.map(
        group_texts,
        batched=True,
        num_proc=num_proc,
        desc=f"Grouping texts in chunks of {tokenizer.max_seq_length}",
        remove_columns=["text"]
    )

    print(len(dataset))
    return dataset.to_iterable_dataset()
```

I hardcoded group_text:
```py
    def group_texts(examples):
        # Concatenate all texts.
        concatenated_examples = {k: list(chain(*examples[k])) for k in examples.keys()}
        total_length = len(concatenated_examples[list(examples.keys())[0]])
        # We drop the small remainder, and if the total_length < max_seq_length  we exclude this batch and return an empty dict.
        # We could add padding if the model supported it instead of this drop, you can customize this part to your needs.
        total_length = (total_length // 512) * 512
        # Split by chunks of max_len.
        result = {
            k: [t[i: i + 512] for i in range(0, total_length, 512)]
            for k, t in concatenated_examples.items()
        }
        # result = {k: [el for el in elements if el] for k, elements in result.items()}
        return result
```

And then I build datasets using the following code:

```py
train1 = build_dataset("d1.txt", ":95%")
train2 = build_dataset("d2.txt", ":95%")
dev1 = build_dataset("d1.txt", "95%:")
dev2 = build_dataset("d2.txt", "95%:")
```

and finally I run
```py
train_dataset = interleave_datasets(
    [train1, train2],
    probabilities=[0.8, 0.2],
    seed=42
)
eval_dataset = interleave_datasets(
    [dev1, dev2],
    probabilities=[0.8, 0.2],
    seed=42
)
```

Then I run the training part which remains mostly untouched:

> CUDA_VISIBLE_DEVICES=1 python custom_dataset.py --model_type bert --per_device_train_batch_size 32 --do_train --output_dir /var/mlm/training-bert/model --max_seq_length 512 --save_steps 10000 --save_total_limit 3 --auto_find_batch_size --logging_dir ./logs-bert --learning_rate 0.0001 --do_train --num_train_epochs 25 --warmup_steps 10000 --max_step 45000 --fp16

### Expected behavior

The model should then train normally, but fails every time at the same step (19).

printing the variables at `inputs[indices_random] = random_words[indices_random]` shows a magnificient empty tensor (, 32) [if I remember well]

### Environment info

transformers[torch] 4.30.2
Ubuntu
A100 0 CUDA 12
Driver Version: 525.116.04


Add KLUE-MRC metrics

The metrics API in `datasets` is deprecated as of version 2.0, and `evaulate` is our new library for metrics. You can add a new metric to it by following [these steps](https://huggingface.co/docs/evaluate/creating_and_sharing).

## Metrics for KLUE-MRC (Korean Language Understanding Evaluation — Machine Reading Comprehension)

Adding metrics for [KLUE-MRC](https://huggingface.co/datasets/klue).
KLUE-MRC is very similar to SQuAD 2.0 but has a slightly different format which is why I added metrics for KLUE-MRC.

Specifically, in the case of [LM Eval Harness](https://github.com/EleutherAI/lm-evaluation-harness), it leverages the scoring script of SQuAD to evaluate SQuAD 2.0 and KorQuAD. But the script isn't suitable for KLUE-MRC because KLUE-MRC is a bit different from SQuAD 2.0. And this is why I added the scoring script for KLUE-MRC.

- [x] All tests passed
- [x] Added a metric card (referred the metric card of SQuAD 2.0)
- [x] Compatibility test with [LM Eval Harness](https://github.com/EleutherAI/lm-evaluation-harness) passed

### References
- [KLUE: Korean Language Understanding Evaluation](https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/file/98dce83da57b0395e163467c9dae521b-Paper-round2.pdf)
- [KLUE on Hugging Face Datasets](https://huggingface.co/datasets/klue)
- #2416