Ensure last tqdm update in `map`

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.011060 / 0.011353 (-0.000293) | 0.005752 / 0.011008 (-0.005256) | 0.120349 / 0.038508 (0.081841) | 0.045303 / 0.023109 (0.022194) | 0.359196 / 0.275898 (0.083298) | 0.406351 / 0.323480 (0.082871) | 0.009474 / 0.007986 (0.001489) | 0.004524 / 0.004328 (0.000195) | 0.091990 / 0.004250 (0.087739) | 0.050034 / 0.037052 (0.012982) | 0.372479 / 0.258489 (0.113990) | 0.418907 / 0.293841 (0.125067) | 0.044300 / 0.128546 (-0.084247) | 0.013989 / 0.075646 (-0.061657) | 0.397406 / 0.419271 (-0.021866) | 0.056070 / 0.043533 (0.012537) | 0.357597 / 0.255139 (0.102458) | 0.382938 / 0.283200 (0.099738) | 0.117060 / 0.141683 (-0.024623) | 1.670869 / 1.452155 (0.218714) | 1.780944 / 1.492716 (0.288227) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.229578 / 0.018006 (0.211572) | 0.493711 / 0.000490 (0.493222) | 0.008413 / 0.000200 (0.008213) | 0.000118 / 0.000054 (0.000063) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033364 / 0.037411 (-0.004047) | 0.135953 / 0.014526 (0.121427) | 0.141942 / 0.176557 (-0.034614) | 0.225891 / 0.737135 (-0.511244) | 0.151010 / 0.296338 (-0.145328) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.470937 / 0.215209 (0.255728) | 4.710258 / 2.077655 (2.632603) | 2.132025 / 1.504120 (0.627905) | 1.913134 / 1.541195 (0.371939) | 2.025993 / 1.468490 (0.557503) | 0.835993 / 4.584777 (-3.748784) | 4.446678 / 3.745712 (0.700965) | 4.260014 / 5.269862 (-1.009847) | 2.193078 / 4.565676 (-2.372598) | 0.100132 / 0.424275 (-0.324143) | 0.014163 / 0.007607 (0.006556) | 0.599252 / 0.226044 (0.373208) | 5.976377 / 2.268929 (3.707448) | 2.678116 / 55.444624 (-52.766508) | 2.309311 / 6.876477 (-4.567166) | 2.410284 / 2.142072 (0.268212) | 1.002415 / 4.805227 (-3.802813) | 0.194588 / 6.500664 (-6.306076) | 0.074921 / 0.075469 (-0.000548) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.432389 / 1.841788 (-0.409399) | 17.915288 / 8.074308 (9.840980) | 17.190906 / 10.191392 (6.999514) | 0.238469 / 0.680424 (-0.441955) | 0.036270 / 0.534201 (-0.497931) | 0.537320 / 0.579283 (-0.041963) | 0.512876 / 0.434364 (0.078512) | 0.629022 / 0.540337 (0.088685) | 0.750109 / 1.386936 (-0.636827) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008544 / 0.011353 (-0.002809) | 0.005933 / 0.011008 (-0.005075) | 0.088879 / 0.038508 (0.050371) | 0.040387 / 0.023109 (0.017278) | 0.406392 / 0.275898 (0.130494) | 0.449572 / 0.323480 (0.126092) | 0.006623 / 0.007986 (-0.001362) | 0.004727 / 0.004328 (0.000398) | 0.086745 / 0.004250 (0.082495) | 0.054335 / 0.037052 (0.017283) | 0.405652 / 0.258489 (0.147163) | 0.473934 / 0.293841 (0.180093) | 0.042157 / 0.128546 (-0.086390) | 0.014249 / 0.075646 (-0.061397) | 0.102130 / 0.419271 (-0.317141) | 0.056815 / 0.043533 (0.013282) | 0.407945 / 0.255139 (0.152806) | 0.431720 / 0.283200 (0.148521) | 0.119901 / 0.141683 (-0.021781) | 1.738381 / 1.452155 (0.286227) | 1.838981 / 1.492716 (0.346265) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.251926 / 0.018006 (0.233919) | 0.498117 / 0.000490 (0.497627) | 0.000439 / 0.000200 (0.000239) | 0.000065 / 0.000054 (0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034526 / 0.037411 (-0.002886) | 0.133038 / 0.014526 (0.118512) | 0.147494 / 0.176557 (-0.029063) | 0.234392 / 0.737135 (-0.502743) | 0.152361 / 0.296338 (-0.143978) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.495144 / 0.215209 (0.279935) | 4.936646 / 2.077655 (2.858991) | 2.385549 / 1.504120 (0.881429) | 2.173817 / 1.541195 (0.632622) | 2.327508 / 1.468490 (0.859018) | 0.851899 / 4.584777 (-3.732878) | 4.820388 / 3.745712 (1.074676) | 2.500304 / 5.269862 (-2.769558) | 1.621246 / 4.565676 (-2.944430) | 0.102858 / 0.424275 (-0.321417) | 0.014719 / 0.007607 (0.007112) | 0.611880 / 0.226044 (0.385836) | 6.100737 / 2.268929 (3.831808) | 2.955681 / 55.444624 (-52.488943) | 2.563533 / 6.876477 (-4.312943) | 2.659030 / 2.142072 (0.516958) | 1.004737 / 4.805227 (-3.800490) | 0.198379 / 6.500664 (-6.302285) | 0.078705 / 0.075469 (0.003236) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.501155 / 1.841788 (-0.340633) | 18.381513 / 8.074308 (10.307205) | 16.173893 / 10.191392 (5.982501) | 0.209497 / 0.680424 (-0.470927) | 0.021640 / 0.534201 (-0.512561) | 0.505905 / 0.579283 (-0.073378) | 0.513446 / 0.434364 (0.079082) | 0.652704 / 0.540337 (0.112366) | 0.761038 / 1.386936 (-0.625898) |

![](https://cml.dev/watermark.png#b8235c92b46b6a63286fcee1a56adae4c0a751d3 "CML watermark")

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009085 / 0.011353 (-0.002268) | 0.004589 / 0.011008 (-0.006419) | 0.100820 / 0.038508 (0.062312) | 0.030677 / 0.023109 (0.007568) | 0.306702 / 0.275898 (0.030804) | 0.360623 / 0.323480 (0.037144) | 0.007377 / 0.007986 (-0.000608) | 0.003480 / 0.004328 (-0.000848) | 0.077813 / 0.004250 (0.073562) | 0.037293 / 0.037052 (0.000241) | 0.314137 / 0.258489 (0.055648) | 0.343394 / 0.293841 (0.049554) | 0.034202 / 0.128546 (-0.094344) | 0.011417 / 0.075646 (-0.064230) | 0.322584 / 0.419271 (-0.096687) | 0.041524 / 0.043533 (-0.002009) | 0.308116 / 0.255139 (0.052977) | 0.324527 / 0.283200 (0.041327) | 0.090973 / 0.141683 (-0.050710) | 1.515941 / 1.452155 (0.063787) | 1.548975 / 1.492716 (0.056259) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.185901 / 0.018006 (0.167895) | 0.420742 / 0.000490 (0.420252) | 0.002958 / 0.000200 (0.002758) | 0.000086 / 0.000054 (0.000032) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024242 / 0.037411 (-0.013170) | 0.098827 / 0.014526 (0.084302) | 0.107609 / 0.176557 (-0.068947) | 0.172228 / 0.737135 (-0.564908) | 0.110042 / 0.296338 (-0.186296) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.429647 / 0.215209 (0.214438) | 4.265406 / 2.077655 (2.187751) | 1.924514 / 1.504120 (0.420394) | 1.709881 / 1.541195 (0.168686) | 1.764872 / 1.468490 (0.296382) | 0.698089 / 4.584777 (-3.886688) | 3.439154 / 3.745712 (-0.306558) | 1.925058 / 5.269862 (-3.344804) | 1.267506 / 4.565676 (-3.298171) | 0.082167 / 0.424275 (-0.342108) | 0.012450 / 0.007607 (0.004843) | 0.523077 / 0.226044 (0.297033) | 5.240422 / 2.268929 (2.971494) | 2.363666 / 55.444624 (-53.080959) | 2.021903 / 6.876477 (-4.854574) | 2.136430 / 2.142072 (-0.005643) | 0.816377 / 4.805227 (-3.988850) | 0.151516 / 6.500664 (-6.349148) | 0.066590 / 0.075469 (-0.008879) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.216477 / 1.841788 (-0.625310) | 13.685044 / 8.074308 (5.610736) | 14.082620 / 10.191392 (3.891228) | 0.148399 / 0.680424 (-0.532025) | 0.028337 / 0.534201 (-0.505864) | 0.405379 / 0.579283 (-0.173904) | 0.405650 / 0.434364 (-0.028714) | 0.492658 / 0.540337 (-0.047679) | 0.578836 / 1.386936 (-0.808100) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006863 / 0.011353 (-0.004490) | 0.004746 / 0.011008 (-0.006262) | 0.075802 / 0.038508 (0.037294) | 0.027950 / 0.023109 (0.004840) | 0.347613 / 0.275898 (0.071715) | 0.401201 / 0.323480 (0.077721) | 0.005765 / 0.007986 (-0.002221) | 0.003567 / 0.004328 (-0.000762) | 0.074188 / 0.004250 (0.069937) | 0.041209 / 0.037052 (0.004157) | 0.346541 / 0.258489 (0.088052) | 0.425729 / 0.293841 (0.131888) | 0.032430 / 0.128546 (-0.096116) | 0.011708 / 0.075646 (-0.063938) | 0.084667 / 0.419271 (-0.334604) | 0.042155 / 0.043533 (-0.001378) | 0.341210 / 0.255139 (0.086071) | 0.389759 / 0.283200 (0.106559) | 0.092640 / 0.141683 (-0.049042) | 1.526093 / 1.452155 (0.073938) | 1.556277 / 1.492716 (0.063561) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.232383 / 0.018006 (0.214377) | 0.412353 / 0.000490 (0.411863) | 0.004009 / 0.000200 (0.003809) | 0.000071 / 0.000054 (0.000017) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025854 / 0.037411 (-0.011557) | 0.102660 / 0.014526 (0.088134) | 0.108420 / 0.176557 (-0.068137) | 0.175834 / 0.737135 (-0.561301) | 0.113472 / 0.296338 (-0.182867) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.443595 / 0.215209 (0.228386) | 4.420959 / 2.077655 (2.343305) | 2.112790 / 1.504120 (0.608670) | 1.908836 / 1.541195 (0.367641) | 1.998340 / 1.468490 (0.529850) | 0.706096 / 4.584777 (-3.878681) | 3.400871 / 3.745712 (-0.344841) | 2.803315 / 5.269862 (-2.466547) | 1.539392 / 4.565676 (-3.026284) | 0.083523 / 0.424275 (-0.340752) | 0.012541 / 0.007607 (0.004934) | 0.543428 / 0.226044 (0.317383) | 5.467416 / 2.268929 (3.198488) | 2.551970 / 55.444624 (-52.892654) | 2.212708 / 6.876477 (-4.663768) | 2.266169 / 2.142072 (0.124096) | 0.809943 / 4.805227 (-3.995284) | 0.152300 / 6.500664 (-6.348364) | 0.068591 / 0.075469 (-0.006878) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.330141 / 1.841788 (-0.511646) | 14.292734 / 8.074308 (6.218426) | 13.556157 / 10.191392 (3.364765) | 0.155949 / 0.680424 (-0.524475) | 0.016464 / 0.534201 (-0.517737) | 0.377906 / 0.579283 (-0.201377) | 0.390385 / 0.434364 (-0.043979) | 0.471867 / 0.540337 (-0.068471) | 0.557794 / 1.386936 (-0.829142) |

![](https://cml.dev/watermark.png#ba50512b76ef315f73bf821b0487296cdb373850 "CML watermark")

I just tried on colab and it didn't finish the progress bar for some reason.

Maybe we need to call `pbar.close()` before `return`

<img width="729" alt="image" src="https://user-images.githubusercontent.com/42851186/220417517-919438a4-5462-4e87-8f84-e9399a9be27c.png">

(just added .close() - let me try quickly if it works now)

it worked ! :)

<img width="575" alt="image" src="https://user-images.githubusercontent.com/42851186/220419220-8108f225-13cb-4968-acff-fe4543d5a324.png">

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008465 / 0.011353 (-0.002888) | 0.004622 / 0.011008 (-0.006387) | 0.100365 / 0.038508 (0.061857) | 0.029453 / 0.023109 (0.006344) | 0.358041 / 0.275898 (0.082143) | 0.424777 / 0.323480 (0.101298) | 0.006930 / 0.007986 (-0.001055) | 0.004756 / 0.004328 (0.000428) | 0.077128 / 0.004250 (0.072878) | 0.036338 / 0.037052 (-0.000715) | 0.367613 / 0.258489 (0.109124) | 0.397798 / 0.293841 (0.103957) | 0.033500 / 0.128546 (-0.095047) | 0.011427 / 0.075646 (-0.064219) | 0.321617 / 0.419271 (-0.097654) | 0.040937 / 0.043533 (-0.002596) | 0.345358 / 0.255139 (0.090219) | 0.366932 / 0.283200 (0.083733) | 0.086506 / 0.141683 (-0.055177) | 1.482434 / 1.452155 (0.030280) | 1.522773 / 1.492716 (0.030057) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.188815 / 0.018006 (0.170809) | 0.404689 / 0.000490 (0.404200) | 0.000390 / 0.000200 (0.000190) | 0.000059 / 0.000054 (0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023165 / 0.037411 (-0.014246) | 0.095934 / 0.014526 (0.081408) | 0.105788 / 0.176557 (-0.070769) | 0.169908 / 0.737135 (-0.567227) | 0.107871 / 0.296338 (-0.188467) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.457543 / 0.215209 (0.242334) | 4.563209 / 2.077655 (2.485554) | 2.172272 / 1.504120 (0.668152) | 1.965064 / 1.541195 (0.423870) | 2.020811 / 1.468490 (0.552321) | 0.705138 / 4.584777 (-3.879638) | 3.353430 / 3.745712 (-0.392283) | 1.861970 / 5.269862 (-3.407892) | 1.159201 / 4.565676 (-3.406476) | 0.083187 / 0.424275 (-0.341088) | 0.012750 / 0.007607 (0.005143) | 0.566377 / 0.226044 (0.340333) | 5.662645 / 2.268929 (3.393717) | 2.609565 / 55.444624 (-52.835059) | 2.244519 / 6.876477 (-4.631957) | 2.284111 / 2.142072 (0.142038) | 0.821974 / 4.805227 (-3.983253) | 0.151080 / 6.500664 (-6.349584) | 0.065373 / 0.075469 (-0.010096) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.230960 / 1.841788 (-0.610828) | 13.930408 / 8.074308 (5.856100) | 13.989082 / 10.191392 (3.797690) | 0.151961 / 0.680424 (-0.528462) | 0.028770 / 0.534201 (-0.505431) | 0.392269 / 0.579283 (-0.187015) | 0.400490 / 0.434364 (-0.033874) | 0.459770 / 0.540337 (-0.080568) | 0.534174 / 1.386936 (-0.852762) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006740 / 0.011353 (-0.004613) | 0.004496 / 0.011008 (-0.006512) | 0.076886 / 0.038508 (0.038377) | 0.027593 / 0.023109 (0.004484) | 0.339570 / 0.275898 (0.063672) | 0.379915 / 0.323480 (0.056435) | 0.004999 / 0.007986 (-0.002987) | 0.004253 / 0.004328 (-0.000076) | 0.074973 / 0.004250 (0.070722) | 0.037321 / 0.037052 (0.000269) | 0.344720 / 0.258489 (0.086230) | 0.398919 / 0.293841 (0.105078) | 0.032146 / 0.128546 (-0.096400) | 0.011694 / 0.075646 (-0.063952) | 0.085134 / 0.419271 (-0.334138) | 0.042328 / 0.043533 (-0.001205) | 0.339384 / 0.255139 (0.084245) | 0.368031 / 0.283200 (0.084831) | 0.092088 / 0.141683 (-0.049595) | 1.492313 / 1.452155 (0.040158) | 1.538406 / 1.492716 (0.045690) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.265619 / 0.018006 (0.247613) | 0.415478 / 0.000490 (0.414988) | 0.030221 / 0.000200 (0.030021) | 0.000277 / 0.000054 (0.000223) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024489 / 0.037411 (-0.012922) | 0.099920 / 0.014526 (0.085395) | 0.108301 / 0.176557 (-0.068256) | 0.179525 / 0.737135 (-0.557610) | 0.111492 / 0.296338 (-0.184847) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.440759 / 0.215209 (0.225550) | 4.382754 / 2.077655 (2.305100) | 2.088686 / 1.504120 (0.584566) | 1.890557 / 1.541195 (0.349363) | 1.947461 / 1.468490 (0.478971) | 0.701751 / 4.584777 (-3.883025) | 3.368896 / 3.745712 (-0.376816) | 1.867238 / 5.269862 (-3.402624) | 1.166787 / 4.565676 (-3.398890) | 0.083427 / 0.424275 (-0.340848) | 0.012406 / 0.007607 (0.004799) | 0.539467 / 0.226044 (0.313423) | 5.376083 / 2.268929 (3.107154) | 2.516566 / 55.444624 (-52.928058) | 2.177991 / 6.876477 (-4.698486) | 2.207438 / 2.142072 (0.065366) | 0.803316 / 4.805227 (-4.001911) | 0.150900 / 6.500664 (-6.349764) | 0.066328 / 0.075469 (-0.009141) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.295308 / 1.841788 (-0.546480) | 14.081343 / 8.074308 (6.007035) | 13.516853 / 10.191392 (3.325461) | 0.160530 / 0.680424 (-0.519894) | 0.016516 / 0.534201 (-0.517685) | 0.380160 / 0.579283 (-0.199123) | 0.443484 / 0.434364 (0.009120) | 0.466645 / 0.540337 (-0.073692) | 0.555339 / 1.386936 (-0.831597) |

![](https://cml.dev/watermark.png#e8a12313cd728e37b4dc4ce67864621ffc79fedb "CML watermark")

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.011321 / 0.011353 (-0.000031) | 0.006365 / 0.011008 (-0.004643) | 0.125613 / 0.038508 (0.087105) | 0.035327 / 0.023109 (0.012218) | 0.391998 / 0.275898 (0.116100) | 0.475402 / 0.323480 (0.151923) | 0.009579 / 0.007986 (0.001593) | 0.005621 / 0.004328 (0.001293) | 0.106097 / 0.004250 (0.101846) | 0.042774 / 0.037052 (0.005722) | 0.420850 / 0.258489 (0.162361) | 0.454501 / 0.293841 (0.160660) | 0.056885 / 0.128546 (-0.071661) | 0.021718 / 0.075646 (-0.053928) | 0.419422 / 0.419271 (0.000150) | 0.056690 / 0.043533 (0.013157) | 0.405375 / 0.255139 (0.150236) | 0.444404 / 0.283200 (0.161204) | 0.136912 / 0.141683 (-0.004771) | 1.846363 / 1.452155 (0.394208) | 1.747433 / 1.492716 (0.254717) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.282260 / 0.018006 (0.264254) | 0.615813 / 0.000490 (0.615323) | 0.000515 / 0.000200 (0.000315) | 0.000106 / 0.000054 (0.000052) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029913 / 0.037411 (-0.007499) | 0.135568 / 0.014526 (0.121042) | 0.134476 / 0.176557 (-0.042081) | 0.206974 / 0.737135 (-0.530161) | 0.136976 / 0.296338 (-0.159362) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.605241 / 0.215209 (0.390032) | 6.125097 / 2.077655 (4.047442) | 2.390102 / 1.504120 (0.885982) | 2.082196 / 1.541195 (0.541001) | 2.226527 / 1.468490 (0.758037) | 1.244807 / 4.584777 (-3.339970) | 5.476437 / 3.745712 (1.730725) | 3.014970 / 5.269862 (-2.254891) | 1.963428 / 4.565676 (-2.602249) | 0.137813 / 0.424275 (-0.286462) | 0.013794 / 0.007607 (0.006187) | 0.766149 / 0.226044 (0.540104) | 7.566103 / 2.268929 (5.297175) | 3.048958 / 55.444624 (-52.395666) | 2.394819 / 6.876477 (-4.481658) | 2.416021 / 2.142072 (0.273949) | 1.369896 / 4.805227 (-3.435331) | 0.245159 / 6.500664 (-6.255506) | 0.076848 / 0.075469 (0.001379) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.530448 / 1.841788 (-0.311340) | 18.580227 / 8.074308 (10.505919) | 20.108470 / 10.191392 (9.917078) | 0.227124 / 0.680424 (-0.453300) | 0.052050 / 0.534201 (-0.482151) | 0.604565 / 0.579283 (0.025282) | 0.686475 / 0.434364 (0.252111) | 0.672298 / 0.540337 (0.131960) | 0.770552 / 1.386936 (-0.616384) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010043 / 0.011353 (-0.001310) | 0.006445 / 0.011008 (-0.004563) | 0.099486 / 0.038508 (0.060978) | 0.037720 / 0.023109 (0.014610) | 0.425571 / 0.275898 (0.149673) | 0.467031 / 0.323480 (0.143551) | 0.007394 / 0.007986 (-0.000591) | 0.005008 / 0.004328 (0.000679) | 0.096176 / 0.004250 (0.091926) | 0.053694 / 0.037052 (0.016641) | 0.418653 / 0.258489 (0.160164) | 0.492441 / 0.293841 (0.198600) | 0.054593 / 0.128546 (-0.073953) | 0.023410 / 0.075646 (-0.052236) | 0.113825 / 0.419271 (-0.305446) | 0.066000 / 0.043533 (0.022467) | 0.418127 / 0.255139 (0.162988) | 0.457416 / 0.283200 (0.174217) | 0.119911 / 0.141683 (-0.021771) | 1.733805 / 1.452155 (0.281651) | 1.961252 / 1.492716 (0.468536) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.296126 / 0.018006 (0.278120) | 0.602169 / 0.000490 (0.601680) | 0.000454 / 0.000200 (0.000254) | 0.000075 / 0.000054 (0.000021) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032970 / 0.037411 (-0.004442) | 0.124071 / 0.014526 (0.109545) | 0.143800 / 0.176557 (-0.032757) | 0.227168 / 0.737135 (-0.509967) | 0.142817 / 0.296338 (-0.153521) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.626239 / 0.215209 (0.411030) | 6.438629 / 2.077655 (4.360974) | 2.760747 / 1.504120 (1.256627) | 2.355419 / 1.541195 (0.814224) | 2.384924 / 1.468490 (0.916434) | 1.210543 / 4.584777 (-3.374234) | 5.440389 / 3.745712 (1.694677) | 5.047939 / 5.269862 (-0.221922) | 2.759618 / 4.565676 (-1.806059) | 0.132757 / 0.424275 (-0.291518) | 0.013163 / 0.007607 (0.005556) | 0.745721 / 0.226044 (0.519677) | 7.660327 / 2.268929 (5.391398) | 3.559385 / 55.444624 (-51.885240) | 2.764344 / 6.876477 (-4.112133) | 2.975274 / 2.142072 (0.833202) | 1.460346 / 4.805227 (-3.344881) | 0.257222 / 6.500664 (-6.243443) | 0.081106 / 0.075469 (0.005637) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.698245 / 1.841788 (-0.143543) | 18.754129 / 8.074308 (10.679821) | 19.065596 / 10.191392 (8.874204) | 0.228237 / 0.680424 (-0.452187) | 0.030688 / 0.534201 (-0.503513) | 0.532561 / 0.579283 (-0.046722) | 0.601133 / 0.434364 (0.166769) | 0.620218 / 0.540337 (0.079881) | 0.751392 / 1.386936 (-0.635545) |

![](https://cml.dev/watermark.png#5f293ff23853fea210388bbef11d1621e54f22e7 "CML watermark")

(the BadZipFile error is unrelated to the changes)

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009368 / 0.011353 (-0.001984) | 0.005143 / 0.011008 (-0.005865) | 0.100675 / 0.038508 (0.062167) | 0.036033 / 0.023109 (0.012924) | 0.297391 / 0.275898 (0.021493) | 0.362230 / 0.323480 (0.038750) | 0.008041 / 0.007986 (0.000055) | 0.004041 / 0.004328 (-0.000287) | 0.075395 / 0.004250 (0.071144) | 0.043020 / 0.037052 (0.005968) | 0.308936 / 0.258489 (0.050447) | 0.343723 / 0.293841 (0.049883) | 0.038416 / 0.128546 (-0.090131) | 0.012086 / 0.075646 (-0.063560) | 0.335102 / 0.419271 (-0.084170) | 0.047718 / 0.043533 (0.004185) | 0.297856 / 0.255139 (0.042717) | 0.317326 / 0.283200 (0.034126) | 0.101462 / 0.141683 (-0.040221) | 1.459965 / 1.452155 (0.007810) | 1.491194 / 1.492716 (-0.001522) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.211311 / 0.018006 (0.193305) | 0.443663 / 0.000490 (0.443174) | 0.003654 / 0.000200 (0.003454) | 0.000076 / 0.000054 (0.000022) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027316 / 0.037411 (-0.010095) | 0.109929 / 0.014526 (0.095403) | 0.117170 / 0.176557 (-0.059387) | 0.182494 / 0.737135 (-0.554641) | 0.124693 / 0.296338 (-0.171646) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.395904 / 0.215209 (0.180695) | 3.950906 / 2.077655 (1.873251) | 1.768807 / 1.504120 (0.264687) | 1.578979 / 1.541195 (0.037784) | 1.689976 / 1.468490 (0.221486) | 0.696458 / 4.584777 (-3.888319) | 3.750491 / 3.745712 (0.004778) | 2.117863 / 5.269862 (-3.151998) | 1.340403 / 4.565676 (-3.225274) | 0.085752 / 0.424275 (-0.338523) | 0.012206 / 0.007607 (0.004599) | 0.505561 / 0.226044 (0.279517) | 5.048721 / 2.268929 (2.779792) | 2.256623 / 55.444624 (-53.188001) | 1.905912 / 6.876477 (-4.970565) | 1.988400 / 2.142072 (-0.153672) | 0.843066 / 4.805227 (-3.962161) | 0.165717 / 6.500664 (-6.334947) | 0.062910 / 0.075469 (-0.012559) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.225668 / 1.841788 (-0.616120) | 14.660082 / 8.074308 (6.585773) | 14.295369 / 10.191392 (4.103977) | 0.171075 / 0.680424 (-0.509348) | 0.029279 / 0.534201 (-0.504922) | 0.441559 / 0.579283 (-0.137724) | 0.445382 / 0.434364 (0.011018) | 0.525350 / 0.540337 (-0.014987) | 0.608493 / 1.386936 (-0.778443) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007288 / 0.011353 (-0.004065) | 0.004999 / 0.011008 (-0.006009) | 0.074656 / 0.038508 (0.036147) | 0.033897 / 0.023109 (0.010788) | 0.345826 / 0.275898 (0.069928) | 0.390891 / 0.323480 (0.067411) | 0.005811 / 0.007986 (-0.002174) | 0.003976 / 0.004328 (-0.000353) | 0.073546 / 0.004250 (0.069295) | 0.047245 / 0.037052 (0.010193) | 0.351851 / 0.258489 (0.093362) | 0.403217 / 0.293841 (0.109376) | 0.036771 / 0.128546 (-0.091775) | 0.012240 / 0.075646 (-0.063407) | 0.086720 / 0.419271 (-0.332552) | 0.049440 / 0.043533 (0.005907) | 0.339520 / 0.255139 (0.084381) | 0.372160 / 0.283200 (0.088961) | 0.100813 / 0.141683 (-0.040870) | 1.436436 / 1.452155 (-0.015718) | 1.514723 / 1.492716 (0.022007) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.231394 / 0.018006 (0.213388) | 0.440825 / 0.000490 (0.440336) | 0.000994 / 0.000200 (0.000794) | 0.000077 / 0.000054 (0.000022) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028999 / 0.037411 (-0.008412) | 0.111391 / 0.014526 (0.096865) | 0.123058 / 0.176557 (-0.053498) | 0.194348 / 0.737135 (-0.542787) | 0.125730 / 0.296338 (-0.170609) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.431950 / 0.215209 (0.216741) | 4.298724 / 2.077655 (2.221069) | 2.064116 / 1.504120 (0.559996) | 1.892062 / 1.541195 (0.350867) | 1.985441 / 1.468490 (0.516951) | 0.707028 / 4.584777 (-3.877749) | 3.812976 / 3.745712 (0.067264) | 3.078704 / 5.269862 (-2.191158) | 1.832737 / 4.565676 (-2.732939) | 0.086182 / 0.424275 (-0.338093) | 0.012289 / 0.007607 (0.004681) | 0.530265 / 0.226044 (0.304220) | 5.283122 / 2.268929 (3.014194) | 2.558491 / 55.444624 (-52.886134) | 2.237046 / 6.876477 (-4.639431) | 2.354548 / 2.142072 (0.212475) | 0.848947 / 4.805227 (-3.956280) | 0.167907 / 6.500664 (-6.332757) | 0.064998 / 0.075469 (-0.010471) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.248287 / 1.841788 (-0.593500) | 14.976327 / 8.074308 (6.902019) | 13.596143 / 10.191392 (3.404751) | 0.145730 / 0.680424 (-0.534694) | 0.017340 / 0.534201 (-0.516861) | 0.430111 / 0.579283 (-0.149172) | 0.433462 / 0.434364 (-0.000902) | 0.540365 / 0.540337 (0.000028) | 0.650586 / 1.386936 (-0.736350) |

![](https://cml.dev/watermark.png#1875c8a4c928aeaccc826f13ffdbf7543112024d "CML watermark")

This PR modifies `map` to:
* ensure the TQDM bar gets the last progress update
* when a map function fails, avoid throwing a chained exception in the single-proc mode


Fix map suffix_template

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.011596 / 0.011353 (0.000244) | 0.005845 / 0.011008 (-0.005164) | 0.121302 / 0.038508 (0.082794) | 0.034306 / 0.023109 (0.011196) | 0.355973 / 0.275898 (0.080075) | 0.419903 / 0.323480 (0.096423) | 0.009049 / 0.007986 (0.001064) | 0.004245 / 0.004328 (-0.000084) | 0.092004 / 0.004250 (0.087753) | 0.042782 / 0.037052 (0.005730) | 0.355805 / 0.258489 (0.097316) | 0.407298 / 0.293841 (0.113457) | 0.052481 / 0.128546 (-0.076066) | 0.020880 / 0.075646 (-0.054766) | 0.379948 / 0.419271 (-0.039324) | 0.061337 / 0.043533 (0.017804) | 0.359829 / 0.255139 (0.104690) | 0.379244 / 0.283200 (0.096044) | 0.116692 / 0.141683 (-0.024990) | 1.733717 / 1.452155 (0.281562) | 1.700246 / 1.492716 (0.207530) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.014622 / 0.018006 (-0.003384) | 0.518777 / 0.000490 (0.518288) | 0.004086 / 0.000200 (0.003886) | 0.000136 / 0.000054 (0.000082) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031208 / 0.037411 (-0.006204) | 0.143003 / 0.014526 (0.128477) | 0.132625 / 0.176557 (-0.043932) | 0.187681 / 0.737135 (-0.549455) | 0.136576 / 0.296338 (-0.159763) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.626516 / 0.215209 (0.411307) | 6.282558 / 2.077655 (4.204904) | 2.702686 / 1.504120 (1.198566) | 2.287445 / 1.541195 (0.746250) | 2.333014 / 1.468490 (0.864524) | 1.227815 / 4.584777 (-3.356962) | 5.545640 / 3.745712 (1.799928) | 4.953226 / 5.269862 (-0.316635) | 2.774549 / 4.565676 (-1.791128) | 0.145257 / 0.424275 (-0.279018) | 0.014887 / 0.007607 (0.007280) | 0.812226 / 0.226044 (0.586182) | 8.002727 / 2.268929 (5.733798) | 3.314852 / 55.444624 (-52.129773) | 2.602348 / 6.876477 (-4.274128) | 2.593511 / 2.142072 (0.451438) | 1.440498 / 4.805227 (-3.364730) | 0.254849 / 6.500664 (-6.245815) | 0.077020 / 0.075469 (0.001551) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.487633 / 1.841788 (-0.354155) | 17.385773 / 8.074308 (9.311465) | 21.775511 / 10.191392 (11.584118) | 0.273514 / 0.680424 (-0.406910) | 0.059644 / 0.534201 (-0.474557) | 0.578710 / 0.579283 (-0.000573) | 0.630221 / 0.434364 (0.195857) | 0.632089 / 0.540337 (0.091752) | 0.762367 / 1.386936 (-0.624569) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009513 / 0.011353 (-0.001840) | 0.006009 / 0.011008 (-0.004999) | 0.087589 / 0.038508 (0.049081) | 0.037487 / 0.023109 (0.014378) | 0.397660 / 0.275898 (0.121762) | 0.474438 / 0.323480 (0.150958) | 0.007373 / 0.007986 (-0.000613) | 0.005839 / 0.004328 (0.001511) | 0.092759 / 0.004250 (0.088509) | 0.052128 / 0.037052 (0.015075) | 0.382378 / 0.258489 (0.123889) | 0.458244 / 0.293841 (0.164403) | 0.057232 / 0.128546 (-0.071314) | 0.020662 / 0.075646 (-0.054984) | 0.110314 / 0.419271 (-0.308957) | 0.063014 / 0.043533 (0.019481) | 0.386020 / 0.255139 (0.130881) | 0.476169 / 0.283200 (0.192970) | 0.118081 / 0.141683 (-0.023602) | 1.724158 / 1.452155 (0.272003) | 1.862257 / 1.492716 (0.369541) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.224288 / 0.018006 (0.206281) | 0.523631 / 0.000490 (0.523141) | 0.004420 / 0.000200 (0.004220) | 0.000127 / 0.000054 (0.000073) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032359 / 0.037411 (-0.005052) | 0.140045 / 0.014526 (0.125519) | 0.138164 / 0.176557 (-0.038393) | 0.181068 / 0.737135 (-0.556067) | 0.143965 / 0.296338 (-0.152374) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.573809 / 0.215209 (0.358600) | 6.083247 / 2.077655 (4.005592) | 2.671258 / 1.504120 (1.167138) | 2.277062 / 1.541195 (0.735868) | 2.299544 / 1.468490 (0.831054) | 1.267351 / 4.584777 (-3.317425) | 5.494461 / 3.745712 (1.748749) | 5.083169 / 5.269862 (-0.186692) | 2.531738 / 4.565676 (-2.033938) | 0.151834 / 0.424275 (-0.272441) | 0.014123 / 0.007607 (0.006516) | 0.800222 / 0.226044 (0.574177) | 7.637624 / 2.268929 (5.368695) | 3.325574 / 55.444624 (-52.119050) | 2.563008 / 6.876477 (-4.313468) | 2.596259 / 2.142072 (0.454187) | 1.459206 / 4.805227 (-3.346021) | 0.237771 / 6.500664 (-6.262893) | 0.071854 / 0.075469 (-0.003615) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.605504 / 1.841788 (-0.236284) | 17.593594 / 8.074308 (9.519285) | 20.618005 / 10.191392 (10.426612) | 0.270938 / 0.680424 (-0.409486) | 0.026205 / 0.534201 (-0.507996) | 0.562223 / 0.579283 (-0.017060) | 0.617571 / 0.434364 (0.183207) | 0.616398 / 0.540337 (0.076060) | 0.715293 / 1.386936 (-0.671643) |

![](https://cml.dev/watermark.png#673dc0dd7d063b2313f7adcc9e0be53d4718f5cf "CML watermark")

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.013213 / 0.011353 (0.001860) | 0.006253 / 0.011008 (-0.004756) | 0.125175 / 0.038508 (0.086667) | 0.037491 / 0.023109 (0.014382) | 0.401379 / 0.275898 (0.125481) | 0.395826 / 0.323480 (0.072346) | 0.009224 / 0.007986 (0.001238) | 0.005163 / 0.004328 (0.000835) | 0.096490 / 0.004250 (0.092239) | 0.042473 / 0.037052 (0.005420) | 0.383713 / 0.258489 (0.125224) | 0.429234 / 0.293841 (0.135393) | 0.063261 / 0.128546 (-0.065285) | 0.020114 / 0.075646 (-0.055532) | 0.401687 / 0.419271 (-0.017585) | 0.062831 / 0.043533 (0.019298) | 0.405211 / 0.255139 (0.150072) | 0.380810 / 0.283200 (0.097610) | 0.109166 / 0.141683 (-0.032517) | 1.869580 / 1.452155 (0.417426) | 1.949947 / 1.492716 (0.457231) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.207481 / 0.018006 (0.189475) | 0.504161 / 0.000490 (0.503671) | 0.008429 / 0.000200 (0.008229) | 0.000101 / 0.000054 (0.000047) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029182 / 0.037411 (-0.008229) | 0.126284 / 0.014526 (0.111758) | 0.140381 / 0.176557 (-0.036175) | 0.175878 / 0.737135 (-0.561257) | 0.138824 / 0.296338 (-0.157514) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.643658 / 0.215209 (0.428449) | 6.396224 / 2.077655 (4.318569) | 2.600702 / 1.504120 (1.096582) | 2.176721 / 1.541195 (0.635526) | 2.216116 / 1.468490 (0.747626) | 1.235069 / 4.584777 (-3.349708) | 5.457228 / 3.745712 (1.711516) | 3.060455 / 5.269862 (-2.209407) | 2.028123 / 4.565676 (-2.537554) | 0.141617 / 0.424275 (-0.282658) | 0.016596 / 0.007607 (0.008989) | 0.804915 / 0.226044 (0.578870) | 7.968821 / 2.268929 (5.699893) | 3.340650 / 55.444624 (-52.103974) | 2.533620 / 6.876477 (-4.342856) | 2.457388 / 2.142072 (0.315315) | 1.486527 / 4.805227 (-3.318700) | 0.253767 / 6.500664 (-6.246897) | 0.082192 / 0.075469 (0.006723) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.470896 / 1.841788 (-0.370892) | 17.566637 / 8.074308 (9.492329) | 23.144148 / 10.191392 (12.952756) | 0.235510 / 0.680424 (-0.444913) | 0.046051 / 0.534201 (-0.488150) | 0.559954 / 0.579283 (-0.019329) | 0.645390 / 0.434364 (0.211026) | 0.690983 / 0.540337 (0.150646) | 0.776252 / 1.386936 (-0.610684) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010564 / 0.011353 (-0.000789) | 0.006150 / 0.011008 (-0.004858) | 0.100030 / 0.038508 (0.061522) | 0.036873 / 0.023109 (0.013764) | 0.448508 / 0.275898 (0.172610) | 0.492593 / 0.323480 (0.169113) | 0.007337 / 0.007986 (-0.000648) | 0.004804 / 0.004328 (0.000475) | 0.099218 / 0.004250 (0.094967) | 0.055513 / 0.037052 (0.018461) | 0.462147 / 0.258489 (0.203658) | 0.510229 / 0.293841 (0.216388) | 0.055307 / 0.128546 (-0.073239) | 0.021989 / 0.075646 (-0.053657) | 0.118487 / 0.419271 (-0.300785) | 0.071752 / 0.043533 (0.028219) | 0.456572 / 0.255139 (0.201433) | 0.475160 / 0.283200 (0.191961) | 0.117472 / 0.141683 (-0.024211) | 1.813212 / 1.452155 (0.361058) | 1.908413 / 1.492716 (0.415696) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.352929 / 0.018006 (0.334923) | 0.543874 / 0.000490 (0.543384) | 0.078529 / 0.000200 (0.078329) | 0.000669 / 0.000054 (0.000614) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033157 / 0.037411 (-0.004254) | 0.162503 / 0.014526 (0.147977) | 0.146424 / 0.176557 (-0.030132) | 0.201781 / 0.737135 (-0.535354) | 0.168110 / 0.296338 (-0.128229) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.644205 / 0.215209 (0.428996) | 6.327519 / 2.077655 (4.249865) | 2.728102 / 1.504120 (1.223982) | 2.306426 / 1.541195 (0.765232) | 2.373125 / 1.468490 (0.904635) | 1.350649 / 4.584777 (-3.234128) | 5.652714 / 3.745712 (1.907002) | 3.175335 / 5.269862 (-2.094526) | 2.222902 / 4.565676 (-2.342775) | 0.160609 / 0.424275 (-0.263666) | 0.015596 / 0.007607 (0.007989) | 0.790357 / 0.226044 (0.564313) | 8.289758 / 2.268929 (6.020830) | 3.479215 / 55.444624 (-51.965410) | 2.860063 / 6.876477 (-4.016413) | 2.806720 / 2.142072 (0.664648) | 1.639046 / 4.805227 (-3.166181) | 0.267017 / 6.500664 (-6.233648) | 0.083990 / 0.075469 (0.008521) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.632262 / 1.841788 (-0.209525) | 17.794357 / 8.074308 (9.720049) | 21.203547 / 10.191392 (11.012155) | 0.250899 / 0.680424 (-0.429525) | 0.024502 / 0.534201 (-0.509699) | 0.519960 / 0.579283 (-0.059323) | 0.615412 / 0.434364 (0.181048) | 0.641914 / 0.540337 (0.101577) | 0.772355 / 1.386936 (-0.614581) |

![](https://cml.dev/watermark.png#32cc4d10243b0feb69650f007d010971fd861dc1 "CML watermark")

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009501 / 0.011353 (-0.001852) | 0.005262 / 0.011008 (-0.005747) | 0.100809 / 0.038508 (0.062301) | 0.036601 / 0.023109 (0.013492) | 0.299612 / 0.275898 (0.023714) | 0.366970 / 0.323480 (0.043490) | 0.007879 / 0.007986 (-0.000107) | 0.004216 / 0.004328 (-0.000113) | 0.076749 / 0.004250 (0.072498) | 0.042081 / 0.037052 (0.005029) | 0.299572 / 0.258489 (0.041083) | 0.339687 / 0.293841 (0.045846) | 0.038706 / 0.128546 (-0.089840) | 0.012295 / 0.075646 (-0.063352) | 0.336172 / 0.419271 (-0.083100) | 0.047524 / 0.043533 (0.003992) | 0.296800 / 0.255139 (0.041661) | 0.331592 / 0.283200 (0.048393) | 0.101191 / 0.141683 (-0.040491) | 1.486200 / 1.452155 (0.034046) | 1.509955 / 1.492716 (0.017239) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.204735 / 0.018006 (0.186728) | 0.446381 / 0.000490 (0.445891) | 0.005177 / 0.000200 (0.004977) | 0.000099 / 0.000054 (0.000045) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028655 / 0.037411 (-0.008756) | 0.116559 / 0.014526 (0.102033) | 0.122551 / 0.176557 (-0.054006) | 0.189764 / 0.737135 (-0.547372) | 0.126446 / 0.296338 (-0.169892) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.400104 / 0.215209 (0.184895) | 4.001524 / 2.077655 (1.923869) | 1.779267 / 1.504120 (0.275147) | 1.580168 / 1.541195 (0.038974) | 1.684100 / 1.468490 (0.215610) | 0.703354 / 4.584777 (-3.881423) | 3.828131 / 3.745712 (0.082419) | 2.098500 / 5.269862 (-3.171362) | 1.331161 / 4.565676 (-3.234516) | 0.085417 / 0.424275 (-0.338858) | 0.012380 / 0.007607 (0.004772) | 0.504189 / 0.226044 (0.278144) | 5.094672 / 2.268929 (2.825743) | 2.264352 / 55.444624 (-53.180272) | 1.909573 / 6.876477 (-4.966904) | 2.005425 / 2.142072 (-0.136648) | 0.840893 / 4.805227 (-3.964335) | 0.164689 / 6.500664 (-6.335975) | 0.062754 / 0.075469 (-0.012715) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.250001 / 1.841788 (-0.591786) | 14.993313 / 8.074308 (6.919005) | 14.880601 / 10.191392 (4.689209) | 0.175141 / 0.680424 (-0.505283) | 0.028952 / 0.534201 (-0.505249) | 0.447073 / 0.579283 (-0.132210) | 0.445993 / 0.434364 (0.011629) | 0.525527 / 0.540337 (-0.014811) | 0.613156 / 1.386936 (-0.773780) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007796 / 0.011353 (-0.003557) | 0.005399 / 0.011008 (-0.005609) | 0.078240 / 0.038508 (0.039732) | 0.035303 / 0.023109 (0.012193) | 0.364603 / 0.275898 (0.088705) | 0.400794 / 0.323480 (0.077314) | 0.006152 / 0.007986 (-0.001834) | 0.004324 / 0.004328 (-0.000004) | 0.074949 / 0.004250 (0.070698) | 0.051939 / 0.037052 (0.014887) | 0.377079 / 0.258489 (0.118590) | 0.413630 / 0.293841 (0.119789) | 0.037567 / 0.128546 (-0.090979) | 0.012793 / 0.075646 (-0.062854) | 0.089013 / 0.419271 (-0.330258) | 0.050748 / 0.043533 (0.007215) | 0.370100 / 0.255139 (0.114961) | 0.384838 / 0.283200 (0.101638) | 0.105840 / 0.141683 (-0.035843) | 1.476490 / 1.452155 (0.024335) | 1.544688 / 1.492716 (0.051972) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.220987 / 0.018006 (0.202981) | 0.443801 / 0.000490 (0.443311) | 0.005747 / 0.000200 (0.005547) | 0.000106 / 0.000054 (0.000051) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030187 / 0.037411 (-0.007225) | 0.118230 / 0.014526 (0.103704) | 0.126810 / 0.176557 (-0.049746) | 0.200482 / 0.737135 (-0.536654) | 0.130831 / 0.296338 (-0.165507) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.423231 / 0.215209 (0.208022) | 4.196576 / 2.077655 (2.118921) | 1.992919 / 1.504120 (0.488799) | 1.809172 / 1.541195 (0.267977) | 1.932706 / 1.468490 (0.464216) | 0.727319 / 4.584777 (-3.857458) | 3.833295 / 3.745712 (0.087583) | 3.527005 / 5.269862 (-1.742857) | 1.937348 / 4.565676 (-2.628329) | 0.088713 / 0.424275 (-0.335562) | 0.012711 / 0.007607 (0.005104) | 0.531385 / 0.226044 (0.305341) | 5.308051 / 2.268929 (3.039123) | 2.493494 / 55.444624 (-52.951131) | 2.168359 / 6.876477 (-4.708118) | 2.258160 / 2.142072 (0.116088) | 0.865629 / 4.805227 (-3.939598) | 0.171281 / 6.500664 (-6.329383) | 0.065746 / 0.075469 (-0.009723) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.290378 / 1.841788 (-0.551409) | 15.900804 / 8.074308 (7.826496) | 14.809614 / 10.191392 (4.618222) | 0.177287 / 0.680424 (-0.503137) | 0.017875 / 0.534201 (-0.516326) | 0.429646 / 0.579283 (-0.149637) | 0.451646 / 0.434364 (0.017282) | 0.545669 / 0.540337 (0.005332) | 0.633215 / 1.386936 (-0.753721) |

![](https://cml.dev/watermark.png#2c67b5f4bc9cea088e977a135644d38da8c144ff "CML watermark")

#5455 introduced a small bug that lead `map` to ignore the `suffix_template` argument and not put suffixes to cached files in multiprocessing.

I fixed this and also improved a few things:

- regarding logging: "Loading cached processed dataset" is now logged only once even in multiprocessing (it used to be logged `num_proc` times)
- regarding new_fingerprint: I made sure that the returned dataset satisfies `ds._fingerprint==new_fingerprint` if `new_fingerprint` is passed to `map`


Remove instructions for `ffmpeg` system package installation on Colab

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.014525 / 0.011353 (0.003172) | 0.006871 / 0.011008 (-0.004137) | 0.135577 / 0.038508 (0.097069) | 0.039620 / 0.023109 (0.016511) | 0.499829 / 0.275898 (0.223931) | 0.571000 / 0.323480 (0.247520) | 0.009726 / 0.007986 (0.001740) | 0.005654 / 0.004328 (0.001325) | 0.104732 / 0.004250 (0.100482) | 0.046849 / 0.037052 (0.009796) | 0.486667 / 0.258489 (0.228178) | 0.543611 / 0.293841 (0.249770) | 0.056414 / 0.128546 (-0.072133) | 0.019974 / 0.075646 (-0.055672) | 0.484878 / 0.419271 (0.065606) | 0.059244 / 0.043533 (0.015711) | 0.490046 / 0.255139 (0.234907) | 0.517427 / 0.283200 (0.234227) | 0.114692 / 0.141683 (-0.026991) | 1.935935 / 1.452155 (0.483780) | 1.990253 / 1.492716 (0.497537) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.271008 / 0.018006 (0.253002) | 0.610964 / 0.000490 (0.610474) | 0.013423 / 0.000200 (0.013223) | 0.000523 / 0.000054 (0.000468) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031940 / 0.037411 (-0.005472) | 0.130755 / 0.014526 (0.116229) | 0.146616 / 0.176557 (-0.029941) | 0.239386 / 0.737135 (-0.497749) | 0.146612 / 0.296338 (-0.149726) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.675383 / 0.215209 (0.460174) | 6.656828 / 2.077655 (4.579174) | 2.741231 / 1.504120 (1.237111) | 2.232921 / 1.541195 (0.691726) | 2.172116 / 1.468490 (0.703626) | 1.221623 / 4.584777 (-3.363154) | 5.683653 / 3.745712 (1.937941) | 5.344137 / 5.269862 (0.074275) | 2.969670 / 4.565676 (-1.596006) | 0.142107 / 0.424275 (-0.282168) | 0.015808 / 0.007607 (0.008201) | 0.767366 / 0.226044 (0.541321) | 8.059605 / 2.268929 (5.790676) | 3.333535 / 55.444624 (-52.111089) | 2.669619 / 6.876477 (-4.206857) | 2.652989 / 2.142072 (0.510917) | 1.526397 / 4.805227 (-3.278830) | 0.265609 / 6.500664 (-6.235055) | 0.082759 / 0.075469 (0.007290) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.631086 / 1.841788 (-0.210701) | 18.701351 / 8.074308 (10.627043) | 22.843802 / 10.191392 (12.652410) | 0.240134 / 0.680424 (-0.440290) | 0.046683 / 0.534201 (-0.487518) | 0.576488 / 0.579283 (-0.002795) | 0.650123 / 0.434364 (0.215759) | 0.661190 / 0.540337 (0.120853) | 0.759563 / 1.386936 (-0.627373) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009883 / 0.011353 (-0.001470) | 0.006692 / 0.011008 (-0.004316) | 0.098550 / 0.038508 (0.060042) | 0.035188 / 0.023109 (0.012078) | 0.463535 / 0.275898 (0.187637) | 0.472762 / 0.323480 (0.149282) | 0.007199 / 0.007986 (-0.000787) | 0.007961 / 0.004328 (0.003632) | 0.093140 / 0.004250 (0.088890) | 0.051752 / 0.037052 (0.014700) | 0.453412 / 0.258489 (0.194922) | 0.502741 / 0.293841 (0.208900) | 0.056006 / 0.128546 (-0.072540) | 0.020164 / 0.075646 (-0.055482) | 0.116828 / 0.419271 (-0.302444) | 0.067205 / 0.043533 (0.023672) | 0.442715 / 0.255139 (0.187576) | 0.472525 / 0.283200 (0.189326) | 0.122767 / 0.141683 (-0.018915) | 1.881366 / 1.452155 (0.429212) | 1.978786 / 1.492716 (0.486069) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.284180 / 0.018006 (0.266174) | 0.601556 / 0.000490 (0.601067) | 0.008455 / 0.000200 (0.008255) | 0.000112 / 0.000054 (0.000057) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033515 / 0.037411 (-0.003896) | 0.136407 / 0.014526 (0.121881) | 0.143341 / 0.176557 (-0.033215) | 0.225394 / 0.737135 (-0.511741) | 0.153343 / 0.296338 (-0.142995) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.688202 / 0.215209 (0.472993) | 6.576502 / 2.077655 (4.498847) | 2.839175 / 1.504120 (1.335055) | 2.481152 / 1.541195 (0.939957) | 2.617227 / 1.468490 (1.148736) | 1.314854 / 4.584777 (-3.269922) | 5.805950 / 3.745712 (2.060238) | 3.188930 / 5.269862 (-2.080932) | 2.141719 / 4.565676 (-2.423957) | 0.145069 / 0.424275 (-0.279206) | 0.014567 / 0.007607 (0.006960) | 0.780000 / 0.226044 (0.553955) | 7.898016 / 2.268929 (5.629088) | 3.549060 / 55.444624 (-51.895564) | 2.856569 / 6.876477 (-4.019907) | 3.117719 / 2.142072 (0.975647) | 1.512560 / 4.805227 (-3.292668) | 0.262689 / 6.500664 (-6.237975) | 0.085979 / 0.075469 (0.010509) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.623550 / 1.841788 (-0.218238) | 19.597063 / 8.074308 (11.522755) | 21.293369 / 10.191392 (11.101977) | 0.263780 / 0.680424 (-0.416643) | 0.027289 / 0.534201 (-0.506912) | 0.560361 / 0.579283 (-0.018922) | 0.646288 / 0.434364 (0.211924) | 0.712699 / 0.540337 (0.172361) | 0.818332 / 1.386936 (-0.568604) |

![](https://cml.dev/watermark.png#b304de5dde30c945ec1397d3b4fe86f3b323ca8b "CML watermark")

Colab now has Ubuntu 20.04 which already has `ffmpeg` of required (>4) version.