Release: 2.14.7

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004943 / 0.011353 (-0.006410) | 0.002900 / 0.011008 (-0.008109) | 0.061495 / 0.038508 (0.022987) | 0.053575 / 0.023109 (0.030466) | 0.249318 / 0.275898 (-0.026580) | 0.271773 / 0.323480 (-0.051706) | 0.003074 / 0.007986 (-0.004911) | 0.003738 / 0.004328 (-0.000590) | 0.047624 / 0.004250 (0.043373) | 0.045141 / 0.037052 (0.008089) | 0.255467 / 0.258489 (-0.003022) | 0.286577 / 0.293841 (-0.007264) | 0.023113 / 0.128546 (-0.105433) | 0.007189 / 0.075646 (-0.068458) | 0.204441 / 0.419271 (-0.214830) | 0.036829 / 0.043533 (-0.006704) | 0.252474 / 0.255139 (-0.002665) | 0.270960 / 0.283200 (-0.012239) | 0.019666 / 0.141683 (-0.122017) | 1.095139 / 1.452155 (-0.357015) | 1.158659 / 1.492716 (-0.334057) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091046 / 0.018006 (0.073040) | 0.298346 / 0.000490 (0.297856) | 0.000215 / 0.000200 (0.000015) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018702 / 0.037411 (-0.018709) | 0.062213 / 0.014526 (0.047687) | 0.073364 / 0.176557 (-0.103193) | 0.119841 / 0.737135 (-0.617294) | 0.074070 / 0.296338 (-0.222268) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282388 / 0.215209 (0.067179) | 2.792029 / 2.077655 (0.714375) | 1.471483 / 1.504120 (-0.032637) | 1.386236 / 1.541195 (-0.154959) | 1.377489 / 1.468490 (-0.091001) | 0.410335 / 4.584777 (-4.174442) | 2.424866 / 3.745712 (-1.320846) | 2.610609 / 5.269862 (-2.659253) | 1.574636 / 4.565676 (-2.991041) | 0.046716 / 0.424275 (-0.377559) | 0.004768 / 0.007607 (-0.002839) | 0.339831 / 0.226044 (0.113787) | 3.297579 / 2.268929 (1.028651) | 1.851410 / 55.444624 (-53.593214) | 1.550048 / 6.876477 (-5.326428) | 1.576647 / 2.142072 (-0.565425) | 0.482538 / 4.805227 (-4.322689) | 0.101381 / 6.500664 (-6.399283) | 0.042066 / 0.075469 (-0.033403) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.972664 / 1.841788 (-0.869123) | 11.580700 / 8.074308 (3.506392) | 10.586747 / 10.191392 (0.395355) | 0.127844 / 0.680424 (-0.552580) | 0.014270 / 0.534201 (-0.519931) | 0.269678 / 0.579283 (-0.309605) | 0.264022 / 0.434364 (-0.170342) | 0.309395 / 0.540337 (-0.230942) | 0.429228 / 1.386936 (-0.957708) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004815 / 0.011353 (-0.006538) | 0.002890 / 0.011008 (-0.008119) | 0.048039 / 0.038508 (0.009531) | 0.053029 / 0.023109 (0.029920) | 0.271346 / 0.275898 (-0.004552) | 0.294488 / 0.323480 (-0.028992) | 0.003983 / 0.007986 (-0.004003) | 0.002439 / 0.004328 (-0.001889) | 0.048250 / 0.004250 (0.044000) | 0.038855 / 0.037052 (0.001803) | 0.284723 / 0.258489 (0.026234) | 0.303604 / 0.293841 (0.009763) | 0.024384 / 0.128546 (-0.104163) | 0.007021 / 0.075646 (-0.068625) | 0.053850 / 0.419271 (-0.365422) | 0.032177 / 0.043533 (-0.011356) | 0.270039 / 0.255139 (0.014900) | 0.289669 / 0.283200 (0.006469) | 0.018840 / 0.141683 (-0.122842) | 1.122191 / 1.452155 (-0.329963) | 1.187083 / 1.492716 (-0.305634) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090609 / 0.018006 (0.072603) | 0.298915 / 0.000490 (0.298425) | 0.000216 / 0.000200 (0.000016) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.020919 / 0.037411 (-0.016492) | 0.070474 / 0.014526 (0.055948) | 0.082421 / 0.176557 (-0.094135) | 0.126967 / 0.737135 (-0.610168) | 0.083447 / 0.296338 (-0.212892) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.300153 / 0.215209 (0.084944) | 2.958992 / 2.077655 (0.881337) | 1.631228 / 1.504120 (0.127108) | 1.497991 / 1.541195 (-0.043204) | 1.536963 / 1.468490 (0.068473) | 0.403047 / 4.584777 (-4.181730) | 2.448782 / 3.745712 (-1.296930) | 2.571954 / 5.269862 (-2.697908) | 1.556346 / 4.565676 (-3.009331) | 0.045992 / 0.424275 (-0.378283) | 0.004785 / 0.007607 (-0.002822) | 0.357448 / 0.226044 (0.131404) | 3.558808 / 2.268929 (1.289880) | 1.992624 / 55.444624 (-53.452001) | 1.695027 / 6.876477 (-5.181450) | 1.695183 / 2.142072 (-0.446889) | 0.477001 / 4.805227 (-4.328226) | 0.097485 / 6.500664 (-6.403179) | 0.040530 / 0.075469 (-0.034939) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.976342 / 1.841788 (-0.865445) | 12.141698 / 8.074308 (4.067390) | 10.881101 / 10.191392 (0.689709) | 0.142443 / 0.680424 (-0.537981) | 0.015583 / 0.534201 (-0.518618) | 0.269727 / 0.579283 (-0.309556) | 0.275890 / 0.434364 (-0.158474) | 0.306351 / 0.540337 (-0.233987) | 0.412003 / 1.386936 (-0.974933) |

![](https://cml.dev/watermark.png#7c744261000fd684f54c54de8ac4f15a726092d7 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004946 / 0.011353 (-0.006407) | 0.002863 / 0.011008 (-0.008146) | 0.061888 / 0.038508 (0.023380) | 0.050664 / 0.023109 (0.027554) | 0.242635 / 0.275898 (-0.033263) | 0.271741 / 0.323480 (-0.051739) | 0.003023 / 0.007986 (-0.004963) | 0.003088 / 0.004328 (-0.001241) | 0.049286 / 0.004250 (0.045036) | 0.044699 / 0.037052 (0.007647) | 0.249581 / 0.258489 (-0.008908) | 0.285633 / 0.293841 (-0.008208) | 0.023048 / 0.128546 (-0.105499) | 0.007235 / 0.075646 (-0.068412) | 0.202989 / 0.419271 (-0.216282) | 0.036357 / 0.043533 (-0.007175) | 0.245980 / 0.255139 (-0.009159) | 0.277486 / 0.283200 (-0.005713) | 0.019215 / 0.141683 (-0.122468) | 1.096456 / 1.452155 (-0.355699) | 1.152196 / 1.492716 (-0.340520) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092026 / 0.018006 (0.074020) | 0.303038 / 0.000490 (0.302549) | 0.000209 / 0.000200 (0.000009) | 0.000048 / 0.000054 (-0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018670 / 0.037411 (-0.018741) | 0.061972 / 0.014526 (0.047446) | 0.072963 / 0.176557 (-0.103594) | 0.119984 / 0.737135 (-0.617151) | 0.074074 / 0.296338 (-0.222265) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282444 / 0.215209 (0.067235) | 2.754571 / 2.077655 (0.676916) | 1.482635 / 1.504120 (-0.021485) | 1.352039 / 1.541195 (-0.189155) | 1.359333 / 1.468490 (-0.109157) | 0.399690 / 4.584777 (-4.185087) | 2.364844 / 3.745712 (-1.380868) | 2.603942 / 5.269862 (-2.665919) | 1.569512 / 4.565676 (-2.996164) | 0.046074 / 0.424275 (-0.378201) | 0.004745 / 0.007607 (-0.002862) | 0.339066 / 0.226044 (0.113022) | 3.363456 / 2.268929 (1.094527) | 1.822213 / 55.444624 (-53.622411) | 1.536622 / 6.876477 (-5.339854) | 1.574772 / 2.142072 (-0.567300) | 0.474418 / 4.805227 (-4.330809) | 0.099572 / 6.500664 (-6.401092) | 0.041824 / 0.075469 (-0.033645) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.956300 / 1.841788 (-0.885487) | 11.648886 / 8.074308 (3.574578) | 10.645700 / 10.191392 (0.454308) | 0.138924 / 0.680424 (-0.541499) | 0.013936 / 0.534201 (-0.520265) | 0.270319 / 0.579283 (-0.308964) | 0.269735 / 0.434364 (-0.164629) | 0.309699 / 0.540337 (-0.230639) | 0.429139 / 1.386936 (-0.957797) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004838 / 0.011353 (-0.006515) | 0.002937 / 0.011008 (-0.008072) | 0.048094 / 0.038508 (0.009586) | 0.053131 / 0.023109 (0.030022) | 0.271893 / 0.275898 (-0.004005) | 0.291025 / 0.323480 (-0.032454) | 0.004058 / 0.007986 (-0.003928) | 0.002410 / 0.004328 (-0.001919) | 0.047939 / 0.004250 (0.043689) | 0.038996 / 0.037052 (0.001944) | 0.274983 / 0.258489 (0.016494) | 0.306175 / 0.293841 (0.012334) | 0.024388 / 0.128546 (-0.104159) | 0.007242 / 0.075646 (-0.068404) | 0.054011 / 0.419271 (-0.365261) | 0.032750 / 0.043533 (-0.010783) | 0.271147 / 0.255139 (0.016008) | 0.288163 / 0.283200 (0.004963) | 0.018383 / 0.141683 (-0.123299) | 1.116134 / 1.452155 (-0.336021) | 1.185964 / 1.492716 (-0.306752) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093289 / 0.018006 (0.075283) | 0.303058 / 0.000490 (0.302568) | 0.000241 / 0.000200 (0.000041) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021422 / 0.037411 (-0.015990) | 0.069974 / 0.014526 (0.055449) | 0.081164 / 0.176557 (-0.095392) | 0.119991 / 0.737135 (-0.617144) | 0.082154 / 0.296338 (-0.214184) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.292298 / 0.215209 (0.077089) | 2.851475 / 2.077655 (0.773821) | 1.558283 / 1.504120 (0.054163) | 1.432431 / 1.541195 (-0.108764) | 1.479282 / 1.468490 (0.010792) | 0.413124 / 4.584777 (-4.171653) | 2.473005 / 3.745712 (-1.272707) | 2.548779 / 5.269862 (-2.721082) | 1.520776 / 4.565676 (-3.044900) | 0.046476 / 0.424275 (-0.377799) | 0.004814 / 0.007607 (-0.002794) | 0.347036 / 0.226044 (0.120992) | 3.424928 / 2.268929 (1.155999) | 1.963274 / 55.444624 (-53.481351) | 1.653794 / 6.876477 (-5.222683) | 1.643874 / 2.142072 (-0.498198) | 0.469086 / 4.805227 (-4.336141) | 0.097417 / 6.500664 (-6.403247) | 0.040468 / 0.075469 (-0.035002) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.972783 / 1.841788 (-0.869005) | 12.122994 / 8.074308 (4.048686) | 10.876396 / 10.191392 (0.685004) | 0.130573 / 0.680424 (-0.549850) | 0.016693 / 0.534201 (-0.517508) | 0.270952 / 0.579283 (-0.308331) | 0.273834 / 0.434364 (-0.160530) | 0.305049 / 0.540337 (-0.235289) | 0.408776 / 1.386936 (-0.978160) |

![](https://cml.dev/watermark.png#e4216e5d57ea07e6b1ed73a3ec2cf845c6e59f70 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004606 / 0.011353 (-0.006747) | 0.002433 / 0.011008 (-0.008576) | 0.061985 / 0.038508 (0.023477) | 0.048853 / 0.023109 (0.025744) | 0.244506 / 0.275898 (-0.031392) | 0.270159 / 0.323480 (-0.053321) | 0.003962 / 0.007986 (-0.004024) | 0.002376 / 0.004328 (-0.001952) | 0.048067 / 0.004250 (0.043817) | 0.041864 / 0.037052 (0.004812) | 0.249743 / 0.258489 (-0.008746) | 0.287723 / 0.293841 (-0.006117) | 0.022954 / 0.128546 (-0.105593) | 0.006845 / 0.075646 (-0.068801) | 0.206313 / 0.419271 (-0.212959) | 0.035780 / 0.043533 (-0.007753) | 0.244286 / 0.255139 (-0.010853) | 0.270026 / 0.283200 (-0.013173) | 0.018177 / 0.141683 (-0.123506) | 1.083998 / 1.452155 (-0.368157) | 1.156086 / 1.492716 (-0.336630) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093754 / 0.018006 (0.075748) | 0.302157 / 0.000490 (0.301667) | 0.000215 / 0.000200 (0.000015) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018745 / 0.037411 (-0.018666) | 0.061707 / 0.014526 (0.047181) | 0.074356 / 0.176557 (-0.102200) | 0.121643 / 0.737135 (-0.615492) | 0.075885 / 0.296338 (-0.220454) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.289156 / 0.215209 (0.073947) | 2.881327 / 2.077655 (0.803672) | 1.483568 / 1.504120 (-0.020552) | 1.355933 / 1.541195 (-0.185262) | 1.389693 / 1.468490 (-0.078797) | 0.402834 / 4.584777 (-4.181943) | 2.390634 / 3.745712 (-1.355078) | 2.596761 / 5.269862 (-2.673101) | 1.527602 / 4.565676 (-3.038074) | 0.046434 / 0.424275 (-0.377841) | 0.004783 / 0.007607 (-0.002824) | 0.341017 / 0.226044 (0.114972) | 3.429023 / 2.268929 (1.160095) | 1.832988 / 55.444624 (-53.611637) | 1.526510 / 6.876477 (-5.349967) | 1.539382 / 2.142072 (-0.602690) | 0.475734 / 4.805227 (-4.329493) | 0.098710 / 6.500664 (-6.401954) | 0.041136 / 0.075469 (-0.034333) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.922023 / 1.841788 (-0.919765) | 11.428215 / 8.074308 (3.353907) | 10.356668 / 10.191392 (0.165276) | 0.139575 / 0.680424 (-0.540848) | 0.014541 / 0.534201 (-0.519660) | 0.271359 / 0.579283 (-0.307924) | 0.266701 / 0.434364 (-0.167663) | 0.309449 / 0.540337 (-0.230888) | 0.422047 / 1.386936 (-0.964889) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004892 / 0.011353 (-0.006461) | 0.002792 / 0.011008 (-0.008216) | 0.048027 / 0.038508 (0.009519) | 0.059256 / 0.023109 (0.036147) | 0.270150 / 0.275898 (-0.005748) | 0.294530 / 0.323480 (-0.028950) | 0.004162 / 0.007986 (-0.003823) | 0.002470 / 0.004328 (-0.001858) | 0.047993 / 0.004250 (0.043743) | 0.040380 / 0.037052 (0.003328) | 0.275247 / 0.258489 (0.016758) | 0.305684 / 0.293841 (0.011843) | 0.025072 / 0.128546 (-0.103474) | 0.007183 / 0.075646 (-0.068463) | 0.054875 / 0.419271 (-0.364397) | 0.033053 / 0.043533 (-0.010480) | 0.271281 / 0.255139 (0.016142) | 0.288057 / 0.283200 (0.004858) | 0.018692 / 0.141683 (-0.122991) | 1.125224 / 1.452155 (-0.326930) | 1.171083 / 1.492716 (-0.321633) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.103102 / 0.018006 (0.085096) | 0.309099 / 0.000490 (0.308609) | 0.000232 / 0.000200 (0.000032) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021532 / 0.037411 (-0.015879) | 0.069927 / 0.014526 (0.055401) | 0.080920 / 0.176557 (-0.095637) | 0.122214 / 0.737135 (-0.614921) | 0.082268 / 0.296338 (-0.214071) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.298121 / 0.215209 (0.082912) | 2.933000 / 2.077655 (0.855345) | 1.608782 / 1.504120 (0.104662) | 1.554083 / 1.541195 (0.012889) | 1.552700 / 1.468490 (0.084209) | 0.400576 / 4.584777 (-4.184201) | 2.412914 / 3.745712 (-1.332798) | 2.545706 / 5.269862 (-2.724155) | 1.548797 / 4.565676 (-3.016879) | 0.045553 / 0.424275 (-0.378722) | 0.004751 / 0.007607 (-0.002857) | 0.343002 / 0.226044 (0.116958) | 3.402866 / 2.268929 (1.133937) | 1.969910 / 55.444624 (-53.474715) | 1.686639 / 6.876477 (-5.189838) | 1.768474 / 2.142072 (-0.373599) | 0.471299 / 4.805227 (-4.333928) | 0.097696 / 6.500664 (-6.402968) | 0.041693 / 0.075469 (-0.033776) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.971380 / 1.841788 (-0.870408) | 12.686033 / 8.074308 (4.611725) | 11.370946 / 10.191392 (1.179554) | 0.138377 / 0.680424 (-0.542047) | 0.015623 / 0.534201 (-0.518578) | 0.270935 / 0.579283 (-0.308348) | 0.276235 / 0.434364 (-0.158129) | 0.310196 / 0.540337 (-0.230141) | 0.416908 / 1.386936 (-0.970028) |

![](https://cml.dev/watermark.png#bf02cff8d70180a9e89328961ded9e3d8510fd22 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004581 / 0.011353 (-0.006772) | 0.002468 / 0.011008 (-0.008541) | 0.061420 / 0.038508 (0.022912) | 0.047685 / 0.023109 (0.024575) | 0.237756 / 0.275898 (-0.038142) | 0.267548 / 0.323480 (-0.055932) | 0.003899 / 0.007986 (-0.004086) | 0.002338 / 0.004328 (-0.001990) | 0.048794 / 0.004250 (0.044543) | 0.042485 / 0.037052 (0.005433) | 0.250165 / 0.258489 (-0.008324) | 0.278791 / 0.293841 (-0.015050) | 0.022371 / 0.128546 (-0.106175) | 0.006923 / 0.075646 (-0.068723) | 0.201401 / 0.419271 (-0.217870) | 0.035867 / 0.043533 (-0.007665) | 0.244628 / 0.255139 (-0.010511) | 0.271137 / 0.283200 (-0.012063) | 0.017257 / 0.141683 (-0.124426) | 1.097261 / 1.452155 (-0.354894) | 1.163314 / 1.492716 (-0.329402) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.089060 / 0.018006 (0.071054) | 0.297489 / 0.000490 (0.296999) | 0.000207 / 0.000200 (0.000007) | 0.000050 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018583 / 0.037411 (-0.018828) | 0.061974 / 0.014526 (0.047449) | 0.073300 / 0.176557 (-0.103256) | 0.118871 / 0.737135 (-0.618264) | 0.075513 / 0.296338 (-0.220826) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.285544 / 0.215209 (0.070335) | 2.799871 / 2.077655 (0.722216) | 1.479871 / 1.504120 (-0.024249) | 1.351128 / 1.541195 (-0.190067) | 1.377540 / 1.468490 (-0.090950) | 0.393056 / 4.584777 (-4.191721) | 2.341791 / 3.745712 (-1.403921) | 2.546854 / 5.269862 (-2.723007) | 1.547368 / 4.565676 (-3.018309) | 0.046056 / 0.424275 (-0.378219) | 0.004765 / 0.007607 (-0.002842) | 0.336384 / 0.226044 (0.110339) | 3.283277 / 2.268929 (1.014348) | 1.784535 / 55.444624 (-53.660089) | 1.557809 / 6.876477 (-5.318667) | 1.581728 / 2.142072 (-0.560344) | 0.470527 / 4.805227 (-4.334700) | 0.098383 / 6.500664 (-6.402281) | 0.041563 / 0.075469 (-0.033906) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.946924 / 1.841788 (-0.894863) | 11.202775 / 8.074308 (3.128467) | 10.249760 / 10.191392 (0.058368) | 0.142337 / 0.680424 (-0.538087) | 0.013784 / 0.534201 (-0.520417) | 0.267237 / 0.579283 (-0.312046) | 0.264142 / 0.434364 (-0.170222) | 0.306343 / 0.540337 (-0.233994) | 0.423681 / 1.386936 (-0.963255) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004786 / 0.011353 (-0.006567) | 0.002398 / 0.011008 (-0.008610) | 0.047325 / 0.038508 (0.008817) | 0.050753 / 0.023109 (0.027644) | 0.271132 / 0.275898 (-0.004766) | 0.290854 / 0.323480 (-0.032626) | 0.003953 / 0.007986 (-0.004033) | 0.002238 / 0.004328 (-0.002090) | 0.047463 / 0.004250 (0.043213) | 0.038504 / 0.037052 (0.001451) | 0.273182 / 0.258489 (0.014693) | 0.303449 / 0.293841 (0.009608) | 0.024069 / 0.128546 (-0.104477) | 0.006712 / 0.075646 (-0.068934) | 0.053032 / 0.419271 (-0.366239) | 0.032221 / 0.043533 (-0.011312) | 0.271770 / 0.255139 (0.016631) | 0.287876 / 0.283200 (0.004677) | 0.018040 / 0.141683 (-0.123643) | 1.138749 / 1.452155 (-0.313405) | 1.192048 / 1.492716 (-0.300668) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.089132 / 0.018006 (0.071126) | 0.298636 / 0.000490 (0.298146) | 0.000220 / 0.000200 (0.000020) | 0.000053 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.020808 / 0.037411 (-0.016603) | 0.069506 / 0.014526 (0.054980) | 0.079412 / 0.176557 (-0.097145) | 0.118188 / 0.737135 (-0.618947) | 0.083044 / 0.296338 (-0.213294) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.293502 / 0.215209 (0.078293) | 2.863692 / 2.077655 (0.786037) | 1.590877 / 1.504120 (0.086757) | 1.483634 / 1.541195 (-0.057561) | 1.502113 / 1.468490 (0.033623) | 0.402170 / 4.584777 (-4.182607) | 2.414188 / 3.745712 (-1.331524) | 2.500146 / 5.269862 (-2.769716) | 1.506977 / 4.565676 (-3.058699) | 0.045849 / 0.424275 (-0.378426) | 0.004755 / 0.007607 (-0.002852) | 0.343073 / 0.226044 (0.117029) | 3.354985 / 2.268929 (1.086056) | 1.952594 / 55.444624 (-53.492030) | 1.664084 / 6.876477 (-5.212392) | 1.664203 / 2.142072 (-0.477869) | 0.475858 / 4.805227 (-4.329370) | 0.097539 / 6.500664 (-6.403125) | 0.040201 / 0.075469 (-0.035268) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.980051 / 1.841788 (-0.861736) | 11.615291 / 8.074308 (3.540983) | 10.492092 / 10.191392 (0.300700) | 0.130450 / 0.680424 (-0.549974) | 0.015883 / 0.534201 (-0.518318) | 0.267575 / 0.579283 (-0.311708) | 0.276981 / 0.434364 (-0.157383) | 0.310221 / 0.540337 (-0.230116) | 0.417143 / 1.386936 (-0.969793) |

![](https://cml.dev/watermark.png#bf02cff8d70180a9e89328961ded9e3d8510fd22 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004721 / 0.011353 (-0.006632) | 0.002931 / 0.011008 (-0.008077) | 0.061948 / 0.038508 (0.023440) | 0.051066 / 0.023109 (0.027957) | 0.245431 / 0.275898 (-0.030467) | 0.295627 / 0.323480 (-0.027852) | 0.003997 / 0.007986 (-0.003988) | 0.002408 / 0.004328 (-0.001920) | 0.048292 / 0.004250 (0.044041) | 0.044716 / 0.037052 (0.007664) | 0.255119 / 0.258489 (-0.003371) | 0.287384 / 0.293841 (-0.006457) | 0.022835 / 0.128546 (-0.105711) | 0.007162 / 0.075646 (-0.068484) | 0.201352 / 0.419271 (-0.217920) | 0.036626 / 0.043533 (-0.006906) | 0.249590 / 0.255139 (-0.005549) | 0.270822 / 0.283200 (-0.012378) | 0.018152 / 0.141683 (-0.123531) | 1.097046 / 1.452155 (-0.355109) | 1.160461 / 1.492716 (-0.332255) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091712 / 0.018006 (0.073705) | 0.299121 / 0.000490 (0.298631) | 0.000244 / 0.000200 (0.000044) | 0.000055 / 0.000054 (0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018998 / 0.037411 (-0.018413) | 0.062811 / 0.014526 (0.048285) | 0.076348 / 0.176557 (-0.100209) | 0.123898 / 0.737135 (-0.613238) | 0.076249 / 0.296338 (-0.220090) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282780 / 0.215209 (0.067571) | 2.739028 / 2.077655 (0.661373) | 1.472564 / 1.504120 (-0.031556) | 1.347343 / 1.541195 (-0.193852) | 1.387130 / 1.468490 (-0.081360) | 0.403348 / 4.584777 (-4.181429) | 2.369924 / 3.745712 (-1.375788) | 2.612875 / 5.269862 (-2.656987) | 1.588079 / 4.565676 (-2.977598) | 0.045233 / 0.424275 (-0.379042) | 0.004767 / 0.007607 (-0.002840) | 0.336614 / 0.226044 (0.110570) | 3.300485 / 2.268929 (1.031556) | 1.834365 / 55.444624 (-53.610259) | 1.559799 / 6.876477 (-5.316677) | 1.601265 / 2.142072 (-0.540808) | 0.468158 / 4.805227 (-4.337069) | 0.099811 / 6.500664 (-6.400853) | 0.042688 / 0.075469 (-0.032782) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.934097 / 1.841788 (-0.907691) | 11.687713 / 8.074308 (3.613405) | 10.412723 / 10.191392 (0.221331) | 0.139276 / 0.680424 (-0.541148) | 0.014042 / 0.534201 (-0.520159) | 0.270306 / 0.579283 (-0.308978) | 0.266609 / 0.434364 (-0.167755) | 0.314179 / 0.540337 (-0.226158) | 0.437744 / 1.386936 (-0.949192) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004893 / 0.011353 (-0.006460) | 0.002952 / 0.011008 (-0.008056) | 0.050441 / 0.038508 (0.011933) | 0.051838 / 0.023109 (0.028729) | 0.271163 / 0.275898 (-0.004735) | 0.293031 / 0.323480 (-0.030449) | 0.003976 / 0.007986 (-0.004010) | 0.002396 / 0.004328 (-0.001933) | 0.048103 / 0.004250 (0.043852) | 0.038732 / 0.037052 (0.001680) | 0.274276 / 0.258489 (0.015787) | 0.305112 / 0.293841 (0.011271) | 0.024112 / 0.128546 (-0.104434) | 0.007203 / 0.075646 (-0.068443) | 0.053502 / 0.419271 (-0.365770) | 0.032360 / 0.043533 (-0.011173) | 0.270154 / 0.255139 (0.015015) | 0.286689 / 0.283200 (0.003489) | 0.018285 / 0.141683 (-0.123397) | 1.141421 / 1.452155 (-0.310734) | 1.244062 / 1.492716 (-0.248654) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090960 / 0.018006 (0.072954) | 0.286134 / 0.000490 (0.285644) | 0.000207 / 0.000200 (0.000007) | 0.000045 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.020789 / 0.037411 (-0.016622) | 0.070850 / 0.014526 (0.056324) | 0.080750 / 0.176557 (-0.095807) | 0.120046 / 0.737135 (-0.617089) | 0.083630 / 0.296338 (-0.212708) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.290654 / 0.215209 (0.075445) | 2.846669 / 2.077655 (0.769014) | 1.561752 / 1.504120 (0.057632) | 1.442968 / 1.541195 (-0.098227) | 1.503551 / 1.468490 (0.035061) | 0.399731 / 4.584777 (-4.185046) | 2.430099 / 3.745712 (-1.315613) | 2.556169 / 5.269862 (-2.713692) | 1.545591 / 4.565676 (-3.020085) | 0.045967 / 0.424275 (-0.378309) | 0.004851 / 0.007607 (-0.002756) | 0.340167 / 0.226044 (0.114122) | 3.392738 / 2.268929 (1.123809) | 1.943577 / 55.444624 (-53.501047) | 1.650057 / 6.876477 (-5.226420) | 1.686872 / 2.142072 (-0.455201) | 0.470305 / 4.805227 (-4.334923) | 0.097296 / 6.500664 (-6.403368) | 0.041399 / 0.075469 (-0.034070) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.985660 / 1.841788 (-0.856128) | 12.300826 / 8.074308 (4.226518) | 10.972591 / 10.191392 (0.781199) | 0.131512 / 0.680424 (-0.548912) | 0.015742 / 0.534201 (-0.518459) | 0.270630 / 0.579283 (-0.308653) | 0.276039 / 0.434364 (-0.158325) | 0.302288 / 0.540337 (-0.238050) | 0.409415 / 1.386936 (-0.977521) |

![](https://cml.dev/watermark.png#bf02cff8d70180a9e89328961ded9e3d8510fd22 "CML watermark")

Release 2.14.7.


Remove token value from warnings

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005135 / 0.011353 (-0.006218) | 0.002950 / 0.011008 (-0.008058) | 0.062316 / 0.038508 (0.023808) | 0.030068 / 0.023109 (0.006959) | 0.251998 / 0.275898 (-0.023900) | 0.274806 / 0.323480 (-0.048674) | 0.003067 / 0.007986 (-0.004919) | 0.003082 / 0.004328 (-0.001247) | 0.048503 / 0.004250 (0.044253) | 0.045167 / 0.037052 (0.008114) | 0.254277 / 0.258489 (-0.004212) | 0.290528 / 0.293841 (-0.003313) | 0.023666 / 0.128546 (-0.104880) | 0.007049 / 0.075646 (-0.068597) | 0.202367 / 0.419271 (-0.216905) | 0.056291 / 0.043533 (0.012758) | 0.251923 / 0.255139 (-0.003216) | 0.273595 / 0.283200 (-0.009605) | 0.019065 / 0.141683 (-0.122618) | 1.100832 / 1.452155 (-0.351322) | 1.266758 / 1.492716 (-0.225959) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094311 / 0.018006 (0.076305) | 0.303199 / 0.000490 (0.302709) | 0.000238 / 0.000200 (0.000039) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019413 / 0.037411 (-0.017999) | 0.062618 / 0.014526 (0.048092) | 0.072850 / 0.176557 (-0.103707) | 0.119124 / 0.737135 (-0.618012) | 0.074044 / 0.296338 (-0.222294) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.273660 / 0.215209 (0.058451) | 2.682371 / 2.077655 (0.604716) | 1.426041 / 1.504120 (-0.078079) | 1.317186 / 1.541195 (-0.224009) | 1.332385 / 1.468490 (-0.136106) | 0.394599 / 4.584777 (-4.190178) | 2.368167 / 3.745712 (-1.377545) | 2.683728 / 5.269862 (-2.586134) | 1.668348 / 4.565676 (-2.897329) | 0.046177 / 0.424275 (-0.378098) | 0.004833 / 0.007607 (-0.002774) | 0.331413 / 0.226044 (0.105369) | 3.278984 / 2.268929 (1.010055) | 1.797600 / 55.444624 (-53.647024) | 1.492202 / 6.876477 (-5.384274) | 1.536039 / 2.142072 (-0.606034) | 0.470601 / 4.805227 (-4.334626) | 0.100833 / 6.500664 (-6.399831) | 0.042787 / 0.075469 (-0.032682) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.959036 / 1.841788 (-0.882752) | 11.632956 / 8.074308 (3.558648) | 10.384574 / 10.191392 (0.193182) | 0.127477 / 0.680424 (-0.552946) | 0.014072 / 0.534201 (-0.520129) | 0.269534 / 0.579283 (-0.309749) | 0.259753 / 0.434364 (-0.174611) | 0.313450 / 0.540337 (-0.226888) | 0.431799 / 1.386936 (-0.955137) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004964 / 0.011353 (-0.006389) | 0.002906 / 0.011008 (-0.008102) | 0.048145 / 0.038508 (0.009637) | 0.056457 / 0.023109 (0.033348) | 0.274131 / 0.275898 (-0.001767) | 0.298534 / 0.323480 (-0.024946) | 0.004145 / 0.007986 (-0.003841) | 0.002415 / 0.004328 (-0.001913) | 0.048558 / 0.004250 (0.044308) | 0.039031 / 0.037052 (0.001978) | 0.278948 / 0.258489 (0.020459) | 0.312358 / 0.293841 (0.018517) | 0.024902 / 0.128546 (-0.103645) | 0.007286 / 0.075646 (-0.068360) | 0.053839 / 0.419271 (-0.365433) | 0.032510 / 0.043533 (-0.011023) | 0.272023 / 0.255139 (0.016884) | 0.293420 / 0.283200 (0.010221) | 0.018932 / 0.141683 (-0.122750) | 1.122792 / 1.452155 (-0.329362) | 1.167385 / 1.492716 (-0.325331) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094574 / 0.018006 (0.076567) | 0.303810 / 0.000490 (0.303321) | 0.000227 / 0.000200 (0.000027) | 0.000056 / 0.000054 (0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021675 / 0.037411 (-0.015737) | 0.070289 / 0.014526 (0.055763) | 0.080345 / 0.176557 (-0.096211) | 0.120220 / 0.737135 (-0.616915) | 0.084080 / 0.296338 (-0.212259) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.300134 / 0.215209 (0.084925) | 2.945831 / 2.077655 (0.868176) | 1.605303 / 1.504120 (0.101183) | 1.480135 / 1.541195 (-0.061059) | 1.526039 / 1.468490 (0.057549) | 0.398264 / 4.584777 (-4.186512) | 2.461391 / 3.745712 (-1.284321) | 2.559929 / 5.269862 (-2.709933) | 1.541391 / 4.565676 (-3.024286) | 0.045319 / 0.424275 (-0.378957) | 0.004834 / 0.007607 (-0.002773) | 0.352186 / 0.226044 (0.126141) | 3.500108 / 2.268929 (1.231180) | 1.966394 / 55.444624 (-53.478230) | 1.675500 / 6.876477 (-5.200977) | 1.683134 / 2.142072 (-0.458938) | 0.465085 / 4.805227 (-4.340142) | 0.097235 / 6.500664 (-6.403429) | 0.040764 / 0.075469 (-0.034705) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.982813 / 1.841788 (-0.858975) | 12.382529 / 8.074308 (4.308221) | 11.082660 / 10.191392 (0.891268) | 0.129113 / 0.680424 (-0.551310) | 0.015718 / 0.534201 (-0.518483) | 0.272776 / 0.579283 (-0.306507) | 0.275513 / 0.434364 (-0.158850) | 0.304933 / 0.540337 (-0.235404) | 0.414591 / 1.386936 (-0.972345) |

![](https://cml.dev/watermark.png#8723b129a64928eba40baf70ffd462060ade9f97 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004400 / 0.011353 (-0.006953) | 0.002580 / 0.011008 (-0.008428) | 0.060975 / 0.038508 (0.022467) | 0.029337 / 0.023109 (0.006228) | 0.248643 / 0.275898 (-0.027255) | 0.274476 / 0.323480 (-0.049004) | 0.003925 / 0.007986 (-0.004061) | 0.002332 / 0.004328 (-0.001997) | 0.049501 / 0.004250 (0.045251) | 0.042730 / 0.037052 (0.005678) | 0.255823 / 0.258489 (-0.002666) | 0.281748 / 0.293841 (-0.012093) | 0.023118 / 0.128546 (-0.105428) | 0.006957 / 0.075646 (-0.068690) | 0.201630 / 0.419271 (-0.217641) | 0.054258 / 0.043533 (0.010725) | 0.252289 / 0.255139 (-0.002850) | 0.267561 / 0.283200 (-0.015639) | 0.016903 / 0.141683 (-0.124780) | 1.104322 / 1.452155 (-0.347833) | 1.160027 / 1.492716 (-0.332689) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096340 / 0.018006 (0.078333) | 0.305187 / 0.000490 (0.304697) | 0.000222 / 0.000200 (0.000022) | 0.000050 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018733 / 0.037411 (-0.018678) | 0.062382 / 0.014526 (0.047856) | 0.072309 / 0.176557 (-0.104248) | 0.119772 / 0.737135 (-0.617364) | 0.074655 / 0.296338 (-0.221683) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.286150 / 0.215209 (0.070941) | 2.770328 / 2.077655 (0.692673) | 1.494593 / 1.504120 (-0.009527) | 1.358611 / 1.541195 (-0.182583) | 1.396308 / 1.468490 (-0.072182) | 0.394806 / 4.584777 (-4.189971) | 2.349100 / 3.745712 (-1.396613) | 2.600541 / 5.269862 (-2.669321) | 1.568975 / 4.565676 (-2.996701) | 0.046212 / 0.424275 (-0.378063) | 0.004821 / 0.007607 (-0.002786) | 0.332286 / 0.226044 (0.106242) | 3.302643 / 2.268929 (1.033714) | 1.838992 / 55.444624 (-53.605633) | 1.571919 / 6.876477 (-5.304557) | 1.574956 / 2.142072 (-0.567117) | 0.464156 / 4.805227 (-4.341071) | 0.097983 / 6.500664 (-6.402681) | 0.042243 / 0.075469 (-0.033226) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.941675 / 1.841788 (-0.900113) | 11.450326 / 8.074308 (3.376017) | 10.169943 / 10.191392 (-0.021449) | 0.137879 / 0.680424 (-0.542545) | 0.013765 / 0.534201 (-0.520436) | 0.268633 / 0.579283 (-0.310650) | 0.265083 / 0.434364 (-0.169281) | 0.302099 / 0.540337 (-0.238238) | 0.423033 / 1.386936 (-0.963903) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004998 / 0.011353 (-0.006355) | 0.003174 / 0.011008 (-0.007834) | 0.047924 / 0.038508 (0.009416) | 0.057598 / 0.023109 (0.034489) | 0.278823 / 0.275898 (0.002925) | 0.334349 / 0.323480 (0.010869) | 0.004053 / 0.007986 (-0.003932) | 0.002554 / 0.004328 (-0.001774) | 0.047797 / 0.004250 (0.043547) | 0.039802 / 0.037052 (0.002749) | 0.278295 / 0.258489 (0.019806) | 0.319597 / 0.293841 (0.025757) | 0.024802 / 0.128546 (-0.103744) | 0.007362 / 0.075646 (-0.068284) | 0.066983 / 0.419271 (-0.352288) | 0.032707 / 0.043533 (-0.010826) | 0.277350 / 0.255139 (0.022211) | 0.296829 / 0.283200 (0.013629) | 0.017902 / 0.141683 (-0.123781) | 1.129765 / 1.452155 (-0.322390) | 1.201940 / 1.492716 (-0.290777) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095631 / 0.018006 (0.077625) | 0.296999 / 0.000490 (0.296510) | 0.000234 / 0.000200 (0.000034) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021547 / 0.037411 (-0.015865) | 0.070003 / 0.014526 (0.055477) | 0.083173 / 0.176557 (-0.093384) | 0.121676 / 0.737135 (-0.615459) | 0.082974 / 0.296338 (-0.213364) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.298982 / 0.215209 (0.083773) | 2.918666 / 2.077655 (0.841011) | 1.582054 / 1.504120 (0.077934) | 1.463804 / 1.541195 (-0.077391) | 1.484384 / 1.468490 (0.015893) | 0.399443 / 4.584777 (-4.185334) | 2.393515 / 3.745712 (-1.352197) | 2.533004 / 5.269862 (-2.736858) | 1.490411 / 4.565676 (-3.075266) | 0.045274 / 0.424275 (-0.379002) | 0.004783 / 0.007607 (-0.002824) | 0.350510 / 0.226044 (0.124465) | 3.437927 / 2.268929 (1.168998) | 1.940115 / 55.444624 (-53.504509) | 1.662025 / 6.876477 (-5.214452) | 1.640621 / 2.142072 (-0.501452) | 0.464014 / 4.805227 (-4.341214) | 0.095506 / 6.500664 (-6.405158) | 0.040172 / 0.075469 (-0.035297) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.975618 / 1.841788 (-0.866169) | 12.561067 / 8.074308 (4.486759) | 11.408037 / 10.191392 (1.216645) | 0.130699 / 0.680424 (-0.549725) | 0.016796 / 0.534201 (-0.517405) | 0.271130 / 0.579283 (-0.308153) | 0.283506 / 0.434364 (-0.150857) | 0.304482 / 0.540337 (-0.235856) | 0.413673 / 1.386936 (-0.973263) |

![](https://cml.dev/watermark.png#723038a73248dd12dc0673d2b341e9295c441ea3 "CML watermark")

Fix #6412


Bug: LayoutLMv3 finetuning on FUNSD Notebook; Arrow Error

Very strange: `datasets-cli env`
>
> Copy-and-paste the text below in your GitHub issue.
>
> - `datasets` version: 2.9.0
> - Platform: macOS-14.0-arm64-arm-64bit
> - Python version: 3.9.13
> - PyArrow version: 8.0.0
> - Pandas version: 1.3.5

After updating datasets and pyarrow on base environment, although I am using a different one called layoutLM

> Copy-and-paste the text below in your GitHub issue.
>
> - `datasets` version: 2.14.6
> - Platform: macOS-14.0-arm64-arm-64bit
> - Python version: 3.9.18
> - Huggingface_hub version: 0.17.3
> - PyArrow version: 14.0.1
> - Pandas version: 2.1.3

> Hi! The latest (patch) release (published a few hours ago) includes a fix for this [PyArrow security issue](https://github.com/advisories/GHSA-5wvp-7f3h-6wmm). To install it, run `pip install -U datasets`.

Thanks for the info and the latest release, it seems this has also solved my issue. First run after the update worked and I am training right now :D
Will close the Issu

### Describe the bug

Arrow issues when running the example Notebook laptop locally on Mac with M1. Works on Google Collab.

**Notebook**: https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv3/Fine_tune_LayoutLMv3_on_FUNSD_(HuggingFace_Trainer).ipynb

**Error**: `ValueError: Arrow type extension<arrow.py_extension_type<pyarrow.lib.UnknownExtensionType>> does not have a datasets dtype equivalent.`

**Caused by**:

```
# we need to define custom features for `set_format` (used later on) to work properly
features = Features({
    'pixel_values': Array3D(dtype="float32", shape=(3, 224, 224)),
    'input_ids': Sequence(feature=Value(dtype='int64')),
    'attention_mask': Sequence(Value(dtype='int64')),
    'bbox': Array2D(dtype="int64", shape=(512, 4)),
    'labels': Sequence(feature=Value(dtype='int64')),
})
```

### Steps to reproduce the bug

Run the notebook provided, locally. If possible also on M1.

### Expected behavior

The cell where features are mapped to Array2D and Array3D should work without any issues.

### Environment info

Tried with Python 3.9 and 3.10 conda envs. Running Mac M1.

`pip show datasets`

> Name: datasets
Version: 2.14.6
Summary: HuggingFace community-driven open-source library of datasets

`pip list`

> Package                   Version
> ------------------------- ------------
> accelerate                0.24.1
> aiohttp                   3.8.6
> aiosignal                 1.3.1
> anyio                     3.5.0
> appnope                   0.1.2
> argon2-cffi               21.3.0
> argon2-cffi-bindings      21.2.0
> asttokens                 2.0.5
> async-timeout             4.0.3
> attrs                     23.1.0
> backcall                  0.2.0
> beautifulsoup4            4.12.2
> bleach                    4.1.0
> certifi                   2023.7.22
> cffi                      1.15.1
> charset-normalizer        3.3.2
> comm                      0.1.2
> datasets                  2.14.6
> debugpy                   1.6.7
> decorator                 5.1.1
> defusedxml                0.7.1
> dill                      0.3.7
> entrypoints               0.4
> exceptiongroup            1.0.4
> executing                 0.8.3
> fastjsonschema            2.16.2
> filelock                  3.13.1
> frozenlist                1.4.0
> fsspec                    2023.10.0
> huggingface-hub           0.17.3
> idna                      3.4
> importlib-metadata        6.0.0
> IProgress                 0.4
> ipykernel                 6.25.0
> ipython                   8.15.0
> ipython-genutils          0.2.0
> jedi                      0.18.1
> Jinja2                    3.1.2
> joblib                    1.3.2
> jsonschema                4.19.2
> jsonschema-specifications 2023.7.1
> jupyter_client            7.4.9
> jupyter_core              5.5.0
> jupyter-server            1.23.4
> jupyterlab-pygments       0.1.2
> MarkupSafe                2.1.1
> matplotlib-inline         0.1.6
> mistune                   2.0.4
> mpmath                    1.3.0
> multidict                 6.0.4
> multiprocess              0.70.15
> nbclassic                 1.0.0
> nbclient                  0.8.0
> nbconvert                 7.10.0
> nbformat                  5.9.2
> nest-asyncio              1.5.6
> networkx                  3.2.1
> notebook                  6.5.4
> notebook_shim             0.2.3
> numpy                     1.26.1
> packaging                 23.1
> pandas                    2.1.3
> pandocfilters             1.5.0
> parso                     0.8.3
> pexpect                   4.8.0
> pickleshare               0.7.5
> Pillow                    10.1.0
> pip                       23.3
> platformdirs              3.10.0
> prometheus-client         0.14.1
> prompt-toolkit            3.0.36
> psutil                    5.9.0
> ptyprocess                0.7.0
> pure-eval                 0.2.2
> pyarrow                   14.0.1
> pycparser                 2.21
> Pygments                  2.15.1
> python-dateutil           2.8.2
> pytz                      2023.3.post1
> PyYAML                    6.0.1
> pyzmq                     23.2.0
> referencing               0.30.2
> regex                     2023.10.3
> requests                  2.31.0
> rpds-py                   0.10.6
> safetensors               0.4.0
> scikit-learn              1.3.2
> scipy                     1.11.3
> Send2Trash                1.8.2
> seqeval                   1.2.2
> setuptools                68.0.0
> six                       1.16.0
> sniffio                   1.2.0
> soupsieve                 2.5
> stack-data                0.2.0
> sympy                     1.12
> terminado                 0.17.1
> threadpoolctl             3.2.0
> tinycss2                  1.2.1
> tokenizers                0.14.1
> torch                     2.1.0
> tornado                   6.3.3
> tqdm                      4.66.1
> traitlets                 5.7.1
> transformers              4.36.0.dev0
> typing_extensions         4.7.1
> tzdata                    2023.3
> urllib3                   2.0.7
> wcwidth                   0.2.5
> webencodings              0.5.1
> websocket-client          0.58.0
> wheel                     0.41.2
> xxhash                    3.4.1
> yarl                      1.9.2
> zipp                      3.11.0


Rename audio_classificiation.py to audio_classification.py

Oh good catch. Can you also rename it in `src/datasets/tasks/__init__.py` ?

Fixed!

(I think, tough word to spell right TBH)

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004737 / 0.011353 (-0.006616) | 0.002446 / 0.011008 (-0.008563) | 0.060928 / 0.038508 (0.022420) | 0.030479 / 0.023109 (0.007370) | 0.238385 / 0.275898 (-0.037513) | 0.265563 / 0.323480 (-0.057917) | 0.002910 / 0.007986 (-0.005076) | 0.002325 / 0.004328 (-0.002004) | 0.047817 / 0.004250 (0.043566) | 0.044243 / 0.037052 (0.007191) | 0.245190 / 0.258489 (-0.013299) | 0.275449 / 0.293841 (-0.018392) | 0.023384 / 0.128546 (-0.105162) | 0.006820 / 0.075646 (-0.068826) | 0.201488 / 0.419271 (-0.217783) | 0.057758 / 0.043533 (0.014225) | 0.245279 / 0.255139 (-0.009860) | 0.266094 / 0.283200 (-0.017106) | 0.019254 / 0.141683 (-0.122429) | 1.107497 / 1.452155 (-0.344658) | 1.161412 / 1.492716 (-0.331304) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094909 / 0.018006 (0.076903) | 0.305185 / 0.000490 (0.304695) | 0.000221 / 0.000200 (0.000021) | 0.000042 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018352 / 0.037411 (-0.019059) | 0.062441 / 0.014526 (0.047915) | 0.072386 / 0.176557 (-0.104171) | 0.118836 / 0.737135 (-0.618299) | 0.074514 / 0.296338 (-0.221824) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.283632 / 0.215209 (0.068423) | 2.751845 / 2.077655 (0.674190) | 1.478620 / 1.504120 (-0.025499) | 1.357221 / 1.541195 (-0.183974) | 1.415297 / 1.468490 (-0.053194) | 0.400093 / 4.584777 (-4.184684) | 2.404607 / 3.745712 (-1.341105) | 2.617572 / 5.269862 (-2.652289) | 1.587622 / 4.565676 (-2.978055) | 0.045997 / 0.424275 (-0.378278) | 0.004872 / 0.007607 (-0.002735) | 0.338901 / 0.226044 (0.112856) | 3.371362 / 2.268929 (1.102434) | 1.870469 / 55.444624 (-53.574155) | 1.561670 / 6.876477 (-5.314807) | 1.573186 / 2.142072 (-0.568886) | 0.478735 / 4.805227 (-4.326492) | 0.098743 / 6.500664 (-6.401921) | 0.041780 / 0.075469 (-0.033689) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.945422 / 1.841788 (-0.896366) | 11.563464 / 8.074308 (3.489156) | 10.368731 / 10.191392 (0.177339) | 0.129910 / 0.680424 (-0.550513) | 0.014014 / 0.534201 (-0.520187) | 0.269036 / 0.579283 (-0.310247) | 0.265516 / 0.434364 (-0.168848) | 0.311082 / 0.540337 (-0.229255) | 0.431510 / 1.386936 (-0.955426) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005068 / 0.011353 (-0.006284) | 0.002989 / 0.011008 (-0.008019) | 0.048213 / 0.038508 (0.009705) | 0.056133 / 0.023109 (0.033024) | 0.283347 / 0.275898 (0.007449) | 0.307505 / 0.323480 (-0.015975) | 0.004041 / 0.007986 (-0.003944) | 0.002477 / 0.004328 (-0.001852) | 0.047771 / 0.004250 (0.043521) | 0.039361 / 0.037052 (0.002309) | 0.283764 / 0.258489 (0.025275) | 0.320644 / 0.293841 (0.026803) | 0.024972 / 0.128546 (-0.103575) | 0.007599 / 0.075646 (-0.068048) | 0.054732 / 0.419271 (-0.364539) | 0.032774 / 0.043533 (-0.010759) | 0.285594 / 0.255139 (0.030455) | 0.301500 / 0.283200 (0.018300) | 0.018181 / 0.141683 (-0.123501) | 1.126311 / 1.452155 (-0.325843) | 1.187147 / 1.492716 (-0.305569) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.097397 / 0.018006 (0.079391) | 0.315112 / 0.000490 (0.314622) | 0.000224 / 0.000200 (0.000024) | 0.000056 / 0.000054 (0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021529 / 0.037411 (-0.015882) | 0.073208 / 0.014526 (0.058682) | 0.081683 / 0.176557 (-0.094874) | 0.120475 / 0.737135 (-0.616660) | 0.083265 / 0.296338 (-0.213073) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.289976 / 0.215209 (0.074767) | 2.839860 / 2.077655 (0.762205) | 1.592635 / 1.504120 (0.088515) | 1.466722 / 1.541195 (-0.074472) | 1.552850 / 1.468490 (0.084360) | 0.418693 / 4.584777 (-4.166084) | 2.526620 / 3.745712 (-1.219093) | 2.706182 / 5.269862 (-2.563680) | 1.618514 / 4.565676 (-2.947162) | 0.046303 / 0.424275 (-0.377972) | 0.004873 / 0.007607 (-0.002734) | 0.345146 / 0.226044 (0.119102) | 3.378448 / 2.268929 (1.109520) | 1.986393 / 55.444624 (-53.458231) | 1.681838 / 6.876477 (-5.194639) | 1.738093 / 2.142072 (-0.403980) | 0.484386 / 4.805227 (-4.320842) | 0.100693 / 6.500664 (-6.399971) | 0.043084 / 0.075469 (-0.032385) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.976399 / 1.841788 (-0.865389) | 13.122968 / 8.074308 (5.048660) | 11.245031 / 10.191392 (1.053639) | 0.134433 / 0.680424 (-0.545991) | 0.017439 / 0.534201 (-0.516762) | 0.274083 / 0.579283 (-0.305200) | 0.287353 / 0.434364 (-0.147011) | 0.309231 / 0.540337 (-0.231106) | 0.418003 / 1.386936 (-0.968933) |

![](https://cml.dev/watermark.png#939f136f255eab68a5bf6441db2a395f8af78511 "CML watermark")