Unpin `tensorflow` maximum version

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006663 / 0.011353 (-0.004690) | 0.004091 / 0.011008 (-0.006918) | 0.084954 / 0.038508 (0.046445) | 0.071869 / 0.023109 (0.048760) | 0.314706 / 0.275898 (0.038808) | 0.352794 / 0.323480 (0.029314) | 0.004027 / 0.007986 (-0.003959) | 0.003371 / 0.004328 (-0.000957) | 0.065456 / 0.004250 (0.061205) | 0.055828 / 0.037052 (0.018775) | 0.316502 / 0.258489 (0.058013) | 0.377979 / 0.293841 (0.084138) | 0.030870 / 0.128546 (-0.097676) | 0.008616 / 0.075646 (-0.067030) | 0.288625 / 0.419271 (-0.130646) | 0.052314 / 0.043533 (0.008781) | 0.322725 / 0.255139 (0.067586) | 0.351810 / 0.283200 (0.068611) | 0.025726 / 0.141683 (-0.115957) | 1.439308 / 1.452155 (-0.012847) | 1.524484 / 1.492716 (0.031768) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.235212 / 0.018006 (0.217206) | 0.444926 / 0.000490 (0.444437) | 0.009887 / 0.000200 (0.009687) | 0.000402 / 0.000054 (0.000347) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028956 / 0.037411 (-0.008455) | 0.084401 / 0.014526 (0.069875) | 0.339686 / 0.176557 (0.163130) | 0.186785 / 0.737135 (-0.550350) | 0.195017 / 0.296338 (-0.101322) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.405480 / 0.215209 (0.190271) | 4.024315 / 2.077655 (1.946661) | 2.056398 / 1.504120 (0.552278) | 1.912099 / 1.541195 (0.370904) | 1.950119 / 1.468490 (0.481629) | 0.486071 / 4.584777 (-4.098706) | 3.578501 / 3.745712 (-0.167211) | 3.268980 / 5.269862 (-2.000881) | 2.018114 / 4.565676 (-2.547563) | 0.057440 / 0.424275 (-0.366835) | 0.007281 / 0.007607 (-0.000326) | 0.474760 / 0.226044 (0.248716) | 4.746908 / 2.268929 (2.477979) | 2.550111 / 55.444624 (-52.894513) | 2.171932 / 6.876477 (-4.704544) | 2.392235 / 2.142072 (0.250162) | 0.585940 / 4.805227 (-4.219287) | 0.136445 / 6.500664 (-6.364219) | 0.062125 / 0.075469 (-0.013344) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.270763 / 1.841788 (-0.571025) | 19.213516 / 8.074308 (11.139208) | 13.992620 / 10.191392 (3.801228) | 0.167356 / 0.680424 (-0.513068) | 0.018261 / 0.534201 (-0.515940) | 0.392489 / 0.579283 (-0.186794) | 0.418845 / 0.434364 (-0.015519) | 0.461824 / 0.540337 (-0.078513) | 0.649661 / 1.386936 (-0.737275) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006675 / 0.011353 (-0.004678) | 0.003913 / 0.011008 (-0.007096) | 0.064943 / 0.038508 (0.026435) | 0.072426 / 0.023109 (0.049317) | 0.400785 / 0.275898 (0.124887) | 0.434359 / 0.323480 (0.110879) | 0.005370 / 0.007986 (-0.002616) | 0.003290 / 0.004328 (-0.001038) | 0.065035 / 0.004250 (0.060785) | 0.054924 / 0.037052 (0.017872) | 0.404442 / 0.258489 (0.145953) | 0.439027 / 0.293841 (0.145186) | 0.032467 / 0.128546 (-0.096080) | 0.008565 / 0.075646 (-0.067081) | 0.070653 / 0.419271 (-0.348619) | 0.048034 / 0.043533 (0.004501) | 0.400869 / 0.255139 (0.145730) | 0.423048 / 0.283200 (0.139848) | 0.022757 / 0.141683 (-0.118926) | 1.516956 / 1.452155 (0.064801) | 1.581599 / 1.492716 (0.088883) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.214761 / 0.018006 (0.196755) | 0.440921 / 0.000490 (0.440431) | 0.007538 / 0.000200 (0.007338) | 0.000087 / 0.000054 (0.000033) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032313 / 0.037411 (-0.005099) | 0.091365 / 0.014526 (0.076839) | 0.106665 / 0.176557 (-0.069891) | 0.158637 / 0.737135 (-0.578498) | 0.104894 / 0.296338 (-0.191445) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.432995 / 0.215209 (0.217786) | 4.339911 / 2.077655 (2.262256) | 2.313139 / 1.504120 (0.809019) | 2.142552 / 1.541195 (0.601357) | 2.279275 / 1.468490 (0.810785) | 0.501133 / 4.584777 (-4.083644) | 3.696160 / 3.745712 (-0.049552) | 3.341886 / 5.269862 (-1.927976) | 2.105972 / 4.565676 (-2.459705) | 0.059268 / 0.424275 (-0.365008) | 0.007568 / 0.007607 (-0.000039) | 0.512546 / 0.226044 (0.286502) | 5.130219 / 2.268929 (2.861290) | 2.808292 / 55.444624 (-52.636332) | 2.478721 / 6.876477 (-4.397755) | 2.679341 / 2.142072 (0.537269) | 0.599022 / 4.805227 (-4.206206) | 0.143761 / 6.500664 (-6.356903) | 0.062061 / 0.075469 (-0.013409) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.430507 / 1.841788 (-0.411281) | 20.458085 / 8.074308 (12.383777) | 15.268356 / 10.191392 (5.076964) | 0.163359 / 0.680424 (-0.517065) | 0.020908 / 0.534201 (-0.513293) | 0.396870 / 0.579283 (-0.182413) | 0.432630 / 0.434364 (-0.001733) | 0.475909 / 0.540337 (-0.064429) | 0.681031 / 1.386936 (-0.705905) |

![](https://cml.dev/watermark.png#fd1dd6aa4c7fa7744c1c1f877573ff59f1529292 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005815 / 0.011353 (-0.005538) | 0.003419 / 0.011008 (-0.007589) | 0.080286 / 0.038508 (0.041778) | 0.056487 / 0.023109 (0.033377) | 0.304414 / 0.275898 (0.028516) | 0.341039 / 0.323480 (0.017559) | 0.004392 / 0.007986 (-0.003594) | 0.002852 / 0.004328 (-0.001477) | 0.062339 / 0.004250 (0.058089) | 0.044683 / 0.037052 (0.007630) | 0.311651 / 0.258489 (0.053162) | 0.357249 / 0.293841 (0.063409) | 0.027300 / 0.128546 (-0.101246) | 0.007963 / 0.075646 (-0.067683) | 0.261948 / 0.419271 (-0.157323) | 0.044952 / 0.043533 (0.001419) | 0.309990 / 0.255139 (0.054851) | 0.340735 / 0.283200 (0.057536) | 0.020786 / 0.141683 (-0.120897) | 1.471378 / 1.452155 (0.019224) | 1.517260 / 1.492716 (0.024543) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.245447 / 0.018006 (0.227441) | 0.418967 / 0.000490 (0.418477) | 0.007039 / 0.000200 (0.006840) | 0.000196 / 0.000054 (0.000142) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022880 / 0.037411 (-0.014532) | 0.071862 / 0.014526 (0.057337) | 0.083009 / 0.176557 (-0.093547) | 0.143414 / 0.737135 (-0.593722) | 0.082896 / 0.296338 (-0.213442) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.390645 / 0.215209 (0.175436) | 3.888104 / 2.077655 (1.810450) | 1.859572 / 1.504120 (0.355452) | 1.683803 / 1.541195 (0.142608) | 1.697902 / 1.468490 (0.229412) | 0.499537 / 4.584777 (-4.085239) | 3.015832 / 3.745712 (-0.729881) | 2.805696 / 5.269862 (-2.464166) | 1.830408 / 4.565676 (-2.735268) | 0.058191 / 0.424275 (-0.366085) | 0.006357 / 0.007607 (-0.001250) | 0.462486 / 0.226044 (0.236442) | 4.634951 / 2.268929 (2.366022) | 2.309364 / 55.444624 (-53.135260) | 1.979521 / 6.876477 (-4.896956) | 2.080011 / 2.142072 (-0.062062) | 0.593086 / 4.805227 (-4.212141) | 0.124856 / 6.500664 (-6.375808) | 0.060172 / 0.075469 (-0.015297) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.251439 / 1.841788 (-0.590349) | 17.068999 / 8.074308 (8.994691) | 13.527209 / 10.191392 (3.335817) | 0.146636 / 0.680424 (-0.533788) | 0.016866 / 0.534201 (-0.517335) | 0.333202 / 0.579283 (-0.246081) | 0.360444 / 0.434364 (-0.073920) | 0.388378 / 0.540337 (-0.151959) | 0.530519 / 1.386936 (-0.856417) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006043 / 0.011353 (-0.005310) | 0.003612 / 0.011008 (-0.007396) | 0.062644 / 0.038508 (0.024135) | 0.056104 / 0.023109 (0.032995) | 0.446328 / 0.275898 (0.170430) | 0.478044 / 0.323480 (0.154564) | 0.004641 / 0.007986 (-0.003345) | 0.002896 / 0.004328 (-0.001432) | 0.062344 / 0.004250 (0.058093) | 0.046339 / 0.037052 (0.009287) | 0.454866 / 0.258489 (0.196377) | 0.484242 / 0.293841 (0.190401) | 0.028602 / 0.128546 (-0.099944) | 0.008075 / 0.075646 (-0.067571) | 0.067980 / 0.419271 (-0.351291) | 0.041339 / 0.043533 (-0.002194) | 0.452911 / 0.255139 (0.197772) | 0.474180 / 0.283200 (0.190981) | 0.019395 / 0.141683 (-0.122288) | 1.432161 / 1.452155 (-0.019993) | 1.505800 / 1.492716 (0.013083) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.216983 / 0.018006 (0.198977) | 0.406232 / 0.000490 (0.405743) | 0.005101 / 0.000200 (0.004902) | 0.000077 / 0.000054 (0.000022) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026295 / 0.037411 (-0.011116) | 0.080490 / 0.014526 (0.065964) | 0.088105 / 0.176557 (-0.088451) | 0.143294 / 0.737135 (-0.593841) | 0.089125 / 0.296338 (-0.207213) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.465512 / 0.215209 (0.250302) | 4.648656 / 2.077655 (2.571002) | 2.598225 / 1.504120 (1.094105) | 2.409588 / 1.541195 (0.868393) | 2.513745 / 1.468490 (1.045255) | 0.507425 / 4.584777 (-4.077352) | 3.130164 / 3.745712 (-0.615548) | 2.836817 / 5.269862 (-2.433045) | 1.836029 / 4.565676 (-2.729647) | 0.058829 / 0.424275 (-0.365446) | 0.006551 / 0.007607 (-0.001056) | 0.537892 / 0.226044 (0.311848) | 5.401079 / 2.268929 (3.132150) | 3.019817 / 55.444624 (-52.424807) | 2.695131 / 6.876477 (-4.181346) | 2.805321 / 2.142072 (0.663248) | 0.595681 / 4.805227 (-4.209546) | 0.124368 / 6.500664 (-6.376296) | 0.060712 / 0.075469 (-0.014757) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.361508 / 1.841788 (-0.480279) | 17.811373 / 8.074308 (9.737065) | 14.482705 / 10.191392 (4.291313) | 0.153193 / 0.680424 (-0.527231) | 0.018347 / 0.534201 (-0.515854) | 0.330900 / 0.579283 (-0.248383) | 0.374948 / 0.434364 (-0.059416) | 0.385615 / 0.540337 (-0.154722) | 0.568077 / 1.386936 (-0.818859) |

![](https://cml.dev/watermark.png#18ef408c21f8efbb2142f050a691b5c916455af3 "CML watermark")

Removes the temporary pin introduced in #6264


Unpin `jax` maximum version

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008410 / 0.011353 (-0.002943) | 0.004888 / 0.011008 (-0.006120) | 0.103342 / 0.038508 (0.064834) | 0.103697 / 0.023109 (0.080587) | 0.416445 / 0.275898 (0.140547) | 0.454604 / 0.323480 (0.131124) | 0.004976 / 0.007986 (-0.003010) | 0.003957 / 0.004328 (-0.000371) | 0.077398 / 0.004250 (0.073148) | 0.069026 / 0.037052 (0.031973) | 0.420484 / 0.258489 (0.161995) | 0.471828 / 0.293841 (0.177987) | 0.037133 / 0.128546 (-0.091413) | 0.010009 / 0.075646 (-0.065637) | 0.349573 / 0.419271 (-0.069698) | 0.063240 / 0.043533 (0.019708) | 0.421554 / 0.255139 (0.166415) | 0.433548 / 0.283200 (0.150348) | 0.029397 / 0.141683 (-0.112286) | 1.716860 / 1.452155 (0.264705) | 1.851264 / 1.492716 (0.358547) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.269733 / 0.018006 (0.251727) | 0.493313 / 0.000490 (0.492823) | 0.010438 / 0.000200 (0.010238) | 0.000401 / 0.000054 (0.000347) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034690 / 0.037411 (-0.002722) | 0.105304 / 0.014526 (0.090778) | 0.115831 / 0.176557 (-0.060726) | 0.185017 / 0.737135 (-0.552118) | 0.117480 / 0.296338 (-0.178859) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.479414 / 0.215209 (0.264205) | 4.785526 / 2.077655 (2.707871) | 2.388412 / 1.504120 (0.884292) | 2.178222 / 1.541195 (0.637027) | 2.248214 / 1.468490 (0.779723) | 0.571723 / 4.584777 (-4.013054) | 4.721250 / 3.745712 (0.975538) | 4.073893 / 5.269862 (-1.195969) | 2.618131 / 4.565676 (-1.947546) | 0.068406 / 0.424275 (-0.355869) | 0.008890 / 0.007607 (0.001283) | 0.564224 / 0.226044 (0.338180) | 5.631412 / 2.268929 (3.362483) | 3.072212 / 55.444624 (-52.372412) | 2.760574 / 6.876477 (-4.115903) | 2.963060 / 2.142072 (0.820987) | 0.708150 / 4.805227 (-4.097077) | 0.160324 / 6.500664 (-6.340340) | 0.075402 / 0.075469 (-0.000067) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.649965 / 1.841788 (-0.191823) | 24.297517 / 8.074308 (16.223209) | 17.658675 / 10.191392 (7.467283) | 0.171399 / 0.680424 (-0.509025) | 0.021172 / 0.534201 (-0.513029) | 0.477196 / 0.579283 (-0.102087) | 0.503900 / 0.434364 (0.069536) | 0.555858 / 0.540337 (0.015520) | 0.824302 / 1.386936 (-0.562634) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008613 / 0.011353 (-0.002740) | 0.004848 / 0.011008 (-0.006160) | 0.078344 / 0.038508 (0.039836) | 0.098976 / 0.023109 (0.075867) | 0.520713 / 0.275898 (0.244815) | 0.566350 / 0.323480 (0.242870) | 0.006658 / 0.007986 (-0.001327) | 0.004043 / 0.004328 (-0.000285) | 0.077881 / 0.004250 (0.073631) | 0.070731 / 0.037052 (0.033678) | 0.519717 / 0.258489 (0.261228) | 0.575623 / 0.293841 (0.281782) | 0.038542 / 0.128546 (-0.090004) | 0.010277 / 0.075646 (-0.065369) | 0.084269 / 0.419271 (-0.335002) | 0.058088 / 0.043533 (0.014555) | 0.541790 / 0.255139 (0.286651) | 0.534915 / 0.283200 (0.251715) | 0.027851 / 0.141683 (-0.113831) | 1.814827 / 1.452155 (0.362672) | 1.898208 / 1.492716 (0.405492) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.244162 / 0.018006 (0.226156) | 0.482895 / 0.000490 (0.482405) | 0.005734 / 0.000200 (0.005534) | 0.000127 / 0.000054 (0.000072) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.039328 / 0.037411 (0.001917) | 0.119795 / 0.014526 (0.105269) | 0.128570 / 0.176557 (-0.047986) | 0.191207 / 0.737135 (-0.545929) | 0.127147 / 0.296338 (-0.169192) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.533545 / 0.215209 (0.318336) | 5.320135 / 2.077655 (3.242480) | 2.924573 / 1.504120 (1.420453) | 2.741351 / 1.541195 (1.200156) | 2.824217 / 1.468490 (1.355727) | 0.595842 / 4.584777 (-3.988935) | 4.343499 / 3.745712 (0.597787) | 3.976546 / 5.269862 (-1.293316) | 2.532541 / 4.565676 (-2.033135) | 0.070480 / 0.424275 (-0.353795) | 0.008868 / 0.007607 (0.001260) | 0.634297 / 0.226044 (0.408253) | 6.327314 / 2.268929 (4.058386) | 3.530741 / 55.444624 (-51.913883) | 3.121435 / 6.876477 (-3.755042) | 3.344473 / 2.142072 (1.202401) | 0.719413 / 4.805227 (-4.085814) | 0.162348 / 6.500664 (-6.338316) | 0.074964 / 0.075469 (-0.000505) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.679095 / 1.841788 (-0.162693) | 25.071620 / 8.074308 (16.997312) | 18.422398 / 10.191392 (8.231006) | 0.223981 / 0.680424 (-0.456443) | 0.026537 / 0.534201 (-0.507664) | 0.513867 / 0.579283 (-0.065416) | 0.535874 / 0.434364 (0.101510) | 0.567971 / 0.540337 (0.027634) | 0.842545 / 1.386936 (-0.544391) |

![](https://cml.dev/watermark.png#d8b871016c25cb3b90ac1ff65a4e54f0454f525e "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006445 / 0.011353 (-0.004908) | 0.003978 / 0.011008 (-0.007030) | 0.084542 / 0.038508 (0.046034) | 0.069231 / 0.023109 (0.046122) | 0.308794 / 0.275898 (0.032896) | 0.339246 / 0.323480 (0.015766) | 0.005269 / 0.007986 (-0.002716) | 0.003285 / 0.004328 (-0.001043) | 0.065336 / 0.004250 (0.061086) | 0.053480 / 0.037052 (0.016428) | 0.316775 / 0.258489 (0.058286) | 0.357885 / 0.293841 (0.064044) | 0.031309 / 0.128546 (-0.097237) | 0.008450 / 0.075646 (-0.067196) | 0.287911 / 0.419271 (-0.131361) | 0.052756 / 0.043533 (0.009223) | 0.321516 / 0.255139 (0.066377) | 0.331998 / 0.283200 (0.048799) | 0.024129 / 0.141683 (-0.117553) | 1.507718 / 1.452155 (0.055563) | 1.571400 / 1.492716 (0.078683) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.237536 / 0.018006 (0.219530) | 0.499691 / 0.000490 (0.499201) | 0.007644 / 0.000200 (0.007444) | 0.000284 / 0.000054 (0.000230) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028243 / 0.037411 (-0.009168) | 0.081556 / 0.014526 (0.067030) | 0.096877 / 0.176557 (-0.079680) | 0.149985 / 0.737135 (-0.587150) | 0.095556 / 0.296338 (-0.200783) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.383215 / 0.215209 (0.168006) | 3.815800 / 2.077655 (1.738145) | 1.832227 / 1.504120 (0.328107) | 1.664001 / 1.541195 (0.122806) | 1.698786 / 1.468490 (0.230296) | 0.487594 / 4.584777 (-4.097183) | 3.569767 / 3.745712 (-0.175945) | 3.262387 / 5.269862 (-2.007475) | 2.017105 / 4.565676 (-2.548572) | 0.057555 / 0.424275 (-0.366720) | 0.007170 / 0.007607 (-0.000437) | 0.460134 / 0.226044 (0.234090) | 4.629800 / 2.268929 (2.360871) | 2.357126 / 55.444624 (-53.087499) | 1.970144 / 6.876477 (-4.906332) | 2.123520 / 2.142072 (-0.018552) | 0.613058 / 4.805227 (-4.192169) | 0.135869 / 6.500664 (-6.364795) | 0.061292 / 0.075469 (-0.014177) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.311294 / 1.841788 (-0.530494) | 18.640807 / 8.074308 (10.566499) | 13.946834 / 10.191392 (3.755442) | 0.163976 / 0.680424 (-0.516448) | 0.018527 / 0.534201 (-0.515674) | 0.390530 / 0.579283 (-0.188753) | 0.412661 / 0.434364 (-0.021703) | 0.459514 / 0.540337 (-0.080823) | 0.635026 / 1.386936 (-0.751910) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006645 / 0.011353 (-0.004708) | 0.003943 / 0.011008 (-0.007066) | 0.064470 / 0.038508 (0.025962) | 0.069895 / 0.023109 (0.046786) | 0.411091 / 0.275898 (0.135193) | 0.437628 / 0.323480 (0.114148) | 0.005214 / 0.007986 (-0.002772) | 0.003281 / 0.004328 (-0.001047) | 0.064434 / 0.004250 (0.060183) | 0.054294 / 0.037052 (0.017241) | 0.413576 / 0.258489 (0.155087) | 0.448793 / 0.293841 (0.154952) | 0.031754 / 0.128546 (-0.096793) | 0.008530 / 0.075646 (-0.067117) | 0.069950 / 0.419271 (-0.349322) | 0.047747 / 0.043533 (0.004214) | 0.411241 / 0.255139 (0.156102) | 0.430076 / 0.283200 (0.146876) | 0.023462 / 0.141683 (-0.118220) | 1.519501 / 1.452155 (0.067346) | 1.575782 / 1.492716 (0.083066) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.231816 / 0.018006 (0.213810) | 0.442802 / 0.000490 (0.442312) | 0.005738 / 0.000200 (0.005539) | 0.000087 / 0.000054 (0.000032) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031426 / 0.037411 (-0.005985) | 0.090758 / 0.014526 (0.076233) | 0.103414 / 0.176557 (-0.073142) | 0.156409 / 0.737135 (-0.580726) | 0.103900 / 0.296338 (-0.192439) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.438897 / 0.215209 (0.223688) | 4.385318 / 2.077655 (2.307663) | 2.352042 / 1.504120 (0.847923) | 2.182228 / 1.541195 (0.641033) | 2.266256 / 1.468490 (0.797766) | 0.492780 / 4.584777 (-4.091997) | 3.665787 / 3.745712 (-0.079925) | 3.315329 / 5.269862 (-1.954533) | 2.027993 / 4.565676 (-2.537684) | 0.058220 / 0.424275 (-0.366055) | 0.007429 / 0.007607 (-0.000178) | 0.508790 / 0.226044 (0.282746) | 5.107093 / 2.268929 (2.838164) | 2.799789 / 55.444624 (-52.644836) | 2.462828 / 6.876477 (-4.413649) | 2.610193 / 2.142072 (0.468120) | 0.588133 / 4.805227 (-4.217094) | 0.133418 / 6.500664 (-6.367246) | 0.059793 / 0.075469 (-0.015676) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.363358 / 1.841788 (-0.478430) | 19.258372 / 8.074308 (11.184064) | 14.730977 / 10.191392 (4.539584) | 0.169493 / 0.680424 (-0.510931) | 0.020462 / 0.534201 (-0.513739) | 0.397980 / 0.579283 (-0.181303) | 0.426638 / 0.434364 (-0.007726) | 0.474249 / 0.540337 (-0.066088) | 0.677640 / 1.386936 (-0.709296) |

![](https://cml.dev/watermark.png#90b3d2619ecb8f01dd12283c30f04dfe6e443795 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006536 / 0.011353 (-0.004817) | 0.003827 / 0.011008 (-0.007181) | 0.084394 / 0.038508 (0.045886) | 0.073166 / 0.023109 (0.050056) | 0.309380 / 0.275898 (0.033482) | 0.338501 / 0.323480 (0.015021) | 0.005346 / 0.007986 (-0.002640) | 0.003273 / 0.004328 (-0.001056) | 0.064606 / 0.004250 (0.060356) | 0.053500 / 0.037052 (0.016447) | 0.313143 / 0.258489 (0.054654) | 0.354364 / 0.293841 (0.060523) | 0.030919 / 0.128546 (-0.097627) | 0.008512 / 0.075646 (-0.067134) | 0.292774 / 0.419271 (-0.126498) | 0.052441 / 0.043533 (0.008908) | 0.310503 / 0.255139 (0.055364) | 0.341211 / 0.283200 (0.058011) | 0.023608 / 0.141683 (-0.118074) | 1.456220 / 1.452155 (0.004065) | 1.540189 / 1.492716 (0.047473) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.234321 / 0.018006 (0.216315) | 0.451809 / 0.000490 (0.451319) | 0.008560 / 0.000200 (0.008360) | 0.000085 / 0.000054 (0.000031) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028165 / 0.037411 (-0.009246) | 0.082548 / 0.014526 (0.068023) | 0.752621 / 0.176557 (0.576065) | 0.263949 / 0.737135 (-0.473187) | 0.097635 / 0.296338 (-0.198704) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.386611 / 0.215209 (0.171402) | 3.847528 / 2.077655 (1.769873) | 1.859173 / 1.504120 (0.355053) | 1.685269 / 1.541195 (0.144074) | 1.715823 / 1.468490 (0.247333) | 0.485272 / 4.584777 (-4.099505) | 3.500724 / 3.745712 (-0.244988) | 3.252149 / 5.269862 (-2.017713) | 2.052914 / 4.565676 (-2.512762) | 0.056794 / 0.424275 (-0.367481) | 0.007317 / 0.007607 (-0.000291) | 0.457924 / 0.226044 (0.231879) | 4.570092 / 2.268929 (2.301163) | 2.328829 / 55.444624 (-53.115796) | 1.986502 / 6.876477 (-4.889975) | 2.164645 / 2.142072 (0.022573) | 0.580455 / 4.805227 (-4.224772) | 0.134415 / 6.500664 (-6.366249) | 0.060506 / 0.075469 (-0.014963) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.267423 / 1.841788 (-0.574364) | 18.653450 / 8.074308 (10.579142) | 13.919682 / 10.191392 (3.728290) | 0.144001 / 0.680424 (-0.536423) | 0.018218 / 0.534201 (-0.515983) | 0.389933 / 0.579283 (-0.189350) | 0.418366 / 0.434364 (-0.015998) | 0.456341 / 0.540337 (-0.083997) | 0.631401 / 1.386936 (-0.755535) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006838 / 0.011353 (-0.004515) | 0.003973 / 0.011008 (-0.007036) | 0.065217 / 0.038508 (0.026709) | 0.068357 / 0.023109 (0.045248) | 0.407960 / 0.275898 (0.132062) | 0.437794 / 0.323480 (0.114314) | 0.005398 / 0.007986 (-0.002587) | 0.003360 / 0.004328 (-0.000969) | 0.065503 / 0.004250 (0.061253) | 0.055676 / 0.037052 (0.018623) | 0.411381 / 0.258489 (0.152892) | 0.446902 / 0.293841 (0.153061) | 0.032156 / 0.128546 (-0.096390) | 0.008702 / 0.075646 (-0.066944) | 0.072295 / 0.419271 (-0.346976) | 0.047722 / 0.043533 (0.004189) | 0.406125 / 0.255139 (0.150986) | 0.428359 / 0.283200 (0.145160) | 0.021901 / 0.141683 (-0.119782) | 1.464186 / 1.452155 (0.012032) | 1.532809 / 1.492716 (0.040093) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.218505 / 0.018006 (0.200499) | 0.447450 / 0.000490 (0.446961) | 0.006509 / 0.000200 (0.006309) | 0.000099 / 0.000054 (0.000045) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031789 / 0.037411 (-0.005622) | 0.091100 / 0.014526 (0.076574) | 0.102812 / 0.176557 (-0.073745) | 0.155988 / 0.737135 (-0.581147) | 0.103983 / 0.296338 (-0.192355) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.436431 / 0.215209 (0.221222) | 4.336072 / 2.077655 (2.258417) | 2.344613 / 1.504120 (0.840493) | 2.173513 / 1.541195 (0.632319) | 2.313134 / 1.468490 (0.844644) | 0.493651 / 4.584777 (-4.091126) | 3.657541 / 3.745712 (-0.088171) | 3.289933 / 5.269862 (-1.979928) | 2.040271 / 4.565676 (-2.525406) | 0.058092 / 0.424275 (-0.366183) | 0.007348 / 0.007607 (-0.000259) | 0.507506 / 0.226044 (0.281462) | 5.093477 / 2.268929 (2.824548) | 2.770579 / 55.444624 (-52.674046) | 2.449507 / 6.876477 (-4.426970) | 2.645470 / 2.142072 (0.503397) | 0.590799 / 4.805227 (-4.214429) | 0.133411 / 6.500664 (-6.367253) | 0.059507 / 0.075469 (-0.015962) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.381148 / 1.841788 (-0.460639) | 19.188716 / 8.074308 (11.114408) | 14.709111 / 10.191392 (4.517719) | 0.191104 / 0.680424 (-0.489320) | 0.019862 / 0.534201 (-0.514339) | 0.395380 / 0.579283 (-0.183903) | 0.424757 / 0.434364 (-0.009607) | 0.468810 / 0.540337 (-0.071527) | 0.687058 / 1.386936 (-0.699878) |

![](https://cml.dev/watermark.png#407169e1ea91ae31f79ff29c4115b04a461279ab "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008872 / 0.011353 (-0.002481) | 0.004824 / 0.011008 (-0.006184) | 0.097012 / 0.038508 (0.058504) | 0.074728 / 0.023109 (0.051619) | 0.400604 / 0.275898 (0.124706) | 0.434316 / 0.323480 (0.110836) | 0.006025 / 0.007986 (-0.001961) | 0.004153 / 0.004328 (-0.000176) | 0.074093 / 0.004250 (0.069842) | 0.057239 / 0.037052 (0.020187) | 0.420611 / 0.258489 (0.162122) | 0.457779 / 0.293841 (0.163938) | 0.047610 / 0.128546 (-0.080936) | 0.014577 / 0.075646 (-0.061069) | 0.414351 / 0.419271 (-0.004921) | 0.063072 / 0.043533 (0.019539) | 0.426141 / 0.255139 (0.171002) | 0.429844 / 0.283200 (0.146644) | 0.034754 / 0.141683 (-0.106929) | 1.620946 / 1.452155 (0.168792) | 1.725831 / 1.492716 (0.233115) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.304712 / 0.018006 (0.286706) | 0.646924 / 0.000490 (0.646434) | 0.014486 / 0.000200 (0.014286) | 0.000626 / 0.000054 (0.000572) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034935 / 0.037411 (-0.002477) | 0.085788 / 0.014526 (0.071262) | 0.107749 / 0.176557 (-0.068807) | 0.170924 / 0.737135 (-0.566211) | 0.134985 / 0.296338 (-0.161354) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.602913 / 0.215209 (0.387704) | 6.041700 / 2.077655 (3.964045) | 2.539970 / 1.504120 (1.035850) | 2.184166 / 1.541195 (0.642972) | 2.241783 / 1.468490 (0.773293) | 0.864601 / 4.584777 (-3.720176) | 5.246955 / 3.745712 (1.501243) | 4.850458 / 5.269862 (-0.419404) | 3.101497 / 4.565676 (-1.464179) | 0.098591 / 0.424275 (-0.325684) | 0.008902 / 0.007607 (0.001295) | 0.732278 / 0.226044 (0.506234) | 7.163557 / 2.268929 (4.894629) | 3.226444 / 55.444624 (-52.218180) | 2.578737 / 6.876477 (-4.297740) | 2.850212 / 2.142072 (0.708140) | 1.026390 / 4.805227 (-3.778837) | 0.217077 / 6.500664 (-6.283587) | 0.080344 / 0.075469 (0.004875) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.687488 / 1.841788 (-0.154300) | 24.686337 / 8.074308 (16.612029) | 21.315989 / 10.191392 (11.124597) | 0.226176 / 0.680424 (-0.454248) | 0.035774 / 0.534201 (-0.498427) | 0.477807 / 0.579283 (-0.101476) | 0.636305 / 0.434364 (0.201941) | 0.553341 / 0.540337 (0.013003) | 0.797267 / 1.386936 (-0.589669) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008955 / 0.011353 (-0.002398) | 0.006099 / 0.011008 (-0.004909) | 0.086306 / 0.038508 (0.047798) | 0.090783 / 0.023109 (0.067674) | 0.554802 / 0.275898 (0.278904) | 0.598778 / 0.323480 (0.275299) | 0.008656 / 0.007986 (0.000670) | 0.004487 / 0.004328 (0.000159) | 0.084194 / 0.004250 (0.079943) | 0.076048 / 0.037052 (0.038996) | 0.533212 / 0.258489 (0.274723) | 0.584029 / 0.293841 (0.290188) | 0.051913 / 0.128546 (-0.076634) | 0.014253 / 0.075646 (-0.061393) | 0.100500 / 0.419271 (-0.318772) | 0.061092 / 0.043533 (0.017560) | 0.516955 / 0.255139 (0.261816) | 0.562754 / 0.283200 (0.279554) | 0.036673 / 0.141683 (-0.105010) | 1.853655 / 1.452155 (0.401501) | 1.968358 / 1.492716 (0.475642) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.308258 / 0.018006 (0.290252) | 0.630492 / 0.000490 (0.630002) | 0.010575 / 0.000200 (0.010375) | 0.000271 / 0.000054 (0.000217) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034762 / 0.037411 (-0.002649) | 0.107314 / 0.014526 (0.092788) | 0.132160 / 0.176557 (-0.044396) | 0.178737 / 0.737135 (-0.558398) | 0.125988 / 0.296338 (-0.170351) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.730738 / 0.215209 (0.515528) | 7.240393 / 2.077655 (5.162738) | 3.557665 / 1.504120 (2.053545) | 3.541425 / 1.541195 (2.000230) | 3.103849 / 1.468490 (1.635359) | 0.926843 / 4.584777 (-3.657934) | 5.818264 / 3.745712 (2.072552) | 5.012984 / 5.269862 (-0.256878) | 3.286085 / 4.565676 (-1.279591) | 0.104879 / 0.424275 (-0.319396) | 0.009010 / 0.007607 (0.001403) | 0.806145 / 0.226044 (0.580101) | 8.263655 / 2.268929 (5.994727) | 4.108932 / 55.444624 (-51.335693) | 3.454613 / 6.876477 (-3.421864) | 3.629045 / 2.142072 (1.486973) | 1.062325 / 4.805227 (-3.742902) | 0.220482 / 6.500664 (-6.280182) | 0.081440 / 0.075469 (0.005970) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.665587 / 1.841788 (-0.176201) | 23.695299 / 8.074308 (15.620991) | 22.917493 / 10.191392 (12.726101) | 0.259033 / 0.680424 (-0.421391) | 0.040118 / 0.534201 (-0.494083) | 0.487329 / 0.579283 (-0.091954) | 0.607482 / 0.434364 (0.173118) | 0.568383 / 0.540337 (0.028045) | 0.824486 / 1.386936 (-0.562450) |

![](https://cml.dev/watermark.png#53592bb8f635a1d6ea3e77acc290efdfb28fcbd7 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007095 / 0.011353 (-0.004258) | 0.004260 / 0.011008 (-0.006748) | 0.084729 / 0.038508 (0.046221) | 0.076498 / 0.023109 (0.053389) | 0.325981 / 0.275898 (0.050083) | 0.357140 / 0.323480 (0.033661) | 0.004325 / 0.007986 (-0.003660) | 0.003632 / 0.004328 (-0.000696) | 0.065075 / 0.004250 (0.060824) | 0.059058 / 0.037052 (0.022006) | 0.331895 / 0.258489 (0.073406) | 0.370782 / 0.293841 (0.076941) | 0.031886 / 0.128546 (-0.096660) | 0.008782 / 0.075646 (-0.066864) | 0.288159 / 0.419271 (-0.131113) | 0.053012 / 0.043533 (0.009479) | 0.319992 / 0.255139 (0.064853) | 0.347061 / 0.283200 (0.063861) | 0.026365 / 0.141683 (-0.115317) | 1.486112 / 1.452155 (0.033958) | 1.570150 / 1.492716 (0.077434) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.277155 / 0.018006 (0.259149) | 0.573507 / 0.000490 (0.573017) | 0.010122 / 0.000200 (0.009922) | 0.000322 / 0.000054 (0.000268) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029076 / 0.037411 (-0.008335) | 0.082517 / 0.014526 (0.067991) | 0.100710 / 0.176557 (-0.075847) | 0.154529 / 0.737135 (-0.582606) | 0.099531 / 0.296338 (-0.196807) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.382058 / 0.215209 (0.166849) | 3.803307 / 2.077655 (1.725652) | 1.834107 / 1.504120 (0.329987) | 1.665703 / 1.541195 (0.124508) | 1.739520 / 1.468490 (0.271030) | 0.490544 / 4.584777 (-4.094233) | 3.577874 / 3.745712 (-0.167838) | 3.327631 / 5.269862 (-1.942231) | 2.056634 / 4.565676 (-2.509043) | 0.057871 / 0.424275 (-0.366404) | 0.007326 / 0.007607 (-0.000281) | 0.453993 / 0.226044 (0.227949) | 4.549179 / 2.268929 (2.280250) | 2.320304 / 55.444624 (-53.124321) | 1.966082 / 6.876477 (-4.910395) | 2.189979 / 2.142072 (0.047907) | 0.586678 / 4.805227 (-4.218549) | 0.134919 / 6.500664 (-6.365745) | 0.061649 / 0.075469 (-0.013820) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.286228 / 1.841788 (-0.555560) | 19.409674 / 8.074308 (11.335366) | 14.290463 / 10.191392 (4.099071) | 0.165766 / 0.680424 (-0.514658) | 0.018200 / 0.534201 (-0.516001) | 0.390526 / 0.579283 (-0.188757) | 0.410953 / 0.434364 (-0.023411) | 0.455921 / 0.540337 (-0.084416) | 0.642271 / 1.386936 (-0.744665) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007288 / 0.011353 (-0.004064) | 0.004348 / 0.011008 (-0.006660) | 0.065935 / 0.038508 (0.027427) | 0.087327 / 0.023109 (0.064218) | 0.413461 / 0.275898 (0.137563) | 0.458904 / 0.323480 (0.135424) | 0.005996 / 0.007986 (-0.001990) | 0.003648 / 0.004328 (-0.000680) | 0.066578 / 0.004250 (0.062328) | 0.062072 / 0.037052 (0.025020) | 0.418469 / 0.258489 (0.159980) | 0.468960 / 0.293841 (0.175119) | 0.032616 / 0.128546 (-0.095930) | 0.008961 / 0.075646 (-0.066686) | 0.072537 / 0.419271 (-0.346734) | 0.048302 / 0.043533 (0.004769) | 0.411845 / 0.255139 (0.156706) | 0.441730 / 0.283200 (0.158530) | 0.025038 / 0.141683 (-0.116645) | 1.519402 / 1.452155 (0.067248) | 1.601791 / 1.492716 (0.109074) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.322494 / 0.018006 (0.304488) | 0.570210 / 0.000490 (0.569720) | 0.025815 / 0.000200 (0.025615) | 0.000166 / 0.000054 (0.000111) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034657 / 0.037411 (-0.002754) | 0.096024 / 0.014526 (0.081498) | 0.109134 / 0.176557 (-0.067422) | 0.162170 / 0.737135 (-0.574965) | 0.110472 / 0.296338 (-0.185866) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.439032 / 0.215209 (0.223823) | 4.385768 / 2.077655 (2.308113) | 2.343261 / 1.504120 (0.839142) | 2.157926 / 1.541195 (0.616731) | 2.299193 / 1.468490 (0.830703) | 0.498961 / 4.584777 (-4.085816) | 3.651909 / 3.745712 (-0.093803) | 3.387587 / 5.269862 (-1.882275) | 2.144553 / 4.565676 (-2.421123) | 0.058242 / 0.424275 (-0.366033) | 0.007416 / 0.007607 (-0.000191) | 0.512714 / 0.226044 (0.286670) | 5.138569 / 2.268929 (2.869641) | 2.778683 / 55.444624 (-52.665941) | 2.532990 / 6.876477 (-4.343487) | 2.782211 / 2.142072 (0.640139) | 0.591881 / 4.805227 (-4.213346) | 0.135005 / 6.500664 (-6.365660) | 0.060965 / 0.075469 (-0.014504) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.356311 / 1.841788 (-0.485477) | 20.029994 / 8.074308 (11.955686) | 14.666570 / 10.191392 (4.475178) | 0.164363 / 0.680424 (-0.516061) | 0.020685 / 0.534201 (-0.513516) | 0.396020 / 0.579283 (-0.183263) | 0.429407 / 0.434364 (-0.004957) | 0.476924 / 0.540337 (-0.063413) | 0.693389 / 1.386936 (-0.693547) |

![](https://cml.dev/watermark.png#292d627e398e30a538a616395f3b5ce4e89bb1e8 "CML watermark")

fix #6299
fix #6202


Support for newer versions of JAX

enhancement

New feature or request

### Feature request

Hi,

I like your idea of adapting the datasets library to be usable with JAX. Thank you for that.

However, in your [setup.py](https://github.com/huggingface/datasets/blob/main/setup.py), you enforce old versions of JAX <= 0.3... It is very cumbersome !

What is the rationale for such a limitation ? Can you remove it please ?

Thanks,

### Motivation

This library is unusable with new versions of JAX ?

### Your contribution

Yes.


Doc readme improvements

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006761 / 0.011353 (-0.004592) | 0.004307 / 0.011008 (-0.006701) | 0.084682 / 0.038508 (0.046174) | 0.083994 / 0.023109 (0.060885) | 0.316612 / 0.275898 (0.040714) | 0.346157 / 0.323480 (0.022678) | 0.004490 / 0.007986 (-0.003495) | 0.003699 / 0.004328 (-0.000629) | 0.066144 / 0.004250 (0.061894) | 0.057958 / 0.037052 (0.020906) | 0.319018 / 0.258489 (0.060529) | 0.367597 / 0.293841 (0.073756) | 0.031146 / 0.128546 (-0.097401) | 0.008814 / 0.075646 (-0.066832) | 0.290971 / 0.419271 (-0.128301) | 0.052769 / 0.043533 (0.009236) | 0.313125 / 0.255139 (0.057986) | 0.330473 / 0.283200 (0.047273) | 0.025922 / 0.141683 (-0.115760) | 1.494989 / 1.452155 (0.042834) | 1.556140 / 1.492716 (0.063423) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.310580 / 0.018006 (0.292574) | 0.563600 / 0.000490 (0.563110) | 0.012344 / 0.000200 (0.012144) | 0.000382 / 0.000054 (0.000328) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031468 / 0.037411 (-0.005943) | 0.084856 / 0.014526 (0.070331) | 0.101371 / 0.176557 (-0.075186) | 0.158735 / 0.737135 (-0.578400) | 0.102451 / 0.296338 (-0.193888) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.402288 / 0.215209 (0.187079) | 4.001351 / 2.077655 (1.923696) | 2.022710 / 1.504120 (0.518590) | 1.850236 / 1.541195 (0.309041) | 1.946779 / 1.468490 (0.478289) | 0.485828 / 4.584777 (-4.098949) | 3.584925 / 3.745712 (-0.160787) | 3.400815 / 5.269862 (-1.869046) | 2.123187 / 4.565676 (-2.442490) | 0.057373 / 0.424275 (-0.366902) | 0.007383 / 0.007607 (-0.000224) | 0.479773 / 0.226044 (0.253729) | 4.805342 / 2.268929 (2.536414) | 2.530151 / 55.444624 (-52.914473) | 2.190136 / 6.876477 (-4.686341) | 2.463666 / 2.142072 (0.321593) | 0.583512 / 4.805227 (-4.221715) | 0.134205 / 6.500664 (-6.366459) | 0.062021 / 0.075469 (-0.013448) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.239532 / 1.841788 (-0.602255) | 20.252941 / 8.074308 (12.178633) | 14.265697 / 10.191392 (4.074305) | 0.158745 / 0.680424 (-0.521679) | 0.018605 / 0.534201 (-0.515596) | 0.394246 / 0.579283 (-0.185037) | 0.415260 / 0.434364 (-0.019104) | 0.462636 / 0.540337 (-0.077701) | 0.645318 / 1.386936 (-0.741618) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007063 / 0.011353 (-0.004290) | 0.004388 / 0.011008 (-0.006621) | 0.064997 / 0.038508 (0.026489) | 0.085135 / 0.023109 (0.062026) | 0.424349 / 0.275898 (0.148451) | 0.456033 / 0.323480 (0.132553) | 0.005745 / 0.007986 (-0.002241) | 0.003705 / 0.004328 (-0.000624) | 0.065835 / 0.004250 (0.061585) | 0.058366 / 0.037052 (0.021314) | 0.421654 / 0.258489 (0.163165) | 0.460334 / 0.293841 (0.166493) | 0.032828 / 0.128546 (-0.095718) | 0.008974 / 0.075646 (-0.066673) | 0.072524 / 0.419271 (-0.346747) | 0.048558 / 0.043533 (0.005025) | 0.413546 / 0.255139 (0.158407) | 0.435765 / 0.283200 (0.152565) | 0.023754 / 0.141683 (-0.117929) | 1.476884 / 1.452155 (0.024730) | 1.560011 / 1.492716 (0.067294) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.318279 / 0.018006 (0.300272) | 0.544990 / 0.000490 (0.544501) | 0.007118 / 0.000200 (0.006918) | 0.000097 / 0.000054 (0.000043) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033352 / 0.037411 (-0.004059) | 0.092921 / 0.014526 (0.078395) | 0.109028 / 0.176557 (-0.067528) | 0.161433 / 0.737135 (-0.575703) | 0.108445 / 0.296338 (-0.187893) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.438925 / 0.215209 (0.223716) | 4.400714 / 2.077655 (2.323059) | 2.403727 / 1.504120 (0.899607) | 2.236472 / 1.541195 (0.695277) | 2.319219 / 1.468490 (0.850729) | 0.490159 / 4.584777 (-4.094618) | 3.647474 / 3.745712 (-0.098238) | 3.433144 / 5.269862 (-1.836718) | 2.145367 / 4.565676 (-2.420310) | 0.057994 / 0.424275 (-0.366281) | 0.007452 / 0.007607 (-0.000155) | 0.513808 / 0.226044 (0.287763) | 5.130792 / 2.268929 (2.861863) | 2.861691 / 55.444624 (-52.582934) | 2.473292 / 6.876477 (-4.403185) | 2.756445 / 2.142072 (0.614372) | 0.586783 / 4.805227 (-4.218444) | 0.134170 / 6.500664 (-6.366494) | 0.061149 / 0.075469 (-0.014320) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.350144 / 1.841788 (-0.491644) | 21.003528 / 8.074308 (12.929220) | 15.174314 / 10.191392 (4.982922) | 0.186535 / 0.680424 (-0.493888) | 0.020821 / 0.534201 (-0.513380) | 0.399210 / 0.579283 (-0.180073) | 0.431942 / 0.434364 (-0.002422) | 0.475395 / 0.540337 (-0.064942) | 0.677457 / 1.386936 (-0.709479) |

![](https://cml.dev/watermark.png#6aa5fc278324a253eab43ad1bc048e822e4ae5c7 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007062 / 0.011353 (-0.004291) | 0.004299 / 0.011008 (-0.006710) | 0.086019 / 0.038508 (0.047511) | 0.085166 / 0.023109 (0.062057) | 0.355804 / 0.275898 (0.079906) | 0.381056 / 0.323480 (0.057577) | 0.005500 / 0.007986 (-0.002486) | 0.003496 / 0.004328 (-0.000833) | 0.064866 / 0.004250 (0.060615) | 0.057399 / 0.037052 (0.020346) | 0.357914 / 0.258489 (0.099425) | 0.395387 / 0.293841 (0.101546) | 0.031763 / 0.128546 (-0.096784) | 0.008665 / 0.075646 (-0.066981) | 0.290097 / 0.419271 (-0.129175) | 0.053297 / 0.043533 (0.009765) | 0.355659 / 0.255139 (0.100520) | 0.378232 / 0.283200 (0.095032) | 0.026015 / 0.141683 (-0.115668) | 1.437121 / 1.452155 (-0.015034) | 1.538798 / 1.492716 (0.046082) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.243518 / 0.018006 (0.225511) | 0.461361 / 0.000490 (0.460871) | 0.009529 / 0.000200 (0.009329) | 0.000473 / 0.000054 (0.000419) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030379 / 0.037411 (-0.007032) | 0.089851 / 0.014526 (0.075325) | 0.098278 / 0.176557 (-0.078278) | 0.157077 / 0.737135 (-0.580058) | 0.098997 / 0.296338 (-0.197341) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.382415 / 0.215209 (0.167206) | 3.801964 / 2.077655 (1.724309) | 1.887680 / 1.504120 (0.383560) | 1.775903 / 1.541195 (0.234709) | 1.851338 / 1.468490 (0.382848) | 0.483616 / 4.584777 (-4.101161) | 3.612977 / 3.745712 (-0.132736) | 3.397700 / 5.269862 (-1.872162) | 2.114572 / 4.565676 (-2.451105) | 0.057250 / 0.424275 (-0.367025) | 0.007362 / 0.007607 (-0.000245) | 0.456873 / 0.226044 (0.230829) | 4.567319 / 2.268929 (2.298391) | 2.399476 / 55.444624 (-53.045148) | 2.054542 / 6.876477 (-4.821935) | 2.343432 / 2.142072 (0.201359) | 0.582319 / 4.805227 (-4.222908) | 0.134045 / 6.500664 (-6.366619) | 0.062726 / 0.075469 (-0.012743) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.283390 / 1.841788 (-0.558398) | 20.358511 / 8.074308 (12.284202) | 14.933989 / 10.191392 (4.742597) | 0.164960 / 0.680424 (-0.515464) | 0.018625 / 0.534201 (-0.515576) | 0.394087 / 0.579283 (-0.185196) | 0.416761 / 0.434364 (-0.017603) | 0.466669 / 0.540337 (-0.073669) | 0.643161 / 1.386936 (-0.743775) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007141 / 0.011353 (-0.004212) | 0.004185 / 0.011008 (-0.006824) | 0.066097 / 0.038508 (0.027588) | 0.088436 / 0.023109 (0.065327) | 0.401189 / 0.275898 (0.125291) | 0.440402 / 0.323480 (0.116922) | 0.005729 / 0.007986 (-0.002257) | 0.003527 / 0.004328 (-0.000801) | 0.065278 / 0.004250 (0.061027) | 0.060866 / 0.037052 (0.023813) | 0.407035 / 0.258489 (0.148546) | 0.443923 / 0.293841 (0.150083) | 0.032922 / 0.128546 (-0.095625) | 0.008739 / 0.075646 (-0.066907) | 0.071800 / 0.419271 (-0.347472) | 0.048994 / 0.043533 (0.005461) | 0.403736 / 0.255139 (0.148597) | 0.419566 / 0.283200 (0.136366) | 0.025369 / 0.141683 (-0.116314) | 1.474980 / 1.452155 (0.022825) | 1.553500 / 1.492716 (0.060784) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.225224 / 0.018006 (0.207218) | 0.462891 / 0.000490 (0.462401) | 0.006958 / 0.000200 (0.006758) | 0.000163 / 0.000054 (0.000108) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034431 / 0.037411 (-0.002980) | 0.100021 / 0.014526 (0.085495) | 0.108339 / 0.176557 (-0.068217) | 0.162762 / 0.737135 (-0.574374) | 0.108951 / 0.296338 (-0.187388) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.435966 / 0.215209 (0.220757) | 4.351744 / 2.077655 (2.274089) | 2.372307 / 1.504120 (0.868187) | 2.192146 / 1.541195 (0.650951) | 2.326839 / 1.468490 (0.858349) | 0.488292 / 4.584777 (-4.096485) | 3.745227 / 3.745712 (-0.000485) | 3.456306 / 5.269862 (-1.813556) | 2.159771 / 4.565676 (-2.405906) | 0.057953 / 0.424275 (-0.366322) | 0.007469 / 0.007607 (-0.000138) | 0.515116 / 0.226044 (0.289071) | 5.162871 / 2.268929 (2.893942) | 2.850336 / 55.444624 (-52.594288) | 2.514700 / 6.876477 (-4.361777) | 2.748843 / 2.142072 (0.606770) | 0.587687 / 4.805227 (-4.217540) | 0.134333 / 6.500664 (-6.366331) | 0.062097 / 0.075469 (-0.013372) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.377082 / 1.841788 (-0.464705) | 21.103127 / 8.074308 (13.028819) | 15.325275 / 10.191392 (5.133883) | 0.166225 / 0.680424 (-0.514199) | 0.020472 / 0.534201 (-0.513729) | 0.395866 / 0.579283 (-0.183417) | 0.444964 / 0.434364 (0.010600) | 0.475367 / 0.540337 (-0.064970) | 0.693325 / 1.386936 (-0.693611) |

![](https://cml.dev/watermark.png#79b5bbbd52ffd90dd958c05b333d7c90a03756cc "CML watermark")

Changes in the doc READMe:
* adds two new sections (to be aligned with `transformers` and `hfh`): "Previewing the documentation" and "Writing documentation examples"
* replaces the mentions of `transformers` with `datasets`
* fixes some dead links