Update README.md

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006757 / 0.011353 (-0.004596) | 0.004233 / 0.011008 (-0.006775) | 0.084123 / 0.038508 (0.045614) | 0.077513 / 0.023109 (0.054404) | 0.357024 / 0.275898 (0.081126) | 0.392956 / 0.323480 (0.069476) | 0.005408 / 0.007986 (-0.002577) | 0.003363 / 0.004328 (-0.000966) | 0.064395 / 0.004250 (0.060145) | 0.054711 / 0.037052 (0.017659) | 0.367287 / 0.258489 (0.108798) | 0.402934 / 0.293841 (0.109093) | 0.031845 / 0.128546 (-0.096701) | 0.008646 / 0.075646 (-0.067000) | 0.288740 / 0.419271 (-0.130531) | 0.053171 / 0.043533 (0.009638) | 0.360711 / 0.255139 (0.105572) | 0.388707 / 0.283200 (0.105507) | 0.025321 / 0.141683 (-0.116361) | 1.500684 / 1.452155 (0.048529) | 1.585747 / 1.492716 (0.093030) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.207329 / 0.018006 (0.189323) | 0.465304 / 0.000490 (0.464814) | 0.003229 / 0.000200 (0.003029) | 0.000080 / 0.000054 (0.000025) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028752 / 0.037411 (-0.008659) | 0.085327 / 0.014526 (0.070802) | 0.332210 / 0.176557 (0.155653) | 0.178779 / 0.737135 (-0.558356) | 0.097765 / 0.296338 (-0.198573) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.403710 / 0.215209 (0.188501) | 4.027069 / 2.077655 (1.949414) | 2.053451 / 1.504120 (0.549331) | 1.906647 / 1.541195 (0.365452) | 1.992507 / 1.468490 (0.524017) | 0.490203 / 4.584777 (-4.094574) | 3.696569 / 3.745712 (-0.049143) | 3.319919 / 5.269862 (-1.949943) | 2.072794 / 4.565676 (-2.492883) | 0.057893 / 0.424275 (-0.366383) | 0.007723 / 0.007607 (0.000116) | 0.485400 / 0.226044 (0.259355) | 4.842891 / 2.268929 (2.573963) | 2.578949 / 55.444624 (-52.865675) | 2.229217 / 6.876477 (-4.647259) | 2.468017 / 2.142072 (0.325945) | 0.595236 / 4.805227 (-4.209992) | 0.135641 / 6.500664 (-6.365023) | 0.061232 / 0.075469 (-0.014237) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.307059 / 1.841788 (-0.534729) | 20.108581 / 8.074308 (12.034273) | 14.438985 / 10.191392 (4.247593) | 0.168878 / 0.680424 (-0.511545) | 0.018208 / 0.534201 (-0.515993) | 0.395986 / 0.579283 (-0.183297) | 0.427440 / 0.434364 (-0.006924) | 0.459917 / 0.540337 (-0.080421) | 0.631379 / 1.386936 (-0.755557) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007002 / 0.011353 (-0.004351) | 0.004120 / 0.011008 (-0.006888) | 0.064817 / 0.038508 (0.026309) | 0.081297 / 0.023109 (0.058188) | 0.405598 / 0.275898 (0.129700) | 0.442360 / 0.323480 (0.118880) | 0.005475 / 0.007986 (-0.002511) | 0.003483 / 0.004328 (-0.000845) | 0.064750 / 0.004250 (0.060499) | 0.058111 / 0.037052 (0.021059) | 0.410154 / 0.258489 (0.151665) | 0.445137 / 0.293841 (0.151296) | 0.033314 / 0.128546 (-0.095232) | 0.008747 / 0.075646 (-0.066899) | 0.071595 / 0.419271 (-0.347676) | 0.048894 / 0.043533 (0.005361) | 0.409162 / 0.255139 (0.154023) | 0.428877 / 0.283200 (0.145677) | 0.024127 / 0.141683 (-0.117556) | 1.521369 / 1.452155 (0.069214) | 1.573505 / 1.492716 (0.080789) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.233199 / 0.018006 (0.215193) | 0.455619 / 0.000490 (0.455129) | 0.003688 / 0.000200 (0.003488) | 0.000081 / 0.000054 (0.000027) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033186 / 0.037411 (-0.004225) | 0.100528 / 0.014526 (0.086003) | 0.105617 / 0.176557 (-0.070940) | 0.159437 / 0.737135 (-0.577698) | 0.108064 / 0.296338 (-0.188274) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.435509 / 0.215209 (0.220300) | 4.339920 / 2.077655 (2.262265) | 2.368983 / 1.504120 (0.864863) | 2.211761 / 1.541195 (0.670566) | 2.301701 / 1.468490 (0.833211) | 0.495144 / 4.584777 (-4.089633) | 3.768882 / 3.745712 (0.023170) | 3.348940 / 5.269862 (-1.920922) | 2.081142 / 4.565676 (-2.484534) | 0.058184 / 0.424275 (-0.366091) | 0.007597 / 0.007607 (-0.000010) | 0.508806 / 0.226044 (0.282762) | 5.089226 / 2.268929 (2.820297) | 2.851930 / 55.444624 (-52.592694) | 2.512144 / 6.876477 (-4.364332) | 2.724461 / 2.142072 (0.582388) | 0.593446 / 4.805227 (-4.211781) | 0.134908 / 6.500664 (-6.365756) | 0.060811 / 0.075469 (-0.014658) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.362279 / 1.841788 (-0.479508) | 20.548216 / 8.074308 (12.473908) | 15.179181 / 10.191392 (4.987789) | 0.170249 / 0.680424 (-0.510175) | 0.020772 / 0.534201 (-0.513429) | 0.398737 / 0.579283 (-0.180546) | 0.441487 / 0.434364 (0.007124) | 0.480096 / 0.540337 (-0.060242) | 0.645825 / 1.386936 (-0.741111) |

![](https://cml.dev/watermark.png#a6fb8b9a833afb25311da395c6e0d9bf770ca2c7 "CML watermark")

fixed a few typos


fix typo in Audio dataset documentation

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006655 / 0.011353 (-0.004698) | 0.004115 / 0.011008 (-0.006893) | 0.083895 / 0.038508 (0.045387) | 0.072770 / 0.023109 (0.049661) | 0.311401 / 0.275898 (0.035503) | 0.341079 / 0.323480 (0.017599) | 0.005488 / 0.007986 (-0.002497) | 0.003530 / 0.004328 (-0.000799) | 0.064691 / 0.004250 (0.060441) | 0.053096 / 0.037052 (0.016044) | 0.314969 / 0.258489 (0.056480) | 0.358245 / 0.293841 (0.064404) | 0.030789 / 0.128546 (-0.097757) | 0.008868 / 0.075646 (-0.066779) | 0.288022 / 0.419271 (-0.131249) | 0.052092 / 0.043533 (0.008559) | 0.310061 / 0.255139 (0.054922) | 0.345369 / 0.283200 (0.062170) | 0.024100 / 0.141683 (-0.117582) | 1.520573 / 1.452155 (0.068418) | 1.593750 / 1.492716 (0.101033) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.242520 / 0.018006 (0.224514) | 0.567963 / 0.000490 (0.567473) | 0.003183 / 0.000200 (0.002983) | 0.000074 / 0.000054 (0.000020) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029473 / 0.037411 (-0.007939) | 0.083012 / 0.014526 (0.068486) | 0.262386 / 0.176557 (0.085830) | 0.155131 / 0.737135 (-0.582004) | 0.099880 / 0.296338 (-0.196458) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.382388 / 0.215209 (0.167179) | 3.816538 / 2.077655 (1.738884) | 1.863422 / 1.504120 (0.359302) | 1.694652 / 1.541195 (0.153457) | 1.738738 / 1.468490 (0.270248) | 0.477073 / 4.584777 (-4.107704) | 3.539244 / 3.745712 (-0.206468) | 3.238469 / 5.269862 (-2.031392) | 2.026154 / 4.565676 (-2.539523) | 0.056111 / 0.424275 (-0.368164) | 0.007615 / 0.007607 (0.000008) | 0.460620 / 0.226044 (0.234576) | 4.596383 / 2.268929 (2.327455) | 2.348645 / 55.444624 (-53.095979) | 1.977465 / 6.876477 (-4.899011) | 2.222828 / 2.142072 (0.080755) | 0.588065 / 4.805227 (-4.217162) | 0.132175 / 6.500664 (-6.368489) | 0.061322 / 0.075469 (-0.014147) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.260623 / 1.841788 (-0.581164) | 19.976475 / 8.074308 (11.902167) | 14.346488 / 10.191392 (4.155096) | 0.145614 / 0.680424 (-0.534810) | 0.018309 / 0.534201 (-0.515892) | 0.393644 / 0.579283 (-0.185639) | 0.405355 / 0.434364 (-0.029009) | 0.458355 / 0.540337 (-0.081982) | 0.630147 / 1.386936 (-0.756789) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006769 / 0.011353 (-0.004584) | 0.004172 / 0.011008 (-0.006836) | 0.064863 / 0.038508 (0.026355) | 0.076831 / 0.023109 (0.053722) | 0.419391 / 0.275898 (0.143493) | 0.439912 / 0.323480 (0.116432) | 0.006249 / 0.007986 (-0.001737) | 0.003571 / 0.004328 (-0.000757) | 0.064877 / 0.004250 (0.060626) | 0.056023 / 0.037052 (0.018971) | 0.419899 / 0.258489 (0.161410) | 0.459334 / 0.293841 (0.165493) | 0.032217 / 0.128546 (-0.096329) | 0.008628 / 0.075646 (-0.067019) | 0.071089 / 0.419271 (-0.348183) | 0.047463 / 0.043533 (0.003930) | 0.414961 / 0.255139 (0.159822) | 0.431408 / 0.283200 (0.148209) | 0.022406 / 0.141683 (-0.119277) | 1.511890 / 1.452155 (0.059735) | 1.580268 / 1.492716 (0.087551) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.280805 / 0.018006 (0.262799) | 0.553766 / 0.000490 (0.553276) | 0.006155 / 0.000200 (0.005955) | 0.000102 / 0.000054 (0.000047) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032980 / 0.037411 (-0.004431) | 0.092981 / 0.014526 (0.078456) | 0.108820 / 0.176557 (-0.067737) | 0.161709 / 0.737135 (-0.575426) | 0.109772 / 0.296338 (-0.186566) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.433659 / 0.215209 (0.218450) | 4.328577 / 2.077655 (2.250923) | 2.316899 / 1.504120 (0.812779) | 2.142645 / 1.541195 (0.601451) | 2.245518 / 1.468490 (0.777028) | 0.489448 / 4.584777 (-4.095329) | 3.630074 / 3.745712 (-0.115638) | 3.322749 / 5.269862 (-1.947112) | 2.062307 / 4.565676 (-2.503370) | 0.058153 / 0.424275 (-0.366122) | 0.007453 / 0.007607 (-0.000154) | 0.507234 / 0.226044 (0.281190) | 5.071830 / 2.268929 (2.802902) | 2.839374 / 55.444624 (-52.605250) | 2.429583 / 6.876477 (-4.446893) | 2.671940 / 2.142072 (0.529868) | 0.588256 / 4.805227 (-4.216972) | 0.135135 / 6.500664 (-6.365530) | 0.060963 / 0.075469 (-0.014506) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.337462 / 1.841788 (-0.504326) | 20.292912 / 8.074308 (12.218604) | 14.871809 / 10.191392 (4.680417) | 0.169214 / 0.680424 (-0.511209) | 0.020450 / 0.534201 (-0.513751) | 0.397094 / 0.579283 (-0.182189) | 0.411623 / 0.434364 (-0.022741) | 0.471560 / 0.540337 (-0.068777) | 0.647293 / 1.386936 (-0.739643) |

![](https://cml.dev/watermark.png#0a068dbf3b446417ffd89d32857608394ec699e6 "CML watermark")

There is a typo in the section of the documentation dedicated to creating an audio dataset. The Dataset is incorrectly suffixed with a `Config`

https://huggingface.co/datasets/indonesian-nlp/librivox-indonesia/blob/main/librivox-indonesia.py#L59


Support saving datasets with custom formatting

Not a fan of pickling this sort of stuff either.
Note that users can also share the code in their dataset documentation.

Requested in https://discuss.huggingface.co/t/using-set-transform-on-a-dataset-leads-to-an-exception/53036.

I am not sure if supporting this is the best idea for the following reasons:

>For this to work, we would have to pickle a custom transform, which means the transform and the objects it references need to be serializable. Also, deserializing these bytes would make `load_from_disk` unsafe, so I'm not sure this is a good idea.

@lhoestq WDYT?


Set dev version

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6220). All of your documentation changes will be reflected on that endpoint.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005950 / 0.011353 (-0.005403) | 0.003578 / 0.011008 (-0.007431) | 0.079327 / 0.038508 (0.040819) | 0.057862 / 0.023109 (0.034752) | 0.317288 / 0.275898 (0.041390) | 0.358210 / 0.323480 (0.034730) | 0.004685 / 0.007986 (-0.003301) | 0.002879 / 0.004328 (-0.001450) | 0.062355 / 0.004250 (0.058105) | 0.045093 / 0.037052 (0.008041) | 0.322520 / 0.258489 (0.064031) | 0.367114 / 0.293841 (0.073273) | 0.027233 / 0.128546 (-0.101313) | 0.007941 / 0.075646 (-0.067705) | 0.260511 / 0.419271 (-0.158761) | 0.044355 / 0.043533 (0.000822) | 0.332993 / 0.255139 (0.077854) | 0.351363 / 0.283200 (0.068163) | 0.020784 / 0.141683 (-0.120899) | 1.429044 / 1.452155 (-0.023111) | 1.489355 / 1.492716 (-0.003362) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.180903 / 0.018006 (0.162897) | 0.421566 / 0.000490 (0.421077) | 0.003259 / 0.000200 (0.003059) | 0.000068 / 0.000054 (0.000014) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023765 / 0.037411 (-0.013646) | 0.072815 / 0.014526 (0.058289) | 0.084592 / 0.176557 (-0.091965) | 0.143556 / 0.737135 (-0.593579) | 0.083591 / 0.296338 (-0.212748) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.401896 / 0.215209 (0.186687) | 4.006344 / 2.077655 (1.928689) | 2.092280 / 1.504120 (0.588160) | 1.937828 / 1.541195 (0.396633) | 2.026901 / 1.468490 (0.558411) | 0.499999 / 4.584777 (-4.084778) | 3.008715 / 3.745712 (-0.736997) | 2.789735 / 5.269862 (-2.480127) | 1.827319 / 4.565676 (-2.738358) | 0.057413 / 0.424275 (-0.366862) | 0.006716 / 0.007607 (-0.000891) | 0.473061 / 0.226044 (0.247016) | 4.733256 / 2.268929 (2.464327) | 2.403922 / 55.444624 (-53.040702) | 2.017466 / 6.876477 (-4.859011) | 2.209710 / 2.142072 (0.067638) | 0.590813 / 4.805227 (-4.214414) | 0.124760 / 6.500664 (-6.375904) | 0.060976 / 0.075469 (-0.014494) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.229172 / 1.841788 (-0.612616) | 17.924644 / 8.074308 (9.850336) | 13.697347 / 10.191392 (3.505955) | 0.128258 / 0.680424 (-0.552166) | 0.016780 / 0.534201 (-0.517421) | 0.329301 / 0.579283 (-0.249982) | 0.344527 / 0.434364 (-0.089837) | 0.379482 / 0.540337 (-0.160855) | 0.513851 / 1.386936 (-0.873085) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005962 / 0.011353 (-0.005391) | 0.003613 / 0.011008 (-0.007396) | 0.062428 / 0.038508 (0.023920) | 0.058151 / 0.023109 (0.035042) | 0.452926 / 0.275898 (0.177027) | 0.489740 / 0.323480 (0.166260) | 0.006137 / 0.007986 (-0.001848) | 0.002890 / 0.004328 (-0.001438) | 0.062880 / 0.004250 (0.058629) | 0.046175 / 0.037052 (0.009123) | 0.452416 / 0.258489 (0.193927) | 0.486047 / 0.293841 (0.192206) | 0.028517 / 0.128546 (-0.100029) | 0.008102 / 0.075646 (-0.067544) | 0.068251 / 0.419271 (-0.351020) | 0.040569 / 0.043533 (-0.002964) | 0.461306 / 0.255139 (0.206167) | 0.477675 / 0.283200 (0.194475) | 0.020944 / 0.141683 (-0.120739) | 1.414300 / 1.452155 (-0.037855) | 1.502108 / 1.492716 (0.009391) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.217786 / 0.018006 (0.199780) | 0.410757 / 0.000490 (0.410267) | 0.002981 / 0.000200 (0.002781) | 0.000081 / 0.000054 (0.000027) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026846 / 0.037411 (-0.010565) | 0.080098 / 0.014526 (0.065572) | 0.090591 / 0.176557 (-0.085965) | 0.144674 / 0.737135 (-0.592461) | 0.091287 / 0.296338 (-0.205052) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.458224 / 0.215209 (0.243015) | 4.590541 / 2.077655 (2.512886) | 2.511251 / 1.504120 (1.007131) | 2.329165 / 1.541195 (0.787970) | 2.379187 / 1.468490 (0.910696) | 0.507485 / 4.584777 (-4.077292) | 3.135011 / 3.745712 (-0.610701) | 2.805913 / 5.269862 (-2.463948) | 1.851382 / 4.565676 (-2.714295) | 0.057981 / 0.424275 (-0.366294) | 0.006557 / 0.007607 (-0.001050) | 0.532496 / 0.226044 (0.306452) | 5.348802 / 2.268929 (3.079874) | 2.993379 / 55.444624 (-52.451245) | 2.636372 / 6.876477 (-4.240104) | 2.753219 / 2.142072 (0.611147) | 0.591989 / 4.805227 (-4.213238) | 0.126691 / 6.500664 (-6.373973) | 0.062359 / 0.075469 (-0.013110) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.345498 / 1.841788 (-0.496290) | 18.335767 / 8.074308 (10.261458) | 15.115449 / 10.191392 (4.924057) | 0.147382 / 0.680424 (-0.533041) | 0.017729 / 0.534201 (-0.516472) | 0.334337 / 0.579283 (-0.244946) | 0.359035 / 0.434364 (-0.075329) | 0.386319 / 0.540337 (-0.154019) | 0.536378 / 1.386936 (-0.850558) |

![](https://cml.dev/watermark.png#f2b028fd83d74e7701e7b8f2d87e740a989505a7 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009136 / 0.011353 (-0.002216) | 0.005567 / 0.011008 (-0.005442) | 0.120320 / 0.038508 (0.081812) | 0.078082 / 0.023109 (0.054973) | 0.405579 / 0.275898 (0.129681) | 0.459714 / 0.323480 (0.136234) | 0.006327 / 0.007986 (-0.001659) | 0.007187 / 0.004328 (0.002859) | 0.084373 / 0.004250 (0.080122) | 0.059727 / 0.037052 (0.022675) | 0.418918 / 0.258489 (0.160429) | 0.486767 / 0.293841 (0.192927) | 0.047715 / 0.128546 (-0.080831) | 0.014417 / 0.075646 (-0.061229) | 0.379847 / 0.419271 (-0.039425) | 0.067472 / 0.043533 (0.023939) | 0.419304 / 0.255139 (0.164166) | 0.466260 / 0.283200 (0.183060) | 0.036872 / 0.141683 (-0.104811) | 1.876273 / 1.452155 (0.424119) | 2.043856 / 1.492716 (0.551140) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.296266 / 0.018006 (0.278260) | 0.601843 / 0.000490 (0.601354) | 0.005663 / 0.000200 (0.005463) | 0.000102 / 0.000054 (0.000048) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033272 / 0.037411 (-0.004139) | 0.098839 / 0.014526 (0.084313) | 0.124658 / 0.176557 (-0.051899) | 0.190226 / 0.737135 (-0.546909) | 0.119288 / 0.296338 (-0.177051) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.600878 / 0.215209 (0.385668) | 6.011749 / 2.077655 (3.934095) | 2.611809 / 1.504120 (1.107689) | 2.314985 / 1.541195 (0.773790) | 2.398988 / 1.468490 (0.930498) | 0.835577 / 4.584777 (-3.749200) | 5.482848 / 3.745712 (1.737136) | 4.965393 / 5.269862 (-0.304469) | 3.082420 / 4.565676 (-1.483256) | 0.098048 / 0.424275 (-0.326227) | 0.009148 / 0.007607 (0.001541) | 0.725721 / 0.226044 (0.499676) | 7.297429 / 2.268929 (5.028501) | 3.558050 / 55.444624 (-51.886575) | 2.815884 / 6.876477 (-4.060593) | 3.094103 / 2.142072 (0.952031) | 1.023617 / 4.805227 (-3.781610) | 0.222453 / 6.500664 (-6.278211) | 0.081707 / 0.075469 (0.006238) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.788327 / 1.841788 (-0.053461) | 25.285829 / 8.074308 (17.211521) | 21.878811 / 10.191392 (11.687419) | 0.215494 / 0.680424 (-0.464930) | 0.032050 / 0.534201 (-0.502151) | 0.505210 / 0.579283 (-0.074073) | 0.623545 / 0.434364 (0.189181) | 0.583342 / 0.540337 (0.043005) | 0.826497 / 1.386936 (-0.560439) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009640 / 0.011353 (-0.001713) | 0.005479 / 0.011008 (-0.005529) | 0.088940 / 0.038508 (0.050432) | 0.084186 / 0.023109 (0.061077) | 0.552290 / 0.275898 (0.276392) | 0.583296 / 0.323480 (0.259816) | 0.006999 / 0.007986 (-0.000987) | 0.004597 / 0.004328 (0.000269) | 0.089407 / 0.004250 (0.085157) | 0.067210 / 0.037052 (0.030157) | 0.554968 / 0.258489 (0.296479) | 0.595635 / 0.293841 (0.301794) | 0.052245 / 0.128546 (-0.076301) | 0.015914 / 0.075646 (-0.059733) | 0.097037 / 0.419271 (-0.322235) | 0.063954 / 0.043533 (0.020421) | 0.533752 / 0.255139 (0.278614) | 0.573789 / 0.283200 (0.290589) | 0.036526 / 0.141683 (-0.105157) | 1.867713 / 1.452155 (0.415558) | 1.996901 / 1.492716 (0.504185) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.414967 / 0.018006 (0.396961) | 0.632367 / 0.000490 (0.631877) | 0.064061 / 0.000200 (0.063861) | 0.000565 / 0.000054 (0.000510) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035953 / 0.037411 (-0.001458) | 0.112603 / 0.014526 (0.098077) | 0.126227 / 0.176557 (-0.050330) | 0.196881 / 0.737135 (-0.540255) | 0.127635 / 0.296338 (-0.168704) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.674735 / 0.215209 (0.459526) | 6.614578 / 2.077655 (4.536923) | 3.208198 / 1.504120 (1.704078) | 2.870412 / 1.541195 (1.329217) | 2.979358 / 1.468490 (1.510868) | 0.872589 / 4.584777 (-3.712187) | 5.501771 / 3.745712 (1.756059) | 4.865191 / 5.269862 (-0.404671) | 3.075281 / 4.565676 (-1.490396) | 0.098048 / 0.424275 (-0.326227) | 0.009121 / 0.007607 (0.001514) | 0.801639 / 0.226044 (0.575595) | 8.062040 / 2.268929 (5.793111) | 3.996693 / 55.444624 (-51.447931) | 3.343770 / 6.876477 (-3.532706) | 3.555977 / 2.142072 (1.413904) | 1.035050 / 4.805227 (-3.770177) | 0.227552 / 6.500664 (-6.273112) | 0.097733 / 0.075469 (0.022264) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.897210 / 1.841788 (0.055422) | 25.762459 / 8.074308 (17.688151) | 22.771290 / 10.191392 (12.579898) | 0.252650 / 0.680424 (-0.427773) | 0.032534 / 0.534201 (-0.501667) | 0.521047 / 0.579283 (-0.058236) | 0.620850 / 0.434364 (0.186486) | 0.612750 / 0.540337 (0.072413) | 0.837486 / 1.386936 (-0.549451) |

![](https://cml.dev/watermark.png#1f522e5bdd73c45f7ba0a03f2ecd4e7de7351f2e "CML watermark")


Release: 2.14.5

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6219). All of your documentation changes will be reflected on that endpoint.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009523 / 0.011353 (-0.001830) | 0.005105 / 0.011008 (-0.005903) | 0.122664 / 0.038508 (0.084156) | 0.084688 / 0.023109 (0.061579) | 0.412057 / 0.275898 (0.136159) | 0.449690 / 0.323480 (0.126210) | 0.006627 / 0.007986 (-0.001358) | 0.004150 / 0.004328 (-0.000178) | 0.082079 / 0.004250 (0.077829) | 0.065289 / 0.037052 (0.028237) | 0.432934 / 0.258489 (0.174445) | 0.492068 / 0.293841 (0.198227) | 0.048317 / 0.128546 (-0.080229) | 0.015582 / 0.075646 (-0.060064) | 0.372050 / 0.419271 (-0.047222) | 0.070649 / 0.043533 (0.027116) | 0.431754 / 0.255139 (0.176615) | 0.473349 / 0.283200 (0.190149) | 0.037293 / 0.141683 (-0.104390) | 1.807537 / 1.452155 (0.355382) | 1.923073 / 1.492716 (0.430357) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.271214 / 0.018006 (0.253208) | 0.592961 / 0.000490 (0.592471) | 0.004062 / 0.000200 (0.003862) | 0.000089 / 0.000054 (0.000035) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034766 / 0.037411 (-0.002645) | 0.093014 / 0.014526 (0.078488) | 0.131332 / 0.176557 (-0.045225) | 0.188110 / 0.737135 (-0.549025) | 0.117617 / 0.296338 (-0.178722) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.668223 / 0.215209 (0.453013) | 6.707031 / 2.077655 (4.629376) | 3.040178 / 1.504120 (1.536058) | 2.641776 / 1.541195 (1.100581) | 2.524057 / 1.468490 (1.055567) | 0.893592 / 4.584777 (-3.691185) | 5.535848 / 3.745712 (1.790136) | 4.867067 / 5.269862 (-0.402794) | 2.999933 / 4.565676 (-1.565743) | 0.103602 / 0.424275 (-0.320673) | 0.008887 / 0.007607 (0.001280) | 0.822214 / 0.226044 (0.596169) | 8.028476 / 2.268929 (5.759547) | 3.708895 / 55.444624 (-51.735730) | 2.858314 / 6.876477 (-4.018163) | 3.101727 / 2.142072 (0.959655) | 1.083136 / 4.805227 (-3.722091) | 0.219588 / 6.500664 (-6.281076) | 0.080151 / 0.075469 (0.004682) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.645819 / 1.841788 (-0.195969) | 24.407887 / 8.074308 (16.333579) | 22.371901 / 10.191392 (12.180509) | 0.219557 / 0.680424 (-0.460867) | 0.037867 / 0.534201 (-0.496334) | 0.484136 / 0.579283 (-0.095147) | 0.620546 / 0.434364 (0.186182) | 0.562272 / 0.540337 (0.021934) | 0.774256 / 1.386936 (-0.612680) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009381 / 0.011353 (-0.001972) | 0.005565 / 0.011008 (-0.005444) | 0.091057 / 0.038508 (0.052549) | 0.078085 / 0.023109 (0.054975) | 0.538929 / 0.275898 (0.263031) | 0.555155 / 0.323480 (0.231675) | 0.007007 / 0.007986 (-0.000978) | 0.004268 / 0.004328 (-0.000060) | 0.086618 / 0.004250 (0.082368) | 0.064117 / 0.037052 (0.027065) | 0.523788 / 0.258489 (0.265299) | 0.586451 / 0.293841 (0.292610) | 0.050804 / 0.128546 (-0.077742) | 0.013964 / 0.075646 (-0.061682) | 0.096008 / 0.419271 (-0.323263) | 0.062242 / 0.043533 (0.018709) | 0.530398 / 0.255139 (0.275259) | 0.568527 / 0.283200 (0.285327) | 0.032456 / 0.141683 (-0.109227) | 1.894975 / 1.452155 (0.442820) | 2.084172 / 1.492716 (0.591455) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.295539 / 0.018006 (0.277533) | 0.588804 / 0.000490 (0.588314) | 0.006445 / 0.000200 (0.006245) | 0.000113 / 0.000054 (0.000059) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033965 / 0.037411 (-0.003447) | 0.111743 / 0.014526 (0.097217) | 0.128805 / 0.176557 (-0.047752) | 0.185013 / 0.737135 (-0.552123) | 0.129400 / 0.296338 (-0.166938) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.749784 / 0.215209 (0.534575) | 7.091075 / 2.077655 (5.013420) | 3.424517 / 1.504120 (1.920397) | 3.069103 / 1.541195 (1.527908) | 3.122431 / 1.468490 (1.653941) | 0.949277 / 4.584777 (-3.635500) | 5.648731 / 3.745712 (1.903019) | 4.937684 / 5.269862 (-0.332178) | 3.198027 / 4.565676 (-1.367650) | 0.100289 / 0.424275 (-0.323987) | 0.009411 / 0.007607 (0.001803) | 0.862604 / 0.226044 (0.636559) | 8.615410 / 2.268929 (6.346482) | 4.306428 / 55.444624 (-51.138196) | 3.591404 / 6.876477 (-3.285073) | 3.823899 / 2.142072 (1.681827) | 1.108006 / 4.805227 (-3.697221) | 0.215330 / 6.500664 (-6.285334) | 0.080755 / 0.075469 (0.005286) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.774914 / 1.841788 (-0.066873) | 25.360983 / 8.074308 (17.286675) | 23.624044 / 10.191392 (13.432652) | 0.226887 / 0.680424 (-0.453537) | 0.032625 / 0.534201 (-0.501576) | 0.499730 / 0.579283 (-0.079553) | 0.647819 / 0.434364 (0.213455) | 0.592239 / 0.540337 (0.051901) | 0.805751 / 1.386936 (-0.581185) |

![](https://cml.dev/watermark.png#0daa82428a0529478801574bcc68e1ed32051f3a "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008656 / 0.011353 (-0.002697) | 0.005545 / 0.011008 (-0.005463) | 0.107936 / 0.038508 (0.069428) | 0.077436 / 0.023109 (0.054327) | 0.391412 / 0.275898 (0.115514) | 0.452811 / 0.323480 (0.129331) | 0.004883 / 0.007986 (-0.003103) | 0.005125 / 0.004328 (0.000796) | 0.080006 / 0.004250 (0.075755) | 0.054425 / 0.037052 (0.017373) | 0.399667 / 0.258489 (0.141178) | 0.458099 / 0.293841 (0.164258) | 0.047302 / 0.128546 (-0.081244) | 0.014153 / 0.075646 (-0.061493) | 0.337281 / 0.419271 (-0.081991) | 0.062153 / 0.043533 (0.018620) | 0.399927 / 0.255139 (0.144788) | 0.407186 / 0.283200 (0.123987) | 0.036759 / 0.141683 (-0.104924) | 1.825935 / 1.452155 (0.373780) | 1.852238 / 1.492716 (0.359522) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.274163 / 0.018006 (0.256157) | 0.615624 / 0.000490 (0.615134) | 0.003782 / 0.000200 (0.003582) | 0.000115 / 0.000054 (0.000060) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026386 / 0.037411 (-0.011026) | 0.101151 / 0.014526 (0.086625) | 0.106115 / 0.176557 (-0.070442) | 0.161253 / 0.737135 (-0.575882) | 0.108861 / 0.296338 (-0.187478) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.587079 / 0.215209 (0.371870) | 6.141743 / 2.077655 (4.064089) | 2.727199 / 1.504120 (1.223079) | 2.526827 / 1.541195 (0.985632) | 2.598321 / 1.468490 (1.129831) | 0.904706 / 4.584777 (-3.680071) | 5.227742 / 3.745712 (1.482030) | 4.621627 / 5.269862 (-0.648234) | 2.931792 / 4.565676 (-1.633885) | 0.089538 / 0.424275 (-0.334737) | 0.008281 / 0.007607 (0.000674) | 0.675773 / 0.226044 (0.449729) | 7.212869 / 2.268929 (4.943941) | 3.541569 / 55.444624 (-51.903056) | 2.804034 / 6.876477 (-4.072443) | 3.080192 / 2.142072 (0.938120) | 1.034577 / 4.805227 (-3.770650) | 0.218727 / 6.500664 (-6.281937) | 0.084548 / 0.075469 (0.009079) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.528974 / 1.841788 (-0.312814) | 21.754329 / 8.074308 (13.680021) | 20.359808 / 10.191392 (10.168416) | 0.234719 / 0.680424 (-0.445705) | 0.026182 / 0.534201 (-0.508019) | 0.448956 / 0.579283 (-0.130327) | 0.577015 / 0.434364 (0.142651) | 0.513675 / 0.540337 (-0.026662) | 0.729780 / 1.386936 (-0.657156) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010427 / 0.011353 (-0.000926) | 0.005126 / 0.011008 (-0.005882) | 0.082759 / 0.038508 (0.044251) | 0.084892 / 0.023109 (0.061783) | 0.543826 / 0.275898 (0.267927) | 0.603050 / 0.323480 (0.279570) | 0.006667 / 0.007986 (-0.001319) | 0.004036 / 0.004328 (-0.000292) | 0.079534 / 0.004250 (0.075283) | 0.067523 / 0.037052 (0.030471) | 0.544845 / 0.258489 (0.286356) | 0.578823 / 0.293841 (0.284982) | 0.054786 / 0.128546 (-0.073760) | 0.014888 / 0.075646 (-0.060759) | 0.095696 / 0.419271 (-0.323576) | 0.064908 / 0.043533 (0.021375) | 0.558087 / 0.255139 (0.302948) | 0.593919 / 0.283200 (0.310719) | 0.039190 / 0.141683 (-0.102493) | 1.828680 / 1.452155 (0.376526) | 1.908891 / 1.492716 (0.416174) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.298926 / 0.018006 (0.280920) | 0.589467 / 0.000490 (0.588977) | 0.005276 / 0.000200 (0.005076) | 0.000112 / 0.000054 (0.000057) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034300 / 0.037411 (-0.003111) | 0.096990 / 0.014526 (0.082464) | 0.109347 / 0.176557 (-0.067209) | 0.171312 / 0.737135 (-0.565823) | 0.121736 / 0.296338 (-0.174603) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.641619 / 0.215209 (0.426410) | 6.365556 / 2.077655 (4.287901) | 2.947989 / 1.504120 (1.443869) | 2.631680 / 1.541195 (1.090485) | 2.602762 / 1.468490 (1.134272) | 0.812767 / 4.584777 (-3.772010) | 5.185753 / 3.745712 (1.440041) | 4.589897 / 5.269862 (-0.679964) | 2.833020 / 4.565676 (-1.732656) | 0.097782 / 0.424275 (-0.326493) | 0.008625 / 0.007607 (0.001018) | 0.741613 / 0.226044 (0.515568) | 7.662905 / 2.268929 (5.393976) | 3.533753 / 55.444624 (-51.910871) | 2.898929 / 6.876477 (-3.977547) | 3.042616 / 2.142072 (0.900544) | 0.933932 / 4.805227 (-3.871296) | 0.195710 / 6.500664 (-6.304954) | 0.066954 / 0.075469 (-0.008515) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.745353 / 1.841788 (-0.096434) | 23.820840 / 8.074308 (15.746532) | 20.892645 / 10.191392 (10.701253) | 0.234853 / 0.680424 (-0.445571) | 0.029149 / 0.534201 (-0.505051) | 0.458953 / 0.579283 (-0.120330) | 0.594278 / 0.434364 (0.159914) | 0.522929 / 0.540337 (-0.017409) | 0.753731 / 1.386936 (-0.633205) |

![](https://cml.dev/watermark.png#de6391d732ea0471ee5bdfb91b8cecc4503da96b "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005976 / 0.011353 (-0.005377) | 0.003636 / 0.011008 (-0.007372) | 0.079946 / 0.038508 (0.041437) | 0.060143 / 0.023109 (0.037034) | 0.314752 / 0.275898 (0.038854) | 0.353714 / 0.323480 (0.030234) | 0.004706 / 0.007986 (-0.003280) | 0.002862 / 0.004328 (-0.001466) | 0.061988 / 0.004250 (0.057737) | 0.045907 / 0.037052 (0.008855) | 0.316118 / 0.258489 (0.057629) | 0.358488 / 0.293841 (0.064647) | 0.027377 / 0.128546 (-0.101170) | 0.007970 / 0.075646 (-0.067677) | 0.261677 / 0.419271 (-0.157594) | 0.045289 / 0.043533 (0.001757) | 0.307931 / 0.255139 (0.052792) | 0.341364 / 0.283200 (0.058165) | 0.021021 / 0.141683 (-0.120662) | 1.440002 / 1.452155 (-0.012153) | 1.502904 / 1.492716 (0.010187) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.201746 / 0.018006 (0.183740) | 0.451114 / 0.000490 (0.450624) | 0.003351 / 0.000200 (0.003151) | 0.000067 / 0.000054 (0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024233 / 0.037411 (-0.013178) | 0.075042 / 0.014526 (0.060516) | 0.085636 / 0.176557 (-0.090920) | 0.144699 / 0.737135 (-0.592436) | 0.085222 / 0.296338 (-0.211117) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.389464 / 0.215209 (0.174255) | 3.889072 / 2.077655 (1.811417) | 1.908307 / 1.504120 (0.404187) | 1.738914 / 1.541195 (0.197719) | 1.866869 / 1.468490 (0.398379) | 0.500536 / 4.584777 (-4.084240) | 3.050155 / 3.745712 (-0.695557) | 2.832259 / 5.269862 (-2.437602) | 1.886657 / 4.565676 (-2.679020) | 0.059214 / 0.424275 (-0.365062) | 0.006711 / 0.007607 (-0.000896) | 0.467753 / 0.226044 (0.241709) | 4.666939 / 2.268929 (2.398011) | 2.471168 / 55.444624 (-52.973456) | 2.223508 / 6.876477 (-4.652968) | 2.176543 / 2.142072 (0.034470) | 0.593461 / 4.805227 (-4.211766) | 0.126216 / 6.500664 (-6.374448) | 0.061495 / 0.075469 (-0.013974) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.301279 / 1.841788 (-0.540509) | 18.317461 / 8.074308 (10.243153) | 13.877813 / 10.191392 (3.686421) | 0.143510 / 0.680424 (-0.536914) | 0.016826 / 0.534201 (-0.517375) | 0.328735 / 0.579283 (-0.250548) | 0.342272 / 0.434364 (-0.092092) | 0.375768 / 0.540337 (-0.164570) | 0.517600 / 1.386936 (-0.869336) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006215 / 0.011353 (-0.005138) | 0.003587 / 0.011008 (-0.007422) | 0.062248 / 0.038508 (0.023740) | 0.059830 / 0.023109 (0.036721) | 0.443278 / 0.275898 (0.167380) | 0.481279 / 0.323480 (0.157799) | 0.004773 / 0.007986 (-0.003213) | 0.002870 / 0.004328 (-0.001459) | 0.062730 / 0.004250 (0.058480) | 0.049422 / 0.037052 (0.012369) | 0.444196 / 0.258489 (0.185707) | 0.498614 / 0.293841 (0.204773) | 0.028477 / 0.128546 (-0.100069) | 0.008009 / 0.075646 (-0.067638) | 0.067919 / 0.419271 (-0.351352) | 0.040416 / 0.043533 (-0.003117) | 0.439460 / 0.255139 (0.184321) | 0.470529 / 0.283200 (0.187329) | 0.020767 / 0.141683 (-0.120916) | 1.478223 / 1.452155 (0.026068) | 1.538580 / 1.492716 (0.045863) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.271321 / 0.018006 (0.253315) | 0.456436 / 0.000490 (0.455946) | 0.011817 / 0.000200 (0.011617) | 0.000115 / 0.000054 (0.000061) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026355 / 0.037411 (-0.011056) | 0.081681 / 0.014526 (0.067155) | 0.091699 / 0.176557 (-0.084858) | 0.146115 / 0.737135 (-0.591021) | 0.094376 / 0.296338 (-0.201963) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.471677 / 0.215209 (0.256468) | 4.702909 / 2.077655 (2.625254) | 2.664882 / 1.504120 (1.160762) | 2.504106 / 1.541195 (0.962911) | 2.573226 / 1.468490 (1.104736) | 0.509679 / 4.584777 (-4.075097) | 3.034970 / 3.745712 (-0.710742) | 2.894704 / 5.269862 (-2.375157) | 1.915148 / 4.565676 (-2.650528) | 0.058312 / 0.424275 (-0.365963) | 0.006615 / 0.007607 (-0.000993) | 0.545339 / 0.226044 (0.319295) | 5.462261 / 2.268929 (3.193332) | 3.101482 / 55.444624 (-52.343143) | 2.755417 / 6.876477 (-4.121060) | 2.931440 / 2.142072 (0.789368) | 0.597521 / 4.805227 (-4.207707) | 0.125676 / 6.500664 (-6.374988) | 0.061798 / 0.075469 (-0.013671) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.356208 / 1.841788 (-0.485579) | 18.912492 / 8.074308 (10.838184) | 14.830128 / 10.191392 (4.638736) | 0.145992 / 0.680424 (-0.534432) | 0.019121 / 0.534201 (-0.515080) | 0.331534 / 0.579283 (-0.247749) | 0.361712 / 0.434364 (-0.072652) | 0.387532 / 0.540337 (-0.152805) | 0.536075 / 1.386936 (-0.850861) |

![](https://cml.dev/watermark.png#de6391d732ea0471ee5bdfb91b8cecc4503da96b "CML watermark")


Rename old push_to_hub configs to "default" in dataset_infos

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006529 / 0.011353 (-0.004823) | 0.004010 / 0.011008 (-0.006998) | 0.086258 / 0.038508 (0.047750) | 0.073775 / 0.023109 (0.050666) | 0.307573 / 0.275898 (0.031675) | 0.337091 / 0.323480 (0.013611) | 0.004251 / 0.007986 (-0.003735) | 0.003886 / 0.004328 (-0.000443) | 0.068238 / 0.004250 (0.063987) | 0.057000 / 0.037052 (0.019948) | 0.321751 / 0.258489 (0.063262) | 0.359227 / 0.293841 (0.065386) | 0.030841 / 0.128546 (-0.097705) | 0.008569 / 0.075646 (-0.067078) | 0.299523 / 0.419271 (-0.119748) | 0.052563 / 0.043533 (0.009030) | 0.312806 / 0.255139 (0.057667) | 0.342273 / 0.283200 (0.059074) | 0.025725 / 0.141683 (-0.115958) | 1.479263 / 1.452155 (0.027108) | 1.554975 / 1.492716 (0.062259) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.316328 / 0.018006 (0.298322) | 0.598993 / 0.000490 (0.598503) | 0.004548 / 0.000200 (0.004348) | 0.000080 / 0.000054 (0.000026) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027399 / 0.037411 (-0.010013) | 0.081683 / 0.014526 (0.067157) | 0.096968 / 0.176557 (-0.079589) | 0.151559 / 0.737135 (-0.585576) | 0.096558 / 0.296338 (-0.199781) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.383117 / 0.215209 (0.167908) | 3.818634 / 2.077655 (1.740979) | 1.878112 / 1.504120 (0.373992) | 1.729031 / 1.541195 (0.187836) | 1.770259 / 1.468490 (0.301769) | 0.484061 / 4.584777 (-4.100716) | 3.596998 / 3.745712 (-0.148715) | 3.246846 / 5.269862 (-2.023016) | 2.019481 / 4.565676 (-2.546195) | 0.057279 / 0.424275 (-0.366996) | 0.007455 / 0.007607 (-0.000152) | 0.465002 / 0.226044 (0.238958) | 4.644669 / 2.268929 (2.375741) | 2.346415 / 55.444624 (-53.098209) | 2.039686 / 6.876477 (-4.836791) | 2.172822 / 2.142072 (0.030750) | 0.582925 / 4.805227 (-4.222302) | 0.134246 / 6.500664 (-6.366418) | 0.060093 / 0.075469 (-0.015376) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.249033 / 1.841788 (-0.592755) | 19.585949 / 8.074308 (11.511641) | 14.100681 / 10.191392 (3.909289) | 0.147138 / 0.680424 (-0.533286) | 0.018307 / 0.534201 (-0.515894) | 0.397939 / 0.579283 (-0.181344) | 0.413916 / 0.434364 (-0.020448) | 0.465688 / 0.540337 (-0.074650) | 0.642140 / 1.386936 (-0.744797) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006627 / 0.011353 (-0.004726) | 0.004173 / 0.011008 (-0.006835) | 0.063850 / 0.038508 (0.025342) | 0.074733 / 0.023109 (0.051623) | 0.398111 / 0.275898 (0.122213) | 0.426344 / 0.323480 (0.102864) | 0.006261 / 0.007986 (-0.001725) | 0.003507 / 0.004328 (-0.000822) | 0.064511 / 0.004250 (0.060260) | 0.056508 / 0.037052 (0.019456) | 0.401750 / 0.258489 (0.143261) | 0.437081 / 0.293841 (0.143240) | 0.031815 / 0.128546 (-0.096732) | 0.008703 / 0.075646 (-0.066943) | 0.071411 / 0.419271 (-0.347861) | 0.048153 / 0.043533 (0.004620) | 0.399221 / 0.255139 (0.144082) | 0.429312 / 0.283200 (0.146112) | 0.022157 / 0.141683 (-0.119526) | 1.485656 / 1.452155 (0.033502) | 1.550967 / 1.492716 (0.058250) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.330575 / 0.018006 (0.312569) | 0.525553 / 0.000490 (0.525064) | 0.004574 / 0.000200 (0.004374) | 0.000093 / 0.000054 (0.000038) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031871 / 0.037411 (-0.005541) | 0.091819 / 0.014526 (0.077293) | 0.105542 / 0.176557 (-0.071015) | 0.158210 / 0.737135 (-0.578926) | 0.107167 / 0.296338 (-0.189172) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.430226 / 0.215209 (0.215017) | 4.293456 / 2.077655 (2.215801) | 2.289538 / 1.504120 (0.785418) | 2.122255 / 1.541195 (0.581060) | 2.181840 / 1.468490 (0.713350) | 0.498529 / 4.584777 (-4.086248) | 3.686636 / 3.745712 (-0.059077) | 3.287279 / 5.269862 (-1.982582) | 2.068397 / 4.565676 (-2.497280) | 0.058775 / 0.424275 (-0.365500) | 0.007583 / 0.007607 (-0.000024) | 0.507165 / 0.226044 (0.281121) | 5.072330 / 2.268929 (2.803401) | 2.796396 / 55.444624 (-52.648228) | 2.409946 / 6.876477 (-4.466531) | 2.657322 / 2.142072 (0.515250) | 0.597744 / 4.805227 (-4.207483) | 0.133803 / 6.500664 (-6.366861) | 0.060231 / 0.075469 (-0.015238) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.333130 / 1.841788 (-0.508658) | 20.545936 / 8.074308 (12.471627) | 14.875020 / 10.191392 (4.683628) | 0.168873 / 0.680424 (-0.511551) | 0.020316 / 0.534201 (-0.513885) | 0.397203 / 0.579283 (-0.182080) | 0.412412 / 0.434364 (-0.021952) | 0.479952 / 0.540337 (-0.060385) | 0.657155 / 1.386936 (-0.729781) |

![](https://cml.dev/watermark.png#13fbee4ca8742460e9baab86a89d9100a294df3e "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007885 / 0.011353 (-0.003468) | 0.005221 / 0.011008 (-0.005787) | 0.099457 / 0.038508 (0.060949) | 0.085867 / 0.023109 (0.062758) | 0.359922 / 0.275898 (0.084024) | 0.406479 / 0.323480 (0.082999) | 0.005001 / 0.007986 (-0.002985) | 0.003678 / 0.004328 (-0.000650) | 0.075647 / 0.004250 (0.071396) | 0.064318 / 0.037052 (0.027265) | 0.372180 / 0.258489 (0.113691) | 0.419206 / 0.293841 (0.125365) | 0.040438 / 0.128546 (-0.088108) | 0.010008 / 0.075646 (-0.065638) | 0.340911 / 0.419271 (-0.078360) | 0.063326 / 0.043533 (0.019793) | 0.359015 / 0.255139 (0.103876) | 0.408601 / 0.283200 (0.125402) | 0.029828 / 0.141683 (-0.111855) | 1.767822 / 1.452155 (0.315667) | 1.829079 / 1.492716 (0.336363) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.234455 / 0.018006 (0.216449) | 0.507786 / 0.000490 (0.507297) | 0.004009 / 0.000200 (0.003809) | 0.000101 / 0.000054 (0.000046) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033374 / 0.037411 (-0.004038) | 0.100817 / 0.014526 (0.086291) | 0.113415 / 0.176557 (-0.063141) | 0.180368 / 0.737135 (-0.556768) | 0.115446 / 0.296338 (-0.180893) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.488976 / 0.215209 (0.273767) | 4.911354 / 2.077655 (2.833699) | 2.623525 / 1.504120 (1.119405) | 2.424400 / 1.541195 (0.883206) | 2.497580 / 1.468490 (1.029089) | 0.561106 / 4.584777 (-4.023671) | 4.265649 / 3.745712 (0.519937) | 3.830267 / 5.269862 (-1.439595) | 2.404727 / 4.565676 (-2.160949) | 0.067303 / 0.424275 (-0.356972) | 0.009177 / 0.007607 (0.001570) | 0.588433 / 0.226044 (0.362388) | 5.871573 / 2.268929 (3.602645) | 3.087845 / 55.444624 (-52.356779) | 2.765381 / 6.876477 (-4.111096) | 3.007863 / 2.142072 (0.865791) | 0.687327 / 4.805227 (-4.117901) | 0.157687 / 6.500664 (-6.342977) | 0.071291 / 0.075469 (-0.004178) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.510931 / 1.841788 (-0.330857) | 22.129590 / 8.074308 (14.055282) | 16.780479 / 10.191392 (6.589087) | 0.168297 / 0.680424 (-0.512127) | 0.021294 / 0.534201 (-0.512907) | 0.464535 / 0.579283 (-0.114748) | 0.480041 / 0.434364 (0.045677) | 0.549185 / 0.540337 (0.008848) | 0.739438 / 1.386936 (-0.647498) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007834 / 0.011353 (-0.003518) | 0.004576 / 0.011008 (-0.006432) | 0.073331 / 0.038508 (0.034823) | 0.084688 / 0.023109 (0.061579) | 0.486367 / 0.275898 (0.210469) | 0.523127 / 0.323480 (0.199647) | 0.006278 / 0.007986 (-0.001708) | 0.003792 / 0.004328 (-0.000537) | 0.075416 / 0.004250 (0.071166) | 0.064053 / 0.037052 (0.027001) | 0.491908 / 0.258489 (0.233419) | 0.529177 / 0.293841 (0.235336) | 0.038483 / 0.128546 (-0.090063) | 0.009560 / 0.075646 (-0.066087) | 0.083431 / 0.419271 (-0.335841) | 0.057114 / 0.043533 (0.013581) | 0.486316 / 0.255139 (0.231177) | 0.512384 / 0.283200 (0.229185) | 0.028452 / 0.141683 (-0.113231) | 1.788886 / 1.452155 (0.336731) | 1.893834 / 1.492716 (0.401118) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.343018 / 0.018006 (0.325011) | 0.513673 / 0.000490 (0.513183) | 0.056778 / 0.000200 (0.056578) | 0.001799 / 0.000054 (0.001745) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.038530 / 0.037411 (0.001119) | 0.109286 / 0.014526 (0.094760) | 0.122812 / 0.176557 (-0.053745) | 0.187780 / 0.737135 (-0.549355) | 0.124083 / 0.296338 (-0.172255) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.509839 / 0.215209 (0.294630) | 5.085840 / 2.077655 (3.008186) | 2.746695 / 1.504120 (1.242575) | 2.542283 / 1.541195 (1.001088) | 2.650243 / 1.468490 (1.181753) | 0.592801 / 4.584777 (-3.991976) | 4.316721 / 3.745712 (0.571009) | 3.811672 / 5.269862 (-1.458189) | 2.433982 / 4.565676 (-2.131695) | 0.066861 / 0.424275 (-0.357414) | 0.008633 / 0.007607 (0.001026) | 0.590482 / 0.226044 (0.364437) | 5.923484 / 2.268929 (3.654556) | 3.282293 / 55.444624 (-52.162332) | 2.882716 / 6.876477 (-3.993761) | 3.139581 / 2.142072 (0.997509) | 0.690702 / 4.805227 (-4.114525) | 0.156781 / 6.500664 (-6.343883) | 0.071487 / 0.075469 (-0.003982) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.604557 / 1.841788 (-0.237231) | 24.000026 / 8.074308 (15.925718) | 17.548685 / 10.191392 (7.357293) | 0.174883 / 0.680424 (-0.505541) | 0.023812 / 0.534201 (-0.510389) | 0.473522 / 0.579283 (-0.105761) | 0.494683 / 0.434364 (0.060319) | 0.593352 / 0.540337 (0.053015) | 0.771852 / 1.386936 (-0.615084) |

![](https://cml.dev/watermark.png#b61c96a806fa97800bc8a66607fb0c78a5d04146 "CML watermark")

thanks! i wonder if we should also fix (change config name) all the old `dataset_infos.json` on the Hub?

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006388 / 0.011353 (-0.004965) | 0.003876 / 0.011008 (-0.007132) | 0.083960 / 0.038508 (0.045452) | 0.068328 / 0.023109 (0.045219) | 0.337958 / 0.275898 (0.062060) | 0.370783 / 0.323480 (0.047303) | 0.003925 / 0.007986 (-0.004060) | 0.004221 / 0.004328 (-0.000107) | 0.064198 / 0.004250 (0.059947) | 0.052681 / 0.037052 (0.015629) | 0.348890 / 0.258489 (0.090401) | 0.389038 / 0.293841 (0.095197) | 0.031133 / 0.128546 (-0.097413) | 0.008566 / 0.075646 (-0.067080) | 0.288169 / 0.419271 (-0.131102) | 0.053290 / 0.043533 (0.009757) | 0.344654 / 0.255139 (0.089515) | 0.381287 / 0.283200 (0.098087) | 0.022350 / 0.141683 (-0.119333) | 1.459933 / 1.452155 (0.007778) | 1.543097 / 1.492716 (0.050380) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.212592 / 0.018006 (0.194586) | 0.461863 / 0.000490 (0.461373) | 0.003468 / 0.000200 (0.003268) | 0.000084 / 0.000054 (0.000029) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026849 / 0.037411 (-0.010563) | 0.081059 / 0.014526 (0.066533) | 0.093986 / 0.176557 (-0.082571) | 0.150328 / 0.737135 (-0.586807) | 0.094253 / 0.296338 (-0.202085) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.382198 / 0.215209 (0.166989) | 3.813878 / 2.077655 (1.736224) | 1.855686 / 1.504120 (0.351566) | 1.672995 / 1.541195 (0.131800) | 1.697705 / 1.468490 (0.229215) | 0.479920 / 4.584777 (-4.104857) | 3.608305 / 3.745712 (-0.137407) | 3.216712 / 5.269862 (-2.053149) | 1.984781 / 4.565676 (-2.580896) | 0.056801 / 0.424275 (-0.367475) | 0.007499 / 0.007607 (-0.000108) | 0.454155 / 0.226044 (0.228110) | 4.531147 / 2.268929 (2.262218) | 2.296149 / 55.444624 (-53.148475) | 1.968701 / 6.876477 (-4.907775) | 2.144286 / 2.142072 (0.002213) | 0.599254 / 4.805227 (-4.205973) | 0.138150 / 6.500664 (-6.362514) | 0.060118 / 0.075469 (-0.015351) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.282486 / 1.841788 (-0.559301) | 19.127792 / 8.074308 (11.053483) | 14.116521 / 10.191392 (3.925129) | 0.163792 / 0.680424 (-0.516632) | 0.018116 / 0.534201 (-0.516085) | 0.390789 / 0.579283 (-0.188494) | 0.409241 / 0.434364 (-0.025123) | 0.457824 / 0.540337 (-0.082513) | 0.624390 / 1.386936 (-0.762546) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006637 / 0.011353 (-0.004716) | 0.003932 / 0.011008 (-0.007076) | 0.063456 / 0.038508 (0.024948) | 0.070062 / 0.023109 (0.046953) | 0.410570 / 0.275898 (0.134672) | 0.436700 / 0.323480 (0.113220) | 0.005324 / 0.007986 (-0.002662) | 0.003263 / 0.004328 (-0.001065) | 0.063590 / 0.004250 (0.059340) | 0.054823 / 0.037052 (0.017770) | 0.408720 / 0.258489 (0.150231) | 0.441493 / 0.293841 (0.147652) | 0.031655 / 0.128546 (-0.096891) | 0.008421 / 0.075646 (-0.067225) | 0.070657 / 0.419271 (-0.348614) | 0.047370 / 0.043533 (0.003837) | 0.408217 / 0.255139 (0.153078) | 0.422178 / 0.283200 (0.138978) | 0.022282 / 0.141683 (-0.119401) | 1.511417 / 1.452155 (0.059262) | 1.570337 / 1.492716 (0.077620) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.224334 / 0.018006 (0.206327) | 0.447589 / 0.000490 (0.447099) | 0.004227 / 0.000200 (0.004027) | 0.000099 / 0.000054 (0.000045) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030797 / 0.037411 (-0.006615) | 0.091276 / 0.014526 (0.076750) | 0.102665 / 0.176557 (-0.073892) | 0.155423 / 0.737135 (-0.581712) | 0.103779 / 0.296338 (-0.192560) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.434509 / 0.215209 (0.219300) | 4.328910 / 2.077655 (2.251255) | 2.311424 / 1.504120 (0.807304) | 2.138380 / 1.541195 (0.597185) | 2.196293 / 1.468490 (0.727803) | 0.482123 / 4.584777 (-4.102654) | 3.597870 / 3.745712 (-0.147842) | 3.222426 / 5.269862 (-2.047435) | 1.994467 / 4.565676 (-2.571210) | 0.057517 / 0.424275 (-0.366758) | 0.007336 / 0.007607 (-0.000271) | 0.504968 / 0.226044 (0.278923) | 5.047940 / 2.268929 (2.779012) | 2.824014 / 55.444624 (-52.620610) | 2.457762 / 6.876477 (-4.418714) | 2.606970 / 2.142072 (0.464897) | 0.580758 / 4.805227 (-4.224469) | 0.132584 / 6.500664 (-6.368080) | 0.059258 / 0.075469 (-0.016211) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.354386 / 1.841788 (-0.487402) | 19.738147 / 8.074308 (11.663839) | 14.858001 / 10.191392 (4.666609) | 0.166074 / 0.680424 (-0.514350) | 0.020181 / 0.534201 (-0.514020) | 0.398333 / 0.579283 (-0.180950) | 0.406969 / 0.434364 (-0.027395) | 0.474515 / 0.540337 (-0.065822) | 0.649571 / 1.386936 (-0.737365) |

![](https://cml.dev/watermark.png#b3ac3b3a9c5f40a29fae71504574cfdeebefe349 "CML watermark")

I would say we should delete all `dataset_infos.json` on the Hub...

@albertvillanova @lhoestq @mariosasko should we really stop supporting it and delete from everywhere?
(bc if not, I've found a bug in updating `dataset_infos.json` with `.push_to_hub` and I'd open a PR to fix it)

We can only delete them for the datasets without namespace and open PRs for the others, so we need to keep supporting them for now

Fix

```python
from datasets import load_dataset_builder

b = load_dataset_builder("lambdalabs/pokemon-blip-captions", "default")
print(b.info)
```

which should return

```
DatasetInfo(
  features={'image': Image(decode=True, id=None), 'text': Value(dtype='string', id=None)},
  dataset_name='pokemon-blip-captions',
  config_name='default',
  version=0.0.0,
  splits={'train': SplitInfo(name='train', num_bytes=119417410.0, num_examples=833, shard_lengths=None, dataset_name='pokemon-blip-captions')},
  download_checksums=None,
  download_size=99672355,
  dataset_size=119417410.0,
  size_in_bytes=219089765.0,
  ...
)
```

instead of and empty dataset info.

The dataset has a dataset_infos.json file with a deprecated config name "lambdalabs--pokemon-blip-captions". We switched those config names to "default" in 2.14, so the builder.info should take this into account.


`Dataset.to_dict()` ignore `decode=True` with Image feature

We need to implement the `Image` type as a PyArrow extension type (to allow us to override the Python conversion) for this to work as expected. For now, it's best to use your approach indeed.

### Describe the bug

`Dataset.to_dict` seems to ignore the decoding instruction passed in features.

### Steps to reproduce the bug

```python
import datasets
import numpy as np
from PIL import Image

img = np.random.randint(0, 256, (5, 5, 3), dtype=np.uint8)
img = Image.fromarray(img)

features = datasets.Features({"image": datasets.Image(decode=True)})
dataset = datasets.Dataset.from_dict({"image": [img]}, features=features)
print({key: dataset[key] for key in dataset.column_names})
# {'image': [<PIL.PngImagePlugin.PngImageFile image mode=RGB size=5x5 at 0x7EFBC80E15B0>]}
print(dataset.to_dict())
# {'image': [{'bytes': b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x05\x00\x00\x00\x05\x08\x02\x00\x00\x00\x02\r\xb1\xb2\x00\x00\x00[IDATx\x9c\x01P\x00\xaf\xff\x01\x13\x1b<7\xe7\xe0\xdc^6\xed\x04\xc7M\xd2\x9f\x00X\x1b\xb0?\x1ba\x15\xc5 o\xd0\x80\xbe\x19/\x01\xec\x95\x1f\x9f\xffj\xfa1\xa7\xc4X\xea\xbe\xa4g\x00\xc4\x15\xdeC\xc7 \xbbaqe\xc8\xb9\xa9q\xe7\x00,?M\xc0)\xdaD`}\xb1\xdci\x1e\xafC\xa9]%.@\xa6\xf0\xb3\x00\x00\x00\x00IEND\xaeB`\x82', 'path': None}]}
```

### Expected behavior

I would expect `{key: dataset[key] for key in dataset.column_names}` and `dataset.to_dict()` to be equivalent. If the previous behavior is expected, then it should be stated [in the doc](https://huggingface.co/docs/datasets/v2.14.4/en/package_reference/main_classes#datasets.Dataset.to_dict).

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-6.2.0-31-generic-x86_64-with-glibc2.35
- Python version: 3.9.12
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
- Pillow 9.5.0
- numpy 1.25.2