Release: 2.13.2

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007801 / 0.011353 (-0.003552) | 0.004831 / 0.011008 (-0.006177) | 0.123101 / 0.038508 (0.084593) | 0.053246 / 0.023109 (0.030137) | 0.381787 / 0.275898 (0.105889) | 0.461822 / 0.323480 (0.138342) | 0.004655 / 0.007986 (-0.003331) | 0.004818 / 0.004328 (0.000490) | 0.090865 / 0.004250 (0.086614) | 0.070626 / 0.037052 (0.033574) | 0.409122 / 0.258489 (0.150633) | 0.449627 / 0.293841 (0.155787) | 0.037477 / 0.128546 (-0.091069) | 0.010677 / 0.075646 (-0.064970) | 0.419970 / 0.419271 (0.000699) | 0.064626 / 0.043533 (0.021093) | 0.379536 / 0.255139 (0.124397) | 0.405790 / 0.283200 (0.122590) | 0.027290 / 0.141683 (-0.114393) | 1.884973 / 1.452155 (0.432819) | 1.960547 / 1.492716 (0.467831) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.259393 / 0.018006 (0.241386) | 0.502130 / 0.000490 (0.501640) | 0.013053 / 0.000200 (0.012853) | 0.000336 / 0.000054 (0.000281) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033459 / 0.037411 (-0.003953) | 0.135888 / 0.014526 (0.121362) | 0.145354 / 0.176557 (-0.031203) | 0.213289 / 0.737135 (-0.523847) | 0.151239 / 0.296338 (-0.145100) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.510817 / 0.215209 (0.295608) | 5.077888 / 2.077655 (3.000234) | 2.502991 / 1.504120 (0.998871) | 2.275566 / 1.541195 (0.734371) | 2.353025 / 1.468490 (0.884535) | 0.659062 / 4.584777 (-3.925715) | 4.411399 / 3.745712 (0.665686) | 2.227395 / 5.269862 (-3.042467) | 1.306771 / 4.565676 (-3.258905) | 0.081121 / 0.424275 (-0.343154) | 0.014252 / 0.007607 (0.006645) | 0.635040 / 0.226044 (0.408996) | 6.357500 / 2.268929 (4.088572) | 3.056647 / 55.444624 (-52.387977) | 2.671997 / 6.876477 (-4.204480) | 2.847955 / 2.142072 (0.705883) | 0.808163 / 4.805227 (-3.997064) | 0.177176 / 6.500664 (-6.323488) | 0.079984 / 0.075469 (0.004515) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.490471 / 1.841788 (-0.351317) | 17.927433 / 8.074308 (9.853124) | 17.744967 / 10.191392 (7.553575) | 0.171034 / 0.680424 (-0.509390) | 0.021432 / 0.534201 (-0.512769) | 0.515745 / 0.579283 (-0.063538) | 0.504746 / 0.434364 (0.070382) | 0.630862 / 0.540337 (0.090524) | 0.755275 / 1.386936 (-0.631662) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008227 / 0.011353 (-0.003126) | 0.004864 / 0.011008 (-0.006144) | 0.092801 / 0.038508 (0.054293) | 0.054996 / 0.023109 (0.031887) | 0.500348 / 0.275898 (0.224450) | 0.565028 / 0.323480 (0.241548) | 0.004792 / 0.007986 (-0.003194) | 0.005052 / 0.004328 (0.000723) | 0.090640 / 0.004250 (0.086390) | 0.074427 / 0.037052 (0.037374) | 0.499908 / 0.258489 (0.241419) | 0.566260 / 0.293841 (0.272419) | 0.040011 / 0.128546 (-0.088536) | 0.010438 / 0.075646 (-0.065208) | 0.099385 / 0.419271 (-0.319887) | 0.060485 / 0.043533 (0.016952) | 0.480603 / 0.255139 (0.225464) | 0.508807 / 0.283200 (0.225607) | 0.025976 / 0.141683 (-0.115707) | 1.870860 / 1.452155 (0.418705) | 1.943460 / 1.492716 (0.450744) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.227753 / 0.018006 (0.209747) | 0.501859 / 0.000490 (0.501369) | 0.008211 / 0.000200 (0.008011) | 0.000127 / 0.000054 (0.000073) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.038329 / 0.037411 (0.000918) | 0.148214 / 0.014526 (0.133688) | 0.162704 / 0.176557 (-0.013852) | 0.218543 / 0.737135 (-0.518592) | 0.162992 / 0.296338 (-0.133347) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.553195 / 0.215209 (0.337986) | 5.568080 / 2.077655 (3.490425) | 2.936616 / 1.504120 (1.432496) | 2.712624 / 1.541195 (1.171429) | 2.713245 / 1.468490 (1.244755) | 0.648593 / 4.584777 (-3.936184) | 4.641361 / 3.745712 (0.895648) | 2.207064 / 5.269862 (-3.062798) | 1.315325 / 4.565676 (-3.250351) | 0.080285 / 0.424275 (-0.343990) | 0.014143 / 0.007607 (0.006536) | 0.672467 / 0.226044 (0.446423) | 6.730262 / 2.268929 (4.461333) | 3.344468 / 55.444624 (-52.100157) | 2.927837 / 6.876477 (-3.948640) | 3.124735 / 2.142072 (0.982662) | 0.795894 / 4.805227 (-4.009333) | 0.170985 / 6.500664 (-6.329679) | 0.077406 / 0.075469 (0.001937) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.598059 / 1.841788 (-0.243729) | 18.531854 / 8.074308 (10.457546) | 18.394895 / 10.191392 (8.203503) | 0.195702 / 0.680424 (-0.484722) | 0.023633 / 0.534201 (-0.510568) | 0.518110 / 0.579283 (-0.061173) | 0.517773 / 0.434364 (0.083409) | 0.617902 / 0.540337 (0.077565) | 0.736459 / 1.386936 (-0.650477) |

![](https://cml.dev/watermark.png#5d4bb03237b74c0009043d50c5b4e4339cb98b2b "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006943 / 0.011353 (-0.004410) | 0.004524 / 0.011008 (-0.006485) | 0.121603 / 0.038508 (0.083095) | 0.047462 / 0.023109 (0.024353) | 0.362393 / 0.275898 (0.086495) | 0.440577 / 0.323480 (0.117098) | 0.004153 / 0.007986 (-0.003832) | 0.003778 / 0.004328 (-0.000550) | 0.090402 / 0.004250 (0.086152) | 0.066268 / 0.037052 (0.029216) | 0.380721 / 0.258489 (0.122232) | 0.442959 / 0.293841 (0.149118) | 0.035228 / 0.128546 (-0.093318) | 0.010217 / 0.075646 (-0.065429) | 0.408587 / 0.419271 (-0.010684) | 0.062609 / 0.043533 (0.019076) | 0.372682 / 0.255139 (0.117543) | 0.389270 / 0.283200 (0.106070) | 0.026699 / 0.141683 (-0.114984) | 1.760476 / 1.452155 (0.308321) | 1.795081 / 1.492716 (0.302365) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.229912 / 0.018006 (0.211906) | 0.476837 / 0.000490 (0.476348) | 0.008178 / 0.000200 (0.007978) | 0.000100 / 0.000054 (0.000045) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031116 / 0.037411 (-0.006296) | 0.126767 / 0.014526 (0.112241) | 0.134242 / 0.176557 (-0.042315) | 0.202120 / 0.737135 (-0.535016) | 0.142777 / 0.296338 (-0.153561) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.470690 / 0.215209 (0.255481) | 4.723198 / 2.077655 (2.645543) | 2.163870 / 1.504120 (0.659750) | 1.914177 / 1.541195 (0.372982) | 2.034529 / 1.468490 (0.566038) | 0.620472 / 4.584777 (-3.964305) | 4.391008 / 3.745712 (0.645296) | 2.100966 / 5.269862 (-3.168896) | 1.225945 / 4.565676 (-3.339732) | 0.076279 / 0.424275 (-0.347996) | 0.013551 / 0.007607 (0.005944) | 0.600989 / 0.226044 (0.374945) | 5.946715 / 2.268929 (3.677787) | 2.665117 / 55.444624 (-52.779508) | 2.320004 / 6.876477 (-4.556473) | 2.413131 / 2.142072 (0.271059) | 0.771908 / 4.805227 (-4.033320) | 0.165438 / 6.500664 (-6.335226) | 0.074512 / 0.075469 (-0.000957) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.432728 / 1.841788 (-0.409060) | 17.398133 / 8.074308 (9.323824) | 16.819152 / 10.191392 (6.627760) | 0.191849 / 0.680424 (-0.488575) | 0.021557 / 0.534201 (-0.512644) | 0.514380 / 0.579283 (-0.064903) | 0.501453 / 0.434364 (0.067089) | 0.634091 / 0.540337 (0.093753) | 0.756786 / 1.386936 (-0.630150) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007946 / 0.011353 (-0.003407) | 0.004751 / 0.011008 (-0.006257) | 0.090190 / 0.038508 (0.051682) | 0.052841 / 0.023109 (0.029732) | 0.480150 / 0.275898 (0.204252) | 0.537509 / 0.323480 (0.214029) | 0.004833 / 0.007986 (-0.003153) | 0.004796 / 0.004328 (0.000467) | 0.090616 / 0.004250 (0.086366) | 0.074325 / 0.037052 (0.037273) | 0.483776 / 0.258489 (0.225287) | 0.552094 / 0.293841 (0.258254) | 0.039240 / 0.128546 (-0.089307) | 0.010416 / 0.075646 (-0.065230) | 0.100275 / 0.419271 (-0.318996) | 0.058086 / 0.043533 (0.014553) | 0.468989 / 0.255139 (0.213850) | 0.485502 / 0.283200 (0.202302) | 0.027514 / 0.141683 (-0.114169) | 1.849625 / 1.452155 (0.397470) | 1.919515 / 1.492716 (0.426798) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.248061 / 0.018006 (0.230055) | 0.475630 / 0.000490 (0.475141) | 0.006248 / 0.000200 (0.006048) | 0.000105 / 0.000054 (0.000050) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.037746 / 0.037411 (0.000335) | 0.141638 / 0.014526 (0.127112) | 0.149530 / 0.176557 (-0.027026) | 0.209255 / 0.737135 (-0.527880) | 0.156447 / 0.296338 (-0.139892) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.544640 / 0.215209 (0.329431) | 5.493152 / 2.077655 (3.415497) | 2.869733 / 1.504120 (1.365613) | 2.624216 / 1.541195 (1.083022) | 2.710818 / 1.468490 (1.242328) | 0.640626 / 4.584777 (-3.944151) | 4.516130 / 3.745712 (0.770418) | 2.128097 / 5.269862 (-3.141765) | 1.278990 / 4.565676 (-3.286686) | 0.077114 / 0.424275 (-0.347161) | 0.013280 / 0.007607 (0.005673) | 0.655552 / 0.226044 (0.429507) | 6.526875 / 2.268929 (4.257947) | 3.347072 / 55.444624 (-52.097553) | 2.992435 / 6.876477 (-3.884041) | 3.124351 / 2.142072 (0.982278) | 0.778523 / 4.805227 (-4.026704) | 0.161873 / 6.500664 (-6.338791) | 0.072897 / 0.075469 (-0.002572) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.587058 / 1.841788 (-0.254730) | 18.170612 / 8.074308 (10.096304) | 17.220483 / 10.191392 (7.029091) | 0.207863 / 0.680424 (-0.472561) | 0.023746 / 0.534201 (-0.510455) | 0.512607 / 0.579283 (-0.066676) | 0.513258 / 0.434364 (0.078894) | 0.597880 / 0.540337 (0.057543) | 0.714974 / 1.386936 (-0.671962) |

![](https://cml.dev/watermark.png#98b1bdd492df953ca7139bb8c9a1771d5c603797 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006224 / 0.011353 (-0.005128) | 0.003857 / 0.011008 (-0.007151) | 0.099786 / 0.038508 (0.061278) | 0.037919 / 0.023109 (0.014810) | 0.315294 / 0.275898 (0.039396) | 0.390178 / 0.323480 (0.066698) | 0.005358 / 0.007986 (-0.002628) | 0.002989 / 0.004328 (-0.001340) | 0.077834 / 0.004250 (0.073583) | 0.053315 / 0.037052 (0.016263) | 0.325155 / 0.258489 (0.066666) | 0.374712 / 0.293841 (0.080871) | 0.029176 / 0.128546 (-0.099370) | 0.008658 / 0.075646 (-0.066988) | 0.314245 / 0.419271 (-0.105027) | 0.046684 / 0.043533 (0.003151) | 0.316473 / 0.255139 (0.061334) | 0.346119 / 0.283200 (0.062919) | 0.022452 / 0.141683 (-0.119230) | 1.540497 / 1.452155 (0.088343) | 1.594888 / 1.492716 (0.102172) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.204349 / 0.018006 (0.186343) | 0.426842 / 0.000490 (0.426353) | 0.003060 / 0.000200 (0.002860) | 0.000073 / 0.000054 (0.000019) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023611 / 0.037411 (-0.013801) | 0.100247 / 0.014526 (0.085721) | 0.107824 / 0.176557 (-0.068733) | 0.166845 / 0.737135 (-0.570291) | 0.112782 / 0.296338 (-0.183556) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.423053 / 0.215209 (0.207844) | 4.235553 / 2.077655 (2.157899) | 1.936589 / 1.504120 (0.432469) | 1.738519 / 1.541195 (0.197325) | 1.787905 / 1.468490 (0.319415) | 0.573362 / 4.584777 (-4.011414) | 3.395272 / 3.745712 (-0.350440) | 1.765977 / 5.269862 (-3.503884) | 1.049596 / 4.565676 (-3.516081) | 0.068868 / 0.424275 (-0.355407) | 0.011028 / 0.007607 (0.003421) | 0.532835 / 0.226044 (0.306791) | 5.314890 / 2.268929 (3.045962) | 2.368733 / 55.444624 (-53.075891) | 2.033959 / 6.876477 (-4.842518) | 2.130481 / 2.142072 (-0.011591) | 0.689360 / 4.805227 (-4.115867) | 0.140271 / 6.500664 (-6.360393) | 0.068198 / 0.075469 (-0.007271) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.237212 / 1.841788 (-0.604576) | 14.182215 / 8.074308 (6.107907) | 14.972608 / 10.191392 (4.781216) | 0.133977 / 0.680424 (-0.546447) | 0.016759 / 0.534201 (-0.517442) | 0.361552 / 0.579283 (-0.217731) | 0.394932 / 0.434364 (-0.039432) | 0.442601 / 0.540337 (-0.097736) | 0.535709 / 1.386936 (-0.851227) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006327 / 0.011353 (-0.005026) | 0.003780 / 0.011008 (-0.007228) | 0.078358 / 0.038508 (0.039850) | 0.037271 / 0.023109 (0.014162) | 0.456766 / 0.275898 (0.180868) | 0.515721 / 0.323480 (0.192241) | 0.004770 / 0.007986 (-0.003216) | 0.002942 / 0.004328 (-0.001387) | 0.077383 / 0.004250 (0.073132) | 0.051773 / 0.037052 (0.014721) | 0.460722 / 0.258489 (0.202233) | 0.519997 / 0.293841 (0.226157) | 0.030461 / 0.128546 (-0.098085) | 0.008622 / 0.075646 (-0.067024) | 0.083271 / 0.419271 (-0.336000) | 0.042242 / 0.043533 (-0.001291) | 0.447691 / 0.255139 (0.192552) | 0.481965 / 0.283200 (0.198765) | 0.019510 / 0.141683 (-0.122173) | 1.536718 / 1.452155 (0.084563) | 1.588433 / 1.492716 (0.095717) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.215880 / 0.018006 (0.197874) | 0.426102 / 0.000490 (0.425612) | 0.003976 / 0.000200 (0.003776) | 0.000079 / 0.000054 (0.000025) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026168 / 0.037411 (-0.011243) | 0.105786 / 0.014526 (0.091260) | 0.113772 / 0.176557 (-0.062785) | 0.166576 / 0.737135 (-0.570559) | 0.117560 / 0.296338 (-0.178779) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.490485 / 0.215209 (0.275276) | 4.890105 / 2.077655 (2.812450) | 2.515099 / 1.504120 (1.010979) | 2.306591 / 1.541195 (0.765396) | 2.383634 / 1.468490 (0.915144) | 0.573780 / 4.584777 (-4.010997) | 3.474394 / 3.745712 (-0.271318) | 1.746795 / 5.269862 (-3.523067) | 1.044678 / 4.565676 (-3.520998) | 0.069176 / 0.424275 (-0.355099) | 0.011045 / 0.007607 (0.003438) | 0.597234 / 0.226044 (0.371189) | 5.979614 / 2.268929 (3.710685) | 3.024203 / 55.444624 (-52.420422) | 2.687502 / 6.876477 (-4.188975) | 2.781637 / 2.142072 (0.639565) | 0.690482 / 4.805227 (-4.114745) | 0.150138 / 6.500664 (-6.350526) | 0.077076 / 0.075469 (0.001607) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.307501 / 1.841788 (-0.534287) | 14.366780 / 8.074308 (6.292471) | 14.966981 / 10.191392 (4.775589) | 0.153829 / 0.680424 (-0.526594) | 0.018047 / 0.534201 (-0.516154) | 0.361391 / 0.579283 (-0.217892) | 0.398345 / 0.434364 (-0.036019) | 0.424574 / 0.540337 (-0.115764) | 0.517165 / 1.386936 (-0.869771) |

![](https://cml.dev/watermark.png#98b1bdd492df953ca7139bb8c9a1771d5c603797 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006944 / 0.011353 (-0.004409) | 0.004504 / 0.011008 (-0.006504) | 0.105224 / 0.038508 (0.066716) | 0.047830 / 0.023109 (0.024721) | 0.339723 / 0.275898 (0.063825) | 0.419249 / 0.323480 (0.095769) | 0.005510 / 0.007986 (-0.002476) | 0.003574 / 0.004328 (-0.000754) | 0.079879 / 0.004250 (0.075628) | 0.066610 / 0.037052 (0.029557) | 0.353818 / 0.258489 (0.095329) | 0.397992 / 0.293841 (0.104151) | 0.031551 / 0.128546 (-0.096995) | 0.009037 / 0.075646 (-0.066610) | 0.355310 / 0.419271 (-0.063961) | 0.054931 / 0.043533 (0.011398) | 0.335153 / 0.255139 (0.080014) | 0.357460 / 0.283200 (0.074260) | 0.026031 / 0.141683 (-0.115652) | 1.546705 / 1.452155 (0.094550) | 1.627324 / 1.492716 (0.134608) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.276708 / 0.018006 (0.258701) | 0.589402 / 0.000490 (0.588912) | 0.009560 / 0.000200 (0.009360) | 0.000095 / 0.000054 (0.000041) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031041 / 0.037411 (-0.006370) | 0.117219 / 0.014526 (0.102693) | 0.125200 / 0.176557 (-0.051356) | 0.181528 / 0.737135 (-0.555607) | 0.131898 / 0.296338 (-0.164440) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.409965 / 0.215209 (0.194756) | 4.102700 / 2.077655 (2.025045) | 1.887578 / 1.504120 (0.383458) | 1.696490 / 1.541195 (0.155295) | 1.821352 / 1.468490 (0.352862) | 0.545422 / 4.584777 (-4.039355) | 3.933784 / 3.745712 (0.188071) | 1.934254 / 5.269862 (-3.335607) | 1.114935 / 4.565676 (-3.450742) | 0.067615 / 0.424275 (-0.356660) | 0.012004 / 0.007607 (0.004397) | 0.522048 / 0.226044 (0.296004) | 5.209224 / 2.268929 (2.940296) | 2.369911 / 55.444624 (-53.074714) | 2.032960 / 6.876477 (-4.843517) | 2.228874 / 2.142072 (0.086802) | 0.673172 / 4.805227 (-4.132055) | 0.147017 / 6.500664 (-6.353647) | 0.067020 / 0.075469 (-0.008449) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.281490 / 1.841788 (-0.560298) | 16.129701 / 8.074308 (8.055393) | 15.474730 / 10.191392 (5.283338) | 0.143934 / 0.680424 (-0.536490) | 0.018311 / 0.534201 (-0.515890) | 0.435940 / 0.579283 (-0.143343) | 0.446846 / 0.434364 (0.012482) | 0.543943 / 0.540337 (0.003605) | 0.648041 / 1.386936 (-0.738895) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007380 / 0.011353 (-0.003973) | 0.004510 / 0.011008 (-0.006499) | 0.080741 / 0.038508 (0.042233) | 0.050907 / 0.023109 (0.027797) | 0.425548 / 0.275898 (0.149650) | 0.487959 / 0.323480 (0.164479) | 0.005887 / 0.007986 (-0.002099) | 0.003689 / 0.004328 (-0.000639) | 0.079588 / 0.004250 (0.075338) | 0.071841 / 0.037052 (0.034788) | 0.425172 / 0.258489 (0.166683) | 0.471185 / 0.293841 (0.177344) | 0.035768 / 0.128546 (-0.092779) | 0.009229 / 0.075646 (-0.066418) | 0.086021 / 0.419271 (-0.333250) | 0.052424 / 0.043533 (0.008891) | 0.413634 / 0.255139 (0.158495) | 0.422310 / 0.283200 (0.139111) | 0.026019 / 0.141683 (-0.115664) | 1.616861 / 1.452155 (0.164707) | 1.653660 / 1.492716 (0.160943) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.280096 / 0.018006 (0.262090) | 0.587853 / 0.000490 (0.587363) | 0.006560 / 0.000200 (0.006360) | 0.000181 / 0.000054 (0.000127) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033747 / 0.037411 (-0.003665) | 0.125089 / 0.014526 (0.110564) | 0.137995 / 0.176557 (-0.038561) | 0.188192 / 0.737135 (-0.548943) | 0.141438 / 0.296338 (-0.154900) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.471524 / 0.215209 (0.256315) | 4.713988 / 2.077655 (2.636334) | 2.414785 / 1.504120 (0.910665) | 2.226815 / 1.541195 (0.685620) | 2.259222 / 1.468490 (0.790732) | 0.551663 / 4.584777 (-4.033114) | 4.031399 / 3.745712 (0.285686) | 1.966917 / 5.269862 (-3.302945) | 1.154487 / 4.565676 (-3.411190) | 0.068500 / 0.424275 (-0.355775) | 0.012127 / 0.007607 (0.004520) | 0.579342 / 0.226044 (0.353298) | 5.757415 / 2.268929 (3.488486) | 2.820012 / 55.444624 (-52.624613) | 2.521783 / 6.876477 (-4.354694) | 2.699994 / 2.142072 (0.557921) | 0.686152 / 4.805227 (-4.119075) | 0.148521 / 6.500664 (-6.352143) | 0.068478 / 0.075469 (-0.006991) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.336260 / 1.841788 (-0.505528) | 17.016935 / 8.074308 (8.942627) | 16.406951 / 10.191392 (6.215559) | 0.166907 / 0.680424 (-0.513517) | 0.020166 / 0.534201 (-0.514035) | 0.437690 / 0.579283 (-0.141593) | 0.480337 / 0.434364 (0.045973) | 0.518065 / 0.540337 (-0.022272) | 0.625904 / 1.386936 (-0.761032) |

![](https://cml.dev/watermark.png#98b1bdd492df953ca7139bb8c9a1771d5c603797 "CML watermark")


Fix checking patterns to infer packaged builder

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006681 / 0.011353 (-0.004672) | 0.003967 / 0.011008 (-0.007041) | 0.085590 / 0.038508 (0.047082) | 0.079285 / 0.023109 (0.056176) | 0.311583 / 0.275898 (0.035685) | 0.345578 / 0.323480 (0.022098) | 0.004115 / 0.007986 (-0.003871) | 0.004286 / 0.004328 (-0.000043) | 0.064405 / 0.004250 (0.060155) | 0.055084 / 0.037052 (0.018032) | 0.316117 / 0.258489 (0.057628) | 0.354737 / 0.293841 (0.060896) | 0.031280 / 0.128546 (-0.097266) | 0.008395 / 0.075646 (-0.067251) | 0.288910 / 0.419271 (-0.130362) | 0.051291 / 0.043533 (0.007759) | 0.309125 / 0.255139 (0.053986) | 0.349673 / 0.283200 (0.066473) | 0.025016 / 0.141683 (-0.116667) | 1.475577 / 1.452155 (0.023422) | 1.558967 / 1.492716 (0.066251) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.208504 / 0.018006 (0.190498) | 0.462270 / 0.000490 (0.461780) | 0.003476 / 0.000200 (0.003276) | 0.000073 / 0.000054 (0.000018) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030371 / 0.037411 (-0.007041) | 0.086157 / 0.014526 (0.071631) | 0.098162 / 0.176557 (-0.078395) | 0.154649 / 0.737135 (-0.582486) | 0.098697 / 0.296338 (-0.197642) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.405883 / 0.215209 (0.190674) | 4.049614 / 2.077655 (1.971959) | 2.075047 / 1.504120 (0.570927) | 1.917782 / 1.541195 (0.376587) | 2.030268 / 1.468490 (0.561778) | 0.483974 / 4.584777 (-4.100803) | 3.542147 / 3.745712 (-0.203566) | 3.305999 / 5.269862 (-1.963863) | 2.052287 / 4.565676 (-2.513390) | 0.057246 / 0.424275 (-0.367029) | 0.007631 / 0.007607 (0.000024) | 0.488189 / 0.226044 (0.262144) | 4.884784 / 2.268929 (2.615856) | 2.576304 / 55.444624 (-52.868320) | 2.241249 / 6.876477 (-4.635228) | 2.490512 / 2.142072 (0.348440) | 0.584495 / 4.805227 (-4.220733) | 0.134741 / 6.500664 (-6.365923) | 0.061639 / 0.075469 (-0.013830) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.317717 / 1.841788 (-0.524071) | 20.098594 / 8.074308 (12.024286) | 14.641051 / 10.191392 (4.449659) | 0.165291 / 0.680424 (-0.515133) | 0.019179 / 0.534201 (-0.515022) | 0.399506 / 0.579283 (-0.179777) | 0.407662 / 0.434364 (-0.026701) | 0.457965 / 0.540337 (-0.082372) | 0.626401 / 1.386936 (-0.760536) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007076 / 0.011353 (-0.004277) | 0.004125 / 0.011008 (-0.006884) | 0.064861 / 0.038508 (0.026353) | 0.082390 / 0.023109 (0.059281) | 0.423227 / 0.275898 (0.147329) | 0.452229 / 0.323480 (0.128750) | 0.005594 / 0.007986 (-0.002392) | 0.003465 / 0.004328 (-0.000863) | 0.064661 / 0.004250 (0.060411) | 0.057945 / 0.037052 (0.020892) | 0.424572 / 0.258489 (0.166083) | 0.465349 / 0.293841 (0.171509) | 0.032687 / 0.128546 (-0.095859) | 0.008573 / 0.075646 (-0.067074) | 0.073020 / 0.419271 (-0.346251) | 0.048423 / 0.043533 (0.004891) | 0.413425 / 0.255139 (0.158286) | 0.433778 / 0.283200 (0.150578) | 0.023942 / 0.141683 (-0.117741) | 1.495190 / 1.452155 (0.043036) | 1.586526 / 1.492716 (0.093810) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.271805 / 0.018006 (0.253799) | 0.454922 / 0.000490 (0.454432) | 0.015386 / 0.000200 (0.015186) | 0.000129 / 0.000054 (0.000074) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033804 / 0.037411 (-0.003607) | 0.099317 / 0.014526 (0.084791) | 0.107207 / 0.176557 (-0.069349) | 0.160926 / 0.737135 (-0.576210) | 0.108669 / 0.296338 (-0.187670) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.430776 / 0.215209 (0.215567) | 4.297622 / 2.077655 (2.219967) | 2.285918 / 1.504120 (0.781798) | 2.109608 / 1.541195 (0.568413) | 2.208326 / 1.468490 (0.739836) | 0.490016 / 4.584777 (-4.094761) | 3.570609 / 3.745712 (-0.175103) | 3.406335 / 5.269862 (-1.863526) | 2.070664 / 4.565676 (-2.495012) | 0.058089 / 0.424275 (-0.366186) | 0.007425 / 0.007607 (-0.000182) | 0.506972 / 0.226044 (0.280927) | 5.078643 / 2.268929 (2.809714) | 2.858973 / 55.444624 (-52.585651) | 2.457344 / 6.876477 (-4.419132) | 2.687727 / 2.142072 (0.545654) | 0.592134 / 4.805227 (-4.213093) | 0.133966 / 6.500664 (-6.366698) | 0.061800 / 0.075469 (-0.013669) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.337167 / 1.841788 (-0.504620) | 20.743951 / 8.074308 (12.669643) | 15.402686 / 10.191392 (5.211294) | 0.164548 / 0.680424 (-0.515876) | 0.020244 / 0.534201 (-0.513957) | 0.399044 / 0.579283 (-0.180239) | 0.414036 / 0.434364 (-0.020328) | 0.474141 / 0.540337 (-0.066197) | 0.654455 / 1.386936 (-0.732482) |

![](https://cml.dev/watermark.png#4de930c45a81a6dff1805bf45f59170e9f953eeb "CML watermark")

Don't ignore results of pattern resolving if `self.data_files` is not None. Otherwise lines 854 and 1037 make no sense.


Unpin fsspec < 2023.9.0

enhancement

New feature or request

Once root issue is fixed, remove temporary pin of fsspec < 2023.9.0 introduced by:
- #6210

Related to issue:
- #6209

After investigation, I think the root issue is related to the new glob behavior with double asterisk `**` they have introduced in:
- https://github.com/fsspec/filesystem_spec/pull/1329


Better list array values handling in cast/embed storage

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008451 / 0.011353 (-0.002902) | 0.005056 / 0.011008 (-0.005952) | 0.086367 / 0.038508 (0.047859) | 0.068030 / 0.023109 (0.044920) | 0.358812 / 0.275898 (0.082914) | 0.385790 / 0.323480 (0.062310) | 0.005608 / 0.007986 (-0.002378) | 0.004262 / 0.004328 (-0.000067) | 0.066618 / 0.004250 (0.062368) | 0.053901 / 0.037052 (0.016849) | 0.398456 / 0.258489 (0.139967) | 0.391681 / 0.293841 (0.097840) | 0.046743 / 0.128546 (-0.081804) | 0.014118 / 0.075646 (-0.061528) | 0.308479 / 0.419271 (-0.110793) | 0.064214 / 0.043533 (0.020681) | 0.367940 / 0.255139 (0.112801) | 0.387204 / 0.283200 (0.104004) | 0.036093 / 0.141683 (-0.105590) | 1.534182 / 1.452155 (0.082027) | 1.598357 / 1.492716 (0.105641) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.265910 / 0.018006 (0.247904) | 0.589453 / 0.000490 (0.588963) | 0.004881 / 0.000200 (0.004681) | 0.000090 / 0.000054 (0.000036) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032540 / 0.037411 (-0.004872) | 0.083153 / 0.014526 (0.068627) | 0.098960 / 0.176557 (-0.077597) | 0.162044 / 0.737135 (-0.575091) | 0.093602 / 0.296338 (-0.202736) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.517056 / 0.215209 (0.301847) | 5.167908 / 2.077655 (3.090253) | 2.359856 / 1.504120 (0.855736) | 2.092448 / 1.541195 (0.551253) | 2.100270 / 1.468490 (0.631780) | 0.742321 / 4.584777 (-3.842456) | 4.845010 / 3.745712 (1.099298) | 4.361808 / 5.269862 (-0.908054) | 2.621941 / 4.565676 (-1.943736) | 0.094907 / 0.424275 (-0.329369) | 0.009357 / 0.007607 (0.001750) | 0.719859 / 0.226044 (0.493814) | 6.929731 / 2.268929 (4.660802) | 3.240862 / 55.444624 (-52.203763) | 2.700817 / 6.876477 (-4.175659) | 2.904600 / 2.142072 (0.762527) | 0.924930 / 4.805227 (-3.880298) | 0.194390 / 6.500664 (-6.306274) | 0.078331 / 0.075469 (0.002862) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.539347 / 1.841788 (-0.302441) | 22.696358 / 8.074308 (14.622050) | 18.791692 / 10.191392 (8.600300) | 0.221376 / 0.680424 (-0.459048) | 0.029824 / 0.534201 (-0.504377) | 0.455604 / 0.579283 (-0.123679) | 0.573169 / 0.434364 (0.138805) | 0.507109 / 0.540337 (-0.033228) | 0.730986 / 1.386936 (-0.655950) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009308 / 0.011353 (-0.002045) | 0.005027 / 0.011008 (-0.005982) | 0.074094 / 0.038508 (0.035586) | 0.068277 / 0.023109 (0.045168) | 0.412716 / 0.275898 (0.136818) | 0.446883 / 0.323480 (0.123403) | 0.005864 / 0.007986 (-0.002122) | 0.003753 / 0.004328 (-0.000575) | 0.072575 / 0.004250 (0.068325) | 0.064434 / 0.037052 (0.027382) | 0.445395 / 0.258489 (0.186906) | 0.464520 / 0.293841 (0.170679) | 0.045303 / 0.128546 (-0.083243) | 0.013120 / 0.075646 (-0.062527) | 0.077830 / 0.419271 (-0.341441) | 0.057303 / 0.043533 (0.013770) | 0.420845 / 0.255139 (0.165706) | 0.431308 / 0.283200 (0.148109) | 0.033908 / 0.141683 (-0.107775) | 1.577667 / 1.452155 (0.125512) | 1.677321 / 1.492716 (0.184604) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.305855 / 0.018006 (0.287849) | 0.601442 / 0.000490 (0.600953) | 0.010722 / 0.000200 (0.010522) | 0.000158 / 0.000054 (0.000104) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029202 / 0.037411 (-0.008209) | 0.094576 / 0.014526 (0.080050) | 0.106734 / 0.176557 (-0.069822) | 0.168114 / 0.737135 (-0.569021) | 0.107241 / 0.296338 (-0.189098) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.643634 / 0.215209 (0.428425) | 6.391757 / 2.077655 (4.314103) | 3.011679 / 1.504120 (1.507559) | 2.379711 / 1.541195 (0.838517) | 2.387444 / 1.468490 (0.918954) | 0.823460 / 4.584777 (-3.761317) | 4.882240 / 3.745712 (1.136528) | 4.091170 / 5.269862 (-1.178691) | 2.688761 / 4.565676 (-1.876915) | 0.094555 / 0.424275 (-0.329720) | 0.008464 / 0.007607 (0.000857) | 0.665949 / 0.226044 (0.439905) | 6.948237 / 2.268929 (4.679309) | 3.384894 / 55.444624 (-52.059730) | 2.675570 / 6.876477 (-4.200907) | 3.073045 / 2.142072 (0.930973) | 0.969780 / 4.805227 (-3.835447) | 0.205859 / 6.500664 (-6.294805) | 0.072548 / 0.075469 (-0.002922) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.563869 / 1.841788 (-0.277919) | 22.431392 / 8.074308 (14.357084) | 19.434811 / 10.191392 (9.243419) | 0.255135 / 0.680424 (-0.425289) | 0.027799 / 0.534201 (-0.506402) | 0.427713 / 0.579283 (-0.151570) | 0.527030 / 0.434364 (0.092666) | 0.503660 / 0.540337 (-0.036678) | 0.730996 / 1.386936 (-0.655940) |

![](https://cml.dev/watermark.png#06c1940953807dbde4bc18af64bd3d87234edf00 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007597 / 0.011353 (-0.003756) | 0.004492 / 0.011008 (-0.006516) | 0.103779 / 0.038508 (0.065271) | 0.079287 / 0.023109 (0.056178) | 0.389651 / 0.275898 (0.113753) | 0.421955 / 0.323480 (0.098475) | 0.006023 / 0.007986 (-0.001963) | 0.003727 / 0.004328 (-0.000602) | 0.078604 / 0.004250 (0.074354) | 0.060810 / 0.037052 (0.023758) | 0.412170 / 0.258489 (0.153681) | 0.436218 / 0.293841 (0.142377) | 0.037282 / 0.128546 (-0.091264) | 0.010341 / 0.075646 (-0.065305) | 0.357652 / 0.419271 (-0.061620) | 0.063320 / 0.043533 (0.019788) | 0.389454 / 0.255139 (0.134315) | 0.433073 / 0.283200 (0.149874) | 0.028449 / 0.141683 (-0.113234) | 1.894107 / 1.452155 (0.441952) | 1.954190 / 1.492716 (0.461474) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.224477 / 0.018006 (0.206471) | 0.510878 / 0.000490 (0.510388) | 0.005013 / 0.000200 (0.004813) | 0.000092 / 0.000054 (0.000037) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032976 / 0.037411 (-0.004436) | 0.101073 / 0.014526 (0.086547) | 0.113990 / 0.176557 (-0.062566) | 0.183499 / 0.737135 (-0.553636) | 0.114283 / 0.296338 (-0.182056) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.473242 / 0.215209 (0.258033) | 4.719800 / 2.077655 (2.642146) | 2.318732 / 1.504120 (0.814612) | 2.102336 / 1.541195 (0.561141) | 2.143618 / 1.468490 (0.675128) | 0.594122 / 4.584777 (-3.990654) | 4.265961 / 3.745712 (0.520249) | 3.794635 / 5.269862 (-1.475226) | 2.394506 / 4.565676 (-2.171170) | 0.070091 / 0.424275 (-0.354184) | 0.009222 / 0.007607 (0.001614) | 0.564496 / 0.226044 (0.338452) | 5.644348 / 2.268929 (3.375419) | 2.934395 / 55.444624 (-52.510229) | 2.429076 / 6.876477 (-4.447401) | 2.592010 / 2.142072 (0.449937) | 0.713371 / 4.805227 (-4.091856) | 0.165019 / 6.500664 (-6.335646) | 0.075913 / 0.075469 (0.000444) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.570836 / 1.841788 (-0.270951) | 22.569763 / 8.074308 (14.495455) | 17.159658 / 10.191392 (6.968266) | 0.185716 / 0.680424 (-0.494708) | 0.021938 / 0.534201 (-0.512263) | 0.487204 / 0.579283 (-0.092079) | 0.472776 / 0.434364 (0.038412) | 0.565052 / 0.540337 (0.024714) | 0.763322 / 1.386936 (-0.623614) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007810 / 0.011353 (-0.003543) | 0.005140 / 0.011008 (-0.005869) | 0.079018 / 0.038508 (0.040510) | 0.080899 / 0.023109 (0.057790) | 0.489213 / 0.275898 (0.213315) | 0.525334 / 0.323480 (0.201854) | 0.006992 / 0.007986 (-0.000994) | 0.003729 / 0.004328 (-0.000599) | 0.079277 / 0.004250 (0.075026) | 0.064883 / 0.037052 (0.027831) | 0.496718 / 0.258489 (0.238229) | 0.534976 / 0.293841 (0.241135) | 0.038790 / 0.128546 (-0.089756) | 0.010122 / 0.075646 (-0.065524) | 0.087669 / 0.419271 (-0.331603) | 0.057959 / 0.043533 (0.014426) | 0.490611 / 0.255139 (0.235472) | 0.518376 / 0.283200 (0.235176) | 0.026561 / 0.141683 (-0.115122) | 1.843241 / 1.452155 (0.391086) | 1.952367 / 1.492716 (0.459651) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.289799 / 0.018006 (0.271792) | 0.486999 / 0.000490 (0.486509) | 0.017481 / 0.000200 (0.017281) | 0.000122 / 0.000054 (0.000068) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.037662 / 0.037411 (0.000250) | 0.113238 / 0.014526 (0.098712) | 0.123918 / 0.176557 (-0.052638) | 0.190484 / 0.737135 (-0.546652) | 0.126473 / 0.296338 (-0.169865) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.530622 / 0.215209 (0.315413) | 5.292093 / 2.077655 (3.214438) | 2.819354 / 1.504120 (1.315234) | 2.609821 / 1.541195 (1.068626) | 2.680090 / 1.468490 (1.211600) | 0.603490 / 4.584777 (-3.981287) | 4.344541 / 3.745712 (0.598828) | 3.874001 / 5.269862 (-1.395861) | 2.445302 / 4.565676 (-2.120375) | 0.071173 / 0.424275 (-0.353102) | 0.009131 / 0.007607 (0.001524) | 0.627273 / 0.226044 (0.401229) | 6.278637 / 2.268929 (4.009709) | 3.433762 / 55.444624 (-52.010862) | 2.973400 / 6.876477 (-3.903077) | 3.188165 / 2.142072 (1.046093) | 0.722824 / 4.805227 (-4.082404) | 0.165154 / 6.500664 (-6.335510) | 0.075268 / 0.075469 (-0.000202) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.652994 / 1.841788 (-0.188794) | 23.309030 / 8.074308 (15.234722) | 18.135649 / 10.191392 (7.944257) | 0.177543 / 0.680424 (-0.502881) | 0.024784 / 0.534201 (-0.509417) | 0.489952 / 0.579283 (-0.089331) | 0.485368 / 0.434364 (0.051004) | 0.580583 / 0.540337 (0.040246) | 0.787843 / 1.386936 (-0.599093) |

![](https://cml.dev/watermark.png#5982039f7814a204fe532240ca6aabe72430d834 "CML watermark")

A bug in `FixedSizeArray.flatten` in `PyArrow<10.0.0` makes CI fail. Colab installs 9.0.0 by default, so we should be able to set the minimal version to `10.0.0` soon. Keeping this PR as a draft in the meantime.

Closing this PR in favor of https://github.com/huggingface/datasets/pull/6283

Use [`array.flatten`](https://arrow.apache.org/docs/python/generated/pyarrow.ListArray.html#pyarrow.ListArray.flatten) that takes `.offset` into account instead of `array.values` in array cast/embed.


Tilde (~) is not supported for data_files

Hi @exs-avianello, is it really needed? Note you can alternatively use `pathlib.Path` among others as it follows:

```python
import datasets
from pathlib import Path

# save a parquet file at ~/path/to/data.parquet

data_files = Path.home() / "path/to/data.parquet"
dataset = datasets.load_dataset("parquet", data_files=data_files)
```

Hi @alvarobartt !

This is definitely just a "nice to have" and I am personally more than happy to just use absolute paths client-side. I just wanted to flag it up in case it can help improve the package even more 🙌 It might not be immediately obvious from the stack trace that the error is triggered by the `~` in the path

### Describe the bug

Attempting to `load_dataset` from a path starting with `~` (as a shorthand for the user's home directory) seems not to be fully working - at least as far as the `parquet` dataset builder is concerned.

(the same file can be loaded correctly if providing its absolute path instead)

I think that this is very similar to https://github.com/huggingface/datasets/issues/5757, but for `data_files` rather than `data_dir`

### Steps to reproduce the bug

```python
import datasets

# save a parquet file at ~/path/to/data.parquet

data_files = "~/path/to/data.parquet"
dataset = datasets.load_dataset("parquet", data_files=data_files)
```
```
Downloading data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 12671.61it/s]
Extracting data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 22671.91it/s]
Generating train split: 0 examples [00:00, ? examples/s]
Traceback (most recent call last):
  File ".venv/lib/python3.11/site-packages/datasets/builder.py", line 1949, in _prepare_split_single
    num_examples, num_bytes = writer.finalize()
                              ^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/datasets/arrow_writer.py", line 598, in finalize
    raise SchemaInferenceError("Please pass `features` or at least one example when writing data")
datasets.arrow_writer.SchemaInferenceError: Please pass `features` or at least one example when writing data

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".venv/lib/python3.11/site-packages/datasets/load.py", line 2133, in load_dataset
    builder_instance.download_and_prepare(
  File ".venv/lib/python3.11/site-packages/datasets/builder.py", line 954, in download_and_prepare
    self._download_and_prepare(
  File ".venv/lib/python3.11/site-packages/datasets/builder.py", line 1049, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File ".venv/lib/python3.11/site-packages/datasets/builder.py", line 1813, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File ".venv/lib/python3.11/site-packages/datasets/builder.py", line 1958, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset

```

### Expected behavior

Can use `~` shorthand in paths when loading local (parquet) datasets.

### Environment info

`datasets                               2.14.3`


Fix empty splitinfo json

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007756 / 0.011353 (-0.003597) | 0.004733 / 0.011008 (-0.006275) | 0.095874 / 0.038508 (0.057366) | 0.081957 / 0.023109 (0.058848) | 0.426430 / 0.275898 (0.150532) | 0.457670 / 0.323480 (0.134190) | 0.004448 / 0.007986 (-0.003537) | 0.004956 / 0.004328 (0.000627) | 0.074195 / 0.004250 (0.069945) | 0.061101 / 0.037052 (0.024048) | 0.435134 / 0.258489 (0.176645) | 0.457245 / 0.293841 (0.163404) | 0.034945 / 0.128546 (-0.093601) | 0.010028 / 0.075646 (-0.065618) | 0.350724 / 0.419271 (-0.068548) | 0.064433 / 0.043533 (0.020901) | 0.417882 / 0.255139 (0.162743) | 0.445087 / 0.283200 (0.161887) | 0.027576 / 0.141683 (-0.114107) | 1.824066 / 1.452155 (0.371912) | 1.957568 / 1.492716 (0.464852) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.238568 / 0.018006 (0.220562) | 0.505289 / 0.000490 (0.504799) | 0.003527 / 0.000200 (0.003327) | 0.000120 / 0.000054 (0.000065) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032839 / 0.037411 (-0.004572) | 0.096708 / 0.014526 (0.082182) | 0.112100 / 0.176557 (-0.064456) | 0.177215 / 0.737135 (-0.559920) | 0.111273 / 0.296338 (-0.185066) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.475200 / 0.215209 (0.259991) | 4.725737 / 2.077655 (2.648082) | 2.414672 / 1.504120 (0.910552) | 2.196357 / 1.541195 (0.655162) | 2.329298 / 1.468490 (0.860808) | 0.575258 / 4.584777 (-4.009519) | 4.343630 / 3.745712 (0.597918) | 3.837665 / 5.269862 (-1.432196) | 2.497970 / 4.565676 (-2.067706) | 0.066467 / 0.424275 (-0.357808) | 0.008680 / 0.007607 (0.001073) | 0.569923 / 0.226044 (0.343878) | 5.634230 / 2.268929 (3.365302) | 2.959222 / 55.444624 (-52.485402) | 2.535954 / 6.876477 (-4.340523) | 2.804844 / 2.142072 (0.662771) | 0.682000 / 4.805227 (-4.123227) | 0.158193 / 6.500664 (-6.342471) | 0.072315 / 0.075469 (-0.003154) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.578148 / 1.841788 (-0.263639) | 22.993419 / 8.074308 (14.919110) | 16.524477 / 10.191392 (6.333085) | 0.169415 / 0.680424 (-0.511009) | 0.021520 / 0.534201 (-0.512681) | 0.455970 / 0.579283 (-0.123313) | 0.489022 / 0.434364 (0.054658) | 0.535656 / 0.540337 (-0.004682) | 0.802341 / 1.386936 (-0.584595) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008002 / 0.011353 (-0.003351) | 0.005577 / 0.011008 (-0.005431) | 0.087803 / 0.038508 (0.049295) | 0.091285 / 0.023109 (0.068176) | 0.500514 / 0.275898 (0.224616) | 0.549770 / 0.323480 (0.226290) | 0.006125 / 0.007986 (-0.001861) | 0.004031 / 0.004328 (-0.000297) | 0.077941 / 0.004250 (0.073691) | 0.071419 / 0.037052 (0.034367) | 0.497570 / 0.258489 (0.239081) | 0.542454 / 0.293841 (0.248613) | 0.040827 / 0.128546 (-0.087719) | 0.011029 / 0.075646 (-0.064617) | 0.088788 / 0.419271 (-0.330484) | 0.056970 / 0.043533 (0.013438) | 0.523934 / 0.255139 (0.268795) | 0.552507 / 0.283200 (0.269308) | 0.029794 / 0.141683 (-0.111889) | 1.817778 / 1.452155 (0.365623) | 1.955843 / 1.492716 (0.463126) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.246992 / 0.018006 (0.228986) | 0.467879 / 0.000490 (0.467390) | 0.005439 / 0.000200 (0.005239) | 0.000110 / 0.000054 (0.000056) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.037774 / 0.037411 (0.000363) | 0.109332 / 0.014526 (0.094806) | 0.120103 / 0.176557 (-0.056454) | 0.185259 / 0.737135 (-0.551876) | 0.126189 / 0.296338 (-0.170149) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.492856 / 0.215209 (0.277646) | 5.033209 / 2.077655 (2.955554) | 2.885551 / 1.504120 (1.381431) | 2.480304 / 1.541195 (0.939109) | 2.579092 / 1.468490 (1.110602) | 0.557671 / 4.584777 (-4.027106) | 4.352765 / 3.745712 (0.607053) | 4.039124 / 5.269862 (-1.230738) | 2.534342 / 4.565676 (-2.031335) | 0.067267 / 0.424275 (-0.357008) | 0.008891 / 0.007607 (0.001284) | 0.591592 / 0.226044 (0.365547) | 5.939982 / 2.268929 (3.671053) | 3.258389 / 55.444624 (-52.186235) | 2.843899 / 6.876477 (-4.032578) | 3.074217 / 2.142072 (0.932144) | 0.695065 / 4.805227 (-4.110162) | 0.156917 / 6.500664 (-6.343747) | 0.070185 / 0.075469 (-0.005284) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.586716 / 1.841788 (-0.255072) | 23.405837 / 8.074308 (15.331529) | 17.200851 / 10.191392 (7.009459) | 0.170073 / 0.680424 (-0.510351) | 0.023345 / 0.534201 (-0.510856) | 0.459192 / 0.579283 (-0.120091) | 0.477419 / 0.434364 (0.043055) | 0.558581 / 0.540337 (0.018244) | 0.814373 / 1.386936 (-0.572563) |

![](https://cml.dev/watermark.png#28bbe5667e6eaa1bb21685791fcf1a4ed1ef1777 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006050 / 0.011353 (-0.005303) | 0.003661 / 0.011008 (-0.007348) | 0.081753 / 0.038508 (0.043245) | 0.061275 / 0.023109 (0.038166) | 0.316278 / 0.275898 (0.040380) | 0.350783 / 0.323480 (0.027303) | 0.004694 / 0.007986 (-0.003291) | 0.003003 / 0.004328 (-0.001326) | 0.062877 / 0.004250 (0.058627) | 0.046985 / 0.037052 (0.009933) | 0.315698 / 0.258489 (0.057208) | 0.364607 / 0.293841 (0.070766) | 0.027365 / 0.128546 (-0.101181) | 0.008016 / 0.075646 (-0.067631) | 0.261379 / 0.419271 (-0.157893) | 0.045173 / 0.043533 (0.001640) | 0.313499 / 0.255139 (0.058360) | 0.339383 / 0.283200 (0.056184) | 0.020855 / 0.141683 (-0.120828) | 1.429851 / 1.452155 (-0.022303) | 1.506112 / 1.492716 (0.013396) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.194872 / 0.018006 (0.176866) | 0.451951 / 0.000490 (0.451462) | 0.002790 / 0.000200 (0.002590) | 0.000070 / 0.000054 (0.000015) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024331 / 0.037411 (-0.013081) | 0.073156 / 0.014526 (0.058630) | 0.084054 / 0.176557 (-0.092502) | 0.145656 / 0.737135 (-0.591480) | 0.084998 / 0.296338 (-0.211340) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.391324 / 0.215209 (0.176115) | 3.898406 / 2.077655 (1.820751) | 1.891175 / 1.504120 (0.387055) | 1.698738 / 1.541195 (0.157543) | 1.774324 / 1.468490 (0.305834) | 0.495129 / 4.584777 (-4.089648) | 3.027027 / 3.745712 (-0.718685) | 2.821423 / 5.269862 (-2.448439) | 1.870761 / 4.565676 (-2.694915) | 0.057029 / 0.424275 (-0.367246) | 0.006715 / 0.007607 (-0.000892) | 0.465801 / 0.226044 (0.239757) | 4.650891 / 2.268929 (2.381962) | 2.425097 / 55.444624 (-53.019527) | 2.134731 / 6.876477 (-4.741745) | 2.312854 / 2.142072 (0.170781) | 0.589668 / 4.805227 (-4.215559) | 0.124673 / 6.500664 (-6.375991) | 0.060887 / 0.075469 (-0.014582) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.243622 / 1.841788 (-0.598166) | 18.501640 / 8.074308 (10.427332) | 13.853099 / 10.191392 (3.661707) | 0.130255 / 0.680424 (-0.550168) | 0.016824 / 0.534201 (-0.517377) | 0.332297 / 0.579283 (-0.246986) | 0.360346 / 0.434364 (-0.074018) | 0.388598 / 0.540337 (-0.151739) | 0.527551 / 1.386936 (-0.859385) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006181 / 0.011353 (-0.005172) | 0.003688 / 0.011008 (-0.007320) | 0.063395 / 0.038508 (0.024887) | 0.062531 / 0.023109 (0.039422) | 0.446565 / 0.275898 (0.170667) | 0.485224 / 0.323480 (0.161744) | 0.004982 / 0.007986 (-0.003004) | 0.002961 / 0.004328 (-0.001367) | 0.063124 / 0.004250 (0.058874) | 0.050234 / 0.037052 (0.013182) | 0.449731 / 0.258489 (0.191242) | 0.487293 / 0.293841 (0.193452) | 0.028528 / 0.128546 (-0.100018) | 0.008210 / 0.075646 (-0.067436) | 0.069520 / 0.419271 (-0.349751) | 0.041026 / 0.043533 (-0.002507) | 0.451370 / 0.255139 (0.196231) | 0.469151 / 0.283200 (0.185951) | 0.021076 / 0.141683 (-0.120607) | 1.439185 / 1.452155 (-0.012970) | 1.492634 / 1.492716 (-0.000082) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.235932 / 0.018006 (0.217926) | 0.430070 / 0.000490 (0.429581) | 0.007347 / 0.000200 (0.007147) | 0.000084 / 0.000054 (0.000029) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026102 / 0.037411 (-0.011309) | 0.081333 / 0.014526 (0.066807) | 0.090111 / 0.176557 (-0.086446) | 0.144578 / 0.737135 (-0.592557) | 0.091961 / 0.296338 (-0.204378) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.455761 / 0.215209 (0.240552) | 4.536345 / 2.077655 (2.458690) | 2.496833 / 1.504120 (0.992713) | 2.323325 / 1.541195 (0.782130) | 2.388364 / 1.468490 (0.919873) | 0.512010 / 4.584777 (-4.072767) | 3.106268 / 3.745712 (-0.639444) | 2.879224 / 5.269862 (-2.390637) | 1.893859 / 4.565676 (-2.671818) | 0.059131 / 0.424275 (-0.365144) | 0.006763 / 0.007607 (-0.000844) | 0.528205 / 0.226044 (0.302161) | 5.296649 / 2.268929 (3.027720) | 2.933787 / 55.444624 (-52.510838) | 2.598258 / 6.876477 (-4.278218) | 2.768195 / 2.142072 (0.626123) | 0.597430 / 4.805227 (-4.207797) | 0.125865 / 6.500664 (-6.374799) | 0.061684 / 0.075469 (-0.013785) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.341194 / 1.841788 (-0.500594) | 18.948225 / 8.074308 (10.873917) | 14.912340 / 10.191392 (4.720948) | 0.146905 / 0.680424 (-0.533519) | 0.017952 / 0.534201 (-0.516249) | 0.332299 / 0.579283 (-0.246984) | 0.362733 / 0.434364 (-0.071631) | 0.388278 / 0.540337 (-0.152060) | 0.546436 / 1.386936 (-0.840500) |

![](https://cml.dev/watermark.png#cb4f8357de001df656f2ea7af27625e189c3995b "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008314 / 0.011353 (-0.003038) | 0.004904 / 0.011008 (-0.006105) | 0.097486 / 0.038508 (0.058978) | 0.074627 / 0.023109 (0.051518) | 0.396395 / 0.275898 (0.120497) | 0.440519 / 0.323480 (0.117039) | 0.005964 / 0.007986 (-0.002022) | 0.004203 / 0.004328 (-0.000126) | 0.079998 / 0.004250 (0.075747) | 0.055158 / 0.037052 (0.018106) | 0.415439 / 0.258489 (0.156950) | 0.476101 / 0.293841 (0.182260) | 0.044761 / 0.128546 (-0.083785) | 0.013966 / 0.075646 (-0.061680) | 0.351279 / 0.419271 (-0.067993) | 0.067250 / 0.043533 (0.023717) | 0.414310 / 0.255139 (0.159171) | 0.458104 / 0.283200 (0.174904) | 0.033678 / 0.141683 (-0.108005) | 1.730539 / 1.452155 (0.278385) | 1.840013 / 1.492716 (0.347297) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.272708 / 0.018006 (0.254702) | 0.593563 / 0.000490 (0.593074) | 0.005153 / 0.000200 (0.004953) | 0.000179 / 0.000054 (0.000125) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029595 / 0.037411 (-0.007816) | 0.087994 / 0.014526 (0.073469) | 0.106066 / 0.176557 (-0.070491) | 0.180491 / 0.737135 (-0.556644) | 0.103707 / 0.296338 (-0.192631) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.566711 / 0.215209 (0.351502) | 5.589034 / 2.077655 (3.511380) | 2.364034 / 1.504120 (0.859914) | 2.119050 / 1.541195 (0.577855) | 2.103823 / 1.468490 (0.635333) | 0.819906 / 4.584777 (-3.764871) | 5.178464 / 3.745712 (1.432752) | 4.433986 / 5.269862 (-0.835875) | 2.825470 / 4.565676 (-1.740207) | 0.096907 / 0.424275 (-0.327368) | 0.008573 / 0.007607 (0.000966) | 0.677607 / 0.226044 (0.451563) | 6.811090 / 2.268929 (4.542162) | 3.140923 / 55.444624 (-52.303701) | 2.492251 / 6.876477 (-4.384225) | 2.660231 / 2.142072 (0.518158) | 0.980573 / 4.805227 (-3.824655) | 0.209028 / 6.500664 (-6.291636) | 0.079413 / 0.075469 (0.003944) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.578861 / 1.841788 (-0.262926) | 22.518269 / 8.074308 (14.443961) | 21.335916 / 10.191392 (11.144524) | 0.211311 / 0.680424 (-0.469113) | 0.033216 / 0.534201 (-0.500985) | 0.473266 / 0.579283 (-0.106017) | 0.581650 / 0.434364 (0.147286) | 0.522442 / 0.540337 (-0.017895) | 0.729039 / 1.386936 (-0.657897) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008349 / 0.011353 (-0.003003) | 0.005856 / 0.011008 (-0.005152) | 0.077855 / 0.038508 (0.039347) | 0.080608 / 0.023109 (0.057499) | 0.512533 / 0.275898 (0.236635) | 0.551862 / 0.323480 (0.228382) | 0.007004 / 0.007986 (-0.000982) | 0.004147 / 0.004328 (-0.000181) | 0.086625 / 0.004250 (0.082374) | 0.065962 / 0.037052 (0.028910) | 0.545590 / 0.258489 (0.287101) | 0.586313 / 0.293841 (0.292472) | 0.048719 / 0.128546 (-0.079827) | 0.014997 / 0.075646 (-0.060649) | 0.089510 / 0.419271 (-0.329761) | 0.060936 / 0.043533 (0.017404) | 0.498455 / 0.255139 (0.243316) | 0.535460 / 0.283200 (0.252260) | 0.034624 / 0.141683 (-0.107059) | 1.717401 / 1.452155 (0.265246) | 1.808772 / 1.492716 (0.316056) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.379504 / 0.018006 (0.361497) | 0.601756 / 0.000490 (0.601266) | 0.061740 / 0.000200 (0.061540) | 0.000497 / 0.000054 (0.000442) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031215 / 0.037411 (-0.006196) | 0.097501 / 0.014526 (0.082975) | 0.117434 / 0.176557 (-0.059122) | 0.166014 / 0.737135 (-0.571121) | 0.116466 / 0.296338 (-0.179873) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.699444 / 0.215209 (0.484235) | 6.329332 / 2.077655 (4.251678) | 3.072812 / 1.504120 (1.568693) | 2.729878 / 1.541195 (1.188683) | 2.933785 / 1.468490 (1.465295) | 0.935858 / 4.584777 (-3.648919) | 5.532532 / 3.745712 (1.786820) | 4.677139 / 5.269862 (-0.592722) | 2.963527 / 4.565676 (-1.602149) | 0.099661 / 0.424275 (-0.324614) | 0.009095 / 0.007607 (0.001488) | 0.751158 / 0.226044 (0.525114) | 7.652588 / 2.268929 (5.383660) | 3.802005 / 55.444624 (-51.642619) | 3.163126 / 6.876477 (-3.713351) | 3.401125 / 2.142072 (1.259052) | 0.998627 / 4.805227 (-3.806600) | 0.203310 / 6.500664 (-6.297354) | 0.073827 / 0.075469 (-0.001642) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.662989 / 1.841788 (-0.178799) | 23.777818 / 8.074308 (15.703510) | 20.855378 / 10.191392 (10.663986) | 0.279892 / 0.680424 (-0.400532) | 0.029303 / 0.534201 (-0.504898) | 0.473681 / 0.579283 (-0.105602) | 0.579148 / 0.434364 (0.144784) | 0.546931 / 0.540337 (0.006593) | 0.769740 / 1.386936 (-0.617196) |

![](https://cml.dev/watermark.png#63114e9cb78fe02dc145f923dec13d545a8d0327 "CML watermark")

If a split is empty, then the JSON split info should mention num_bytes = 0 and num_examples = 0.

Until now they were omited because the JSON dumps ignore the fields that are equal to the default values.

This is needed in datasets-server since we parse this information to the viewer