Don't skip hidden files in `dl_manager.iter_files` when they are given as input

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005894 / 0.011353 (-0.005459) | 0.003621 / 0.011008 (-0.007387) | 0.080446 / 0.038508 (0.041938) | 0.056800 / 0.023109 (0.033691) | 0.326485 / 0.275898 (0.050587) | 0.376207 / 0.323480 (0.052727) | 0.004640 / 0.007986 (-0.003346) | 0.002795 / 0.004328 (-0.001533) | 0.062815 / 0.004250 (0.058565) | 0.045761 / 0.037052 (0.008709) | 0.341417 / 0.258489 (0.082928) | 0.373129 / 0.293841 (0.079288) | 0.027226 / 0.128546 (-0.101321) | 0.007873 / 0.075646 (-0.067774) | 0.261737 / 0.419271 (-0.157535) | 0.044648 / 0.043533 (0.001115) | 0.320195 / 0.255139 (0.065056) | 0.381892 / 0.283200 (0.098692) | 0.020431 / 0.141683 (-0.121252) | 1.405332 / 1.452155 (-0.046823) | 1.455592 / 1.492716 (-0.037125) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.191539 / 0.018006 (0.173533) | 0.423655 / 0.000490 (0.423165) | 0.002741 / 0.000200 (0.002541) | 0.000069 / 0.000054 (0.000014) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023952 / 0.037411 (-0.013459) | 0.073387 / 0.014526 (0.058861) | 0.083746 / 0.176557 (-0.092810) | 0.144977 / 0.737135 (-0.592159) | 0.083808 / 0.296338 (-0.212530) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.436228 / 0.215209 (0.221019) | 4.370510 / 2.077655 (2.292855) | 2.340426 / 1.504120 (0.836306) | 2.202215 / 1.541195 (0.661021) | 2.258528 / 1.468490 (0.790037) | 0.503455 / 4.584777 (-4.081322) | 3.043695 / 3.745712 (-0.702017) | 2.784033 / 5.269862 (-2.485829) | 1.847956 / 4.565676 (-2.717721) | 0.057702 / 0.424275 (-0.366573) | 0.006703 / 0.007607 (-0.000904) | 0.510628 / 0.226044 (0.284583) | 5.101890 / 2.268929 (2.832961) | 2.816469 / 55.444624 (-52.628155) | 2.474220 / 6.876477 (-4.402257) | 2.617851 / 2.142072 (0.475779) | 0.593585 / 4.805227 (-4.211642) | 0.125895 / 6.500664 (-6.374769) | 0.062170 / 0.075469 (-0.013299) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.238792 / 1.841788 (-0.602996) | 18.096417 / 8.074308 (10.022108) | 13.548778 / 10.191392 (3.357386) | 0.144878 / 0.680424 (-0.535546) | 0.016644 / 0.534201 (-0.517557) | 0.334556 / 0.579283 (-0.244728) | 0.343680 / 0.434364 (-0.090684) | 0.383093 / 0.540337 (-0.157244) | 0.525075 / 1.386936 (-0.861861) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006125 / 0.011353 (-0.005228) | 0.003668 / 0.011008 (-0.007340) | 0.062650 / 0.038508 (0.024142) | 0.058882 / 0.023109 (0.035772) | 0.454643 / 0.275898 (0.178745) | 0.486659 / 0.323480 (0.163179) | 0.005558 / 0.007986 (-0.002427) | 0.002858 / 0.004328 (-0.001471) | 0.062603 / 0.004250 (0.058353) | 0.049701 / 0.037052 (0.012649) | 0.455903 / 0.258489 (0.197413) | 0.491544 / 0.293841 (0.197703) | 0.028581 / 0.128546 (-0.099965) | 0.008040 / 0.075646 (-0.067607) | 0.068314 / 0.419271 (-0.350957) | 0.040637 / 0.043533 (-0.002896) | 0.450288 / 0.255139 (0.195149) | 0.476330 / 0.283200 (0.193131) | 0.018989 / 0.141683 (-0.122693) | 1.455122 / 1.452155 (0.002967) | 1.496941 / 1.492716 (0.004225) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.227382 / 0.018006 (0.209376) | 0.432637 / 0.000490 (0.432147) | 0.002727 / 0.000200 (0.002527) | 0.000073 / 0.000054 (0.000019) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026125 / 0.037411 (-0.011286) | 0.081342 / 0.014526 (0.066817) | 0.091227 / 0.176557 (-0.085329) | 0.145175 / 0.737135 (-0.591960) | 0.091988 / 0.296338 (-0.204351) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.454293 / 0.215209 (0.239083) | 4.537912 / 2.077655 (2.460257) | 2.489146 / 1.504120 (0.985026) | 2.307166 / 1.541195 (0.765971) | 2.380866 / 1.468490 (0.912376) | 0.509015 / 4.584777 (-4.075762) | 3.111069 / 3.745712 (-0.634644) | 2.839181 / 5.269862 (-2.430681) | 1.874630 / 4.565676 (-2.691047) | 0.058540 / 0.424275 (-0.365735) | 0.006693 / 0.007607 (-0.000914) | 0.528408 / 0.226044 (0.302363) | 5.285802 / 2.268929 (3.016874) | 2.952090 / 55.444624 (-52.492534) | 2.591496 / 6.876477 (-4.284980) | 2.741080 / 2.142072 (0.599007) | 0.595610 / 4.805227 (-4.209617) | 0.124387 / 6.500664 (-6.376277) | 0.061032 / 0.075469 (-0.014437) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.365816 / 1.841788 (-0.475972) | 18.684534 / 8.074308 (10.610226) | 14.540438 / 10.191392 (4.349046) | 0.146793 / 0.680424 (-0.533631) | 0.018165 / 0.534201 (-0.516036) | 0.333794 / 0.579283 (-0.245489) | 0.345533 / 0.434364 (-0.088830) | 0.384453 / 0.540337 (-0.155885) | 0.529104 / 1.386936 (-0.857832) |

![](https://cml.dev/watermark.png#6c884967dd5f4e8aa3d1f3c2e3a414ae53afe261 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006121 / 0.011353 (-0.005232) | 0.003683 / 0.011008 (-0.007325) | 0.083329 / 0.038508 (0.044821) | 0.063350 / 0.023109 (0.040241) | 0.329959 / 0.275898 (0.054061) | 0.396111 / 0.323480 (0.072631) | 0.003554 / 0.007986 (-0.004432) | 0.002907 / 0.004328 (-0.001421) | 0.064152 / 0.004250 (0.059902) | 0.049182 / 0.037052 (0.012130) | 0.343862 / 0.258489 (0.085373) | 0.414568 / 0.293841 (0.120727) | 0.027157 / 0.128546 (-0.101389) | 0.007957 / 0.075646 (-0.067689) | 0.261868 / 0.419271 (-0.157404) | 0.044938 / 0.043533 (0.001405) | 0.318470 / 0.255139 (0.063331) | 0.393319 / 0.283200 (0.110119) | 0.022848 / 0.141683 (-0.118835) | 1.419916 / 1.452155 (-0.032238) | 1.508783 / 1.492716 (0.016067) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.200530 / 0.018006 (0.182523) | 0.433586 / 0.000490 (0.433097) | 0.002063 / 0.000200 (0.001863) | 0.000070 / 0.000054 (0.000016) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024803 / 0.037411 (-0.012609) | 0.075894 / 0.014526 (0.061368) | 0.086488 / 0.176557 (-0.090069) | 0.149058 / 0.737135 (-0.588077) | 0.087046 / 0.296338 (-0.209292) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.390771 / 0.215209 (0.175562) | 3.886178 / 2.077655 (1.808523) | 1.868626 / 1.504120 (0.364506) | 1.708532 / 1.541195 (0.167338) | 1.788491 / 1.468490 (0.320001) | 0.505706 / 4.584777 (-4.079071) | 3.062094 / 3.745712 (-0.683618) | 2.898559 / 5.269862 (-2.371302) | 1.901225 / 4.565676 (-2.664452) | 0.058366 / 0.424275 (-0.365909) | 0.006851 / 0.007607 (-0.000756) | 0.465382 / 0.226044 (0.239337) | 4.650187 / 2.268929 (2.381258) | 2.316152 / 55.444624 (-53.128472) | 1.989597 / 6.876477 (-4.886879) | 2.169266 / 2.142072 (0.027194) | 0.593257 / 4.805227 (-4.211970) | 0.126440 / 6.500664 (-6.374224) | 0.062227 / 0.075469 (-0.013242) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.283591 / 1.841788 (-0.558197) | 18.384667 / 8.074308 (10.310358) | 14.079611 / 10.191392 (3.888219) | 0.150453 / 0.680424 (-0.529971) | 0.017100 / 0.534201 (-0.517101) | 0.330503 / 0.579283 (-0.248780) | 0.348134 / 0.434364 (-0.086230) | 0.385726 / 0.540337 (-0.154612) | 0.529147 / 1.386936 (-0.857789) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006168 / 0.011353 (-0.005185) | 0.003801 / 0.011008 (-0.007208) | 0.063168 / 0.038508 (0.024660) | 0.062331 / 0.023109 (0.039221) | 0.448321 / 0.275898 (0.172423) | 0.484416 / 0.323480 (0.160937) | 0.004827 / 0.007986 (-0.003159) | 0.002848 / 0.004328 (-0.001480) | 0.062736 / 0.004250 (0.058486) | 0.049128 / 0.037052 (0.012075) | 0.449276 / 0.258489 (0.190787) | 0.499035 / 0.293841 (0.205194) | 0.028577 / 0.128546 (-0.099969) | 0.008114 / 0.075646 (-0.067532) | 0.068297 / 0.419271 (-0.350974) | 0.040835 / 0.043533 (-0.002698) | 0.453556 / 0.255139 (0.198417) | 0.475420 / 0.283200 (0.192220) | 0.020292 / 0.141683 (-0.121390) | 1.472226 / 1.452155 (0.020071) | 1.523809 / 1.492716 (0.031093) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.230662 / 0.018006 (0.212655) | 0.439697 / 0.000490 (0.439207) | 0.009899 / 0.000200 (0.009699) | 0.000087 / 0.000054 (0.000033) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026418 / 0.037411 (-0.010993) | 0.082188 / 0.014526 (0.067662) | 0.091039 / 0.176557 (-0.085518) | 0.146646 / 0.737135 (-0.590489) | 0.091693 / 0.296338 (-0.204645) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.462086 / 0.215209 (0.246877) | 4.620925 / 2.077655 (2.543271) | 2.539234 / 1.504120 (1.035114) | 2.371178 / 1.541195 (0.829983) | 2.440538 / 1.468490 (0.972048) | 0.511047 / 4.584777 (-4.073730) | 3.082088 / 3.745712 (-0.663624) | 2.918162 / 5.269862 (-2.351700) | 1.899651 / 4.565676 (-2.666025) | 0.059003 / 0.424275 (-0.365272) | 0.006746 / 0.007607 (-0.000861) | 0.537863 / 0.226044 (0.311819) | 5.382355 / 2.268929 (3.113426) | 3.060091 / 55.444624 (-52.384534) | 2.754969 / 6.876477 (-4.121507) | 2.863156 / 2.142072 (0.721084) | 0.606888 / 4.805227 (-4.198339) | 0.127448 / 6.500664 (-6.373216) | 0.062975 / 0.075469 (-0.012494) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.336065 / 1.841788 (-0.505722) | 19.019902 / 8.074308 (10.945594) | 15.057979 / 10.191392 (4.866587) | 0.160646 / 0.680424 (-0.519778) | 0.018340 / 0.534201 (-0.515861) | 0.341664 / 0.579283 (-0.237619) | 0.356536 / 0.434364 (-0.077828) | 0.393974 / 0.540337 (-0.146363) | 0.546036 / 1.386936 (-0.840900) |

![](https://cml.dev/watermark.png#fd04e445bd36d7eb4af4d5a6b8519ab8e306ecf5 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007220 / 0.011353 (-0.004132) | 0.004537 / 0.011008 (-0.006471) | 0.087333 / 0.038508 (0.048825) | 0.095637 / 0.023109 (0.072528) | 0.323819 / 0.275898 (0.047921) | 0.358838 / 0.323480 (0.035358) | 0.005910 / 0.007986 (-0.002076) | 0.003781 / 0.004328 (-0.000548) | 0.064565 / 0.004250 (0.060315) | 0.062818 / 0.037052 (0.025766) | 0.322595 / 0.258489 (0.064106) | 0.371865 / 0.293841 (0.078024) | 0.031667 / 0.128546 (-0.096880) | 0.009068 / 0.075646 (-0.066579) | 0.290574 / 0.419271 (-0.128697) | 0.054618 / 0.043533 (0.011085) | 0.314708 / 0.255139 (0.059569) | 0.336647 / 0.283200 (0.053447) | 0.027070 / 0.141683 (-0.114613) | 1.500640 / 1.452155 (0.048485) | 1.586775 / 1.492716 (0.094059) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.294461 / 0.018006 (0.276455) | 0.580125 / 0.000490 (0.579635) | 0.008165 / 0.000200 (0.007965) | 0.000320 / 0.000054 (0.000266) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032352 / 0.037411 (-0.005059) | 0.092187 / 0.014526 (0.077661) | 0.104993 / 0.176557 (-0.071564) | 0.162738 / 0.737135 (-0.574397) | 0.103242 / 0.296338 (-0.193096) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.396732 / 0.215209 (0.181523) | 3.955049 / 2.077655 (1.877394) | 1.876762 / 1.504120 (0.372642) | 1.698477 / 1.541195 (0.157282) | 1.847086 / 1.468490 (0.378596) | 0.488306 / 4.584777 (-4.096471) | 3.658922 / 3.745712 (-0.086790) | 3.559050 / 5.269862 (-1.710812) | 2.187363 / 4.565676 (-2.378313) | 0.059795 / 0.424275 (-0.364480) | 0.008966 / 0.007607 (0.001359) | 0.474212 / 0.226044 (0.248168) | 4.732540 / 2.268929 (2.463611) | 2.466370 / 55.444624 (-52.978254) | 2.112105 / 6.876477 (-4.764372) | 2.414624 / 2.142072 (0.272552) | 0.595447 / 4.805227 (-4.209780) | 0.136705 / 6.500664 (-6.363959) | 0.062267 / 0.075469 (-0.013202) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.266518 / 1.841788 (-0.575270) | 21.009975 / 8.074308 (12.935666) | 14.823960 / 10.191392 (4.632568) | 0.165630 / 0.680424 (-0.514793) | 0.018499 / 0.534201 (-0.515702) | 0.396720 / 0.579283 (-0.182563) | 0.424807 / 0.434364 (-0.009557) | 0.463326 / 0.540337 (-0.077011) | 0.653132 / 1.386936 (-0.733804) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007789 / 0.011353 (-0.003564) | 0.004720 / 0.011008 (-0.006288) | 0.066656 / 0.038508 (0.028148) | 0.094219 / 0.023109 (0.071109) | 0.414965 / 0.275898 (0.139067) | 0.454808 / 0.323480 (0.131328) | 0.006088 / 0.007986 (-0.001898) | 0.003980 / 0.004328 (-0.000349) | 0.066048 / 0.004250 (0.061797) | 0.065875 / 0.037052 (0.028823) | 0.419994 / 0.258489 (0.161505) | 0.462001 / 0.293841 (0.168160) | 0.033534 / 0.128546 (-0.095013) | 0.009010 / 0.075646 (-0.066636) | 0.072778 / 0.419271 (-0.346493) | 0.049834 / 0.043533 (0.006301) | 0.411003 / 0.255139 (0.155864) | 0.430918 / 0.283200 (0.147718) | 0.025664 / 0.141683 (-0.116019) | 1.526771 / 1.452155 (0.074616) | 1.634767 / 1.492716 (0.142051) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.271180 / 0.018006 (0.253174) | 0.576704 / 0.000490 (0.576214) | 0.004362 / 0.000200 (0.004162) | 0.000112 / 0.000054 (0.000058) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035648 / 0.037411 (-0.001763) | 0.102407 / 0.014526 (0.087881) | 0.111613 / 0.176557 (-0.064944) | 0.166173 / 0.737135 (-0.570962) | 0.113371 / 0.296338 (-0.182967) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.436031 / 0.215209 (0.220822) | 4.347071 / 2.077655 (2.269416) | 2.366937 / 1.504120 (0.862817) | 2.216356 / 1.541195 (0.675161) | 2.335933 / 1.468490 (0.867443) | 0.490484 / 4.584777 (-4.094293) | 3.730656 / 3.745712 (-0.015056) | 3.497248 / 5.269862 (-1.772613) | 2.215729 / 4.565676 (-2.349947) | 0.057905 / 0.424275 (-0.366370) | 0.007983 / 0.007607 (0.000376) | 0.510413 / 0.226044 (0.284369) | 5.114502 / 2.268929 (2.845574) | 2.871599 / 55.444624 (-52.573026) | 2.537514 / 6.876477 (-4.338962) | 2.819135 / 2.142072 (0.677063) | 0.588397 / 4.805227 (-4.216830) | 0.134665 / 6.500664 (-6.365999) | 0.063349 / 0.075469 (-0.012120) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.352962 / 1.841788 (-0.488826) | 21.628664 / 8.074308 (13.554356) | 15.962105 / 10.191392 (5.770713) | 0.167781 / 0.680424 (-0.512643) | 0.020965 / 0.534201 (-0.513236) | 0.402809 / 0.579283 (-0.176474) | 0.435153 / 0.434364 (0.000789) | 0.481394 / 0.540337 (-0.058944) | 0.658068 / 1.386936 (-0.728868) |

![](https://cml.dev/watermark.png#12adf38b90fde8e2a4e46fcbb023ee23b5c4e98c "CML watermark")

Required for `load_dataset(<format>, data_files=["path/to/.hidden_file"])` to work as expected


Apply inference on all images in the dataset

From what I see, `MMSegInferencer` supports NumPy arrays, so replace the line `image_path = example['image']` with `image_path = np.array(example['image'])` to fix the issue (`example["image"]` is a `PIL.Image` object).

https://github.com/huggingface/datasets/issues/6246 resolved the `add_column` error, so I'm closing this issue :)

### Describe the bug

```
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
Cell In[14], line 11
      9 for idx, example in enumerate(dataset['train']):
     10     image_path = example['image']
---> 11     mask_image = process_image(image_path)
     12     mask_image.save(f"mask_{idx}.png")

Cell In[14], line 4, in process_image(image_path)
      2 def process_image(image_path):
      3     print("Processing image:", image_path)
----> 4     result = inferencer(image_path)['predictions']
      5     mask = np.where(result == 12, 255, 0).astype('uint8')
      6     return Image.fromarray(mask)

File /usr/local/lib/python3.10/dist-packages/mmseg/apis/mmseg_inferencer.py:183, in MMSegInferencer.__call__(self, inputs, return_datasamples, batch_size, show, wait_time, out_dir, img_out_dir, pred_out_dir, **kwargs)
    180     pred_out_dir = ''
    181     img_out_dir = ''
--> 183 return super().__call__(
    184     inputs=inputs,
    185     return_datasamples=return_datasamples,
    186     batch_size=batch_size,
    187     show=show,
    188     wait_time=wait_time,
    189     img_out_dir=img_out_dir,
    190     pred_out_dir=pred_out_dir,
    191     **kwargs)

File /usr/local/lib/python3.10/dist-packages/mmengine/infer/infer.py:221, in BaseInferencer.__call__(self, inputs, return_datasamples, batch_size, **kwargs)
    218 inputs = self.preprocess(
    219     ori_inputs, batch_size=batch_size, **preprocess_kwargs)
    220 preds = []
--> 221 for data in (track(inputs, description='Inference')
    222              if self.show_progress else inputs):
    223     preds.extend(self.forward(data, **forward_kwargs))
    224 visualization = self.visualize(
    225     ori_inputs, preds,
    226     **visualize_kwargs)  # type: ignore  # noqa: E501

File /usr/local/lib/python3.10/dist-packages/rich/progress.py:168, in track(sequence, description, total, auto_refresh, console, transient, get_time, refresh_per_second, style, complete_style, finished_style, pulse_style, update_period, disable, show_speed)
    157 progress = Progress(
    158     *columns,
    159     auto_refresh=auto_refresh,
   (...)
    164     disable=disable,
    165 )
    167 with progress:
--> 168     yield from progress.track(
    169         sequence, total=total, description=description, update_period=update_period
    170     )

File /usr/local/lib/python3.10/dist-packages/rich/progress.py:1210, in Progress.track(self, sequence, total, task_id, description, update_period)
   1208 if self.live.auto_refresh:
   1209     with _TrackThread(self, task_id, update_period) as track_thread:
-> 1210         for value in sequence:
   1211             yield value
   1212             track_thread.completed += 1

File /usr/local/lib/python3.10/dist-packages/mmengine/infer/infer.py:291, in BaseInferencer.preprocess(self, inputs, batch_size, **kwargs)
    266 """Process the inputs into a model-feedable format.
    267
    268 Customize your preprocess by overriding this method. Preprocess should
   (...)
    287     Any: Data processed by the ``pipeline`` and ``collate_fn``.
    288 """
    289 chunked_data = self._get_chunk_data(
    290     map(self.pipeline, inputs), batch_size)
--> 291 yield from map(self.collate_fn, chunked_data)

File /usr/local/lib/python3.10/dist-packages/mmengine/infer/infer.py:588, in BaseInferencer._get_chunk_data(self, inputs, chunk_size)
    586 chunk_data = []
    587 for _ in range(chunk_size):
--> 588     processed_data = next(inputs_iter)
    589     chunk_data.append(processed_data)
    590 yield chunk_data

File /usr/local/lib/python3.10/dist-packages/mmcv/transforms/base.py:12, in BaseTransform.__call__(self, results)
      9 def __call__(self,
     10              results: Dict) -> Optional[Union[Dict, Tuple[List, List]]]:
---> 12     return self.transform(results)

File /usr/local/lib/python3.10/dist-packages/mmcv/transforms/wrappers.py:88, in Compose.transform(self, results)
     79 """Call function to apply transforms sequentially.
     80
     81 Args:
   (...)
     85     dict or None: Transformed results.
     86 """
     87 for t in self.transforms:
---> 88     results = t(results)  # type: ignore
     89     if results is None:
     90         return None

File /usr/local/lib/python3.10/dist-packages/mmcv/transforms/base.py:12, in BaseTransform.__call__(self, results)
      9 def __call__(self,
     10              results: Dict) -> Optional[Union[Dict, Tuple[List, List]]]:
---> 12     return self.transform(results)

File /usr/local/lib/python3.10/dist-packages/mmseg/datasets/transforms/loading.py:496, in InferencerLoader.transform(self, single_input)
    494     inputs = single_input
    495 else:
--> 496     raise NotImplementedError
    498 if 'img' in inputs:
    499     return self.from_ndarray(inputs)

NotImplementedError:
````

### Steps to reproduce the bug

```
from datasets import load_dataset
dataset = load_dataset('Andyrasika/cat_kingdom')
dataset

from mmseg.apis import MMSegInferencer

checkpoint_name = 'segformer_mit-b5_8xb2-160k_ade20k-640x640'

inferencer = MMSegInferencer(model=checkpoint_name)

# Define a function to apply the code to each image in the dataset
def process_image(image_path):
    print("Processing image:", image_path)
    result = inferencer(image_path)['predictions']
    mask = np.where(result == 12, 255, 0).astype('uint8')
    return Image.fromarray(mask)

# Process and save masks for each image in the dataset
for idx, example in enumerate(dataset['train']):
    image_path = example['image']
    mask_image = process_image(image_path)
    mask_image.save(f"mask_{idx}.png")
```

### Expected behavior

create a separate column with masks in the dataset and further shows as a separate column in hub

### Environment info

jupyter notebook RTX 3090


Remove RGB -> BGR image conversion in Object Detection tutorial

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009443 / 0.011353 (-0.001910) | 0.005274 / 0.011008 (-0.005734) | 0.105950 / 0.038508 (0.067441) | 0.079947 / 0.023109 (0.056837) | 0.414248 / 0.275898 (0.138350) | 0.440611 / 0.323480 (0.117131) | 0.006779 / 0.007986 (-0.001206) | 0.004301 / 0.004328 (-0.000028) | 0.080616 / 0.004250 (0.076366) | 0.061425 / 0.037052 (0.024372) | 0.418460 / 0.258489 (0.159971) | 0.468108 / 0.293841 (0.174267) | 0.051090 / 0.128546 (-0.077456) | 0.014133 / 0.075646 (-0.061513) | 0.376121 / 0.419271 (-0.043151) | 0.070715 / 0.043533 (0.027182) | 0.415435 / 0.255139 (0.160296) | 0.457925 / 0.283200 (0.174725) | 0.053653 / 0.141683 (-0.088030) | 1.872681 / 1.452155 (0.420527) | 1.961187 / 1.492716 (0.468470) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.255829 / 0.018006 (0.237823) | 0.574224 / 0.000490 (0.573735) | 0.007597 / 0.000200 (0.007397) | 0.000098 / 0.000054 (0.000044) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032562 / 0.037411 (-0.004849) | 0.097528 / 0.014526 (0.083003) | 0.113487 / 0.176557 (-0.063070) | 0.185670 / 0.737135 (-0.551465) | 0.118909 / 0.296338 (-0.177430) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.611441 / 0.215209 (0.396232) | 5.908576 / 2.077655 (3.830921) | 2.586758 / 1.504120 (1.082638) | 2.310199 / 1.541195 (0.769004) | 2.333396 / 1.468490 (0.864906) | 0.900884 / 4.584777 (-3.683893) | 5.438304 / 3.745712 (1.692591) | 4.806611 / 5.269862 (-0.463250) | 2.970631 / 4.565676 (-1.595046) | 0.097861 / 0.424275 (-0.326414) | 0.009873 / 0.007607 (0.002266) | 0.739553 / 0.226044 (0.513509) | 7.104953 / 2.268929 (4.836024) | 3.150128 / 55.444624 (-52.294497) | 2.469552 / 6.876477 (-4.406924) | 2.709206 / 2.142072 (0.567133) | 0.983081 / 4.805227 (-3.822147) | 0.205150 / 6.500664 (-6.295514) | 0.075947 / 0.075469 (0.000478) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.631255 / 1.841788 (-0.210532) | 24.213679 / 8.074308 (16.139370) | 21.514481 / 10.191392 (11.323089) | 0.220360 / 0.680424 (-0.460063) | 0.031663 / 0.534201 (-0.502538) | 0.516029 / 0.579283 (-0.063254) | 0.591461 / 0.434364 (0.157097) | 0.612398 / 0.540337 (0.072061) | 0.807609 / 1.386936 (-0.579328) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009443 / 0.011353 (-0.001910) | 0.005510 / 0.011008 (-0.005498) | 0.085722 / 0.038508 (0.047214) | 0.076256 / 0.023109 (0.053146) | 0.604248 / 0.275898 (0.328349) | 0.596222 / 0.323480 (0.272742) | 0.006786 / 0.007986 (-0.001200) | 0.004135 / 0.004328 (-0.000193) | 0.085934 / 0.004250 (0.081683) | 0.065890 / 0.037052 (0.028838) | 0.592080 / 0.258489 (0.333591) | 0.624560 / 0.293841 (0.330719) | 0.048200 / 0.128546 (-0.080346) | 0.015477 / 0.075646 (-0.060169) | 0.097042 / 0.419271 (-0.322230) | 0.060513 / 0.043533 (0.016981) | 0.557171 / 0.255139 (0.302032) | 0.582057 / 0.283200 (0.298858) | 0.035678 / 0.141683 (-0.106005) | 1.894947 / 1.452155 (0.442792) | 1.956652 / 1.492716 (0.463936) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.268927 / 0.018006 (0.250921) | 0.566086 / 0.000490 (0.565597) | 0.007190 / 0.000200 (0.006990) | 0.000101 / 0.000054 (0.000047) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.042090 / 0.037411 (0.004679) | 0.109618 / 0.014526 (0.095092) | 0.126588 / 0.176557 (-0.049968) | 0.200426 / 0.737135 (-0.536709) | 0.127032 / 0.296338 (-0.169306) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.669773 / 0.215209 (0.454564) | 6.453417 / 2.077655 (4.375763) | 3.119147 / 1.504120 (1.615027) | 2.818632 / 1.541195 (1.277437) | 2.930880 / 1.468490 (1.462390) | 0.922164 / 4.584777 (-3.662612) | 5.769564 / 3.745712 (2.023852) | 4.885108 / 5.269862 (-0.384754) | 3.041640 / 4.565676 (-1.524037) | 0.100186 / 0.424275 (-0.324090) | 0.009417 / 0.007607 (0.001810) | 0.783138 / 0.226044 (0.557094) | 8.113361 / 2.268929 (5.844432) | 4.018630 / 55.444624 (-51.425995) | 3.246772 / 6.876477 (-3.629704) | 3.520690 / 2.142072 (1.378618) | 1.063686 / 4.805227 (-3.741541) | 0.218667 / 6.500664 (-6.281997) | 0.084169 / 0.075469 (0.008700) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.791949 / 1.841788 (-0.049839) | 23.148341 / 8.074308 (15.074033) | 23.321125 / 10.191392 (13.129733) | 0.245391 / 0.680424 (-0.435032) | 0.031911 / 0.534201 (-0.502290) | 0.470707 / 0.579283 (-0.108576) | 0.608195 / 0.434364 (0.173832) | 0.559590 / 0.540337 (0.019253) | 0.786007 / 1.386936 (-0.600929) |

![](https://cml.dev/watermark.png#8e071f565cc0801f73f7f34fba92dc30a43946a9 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008428 / 0.011353 (-0.002925) | 0.004064 / 0.011008 (-0.006944) | 0.088421 / 0.038508 (0.049913) | 0.078042 / 0.023109 (0.054933) | 0.306356 / 0.275898 (0.030458) | 0.349766 / 0.323480 (0.026286) | 0.004086 / 0.007986 (-0.003900) | 0.003900 / 0.004328 (-0.000428) | 0.068379 / 0.004250 (0.064129) | 0.056214 / 0.037052 (0.019161) | 0.310211 / 0.258489 (0.051722) | 0.363692 / 0.293841 (0.069851) | 0.050421 / 0.128546 (-0.078125) | 0.011661 / 0.075646 (-0.063985) | 0.298400 / 0.419271 (-0.120871) | 0.063503 / 0.043533 (0.019970) | 0.339799 / 0.255139 (0.084660) | 0.359479 / 0.283200 (0.076279) | 0.039265 / 0.141683 (-0.102418) | 1.390578 / 1.452155 (-0.061576) | 1.573333 / 1.492716 (0.080617) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.260442 / 0.018006 (0.242436) | 0.560390 / 0.000490 (0.559900) | 0.003926 / 0.000200 (0.003726) | 0.000083 / 0.000054 (0.000029) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025809 / 0.037411 (-0.011602) | 0.081902 / 0.014526 (0.067376) | 0.093655 / 0.176557 (-0.082901) | 0.149432 / 0.737135 (-0.587703) | 0.099059 / 0.296338 (-0.197279) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.505644 / 0.215209 (0.290435) | 5.108292 / 2.077655 (3.030638) | 2.121689 / 1.504120 (0.617569) | 1.846576 / 1.541195 (0.305381) | 1.836587 / 1.468490 (0.368097) | 0.708088 / 4.584777 (-3.876689) | 4.562630 / 3.745712 (0.816918) | 3.934747 / 5.269862 (-1.335115) | 2.453409 / 4.565676 (-2.112267) | 0.081908 / 0.424275 (-0.342367) | 0.012996 / 0.007607 (0.005389) | 0.636588 / 0.226044 (0.410544) | 6.361086 / 2.268929 (4.092157) | 2.911681 / 55.444624 (-52.532943) | 2.271809 / 6.876477 (-4.604667) | 2.670327 / 2.142072 (0.528254) | 0.943688 / 4.805227 (-3.861539) | 0.191677 / 6.500664 (-6.308988) | 0.066008 / 0.075469 (-0.009461) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.400139 / 1.841788 (-0.441648) | 21.896198 / 8.074308 (13.821890) | 17.853604 / 10.191392 (7.662212) | 0.226603 / 0.680424 (-0.453821) | 0.026682 / 0.534201 (-0.507518) | 0.460131 / 0.579283 (-0.119152) | 0.536790 / 0.434364 (0.102427) | 0.492913 / 0.540337 (-0.047424) | 0.724290 / 1.386936 (-0.662646) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007795 / 0.011353 (-0.003557) | 0.009045 / 0.011008 (-0.001963) | 0.085480 / 0.038508 (0.046972) | 0.071881 / 0.023109 (0.048772) | 0.514520 / 0.275898 (0.238622) | 0.569762 / 0.323480 (0.246282) | 0.006126 / 0.007986 (-0.001859) | 0.004153 / 0.004328 (-0.000175) | 0.072150 / 0.004250 (0.067900) | 0.056511 / 0.037052 (0.019458) | 0.484097 / 0.258489 (0.225607) | 0.532673 / 0.293841 (0.238832) | 0.040974 / 0.128546 (-0.087572) | 0.012071 / 0.075646 (-0.063575) | 0.102608 / 0.419271 (-0.316663) | 0.052893 / 0.043533 (0.009360) | 0.485832 / 0.255139 (0.230693) | 0.530479 / 0.283200 (0.247280) | 0.031556 / 0.141683 (-0.110127) | 1.737508 / 1.452155 (0.285354) | 1.834637 / 1.492716 (0.341921) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.423314 / 0.018006 (0.405308) | 0.614163 / 0.000490 (0.613673) | 0.052784 / 0.000200 (0.052584) | 0.000206 / 0.000054 (0.000151) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031728 / 0.037411 (-0.005684) | 0.088048 / 0.014526 (0.073522) | 0.105759 / 0.176557 (-0.070798) | 0.181433 / 0.737135 (-0.555703) | 0.103133 / 0.296338 (-0.193205) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.659710 / 0.215209 (0.444501) | 5.876378 / 2.077655 (3.798723) | 2.899444 / 1.504120 (1.395324) | 2.871592 / 1.541195 (1.330397) | 2.861205 / 1.468490 (1.392715) | 0.879452 / 4.584777 (-3.705325) | 5.395988 / 3.745712 (1.650275) | 4.548359 / 5.269862 (-0.721502) | 2.946601 / 4.565676 (-1.619076) | 0.099832 / 0.424275 (-0.324443) | 0.008958 / 0.007607 (0.001351) | 0.778480 / 0.226044 (0.552435) | 7.672282 / 2.268929 (5.403354) | 3.963701 / 55.444624 (-51.480923) | 3.154950 / 6.876477 (-3.721527) | 3.351070 / 2.142072 (1.208997) | 1.059459 / 4.805227 (-3.745768) | 0.212035 / 6.500664 (-6.288629) | 0.076941 / 0.075469 (0.001472) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.639813 / 1.841788 (-0.201975) | 24.807517 / 8.074308 (16.733208) | 20.662500 / 10.191392 (10.471108) | 0.244486 / 0.680424 (-0.435937) | 0.032335 / 0.534201 (-0.501866) | 0.470896 / 0.579283 (-0.108387) | 0.581561 / 0.434364 (0.147197) | 0.495158 / 0.540337 (-0.045179) | 0.788350 / 1.386936 (-0.598586) |

![](https://cml.dev/watermark.png#99641ced2e08a28cb876f483babcdd43f7dd76d2 "CML watermark")

Fix #6225


Add push_to_hub with multiple configs docs

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005920 / 0.011353 (-0.005433) | 0.003623 / 0.011008 (-0.007385) | 0.079283 / 0.038508 (0.040775) | 0.058325 / 0.023109 (0.035216) | 0.313733 / 0.275898 (0.037835) | 0.360790 / 0.323480 (0.037310) | 0.004653 / 0.007986 (-0.003332) | 0.002876 / 0.004328 (-0.001452) | 0.062137 / 0.004250 (0.057886) | 0.045084 / 0.037052 (0.008031) | 0.328569 / 0.258489 (0.070079) | 0.368965 / 0.293841 (0.075124) | 0.027085 / 0.128546 (-0.101461) | 0.008051 / 0.075646 (-0.067595) | 0.260222 / 0.419271 (-0.159050) | 0.045477 / 0.043533 (0.001944) | 0.315344 / 0.255139 (0.060205) | 0.348215 / 0.283200 (0.065015) | 0.021352 / 0.141683 (-0.120331) | 1.432200 / 1.452155 (-0.019955) | 1.509217 / 1.492716 (0.016501) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.199843 / 0.018006 (0.181837) | 0.427925 / 0.000490 (0.427435) | 0.002903 / 0.000200 (0.002703) | 0.000067 / 0.000054 (0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023121 / 0.037411 (-0.014291) | 0.072451 / 0.014526 (0.057925) | 0.083260 / 0.176557 (-0.093296) | 0.142879 / 0.737135 (-0.594257) | 0.084053 / 0.296338 (-0.212286) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.394922 / 0.215209 (0.179713) | 3.956111 / 2.077655 (1.878456) | 1.926411 / 1.504120 (0.422291) | 1.743840 / 1.541195 (0.202646) | 1.776957 / 1.468490 (0.308467) | 0.502134 / 4.584777 (-4.082643) | 3.001721 / 3.745712 (-0.743991) | 2.852496 / 5.269862 (-2.417365) | 1.862794 / 4.565676 (-2.702883) | 0.057544 / 0.424275 (-0.366731) | 0.006751 / 0.007607 (-0.000856) | 0.470619 / 0.226044 (0.244575) | 4.696674 / 2.268929 (2.427746) | 2.326545 / 55.444624 (-53.118080) | 1.980888 / 6.876477 (-4.895589) | 2.139172 / 2.142072 (-0.002901) | 0.590256 / 4.805227 (-4.214971) | 0.125815 / 6.500664 (-6.374849) | 0.061000 / 0.075469 (-0.014469) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.261948 / 1.841788 (-0.579839) | 18.317473 / 8.074308 (10.243165) | 13.810883 / 10.191392 (3.619491) | 0.146180 / 0.680424 (-0.534244) | 0.016701 / 0.534201 (-0.517500) | 0.330731 / 0.579283 (-0.248552) | 0.345103 / 0.434364 (-0.089261) | 0.374449 / 0.540337 (-0.165889) | 0.522463 / 1.386936 (-0.864473) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006217 / 0.011353 (-0.005136) | 0.003678 / 0.011008 (-0.007331) | 0.062321 / 0.038508 (0.023813) | 0.059256 / 0.023109 (0.036147) | 0.444501 / 0.275898 (0.168603) | 0.475881 / 0.323480 (0.152401) | 0.004863 / 0.007986 (-0.003123) | 0.002916 / 0.004328 (-0.001412) | 0.062197 / 0.004250 (0.057946) | 0.048449 / 0.037052 (0.011396) | 0.443680 / 0.258489 (0.185191) | 0.484570 / 0.293841 (0.190729) | 0.028694 / 0.128546 (-0.099852) | 0.008096 / 0.075646 (-0.067550) | 0.068347 / 0.419271 (-0.350924) | 0.041031 / 0.043533 (-0.002502) | 0.443907 / 0.255139 (0.188768) | 0.469888 / 0.283200 (0.186689) | 0.020237 / 0.141683 (-0.121445) | 1.438484 / 1.452155 (-0.013671) | 1.512652 / 1.492716 (0.019936) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.243118 / 0.018006 (0.225111) | 0.416797 / 0.000490 (0.416308) | 0.010421 / 0.000200 (0.010221) | 0.000082 / 0.000054 (0.000028) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026191 / 0.037411 (-0.011220) | 0.080881 / 0.014526 (0.066355) | 0.093207 / 0.176557 (-0.083349) | 0.146708 / 0.737135 (-0.590428) | 0.091676 / 0.296338 (-0.204663) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.461475 / 0.215209 (0.246266) | 4.617351 / 2.077655 (2.539696) | 2.564369 / 1.504120 (1.060249) | 2.393263 / 1.541195 (0.852068) | 2.447343 / 1.468490 (0.978853) | 0.508764 / 4.584777 (-4.076013) | 3.075460 / 3.745712 (-0.670252) | 2.884683 / 5.269862 (-2.385179) | 1.866432 / 4.565676 (-2.699244) | 0.058759 / 0.424275 (-0.365516) | 0.006591 / 0.007607 (-0.001016) | 0.537718 / 0.226044 (0.311674) | 5.378709 / 2.268929 (3.109781) | 3.006751 / 55.444624 (-52.437873) | 2.666653 / 6.876477 (-4.209824) | 2.847559 / 2.142072 (0.705486) | 0.596878 / 4.805227 (-4.208350) | 0.125073 / 6.500664 (-6.375591) | 0.061345 / 0.075469 (-0.014124) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.349066 / 1.841788 (-0.492721) | 18.684735 / 8.074308 (10.610427) | 15.128142 / 10.191392 (4.936750) | 0.149254 / 0.680424 (-0.531170) | 0.017911 / 0.534201 (-0.516290) | 0.344057 / 0.579283 (-0.235226) | 0.363474 / 0.434364 (-0.070890) | 0.399425 / 0.540337 (-0.140912) | 0.549329 / 1.386936 (-0.837607) |

![](https://cml.dev/watermark.png#e675a2396efb5204a4553721001f3b46aa4cc334 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005843 / 0.011353 (-0.005510) | 0.003549 / 0.011008 (-0.007460) | 0.082318 / 0.038508 (0.043810) | 0.056835 / 0.023109 (0.033726) | 0.312968 / 0.275898 (0.037070) | 0.345918 / 0.323480 (0.022438) | 0.003239 / 0.007986 (-0.004747) | 0.002762 / 0.004328 (-0.001567) | 0.062362 / 0.004250 (0.058111) | 0.045934 / 0.037052 (0.008882) | 0.317035 / 0.258489 (0.058546) | 0.358473 / 0.293841 (0.064632) | 0.027311 / 0.128546 (-0.101235) | 0.007994 / 0.075646 (-0.067652) | 0.261565 / 0.419271 (-0.157706) | 0.044942 / 0.043533 (0.001410) | 0.313092 / 0.255139 (0.057953) | 0.339021 / 0.283200 (0.055821) | 0.021555 / 0.141683 (-0.120127) | 1.421232 / 1.452155 (-0.030923) | 1.487597 / 1.492716 (-0.005119) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.206432 / 0.018006 (0.188425) | 0.421932 / 0.000490 (0.421442) | 0.002825 / 0.000200 (0.002625) | 0.000065 / 0.000054 (0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022795 / 0.037411 (-0.014616) | 0.072666 / 0.014526 (0.058140) | 0.082779 / 0.176557 (-0.093778) | 0.142320 / 0.737135 (-0.594815) | 0.083343 / 0.296338 (-0.212995) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.394227 / 0.215209 (0.179018) | 3.931858 / 2.077655 (1.854203) | 1.909953 / 1.504120 (0.405833) | 1.711298 / 1.541195 (0.170104) | 1.745816 / 1.468490 (0.277326) | 0.503670 / 4.584777 (-4.081107) | 3.053677 / 3.745712 (-0.692035) | 2.802597 / 5.269862 (-2.467264) | 1.825315 / 4.565676 (-2.740362) | 0.057741 / 0.424275 (-0.366534) | 0.006581 / 0.007607 (-0.001027) | 0.463597 / 0.226044 (0.237552) | 4.638821 / 2.268929 (2.369893) | 2.301266 / 55.444624 (-53.143358) | 1.967111 / 6.876477 (-4.909365) | 2.097756 / 2.142072 (-0.044317) | 0.589840 / 4.805227 (-4.215387) | 0.125538 / 6.500664 (-6.375126) | 0.061203 / 0.075469 (-0.014266) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.291815 / 1.841788 (-0.549973) | 17.997040 / 8.074308 (9.922732) | 13.616252 / 10.191392 (3.424860) | 0.137349 / 0.680424 (-0.543075) | 0.016626 / 0.534201 (-0.517575) | 0.329611 / 0.579283 (-0.249672) | 0.346592 / 0.434364 (-0.087772) | 0.379521 / 0.540337 (-0.160817) | 0.528058 / 1.386936 (-0.858878) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006073 / 0.011353 (-0.005280) | 0.003594 / 0.011008 (-0.007414) | 0.062537 / 0.038508 (0.024029) | 0.057503 / 0.023109 (0.034394) | 0.449427 / 0.275898 (0.173529) | 0.482729 / 0.323480 (0.159249) | 0.004690 / 0.007986 (-0.003295) | 0.002901 / 0.004328 (-0.001428) | 0.062421 / 0.004250 (0.058171) | 0.046405 / 0.037052 (0.009353) | 0.456578 / 0.258489 (0.198089) | 0.492268 / 0.293841 (0.198427) | 0.028283 / 0.128546 (-0.100263) | 0.008028 / 0.075646 (-0.067618) | 0.067885 / 0.419271 (-0.351387) | 0.041273 / 0.043533 (-0.002260) | 0.449870 / 0.255139 (0.194731) | 0.472305 / 0.283200 (0.189106) | 0.018556 / 0.141683 (-0.123127) | 1.449016 / 1.452155 (-0.003138) | 1.490839 / 1.492716 (-0.001877) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.226569 / 0.018006 (0.208563) | 0.417106 / 0.000490 (0.416616) | 0.002784 / 0.000200 (0.002584) | 0.000072 / 0.000054 (0.000018) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025803 / 0.037411 (-0.011608) | 0.081084 / 0.014526 (0.066559) | 0.091851 / 0.176557 (-0.084706) | 0.143982 / 0.737135 (-0.593153) | 0.090511 / 0.296338 (-0.205827) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.463664 / 0.215209 (0.248454) | 4.634528 / 2.077655 (2.556874) | 2.574739 / 1.504120 (1.070619) | 2.412857 / 1.541195 (0.871662) | 2.442858 / 1.468490 (0.974368) | 0.511990 / 4.584777 (-4.072787) | 3.070345 / 3.745712 (-0.675367) | 2.842290 / 5.269862 (-2.427571) | 1.846727 / 4.565676 (-2.718950) | 0.058852 / 0.424275 (-0.365424) | 0.006624 / 0.007607 (-0.000983) | 0.539616 / 0.226044 (0.313571) | 5.410784 / 2.268929 (3.141856) | 3.065593 / 55.444624 (-52.379031) | 2.677930 / 6.876477 (-4.198547) | 2.817548 / 2.142072 (0.675476) | 0.602672 / 4.805227 (-4.202555) | 0.125689 / 6.500664 (-6.374975) | 0.062007 / 0.075469 (-0.013462) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.335336 / 1.841788 (-0.506452) | 18.310099 / 8.074308 (10.235791) | 14.818452 / 10.191392 (4.627060) | 0.154001 / 0.680424 (-0.526423) | 0.017892 / 0.534201 (-0.516309) | 0.345989 / 0.579283 (-0.233294) | 0.352108 / 0.434364 (-0.082256) | 0.394333 / 0.540337 (-0.146004) | 0.547680 / 1.386936 (-0.839256) |

![](https://cml.dev/watermark.png#d058d6e9b849acb5bc61d7df597a94253b487eb6 "CML watermark")


Conversion from RGB to BGR in Object Detection tutorial

The [tutorial](https://huggingface.co/docs/datasets/main/en/object_detection) mentions the necessity of conversion the input image from BGR to RGB

> albumentations expects the image to be in BGR format, not RGB, so you’ll have to convert the image before applying the transform.

[Link to tutorial](https://github.com/huggingface/datasets/blob/0a068dbf3b446417ffd89d32857608394ec699e6/docs/source/object_detection.mdx#L77)

However, relevant albumentations' tutorials [on channels conversion](https://albumentations.ai/docs/examples/example/#read-the-image-from-the-disk-and-convert-it-from-the-bgr-color-space-to-the-rgb-color-space) and  [on boxes](https://albumentations.ai/docs/examples/example_bboxes/) imply that it's not really true no more.

I suggest removing this outdated conversion from the tutorial.


Ignore `dataset_info.json` in data files resolution

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009450 / 0.011353 (-0.001903) | 0.007339 / 0.011008 (-0.003669) | 0.110150 / 0.038508 (0.071641) | 0.087794 / 0.023109 (0.064685) | 0.472099 / 0.275898 (0.196201) | 0.476622 / 0.323480 (0.153142) | 0.005057 / 0.007986 (-0.002929) | 0.005262 / 0.004328 (0.000933) | 0.103059 / 0.004250 (0.098808) | 0.069815 / 0.037052 (0.032763) | 0.489377 / 0.258489 (0.230888) | 0.547087 / 0.293841 (0.253247) | 0.048883 / 0.128546 (-0.079663) | 0.019192 / 0.075646 (-0.056454) | 0.410865 / 0.419271 (-0.008407) | 0.076215 / 0.043533 (0.032682) | 0.484825 / 0.255139 (0.229686) | 0.519035 / 0.283200 (0.235835) | 0.042030 / 0.141683 (-0.099653) | 1.909630 / 1.452155 (0.457475) | 2.120869 / 1.492716 (0.628153) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.267600 / 0.018006 (0.249594) | 0.619135 / 0.000490 (0.618645) | 0.005897 / 0.000200 (0.005697) | 0.000142 / 0.000054 (0.000087) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033265 / 0.037411 (-0.004146) | 0.104476 / 0.014526 (0.089950) | 0.129199 / 0.176557 (-0.047358) | 0.196898 / 0.737135 (-0.540238) | 0.118852 / 0.296338 (-0.177487) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.598908 / 0.215209 (0.383699) | 6.263096 / 2.077655 (4.185441) | 2.672134 / 1.504120 (1.168014) | 2.428706 / 1.541195 (0.887511) | 2.431651 / 1.468490 (0.963161) | 0.918465 / 4.584777 (-3.666312) | 5.667857 / 3.745712 (1.922145) | 5.113696 / 5.269862 (-0.156166) | 3.276805 / 4.565676 (-1.288872) | 0.101829 / 0.424275 (-0.322446) | 0.010224 / 0.007607 (0.002617) | 0.741547 / 0.226044 (0.515502) | 7.517002 / 2.268929 (5.248073) | 3.546353 / 55.444624 (-51.898272) | 2.845956 / 6.876477 (-4.030521) | 3.172777 / 2.142072 (1.030705) | 1.153485 / 4.805227 (-3.651742) | 0.225758 / 6.500664 (-6.274906) | 0.084333 / 0.075469 (0.008864) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.704645 / 1.841788 (-0.137143) | 27.044110 / 8.074308 (18.969801) | 24.653837 / 10.191392 (14.462445) | 0.235452 / 0.680424 (-0.444971) | 0.029285 / 0.534201 (-0.504916) | 0.576122 / 0.579283 (-0.003161) | 0.626263 / 0.434364 (0.191899) | 0.600201 / 0.540337 (0.059864) | 0.838406 / 1.386936 (-0.548530) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.013754 / 0.011353 (0.002401) | 0.005954 / 0.011008 (-0.005054) | 0.089766 / 0.038508 (0.051258) | 0.096126 / 0.023109 (0.073017) | 0.556455 / 0.275898 (0.280557) | 0.579302 / 0.323480 (0.255822) | 0.009222 / 0.007986 (0.001236) | 0.006128 / 0.004328 (0.001800) | 0.099725 / 0.004250 (0.095475) | 0.075642 / 0.037052 (0.038589) | 0.556645 / 0.258489 (0.298156) | 0.615898 / 0.293841 (0.322057) | 0.057728 / 0.128546 (-0.070818) | 0.016746 / 0.075646 (-0.058900) | 0.098053 / 0.419271 (-0.321219) | 0.066676 / 0.043533 (0.023143) | 0.534156 / 0.255139 (0.279017) | 0.590020 / 0.283200 (0.306820) | 0.038782 / 0.141683 (-0.102901) | 1.952301 / 1.452155 (0.500146) | 2.104255 / 1.492716 (0.611539) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.305945 / 0.018006 (0.287939) | 0.643915 / 0.000490 (0.643426) | 0.006268 / 0.000200 (0.006068) | 0.000156 / 0.000054 (0.000102) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.039891 / 0.037411 (0.002479) | 0.117888 / 0.014526 (0.103363) | 0.134230 / 0.176557 (-0.042326) | 0.212544 / 0.737135 (-0.524591) | 0.128858 / 0.296338 (-0.167480) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.718165 / 0.215209 (0.502955) | 7.023867 / 2.077655 (4.946212) | 3.391344 / 1.504120 (1.887224) | 3.021248 / 1.541195 (1.480053) | 3.010217 / 1.468490 (1.541727) | 0.932608 / 4.584777 (-3.652169) | 5.787536 / 3.745712 (2.041824) | 5.221305 / 5.269862 (-0.048557) | 3.282552 / 4.565676 (-1.283125) | 0.105486 / 0.424275 (-0.318789) | 0.009800 / 0.007607 (0.002193) | 0.839358 / 0.226044 (0.613314) | 8.279712 / 2.268929 (6.010784) | 4.118466 / 55.444624 (-51.326158) | 3.407738 / 6.876477 (-3.468739) | 3.632538 / 2.142072 (1.490466) | 1.109673 / 4.805227 (-3.695555) | 0.216541 / 6.500664 (-6.284123) | 0.094031 / 0.075469 (0.018562) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.983979 / 1.841788 (0.142191) | 27.125882 / 8.074308 (19.051573) | 24.714002 / 10.191392 (14.522610) | 0.264417 / 0.680424 (-0.416007) | 0.034783 / 0.534201 (-0.499418) | 0.533304 / 0.579283 (-0.045979) | 0.647798 / 0.434364 (0.213434) | 0.588680 / 0.540337 (0.048343) | 0.854250 / 1.386936 (-0.532686) |

![](https://cml.dev/watermark.png#491604b46b1fd8d6fd1b7531f7917ccd657665a6 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006664 / 0.011353 (-0.004689) | 0.004164 / 0.011008 (-0.006844) | 0.085192 / 0.038508 (0.046684) | 0.073578 / 0.023109 (0.050469) | 0.356379 / 0.275898 (0.080481) | 0.389381 / 0.323480 (0.065902) | 0.005527 / 0.007986 (-0.002459) | 0.003488 / 0.004328 (-0.000840) | 0.065640 / 0.004250 (0.061390) | 0.055013 / 0.037052 (0.017960) | 0.358002 / 0.258489 (0.099513) | 0.400663 / 0.293841 (0.106822) | 0.030937 / 0.128546 (-0.097609) | 0.008838 / 0.075646 (-0.066808) | 0.287488 / 0.419271 (-0.131784) | 0.051503 / 0.043533 (0.007971) | 0.353945 / 0.255139 (0.098806) | 0.388778 / 0.283200 (0.105579) | 0.023346 / 0.141683 (-0.118337) | 1.479621 / 1.452155 (0.027466) | 1.559164 / 1.492716 (0.066448) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.245160 / 0.018006 (0.227154) | 0.561890 / 0.000490 (0.561400) | 0.004339 / 0.000200 (0.004139) | 0.000083 / 0.000054 (0.000028) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028460 / 0.037411 (-0.008952) | 0.082046 / 0.014526 (0.067520) | 0.098005 / 0.176557 (-0.078552) | 0.154171 / 0.737135 (-0.582965) | 0.097632 / 0.296338 (-0.198707) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.389993 / 0.215209 (0.174784) | 3.893287 / 2.077655 (1.815632) | 1.885668 / 1.504120 (0.381549) | 1.715055 / 1.541195 (0.173860) | 1.778008 / 1.468490 (0.309518) | 0.482818 / 4.584777 (-4.101959) | 3.572153 / 3.745712 (-0.173559) | 3.267666 / 5.269862 (-2.002196) | 2.088394 / 4.565676 (-2.477282) | 0.056961 / 0.424275 (-0.367314) | 0.007784 / 0.007607 (0.000177) | 0.466586 / 0.226044 (0.240542) | 4.652505 / 2.268929 (2.383576) | 2.491392 / 55.444624 (-52.953233) | 2.127600 / 6.876477 (-4.748877) | 2.296778 / 2.142072 (0.154705) | 0.582332 / 4.805227 (-4.222895) | 0.134372 / 6.500664 (-6.366292) | 0.061737 / 0.075469 (-0.013732) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.253647 / 1.841788 (-0.588140) | 19.802353 / 8.074308 (11.728045) | 14.262815 / 10.191392 (4.071423) | 0.169489 / 0.680424 (-0.510935) | 0.018108 / 0.534201 (-0.516093) | 0.391711 / 0.579283 (-0.187572) | 0.406169 / 0.434364 (-0.028195) | 0.456728 / 0.540337 (-0.083609) | 0.633538 / 1.386936 (-0.753398) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006661 / 0.011353 (-0.004692) | 0.004181 / 0.011008 (-0.006827) | 0.064945 / 0.038508 (0.026437) | 0.073965 / 0.023109 (0.050856) | 0.406549 / 0.275898 (0.130651) | 0.441568 / 0.323480 (0.118089) | 0.005579 / 0.007986 (-0.002407) | 0.003523 / 0.004328 (-0.000805) | 0.065270 / 0.004250 (0.061019) | 0.055596 / 0.037052 (0.018544) | 0.407701 / 0.258489 (0.149212) | 0.444609 / 0.293841 (0.150768) | 0.031749 / 0.128546 (-0.096797) | 0.008680 / 0.075646 (-0.066966) | 0.071154 / 0.419271 (-0.348117) | 0.047376 / 0.043533 (0.003843) | 0.406409 / 0.255139 (0.151270) | 0.420477 / 0.283200 (0.137278) | 0.023707 / 0.141683 (-0.117976) | 1.484516 / 1.452155 (0.032361) | 1.568493 / 1.492716 (0.075777) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.266534 / 0.018006 (0.248528) | 0.573806 / 0.000490 (0.573316) | 0.006247 / 0.000200 (0.006048) | 0.000165 / 0.000054 (0.000110) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033436 / 0.037411 (-0.003976) | 0.091947 / 0.014526 (0.077421) | 0.105556 / 0.176557 (-0.071000) | 0.162094 / 0.737135 (-0.575041) | 0.107879 / 0.296338 (-0.188459) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.429126 / 0.215209 (0.213917) | 4.281329 / 2.077655 (2.203675) | 2.295406 / 1.504120 (0.791286) | 2.123336 / 1.541195 (0.582141) | 2.190804 / 1.468490 (0.722314) | 0.492972 / 4.584777 (-4.091805) | 3.638485 / 3.745712 (-0.107227) | 3.304576 / 5.269862 (-1.965285) | 2.063694 / 4.565676 (-2.501983) | 0.058549 / 0.424275 (-0.365726) | 0.007591 / 0.007607 (-0.000016) | 0.504268 / 0.226044 (0.278223) | 5.031990 / 2.268929 (2.763061) | 2.773173 / 55.444624 (-52.671451) | 2.430789 / 6.876477 (-4.445688) | 2.699900 / 2.142072 (0.557828) | 0.593220 / 4.805227 (-4.212007) | 0.133710 / 6.500664 (-6.366954) | 0.059840 / 0.075469 (-0.015629) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.351158 / 1.841788 (-0.490629) | 20.176310 / 8.074308 (12.102002) | 14.933202 / 10.191392 (4.741810) | 0.169920 / 0.680424 (-0.510503) | 0.020156 / 0.534201 (-0.514045) | 0.397440 / 0.579283 (-0.181843) | 0.409395 / 0.434364 (-0.024969) | 0.471066 / 0.540337 (-0.069271) | 0.642670 / 1.386936 (-0.744266) |

![](https://cml.dev/watermark.png#cf90ca7fbfd9c4639cc3faf0a349eb26490e38fc "CML watermark")

`save_to_disk` creates this file, but also [`HugginFaceDatasetSever`](https://github.com/gradio-app/gradio/blob/26fef8c7f85a006c7e25cdbed1792df19c512d02/gradio/flagging.py#L214), so this is needed to avoid issues such as [this one](https://discord.com/channels/879548962464493619/1149295819938349107/1149295819938349107).