Unpin responses

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006113 / 0.011353 (-0.005239) | 0.004195 / 0.011008 (-0.006813) | 0.098103 / 0.038508 (0.059595) | 0.027970 / 0.023109 (0.004860) | 0.300992 / 0.275898 (0.025094) | 0.335402 / 0.323480 (0.011922) | 0.005079 / 0.007986 (-0.002906) | 0.003516 / 0.004328 (-0.000813) | 0.077311 / 0.004250 (0.073061) | 0.037863 / 0.037052 (0.000810) | 0.302638 / 0.258489 (0.044149) | 0.346554 / 0.293841 (0.052713) | 0.025218 / 0.128546 (-0.103328) | 0.008630 / 0.075646 (-0.067017) | 0.319748 / 0.419271 (-0.099523) | 0.049182 / 0.043533 (0.005650) | 0.306233 / 0.255139 (0.051094) | 0.331040 / 0.283200 (0.047840) | 0.089203 / 0.141683 (-0.052480) | 1.496104 / 1.452155 (0.043949) | 1.567878 / 1.492716 (0.075162) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.215774 / 0.018006 (0.197768) | 0.436810 / 0.000490 (0.436320) | 0.000307 / 0.000200 (0.000107) | 0.000059 / 0.000054 (0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024102 / 0.037411 (-0.013310) | 0.095459 / 0.014526 (0.080933) | 0.106564 / 0.176557 (-0.069992) | 0.169894 / 0.737135 (-0.567241) | 0.109152 / 0.296338 (-0.187186) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.429066 / 0.215209 (0.213857) | 4.297385 / 2.077655 (2.219730) | 2.054854 / 1.504120 (0.550734) | 1.846844 / 1.541195 (0.305649) | 1.840807 / 1.468490 (0.372317) | 0.553193 / 4.584777 (-4.031584) | 3.366788 / 3.745712 (-0.378924) | 1.727337 / 5.269862 (-3.542525) | 0.994357 / 4.565676 (-3.571319) | 0.067790 / 0.424275 (-0.356485) | 0.012002 / 0.007607 (0.004395) | 0.533335 / 0.226044 (0.307291) | 5.341341 / 2.268929 (3.072412) | 2.543581 / 55.444624 (-52.901043) | 2.220374 / 6.876477 (-4.656103) | 2.321656 / 2.142072 (0.179583) | 0.654408 / 4.805227 (-4.150819) | 0.134693 / 6.500664 (-6.365971) | 0.066926 / 0.075469 (-0.008544) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.209463 / 1.841788 (-0.632325) | 13.568221 / 8.074308 (5.493913) | 13.965418 / 10.191392 (3.774026) | 0.145049 / 0.680424 (-0.535375) | 0.016936 / 0.534201 (-0.517265) | 0.371587 / 0.579283 (-0.207696) | 0.386363 / 0.434364 (-0.048001) | 0.437137 / 0.540337 (-0.103201) | 0.514779 / 1.386936 (-0.872157) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006245 / 0.011353 (-0.005108) | 0.004232 / 0.011008 (-0.006776) | 0.075682 / 0.038508 (0.037174) | 0.027858 / 0.023109 (0.004749) | 0.425325 / 0.275898 (0.149427) | 0.466732 / 0.323480 (0.143253) | 0.005240 / 0.007986 (-0.002745) | 0.003506 / 0.004328 (-0.000823) | 0.075294 / 0.004250 (0.071044) | 0.041677 / 0.037052 (0.004624) | 0.426552 / 0.258489 (0.168063) | 0.469452 / 0.293841 (0.175611) | 0.025443 / 0.128546 (-0.103104) | 0.008526 / 0.075646 (-0.067120) | 0.082190 / 0.419271 (-0.337081) | 0.040906 / 0.043533 (-0.002626) | 0.428406 / 0.255139 (0.173267) | 0.446795 / 0.283200 (0.163595) | 0.093837 / 0.141683 (-0.047846) | 1.518639 / 1.452155 (0.066484) | 1.620214 / 1.492716 (0.127498) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.223259 / 0.018006 (0.205253) | 0.425077 / 0.000490 (0.424588) | 0.001980 / 0.000200 (0.001780) | 0.000077 / 0.000054 (0.000023) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025813 / 0.037411 (-0.011599) | 0.103062 / 0.014526 (0.088536) | 0.108958 / 0.176557 (-0.067598) | 0.161591 / 0.737135 (-0.575544) | 0.112130 / 0.296338 (-0.184209) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.472843 / 0.215209 (0.257634) | 4.713281 / 2.077655 (2.635626) | 2.458216 / 1.504120 (0.954096) | 2.272467 / 1.541195 (0.731273) | 2.324456 / 1.468490 (0.855965) | 0.554686 / 4.584777 (-4.030091) | 3.445079 / 3.745712 (-0.300634) | 3.451896 / 5.269862 (-1.817966) | 1.431065 / 4.565676 (-3.134612) | 0.067868 / 0.424275 (-0.356407) | 0.012093 / 0.007607 (0.004486) | 0.573571 / 0.226044 (0.347526) | 5.820452 / 2.268929 (3.551523) | 2.934858 / 55.444624 (-52.509767) | 2.602719 / 6.876477 (-4.273758) | 2.645999 / 2.142072 (0.503927) | 0.660688 / 4.805227 (-4.144540) | 0.137490 / 6.500664 (-6.363174) | 0.068311 / 0.075469 (-0.007158) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.321709 / 1.841788 (-0.520079) | 14.592346 / 8.074308 (6.518038) | 14.520748 / 10.191392 (4.329356) | 0.132689 / 0.680424 (-0.547735) | 0.016422 / 0.534201 (-0.517779) | 0.370071 / 0.579283 (-0.209212) | 0.397091 / 0.434364 (-0.037273) | 0.431979 / 0.540337 (-0.108358) | 0.509965 / 1.386936 (-0.876971) |

![](https://cml.dev/watermark.png#8bcd061ab2082a0862f30329bc52f6e0d321805c "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006182 / 0.011353 (-0.005171) | 0.004153 / 0.011008 (-0.006855) | 0.095715 / 0.038508 (0.057207) | 0.032457 / 0.023109 (0.009347) | 0.314961 / 0.275898 (0.039063) | 0.353696 / 0.323480 (0.030216) | 0.005256 / 0.007986 (-0.002729) | 0.004870 / 0.004328 (0.000541) | 0.072442 / 0.004250 (0.068192) | 0.046102 / 0.037052 (0.009050) | 0.324410 / 0.258489 (0.065921) | 0.366861 / 0.293841 (0.073020) | 0.027088 / 0.128546 (-0.101458) | 0.008572 / 0.075646 (-0.067075) | 0.325988 / 0.419271 (-0.093284) | 0.049494 / 0.043533 (0.005961) | 0.311221 / 0.255139 (0.056082) | 0.359720 / 0.283200 (0.076521) | 0.095101 / 0.141683 (-0.046581) | 1.472821 / 1.452155 (0.020667) | 1.516157 / 1.492716 (0.023441) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.210456 / 0.018006 (0.192450) | 0.439440 / 0.000490 (0.438950) | 0.003764 / 0.000200 (0.003564) | 0.000087 / 0.000054 (0.000033) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024076 / 0.037411 (-0.013335) | 0.104886 / 0.014526 (0.090360) | 0.114164 / 0.176557 (-0.062393) | 0.167289 / 0.737135 (-0.569847) | 0.116457 / 0.296338 (-0.179882) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.400039 / 0.215209 (0.184830) | 3.973243 / 2.077655 (1.895588) | 1.801991 / 1.504120 (0.297871) | 1.592017 / 1.541195 (0.050822) | 1.612564 / 1.468490 (0.144074) | 0.527475 / 4.584777 (-4.057302) | 3.676246 / 3.745712 (-0.069466) | 1.806423 / 5.269862 (-3.463438) | 1.176921 / 4.565676 (-3.388756) | 0.065902 / 0.424275 (-0.358373) | 0.012245 / 0.007607 (0.004638) | 0.490883 / 0.226044 (0.264838) | 4.905270 / 2.268929 (2.636341) | 2.218694 / 55.444624 (-53.225930) | 1.903074 / 6.876477 (-4.973403) | 1.979505 / 2.142072 (-0.162567) | 0.644415 / 4.805227 (-4.160812) | 0.142433 / 6.500664 (-6.358231) | 0.063564 / 0.075469 (-0.011905) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.193756 / 1.841788 (-0.648032) | 14.673103 / 8.074308 (6.598795) | 13.410951 / 10.191392 (3.219559) | 0.159175 / 0.680424 (-0.521249) | 0.017076 / 0.534201 (-0.517125) | 0.388880 / 0.579283 (-0.190403) | 0.409974 / 0.434364 (-0.024390) | 0.454494 / 0.540337 (-0.085844) | 0.556873 / 1.386936 (-0.830063) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006107 / 0.011353 (-0.005246) | 0.004433 / 0.011008 (-0.006575) | 0.073892 / 0.038508 (0.035384) | 0.032386 / 0.023109 (0.009277) | 0.370339 / 0.275898 (0.094441) | 0.388996 / 0.323480 (0.065516) | 0.005438 / 0.007986 (-0.002548) | 0.003875 / 0.004328 (-0.000454) | 0.073867 / 0.004250 (0.069617) | 0.048350 / 0.037052 (0.011298) | 0.380328 / 0.258489 (0.121839) | 0.411373 / 0.293841 (0.117532) | 0.028183 / 0.128546 (-0.100363) | 0.008924 / 0.075646 (-0.066723) | 0.082484 / 0.419271 (-0.336787) | 0.047321 / 0.043533 (0.003788) | 0.371702 / 0.255139 (0.116563) | 0.380535 / 0.283200 (0.097335) | 0.100772 / 0.141683 (-0.040911) | 1.475038 / 1.452155 (0.022883) | 1.564293 / 1.492716 (0.071577) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.214589 / 0.018006 (0.196583) | 0.437193 / 0.000490 (0.436703) | 0.003676 / 0.000200 (0.003476) | 0.000094 / 0.000054 (0.000040) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027991 / 0.037411 (-0.009421) | 0.111154 / 0.014526 (0.096628) | 0.120365 / 0.176557 (-0.056191) | 0.173601 / 0.737135 (-0.563535) | 0.126244 / 0.296338 (-0.170094) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.442848 / 0.215209 (0.227639) | 4.398336 / 2.077655 (2.320681) | 2.217058 / 1.504120 (0.712938) | 2.011155 / 1.541195 (0.469960) | 2.123086 / 1.468490 (0.654596) | 0.525857 / 4.584777 (-4.058920) | 3.730191 / 3.745712 (-0.015521) | 3.517680 / 5.269862 (-1.752181) | 1.557940 / 4.565676 (-3.007736) | 0.066309 / 0.424275 (-0.357967) | 0.011788 / 0.007607 (0.004181) | 0.548506 / 0.226044 (0.322462) | 5.483615 / 2.268929 (3.214687) | 2.663784 / 55.444624 (-52.780840) | 2.325744 / 6.876477 (-4.550732) | 2.344179 / 2.142072 (0.202106) | 0.644217 / 4.805227 (-4.161010) | 0.141546 / 6.500664 (-6.359118) | 0.063730 / 0.075469 (-0.011739) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.296032 / 1.841788 (-0.545756) | 14.903729 / 8.074308 (6.829421) | 14.505409 / 10.191392 (4.314017) | 0.170478 / 0.680424 (-0.509946) | 0.017876 / 0.534201 (-0.516325) | 0.401047 / 0.579283 (-0.178236) | 0.417855 / 0.434364 (-0.016509) | 0.472138 / 0.540337 (-0.068200) | 0.570859 / 1.386936 (-0.816077) |

![](https://cml.dev/watermark.png#5a4d530965eb35c66955ef89df79210c66b7f5e6 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008495 / 0.011353 (-0.002858) | 0.005322 / 0.011008 (-0.005686) | 0.125471 / 0.038508 (0.086962) | 0.034604 / 0.023109 (0.011495) | 0.419831 / 0.275898 (0.143933) | 0.415707 / 0.323480 (0.092227) | 0.007471 / 0.007986 (-0.000515) | 0.005441 / 0.004328 (0.001112) | 0.095412 / 0.004250 (0.091162) | 0.053865 / 0.037052 (0.016812) | 0.375257 / 0.258489 (0.116768) | 0.438114 / 0.293841 (0.144273) | 0.046183 / 0.128546 (-0.082363) | 0.013663 / 0.075646 (-0.061984) | 0.438317 / 0.419271 (0.019045) | 0.065665 / 0.043533 (0.022133) | 0.387640 / 0.255139 (0.132501) | 0.431350 / 0.283200 (0.148150) | 0.112841 / 0.141683 (-0.028842) | 1.778639 / 1.452155 (0.326484) | 1.891948 / 1.492716 (0.399232) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.284371 / 0.018006 (0.266365) | 0.598247 / 0.000490 (0.597758) | 0.013674 / 0.000200 (0.013474) | 0.000483 / 0.000054 (0.000428) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032437 / 0.037411 (-0.004974) | 0.120547 / 0.014526 (0.106021) | 0.129845 / 0.176557 (-0.046711) | 0.203455 / 0.737135 (-0.533680) | 0.140039 / 0.296338 (-0.156300) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.596549 / 0.215209 (0.381340) | 6.138766 / 2.077655 (4.061111) | 2.515506 / 1.504120 (1.011386) | 2.124472 / 1.541195 (0.583277) | 2.160812 / 1.468490 (0.692322) | 0.898965 / 4.584777 (-3.685812) | 5.588152 / 3.745712 (1.842440) | 2.717580 / 5.269862 (-2.552282) | 1.683641 / 4.565676 (-2.882036) | 0.108045 / 0.424275 (-0.316230) | 0.014089 / 0.007607 (0.006481) | 0.749567 / 0.226044 (0.523523) | 7.518051 / 2.268929 (5.249123) | 3.198238 / 55.444624 (-52.246386) | 2.575156 / 6.876477 (-4.301321) | 2.725818 / 2.142072 (0.583745) | 1.149338 / 4.805227 (-3.655889) | 0.220443 / 6.500664 (-6.280221) | 0.081452 / 0.075469 (0.005983) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.624462 / 1.841788 (-0.217325) | 18.204963 / 8.074308 (10.130655) | 21.379169 / 10.191392 (11.187777) | 0.248520 / 0.680424 (-0.431903) | 0.030121 / 0.534201 (-0.504080) | 0.499542 / 0.579283 (-0.079741) | 0.599783 / 0.434364 (0.165419) | 0.597642 / 0.540337 (0.057305) | 0.681948 / 1.386936 (-0.704988) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008431 / 0.011353 (-0.002921) | 0.006143 / 0.011008 (-0.004865) | 0.107531 / 0.038508 (0.069023) | 0.036308 / 0.023109 (0.013199) | 0.480555 / 0.275898 (0.204657) | 0.556407 / 0.323480 (0.232927) | 0.007614 / 0.007986 (-0.000372) | 0.004749 / 0.004328 (0.000421) | 0.105734 / 0.004250 (0.101484) | 0.051619 / 0.037052 (0.014567) | 0.514821 / 0.258489 (0.256332) | 0.562143 / 0.293841 (0.268302) | 0.042957 / 0.128546 (-0.085589) | 0.015142 / 0.075646 (-0.060505) | 0.143161 / 0.419271 (-0.276111) | 0.061910 / 0.043533 (0.018377) | 0.496923 / 0.255139 (0.241784) | 0.556302 / 0.283200 (0.273102) | 0.136700 / 0.141683 (-0.004983) | 1.886184 / 1.452155 (0.434029) | 2.004087 / 1.492716 (0.511371) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.235530 / 0.018006 (0.217523) | 0.600796 / 0.000490 (0.600306) | 0.009074 / 0.000200 (0.008874) | 0.000203 / 0.000054 (0.000149) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036345 / 0.037411 (-0.001066) | 0.126112 / 0.014526 (0.111586) | 0.143369 / 0.176557 (-0.033188) | 0.211381 / 0.737135 (-0.525755) | 0.151095 / 0.296338 (-0.145243) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.695022 / 0.215209 (0.479813) | 6.685981 / 2.077655 (4.608326) | 3.104521 / 1.504120 (1.600401) | 2.758323 / 1.541195 (1.217128) | 2.706286 / 1.468490 (1.237796) | 0.941182 / 4.584777 (-3.643595) | 5.715839 / 3.745712 (1.970127) | 5.089636 / 5.269862 (-0.180226) | 2.594739 / 4.565676 (-1.970937) | 0.112621 / 0.424275 (-0.311655) | 0.014001 / 0.007607 (0.006394) | 0.812990 / 0.226044 (0.586945) | 8.060890 / 2.268929 (5.791961) | 3.832506 / 55.444624 (-51.612119) | 3.148051 / 6.876477 (-3.728425) | 3.110096 / 2.142072 (0.968023) | 1.105050 / 4.805227 (-3.700178) | 0.219835 / 6.500664 (-6.280829) | 0.078600 / 0.075469 (0.003131) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.707551 / 1.841788 (-0.134237) | 19.238194 / 8.074308 (11.163885) | 22.167076 / 10.191392 (11.975684) | 0.233458 / 0.680424 (-0.446966) | 0.025131 / 0.534201 (-0.509070) | 0.525241 / 0.579283 (-0.054042) | 0.649666 / 0.434364 (0.215303) | 0.602941 / 0.540337 (0.062603) | 0.718472 / 1.386936 (-0.668464) |

![](https://cml.dev/watermark.png#ac3a42c525d91cb630273702a0c110a71c9bf54b "CML watermark")

Fix #5906


Raise error in `DatasetBuilder.as_dataset` when `file_format` is not `"arrow"`

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006416 / 0.011353 (-0.004937) | 0.004278 / 0.011008 (-0.006731) | 0.097562 / 0.038508 (0.059054) | 0.029488 / 0.023109 (0.006379) | 0.308648 / 0.275898 (0.032750) | 0.339879 / 0.323480 (0.016399) | 0.005288 / 0.007986 (-0.002697) | 0.005033 / 0.004328 (0.000704) | 0.074666 / 0.004250 (0.070416) | 0.034888 / 0.037052 (-0.002164) | 0.309960 / 0.258489 (0.051471) | 0.344276 / 0.293841 (0.050435) | 0.025564 / 0.128546 (-0.102982) | 0.008579 / 0.075646 (-0.067067) | 0.319796 / 0.419271 (-0.099476) | 0.044786 / 0.043533 (0.001253) | 0.308888 / 0.255139 (0.053749) | 0.334001 / 0.283200 (0.050802) | 0.089917 / 0.141683 (-0.051766) | 1.456696 / 1.452155 (0.004541) | 1.542273 / 1.492716 (0.049557) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.213236 / 0.018006 (0.195230) | 0.425139 / 0.000490 (0.424650) | 0.008831 / 0.000200 (0.008631) | 0.000209 / 0.000054 (0.000155) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023990 / 0.037411 (-0.013421) | 0.096787 / 0.014526 (0.082261) | 0.105783 / 0.176557 (-0.070774) | 0.167182 / 0.737135 (-0.569954) | 0.108896 / 0.296338 (-0.187442) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.419844 / 0.215209 (0.204635) | 4.201909 / 2.077655 (2.124254) | 1.910784 / 1.504120 (0.406664) | 1.685183 / 1.541195 (0.143988) | 1.716927 / 1.468490 (0.248437) | 0.548261 / 4.584777 (-4.036516) | 3.414168 / 3.745712 (-0.331544) | 1.695446 / 5.269862 (-3.574415) | 0.989668 / 4.565676 (-3.576008) | 0.067328 / 0.424275 (-0.356948) | 0.012084 / 0.007607 (0.004477) | 0.523799 / 0.226044 (0.297754) | 5.240589 / 2.268929 (2.971661) | 2.331618 / 55.444624 (-53.113007) | 1.996094 / 6.876477 (-4.880383) | 2.105450 / 2.142072 (-0.036623) | 0.654614 / 4.805227 (-4.150613) | 0.134721 / 6.500664 (-6.365943) | 0.066227 / 0.075469 (-0.009242) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.196266 / 1.841788 (-0.645521) | 13.990045 / 8.074308 (5.915737) | 13.928126 / 10.191392 (3.736734) | 0.142600 / 0.680424 (-0.537824) | 0.016462 / 0.534201 (-0.517739) | 0.363113 / 0.579283 (-0.216170) | 0.428590 / 0.434364 (-0.005773) | 0.452594 / 0.540337 (-0.087743) | 0.551678 / 1.386936 (-0.835258) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005992 / 0.011353 (-0.005361) | 0.004161 / 0.011008 (-0.006847) | 0.076098 / 0.038508 (0.037589) | 0.028559 / 0.023109 (0.005450) | 0.411696 / 0.275898 (0.135798) | 0.444519 / 0.323480 (0.121040) | 0.004965 / 0.007986 (-0.003021) | 0.003452 / 0.004328 (-0.000876) | 0.075107 / 0.004250 (0.070857) | 0.037305 / 0.037052 (0.000252) | 0.429728 / 0.258489 (0.171239) | 0.444313 / 0.293841 (0.150472) | 0.025278 / 0.128546 (-0.103268) | 0.008527 / 0.075646 (-0.067120) | 0.081502 / 0.419271 (-0.337770) | 0.041237 / 0.043533 (-0.002296) | 0.417848 / 0.255139 (0.162709) | 0.426615 / 0.283200 (0.143415) | 0.094641 / 0.141683 (-0.047041) | 1.525141 / 1.452155 (0.072987) | 1.615608 / 1.492716 (0.122892) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.192867 / 0.018006 (0.174861) | 0.414979 / 0.000490 (0.414490) | 0.000815 / 0.000200 (0.000615) | 0.000068 / 0.000054 (0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025354 / 0.037411 (-0.012058) | 0.102085 / 0.014526 (0.087559) | 0.107930 / 0.176557 (-0.068626) | 0.160483 / 0.737135 (-0.576652) | 0.112341 / 0.296338 (-0.183997) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.446938 / 0.215209 (0.231728) | 4.480057 / 2.077655 (2.402402) | 2.154825 / 1.504120 (0.650705) | 1.942774 / 1.541195 (0.401580) | 1.996418 / 1.468490 (0.527928) | 0.556728 / 4.584777 (-4.028049) | 3.441228 / 3.745712 (-0.304484) | 3.004179 / 5.269862 (-2.265683) | 1.314104 / 4.565676 (-3.251573) | 0.068670 / 0.424275 (-0.355606) | 0.011972 / 0.007607 (0.004365) | 0.556604 / 0.226044 (0.330560) | 5.561783 / 2.268929 (3.292855) | 2.631262 / 55.444624 (-52.813363) | 2.262143 / 6.876477 (-4.614333) | 2.364243 / 2.142072 (0.222170) | 0.660621 / 4.805227 (-4.144607) | 0.137371 / 6.500664 (-6.363293) | 0.069104 / 0.075469 (-0.006365) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.305706 / 1.841788 (-0.536081) | 14.015932 / 8.074308 (5.941624) | 14.353580 / 10.191392 (4.162187) | 0.146172 / 0.680424 (-0.534251) | 0.016699 / 0.534201 (-0.517502) | 0.357970 / 0.579283 (-0.221313) | 0.389067 / 0.434364 (-0.045297) | 0.415470 / 0.540337 (-0.124867) | 0.501359 / 1.386936 (-0.885577) |

![](https://cml.dev/watermark.png#b2b837b4e7267db9e32d2613d8bf8d70d2ce0b47 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006800 / 0.011353 (-0.004552) | 0.004721 / 0.011008 (-0.006287) | 0.097760 / 0.038508 (0.059252) | 0.034192 / 0.023109 (0.011083) | 0.298240 / 0.275898 (0.022342) | 0.331119 / 0.323480 (0.007639) | 0.005826 / 0.007986 (-0.002160) | 0.003968 / 0.004328 (-0.000360) | 0.073833 / 0.004250 (0.069582) | 0.046288 / 0.037052 (0.009236) | 0.303018 / 0.258489 (0.044529) | 0.342163 / 0.293841 (0.048322) | 0.028504 / 0.128546 (-0.100042) | 0.009031 / 0.075646 (-0.066615) | 0.331617 / 0.419271 (-0.087655) | 0.060911 / 0.043533 (0.017379) | 0.304044 / 0.255139 (0.048905) | 0.328959 / 0.283200 (0.045759) | 0.113174 / 0.141683 (-0.028509) | 1.424652 / 1.452155 (-0.027502) | 1.531392 / 1.492716 (0.038676) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.206175 / 0.018006 (0.188169) | 0.435916 / 0.000490 (0.435426) | 0.002587 / 0.000200 (0.002387) | 0.000083 / 0.000054 (0.000029) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026996 / 0.037411 (-0.010415) | 0.106722 / 0.014526 (0.092196) | 0.117655 / 0.176557 (-0.058902) | 0.176969 / 0.737135 (-0.560166) | 0.122577 / 0.296338 (-0.173762) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.396086 / 0.215209 (0.180877) | 3.972465 / 2.077655 (1.894811) | 1.800798 / 1.504120 (0.296678) | 1.616747 / 1.541195 (0.075552) | 1.680711 / 1.468490 (0.212221) | 0.526479 / 4.584777 (-4.058298) | 3.791528 / 3.745712 (0.045816) | 2.989518 / 5.269862 (-2.280344) | 1.463221 / 4.565676 (-3.102455) | 0.065649 / 0.424275 (-0.358626) | 0.012155 / 0.007607 (0.004548) | 0.500241 / 0.226044 (0.274197) | 5.008895 / 2.268929 (2.739966) | 2.315288 / 55.444624 (-53.129336) | 1.959409 / 6.876477 (-4.917067) | 2.102371 / 2.142072 (-0.039701) | 0.639611 / 4.805227 (-4.165617) | 0.140101 / 6.500664 (-6.360563) | 0.063599 / 0.075469 (-0.011870) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.206729 / 1.841788 (-0.635059) | 15.127250 / 8.074308 (7.052942) | 14.397228 / 10.191392 (4.205836) | 0.148802 / 0.680424 (-0.531622) | 0.017628 / 0.534201 (-0.516573) | 0.396150 / 0.579283 (-0.183133) | 0.435826 / 0.434364 (0.001462) | 0.471215 / 0.540337 (-0.069122) | 0.559413 / 1.386936 (-0.827523) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006479 / 0.011353 (-0.004874) | 0.004520 / 0.011008 (-0.006488) | 0.074395 / 0.038508 (0.035887) | 0.033400 / 0.023109 (0.010291) | 0.388411 / 0.275898 (0.112513) | 0.396714 / 0.323480 (0.073234) | 0.005736 / 0.007986 (-0.002250) | 0.004038 / 0.004328 (-0.000291) | 0.073595 / 0.004250 (0.069345) | 0.045207 / 0.037052 (0.008155) | 0.378096 / 0.258489 (0.119607) | 0.417830 / 0.293841 (0.123989) | 0.028365 / 0.128546 (-0.100181) | 0.008887 / 0.075646 (-0.066760) | 0.080766 / 0.419271 (-0.338505) | 0.046923 / 0.043533 (0.003390) | 0.376190 / 0.255139 (0.121051) | 0.385875 / 0.283200 (0.102675) | 0.107542 / 0.141683 (-0.034141) | 1.409257 / 1.452155 (-0.042898) | 1.518475 / 1.492716 (0.025759) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.223299 / 0.018006 (0.205292) | 0.440640 / 0.000490 (0.440150) | 0.000397 / 0.000200 (0.000197) | 0.000056 / 0.000054 (0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031388 / 0.037411 (-0.006024) | 0.113078 / 0.014526 (0.098552) | 0.124398 / 0.176557 (-0.052159) | 0.173802 / 0.737135 (-0.563333) | 0.129555 / 0.296338 (-0.166783) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.440220 / 0.215209 (0.225011) | 4.398052 / 2.077655 (2.320398) | 2.188396 / 1.504120 (0.684276) | 1.997811 / 1.541195 (0.456616) | 2.093338 / 1.468490 (0.624847) | 0.519597 / 4.584777 (-4.065180) | 3.885795 / 3.745712 (0.140083) | 2.896327 / 5.269862 (-2.373534) | 1.245785 / 4.565676 (-3.319891) | 0.065675 / 0.424275 (-0.358600) | 0.011729 / 0.007607 (0.004121) | 0.541526 / 0.226044 (0.315482) | 5.406763 / 2.268929 (3.137834) | 2.722914 / 55.444624 (-52.721711) | 2.471111 / 6.876477 (-4.405366) | 2.541488 / 2.142072 (0.399415) | 0.633566 / 4.805227 (-4.171661) | 0.139622 / 6.500664 (-6.361042) | 0.064220 / 0.075469 (-0.011249) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.296097 / 1.841788 (-0.545690) | 15.095320 / 8.074308 (7.021012) | 14.300821 / 10.191392 (4.109429) | 0.145470 / 0.680424 (-0.534954) | 0.017496 / 0.534201 (-0.516705) | 0.400589 / 0.579283 (-0.178694) | 0.423091 / 0.434364 (-0.011273) | 0.468258 / 0.540337 (-0.072079) | 0.570873 / 1.386936 (-0.816063) |

![](https://cml.dev/watermark.png#aee6c67034d6ff298b2153a2fcdab97f14ee6d66 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005918 / 0.011353 (-0.005435) | 0.004393 / 0.011008 (-0.006615) | 0.091677 / 0.038508 (0.053169) | 0.033546 / 0.023109 (0.010437) | 0.344682 / 0.275898 (0.068784) | 0.388906 / 0.323480 (0.065426) | 0.005412 / 0.007986 (-0.002574) | 0.004909 / 0.004328 (0.000580) | 0.082589 / 0.004250 (0.078339) | 0.045242 / 0.037052 (0.008190) | 0.339191 / 0.258489 (0.080702) | 0.349673 / 0.293841 (0.055832) | 0.026805 / 0.128546 (-0.101742) | 0.007529 / 0.075646 (-0.068117) | 0.319108 / 0.419271 (-0.100164) | 0.049482 / 0.043533 (0.005949) | 0.320013 / 0.255139 (0.064874) | 0.342059 / 0.283200 (0.058859) | 0.096623 / 0.141683 (-0.045060) | 1.458204 / 1.452155 (0.006049) | 1.571172 / 1.492716 (0.078455) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.235171 / 0.018006 (0.217165) | 0.479678 / 0.000490 (0.479188) | 0.006627 / 0.000200 (0.006427) | 0.000257 / 0.000054 (0.000202) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025716 / 0.037411 (-0.011696) | 0.107730 / 0.014526 (0.093204) | 0.111595 / 0.176557 (-0.064962) | 0.171316 / 0.737135 (-0.565819) | 0.118962 / 0.296338 (-0.177377) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.376318 / 0.215209 (0.161109) | 4.039484 / 2.077655 (1.961829) | 1.811548 / 1.504120 (0.307428) | 1.646728 / 1.541195 (0.105533) | 1.688071 / 1.468490 (0.219581) | 0.551256 / 4.584777 (-4.033520) | 4.153931 / 3.745712 (0.408218) | 3.424154 / 5.269862 (-1.845707) | 1.734860 / 4.565676 (-2.830816) | 0.067753 / 0.424275 (-0.356522) | 0.012699 / 0.007607 (0.005092) | 0.505722 / 0.226044 (0.279677) | 4.997321 / 2.268929 (2.728392) | 2.258755 / 55.444624 (-53.185869) | 1.954382 / 6.876477 (-4.922095) | 1.967545 / 2.142072 (-0.174527) | 0.630489 / 4.805227 (-4.174738) | 0.138738 / 6.500664 (-6.361926) | 0.064907 / 0.075469 (-0.010562) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.209634 / 1.841788 (-0.632154) | 15.055062 / 8.074308 (6.980754) | 12.721606 / 10.191392 (2.530214) | 0.164908 / 0.680424 (-0.515516) | 0.019528 / 0.534201 (-0.514673) | 0.400136 / 0.579283 (-0.179147) | 0.451640 / 0.434364 (0.017276) | 0.466272 / 0.540337 (-0.074065) | 0.553258 / 1.386936 (-0.833679) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006341 / 0.011353 (-0.005011) | 0.004617 / 0.011008 (-0.006391) | 0.077953 / 0.038508 (0.039445) | 0.031104 / 0.023109 (0.007995) | 0.360328 / 0.275898 (0.084430) | 0.408403 / 0.323480 (0.084923) | 0.005704 / 0.007986 (-0.002282) | 0.003588 / 0.004328 (-0.000741) | 0.071441 / 0.004250 (0.067190) | 0.043520 / 0.037052 (0.006468) | 0.375798 / 0.258489 (0.117309) | 0.400955 / 0.293841 (0.107114) | 0.028166 / 0.128546 (-0.100381) | 0.008578 / 0.075646 (-0.067068) | 0.086673 / 0.419271 (-0.332598) | 0.046424 / 0.043533 (0.002891) | 0.367276 / 0.255139 (0.112137) | 0.414550 / 0.283200 (0.131351) | 0.097355 / 0.141683 (-0.044328) | 1.465191 / 1.452155 (0.013036) | 1.555028 / 1.492716 (0.062312) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.196642 / 0.018006 (0.178636) | 0.464221 / 0.000490 (0.463731) | 0.002726 / 0.000200 (0.002526) | 0.000110 / 0.000054 (0.000055) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028078 / 0.037411 (-0.009333) | 0.110762 / 0.014526 (0.096236) | 0.122212 / 0.176557 (-0.054344) | 0.164758 / 0.737135 (-0.572377) | 0.133969 / 0.296338 (-0.162370) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.448134 / 0.215209 (0.232925) | 4.339335 / 2.077655 (2.261680) | 2.129209 / 1.504120 (0.625089) | 1.957805 / 1.541195 (0.416611) | 1.994038 / 1.468490 (0.525548) | 0.497101 / 4.584777 (-4.087676) | 4.114432 / 3.745712 (0.368720) | 3.437305 / 5.269862 (-1.832556) | 1.692810 / 4.565676 (-2.872866) | 0.071077 / 0.424275 (-0.353198) | 0.012735 / 0.007607 (0.005128) | 0.534393 / 0.226044 (0.308348) | 5.217445 / 2.268929 (2.948517) | 2.594858 / 55.444624 (-52.849766) | 2.317464 / 6.876477 (-4.559012) | 2.337974 / 2.142072 (0.195902) | 0.622291 / 4.805227 (-4.182936) | 0.144934 / 6.500664 (-6.355730) | 0.068524 / 0.075469 (-0.006945) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.310601 / 1.841788 (-0.531187) | 15.771527 / 8.074308 (7.697219) | 13.952032 / 10.191392 (3.760640) | 0.212473 / 0.680424 (-0.467951) | 0.017963 / 0.534201 (-0.516238) | 0.400755 / 0.579283 (-0.178528) | 0.439817 / 0.434364 (0.005453) | 0.472614 / 0.540337 (-0.067724) | 0.558410 / 1.386936 (-0.828526) |

![](https://cml.dev/watermark.png#1b51429d02a0da1ff798873afe655309136c5689 "CML watermark")

Raise an error in  `DatasetBuilder.as_dataset` when `file_format != "arrow"` (and fix the docstring)

Fix #5874


array is too big; `arr.size * arr.dtype.itemsize` is larger than the maximum possible size in Datasets

### Describe the bug

When using the `filter` or `map` function to preprocess a dataset, a ValueError is encountered with the error message "array is too big; arr.size * arr.dtype.itemsize is larger than the maximum possible size."

Detailed error message:
Traceback (most recent call last):
  File "data_processing.py", line 26, in <module>
    processed_dataset[split] = samromur_children[split].map(prepare_dataset, cache_file_name=cache_dict[split],writer_batch_size = 50)
  File "/projects/zhwa3087/software/anaconda/envs/mycustomenv/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 2405, in map
    desc=desc,
  File "/projects/zhwa3087/software/anaconda/envs/mycustomenv/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 557, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/projects/zhwa3087/software/anaconda/envs/mycustomenv/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 524, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/projects/zhwa3087/software/anaconda/envs/mycustomenv/lib/python3.7/site-packages/datasets/fingerprint.py", line 480, in wrapper
    out = func(self, *args, **kwargs)
  File "/projects/zhwa3087/software/anaconda/envs/mycustomenv/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 2756, in _map_single
    example = apply_function_on_filtered_inputs(example, i, offset=offset)
  File "/projects/zhwa3087/software/anaconda/envs/mycustomenv/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 2655, in apply_function_on_filtered_inputs
    processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
  File "/projects/zhwa3087/software/anaconda/envs/mycustomenv/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 2347, in decorated
    result = f(decorated_item, *args, **kwargs)
  File "data_processing.py", line 11, in prepare_dataset
    audio = batch["audio"]
  File "/projects/zhwa3087/software/anaconda/envs/mycustomenv/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 123, in __getitem__
    value = decode_nested_example(self.features[key], value) if value is not None else None
  File "/projects/zhwa3087/software/anaconda/envs/mycustomenv/lib/python3.7/site-packages/datasets/features/features.py", line 1260, in decode_nested_example
    return schema.decode_example(obj, token_per_repo_id=token_per_repo_id) if obj is not None else None
  File "/projects/zhwa3087/software/anaconda/envs/mycustomenv/lib/python3.7/site-packages/datasets/features/audio.py", line 156, in decode_example
    array, sampling_rate = self._decode_non_mp3_path_like(path, token_per_repo_id=token_per_repo_id)
  File "/projects/zhwa3087/software/anaconda/envs/mycustomenv/lib/python3.7/site-packages/datasets/features/audio.py", line 257, in _decode_non_mp3_path_like
    array, sampling_rate = librosa.load(f, sr=self.sampling_rate, mono=self.mono)
  File "/projects/zhwa3087/software/anaconda/envs/mycustomenv/lib/python3.7/site-packages/librosa/core/audio.py", line 176, in load
    y, sr_native = __soundfile_load(path, offset, duration, dtype)
  File "/projects/zhwa3087/software/anaconda/envs/mycustomenv/lib/python3.7/site-packages/librosa/core/audio.py", line 222, in __soundfile_load
    y = sf_desc.read(frames=frame_duration, dtype=dtype, always_2d=False).T
  File "/projects/zhwa3087/software/anaconda/envs/mycustomenv/lib/python3.7/site-packages/soundfile.py", line 891, in read
    out = self._create_empty_array(frames, always_2d, dtype)
  File "/projects/zhwa3087/software/anaconda/envs/mycustomenv/lib/python3.7/site-packages/soundfile.py", line 1323, in _create_empty_array
    return np.empty(shape, dtype, order='C')
ValueError: array is too big; `arr.size * arr.dtype.itemsize` is larger than the maximum possible size.

### Steps to reproduce the bug

```python
from datasets import load_dataset, DatasetDict
from transformers import WhisperFeatureExtractor
from transformers import WhisperTokenizer

samromur_children= load_dataset("language-and-voice-lab/samromur_children")
feature_extractor = WhisperFeatureExtractor.from_pretrained("openai/whisper-small")
tokenizer = WhisperTokenizer.from_pretrained("openai/whisper-small", language="icelandic", task="transcribe")

def prepare_dataset(batch):
    # load and resample audio data from 48 to 16kHz
    audio = batch["audio"]

    # compute log-Mel input features from input audio array
    batch["input_features"] = feature_extractor(audio["array"], sampling_rate=16000).input_features[0]

    # encode target text to label ids
    batch["labels"] = tokenizer(batch["normalized_text"]).input_ids
    return batch

cache_dict = {"train": "./cache/audio_train.cache", \
              "validation": "./cache/audio_validation.cache", \
              "test": "./cache/audio_test.cache"}
filter_cache_dict = {"train": "./cache/filter_train.arrow", \
                "validation": "./cache/filter_validation.arrow", \
                "test": "./cache/filter_test.arrow"}

print("before filtering")
print(samromur_children)
#filter the dataset to only include examples with more than 2 seconds of audio
samromur_children = samromur_children.filter(lambda example: example["audio"]["array"].shape[0] > 16000*2, cache_file_names=filter_cache_dict)
print("after filtering")
print(samromur_children)
processed_dataset = DatasetDict()
# processed_dataset = samromur_children.map(prepare_dataset, cache_file_names=cache_dict, num_proc=10,)
for split in ["train", "validation", "test"]:
    processed_dataset[split] = samromur_children[split].map(prepare_dataset, cache_file_name=cache_dict[split])
```

### Expected behavior

The dataset is successfully processed and ready to train the model.

### Environment info

Python version:  3.7.13
datasets package version: 2.4.0
librosa package version: 0.10.0.post2


I tried to load a custom dataset using the following statement: dataset = load_dataset('json', data_files=data_files). The dataset contains 50 million text-image pairs, but an error occurred.

Thanks for reporting, @cjt222.

What is the structure of your JSON files. Please note that it is normally simpler if the data file format is JSON-Lines instead.

> Thanks for reporting, @cjt222.
>
> What is the structure of your JSON files. Please note that it is normally simpler if the data file format is JSON-Lines instead.

Thanks! I have encountered similar problems. I modify the json format from list to line and works!

### Describe the bug

File "/home/kas/.conda/envs/diffusers/lib/python3.7/site-packages/datasets/builder.py", line 1858, in _prepare_split_single
Downloading and preparing dataset json/default to /home/kas/diffusers/examples/dreambooth/cache_data/datasets/json/default-acf423d8c6ef99d0/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s] Downloading data files: 100%|██████████| 1/1 [00:00<00:00, 84.35it/s]
Extracting data files: 0%| | 0/1 [00:00<?, ?it/s] for _, table in generator:
File "/home/kas/.conda/envs/diffusers/lib/python3.7/site-packages/datasets/packaged_modules/json/json.py", line 114, in _generate_tables
io.BytesIO(batch), read_options=paj.ReadOptions(block_size=block_size)
File "pyarrow/_json.pyx", line 258, in pyarrow._json.read_json
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 27.72it/s]
Generating train split: 0 examples [00:00, ? examples/s] File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 125, in pyarrow.lib.check_status
pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2390448764

### Steps to reproduce the bug

1、data_files = ["1.json", "2.json", "3.json"]
2、dataset = load_dataset('json', data_files=data_files)

### Expected behavior

Read the dataset normally.

### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-4.15.0-29-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.16
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 1.3.5


Missing elements in `map` a batched dataset

Hi ! in your code batching is **only used within** `map`, to process examples in batch. The dataset itself however is not batched and returns elements one by one.

To iterate on batches, you can do
```python
for batch in dataset.iter(batch_size=8):
    ...
```

### Describe the bug

As outlined [here](https://discuss.huggingface.co/t/length-error-using-map-with-datasets/40969/3?u=sachin), the following collate function drops 5 out of possible 6 elements in the batch (it is 6 because out of the eight, two are bad links in laion). A reproducible [kaggle kernel ](https://www.kaggle.com/sachin/laion-hf-dataset/edit) can be found here.

The weirdest part is when inspecting the sizes of the tensors as shown below, both `tokenized_captions["input_ids"]` and `image_features` show the correct shapes. Simply the output only has one element (with the batch dimension squeezed out).

```python
class CollateFn:
    def get_image(self, url):
        try:
            response = requests.get(url)
            return Image.open(io.BytesIO(response.content)).convert("RGB")
        except PIL.UnidentifiedImageError:
            logger.info(f"Reading error: Could not transform f{url}")
            return None
        except requests.exceptions.ConnectionError:
            logger.info(f"Connection error: Could not transform f{url}")
            return None

    def __call__(self, batch):
        images = [self.get_image(url) for url in batch["url"]]
        captions = [caption for caption, image in zip(batch["caption"], images) if image is not None]
        images = [image for image in images if image is not None]

        tokenized_captions = tokenizer(
            captions,
            padding="max_length",
            truncation=True,
            max_length=tokenizer.model_max_length,
            return_tensors="pt",
        )

        image_features = torch.stack([torch.Tensor(feature_extractor(image)["pixel_values"][0]) for image in images])
#         import pdb; pdb.set_trace()
        return {"input_ids": tokenized_captions["input_ids"], "images": image_features}

collate_fn = CollateFn()
laion_ds = datasets.load_dataset("laion/laion400m", split="train", streaming=True)
laion_ds_batched = laion_ds.map(collate_fn, batched=True, batch_size=8, remove_columns=next(iter(laion_ds)).keys())
```

### Steps to reproduce the bug

A reproducible [kaggle kernel ](https://www.kaggle.com/sachin/laion-hf-dataset/edit) can be found here.

### Expected behavior

Would expect `next(iter(laion_ds_batched))` to produce two tensors of shape `(batch_size, 77)` and `batch_size, image_shape`.

### Environment info

datasets==2.12.0
python==3.10


Cannot use both set_format and set_transform

Currently, it's not possible to chain `set_format`/`set_transform` calls (plus, this is a breaking change if we decide to implement it), so I see two possible solutions:
* using `set_format`/`set_transform` for the 1st transform and then passing the transformed example/batch to the 2nd transform
* implementing and registering a custom formatter (the relevant code is [here](https://github.com/huggingface/datasets/tree/main/src/datasets/formatting))

Btw, your example requires a single `set_format` call:
```python
ds.set_format("torch", columns=["image"], output_all_columns=True, dtype=torch.double)
```

Hey Mario,
Thanks, for getting back to me. the toDouble was just an example my real life case requires many more transforms.

What do you mean by:
> using set_format/set_transform for the 1st transform and then passing the transformed example/batch to the 2nd transform

How would that go, I thought you can't chain them?

As for the custom formatter, is it possible to reference an existing formatter, in my case `torch_formatter` inside of my custom formatter?

maybe I can inherit from it and just call `super.recursive_tensorize()`?

> How would that go, I thought you can't chain them?

Yes, they cannot be chained. This is what I meant:
```python
ds.set_transform(first_transform)
# calling the 2nd transform on each accessed batch
second_transform(ds[2:3])
```

> As for the custom formatter, is it possible to reference an existing formatter, in my case torch_formatter inside of my custom formatter?
>
>maybe I can inherit from it and just call super.recursive_tensorize()?

Yes, subclassing makes the most sense.

### Describe the bug

I need to process some data using the set_transform method but I also need the data to be formatted for pytorch before processing it.

I don't see anywhere in the documentation something that says that both methods cannot be used at the same time.

### Steps to reproduce the bug

```
from datasets import load_dataset
ds = load_dataset("mnist", split="train")
ds.set_format(type="torch")
def transform(entry):
    return entry["image"].double()
ds.set_transform(transform)

print(ds[0])
```

### Expected behavior

It should print the pytorch tensor image as a double, but it errors because "entry" in the transform function doesn't receive a pytorch tensor to begin with, it receives a PIL Image -> entry.double() errors because entry isn't a pytorch tensor.

### Environment info
Latest versions.

### Note:
It would be at least handy to have access to a function that can do the dataset.set_format in the set_transform function.

Something like:
```
from datasets import load_dataset, do_format
ds = load_dataset("mnist", split="train")
def transform(entry):
    entry = do_format(entry, type="torch")
    return entry["image"].double()
ds.set_transform(transform)

print(ds[0])
```


Use more efficient and idiomatic way to construct list.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008156 / 0.011353 (-0.003197) | 0.005563 / 0.011008 (-0.005445) | 0.118319 / 0.038508 (0.079810) | 0.044305 / 0.023109 (0.021195) | 0.366221 / 0.275898 (0.090323) | 0.407585 / 0.323480 (0.084105) | 0.006961 / 0.007986 (-0.001024) | 0.004841 / 0.004328 (0.000513) | 0.089949 / 0.004250 (0.085698) | 0.062197 / 0.037052 (0.025144) | 0.360721 / 0.258489 (0.102232) | 0.415332 / 0.293841 (0.121491) | 0.035709 / 0.128546 (-0.092837) | 0.010617 / 0.075646 (-0.065030) | 0.397454 / 0.419271 (-0.021817) | 0.063490 / 0.043533 (0.019958) | 0.374289 / 0.255139 (0.119150) | 0.382827 / 0.283200 (0.099628) | 0.121014 / 0.141683 (-0.020669) | 1.729933 / 1.452155 (0.277779) | 1.896222 / 1.492716 (0.403506) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.254030 / 0.018006 (0.236023) | 0.491225 / 0.000490 (0.490736) | 0.018933 / 0.000200 (0.018734) | 0.000413 / 0.000054 (0.000358) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033085 / 0.037411 (-0.004327) | 0.132837 / 0.014526 (0.118311) | 0.143275 / 0.176557 (-0.033282) | 0.215800 / 0.737135 (-0.521335) | 0.149802 / 0.296338 (-0.146536) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.474688 / 0.215209 (0.259479) | 4.743223 / 2.077655 (2.665569) | 2.163107 / 1.504120 (0.658988) | 1.946396 / 1.541195 (0.405201) | 2.057538 / 1.468490 (0.589047) | 0.618836 / 4.584777 (-3.965941) | 4.605934 / 3.745712 (0.860222) | 2.201537 / 5.269862 (-3.068324) | 1.275758 / 4.565676 (-3.289919) | 0.077782 / 0.424275 (-0.346493) | 0.014830 / 0.007607 (0.007223) | 0.593372 / 0.226044 (0.367328) | 5.927000 / 2.268929 (3.658072) | 2.687293 / 55.444624 (-52.757331) | 2.301797 / 6.876477 (-4.574679) | 2.489928 / 2.142072 (0.347856) | 0.756779 / 4.805227 (-4.048449) | 0.168065 / 6.500664 (-6.332600) | 0.077276 / 0.075469 (0.001807) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.608169 / 1.841788 (-0.233619) | 19.048790 / 8.074308 (10.974482) | 16.100228 / 10.191392 (5.908836) | 0.215346 / 0.680424 (-0.465077) | 0.022293 / 0.534201 (-0.511907) | 0.535899 / 0.579283 (-0.043384) | 0.533729 / 0.434364 (0.099365) | 0.562697 / 0.540337 (0.022360) | 0.764082 / 1.386936 (-0.622854) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010087 / 0.011353 (-0.001266) | 0.005357 / 0.011008 (-0.005651) | 0.092678 / 0.038508 (0.054170) | 0.041207 / 0.023109 (0.018098) | 0.437464 / 0.275898 (0.161566) | 0.527867 / 0.323480 (0.204387) | 0.006861 / 0.007986 (-0.001125) | 0.006131 / 0.004328 (0.001802) | 0.093741 / 0.004250 (0.089490) | 0.064142 / 0.037052 (0.027090) | 0.433577 / 0.258489 (0.175088) | 0.537148 / 0.293841 (0.243307) | 0.035339 / 0.128546 (-0.093207) | 0.010432 / 0.075646 (-0.065214) | 0.102838 / 0.419271 (-0.316434) | 0.057905 / 0.043533 (0.014372) | 0.437956 / 0.255139 (0.182817) | 0.509562 / 0.283200 (0.226362) | 0.120620 / 0.141683 (-0.021063) | 1.798686 / 1.452155 (0.346531) | 2.013290 / 1.492716 (0.520574) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.249067 / 0.018006 (0.231061) | 0.462219 / 0.000490 (0.461729) | 0.000476 / 0.000200 (0.000276) | 0.000068 / 0.000054 (0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033988 / 0.037411 (-0.003424) | 0.135863 / 0.014526 (0.121337) | 0.144082 / 0.176557 (-0.032474) | 0.201715 / 0.737135 (-0.535421) | 0.152079 / 0.296338 (-0.144259) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.522820 / 0.215209 (0.307611) | 5.216723 / 2.077655 (3.139068) | 2.582355 / 1.504120 (1.078235) | 2.352799 / 1.541195 (0.811604) | 2.451943 / 1.468490 (0.983453) | 0.620381 / 4.584777 (-3.964396) | 4.537841 / 3.745712 (0.792129) | 2.206431 / 5.269862 (-3.063431) | 1.269865 / 4.565676 (-3.295811) | 0.078744 / 0.424275 (-0.345531) | 0.014375 / 0.007607 (0.006768) | 0.648215 / 0.226044 (0.422171) | 6.482809 / 2.268929 (4.213881) | 3.210670 / 55.444624 (-52.233954) | 2.847485 / 6.876477 (-4.028992) | 2.820946 / 2.142072 (0.678873) | 0.762711 / 4.805227 (-4.042516) | 0.171235 / 6.500664 (-6.329429) | 0.080230 / 0.075469 (0.004761) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.646840 / 1.841788 (-0.194948) | 19.400451 / 8.074308 (11.326142) | 16.758845 / 10.191392 (6.567453) | 0.171377 / 0.680424 (-0.509046) | 0.020400 / 0.534201 (-0.513801) | 0.467675 / 0.579283 (-0.111608) | 0.529745 / 0.434364 (0.095381) | 0.605989 / 0.540337 (0.065652) | 0.694659 / 1.386936 (-0.692277) |

![](https://cml.dev/watermark.png#006bf33ac5c308f9c70f4df4868abd539eb6c366 "CML watermark")

It's faster because all the items are the same object, but this also means modifying one of them will alter each unless these items are immutable, and they are in this case (tuples). So we should be careful when using this idiom.

Using `*` is ~2X faster according to [benchmark](https://colab.research.google.com/gist/ttsugriy/c964a2604edf70c41911b10335729b6a/for-vs-mult.ipynb) with just 4 patterns. This doesn't matter much since this tiny difference is not going to be noticeable, but why not?


Unbearably slow sorting on big mapped datasets

Hi ! `shard` currently returns a slow dataset by default, with examples evenly distributed in the dataset.

You can get a fast dataset using `contiguous=True` (which should be the default imo):

```python
dataset = dataset.shard(10, 0, contiguous=True)
```

This way you don't need to flatten_indices() and sort should be fast as well

@lhoestq

> contiguous=True (which should be the default imo)

For `IterableDataset`, it's not possible to implement contiguous sharding without knowing the number of examples in advance, so setting the default value to `contiguous=True` would result in an inconsistency between `Dataset` and `IterableDataset` (when we add `IterableDataset.shard`)

Actually sharded iterable datasets are made of sub iterables that generally yield contiguous data no ? So in a way it's possible to shard an iterable dataset contiguously.

If the dataset is made of one shard it's indeed not possible to shard it contiguously though

> Actually sharded iterable datasets are made of sub iterables that generally yield contiguous data no ? So in a way it's possible to shard an iterable dataset contiguously.

But sharding an iterable dataset by sharding its `gen_kwargs` would still yield approximate shards(not equal to `Dataset.shard`), no?

I understand the issue doesn't exist with non-mapped datasets, but if flattening is so much more efficient than sorting the indices, that's an issue in itself.

There are plenty of issues people posted for which the root cause turns out to be the same. It seems like mapped datasets are terribly inefficient. I think I saw some issue like that somewhere (about the mapped datasets in general), but can't find it now.

Maybe indices should be flattened before any additional processing, then.

### Describe the bug

For me, with ~40k lines, sorting took 3.5 seconds on a flattened dataset (including the flatten operation) and 22.7 seconds on a mapped dataset (right after sharding), which is about x5 slowdown. Moreover, it seems like it slows down exponentially with bigger datasets (wasn't able to sort 700k lines at all, with flattening takes about a minute).

### Steps to reproduce the bug

```Python
from datasets import load_dataset
import time

dataset = load_dataset("xnli", "en", split="train")

dataset = dataset.shard(10, 0)

print(len(dataset))

t = time.time()

# dataset = dataset.flatten_indices()  # uncomment this line and it's fast

dataset = dataset.sort("label", reverse=True, load_from_cache_file=False)

print(f"finished in {time.time() - t:.4f} seconds")

```

### Expected behavior

Expect sorting to take the same or less time than flattening and then sorting.

### Environment info

- `datasets` version: 2.12.1.dev0 (same with 2.12.0 too)
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.10.10
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1


Add `flatten_indices` to `DatasetDict`

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006192 / 0.011353 (-0.005161) | 0.004410 / 0.011008 (-0.006598) | 0.095990 / 0.038508 (0.057482) | 0.032662 / 0.023109 (0.009553) | 0.322827 / 0.275898 (0.046929) | 0.352542 / 0.323480 (0.029062) | 0.005398 / 0.007986 (-0.002588) | 0.003926 / 0.004328 (-0.000403) | 0.075131 / 0.004250 (0.070880) | 0.046205 / 0.037052 (0.009153) | 0.330957 / 0.258489 (0.072468) | 0.360166 / 0.293841 (0.066325) | 0.027880 / 0.128546 (-0.100666) | 0.008813 / 0.075646 (-0.066833) | 0.327316 / 0.419271 (-0.091955) | 0.050071 / 0.043533 (0.006539) | 0.319939 / 0.255139 (0.064800) | 0.331593 / 0.283200 (0.048393) | 0.096745 / 0.141683 (-0.044938) | 1.445165 / 1.452155 (-0.006990) | 1.515538 / 1.492716 (0.022821) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.209365 / 0.018006 (0.191358) | 0.437007 / 0.000490 (0.436518) | 0.003207 / 0.000200 (0.003007) | 0.000088 / 0.000054 (0.000033) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027261 / 0.037411 (-0.010151) | 0.105101 / 0.014526 (0.090575) | 0.117163 / 0.176557 (-0.059394) | 0.176237 / 0.737135 (-0.560898) | 0.122559 / 0.296338 (-0.173779) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.406792 / 0.215209 (0.191583) | 4.060831 / 2.077655 (1.983176) | 1.829691 / 1.504120 (0.325571) | 1.633155 / 1.541195 (0.091960) | 1.704817 / 1.468490 (0.236327) | 0.525325 / 4.584777 (-4.059452) | 3.752907 / 3.745712 (0.007194) | 1.857513 / 5.269862 (-3.412349) | 1.222237 / 4.565676 (-3.343439) | 0.065941 / 0.424275 (-0.358334) | 0.012498 / 0.007607 (0.004891) | 0.495009 / 0.226044 (0.268965) | 4.968074 / 2.268929 (2.699145) | 2.277898 / 55.444624 (-53.166727) | 1.936656 / 6.876477 (-4.939821) | 1.970698 / 2.142072 (-0.171374) | 0.635221 / 4.805227 (-4.170006) | 0.140539 / 6.500664 (-6.360125) | 0.064111 / 0.075469 (-0.011358) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.238151 / 1.841788 (-0.603637) | 14.681262 / 8.074308 (6.606954) | 13.405525 / 10.191392 (3.214133) | 0.163225 / 0.680424 (-0.517199) | 0.017282 / 0.534201 (-0.516918) | 0.395526 / 0.579283 (-0.183757) | 0.429156 / 0.434364 (-0.005208) | 0.470806 / 0.540337 (-0.069531) | 0.571290 / 1.386936 (-0.815646) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006444 / 0.011353 (-0.004909) | 0.004388 / 0.011008 (-0.006621) | 0.075004 / 0.038508 (0.036496) | 0.032904 / 0.023109 (0.009795) | 0.375360 / 0.275898 (0.099462) | 0.413684 / 0.323480 (0.090204) | 0.005854 / 0.007986 (-0.002132) | 0.005504 / 0.004328 (0.001175) | 0.075049 / 0.004250 (0.070799) | 0.047973 / 0.037052 (0.010920) | 0.377943 / 0.258489 (0.119454) | 0.427039 / 0.293841 (0.133198) | 0.028248 / 0.128546 (-0.100298) | 0.008972 / 0.075646 (-0.066674) | 0.081848 / 0.419271 (-0.337424) | 0.047935 / 0.043533 (0.004402) | 0.377980 / 0.255139 (0.122841) | 0.407856 / 0.283200 (0.124656) | 0.103454 / 0.141683 (-0.038229) | 1.469051 / 1.452155 (0.016896) | 1.590657 / 1.492716 (0.097941) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.192380 / 0.018006 (0.174374) | 0.440995 / 0.000490 (0.440505) | 0.004082 / 0.000200 (0.003882) | 0.000096 / 0.000054 (0.000042) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029584 / 0.037411 (-0.007828) | 0.110051 / 0.014526 (0.095525) | 0.121196 / 0.176557 (-0.055361) | 0.172249 / 0.737135 (-0.564886) | 0.125380 / 0.296338 (-0.170958) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.435218 / 0.215209 (0.220009) | 4.354811 / 2.077655 (2.277156) | 2.102050 / 1.504120 (0.597930) | 1.913454 / 1.541195 (0.372260) | 1.974624 / 1.468490 (0.506134) | 0.529975 / 4.584777 (-4.054802) | 3.801605 / 3.745712 (0.055893) | 3.162408 / 5.269862 (-2.107454) | 1.599576 / 4.565676 (-2.966101) | 0.066710 / 0.424275 (-0.357565) | 0.012158 / 0.007607 (0.004551) | 0.549187 / 0.226044 (0.323142) | 5.489930 / 2.268929 (3.221002) | 2.646787 / 55.444624 (-52.797837) | 2.311915 / 6.876477 (-4.564562) | 2.335645 / 2.142072 (0.193572) | 0.641067 / 4.805227 (-4.164160) | 0.142227 / 6.500664 (-6.358437) | 0.065303 / 0.075469 (-0.010166) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.283209 / 1.841788 (-0.558579) | 15.241809 / 8.074308 (7.167501) | 14.131471 / 10.191392 (3.940079) | 0.143921 / 0.680424 (-0.536503) | 0.017497 / 0.534201 (-0.516704) | 0.402236 / 0.579283 (-0.177047) | 0.418917 / 0.434364 (-0.015447) | 0.461745 / 0.540337 (-0.078593) | 0.560212 / 1.386936 (-0.826724) |

![](https://cml.dev/watermark.png#7098922130cabfbfa6b8a3885ff2e6f032d6203d "CML watermark")

Add `flatten_indices` to `DatasetDict` for convinience


Could you unpin responses version?

### Describe the bug

Could you unpin [this](https://github.com/huggingface/datasets/blob/main/setup.py#L139) or move it to test requirements? This is a testing library and we also use it for our tests as well. We do not want to use a very outdated version.

### Steps to reproduce the bug

could not install this library due to dependency conflict.

### Expected behavior

can install datasets

### Environment info

linux 64


Offer an alternative to Iterable Dataset that allows lazy loading and processing while skipping batches efficiently

enhancement

New feature or request

We plan to improve this eventually (see https://github.com/huggingface/datasets/issues/5454 and https://github.com/huggingface/datasets/issues/5380).

> Is it possible to lazily load samples of a mapped dataset ? I'm used to [dataset scripts](https://huggingface.co/docs/datasets/dataset_script), maybe something can be done there.
If not, I could do it using a plain Pytorch dataset. Then I would need to convert it to a datasets' dataset to get all the features of datasets. Is it something possible ?

Yes, by creating a mapped dataset that stores audio URLs. Indexing a dataset in such format only downloads and decodes the bytes of the accessed samples (without storing them on disk).

You can do the following to create this dataset:
```python

def gen():
    # Generator that yields (audio URL, text) pairs as dict
    ...
    yield {"audio": "audio_url", "text": "some text"}

features = Features({"audio": datasets.Audio(), "text": datasets.Value("string")})
ds = Dataset.from_generator(gen, features=features)
ds[2:5] # downloads and decodes the samples each time they are accessed
```

### Feature request

I would like a way to resume training from a checkpoint without waiting for a very long time when using an iterable dataset.

### Motivation

I am training models on the speech-recognition task. I have very large datasets that I can't comfortably store on a disk and also quite computationally intensive audio processing to do. As a result I want to load data from my remote when it is needed and perform all processing on the fly.

I am currently using the iterable dataset feature of _datasets_. It does everything I need with one exception. My issue is that when resuming training at a step n, we have to download all the data and perform the processing of steps < n, just to get the iterable at the right step. In my case it takes almost as long as training for the same steps, which make resuming training from a checkpoint useless in practice.

I understand that the nature of iterators make it probably nearly impossible to quickly resume training.

I thought about a possible solution nonetheless :

I could in fact index my large dataset and make it a mapped dataset. Then I could use set_transform to perform the processing on the fly. Finally, if I'm not mistaken, the _accelerate_ package allows to [skip steps efficiently](https://github.com/huggingface/accelerate/blob/a73898027a211c3f6dc4460351b0ec246aa824aa/src/accelerate/data_loader.py#L827) for a mapped dataset.

Is it possible to lazily load samples of a mapped dataset ? I'm used to [dataset scripts](https://huggingface.co/docs/datasets/dataset_script), maybe something can be done there.
If not, I could do it using a plain _Pytorch_ dataset. Then I would need to convert it to a _datasets_' dataset to get all the features of _datasets_. Is it something possible ?

### Your contribution

I could provide a PR to allow lazy loading of mapped dataset or the conversion of a mapped _Pytorch_ dataset into a _Datasets_ dataset if you think it is an useful new feature.