Fix cast from fixed size list to variable size list

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006784 / 0.011353 (-0.004569) | 0.004051 / 0.011008 (-0.006957) | 0.083790 / 0.038508 (0.045282) | 0.081219 / 0.023109 (0.058110) | 0.313195 / 0.275898 (0.037297) | 0.336954 / 0.323480 (0.013475) | 0.004324 / 0.007986 (-0.003662) | 0.004516 / 0.004328 (0.000188) | 0.065051 / 0.004250 (0.060801) | 0.057647 / 0.037052 (0.020595) | 0.316675 / 0.258489 (0.058186) | 0.357936 / 0.293841 (0.064095) | 0.030980 / 0.128546 (-0.097566) | 0.008844 / 0.075646 (-0.066802) | 0.287027 / 0.419271 (-0.132245) | 0.052130 / 0.043533 (0.008597) | 0.308125 / 0.255139 (0.052986) | 0.337345 / 0.283200 (0.054145) | 0.025781 / 0.141683 (-0.115902) | 1.466161 / 1.452155 (0.014006) | 1.565824 / 1.492716 (0.073108) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.299112 / 0.018006 (0.281106) | 0.640520 / 0.000490 (0.640030) | 0.008846 / 0.000200 (0.008647) | 0.000273 / 0.000054 (0.000219) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029853 / 0.037411 (-0.007559) | 0.081697 / 0.014526 (0.067172) | 0.099110 / 0.176557 (-0.077447) | 0.155864 / 0.737135 (-0.581271) | 0.098749 / 0.296338 (-0.197590) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.385722 / 0.215209 (0.170512) | 3.851490 / 2.077655 (1.773835) | 1.851995 / 1.504120 (0.347875) | 1.660398 / 1.541195 (0.119204) | 1.769370 / 1.468490 (0.300879) | 0.481523 / 4.584777 (-4.103254) | 3.550449 / 3.745712 (-0.195263) | 3.424782 / 5.269862 (-1.845079) | 2.106470 / 4.565676 (-2.459206) | 0.056500 / 0.424275 (-0.367775) | 0.007891 / 0.007607 (0.000284) | 0.465564 / 0.226044 (0.239520) | 4.662892 / 2.268929 (2.393964) | 2.305424 / 55.444624 (-53.139201) | 1.980524 / 6.876477 (-4.895953) | 2.218423 / 2.142072 (0.076350) | 0.584662 / 4.805227 (-4.220565) | 0.132325 / 6.500664 (-6.368340) | 0.060773 / 0.075469 (-0.014696) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.254261 / 1.841788 (-0.587527) | 19.479805 / 8.074308 (11.405497) | 14.222687 / 10.191392 (4.031295) | 0.149829 / 0.680424 (-0.530595) | 0.018630 / 0.534201 (-0.515571) | 0.395284 / 0.579283 (-0.183999) | 0.413385 / 0.434364 (-0.020978) | 0.462931 / 0.540337 (-0.077406) | 0.645359 / 1.386936 (-0.741577) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006991 / 0.011353 (-0.004362) | 0.004306 / 0.011008 (-0.006702) | 0.065213 / 0.038508 (0.026705) | 0.082442 / 0.023109 (0.059332) | 0.411294 / 0.275898 (0.135396) | 0.452176 / 0.323480 (0.128696) | 0.005802 / 0.007986 (-0.002183) | 0.003556 / 0.004328 (-0.000772) | 0.066163 / 0.004250 (0.061913) | 0.060680 / 0.037052 (0.023628) | 0.416975 / 0.258489 (0.158486) | 0.456353 / 0.293841 (0.162512) | 0.033584 / 0.128546 (-0.094963) | 0.008687 / 0.075646 (-0.066959) | 0.071300 / 0.419271 (-0.347972) | 0.049382 / 0.043533 (0.005849) | 0.409329 / 0.255139 (0.154190) | 0.434829 / 0.283200 (0.151629) | 0.022966 / 0.141683 (-0.118716) | 1.493847 / 1.452155 (0.041692) | 1.582372 / 1.492716 (0.089656) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.280578 / 0.018006 (0.262572) | 0.538122 / 0.000490 (0.537632) | 0.004515 / 0.000200 (0.004315) | 0.000098 / 0.000054 (0.000043) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033383 / 0.037411 (-0.004028) | 0.093426 / 0.014526 (0.078901) | 0.109314 / 0.176557 (-0.067242) | 0.162349 / 0.737135 (-0.574786) | 0.109849 / 0.296338 (-0.186490) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.431073 / 0.215209 (0.215864) | 4.311942 / 2.077655 (2.234287) | 2.291170 / 1.504120 (0.787051) | 2.132266 / 1.541195 (0.591072) | 2.236526 / 1.468490 (0.768036) | 0.492001 / 4.584777 (-4.092776) | 3.523013 / 3.745712 (-0.222699) | 3.413481 / 5.269862 (-1.856381) | 2.112979 / 4.565676 (-2.452698) | 0.058654 / 0.424275 (-0.365621) | 0.007729 / 0.007607 (0.000121) | 0.512027 / 0.226044 (0.285982) | 5.125264 / 2.268929 (2.856336) | 2.836281 / 55.444624 (-52.608344) | 2.447253 / 6.876477 (-4.429224) | 2.711908 / 2.142072 (0.569835) | 0.592598 / 4.805227 (-4.212629) | 0.134837 / 6.500664 (-6.365827) | 0.059813 / 0.075469 (-0.015656) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.373464 / 1.841788 (-0.468323) | 20.548983 / 8.074308 (12.474675) | 14.799833 / 10.191392 (4.608441) | 0.168601 / 0.680424 (-0.511823) | 0.020358 / 0.534201 (-0.513843) | 0.398790 / 0.579283 (-0.180494) | 0.416921 / 0.434364 (-0.017443) | 0.480542 / 0.540337 (-0.059795) | 0.645062 / 1.386936 (-0.741874) |

![](https://cml.dev/watermark.png#afd6fc193a91cb0461c8bf3b64db6943c23de846 "CML watermark")

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008616 / 0.011353 (-0.002737) | 0.004957 / 0.011008 (-0.006051) | 0.102629 / 0.038508 (0.064121) | 0.080492 / 0.023109 (0.057383) | 0.461817 / 0.275898 (0.185919) | 0.487964 / 0.323480 (0.164484) | 0.006336 / 0.007986 (-0.001649) | 0.004607 / 0.004328 (0.000278) | 0.074311 / 0.004250 (0.070061) | 0.060368 / 0.037052 (0.023315) | 0.458076 / 0.258489 (0.199587) | 0.493028 / 0.293841 (0.199187) | 0.044153 / 0.128546 (-0.084394) | 0.014066 / 0.075646 (-0.061581) | 0.369848 / 0.419271 (-0.049424) | 0.061690 / 0.043533 (0.018157) | 0.439728 / 0.255139 (0.184590) | 0.484706 / 0.283200 (0.201506) | 0.034657 / 0.141683 (-0.107026) | 1.710591 / 1.452155 (0.258437) | 1.900225 / 1.492716 (0.407509) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.308837 / 0.018006 (0.290831) | 0.579561 / 0.000490 (0.579072) | 0.010163 / 0.000200 (0.009963) | 0.000613 / 0.000054 (0.000558) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028108 / 0.037411 (-0.009303) | 0.085072 / 0.014526 (0.070546) | 0.103375 / 0.176557 (-0.073182) | 0.173765 / 0.737135 (-0.563371) | 0.102460 / 0.296338 (-0.193879) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.602642 / 0.215209 (0.387433) | 5.582537 / 2.077655 (3.504882) | 2.405553 / 1.504120 (0.901434) | 2.057298 / 1.541195 (0.516103) | 2.223787 / 1.468490 (0.755297) | 0.846138 / 4.584777 (-3.738638) | 5.290306 / 3.745712 (1.544594) | 4.836066 / 5.269862 (-0.433795) | 2.951901 / 4.565676 (-1.613775) | 0.099432 / 0.424275 (-0.324843) | 0.009198 / 0.007607 (0.001591) | 0.731370 / 0.226044 (0.505325) | 6.663026 / 2.268929 (4.394098) | 3.200932 / 55.444624 (-52.243692) | 2.486654 / 6.876477 (-4.389823) | 2.833195 / 2.142072 (0.691123) | 0.989481 / 4.805227 (-3.815746) | 0.205176 / 6.500664 (-6.295488) | 0.073760 / 0.075469 (-0.001709) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.745494 / 1.841788 (-0.096294) | 24.649294 / 8.074308 (16.574986) | 22.312182 / 10.191392 (12.120790) | 0.245207 / 0.680424 (-0.435217) | 0.031971 / 0.534201 (-0.502230) | 0.495179 / 0.579283 (-0.084104) | 0.603233 / 0.434364 (0.168869) | 0.560906 / 0.540337 (0.020569) | 0.788292 / 1.386936 (-0.598644) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008922 / 0.011353 (-0.002431) | 0.005203 / 0.011008 (-0.005805) | 0.074414 / 0.038508 (0.035906) | 0.077552 / 0.023109 (0.054443) | 0.547217 / 0.275898 (0.271319) | 0.625298 / 0.323480 (0.301818) | 0.006135 / 0.007986 (-0.001851) | 0.004163 / 0.004328 (-0.000165) | 0.078014 / 0.004250 (0.073764) | 0.064484 / 0.037052 (0.027431) | 0.562356 / 0.258489 (0.303867) | 0.643613 / 0.293841 (0.349772) | 0.050155 / 0.128546 (-0.078391) | 0.013665 / 0.075646 (-0.061981) | 0.090224 / 0.419271 (-0.329048) | 0.063852 / 0.043533 (0.020319) | 0.560914 / 0.255139 (0.305775) | 0.591531 / 0.283200 (0.308331) | 0.036491 / 0.141683 (-0.105192) | 1.670898 / 1.452155 (0.218743) | 1.783924 / 1.492716 (0.291208) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.312764 / 0.018006 (0.294758) | 0.611116 / 0.000490 (0.610626) | 0.006367 / 0.000200 (0.006167) | 0.000130 / 0.000054 (0.000075) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033967 / 0.037411 (-0.003445) | 0.101550 / 0.014526 (0.087025) | 0.116953 / 0.176557 (-0.059604) | 0.180061 / 0.737135 (-0.557075) | 0.115220 / 0.296338 (-0.181118) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.642110 / 0.215209 (0.426901) | 6.361381 / 2.077655 (4.283727) | 2.948175 / 1.504120 (1.444055) | 2.633935 / 1.541195 (1.092740) | 2.822150 / 1.468490 (1.353660) | 0.931412 / 4.584777 (-3.653365) | 5.428540 / 3.745712 (1.682828) | 4.672920 / 5.269862 (-0.596941) | 3.102046 / 4.565676 (-1.463630) | 0.100825 / 0.424275 (-0.323450) | 0.009464 / 0.007607 (0.001857) | 0.774102 / 0.226044 (0.548058) | 7.715003 / 2.268929 (5.446074) | 3.987807 / 55.444624 (-51.456817) | 3.089129 / 6.876477 (-3.787347) | 3.333247 / 2.142072 (1.191174) | 1.012427 / 4.805227 (-3.792800) | 0.200662 / 6.500664 (-6.300002) | 0.072422 / 0.075469 (-0.003047) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.680364 / 1.841788 (-0.161424) | 24.484576 / 8.074308 (16.410268) | 21.920990 / 10.191392 (11.729598) | 0.218604 / 0.680424 (-0.461820) | 0.035818 / 0.534201 (-0.498383) | 0.470648 / 0.579283 (-0.108635) | 0.585108 / 0.434364 (0.150744) | 0.539152 / 0.540337 (-0.001185) | 0.763999 / 1.386936 (-0.622937) |

![](https://cml.dev/watermark.png#cfed1d09ed6c680085624d96eb99bfb2b0b27599 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006304 / 0.011353 (-0.005049) | 0.003884 / 0.011008 (-0.007125) | 0.084847 / 0.038508 (0.046339) | 0.069372 / 0.023109 (0.046263) | 0.318876 / 0.275898 (0.042978) | 0.344733 / 0.323480 (0.021253) | 0.005139 / 0.007986 (-0.002847) | 0.003203 / 0.004328 (-0.001125) | 0.065758 / 0.004250 (0.061507) | 0.054189 / 0.037052 (0.017137) | 0.317475 / 0.258489 (0.058986) | 0.359310 / 0.293841 (0.065469) | 0.030639 / 0.128546 (-0.097908) | 0.008657 / 0.075646 (-0.066989) | 0.289127 / 0.419271 (-0.130144) | 0.052344 / 0.043533 (0.008811) | 0.316122 / 0.255139 (0.060983) | 0.338339 / 0.283200 (0.055140) | 0.022677 / 0.141683 (-0.119006) | 1.551629 / 1.452155 (0.099474) | 1.617917 / 1.492716 (0.125201) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.231067 / 0.018006 (0.213061) | 0.450559 / 0.000490 (0.450070) | 0.008484 / 0.000200 (0.008284) | 0.000234 / 0.000054 (0.000179) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027054 / 0.037411 (-0.010357) | 0.081560 / 0.014526 (0.067034) | 0.094162 / 0.176557 (-0.082395) | 0.148583 / 0.737135 (-0.588552) | 0.093596 / 0.296338 (-0.202742) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.388616 / 0.215209 (0.173407) | 3.874905 / 2.077655 (1.797251) | 1.915845 / 1.504120 (0.411725) | 1.746410 / 1.541195 (0.205215) | 1.828789 / 1.468490 (0.360299) | 0.483270 / 4.584777 (-4.101506) | 3.489157 / 3.745712 (-0.256555) | 3.190086 / 5.269862 (-2.079776) | 1.978023 / 4.565676 (-2.587653) | 0.056290 / 0.424275 (-0.367985) | 0.007585 / 0.007607 (-0.000022) | 0.467051 / 0.226044 (0.241007) | 4.665971 / 2.268929 (2.397043) | 2.418550 / 55.444624 (-53.026075) | 2.048338 / 6.876477 (-4.828139) | 2.225275 / 2.142072 (0.083203) | 0.576601 / 4.805227 (-4.228626) | 0.131960 / 6.500664 (-6.368704) | 0.060177 / 0.075469 (-0.015292) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.249797 / 1.841788 (-0.591991) | 18.552939 / 8.074308 (10.478631) | 14.016616 / 10.191392 (3.825224) | 0.162869 / 0.680424 (-0.517555) | 0.018105 / 0.534201 (-0.516096) | 0.394838 / 0.579283 (-0.184445) | 0.403378 / 0.434364 (-0.030986) | 0.460931 / 0.540337 (-0.079407) | 0.637365 / 1.386936 (-0.749571) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006497 / 0.011353 (-0.004856) | 0.003928 / 0.011008 (-0.007080) | 0.063958 / 0.038508 (0.025450) | 0.069609 / 0.023109 (0.046500) | 0.401599 / 0.275898 (0.125701) | 0.428128 / 0.323480 (0.104648) | 0.005296 / 0.007986 (-0.002689) | 0.003332 / 0.004328 (-0.000996) | 0.063903 / 0.004250 (0.059652) | 0.056303 / 0.037052 (0.019250) | 0.400704 / 0.258489 (0.142214) | 0.435982 / 0.293841 (0.142141) | 0.032434 / 0.128546 (-0.096112) | 0.008570 / 0.075646 (-0.067077) | 0.070788 / 0.419271 (-0.348483) | 0.048252 / 0.043533 (0.004719) | 0.403269 / 0.255139 (0.148130) | 0.419796 / 0.283200 (0.136596) | 0.022598 / 0.141683 (-0.119085) | 1.481627 / 1.452155 (0.029472) | 1.578388 / 1.492716 (0.085672) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.224552 / 0.018006 (0.206546) | 0.444059 / 0.000490 (0.443570) | 0.003757 / 0.000200 (0.003557) | 0.000225 / 0.000054 (0.000171) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032173 / 0.037411 (-0.005239) | 0.092562 / 0.014526 (0.078036) | 0.104972 / 0.176557 (-0.071584) | 0.156467 / 0.737135 (-0.580669) | 0.104274 / 0.296338 (-0.192065) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.441693 / 0.215209 (0.226484) | 4.400217 / 2.077655 (2.322562) | 2.393862 / 1.504120 (0.889742) | 2.281178 / 1.541195 (0.739983) | 2.339895 / 1.468490 (0.871405) | 0.488734 / 4.584777 (-4.096043) | 3.523352 / 3.745712 (-0.222360) | 3.216761 / 5.269862 (-2.053101) | 2.007553 / 4.565676 (-2.558123) | 0.058050 / 0.424275 (-0.366225) | 0.007566 / 0.007607 (-0.000041) | 0.515439 / 0.226044 (0.289394) | 5.155086 / 2.268929 (2.886157) | 2.864958 / 55.444624 (-52.579666) | 2.592460 / 6.876477 (-4.284016) | 2.800449 / 2.142072 (0.658376) | 0.588441 / 4.805227 (-4.216786) | 0.131589 / 6.500664 (-6.369075) | 0.059075 / 0.075469 (-0.016394) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.353889 / 1.841788 (-0.487898) | 18.938285 / 8.074308 (10.863977) | 14.937141 / 10.191392 (4.745749) | 0.168811 / 0.680424 (-0.511613) | 0.020118 / 0.534201 (-0.514083) | 0.394791 / 0.579283 (-0.184492) | 0.414434 / 0.434364 (-0.019930) | 0.466821 / 0.540337 (-0.073517) | 0.629894 / 1.386936 (-0.757042) |

![](https://cml.dev/watermark.png#23921b08390db7dbb3186a8de40dc49a4066da76 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005959 / 0.011353 (-0.005394) | 0.004164 / 0.011008 (-0.006844) | 0.082336 / 0.038508 (0.043828) | 0.070344 / 0.023109 (0.047234) | 0.348032 / 0.275898 (0.072134) | 0.366328 / 0.323480 (0.042848) | 0.003882 / 0.007986 (-0.004104) | 0.003619 / 0.004328 (-0.000709) | 0.063343 / 0.004250 (0.059093) | 0.056617 / 0.037052 (0.019564) | 0.351625 / 0.258489 (0.093136) | 0.395839 / 0.293841 (0.101998) | 0.030842 / 0.128546 (-0.097704) | 0.008363 / 0.075646 (-0.067284) | 0.300535 / 0.419271 (-0.118737) | 0.053303 / 0.043533 (0.009770) | 0.354782 / 0.255139 (0.099643) | 0.364918 / 0.283200 (0.081719) | 0.025365 / 0.141683 (-0.116318) | 1.555009 / 1.452155 (0.102854) | 1.597443 / 1.492716 (0.104727) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.239808 / 0.018006 (0.221801) | 0.488164 / 0.000490 (0.487675) | 0.013183 / 0.000200 (0.012983) | 0.000483 / 0.000054 (0.000429) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027938 / 0.037411 (-0.009473) | 0.078521 / 0.014526 (0.063995) | 0.095498 / 0.176557 (-0.081059) | 0.150884 / 0.737135 (-0.586251) | 0.097577 / 0.296338 (-0.198762) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.384546 / 0.215209 (0.169337) | 4.037707 / 2.077655 (1.960053) | 1.940321 / 1.504120 (0.436201) | 1.716741 / 1.541195 (0.175546) | 1.837200 / 1.468490 (0.368710) | 0.502112 / 4.584777 (-4.082665) | 3.770452 / 3.745712 (0.024740) | 3.325691 / 5.269862 (-1.944171) | 2.015622 / 4.565676 (-2.550055) | 0.056246 / 0.424275 (-0.368029) | 0.007320 / 0.007607 (-0.000287) | 0.445553 / 0.226044 (0.219509) | 4.567233 / 2.268929 (2.298304) | 2.319531 / 55.444624 (-53.125093) | 1.968664 / 6.876477 (-4.907813) | 2.122349 / 2.142072 (-0.019724) | 0.573688 / 4.805227 (-4.231540) | 0.131410 / 6.500664 (-6.369254) | 0.062767 / 0.075469 (-0.012702) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.255244 / 1.841788 (-0.586543) | 19.042480 / 8.074308 (10.968172) | 13.935342 / 10.191392 (3.743950) | 0.161259 / 0.680424 (-0.519165) | 0.020582 / 0.534201 (-0.513619) | 0.391365 / 0.579283 (-0.187918) | 0.417462 / 0.434364 (-0.016902) | 0.473121 / 0.540337 (-0.067216) | 0.674768 / 1.386936 (-0.712168) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006299 / 0.011353 (-0.005054) | 0.003969 / 0.011008 (-0.007040) | 0.063558 / 0.038508 (0.025050) | 0.073847 / 0.023109 (0.050738) | 0.407064 / 0.275898 (0.131166) | 0.440695 / 0.323480 (0.117215) | 0.005783 / 0.007986 (-0.002203) | 0.003517 / 0.004328 (-0.000812) | 0.065721 / 0.004250 (0.061470) | 0.056390 / 0.037052 (0.019338) | 0.419019 / 0.258489 (0.160530) | 0.450721 / 0.293841 (0.156880) | 0.034094 / 0.128546 (-0.094452) | 0.008594 / 0.075646 (-0.067052) | 0.069254 / 0.419271 (-0.350017) | 0.049218 / 0.043533 (0.005685) | 0.413312 / 0.255139 (0.158173) | 0.439454 / 0.283200 (0.156255) | 0.021481 / 0.141683 (-0.120202) | 1.517536 / 1.452155 (0.065382) | 1.530532 / 1.492716 (0.037815) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.235392 / 0.018006 (0.217386) | 0.477371 / 0.000490 (0.476881) | 0.007070 / 0.000200 (0.006870) | 0.000132 / 0.000054 (0.000077) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031909 / 0.037411 (-0.005502) | 0.092459 / 0.014526 (0.077933) | 0.105795 / 0.176557 (-0.070761) | 0.157745 / 0.737135 (-0.579390) | 0.104187 / 0.296338 (-0.192152) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.424385 / 0.215209 (0.209176) | 4.445371 / 2.077655 (2.367716) | 2.423639 / 1.504120 (0.919519) | 2.188167 / 1.541195 (0.646972) | 2.171023 / 1.468490 (0.702532) | 0.483566 / 4.584777 (-4.101211) | 3.825702 / 3.745712 (0.079990) | 3.276350 / 5.269862 (-1.993512) | 2.063075 / 4.565676 (-2.502602) | 0.061628 / 0.424275 (-0.362647) | 0.008176 / 0.007607 (0.000569) | 0.506697 / 0.226044 (0.280653) | 5.067924 / 2.268929 (2.798995) | 2.785567 / 55.444624 (-52.659057) | 2.457340 / 6.876477 (-4.419137) | 2.599646 / 2.142072 (0.457574) | 0.581550 / 4.805227 (-4.223677) | 0.131712 / 6.500664 (-6.368952) | 0.058776 / 0.075469 (-0.016693) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.356639 / 1.841788 (-0.485148) | 20.103463 / 8.074308 (12.029155) | 14.481010 / 10.191392 (4.289618) | 0.162870 / 0.680424 (-0.517554) | 0.023197 / 0.534201 (-0.511004) | 0.413042 / 0.579283 (-0.166241) | 0.427494 / 0.434364 (-0.006870) | 0.508457 / 0.540337 (-0.031880) | 0.662412 / 1.386936 (-0.724524) |

![](https://cml.dev/watermark.png#05fe5c06d42f84408b933c2809acb9b7449cbbb3 "CML watermark")

Fix #6242


Data alteration when loading dataset with unspecified inner sequence length

While this issue may seem specific, it led to a silent problem in my workflow that took days to diagnose. If this feature is not intended to be supported, an error should be raised when encountering this configuration to prevent such issues.

Thanks for reporting! This is a MRE:

```python
import pyarrow as pa
from datasets.table import cast_array_to_feature
from datasets import Sequence, Value
data = [
    [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]],
    [[7.0, 8.0, 9.0], [10.0, 11.0, 12.0]],
]
arr = pa.array(data, pa.list_(pa.list_(pa.float32(), 3)))
cast_array_to_feature(arr, Sequence(Sequence(Value("float32"))))
```

I've opened a PR with a fix.

### Describe the bug

When a dataset saved with a specified inner sequence length is loaded without specifying that length, the original data is altered and becomes inconsistent.

### Steps to reproduce the bug

```python
from datasets import Dataset, Features, Value, Sequence, load_dataset

# Repository ID
repo_id = "my_repo_id"

# Define features with a specific length of 3 for each inner sequence
specified_features = Features({"key": Sequence(Sequence(Value("float32"), length=3))})

# Create a dataset with the specified features
data = [
    [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]],
    [[7.0, 8.0, 9.0], [10.0, 11.0, 12.0]],
]
dataset = Dataset.from_dict({"key": data}, features=specified_features)

# Push the dataset to the hub
dataset.push_to_hub(repo_id)

# Define features without specifying the length
unspecified_features = Features({"key": Sequence(Sequence(Value("float32")))})

# Load the dataset from the hub with this new feature definition
dataset = load_dataset(f"qgallouedec/{repo_id}", split="train", features=unspecified_features)

# The obtained data is altered
print(dataset.to_dict())  # {'key': [[[1.0], [2.0]], [[3.0], [4.0]]]}
```

### Expected behavior

```python
print(dataset.to_dict())  # {'key': [[[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]], [[7.0, 8.0, 9.0], [10.0, 11.0, 12.0]]]}
```

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-6.2.0-32-generic-x86_64-with-glibc2.35
- Python version: 3.9.12
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 2.0.3


Remove unused global variables in `audio.py`

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006753 / 0.011353 (-0.004600) | 0.004027 / 0.011008 (-0.006982) | 0.084200 / 0.038508 (0.045692) | 0.072233 / 0.023109 (0.049124) | 0.361535 / 0.275898 (0.085637) | 0.386196 / 0.323480 (0.062716) | 0.004047 / 0.007986 (-0.003939) | 0.003416 / 0.004328 (-0.000912) | 0.064724 / 0.004250 (0.060474) | 0.055740 / 0.037052 (0.018688) | 0.360422 / 0.258489 (0.101933) | 0.399230 / 0.293841 (0.105389) | 0.031537 / 0.128546 (-0.097009) | 0.008630 / 0.075646 (-0.067016) | 0.289652 / 0.419271 (-0.129620) | 0.052881 / 0.043533 (0.009348) | 0.359538 / 0.255139 (0.104399) | 0.379410 / 0.283200 (0.096211) | 0.024539 / 0.141683 (-0.117144) | 1.470891 / 1.452155 (0.018736) | 1.578879 / 1.492716 (0.086163) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.239200 / 0.018006 (0.221194) | 0.462100 / 0.000490 (0.461610) | 0.009055 / 0.000200 (0.008856) | 0.000406 / 0.000054 (0.000352) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028736 / 0.037411 (-0.008675) | 0.088051 / 0.014526 (0.073525) | 0.098101 / 0.176557 (-0.078456) | 0.152399 / 0.737135 (-0.584737) | 0.098776 / 0.296338 (-0.197563) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.401761 / 0.215209 (0.186552) | 4.014143 / 2.077655 (1.936488) | 2.033255 / 1.504120 (0.529135) | 1.855347 / 1.541195 (0.314152) | 1.996144 / 1.468490 (0.527654) | 0.488545 / 4.584777 (-4.096232) | 3.712030 / 3.745712 (-0.033682) | 3.439725 / 5.269862 (-1.830137) | 2.119289 / 4.565676 (-2.446388) | 0.057523 / 0.424275 (-0.366752) | 0.007780 / 0.007607 (0.000173) | 0.479522 / 0.226044 (0.253477) | 4.798218 / 2.268929 (2.529290) | 2.543816 / 55.444624 (-52.900809) | 2.180392 / 6.876477 (-4.696085) | 2.427195 / 2.142072 (0.285122) | 0.602071 / 4.805227 (-4.203156) | 0.133450 / 6.500664 (-6.367214) | 0.061975 / 0.075469 (-0.013494) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.250040 / 1.841788 (-0.591748) | 19.532327 / 8.074308 (11.458019) | 14.200298 / 10.191392 (4.008906) | 0.165165 / 0.680424 (-0.515259) | 0.018326 / 0.534201 (-0.515875) | 0.389788 / 0.579283 (-0.189495) | 0.419301 / 0.434364 (-0.015063) | 0.452645 / 0.540337 (-0.087693) | 0.643409 / 1.386936 (-0.743527) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007040 / 0.011353 (-0.004313) | 0.004157 / 0.011008 (-0.006851) | 0.065439 / 0.038508 (0.026931) | 0.083210 / 0.023109 (0.060101) | 0.406707 / 0.275898 (0.130809) | 0.442759 / 0.323480 (0.119279) | 0.006321 / 0.007986 (-0.001665) | 0.003684 / 0.004328 (-0.000645) | 0.064517 / 0.004250 (0.060266) | 0.060676 / 0.037052 (0.023624) | 0.413395 / 0.258489 (0.154906) | 0.446776 / 0.293841 (0.152935) | 0.032542 / 0.128546 (-0.096004) | 0.008614 / 0.075646 (-0.067033) | 0.071760 / 0.419271 (-0.347511) | 0.049646 / 0.043533 (0.006113) | 0.402409 / 0.255139 (0.147270) | 0.422775 / 0.283200 (0.139575) | 0.024846 / 0.141683 (-0.116836) | 1.522915 / 1.452155 (0.070761) | 1.566518 / 1.492716 (0.073802) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.234478 / 0.018006 (0.216472) | 0.461318 / 0.000490 (0.460828) | 0.006304 / 0.000200 (0.006105) | 0.000105 / 0.000054 (0.000051) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036904 / 0.037411 (-0.000508) | 0.102144 / 0.014526 (0.087619) | 0.108985 / 0.176557 (-0.067572) | 0.162609 / 0.737135 (-0.574526) | 0.110295 / 0.296338 (-0.186044) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.438735 / 0.215209 (0.223526) | 4.377602 / 2.077655 (2.299948) | 2.375305 / 1.504120 (0.871185) | 2.215877 / 1.541195 (0.674682) | 2.317468 / 1.468490 (0.848978) | 0.495137 / 4.584777 (-4.089640) | 3.726323 / 3.745712 (-0.019389) | 3.493785 / 5.269862 (-1.776077) | 2.177891 / 4.565676 (-2.387785) | 0.058975 / 0.424275 (-0.365300) | 0.007897 / 0.007607 (0.000290) | 0.514063 / 0.226044 (0.288019) | 5.132714 / 2.268929 (2.863786) | 2.914125 / 55.444624 (-52.530499) | 2.532912 / 6.876477 (-4.343564) | 2.776438 / 2.142072 (0.634365) | 0.624831 / 4.805227 (-4.180396) | 0.135023 / 6.500664 (-6.365641) | 0.062040 / 0.075469 (-0.013429) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.359970 / 1.841788 (-0.481818) | 20.816464 / 8.074308 (12.742156) | 16.103544 / 10.191392 (5.912152) | 0.149120 / 0.680424 (-0.531304) | 0.020279 / 0.534201 (-0.513922) | 0.408727 / 0.579283 (-0.170556) | 0.436191 / 0.434364 (0.001827) | 0.485056 / 0.540337 (-0.055281) | 0.737727 / 1.386936 (-0.649209) |

![](https://cml.dev/watermark.png#d15280f435b7e27c9350a0cc37a07dbc5e2ea9ca "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008102 / 0.011353 (-0.003251) | 0.004886 / 0.011008 (-0.006123) | 0.090482 / 0.038508 (0.051974) | 0.071594 / 0.023109 (0.048485) | 0.428678 / 0.275898 (0.152780) | 0.442179 / 0.323480 (0.118699) | 0.004329 / 0.007986 (-0.003657) | 0.003756 / 0.004328 (-0.000573) | 0.087125 / 0.004250 (0.082874) | 0.055159 / 0.037052 (0.018107) | 0.437646 / 0.258489 (0.179157) | 0.446665 / 0.293841 (0.152824) | 0.046402 / 0.128546 (-0.082145) | 0.014248 / 0.075646 (-0.061398) | 0.331401 / 0.419271 (-0.087871) | 0.062010 / 0.043533 (0.018478) | 0.434774 / 0.255139 (0.179635) | 0.441063 / 0.283200 (0.157863) | 0.037424 / 0.141683 (-0.104258) | 1.720276 / 1.452155 (0.268121) | 1.731491 / 1.492716 (0.238775) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.302935 / 0.018006 (0.284929) | 0.590556 / 0.000490 (0.590067) | 0.014473 / 0.000200 (0.014274) | 0.000712 / 0.000054 (0.000658) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031289 / 0.037411 (-0.006122) | 0.091175 / 0.014526 (0.076649) | 0.112895 / 0.176557 (-0.063661) | 0.199558 / 0.737135 (-0.537577) | 0.113397 / 0.296338 (-0.182942) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.571586 / 0.215209 (0.356377) | 5.706894 / 2.077655 (3.629240) | 2.512701 / 1.504120 (1.008581) | 2.151705 / 1.541195 (0.610510) | 2.252738 / 1.468490 (0.784248) | 0.857524 / 4.584777 (-3.727253) | 5.189027 / 3.745712 (1.443315) | 4.464979 / 5.269862 (-0.804882) | 2.787486 / 4.565676 (-1.778190) | 0.090161 / 0.424275 (-0.334115) | 0.008649 / 0.007607 (0.001042) | 0.703367 / 0.226044 (0.477322) | 7.128971 / 2.268929 (4.860043) | 3.437475 / 55.444624 (-52.007149) | 2.562291 / 6.876477 (-4.314186) | 2.753419 / 2.142072 (0.611346) | 0.981964 / 4.805227 (-3.823263) | 0.194533 / 6.500664 (-6.306131) | 0.069659 / 0.075469 (-0.005810) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.510356 / 1.841788 (-0.331431) | 22.414117 / 8.074308 (14.339809) | 20.325418 / 10.191392 (10.134025) | 0.226823 / 0.680424 (-0.453601) | 0.029123 / 0.534201 (-0.505078) | 0.454656 / 0.579283 (-0.124627) | 0.559588 / 0.434364 (0.125224) | 0.547386 / 0.540337 (0.007048) | 0.770169 / 1.386936 (-0.616767) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010167 / 0.011353 (-0.001186) | 0.005164 / 0.011008 (-0.005844) | 0.094897 / 0.038508 (0.056388) | 0.078027 / 0.023109 (0.054918) | 0.474442 / 0.275898 (0.198544) | 0.503362 / 0.323480 (0.179882) | 0.006988 / 0.007986 (-0.000998) | 0.005369 / 0.004328 (0.001041) | 0.079547 / 0.004250 (0.075297) | 0.059382 / 0.037052 (0.022329) | 0.468759 / 0.258489 (0.210270) | 0.566780 / 0.293841 (0.272939) | 0.050791 / 0.128546 (-0.077755) | 0.013191 / 0.075646 (-0.062455) | 0.086086 / 0.419271 (-0.333186) | 0.060399 / 0.043533 (0.016866) | 0.492985 / 0.255139 (0.237846) | 0.509139 / 0.283200 (0.225940) | 0.034537 / 0.141683 (-0.107146) | 1.699166 / 1.452155 (0.247011) | 1.789781 / 1.492716 (0.297065) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.278776 / 0.018006 (0.260769) | 0.615877 / 0.000490 (0.615387) | 0.009062 / 0.000200 (0.008862) | 0.000112 / 0.000054 (0.000057) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032931 / 0.037411 (-0.004481) | 0.094796 / 0.014526 (0.080270) | 0.126697 / 0.176557 (-0.049859) | 0.168172 / 0.737135 (-0.568963) | 0.113906 / 0.296338 (-0.182433) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.602378 / 0.215209 (0.387169) | 5.987708 / 2.077655 (3.910054) | 2.800339 / 1.504120 (1.296219) | 2.474127 / 1.541195 (0.932932) | 2.502387 / 1.468490 (1.033897) | 0.808147 / 4.584777 (-3.776630) | 5.212691 / 3.745712 (1.466979) | 4.479452 / 5.269862 (-0.790409) | 2.831960 / 4.565676 (-1.733717) | 0.086777 / 0.424275 (-0.337498) | 0.009492 / 0.007607 (0.001885) | 0.716848 / 0.226044 (0.490803) | 7.099904 / 2.268929 (4.830975) | 3.794708 / 55.444624 (-51.649916) | 2.859826 / 6.876477 (-4.016650) | 3.109673 / 2.142072 (0.967600) | 0.936776 / 4.805227 (-3.868451) | 0.195152 / 6.500664 (-6.305512) | 0.074184 / 0.075469 (-0.001285) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.585419 / 1.841788 (-0.256369) | 22.420377 / 8.074308 (14.346068) | 20.761533 / 10.191392 (10.570141) | 0.228480 / 0.680424 (-0.451943) | 0.030944 / 0.534201 (-0.503257) | 0.444717 / 0.579283 (-0.134566) | 0.579632 / 0.434364 (0.145268) | 0.521669 / 0.540337 (-0.018669) | 0.748274 / 1.386936 (-0.638662) |

![](https://cml.dev/watermark.png#94e07965a400e6901f12e6f0f25c7090656c828c "CML watermark")


Dataloader stuck on multiple GPUs

What type of dataset are you using in this script? `torch.utils.data.Dataset` or `datasets.Dataset`? Please share the `datasets` package version if it's the latter. Otherwise, it's better to move this issue to the `accelerate` repo.

Very sorry, I thought I had a repo in `accelerate!`
I will close this issue and repo the issue in the appropriate place.

### Describe the bug

I am trying to get CLIP to fine-tuning with my code.
When I tried to run it on multiple GPUs using accelerate, I encountered the following phenomenon.

- Validation dataloader stuck in 2nd epoch only on multi-GPU
Specifically, when the "for inputs in valid_loader:" process is finished, it does not proceed to the next step. train_loader process is completed. Also, both train and valid are working correctly in the first epoch.
The accelerate command at that time is as follows.
`accelerate launch --multi_gpu --num_processes=2 {script_name.py} {--arg1} {--arg2} ...`

- This will not happen when single GPU is used.
`CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...`
- Setting num_workers=0 in dataloader did not change the result.

### Steps to reproduce the bug

1. The codes for fine-tuning the regular CLIP were updated for accelerate.
2. Run the code with the accelerate command as `accelerate launch --multi_gpu --num_processes=2 {script_name.py} {--arg1} {--arg2} ...` and the above problem will occur.
3. CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...` , it works fine.

### Expected behavior

It Should end normally as if it was run on a single GPU.

### Environment info

Since `datasets-cli env` did not work, the environment is described below.

- OS: Ubuntu 22.04 with Docker
- Docker: 24.0.5, build ced0996
- Python: 3.10.12
- torch==2.0.1
- accelerate==0.21.0
- transformers==4.33.1


Load local audio data doesn't work

I think this is the same issue as https://github.com/huggingface/datasets/issues/4776. Maybe installing `ffmpeg` can fix it:
```python
add-apt-repository -y ppa:savoury1/ffmpeg4
apt-get -qq install -y ffmpeg
```

However, the best solution is to use a newer version of `datasets`. In the recent releases, we've replaced `torchaudio` with `soundfile`, which is easier to install and faster.

### Describe the bug

I get a RuntimeError from the following code:

```python
 audio_dataset = Dataset.from_dict({"audio": ["/kaggle/input/bengaliai-speech/train_mp3s/000005f3362c.mp3"]}).cast_column("audio", Audio())
 audio_dataset[0]
 ```

 ### Traceback

<details>

```python
RuntimeError                              Traceback (most recent call last)
Cell In[33], line 1
----> 1 train_dataset[0]

File /opt/conda/lib/python3.10/site-packages/datasets/arrow_dataset.py:1764, in Dataset.__getitem__(self, key)
   1762 def __getitem__(self, key):  # noqa: F811
   1763     """Can be used to index columns (by string names) or rows (by integer index or iterable of indices or bools)."""
-> 1764     return self._getitem(
   1765         key,
   1766     )

File /opt/conda/lib/python3.10/site-packages/datasets/arrow_dataset.py:1749, in Dataset._getitem(self, key, decoded, **kwargs)
   1747 formatter = get_formatter(format_type, features=self.features, decoded=decoded, **format_kwargs)
   1748 pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
-> 1749 formatted_output = format_table(
   1750     pa_subtable, key, formatter=formatter, format_columns=format_columns, output_all_columns=output_all_columns
   1751 )
   1752 return formatted_output

File /opt/conda/lib/python3.10/site-packages/datasets/formatting/formatting.py:532, in format_table(table, key, formatter, format_columns, output_all_columns)
    530 python_formatter = PythonFormatter(features=None)
    531 if format_columns is None:
--> 532     return formatter(pa_table, query_type=query_type)
    533 elif query_type == "column":
    534     if key in format_columns:

File /opt/conda/lib/python3.10/site-packages/datasets/formatting/formatting.py:281, in Formatter.__call__(self, pa_table, query_type)
    279 def __call__(self, pa_table: pa.Table, query_type: str) -> Union[RowFormat, ColumnFormat, BatchFormat]:
    280     if query_type == "row":
--> 281         return self.format_row(pa_table)
    282     elif query_type == "column":
    283         return self.format_column(pa_table)

File /opt/conda/lib/python3.10/site-packages/datasets/formatting/formatting.py:312, in PythonFormatter.format_row(self, pa_table)
    310 row = self.python_arrow_extractor().extract_row(pa_table)
    311 if self.decoded:
--> 312     row = self.python_features_decoder.decode_row(row)
    313 return row

File /opt/conda/lib/python3.10/site-packages/datasets/formatting/formatting.py:221, in PythonFeaturesDecoder.decode_row(self, row)
    220 def decode_row(self, row: dict) -> dict:
--> 221     return self.features.decode_example(row) if self.features else row

File /opt/conda/lib/python3.10/site-packages/datasets/features/features.py:1386, in Features.decode_example(self, example)
   1376 def decode_example(self, example: dict):
   1377     """Decode example with custom feature decoding.
   1378
   1379     Args:
   (...)
   1383         :obj:`dict[str, Any]`
   1384     """
-> 1386     return {
   1387         column_name: decode_nested_example(feature, value)
   1388         if self._column_requires_decoding[column_name]
   1389         else value
   1390         for column_name, (feature, value) in zip_dict(
   1391             {key: value for key, value in self.items() if key in example}, example
   1392         )
   1393     }

File /opt/conda/lib/python3.10/site-packages/datasets/features/features.py:1387, in <dictcomp>(.0)
   1376 def decode_example(self, example: dict):
   1377     """Decode example with custom feature decoding.
   1378
   1379     Args:
   (...)
   1383         :obj:`dict[str, Any]`
   1384     """
   1386     return {
-> 1387         column_name: decode_nested_example(feature, value)
   1388         if self._column_requires_decoding[column_name]
   1389         else value
   1390         for column_name, (feature, value) in zip_dict(
   1391             {key: value for key, value in self.items() if key in example}, example
   1392         )
   1393     }

File /opt/conda/lib/python3.10/site-packages/datasets/features/features.py:1087, in decode_nested_example(schema, obj)
   1085 # Object with special decoding:
   1086 elif isinstance(schema, (Audio, Image)):
-> 1087     return schema.decode_example(obj) if obj is not None else None
   1088 return obj

File /opt/conda/lib/python3.10/site-packages/datasets/features/audio.py:103, in Audio.decode_example(self, value)
    101     raise ValueError(f"An audio sample should have one of 'path' or 'bytes' but both are None in {value}.")
    102 elif path is not None and path.endswith("mp3"):
--> 103     array, sampling_rate = self._decode_mp3(file if file else path)
    104 elif path is not None and path.endswith("opus"):
    105     if file:

File /opt/conda/lib/python3.10/site-packages/datasets/features/audio.py:241, in Audio._decode_mp3(self, path_or_file)
    238 except RuntimeError as err:
    239     raise ImportError("To support decoding 'mp3' audio files, please install 'sox'.") from err
--> 241 array, sampling_rate = torchaudio.load(path_or_file, format="mp3")
    242 if self.sampling_rate and self.sampling_rate != sampling_rate:
    243     if not hasattr(self, "_resampler") or self._resampler.orig_freq != sampling_rate:

File /opt/conda/lib/python3.10/site-packages/torchaudio/backend/sox_io_backend.py:256, in load(filepath, frame_offset, num_frames, normalize, channels_first, format)
    254 if ret is not None:
    255     return ret
--> 256 return _fallback_load(filepath, frame_offset, num_frames, normalize, channels_first, format)

File /opt/conda/lib/python3.10/site-packages/torchaudio/backend/sox_io_backend.py:30, in _fail_load(filepath, frame_offset, num_frames, normalize, channels_first, format)
     22 def _fail_load(
     23     filepath: str,
     24     frame_offset: int = 0,
   (...)
     28     format: Optional[str] = None,
     29 ) -> Tuple[torch.Tensor, int]:
---> 30     raise RuntimeError("Failed to load audio from {}".format(filepath))

RuntimeError: Failed to load audio from /kaggle/input/bengaliai-speech/train_mp3s/000005f3362c.mp3
```

</details>

### Steps to reproduce the bug

1. - Create a custom dataset using Local files of type mp3.
3. - Try to read the first audio item.

### Expected behavior

Expected output
```python
audio_dataset[0]["audio"]
{'array': array([ 0.        ,  0.00024414, -0.00024414, ..., -0.00024414,
         0.        ,  0.        ], dtype=float32),
 'path': 'path/to/audio_1',
 'sampling_rate': 16000}
 ```

### Environment info

N/A


`dataset.filter` ALWAYS removes the first item from the dataset when using batched=True

`filter` treats the function's output as a (selection) mask - `True` keeps the sample, and `False` drops it. In your case, `bool(0)` evaluates to `False`, so dropping the first sample is the correct behavior.

Oh gosh! 🤦  I totally misunderstood the API! My apologies!

### Describe the bug

If you call batched=True when calling `filter`, the first item is _always_ filtered out, regardless of the filter condition.

### Steps to reproduce the bug

Here's a minimal example:
```python
def filter_batch_always_true(batch, indices):
    print("First index being passed into this filter function: ", indices[0])
    return indices  # Keep all indices

data = {"value": list(range(10))}
dataset = Dataset.from_dict(data)

filtered_dataset = dataset.filter(filter_batch_always_true, with_indices=True, batched=True)
print("Length of original dataset: ", len(dataset))
print("Length of filtered_dataset: ", len(filtered_dataset))
print("Is equal to original? ", len(filtered_dataset) == len(dataset))
print("First item of filtered dataset: ", filtered_dataset[0])
print("Last item of filtered dataset: ", filtered_dataset[-1])
```
prints:
```
First index being passed into this filter function:  0
Length of original dataset:  10
Length of filtered_dataset:  9
Is equal to original?  False
First item of filtered dataset:  {'value': 1}
Last item of filtered dataset:  {'value': 9}
```

### Expected behavior

Filter should respect the filter condition.

### Environment info

- `datasets` version: 2.14.4
- Platform: macOS-13.5-arm64-arm-64bit
- Python version: 3.9.18
- Huggingface_hub version: 0.17.1
- PyArrow version: 10.0.1
- Pandas version: 2.0.2


Tokenization with multiple workers is too slow

[This](https://huggingface.co/docs/datasets/nlp_process#map) is the most performant way to tokenize a dataset (`batched=True, num_proc=None, return_tensors="np"`)

If`tokenizer.is_fast` returns `True`, `num_proc` must be `None/1` to benefit from the fast tokenizers' parallelism (the fast tokenizers are implemented in Rust, and Rust multi-threading doesn't work well with Python multi-processing)

I am trying to tokenize a few million documents with multiple workers but the tokenization process is taking forever.

Code snippet:

```
raw_datasets.map(
            encode_function,
            batched=False,
            num_proc=args.preprocessing_num_workers,
            load_from_cache_file=not args.overwrite_cache,
            remove_columns=[name for name in raw_datasets["train"].column_names if name not in ["input_ids", "labels", "attention_mask"]],
            desc="Tokenizing data",
        )
```

Details:

```
transformers==4.28.0.dev0
datasets==4.28.0.dev0
preprocessing_num_workers==48
```
tokenizer == decapoda-research/llama-7b-hf


Support buffer shuffle for to_tf_dataset

enhancement

New feature or request

Hey! You can implement this yourself, just:

1) Create the dataset with `to_tf_dataset()` with `shuffle=False`
2) Add an `unbatch()` at the end (or use batch_size=1)
3) Add a `shuffle()` to the resulting dataset with your desired buffer size
4) Add a `batch()` at the end again to re-batch your dataset.

Note that the way we construct datasets in `to_tf_dataset()`, we don't actually shuffle the entire dataset in-memory, using `tf.data.Dataset.shuffle()`! Instead, we shuffle an index array and then load from the dataset with that. This means that shuffling with `tf.data.Dataset.shuffle()` will probably be slower and use more memory than our approach - I don't think adding the option for smaller shuffle buffers will actually save you memory on this!

Thanks for your reply! @Rocketknight1
"We don't actually shuffle the entire dataset in-memory, using tf.data.Dataset.shuffle()! Instead, we shuffle an index array and then load from the dataset with that."
In such case, there will be random access to dataset data during shuffling. When the dataset is large, the performance can be X10 times slow. I have tried many ways with to_tf_dataset() trying to achieve comparable performance with tf.data.Dataset().shuffle(buffer_size).batch(). But the performance with to_tf_dataset() is still slow.

### Feature request

I'm using to_tf_dataset to convert a large dataset to tf.data.Dataset and use Keras fit to train model.
Currently, to_tf_dataset only supports full size shuffle, which can be very slow on large dataset.
tf.data.Dataset support buffer shuffle by default.
shuffle(
    buffer_size, seed=None, reshuffle_each_iteration=None, name=None
)

### Motivation

I'm very frustrated to find the loading with shuffling large dataset is very slow. It seems impossible to shuffle before training Keras with big dataset.

### Your contribution

NA


Support multiprocessing for download/extract nestedly

New feature or request

### Feature request

Current multiprocessing for download/extract is not done nestedly. For example, when processing SlimPajama, there is only 3 processes (for train/test/val), while there are many files inside these 3 folders

```
Downloading data files #0:   0%|          | 0/1 [00:00<?, ?obj/s]
Downloading data files #1:   0%|          | 0/1 [00:00<?, ?obj/s]
Downloading data files #2:   0%|          | 0/1 [00:00<?, ?obj/s]

Extracting data files #0:   0%|          | 0/1 [00:00<?, ?obj/s]
Extracting data files #1:   0%|          | 0/1 [00:00<?, ?obj/s]
Extracting data files #2:   0%|          | 0/1 [00:00<?, ?obj/s]
```

### Motivation

speedup dataset loading

### Your contribution

I can help test the feature


Update README.md

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008370 / 0.011353 (-0.002983) | 0.004674 / 0.011008 (-0.006334) | 0.103912 / 0.038508 (0.065404) | 0.101668 / 0.023109 (0.078559) | 0.417945 / 0.275898 (0.142047) | 0.454805 / 0.323480 (0.131325) | 0.004763 / 0.007986 (-0.003223) | 0.003934 / 0.004328 (-0.000394) | 0.078446 / 0.004250 (0.074196) | 0.068383 / 0.037052 (0.031331) | 0.415100 / 0.258489 (0.156611) | 0.475272 / 0.293841 (0.181431) | 0.036884 / 0.128546 (-0.091662) | 0.010097 / 0.075646 (-0.065549) | 0.354962 / 0.419271 (-0.064309) | 0.062688 / 0.043533 (0.019155) | 0.420643 / 0.255139 (0.165504) | 0.446504 / 0.283200 (0.163304) | 0.029075 / 0.141683 (-0.112608) | 1.791517 / 1.452155 (0.339363) | 1.859820 / 1.492716 (0.367104) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.246929 / 0.018006 (0.228923) | 0.519593 / 0.000490 (0.519103) | 0.006848 / 0.000200 (0.006648) | 0.000168 / 0.000054 (0.000114) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035179 / 0.037411 (-0.002232) | 0.115582 / 0.014526 (0.101057) | 0.128235 / 0.176557 (-0.048321) | 0.187123 / 0.737135 (-0.550012) | 0.120862 / 0.296338 (-0.175477) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.463406 / 0.215209 (0.248197) | 4.615517 / 2.077655 (2.537863) | 2.250513 / 1.504120 (0.746393) | 2.061226 / 1.541195 (0.520032) | 2.189938 / 1.468490 (0.721448) | 0.582984 / 4.584777 (-4.001793) | 4.299464 / 3.745712 (0.553751) | 4.037274 / 5.269862 (-1.232588) | 2.608967 / 4.565676 (-1.956710) | 0.068944 / 0.424275 (-0.355331) | 0.009501 / 0.007607 (0.001894) | 0.567436 / 0.226044 (0.341392) | 5.662738 / 2.268929 (3.393809) | 2.849094 / 55.444624 (-52.595530) | 2.461013 / 6.876477 (-4.415464) | 2.663245 / 2.142072 (0.521172) | 0.704528 / 4.805227 (-4.100699) | 0.163583 / 6.500664 (-6.337081) | 0.075719 / 0.075469 (0.000250) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.604743 / 1.841788 (-0.237044) | 24.512054 / 8.074308 (16.437746) | 17.870939 / 10.191392 (7.679547) | 0.199188 / 0.680424 (-0.481236) | 0.023820 / 0.534201 (-0.510381) | 0.487520 / 0.579283 (-0.091763) | 0.512543 / 0.434364 (0.078179) | 0.575138 / 0.540337 (0.034801) | 0.759863 / 1.386936 (-0.627073) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010516 / 0.011353 (-0.000837) | 0.004779 / 0.011008 (-0.006229) | 0.078482 / 0.038508 (0.039974) | 0.108533 / 0.023109 (0.085424) | 0.498692 / 0.275898 (0.222794) | 0.534698 / 0.323480 (0.211218) | 0.007624 / 0.007986 (-0.000362) | 0.003938 / 0.004328 (-0.000391) | 0.077317 / 0.004250 (0.073067) | 0.078056 / 0.037052 (0.041004) | 0.493648 / 0.258489 (0.235159) | 0.540891 / 0.293841 (0.247050) | 0.040377 / 0.128546 (-0.088169) | 0.010155 / 0.075646 (-0.065491) | 0.084384 / 0.419271 (-0.334888) | 0.061419 / 0.043533 (0.017886) | 0.494474 / 0.255139 (0.239335) | 0.524656 / 0.283200 (0.241456) | 0.029052 / 0.141683 (-0.112631) | 1.794584 / 1.452155 (0.342429) | 1.939987 / 1.492716 (0.447270) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.377404 / 0.018006 (0.359398) | 0.516562 / 0.000490 (0.516072) | 0.109555 / 0.000200 (0.109356) | 0.001126 / 0.000054 (0.001071) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.039793 / 0.037411 (0.002382) | 0.123001 / 0.014526 (0.108475) | 0.127536 / 0.176557 (-0.049021) | 0.191681 / 0.737135 (-0.545455) | 0.128590 / 0.296338 (-0.167748) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.513689 / 0.215209 (0.298480) | 5.135114 / 2.077655 (3.057459) | 2.797885 / 1.504120 (1.293765) | 2.715332 / 1.541195 (1.174137) | 2.746437 / 1.468490 (1.277947) | 0.596480 / 4.584777 (-3.988297) | 4.382013 / 3.745712 (0.636301) | 3.965956 / 5.269862 (-1.303906) | 2.545206 / 4.565676 (-2.020471) | 0.069620 / 0.424275 (-0.354655) | 0.009321 / 0.007607 (0.001714) | 0.612424 / 0.226044 (0.386379) | 6.107037 / 2.268929 (3.838109) | 3.447246 / 55.444624 (-51.997379) | 3.073262 / 6.876477 (-3.803215) | 3.280185 / 2.142072 (1.138113) | 0.704776 / 4.805227 (-4.100451) | 0.160488 / 6.500664 (-6.340176) | 0.075730 / 0.075469 (0.000261) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.697035 / 1.841788 (-0.144753) | 24.766118 / 8.074308 (16.691809) | 18.476699 / 10.191392 (8.285307) | 0.176594 / 0.680424 (-0.503830) | 0.024249 / 0.534201 (-0.509952) | 0.478743 / 0.579283 (-0.100541) | 0.518774 / 0.434364 (0.084410) | 0.581498 / 0.540337 (0.041161) | 0.797784 / 1.386936 (-0.589152) |

![](https://cml.dev/watermark.png#722cea0f4929ff4ffcdbb7ca6b72cba229b9701a "CML watermark")

fixed a typo


Improve error message for missing function parameters

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006681 / 0.011353 (-0.004672) | 0.004132 / 0.011008 (-0.006876) | 0.085045 / 0.038508 (0.046536) | 0.077680 / 0.023109 (0.054571) | 0.382042 / 0.275898 (0.106144) | 0.412932 / 0.323480 (0.089452) | 0.005339 / 0.007986 (-0.002646) | 0.003408 / 0.004328 (-0.000921) | 0.065280 / 0.004250 (0.061030) | 0.055732 / 0.037052 (0.018680) | 0.400231 / 0.258489 (0.141742) | 0.432497 / 0.293841 (0.138656) | 0.031532 / 0.128546 (-0.097014) | 0.008721 / 0.075646 (-0.066925) | 0.289612 / 0.419271 (-0.129660) | 0.053089 / 0.043533 (0.009556) | 0.383300 / 0.255139 (0.128161) | 0.401204 / 0.283200 (0.118004) | 0.023582 / 0.141683 (-0.118100) | 1.493854 / 1.452155 (0.041699) | 1.583497 / 1.492716 (0.090781) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.239163 / 0.018006 (0.221157) | 0.469555 / 0.000490 (0.469065) | 0.008325 / 0.000200 (0.008125) | 0.000113 / 0.000054 (0.000059) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028975 / 0.037411 (-0.008436) | 0.084195 / 0.014526 (0.069669) | 0.189394 / 0.176557 (0.012837) | 0.158010 / 0.737135 (-0.579125) | 0.097502 / 0.296338 (-0.198837) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.383085 / 0.215209 (0.167876) | 3.827030 / 2.077655 (1.749375) | 1.872279 / 1.504120 (0.368159) | 1.705808 / 1.541195 (0.164613) | 1.833706 / 1.468490 (0.365216) | 0.484744 / 4.584777 (-4.100033) | 3.658221 / 3.745712 (-0.087491) | 3.398462 / 5.269862 (-1.871399) | 2.064974 / 4.565676 (-2.500703) | 0.057740 / 0.424275 (-0.366535) | 0.007926 / 0.007607 (0.000319) | 0.465358 / 0.226044 (0.239314) | 4.652951 / 2.268929 (2.384022) | 2.328390 / 55.444624 (-53.116235) | 2.000606 / 6.876477 (-4.875870) | 2.268391 / 2.142072 (0.126318) | 0.586537 / 4.805227 (-4.218690) | 0.134749 / 6.500664 (-6.365915) | 0.061276 / 0.075469 (-0.014193) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.337913 / 1.841788 (-0.503875) | 20.232122 / 8.074308 (12.157814) | 14.478579 / 10.191392 (4.287187) | 0.167545 / 0.680424 (-0.512878) | 0.018745 / 0.534201 (-0.515456) | 0.401209 / 0.579283 (-0.178074) | 0.425748 / 0.434364 (-0.008616) | 0.462539 / 0.540337 (-0.077798) | 0.652446 / 1.386936 (-0.734490) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007159 / 0.011353 (-0.004194) | 0.004091 / 0.011008 (-0.006917) | 0.066202 / 0.038508 (0.027694) | 0.083096 / 0.023109 (0.059987) | 0.402160 / 0.275898 (0.126261) | 0.440565 / 0.323480 (0.117085) | 0.005757 / 0.007986 (-0.002228) | 0.003445 / 0.004328 (-0.000884) | 0.065498 / 0.004250 (0.061248) | 0.059787 / 0.037052 (0.022735) | 0.407017 / 0.258489 (0.148528) | 0.448270 / 0.293841 (0.154429) | 0.033606 / 0.128546 (-0.094941) | 0.008744 / 0.075646 (-0.066902) | 0.072902 / 0.419271 (-0.346369) | 0.050144 / 0.043533 (0.006611) | 0.401069 / 0.255139 (0.145930) | 0.426389 / 0.283200 (0.143189) | 0.023297 / 0.141683 (-0.118386) | 1.506152 / 1.452155 (0.053998) | 1.570211 / 1.492716 (0.077495) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.235759 / 0.018006 (0.217753) | 0.488410 / 0.000490 (0.487921) | 0.004587 / 0.000200 (0.004387) | 0.000115 / 0.000054 (0.000060) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034123 / 0.037411 (-0.003289) | 0.102163 / 0.014526 (0.087638) | 0.110892 / 0.176557 (-0.065664) | 0.166000 / 0.737135 (-0.571135) | 0.110845 / 0.296338 (-0.185494) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.431397 / 0.215209 (0.216188) | 4.291540 / 2.077655 (2.213885) | 2.298248 / 1.504120 (0.794128) | 2.134752 / 1.541195 (0.593557) | 2.207913 / 1.468490 (0.739423) | 0.490607 / 4.584777 (-4.094170) | 3.683078 / 3.745712 (-0.062635) | 3.314266 / 5.269862 (-1.955596) | 2.059488 / 4.565676 (-2.506188) | 0.057876 / 0.424275 (-0.366399) | 0.007696 / 0.007607 (0.000089) | 0.512186 / 0.226044 (0.286142) | 5.124071 / 2.268929 (2.855142) | 2.803913 / 55.444624 (-52.640711) | 2.428558 / 6.876477 (-4.447919) | 2.655207 / 2.142072 (0.513135) | 0.584589 / 4.805227 (-4.220638) | 0.133518 / 6.500664 (-6.367146) | 0.060729 / 0.075469 (-0.014740) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.352916 / 1.841788 (-0.488872) | 20.249632 / 8.074308 (12.175323) | 15.283079 / 10.191392 (5.091686) | 0.157601 / 0.680424 (-0.522823) | 0.019650 / 0.534201 (-0.514551) | 0.396398 / 0.579283 (-0.182885) | 0.430111 / 0.434364 (-0.004252) | 0.480627 / 0.540337 (-0.059710) | 0.642165 / 1.386936 (-0.744771) |

![](https://cml.dev/watermark.png#9b21e181b642bd55b3ef68c1948bfbcd388136d6 "CML watermark")

The error message in the fingerprint module was missing the f-string 'f' symbol, so the error message returned by fingerprint.py, line 469 was literally "function {func} is missing parameters {fingerprint_names} in signature."

This has been fixed.


Overwrite legacy default config name in `dataset_infos.json` in packaged datasets

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6231). All of your documentation changes will be reflected on that endpoint.

realized that this pr is still not merged, @lhoestq maybe you can take a look at it?

I think https://github.com/huggingface/datasets/pull/6218 fixed the issue (a bit differently though)

@lhoestq yeah the pr you're referencing doesn't fix the problem when two semantically analogous configs occur in datasets_info.json, i suggest to rewrite the legacy one if it exists during .push_to_hub

Only the old versions of `datasets` use the JSON file over the README and they can only load one config so the name doesn't really matter.

That's why I chose to load the info from the JSON no matter the name (no check to see if it's "username--dataset_name") in my previous PR.

I think you can remove the old info without even checking the name. In this case maybe no need to update load.py

(also minor: not checking the name makes it more robust to dataset renaming)

@lhoestq okay makes sense... so you think it's not a problem that in some cases we might end up with `dataset_infos.json` having two keys in it?

> @lhoestq okay makes sense... so you think it's not a problem that in some cases we might end up with dataset_infos.json having two keys in it?

Ideally they should have only one config no ? Since old versions of `datasets` simply load the first config in the JSON.
We can overwrite it with the new default one (and no matter the name of the outdated config in the JSON)

Currently if we push data as default config with `.push_to_hub` to a repo that has a legacy `dataset_infos.json` file containing a legacy default config name like `{username}--{dataset_name}`, new key `"default"` is added to `dataset_infos.json` along with the legacy one. I think the legacy one should be dropped in this case.

Also, in `load.py` I suggest to check if a legacy config name is indeed a legacy config name because after this fix it might not be the case (this check was first introduced in https://github.com/huggingface/datasets/pull/6218)