Less zip false positives

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006998 / 0.011353 (-0.004355) | 0.005093 / 0.011008 (-0.005916) | 0.100490 / 0.038508 (0.061982) | 0.032736 / 0.023109 (0.009627) | 0.297738 / 0.275898 (0.021840) | 0.322255 / 0.323480 (-0.001225) | 0.005583 / 0.007986 (-0.002402) | 0.004007 / 0.004328 (-0.000321) | 0.075863 / 0.004250 (0.071613) | 0.044212 / 0.037052 (0.007159) | 0.300033 / 0.258489 (0.041544) | 0.341997 / 0.293841 (0.048156) | 0.036172 / 0.128546 (-0.092374) | 0.012176 / 0.075646 (-0.063471) | 0.356052 / 0.419271 (-0.063220) | 0.050438 / 0.043533 (0.006905) | 0.294677 / 0.255139 (0.039538) | 0.318050 / 0.283200 (0.034850) | 0.104733 / 0.141683 (-0.036950) | 1.435681 / 1.452155 (-0.016474) | 1.534793 / 1.492716 (0.042076) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.242815 / 0.018006 (0.224809) | 0.565983 / 0.000490 (0.565494) | 0.006800 / 0.000200 (0.006600) | 0.000124 / 0.000054 (0.000070) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026548 / 0.037411 (-0.010863) | 0.104816 / 0.014526 (0.090290) | 0.116222 / 0.176557 (-0.060335) | 0.172143 / 0.737135 (-0.564992) | 0.121631 / 0.296338 (-0.174707) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.400126 / 0.215209 (0.184917) | 4.004538 / 2.077655 (1.926883) | 1.798822 / 1.504120 (0.294702) | 1.595191 / 1.541195 (0.053996) | 1.645777 / 1.468490 (0.177287) | 0.705643 / 4.584777 (-3.879134) | 3.750887 / 3.745712 (0.005175) | 2.136547 / 5.269862 (-3.133315) | 1.475881 / 4.565676 (-3.089795) | 0.086921 / 0.424275 (-0.337354) | 0.012379 / 0.007607 (0.004771) | 0.505824 / 0.226044 (0.279779) | 5.052364 / 2.268929 (2.783435) | 2.279983 / 55.444624 (-53.164641) | 1.932253 / 6.876477 (-4.944224) | 2.051359 / 2.142072 (-0.090714) | 0.851906 / 4.805227 (-3.953321) | 0.169566 / 6.500664 (-6.331098) | 0.064600 / 0.075469 (-0.010869) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.165859 / 1.841788 (-0.675929) | 15.049950 / 8.074308 (6.975642) | 14.095981 / 10.191392 (3.904589) | 0.151779 / 0.680424 (-0.528645) | 0.017537 / 0.534201 (-0.516664) | 0.420164 / 0.579283 (-0.159119) | 0.418932 / 0.434364 (-0.015432) | 0.488749 / 0.540337 (-0.051588) | 0.582359 / 1.386936 (-0.804577) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007426 / 0.011353 (-0.003927) | 0.005248 / 0.011008 (-0.005761) | 0.074118 / 0.038508 (0.035610) | 0.034223 / 0.023109 (0.011114) | 0.337780 / 0.275898 (0.061882) | 0.376300 / 0.323480 (0.052820) | 0.006142 / 0.007986 (-0.001843) | 0.004246 / 0.004328 (-0.000083) | 0.074177 / 0.004250 (0.069926) | 0.052698 / 0.037052 (0.015646) | 0.340229 / 0.258489 (0.081740) | 0.396172 / 0.293841 (0.102331) | 0.037293 / 0.128546 (-0.091253) | 0.012514 / 0.075646 (-0.063132) | 0.087144 / 0.419271 (-0.332128) | 0.051922 / 0.043533 (0.008390) | 0.333188 / 0.255139 (0.078049) | 0.355420 / 0.283200 (0.072220) | 0.110273 / 0.141683 (-0.031410) | 1.447826 / 1.452155 (-0.004329) | 1.561135 / 1.492716 (0.068419) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.269203 / 0.018006 (0.251197) | 0.551997 / 0.000490 (0.551508) | 0.001558 / 0.000200 (0.001359) | 0.000090 / 0.000054 (0.000035) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029511 / 0.037411 (-0.007900) | 0.108614 / 0.014526 (0.094089) | 0.123438 / 0.176557 (-0.053118) | 0.171596 / 0.737135 (-0.565539) | 0.126828 / 0.296338 (-0.169511) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.420520 / 0.215209 (0.205310) | 4.175672 / 2.077655 (2.098017) | 1.982220 / 1.504120 (0.478101) | 1.788575 / 1.541195 (0.247381) | 1.860840 / 1.468490 (0.392349) | 0.706730 / 4.584777 (-3.878047) | 3.858718 / 3.745712 (0.113005) | 3.069389 / 5.269862 (-2.200472) | 1.827603 / 4.565676 (-2.738073) | 0.087893 / 0.424275 (-0.336382) | 0.012613 / 0.007607 (0.005006) | 0.524177 / 0.226044 (0.298132) | 5.177077 / 2.268929 (2.908148) | 2.494397 / 55.444624 (-52.950227) | 2.189484 / 6.876477 (-4.686992) | 2.217626 / 2.142072 (0.075554) | 0.846326 / 4.805227 (-3.958901) | 0.176558 / 6.500664 (-6.324106) | 0.065018 / 0.075469 (-0.010451) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.268618 / 1.841788 (-0.573170) | 15.132711 / 8.074308 (7.058403) | 14.585530 / 10.191392 (4.394138) | 0.163454 / 0.680424 (-0.516970) | 0.017442 / 0.534201 (-0.516759) | 0.421746 / 0.579283 (-0.157537) | 0.425412 / 0.434364 (-0.008952) | 0.499178 / 0.540337 (-0.041159) | 0.595458 / 1.386936 (-0.791478) |

![](https://cml.dev/watermark.png#ab77e58cd32413f4ef4828134a2470ebd53bb542 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007980 / 0.011353 (-0.003373) | 0.005414 / 0.011008 (-0.005594) | 0.099226 / 0.038508 (0.060718) | 0.035442 / 0.023109 (0.012332) | 0.304851 / 0.275898 (0.028952) | 0.337144 / 0.323480 (0.013664) | 0.006162 / 0.007986 (-0.001823) | 0.004151 / 0.004328 (-0.000177) | 0.074708 / 0.004250 (0.070458) | 0.049690 / 0.037052 (0.012638) | 0.307658 / 0.258489 (0.049168) | 0.358472 / 0.293841 (0.064631) | 0.037181 / 0.128546 (-0.091365) | 0.012259 / 0.075646 (-0.063387) | 0.335426 / 0.419271 (-0.083846) | 0.050790 / 0.043533 (0.007257) | 0.301715 / 0.255139 (0.046576) | 0.320834 / 0.283200 (0.037634) | 0.102357 / 0.141683 (-0.039326) | 1.454750 / 1.452155 (0.002596) | 1.571994 / 1.492716 (0.079278) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.218708 / 0.018006 (0.200702) | 0.444391 / 0.000490 (0.443901) | 0.005717 / 0.000200 (0.005517) | 0.000089 / 0.000054 (0.000035) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028017 / 0.037411 (-0.009395) | 0.112753 / 0.014526 (0.098227) | 0.121003 / 0.176557 (-0.055554) | 0.181085 / 0.737135 (-0.556050) | 0.127211 / 0.296338 (-0.169127) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.400803 / 0.215209 (0.185594) | 4.007315 / 2.077655 (1.929660) | 1.826911 / 1.504120 (0.322791) | 1.637799 / 1.541195 (0.096605) | 1.699754 / 1.468490 (0.231264) | 0.709413 / 4.584777 (-3.875364) | 4.008904 / 3.745712 (0.263192) | 3.916540 / 5.269862 (-1.353322) | 1.902102 / 4.565676 (-2.663575) | 0.089048 / 0.424275 (-0.335227) | 0.012763 / 0.007607 (0.005155) | 0.498957 / 0.226044 (0.272913) | 4.979865 / 2.268929 (2.710937) | 2.301987 / 55.444624 (-53.142637) | 1.929404 / 6.876477 (-4.947073) | 2.107839 / 2.142072 (-0.034233) | 0.857253 / 4.805227 (-3.947974) | 0.171935 / 6.500664 (-6.328729) | 0.066753 / 0.075469 (-0.008716) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.186811 / 1.841788 (-0.654977) | 15.866319 / 8.074308 (7.792011) | 14.738555 / 10.191392 (4.547163) | 0.142879 / 0.680424 (-0.537544) | 0.017679 / 0.534201 (-0.516522) | 0.422840 / 0.579283 (-0.156443) | 0.450307 / 0.434364 (0.015943) | 0.491802 / 0.540337 (-0.048536) | 0.588837 / 1.386936 (-0.798099) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007659 / 0.011353 (-0.003694) | 0.005331 / 0.011008 (-0.005678) | 0.075360 / 0.038508 (0.036852) | 0.034011 / 0.023109 (0.010902) | 0.354488 / 0.275898 (0.078590) | 0.401781 / 0.323480 (0.078301) | 0.005806 / 0.007986 (-0.002179) | 0.004029 / 0.004328 (-0.000300) | 0.073822 / 0.004250 (0.069572) | 0.049067 / 0.037052 (0.012015) | 0.364483 / 0.258489 (0.105994) | 0.405637 / 0.293841 (0.111796) | 0.037166 / 0.128546 (-0.091380) | 0.012397 / 0.075646 (-0.063249) | 0.087346 / 0.419271 (-0.331926) | 0.050888 / 0.043533 (0.007355) | 0.334796 / 0.255139 (0.079657) | 0.387681 / 0.283200 (0.104481) | 0.105056 / 0.141683 (-0.036627) | 1.471630 / 1.452155 (0.019475) | 1.554764 / 1.492716 (0.062047) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.231825 / 0.018006 (0.213819) | 0.449746 / 0.000490 (0.449256) | 0.000888 / 0.000200 (0.000688) | 0.000078 / 0.000054 (0.000023) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030363 / 0.037411 (-0.007049) | 0.115234 / 0.014526 (0.100708) | 0.123005 / 0.176557 (-0.053551) | 0.172772 / 0.737135 (-0.564363) | 0.127818 / 0.296338 (-0.168520) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.425761 / 0.215209 (0.210552) | 4.237950 / 2.077655 (2.160295) | 1.992045 / 1.504120 (0.487925) | 1.801622 / 1.541195 (0.260427) | 1.918477 / 1.468490 (0.449987) | 0.722730 / 4.584777 (-3.862047) | 4.015968 / 3.745712 (0.270256) | 3.720412 / 5.269862 (-1.549450) | 1.763111 / 4.565676 (-2.802566) | 0.089041 / 0.424275 (-0.335234) | 0.012608 / 0.007607 (0.005001) | 0.522645 / 0.226044 (0.296601) | 5.227108 / 2.268929 (2.958180) | 2.444714 / 55.444624 (-52.999910) | 2.109745 / 6.876477 (-4.766732) | 2.194042 / 2.142072 (0.051969) | 0.871781 / 4.805227 (-3.933447) | 0.173149 / 6.500664 (-6.327515) | 0.066192 / 0.075469 (-0.009277) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.312051 / 1.841788 (-0.529737) | 16.024315 / 8.074308 (7.950007) | 15.123823 / 10.191392 (4.932431) | 0.163997 / 0.680424 (-0.516427) | 0.017595 / 0.534201 (-0.516606) | 0.426379 / 0.579283 (-0.152904) | 0.467709 / 0.434364 (0.033345) | 0.498308 / 0.540337 (-0.042030) | 0.591426 / 1.386936 (-0.795510) |

![](https://cml.dev/watermark.png#13488cc110b67090289794f48d5c84a4fd0c063a "CML watermark")

CI is failing due to unrelated issues, hopefully https://github.com/huggingface/datasets/pull/5642 fixes it

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006478 / 0.011353 (-0.004875) | 0.004347 / 0.011008 (-0.006661) | 0.097103 / 0.038508 (0.058595) | 0.027650 / 0.023109 (0.004541) | 0.372355 / 0.275898 (0.096457) | 0.408794 / 0.323480 (0.085314) | 0.005034 / 0.007986 (-0.002952) | 0.003252 / 0.004328 (-0.001076) | 0.074068 / 0.004250 (0.069818) | 0.035542 / 0.037052 (-0.001510) | 0.367392 / 0.258489 (0.108903) | 0.409644 / 0.293841 (0.115803) | 0.031745 / 0.128546 (-0.096801) | 0.011501 / 0.075646 (-0.064145) | 0.323355 / 0.419271 (-0.095917) | 0.043065 / 0.043533 (-0.000467) | 0.377313 / 0.255139 (0.122174) | 0.395326 / 0.283200 (0.112127) | 0.087101 / 0.141683 (-0.054582) | 1.461228 / 1.452155 (0.009073) | 1.529413 / 1.492716 (0.036696) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.199245 / 0.018006 (0.181239) | 0.409978 / 0.000490 (0.409488) | 0.002655 / 0.000200 (0.002455) | 0.000070 / 0.000054 (0.000016) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023903 / 0.037411 (-0.013508) | 0.097855 / 0.014526 (0.083330) | 0.106405 / 0.176557 (-0.070152) | 0.166889 / 0.737135 (-0.570247) | 0.110256 / 0.296338 (-0.186082) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.440351 / 0.215209 (0.225142) | 4.382848 / 2.077655 (2.305194) | 2.049602 / 1.504120 (0.545482) | 1.824638 / 1.541195 (0.283443) | 1.850519 / 1.468490 (0.382029) | 0.702652 / 4.584777 (-3.882125) | 3.394571 / 3.745712 (-0.351141) | 1.940608 / 5.269862 (-3.329254) | 1.263961 / 4.565676 (-3.301716) | 0.083985 / 0.424275 (-0.340290) | 0.013046 / 0.007607 (0.005439) | 0.538272 / 0.226044 (0.312228) | 5.407563 / 2.268929 (3.138634) | 2.519207 / 55.444624 (-52.925418) | 2.153379 / 6.876477 (-4.723098) | 2.394512 / 2.142072 (0.252439) | 0.812840 / 4.805227 (-3.992387) | 0.152868 / 6.500664 (-6.347796) | 0.067823 / 0.075469 (-0.007646) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.220031 / 1.841788 (-0.621757) | 13.781237 / 8.074308 (5.706929) | 14.203975 / 10.191392 (4.012583) | 0.141077 / 0.680424 (-0.539347) | 0.016518 / 0.534201 (-0.517682) | 0.379079 / 0.579283 (-0.200204) | 0.378916 / 0.434364 (-0.055448) | 0.434589 / 0.540337 (-0.105749) | 0.521129 / 1.386936 (-0.865807) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006997 / 0.011353 (-0.004356) | 0.004599 / 0.011008 (-0.006410) | 0.078700 / 0.038508 (0.040192) | 0.027902 / 0.023109 (0.004793) | 0.344406 / 0.275898 (0.068508) | 0.392918 / 0.323480 (0.069438) | 0.005175 / 0.007986 (-0.002811) | 0.004755 / 0.004328 (0.000427) | 0.077707 / 0.004250 (0.073457) | 0.039409 / 0.037052 (0.002357) | 0.343250 / 0.258489 (0.084761) | 0.405544 / 0.293841 (0.111703) | 0.032286 / 0.128546 (-0.096260) | 0.011674 / 0.075646 (-0.063972) | 0.087633 / 0.419271 (-0.331639) | 0.043346 / 0.043533 (-0.000186) | 0.355076 / 0.255139 (0.099937) | 0.382155 / 0.283200 (0.098955) | 0.090914 / 0.141683 (-0.050769) | 1.518369 / 1.452155 (0.066215) | 1.583530 / 1.492716 (0.090813) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.160369 / 0.018006 (0.142362) | 0.406844 / 0.000490 (0.406354) | 0.002651 / 0.000200 (0.002451) | 0.000080 / 0.000054 (0.000025) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025295 / 0.037411 (-0.012116) | 0.101490 / 0.014526 (0.086964) | 0.108825 / 0.176557 (-0.067732) | 0.161673 / 0.737135 (-0.575462) | 0.113610 / 0.296338 (-0.182729) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.443514 / 0.215209 (0.228305) | 4.436722 / 2.077655 (2.359067) | 2.144008 / 1.504120 (0.639888) | 2.005324 / 1.541195 (0.464129) | 2.123356 / 1.468490 (0.654866) | 0.697217 / 4.584777 (-3.887560) | 3.401105 / 3.745712 (-0.344607) | 1.874621 / 5.269862 (-3.395240) | 1.165069 / 4.565676 (-3.400608) | 0.082799 / 0.424275 (-0.341476) | 0.012806 / 0.007607 (0.005199) | 0.542688 / 0.226044 (0.316644) | 5.420963 / 2.268929 (3.152034) | 2.579034 / 55.444624 (-52.865590) | 2.240201 / 6.876477 (-4.636276) | 2.261309 / 2.142072 (0.119237) | 0.800246 / 4.805227 (-4.004981) | 0.150380 / 6.500664 (-6.350285) | 0.066880 / 0.075469 (-0.008589) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.281721 / 1.841788 (-0.560067) | 13.906361 / 8.074308 (5.832053) | 14.135336 / 10.191392 (3.943944) | 0.128865 / 0.680424 (-0.551559) | 0.016452 / 0.534201 (-0.517749) | 0.373563 / 0.579283 (-0.205720) | 0.385321 / 0.434364 (-0.049043) | 0.437198 / 0.540337 (-0.103139) | 0.530720 / 1.386936 (-0.856216) |

![](https://cml.dev/watermark.png#e2f8e17f3c8f8d0cb77a4c566a78e31fab47108c "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008099 / 0.011353 (-0.003254) | 0.005093 / 0.011008 (-0.005916) | 0.106258 / 0.038508 (0.067750) | 0.037051 / 0.023109 (0.013942) | 0.347960 / 0.275898 (0.072062) | 0.370849 / 0.323480 (0.047369) | 0.006122 / 0.007986 (-0.001863) | 0.004094 / 0.004328 (-0.000235) | 0.079549 / 0.004250 (0.075299) | 0.046563 / 0.037052 (0.009510) | 0.332735 / 0.258489 (0.074246) | 0.417061 / 0.293841 (0.123220) | 0.038105 / 0.128546 (-0.090441) | 0.011886 / 0.075646 (-0.063760) | 0.342103 / 0.419271 (-0.077169) | 0.053233 / 0.043533 (0.009700) | 0.344754 / 0.255139 (0.089615) | 0.355354 / 0.283200 (0.072155) | 0.101059 / 0.141683 (-0.040624) | 1.518561 / 1.452155 (0.066406) | 1.558652 / 1.492716 (0.065935) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.225919 / 0.018006 (0.207913) | 0.518539 / 0.000490 (0.518049) | 0.006230 / 0.000200 (0.006030) | 0.000124 / 0.000054 (0.000070) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026782 / 0.037411 (-0.010629) | 0.108457 / 0.014526 (0.093931) | 0.125203 / 0.176557 (-0.051353) | 0.175726 / 0.737135 (-0.561409) | 0.127051 / 0.296338 (-0.169287) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.416427 / 0.215209 (0.201217) | 4.168851 / 2.077655 (2.091196) | 1.962238 / 1.504120 (0.458118) | 1.825224 / 1.541195 (0.284029) | 1.831200 / 1.468490 (0.362710) | 0.765526 / 4.584777 (-3.819250) | 4.303957 / 3.745712 (0.558245) | 2.193467 / 5.269862 (-3.076395) | 1.654605 / 4.565676 (-2.911071) | 0.096709 / 0.424275 (-0.327566) | 0.013792 / 0.007607 (0.006185) | 0.537862 / 0.226044 (0.311818) | 5.152230 / 2.268929 (2.883302) | 2.520938 / 55.444624 (-52.923686) | 2.108422 / 6.876477 (-4.768054) | 2.214220 / 2.142072 (0.072147) | 0.834320 / 4.805227 (-3.970907) | 0.170635 / 6.500664 (-6.330029) | 0.063131 / 0.075469 (-0.012338) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.215767 / 1.841788 (-0.626020) | 15.254781 / 8.074308 (7.180473) | 14.360764 / 10.191392 (4.169372) | 0.172511 / 0.680424 (-0.507913) | 0.020161 / 0.534201 (-0.514040) | 0.426936 / 0.579283 (-0.152347) | 0.438771 / 0.434364 (0.004407) | 0.486973 / 0.540337 (-0.053364) | 0.584238 / 1.386936 (-0.802698) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006777 / 0.011353 (-0.004576) | 0.005304 / 0.011008 (-0.005704) | 0.073717 / 0.038508 (0.035209) | 0.033604 / 0.023109 (0.010494) | 0.340448 / 0.275898 (0.064550) | 0.351861 / 0.323480 (0.028381) | 0.005786 / 0.007986 (-0.002199) | 0.005013 / 0.004328 (0.000685) | 0.071263 / 0.004250 (0.067012) | 0.048189 / 0.037052 (0.011137) | 0.339457 / 0.258489 (0.080968) | 0.384383 / 0.293841 (0.090542) | 0.035563 / 0.128546 (-0.092983) | 0.011509 / 0.075646 (-0.064137) | 0.083722 / 0.419271 (-0.335550) | 0.048886 / 0.043533 (0.005353) | 0.350184 / 0.255139 (0.095045) | 0.361037 / 0.283200 (0.077837) | 0.105191 / 0.141683 (-0.036492) | 1.503247 / 1.452155 (0.051093) | 1.582298 / 1.492716 (0.089581) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.221687 / 0.018006 (0.203681) | 0.466489 / 0.000490 (0.465999) | 0.000484 / 0.000200 (0.000284) | 0.000069 / 0.000054 (0.000015) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027978 / 0.037411 (-0.009434) | 0.119572 / 0.014526 (0.105047) | 0.133530 / 0.176557 (-0.043026) | 0.177892 / 0.737135 (-0.559243) | 0.127045 / 0.296338 (-0.169294) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.430198 / 0.215209 (0.214989) | 4.435512 / 2.077655 (2.357858) | 2.007183 / 1.504120 (0.503063) | 1.799230 / 1.541195 (0.258036) | 1.884750 / 1.468490 (0.416260) | 0.745232 / 4.584777 (-3.839545) | 4.088069 / 3.745712 (0.342357) | 4.114669 / 5.269862 (-1.155193) | 2.374086 / 4.565676 (-2.191590) | 0.089154 / 0.424275 (-0.335121) | 0.012938 / 0.007607 (0.005331) | 0.505954 / 0.226044 (0.279909) | 5.194226 / 2.268929 (2.925298) | 2.487230 / 55.444624 (-52.957394) | 2.163353 / 6.876477 (-4.713124) | 2.177879 / 2.142072 (0.035807) | 0.828728 / 4.805227 (-3.976499) | 0.171157 / 6.500664 (-6.329507) | 0.062883 / 0.075469 (-0.012586) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.275906 / 1.841788 (-0.565882) | 15.235484 / 8.074308 (7.161176) | 14.467396 / 10.191392 (4.276004) | 0.198994 / 0.680424 (-0.481430) | 0.020203 / 0.534201 (-0.513998) | 0.447904 / 0.579283 (-0.131380) | 0.454210 / 0.434364 (0.019846) | 0.528062 / 0.540337 (-0.012275) | 0.619311 / 1.386936 (-0.767625) |

![](https://cml.dev/watermark.png#11cd0f73acbce1d16174f2555e56fda511d5a08b "CML watermark")

`zipfile.is_zipfile` return false positives for some Parquet files. It causes errors when loading certain parquet datasets, where some files are considered ZIP files by `zipfile.is_zipfile`

This is a known issue: https://github.com/python/cpython/issues/72680

At first I wanted to rely only on magic numbers, but then I found that someone contributed a [fix to is_zipfile](https://github.com/python/cpython/pull/5053) - do you think we should use it @albertvillanova or not ?

IMO it's ok to rely on magic numbers only for now, since in streaming mode we've had no issue checking only the magic number so far.

Close https://github.com/huggingface/datasets/issues/5639


Parquet file wrongly recognized as zip prevents loading a dataset

### Describe the bug

When trying to `load_dataset_builder` for `HuggingFaceGECLM/StackExchange_Mar2023`, extraction fails, because parquet file [devops-00000-of-00001-22fe902fd8702892.parquet](https://huggingface.co/datasets/HuggingFaceGECLM/StackExchange_Mar2023/resolve/1f8c9a2ab6f7d0f9ae904b8b922e4384592ae1a5/data/devops-00000-of-00001-22fe902fd8702892.parquet) is wrongly identified by python as being a zip not a parquet.
(Full thread on [Slack](https://huggingface.slack.com/archives/C02V51Q3800/p1678890880803599))

### Steps to reproduce the bug

```python
from datasets import load_dataset_builder

ds = load_dataset_builder("HuggingFaceGECLM/StackExchange_Mar2023")
```

### Expected behavior

Loading the file normally.

### Environment info

- `datasets` version: 2.3.2
- Platform: Linux-5.14.0-1058-oem-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


xPath to implement all operations for Path

enhancement

New feature or request

 I think https://github.com/fsspec/universal_pathlib is the project you are looking for.

`xPath` has the methods often used in dataset scripts, and `mkdir` is not one of them (`dl_manager`'s role is to "interact" with the file system, so using `mkdir` is discouraged).

Right is there a difference between UPath and xPath? Typically is xPath less well implemented compared to Upath, ie missing some implementations of some methods? Or are there methods in xPath that are not implemented with UPath?

`xPath` is an internal component (it doesn't have a leading underscore in the name, but it should) not meant to be used outside of `datasets`, and it's only tested on HTTP URLs, not S3.

Okay I understand that xPath won't support my usecase. What I was perhaps getting to is why not use UPath in `datasets` instead of `xPath` if UPath seems to have strictly more robust implementations.

It seems like `universal_pathlib` does not support `fsspec` URL chaining (`::` is the chaining symbol) and "compression" filesystems (e.g., `zip`), but this is what we need to access and stream files from within an archive (e.g., we want to stream URLs such as this one: `zip://data.parquet::https://www.dummyurl.com/archive.zip`)

### Feature request

Current xPath implementation is a great extension of Path in order to work with remote objects. However some methods such as `mkdir` are not implemented correctly. It should instead rely on `fsspec` methods, instead of defaulting do `Path` methods which only work locally.

### Motivation

I'm using xPath to interact with remote objects.

### Your contribution

I could try to make a PR. I'm a bit unfamiliar with chaining right now.


IterableDataset with_format does not support 'device' keyword for jax

Hi! Yes, only `torch` is currently supported. Unlike `Dataset`, `IterableDataset` is not PyArrow-backed, so we cannot simply call `to_numpy` on the underlying subtables to format them numerically. Instead, we must manually convert examples to (numeric) arrays while preserving consistency with `Dataset`, which is not trivial, so this is still a to-do.

Any plans to support it in the future? Or would streaming dataset be left without support for jax and tensorflow?

### Describe the bug

As seen here: https://huggingface.co/docs/datasets/use_with_jax dataset.with_format() supports the keyword 'device', to put data on a specific device when loaded as jax. However, when called on an IterableDataset, I got the error `TypeError: with_format() got an unexpected keyword argument 'device'`

Looking over the code, it seems IterableDataset support only pytorch and no support for jax device keyword?
https://github.com/huggingface/datasets/blob/fc5c84f36684343bff3e424cb0fd1ac5ecdd66da/src/datasets/iterable_dataset.py#L1029

### Steps to reproduce the bug

1. Load an IterableDataset (tested in streaming mode)
2. Call with_format('jax',device=device)

### Expected behavior

I expect to call `with_format('jax', device=device)`  as per [documentation](https://huggingface.co/docs/datasets/use_with_jax) without error

### Environment info

Tested with installing newest (dev) and also pip release (2.10.1).
- `datasets` version: 2.10.2.dev0
- Platform: Linux-5.15.89+-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- Huggingface_hub version: 0.12.1
- PyArrow version: 11.0.0
- Pandas version: 1.3.5


Fix CI: ignore C901 ("some_func" is to complex) in `ruff`

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006529 / 0.011353 (-0.004824) | 0.004527 / 0.011008 (-0.006481) | 0.098051 / 0.038508 (0.059543) | 0.028058 / 0.023109 (0.004949) | 0.368543 / 0.275898 (0.092645) | 0.397126 / 0.323480 (0.073646) | 0.005072 / 0.007986 (-0.002913) | 0.003377 / 0.004328 (-0.000952) | 0.076867 / 0.004250 (0.072617) | 0.040121 / 0.037052 (0.003069) | 0.373422 / 0.258489 (0.114933) | 0.403969 / 0.293841 (0.110128) | 0.031485 / 0.128546 (-0.097061) | 0.011673 / 0.075646 (-0.063973) | 0.321837 / 0.419271 (-0.097434) | 0.042828 / 0.043533 (-0.000704) | 0.370391 / 0.255139 (0.115252) | 0.391737 / 0.283200 (0.108538) | 0.084764 / 0.141683 (-0.056919) | 1.463114 / 1.452155 (0.010959) | 1.527042 / 1.492716 (0.034325) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.200964 / 0.018006 (0.182958) | 0.403967 / 0.000490 (0.403477) | 0.002439 / 0.000200 (0.002239) | 0.000070 / 0.000054 (0.000016) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023531 / 0.037411 (-0.013880) | 0.097424 / 0.014526 (0.082899) | 0.104854 / 0.176557 (-0.071703) | 0.165682 / 0.737135 (-0.571453) | 0.109416 / 0.296338 (-0.186922) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.431041 / 0.215209 (0.215832) | 4.326039 / 2.077655 (2.248384) | 2.085123 / 1.504120 (0.581003) | 1.922720 / 1.541195 (0.381525) | 2.006608 / 1.468490 (0.538118) | 0.703348 / 4.584777 (-3.881428) | 3.441516 / 3.745712 (-0.304196) | 1.875244 / 5.269862 (-3.394618) | 1.181341 / 4.565676 (-3.384336) | 0.083442 / 0.424275 (-0.340833) | 0.012966 / 0.007607 (0.005359) | 0.536047 / 0.226044 (0.310002) | 5.354856 / 2.268929 (3.085927) | 2.451064 / 55.444624 (-52.993560) | 2.076110 / 6.876477 (-4.800367) | 2.196507 / 2.142072 (0.054435) | 0.811196 / 4.805227 (-3.994032) | 0.152547 / 6.500664 (-6.348118) | 0.067978 / 0.075469 (-0.007491) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.196169 / 1.841788 (-0.645618) | 13.697234 / 8.074308 (5.622926) | 13.966652 / 10.191392 (3.775260) | 0.143735 / 0.680424 (-0.536688) | 0.016484 / 0.534201 (-0.517717) | 0.382349 / 0.579283 (-0.196934) | 0.401507 / 0.434364 (-0.032857) | 0.447297 / 0.540337 (-0.093041) | 0.529779 / 1.386936 (-0.857157) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006698 / 0.011353 (-0.004655) | 0.004608 / 0.011008 (-0.006400) | 0.076220 / 0.038508 (0.037712) | 0.027340 / 0.023109 (0.004231) | 0.344095 / 0.275898 (0.068197) | 0.374715 / 0.323480 (0.051235) | 0.004883 / 0.007986 (-0.003102) | 0.004658 / 0.004328 (0.000330) | 0.075381 / 0.004250 (0.071130) | 0.036099 / 0.037052 (-0.000953) | 0.340382 / 0.258489 (0.081893) | 0.383488 / 0.293841 (0.089647) | 0.031534 / 0.128546 (-0.097012) | 0.011735 / 0.075646 (-0.063912) | 0.085895 / 0.419271 (-0.333377) | 0.042226 / 0.043533 (-0.001306) | 0.340301 / 0.255139 (0.085162) | 0.366079 / 0.283200 (0.082879) | 0.088828 / 0.141683 (-0.052854) | 1.487880 / 1.452155 (0.035725) | 1.561318 / 1.492716 (0.068601) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.226366 / 0.018006 (0.208360) | 0.408934 / 0.000490 (0.408444) | 0.000396 / 0.000200 (0.000196) | 0.000060 / 0.000054 (0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024521 / 0.037411 (-0.012891) | 0.100167 / 0.014526 (0.085641) | 0.106480 / 0.176557 (-0.070077) | 0.156377 / 0.737135 (-0.580758) | 0.111709 / 0.296338 (-0.184630) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.436138 / 0.215209 (0.220928) | 4.370919 / 2.077655 (2.293265) | 2.066402 / 1.504120 (0.562282) | 1.862157 / 1.541195 (0.320962) | 1.920701 / 1.468490 (0.452211) | 0.695517 / 4.584777 (-3.889260) | 3.435558 / 3.745712 (-0.310154) | 1.864000 / 5.269862 (-3.405861) | 1.164134 / 4.565676 (-3.401543) | 0.083006 / 0.424275 (-0.341269) | 0.012751 / 0.007607 (0.005144) | 0.535405 / 0.226044 (0.309360) | 5.368530 / 2.268929 (3.099602) | 2.494197 / 55.444624 (-52.950427) | 2.161370 / 6.876477 (-4.715107) | 2.180345 / 2.142072 (0.038272) | 0.808076 / 4.805227 (-3.997151) | 0.151891 / 6.500664 (-6.348773) | 0.067643 / 0.075469 (-0.007826) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.334245 / 1.841788 (-0.507543) | 14.112805 / 8.074308 (6.038497) | 14.152303 / 10.191392 (3.960911) | 0.153492 / 0.680424 (-0.526932) | 0.016542 / 0.534201 (-0.517659) | 0.376013 / 0.579283 (-0.203270) | 0.386528 / 0.434364 (-0.047836) | 0.436461 / 0.540337 (-0.103876) | 0.519278 / 1.386936 (-0.867658) |

![](https://cml.dev/watermark.png#ce1d1076fc55ac49277398304e551f0b56c3c9e2 "CML watermark")

idk if I should have added this ignore to `ruff` too, but I added :)


Pass custom metadata filename to Image/Audio folders

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5635). All of your documentation changes will be reflected on that endpoint.

I'm not a big fan of this new param - I find assigning metadata files to splits via the `data_files` param cleaner. Also, assuming that the metadata filename is `metadata.json`/`metadata.csv` (I don't think we should allow other names), a user can do `load_dataset("imagefolder", data_dir="data")` to load a dataset with that structure.

@mariosasko  I don't really like this change in it's current state either but passing specific files with `data_files` also looks not quite user-friendly to me. The idea of providing specific parameter for metadata filename seems natural to me but I don't see a way for implementing it without some ugly changes in `load.py` (passing the param to factories and creating metadata patterns on the fly). Why don't you like this parameter?

For context: this PR emerged from the case where users wanted to use different metadata files with the same large set of images without copying directories on disk and it's not possible with `data_files` approach.

edit: ah no, it's possible if one  puts metadata files in different subdirs (so that the filenames can be left the same)

>For context: this PR emerged from the case where users wanted to use different metadata files with the same large set of images without copying directories on disk and it's not possible with data_files approach.
>
>edit: ah no, it's possible if one puts metadata files in different subdirs (so that the filenames can be left the same)

Seems low prio, but one way to address this would be by allowing to pass  "exclude patterns" to `data_files`

This is a quick fix.
Now it requires to pass data via `data_files` parameters and include a required metadata file there and pass its filename as `metadata_filename` parameter.
For example, with the structure like:
```
data
   images_dir/
      im1.jpg
      im2.jpg
      ...
   metadata_dir/
      meta_file1.jsonl
      meta_file2.jsonl
      ...
```
to load data with `metadata_file1.jsonl` do:
```python
ds = load_dataset("imagefolder", data_files=["data/images_dir/**", "data/metadata_dir/meta_file1.jsonl"], metadata_filename="meta_file1.jsonl")
```

Note that if you have multiple splits, metadata file should be specified in each of them in `data_files`, smth like:
```python
data_files={
"train": ["data/train/**", "data/metadata_dir/meta_file1.jsonl"],
"test": ["data/train/**", "data/metadata_dir/meta_file1.jsonl"]
}
```


Not all progress bars are showing up when they should for downloading dataset

Hi!

By default, tqdm has `leave=True` to "keep all traces of the progress bar upon the termination of iteration". However, we use `leave=False` in some places (as of recently), which removes the bar once the iteration is over.

I feel like our TQDM bars are noisy, so I think we should always set `leave=False` and also use the `delay` parameter to display progress bars only for tasks that take time (e.g., more than 3s). What do you think about this? Do you find these bars useful (after the dataset generation is over)?

Hi sorry for the late update. I think the problem still exists despite the `leave` flag

<img width="1105" alt="image" src="https://user-images.githubusercontent.com/110427462/226501615-5b02fb02-fd5f-4eda-b1f7-a7ed6570892d.png">

```
Package                  Version
------------------------ ---------
aiofiles                 22.1.0
aiohttp                  3.8.4
aiosignal                1.3.1
aiosqlite                0.18.0
anyio                    3.6.2
appnope                  0.1.3
argon2-cffi              21.3.0
argon2-cffi-bindings     21.2.0
arrow                    1.2.3
asttokens                2.2.1
async-generator          1.10
async-timeout            4.0.2
attrs                    22.2.0
Babel                    2.12.1
backcall                 0.2.0
beautifulsoup4           4.11.2
bleach                   6.0.0
brotlipy                 0.7.0
certifi                  2022.12.7
cffi                     1.15.1
cfgv                     3.3.1
charset-normalizer       2.1.1
comm                     0.1.2
conda                    22.9.0
conda-package-handling   2.0.2
conda_package_streaming  0.7.0
coverage                 7.2.1
cryptography             38.0.4
datasets                 2.8.0
debugpy                  1.6.6
decorator                5.1.1
defusedxml               0.7.1
dill                     0.3.6
distlib                  0.3.6
distro                   1.4.0
entrypoints              0.4
exceptiongroup           1.1.0
executing                1.2.0
fastjsonschema           2.16.3
filelock                 3.9.0
flaky                    3.7.0
fqdn                     1.5.1
frozenlist               1.3.3
fsspec                   2023.3.0
huggingface-hub          0.10.1
identify                 2.5.18
idna                     3.4
iniconfig                2.0.0
ipykernel                6.12.1
ipyparallel              8.4.1
ipython                  7.32.0
ipython-genutils         0.2.0
ipywidgets               8.0.4
isoduration              20.11.0
jedi                     0.18.2
Jinja2                   3.1.2
json5                    0.9.11
jsonpointer              2.3
jsonschema               4.17.3
jupyter_client           8.0.3
jupyter_core             5.2.0
jupyter-events           0.6.3
jupyter_server           2.4.0
jupyter_server_fileid    0.8.0
jupyter_server_terminals 0.4.4
jupyter_server_ydoc      0.6.1
jupyter-ydoc             0.2.2
jupyterlab               3.6.1
jupyterlab-pygments      0.2.2
jupyterlab_server        2.20.0
jupyterlab-widgets       3.0.5
libmambapy               1.1.0
mamba                    1.1.0
MarkupSafe               2.1.2
matplotlib-inline        0.1.6
mistune                  2.0.5
multidict                6.0.4
multiprocess             0.70.14
nbclassic                0.5.3
nbclient                 0.7.2
nbconvert                7.2.9
nbformat                 5.7.3
nest-asyncio             1.5.6
nodeenv                  1.7.0
notebook                 6.5.3
notebook_shim            0.2.2
numpy                    1.24.2
outcome                  1.2.0
packaging                23.0
pandas                   1.5.3
pandocfilters            1.5.0
parso                    0.8.3
pexpect                  4.8.0
pickleshare              0.7.5
pip                      22.3.1
platformdirs             3.0.0
plotly                   5.13.1
pluggy                   1.0.0
pre-commit               3.1.0
prometheus-client        0.16.0
prompt-toolkit           3.0.38
psutil                   5.9.4
ptyprocess               0.7.0
pure-eval                0.2.2
pyarrow                  11.0.0
pycosat                  0.6.4
pycparser                2.21
Pygments                 2.14.0
pyOpenSSL                22.1.0
pyrsistent               0.19.3
PySocks                  1.7.1
pytest                   7.2.1
pytest-asyncio           0.20.3
pytest-cov               4.0.0
pytest-timeout           2.1.0
python-dateutil          2.8.2
python-json-logger       2.0.7
pytz                     2022.7.1
PyYAML                   6.0
pyzmq                    25.0.0
requests                 2.28.1
responses                0.18.0
rfc3339-validator        0.1.4
rfc3986-validator        0.1.1
ruamel-yaml-conda        0.15.80
Send2Trash               1.8.0
setuptools               65.6.3
simplegeneric            0.8.1
six                      1.16.0
sniffio                  1.3.0
sortedcontainers         2.4.0
soupsieve                2.4
stack-data               0.6.2
tenacity                 8.2.2
terminado                0.17.1
tinycss2                 1.2.1
tomli                    2.0.1
toolz                    0.12.0
tornado                  6.2
tqdm                     4.65.0
traitlets                5.8.1
trio                     0.22.0
typing_extensions        4.5.0
uri-template             1.2.0
urllib3                  1.26.13
virtualenv               20.19.0
wcwidth                  0.2.6
webcolors                1.12
webencodings             0.5.1
websocket-client         1.5.1
wheel                    0.38.4
widgetsnbextension       4.0.5
xxhash                   3.2.0
y-py                     0.5.9
yarl                     1.8.2
ypy-websocket            0.8.2
zstandard                0.19.0
```

Any idea why this is happening? I debugged this to know the tqdm.pbar value is not being updated properly and its not the kernel not sending the comm messages to the IProgress bar

### Describe the bug

During downloading the rotten tomatoes dataset, not all progress bars are displayed properly. This might be related to [this ticket](https://github.com/huggingface/datasets/issues/5117) as it raised the same concern but its not clear if the fix solves this issue too.

ipywidgets

tqdm

### Steps to reproduce the bug

1. Run this line
```
from datasets import load_dataset

rotten_tomatoes = load_dataset("rotten_tomatoes", split="train")
```

### Expected behavior

all progress bars for builder script, metadata, readme, training, validation, and test set

### Environment info

requirements.txt
```
aiofiles==22.1.0
aiohttp==3.8.4
aiosignal==1.3.1
aiosqlite==0.18.0
anyio==3.6.2
appnope==0.1.3
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
arrow==1.2.3
asttokens==2.2.1
async-generator==1.10
async-timeout==4.0.2
attrs==22.2.0
Babel==2.12.1
backcall==0.2.0
beautifulsoup4==4.11.2
bleach==6.0.0
brotlipy @ file:///Users/runner/miniforge3/conda-bld/brotlipy_1666764961872/work
certifi==2022.12.7
cffi @ file:///Users/runner/miniforge3/conda-bld/cffi_1671179414629/work
cfgv==3.3.1
charset-normalizer @ file:///home/conda/feedstock_root/build_artifacts/charset-normalizer_1661170624537/work
comm==0.1.2
conda==22.9.0
conda-package-handling @ file:///home/conda/feedstock_root/build_artifacts/conda-package-handling_1669907009957/work
conda_package_streaming @ file:///home/conda/feedstock_root/build_artifacts/conda-package-streaming_1669733752472/work
coverage==7.2.1
cryptography @ file:///Users/runner/miniforge3/conda-bld/cryptography_1669592251328/work
datasets==2.1.0
debugpy==1.6.6
decorator==5.1.1
defusedxml==0.7.1
dill==0.3.6
distlib==0.3.6
distro==1.4.0
entrypoints==0.4
exceptiongroup==1.1.0
executing==1.2.0
fastjsonschema==2.16.3
filelock==3.9.0
flaky==3.7.0
fqdn==1.5.1
frozenlist==1.3.3
fsspec==2023.3.0
huggingface-hub==0.10.1
identify==2.5.18
idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1663625384323/work
iniconfig==2.0.0
ipykernel==6.12.1
ipyparallel==8.4.1
ipython==7.32.0
ipython-genutils==0.2.0
ipywidgets==8.0.4
isoduration==20.11.0
jedi==0.18.2
Jinja2==3.1.2
json5==0.9.11
jsonpointer==2.3
jsonschema==4.17.3
jupyter-events==0.6.3
jupyter-ydoc==0.2.2
jupyter_client==8.0.3
jupyter_core==5.2.0
jupyter_server==2.4.0
jupyter_server_fileid==0.8.0
jupyter_server_terminals==0.4.4
jupyter_server_ydoc==0.6.1
jupyterlab==3.6.1
jupyterlab-pygments==0.2.2
jupyterlab-widgets==3.0.5
jupyterlab_server==2.20.0
libmambapy @ file:///Users/runner/miniforge3/conda-bld/mamba-split_1671598370072/work/libmambapy
mamba @ file:///Users/runner/miniforge3/conda-bld/mamba-split_1671598370072/work/mamba
MarkupSafe==2.1.2
matplotlib-inline==0.1.6
mistune==2.0.5
multidict==6.0.4
multiprocess==0.70.14
nbclassic==0.5.3
nbclient==0.7.2
nbconvert==7.2.9
nbformat==5.7.3
nest-asyncio==1.5.6
nodeenv==1.7.0
notebook==6.5.3
notebook_shim==0.2.2
numpy==1.24.2
outcome==1.2.0
packaging==23.0
pandas==1.5.3
pandocfilters==1.5.0
parso==0.8.3
pexpect==4.8.0
pickleshare==0.7.5
platformdirs==3.0.0
plotly==5.13.1
pluggy==1.0.0
pre-commit==3.1.0
prometheus-client==0.16.0
prompt-toolkit==3.0.38
psutil==5.9.4
ptyprocess==0.7.0
pure-eval==0.2.2
pyarrow==11.0.0
pycosat @ file:///Users/runner/miniforge3/conda-bld/pycosat_1666836580084/work
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1636257122734/work
Pygments==2.14.0
pyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1665350324128/work
pyrsistent==0.19.3
PySocks @ file:///home/conda/feedstock_root/build_artifacts/pysocks_1661604839144/work
pytest==7.2.1
pytest-asyncio==0.20.3
pytest-cov==4.0.0
pytest-timeout==2.1.0
python-dateutil==2.8.2
python-json-logger==2.0.7
pytz==2022.7.1
PyYAML==6.0
pyzmq==25.0.0
requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1661872987712/work
responses==0.18.0
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
ruamel-yaml-conda @ file:///Users/runner/miniforge3/conda-bld/ruamel_yaml_1666819760545/work
Send2Trash==1.8.0
simplegeneric==0.8.1
six==1.16.0
sniffio==1.3.0
sortedcontainers==2.4.0
soupsieve==2.4
stack-data==0.6.2
tenacity==8.2.2
terminado==0.17.1
tinycss2==1.2.1
tomli==2.0.1
toolz @ file:///home/conda/feedstock_root/build_artifacts/toolz_1657485559105/work
tornado==6.2
tqdm==4.64.1
traitlets==5.8.1
trio==0.22.0
typing_extensions==4.5.0
uri-template==1.2.0
urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1669259737463/work
virtualenv==20.19.0
wcwidth==0.2.6
webcolors==1.12
webencodings==0.5.1
websocket-client==1.5.1
widgetsnbextension==4.0.5
xxhash==3.2.0
y-py==0.5.9
yarl==1.8.2
ypy-websocket==0.8.2
zstandard==0.19.0
```


Cannot import datasets

Okay, the issue was likely caused by mixing `conda` and `pip` usage - I forgot that I have already used `pip` in this environment previously and that it was 'spoiled' because of it. Creating another environment and installing `datasets` by pip with other packages from the `requirements.txt` file solved the problem.

### Describe the bug

Hi,

I cannot even import the library :( I installed it by running:
```
$ conda install datasets
```
Then I realized I should maybe use the huggingface channel, because I encountered the error below, so I ran:
```
$ conda remove datasets
$ conda install -c huggingface datasets
```
Please see 'steps to reproduce the bug' for the specific error, as steps to reproduce is just importing the library

### Steps to reproduce the bug

```
$ python3
Python 3.8.15 (default, Nov 24 2022, 15:19:38)
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import datasets
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jack/.conda/envs/jack_zpp/lib/python3.8/site-packages/datasets/__init__.py", line 33, in <module>
    from .arrow_dataset import Dataset, concatenate_datasets
  File "/home/jack/.conda/envs/jack_zpp/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 59, in <module>
    from .arrow_reader import ArrowReader
  File "/home/jack/.conda/envs/jack_zpp/lib/python3.8/site-packages/datasets/arrow_reader.py", line 27, in <module>
    import pyarrow.parquet as pq
  File "/home/jack/.conda/envs/jack_zpp/lib/python3.8/site-packages/pyarrow/parquet/__init__.py", line 20, in <module>
    from .core import *
  File "/home/jack/.conda/envs/jack_zpp/lib/python3.8/site-packages/pyarrow/parquet/core.py", line 37, in <module>
    from pyarrow._parquet import (ParquetReader, Statistics,  # noqa
ImportError: cannot import name 'FileEncryptionProperties' from 'pyarrow._parquet' (/home/jack/.conda/envs/jack_zpp/lib/python3.8/site-packages/pyarrow/_parquet.cpython-38-x86_64-linux-gnu.so)
```

### Expected behavior

I would expect for the statement `import datasets` to cause no error

### Environment info

Output of `conda list`:
```
# packages in environment at /home/jack/.conda/envs/pbalawender_zpp:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main
_openmp_mutex             5.1                       1_gnu
abseil-cpp                20210324.2           h2531618_0
advertools                0.13.2                   pypi_0    pypi
aiofiles                  0.8.0                    pypi_0    pypi
aiohttp                   3.8.3            py38h5eee18b_0
aiosignal                 1.2.0              pyhd3eb1b0_0
aiosqlite                 0.17.0                   pypi_0    pypi
anyio                     3.6.2                    pypi_0    pypi
aquirdturtle-collapsible-headings 3.1.0                    pypi_0    pypi
argon2-cffi               21.3.0                   pypi_0    pypi
argon2-cffi-bindings      21.2.0                   pypi_0    pypi
arrow                     1.2.3                    pypi_0    pypi
arrow-cpp                 3.0.0            py38h6b21186_4
asttokens                 2.2.0                    pypi_0    pypi
async-timeout             4.0.2            py38h06a4308_0
attrs                     22.1.0           py38h06a4308_0
automat                   22.10.0                  pypi_0    pypi
aws-c-common              0.4.57               he6710b0_1
aws-c-event-stream        0.1.6                h2531618_5
aws-checksums             0.1.9                he6710b0_0
aws-sdk-cpp               1.8.185              hce553d0_0
babel                     2.11.0                   pypi_0    pypi
backcall                  0.2.0              pyhd3eb1b0_0
beautifulsoup4            4.11.1                   pypi_0    pypi
blas                      1.0                         mkl
bleach                    5.0.1                    pypi_0    pypi
boost-cpp                 1.73.0              h27cfd23_11
bottleneck                1.3.5            py38h7deecbd_0
brotli                    1.0.9                h5eee18b_7
brotli-bin                1.0.9                h5eee18b_7
brotlipy                  0.7.0           py38h27cfd23_1003
bzip2                     1.0.8                h7b6447c_0
c-ares                    1.18.1               h7f8727e_0
ca-certificates           2023.01.10           h06a4308_0
certifi                   2022.9.24                pypi_0    pypi
cffi                      1.15.1           py38h5eee18b_3
charset-normalizer        2.1.1                    pypi_0    pypi
click                     8.1.3                    pypi_0    pypi
constantly                15.1.0                   pypi_0    pypi
contourpy                 1.0.6                    pypi_0    pypi
cryptography              38.0.4                   pypi_0    pypi
cssselect                 1.2.0                    pypi_0    pypi
cudatoolkit               10.1.243            h8cb64d8_10    conda-forge
cycler                    0.11.0                   pypi_0    pypi
dacite                    1.6.0                    pypi_0    pypi
dataclasses               0.8                pyh6d0b6a4_7
datasets                  1.18.4                     py_0    huggingface
datetime                  4.7                      pypi_0    pypi
debugpy                   1.6.4                    pypi_0    pypi
decorator                 5.1.1              pyhd3eb1b0_0
defusedxml                0.7.1                    pypi_0    pypi
dill                      0.3.6            py38h06a4308_0
docker-pycreds            0.4.0                    pypi_0    pypi
double-conversion         3.1.5                he6710b0_1
entrypoints               0.4              py38h06a4308_0
executing                 0.8.3              pyhd3eb1b0_0
filelock                  3.8.0                    pypi_0    pypi
flake8                    6.0.0                    pypi_0    pypi
flask                     2.1.3            py38h06a4308_0
flit-core                 3.6.0              pyhd3eb1b0_0
fonttools                 4.38.0                   pypi_0    pypi
fqdn                      1.5.1                    pypi_0    pypi
freetype                  2.12.1               h4a9f257_0
frozenlist                1.3.3            py38h5eee18b_0
fsspec                    2022.11.0        py38h06a4308_0
gensim                    4.2.0                    pypi_0    pypi
gflags                    2.2.2                he6710b0_0
giflib                    5.2.1                h5eee18b_3
gitdb                     4.0.10                   pypi_0    pypi
gitpython                 3.1.30                   pypi_0    pypi
glog                      0.5.0                h2531618_0
grpc-cpp                  1.39.0               hae934f6_5
huggingface-hub           0.11.1                   pypi_0    pypi
huggingface_hub           0.13.1                     py_0    huggingface
hyperlink                 21.0.0                   pypi_0    pypi
icu                       58.2                 he6710b0_3
idna                      3.4              py38h06a4308_0
importlib-metadata        5.1.0                    pypi_0    pypi
importlib_metadata        4.11.3               hd3eb1b0_0
importlib_resources       5.2.0              pyhd3eb1b0_1
incremental               22.10.0                  pypi_0    pypi
intel-openmp              2021.4.0          h06a4308_3561
ipykernel                 6.17.1             pyh210e3f2_0    conda-forge
ipython                   8.7.0                    pypi_0    pypi
ipython-genutils          0.2.0                    pypi_0    pypi
ipywidgets                8.0.2              pyhd8ed1ab_1    conda-forge
isoduration               20.11.0                  pypi_0    pypi
itemadapter               0.7.0                    pypi_0    pypi
itemloaders               1.0.6                    pypi_0    pypi
itsdangerous              2.0.1              pyhd3eb1b0_0
jedi                      0.18.2                   pypi_0    pypi
jinja2                    3.1.2            py38h06a4308_0
jmespath                  1.0.1                    pypi_0    pypi
joblib                    1.2.0                    pypi_0    pypi
jpeg                      9b                   h024ee3a_2
json5                     0.9.10                   pypi_0    pypi
jsonpickle                3.0.0                    pypi_0    pypi
jsonpointer               2.3                      pypi_0    pypi
jsonschema                4.17.3           py38h06a4308_0
jupyter-core              5.1.0                    pypi_0    pypi
jupyter-events            0.5.0                    pypi_0    pypi
jupyter-server            1.23.3                   pypi_0    pypi
jupyter-server-fileid     0.6.0                    pypi_0    pypi
jupyter-server-ydoc       0.4.0                    pypi_0    pypi
jupyter-ydoc              0.2.2                    pypi_0    pypi
jupyter_client            7.4.9            py38h06a4308_0
jupyter_core              5.2.0            py38h06a4308_0
jupyterlab                3.6.0a4                  pypi_0    pypi
jupyterlab-pygments       0.2.2                    pypi_0    pypi
jupyterlab-server         2.16.3                   pypi_0    pypi
jupyterlab_widgets        3.0.3              pyhd8ed1ab_0    conda-forge
kiwisolver                1.4.4                    pypi_0    pypi
krb5                      1.19.4               h568e23c_0
lcms2                     2.12                 h3be6417_0
ld_impl_linux-64          2.38                 h1181459_1
libboost                  1.73.0              h3ff78a5_11
libbrotlicommon           1.0.9                h5eee18b_7
libbrotlidec              1.0.9                h5eee18b_7
libbrotlienc              1.0.9                h5eee18b_7
libcurl                   7.88.1               h91b91d3_0
libedit                   3.1.20221030         h5eee18b_0
libev                     4.33                 h7f8727e_1
libevent                  2.1.12               h8f2d780_0
libffi                    3.4.2                h6a678d5_6
libgcc-ng                 11.2.0               h1234567_1
libgomp                   11.2.0               h1234567_1
libnghttp2                1.46.0               hce63b2e_0
libpng                    1.6.39               h5eee18b_0
libprotobuf               3.17.2               h4ff587b_1
libsodium                 1.0.18               h7b6447c_0
libssh2                   1.10.0               h8f2d780_0
libstdcxx-ng              11.2.0               h1234567_1
libthrift                 0.14.2               hcc01f38_0
libtiff                   4.1.0                h2733197_1
libuv                     1.44.2               h5eee18b_0
libwebp                   1.2.0                h89dd481_0
lz4-c                     1.9.4                h6a678d5_0
markupsafe                2.1.1            py38h7f8727e_0
matplotlib                3.6.2                    pypi_0    pypi
matplotlib-inline         0.1.6            py38h06a4308_0
mccabe                    0.7.0                    pypi_0    pypi
mistune                   2.0.4                    pypi_0    pypi
mkl                       2021.4.0           h06a4308_640
mkl-service               2.4.0            py38h7f8727e_0
mkl_fft                   1.3.1            py38hd3c417c_0
mkl_random                1.2.2            py38h51133e4_0
morfeusz2                 1.99.6                   pypi_0    pypi
multidict                 6.0.2            py38h5eee18b_0
multiprocess              0.70.14          py38h06a4308_0
nbclassic                 0.4.8                    pypi_0    pypi
nbclient                  0.7.2                    pypi_0    pypi
nbconvert                 7.2.5                    pypi_0    pypi
nbformat                  5.7.0            py38h06a4308_0
ncurses                   6.4                  h6a678d5_0
nest-asyncio              1.5.6            py38h06a4308_0
ninja                     1.10.2               h06a4308_5
ninja-base                1.10.2               hd09550d_5
notebook                  6.5.2                    pypi_0    pypi
notebook-shim             0.2.2                    pypi_0    pypi
numexpr                   2.8.4            py38he184ba9_0
numpy                     1.23.5           py38h14f4228_0
numpy-base                1.23.5           py38h31eccc5_0
oauthlib                  3.2.2                    pypi_0    pypi
opencv-python             4.6.0.66                 pypi_0    pypi
openssl                   1.1.1t               h7f8727e_0
orc                       1.6.9                ha97a36c_3
packaging                 22.0             py38h06a4308_0
pandas                    1.5.2                    pypi_0    pypi
pandocfilters             1.5.0                    pypi_0    pypi
parsel                    1.7.0                    pypi_0    pypi
parso                     0.8.3              pyhd3eb1b0_0
pathlib                   1.0.1                    pypi_0    pypi
pathtools                 0.1.2                    pypi_0    pypi
pexpect                   4.8.0              pyhd3eb1b0_3
pickleshare               0.7.5           pyhd3eb1b0_1003
pillow                    9.3.0                    pypi_0    pypi
pip                       22.2.2           py38h06a4308_0
pkgutil-resolve-name      1.3.10           py38h06a4308_0
platformdirs              2.5.4                    pypi_0    pypi
prometheus-client         0.15.0                   pypi_0    pypi
promise                   2.3                      pypi_0    pypi
prompt-toolkit            3.0.33                   pypi_0    pypi
protego                   0.2.1                    pypi_0    pypi
protobuf                  4.21.12                  pypi_0    pypi
psutil                    5.9.0            py38h5eee18b_0
ptyprocess                0.7.0              pyhd3eb1b0_2
pure_eval                 0.2.2              pyhd3eb1b0_0
pyarrow                   10.0.1                   pypi_0    pypi
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pycodestyle               2.10.0                   pypi_0    pypi
pycparser                 2.21               pyhd3eb1b0_0
pydispatcher              2.0.6                    pypi_0    pypi
pyflakes                  3.0.1                    pypi_0    pypi
pygments                  2.11.2             pyhd3eb1b0_0
pyopenssl                 22.1.0                   pypi_0    pypi
pyrsistent                0.18.0           py38heee7806_0
pysocks                   1.7.1            py38h06a4308_0
python                    3.8.15               h7a1cb2a_2
python-dateutil           2.8.2              pyhd3eb1b0_0
python-dotenv             0.21.0                   pypi_0    pypi
python-fastjsonschema     2.16.2           py38h06a4308_0
python-json-logger        2.0.4                    pypi_0    pypi
python-xxhash             2.0.2            py38h5eee18b_1
pytorch                   1.7.1           py3.8_cuda10.1.243_cudnn7.6.3_0    pytorch
pytz                      2022.6                   pypi_0    pypi
pyyaml                    6.0              py38h5eee18b_1
pyzmq                     23.2.0           py38h6a678d5_0
queuelib                  1.6.2                    pypi_0    pypi
re2                       2022.04.01           h295c915_0
readline                  8.2                  h5eee18b_0
regex                     2022.10.31               pypi_0    pypi
requests                  2.28.1           py38h06a4308_0
requests-file             1.5.1                    pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
rfc3339-validator         0.1.4                    pypi_0    pypi
rfc3986-validator         0.1.1                    pypi_0    pypi
scikit-learn              1.1.3                    pypi_0    pypi
scipy                     1.9.3                    pypi_0    pypi
scrapy                    2.7.1                    pypi_0    pypi
seaborn                   0.12.1                   pypi_0    pypi
send2trash                1.8.0                    pypi_0    pypi
sentry-sdk                1.12.1                   pypi_0    pypi
service-identity          21.1.0                   pypi_0    pypi
setproctitle              1.3.2                    pypi_0    pypi
setuptools                65.6.3                   pypi_0    pypi
shortuuid                 1.0.11                   pypi_0    pypi
six                       1.16.0             pyhd3eb1b0_1
smart-open                6.2.0                    pypi_0    pypi
smmap                     5.0.0                    pypi_0    pypi
snappy                    1.1.9                h295c915_0
sniffio                   1.3.0                    pypi_0    pypi
soupsieve                 2.3.2.post1              pypi_0    pypi
sqlite                    3.40.1               h5082296_0
stack-data                0.6.2                    pypi_0    pypi
stack_data                0.2.0              pyhd3eb1b0_0
terminado                 0.17.0                   pypi_0    pypi
threadpoolctl             3.1.0                    pypi_0    pypi
tinycss2                  1.2.1                    pypi_0    pypi
tk                        8.6.12               h1ccaba5_0
tldextract                3.4.0                    pypi_0    pypi
tokenizers                0.13.2                   pypi_0    pypi
tomli                     2.0.1                    pypi_0    pypi
torchvision               0.8.2                py38_cu101    pytorch
tornado                   6.2              py38h5eee18b_0
tqdm                      4.64.1           py38h06a4308_0
traitlets                 5.6.0                    pypi_0    pypi
transformers              4.25.1                   pypi_0    pypi
tweepy                    4.12.1                   pypi_0    pypi
twisted                   22.10.0                  pypi_0    pypi
twython                   3.9.1                    pypi_0    pypi
typing-extensions         4.4.0            py38h06a4308_0
typing_extensions         4.4.0            py38h06a4308_0
uri-template              1.2.0                    pypi_0    pypi
uriparser                 0.9.3                he6710b0_1
urllib3                   1.26.13                  pypi_0    pypi
utf8proc                  2.6.1                h27cfd23_0
w3lib                     2.1.0                    pypi_0    pypi
wandb                     0.13.7                   pypi_0    pypi
wcwidth                   0.2.5              pyhd3eb1b0_0
webcolors                 1.12                     pypi_0    pypi
webencodings              0.5.1                    pypi_0    pypi
websocket-client          1.4.2                    pypi_0    pypi
werkzeug                  2.2.2            py38h06a4308_0
wheel                     0.38.4           py38h06a4308_0
widgetsnbextension        4.0.3            py38h06a4308_0
xxhash                    0.8.0                h7f8727e_3
xz                        5.2.10               h5eee18b_1
y-py                      0.5.4                    pypi_0    pypi
yaml                      0.2.5                h7b6447c_0
yarl                      1.8.1            py38h5eee18b_0
ypy-websocket             0.5.0                    pypi_0    pypi
zeromq                    4.3.4                h2531618_0
zipp                      3.11.0           py38h06a4308_0
zlib                      1.2.13               h5eee18b_0
zope-interface            5.5.2                    pypi_0    pypi
zstd                      1.4.9                haebb681_0
```


Dataset cannot convert too large dictionnary

Answered on the forum:

> To fix the overflow error, we need to merge [support LargeListArray in pyarrow by xwwwwww · Pull Request #4800 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/pull/4800), which adds support for the large lists. However, before merging it, we need to come up with a cleaner API for large lists. I hope to find some time to address this before Datasets 3.0.

### Describe the bug

Hello everyone!

I tried to build a new dataset with the command "dict_valid = datasets.Dataset.from_dict({'input_values': values_array})".
However, I have a very large dataset (~400Go) and it seems that dataset cannot handle this.

Indeed, I can create the dataset until a certain size of my dictionnary, and then I have the error "OverflowError: Python int too large to convert to C long".

Do you know how to solve this problem?
Unfortunately I cannot give a reproductible code because I cannot share a so large file, but you can find the code below (it's a test on only a part of the validation data ~10Go, but it's already the case).

Thank you!

### Steps to reproduce the bug

SAVE_DIR = './data/'
features = h5py.File(SAVE_DIR+'features.hdf5','r')

valid_data = features["validation"]["data/features"]

v_array_values = [np.float32(item[()]) for item in valid_data.values()]
for i in range(len(v_array_values)):
    v_array_values[i] = v_array_values[i].round(decimals=5)

dict_valid = datasets.Dataset.from_dict({'input_values': v_array_values})

### Expected behavior

The code is expected to give me a Huggingface dataset.

### Environment info

python: 3.8.15
numpy: 1.22.3
datasets: 2.3.2
pyarrow: 8.0.0


Custom split names

New feature or request

Hi!

You can also use names other than "train", "validation"  and "test". As an example, check the [script](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0/blob/e095840f23f3dffc1056c078c2f9320dad9ca74d/common_voice_11_0.py#L139) of the Common Voice 11 dataset.

### Feature request

Hi,
I participated in multiple NLP tasks where there are more than just train, test, validation splits, there could be multiple validation sets or test sets. But it seems currently only those mentioned three splits supported. It would be nice to have the support for more splits on the hub. (currently i can have more splits when I am loading datasets from urls, but not hub)

### Motivation

Easier access to more splits

### Your contribution

No


adds early exit if url is `PathLike`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5630). All of your documentation changes will be reflected on that endpoint.

Closes #4864

Should fix errors thrown when attempting to load `json` dataset using `pathlib.Path` in `data_files` argument.


load_dataset gives "403" error when using Financial phrasebank

Hi! You seem to be using an outdated version of `datasets` that downloads the older script version. To avoid the error, you can either pass `revision="main"` to `load_dataset` (this can fail if a script uses newer features of the lib) or update your installation with `pip install -U datasets` (better solution).

When I try to load this dataset, I receive the following error:

ConnectionError: Couldn't reach https://www.researchgate.net/profile/Pekka_Malo/publication/251231364_FinancialPhraseBank-v10/data/0c96051eee4fb1d56e000000/FinancialPhraseBank-v10.zip (error 403)

Has this been seen before? Thanks. The website loads when I try to access it manually.


add kwargs to index search

This PR proposes to add kwargs to index search methods.

This is particularly useful for setting the timeout of a query on elasticsearch.

A typical use case would be:
```python
dset.add_elasticsearch_index("filename", es_client=es_client)
scores, examples = dset.get_nearest_examples("filename", "my_name-train_29", request_timeout=60)
```


Unable to load AutoTrain-generated dataset from the hub

The AutoTrain format is not supported right now. I think it would require a dedicated dataset builder

Okay, good to know. Thanks for the reply. For now I will just have to
manage the split manually before training, because I can’t find any way of
pulling out file indices or file names from the autogenerated split. The
file names field of the image dataset (loaded directly from arrow file) is
missing, just fyi (for anyone else this might be relevant too).

On Fri, Mar 10, 2023 at 7:02 PM Quentin Lhoest ***@***.***>
wrote:

> The AutoTrain format is not supported right now. I think it would require
> a dedicated dataset builder
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/5627#issuecomment-1464734308>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACBJ4F5A353MCZ76OGRJ6CTW3PFI7ANCNFSM6AAAAAAVWXNUTE>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>

### Describe the bug

DatasetGenerationError: An error occurred while generating the dataset -> ValueError: Couldn't cast ... because column names don't match
```
ValueError: Couldn't cast
_data_files: list<item: struct<filename: string>>
  child 0, item: struct<filename: string>
      child 0, filename: string
_fingerprint: string
_format_columns: list<item: string>
  child 0, item: string
_format_kwargs: struct<>
_format_type: null
_indexes: struct<>
_output_all_columns: bool
_split: null
to
{'citation': Value(dtype='string', id=None), 'description': Value(dtype='string', id=None), 'features': {'image': {'_type': Value(dtype='string', id=None)}, 'target': {'names': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), '_type': Value(dtype='string', id=None)}}, 'homepage': Value(dtype='string', id=None), 'license': Value(dtype='string', id=None), 'splits': {'train': {'name': Value(dtype='string', id=None), 'num_bytes': Value(dtype='int64', id=None), 'num_examples': Value(dtype='int64', id=None), 'dataset_name': Value(dtype='null', id=None)}}}
because column names don't match
```

### Steps to reproduce the bug

Steps to reproduce:

1. `pip install datasets==2.10.1`
2. Attempt to load (private dataset). Note that I'm authenticated via ` huggingface-cli login`

```
from datasets import load_dataset

# load dataset
dataset = "ijmiller2/autotrain-data-betterbin-vision-10000"
dataset = load_dataset(dataset)
```

Here's the full traceback:

```Downloading and preparing dataset json/ijmiller2--autotrain-data-betterbin-vision-10000 to /Users/ian/.cache/huggingface/datasets/ijmiller2___json/ijmiller2--autotrain-data-betterbin-vision-10000-2eae034a9ff8a1a9/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51...
Downloading data files: 100%|███████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 2383.80it/s]
Extracting data files: 100%|█████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 505.95it/s]

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File ~/anaconda3/envs/betterbin/lib/python3.8/site-packages/datasets/builder.py:1874, in ArrowBasedBuilder._prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
   1868     writer = writer_class(
   1869         features=writer._features,
   1870         path=fpath.replace("SSSSS", f"{shard_id:05d}").replace("JJJJJ", f"{job_id:05d}"),
   1871         storage_options=self._fs.storage_options,
   1872         embed_local_files=embed_local_files,
   1873     )
-> 1874 writer.write_table(table)
   1875 num_examples_progress_update += len(table)

File ~/anaconda3/envs/betterbin/lib/python3.8/site-packages/datasets/arrow_writer.py:568, in ArrowWriter.write_table(self, pa_table, writer_batch_size)
    567 pa_table = pa_table.combine_chunks()
--> 568 pa_table = table_cast(pa_table, self._schema)
    569 if self.embed_local_files:

File ~/anaconda3/envs/betterbin/lib/python3.8/site-packages/datasets/table.py:2312, in table_cast(table, schema)
   2311 if table.schema != schema:
-> 2312     return cast_table_to_schema(table, schema)
   2313 elif table.schema.metadata != schema.metadata:

File ~/anaconda3/envs/betterbin/lib/python3.8/site-packages/datasets/table.py:2270, in cast_table_to_schema(table, schema)
   2269 if sorted(table.column_names) != sorted(features):
-> 2270     raise ValueError(f"Couldn't cast\n{table.schema}\nto\n{features}\nbecause column names don't match")
   2271 arrays = [cast_array_to_feature(table[name], feature) for name, feature in features.items()]

ValueError: Couldn't cast
_data_files: list<item: struct<filename: string>>
  child 0, item: struct<filename: string>
      child 0, filename: string
_fingerprint: string
_format_columns: list<item: string>
  child 0, item: string
_format_kwargs: struct<>
_format_type: null
_indexes: struct<>
_output_all_columns: bool
_split: null
to
{'citation': Value(dtype='string', id=None), 'description': Value(dtype='string', id=None), 'features': {'image': {'_type': Value(dtype='string', id=None)}, 'target': {'names': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), '_type': Value(dtype='string', id=None)}}, 'homepage': Value(dtype='string', id=None), 'license': Value(dtype='string', id=None), 'splits': {'train': {'name': Value(dtype='string', id=None), 'num_bytes': Value(dtype='int64', id=None), 'num_examples': Value(dtype='int64', id=None), 'dataset_name': Value(dtype='null', id=None)}}}
because column names don't match

The above exception was the direct cause of the following exception:

DatasetGenerationError                    Traceback (most recent call last)
Input In [8], in <cell line: 6>()
      4 # load dataset
      5 dataset = "ijmiller2/autotrain-data-betterbin-vision-10000"
----> 6 dataset = load_dataset(dataset)

File ~/anaconda3/envs/betterbin/lib/python3.8/site-packages/datasets/load.py:1782, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, num_proc, **config_kwargs)
   1779 try_from_hf_gcs = path not in _PACKAGED_DATASETS_MODULES
   1781 # Download and prepare data
-> 1782 builder_instance.download_and_prepare(
   1783     download_config=download_config,
   1784     download_mode=download_mode,
   1785     verification_mode=verification_mode,
   1786     try_from_hf_gcs=try_from_hf_gcs,
   1787     num_proc=num_proc,
   1788 )
   1790 # Build dataset for splits
   1791 keep_in_memory = (
   1792     keep_in_memory if keep_in_memory is not None else is_small_dataset(builder_instance.info.dataset_size)
   1793 )

File ~/anaconda3/envs/betterbin/lib/python3.8/site-packages/datasets/builder.py:872, in DatasetBuilder.download_and_prepare(self, output_dir, download_config, download_mode, verification_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, file_format, max_shard_size, num_proc, storage_options, **download_and_prepare_kwargs)
    870     if num_proc is not None:
    871         prepare_split_kwargs["num_proc"] = num_proc
--> 872     self._download_and_prepare(
    873         dl_manager=dl_manager,
    874         verification_mode=verification_mode,
    875         **prepare_split_kwargs,
    876         **download_and_prepare_kwargs,
    877     )
    878 # Sync info
    879 self.info.dataset_size = sum(split.num_bytes for split in self.info.splits.values())

File ~/anaconda3/envs/betterbin/lib/python3.8/site-packages/datasets/builder.py:967, in DatasetBuilder._download_and_prepare(self, dl_manager, verification_mode, **prepare_split_kwargs)
    963 split_dict.add(split_generator.split_info)
    965 try:
    966     # Prepare split will record examples associated to the split
--> 967     self._prepare_split(split_generator, **prepare_split_kwargs)
    968 except OSError as e:
    969     raise OSError(
    970         "Cannot find data file. "
    971         + (self.manual_download_instructions or "")
    972         + "\nOriginal error:\n"
    973         + str(e)
    974     ) from None

File ~/anaconda3/envs/betterbin/lib/python3.8/site-packages/datasets/builder.py:1749, in ArrowBasedBuilder._prepare_split(self, split_generator, file_format, num_proc, max_shard_size)
   1747 job_id = 0
   1748 with pbar:
-> 1749     for job_id, done, content in self._prepare_split_single(
   1750         gen_kwargs=gen_kwargs, job_id=job_id, **_prepare_split_args
   1751     ):
   1752         if done:
   1753             result = content

File ~/anaconda3/envs/betterbin/lib/python3.8/site-packages/datasets/builder.py:1892, in ArrowBasedBuilder._prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
   1890     if isinstance(e, SchemaInferenceError) and e.__context__ is not None:
   1891         e = e.__context__
-> 1892     raise DatasetGenerationError("An error occurred while generating the dataset") from e
   1894 yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_shards, shard_lengths)

DatasetGenerationError: An error occurred while generating the dataset
```

### Expected behavior

I'm ultimately trying to generate my own performance metrics on validation data (before putting an endpoint into production) and so was hoping to load all or at least the validation subset from the hub.

I'm expecting the `load_dataset()` function to work as shown in the documentation [here](https://huggingface.co/docs/datasets/loading#hugging-face-hub):

```python
dataset = load_dataset(
  "lhoestq/custom_squad",
  revision="main"  # tag name, or branch name, or commit hash
)
```

### Environment info

- `datasets` version: 2.10.1
- Platform: macOS-13.2.1-arm64-arm-64bit
- Python version: 3.8.13
- PyArrow version: 9.0.0
- Pandas version: 1.4.4