Use yaml instead of get data patterns when possible

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006829 / 0.011353 (-0.004524) | 0.004535 / 0.011008 (-0.006473) | 0.085255 / 0.038508 (0.046747) | 0.080861 / 0.023109 (0.057752) | 0.366023 / 0.275898 (0.090125) | 0.403095 / 0.323480 (0.079615) | 0.005615 / 0.007986 (-0.002370) | 0.003830 / 0.004328 (-0.000498) | 0.064502 / 0.004250 (0.060251) | 0.053916 / 0.037052 (0.016863) | 0.366010 / 0.258489 (0.107521) | 0.414565 / 0.293841 (0.120724) | 0.031500 / 0.128546 (-0.097046) | 0.009252 / 0.075646 (-0.066394) | 0.289584 / 0.419271 (-0.129688) | 0.052984 / 0.043533 (0.009451) | 0.352626 / 0.255139 (0.097487) | 0.390964 / 0.283200 (0.107764) | 0.025118 / 0.141683 (-0.116565) | 1.462316 / 1.452155 (0.010161) | 1.565682 / 1.492716 (0.072966) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.294432 / 0.018006 (0.276426) | 0.618366 / 0.000490 (0.617876) | 0.003270 / 0.000200 (0.003071) | 0.000081 / 0.000054 (0.000027) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031194 / 0.037411 (-0.006217) | 0.088892 / 0.014526 (0.074366) | 0.102580 / 0.176557 (-0.073977) | 0.159449 / 0.737135 (-0.577686) | 0.104434 / 0.296338 (-0.191905) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.385690 / 0.215209 (0.170481) | 3.832782 / 2.077655 (1.755128) | 1.862521 / 1.504120 (0.358401) | 1.685674 / 1.541195 (0.144479) | 1.724984 / 1.468490 (0.256494) | 0.483700 / 4.584777 (-4.101077) | 3.664154 / 3.745712 (-0.081558) | 3.323023 / 5.269862 (-1.946839) | 2.055958 / 4.565676 (-2.509718) | 0.056990 / 0.424275 (-0.367285) | 0.007674 / 0.007607 (0.000067) | 0.460642 / 0.226044 (0.234598) | 4.609964 / 2.268929 (2.341036) | 2.434868 / 55.444624 (-53.009756) | 2.003347 / 6.876477 (-4.873130) | 2.209520 / 2.142072 (0.067448) | 0.629363 / 4.805227 (-4.175864) | 0.135434 / 6.500664 (-6.365230) | 0.060498 / 0.075469 (-0.014971) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.253917 / 1.841788 (-0.587870) | 19.988953 / 8.074308 (11.914645) | 14.353739 / 10.191392 (4.162347) | 0.165987 / 0.680424 (-0.514437) | 0.018299 / 0.534201 (-0.515902) | 0.395532 / 0.579283 (-0.183751) | 0.418708 / 0.434364 (-0.015656) | 0.460865 / 0.540337 (-0.079472) | 0.633925 / 1.386936 (-0.753011) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006631 / 0.011353 (-0.004722) | 0.004109 / 0.011008 (-0.006899) | 0.065003 / 0.038508 (0.026495) | 0.080407 / 0.023109 (0.057297) | 0.362966 / 0.275898 (0.087068) | 0.389727 / 0.323480 (0.066247) | 0.005588 / 0.007986 (-0.002397) | 0.003517 / 0.004328 (-0.000812) | 0.065821 / 0.004250 (0.061570) | 0.057614 / 0.037052 (0.020561) | 0.367422 / 0.258489 (0.108932) | 0.400706 / 0.293841 (0.106865) | 0.031560 / 0.128546 (-0.096986) | 0.008659 / 0.075646 (-0.066987) | 0.070756 / 0.419271 (-0.348516) | 0.049821 / 0.043533 (0.006288) | 0.360836 / 0.255139 (0.105697) | 0.383981 / 0.283200 (0.100781) | 0.023719 / 0.141683 (-0.117963) | 1.485197 / 1.452155 (0.033043) | 1.544899 / 1.492716 (0.052182) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.336480 / 0.018006 (0.318474) | 0.532839 / 0.000490 (0.532349) | 0.003767 / 0.000200 (0.003567) | 0.000087 / 0.000054 (0.000032) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034132 / 0.037411 (-0.003280) | 0.090131 / 0.014526 (0.075605) | 0.104086 / 0.176557 (-0.072471) | 0.158385 / 0.737135 (-0.578751) | 0.106417 / 0.296338 (-0.189922) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.416462 / 0.215209 (0.201253) | 4.160409 / 2.077655 (2.082755) | 2.195355 / 1.504120 (0.691235) | 2.051234 / 1.541195 (0.510040) | 2.012116 / 1.468490 (0.543626) | 0.477414 / 4.584777 (-4.107363) | 3.590326 / 3.745712 (-0.155386) | 3.318490 / 5.269862 (-1.951371) | 2.064124 / 4.565676 (-2.501553) | 0.057040 / 0.424275 (-0.367235) | 0.007283 / 0.007607 (-0.000324) | 0.480490 / 0.226044 (0.254445) | 4.804013 / 2.268929 (2.535084) | 2.625940 / 55.444624 (-52.818685) | 2.231537 / 6.876477 (-4.644939) | 2.441649 / 2.142072 (0.299576) | 0.573207 / 4.805227 (-4.232020) | 0.131685 / 6.500664 (-6.368979) | 0.060112 / 0.075469 (-0.015357) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.358587 / 1.841788 (-0.483200) | 20.457562 / 8.074308 (12.383254) | 14.236304 / 10.191392 (4.044912) | 0.152860 / 0.680424 (-0.527563) | 0.018466 / 0.534201 (-0.515735) | 0.401391 / 0.579283 (-0.177893) | 0.410252 / 0.434364 (-0.024111) | 0.484335 / 0.540337 (-0.056002) | 0.663818 / 1.386936 (-0.723118) |

![](https://cml.dev/watermark.png#acac88873abcb585892dc361eb9f6a70a1fd9a59 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007725 / 0.011353 (-0.003628) | 0.004448 / 0.011008 (-0.006560) | 0.098689 / 0.038508 (0.060180) | 0.082919 / 0.023109 (0.059809) | 0.380707 / 0.275898 (0.104809) | 0.452977 / 0.323480 (0.129497) | 0.004430 / 0.007986 (-0.003555) | 0.003712 / 0.004328 (-0.000616) | 0.076675 / 0.004250 (0.072425) | 0.062281 / 0.037052 (0.025228) | 0.403370 / 0.258489 (0.144881) | 0.464557 / 0.293841 (0.170716) | 0.035646 / 0.128546 (-0.092900) | 0.009776 / 0.075646 (-0.065870) | 0.341955 / 0.419271 (-0.077316) | 0.059515 / 0.043533 (0.015983) | 0.388421 / 0.255139 (0.133282) | 0.439496 / 0.283200 (0.156296) | 0.029090 / 0.141683 (-0.112593) | 1.727473 / 1.452155 (0.275319) | 1.810448 / 1.492716 (0.317732) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.221215 / 0.018006 (0.203208) | 0.486660 / 0.000490 (0.486171) | 0.005467 / 0.000200 (0.005267) | 0.000110 / 0.000054 (0.000056) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032491 / 0.037411 (-0.004920) | 0.094446 / 0.014526 (0.079920) | 0.110339 / 0.176557 (-0.066217) | 0.175004 / 0.737135 (-0.562131) | 0.109209 / 0.296338 (-0.187129) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.453966 / 0.215209 (0.238757) | 4.515842 / 2.077655 (2.438187) | 2.240512 / 1.504120 (0.736392) | 2.059911 / 1.541195 (0.518717) | 2.150635 / 1.468490 (0.682145) | 0.564509 / 4.584777 (-4.020268) | 4.055208 / 3.745712 (0.309496) | 3.614084 / 5.269862 (-1.655778) | 2.295760 / 4.565676 (-2.269917) | 0.066507 / 0.424275 (-0.357768) | 0.008909 / 0.007607 (0.001302) | 0.542604 / 0.226044 (0.316560) | 5.412162 / 2.268929 (3.143233) | 2.758757 / 55.444624 (-52.685867) | 2.430693 / 6.876477 (-4.445784) | 2.669866 / 2.142072 (0.527793) | 0.681756 / 4.805227 (-4.123471) | 0.156524 / 6.500664 (-6.344140) | 0.069499 / 0.075469 (-0.005970) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.571591 / 1.841788 (-0.270197) | 22.543437 / 8.074308 (14.469129) | 16.068426 / 10.191392 (5.877034) | 0.169860 / 0.680424 (-0.510564) | 0.021216 / 0.534201 (-0.512985) | 0.468745 / 0.579283 (-0.110538) | 0.475924 / 0.434364 (0.041560) | 0.535574 / 0.540337 (-0.004763) | 0.733823 / 1.386936 (-0.653113) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008038 / 0.011353 (-0.003315) | 0.004565 / 0.011008 (-0.006443) | 0.076892 / 0.038508 (0.038384) | 0.089559 / 0.023109 (0.066450) | 0.456752 / 0.275898 (0.180854) | 0.497282 / 0.323480 (0.173802) | 0.005991 / 0.007986 (-0.001995) | 0.003784 / 0.004328 (-0.000545) | 0.076339 / 0.004250 (0.072089) | 0.066050 / 0.037052 (0.028998) | 0.462708 / 0.258489 (0.204219) | 0.503711 / 0.293841 (0.209870) | 0.037098 / 0.128546 (-0.091448) | 0.009869 / 0.075646 (-0.065777) | 0.083678 / 0.419271 (-0.335594) | 0.058166 / 0.043533 (0.014633) | 0.461839 / 0.255139 (0.206700) | 0.481546 / 0.283200 (0.198347) | 0.027755 / 0.141683 (-0.113928) | 1.738490 / 1.452155 (0.286335) | 1.832276 / 1.492716 (0.339560) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.329935 / 0.018006 (0.311929) | 0.497438 / 0.000490 (0.496949) | 0.034644 / 0.000200 (0.034444) | 0.000199 / 0.000054 (0.000145) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035427 / 0.037411 (-0.001984) | 0.105689 / 0.014526 (0.091163) | 0.117706 / 0.176557 (-0.058850) | 0.177862 / 0.737135 (-0.559273) | 0.116791 / 0.296338 (-0.179547) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.484851 / 0.215209 (0.269642) | 4.804346 / 2.077655 (2.726691) | 2.494801 / 1.504120 (0.990681) | 2.320185 / 1.541195 (0.778990) | 2.374090 / 1.468490 (0.905600) | 0.567397 / 4.584777 (-4.017380) | 4.087402 / 3.745712 (0.341690) | 3.794245 / 5.269862 (-1.475616) | 2.378481 / 4.565676 (-2.187195) | 0.068228 / 0.424275 (-0.356047) | 0.008740 / 0.007607 (0.001133) | 0.574876 / 0.226044 (0.348832) | 5.742644 / 2.268929 (3.473716) | 3.047661 / 55.444624 (-52.396963) | 2.729742 / 6.876477 (-4.146735) | 2.852510 / 2.142072 (0.710438) | 0.679450 / 4.805227 (-4.125777) | 0.156162 / 6.500664 (-6.344502) | 0.074051 / 0.075469 (-0.001418) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.576182 / 1.841788 (-0.265605) | 23.298147 / 8.074308 (15.223839) | 16.344621 / 10.191392 (6.153229) | 0.167571 / 0.680424 (-0.512852) | 0.021423 / 0.534201 (-0.512778) | 0.464511 / 0.579283 (-0.114772) | 0.453257 / 0.434364 (0.018893) | 0.563439 / 0.540337 (0.023102) | 0.764759 / 1.386936 (-0.622177) |

![](https://cml.dev/watermark.png#e8dc4b32b0d91bdb0971f8203ee37e6588c7770e "CML watermark")

This should also fix https://github.com/huggingface/datasets/issues/6140, so please link it with this PR before merging.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006719 / 0.011353 (-0.004634) | 0.004299 / 0.011008 (-0.006709) | 0.085296 / 0.038508 (0.046788) | 0.085144 / 0.023109 (0.062035) | 0.361703 / 0.275898 (0.085805) | 0.397721 / 0.323480 (0.074241) | 0.005920 / 0.007986 (-0.002065) | 0.003853 / 0.004328 (-0.000476) | 0.065633 / 0.004250 (0.061383) | 0.057000 / 0.037052 (0.019947) | 0.379981 / 0.258489 (0.121492) | 0.419041 / 0.293841 (0.125200) | 0.031225 / 0.128546 (-0.097322) | 0.008868 / 0.075646 (-0.066779) | 0.288808 / 0.419271 (-0.130463) | 0.052391 / 0.043533 (0.008859) | 0.362349 / 0.255139 (0.107210) | 0.399858 / 0.283200 (0.116658) | 0.025843 / 0.141683 (-0.115840) | 1.498988 / 1.452155 (0.046834) | 1.547290 / 1.492716 (0.054574) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.278091 / 0.018006 (0.260085) | 0.621794 / 0.000490 (0.621305) | 0.003770 / 0.000200 (0.003570) | 0.000084 / 0.000054 (0.000029) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029128 / 0.037411 (-0.008283) | 0.082061 / 0.014526 (0.067536) | 0.101758 / 0.176557 (-0.074799) | 0.155724 / 0.737135 (-0.581411) | 0.102173 / 0.296338 (-0.194165) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.387145 / 0.215209 (0.171935) | 3.868262 / 2.077655 (1.790607) | 1.886440 / 1.504120 (0.382320) | 1.723305 / 1.541195 (0.182111) | 1.805411 / 1.468490 (0.336921) | 0.485024 / 4.584777 (-4.099753) | 3.637859 / 3.745712 (-0.107853) | 3.319593 / 5.269862 (-1.950269) | 2.087860 / 4.565676 (-2.477817) | 0.056992 / 0.424275 (-0.367283) | 0.007623 / 0.007607 (0.000016) | 0.468182 / 0.226044 (0.242138) | 4.681112 / 2.268929 (2.412183) | 2.407010 / 55.444624 (-53.037614) | 2.026604 / 6.876477 (-4.849872) | 2.298158 / 2.142072 (0.156086) | 0.581839 / 4.805227 (-4.223388) | 0.132101 / 6.500664 (-6.368563) | 0.060472 / 0.075469 (-0.014997) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.236422 / 1.841788 (-0.605365) | 20.505168 / 8.074308 (12.430860) | 14.356081 / 10.191392 (4.164689) | 0.148808 / 0.680424 (-0.531616) | 0.018433 / 0.534201 (-0.515768) | 0.391323 / 0.579283 (-0.187960) | 0.413142 / 0.434364 (-0.021222) | 0.453484 / 0.540337 (-0.086853) | 0.620771 / 1.386936 (-0.766165) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007030 / 0.011353 (-0.004323) | 0.004430 / 0.011008 (-0.006578) | 0.065578 / 0.038508 (0.027070) | 0.090751 / 0.023109 (0.067642) | 0.389121 / 0.275898 (0.113223) | 0.424657 / 0.323480 (0.101177) | 0.006575 / 0.007986 (-0.001410) | 0.003855 / 0.004328 (-0.000473) | 0.066175 / 0.004250 (0.061925) | 0.063255 / 0.037052 (0.026202) | 0.397161 / 0.258489 (0.138672) | 0.435291 / 0.293841 (0.141450) | 0.031622 / 0.128546 (-0.096925) | 0.008900 / 0.075646 (-0.066747) | 0.071694 / 0.419271 (-0.347577) | 0.049161 / 0.043533 (0.005628) | 0.386214 / 0.255139 (0.131075) | 0.404571 / 0.283200 (0.121372) | 0.024821 / 0.141683 (-0.116862) | 1.489514 / 1.452155 (0.037359) | 1.576139 / 1.492716 (0.083423) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.289884 / 0.018006 (0.271878) | 0.629342 / 0.000490 (0.628852) | 0.004799 / 0.000200 (0.004599) | 0.000160 / 0.000054 (0.000106) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032081 / 0.037411 (-0.005331) | 0.088152 / 0.014526 (0.073626) | 0.107289 / 0.176557 (-0.069267) | 0.164598 / 0.737135 (-0.572537) | 0.108395 / 0.296338 (-0.187944) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.426723 / 0.215209 (0.211514) | 4.267719 / 2.077655 (2.190064) | 2.289657 / 1.504120 (0.785537) | 2.117435 / 1.541195 (0.576240) | 2.187292 / 1.468490 (0.718802) | 0.478387 / 4.584777 (-4.106390) | 3.625096 / 3.745712 (-0.120616) | 3.408036 / 5.269862 (-1.861826) | 2.124117 / 4.565676 (-2.441559) | 0.056537 / 0.424275 (-0.367738) | 0.007489 / 0.007607 (-0.000118) | 0.502434 / 0.226044 (0.276389) | 5.025357 / 2.268929 (2.756428) | 2.740554 / 55.444624 (-52.704070) | 2.418841 / 6.876477 (-4.457635) | 2.730764 / 2.142072 (0.588691) | 0.600013 / 4.805227 (-4.205214) | 0.133039 / 6.500664 (-6.367625) | 0.061466 / 0.075469 (-0.014003) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.330211 / 1.841788 (-0.511577) | 21.092100 / 8.074308 (13.017792) | 14.463054 / 10.191392 (4.271662) | 0.154149 / 0.680424 (-0.526274) | 0.018891 / 0.534201 (-0.515310) | 0.393078 / 0.579283 (-0.186205) | 0.415279 / 0.434364 (-0.019085) | 0.479469 / 0.540337 (-0.060868) | 0.659953 / 1.386936 (-0.726983) |

![](https://cml.dev/watermark.png#5ca2ba050340829b4dd44791afc15db0d82a3276 "CML watermark")

This would make the data files resolution faster: no need to list all the data files to infer the dataset builder to use.

fix https://github.com/huggingface/datasets/issues/6140


good first issue

Good for newcomers

Makes sense, I guess this can be fixed in the load_dataset_builder method.
It concerns every packaged builder I think (see values in `_PACKAGED_DATASETS_MODULES`)

I think the behavior is related to these lines, which short circuited the error handling.
https://github.com/huggingface/datasets/blob/664a1cb72ea1e6ef7c47e671e2686ca4a35e8d63/src/datasets/load.py#L946-L952

So should data_dir be checked here or still delegating to actual `DatasetModule`? In that case, how to properly set `data_files` here.

This is location in PackagedDatasetModuleFactory.get_module seems the be the right place to check if at least data_dir or data_files are passed

@mariosasko can you please assign this issue to me,I want to work on this

@mariosasko is this issue still open? i would love to kickstart my journey to open source with this issue!
Regards
zutarich

@zutarich It is unless @debrupf2946 is working on it.

I am working and will open a pull request soon @Etelis

i think its working as expected . Heres the log i get for the same line -

![image](https://github.com/huggingface/datasets/assets/63234112/8a857ec5-8dd0-4b01-b3a7-7c93444b9558)

### Describe the bug

FolderBase Dataset automatically resolves under current directory when data_dir is not specified.

For example:
```
load_dataset("audiofolder")
```
takes long time to resolve and collect data_files from current directory. But I think it should reach out to this line for error handling https://github.com/huggingface/datasets/blob/cb8c5de5145c7e7eee65391cb7f4d92f0d565d62/src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py#L58-L59

### Steps to reproduce the bug

```
load_dataset("audiofolder")
```

### Expected behavior

Error report

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.15.0-78-generic-x86_64-with-glibc2.17
- Python version: 3.8.15
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.3


Faster sorting for single key items

enhancement

New feature or request

`Dataset.sort` essentially does the same thing except it uses `pyarrow.compute.sort_indices` which doesn't involve copying the data into python objects (saving memory)

```python
sort_keys = [(col, "ascending") for col in column_names]
indices = pc.sort_indices(self.data, sort_keys=sort_keys)
return self.select(indices)
```

Ok interesting, I'll continue debugging to see what is going wrong on my end.

### Feature request

A faster way to sort a dataset which contains a large number of rows.

### Motivation

The current sorting implementations took significantly longer than expected when I was running on a dataset trying to sort by timestamps.

**Code snippet:**
```python
ds = datasets.load_dataset( "json", **{"data_files": {"train": "path-to-jsonlines"}, "split": "train"}, num_proc=os.cpu_count(), keep_in_memory=True)
sorted_ds = ds.sort("pubDate", keep_in_memory=True)
```

However, once I switched to a different method which
1. unpacked to a list of tuples
2. sorted tuples by key
3. run `.select` with the sorted list of indices
It was significantly faster (orders of magnitude, especially with M's of rows)

### Your contribution

I'd be happy to implement a crude single key sorting algorithm so that other users can benefit from this trick. Broadly, this would take a `Dataset` and perform;

```python
# ds is a Dataset object
# key_name is the sorting key

class Dataset:
   ...
   def _sort(key_name: str) -> Dataset:
      index_keys = [(i,x) for i,x in enumerate(self[key_name])]
      sorted_rows = sorted(row_pubdate, key=lambda x: x[1])
      sorted_indicies = [x[0] for x in sorted_rows]
      return self.select(sorted_indicies)
```


Allow dataset implement .take

New feature or request

```
    dataset = IterableDataset(dataset) if type(dataset) != IterableDataset else dataset  # to force dataset.take(batch_size) to work in non-streaming mode
   ```

Feel free to work on this. In addition, `IterableDataset` supports `skip`, so we should also add this method to `Dataset`.

### Feature request

I want to do:
```
dataset.take(512)
```

but it only works with streaming = True

### Motivation

uniform interface to data sets. Really surprising the above only works with streaming = True.

### Your contribution

Should be trivial to copy paste the IterableDataset .take to use the local path in the data (when streaming = False)


Dataset.from_parquet cannot load subset of columns

Looks like this regression was introduced in `datasets==2.13.0` (`2.12.0` could load a subset of columns)

This does not appear to be fixed by https://github.com/huggingface/datasets/pull/6045 (bug still exists on `main`)

### Describe the bug

When using `Dataset.from_parquet(path_or_paths, columns=[...])` and a subset of columns, loading fails with a variant of the following

```
ValueError: Couldn't cast
a: int64
-- schema metadata --
pandas: '{"index_columns": [], "column_indexes": [], "columns": [{"name":' + 273
to
{'a': Value(dtype='int64', id=None), 'b': Value(dtype='int64', id=None)}
because column names don't match

The above exception was the direct cause of the following exception:
```

Looks to be triggered by https://github.com/huggingface/datasets/blob/c02a44715c036b5261686669727394b1308a3a4b/src/datasets/table.py#L2285-L2286

### Steps to reproduce the bug

```
import pandas as pd
from datasets import Dataset

pd.DataFrame([{"a": 1, "b": 2}]).to_parquet("test.pq")
Dataset.from_parquet("test.pq", columns=["a"])
```

### Expected behavior

A subset of columns should be loaded without error

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.10.0-23-cloud-amd64-x86_64-with-glibc2.2.5
- Python version: 3.8.16
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3


Ignore parallel warning in map_nested

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006818 / 0.011353 (-0.004534) | 0.004166 / 0.011008 (-0.006842) | 0.086756 / 0.038508 (0.048248) | 0.084444 / 0.023109 (0.061335) | 0.319249 / 0.275898 (0.043351) | 0.358689 / 0.323480 (0.035209) | 0.004344 / 0.007986 (-0.003641) | 0.003564 / 0.004328 (-0.000765) | 0.065021 / 0.004250 (0.060771) | 0.055991 / 0.037052 (0.018939) | 0.319573 / 0.258489 (0.061084) | 0.373239 / 0.293841 (0.079398) | 0.031431 / 0.128546 (-0.097115) | 0.008671 / 0.075646 (-0.066975) | 0.288484 / 0.419271 (-0.130788) | 0.053501 / 0.043533 (0.009968) | 0.316934 / 0.255139 (0.061795) | 0.354233 / 0.283200 (0.071034) | 0.028088 / 0.141683 (-0.113595) | 1.510905 / 1.452155 (0.058750) | 1.568614 / 1.492716 (0.075898) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.292343 / 0.018006 (0.274337) | 0.592309 / 0.000490 (0.591819) | 0.003850 / 0.000200 (0.003650) | 0.000084 / 0.000054 (0.000030) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033510 / 0.037411 (-0.003901) | 0.089546 / 0.014526 (0.075020) | 0.104909 / 0.176557 (-0.071648) | 0.162219 / 0.737135 (-0.574916) | 0.104137 / 0.296338 (-0.192202) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.407993 / 0.215209 (0.192784) | 4.063423 / 2.077655 (1.985768) | 2.050237 / 1.504120 (0.546117) | 1.888939 / 1.541195 (0.347744) | 2.015195 / 1.468490 (0.546704) | 0.492617 / 4.584777 (-4.092160) | 3.595871 / 3.745712 (-0.149841) | 3.320467 / 5.269862 (-1.949395) | 2.099987 / 4.565676 (-2.465690) | 0.058513 / 0.424275 (-0.365762) | 0.007709 / 0.007607 (0.000102) | 0.479277 / 0.226044 (0.253233) | 4.790712 / 2.268929 (2.521783) | 2.517292 / 55.444624 (-52.927332) | 2.167461 / 6.876477 (-4.709016) | 2.432011 / 2.142072 (0.289939) | 0.600537 / 4.805227 (-4.204690) | 0.133538 / 6.500664 (-6.367126) | 0.059621 / 0.075469 (-0.015848) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.280375 / 1.841788 (-0.561413) | 20.777971 / 8.074308 (12.703663) | 14.869539 / 10.191392 (4.678147) | 0.159372 / 0.680424 (-0.521052) | 0.018096 / 0.534201 (-0.516105) | 0.393945 / 0.579283 (-0.185338) | 0.409598 / 0.434364 (-0.024766) | 0.459202 / 0.540337 (-0.081136) | 0.632298 / 1.386936 (-0.754638) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006694 / 0.011353 (-0.004659) | 0.004299 / 0.011008 (-0.006709) | 0.064880 / 0.038508 (0.026372) | 0.083233 / 0.023109 (0.060124) | 0.366488 / 0.275898 (0.090590) | 0.405049 / 0.323480 (0.081569) | 0.005602 / 0.007986 (-0.002384) | 0.003623 / 0.004328 (-0.000705) | 0.064410 / 0.004250 (0.060160) | 0.057962 / 0.037052 (0.020910) | 0.365318 / 0.258489 (0.106829) | 0.403151 / 0.293841 (0.109310) | 0.031285 / 0.128546 (-0.097261) | 0.008867 / 0.075646 (-0.066780) | 0.071137 / 0.419271 (-0.348135) | 0.048398 / 0.043533 (0.004865) | 0.360187 / 0.255139 (0.105048) | 0.383872 / 0.283200 (0.100673) | 0.023232 / 0.141683 (-0.118451) | 1.526980 / 1.452155 (0.074826) | 1.587265 / 1.492716 (0.094549) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.362603 / 0.018006 (0.344596) | 0.557034 / 0.000490 (0.556544) | 0.025303 / 0.000200 (0.025103) | 0.000562 / 0.000054 (0.000508) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030636 / 0.037411 (-0.006775) | 0.088085 / 0.014526 (0.073559) | 0.103238 / 0.176557 (-0.073318) | 0.155208 / 0.737135 (-0.581928) | 0.106661 / 0.296338 (-0.189678) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.413660 / 0.215209 (0.198451) | 4.122717 / 2.077655 (2.045063) | 2.097656 / 1.504120 (0.593536) | 1.931995 / 1.541195 (0.390801) | 2.071497 / 1.468490 (0.603007) | 0.490257 / 4.584777 (-4.094520) | 3.588076 / 3.745712 (-0.157636) | 3.423087 / 5.269862 (-1.846774) | 2.147974 / 4.565676 (-2.417703) | 0.058783 / 0.424275 (-0.365492) | 0.007456 / 0.007607 (-0.000151) | 0.492350 / 0.226044 (0.266305) | 4.935935 / 2.268929 (2.667006) | 2.604217 / 55.444624 (-52.840407) | 2.333723 / 6.876477 (-4.542754) | 2.585293 / 2.142072 (0.443220) | 0.608800 / 4.805227 (-4.196427) | 0.135806 / 6.500664 (-6.364858) | 0.062716 / 0.075469 (-0.012753) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.347359 / 1.841788 (-0.494429) | 21.420505 / 8.074308 (13.346197) | 14.325914 / 10.191392 (4.134522) | 0.159617 / 0.680424 (-0.520806) | 0.018769 / 0.534201 (-0.515432) | 0.399677 / 0.579283 (-0.179606) | 0.402992 / 0.434364 (-0.031372) | 0.484629 / 0.540337 (-0.055709) | 0.656007 / 1.386936 (-0.730929) |

![](https://cml.dev/watermark.png#ac94bb10d5c00ce8fdaf461eb1ff4b8572cfe956 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007291 / 0.011353 (-0.004062) | 0.004501 / 0.011008 (-0.006508) | 0.097529 / 0.038508 (0.059021) | 0.079257 / 0.023109 (0.056147) | 0.356390 / 0.275898 (0.080492) | 0.390065 / 0.323480 (0.066585) | 0.006071 / 0.007986 (-0.001914) | 0.003783 / 0.004328 (-0.000546) | 0.074598 / 0.004250 (0.070348) | 0.059626 / 0.037052 (0.022574) | 0.395344 / 0.258489 (0.136855) | 0.418564 / 0.293841 (0.124723) | 0.041843 / 0.128546 (-0.086704) | 0.009293 / 0.075646 (-0.066354) | 0.332668 / 0.419271 (-0.086604) | 0.065753 / 0.043533 (0.022220) | 0.357285 / 0.255139 (0.102146) | 0.402974 / 0.283200 (0.119775) | 0.028714 / 0.141683 (-0.112968) | 1.733913 / 1.452155 (0.281759) | 1.802574 / 1.492716 (0.309858) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.253114 / 0.018006 (0.235108) | 0.606338 / 0.000490 (0.605848) | 0.006871 / 0.000200 (0.006671) | 0.000126 / 0.000054 (0.000072) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031850 / 0.037411 (-0.005562) | 0.095148 / 0.014526 (0.080622) | 0.111499 / 0.176557 (-0.065057) | 0.174653 / 0.737135 (-0.562483) | 0.109396 / 0.296338 (-0.186943) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.440442 / 0.215209 (0.225233) | 4.408792 / 2.077655 (2.331137) | 2.149778 / 1.504120 (0.645658) | 1.922430 / 1.541195 (0.381235) | 2.029281 / 1.468490 (0.560791) | 0.611586 / 4.584777 (-3.973191) | 4.204571 / 3.745712 (0.458859) | 3.638194 / 5.269862 (-1.631668) | 2.336146 / 4.565676 (-2.229531) | 0.065383 / 0.424275 (-0.358892) | 0.008441 / 0.007607 (0.000834) | 0.527357 / 0.226044 (0.301313) | 5.247892 / 2.268929 (2.978963) | 2.654005 / 55.444624 (-52.790620) | 2.256596 / 6.876477 (-4.619881) | 2.432191 / 2.142072 (0.290119) | 0.672759 / 4.805227 (-4.132469) | 0.148494 / 6.500664 (-6.352170) | 0.068248 / 0.075469 (-0.007221) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.544250 / 1.841788 (-0.297538) | 21.882016 / 8.074308 (13.807708) | 16.470182 / 10.191392 (6.278790) | 0.166107 / 0.680424 (-0.514317) | 0.021305 / 0.534201 (-0.512896) | 0.445069 / 0.579283 (-0.134214) | 0.500631 / 0.434364 (0.066267) | 0.525801 / 0.540337 (-0.014536) | 0.806534 / 1.386936 (-0.580402) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007322 / 0.011353 (-0.004030) | 0.004206 / 0.011008 (-0.006802) | 0.074827 / 0.038508 (0.036319) | 0.084759 / 0.023109 (0.061650) | 0.421204 / 0.275898 (0.145306) | 0.464442 / 0.323480 (0.140962) | 0.006523 / 0.007986 (-0.001463) | 0.003613 / 0.004328 (-0.000716) | 0.073796 / 0.004250 (0.069545) | 0.066609 / 0.037052 (0.029557) | 0.430108 / 0.258489 (0.171619) | 0.463165 / 0.293841 (0.169324) | 0.036015 / 0.128546 (-0.092532) | 0.009696 / 0.075646 (-0.065951) | 0.083326 / 0.419271 (-0.335946) | 0.056804 / 0.043533 (0.013271) | 0.423333 / 0.255139 (0.168194) | 0.450538 / 0.283200 (0.167338) | 0.027067 / 0.141683 (-0.114616) | 1.700563 / 1.452155 (0.248408) | 1.748738 / 1.492716 (0.256021) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.395682 / 0.018006 (0.377675) | 0.540192 / 0.000490 (0.539702) | 0.140049 / 0.000200 (0.139849) | 0.000694 / 0.000054 (0.000639) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036643 / 0.037411 (-0.000769) | 0.104422 / 0.014526 (0.089896) | 0.113072 / 0.176557 (-0.063484) | 0.179561 / 0.737135 (-0.557575) | 0.118620 / 0.296338 (-0.177718) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.476547 / 0.215209 (0.261338) | 4.716009 / 2.077655 (2.638354) | 2.412111 / 1.504120 (0.907991) | 2.246389 / 1.541195 (0.705194) | 2.307058 / 1.468490 (0.838568) | 0.552759 / 4.584777 (-4.032018) | 4.172484 / 3.745712 (0.426771) | 3.848419 / 5.269862 (-1.421443) | 2.310338 / 4.565676 (-2.255339) | 0.071757 / 0.424275 (-0.352518) | 0.011206 / 0.007607 (0.003599) | 0.609526 / 0.226044 (0.383482) | 5.583065 / 2.268929 (3.314136) | 3.081227 / 55.444624 (-52.363397) | 2.637782 / 6.876477 (-4.238695) | 2.887561 / 2.142072 (0.745489) | 0.667227 / 4.805227 (-4.138000) | 0.154421 / 6.500664 (-6.346243) | 0.070772 / 0.075469 (-0.004697) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.605500 / 1.841788 (-0.236288) | 22.872717 / 8.074308 (14.798409) | 15.865333 / 10.191392 (5.673941) | 0.170353 / 0.680424 (-0.510071) | 0.021854 / 0.534201 (-0.512347) | 0.461467 / 0.579283 (-0.117816) | 0.477743 / 0.434364 (0.043379) | 0.597234 / 0.540337 (0.056896) | 0.800416 / 1.386936 (-0.586520) |

![](https://cml.dev/watermark.png#a7f8d9019e7cb104eac4106bdc6ec0292f0dc61a "CML watermark")

This warning message was shown every time you pass num_proc to `load_dataset` because of `map_nested`

```
parallel_map is experimental and might be subject to breaking changes in the future
```

This PR removes it for `map_nested`. If someone uses another parallel backend they're already warned when `parallel_backend` is called anyway


ValueError when running BeamBasedBuilder with GCS path in cache_dir

The cause of the error seems to be that `datasets` adds "gcs://" as a schema, while `beam` checks only "gs://".

datasets: https://github.com/huggingface/datasets/blob/c02a44715c036b5261686669727394b1308a3a4b/src/datasets/builder.py#L822

beam: [link](https://github.com/apache/beam/blob/25e1a64641b1c8a3c0a6c75c6e86031b87307f22/sdks/python/apache_beam/io/filesystems.py#L98-L101)
```
      systems = [
          fs for fs in FileSystem.get_all_subclasses()
          if fs.scheme() == path_scheme
      ]
```

Hi! We've deprecated the Beam API, as we don't have the bandwidth to support it properly...

### Describe the bug

When running the BeamBasedBuilder with a GCS path specified in the cache_dir, the following ValueError occurs:

```
ValueError: Unable to get filesystem from specified path, please use the correct path or ensure the required dependency is installed, e.g., pip install apache-beam[gcp]. Path specified: gcs://my-bucket/huggingface_datasets/my_beam_dataset/default/0.0.0/my_beam_dataset-train [while running 'train/Save to parquet/Write/WriteImpl/InitializeWrite']
```

Same error occurs after running `pip install apache-beam[gcp]` as instructed.

### Steps to reproduce the bug

Put `my_beam_dataset.py`:

```python
import datasets

class MyBeamDataset(datasets.BeamBasedBuilder):
    def _info(self):
        features = datasets.Features({"value": datasets.Value("int64")})
        return datasets.DatasetInfo(features=features)

    def _split_generators(self, dl_manager, pipeline):
        return [datasets.SplitGenerator(name=datasets.Split.TRAIN, gen_kwargs={})]

    def _build_pcollection(self, pipeline):
        import apache_beam as beam

        return pipeline | beam.Create([{"value": i} for i in range(10)])
```

Run:
```bash
datasets-cli run_beam my_beam_dataset.py --cache_dir=gs://my-bucket/huggingface_datasets/ --beam_pipeline_options="runner=DirectRunner"
```

### Expected behavior

Running the BeamBasedBuilder with a GCS cache path without any errors.

### Environment info

- `datasets` version: 2.14.4
- Platform: macOS-13.4-arm64-arm-64bit
- Python version: 3.9.17
- Huggingface_hub version: 0.16.4
- PyArrow version: 9.0.0
- Pandas version: 2.0.3


DatasetGenerationError when load glue benchmark datasets from `load_dataset`

I've tried clear the .cache file, doesn't work.

This issue can happen if there is a directory named "glue" relative to the Python script with the `load_dataset` call (similar issue to this one: https://github.com/huggingface/datasets/issues/5228). Is this the case?

> This issue can happen if there is a directory named "glue" relative to the Python script with the `load_dataset` call (similar issue to this one: #5228). Is this the case?

Thats correct!
Sorry for my late response.

### Describe the bug

Package version: datasets-2.14.4

When I run the codes:

```
from datasets import load_dataset

dataset = load_dataset("glue", "ax")
```

I got the following errors:

---------------------------------------------------------------------------
SchemaInferenceError                      Traceback (most recent call last)
File ~/anaconda3/envs/python3/lib/python3.10/site-packages/datasets/builder.py:1949, in ArrowBasedBuilder._prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
   1948 num_shards = shard_id + 1
-> 1949 num_examples, num_bytes = writer.finalize()
   1950 writer.close()

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/datasets/arrow_writer.py:598, in ArrowWriter.finalize(self, close_stream)
    597         self.stream.close()
--> 598     raise SchemaInferenceError("Please pass `features` or at least one example when writing data")
    599 logger.debug(
    600     f"Done writing {self._num_examples} {self.unit} in {self._num_bytes} bytes {self._path if self._path else ''}."
    601 )

SchemaInferenceError: Please pass `features` or at least one example when writing data

The above exception was the direct cause of the following exception:

DatasetGenerationError                    Traceback (most recent call last)
Cell In[5], line 3
      1 from datasets import load_dataset
----> 3 dataset = load_dataset("glue", "ax")

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/datasets/load.py:2136, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, ignore_verifications, keep_in_memory, save_infos, revision, token, use_auth_token, task, streaming, num_proc, storage_options, **config_kwargs)
   2133 try_from_hf_gcs = path not in _PACKAGED_DATASETS_MODULES
   2135 # Download and prepare data
-> 2136 builder_instance.download_and_prepare(
   2137     download_config=download_config,
   2138     download_mode=download_mode,
   2139     verification_mode=verification_mode,
   2140     try_from_hf_gcs=try_from_hf_gcs,
   2141     num_proc=num_proc,
   2142     storage_options=storage_options,
   2143 )
   2145 # Build dataset for splits
   2146 keep_in_memory = (
   2147     keep_in_memory if keep_in_memory is not None else is_small_dataset(builder_instance.info.dataset_size)
   2148 )

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/datasets/builder.py:954, in DatasetBuilder.download_and_prepare(self, output_dir, download_config, download_mode, verification_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, file_format, max_shard_size, num_proc, storage_options, **download_and_prepare_kwargs)
    952     if num_proc is not None:
    953         prepare_split_kwargs["num_proc"] = num_proc
--> 954     self._download_and_prepare(
    955         dl_manager=dl_manager,
    956         verification_mode=verification_mode,
    957         **prepare_split_kwargs,
    958         **download_and_prepare_kwargs,
    959     )
    960 # Sync info
    961 self.info.dataset_size = sum(split.num_bytes for split in self.info.splits.values())

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/datasets/builder.py:1049, in DatasetBuilder._download_and_prepare(self, dl_manager, verification_mode, **prepare_split_kwargs)
   1045 split_dict.add(split_generator.split_info)
   1047 try:
   1048     # Prepare split will record examples associated to the split
-> 1049     self._prepare_split(split_generator, **prepare_split_kwargs)
   1050 except OSError as e:
   1051     raise OSError(
   1052         "Cannot find data file. "
   1053         + (self.manual_download_instructions or "")
   1054         + "\nOriginal error:\n"
   1055         + str(e)
   1056     ) from None

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/datasets/builder.py:1813, in ArrowBasedBuilder._prepare_split(self, split_generator, file_format, num_proc, max_shard_size)
   1811 job_id = 0
   1812 with pbar:
-> 1813     for job_id, done, content in self._prepare_split_single(
   1814         gen_kwargs=gen_kwargs, job_id=job_id, **_prepare_split_args
   1815     ):
   1816         if done:
   1817             result = content

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/datasets/builder.py:1958, in ArrowBasedBuilder._prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
   1956     if isinstance(e, SchemaInferenceError) and e.__context__ is not None:
   1957         e = e.__context__
-> 1958     raise DatasetGenerationError("An error occurred while generating the dataset") from e
   1960 yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_shards, shard_lengths)

DatasetGenerationError: An error occurred while generating the dataset

### Steps to reproduce the bug

from datasets import load_dataset

dataset = load_dataset("glue", "ax")

### Expected behavior

When generating the train split:
Generating train split:
0/0 [00:00<?, ? examples/s]

It raise the error:
DatasetGenerationError: An error occurred while generating the dataset

### Environment info

datasets-2.14.4.
Python 3.10


custom load dataset to hub

> This is an issue for the [Datasets repo](https://github.com/huggingface/datasets).

Thanks @sgugger , I guess I will wait for them to address the issue . Looking forward to hearing from them

> You can use `.push_to_hub("<username>/<repo>")` to push a `Dataset` to the Hub.

how about subset? like `.load_dataset("<username>/<repo>", "<subset>")`, how can I upload multi-dataset in one repo? thanks a lot !

> > You can use `.push_to_hub("<username>/<repo>")` to push a `Dataset` to the Hub.
>
> how about subset? like `.load_dataset("<username>/<repo>", "<subset>")`, how can I upload multi-dataset in one repo? thanks a lot !

I solved it by upgrading `Datasets` version to 2.15.0

### System Info

kaggle notebook

i transformed dataset:
```
dataset = load_dataset("Dahoas/first-instruct-human-assistant-prompt")
```
to
formatted_dataset:
```
Dataset({
    features: ['message_tree_id', 'message_tree_text'],
    num_rows: 33143
})
```
but would like to know how to upload to hub

### Who can help?

@ArthurZucker @younesbelkada

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

shared above

### Expected behavior

load dataset to hub


Export to_iterable_dataset to document

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006076 / 0.011353 (-0.005277) | 0.003730 / 0.011008 (-0.007279) | 0.080778 / 0.038508 (0.042270) | 0.062970 / 0.023109 (0.039860) | 0.395864 / 0.275898 (0.119966) | 0.430024 / 0.323480 (0.106544) | 0.004823 / 0.007986 (-0.003162) | 0.002949 / 0.004328 (-0.001379) | 0.062423 / 0.004250 (0.058172) | 0.047343 / 0.037052 (0.010291) | 0.403153 / 0.258489 (0.144664) | 0.443666 / 0.293841 (0.149825) | 0.027798 / 0.128546 (-0.100748) | 0.008056 / 0.075646 (-0.067590) | 0.262260 / 0.419271 (-0.157011) | 0.045958 / 0.043533 (0.002425) | 0.391349 / 0.255139 (0.136210) | 0.421831 / 0.283200 (0.138632) | 0.021837 / 0.141683 (-0.119846) | 1.485509 / 1.452155 (0.033355) | 1.542940 / 1.492716 (0.050224) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.196831 / 0.018006 (0.178825) | 0.435774 / 0.000490 (0.435285) | 0.003647 / 0.000200 (0.003447) | 0.000065 / 0.000054 (0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023756 / 0.037411 (-0.013655) | 0.075737 / 0.014526 (0.061211) | 0.303703 / 0.176557 (0.127146) | 0.164862 / 0.737135 (-0.572273) | 0.198483 / 0.296338 (-0.097855) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.405220 / 0.215209 (0.190011) | 4.065983 / 2.077655 (1.988328) | 2.043001 / 1.504120 (0.538881) | 1.853318 / 1.541195 (0.312123) | 1.977452 / 1.468490 (0.508962) | 0.500897 / 4.584777 (-4.083880) | 3.065756 / 3.745712 (-0.679956) | 2.924096 / 5.269862 (-2.345765) | 1.876194 / 4.565676 (-2.689482) | 0.057774 / 0.424275 (-0.366501) | 0.006809 / 0.007607 (-0.000798) | 0.470979 / 0.226044 (0.244934) | 4.719546 / 2.268929 (2.450618) | 2.449651 / 55.444624 (-52.994973) | 2.211817 / 6.876477 (-4.664660) | 2.398760 / 2.142072 (0.256687) | 0.590608 / 4.805227 (-4.214619) | 0.125836 / 6.500664 (-6.374829) | 0.060759 / 0.075469 (-0.014710) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.243609 / 1.841788 (-0.598179) | 18.836193 / 8.074308 (10.761885) | 13.835053 / 10.191392 (3.643661) | 0.129708 / 0.680424 (-0.550716) | 0.016708 / 0.534201 (-0.517493) | 0.337219 / 0.579283 (-0.242065) | 0.359045 / 0.434364 (-0.075319) | 0.383329 / 0.540337 (-0.157009) | 0.539629 / 1.386936 (-0.847307) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006073 / 0.011353 (-0.005280) | 0.003713 / 0.011008 (-0.007295) | 0.062642 / 0.038508 (0.024134) | 0.062618 / 0.023109 (0.039508) | 0.362029 / 0.275898 (0.086130) | 0.401924 / 0.323480 (0.078445) | 0.004689 / 0.007986 (-0.003297) | 0.002945 / 0.004328 (-0.001384) | 0.062720 / 0.004250 (0.058470) | 0.048901 / 0.037052 (0.011848) | 0.363780 / 0.258489 (0.105291) | 0.405111 / 0.293841 (0.111270) | 0.027738 / 0.128546 (-0.100808) | 0.008046 / 0.075646 (-0.067600) | 0.067752 / 0.419271 (-0.351519) | 0.041955 / 0.043533 (-0.001577) | 0.361615 / 0.255139 (0.106476) | 0.388762 / 0.283200 (0.105562) | 0.021302 / 0.141683 (-0.120380) | 1.473527 / 1.452155 (0.021372) | 1.529753 / 1.492716 (0.037037) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.300446 / 0.018006 (0.282440) | 0.425844 / 0.000490 (0.425354) | 0.054507 / 0.000200 (0.054307) | 0.000282 / 0.000054 (0.000228) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025478 / 0.037411 (-0.011933) | 0.078298 / 0.014526 (0.063772) | 0.087647 / 0.176557 (-0.088909) | 0.138978 / 0.737135 (-0.598157) | 0.088396 / 0.296338 (-0.207942) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.421345 / 0.215209 (0.206136) | 4.209188 / 2.077655 (2.131533) | 2.260731 / 1.504120 (0.756611) | 2.072329 / 1.541195 (0.531134) | 2.086778 / 1.468490 (0.618288) | 0.495425 / 4.584777 (-4.089352) | 2.987519 / 3.745712 (-0.758194) | 2.895106 / 5.269862 (-2.374756) | 1.874637 / 4.565676 (-2.691039) | 0.057080 / 0.424275 (-0.367195) | 0.006402 / 0.007607 (-0.001205) | 0.498233 / 0.226044 (0.272188) | 4.974385 / 2.268929 (2.705457) | 2.671755 / 55.444624 (-52.772870) | 2.356120 / 6.876477 (-4.520357) | 2.531374 / 2.142072 (0.389301) | 0.581955 / 4.805227 (-4.223272) | 0.125491 / 6.500664 (-6.375173) | 0.062267 / 0.075469 (-0.013202) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.307233 / 1.841788 (-0.534555) | 18.929740 / 8.074308 (10.855431) | 14.029693 / 10.191392 (3.838301) | 0.161992 / 0.680424 (-0.518431) | 0.017127 / 0.534201 (-0.517074) | 0.336644 / 0.579283 (-0.242639) | 0.336550 / 0.434364 (-0.097814) | 0.400554 / 0.540337 (-0.139783) | 0.560725 / 1.386936 (-0.826211) |

![](https://cml.dev/watermark.png#cb8c5de5145c7e7eee65391cb7f4d92f0d565d62 "CML watermark")

Fix the export of a missing method of `Dataset`


NIH exporter file not found

another file not found:
```
Traceback (most recent call last):
  File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/implementations/http.py", line 417, in _info
    await _file_info(
  File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/implementations/http.py", line 837, in _file_info
    r.raise_for_status()
  File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1005, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('https://the-eye.eu/public/AI/pile_preliminary_components/pile_uspto.tar')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/lfs/ampere1/0/brando9/.vscode-server-insiders/extensions/ms-python.python-2023.14.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
    cli.main()
  File "/lfs/ampere1/0/brando9/.vscode-server-insiders/extensions/ms-python.python-2023.14.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "/lfs/ampere1/0/brando9/.vscode-server-insiders/extensions/ms-python.python-2023.14.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
    runpy.run_path(target, run_name="__main__")
  File "/lfs/ampere1/0/brando9/.vscode-server-insiders/extensions/ms-python.python-2023.14.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/lfs/ampere1/0/brando9/.vscode-server-insiders/extensions/ms-python.python-2023.14.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/lfs/ampere1/0/brando9/.vscode-server-insiders/extensions/ms-python.python-2023.14.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "/lfs/ampere1/0/brando9/beyond-scale-language-data-diversity/src/diversity/div_coeff.py", line 526, in <module>
    experiment_compute_diveristy_coeff_single_dataset_then_combined_datasets_with_domain_weights()
  File "/lfs/ampere1/0/brando9/beyond-scale-language-data-diversity/src/diversity/div_coeff.py", line 475, in experiment_compute_diveristy_coeff_single_dataset_then_combined_datasets_with_domain_weights
    column_names = next(iter(dataset)).keys()
  File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 1353, in __iter__
    for key, example in ex_iterable:
  File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 207, in __iter__
    yield from self.generate_examples_fn(**self.kwargs)
  File "/lfs/ampere1/0/brando9/.cache/huggingface/modules/datasets_modules/datasets/EleutherAI--pile/ebea56d358e91cf4d37b0fde361d563bed1472fbd8221a21b38fc8bb4ba554fb/pile.py", line 257, in _generate_examples
    for path, file in files[subset]:
  File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/datasets/download/streaming_download_manager.py", line 840, in __iter__
    yield from self.generator(*self.args, **self.kwargs)
  File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/datasets/download/streaming_download_manager.py", line 891, in _iter_from_urlpath
    with xopen(urlpath, "rb", download_config=download_config) as f:
  File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/datasets/download/streaming_download_manager.py", line 496, in xopen
    file_obj = fsspec.open(file, mode=mode, *args, **kwargs).open()
  File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/core.py", line 134, in open
    return self.__enter__()
  File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/core.py", line 102, in __enter__
    f = self.fs.open(self.path, mode=mode)
  File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/spec.py", line 1241, in open
    f = self._open(
  File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/implementations/http.py", line 356, in _open
    size = size or self.info(path, **kwargs)["size"]
  File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/asyn.py", line 121, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/asyn.py", line 106, in sync
    raise return_result
  File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/asyn.py", line 61, in _runner
    result[0] = await coro
  File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/implementations/http.py", line 430, in _info
    raise FileNotFoundError(url) from exc
FileNotFoundError: https://the-eye.eu/public/AI/pile_preliminary_components/pile_uspto.tar
```

```
FileNotFoundError: https://the-eye.eu/public/AI/pile_preliminary_components/pile_uspto.tar
```
most relevant line I think.

link to tweet: https://twitter.com/BrandoHablando/status/1690081313519489024?s=20 about issue

this seems to work but it's rather annoying.

Summary of how to make it work:
1. get urls to parquet files into a list
2. load list to load_dataset via `load_dataset('parquet', data_files=urls)` (note api names to hf are really confusing sometimes)
3. then it should work, print a batch of text.

presudo code
```python
urls_hacker_news = [
    "https://huggingface.co/datasets/EleutherAI/pile/resolve/refs%2Fconvert%2Fparquet/hacker_news/pile-train-00000-of-00004.parquet",
    "https://huggingface.co/datasets/EleutherAI/pile/resolve/refs%2Fconvert%2Fparquet/hacker_news/pile-train-00001-of-00004.parquet",
    "https://huggingface.co/datasets/EleutherAI/pile/resolve/refs%2Fconvert%2Fparquet/hacker_news/pile-train-00002-of-00004.parquet",
    "https://huggingface.co/datasets/EleutherAI/pile/resolve/refs%2Fconvert%2Fparquet/hacker_news/pile-train-00003-of-00004.parquet"
]

...

    # streaming = False
    from diversity.pile_subset_urls import urls_hacker_news
    path, name, data_files = 'parquet', 'hacker_news', urls_hacker_news
    # not changing
    batch_size = 512
    today = datetime.datetime.now().strftime('%Y-m%m-d%d-t%Hh_%Mm_%Ss')
    run_name = f'{path} div_coeff_{num_batches=} ({today=} ({name=}) {data_mixture_name=} {probabilities=})'
    print(f'{run_name=}')

    # - Init wandb
    debug: bool = mode == 'dryrun'
    run = wandb.init(mode=mode, project="beyond-scale", name=run_name, save_code=True)
    wandb.config.update({"num_batches": num_batches, "path": path, "name": name, "today": today, 'probabilities': probabilities, 'batch_size': batch_size, 'debug': debug, 'data_mixture_name': data_mixture_name, 'streaming': streaming, 'data_files': data_files})
    # run.notify_on_failure() # https://community.wandb.ai/t/how-do-i-set-the-wandb-alert-programatically-for-my-current-run/4891
    print(f'{debug=}')
    print(f'{wandb.config=}')

    # -- Get probe network
    from datasets import load_dataset
    import torch
    from transformers import GPT2Tokenizer, GPT2LMHeadModel

    tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
    if tokenizer.pad_token_id is None:
        tokenizer.pad_token = tokenizer.eos_token
    probe_network = GPT2LMHeadModel.from_pretrained("gpt2")
    device = torch.device(f"cuda:{0}" if torch.cuda.is_available() else "cpu")
    probe_network = probe_network.to(device)

    # -- Get data set
    def my_load_dataset(path, name):
        print(f'{path=} {name=} {streaming=}')
        if path == 'json' or path == 'bin' or path == 'csv':
            print(f'{data_files_prefix+name=}')
            return load_dataset(path, data_files=data_files_prefix+name, streaming=streaming, split="train").with_format("torch")
        elif path == 'parquet':
            print(f'{data_files=}')
            return load_dataset(path, data_files=data_files, streaming=streaming, split="train").with_format("torch")
        else:
            return load_dataset(path, name, streaming=streaming, split="train").with_format("torch")
    # - get data set for real now
    if isinstance(path, str):
        dataset = my_load_dataset(path, name)
    else:
        print('-- interleaving datasets')
        datasets = [my_load_dataset(path, name).with_format("torch") for path, name in zip(path, name)]
        [print(f'{dataset.description=}') for dataset in datasets]
        dataset = interleave_datasets(datasets, probabilities)
    print(f'{dataset=}')
    batch = dataset.take(batch_size)
    print(f'{next(iter(batch))=}')
    column_names = next(iter(batch)).keys()
    print(f'{column_names=}')

    # - Prepare functions to tokenize batch
    def preprocess(examples):
        return tokenizer(examples["text"], padding="max_length", max_length=128, truncation=True, return_tensors="pt")
    remove_columns = column_names  # remove all keys that are not tensors to avoid bugs in collate function in task2vec's pytorch data loader
    def map(batch):
        return batch.map(preprocess, batched=True, remove_columns=remove_columns)
    tokenized_batch = map(batch)
    print(f'{next(iter(tokenized_batch))=}')
```

https://stackoverflow.com/questions/76891189/how-to-download-data-from-hugging-face-that-is-visible-on-the-data-viewer-but-th/76902681#76902681

https://discuss.huggingface.co/t/how-to-download-data-from-hugging-face-that-is-visible-on-the-data-viewer-but-the-files-are-not-available/50555/5?u=severo

### Describe the bug

can't use or download the nih exporter pile data.
```
15     experiment_compute_diveristy_coeff_single_dataset_then_combined_datasets_with_domain_weights()
16   File "/lfs/ampere1/0/brando9/beyond-scale-language-data-diversity/src/diversity/div_coeff.py", line 474, in experiment_compute_diveristy_coeff_single_dataset_then_combined_datasets_with_domain_weights
17     column_names = next(iter(dataset)).keys()
18   File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 1353, in __iter__
19     for key, example in ex_iterable:
20   File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 207, in __iter__
21     yield from self.generate_examples_fn(**self.kwargs)
22   File "/lfs/ampere1/0/brando9/.cache/huggingface/modules/datasets_modules/datasets/EleutherAI--pile/ebea56d358e91cf4d37b0fde361d563bed1472fbd8221a21b38fc8bb4ba554fb/pile.py", line 236, in _generate_examples
23     with zstd.open(open(files[subset], "rb"), "rt", encoding="utf-8") as f:
24   File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/datasets/streaming.py", line 74, in wrapper
25     return function(*args, download_config=download_config, **kwargs)
26   File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/datasets/download/streaming_download_manager.py", line 496, in xopen
27     file_obj = fsspec.open(file, mode=mode, *args, **kwargs).open()
28   File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/core.py", line 134, in open
29     return self.__enter__()
30   File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/core.py", line 102, in __enter__
31     f = self.fs.open(self.path, mode=mode)
32   File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/spec.py", line 1241, in open
33     f = self._open(
34   File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/implementations/http.py", line 356, in _open
35     size = size or self.info(path, **kwargs)["size"]
36   File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/asyn.py", line 121, in wrapper
37     return sync(self.loop, func, *args, **kwargs)
38   File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/asyn.py", line 106, in sync
39     raise return_result
40   File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/asyn.py", line 61, in _runner
41     result[0] = await coro
42   File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/site-packages/fsspec/implementations/http.py", line 430, in _info
43     raise FileNotFoundError(url) from exc
44 FileNotFoundError: https://the-eye.eu/public/AI/pile_preliminary_components/NIH_ExPORTER_awarded_grant_text.jsonl.zst
```

### Steps to reproduce the bug

run this:
```
from datasets import load_dataset
path, name = 'EleutherAI/pile', 'nih_exporter'

# -- Get data set
dataset = load_dataset(path, name, streaming=True, split="train").with_format("torch")
batch = dataset.take(512)
print(f'{batch=}')
```

### Expected behavior

print the batch

### Environment info

```
(beyond_scale) brando9@ampere1:~/beyond-scale-language-data-diversity$ datasets-cli env

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 2.14.4
- Platform: Linux-5.4.0-122-generic-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
```


the-stack-dedup fails to generate

It seems that some parquet files have additional columns.

I ran a scan and found that two files have the additional `__id__` column:

1. `hf://datasets/bigcode/the-stack-dedup/data/numpy/data-00000-of-00001.parquet`
2. `hf://datasets/bigcode/the-stack-dedup/data/omgrofl/data-00000-of-00001.parquet`

We should open a PR to fix those two files

The files have been fixed ! I'm closing this one but feel free to re-open if you still have the issue

### Describe the bug

I'm getting an error generating the-stack-dedup with datasets 2.13.1, and with 2.14.4 nothing happens.

### Steps to reproduce the bug

My code:

```
import os
import datasets as ds

MY_CACHE_DIR = "/home/ubuntu/the-stack-dedup-local"
MY_TOKEN="my-token"

the_stack_ds = ds.load_dataset("bigcode/the-stack-dedup", split="train", download_mode="reuse_cache_if_exists", cache_dir=MY_CACHE_DIR, use_auth_token=MY_TOKEN, num_proc=64)
```

The exception:

```
Generating train split: 233248251 examples [54:31, 57280.00 examples/s]
multiprocess.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/datasets/build
er.py", line 1879, in _prepare_split_single
    for _, table in generator:
  File "/home/ubuntu/.local/lib/python3.10/site-packages/datasets/packa
ged_modules/parquet/parquet.py", line 82, in _generate_tables
    yield f"{file_idx}_{batch_idx}", self._cast_table(pa_table)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/datasets/packa
ged_modules/parquet/parquet.py", line 61, in _cast_table
    pa_table = table_cast(pa_table, self.info.features.arrow_schema)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/datasets/table
.py", line 2324, in table_cast
    return cast_table_to_schema(table, schema)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/datasets/table
.py", line 2282, in cast_table_to_schema
    raise ValueError(f"Couldn't cast\n{table.schema}\nto\n{features}\nb
ecause column names don't match")
ValueError: Couldn't cast
hexsha: string
size: int64
ext: string
lang: string
max_stars_repo_path: string
max_stars_repo_name: string
max_stars_repo_head_hexsha: string
max_stars_repo_licenses: list<item: string>
  child 0, item: string
max_stars_count: int64
max_stars_repo_stars_event_min_datetime: string
max_stars_repo_stars_event_max_datetime: string
max_issues_repo_path: string
max_issues_repo_name: string
max_issues_repo_head_hexsha: string
max_issues_repo_licenses: list<item: string>
  child 0, item: string
max_issues_count: int64
max_issues_repo_issues_event_min_datetime: string
max_issues_repo_issues_event_max_datetime: string
max_forks_repo_path: string
max_forks_repo_name: string
max_forks_repo_head_hexsha: string
max_forks_repo_licenses: list<item: string>
  child 0, item: string
max_forks_count: int64
max_forks_repo_forks_event_min_datetime: string
max_forks_repo_forks_event_max_datetime: string
content: string
avg_line_length: double
max_line_length: int64
alphanum_fraction: double
__id__: int64
-- schema metadata --
huggingface: '{"info": {"features": {"hexsha": {"dtype": "string", "_type' + 1979
to
{'hexsha': Value(dtype='string', id=None), 'size': Value(dtype='int64', id=None), 'ext': Value(dtype='string', id=None), 'lang': Value(dtype='string', id=None), 'max_stars_repo_path': Value(dtype='string', id=None), 'max_stars_repo_name': Value(dtype='string', id=None), 'max_stars_repo_head_hexsha': Value(dtype='string', id=None), 'max_stars_repo_licenses': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'max_stars_count': Value(dtype='int64', id=None), 'max_stars_repo_stars_event_min_datetime': Value(dtype='string', id=None), 'max_stars_repo_stars_event_max_datetime': Value(dtype='string', id=None), 'max_issues_repo_path': Value(dtype='string', id=None), 'max_issues_repo_name': Value(dtype='string', id=None), 'max_issues_repo_head_hexsha': Value(dtype='string', id=None), 'max_issues_repo_licenses': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'max_issues_count': Value(dtype='int64', id=None), 'max_issues_repo_issues_event_min_datetime': Value(dtype='string', id=None), 'max_issues_repo_issues_event_max_datetime': Value(dtype='string', id=None), 'max_forks_repo_path': Value(dtype='string', id=None), 'max_forks_repo_name': Value(dtype='string', id=None), 'max_forks_repo_head_hexsha': Value(dtype='string', id=None), 'max_forks_repo_licenses': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'max_forks_count': Value(dtype='int64', id=None), 'max_forks_repo_forks_event_min_datetime': Value(dtype='string', id=None), 'max_forks_repo_forks_event_max_datetime': Value(dtype='string', id=None), 'content': Value(dtype='string', id=None), 'avg_line_length': Value(dtype='float64', id=None), 'max_line_length': Value(dtype='int64', id=None), 'alphanum_fraction': Value(dtype='float64', id=None)}
because column names don't match

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/multiprocess/p
ool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/ubuntu/.local/lib/python3.10/site-packages/datasets/utils
/py_utils.py", line 1328, in _write_generator_to_queue
    for i, result in enumerate(func(**kwargs)):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/datasets/build
er.py", line 1912, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating th
e dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while genera
ting the dataset
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/download_the_stack.py", line 7, in <module>
    the_stack_ds = ds.load_dataset("bigcode/the-stack-dedup", split="tr
ain", download_mode="reuse_cache_if_exists", cache_dir=MY_CACHE_DIR, us
e_auth_token=MY_TOKEN, num_proc=64)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/datasets/load.
py", line 1809, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/datasets/build
er.py", line 909, in download_and_prepare
    self._download_and_prepare(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/datasets/build
er.py", line 1004, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/datasets/build
er.py", line 1796, in _prepare_split
    for job_id, done, content in iflatmap_unordered(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/datasets/utils
/py_utils.py", line 1354, in iflatmap_unordered
    [async_result.get(timeout=0.05) for async_result in async_results]
  File "/home/ubuntu/.local/lib/python3.10/site-packages/datasets/utils
/py_utils.py", line 1354, in <listcomp>
    [async_result.get(timeout=0.05) for async_result in async_results]
  File "/home/ubuntu/.local/lib/python3.10/site-packages/multiprocess/p
ool.py", line 774, in get
    raise self._value
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
```

### Expected behavior

The dataset downloads properly. @lhoestq @loub

### Environment info

Datasets 2.13.1, large VM with 2TB RAM, Ubuntu 20.04


Hi! I cannot reproduce this error on my machine or in Colab. Which version of `fsspec` do you have installed?

### Describe the bug

Hello, when I ran the [code snippet](https://huggingface.co/docs/datasets/v2.14.4/en/loading#json) on the document, I encountered the following problem:

```
Python 3.10.9 (main, Mar  1 2023, 18:23:06) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from datasets import load_dataset
>>> base_url = "https://rajpurkar.github.io/SQuAD-explorer/dataset/"
>>> dataset = load_dataset("json", data_files={"train": base_url + "train-v1.1.json", "validation": base_url + "dev-v1.1.json"}, field="data")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/liushuai/anaconda3/lib/python3.10/site-packages/datasets/load.py", line 2112, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/liushuai/anaconda3/lib/python3.10/site-packages/datasets/load.py", line 1798, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/home/liushuai/anaconda3/lib/python3.10/site-packages/datasets/load.py", line 1413, in dataset_module_factory
    ).get_module()
  File "/home/liushuai/anaconda3/lib/python3.10/site-packages/datasets/load.py", line 949, in get_module
    data_files = DataFilesDict.from_patterns(
  File "/home/liushuai/anaconda3/lib/python3.10/site-packages/datasets/data_files.py", line 672, in from_patterns
    DataFilesList.from_patterns(
  File "/home/liushuai/anaconda3/lib/python3.10/site-packages/datasets/data_files.py", line 578, in from_patterns
    resolve_pattern(
  File "/home/liushuai/anaconda3/lib/python3.10/site-packages/datasets/data_files.py", line 340, in resolve_pattern
    for filepath, info in fs.glob(pattern, detail=True).items()
  File "/home/liushuai/anaconda3/lib/python3.10/site-packages/fsspec/asyn.py", line 113, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/liushuai/anaconda3/lib/python3.10/site-packages/fsspec/asyn.py", line 98, in sync
    raise return_result
  File "/home/liushuai/anaconda3/lib/python3.10/site-packages/fsspec/asyn.py", line 53, in _runner
    result[0] = await coro
  File "/home/liushuai/anaconda3/lib/python3.10/site-packages/fsspec/implementations/http.py", line 449, in _glob
    elif await self._exists(path):
  File "/home/liushuai/anaconda3/lib/python3.10/site-packages/fsspec/implementations/http.py", line 306, in _exists
    r = await session.get(self.encode_url(path), **kw)
  File "/home/liushuai/anaconda3/lib/python3.10/site-packages/aiohttp/client.py", line 922, in get
    self._request(hdrs.METH_GET, url, allow_redirects=allow_redirects, **kwargs)
TypeError: ClientSession._request() got an unexpected keyword argument 'https'
```

### Steps to reproduce the bug

```
from datasets import load_dataset
base_url = "https://rajpurkar.github.io/SQuAD-explorer/dataset/"
dataset = load_dataset("json", data_files={"train": base_url + "train-v1.1.json", "validation": base_url + "dev-v1.1.json"}, field="data")
```

### Expected behavior

able to load normally

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.4.54-2-x86_64-with-glibc2.27
- Python version: 3.10.9
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.3


Misalignment between file format specified in configs metadata YAML and the inferred builder

bug

Something isn't working

There is a misalignment between the format of the `data_files` specified in the configs metadata YAML (CSV):
```yaml
configs:
  - config_name: default
    data_files:
      - split: train
        path: data.csv
```
and the inferred builder (JSON). Note there are multiple JSON files in the repo, but they do not appear in the configs metadata YAML.

See: https://huggingface.co/datasets/freddyaboulton/chatinterface_with_image_csv/discussions/1

CC: @freddyaboulton @polinaeterna


Offline dataset viewer

enhancement

New feature or request

dataset-viewer

Related to the dataset viewer on huggingface.co

Hi, thanks for the suggestion. It's not possible at the moment. The viewer is part of the Hub codebase and only works on public datasets. Also, it relies on [Datasets Server](https://github.com/huggingface/datasets-server/), which prepares the data and provides an API to access the rows, size, etc.

If you're interested in hosting your data as a private dataset on the Hub, you might want to look at https://github.com/huggingface/datasets-server/issues/39.

Hi, we are building an offline dataset viewer: https://github.com/Renumics/spotlight
It supports many HF datasets, but currently you have to use it via Pandas:
df=ds.to_pandas()
spotlight.show(df)

Would love to hear from you if that works for your use case. If not, feel free to open an issue on the repo: https://github.com/Renumics/spotlight/issues

@ssuwelack thank you! I will definitely try it out.

Related issues:
- https://github.com/huggingface/datasets-server/issues/213
- https://github.com/huggingface/datasets-server/issues/441
- https://github.com/huggingface/datasets/issues/6014

Closing for now, as developing and maintaining an offline viewer is not planned.

@yuvalkirstain the dataset viewer is now available on private datasets for [PRO users](https://huggingface.co/pricing#pro) and [Enterprise Hub orgs](https://huggingface.co/enterprise). Would it fit your needs?

### Feature request

The dataset viewer feature is very nice. It enables to the user to easily view the dataset. However, when working for private companies we cannot always upload the dataset to the hub. Is there a way to create dataset viewer offline? I.e. to run a code that will open some kind of html or something that makes it easy to view the dataset.

### Motivation

I want to easily view my dataset even when it is hosted locally.

### Your contribution

N.A.