Support huggingface-hub pre-releases

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6516). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005309 / 0.011353 (-0.006044) | 0.003231 / 0.011008 (-0.007777) | 0.062690 / 0.038508 (0.024182) | 0.050811 / 0.023109 (0.027701) | 0.258319 / 0.275898 (-0.017579) | 0.275977 / 0.323480 (-0.047503) | 0.002842 / 0.007986 (-0.005143) | 0.002606 / 0.004328 (-0.001723) | 0.048672 / 0.004250 (0.044421) | 0.038730 / 0.037052 (0.001677) | 0.258531 / 0.258489 (0.000042) | 0.289327 / 0.293841 (-0.004514) | 0.027994 / 0.128546 (-0.100552) | 0.010446 / 0.075646 (-0.065200) | 0.207152 / 0.419271 (-0.212119) | 0.035839 / 0.043533 (-0.007693) | 0.258416 / 0.255139 (0.003277) | 0.274348 / 0.283200 (-0.008851) | 0.019661 / 0.141683 (-0.122022) | 1.103688 / 1.452155 (-0.348466) | 1.207711 / 1.492716 (-0.285006) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090693 / 0.018006 (0.072687) | 0.300648 / 0.000490 (0.300158) | 0.000215 / 0.000200 (0.000015) | 0.000053 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018589 / 0.037411 (-0.018822) | 0.061056 / 0.014526 (0.046530) | 0.074512 / 0.176557 (-0.102044) | 0.121260 / 0.737135 (-0.615875) | 0.073111 / 0.296338 (-0.223227) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.285811 / 0.215209 (0.070602) | 2.785081 / 2.077655 (0.707426) | 1.469493 / 1.504120 (-0.034627) | 1.346389 / 1.541195 (-0.194806) | 1.391866 / 1.468490 (-0.076624) | 0.567304 / 4.584777 (-4.017473) | 2.407150 / 3.745712 (-1.338562) | 2.809915 / 5.269862 (-2.459946) | 1.741185 / 4.565676 (-2.824491) | 0.063073 / 0.424275 (-0.361202) | 0.004974 / 0.007607 (-0.002633) | 0.336431 / 0.226044 (0.110386) | 3.331371 / 2.268929 (1.062443) | 1.841466 / 55.444624 (-53.603159) | 1.559065 / 6.876477 (-5.317411) | 1.585033 / 2.142072 (-0.557039) | 0.647469 / 4.805227 (-4.157759) | 0.117488 / 6.500664 (-6.383176) | 0.042535 / 0.075469 (-0.032934) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.936409 / 1.841788 (-0.905379) | 11.301514 / 8.074308 (3.227206) | 10.500465 / 10.191392 (0.309073) | 0.131316 / 0.680424 (-0.549107) | 0.014007 / 0.534201 (-0.520194) | 0.286932 / 0.579283 (-0.292351) | 0.263516 / 0.434364 (-0.170848) | 0.340883 / 0.540337 (-0.199454) | 0.443589 / 1.386936 (-0.943347) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005204 / 0.011353 (-0.006149) | 0.003472 / 0.011008 (-0.007536) | 0.049235 / 0.038508 (0.010727) | 0.050668 / 0.023109 (0.027559) | 0.270198 / 0.275898 (-0.005700) | 0.293942 / 0.323480 (-0.029538) | 0.003964 / 0.007986 (-0.004022) | 0.002596 / 0.004328 (-0.001733) | 0.048654 / 0.004250 (0.044404) | 0.039411 / 0.037052 (0.002358) | 0.271938 / 0.258489 (0.013449) | 0.304308 / 0.293841 (0.010467) | 0.029042 / 0.128546 (-0.099504) | 0.010414 / 0.075646 (-0.065232) | 0.058273 / 0.419271 (-0.360999) | 0.032507 / 0.043533 (-0.011025) | 0.271671 / 0.255139 (0.016532) | 0.289850 / 0.283200 (0.006650) | 0.017292 / 0.141683 (-0.124391) | 1.126160 / 1.452155 (-0.325995) | 1.177365 / 1.492716 (-0.315351) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091158 / 0.018006 (0.073152) | 0.299143 / 0.000490 (0.298653) | 0.000217 / 0.000200 (0.000017) | 0.000042 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022558 / 0.037411 (-0.014853) | 0.076139 / 0.014526 (0.061613) | 0.088344 / 0.176557 (-0.088212) | 0.126640 / 0.737135 (-0.610495) | 0.089736 / 0.296338 (-0.206602) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295351 / 0.215209 (0.080142) | 2.895779 / 2.077655 (0.818125) | 1.585886 / 1.504120 (0.081766) | 1.458601 / 1.541195 (-0.082594) | 1.468880 / 1.468490 (0.000390) | 0.554686 / 4.584777 (-4.030091) | 2.466276 / 3.745712 (-1.279437) | 2.741938 / 5.269862 (-2.527924) | 1.711793 / 4.565676 (-2.853883) | 0.062928 / 0.424275 (-0.361347) | 0.005177 / 0.007607 (-0.002430) | 0.343908 / 0.226044 (0.117863) | 3.393360 / 2.268929 (1.124431) | 1.928800 / 55.444624 (-53.515824) | 1.652181 / 6.876477 (-5.224296) | 1.643667 / 2.142072 (-0.498405) | 0.632829 / 4.805227 (-4.172398) | 0.114583 / 6.500664 (-6.386081) | 0.041248 / 0.075469 (-0.034221) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.986196 / 1.841788 (-0.855592) | 12.006772 / 8.074308 (3.932464) | 10.522661 / 10.191392 (0.331269) | 0.133710 / 0.680424 (-0.546713) | 0.016714 / 0.534201 (-0.517487) | 0.286502 / 0.579283 (-0.292781) | 0.280090 / 0.434364 (-0.154273) | 0.326063 / 0.540337 (-0.214275) | 0.548485 / 1.386936 (-0.838452) |

![](https://cml.dev/watermark.png#3f149204a2a5948287adcade5e90707aa5207a92 "CML watermark")

Support `huggingface-hub` pre-releases.

This way we will have our CI green when testing `huggingface-hub` release candidates. See: https://github.com/huggingface/datasets/tree/ci-test-huggingface-hub-v0.20.0.rc1

Close #6513.


Why call http_head() when fsspec_head()  succeeds


Cache backward compatibility with 2.15.0

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6514). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

it's hard to tell if this works as expected without a test but i guess it's not trivial to implement such a test.

i tried to reproduce locally (with this branch merged into the lazy-resolve-and-cache-reload) and it didn't work.
I run:
```
 ds = load_dataset("polinaeterna/audiofolder_two_configs_in_metadata", "v2", data_files="v2/train/*")
```
and i got this in the cache:
```
v2-374bfde4f55442bc/
└── 0.0.0
    ├── 5a2339ad2bb7caf6a6daf2f213204e3ac03a13a5  # - from this pr
    │   ├── audiofolder_two_configs_in_metadata-train.arrow
    │   └── dataset_info.json
    ├── 5a2339ad2bb7caf6a6daf2f213204e3ac03a13a5_builder.lock
    ├── 5a2339ad2bb7caf6a6daf2f213204e3ac03a13a5.incomplete_info.lock
    ├── 7896925d64deea5d  # from 2.15.0
    │   ├── audiofolder_two_configs_in_metadata-train.arrow
    │   └── dataset_info.json
    ├── 7896925d64deea5d_builder.lock
    └── 7896925d64deea5d.incomplete_info.lock
```
so the first hash (the top-level dir v2-374bfde4f55442bc) matches but the second (after version) doesn't.
maybe i did something wrong though.

also i'm not sure if this is worth too much effort, maybe nobody notices if their datasets will be generated again :D idk

I just pushed a fix, it should work just fine now :)

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004798 / 0.011353 (-0.006555) | 0.003203 / 0.011008 (-0.007805) | 0.062247 / 0.038508 (0.023738) | 0.029906 / 0.023109 (0.006797) | 0.259370 / 0.275898 (-0.016528) | 0.276084 / 0.323480 (-0.047396) | 0.002910 / 0.007986 (-0.005076) | 0.002364 / 0.004328 (-0.001964) | 0.048080 / 0.004250 (0.043830) | 0.041168 / 0.037052 (0.004116) | 0.259833 / 0.258489 (0.001343) | 0.289882 / 0.293841 (-0.003959) | 0.026790 / 0.128546 (-0.101756) | 0.010336 / 0.075646 (-0.065311) | 0.209628 / 0.419271 (-0.209643) | 0.035080 / 0.043533 (-0.008452) | 0.256278 / 0.255139 (0.001139) | 0.279502 / 0.283200 (-0.003697) | 0.019755 / 0.141683 (-0.121928) | 1.121552 / 1.452155 (-0.330602) | 1.174360 / 1.492716 (-0.318356) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093510 / 0.018006 (0.075504) | 0.302065 / 0.000490 (0.301575) | 0.000214 / 0.000200 (0.000014) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.017652 / 0.037411 (-0.019759) | 0.060512 / 0.014526 (0.045986) | 0.072441 / 0.176557 (-0.104115) | 0.118058 / 0.737135 (-0.619078) | 0.072657 / 0.296338 (-0.223682) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.283949 / 0.215209 (0.068740) | 2.803275 / 2.077655 (0.725620) | 1.527353 / 1.504120 (0.023233) | 1.408176 / 1.541195 (-0.133019) | 1.375335 / 1.468490 (-0.093155) | 0.546426 / 4.584777 (-4.038351) | 2.402210 / 3.745712 (-1.343502) | 2.765879 / 5.269862 (-2.503982) | 1.703722 / 4.565676 (-2.861955) | 0.062669 / 0.424275 (-0.361606) | 0.005006 / 0.007607 (-0.002601) | 0.337941 / 0.226044 (0.111897) | 3.385494 / 2.268929 (1.116566) | 1.817360 / 55.444624 (-53.627264) | 1.548594 / 6.876477 (-5.327883) | 1.548610 / 2.142072 (-0.593463) | 0.630188 / 4.805227 (-4.175040) | 0.117079 / 6.500664 (-6.383585) | 0.042077 / 0.075469 (-0.033392) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.941606 / 1.841788 (-0.900182) | 11.226277 / 8.074308 (3.151969) | 10.118005 / 10.191392 (-0.073387) | 0.130408 / 0.680424 (-0.550015) | 0.014419 / 0.534201 (-0.519782) | 0.284812 / 0.579283 (-0.294471) | 0.266951 / 0.434364 (-0.167413) | 0.322251 / 0.540337 (-0.218087) | 0.415014 / 1.386936 (-0.971922) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005192 / 0.011353 (-0.006161) | 0.003028 / 0.011008 (-0.007980) | 0.048322 / 0.038508 (0.009814) | 0.030550 / 0.023109 (0.007441) | 0.264360 / 0.275898 (-0.011538) | 0.289544 / 0.323480 (-0.033936) | 0.004053 / 0.007986 (-0.003933) | 0.002480 / 0.004328 (-0.001848) | 0.048215 / 0.004250 (0.043964) | 0.044208 / 0.037052 (0.007156) | 0.263943 / 0.258489 (0.005454) | 0.297648 / 0.293841 (0.003807) | 0.029315 / 0.128546 (-0.099231) | 0.010533 / 0.075646 (-0.065114) | 0.057021 / 0.419271 (-0.362251) | 0.053751 / 0.043533 (0.010218) | 0.265153 / 0.255139 (0.010014) | 0.284988 / 0.283200 (0.001788) | 0.018459 / 0.141683 (-0.123224) | 1.225657 / 1.452155 (-0.226498) | 1.195737 / 1.492716 (-0.296979) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093030 / 0.018006 (0.075024) | 0.301022 / 0.000490 (0.300533) | 0.000228 / 0.000200 (0.000028) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022073 / 0.037411 (-0.015339) | 0.075912 / 0.014526 (0.061386) | 0.087628 / 0.176557 (-0.088929) | 0.125607 / 0.737135 (-0.611529) | 0.088568 / 0.296338 (-0.207770) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.303482 / 0.215209 (0.088273) | 2.965987 / 2.077655 (0.888333) | 1.615273 / 1.504120 (0.111153) | 1.482851 / 1.541195 (-0.058344) | 1.562627 / 1.468490 (0.094137) | 0.563626 / 4.584777 (-4.021151) | 2.448741 / 3.745712 (-1.296971) | 2.761006 / 5.269862 (-2.508855) | 1.711242 / 4.565676 (-2.854434) | 0.064593 / 0.424275 (-0.359682) | 0.005044 / 0.007607 (-0.002563) | 0.354131 / 0.226044 (0.128087) | 3.511698 / 2.268929 (1.242770) | 1.951087 / 55.444624 (-53.493538) | 1.682171 / 6.876477 (-5.194305) | 1.666330 / 2.142072 (-0.475742) | 0.654880 / 4.805227 (-4.150347) | 0.118544 / 6.500664 (-6.382120) | 0.040753 / 0.075469 (-0.034717) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.967771 / 1.841788 (-0.874017) | 12.017277 / 8.074308 (3.942969) | 10.624947 / 10.191392 (0.433555) | 0.128834 / 0.680424 (-0.551590) | 0.015739 / 0.534201 (-0.518462) | 0.285906 / 0.579283 (-0.293377) | 0.273659 / 0.434364 (-0.160705) | 0.324044 / 0.540337 (-0.216293) | 0.419469 / 1.386936 (-0.967467) |

![](https://cml.dev/watermark.png#2afbf785f8d0551cdd65a81c5c3228e469613724 "CML watermark")

...for datasets without scripts

It takes into account the changes in cache from
- https://github.com/huggingface/datasets/pull/6493: switch to `config/version/commit_sha` schema
- https://github.com/huggingface/datasets/pull/6454: fix `DataFilesDict` keys ordering when hashing

requires https://github.com/huggingface/datasets/pull/6493 to be merged


Support huggingface-hub 0.20.0

CI to test the support of `huggingface-hub` 0.20.0: https://github.com/huggingface/datasets/compare/main...ci-test-huggingface-hub-v0.20.0.rc1

We need to merge:
- #6510
- #6512
- #6516


Remove deprecated HfFolder

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6512). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005468 / 0.011353 (-0.005885) | 0.003447 / 0.011008 (-0.007561) | 0.062569 / 0.038508 (0.024061) | 0.049427 / 0.023109 (0.026318) | 0.238463 / 0.275898 (-0.037435) | 0.268320 / 0.323480 (-0.055159) | 0.002834 / 0.007986 (-0.005151) | 0.002679 / 0.004328 (-0.001649) | 0.048613 / 0.004250 (0.044363) | 0.038793 / 0.037052 (0.001741) | 0.247710 / 0.258489 (-0.010779) | 0.277557 / 0.293841 (-0.016284) | 0.027134 / 0.128546 (-0.101412) | 0.010346 / 0.075646 (-0.065301) | 0.205782 / 0.419271 (-0.213490) | 0.035549 / 0.043533 (-0.007983) | 0.241667 / 0.255139 (-0.013472) | 0.268358 / 0.283200 (-0.014842) | 0.017119 / 0.141683 (-0.124563) | 1.108487 / 1.452155 (-0.343668) | 1.177519 / 1.492716 (-0.315197) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090925 / 0.018006 (0.072919) | 0.310422 / 0.000490 (0.309932) | 0.000212 / 0.000200 (0.000012) | 0.000053 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018912 / 0.037411 (-0.018499) | 0.061534 / 0.014526 (0.047008) | 0.073608 / 0.176557 (-0.102949) | 0.119278 / 0.737135 (-0.617858) | 0.074698 / 0.296338 (-0.221640) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.287224 / 0.215209 (0.072014) | 2.792022 / 2.077655 (0.714367) | 1.474605 / 1.504120 (-0.029515) | 1.348714 / 1.541195 (-0.192481) | 1.381339 / 1.468490 (-0.087151) | 0.553033 / 4.584777 (-4.031744) | 2.360745 / 3.745712 (-1.384967) | 2.779281 / 5.269862 (-2.490580) | 1.743922 / 4.565676 (-2.821754) | 0.063817 / 0.424275 (-0.360458) | 0.004954 / 0.007607 (-0.002653) | 0.340039 / 0.226044 (0.113994) | 3.336771 / 2.268929 (1.067843) | 1.825573 / 55.444624 (-53.619051) | 1.525362 / 6.876477 (-5.351115) | 1.578793 / 2.142072 (-0.563280) | 0.638432 / 4.805227 (-4.166795) | 0.117601 / 6.500664 (-6.383063) | 0.041890 / 0.075469 (-0.033579) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.936896 / 1.841788 (-0.904892) | 11.426979 / 8.074308 (3.352671) | 10.636043 / 10.191392 (0.444651) | 0.136172 / 0.680424 (-0.544252) | 0.014249 / 0.534201 (-0.519952) | 0.287806 / 0.579283 (-0.291477) | 0.266046 / 0.434364 (-0.168318) | 0.326155 / 0.540337 (-0.214183) | 0.455508 / 1.386936 (-0.931428) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005199 / 0.011353 (-0.006154) | 0.003476 / 0.011008 (-0.007532) | 0.050519 / 0.038508 (0.012011) | 0.050732 / 0.023109 (0.027623) | 0.270140 / 0.275898 (-0.005758) | 0.295539 / 0.323480 (-0.027941) | 0.004057 / 0.007986 (-0.003928) | 0.002771 / 0.004328 (-0.001558) | 0.049157 / 0.004250 (0.044906) | 0.039863 / 0.037052 (0.002811) | 0.275934 / 0.258489 (0.017445) | 0.306971 / 0.293841 (0.013130) | 0.029405 / 0.128546 (-0.099141) | 0.010746 / 0.075646 (-0.064900) | 0.058427 / 0.419271 (-0.360845) | 0.032448 / 0.043533 (-0.011085) | 0.271851 / 0.255139 (0.016712) | 0.290337 / 0.283200 (0.007138) | 0.019145 / 0.141683 (-0.122538) | 1.112232 / 1.452155 (-0.339922) | 1.215153 / 1.492716 (-0.277564) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.088590 / 0.018006 (0.070584) | 0.299047 / 0.000490 (0.298558) | 0.000219 / 0.000200 (0.000019) | 0.000050 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022755 / 0.037411 (-0.014656) | 0.078720 / 0.014526 (0.064194) | 0.089051 / 0.176557 (-0.087505) | 0.129330 / 0.737135 (-0.607805) | 0.090645 / 0.296338 (-0.205693) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.294083 / 0.215209 (0.078874) | 2.907195 / 2.077655 (0.829540) | 1.607392 / 1.504120 (0.103272) | 1.481931 / 1.541195 (-0.059263) | 1.486934 / 1.468490 (0.018444) | 0.574093 / 4.584777 (-4.010684) | 2.439775 / 3.745712 (-1.305937) | 2.739818 / 5.269862 (-2.530044) | 1.753922 / 4.565676 (-2.811755) | 0.063738 / 0.424275 (-0.360537) | 0.005219 / 0.007607 (-0.002388) | 0.350342 / 0.226044 (0.124297) | 3.463644 / 2.268929 (1.194716) | 1.971598 / 55.444624 (-53.473026) | 1.671752 / 6.876477 (-5.204724) | 1.686504 / 2.142072 (-0.455569) | 0.655870 / 4.805227 (-4.149357) | 0.117580 / 6.500664 (-6.383084) | 0.041210 / 0.075469 (-0.034259) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.996305 / 1.841788 (-0.845482) | 12.426361 / 8.074308 (4.352053) | 10.600309 / 10.191392 (0.408917) | 0.129728 / 0.680424 (-0.550695) | 0.015267 / 0.534201 (-0.518934) | 0.285444 / 0.579283 (-0.293839) | 0.272375 / 0.434364 (-0.161989) | 0.323478 / 0.540337 (-0.216860) | 0.547566 / 1.386936 (-0.839370) |

![](https://cml.dev/watermark.png#a91582de288d98e94bcb5ab634ca1cfeeff544c5 "CML watermark")

...and use `huggingface_hub.get_token()` instead


Implement get dataset default config name

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6511). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@huggingface/datasets, this PR is ready for review.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005343 / 0.011353 (-0.006010) | 0.003521 / 0.011008 (-0.007487) | 0.061835 / 0.038508 (0.023327) | 0.052633 / 0.023109 (0.029524) | 0.243897 / 0.275898 (-0.032001) | 0.272961 / 0.323480 (-0.050519) | 0.003013 / 0.007986 (-0.004973) | 0.002692 / 0.004328 (-0.001636) | 0.050099 / 0.004250 (0.045848) | 0.045381 / 0.037052 (0.008329) | 0.249981 / 0.258489 (-0.008508) | 0.276789 / 0.293841 (-0.017052) | 0.027929 / 0.128546 (-0.100617) | 0.010933 / 0.075646 (-0.064714) | 0.206757 / 0.419271 (-0.212514) | 0.035334 / 0.043533 (-0.008199) | 0.249411 / 0.255139 (-0.005728) | 0.268893 / 0.283200 (-0.014306) | 0.019175 / 0.141683 (-0.122507) | 1.106932 / 1.452155 (-0.345223) | 1.177819 / 1.492716 (-0.314897) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092895 / 0.018006 (0.074889) | 0.303658 / 0.000490 (0.303169) | 0.000214 / 0.000200 (0.000014) | 0.000050 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018978 / 0.037411 (-0.018434) | 0.060459 / 0.014526 (0.045934) | 0.072900 / 0.176557 (-0.103657) | 0.119803 / 0.737135 (-0.617332) | 0.074349 / 0.296338 (-0.221989) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.283715 / 0.215209 (0.068505) | 2.752394 / 2.077655 (0.674739) | 1.446619 / 1.504120 (-0.057501) | 1.319612 / 1.541195 (-0.221582) | 1.374769 / 1.468490 (-0.093721) | 0.571543 / 4.584777 (-4.013234) | 2.389106 / 3.745712 (-1.356607) | 2.797837 / 5.269862 (-2.472025) | 1.737615 / 4.565676 (-2.828062) | 0.063268 / 0.424275 (-0.361007) | 0.005118 / 0.007607 (-0.002489) | 0.340238 / 0.226044 (0.114193) | 3.366207 / 2.268929 (1.097278) | 1.845934 / 55.444624 (-53.598690) | 1.540640 / 6.876477 (-5.335837) | 1.585489 / 2.142072 (-0.556584) | 0.641178 / 4.805227 (-4.164049) | 0.118701 / 6.500664 (-6.381964) | 0.042719 / 0.075469 (-0.032750) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.946706 / 1.841788 (-0.895082) | 11.846230 / 8.074308 (3.771921) | 10.459268 / 10.191392 (0.267876) | 0.130557 / 0.680424 (-0.549867) | 0.014292 / 0.534201 (-0.519909) | 0.287455 / 0.579283 (-0.291828) | 0.265213 / 0.434364 (-0.169151) | 0.325670 / 0.540337 (-0.214667) | 0.422800 / 1.386936 (-0.964136) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005454 / 0.011353 (-0.005899) | 0.003567 / 0.011008 (-0.007441) | 0.048696 / 0.038508 (0.010188) | 0.058844 / 0.023109 (0.035735) | 0.277011 / 0.275898 (0.001113) | 0.302544 / 0.323480 (-0.020936) | 0.004077 / 0.007986 (-0.003908) | 0.002720 / 0.004328 (-0.001609) | 0.058251 / 0.004250 (0.054001) | 0.040946 / 0.037052 (0.003893) | 0.276261 / 0.258489 (0.017772) | 0.352827 / 0.293841 (0.058986) | 0.029915 / 0.128546 (-0.098632) | 0.010562 / 0.075646 (-0.065084) | 0.057836 / 0.419271 (-0.361436) | 0.033129 / 0.043533 (-0.010404) | 0.276053 / 0.255139 (0.020914) | 0.292045 / 0.283200 (0.008846) | 0.020504 / 0.141683 (-0.121179) | 1.129746 / 1.452155 (-0.322409) | 1.190888 / 1.492716 (-0.301829) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095202 / 0.018006 (0.077196) | 0.303956 / 0.000490 (0.303466) | 0.000226 / 0.000200 (0.000026) | 0.000054 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021960 / 0.037411 (-0.015451) | 0.076209 / 0.014526 (0.061683) | 0.088813 / 0.176557 (-0.087744) | 0.129061 / 0.737135 (-0.608074) | 0.091202 / 0.296338 (-0.205136) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.301394 / 0.215209 (0.086185) | 2.948057 / 2.077655 (0.870403) | 1.591371 / 1.504120 (0.087251) | 1.463515 / 1.541195 (-0.077680) | 1.516477 / 1.468490 (0.047987) | 0.577223 / 4.584777 (-4.007554) | 2.506716 / 3.745712 (-1.238996) | 2.833385 / 5.269862 (-2.436477) | 1.808896 / 4.565676 (-2.756781) | 0.063241 / 0.424275 (-0.361034) | 0.005057 / 0.007607 (-0.002550) | 0.350108 / 0.226044 (0.124063) | 3.470252 / 2.268929 (1.201324) | 1.925689 / 55.444624 (-53.518935) | 1.667521 / 6.876477 (-5.208955) | 1.690909 / 2.142072 (-0.451164) | 0.647070 / 4.805227 (-4.158157) | 0.117596 / 6.500664 (-6.383068) | 0.042431 / 0.075469 (-0.033038) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.977297 / 1.841788 (-0.864490) | 12.947399 / 8.074308 (4.873091) | 10.964949 / 10.191392 (0.773557) | 0.130905 / 0.680424 (-0.549518) | 0.015207 / 0.534201 (-0.518994) | 0.288151 / 0.579283 (-0.291132) | 0.281817 / 0.434364 (-0.152547) | 0.326398 / 0.540337 (-0.213940) | 0.421354 / 1.386936 (-0.965582) |

![](https://cml.dev/watermark.png#8b04288f0b94c987a278c5bb8459746bc35ba367 "CML watermark")

Implement `get_dataset_default_config_name`.

Now that we support setting a configuration as default in `push_to_hub` (see #6500), we need a programmatically way to know in advance which is the default configuration. This will be used in the Space to convert script-datasets to Parquet: https://huggingface.co/spaces/albertvillanova/convert-dataset-to-parquet

Follow-up of:
- #6500

CC: @severo


Replace `list_files_info` with `list_repo_tree` in `push_to_hub`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6510). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

CI errors are unrelated to the changes, so I'm merging.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005161 / 0.011353 (-0.006192) | 0.003494 / 0.011008 (-0.007515) | 0.062601 / 0.038508 (0.024093) | 0.052876 / 0.023109 (0.029767) | 0.255595 / 0.275898 (-0.020303) | 0.283108 / 0.323480 (-0.040371) | 0.003856 / 0.007986 (-0.004130) | 0.002686 / 0.004328 (-0.001642) | 0.048604 / 0.004250 (0.044353) | 0.037886 / 0.037052 (0.000834) | 0.252902 / 0.258489 (-0.005587) | 0.286906 / 0.293841 (-0.006935) | 0.028570 / 0.128546 (-0.099976) | 0.010684 / 0.075646 (-0.064962) | 0.208154 / 0.419271 (-0.211118) | 0.036169 / 0.043533 (-0.007364) | 0.276026 / 0.255139 (0.020887) | 0.272274 / 0.283200 (-0.010925) | 0.017690 / 0.141683 (-0.123993) | 1.202400 / 1.452155 (-0.249755) | 1.231223 / 1.492716 (-0.261494) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095229 / 0.018006 (0.077222) | 0.302205 / 0.000490 (0.301716) | 0.000226 / 0.000200 (0.000026) | 0.000045 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018877 / 0.037411 (-0.018534) | 0.062286 / 0.014526 (0.047760) | 0.075191 / 0.176557 (-0.101366) | 0.121419 / 0.737135 (-0.615716) | 0.075641 / 0.296338 (-0.220697) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282914 / 0.215209 (0.067705) | 2.769156 / 2.077655 (0.691501) | 1.480219 / 1.504120 (-0.023901) | 1.355742 / 1.541195 (-0.185453) | 1.399740 / 1.468490 (-0.068750) | 0.556365 / 4.584777 (-4.028412) | 2.399679 / 3.745712 (-1.346033) | 2.850510 / 5.269862 (-2.419351) | 1.781428 / 4.565676 (-2.784249) | 0.063045 / 0.424275 (-0.361230) | 0.004931 / 0.007607 (-0.002676) | 0.343743 / 0.226044 (0.117698) | 3.374907 / 2.268929 (1.105978) | 1.857774 / 55.444624 (-53.586851) | 1.577154 / 6.876477 (-5.299323) | 1.626597 / 2.142072 (-0.515475) | 0.653991 / 4.805227 (-4.151236) | 0.121306 / 6.500664 (-6.379358) | 0.042131 / 0.075469 (-0.033339) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.948826 / 1.841788 (-0.892962) | 11.922497 / 8.074308 (3.848188) | 10.592334 / 10.191392 (0.400942) | 0.129145 / 0.680424 (-0.551279) | 0.014652 / 0.534201 (-0.519549) | 0.286074 / 0.579283 (-0.293210) | 0.265338 / 0.434364 (-0.169026) | 0.346872 / 0.540337 (-0.193466) | 0.450480 / 1.386936 (-0.936456) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005305 / 0.011353 (-0.006048) | 0.003583 / 0.011008 (-0.007426) | 0.049855 / 0.038508 (0.011347) | 0.052882 / 0.023109 (0.029773) | 0.268429 / 0.275898 (-0.007469) | 0.293375 / 0.323480 (-0.030105) | 0.004052 / 0.007986 (-0.003934) | 0.002685 / 0.004328 (-0.001644) | 0.049206 / 0.004250 (0.044955) | 0.040187 / 0.037052 (0.003135) | 0.270112 / 0.258489 (0.011623) | 0.306380 / 0.293841 (0.012539) | 0.029161 / 0.128546 (-0.099386) | 0.010948 / 0.075646 (-0.064698) | 0.057721 / 0.419271 (-0.361550) | 0.032628 / 0.043533 (-0.010905) | 0.267458 / 0.255139 (0.012319) | 0.291905 / 0.283200 (0.008705) | 0.018096 / 0.141683 (-0.123587) | 1.112744 / 1.452155 (-0.339410) | 1.161962 / 1.492716 (-0.330754) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.097449 / 0.018006 (0.079443) | 0.304270 / 0.000490 (0.303780) | 0.000235 / 0.000200 (0.000035) | 0.000060 / 0.000054 (0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023550 / 0.037411 (-0.013861) | 0.078246 / 0.014526 (0.063720) | 0.091229 / 0.176557 (-0.085327) | 0.130624 / 0.737135 (-0.606511) | 0.092767 / 0.296338 (-0.203571) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.284962 / 0.215209 (0.069753) | 2.761090 / 2.077655 (0.683435) | 1.545409 / 1.504120 (0.041289) | 1.424573 / 1.541195 (-0.116622) | 1.438869 / 1.468490 (-0.029621) | 0.571281 / 4.584777 (-4.013496) | 2.419493 / 3.745712 (-1.326219) | 2.802611 / 5.269862 (-2.467251) | 1.749880 / 4.565676 (-2.815796) | 0.062566 / 0.424275 (-0.361709) | 0.005243 / 0.007607 (-0.002364) | 0.344653 / 0.226044 (0.118608) | 3.367488 / 2.268929 (1.098559) | 1.925871 / 55.444624 (-53.518754) | 1.624258 / 6.876477 (-5.252219) | 1.663742 / 2.142072 (-0.478330) | 0.634553 / 4.805227 (-4.170675) | 0.116745 / 6.500664 (-6.383919) | 0.041734 / 0.075469 (-0.033735) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.006808 / 1.841788 (-0.834980) | 12.499711 / 8.074308 (4.425403) | 10.956260 / 10.191392 (0.764868) | 0.132393 / 0.680424 (-0.548031) | 0.015924 / 0.534201 (-0.518277) | 0.289837 / 0.579283 (-0.289446) | 0.281565 / 0.434364 (-0.152799) | 0.337393 / 0.540337 (-0.202945) | 0.560385 / 1.386936 (-0.826551) |

![](https://cml.dev/watermark.png#3f699ab27ef2c0c23dc3a514b5bb155485ff6913 "CML watermark")

Starting from `huggingface_hub` 0.20.0, `list_files_info` will be deprecated in favor of `list_repo_tree` (see https://github.com/huggingface/huggingface_hub/pull/1910)


Better cast error when generating dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6509). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I created `DatatasetGenerationCastError` in `exceptions.py` that inherits from `DatasetGenerationError` (for backward compatibility) that inherits from `DatasetsError`.

I also added a help message at the end of the error:

```
Please either edit the data files to have matching columns, or separate them into different configurations (see docs at https://hf.co/docs/hub/datasets-manual-configuration#multiple-configurations)
```

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004991 / 0.011353 (-0.006361) | 0.003362 / 0.011008 (-0.007646) | 0.062093 / 0.038508 (0.023585) | 0.051533 / 0.023109 (0.028424) | 0.247508 / 0.275898 (-0.028390) | 0.275593 / 0.323480 (-0.047886) | 0.003828 / 0.007986 (-0.004158) | 0.002573 / 0.004328 (-0.001755) | 0.047727 / 0.004250 (0.043477) | 0.037029 / 0.037052 (-0.000023) | 0.250359 / 0.258489 (-0.008130) | 0.282640 / 0.293841 (-0.011201) | 0.027853 / 0.128546 (-0.100693) | 0.010247 / 0.075646 (-0.065400) | 0.206826 / 0.419271 (-0.212445) | 0.035837 / 0.043533 (-0.007695) | 0.251795 / 0.255139 (-0.003344) | 0.275654 / 0.283200 (-0.007545) | 0.017722 / 0.141683 (-0.123960) | 1.120287 / 1.452155 (-0.331868) | 1.203087 / 1.492716 (-0.289630) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092320 / 0.018006 (0.074314) | 0.300079 / 0.000490 (0.299589) | 0.000211 / 0.000200 (0.000011) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018193 / 0.037411 (-0.019218) | 0.061310 / 0.014526 (0.046784) | 0.072433 / 0.176557 (-0.104124) | 0.119092 / 0.737135 (-0.618043) | 0.074044 / 0.296338 (-0.222294) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.297184 / 0.215209 (0.081975) | 2.805197 / 2.077655 (0.727543) | 1.521326 / 1.504120 (0.017206) | 1.374321 / 1.541195 (-0.166874) | 1.388767 / 1.468490 (-0.079723) | 0.571865 / 4.584777 (-4.012912) | 2.385213 / 3.745712 (-1.360499) | 2.726840 / 5.269862 (-2.543021) | 1.725352 / 4.565676 (-2.840325) | 0.063012 / 0.424275 (-0.361263) | 0.004911 / 0.007607 (-0.002697) | 0.336430 / 0.226044 (0.110385) | 3.390616 / 2.268929 (1.121688) | 1.846398 / 55.444624 (-53.598227) | 1.576797 / 6.876477 (-5.299680) | 1.579445 / 2.142072 (-0.562627) | 0.652515 / 4.805227 (-4.152712) | 0.118393 / 6.500664 (-6.382271) | 0.042155 / 0.075469 (-0.033314) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.942269 / 1.841788 (-0.899518) | 11.318258 / 8.074308 (3.243950) | 10.299948 / 10.191392 (0.108556) | 0.136088 / 0.680424 (-0.544336) | 0.013682 / 0.534201 (-0.520519) | 0.287549 / 0.579283 (-0.291734) | 0.258346 / 0.434364 (-0.176018) | 0.337146 / 0.540337 (-0.203191) | 0.443922 / 1.386936 (-0.943014) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005302 / 0.011353 (-0.006051) | 0.003234 / 0.011008 (-0.007774) | 0.049159 / 0.038508 (0.010651) | 0.050459 / 0.023109 (0.027350) | 0.273718 / 0.275898 (-0.002180) | 0.296997 / 0.323480 (-0.026483) | 0.003948 / 0.007986 (-0.004038) | 0.002590 / 0.004328 (-0.001739) | 0.048129 / 0.004250 (0.043879) | 0.039369 / 0.037052 (0.002317) | 0.276469 / 0.258489 (0.017980) | 0.306359 / 0.293841 (0.012519) | 0.028864 / 0.128546 (-0.099682) | 0.010253 / 0.075646 (-0.065394) | 0.058264 / 0.419271 (-0.361008) | 0.032451 / 0.043533 (-0.011082) | 0.277336 / 0.255139 (0.022197) | 0.296137 / 0.283200 (0.012937) | 0.018094 / 0.141683 (-0.123589) | 1.119539 / 1.452155 (-0.332615) | 1.163116 / 1.492716 (-0.329600) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092578 / 0.018006 (0.074572) | 0.300756 / 0.000490 (0.300267) | 0.000222 / 0.000200 (0.000022) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022333 / 0.037411 (-0.015078) | 0.076632 / 0.014526 (0.062107) | 0.087829 / 0.176557 (-0.088727) | 0.127686 / 0.737135 (-0.609449) | 0.091314 / 0.296338 (-0.205024) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.297499 / 0.215209 (0.082290) | 2.889775 / 2.077655 (0.812120) | 1.598976 / 1.504120 (0.094856) | 1.478805 / 1.541195 (-0.062389) | 1.481818 / 1.468490 (0.013328) | 0.557972 / 4.584777 (-4.026804) | 2.453248 / 3.745712 (-1.292464) | 2.771823 / 5.269862 (-2.498039) | 1.721527 / 4.565676 (-2.844150) | 0.062786 / 0.424275 (-0.361489) | 0.005298 / 0.007607 (-0.002309) | 0.346660 / 0.226044 (0.120615) | 3.412262 / 2.268929 (1.143334) | 1.940240 / 55.444624 (-53.504384) | 1.654015 / 6.876477 (-5.222461) | 1.652039 / 2.142072 (-0.490034) | 0.636870 / 4.805227 (-4.168357) | 0.116213 / 6.500664 (-6.384451) | 0.040937 / 0.075469 (-0.034532) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.001605 / 1.841788 (-0.840183) | 11.986592 / 8.074308 (3.912284) | 10.231288 / 10.191392 (0.039896) | 0.130242 / 0.680424 (-0.550182) | 0.015764 / 0.534201 (-0.518437) | 0.289257 / 0.579283 (-0.290026) | 0.275996 / 0.434364 (-0.158368) | 0.323089 / 0.540337 (-0.217248) | 0.556383 / 1.386936 (-0.830553) |

![](https://cml.dev/watermark.png#773324159ad4afd7931588a710839b76670ddf87 "CML watermark")

I want to improve the error message for datasets like https://huggingface.co/datasets/m-a-p/COIG-CQIA

Cc @albertvillanova @severo is this new error ok ? Or should I use a dedicated error class ?

New:

```python
Traceback (most recent call last):
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 1920, in _prepare_split_single
    writer.write_table(table)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/arrow_writer.py", line 574, in write_table
    pa_table = table_cast(pa_table, self._schema)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/table.py", line 2322, in table_cast
    return cast_table_to_schema(table, schema)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/table.py", line 2276, in cast_table_to_schema
    raise CastError(
datasets.table.CastError: Couldn't cast
instruction: string
other: string
index: string
domain: list<item: string>
  child 0, item: string
output: string
task_type: struct<major: list<item: string>, minor: list<item: string>>
  child 0, major: list<item: string>
      child 0, item: string
  child 1, minor: list<item: string>
      child 0, item: string
task_name_in_eng: string
input: string
to
{'answer_from': Value(dtype='string', id=None), 'instruction': Value(dtype='string', id=None), 'human_verified': Value(dtype='bool', id=None), 'domain': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'output': Value(dtype='string', id=None), 'task_type': {'major': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'minor': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}, 'copyright': Value(dtype='string', id=None), 'input': Value(dtype='string', id=None)}
because column names don't match

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/quentinlhoest/hf/datasets/playground/ttest.py", line 74, in <module>
    load_dataset("m-a-p/COIG-CQIA")
  File "/Users/quentinlhoest/hf/datasets/src/datasets/load.py", line 2529, in load_dataset
    builder_instance.download_and_prepare(
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 936, in download_and_prepare
    self._download_and_prepare(
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 1031, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 1791, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 1922, in _prepare_split_single
    raise DatasetGenerationCastError.from_cast_error(
datasets.exceptions.DatasetGenerationCastError: An error occurred while generating the dataset

All the data files must have the same columns, but at some point there are 3 new columns (other, index, task_name_in_eng) and 3 missing columns (answer_from, copyright, human_verified).

This happened while the json dataset builder was generating data using

hf://datasets/m-a-p/COIG-CQIA/coig_pc/coig_pc_core_sample.json (at revision b7b7ecf290f6515036c7c04bd8537228ac2eb474)

Please either edit the data files to have matching columns, or separate them into different configurations (see docs at https://hf.co/docs/hub/datasets-manual-configuration#multiple-configurations)
```

Previously:

```python
Traceback (most recent call last):
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 1931, in _prepare_split_single
    writer.write_table(table)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/arrow_writer.py", line 574, in write_table
    pa_table = table_cast(pa_table, self._schema)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/table.py", line 2295, in table_cast
    return cast_table_to_schema(table, schema)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/table.py", line 2253, in cast_table_to_schema
    raise ValueError(f"Couldn't cast\n{table.schema}\nto\n{features}\nbecause column names don't match")
ValueError: Couldn't cast
task_type: struct<major: list<item: string>, minor: list<item: string>>
  child 0, major: list<item: string>
      child 0, item: string
  child 1, minor: list<item: string>
      child 0, item: string
other: string
instruction: string
task_name_in_eng: string
domain: list<item: string>
  child 0, item: string
index: string
output: string
input: string
to
{'human_verified': Value(dtype='bool', id=None), 'task_type': {'major': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'minor': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}, 'answer_from': Value(dtype='string', id=None), 'copyright': Value(dtype='string', id=None), 'instruction': Value(dtype='string', id=None), 'domain': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'output': Value(dtype='string', id=None), 'input': Value(dtype='string', id=None)}
because column names don't match

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/quentinlhoest/hf/datasets/playground/ttest.py", line 74, in <module>
    load_dataset("m-a-p/COIG-CQIA")
  File "/Users/quentinlhoest/hf/datasets/src/datasets/load.py", line 2529, in load_dataset
    builder_instance.download_and_prepare(
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 949, in download_and_prepare
    self._download_and_prepare(
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 1044, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 1804, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/Users/quentinlhoest/hf/datasets/src/datasets/builder.py", line 1949, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
```


Read GeoParquet files using parquet reader

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6508). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Cool ! Do you mind writing a test using a geoparquet file in `tests/io/test_parquet.py` ?

I'm not too familiar with geoparquet, does it use e.g. pyarrow extension types ? or schema metadata ?

> Geometry columns MUST be stored using the BYTE_ARRAY parquet type. They MUST be encoded as [WKB](https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry#Well-known_binary).

https://github.com/opengeospatial/geoparquet/blob/main/format-specs/geoparquet.md#geometry-columns

It has metadata:

> File metadata indicating things like the version of this specification used
> Column metadata with additional metadata for each geometry column

https://github.com/opengeospatial/geoparquet/blob/main/format-specs/geoparquet.md#metadata

The specification is very short by the way:

https://github.com/opengeospatial/geoparquet/blob/main/format-specs/geoparquet.md

https://github.com/opengeospatial/geoparquet/blob/main/format-specs/compatible-parquet.md is also worth reading for this PR

> Cool ! Do you mind writing a test using a geoparquet file in `tests/io/test_parquet.py` ?

Yep, let me do that do that later today!

> I'm not too familiar with geoparquet, does it use e.g. pyarrow extension types ? or schema metadata ?

GeoParquet is a Parquet file with a `geometry` column that is encoded in a Binary format (technically WKB as @severo mentioned above). It is not a pyarrow extension type (though that would be cool). Regular `parquet` readers such as `pyarrow` would thus see the column as a binary column, while libraries such as `geopandas` which implement a GeoParquet reader would look at the schema metadata.

E.g. taking this [file](https://huggingface.co/datasets/weiji14/clay_vector_embeddings/resolve/862b1602f326421adc99375912c08603a9f2cc5c/32VLM_v01.gpq) as an example, this is how the 'geo' schema looks like:

```python
import pyarrow.parquet as pq

schema = pq.read_schema(where="32VLM_v01.gpq")
print(schema.metadata[b"geo"])
```

```
{
    "primary_column": "geometry",
    "columns": {
        "geometry": {
            "encoding": "WKB",
            "crs": {
                "$schema": "https://proj.org/schemas/v0.7/projjson.schema.json",
                "type": "GeographicCRS",
                "name": "WGS 84 (CRS84)",
                "datum_ensemble": {
                    "name": "World Geodetic System 1984 ensemble",
                    "members": [
                        {"name": "World Geodetic System 1984 (Transit)"},
                        {"name": "World Geodetic System 1984 (G730)"},
                        {"name": "World Geodetic System 1984 (G873)"},
                        {"name": "World Geodetic System 1984 (G1150)"},
                        {"name": "World Geodetic System 1984 (G1674)"},
                        {"name": "World Geodetic System 1984 (G1762)"},
                        {"name": "World Geodetic System 1984 (G2139)"},
                    ],
                    "ellipsoid": {
                        "name": "WGS 84",
                        "semi_major_axis": 6378137,
                        "inverse_flattening": 298.257223563,
                    },
                    "accuracy": "2.0",
                    "id": {"authority": "EPSG", "code": 6326},
                },
                "coordinate_system": {
                    "subtype": "ellipsoidal",
                    "axis": [
                        {
                            "name": "Geodetic longitude",
                            "abbreviation": "Lon",
                            "direction": "east",
                            "unit": "degree",
                        },
                        {
                            "name": "Geodetic latitude",
                            "abbreviation": "Lat",
                            "direction": "north",
                            "unit": "degree",
                        },
                    ],
                },
                "scope": "Not known.",
                "area": "World.",
                "bbox": {
                    "south_latitude": -90,
                    "west_longitude": -180,
                    "north_latitude": 90,
                    "east_longitude": 180,
                },
                "id": {"authority": "OGC", "code": "CRS84"},
            },
            "geometry_types": ["Polygon"],
            "bbox": [
                5.370542846111244,
                59.42344573656881,
                7.368267282586697,
                60.42591328670696,
            ],
        }
    },
    "version": "1.0.0",
    "creator": {"library": "geopandas", "version": "0.14.1"},
}
```

We can continue the discussion on how to handle this extra 'geo' schema metadata in #6438. I'd like to keep this PR small by just piggy-backing off the default Parquet reader for now, which would just show the 'geometry' column as a binary column.

Thanks ! Also if you can make sure that doing `ds.to_parquet("path/to/output.geoparquet")` also saves as a valid geoparquet files (including the schema metadata) that would be awesome.

It would also enable `push_to_hub` to save geoparquet files

> Thanks ! Also if you can make sure that doing `ds.to_parquet("path/to/output.geoparquet")` also saves as a valid geoparquet files (including the schema metadata) that would be awesome.
>
> It would also enable `push_to_hub` to save geoparquet files

Hmm, it should be possible to let PyArrow save a Parquet file with a geometry WKB column, but saving the GeoParquet schema metadata won't be easy without introducing [`geopandas`](https://github.com/geopandas/geopandas) as a dependency. Does this need to be done in this PR, or can it be a separate one?

I see, then let's keep it like this for now.
I just checked and it would require to add support for keeping the schema metadata in `Features` anyway.

Feel free to fix your code formatting using

```
make style
```

and we can merge this PR :)

Cool, linted to remove the extra blank line at 7088f585557807a63673cdc58900d7ce56146cf7. :rocket:

The previous CI failure at https://github.com/huggingface/datasets/actions/runs/7482863299/job/20668381959#step:6:5299 says `datasets.exceptions.DefunctDatasetError: Dataset 'eli5' is defunct and no longer accessible due to unavailability of the source data` which seems unrelated, might be to do with https://github.com/huggingface/datasets/issues/6605. I've updated the PR branch with changes from `main` again, could someone re-run the tests and merge if ok? Thanks!

sorry, it took me some time to fix the CI on the `main` branch

will merge once it's green :)

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005467 / 0.011353 (-0.005886) | 0.003696 / 0.011008 (-0.007313) | 0.063298 / 0.038508 (0.024790) | 0.032209 / 0.023109 (0.009100) | 0.246307 / 0.275898 (-0.029591) | 0.276864 / 0.323480 (-0.046616) | 0.003941 / 0.007986 (-0.004044) | 0.002616 / 0.004328 (-0.001713) | 0.049543 / 0.004250 (0.045292) | 0.044886 / 0.037052 (0.007833) | 0.266502 / 0.258489 (0.008013) | 0.288401 / 0.293841 (-0.005440) | 0.027911 / 0.128546 (-0.100635) | 0.011011 / 0.075646 (-0.064636) | 0.207972 / 0.419271 (-0.211299) | 0.036324 / 0.043533 (-0.007209) | 0.259450 / 0.255139 (0.004311) | 0.267317 / 0.283200 (-0.015883) | 0.018857 / 0.141683 (-0.122826) | 1.145350 / 1.452155 (-0.306805) | 1.204204 / 1.492716 (-0.288513) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.103864 / 0.018006 (0.085858) | 0.306941 / 0.000490 (0.306451) | 0.000218 / 0.000200 (0.000018) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018391 / 0.037411 (-0.019020) | 0.064600 / 0.014526 (0.050074) | 0.075454 / 0.176557 (-0.101102) | 0.120913 / 0.737135 (-0.616223) | 0.076998 / 0.296338 (-0.219341) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.279491 / 0.215209 (0.064282) | 2.742471 / 2.077655 (0.664816) | 1.447980 / 1.504120 (-0.056140) | 1.328202 / 1.541195 (-0.212992) | 1.397291 / 1.468490 (-0.071199) | 0.585726 / 4.584777 (-3.999051) | 2.385132 / 3.745712 (-1.360580) | 2.874888 / 5.269862 (-2.394974) | 1.820177 / 4.565676 (-2.745500) | 0.063876 / 0.424275 (-0.360399) | 0.004946 / 0.007607 (-0.002661) | 0.336445 / 0.226044 (0.110401) | 3.396813 / 2.268929 (1.127885) | 1.832644 / 55.444624 (-53.611981) | 1.581304 / 6.876477 (-5.295172) | 1.607243 / 2.142072 (-0.534829) | 0.662752 / 4.805227 (-4.142476) | 0.119494 / 6.500664 (-6.381170) | 0.042573 / 0.075469 (-0.032896) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.936784 / 1.841788 (-0.905003) | 12.154288 / 8.074308 (4.079980) | 10.944835 / 10.191392 (0.753443) | 0.132856 / 0.680424 (-0.547568) | 0.015197 / 0.534201 (-0.519004) | 0.290647 / 0.579283 (-0.288636) | 0.273498 / 0.434364 (-0.160866) | 0.324893 / 0.540337 (-0.215444) | 0.427905 / 1.386936 (-0.959032) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005695 / 0.011353 (-0.005658) | 0.003562 / 0.011008 (-0.007446) | 0.050117 / 0.038508 (0.011608) | 0.033876 / 0.023109 (0.010767) | 0.275514 / 0.275898 (-0.000384) | 0.298460 / 0.323480 (-0.025020) | 0.004240 / 0.007986 (-0.003745) | 0.002738 / 0.004328 (-0.001591) | 0.048518 / 0.004250 (0.044268) | 0.049064 / 0.037052 (0.012012) | 0.287094 / 0.258489 (0.028605) | 0.314281 / 0.293841 (0.020440) | 0.057861 / 0.128546 (-0.070686) | 0.010893 / 0.075646 (-0.064753) | 0.062251 / 0.419271 (-0.357020) | 0.036788 / 0.043533 (-0.006745) | 0.272431 / 0.255139 (0.017292) | 0.292022 / 0.283200 (0.008822) | 0.019874 / 0.141683 (-0.121809) | 1.156939 / 1.452155 (-0.295216) | 1.237966 / 1.492716 (-0.254751) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096156 / 0.018006 (0.078150) | 0.306652 / 0.000490 (0.306162) | 0.000230 / 0.000200 (0.000031) | 0.000059 / 0.000054 (0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022965 / 0.037411 (-0.014447) | 0.081349 / 0.014526 (0.066823) | 0.089035 / 0.176557 (-0.087521) | 0.128831 / 0.737135 (-0.608304) | 0.090321 / 0.296338 (-0.206017) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.293110 / 0.215209 (0.077901) | 2.884493 / 2.077655 (0.806839) | 1.582522 / 1.504120 (0.078402) | 1.518977 / 1.541195 (-0.022218) | 1.528449 / 1.468490 (0.059959) | 0.577369 / 4.584777 (-4.007408) | 2.473060 / 3.745712 (-1.272652) | 3.104363 / 5.269862 (-2.165499) | 1.916529 / 4.565676 (-2.649147) | 0.064594 / 0.424275 (-0.359682) | 0.005386 / 0.007607 (-0.002221) | 0.353336 / 0.226044 (0.127292) | 3.471914 / 2.268929 (1.202985) | 1.959222 / 55.444624 (-53.485402) | 1.677153 / 6.876477 (-5.199324) | 1.716961 / 2.142072 (-0.425112) | 0.658355 / 4.805227 (-4.146873) | 0.117296 / 6.500664 (-6.383368) | 0.041139 / 0.075469 (-0.034330) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.025220 / 1.841788 (-0.816567) | 14.510987 / 8.074308 (6.436679) | 11.851428 / 10.191392 (1.660036) | 0.143759 / 0.680424 (-0.536665) | 0.015644 / 0.534201 (-0.518557) | 0.296824 / 0.579283 (-0.282459) | 0.281566 / 0.434364 (-0.152798) | 0.335094 / 0.540337 (-0.205244) | 0.425199 / 1.386936 (-0.961737) |

![](https://cml.dev/watermark.png#fabc2c8cee8822572115893715b76dfdabac1631 "CML watermark")

Let GeoParquet files with the file extension `*.geoparquet` or `*.gpq` be readable by the default parquet reader.

Those two file extensions are the ones most commonly used for GeoParquet files, and is included in the `gpq` validator tool at https://github.com/planetlabs/gpq/blob/e5576b4ee7306b4d2259d56c879465a9364dab90/cmd/gpq/command/convert.go#L73-L75

Addresses https://github.com/huggingface/datasets/issues/6438


where is glue_metric.py> @Frankie123421 what was the resolution to this?

              > @Frankie123421 what was the resolution to this?

use glue_metric.py instead of glue.py in load_metric

_Originally posted by @Frankie123421 in https://github.com/huggingface/datasets/issues/2117#issuecomment-905093763_


Incorrect test set labels for RTE and CoLA datasets via load_dataset

As this is a specific issue of the "glue" dataset, I have transferred it to the dataset Discussion page: https://huggingface.co/datasets/glue/discussions/15

Let's continue the discussion there!

### Describe the bug

The test set labels for the RTE and CoLA datasets when loading via datasets load_dataset are all -1.
Edit: It appears this is also the case for every other dataset except for MRPC (stsb, sst2, qqp, mnli (both matched and mismatched), qnli, wnli, ax). Is this intended behavior to safeguard the test set for evaluation purposes?

### Steps to reproduce the bug

!pip install datasets
from datasets import load_dataset

rte_data = load_dataset('glue', 'rte')
cola_data = load_dataset('glue', 'cola')
print(rte_data['test'][0:30]['label'])
print(cola_data['test'][0:30]['label'])

Output:
[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]
[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]

The non-label test data seems to be fine:
e.g. rte_data['test'][1] is:
{'sentence1': "Authorities in Brazil say that more than 200 people are being held hostage in a prison in the country's remote, Amazonian-jungle state of Rondonia.",
 'sentence2': 'Authorities in Brazil hold 200 people as hostage.',
 'label': -1,
 'idx': 1}
Training and validation data are also fine:
e.g. rte_data['train][0] is:
{'sentence1': 'No Weapons of Mass Destruction Found in Iraq Yet.',
 'sentence2': 'Weapons of Mass Destruction Found in Iraq.',
 'label': 1,
 'idx': 0}

### Expected behavior

Expected the labels to be binary 0/1 values; Got all -1s instead

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.19.4
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0


Got stuck when I trying to load a dataset

> I ran into the same problem when I used a server cluster (Slurm system managed) that couldn't load any of the huggingface datasets or models, but it worked on my laptop. I suspected some system configuration-related problem, but I had no idea. My problems are consistent with [issue #2618](https://github.com/huggingface/datasets/issues/2618). All the huggingface-related libraries I use are the latest versions.

have you solved this issue yet? i met the same problem on server but everything works on laptop. I think maybe the filelock repo is contradictory with file system.

I am having the same issue on a computing cluster but this works on my laptop as well.  I instead have this error:
`/home/.conda/envs/py10/lib/python3.10/site-packages/filelock/_unix.py", line 43, in _acquire
    fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
OSError: [Errno 5] Input/output error`

the load_dataset command does not work on server for local or hosted hugging-face datasets, and I have tried for several files

> In my case, `.cahce` was in a shared folder. Moving it into the user's home folder fixed the problem. #2618 for more details在我的情况下， `.cahce` 在一个共享文件夹中。将其移动到用户的主文件夹中解决了问题。 #2618 获取更多详细信息。

Can you be more specific? thank.

### Describe the bug

Hello, everyone. I met a problem when I am trying to load a data file using load_dataset method on a Debian 10 system. The data file is not very large, only 1.63MB with 600 records.
Here is my code:

from datasets import load_dataset

dataset = load_dataset('json', data_files='mypath/oaast_rm_zh.json')

I waited it for 20 minutes. It still no response. I cannot using Ctrl+C to cancel the command. I have to use Ctrl+Z to kill it. I also try it with a txt file, it still no response in a long time.

I can load the same file successfully using my laptop (windows 10, python 3.8.5, datasets==2.14.5). I can also make it on another computer (Ubuntu 20.04.5 LTS, python 3.10.13, datasets 2.14.7). It only takes me 1-2 miniutes.

Could you give me some suggestions? Thank you.

### Steps to reproduce the bug

from datasets import load_dataset

dataset = load_dataset('json', data_files='mypath/oaast_rm_zh.json')

### Expected behavior

I hope it can load the file successfully.

### Environment info

OS: Debian GNU/Linux 10
Python: Python 3.10.13
Pip list:
Package                   Version
------------------------- ------------
accelerate                0.25.0
addict                    2.4.0
aiofiles                  23.2.1
aiohttp                   3.9.1
aiosignal                 1.3.1
aliyun-python-sdk-core    2.14.0
aliyun-python-sdk-kms     2.16.2
altair                    5.2.0
annotated-types           0.6.0
anyio                     3.7.1
async-timeout             4.0.3
attrs                     23.1.0
certifi                   2023.11.17
cffi                      1.16.0
charset-normalizer        3.3.2
click                     8.1.7
contourpy                 1.2.0
crcmod                    1.7
cryptography              41.0.7
cycler                    0.12.1
datasets                  2.14.7
dill                      0.3.7
docstring-parser          0.15
einops                    0.7.0
exceptiongroup            1.2.0
fastapi                   0.105.0
ffmpy                     0.3.1
filelock                  3.13.1
fonttools                 4.46.0
frozenlist                1.4.1
fsspec                    2023.10.0
gast                      0.5.4
gradio                    3.50.2
gradio_client             0.6.1
h11                       0.14.0
httpcore                  1.0.2
httpx                     0.25.2
huggingface-hub           0.19.4
idna                      3.6
importlib-metadata        7.0.0
importlib-resources       6.1.1
jieba                     0.42.1
Jinja2                    3.1.2
jmespath                  0.10.0
joblib                    1.3.2
jsonschema                4.20.0
jsonschema-specifications 2023.11.2
kiwisolver                1.4.5
markdown-it-py            3.0.0
MarkupSafe                2.1.3
matplotlib                3.8.2
mdurl                     0.1.2
modelscope                1.10.0
mpmath                    1.3.0
multidict                 6.0.4
multiprocess              0.70.15
networkx                  3.2.1
nltk                      3.8.1
numpy                     1.26.2
nvidia-cublas-cu12        12.1.3.1
nvidia-cuda-cupti-cu12    12.1.105
nvidia-cuda-nvrtc-cu12    12.1.105
nvidia-cuda-runtime-cu12  12.1.105
nvidia-cudnn-cu12         8.9.2.26
nvidia-cufft-cu12         11.0.2.54
nvidia-curand-cu12        10.3.2.106
nvidia-cusolver-cu12      11.4.5.107
nvidia-cusparse-cu12      12.1.0.106
nvidia-nccl-cu12          2.18.1
nvidia-nvjitlink-cu12     12.3.101
nvidia-nvtx-cu12          12.1.105
orjson                    3.9.10
oss2                      2.18.3
packaging                 23.2
pandas                    2.1.4
peft                      0.7.1
Pillow                    10.1.0
pip                       23.3.1
platformdirs              4.1.0
protobuf                  4.25.1
psutil                    5.9.6
pyarrow                   14.0.1
pyarrow-hotfix            0.6
pycparser                 2.21
pycryptodome              3.19.0
pydantic                  2.5.2
pydantic_core             2.14.5
pydub                     0.25.1
Pygments                  2.17.2
pyparsing                 3.1.1
python-dateutil           2.8.2
python-multipart          0.0.6
pytz                      2023.3.post1
PyYAML                    6.0.1
referencing               0.32.0
regex                     2023.10.3
requests                  2.31.0
rich                      13.7.0
rouge-chinese             1.0.3
rpds-py                   0.13.2
safetensors               0.4.1
scipy                     1.11.4
semantic-version          2.10.0
sentencepiece             0.1.99
setuptools                68.2.2
shtab                     1.6.5
simplejson                3.19.2
six                       1.16.0
sniffio                   1.3.0
sortedcontainers          2.4.0
sse-starlette             1.8.2
starlette                 0.27.0
sympy                     1.12
tiktoken                  0.5.2
tokenizers                0.15.0
tomli                     2.0.1
toolz                     0.12.0
torch                     2.1.2
tqdm                      4.66.1
transformers              4.36.1
triton                    2.1.0
trl                       0.7.4
typing_extensions         4.9.0
tyro                      0.6.0
tzdata                    2023.3
urllib3                   2.1.0
uvicorn                   0.24.0.post1
websockets                11.0.3
wheel                     0.41.2
xxhash                    3.4.1
yapf                      0.40.2
yarl                      1.9.4
zipp                      3.17.0


Error Pushing to Hub

### Describe the bug

Error when trying to push a dataset in a special format to hub

### Steps to reproduce the bug

```
import datasets
from datasets import Dataset
dataset_dict = {
    "filename": ["apple", "banana"],
    "token": [[[1,2],[3,4]],[[1,2],[3,4]]],
    "label": [0, 1],
}
dataset = Dataset.from_dict(dataset_dict)
dataset = dataset.cast_column("token", datasets.features.features.Array2D(shape=(2, 2),dtype="int16"))

dataset.push_to_hub("SequenceModel/imagenet_val_256")
```
Error:
```
...

ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/tuple'
  in "<unicode string>", line 8, column 16:
            shape: !!python/tuple
                   ^
```

### Expected behavior

Dataset being pushed to hub

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-5.19.0-1022-gcp-x86_64-with-glibc2.35
- Python version: 3.11.5
- `huggingface_hub` version: 0.19.4
- PyArrow version: 14.0.1
- Pandas version: 2.1.4
- `fsspec` version: 2023.10.0