Fix streaming xnli

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6503). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005003 / 0.011353 (-0.006350) | 0.003020 / 0.011008 (-0.007988) | 0.061370 / 0.038508 (0.022862) | 0.050996 / 0.023109 (0.027887) | 0.243434 / 0.275898 (-0.032464) | 0.266317 / 0.323480 (-0.057163) | 0.003888 / 0.007986 (-0.004098) | 0.002607 / 0.004328 (-0.001721) | 0.047541 / 0.004250 (0.043290) | 0.037933 / 0.037052 (0.000881) | 0.259695 / 0.258489 (0.001206) | 0.279374 / 0.293841 (-0.014467) | 0.027258 / 0.128546 (-0.101288) | 0.010184 / 0.075646 (-0.065462) | 0.207412 / 0.419271 (-0.211860) | 0.034978 / 0.043533 (-0.008554) | 0.247871 / 0.255139 (-0.007267) | 0.265273 / 0.283200 (-0.017927) | 0.017886 / 0.141683 (-0.123796) | 1.090451 / 1.452155 (-0.361704) | 1.152034 / 1.492716 (-0.340682) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094383 / 0.018006 (0.076377) | 0.301151 / 0.000490 (0.300661) | 0.000211 / 0.000200 (0.000011) | 0.000049 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018927 / 0.037411 (-0.018484) | 0.062152 / 0.014526 (0.047626) | 0.072177 / 0.176557 (-0.104380) | 0.119792 / 0.737135 (-0.617343) | 0.073333 / 0.296338 (-0.223005) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282671 / 0.215209 (0.067462) | 2.721148 / 2.077655 (0.643494) | 1.472689 / 1.504120 (-0.031431) | 1.355226 / 1.541195 (-0.185969) | 1.375935 / 1.468490 (-0.092556) | 0.562600 / 4.584777 (-4.022177) | 2.364046 / 3.745712 (-1.381666) | 2.714984 / 5.269862 (-2.554878) | 1.738413 / 4.565676 (-2.827263) | 0.062564 / 0.424275 (-0.361711) | 0.004964 / 0.007607 (-0.002643) | 0.341300 / 0.226044 (0.115255) | 3.345187 / 2.268929 (1.076259) | 1.857822 / 55.444624 (-53.586803) | 1.581002 / 6.876477 (-5.295475) | 1.585919 / 2.142072 (-0.556153) | 0.640105 / 4.805227 (-4.165122) | 0.117880 / 6.500664 (-6.382784) | 0.042032 / 0.075469 (-0.033437) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.962701 / 1.841788 (-0.879086) | 11.309251 / 8.074308 (3.234943) | 10.462520 / 10.191392 (0.271128) | 0.127399 / 0.680424 (-0.553025) | 0.014549 / 0.534201 (-0.519652) | 0.297017 / 0.579283 (-0.282266) | 0.266152 / 0.434364 (-0.168212) | 0.349252 / 0.540337 (-0.191085) | 0.457015 / 1.386936 (-0.929921) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005341 / 0.011353 (-0.006012) | 0.003108 / 0.011008 (-0.007900) | 0.048862 / 0.038508 (0.010353) | 0.053354 / 0.023109 (0.030245) | 0.274499 / 0.275898 (-0.001399) | 0.296698 / 0.323480 (-0.026782) | 0.003974 / 0.007986 (-0.004012) | 0.002631 / 0.004328 (-0.001697) | 0.048013 / 0.004250 (0.043762) | 0.040416 / 0.037052 (0.003363) | 0.276581 / 0.258489 (0.018092) | 0.301296 / 0.293841 (0.007455) | 0.029049 / 0.128546 (-0.099497) | 0.010253 / 0.075646 (-0.065393) | 0.057157 / 0.419271 (-0.362114) | 0.031830 / 0.043533 (-0.011703) | 0.274341 / 0.255139 (0.019202) | 0.292583 / 0.283200 (0.009383) | 0.018449 / 0.141683 (-0.123234) | 1.145099 / 1.452155 (-0.307055) | 1.192958 / 1.492716 (-0.299758) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091596 / 0.018006 (0.073590) | 0.300917 / 0.000490 (0.300427) | 0.000225 / 0.000200 (0.000025) | 0.000054 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021657 / 0.037411 (-0.015754) | 0.068464 / 0.014526 (0.053938) | 0.079869 / 0.176557 (-0.096687) | 0.117523 / 0.737135 (-0.619613) | 0.081257 / 0.296338 (-0.215082) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.294876 / 0.215209 (0.079667) | 2.879372 / 2.077655 (0.801718) | 1.619887 / 1.504120 (0.115767) | 1.482154 / 1.541195 (-0.059041) | 1.494656 / 1.468490 (0.026166) | 0.558914 / 4.584777 (-4.025862) | 2.420948 / 3.745712 (-1.324765) | 2.728992 / 5.269862 (-2.540869) | 1.722135 / 4.565676 (-2.843542) | 0.062182 / 0.424275 (-0.362093) | 0.004933 / 0.007607 (-0.002674) | 0.342759 / 0.226044 (0.116715) | 3.424083 / 2.268929 (1.155154) | 1.950673 / 55.444624 (-53.493951) | 1.683126 / 6.876477 (-5.193351) | 1.673135 / 2.142072 (-0.468937) | 0.633711 / 4.805227 (-4.171516) | 0.114898 / 6.500664 (-6.385766) | 0.040332 / 0.075469 (-0.035137) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.975102 / 1.841788 (-0.866685) | 11.975731 / 8.074308 (3.901423) | 10.961103 / 10.191392 (0.769711) | 0.131152 / 0.680424 (-0.549272) | 0.016268 / 0.534201 (-0.517933) | 0.285031 / 0.579283 (-0.294252) | 0.279556 / 0.434364 (-0.154808) | 0.324183 / 0.540337 (-0.216154) | 0.571404 / 1.386936 (-0.815532) |

![](https://cml.dev/watermark.png#4f67312956fc15572b6a0ca0dfcc0ceb90fbb794 "CML watermark")

This code was failing

```python

In [1]: from datasets import load_dataset

In [2]:
   ...:         ds = load_dataset("xnli", "all_languages", split="test", streaming=True)
   ...:
   ...:         sample_data = next(iter(ds))["premise"]  # pick up one data
   ...:         input_text = list(sample_data.values())
```

```
File ~/hf/datasets/src/datasets/features/translation.py:104, in TranslationVariableLanguages.encode_example(self, translation_dict)
    102     return translation_dict
    103 elif self.languages and set(translation_dict) - lang_set:
--> 104     raise ValueError(
    105         f'Some languages in example ({", ".join(sorted(set(translation_dict) - lang_set))}) are not in valid set ({", ".join(lang_set)}).'
    106     )
    108 # Convert dictionary into tuples, splitting out cases where there are
    109 # multiple translations for a single language.
    110 translation_tuples = []

ValueError: Some languages in example (language, translation) are not in valid set (ur, fr, hi, sw, vi, el, de, th, en, tr, zh, ar, bg, ru, es).
```

because in streaming mode we expect features encode methods to be no-ops if the example is already encoded.

I fixed `TranslationVariableLanguages` to account for that


Pickle support for `torch.Generator` objects

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6502). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005472 / 0.011353 (-0.005881) | 0.003715 / 0.011008 (-0.007293) | 0.063257 / 0.038508 (0.024749) | 0.060683 / 0.023109 (0.037574) | 0.250885 / 0.275898 (-0.025013) | 0.271685 / 0.323480 (-0.051795) | 0.003051 / 0.007986 (-0.004934) | 0.002799 / 0.004328 (-0.001530) | 0.049113 / 0.004250 (0.044863) | 0.038965 / 0.037052 (0.001912) | 0.252688 / 0.258489 (-0.005801) | 0.282536 / 0.293841 (-0.011305) | 0.028722 / 0.128546 (-0.099824) | 0.010586 / 0.075646 (-0.065060) | 0.205145 / 0.419271 (-0.214127) | 0.036996 / 0.043533 (-0.006537) | 0.248874 / 0.255139 (-0.006265) | 0.266148 / 0.283200 (-0.017051) | 0.018540 / 0.141683 (-0.123143) | 1.120216 / 1.452155 (-0.331938) | 1.191072 / 1.492716 (-0.301644) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095721 / 0.018006 (0.077714) | 0.313401 / 0.000490 (0.312911) | 0.000234 / 0.000200 (0.000034) | 0.000053 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018604 / 0.037411 (-0.018807) | 0.061571 / 0.014526 (0.047045) | 0.075343 / 0.176557 (-0.101213) | 0.121272 / 0.737135 (-0.615864) | 0.076448 / 0.296338 (-0.219890) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.286885 / 0.215209 (0.071676) | 2.809100 / 2.077655 (0.731445) | 1.485365 / 1.504120 (-0.018755) | 1.367672 / 1.541195 (-0.173523) | 1.423570 / 1.468490 (-0.044920) | 0.571063 / 4.584777 (-4.013714) | 2.385248 / 3.745712 (-1.360464) | 2.855251 / 5.269862 (-2.414610) | 1.799371 / 4.565676 (-2.766306) | 0.063491 / 0.424275 (-0.360784) | 0.004942 / 0.007607 (-0.002665) | 0.346181 / 0.226044 (0.120137) | 3.388123 / 2.268929 (1.119195) | 1.819093 / 55.444624 (-53.625532) | 1.552998 / 6.876477 (-5.323479) | 1.627930 / 2.142072 (-0.514143) | 0.653438 / 4.805227 (-4.151789) | 0.123831 / 6.500664 (-6.376833) | 0.043340 / 0.075469 (-0.032129) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.952167 / 1.841788 (-0.889621) | 12.149515 / 8.074308 (4.075207) | 10.665085 / 10.191392 (0.473693) | 0.127768 / 0.680424 (-0.552656) | 0.014022 / 0.534201 (-0.520179) | 0.285959 / 0.579283 (-0.293324) | 0.269727 / 0.434364 (-0.164637) | 0.336646 / 0.540337 (-0.203692) | 0.442932 / 1.386936 (-0.944005) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005351 / 0.011353 (-0.006002) | 0.003561 / 0.011008 (-0.007448) | 0.048890 / 0.038508 (0.010382) | 0.054093 / 0.023109 (0.030984) | 0.274397 / 0.275898 (-0.001501) | 0.296980 / 0.323480 (-0.026500) | 0.004126 / 0.007986 (-0.003860) | 0.002751 / 0.004328 (-0.001578) | 0.049131 / 0.004250 (0.044880) | 0.040769 / 0.037052 (0.003716) | 0.279147 / 0.258489 (0.020658) | 0.302014 / 0.293841 (0.008173) | 0.029847 / 0.128546 (-0.098699) | 0.010710 / 0.075646 (-0.064936) | 0.057626 / 0.419271 (-0.361645) | 0.032801 / 0.043533 (-0.010732) | 0.272698 / 0.255139 (0.017559) | 0.289238 / 0.283200 (0.006039) | 0.017876 / 0.141683 (-0.123807) | 1.152059 / 1.452155 (-0.300096) | 1.212289 / 1.492716 (-0.280427) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092914 / 0.018006 (0.074908) | 0.303092 / 0.000490 (0.302603) | 0.000214 / 0.000200 (0.000014) | 0.000058 / 0.000054 (0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022074 / 0.037411 (-0.015337) | 0.070109 / 0.014526 (0.055583) | 0.083360 / 0.176557 (-0.093196) | 0.122445 / 0.737135 (-0.614690) | 0.083625 / 0.296338 (-0.212714) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282788 / 0.215209 (0.067579) | 2.789229 / 2.077655 (0.711574) | 1.571077 / 1.504120 (0.066957) | 1.452627 / 1.541195 (-0.088567) | 1.493176 / 1.468490 (0.024686) | 0.556892 / 4.584777 (-4.027885) | 2.442771 / 3.745712 (-1.302941) | 2.826316 / 5.269862 (-2.443545) | 1.758276 / 4.565676 (-2.807401) | 0.063039 / 0.424275 (-0.361236) | 0.004928 / 0.007607 (-0.002679) | 0.338247 / 0.226044 (0.112202) | 3.346344 / 2.268929 (1.077416) | 1.952520 / 55.444624 (-53.492104) | 1.664520 / 6.876477 (-5.211956) | 1.701528 / 2.142072 (-0.440544) | 0.634746 / 4.805227 (-4.170481) | 0.116879 / 6.500664 (-6.383786) | 0.040990 / 0.075469 (-0.034479) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.969521 / 1.841788 (-0.872267) | 12.431395 / 8.074308 (4.357087) | 10.907503 / 10.191392 (0.716111) | 0.131028 / 0.680424 (-0.549396) | 0.015239 / 0.534201 (-0.518962) | 0.290793 / 0.579283 (-0.288490) | 0.275072 / 0.434364 (-0.159292) | 0.331036 / 0.540337 (-0.209301) | 0.567858 / 1.386936 (-0.819078) |

![](https://cml.dev/watermark.png#092118fc00f7dd718ab3643739d7b23ff16c9eff "CML watermark")

Fix for https://discuss.huggingface.co/t/caching-a-dataset-processed-with-randomness/65616


OverflowError: value too large to convert to int32_t

### Describe the bug

![image](https://github.com/huggingface/datasets/assets/47747764/f58044fb-ddda-48b6-ba68-7bbfef781630)

### Steps to reproduce the bug

just loading datasets

### Expected behavior

how can I fix it

### Environment info

pip install /mnt/cluster/zhangfan/study_info/LLaMA-Factory/peft-0.6.0-py3-none-any.whl
pip install huggingface_hub-0.19.4-py3-none-any.whl tokenizers-0.15.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl transformers-4.36.1-py3-none-any.whl pyarrow_hotfix-0.6-py3-none-any.whl datasets-2.15.0-py3-none-any.whl tyro-0.5.18-py3-none-any.whl trl-0.7.4-py3-none-any.whl

done


Enable setting config as default when push_to_hub

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6500). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This is ready for review @huggingface/datasets.

Also what if the config is being overwritten and it was the default config and the user doesn't pass `set_default` ?
I'd expect the config to keep being the default one but lmk what you think

How can you unset a config as the default one? In the case you mentioned, I would expect the config not being the default one.

Maybe by passing `set_default=False` ? (set_default can be None by default)

I think that way we are unnecessarily complicating the logic of `push_to_hub` and as I told you, I would expect the contrary: the result of calling `push_to_hub` with a determined set of arguments should always be the same, independently of previous calls and the current state of the config on the Hub. Push to hub should be somehow stateless in that sense, and IMO the user expects that the push overwrites previous config if already present on the Hub. I find very confusing making it to partially update the config on the Hub.

That makes sense, having it stateless is simpler and no need to do something too fancy indeed

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005329 / 0.011353 (-0.006024) | 0.002998 / 0.011008 (-0.008010) | 0.063756 / 0.038508 (0.025248) | 0.051713 / 0.023109 (0.028603) | 0.248135 / 0.275898 (-0.027763) | 0.269136 / 0.323480 (-0.054344) | 0.002970 / 0.007986 (-0.005015) | 0.002566 / 0.004328 (-0.001763) | 0.048110 / 0.004250 (0.043859) | 0.038415 / 0.037052 (0.001363) | 0.254012 / 0.258489 (-0.004477) | 0.281915 / 0.293841 (-0.011926) | 0.027503 / 0.128546 (-0.101043) | 0.010370 / 0.075646 (-0.065276) | 0.208965 / 0.419271 (-0.210306) | 0.035508 / 0.043533 (-0.008024) | 0.249116 / 0.255139 (-0.006023) | 0.266350 / 0.283200 (-0.016850) | 0.018440 / 0.141683 (-0.123243) | 1.101089 / 1.452155 (-0.351066) | 1.164870 / 1.492716 (-0.327847) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090909 / 0.018006 (0.072903) | 0.298041 / 0.000490 (0.297551) | 0.000211 / 0.000200 (0.000012) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018137 / 0.037411 (-0.019275) | 0.059574 / 0.014526 (0.045048) | 0.071754 / 0.176557 (-0.104803) | 0.117980 / 0.737135 (-0.619155) | 0.072903 / 0.296338 (-0.223435) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282844 / 0.215209 (0.067635) | 2.740916 / 2.077655 (0.663261) | 1.444546 / 1.504120 (-0.059574) | 1.321904 / 1.541195 (-0.219291) | 1.356957 / 1.468490 (-0.111533) | 0.568389 / 4.584777 (-4.016388) | 2.354042 / 3.745712 (-1.391671) | 2.719427 / 5.269862 (-2.550435) | 1.719616 / 4.565676 (-2.846061) | 0.062537 / 0.424275 (-0.361738) | 0.004915 / 0.007607 (-0.002692) | 0.334716 / 0.226044 (0.108672) | 3.299499 / 2.268929 (1.030571) | 1.814629 / 55.444624 (-53.629996) | 1.515245 / 6.876477 (-5.361232) | 1.553085 / 2.142072 (-0.588987) | 0.643859 / 4.805227 (-4.161368) | 0.116650 / 6.500664 (-6.384014) | 0.041432 / 0.075469 (-0.034037) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.948227 / 1.841788 (-0.893561) | 11.331103 / 8.074308 (3.256795) | 10.209658 / 10.191392 (0.018266) | 0.126721 / 0.680424 (-0.553703) | 0.013638 / 0.534201 (-0.520563) | 0.282540 / 0.579283 (-0.296743) | 0.262635 / 0.434364 (-0.171729) | 0.335357 / 0.540337 (-0.204981) | 0.441798 / 1.386936 (-0.945138) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005200 / 0.011353 (-0.006153) | 0.003012 / 0.011008 (-0.007996) | 0.047571 / 0.038508 (0.009063) | 0.055069 / 0.023109 (0.031959) | 0.271150 / 0.275898 (-0.004748) | 0.294957 / 0.323480 (-0.028523) | 0.003922 / 0.007986 (-0.004064) | 0.002627 / 0.004328 (-0.001702) | 0.047777 / 0.004250 (0.043527) | 0.039507 / 0.037052 (0.002454) | 0.276314 / 0.258489 (0.017825) | 0.300436 / 0.293841 (0.006595) | 0.028951 / 0.128546 (-0.099595) | 0.010583 / 0.075646 (-0.065063) | 0.056535 / 0.419271 (-0.362737) | 0.032654 / 0.043533 (-0.010879) | 0.272945 / 0.255139 (0.017806) | 0.291909 / 0.283200 (0.008709) | 0.017545 / 0.141683 (-0.124138) | 1.195897 / 1.452155 (-0.256258) | 1.171855 / 1.492716 (-0.320861) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091919 / 0.018006 (0.073913) | 0.299297 / 0.000490 (0.298807) | 0.000225 / 0.000200 (0.000025) | 0.000051 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022271 / 0.037411 (-0.015140) | 0.068903 / 0.014526 (0.054377) | 0.083767 / 0.176557 (-0.092790) | 0.120239 / 0.737135 (-0.616896) | 0.083448 / 0.296338 (-0.212891) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295353 / 0.215209 (0.080144) | 2.911452 / 2.077655 (0.833798) | 1.577941 / 1.504120 (0.073821) | 1.454514 / 1.541195 (-0.086681) | 1.459575 / 1.468490 (-0.008915) | 0.572475 / 4.584777 (-4.012302) | 2.443634 / 3.745712 (-1.302078) | 2.801171 / 5.269862 (-2.468691) | 1.724214 / 4.565676 (-2.841462) | 0.063539 / 0.424275 (-0.360736) | 0.004939 / 0.007607 (-0.002668) | 0.347705 / 0.226044 (0.121660) | 3.489591 / 2.268929 (1.220663) | 1.944952 / 55.444624 (-53.499672) | 1.652810 / 6.876477 (-5.223667) | 1.656361 / 2.142072 (-0.485712) | 0.647052 / 4.805227 (-4.158176) | 0.117286 / 6.500664 (-6.383379) | 0.040979 / 0.075469 (-0.034490) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.971761 / 1.841788 (-0.870027) | 11.770547 / 8.074308 (3.696239) | 10.402502 / 10.191392 (0.211110) | 0.128280 / 0.680424 (-0.552144) | 0.015160 / 0.534201 (-0.519041) | 0.286706 / 0.579283 (-0.292578) | 0.274539 / 0.434364 (-0.159825) | 0.324591 / 0.540337 (-0.215747) | 0.573846 / 1.386936 (-0.813090) |

![](https://cml.dev/watermark.png#3329be80b9abfe83285ef940a590a4e9f68835a3 "CML watermark")

Fix #6497.


docs: add reference Git over SSH

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6499). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005701 / 0.011353 (-0.005652) | 0.003546 / 0.011008 (-0.007463) | 0.063335 / 0.038508 (0.024827) | 0.051987 / 0.023109 (0.028878) | 0.240429 / 0.275898 (-0.035469) | 0.260659 / 0.323480 (-0.062820) | 0.003866 / 0.007986 (-0.004120) | 0.002617 / 0.004328 (-0.001712) | 0.048653 / 0.004250 (0.044403) | 0.038176 / 0.037052 (0.001124) | 0.245496 / 0.258489 (-0.012993) | 0.277141 / 0.293841 (-0.016700) | 0.027886 / 0.128546 (-0.100660) | 0.010738 / 0.075646 (-0.064908) | 0.211255 / 0.419271 (-0.208016) | 0.045205 / 0.043533 (0.001672) | 0.243062 / 0.255139 (-0.012077) | 0.262877 / 0.283200 (-0.020323) | 0.023426 / 0.141683 (-0.118257) | 1.092247 / 1.452155 (-0.359908) | 1.161074 / 1.492716 (-0.331642) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090488 / 0.018006 (0.072482) | 0.300993 / 0.000490 (0.300504) | 0.000212 / 0.000200 (0.000012) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018543 / 0.037411 (-0.018868) | 0.061418 / 0.014526 (0.046892) | 0.073242 / 0.176557 (-0.103314) | 0.120757 / 0.737135 (-0.616378) | 0.073967 / 0.296338 (-0.222372) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282341 / 0.215209 (0.067132) | 2.741106 / 2.077655 (0.663451) | 1.416573 / 1.504120 (-0.087547) | 1.287904 / 1.541195 (-0.253291) | 1.309425 / 1.468490 (-0.159065) | 0.582592 / 4.584777 (-4.002184) | 2.404866 / 3.745712 (-1.340846) | 2.895397 / 5.269862 (-2.374464) | 1.799864 / 4.565676 (-2.765812) | 0.064386 / 0.424275 (-0.359889) | 0.004920 / 0.007607 (-0.002687) | 0.330879 / 0.226044 (0.104835) | 3.287064 / 2.268929 (1.018135) | 1.765169 / 55.444624 (-53.679456) | 1.490442 / 6.876477 (-5.386034) | 1.530960 / 2.142072 (-0.611113) | 0.655939 / 4.805227 (-4.149288) | 0.118529 / 6.500664 (-6.382135) | 0.042350 / 0.075469 (-0.033119) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.959027 / 1.841788 (-0.882761) | 11.911284 / 8.074308 (3.836976) | 10.576898 / 10.191392 (0.385506) | 0.141038 / 0.680424 (-0.539386) | 0.014184 / 0.534201 (-0.520017) | 0.305335 / 0.579283 (-0.273948) | 0.267531 / 0.434364 (-0.166832) | 0.353362 / 0.540337 (-0.186975) | 0.466258 / 1.386936 (-0.920678) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005194 / 0.011353 (-0.006159) | 0.003561 / 0.011008 (-0.007448) | 0.049181 / 0.038508 (0.010673) | 0.056664 / 0.023109 (0.033555) | 0.267142 / 0.275898 (-0.008756) | 0.291871 / 0.323480 (-0.031609) | 0.003996 / 0.007986 (-0.003990) | 0.003147 / 0.004328 (-0.001181) | 0.048527 / 0.004250 (0.044276) | 0.040239 / 0.037052 (0.003187) | 0.269728 / 0.258489 (0.011239) | 0.295531 / 0.293841 (0.001690) | 0.030316 / 0.128546 (-0.098231) | 0.010666 / 0.075646 (-0.064981) | 0.058176 / 0.419271 (-0.361095) | 0.033218 / 0.043533 (-0.010315) | 0.265383 / 0.255139 (0.010244) | 0.285102 / 0.283200 (0.001902) | 0.018295 / 0.141683 (-0.123388) | 1.117830 / 1.452155 (-0.334325) | 1.196919 / 1.492716 (-0.295798) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.088547 / 0.018006 (0.070541) | 0.293220 / 0.000490 (0.292730) | 0.000211 / 0.000200 (0.000011) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022060 / 0.037411 (-0.015351) | 0.071973 / 0.014526 (0.057448) | 0.081721 / 0.176557 (-0.094836) | 0.119990 / 0.737135 (-0.617145) | 0.081639 / 0.296338 (-0.214700) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.293712 / 0.215209 (0.078503) | 2.872986 / 2.077655 (0.795331) | 1.568944 / 1.504120 (0.064824) | 1.434555 / 1.541195 (-0.106639) | 1.457747 / 1.468490 (-0.010743) | 0.559296 / 4.584777 (-4.025481) | 2.471845 / 3.745712 (-1.273867) | 2.840916 / 5.269862 (-2.428946) | 1.754909 / 4.565676 (-2.810768) | 0.064585 / 0.424275 (-0.359690) | 0.004992 / 0.007607 (-0.002615) | 0.349149 / 0.226044 (0.123104) | 3.385906 / 2.268929 (1.116977) | 1.940644 / 55.444624 (-53.503980) | 1.638300 / 6.876477 (-5.238177) | 1.649939 / 2.142072 (-0.492133) | 0.645680 / 4.805227 (-4.159547) | 0.118080 / 6.500664 (-6.382584) | 0.040643 / 0.075469 (-0.034826) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.969965 / 1.841788 (-0.871822) | 12.099766 / 8.074308 (4.025457) | 10.550650 / 10.191392 (0.359258) | 0.131736 / 0.680424 (-0.548688) | 0.015483 / 0.534201 (-0.518718) | 0.289231 / 0.579283 (-0.290052) | 0.287505 / 0.434364 (-0.146858) | 0.327326 / 0.540337 (-0.213011) | 0.570364 / 1.386936 (-0.816572) |

![](https://cml.dev/watermark.png#533c38cef16111e9e8154eeb76c207f1f4936ddf "CML watermark")

see https://discuss.huggingface.co/t/update-datasets-getting-started-to-new-git-security/65893


Fallback on dataset script if user wants to load default config

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6498). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> I was just thinking: what if the user does not pass a config name and the dataset has only a config with a name different from "default"?

You mean if there is a DEFAULT_CONFIG_NAME defined in the script but the dataset only has one configuration ? We can't easily get the number of configs without running the python code so I don't think we can support detect this case

Most datasets with a script don't define DEFAULT_CONFIG_NAME if there is only one configuration anyway.

So there is no issue e.g. for `squad`

> I was trying to mean the case where DEFAULT_CONFIG_NAME is None but there is only a single config in BUILDER_CONFIGS, with a name different from "default".

In this case we can detect if "DEFAULT_CONFIG_NAME" is not mentioned and use the Parquet export. If it is mentioned (and maybe it is set to None or to the single config) I consider that it may have multiple configs and fall back on using the script

... but the user does not pass the config name.

In this case we load the single configuration (this is how a DatasetBuilder works)

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005122 / 0.011353 (-0.006231) | 0.003565 / 0.011008 (-0.007443) | 0.062706 / 0.038508 (0.024198) | 0.049314 / 0.023109 (0.026205) | 0.247325 / 0.275898 (-0.028573) | 0.269788 / 0.323480 (-0.053692) | 0.003895 / 0.007986 (-0.004090) | 0.002788 / 0.004328 (-0.001540) | 0.048615 / 0.004250 (0.044365) | 0.037591 / 0.037052 (0.000539) | 0.253495 / 0.258489 (-0.004994) | 0.281200 / 0.293841 (-0.012641) | 0.027712 / 0.128546 (-0.100834) | 0.010901 / 0.075646 (-0.064745) | 0.205577 / 0.419271 (-0.213694) | 0.035989 / 0.043533 (-0.007544) | 0.252978 / 0.255139 (-0.002161) | 0.268042 / 0.283200 (-0.015157) | 0.017857 / 0.141683 (-0.123826) | 1.096633 / 1.452155 (-0.355521) | 1.147026 / 1.492716 (-0.345691) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095609 / 0.018006 (0.077603) | 0.311941 / 0.000490 (0.311451) | 0.000211 / 0.000200 (0.000011) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019042 / 0.037411 (-0.018369) | 0.060549 / 0.014526 (0.046023) | 0.074761 / 0.176557 (-0.101796) | 0.121729 / 0.737135 (-0.615406) | 0.075661 / 0.296338 (-0.220677) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.284774 / 0.215209 (0.069565) | 2.764576 / 2.077655 (0.686921) | 1.489926 / 1.504120 (-0.014194) | 1.387276 / 1.541195 (-0.153919) | 1.400931 / 1.468490 (-0.067559) | 0.555623 / 4.584777 (-4.029154) | 2.409488 / 3.745712 (-1.336224) | 2.781053 / 5.269862 (-2.488808) | 1.750472 / 4.565676 (-2.815204) | 0.062232 / 0.424275 (-0.362043) | 0.004974 / 0.007607 (-0.002633) | 0.336324 / 0.226044 (0.110280) | 3.286619 / 2.268929 (1.017691) | 1.825070 / 55.444624 (-53.619554) | 1.537993 / 6.876477 (-5.338484) | 1.586520 / 2.142072 (-0.555553) | 0.640090 / 4.805227 (-4.165138) | 0.117637 / 6.500664 (-6.383027) | 0.042318 / 0.075469 (-0.033151) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.964051 / 1.841788 (-0.877736) | 11.706259 / 8.074308 (3.631951) | 10.752311 / 10.191392 (0.560919) | 0.128117 / 0.680424 (-0.552307) | 0.014001 / 0.534201 (-0.520200) | 0.286255 / 0.579283 (-0.293028) | 0.263810 / 0.434364 (-0.170554) | 0.329347 / 0.540337 (-0.210991) | 0.437349 / 1.386936 (-0.949587) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005303 / 0.011353 (-0.006050) | 0.003586 / 0.011008 (-0.007422) | 0.049339 / 0.038508 (0.010831) | 0.051287 / 0.023109 (0.028178) | 0.274397 / 0.275898 (-0.001501) | 0.292977 / 0.323480 (-0.030503) | 0.004029 / 0.007986 (-0.003957) | 0.002727 / 0.004328 (-0.001602) | 0.048779 / 0.004250 (0.044528) | 0.040075 / 0.037052 (0.003022) | 0.277676 / 0.258489 (0.019187) | 0.301963 / 0.293841 (0.008122) | 0.029340 / 0.128546 (-0.099206) | 0.010714 / 0.075646 (-0.064932) | 0.057253 / 0.419271 (-0.362018) | 0.033426 / 0.043533 (-0.010107) | 0.276673 / 0.255139 (0.021534) | 0.291053 / 0.283200 (0.007854) | 0.017660 / 0.141683 (-0.124023) | 1.122354 / 1.452155 (-0.329800) | 1.180381 / 1.492716 (-0.312335) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091903 / 0.018006 (0.073897) | 0.300720 / 0.000490 (0.300231) | 0.000288 / 0.000200 (0.000088) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021521 / 0.037411 (-0.015890) | 0.068233 / 0.014526 (0.053707) | 0.081245 / 0.176557 (-0.095312) | 0.119996 / 0.737135 (-0.617139) | 0.082483 / 0.296338 (-0.213856) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.302776 / 0.215209 (0.087567) | 2.950776 / 2.077655 (0.873122) | 1.631032 / 1.504120 (0.126912) | 1.502021 / 1.541195 (-0.039174) | 1.514213 / 1.468490 (0.045723) | 0.578246 / 4.584777 (-4.006531) | 2.443768 / 3.745712 (-1.301944) | 2.827811 / 5.269862 (-2.442051) | 1.771529 / 4.565676 (-2.794148) | 0.064479 / 0.424275 (-0.359797) | 0.005061 / 0.007607 (-0.002546) | 0.350966 / 0.226044 (0.124922) | 3.458616 / 2.268929 (1.189687) | 1.967917 / 55.444624 (-53.476707) | 1.704661 / 6.876477 (-5.171815) | 1.698895 / 2.142072 (-0.443178) | 0.663259 / 4.805227 (-4.141968) | 0.122140 / 6.500664 (-6.378525) | 0.041099 / 0.075469 (-0.034371) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.972080 / 1.841788 (-0.869708) | 12.123286 / 8.074308 (4.048978) | 10.819854 / 10.191392 (0.628462) | 0.131486 / 0.680424 (-0.548938) | 0.015785 / 0.534201 (-0.518416) | 0.290048 / 0.579283 (-0.289235) | 0.277822 / 0.434364 (-0.156542) | 0.325949 / 0.540337 (-0.214388) | 0.577681 / 1.386936 (-0.809255) |

![](https://cml.dev/watermark.png#30f6a2d9af183eba4501f0b8d90e9200bdca6bb1 "CML watermark")

Right now this code is failing on `main`:

```python
load_dataset("openbookqa")
```

This is because it tries to load the dataset from the Parquet export but the dataset has multiple configurations and the Parquet export doesn't know which one is the default one.

I fixed this by simply falling back on using the dataset script (which tells the user to pass `trust_remote_code=True`):

```python
load_dataset("openbookqa", trust_remote_code=True)
```

Note that if the user happened to specify a config name I don't fall back on the script since we can use the Parquet export in this case (no need to know which config is the default)

```python
load_dataset("openbookqa", "main")
```


Support setting a default config name in push_to_hub

enhancement

New feature or request

In order to convert script-datasets to no-script datasets, we need to support setting a default config name for those scripts that set one.


Error when writing a dataset to HF Hub: A commit has happened since. Please refresh and try again.

I transferred from datasets-server, since the issue is more about `datasets` and the integration with `huggingface_hub`.

**Describe the bug**

Getting a `412 Client Error: Precondition Failed` when trying to write a dataset to the HF hub.

```
huggingface_hub.utils._errors.HfHubHTTPError: 412 Client Error: Precondition Failed for url: https://huggingface.co/api/datasets/GLorr/test-dask/commit/main (Request ID: Root=1-657ae26f-3bd92bf861bb254b2cc0826c;50a09ab7-9347-406a-ba49-69f98abee9cc)

A commit has happened since. Please refresh and try again.
```

**Steps to reproduce the bug**
This is a minimal reproducer:
```
import dask.dataframe as dd
import pandas as pd
import random
import os

import huggingface_hub

import datasets

huggingface_hub.login(token=os.getenv("HF_TOKEN"))

data = {"number": [random.randint(0,10) for _ in range(1000)]}
df = pd.DataFrame.from_dict(data)
dataframe = dd.from_pandas(df, npartitions=1)
dataframe = dataframe.repartition(npartitions=3)

schema = datasets.Features({"number": datasets.Value("int64")}).arrow_schema

repo_id = "GLorr/test-dask"
repo_path = f"hf://datasets/{repo_id}"
huggingface_hub.create_repo(repo_id=repo_id, repo_type="dataset", exist_ok=True)
dd.to_parquet(dataframe, path=f"{repo_path}/data", schema=schema)
```

**Expected behavior**
Would expect to write to the hub without any problem.

**Environment info**
```
datasets==2.15.0
huggingface-hub==0.19.4
```


Image Data loaded Twice

### Describe the bug

![1702472610561](https://github.com/huggingface/datasets/assets/28867010/4b7ef5e7-32c3-4b73-84cb-5de059caa0b6)
When I learn from https://huggingface.co/docs/datasets/image_load and try to load image data from a folder. I noticed that the image was read twice in the returned data. As you can see in the attached image, there are only four images in the train folder, but reading brings up eight images

### Steps to reproduce the bug

from datasets import Dataset, load_dataset

dataset = load_dataset("imagefolder", data_dir="data/", drop_labels=False)
# print(dataset["train"][0]["image"] == dataset["train"][1]["image"])

print(dataset)
print(dataset["train"]["image"])
print(len(dataset["train"]["image"]))

### Expected behavior

DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 8
    })
})
[<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=2877x2129 at 0x1BD1D1CA8B0>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=2877x2129 at 0x1BD1D2452E0>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=4208x3120 at 0x1BD1D245310>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=4208x3120 at 0x1BD1D2453A0>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=2877x2129 at 0x1BD1D245460>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=2877x2129 at 0x1BD1D245430>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=4208x3120 at 0x1BD1D2454F0>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=4208x3120 at 0x1BD1D245550>]
8

### Environment info

- `datasets` version: 2.14.5
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.9.17
- Huggingface_hub version: 0.19.4
- PyArrow version: 13.0.0
- Pandas version: 2.0.3


Newline characters don't behave as expected when calling dataset.info

### System Info

- `transformers` version: 4.32.1
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.11.5
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.1+cpu (False)
- Tensorflow version (GPU?): 2.15.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@marios

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

[Source](https://huggingface.co/docs/datasets/v2.2.1/en/access)
```
from datasets import load_dataset
dataset = load_dataset('glue', 'mrpc', split='train')
dataset.info
```

DatasetInfo(description='GLUE, the General Language Understanding Evaluation benchmark\n(https://gluebenchmark.com/) is a collection of resources for training,\nevaluating, and analyzing natural language understanding systems.\n\n', citation='@inproceedings{dolan2005automatically,\n  title={Automatically constructing a corpus of sentential paraphrases},\n  author={Dolan, William B and Brockett, Chris},\n  booktitle={Proceedings of the Third International Workshop on Paraphrasing (IWP2005)},\n  year={2005}\n}\n@inproceedings{wang2019glue,\n  title={{GLUE}: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding},\n  author={Wang, Alex and Singh, Amanpreet and Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel R.},\n  note={In the Proceedings of ICLR.},\n  year={2019}\n}\n', homepage='https://www.microsoft.com/en-us/download/details.aspx?id=52398', license='', features={'sentence1': Value(dtype='string', id=None), 'sentence2': Value(dtype='string', id=None), 'label': ClassLabel(names=['not_equivalent', 'equivalent'], id=None), 'idx': Value(dtype='int32', id=None)}, post_processed=None, supervised_keys=None, task_templates=None, builder_name='glue', dataset_name=None, config_name='mrpc', version=1.0.0, splits={'train': SplitInfo(name='train', num_bytes=943843, num_examples=3668, shard_lengths=None, dataset_name='glue'), 'validation': SplitInfo(name='validation', num_bytes=105879, num_examples=408, shard_lengths=None, dataset_name='glue'), 'test': SplitInfo(name='test', num_bytes=442410, num_examples=1725, shard_lengths=None, dataset_name='glue')}, download_checksums={'https://dl.fbaipublicfiles.com/glue/data/mrpc_dev_ids.tsv': {'num_bytes': 6222, 'checksum': None}, 'https://dl.fbaipublicfiles.com/senteval/senteval_data/msr_paraphrase_train.txt': {'num_bytes': 1047044, 'checksum': None}, 'https://dl.fbaipublicfiles.com/senteval/senteval_data/msr_paraphrase_test.txt': {'num_bytes': 441275, 'checksum': None}}, download_size=1494541, post_processing_size=None, dataset_size=1492132, size_in_bytes=2986673)

### Expected behavior

```
from datasets import load_dataset
dataset = load_dataset('glue', 'mrpc', split='train')
dataset.info
```
DatasetInfo(
    description='GLUE, the General Language Understanding Evaluation benchmark\n(https://gluebenchmark.com/) is a collection of resources for training,\nevaluating, and analyzing natural language understanding systems.\n\n',
    citation='@inproceedings{dolan2005automatically,\n  title={Automatically constructing a corpus of sentential paraphrases},\n  author={Dolan, William B and Brockett, Chris},\n  booktitle={Proceedings of the Third International Workshop on Paraphrasing (IWP2005)},\n  year={2005}\n}\n@inproceedings{wang2019glue,\n  title={{GLUE}: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding},\n  author={Wang, Alex and Singh, Amanpreet and Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel R.},\n  note={In the Proceedings of ICLR.},\n  year={2019}\n}\n', homepage='https://www.microsoft.com/en-us/download/details.aspx?id=52398',
    license='',
    features={'sentence1': Value(dtype='string', id=None), 'sentence2': Value(dtype='string', id=None), 'label': ClassLabel(num_classes=2, names=['not_equivalent', 'equivalent'], names_file=None, id=None), 'idx': Value(dtype='int32', id=None)}, post_processed=None, supervised_keys=None, builder_name='glue', config_name='mrpc', version=1.0.0, splits={'train': SplitInfo(name='train', num_bytes=943851, num_examples=3668, dataset_name='glue'), 'validation': SplitInfo(name='validation', num_bytes=105887, num_examples=408, dataset_name='glue'), 'test': SplitInfo(name='test', num_bytes=442418, num_examples=1725, dataset_name='glue')},
    download_checksums={'https://dl.fbaipublicfiles.com/glue/data/mrpc_dev_ids.tsv': {'num_bytes': 6222, 'checksum': '971d7767d81b997fd9060ade0ec23c4fc31cbb226a55d1bd4a1bac474eb81dc7'}, 'https://dl.fbaipublicfiles.com/senteval/senteval_data/msr_paraphrase_train.txt': {'num_bytes': 1047044, 'checksum': '60a9b09084528f0673eedee2b69cb941920f0b8cd0eeccefc464a98768457f89'}, 'https://dl.fbaipublicfiles.com/senteval/senteval_data/msr_paraphrase_test.txt': {'num_bytes': 441275, 'checksum': 'a04e271090879aaba6423d65b94950c089298587d9c084bf9cd7439bd785f784'}},
    download_size=1494541,
    post_processing_size=None,
    dataset_size=1492156,
    size_in_bytes=2986697
)


Lazy data files resolution and offline cache reload

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6493). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Naive question: is there any breaking change when loading?

No breaking changes except that the cache folders are different

e.g. for glue sst2 (has parquet export)

```
This branch (new format is config/version/commit_sha)
~/.cache/huggingface/datasets/glue/sst2/1.0.0/fd8e86499fa5c264fcaad392a8f49ddf58bf4037
On main
~/.cache/huggingface/datasets/glue/sst2/0.0.0/74a75637ac4acd3f
On 2.15.0
~/.cache/huggingface/datasets/glue/sst2/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad
```

e.g. for wikimedia/wikipedia 20231101.ab (has metadata configs)

```
This branch (new format is config/version/commit_sha)
~/.cache/huggingface/datasets/wikimedia___wikipedia/20231101.ab/0.0.0/4cb9b0d719291f1a10f96f67d609c5d442980dc9
On main (takes ages to load)
~/.cache/huggingface/datasets/wikimedia___wikipedia/20231101.ab/0.0.0/cfa627e27933df13
On 2.15.0 (takes ages to load)
~/.cache/huggingface/datasets/wikimedia___wikipedia/20231101.ab/0.0.0/e92ee7a91c466564
```

e.g. for lhoestq/demo1 (no metadata configs)

```
This branch (new format is config/version/commit_sha)
~/.cache/huggingface/datasets/lhoestq___demo1/default/0.0.0/87ecf163bedca9d80598b528940a9c4f99e14c11
On main
~/.cache/huggingface/datasets/lhoestq___demo1/default-8a4a0b7a240d3c5e/0.0.0/eea64c71ca8b46dd3f537ed218fc9bf495d5707789152eb2764f5c78fa66d59d
On 2.15.0
~/.cache/huggingface/datasets/lhoestq___demo1/default-59d4029e0bb36ae0/0.0.0/eea64c71ca8b46dd3f537ed218fc9bf495d5707789152eb2764f5c78fa66d59d
```

There was a last bug I just fixed: if you modify a dataset and reload it from the hub it won't download the new version - I think I need to use another hash to name the cache directory
edit: fixed

I switched to using the git commit sha for the cache directory, which is now `config/version/commit_sha` :) much cleaner than before.

And for local file it's a hash that takes into account the resolved files (and their last modified dates)

I also ran the `transformers` CI on this branch and it's green

FYI `huggingface_hub` will have a release on tuesday/wednesday (will speed up load_dataset data files resolution which is now needed for datasets loaded from parquet export) so we can aim on merging this around the same time and do a release on thursday

Merging this one, and hopefully the cache backward compatibility PR soon too :)

Then it will be release time

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005444 / 0.011353 (-0.005909) | 0.003562 / 0.011008 (-0.007446) | 0.063183 / 0.038508 (0.024675) | 0.048885 / 0.023109 (0.025776) | 0.248422 / 0.275898 (-0.027476) | 0.277844 / 0.323480 (-0.045636) | 0.003019 / 0.007986 (-0.004966) | 0.002660 / 0.004328 (-0.001669) | 0.048928 / 0.004250 (0.044677) | 0.044850 / 0.037052 (0.007798) | 0.248505 / 0.258489 (-0.009984) | 0.282231 / 0.293841 (-0.011610) | 0.028302 / 0.128546 (-0.100244) | 0.010829 / 0.075646 (-0.064818) | 0.206738 / 0.419271 (-0.212533) | 0.035485 / 0.043533 (-0.008048) | 0.244575 / 0.255139 (-0.010564) | 0.281411 / 0.283200 (-0.001789) | 0.019563 / 0.141683 (-0.122120) | 1.113769 / 1.452155 (-0.338386) | 1.176831 / 1.492716 (-0.315885) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.004718 / 0.018006 (-0.013288) | 0.304103 / 0.000490 (0.303614) | 0.000214 / 0.000200 (0.000014) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019642 / 0.037411 (-0.017769) | 0.060275 / 0.014526 (0.045749) | 0.073072 / 0.176557 (-0.103484) | 0.119789 / 0.737135 (-0.617346) | 0.074535 / 0.296338 (-0.221804) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.278799 / 0.215209 (0.063590) | 2.725320 / 2.077655 (0.647665) | 1.419048 / 1.504120 (-0.085071) | 1.335041 / 1.541195 (-0.206154) | 1.373029 / 1.468490 (-0.095461) | 0.566774 / 4.584777 (-4.018003) | 2.383796 / 3.745712 (-1.361916) | 2.734804 / 5.269862 (-2.535057) | 1.712277 / 4.565676 (-2.853399) | 0.062119 / 0.424275 (-0.362156) | 0.004949 / 0.007607 (-0.002658) | 0.336126 / 0.226044 (0.110082) | 3.298602 / 2.268929 (1.029674) | 1.842815 / 55.444624 (-53.601809) | 1.544028 / 6.876477 (-5.332449) | 1.566717 / 2.142072 (-0.575355) | 0.643006 / 4.805227 (-4.162221) | 0.118241 / 6.500664 (-6.382423) | 0.042453 / 0.075469 (-0.033016) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.949015 / 1.841788 (-0.892773) | 11.717958 / 8.074308 (3.643649) | 10.482448 / 10.191392 (0.291056) | 0.128564 / 0.680424 (-0.551860) | 0.014792 / 0.534201 (-0.519408) | 0.288636 / 0.579283 (-0.290647) | 0.263345 / 0.434364 (-0.171019) | 0.325753 / 0.540337 (-0.214584) | 0.421294 / 1.386936 (-0.965642) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005367 / 0.011353 (-0.005985) | 0.003802 / 0.011008 (-0.007206) | 0.049322 / 0.038508 (0.010814) | 0.055201 / 0.023109 (0.032092) | 0.287811 / 0.275898 (0.011913) | 0.305141 / 0.323480 (-0.018339) | 0.004095 / 0.007986 (-0.003890) | 0.002733 / 0.004328 (-0.001595) | 0.049508 / 0.004250 (0.045258) | 0.039199 / 0.037052 (0.002147) | 0.282719 / 0.258489 (0.024230) | 0.311156 / 0.293841 (0.017315) | 0.029469 / 0.128546 (-0.099077) | 0.010709 / 0.075646 (-0.064937) | 0.057646 / 0.419271 (-0.361626) | 0.032696 / 0.043533 (-0.010837) | 0.285087 / 0.255139 (0.029948) | 0.294142 / 0.283200 (0.010942) | 0.019779 / 0.141683 (-0.121904) | 1.176844 / 1.452155 (-0.275310) | 1.190925 / 1.492716 (-0.301792) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092885 / 0.018006 (0.074879) | 0.301129 / 0.000490 (0.300640) | 0.000232 / 0.000200 (0.000032) | 0.000049 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023202 / 0.037411 (-0.014210) | 0.076850 / 0.014526 (0.062325) | 0.090058 / 0.176557 (-0.086499) | 0.128091 / 0.737135 (-0.609045) | 0.091098 / 0.296338 (-0.205240) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.292973 / 0.215209 (0.077764) | 2.876022 / 2.077655 (0.798367) | 1.672115 / 1.504120 (0.167995) | 1.555103 / 1.541195 (0.013909) | 1.559832 / 1.468490 (0.091342) | 0.558017 / 4.584777 (-4.026760) | 2.428448 / 3.745712 (-1.317264) | 2.812024 / 5.269862 (-2.457837) | 1.738470 / 4.565676 (-2.827207) | 0.062669 / 0.424275 (-0.361607) | 0.005071 / 0.007607 (-0.002536) | 0.351804 / 0.226044 (0.125759) | 3.412207 / 2.268929 (1.143279) | 2.023478 / 55.444624 (-53.421147) | 1.761281 / 6.876477 (-5.115195) | 1.770789 / 2.142072 (-0.371283) | 0.643062 / 4.805227 (-4.162165) | 0.116616 / 6.500664 (-6.384048) | 0.041816 / 0.075469 (-0.033653) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.988430 / 1.841788 (-0.853357) | 12.278636 / 8.074308 (4.204328) | 11.066185 / 10.191392 (0.874793) | 0.141191 / 0.680424 (-0.539233) | 0.015547 / 0.534201 (-0.518654) | 0.288045 / 0.579283 (-0.291238) | 0.279651 / 0.434364 (-0.154713) | 0.329869 / 0.540337 (-0.210469) | 0.420391 / 1.386936 (-0.966545) |

![](https://cml.dev/watermark.png#ef3b5dd3633995c95d77f35fb17f89ff44990bc4 "CML watermark")

Includes both https://github.com/huggingface/datasets/pull/6458 and https://github.com/huggingface/datasets/pull/6459

This PR should be merged instead of the two individually, since they are conflicting

## Offline cache reload

 it can reload datasets that were pushed to hub if they exist in the cache.

example:

```python
>>> Dataset.from_dict({"a": [1, 2]}).push_to_hub("lhoestq/tmp")
>>> load_dataset("lhoestq/tmp")
DatasetDict({
    train: Dataset({
        features: ['a'],
        num_rows: 2
    })
})
```

and later, without connection:

```python
>>> load_dataset("lhoestq/tmp")
Using the latest cached version of the dataset since lhoestq/tmp couldn't be found on the Hugging Face Hub
Found the latest cached dataset configuration 'default' at /Users/quentinlhoest/.cache/huggingface/datasets/lhoestq___tmp/default/0.0.0/da0e902a945afeb9 (last modified on Wed Dec 13 14:55:52 2023).
DatasetDict({
    train: Dataset({
        features: ['a'],
        num_rows: 2
    })
})
```

- Updated `CachedDatasetModuleFactory` to look for datasets in the cache at `<namespace>___<dataset_name>/<config_id>`
- Since the metadata configs parameters are not available in offline mode, we don't know which folder to load (config_id and hash change), so I simply load the latest one
  - I instantiate a BuilderConfig even if there is no metadata config with the right config_name
  - Its config_id is equal to the config_name to be able to retrieve it in the cache (no more suffix for configs from metadata configs)
  - We can reload this config if offline mode by specifying the right config_name (same as online !)
- Consequences of this change:
  - Only when there are user's parameters it creates a custom builder config with config_id = config_name + user parameters hash
  - the hash used to name the cache folder takes into account the metadata config and the dataset info, so that the right cache can be reloaded when there is internet connection without redownloading the data or resolving the data files. For local directories I hash the builder configs and dataset info, and for datasets on the hub I use the commit sha as hash.
  - cache directories now look like `config/version/commit_sha` for hub datasets which is clean :)

Fix https://github.com/huggingface/datasets/issues/3547

## Lazy data files resolution

this makes this code run in 2sec instead of >10sec

```python
from datasets import load_dataset

ds = load_dataset("glue", "sst2", streaming=True, trust_remote_code=False)
```

For some datasets with many configs and files it can be up to 100x faster.
This is particularly important now that some datasets will be loaded from the Parquet export instead of the scripts.

The data files are only resolved in the builder `__init__`. To do so I added DataFilesPatternsList and DataFilesPatternsDict that have `.resolve()` to return resolved DataFilesList and DataFilesDict


Make push_to_hub return CommitInfo

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6492). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This PR is ready to review @huggingface/datasets.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005093 / 0.011353 (-0.006259) | 0.003695 / 0.011008 (-0.007313) | 0.064648 / 0.038508 (0.026140) | 0.054677 / 0.023109 (0.031568) | 0.242007 / 0.275898 (-0.033891) | 0.265216 / 0.323480 (-0.058264) | 0.003847 / 0.007986 (-0.004138) | 0.003773 / 0.004328 (-0.000556) | 0.048595 / 0.004250 (0.044345) | 0.038122 / 0.037052 (0.001070) | 0.245698 / 0.258489 (-0.012791) | 0.278095 / 0.293841 (-0.015746) | 0.027488 / 0.128546 (-0.101058) | 0.011002 / 0.075646 (-0.064644) | 0.211443 / 0.419271 (-0.207829) | 0.035664 / 0.043533 (-0.007869) | 0.244754 / 0.255139 (-0.010385) | 0.261078 / 0.283200 (-0.022121) | 0.017768 / 0.141683 (-0.123915) | 1.130765 / 1.452155 (-0.321390) | 1.189825 / 1.492716 (-0.302891) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093027 / 0.018006 (0.075021) | 0.302193 / 0.000490 (0.301703) | 0.000207 / 0.000200 (0.000007) | 0.000045 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018413 / 0.037411 (-0.018999) | 0.062715 / 0.014526 (0.048190) | 0.073287 / 0.176557 (-0.103269) | 0.120394 / 0.737135 (-0.616741) | 0.077573 / 0.296338 (-0.218765) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.284445 / 0.215209 (0.069236) | 2.780718 / 2.077655 (0.703063) | 1.460988 / 1.504120 (-0.043132) | 1.345799 / 1.541195 (-0.195395) | 1.399892 / 1.468490 (-0.068598) | 0.576051 / 4.584777 (-4.008726) | 2.418792 / 3.745712 (-1.326921) | 2.901330 / 5.269862 (-2.368532) | 1.765083 / 4.565676 (-2.800593) | 0.063555 / 0.424275 (-0.360720) | 0.004991 / 0.007607 (-0.002616) | 0.339657 / 0.226044 (0.113613) | 3.372963 / 2.268929 (1.104034) | 1.853667 / 55.444624 (-53.590958) | 1.552022 / 6.876477 (-5.324454) | 1.616452 / 2.142072 (-0.525620) | 0.652309 / 4.805227 (-4.152919) | 0.121125 / 6.500664 (-6.379539) | 0.042420 / 0.075469 (-0.033049) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.954514 / 1.841788 (-0.887274) | 11.853736 / 8.074308 (3.779428) | 10.624571 / 10.191392 (0.433179) | 0.134118 / 0.680424 (-0.546306) | 0.014200 / 0.534201 (-0.520001) | 0.290106 / 0.579283 (-0.289177) | 0.270637 / 0.434364 (-0.163727) | 0.336155 / 0.540337 (-0.204182) | 0.443962 / 1.386936 (-0.942974) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005282 / 0.011353 (-0.006071) | 0.003526 / 0.011008 (-0.007482) | 0.048994 / 0.038508 (0.010486) | 0.055345 / 0.023109 (0.032236) | 0.271587 / 0.275898 (-0.004311) | 0.294676 / 0.323480 (-0.028804) | 0.003989 / 0.007986 (-0.003996) | 0.002594 / 0.004328 (-0.001735) | 0.048310 / 0.004250 (0.044059) | 0.039945 / 0.037052 (0.002893) | 0.277304 / 0.258489 (0.018815) | 0.312017 / 0.293841 (0.018176) | 0.028364 / 0.128546 (-0.100182) | 0.010683 / 0.075646 (-0.064963) | 0.057990 / 0.419271 (-0.361281) | 0.032418 / 0.043533 (-0.011115) | 0.273835 / 0.255139 (0.018697) | 0.288585 / 0.283200 (0.005385) | 0.018964 / 0.141683 (-0.122719) | 1.148863 / 1.452155 (-0.303292) | 1.195684 / 1.492716 (-0.297032) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091967 / 0.018006 (0.073960) | 0.303236 / 0.000490 (0.302747) | 0.000214 / 0.000200 (0.000015) | 0.000051 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021960 / 0.037411 (-0.015452) | 0.068744 / 0.014526 (0.054218) | 0.081167 / 0.176557 (-0.095390) | 0.119623 / 0.737135 (-0.617513) | 0.084965 / 0.296338 (-0.211373) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.297740 / 0.215209 (0.082531) | 2.924856 / 2.077655 (0.847201) | 1.602080 / 1.504120 (0.097960) | 1.494083 / 1.541195 (-0.047112) | 1.544662 / 1.468490 (0.076172) | 0.581212 / 4.584777 (-4.003565) | 2.451064 / 3.745712 (-1.294648) | 2.875213 / 5.269862 (-2.394649) | 1.780777 / 4.565676 (-2.784900) | 0.063751 / 0.424275 (-0.360524) | 0.004967 / 0.007607 (-0.002641) | 0.350321 / 0.226044 (0.124276) | 3.449585 / 2.268929 (1.180657) | 1.977666 / 55.444624 (-53.466958) | 1.685125 / 6.876477 (-5.191351) | 1.734466 / 2.142072 (-0.407606) | 0.657477 / 4.805227 (-4.147750) | 0.116767 / 6.500664 (-6.383898) | 0.041400 / 0.075469 (-0.034069) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.985751 / 1.841788 (-0.856037) | 12.300065 / 8.074308 (4.225756) | 10.608238 / 10.191392 (0.416846) | 0.139907 / 0.680424 (-0.540517) | 0.015379 / 0.534201 (-0.518822) | 0.283528 / 0.579283 (-0.295755) | 0.278751 / 0.434364 (-0.155613) | 0.328811 / 0.540337 (-0.211527) | 0.584041 / 1.386936 (-0.802895) |

![](https://cml.dev/watermark.png#ef0f986518bd252c5314a7e3a419dedcbb166630 "CML watermark")

Make `push_to_hub` return `CommitInfo`.

This is useful, for example, if we pass `create_pr=True` and we want to know the created PR ID.

CC: @severo for the use case in https://huggingface.co/datasets/jmhessel/newyorker_caption_contest/discussions/4


Fix metrics dead link

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6491). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005193 / 0.011353 (-0.006160) | 0.003246 / 0.011008 (-0.007762) | 0.063053 / 0.038508 (0.024545) | 0.049636 / 0.023109 (0.026527) | 0.240990 / 0.275898 (-0.034908) | 0.263732 / 0.323480 (-0.059747) | 0.004062 / 0.007986 (-0.003923) | 0.002681 / 0.004328 (-0.001648) | 0.048527 / 0.004250 (0.044277) | 0.044159 / 0.037052 (0.007107) | 0.248031 / 0.258489 (-0.010458) | 0.275705 / 0.293841 (-0.018136) | 0.028210 / 0.128546 (-0.100336) | 0.010314 / 0.075646 (-0.065332) | 0.209887 / 0.419271 (-0.209384) | 0.035649 / 0.043533 (-0.007884) | 0.251321 / 0.255139 (-0.003818) | 0.266672 / 0.283200 (-0.016528) | 0.017382 / 0.141683 (-0.124301) | 1.088937 / 1.452155 (-0.363217) | 1.143692 / 1.492716 (-0.349024) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092558 / 0.018006 (0.074552) | 0.301648 / 0.000490 (0.301159) | 0.000208 / 0.000200 (0.000008) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018305 / 0.037411 (-0.019106) | 0.059836 / 0.014526 (0.045310) | 0.072926 / 0.176557 (-0.103631) | 0.119826 / 0.737135 (-0.617309) | 0.074357 / 0.296338 (-0.221982) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.279051 / 0.215209 (0.063842) | 2.711402 / 2.077655 (0.633747) | 1.431782 / 1.504120 (-0.072338) | 1.316592 / 1.541195 (-0.224603) | 1.352062 / 1.468490 (-0.116428) | 0.562553 / 4.584777 (-4.022224) | 2.387719 / 3.745712 (-1.357993) | 2.693330 / 5.269862 (-2.576532) | 1.682040 / 4.565676 (-2.883636) | 0.061832 / 0.424275 (-0.362443) | 0.005066 / 0.007607 (-0.002541) | 0.332730 / 0.226044 (0.106685) | 3.315503 / 2.268929 (1.046575) | 1.787129 / 55.444624 (-53.657496) | 1.508955 / 6.876477 (-5.367522) | 1.512620 / 2.142072 (-0.629453) | 0.637120 / 4.805227 (-4.168107) | 0.116005 / 6.500664 (-6.384660) | 0.041973 / 0.075469 (-0.033496) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.936996 / 1.841788 (-0.904792) | 11.485975 / 8.074308 (3.411667) | 10.604481 / 10.191392 (0.413089) | 0.130803 / 0.680424 (-0.549621) | 0.014561 / 0.534201 (-0.519640) | 0.285905 / 0.579283 (-0.293378) | 0.271573 / 0.434364 (-0.162791) | 0.329206 / 0.540337 (-0.211132) | 0.411977 / 1.386936 (-0.974959) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005333 / 0.011353 (-0.006020) | 0.003519 / 0.011008 (-0.007489) | 0.050880 / 0.038508 (0.012372) | 0.053681 / 0.023109 (0.030571) | 0.269359 / 0.275898 (-0.006539) | 0.291498 / 0.323480 (-0.031982) | 0.004006 / 0.007986 (-0.003979) | 0.002676 / 0.004328 (-0.001653) | 0.049652 / 0.004250 (0.045401) | 0.040588 / 0.037052 (0.003536) | 0.271701 / 0.258489 (0.013212) | 0.308384 / 0.293841 (0.014543) | 0.028713 / 0.128546 (-0.099833) | 0.010423 / 0.075646 (-0.065223) | 0.058099 / 0.419271 (-0.361172) | 0.032372 / 0.043533 (-0.011161) | 0.269395 / 0.255139 (0.014256) | 0.292252 / 0.283200 (0.009052) | 0.020038 / 0.141683 (-0.121645) | 1.124761 / 1.452155 (-0.327393) | 1.177609 / 1.492716 (-0.315107) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092187 / 0.018006 (0.074181) | 0.301936 / 0.000490 (0.301446) | 0.000230 / 0.000200 (0.000030) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022932 / 0.037411 (-0.014480) | 0.076552 / 0.014526 (0.062027) | 0.088729 / 0.176557 (-0.087827) | 0.127198 / 0.737135 (-0.609937) | 0.091902 / 0.296338 (-0.204436) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.299964 / 0.215209 (0.084755) | 2.929352 / 2.077655 (0.851697) | 1.598715 / 1.504120 (0.094595) | 1.462438 / 1.541195 (-0.078756) | 1.474308 / 1.468490 (0.005818) | 0.567120 / 4.584777 (-4.017657) | 2.481757 / 3.745712 (-1.263955) | 2.795375 / 5.269862 (-2.474487) | 1.740346 / 4.565676 (-2.825331) | 0.064048 / 0.424275 (-0.360227) | 0.004995 / 0.007607 (-0.002612) | 0.349084 / 0.226044 (0.123040) | 3.417679 / 2.268929 (1.148750) | 1.910615 / 55.444624 (-53.534009) | 1.694120 / 6.876477 (-5.182356) | 1.658654 / 2.142072 (-0.483419) | 0.638158 / 4.805227 (-4.167069) | 0.115509 / 6.500664 (-6.385156) | 0.040650 / 0.075469 (-0.034819) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.988081 / 1.841788 (-0.853707) | 12.210089 / 8.074308 (4.135781) | 11.090203 / 10.191392 (0.898811) | 0.131861 / 0.680424 (-0.548563) | 0.015461 / 0.534201 (-0.518740) | 0.287737 / 0.579283 (-0.291546) | 0.284170 / 0.434364 (-0.150194) | 0.324949 / 0.540337 (-0.215388) | 0.414912 / 1.386936 (-0.972024) |

![](https://cml.dev/watermark.png#cf71653947cecd84050daf0448dc5a73c2c071f3 "CML watermark")


`load_dataset(...,save_infos=True)` not working without loading script

Also, once the README.md exists in the python environment it is used when loading another dataset in the same format (e.g. json) since it always resolves the path to the same directory.
The consequence here is any other dataset won't load because of infos mismatch.
To reproduce this aspect:
1. Do a `load_datasets(...,save_infos=True)` with one dataset without a loading script
2. Try to load another dataset without a loading script in the same format (e.g. json) but with a different schema

### Describe the bug

It seems that saving a dataset infos back into the card file is not working for datasets without a loading script.

After tracking the problem a bit it looks like saving the infos uses `Builder.get_imported_module_dir()` as its destination directory.
Internally this is a call to `inspect.getfile()` but since the actual builder class used is dynamically created (cf. `datasets.load.configure_builder_class`) this method actually return te path to the parent builder class (e.g. `datasets.packaged_modules.json.JSON`).

### Steps to reproduce the bug

1. Have a local dataset without any loading script
2. Make sure there are no dataset infos in the README.md
3. Load with `save_infos=True`
4. No change in the dataset README.md
5. A new README.md file is created in the directory of the parent builder class (e.g. for json in `.../site-packages/datasets/packaged_modules/json/README.md`)

### Expected behavior

The dataset README.md should be updated and no file should be created in the python environment.

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-6.2.0-37-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.19.4
- PyArrow version: 14.0.1
- Pandas version: 2.1.3
- `fsspec` version: 2023.6.0


load_dataset imageflder for aws s3 path

New feature or request

### Feature request

I would like to load a dataset from S3 using the imagefolder option
something like
`dataset = datasets.load_dataset('imagefolder', data_dir='s3://.../lsun/train/bedroom', fs=S3FileSystem(), streaming=True)  `

### Motivation

no need of data_files

### Your contribution

no experience with this


429 Client Error

Transferring repos as this is a datasets issue

I'm getting a similar issue even though I've already downloaded the dataset 😅

```
huggingface_hub.utils._errors.HfHubHTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/api/datasets/HuggingFaceM4/WebSight
```

Hello, I was downloading the following dataset and after 20% of data was downloaded, I started getting error 429. It is not resolved since a few days. How should I resolve it?
Thanks

Dataset:
https://huggingface.co/datasets/cerebras/SlimPajama-627B

Error:
`requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/train/chunk1/example_train_3300.jsonl.zst`


Update builder hash with info

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6487). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Closing this one in favor of https://github.com/huggingface/datasets/pull/6458/commits/565c294fc12bc547730a023a610ed4f92313d8fb in https://github.com/huggingface/datasets/pull/6458

Currently if you change the `dataset_info` of a dataset (e.g. in the YAML part of the README.md), the cache ignores this change.

This is problematic because you want to regenerate a dataset if you change the features or the split sizes for example (e.g. after push_to_hub)

Ideally we should take the resolved files into account as well but this will be for another PR