Multi gpu docs

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6550). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks @lhoestq . This is a very important fix for code to run on multiple GPUs. Otherwise, only one GPU is working. I wish it can be merged soon.
I also wrote a [blog post](https://forrestbao.github.io/2024/01/30/datasets_map_with_rank_multiple_GPUs.html) with a complete  example in case it can be helpful to someone. Please feel free to use complete example in any documentation.

Thanks a lot @forrestbao ! I reused parts of your code for the documentation, I'm sure it will be useful to many people !

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005662 / 0.011353 (-0.005691) | 0.003930 / 0.011008 (-0.007078) | 0.063807 / 0.038508 (0.025299) | 0.030227 / 0.023109 (0.007118) | 0.235338 / 0.275898 (-0.040560) | 0.264433 / 0.323480 (-0.059047) | 0.004226 / 0.007986 (-0.003759) | 0.002847 / 0.004328 (-0.001481) | 0.048998 / 0.004250 (0.044747) | 0.042713 / 0.037052 (0.005660) | 0.250504 / 0.258489 (-0.007985) | 0.281101 / 0.293841 (-0.012740) | 0.029123 / 0.128546 (-0.099423) | 0.011388 / 0.075646 (-0.064258) | 0.211342 / 0.419271 (-0.207930) | 0.036437 / 0.043533 (-0.007096) | 0.238909 / 0.255139 (-0.016230) | 0.255853 / 0.283200 (-0.027347) | 0.018852 / 0.141683 (-0.122831) | 1.131870 / 1.452155 (-0.320284) | 1.209007 / 1.492716 (-0.283710) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092433 / 0.018006 (0.074427) | 0.303045 / 0.000490 (0.302556) | 0.000291 / 0.000200 (0.000091) | 0.000079 / 0.000054 (0.000025) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018349 / 0.037411 (-0.019062) | 0.062527 / 0.014526 (0.048002) | 0.075347 / 0.176557 (-0.101210) | 0.120587 / 0.737135 (-0.616549) | 0.075171 / 0.296338 (-0.221167) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.288364 / 0.215209 (0.073155) | 2.775779 / 2.077655 (0.698124) | 1.490875 / 1.504120 (-0.013245) | 1.375451 / 1.541195 (-0.165744) | 1.398923 / 1.468490 (-0.069567) | 0.588659 / 4.584777 (-3.996117) | 2.458114 / 3.745712 (-1.287598) | 2.928910 / 5.269862 (-2.340951) | 1.834221 / 4.565676 (-2.731456) | 0.064503 / 0.424275 (-0.359772) | 0.005028 / 0.007607 (-0.002580) | 0.340386 / 0.226044 (0.114341) | 3.408697 / 2.268929 (1.139769) | 1.843613 / 55.444624 (-53.601012) | 1.569300 / 6.876477 (-5.307177) | 1.636761 / 2.142072 (-0.505312) | 0.687854 / 4.805227 (-4.117374) | 0.123462 / 6.500664 (-6.377202) | 0.042877 / 0.075469 (-0.032593) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.984054 / 1.841788 (-0.857734) | 12.243934 / 8.074308 (4.169626) | 10.835244 / 10.191392 (0.643852) | 0.131609 / 0.680424 (-0.548815) | 0.014000 / 0.534201 (-0.520201) | 0.292070 / 0.579283 (-0.287213) | 0.271958 / 0.434364 (-0.162406) | 0.326866 / 0.540337 (-0.213471) | 0.440880 / 1.386936 (-0.946056) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005954 / 0.011353 (-0.005399) | 0.004123 / 0.011008 (-0.006885) | 0.050371 / 0.038508 (0.011863) | 0.034387 / 0.023109 (0.011277) | 0.273254 / 0.275898 (-0.002644) | 0.297785 / 0.323480 (-0.025695) | 0.004619 / 0.007986 (-0.003367) | 0.002884 / 0.004328 (-0.001444) | 0.050236 / 0.004250 (0.045986) | 0.048586 / 0.037052 (0.011533) | 0.283878 / 0.258489 (0.025389) | 0.315218 / 0.293841 (0.021377) | 0.060688 / 0.128546 (-0.067859) | 0.011991 / 0.075646 (-0.063655) | 0.059518 / 0.419271 (-0.359753) | 0.036113 / 0.043533 (-0.007420) | 0.274767 / 0.255139 (0.019628) | 0.290620 / 0.283200 (0.007420) | 0.020070 / 0.141683 (-0.121613) | 1.164635 / 1.452155 (-0.287519) | 1.189482 / 1.492716 (-0.303234) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095171 / 0.018006 (0.077165) | 0.307129 / 0.000490 (0.306639) | 0.000227 / 0.000200 (0.000027) | 0.000045 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022777 / 0.037411 (-0.014634) | 0.076761 / 0.014526 (0.062235) | 0.087654 / 0.176557 (-0.088902) | 0.126729 / 0.737135 (-0.610406) | 0.089491 / 0.296338 (-0.206847) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.292208 / 0.215209 (0.076999) | 2.890491 / 2.077655 (0.812836) | 1.625696 / 1.504120 (0.121576) | 1.463484 / 1.541195 (-0.077710) | 1.490889 / 1.468490 (0.022399) | 0.582155 / 4.584777 (-4.002622) | 2.492209 / 3.745712 (-1.253503) | 2.817020 / 5.269862 (-2.452842) | 1.806812 / 4.565676 (-2.758864) | 0.065830 / 0.424275 (-0.358445) | 0.005089 / 0.007607 (-0.002518) | 0.356067 / 0.226044 (0.130022) | 3.489652 / 2.268929 (1.220723) | 1.959276 / 55.444624 (-53.485348) | 1.678819 / 6.876477 (-5.197657) | 1.853581 / 2.142072 (-0.288491) | 0.660515 / 4.805227 (-4.144712) | 0.119884 / 6.500664 (-6.380780) | 0.041713 / 0.075469 (-0.033757) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.021701 / 1.841788 (-0.820087) | 12.918290 / 8.074308 (4.843982) | 11.469371 / 10.191392 (1.277979) | 0.144830 / 0.680424 (-0.535594) | 0.015858 / 0.534201 (-0.518343) | 0.290136 / 0.579283 (-0.289148) | 0.277894 / 0.434364 (-0.156470) | 0.330091 / 0.540337 (-0.210247) | 0.422697 / 1.386936 (-0.964240) |

![](https://cml.dev/watermark.png#13b36ee5c6d77f7eacbb4dd545a21e785db7fd3e "CML watermark")

after discussions in https://github.com/huggingface/datasets/pull/6415


Loading from hf hub with clearer error message

enhancement

New feature or request

Maybe we can add a helper message like `Maybe try again using "hf://path/without/resolve"` if the path contains `/resolve/` ?

e.g.

```
FileNotFoundError: Unable to find 'hf://datasets/HuggingFaceTB/eval_data/resolve/main/eval_data_context_and_answers.json'
It looks like you used parts of the URL of the file from the Hugging Face website, but you should remove the "/resolve/<revision>" part to have a valid `hf://` path.
Please try again using this path instead:
    hf://datasets/HuggingFaceTB/eval_data/eval_data_context_and_answers.json
```

and suggest `f"hf://datasets/HuggingFaceTB/eval_data@{revision}/eval_data_context_and_answers.json"` if revision != "main"

EDIT: I think this message should also be raised from the  `huggingface_hub`'s `HfFileSystem` implementation

### Feature request

Shouldn't this kinda work ?
```
Dataset.from_json("hf://datasets/HuggingFaceTB/eval_data/resolve/main/eval_data_context_and_answers.json")
```
I got an error
```
File ~/miniconda3/envs/datatrove/lib/python3.10/site-packages/datasets/data_files.py:380, in resolve_pattern(pattern, base_path, allowed_extensions, download_config)
    378     if allowed_extensions is not None:
    379         error_msg += f" with any supported extension {list(allowed_extensions)}"
--> 380     raise FileNotFoundError(error_msg)
    381 return out

FileNotFoundError: Unable to find 'hf://datasets/HuggingFaceTB/eval_data/resolve/main/eval_data_context_and_answers.json'
(I'm logged in)
```

Fix: the correct path is
```
hf://datasets/HuggingFaceTB/eval_data/eval_data_context_and_answers.json
```

Proposal: raise a clearer error

### Motivation

Clearer error message

### Your contribution

Can open a PR


Skip if a dataset has issues

It looks like a transient DNS issue. It should work fine now if you try again.

There is no parameter in load_dataset to skip failed downloads. In your case it would have skipped every single subsequent download until the DNS issue was resolved anyway.

### Describe the bug

Hello everyone,
I'm using **load_datasets** from **huggingface** to download the datasets and I'm facing an issue, the download starts but it reaches some state and then  fails with the following error:
Couldn't reach https://huggingface.co/datasets/wikimedia/wikipedia/resolve/4cb9b0d719291f1a10f96f67d609c5d442980dc9/20231101.ext/train-00000-of-00001.parquet

Failed to resolve \'huggingface.co\' ([Errno -3] Temporary failure in name resolution)"))')))

![image](https://github.com/huggingface/datasets/assets/143214684/8847d9cb-529e-4eda-9c76-282713dfa3af)

so I was wondering is there a parameter to be passed to load_dataset() to skip files that can't be downloaded??

### Steps to reproduce the bug

Parameter to be passed to load_dataset() of huggingface to skip files that can't be downloaded??

### Expected behavior

load_dataset() finishes without error

### Environment info

None


set dev version

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6547). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004855 / 0.011353 (-0.006498) | 0.003552 / 0.011008 (-0.007456) | 0.062328 / 0.038508 (0.023820) | 0.031142 / 0.023109 (0.008032) | 0.247726 / 0.275898 (-0.028172) | 0.270951 / 0.323480 (-0.052528) | 0.002887 / 0.007986 (-0.005099) | 0.002663 / 0.004328 (-0.001665) | 0.047888 / 0.004250 (0.043638) | 0.042932 / 0.037052 (0.005880) | 0.253660 / 0.258489 (-0.004829) | 0.274997 / 0.293841 (-0.018844) | 0.027200 / 0.128546 (-0.101347) | 0.010851 / 0.075646 (-0.064796) | 0.206566 / 0.419271 (-0.212706) | 0.035311 / 0.043533 (-0.008222) | 0.254146 / 0.255139 (-0.000993) | 0.269074 / 0.283200 (-0.014126) | 0.019221 / 0.141683 (-0.122462) | 1.101986 / 1.452155 (-0.350169) | 1.155541 / 1.492716 (-0.337175) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.004749 / 0.018006 (-0.013257) | 0.301627 / 0.000490 (0.301138) | 0.000208 / 0.000200 (0.000008) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018205 / 0.037411 (-0.019206) | 0.060420 / 0.014526 (0.045894) | 0.072533 / 0.176557 (-0.104023) | 0.119807 / 0.737135 (-0.617328) | 0.073249 / 0.296338 (-0.223089) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.284947 / 0.215209 (0.069738) | 2.796939 / 2.077655 (0.719285) | 1.486076 / 1.504120 (-0.018043) | 1.358247 / 1.541195 (-0.182948) | 1.383680 / 1.468490 (-0.084811) | 0.550253 / 4.584777 (-4.034524) | 2.364783 / 3.745712 (-1.380929) | 2.765631 / 5.269862 (-2.504230) | 1.695694 / 4.565676 (-2.869983) | 0.061519 / 0.424275 (-0.362756) | 0.004914 / 0.007607 (-0.002693) | 0.340370 / 0.226044 (0.114325) | 3.313175 / 2.268929 (1.044247) | 1.805421 / 55.444624 (-53.639203) | 1.532151 / 6.876477 (-5.344325) | 1.541195 / 2.142072 (-0.600878) | 0.625266 / 4.805227 (-4.179961) | 0.119980 / 6.500664 (-6.380684) | 0.042334 / 0.075469 (-0.033135) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.952893 / 1.841788 (-0.888895) | 11.322232 / 8.074308 (3.247924) | 9.982108 / 10.191392 (-0.209284) | 0.130034 / 0.680424 (-0.550389) | 0.013192 / 0.534201 (-0.521009) | 0.286041 / 0.579283 (-0.293243) | 0.269802 / 0.434364 (-0.164562) | 0.323582 / 0.540337 (-0.216755) | 0.428641 / 1.386936 (-0.958295) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005071 / 0.011353 (-0.006282) | 0.003368 / 0.011008 (-0.007640) | 0.049003 / 0.038508 (0.010495) | 0.029507 / 0.023109 (0.006398) | 0.271859 / 0.275898 (-0.004039) | 0.294660 / 0.323480 (-0.028820) | 0.004218 / 0.007986 (-0.003767) | 0.002686 / 0.004328 (-0.001642) | 0.047947 / 0.004250 (0.043696) | 0.044499 / 0.037052 (0.007447) | 0.273982 / 0.258489 (0.015493) | 0.303393 / 0.293841 (0.009552) | 0.029649 / 0.128546 (-0.098898) | 0.010555 / 0.075646 (-0.065091) | 0.057553 / 0.419271 (-0.361718) | 0.051686 / 0.043533 (0.008153) | 0.274079 / 0.255139 (0.018940) | 0.292535 / 0.283200 (0.009335) | 0.019211 / 0.141683 (-0.122472) | 1.130629 / 1.452155 (-0.321526) | 1.196791 / 1.492716 (-0.295925) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093617 / 0.018006 (0.075611) | 0.302698 / 0.000490 (0.302209) | 0.000222 / 0.000200 (0.000022) | 0.000046 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022830 / 0.037411 (-0.014581) | 0.077061 / 0.014526 (0.062535) | 0.089464 / 0.176557 (-0.087092) | 0.127487 / 0.737135 (-0.609649) | 0.092133 / 0.296338 (-0.204205) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295362 / 0.215209 (0.080153) | 2.902251 / 2.077655 (0.824596) | 1.600508 / 1.504120 (0.096388) | 1.477763 / 1.541195 (-0.063431) | 1.492242 / 1.468490 (0.023752) | 0.569347 / 4.584777 (-4.015430) | 2.449873 / 3.745712 (-1.295839) | 2.787207 / 5.269862 (-2.482655) | 1.723852 / 4.565676 (-2.841825) | 0.063076 / 0.424275 (-0.361199) | 0.005060 / 0.007607 (-0.002547) | 0.349614 / 0.226044 (0.123569) | 3.429735 / 2.268929 (1.160806) | 1.953883 / 55.444624 (-53.490741) | 1.664232 / 6.876477 (-5.212245) | 1.648864 / 2.142072 (-0.493209) | 0.640295 / 4.805227 (-4.164932) | 0.117053 / 6.500664 (-6.383611) | 0.041314 / 0.075469 (-0.034156) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.970663 / 1.841788 (-0.871125) | 12.144810 / 8.074308 (4.070502) | 10.938985 / 10.191392 (0.747593) | 0.140502 / 0.680424 (-0.539922) | 0.015522 / 0.534201 (-0.518679) | 0.286629 / 0.579283 (-0.292654) | 0.283695 / 0.434364 (-0.150669) | 0.327298 / 0.540337 (-0.213039) | 0.424635 / 1.386936 (-0.962301) |

![](https://cml.dev/watermark.png#e23a59ef7ba2b50d4e5588825c41212a3cfd1331 "CML watermark")


Release: 2.16.1

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6546). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005415 / 0.011353 (-0.005938) | 0.003733 / 0.011008 (-0.007275) | 0.064178 / 0.038508 (0.025670) | 0.033162 / 0.023109 (0.010053) | 0.249799 / 0.275898 (-0.026099) | 0.274875 / 0.323480 (-0.048605) | 0.002977 / 0.007986 (-0.005009) | 0.002696 / 0.004328 (-0.001633) | 0.050042 / 0.004250 (0.045792) | 0.047127 / 0.037052 (0.010074) | 0.250865 / 0.258489 (-0.007624) | 0.289758 / 0.293841 (-0.004083) | 0.028007 / 0.128546 (-0.100539) | 0.010671 / 0.075646 (-0.064975) | 0.207123 / 0.419271 (-0.212148) | 0.036403 / 0.043533 (-0.007130) | 0.261527 / 0.255139 (0.006388) | 0.277277 / 0.283200 (-0.005922) | 0.019418 / 0.141683 (-0.122264) | 1.118019 / 1.452155 (-0.334136) | 1.180254 / 1.492716 (-0.312462) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.004604 / 0.018006 (-0.013402) | 0.308129 / 0.000490 (0.307639) | 0.000202 / 0.000200 (0.000002) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018400 / 0.037411 (-0.019011) | 0.060777 / 0.014526 (0.046251) | 0.073059 / 0.176557 (-0.103498) | 0.119677 / 0.737135 (-0.617458) | 0.074076 / 0.296338 (-0.222263) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.275353 / 0.215209 (0.060144) | 2.694079 / 2.077655 (0.616424) | 1.419670 / 1.504120 (-0.084450) | 1.302079 / 1.541195 (-0.239116) | 1.342077 / 1.468490 (-0.126413) | 0.549794 / 4.584777 (-4.034983) | 2.377149 / 3.745712 (-1.368563) | 2.800362 / 5.269862 (-2.469500) | 1.728152 / 4.565676 (-2.837524) | 0.061774 / 0.424275 (-0.362501) | 0.004898 / 0.007607 (-0.002709) | 0.330996 / 0.226044 (0.104952) | 3.262010 / 2.268929 (0.993082) | 1.761106 / 55.444624 (-53.683518) | 1.489783 / 6.876477 (-5.386694) | 1.532470 / 2.142072 (-0.609602) | 0.648814 / 4.805227 (-4.156414) | 0.116893 / 6.500664 (-6.383771) | 0.042167 / 0.075469 (-0.033303) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.937679 / 1.841788 (-0.904109) | 11.621632 / 8.074308 (3.547324) | 10.226177 / 10.191392 (0.034785) | 0.129242 / 0.680424 (-0.551182) | 0.014884 / 0.534201 (-0.519317) | 0.287619 / 0.579283 (-0.291664) | 0.261677 / 0.434364 (-0.172687) | 0.336361 / 0.540337 (-0.203976) | 0.426461 / 1.386936 (-0.960475) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005246 / 0.011353 (-0.006106) | 0.003533 / 0.011008 (-0.007475) | 0.051691 / 0.038508 (0.013182) | 0.031551 / 0.023109 (0.008442) | 0.297884 / 0.275898 (0.021986) | 0.323100 / 0.323480 (-0.000380) | 0.004101 / 0.007986 (-0.003884) | 0.002668 / 0.004328 (-0.001661) | 0.048764 / 0.004250 (0.044513) | 0.045429 / 0.037052 (0.008377) | 0.300107 / 0.258489 (0.041618) | 0.335650 / 0.293841 (0.041809) | 0.030061 / 0.128546 (-0.098485) | 0.010878 / 0.075646 (-0.064768) | 0.058561 / 0.419271 (-0.360710) | 0.052829 / 0.043533 (0.009296) | 0.302704 / 0.255139 (0.047565) | 0.320527 / 0.283200 (0.037327) | 0.018995 / 0.141683 (-0.122688) | 1.144050 / 1.452155 (-0.308105) | 1.255275 / 1.492716 (-0.237441) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092708 / 0.018006 (0.074701) | 0.305204 / 0.000490 (0.304714) | 0.000224 / 0.000200 (0.000024) | 0.000055 / 0.000054 (0.000000) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021607 / 0.037411 (-0.015805) | 0.075938 / 0.014526 (0.061412) | 0.090864 / 0.176557 (-0.085693) | 0.128248 / 0.737135 (-0.608887) | 0.090322 / 0.296338 (-0.206017) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.302095 / 0.215209 (0.086886) | 2.925686 / 2.077655 (0.848032) | 1.617767 / 1.504120 (0.113648) | 1.477975 / 1.541195 (-0.063220) | 1.508576 / 1.468490 (0.040086) | 0.574376 / 4.584777 (-4.010401) | 2.467483 / 3.745712 (-1.278229) | 2.832500 / 5.269862 (-2.437362) | 1.765233 / 4.565676 (-2.800443) | 0.064105 / 0.424275 (-0.360170) | 0.005090 / 0.007607 (-0.002517) | 0.349819 / 0.226044 (0.123774) | 3.468916 / 2.268929 (1.199987) | 1.946499 / 55.444624 (-53.498126) | 1.684369 / 6.876477 (-5.192107) | 1.711036 / 2.142072 (-0.431036) | 0.650153 / 4.805227 (-4.155075) | 0.116598 / 6.500664 (-6.384066) | 0.041213 / 0.075469 (-0.034256) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.990842 / 1.841788 (-0.850946) | 12.348468 / 8.074308 (4.274160) | 11.174441 / 10.191392 (0.983049) | 0.140950 / 0.680424 (-0.539473) | 0.016100 / 0.534201 (-0.518101) | 0.286486 / 0.579283 (-0.292797) | 0.282054 / 0.434364 (-0.152310) | 0.324261 / 0.540337 (-0.216076) | 0.420717 / 1.386936 (-0.966219) |

![](https://cml.dev/watermark.png#7b2bcd76457de720454c3ac304f2ed5c6f40acaa "CML watermark")


`image` column not automatically inferred if image dataset only contains 1 image

### Describe the bug

By default, the standard Image Dataset maps out `file_name` to `image` when loading an Image Dataset.

However, if the dataset contains only 1 image, this does not take place

### Steps to reproduce the bug

Input
(dataset with one image `multimodalart/repro_1_image`)
```py
from datasets import load_dataset
dataset = load_dataset("multimodalart/repro_1_image")
dataset
```

Output:
```py
DatasetDict({
    train: Dataset({
        features: ['file_name', 'prompt'],
        num_rows: 1
    })
})
```

Input
(dataset with 2+ images `multimodalart/repro_2_image`)
```py
from datasets import load_dataset
dataset = load_dataset("multimodalart/repro_2_image")
dataset
```

Output:
```py
DatasetDict({
    train: Dataset({
        features: ['image', 'prompt'],
        num_rows: 2
    })
})
```

### Expected behavior

Expected to map `file_name`  → `image` for all dataset sizes, including 1.

### Environment info

Both latest main and 2.16.0


Fix custom configs from script

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6544). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005462 / 0.011353 (-0.005891) | 0.003918 / 0.011008 (-0.007090) | 0.065021 / 0.038508 (0.026513) | 0.032620 / 0.023109 (0.009511) | 0.249794 / 0.275898 (-0.026104) | 0.277330 / 0.323480 (-0.046150) | 0.002962 / 0.007986 (-0.005023) | 0.003435 / 0.004328 (-0.000894) | 0.048992 / 0.004250 (0.044742) | 0.046841 / 0.037052 (0.009788) | 0.252459 / 0.258489 (-0.006030) | 0.287889 / 0.293841 (-0.005952) | 0.028322 / 0.128546 (-0.100224) | 0.011214 / 0.075646 (-0.064432) | 0.208555 / 0.419271 (-0.210717) | 0.037004 / 0.043533 (-0.006529) | 0.262537 / 0.255139 (0.007398) | 0.307418 / 0.283200 (0.024218) | 0.021552 / 0.141683 (-0.120131) | 1.144252 / 1.452155 (-0.307903) | 1.195687 / 1.492716 (-0.297029) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.004766 / 0.018006 (-0.013240) | 0.301926 / 0.000490 (0.301436) | 0.000218 / 0.000200 (0.000018) | 0.000042 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.017891 / 0.037411 (-0.019521) | 0.066848 / 0.014526 (0.052322) | 0.075522 / 0.176557 (-0.101035) | 0.120762 / 0.737135 (-0.616374) | 0.075980 / 0.296338 (-0.220359) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.284843 / 0.215209 (0.069634) | 2.816260 / 2.077655 (0.738605) | 1.484370 / 1.504120 (-0.019750) | 1.362090 / 1.541195 (-0.179104) | 1.421729 / 1.468490 (-0.046762) | 0.561673 / 4.584777 (-4.023104) | 2.370793 / 3.745712 (-1.374919) | 2.982639 / 5.269862 (-2.287223) | 1.834614 / 4.565676 (-2.731063) | 0.063158 / 0.424275 (-0.361117) | 0.005044 / 0.007607 (-0.002563) | 0.339834 / 0.226044 (0.113790) | 3.369051 / 2.268929 (1.100122) | 1.821040 / 55.444624 (-53.623584) | 1.544009 / 6.876477 (-5.332468) | 1.603902 / 2.142072 (-0.538171) | 0.638151 / 4.805227 (-4.167076) | 0.117012 / 6.500664 (-6.383652) | 0.042999 / 0.075469 (-0.032470) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.941809 / 1.841788 (-0.899978) | 12.279635 / 8.074308 (4.205326) | 10.212876 / 10.191392 (0.021484) | 0.129904 / 0.680424 (-0.550519) | 0.014210 / 0.534201 (-0.519991) | 0.286140 / 0.579283 (-0.293143) | 0.267453 / 0.434364 (-0.166911) | 0.324417 / 0.540337 (-0.215921) | 0.428262 / 1.386936 (-0.958674) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005351 / 0.011353 (-0.006002) | 0.003591 / 0.011008 (-0.007417) | 0.048755 / 0.038508 (0.010247) | 0.030857 / 0.023109 (0.007748) | 0.270301 / 0.275898 (-0.005597) | 0.294459 / 0.323480 (-0.029021) | 0.004265 / 0.007986 (-0.003720) | 0.002712 / 0.004328 (-0.001616) | 0.047725 / 0.004250 (0.043475) | 0.048392 / 0.037052 (0.011339) | 0.274226 / 0.258489 (0.015737) | 0.304010 / 0.293841 (0.010169) | 0.029283 / 0.128546 (-0.099263) | 0.011196 / 0.075646 (-0.064450) | 0.057213 / 0.419271 (-0.362058) | 0.057504 / 0.043533 (0.013971) | 0.266091 / 0.255139 (0.010952) | 0.285991 / 0.283200 (0.002791) | 0.020030 / 0.141683 (-0.121653) | 1.121514 / 1.452155 (-0.330641) | 1.192608 / 1.492716 (-0.300108) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095041 / 0.018006 (0.077035) | 0.301255 / 0.000490 (0.300765) | 0.000218 / 0.000200 (0.000018) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022265 / 0.037411 (-0.015146) | 0.078416 / 0.014526 (0.063890) | 0.091097 / 0.176557 (-0.085460) | 0.129864 / 0.737135 (-0.607272) | 0.091683 / 0.296338 (-0.204655) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.294104 / 0.215209 (0.078895) | 2.886809 / 2.077655 (0.809154) | 1.601931 / 1.504120 (0.097811) | 1.469353 / 1.541195 (-0.071842) | 1.525132 / 1.468490 (0.056642) | 0.565164 / 4.584777 (-4.019613) | 2.432873 / 3.745712 (-1.312839) | 2.885849 / 5.269862 (-2.384013) | 1.780474 / 4.565676 (-2.785203) | 0.064358 / 0.424275 (-0.359917) | 0.005186 / 0.007607 (-0.002421) | 0.349374 / 0.226044 (0.123329) | 3.424751 / 2.268929 (1.155823) | 1.956874 / 55.444624 (-53.487750) | 1.679002 / 6.876477 (-5.197475) | 1.718821 / 2.142072 (-0.423252) | 0.656974 / 4.805227 (-4.148254) | 0.120645 / 6.500664 (-6.380019) | 0.042355 / 0.075469 (-0.033114) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.001923 / 1.841788 (-0.839864) | 13.208127 / 8.074308 (5.133819) | 11.164863 / 10.191392 (0.973471) | 0.131964 / 0.680424 (-0.548460) | 0.015344 / 0.534201 (-0.518857) | 0.287961 / 0.579283 (-0.291322) | 0.273986 / 0.434364 (-0.160378) | 0.327280 / 0.540337 (-0.213058) | 0.426761 / 1.386936 (-0.960175) |

![](https://cml.dev/watermark.png#ebb913eca80807521239efece1ff305625cb89b4 "CML watermark")

Thanks for the fix and the patch release. This confirms that, as I suggested in the Summer, maybe we should avoid making a release right before leaving on holidays.

We should not use the parquet export when the user is passing config_kwargs

I also fixed a regression that would disallow creating a custom config when a dataset has multiple predefined configs

fix https://github.com/huggingface/datasets/issues/6533


Fix dl_manager.extract returning FileNotFoundError

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6543). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004950 / 0.011353 (-0.006403) | 0.003502 / 0.011008 (-0.007506) | 0.062517 / 0.038508 (0.024009) | 0.030965 / 0.023109 (0.007856) | 0.250661 / 0.275898 (-0.025237) | 0.279165 / 0.323480 (-0.044314) | 0.002960 / 0.007986 (-0.005026) | 0.003382 / 0.004328 (-0.000946) | 0.048174 / 0.004250 (0.043923) | 0.042975 / 0.037052 (0.005922) | 0.248079 / 0.258489 (-0.010410) | 0.283770 / 0.293841 (-0.010070) | 0.027935 / 0.128546 (-0.100611) | 0.010634 / 0.075646 (-0.065012) | 0.207039 / 0.419271 (-0.212233) | 0.035863 / 0.043533 (-0.007670) | 0.257426 / 0.255139 (0.002287) | 0.274222 / 0.283200 (-0.008978) | 0.017590 / 0.141683 (-0.124093) | 1.126889 / 1.452155 (-0.325266) | 1.160795 / 1.492716 (-0.331921) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.004298 / 0.018006 (-0.013708) | 0.301366 / 0.000490 (0.300876) | 0.000202 / 0.000200 (0.000002) | 0.000050 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018159 / 0.037411 (-0.019252) | 0.060566 / 0.014526 (0.046041) | 0.072500 / 0.176557 (-0.104057) | 0.119612 / 0.737135 (-0.617523) | 0.074467 / 0.296338 (-0.221871) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.281859 / 0.215209 (0.066650) | 2.760157 / 2.077655 (0.682502) | 1.450632 / 1.504120 (-0.053487) | 1.326636 / 1.541195 (-0.214559) | 1.363381 / 1.468490 (-0.105109) | 0.576199 / 4.584777 (-4.008578) | 2.355776 / 3.745712 (-1.389936) | 2.807308 / 5.269862 (-2.462553) | 1.745449 / 4.565676 (-2.820228) | 0.063413 / 0.424275 (-0.360862) | 0.004978 / 0.007607 (-0.002630) | 0.332738 / 0.226044 (0.106693) | 3.267677 / 2.268929 (0.998748) | 1.766074 / 55.444624 (-53.678551) | 1.500853 / 6.876477 (-5.375624) | 1.532434 / 2.142072 (-0.609639) | 0.648238 / 4.805227 (-4.156989) | 0.116030 / 6.500664 (-6.384634) | 0.042018 / 0.075469 (-0.033451) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.934325 / 1.841788 (-0.907463) | 11.439765 / 8.074308 (3.365457) | 9.958624 / 10.191392 (-0.232768) | 0.130295 / 0.680424 (-0.550129) | 0.014437 / 0.534201 (-0.519764) | 0.286073 / 0.579283 (-0.293210) | 0.262430 / 0.434364 (-0.171934) | 0.323905 / 0.540337 (-0.216432) | 0.416615 / 1.386936 (-0.970321) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005149 / 0.011353 (-0.006204) | 0.003517 / 0.011008 (-0.007491) | 0.048658 / 0.038508 (0.010150) | 0.029638 / 0.023109 (0.006529) | 0.271002 / 0.275898 (-0.004896) | 0.324910 / 0.323480 (0.001430) | 0.004086 / 0.007986 (-0.003900) | 0.002609 / 0.004328 (-0.001719) | 0.047806 / 0.004250 (0.043556) | 0.045422 / 0.037052 (0.008369) | 0.274317 / 0.258489 (0.015828) | 0.304544 / 0.293841 (0.010703) | 0.029318 / 0.128546 (-0.099229) | 0.010626 / 0.075646 (-0.065020) | 0.057838 / 0.419271 (-0.361434) | 0.052408 / 0.043533 (0.008875) | 0.267736 / 0.255139 (0.012597) | 0.292024 / 0.283200 (0.008824) | 0.019244 / 0.141683 (-0.122439) | 1.167728 / 1.452155 (-0.284427) | 1.226364 / 1.492716 (-0.266352) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092441 / 0.018006 (0.074435) | 0.310316 / 0.000490 (0.309827) | 0.000218 / 0.000200 (0.000018) | 0.000049 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021818 / 0.037411 (-0.015594) | 0.076515 / 0.014526 (0.061989) | 0.089179 / 0.176557 (-0.087377) | 0.127034 / 0.737135 (-0.610102) | 0.089646 / 0.296338 (-0.206692) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.292056 / 0.215209 (0.076847) | 2.841410 / 2.077655 (0.763756) | 1.550626 / 1.504120 (0.046506) | 1.426204 / 1.541195 (-0.114990) | 1.445838 / 1.468490 (-0.022652) | 0.555777 / 4.584777 (-4.029000) | 2.441077 / 3.745712 (-1.304635) | 2.773445 / 5.269862 (-2.496416) | 1.728951 / 4.565676 (-2.836726) | 0.062579 / 0.424275 (-0.361697) | 0.005063 / 0.007607 (-0.002544) | 0.350749 / 0.226044 (0.124705) | 3.461702 / 2.268929 (1.192773) | 1.892506 / 55.444624 (-53.552118) | 1.625958 / 6.876477 (-5.250519) | 1.649175 / 2.142072 (-0.492898) | 0.636123 / 4.805227 (-4.169105) | 0.116548 / 6.500664 (-6.384116) | 0.041174 / 0.075469 (-0.034295) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.973468 / 1.841788 (-0.868320) | 12.104761 / 8.074308 (4.030453) | 11.131691 / 10.191392 (0.940299) | 0.132309 / 0.680424 (-0.548115) | 0.016191 / 0.534201 (-0.518010) | 0.284748 / 0.579283 (-0.294535) | 0.282661 / 0.434364 (-0.151703) | 0.323797 / 0.540337 (-0.216540) | 0.417767 / 1.386936 (-0.969169) |

![](https://cml.dev/watermark.png#72b440325f6a84d341ea57539d8c368a001e2e75 "CML watermark")

The dl_manager base path is remote (e.g. a hf:// path), so local cached paths should be passed as absolute paths.
This could happen if users provide a relative path as `cache_dir`

fix https://github.com/huggingface/datasets/issues/6536


Datasets : wikipedia 20220301.en error

Hi ! We now recommend using the `wikimedia/wikipedia` dataset, can you try loading this one instead ?

```python
wiki_dataset  = load_dataset("wikimedia/wikipedia", "20231101.en")
```

This bug has been fixed in `2.16.1` thanks to https://github.com/huggingface/datasets/pull/6544, feel free to update `datasets` and re-run your code :)

```
pip install -U datasets
```

### Describe the bug

When I used load_dataset to download this data set, the following error occurred. The main problem was that the target data did not exist.

### Steps to reproduce the bug

1.I tried downloading directly.
```python
wiki_dataset = load_dataset("wikipedia", "20220301.en")
```
An exception occurred
```
MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory).
Example of usage:
	`load_dataset('wikipedia', '20220301.en', beam_runner='DirectRunner')`
```
2.I modified the code as prompted.
```python
wiki_dataset = load_dataset('wikipedia', '20220301.en', beam_runner='DirectRunner')
```
An exception occurred:
```
FileNotFoundError: Couldn't find file at https://dumps.wikimedia.org/enwiki/20220301/dumpstatus.json
```

### Expected behavior

I searched in the parent directory of the corresponding URL, but there was no corresponding "20220301" directory.
I really need this data set and hope to provide a download method.

### Environment info

python 3.8
datasets 2.16.0
apache-beam 2.52.0
dill 0.3.7


Dataset not loading successfully.

This is a problem with your environment. You should be able to fix it by upgrading `numpy` based on [this](https://github.com/numpy/numpy/issues/23570) issue.

Then, this shouldn't throw an error on your machine:
```python
import numpy
numpy._no_nep50_warning
```

If it does, run `python -m pip install numpy` to ensure the correct `pip` is used for the package installation.

Your suggestion to run `python -m pip install numpy` proved to be successful, and my issue has been resolved. I am grateful for your assistance, @mariosasko

### Describe the bug

When I run down the below code shows this error: AttributeError: module 'numpy' has no attribute '_no_nep50_warning'
I also added this issue in transformers library please check out: [link](https://github.com/huggingface/transformers/issues/28099)

### Steps to reproduce the bug

## Reproduction
Hi, please check this line of code, when I run Show attribute error.

```
from datasets import load_dataset
from transformers import WhisperProcessor, WhisperForConditionalGeneration

# Select an audio file and read it:
ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
audio_sample = ds[0]["audio"]
waveform = audio_sample["array"]
sampling_rate = audio_sample["sampling_rate"]

# Load the Whisper model in Hugging Face format:
processor = WhisperProcessor.from_pretrained("openai/whisper-tiny.en")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny.en")

# Use the model and processor to transcribe the audio:
input_features = processor(
    waveform, sampling_rate=sampling_rate, return_tensors="pt"
).input_features

# Generate token ids
predicted_ids = model.generate(input_features)

# Decode token ids to text
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)

transcription[0]

```

**Attribute Error**
```
AttributeError                            Traceback (most recent call last)
Cell In[9], line 6
      4 # Select an audio file and read it:
      5 ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
----> 6 audio_sample = ds[0]["audio"]
      7 waveform = audio_sample["array"]
      8 sampling_rate = audio_sample["sampling_rate"]

File /opt/pytorch/lib/python3.8/site-packages/datasets/arrow_dataset.py:2795, in Dataset.__getitem__(self, key)
   2793 def __getitem__(self, key):  # noqa: F811
   2794     """Can be used to index columns (by string names) or rows (by integer index or iterable of indices or bools)."""
-> 2795     return self._getitem(key)

File /opt/pytorch/lib/python3.8/site-packages/datasets/arrow_dataset.py:2780, in Dataset._getitem(self, key, **kwargs)
   2778 formatter = get_formatter(format_type, features=self._info.features, **format_kwargs)
   2779 pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
-> 2780 formatted_output = format_table(
   2781     pa_subtable, key, formatter=formatter, format_columns=format_columns, output_all_columns=output_all_columns
   2782 )
   2783 return formatted_output

File /opt/pytorch/lib/python3.8/site-packages/datasets/formatting/formatting.py:629, in format_table(table, key, formatter, format_columns, output_all_columns)
    627 python_formatter = PythonFormatter(features=formatter.features)
    628 if format_columns is None:
--> 629     return formatter(pa_table, query_type=query_type)
    630 elif query_type == "column":
    631     if key in format_columns:

File /opt/pytorch/lib/python3.8/site-packages/datasets/formatting/formatting.py:396, in Formatter.__call__(self, pa_table, query_type)
    394 def __call__(self, pa_table: pa.Table, query_type: str) -> Union[RowFormat, ColumnFormat, BatchFormat]:
    395     if query_type == "row":
--> 396         return self.format_row(pa_table)
    397     elif query_type == "column":
    398         return self.format_column(pa_table)

File /opt/pytorch/lib/python3.8/site-packages/datasets/formatting/formatting.py:437, in PythonFormatter.format_row(self, pa_table)
    435     return LazyRow(pa_table, self)
    436 row = self.python_arrow_extractor().extract_row(pa_table)
--> 437 row = self.python_features_decoder.decode_row(row)
    438 return row

File /opt/pytorch/lib/python3.8/site-packages/datasets/formatting/formatting.py:215, in PythonFeaturesDecoder.decode_row(self, row)
    214 def decode_row(self, row: dict) -> dict:
--> 215     return self.features.decode_example(row) if self.features else row

File /opt/pytorch/lib/python3.8/site-packages/datasets/features/features.py:1917, in Features.decode_example(self, example, token_per_repo_id)
   1903 def decode_example(self, example: dict, token_per_repo_id: Optional[Dict[str, Union[str, bool, None]]] = None):
   1904     """Decode example with custom feature decoding.
   1905
   1906     Args:
   (...)
   1914         `dict[str, Any]`
   1915     """
-> 1917     return {
   1918         column_name: decode_nested_example(feature, value, token_per_repo_id=token_per_repo_id)
   1919         if self._column_requires_decoding[column_name]
   1920         else value
   1921         for column_name, (feature, value) in zip_dict(
   1922             {key: value for key, value in self.items() if key in example}, example
   1923         )
   1924     }

File /opt/pytorch/lib/python3.8/site-packages/datasets/features/features.py:1918, in <dictcomp>(.0)
   1903 def decode_example(self, example: dict, token_per_repo_id: Optional[Dict[str, Union[str, bool, None]]] = None):
   1904     """Decode example with custom feature decoding.
   1905
   1906     Args:
   (...)
   1914         `dict[str, Any]`
   1915     """
   1917     return {
-> 1918         column_name: decode_nested_example(feature, value, token_per_repo_id=token_per_repo_id)
   1919         if self._column_requires_decoding[column_name]
   1920         else value
   1921         for column_name, (feature, value) in zip_dict(
   1922             {key: value for key, value in self.items() if key in example}, example
   1923         )
   1924     }

File /opt/pytorch/lib/python3.8/site-packages/datasets/features/features.py:1339, in decode_nested_example(schema, obj, token_per_repo_id)
   1336 elif isinstance(schema, (Audio, Image)):
   1337     # we pass the token to read and decode files from private repositories in streaming mode
   1338     if obj is not None and schema.decode:
-> 1339         return schema.decode_example(obj, token_per_repo_id=token_per_repo_id)
   1340 return obj

File /opt/pytorch/lib/python3.8/site-packages/datasets/features/audio.py:191, in Audio.decode_example(self, value, token_per_repo_id)
    189 array = array.T
    190 if self.mono:
--> 191     array = librosa.to_mono(array)
    192 if self.sampling_rate and self.sampling_rate != sampling_rate:
    193     array = librosa.resample(array, orig_sr=sampling_rate, target_sr=self.sampling_rate)

File /opt/pytorch/lib/python3.8/site-packages/lazy_loader/__init__.py:78, in attach.<locals>.__getattr__(name)
     76 submod_path = f"{package_name}.{attr_to_modules[name]}"
     77 submod = importlib.import_module(submod_path)
---> 78 attr = getattr(submod, name)
     80 # If the attribute lives in a file (module) with the same
     81 # name as the attribute, ensure that the attribute and *not*
     82 # the module is accessible on the package.
     83 if name == attr_to_modules[name]:

File /opt/pytorch/lib/python3.8/site-packages/lazy_loader/__init__.py:77, in attach.<locals>.__getattr__(name)
     75 elif name in attr_to_modules:
     76     submod_path = f"{package_name}.{attr_to_modules[name]}"
---> 77     submod = importlib.import_module(submod_path)
     78     attr = getattr(submod, name)
     80     # If the attribute lives in a file (module) with the same
     81     # name as the attribute, ensure that the attribute and *not*
     82     # the module is accessible on the package.

File /usr/lib/python3.8/importlib/__init__.py:127, in import_module(name, package)
    125             break
    126         level += 1
--> 127 return _bootstrap._gcd_import(name[level:], package, level)

File <frozen importlib._bootstrap>:1014, in _gcd_import(name, package, level)

File <frozen importlib._bootstrap>:991, in _find_and_load(name, import_)

File <frozen importlib._bootstrap>:975, in _find_and_load_unlocked(name, import_)

File <frozen importlib._bootstrap>:671, in _load_unlocked(spec)

File <frozen importlib._bootstrap_external>:848, in exec_module(self, module)

File <frozen importlib._bootstrap>:219, in _call_with_frames_removed(f, *args, **kwds)

File /opt/pytorch/lib/python3.8/site-packages/librosa/core/audio.py:13
     11 import audioread
     12 import numpy as np
---> 13 import scipy.signal
     14 import soxr
     15 import lazy_loader as lazy

File /opt/pytorch/lib/python3.8/site-packages/scipy/signal/__init__.py:323
    314 from ._spline import (  # noqa: F401
    315     cspline2d,
    316     qspline2d,
   (...)
    319     symiirorder2,
    320 )
    322 from ._bsplines import *
--> 323 from ._filter_design import *
    324 from ._fir_filter_design import *
    325 from ._ltisys import *

File /opt/pytorch/lib/python3.8/site-packages/scipy/signal/_filter_design.py:16
     13 from numpy.polynomial.polynomial import polyval as npp_polyval
     14 from numpy.polynomial.polynomial import polyvalfromroots
---> 16 from scipy import special, optimize, fft as sp_fft
     17 from scipy.special import comb
     18 from scipy._lib._util import float_factorial

File /opt/pytorch/lib/python3.8/site-packages/scipy/optimize/__init__.py:405
      1 """
      2 =====================================================
      3 Optimization and root finding (:mod:`scipy.optimize`)
   (...)
    401
    402 """
    404 from ._optimize import *
--> 405 from ._minimize import *
    406 from ._root import *
    407 from ._root_scalar import *

File /opt/pytorch/lib/python3.8/site-packages/scipy/optimize/_minimize.py:26
     24 from ._trustregion_krylov import _minimize_trust_krylov
     25 from ._trustregion_exact import _minimize_trustregion_exact
---> 26 from ._trustregion_constr import _minimize_trustregion_constr
     28 # constrained minimization
     29 from ._lbfgsb_py import _minimize_lbfgsb

File /opt/pytorch/lib/python3.8/site-packages/scipy/optimize/_trustregion_constr/__init__.py:4
      1 """This module contains the equality constrained SQP solver."""
----> 4 from .minimize_trustregion_constr import _minimize_trustregion_constr
      6 __all__ = ['_minimize_trustregion_constr']

File /opt/pytorch/lib/python3.8/site-packages/scipy/optimize/_trustregion_constr/minimize_trustregion_constr.py:5
      3 from scipy.sparse.linalg import LinearOperator
      4 from .._differentiable_functions import VectorFunction
----> 5 from .._constraints import (
      6     NonlinearConstraint, LinearConstraint, PreparedConstraint, strict_bounds)
      7 from .._hessian_update_strategy import BFGS
      8 from .._optimize import OptimizeResult

File /opt/pytorch/lib/python3.8/site-packages/scipy/optimize/_constraints.py:8
      6 from ._optimize import OptimizeWarning
      7 from warnings import warn, catch_warnings, simplefilter
----> 8 from numpy.testing import suppress_warnings
      9 from scipy.sparse import issparse
     12 def _arr_to_scalar(x):
     13     # If x is a numpy array, return x.item().  This will
     14     # fail if the array has more than one element.

File /opt/pytorch/lib/python3.8/site-packages/numpy/testing/__init__.py:11
      8 from unittest import TestCase
     10 from . import _private
---> 11 from ._private.utils import *
     12 from ._private.utils import (_assert_valid_refcount, _gen_alignment_data)
     13 from ._private import extbuild, decorators as dec

File /opt/pytorch/lib/python3.8/site-packages/numpy/testing/_private/utils.py:480
    476         pprint.pprint(desired, msg)
    477         raise AssertionError(msg.getvalue())
--> 480 @np._no_nep50_warning()
    481 def assert_almost_equal(actual,desired,decimal=7,err_msg='',verbose=True):
    482     """
    483     Raises an AssertionError if two items are not equal up to desired
    484     precision.
   (...)
    548
    549     """
    550     __tracebackhide__ = True  # Hide traceback for py.test

File /opt/pytorch/lib/python3.8/site-packages/numpy/__init__.py:313, in __getattr__(attr)
    305     raise AttributeError(__former_attrs__[attr])
    307 # Importing Tester requires importing all of UnitTest which is not a
    308 # cheap import Since it is mainly used in test suits, we lazy import it
    309 # here to save on the order of 10 ms of import time for most users
    310 #
    311 # The previous way Tester was imported also had a side effect of adding
    312 # the full `numpy.testing` namespace
--> 313 if attr == 'testing':
    314     import numpy.testing as testing
    315     return testing

AttributeError: module 'numpy' has no attribute '_no_nep50_warning'

```

### Expected behavior

``` ' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.' ```

Also, make sure this script is provided for your official website so please update:
[script](https://huggingface.co/docs/transformers/model_doc/whisper)

### Environment info

**System Info**
* transformers -> 4.36.1
* datasets -> 2.15.0
* huggingface_hub -> 0.19.4
* python -> 3.8.10
* accelerate -> 0.25.0
* pytorch -> 2.0.1+cpu
* Using GPU in Script -> No


Extreme inefficiency for `save_to_disk` when merging datasets

Concatenating datasets doesn't create any indices mapping - so flattening indices is not needed (unless you shuffle the dataset).
Can you share the snippet of code you are using to merge your datasets and save them to disk ?

### Describe the bug

Hi, I tried to merge in total 22M sequences of data, where each sequence is of maximum length 2000. I found that merging these datasets and then `save_to_disk` is extremely slow because of flattening the indices. Wondering if you have any suggestions or guidance on this. Thank you very much!

### Steps to reproduce the bug

The source data is too big to demonstrate

### Expected behavior

The source data is too big to demonstrate

### Environment info

python                     3.9.0
datasets                  2.7.0
pytorch                   2.0.0
tokenizers                0.13.1
transformers              4.31.0


'Repo card metadata block was not found' when loading a pragmeval dataset

### Describe the bug

I can't load dataset subsets of 'pragmeval'.

The funny thing is I ran the dataset author's [colab notebook](https://colab.research.google.com/drive/1sg--LF4z7XR1wxAOfp0-3d4J6kQ9nj_A?usp=sharing) and it works just fine. I tried to install exactly the same packages that are installed on colab using poetry, so my environment info only differs from the one from colab in linux version - I still get the same bug outside colab.

### Steps to reproduce the bug

Install dependencies with poetry

pyproject.toml
```
[tool.poetry]
name = "project"
version = "0.1.0"
description = ""
authors = []

[tool.poetry.dependencies]
python = "^3.10"
datasets = "2.16.0"
pandas = "1.5.3"
pyarrow = "10.0.1"
huggingface-hub = "0.19.4"
fsspec = "2023.6.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
```

`poetry run python -c "import datasets; print(datasets.get_dataset_config_names('pragmeval'))`

prints ['default']

### Expected behavior

The command should print

```
['emergent',
 'emobank-arousal',
 'emobank-dominance',
 'emobank-valence',
 'gum',
 'mrda',
 'pdtb',
 'persuasiveness-claimtype',
 'persuasiveness-eloquence',
 'persuasiveness-premisetype',
 'persuasiveness-relevance',
 'persuasiveness-specificity',
 'persuasiveness-strength',
 'sarcasm',
 'squinky-formality',
 'squinky-implicature',
 'squinky-informativeness',
 'stac',
 'switchboard',
 'verifiability']
```

### Environment info

- `datasets` version: 2.16.0
- Platform: Linux-6.2.0-37-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.19.4
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0


ImportError: cannot import name 'SchemaInferenceError' from 'datasets.arrow_writer' (/opt/conda/lib/python3.10/site-packages/datasets/arrow_writer.py)

> Hi ! Are you sure you have `datasets` 2.16 ? I just checked and on 2.16 I can run `from datasets.arrow_writer import SchemaInferenceError` without error

Yes, I am sure

```
!pip show datasets
Name: datasets
Version: 2.16.1
Summary: HuggingFace community-driven open-source library of datasets
Home-page: https://github.com/huggingface/datasets
Author: HuggingFace Inc.
Author-email: thomas@huggingface.co
License: Apache 2.0
Location: /opt/conda/lib/python3.10/site-packages
Requires: aiohttp, dill, filelock, fsspec, huggingface-hub, multiprocess, numpy, packaging, pandas, pyarrow, pyarrow-hotfix, pyyaml, requests, tqdm, xxhash
Required-by: trl
```

> I have the same issue - using with datasets version 2.16.1. Also this is on a kaggle notebook - other people with the same issue also seem to be having it on kaggle?

Don't know about other people. But I am having this issue whose solution I can't find anywhere. And this issue still persists.

> I have the same issue now and didn't have this problem around 2 weeks ago.

Same here

> I was having the same issue but the datasets version was 2.6.1, after I updated it to latest(2.16), error is gone while importing.

I also have datasets version 2.16, but the error is still there.

> Can you try re-installing `datasets` ?

I tried re-installing. Still getting the same error.

> > Can you try re-installing `datasets` ?
>
> I tried re-installing. Still getting the same error.

In kaggle I used:
- `%pip install -U datasets`
and then restarted runtime and then everything works fine.

> > > Can you try re-installing `datasets` ?
> >
> >
> > I tried re-installing. Still getting the same error.
>
> In kaggle I used:
>
> * `%pip install -U datasets`
>   and then restarted runtime and then everything works fine.

Yes, this is working. When I restart the runtime after installing packages, it's working perfectly. Thank you so much. But why do we need to restart runtime every time after installing packages?

> > > > Can you try re-installing `datasets` ?
> > >
> > >
> > > I tried re-installing. Still getting the same error.
> >
> >
> > In kaggle I used:
> >
> > * `%pip install -U datasets`
> >   and then restarted runtime and then everything works fine.
>
> Yes, this is working. When I restart the runtime after installing packages, it's working perfectly. Thank you so much. But why do we need to restart runtime every time after installing packages?
For some packages it is required.
https://stackoverflow.com/questions/57831187/need-to-restart-runtime-before-import-an-installed-package-in-colab

> > > > > Can you try re-installing `datasets` ?
> > > >
> > > >
> > > > I tried re-installing. Still getting the same error.
> > >
> > >
> > > In kaggle I used:
> > >
> > > * `%pip install -U datasets`
> > >   and then restarted runtime and then everything works fine.
> >
> >
> > Yes, this is working. When I restart the runtime after installing packages, it's working perfectly. Thank you so much. But why do we need to restart runtime every time after installing packages?
> > For some packages it is required.
> > https://stackoverflow.com/questions/57831187/need-to-restart-runtime-before-import-an-installed-package-in-colab

Thank you for your assistance. I dedicated the past 2-3 weeks to resolving this issue. Interestingly, it runs flawlessly in Colab without requiring a runtime restart. However, the problem persisted exclusively in Kaggle. I appreciate your help once again. Thank you.

Closing this issue as it is not related to the datasets library; rather, it's linked to platform-related issues.

### Describe the bug

While importing from packages getting the error
Code:

```
import os
import torch
from datasets import load_dataset, Dataset
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    HfArgumentParser,
    TrainingArguments,
    pipeline,
    logging
)
from peft import LoraConfig, PeftModel
from trl import SFTTrainer
from huggingface_hub import login
import pandas as pd
```

Error:
````
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In[5], line 14
      4 from transformers import (
      5     AutoModelForCausalLM,
      6     AutoTokenizer,
   (...)
     11     logging
     12 )
     13 from peft import LoraConfig, PeftModel
---> 14 from trl import SFTTrainer
     15 from huggingface_hub import login
     16 import pandas as pd

File /opt/conda/lib/python3.10/site-packages/trl/__init__.py:21
      8 from .import_utils import (
      9     is_diffusers_available,
     10     is_npu_available,
   (...)
     13     is_xpu_available,
     14 )
     15 from .models import (
     16     AutoModelForCausalLMWithValueHead,
     17     AutoModelForSeq2SeqLMWithValueHead,
     18     PreTrainedModelWrapper,
     19     create_reference_model,
     20 )
---> 21 from .trainer import (
     22     DataCollatorForCompletionOnlyLM,
     23     DPOTrainer,
     24     IterativeSFTTrainer,
     25     PPOConfig,
     26     PPOTrainer,
     27     RewardConfig,
     28     RewardTrainer,
     29     SFTTrainer,
     30 )
     33 if is_diffusers_available():
     34     from .models import (
     35         DDPOPipelineOutput,
     36         DDPOSchedulerOutput,
     37         DDPOStableDiffusionPipeline,
     38         DefaultDDPOStableDiffusionPipeline,
     39     )

File /opt/conda/lib/python3.10/site-packages/trl/trainer/__init__.py:44
     42 from .ppo_trainer import PPOTrainer
     43 from .reward_trainer import RewardTrainer, compute_accuracy
---> 44 from .sft_trainer import SFTTrainer
     45 from .training_configs import RewardConfig

File /opt/conda/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:23
     21 import torch.nn as nn
     22 from datasets import Dataset
---> 23 from datasets.arrow_writer import SchemaInferenceError
     24 from datasets.builder import DatasetGenerationError
     25 from transformers import (
     26     AutoModelForCausalLM,
     27     AutoTokenizer,
   (...)
     33     TrainingArguments,
     34 )

ImportError: cannot import name 'SchemaInferenceError' from 'datasets.arrow_writer' (/opt/conda/lib/python3.10/site-packages/datasets/arrow_writer.py

````

transformers version: 4.36.2
python version: 3.10.12
datasets version: 2.16.1

### Steps to reproduce the bug

1. Install packages
```
!pip install -U datasets trl accelerate peft bitsandbytes transformers trl huggingface_hub
```

2. import packages
```
import os
import torch
from datasets import load_dataset, Dataset
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    HfArgumentParser,
    TrainingArguments,
    pipeline,
    logging
)
from peft import LoraConfig, PeftModel
from trl import SFTTrainer
from huggingface_hub import login
import pandas as pd
```

### Expected behavior

No error while importing

### Environment info

- `datasets` version: 2.16.0
- Platform: Linux-5.15.133+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.20.1
- PyArrow version: 11.0.0
- Pandas version: 2.1.4
- `fsspec` version: 2023.10.0


Adding support for netCDF (*.nc) files

New feature or request

Conceptually, we can use xarray to load the netCDF file, then xarray -> pandas -> pyarrow.

I'd still need to verify that such a conversion would be lossless, especially for multi-dimensional data.

### Feature request

netCDF (*.nc) is a file format for storing multidimensional scientific data, which is used by packages like `xarray` (labelled multi-dimensional arrays in Python). It would be nice to have native support for netCDF in `datasets`.

### Motivation

When uploading *.nc files onto Huggingface Hub through the `datasets` API, I would like to be able to preview the dataset without converting it to another format.

### Your contribution

I can submit a PR, provided I have the time.


datasets.load_dataset raises FileNotFoundError for datasets==2.16.0

Hi ! Thanks for reporting

This is a bug in 2.16.0 for some datasets when `cache_dir` is a relative path. I opened https://github.com/huggingface/datasets/pull/6543 to fix this

We just released 2.16.1 with a fix:

```
pip install -U datasets
```

### Describe the bug

Seems `datasets.load_dataset` raises FileNotFoundError for some hub datasets with the latest `datasets==2.16.0`

### Steps to reproduce the bug

For example `pip install datasets==2.16.0`
then

```python
import datasets

datasets.load_dataset("wentingzhao/anthropic-hh-first-prompt", cache_dir='cache1')["train"]

```

This will raise:

```bash
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/xxx/miniconda3/envs/env/lib/python3.9/site-packages/datasets/load.py", line 2545, in load_dataset
    builder_instance.download_and_prepare(
  File "/Users/xxx/miniconda3/envs/env/lib/python3.9/site-packages/datasets/builder.py", line 1003, in download_and_prepare
    self._download_and_prepare(
  File "/Users/xxx/miniconda3/envs/env/lib/python3.9/site-packages/datasets/builder.py", line 1076, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/Users/xxx/miniconda3/envs/env/lib/python3.9/site-packages/datasets/packaged_modules/parquet/parquet.py", line 43, in _split_generators
    data_files = dl_manager.download_and_extract(self.config.data_files)
  File "/Users/xxx/miniconda3/envs/env/lib/python3.9/site-packages/datasets/download/download_manager.py", line 566, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/Users/xxx/miniconda3/envs/env/lib/python3.9/site-packages/datasets/download/download_manager.py", line 539, in extract
    extracted_paths = map_nested(
  File "/Users/xxx/miniconda3/envs/env/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 466, in map_nested
    mapped = [
  File "/Users/xxx/miniconda3/envs/env/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 467, in <listcomp>
    _single_map_nested((function, obj, types, None, True, None))
  File "/Users/xxx/miniconda3/envs/env/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 387, in _single_map_nested
    mapped = [_single_map_nested((function, v, types, None, True, None)) for v in pbar]
  File "/Users/xxx/miniconda3/envs/env/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 387, in <listcomp>
    mapped = [_single_map_nested((function, v, types, None, True, None)) for v in pbar]
  File "/Users/xxx/miniconda3/envs/env/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 370, in _single_map_nested
    return function(data_struct)
  File "/Users/xxx/miniconda3/envs/env/lib/python3.9/site-packages/datasets/download/download_manager.py", line 451, in _download
    out = cached_path(url_or_filename, download_config=download_config)
  File "/Users/xxx/miniconda3/envs/env/lib/python3.9/site-packages/datasets/utils/file_utils.py", line 188, in cached_path
    output_path = get_from_cache(
  File "/Users/xxx/miniconda3/envs/env/lib/python3.9/site-packages/datasets/utils/file_utils.py", line 570, in get_from_cache
    raise FileNotFoundError(f"Couldn't find file at {url}")
FileNotFoundError: Couldn't find file at https://huggingface.co/datasets/wentingzhao/anthropic-hh-first-prompt/resolve/11b393a5545f706a357ebcd4a5285d93db176715/cache1/downloads/87d66c365626feca116cba323c4856c9aae056e4503f09f23e34aa085eb9de15
```

However, seems it works fine for some datasets, for example, if works fine for `datasets.load_dataset("ag_news", cache_dir='cache2')["test"]`

But the dataset works fine for datasets==2.15.0, for example `pip install datasets==2.15.0`,
then

```python
import datasets

datasets.load_dataset("wentingzhao/anthropic-hh-first-prompt", cache_dir='cache3')["train"]

Dataset({
    features: ['user', 'system', 'source'],
    num_rows: 8552
})
```

### Expected behavior

2.16.0 should work as same as 2.15.0 for all datasets

### Environment info

python3.9
conda env
tested on MacOS and Linux


IndexError: Invalid key: 47682 is out of bounds for size 0 while using PEFT

This is surely the same issue as https://discuss.huggingface.co/t/indexerror-invalid-key-16-is-out-of-bounds-for-size-0/14298/25 that comes from the `transformers` `Trainer`. You should add `remove_unused_columns=False` to `TrainingArguments`

Also check your logs: the `Trainer` should log the length of your dataset before training starts and it surely showed length=0.

the same error
IndexError: Invalid key: 22330 is out of bounds for size 0

### Describe the bug

I am trying to fine-tune the t5 model on the paraphrasing task. While running the same code without-

model = get_peft_model(model, config)

the model trains without any issues. However, using the model returned from get_peft_model raises the following error due to datasets-

IndexError: Invalid key: 47682 is out of bounds for size 0.

I had raised this in https://github.com/huggingface/peft/issues/1299#issue-2056173386 and they suggested that I raise it here.

Here is the complete error-

IndexError Traceback (most recent call last)
in <cell line: 1>()
----> 1 trainer.train()

11 frames
[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
1553 hf_hub_utils.enable_progress_bars()
1554 else:
-> 1555 return inner_training_loop(
1556 args=args,
1557 resume_from_checkpoint=resume_from_checkpoint,

[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in _inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)
1836
1837 step = -1
-> 1838 for step, inputs in enumerate(epoch_iterator):
1839 total_batched_samples += 1
1840 if rng_to_sync:

[/usr/local/lib/python3.10/dist-packages/accelerate/data_loader.py](https://localhost:8080/#) in iter(self)
446 # We iterate one batch ahead to check when we are at the end
447 try:
--> 448 current_batch = next(dataloader_iter)
449 except StopIteration:
450 yield

[/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py](https://localhost:8080/#) in next(self)
628 # TODO(https://github.com/pytorch/pytorch/issues/76750)
629 self._reset() # type: ignore[call-arg]
--> 630 data = self._next_data()
631 self._num_yielded += 1
632 if self._dataset_kind == _DatasetKind.Iterable and \

[/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py](https://localhost:8080/#) in _next_data(self)
672 def _next_data(self):
673 index = self._next_index() # may raise StopIteration
--> 674 data = self._dataset_fetcher.fetch(index) # may raise StopIteration
675 if self._pin_memory:
676 data = _utils.pin_memory.pin_memory(data, self._pin_memory_device)

[/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py](https://localhost:8080/#) in fetch(self, possibly_batched_index)
47 if self.auto_collation:
48 if hasattr(self.dataset, "getitems") and self.dataset.getitems:
---> 49 data = self.dataset.getitems(possibly_batched_index)
50 else:
51 data = [self.dataset[idx] for idx in possibly_batched_index]

[/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in getitems(self, keys)
2802 def getitems(self, keys: List) -> List:
2803 """Can be used to get a batch using a list of integers indices."""
-> 2804 batch = self.getitem(keys)
2805 n_examples = len(batch[next(iter(batch))])
2806 return [{col: array[i] for col, array in batch.items()} for i in range(n_examples)]

[/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in getitem(self, key)
2798 def getitem(self, key): # noqa: F811
2799 """Can be used to index columns (by string names) or rows (by integer index or iterable of indices or bools)."""
-> 2800 return self._getitem(key)
2801
2802 def getitems(self, keys: List) -> List:

[/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in _getitem(self, key, **kwargs)
2782 format_kwargs = format_kwargs if format_kwargs is not None else {}
2783 formatter = get_formatter(format_type, features=self._info.features, **format_kwargs)
-> 2784 pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
2785 formatted_output = format_table(
2786 pa_subtable, key, formatter=formatter, format_columns=format_columns, output_all_columns=output_all_columns

[/usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py](https://localhost:8080/#) in query_table(table, key, indices)
581 else:
582 size = indices.num_rows if indices is not None else table.num_rows
--> 583 _check_valid_index_key(key, size)
584 # Query the main table
585 if indices is None:

[/usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py](https://localhost:8080/#) in _check_valid_index_key(key, size)
534 elif isinstance(key, Iterable):
535 if len(key) > 0:
--> 536 _check_valid_index_key(int(max(key)), size=size)
537 _check_valid_index_key(int(min(key)), size=size)
538 else:

[/usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py](https://localhost:8080/#) in _check_valid_index_key(key, size)
524 if isinstance(key, int):
525 if (key < 0 and key + size < 0) or (key >= size):
--> 526 raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
527 return
528 elif isinstance(key, slice):

IndexError: Invalid key: 47682 is out of bounds for size 0

### Steps to reproduce the bug

device = "cuda:0" if torch.cuda.is_available() else "cpu"

#defining model name for tokenizer and model loading
model_name= "t5-small"
#loading the tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name)

def preprocess_function(data, tokenizer):
inputs = [f"Paraphrase this sentence: {doc}" for doc in data["text"]]
model_inputs = tokenizer(inputs, max_length=150, truncation=True)
labels = [ast.literal_eval(i)[0] for i in data['paraphrases']]
labels = tokenizer(labels, max_length=150, truncation=True)
model_inputs["labels"] = labels["input_ids"]
return model_inputs

train_dataset = load_dataset("humarin/chatgpt-paraphrases", split="train").shuffle(seed=42).select(range(50000))
val_dataset = load_dataset("humarin/chatgpt-paraphrases", split="train").shuffle(seed=42).select(range(50000,55000))

tokenized_train = train_dataset.map(lambda batch: preprocess_function(batch, tokenizer), batched=True)
tokenized_val = val_dataset.map(lambda batch: preprocess_function(batch, tokenizer), batched=True)

def print_trainable_parameters(model):
"""
Prints the number of trainable parameters in the model.
"""
trainable_params = 0
all_param = 0
for _, param in model.named_parameters():
all_param += param.numel()
if param.requires_grad:
trainable_params += param.numel()
print(
f"trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param}"
)

config = LoraConfig(
r=16, #attention heads
lora_alpha=32, #alpha scaling
lora_dropout=0.05,
bias="none",
task_type="Seq2Seq"
)

#loading the model
model = AutoModelForSeq2SeqLM.from_pretrained(model_name).to(device)

model = get_peft_model(model, config)
print_trainable_parameters(model)

#loading the data collator
data_collator = DataCollatorForSeq2Seq(
tokenizer=tokenizer,
model=model,
label_pad_token_id=-100,
padding="longest"
)

#defining the training arguments
training_args = Seq2SeqTrainingArguments(
output_dir=os.getcwd(),
evaluation_strategy="epoch",
save_strategy="epoch",
learning_rate=2e-5,
per_device_train_batch_size=16,
per_device_eval_batch_size=16,
weight_decay=1e-3,
save_total_limit=3,
load_best_model_at_end=True,
num_train_epochs=1,
predict_with_generate=True
)

def compute_metric_with_extra(tokenizer):
def compute_metrics(eval_preds):
metric = evaluate.load('rouge')
preds, labels = eval_preds

    # decode preds and labels
    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
    decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)
    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

    # rougeLSum expects newline after each sentence
    decoded_preds = ["\n".join(nltk.sent_tokenize(pred.strip())) for pred in decoded_preds]
    decoded_labels = ["\n".join(nltk.sent_tokenize(label.strip())) for label in decoded_labels]

    result = metric.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)
    return result

return compute_metrics
trainer = Seq2SeqTrainer(
model=model,
args=training_args,
train_dataset=tokenized_train,
eval_dataset=tokenized_val,
tokenizer=tokenizer,
data_collator=data_collator,
compute_metrics= compute_metric_with_extra(tokenizer)
)

trainer.train()

### Expected behavior

I would want the trainer to train normally as it was before I used-

model = get_peft_model(model, config)

### Environment info

datasets version- 2.16.0
peft version- 0.7.1
transformers version- 4.35.2
accelerate version- 0.25.0
python- 3.10.12
enviroment- google colab


How to configure multiple folders in the same zip package

How should I write "config" in readme when all the data, such as train test, is in a zip file

train floder and test floder in data.zip


ted_talks_iwslt | Error: Config name is missing

Hi ! Thanks for reporting. I opened https://github.com/huggingface/datasets/pull/6544 to fix this

### Describe the bug

Running load_dataset using the newest `datasets` library like below on the ted_talks_iwslt using year pair data will throw an error "Config name is missing"

see also:
https://huggingface.co/datasets/ted_talks_iwslt/discussions/3

likely caused by #6493, where the `and not config_kwargs` part in the if logic was removed
 https://github.com/huggingface/datasets/blob/ef3b5dd3633995c95d77f35fb17f89ff44990bc4/src/datasets/builder.py#L512

### Steps to reproduce the bug

run:
```python
load_dataset("ted_talks_iwslt", language_pair=("ja", "en"), year="2015")
```

### Expected behavior

Load the data without error

### Environment info

datasets 2.16.0


[Feature request] Indexing datasets by a customly-defined id field to enable random access dataset items via the id

New feature or request

You can simply use a python dict as index:

```python
>>> from datasets import load_dataset
>>> ds = load_dataset("BeIR/dbpedia-entity", "corpus", split="corpus")
>>> index = {key: idx for idx, key in enumerate(ds["_id"])}
>>> ds[index["<dbpedia:Pikachu>"]]
{'_id': '<dbpedia:Pikachu>',
 'title': 'Pikachu',
 'text': 'Pikachu (Japanese: ピカチュウ) are a fictional species of Pokémon.  Pokémon are fictional creatures that appear in an assortment of comic books, animated movies and television shows, video games, and trading card games licensed by The Pokémon Company, a Japanese corporation. The Pikachu design was conceived by Ken Sugimori.'}
```

Thanks for your reply. Yes, I can do that, but it is time-consuming to do that every time I launch the program (some datasets are extremely big). HF Datasets has a nice feature to support instant data loading and efficient random access via row ids. I'm curious if this beneficial feature could be further extended to custom data columns.

+1 on the issue I think it would be extremely useful

+1 - currently having to manually implement this

### Feature request

Some datasets may contain an id-like field, for example the `id` field in [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) and the `_id` field in [BeIR/dbpedia-entity](https://huggingface.co/datasets/BeIR/dbpedia-entity). HF datasets support efficient random access via row, but not via this kinds of id fields. I wonder if it is possible to add support for indexing by a custom "id-like" field to enable random access via such ids. The ids may be numbers or strings.

### Motivation

In some cases, especially during inference/evaluation, I may want to find out the item that has a specified id, defined by the dataset itself.

For example, in a typical re-ranking setting in information retrieval, the user may want to re-rank the set of candidate documents of each query. The input is usually presented in a TREC-style run file, with the following format:

```
<qid> Q0 <docno> <rank> <score> <tag>
```

The re-ranking program should be able to fetch the queries and documents according to the `<qid>` and `<docno>`, which are the original id defined in the query/document datasets. To accomplish this, I have to iterate over the whole HF dataset to get the mapping from real ids to row ids every time I start the program, which is time-consuming. Thus I want HF dataset to provide options for users to index by a custom id column, not by row.

### Your contribution

I'm not an expert in this project and I'm afraid that I'm not able to make contributions on the code.