Docs for creating a loading script for image datasets

documentation

Improvements or additions to documentation

IMO it would make more sense to add a "Create image dataset" page with two main sections - a no-code approach with `imagefolder` + metadata (preferred way), and with a loading script (advanced). It should be clear when to choose which. If we leave this as-is, the user who jumps straight to the Vision section could be under the impression that writing a loading script is the preferred way to share a vision dataset due to how this subsection starts:
```
Write a dataset loading script to share a dataset.
```

Also, I think a note explaining how to make a dataset gated/disable the viewer to hide the data would be beneficial (it's pretty common to require submitting a form to access a CV dataset).

Great suggestion @mariosasko! I added your suggestions, let me know what you think. For gated dataset access, I just added a tip referring users to the relevant docs since it's more of a Hub feature than `datasets` feature.

Thanks, looks much better now :). I would also move the sections explaining how to create an `imagefolder` for the specific task from the [loading page](https://raw.githubusercontent.com/huggingface/datasets/main/docs/source/image_load.mdx) to this one. IMO it makes more sense to have the basic info (imagefolder structure + `load_dataset` call) there + a link to this page for info on how to create an image folder dataset.

Good idea! Moved everything about `imagefolder` + metadata to the create an image dataset section since the `load_dataset` call is the same for different computer vision tasks.

Thanks for all the feedbacks! 🥰

What do you think about creating how to share an `ImageFolder` dataset in a separate PR? I think we should create a new section under `Vision` for how to share an image dataset.

I love it thanks ! I think moving forward we can use CSV instead of JSON Lines in the docs ;)

This PR is a first draft of creating a loading script for image datasets. Feel free to let me know if there are any specificities I'm missing for this. 🙂

To do:
- [x] Document how to create different configurations.


bug

Something isn't working

Thanks for reporting @conceptofmind.

Could you please give details about your environment?
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:
```

Hi @albertvillanova ,

Here is the environment information:
```
- `datasets` version: 2.3.2
- Platform: Linux-5.4.0-122-generic-x86_64-with-glibc2.27
- Python version: 3.9.12
- PyArrow version: 7.0.0
- Pandas version: 1.4.2
```
Thanks,

Enrico

I think this issue is solved here https://discuss.huggingface.co/t/minhash-deduplication/19992/12?u=loubnabnl, this only happens for very large datasets we will update it in CodeParrot code

Hi @loubnabnl,

Yes, the issue is solved in the discussion thread.

I will close this issue.

Thank you again for all of your help.

Enrico

Thanks @loubnabnl for pointing out the solution to this issue.

## Describe the bug
Following the example in CodeParrot, I receive an array size limitation error when deduplicating larger datasets.

## Steps to reproduce the bug
```python
dataset_name = "the_pile"
ds = load_dataset(dataset_name, split="train")
ds = ds.map(preprocess, num_proc=num_workers)
uniques = set(ds.unique("hash"))
```
Gists for minimum reproducible example:
https://gist.github.com/conceptofmind/c5804428ea1bd89767815f9cd5f02d9a
https://gist.github.com/conceptofmind/feafb07e236f28d79c2d4b28ffbdb6e2

## Expected results
Chunking and writing out a deduplicated dataset.

## Actual results
```
return dataset._data.column(column).unique().to_pylist()
File "pyarrow/table.pxi", line 394, in pyarrow.lib.ChunkedArray.unique
File "pyarrow/_compute.pyx", line 531, in pyarrow._compute.call_function
File "pyarrow/_compute.pyx", line 330, in pyarrow._compute.Function.call
File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 124, in pyarrow.lib.check_status
pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2147483648
```


Fix label renaming and add a battery of tests

_The documentation is not available anymore as the PR was closed or merged._

Why don't we deprecate label renaming already instead ?

I think it'll break a lot of workflows if we deprecate it now! There isn't really a non-deprecated workflow yet - once we've added the `auto_rename_labels` option, then we can have `prepare_tf_dataset` on the `transformers` side use that, and then we can consider setting the default option to `False`, or beginning to deprecate it somehow.

I'm worried it's a bit of a waste of time to continue working on this behavior that shouldn't be here in the first place. Do you have a plan in mind ?

@lhoestq Broadly! The plan is:

1) Create the `auto_rename_labels` flag with this PR and skip label renaming if it isn't set. Leave it as `True` for backward compatibility.
2) Add the label renaming logic to `model.prepare_tf_dataset` in `transformers`. That method calls `to_tf_dataset()` right now. Once the label renaming logic is moved there, `model.prepare_tf_dataset` will set `auto_rename_labels=False` when calling `to_tf_dataset()`, and do label renaming itself.

After step 2, `auto_rename_labels` is now only necessary for backward compatibility when users use `to_tf_dataset` directly. I want to leave it alone for a while because the `model.prepare_tf_dataset` workflow is very new. However, once it is established, we can deprecate `auto_rename_labels` and then finally remove it from the `datasets` code and keep it in `transformers` where it belongs.

I see ! Could it be possible to not add `auto_rename_labels` at all, since you want to remove it at the end ? Something roughly like this:
1. show a warning in `to_tf_dataset` whevener a label is renamed automatically, saying that in the next major release this will be removed
1. add the label renaming logic in `transformers` (to not have the warning)
1. after some time, do a major release 3.0.0 and remove label renaming completely in `to_tf_dataset`

What do you think ? cc @LysandreJik in case you have an opinion on this process.

@lhoestq I think that plan is mostly good, but if we make the change to `datasets` first then all users will keep getting deprecation warnings until we update the method in `transformers` and release a new version.

I think we can follow your plan, but make the change to `transformers` first and wait for a new release before changing `datasets` - that way there are no visible warnings or API changes for users using `prepare_tf_dataset`. It also gives us more time to update the docs and try to move people to `prepare_tf_dataset` so they aren't confused by this!

Sounds good to me ! To summarize:
1. add the label renaming logic in `transformers` + release
1. show a warning in `to_tf_dataset` whevener a label is renamed automatically, saying that in the next major release this will be removed + minor release
1. after some time, do a major release 3.0.0 and remove label renaming completely in `to_tf_dataset`

@lhoestq Are you okay with me merging this for now?

Can you remove `auto_rename_labels` ? I don't think it's a good idea to add it if the plan is to remove it later

Right now, the `auto_rename_labels` behaviour happens in all cases! Making it an option is the first step in the process of disabling it (and moving the functionality to `transformers`) and then finally deprecating it.

This PR makes some changes to label renaming in `to_tf_dataset()`, both to fix some issues when users input something we weren't expecting, and also to make it easier to deprecate label renaming in future, if/when we want to move this special-casing logic to a function in `transformers`.

The main changes are:
- Label renaming now only happens when the `auto_rename_labels` argument is set. For backward compatibility, this defaults to `True` for now.
- If the user requests "label" but the data collator renames that column to "labels", the label renaming logic will now handle that case correctly.
- Added a battery of tests to make this more reliable in future.
- Adds an optimization to loading in `to_tf_dataset()` for unshuffled datasets (uses slicing instead of a list of indices)

Fixes #4772


Remove apache_beam import from module level in natural_questions dataset

Instead of importing `apache_beam` at the module level, import it in the method `_build_pcollection`.

Fix #4779.


Loading natural_questions requires apache_beam even with existing preprocessed data

Something isn't working

## Describe the bug
When loading "natural_questions", the package "apache_beam" is required:
```
ImportError: To be able to use natural_questions, you need to install the following dependency: apache_beam.
Please install it using 'pip install apache_beam' for instance'
```

This requirement is unnecessary, once there exists preprocessed data and the script just needs to download it.

## Steps to reproduce the bug
```python
load_dataset("natural_questions", "dev", split="validation", revision="main")
```

## Expected results
No ImportError raised.

## Actual results
```
ImportError                               Traceback (most recent call last)
[<ipython-input-3-c938e7c05d02>](https://localhost:8080/#) in <module>()
----> 1 from datasets import load_dataset; ds = load_dataset("natural_questions", "dev", split="validation", revision="main")

[/usr/local/lib/python3.7/dist-packages/datasets/load.py](https://localhost:8080/#) in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1732         revision=revision,
   1733         use_auth_token=use_auth_token,
-> 1734         **config_kwargs,
   1735     )
   1736

[/usr/local/lib/python3.7/dist-packages/datasets/load.py](https://localhost:8080/#) in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, use_auth_token, **config_kwargs)
   1504         download_mode=download_mode,
   1505         data_dir=data_dir,
-> 1506         data_files=data_files,
   1507     )
   1508

[/usr/local/lib/python3.7/dist-packages/datasets/load.py](https://localhost:8080/#) in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1245                         f"Couldn't find '{path}' on the Hugging Face Hub either: {type(e1).__name__}: {e1}"
   1246                     ) from None
-> 1247                 raise e1 from None
   1248     else:
   1249         raise FileNotFoundError(

[/usr/local/lib/python3.7/dist-packages/datasets/load.py](https://localhost:8080/#) in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1180                     download_config=download_config,
   1181                     download_mode=download_mode,
-> 1182                     dynamic_modules_path=dynamic_modules_path,
   1183                 ).get_module()
   1184             elif path.count("/") == 1:  # community dataset on the Hub

[/usr/local/lib/python3.7/dist-packages/datasets/load.py](https://localhost:8080/#) in get_module(self)
    490             base_path=hf_github_url(path=self.name, name="", revision=revision),
    491             imports=imports,
--> 492             download_config=self.download_config,
    493         )
    494         additional_files = [(config.DATASETDICT_INFOS_FILENAME, dataset_infos_path)] if dataset_infos_path else []

[/usr/local/lib/python3.7/dist-packages/datasets/load.py](https://localhost:8080/#) in _download_additional_modules(name, base_path, imports, download_config)
    214         _them_str = "them" if len(needs_to_be_installed) > 1 else "it"
    215         raise ImportError(
--> 216             f"To be able to use {name}, you need to install the following {_depencencies_str}: "
    217             f"{', '.join(needs_to_be_installed)}.\nPlease install {_them_str} using 'pip install "
    218             f"{' '.join(needs_to_be_installed.values())}' for instance'"

ImportError: To be able to use natural_questions, you need to install the following dependency: apache_beam.
Please install it using 'pip install apache_beam' for instance'
```

## Environment info
Colab notebook.


Update local loading script docs

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4778). All of your documentation changes will be reflected on that endpoint.

I would rather have a section in the docs that explains how to modify the script of an existing dataset (`inspect_dataset` + modification + `load_dataset`) instead of focusing on the GH datasets bundled with the source (only applicable for devs).

Good idea! I went with @mariosasko's suggestion to use `inspect_dataset` instead of cloning a dataset repository since it's a good opportunity to show off more of the library's lesser-known functions if that's ok with everyone :)

One advantage of cloning the repo is that it fetches potential data files referenced inside a script using relative paths, so if we decide to use `inspect_dataset`, we should at least add a tip to explain this limitation and how to circumvent it.

Oh you're right. Calling `load_dataset` on the modified script without having the files that come with it is not ideal. I agree it should be `git clone` instead - and inspect is for inspection only ^^'

This PR clarifies the local loading script section to include how to load a dataset after you've modified the local loading script (closes #4732).


Require torchaudio<0.12.0 to avoid RuntimeError

Related to:
- https://github.com/huggingface/transformers/issues/18379

Fix partially #4776.


RuntimeError when using torchaudio 0.12.0 to load MP3 audio file

Requiring torchaudio<0.12.0 isn't really a viable solution because that implies torch<0.12.0 which means no sm_86 CUDA support which means no RTX 3090 support in PyTorch.

But in my case, the error only occurs if `_fallback_load` resolves to `_fail_load` inside torchaudio 0.12.0 which is only the case if FFMPEG initialization failed:  https://github.com/pytorch/audio/blob/b1f510fa5681e92ee82bdc6b2d1ed896799fc32c/torchaudio/backend/sox_io_backend.py#L36-L47

That means the proper solution for torchaudio>=0.12.0 is to check `torchaudio._extension._FFMPEG_INITIALIZED` and if it is False, then we need to remind the user to install a dynamically linked ffmpeg 4.1.8 and then maybe call `torchaudio._extension._init_ffmpeg()` to force a user-visible exception showing the missing ffmpeg dynamic library name.

On my system, installing

- libavcodec.so.58
- libavdevice.so.58
- libavfilter.so.7
- libavformat.so.58
- libavutil.so.56
- libswresample.so.3
- libswscale.so.5

from ffmpeg 4.1.8 made HF datasets 2.3.2 work just fine with  torchaudio 0.12.1+cu116:

```python3
import sox, torchaudio, datasets
print('torchaudio', torchaudio.__version__)
print('datasets', datasets.__version__)
torchaudio._extension._init_ffmpeg()
print(torchaudio._extension._FFMPEG_INITIALIZED)
waveform, sample_rate = torchaudio.load('/workspace/.cache/huggingface/datasets/downloads/extracted/8e5aa88585efa2a4c74c6664b576550d32b7ff9c3d1d17cc04f44f11338c3dc6/cv-corpus-8.0-2022-01-19/en/clips/common_voice_en_100038.mp3', format='mp3')
print(waveform.shape)
```

```
torchaudio 0.12.1+cu116
datasets 2.3.2
True
torch.Size([1, 369792])
```

Closing as we no longer use `torchaudio` for decoding MP3 files.

Current version of `torchaudio` (0.12.0) raises a RuntimeError when trying to use `sox_io` backend but non-Python dependency `sox` is not installed:
https://github.com/pytorch/audio/blob/2e1388401c434011e9f044b40bc8374f2ddfc414/torchaudio/backend/sox_io_backend.py#L21-L29
```python
def _fail_load(
    filepath: str,
    frame_offset: int = 0,
    num_frames: int = -1,
    normalize: bool = True,
    channels_first: bool = True,
    format: Optional[str] = None,
) -> Tuple[torch.Tensor, int]:
    raise RuntimeError("Failed to load audio from {}".format(filepath))
```

Maybe we should raise a more actionable error message so that the user knows how to fix it.

UPDATE:
- this is an incompatibility of latest torchaudio (0.12.0) and the sox backend

TODO:
- [x] as a temporary solution, we should recommend installing torchaudio<0.12.0
  - #4777
  - #4785
- [ ] however, a stable solution must be found for torchaudio>=0.12.0

Related to:
- https://github.com/huggingface/transformers/issues/18379


Streaming not supported in Theivaprakasham/wildreceipt

Thanks for reporting @NitishkKarra.

The root source of the issue is that streaming mode is not supported out-of-the-box for that dataset, because it contains a TAR file.

We have opened a discussion in the corresponding Hub dataset page, pointing out this issue: https://huggingface.co/datasets/Theivaprakasham/wildreceipt/discussions/1

I'm closing this issue here, so this discussion is transferred there instead.

### Link

_No response_

### Description

_No response_

### Owner

_No response_


Training hangs at the end of epoch, with set_transform/with_transform+multiple workers

Something isn't working

## Describe the bug
I use load_dataset() (I tried with [wiki](https://huggingface.co/datasets/wikipedia) and my own json data) and use set_transform/with_transform for preprocessing. But it hangs at the end of the 1st epoch if dataloader_num_workers>=1. No problem with single worker.

## Steps to reproduce the bug
```python
train_dataset = datasets.load_dataset("wikipedia", "20220301.en",
                                      split='train',
                                      cache_dir=model_args.cache_dir,
                                      streaming=False)
train_dataset.set_transform(psg_parse_fn)
train_dataloader = DataLoader(
            train_dataset,
            batch_size=args.train_batch_size,
            sampler=DistributedSampler(train_dataset),
            collate_fn=data_collator,
            drop_last=args.dataloader_drop_last,
            num_workers=args.dataloader_num_workers,
        )
```

## Expected results

## Actual results
It simply hangs. The ending step is num_example/batch_size (one epoch).

## Environment info
- `datasets` version: 2.4.1.dev0
- Platform: Linux-5.4.170+-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 8.0.0
- Pandas version: 1.4.1


Document loading from relative path

_The documentation is not available anymore as the PR was closed or merged._

Thanks for the feedback!

I agree that adding it to `load_hub.mdx` is probably a bit too specific, especially for beginners reading the tutorials. Since this clarification is closely related to loading from the Hub (the only difference being the presence/absence of a loading script), I think it makes the most sense to keep it somewhere in `loading.mdx`. What do you think about adding a Warning in Loading >>> Hugging Face Hub that explains the difference between relative/absolute paths when there is a script?

What about updating the section about "manual download" ? I think it goes there no ?

https://huggingface.co/docs/datasets/v2.4.0/en/loading#manual-download

Thanks ! Pinging @albertvillanova to review this change, and then I think we're good to merge

This PR describes loading a dataset from the Hub by specifying a relative path in `data_dir` or `data_files` in `load_dataset` (see #4757).


AssertionError when using label_cols in to_tf_dataset

Something isn't working

Hi @lehrig, this is caused by the data collator renaming "label" to "labels". If you set `label_cols=["labels"]` in the call it will work correctly. However, I agree that the cause of the bug is not obvious, so I'll see if I can make a PR to clarify things when the collator renames columns.

Thanks - and wow, that appears like a strange side-effect of the data collator. Is that really needed?

Why not make it more explicit? For example, extend `DefaultDataCollator` with an optional property `label_col_name` to be used as label column; only when it is not provided default to `labels` (and document that this happens) for backwards-compatibility?

Haha, I honestly have no idea why our data collators rename `"label"` (the standard label column name in our datasets) to `"labels"` (the standard label column name input to our models). It's been a pain point when I design TF data pipelines, though, because I don't want to hardcode things like that - especially in `datasets`, because the renaming is something that happens purely at the `transformers` end. I don't think I could make the change in the data collators themselves at this point, because it would break backward compatibility for everything in PyTorch as well as TF.

In the most recent version of `transformers` we added a [prepare_tf_dataset](https://huggingface.co/docs/transformers/main_classes/model#transformers.TFPreTrainedModel.prepare_tf_dataset) method to our models which takes care of these details for you, and even chooses appropriate columns and labels for the model you're using. In future we might make that the officially recommended way to convert HF datasets to `tf.data.Dataset`.

Interesting, that'd be great especially for clarity. https://huggingface.co/docs/datasets/use_with_tensorflow#data-loading already improved clarity, yet, all those options will still confuse people. Looking forward to those advances in the hope there'll be only 1 way in the future ;)

Anyways, I am happy for the time being with the work-around you provided. Thank you!

## Describe the bug
An incorrect `AssertionError` is raised when using `label_cols` in `to_tf_dataset` and the label's key name is `label`.

The assertion is in this line:
https://github.com/huggingface/datasets/blob/2.4.0/src/datasets/arrow_dataset.py#L475

## Steps to reproduce the bug
```python
from datasets import load_dataset
from transformers import DefaultDataCollator

dataset = load_dataset('glue', 'mrpc', split='train')

tf_dataset = dataset.to_tf_dataset(
    columns=["sentence1", "sentence2", "idx"],
    label_cols=["label"],
    batch_size=16,
    collate_fn=DefaultDataCollator(return_tensors="tf"),
)
```

## Expected results
No assertion error.

## Actual results
```
AssertionError: in user code:

    File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 475, in split_features_and_labels  *
        assert set(features.keys()).union(labels.keys()) == set(input_batch.keys())
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: Linux-4.18.0-305.45.1.el8_4.ppc64le-ppc64le-with-glibc2.17
- Python version: 3.8.13
- PyArrow version: 7.0.0
- Pandas version: 1.4.3


Remove dummy data generation docs

This PR removes instructions to generate dummy data since that is no longer necessary for datasets that are uploaded to the Hub instead of our GitHub repo.

Close #4744


fix typo

> good catch thanks ! Can you check if the same typo is also present in `add_elasticsearch_index` ? It has a very similar signature

fixed

By defaul -> By default


Fail to process SQuADv1.1 datasets with max_seq_length=128, doc_stride=96.

Something isn't working

## Describe the bug
datasets fail to process SQuADv1.1 with max_seq_length=128, doc_stride=96 when calling datasets["train"].train_dataset.map().

## Steps to reproduce the bug

I used huggingface[ TF2 question-answering examples](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering). And my scripts are as follows:

```
python run_qa.py \
  --model_name_or_path $BERT_DIR \
  --dataset_name $SQUAD_DIR \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2 \
  --max_seq_length 128 \
  --doc_stride 96 \
  --output_dir $OUTPUT \
  --save_steps 10000 \
  --overwrite_cache \
  --overwrite_output_dir \

```

## Expected results
Normally process SQuADv1.1 datasets with max_seq_length=128, doc_stride=96.

## Actual results
```
INFO:__main__:Padding all batches to max length because argument was set or we're on TPU.
WARNING:datasets.fingerprint:Parameter 'function'=<function main.<locals>.prepare_train_features at 0x7f15bc2d07a0> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.
  0%|                                                                                | 0/88 [00:00<?, ?ba/s]thread '<unnamed>' panicked at 'assertion failed: stride < max_len', /__w/tokenizers/tokenizers/tokenizers/src/tokenizer/encoding.rs:311:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
  0%|                                                                                | 0/88 [00:00<?, ?ba/s]
Traceback (most recent call last):
  File "run_qa.py", line 743, in <module>
    main()
  File "run_qa.py", line 485, in main
    load_from_cache_file=not data_args.overwrite_cache,
  File "/anaconda3/envs/py37/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 2394, in map
    desc=desc,
  File "/anaconda3/envs/py37/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 551, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/anaconda3/envs/py37/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 518, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/anaconda3/envs/py37/lib/python3.7/site-packages/datasets/fingerprint.py", line 458, in wrapper
    out = func(self, *args, **kwargs)
  File "anaconda3/envs/py37/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 2768, in _map_single
    offset=offset,
  File "anaconda3/envs/py37/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 2644, in apply_function_on_filtered_inputs
    processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
  File "anaconda3/envs/py37/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 2336, in decorated
    result = f(decorated_item, *args, **kwargs)
  File "run_qa.py", line 410, in prepare_train_features
    padding=padding,
  File "anaconda3/envs/py37/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 2512, in __call__
    **kwargs,
  File "anaconda3/envs/py37/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 2703, in batch_encode_plus
    **kwargs,
  File "anaconda3/envs/py37/lib/python3.7/site-packages/transformers/tokenization_utils_fast.py", line 429, in _batch_encode_plus
    is_pretokenized=is_split_into_words,
pyo3_runtime.PanicException: assertion failed: stride < max_len
Traceback (most recent call last):
  File "./data/SQuADv1.1/evaluate-v1.1.py", line 92, in <module>
    with open(args.prediction_file) as prediction_file:
FileNotFoundError: [Errno 2] No such file or directory: './output/bert_base_squadv1.1_tf2/eval_predictions.json'

```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Ubuntu, pytorch=1.11.0, tensorflow-gpu=2.9.1
- Python version: 2.7
- PyArrow version: 8.0.0


Unpin rouge_score test dependency

Once `rouge-score` has made the 0.1.2 release to fix their issue https://github.com/google-research/google-research/issues/1212, we can unpin it.

Related to:
- #4735


Add 2.4.0 version added to docstrings


Fix version in map_nested docstring

After latest release, `map_nested` docstring needs being updated with the right version for versionchanged and versionadded.


Update CI badge

Replace the old CircleCI badge with a new one for GH Actions.


More rigorous shape inference in to_tf_dataset

`tf.data` needs to know the shape of tensors emitted from a `tf.data.Dataset`. Although `None` dimensions are possible, overusing them can cause problems - Keras uses the dataset tensor spec at compile-time, and so saying that a dimension is `None` when it's actually constant can hurt performance, or even cause training to fail for dimensions that are needed to determine the shape of weight tensors!

The compromise I used here was to sample several batches from the underlying dataset and apply the `collate_fn` to them, and then to see which dimensions were "empirically variable". There's an obvious problem here, though - if you sample 10 batches and they all have the same shape on a certain dimension, there's still a small chance that the 11th batch will be different, and Keras will throw an error  if a dataset tries to emit a tensor whose shape doesn't match the spec.

I encountered this bug in practice once or twice for datasets that were mostly-but-not-totally constant on a given dimension, and I still don't have a perfect solution, but this PR should greatly reduce the risk. It samples many more batches, and also samples very small batches (size 2) - this increases the variability, making it more likely that a few outlier samples will be detected.

Ideally, of course, we'd determine the full output shape analytically, but that's surprisingly tricky when the `collate_fn` can be any arbitrary Python code!


Improve features resolution in streaming

Just took your comment into account @mariosasko , let me know if it's good for you now :)

`IterableDataset._resolve_features` was returning the features sorted alphabetically by column name, which is not consistent with non-streaming. I changed this and used the order of columns from the data themselves. It was causing some inconsistencies in the dataset viewer as well.

I also fixed `interleave_datasets` that was not filling missing columns with None, because it was not using the columns from `IterableDataset._resolve_features`

cc @severo


parallel searching in multi-gpu setting using faiss

And I don't see any speed up when increasing the number of GPUs while calling `get_nearest_examples_batch`.

Hi ! Yes search_batch uses FAISS search which happens in parallel across the GPUs

> And I don't see any speed up when increasing the number of GPUs while calling get_nearest_examples_batch.

That's unexpected, can you share the code you're running ?

here is the code snippet

```python

# add faiss index
source_dataset = load_dataset(source_path)
queries = load_dataset(query_path)
gpu = [0,1,2,3]
source_dataset.add_faiss_index(
                "embedding",
                device=gpu,
            )

# batch query
batch_size = 32
for i in tqdm(range(0, len(queries), batch_size)):
    if i + batch_size >= len(queries):
        batched_queries = queries[i:]
    else:
        batched_queries = queries[i:i+batch_size]

    batched_query_embeddings = np.stack([i for i in batched_queries['embedding']], axis=0)
    scores, candidates = source_dataset.get_nearest_examples_batch(
        "embedding",
        batched_query_embeddings,
        k=5
    )
```

The code looks all good to me, do you see all the GPUs being utilized ? What version of faiss are you using ?

I can see the memory usage of all the GPUs.
My version of `faiss-gpu` is `1.7.2`

It looks all good to me then ^^ though you said you didn't experienced speed improvements by adding more GPUs ? What size is your source dataset and what time differences did you experience ?

query set: 1e6
source dataset: 1e6
embedding size: 768
index: Flat
topk: 20
GPU: V100

The time taken to traverse the query set once is about 1.5h, which is almost not influenced by the value of query batch size or the number of GPUs according to my experiments.

Hmmm the number of GPUs should divide the time, something is going wrong. Can you check that adding more GPU does divide the memory used per GPU ? Maybe it can be worth looking at similar issues in the FAISS repository or create a noew issue over there to understand what's going on

> Can you check that adding more GPU does divide the memory used per GPU

The memory used per GPU is unchanged while adding more GPU. Is this unexpected?

I used to think that every GPU loads all the source vectors and the data parallelism is at the query level. 😆

> I used to think that every GPU loads all the source vectors and the data parallelism is at the query level. 😆

Oh indeed that's possible, I wasn't sure. Anyway you can check that calling get_nearest_examples_batch simply calls search under the hood:

https://github.com/huggingface/datasets/blob/f90f71fbbb33889fe75a3ffc101cdf16a88a3453/src/datasets/search.py#L375

Here is a runnable script.
Multi-GPU searching still does not work in my experiments.

```python
import os
from tqdm import tqdm
import numpy as np
import datasets
from datasets import Dataset

class DPRSelector:

    def __init__(self, source, target, index_name, gpu=None):
        self.source = source
        self.target = target
        self.index_name = index_name

        cache_path = 'embedding.faiss'

        if not os.path.exists(cache_path):
            self.source.add_faiss_index(
                column="embedding",
                index_name=index_name,
                device=gpu,
            )
            self.source.save_faiss_index(index_name, cache_path)
        else:
            self.source.load_faiss_index(
                index_name,
                cache_path,
                device=gpu
            )
        print('index builded!')

    def build_dataset(self, top_k, batch_size):
        print('start search')

        for i in tqdm(range(0, len(self.target), batch_size)):
            if i + batch_size >= len(self.target):
                batched_queries = self.target[i:]
            else:
                batched_queries = self.target[i:i+batch_size]

            batched_query_embeddings = np.stack([i for i in batched_queries['embedding']], axis=0)
            search_res = self.source.get_nearest_examples_batch(
                self.index_name,
                batched_query_embeddings,
                k=top_k
            )

        print('finish search')

def get_pseudo_dataset():
    pseudo_dict = {"embedding": np.zeros((1000000, 768), dtype=np.float32)}
    print('generate pseudo data')

    dataset = Dataset.from_dict(pseudo_dict)
    def list_to_array(data):
        return {"embedding": [np.array(vector, dtype=np.float32) for vector in data["embedding"]]}
    dataset.set_transform(list_to_array, columns='embedding', output_all_columns=True)

    print('build dataset')
    return dataset

if __name__=="__main__":

    np.random.seed(42)

    source_dataset = get_pseudo_dataset()
    target_dataset = get_pseudo_dataset()

    gpu = [0,1,2,3,4,5,6,7]
    selector = DPRSelector(source_dataset, target_dataset, "embedding",  gpu=gpu)

    selector.build_dataset(top_k=20, batch_size=32)
```

@lhoestq Hi, could you please test the code above if you have time? 😄

Maybe @albertvillanova you can take a look ? I won't be available in the following days

@albertvillanova Hi, can you help with this issue?

Hi @xwwwwww I'm investigating it, but I'm not an expert in Faiss. In principle, it is weird that your code does not work properly because it seems right...

> Have you tried passing `gpu=-1` and check if there is a speedup?

yes, there is a speed up using GPU compared with CPU.

When passing `device=-1`, ALL existing GPUs are used (multi GPU): this is the maximum speedup you can get. To know the number of total GPUs:
```
import faiss

ngpus = faiss.get_num_gpus()
print(ngpus)
```

When passing a list of integers to `device`, then only that number of GPUs are used (multi GPU as well)
- the speedup should be proportional (more or less) to the ratio of the number of elements passed to `device` over `ngpus`
- if this is not the case, then there is an issue in the implementation of this use case (however, I have reviewed the code and in principle I can't find any evident bug)

When passing a positive integer to `device`, then only a single GPU is used.
- this time should be more or less proportional to the time when passing `device=-1` over `ngpus`

Thanks for your help!
Have you run the code and replicated the same experimental results (i.e., no speedup while increasing the number of GPUs)?

@albertvillanova @lhoestq Sorry for the bother, is there any progress on this issue? 😃

I can confirm `add_faiss_index` calls `index = faiss.index_cpu_to_gpus_list(index, gpus=list(device))`.

Could this be an issue with your environment ? Could you try running with 1 and 8 GPUs with a code similar to[ this one from the FAISS examples](https://github.com/facebookresearch/faiss/blob/main/tutorial/python/5-Multiple-GPUs.py) but using `gpu_index = faiss.index_cpu_to_gpus_list(cpu_index, gpus=list(device))`, and see if the speed changes ?

Hi, I test the FAISS example and the speed indeed changes.  I set `nb=1000000`, `nq=1000000` and `d=64`

| num GPUS | time cost |
| -------- | --------- |
| 1        | 28.53     |
| 5        | 7.16      |

Ok the benchmark is great, not sure why it doesn't speed up the index in your case though. You can try running the benchmark with the same settings as your actual dataset
```
query set: 1e6
source dataset: 1e6
embedding size: 768
index: Flat
topk: 20
GPU: V100
```

Note that you can still pass a FAISS index you built yourself to a dataset using https://huggingface.co/docs/datasets/v2.4.0/en/package_reference/main_classes#datasets.Dataset.add_faiss_index_from_external_arrays

> Here is a runnable script. Multi-GPU searching still does not work in my experiments.
>
> ```python
> import os
> from tqdm import tqdm
> import numpy as np
> import datasets
> from datasets import Dataset
>
> class DPRSelector:
>
>     def __init__(self, source, target, index_name, gpu=None):
>         self.source = source
>         self.target = target
>         self.index_name = index_name
>
>         cache_path = 'embedding.faiss'
>
>         if not os.path.exists(cache_path):
>             self.source.add_faiss_index(
>                 column="embedding",
>                 index_name=index_name,
>                 device=gpu,
>             )
>             self.source.save_faiss_index(index_name, cache_path)
>         else:
>             self.source.load_faiss_index(
>                 index_name,
>                 cache_path,
>                 device=gpu
>             )
>         print('index builded!')
>
>     def build_dataset(self, top_k, batch_size):
>         print('start search')
>
>         for i in tqdm(range(0, len(self.target), batch_size)):
>             if i + batch_size >= len(self.target):
>                 batched_queries = self.target[i:]
>             else:
>                 batched_queries = self.target[i:i+batch_size]
>
>
>             batched_query_embeddings = np.stack([i for i in batched_queries['embedding']], axis=0)
>             search_res = self.source.get_nearest_examples_batch(
>                 self.index_name,
>                 batched_query_embeddings,
>                 k=top_k
>             )
>
>         print('finish search')
>
>
> def get_pseudo_dataset():
>     pseudo_dict = {"embedding": np.zeros((1000000, 768), dtype=np.float32)}
>     print('generate pseudo data')
>
>     dataset = Dataset.from_dict(pseudo_dict)
>     def list_to_array(data):
>         return {"embedding": [np.array(vector, dtype=np.float32) for vector in data["embedding"]]}
>     dataset.set_transform(list_to_array, columns='embedding', output_all_columns=True)
>
>     print('build dataset')
>     return dataset
>
>
>
> if __name__=="__main__":
>
>     np.random.seed(42)
>
>
>     source_dataset = get_pseudo_dataset()
>     target_dataset = get_pseudo_dataset()
>
>     gpu = [0,1,2,3,4,5,6,7]
>     selector = DPRSelector(source_dataset, target_dataset, "embedding",  gpu=gpu)
>
>     selector.build_dataset(top_k=20, batch_size=32)
> ```

By the way,  have you run this toy example and replicated my experiment results? I think it is a more direct way to figure this out :)

Hi,

I have a similar question and would like to know if there's any progress in this issue.

`dataset.add_faiss_index(column="embedding")`, this takes around 5minutes to add the index.

`dataset.add_faiss_index(column="embedding", device=-1)`, this ran for more than 10minutes and still didn't complete execution.

Now, I don't understand why that's the case as I expected for GPU the indexing should be faster

While I notice that  `add_faiss_index` has supported assigning multiple GPUs, I am still confused about how it works.

Does the `search-batch` function automatically parallelizes the input queries to different gpus?https://github.com/huggingface/datasets/blob/d76599bdd4d186b2e7c4f468b05766016055a0a5/src/datasets/search.py#L360


Issue with offline mode

Something isn't working

Hi @SaulLu, thanks for reporting.

I think offline mode is not supported for datasets containing only data files (without any loading script). I'm having a look into this...

Thanks for your feedback!

To give you a little more info, if you don't set the offline mode flag, the script will load the cache. I first noticed this behavior with the `evaluate` library, and while trying to understand the downloading flow I realized that I had a similar error with datasets.

This is related to https://github.com/huggingface/datasets/issues/3547

#5331 will be helpful to fix this, as it updates the cache directory template to be aligned with the other datasets

This issue has been fixed in `datasets` 2.16 by https://github.com/huggingface/datasets/pull/6493. The cache is now working properly :)

You just have to update `datasets`:

```
pip install -U datasets
```

I'm on version 2.17.0, and this exact problem is still persisting.

Can you share some code to reproduce your issue ?

Also make sure your cache was populated with recent versions of `datasets`. Datasets cached with old versions may not be reloadable in offline mode, though we did our best to keep as much backward compatibility as possible.

I'm not sure if this is related @lhoestq but I am experiencing a similar issue when using offline mode:

```bash
$ python -c "from datasets import load_dataset; load_dataset('openai_humaneval', split='test')"
$ HF_DATASETS_OFFLINE=1 python -c "from datasets import load_dataset; load_dataset('openai_humaneval', split='test')"
Using the latest cached version of the dataset since openai_humaneval couldn't be found on the Hugging Face Hub (offline mode is enabled).
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/dodrio/scratch/projects/2023_071/alignment-handbook/.venv/lib/python3.10/site-packages/datasets/load.py", line 2556, in load_dataset
    builder_instance = load_dataset_builder(
  File "/dodrio/scratch/projects/2023_071/alignment-handbook/.venv/lib/python3.10/site-packages/datasets/load.py", line 2265, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/dodrio/scratch/projects/2023_071/alignment-handbook/.venv/lib/python3.10/site-packages/datasets/packaged_modules/cache/cache.py", line 122, in __init__
    config_name, version, hash = _find_hash_in_cache(
  File "/dodrio/scratch/projects/2023_071/alignment-handbook/.venv/lib/python3.10/site-packages/datasets/packaged_modules/cache/cache.py", line 48, in _find_hash_in_cache
    raise ValueError(
ValueError: Couldn't find cache for openai_humaneval for config 'default'
Available configs in the cache: ['openai_humaneval']
```

Thanks for reporting @BramVanroy, I managed to reproduce and I opened a fix here: https://github.com/huggingface/datasets/pull/6741

Awesome, thanks for the quick fix @lhoestq! Looking forward to update my dependency version list.

> Thanks for reporting @BramVanroy, I managed to reproduce and I opened a fix here: #6741

Thanks a lot！ I have faced the same problem. Can I use your fix code to directly replace the existing version code? I noticed that this fix has not been merged yet. Will it affect other functionalities?

I just merged the fix, you can install `datasets` from source or wait for the patch release which will be out in the coming days

## Describe the bug
I can't retrieve a cached dataset with offline mode enabled

## Steps to reproduce the bug

To reproduce my issue, first, you'll need to run a script that will cache the dataset
```python
import os
os.environ["HF_DATASETS_OFFLINE"] = "0"

import datasets

datasets.logging.set_verbosity_info()
ds_name = "SaulLu/toy_struc_dataset"
ds = datasets.load_dataset(ds_name)
print(ds)
```
then, you can try to reload it in offline mode:
```python
import os
os.environ["HF_DATASETS_OFFLINE"] = "1"

import datasets

datasets.logging.set_verbosity_info()
ds_name = "SaulLu/toy_struc_dataset"
ds = datasets.load_dataset(ds_name)
print(ds)
```

## Expected results
I would have expected the 2nd snippet not to return any errors

## Actual results
The 2nd snippet returns:
```
Traceback (most recent call last):
  File "/home/lucile_huggingface_co/sandbox/evaluate/test_cache_datasets.py", line 8, in <module>
    ds = datasets.load_dataset(ds_name)
  File "/home/lucile_huggingface_co/anaconda3/envs/evaluate-dev/lib/python3.8/site-packages/datasets/load.py", line 1723, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/lucile_huggingface_co/anaconda3/envs/evaluate-dev/lib/python3.8/site-packages/datasets/load.py", line 1500, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/home/lucile_huggingface_co/anaconda3/envs/evaluate-dev/lib/python3.8/site-packages/datasets/load.py", line 1241, in dataset_module_factory
    raise ConnectionError(f"Couln't reach the Hugging Face Hub for dataset '{path}': {e1}") from None
ConnectionError: Couln't reach the Hugging Face Hub for dataset 'SaulLu/toy_struc_dataset': Offline mode is enabled.
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: Linux-4.19.0-21-cloud-amd64-x86_64-with-glibc2.17
- Python version: 3.8.13
- PyArrow version: 8.0.0
- Pandas version: 1.4.3

Maybe I'm misunderstanding something in the use of the offline mode (see [doc](https://huggingface.co/docs/datasets/v2.4.0/en/loading#offline)), is that the case?


Dataset Viewer issue for Toygar/turkish-offensive-language-detection

dataset-viewer

Related to the dataset viewer on huggingface.co

I refreshed the dataset viewer manually, it's fixed now. Sorry for the inconvenience.
<img width="1557" alt="Capture d’écran 2022-07-28 à 09 17 39" src="https://user-images.githubusercontent.com/1676121/181514666-92d7f8e1-ddc1-4769-84f3-f1edfdb902e8.png">

### Link

https://huggingface.co/datasets/Toygar/turkish-offensive-language-detection

### Description

Status code:   400
Exception:     Status400Error
Message:       The dataset does not exist.

Hi, I provided train.csv, test.csv and valid.csv files. However, viewer says dataset does not exist.
Should I need to do anything else?

### Owner

Yes


Document better when relative paths are transformed to URLs

As discussed with @ydshieh, when passing a relative path as `data_dir` to `load_dataset` of a dataset hosted on the Hub, the relative path is transformed to the corresponding URL of the Hub dataset.

Currently, we mention this in our docs here: [Create a dataset loading script > Download data files and organize splits](https://huggingface.co/docs/datasets/v2.4.0/en/dataset_script#download-data-files-and-organize-splits)
> If the data files live in the same folder or repository of the dataset script, you can just pass the relative paths to the files instead of URLs.

Maybe we should document better how relative paths are handled, not only when creating a dataset loading script, but also when passing to `load_dataset`:
- `data_dir`
- `data_files`

CC: @stevhliu


Datasets.map causes incorrect overflow_to_sample_mapping when used with tokenizers and small batch size

bug

Something isn't working

I've built a minimal example that shows this bug without `n_proc`. It seems like it's a problem any way of using **tokenizers, `overflow_to_sample_mapping`, and Dataset.map, with a small batch size**:

```
import datasets
import transformers
pretrained = 'deepset/tinyroberta-squad2'
tokenizer = transformers.AutoTokenizer.from_pretrained(pretrained)

questions = ['Can you tell me why?', 'What time is it?']
contexts = ['This is context zero', 'Another paragraph goes here']

def tok(questions, contexts):
    return tokenizer(text=questions,
            text_pair=contexts,
            truncation='only_second',
            return_overflowing_tokens=True,
            )
print(tok(questions, contexts)['overflow_to_sample_mapping'])
assert tok(questions, contexts)['overflow_to_sample_mapping'] == [0, 1] # PASSES

def tok2(d):
    return tok(d['question'], d['context'])

def tok2(d):
    return tok(d['question'], d['context'])

ds = datasets.Dataset.from_dict({'question': questions, 'context': contexts})
tokens = ds.map(tok2, batched=True, batch_size=1)
print(tokens['overflow_to_sample_mapping'])
assert tokens['overflow_to_sample_mapping'] == [0, 1] # FAILS produces [0,0]
```

Note that even if the batch size would be larger, there will be instances where we will not have a lot of data, and end up using small batches.  This can occur e.g. if `n_proc` causes batches to be underfill.  I imagine it can also occur in other ways, e.g. the final leftover batch at the end.

A larger batch size does _not_ have this behavior:

```
def tok2(d):
    return tok(d['question'], d['context'])

ds = datasets.Dataset.from_dict({'question': questions, 'context': contexts})
tokens = ds.map(tok2, batched=True, batch_size=2)
print(tokens['overflow_to_sample_mapping'])
assert tokens['overflow_to_sample_mapping'] == [0, 1] # PASSES
```

I was trying the [Question answering](https://huggingface.co/learn/nlp-course/chapter7/7#question-answering) tutorial on Hugging face when i faced the same problem.  The preprocessing step is [here](https://huggingface.co/learn/nlp-course/chapter7/7#processing-the-validation-data). i have changed ```max_length=200, stride=50```,

```
validation_dataset = raw_datasets['validation'].select(range(8)).map(
    preprocess_validation_examples,
    batched=True,
    remove_columns=raw_datasets["validation"].column_names,
    num_proc=1
)
print(validation_dataset['overflow_to_sample_mapping'])
print(validation_dataset['example_id'])
```
result

```
[0, 1, 2, 3, 4, 5, 6, 7]
['56be4db0acb8001400a502ec', '56be4db0acb8001400a502ed', '56be4db0acb8001400a502ee',
'56be4db0acb8001400a502ef', '56be4db0acb8001400a502f0', '56be8e613aeaaa14008c90d1',
'56be8e613aeaaa14008c90d2', '56be8e613aeaaa14008c90d3']
```
when ```num_proc=2```, result -

```
[0, 1, 2, 3, 0, 1, 2, 3]
['56be4db0acb8001400a502ec', '56be4db0acb8001400a502ed', '56be4db0acb8001400a502ee',
'56be4db0acb8001400a502ef', '56be4db0acb8001400a502f0', '56be8e613aeaaa14008c90d1',
'56be8e613aeaaa14008c90d2', '56be8e613aeaaa14008c90d3']
```

when ```num_proc=3```, result -

```
[0, 1, 2, 0, 1, 2, 0, 1]
['56be4db0acb8001400a502ec', '56be4db0acb8001400a502ed', '56be4db0acb8001400a502ee',
'56be4db0acb8001400a502ef', '56be4db0acb8001400a502f0', '56be8e613aeaaa14008c90d1',
'56be8e613aeaaa14008c90d2', '56be8e613aeaaa14008c90d3']
```

The```overflow_to_sample_mapping``` changes with ```num_proc```, but ```example_id``` field remains the same . It seems that each process in ```map``` has its own counter for overflow_to_sample_mapping. If you are using ```overflow_to_sample_mapping``` inside the ```preprocess_validation_examples``` function, then there is no issue.

## Describe the bug
When using `tokenizer`, we can retrieve the field `overflow_to_sample_mapping`, since long samples will be overflown into multiple token sequences.

However, when tokenizing is done via `Dataset.map`, with `n_proc > 1`, the `overflow_to_sample_mapping` field is wrong.  This seems to be because each tokenizer only looks at its share of the samples, and maps to the index _within its share_, but then `Dataset.map` collates them together.

## Steps to reproduce the bug

1. Make a dataset of 3 strings.
2. Tokenize via Dataset.map with n_proc = 8
3. Inspect the `overflow_to_sample_mapping` field

## Expected results
`[0, 1, 2]`

## Actual results
`[0, 0, 0]`

Notes:

1. I have not yet extracted a minimal example, but the above works reliably
2. If the dataset is large, I've yet to determine if this bug still happens a. not at all b. always c. on the small, leftover batch at the end.


Remove "unkown" language tags

_The documentation is not available anymore as the PR was closed or merged._

Following https://github.com/huggingface/datasets/pull/4753 there was still a "unknown" langauge tag in `wikipedia` so the job at https://github.com/huggingface/datasets/runs/7542567336?check_suite_focus=true failed for wikipedia


Add `language_bcp47` tag

Following (internal) https://github.com/huggingface/moon-landing/pull/3509, we need to move the bcp47 tags to `language_bcp47` and keep the `language` tag for iso 639 1-2-3 codes. In particular I made sure that all the tags in `languages` are not longer than 3 characters. I moved the rest to `language_bcp47` and fixed some of them.

After this PR is merged I think we can simplify the language validation from the DatasetMetadata class (and keep it bare-bone just for the tagging app)

PS: the CI is failing because of missing content in dataset cards that are unrelated to this PR


DatasetInfo issue when testing multiple configs: mixed task_templates

Something isn't working

I've narrowed down the issue to the `dataset_module_factory` which already creates a `dataset_infos.json` file down in the `.cache/modules/dataset_modules/..` folder. That JSON file already contains the wrong task_templates for `unfiltered`.

Ugh. Found the issue: apparently `datasets` was reusing the already existing `dataset_infos.json` that is inside `datasets/datasets/hebban-reviews`! Is this desired behavior?

Perhaps when `--save_infos` and `--all_configs` are given, an existing `dataset_infos.json` file should first be deleted before continuing with the test? Because that would assume that the user wants to create a new infos file for all configs anyway.

Hi! I think this is a reasonable solution. Would you be interested in submitting a PR?

## Describe the bug
When running the `datasets-cli test` it would seem that some config properties in a DatasetInfo get mangled, leading to issues, e.g., about the ClassLabel.

## Steps to reproduce the bug

In summary, what I want to do is create three configs:
- unfiltered: no classlabel, no tasks. Gets data from unfiltered.json.gz (I'd want this without splits, just one chunk of data, but that does not seem possible?)
- filtered_sentiment: `review_sentiment` as ClassLabel, TextClassification task with `review_sentiment` as label. Gets train/test split from respective json.gz files
- filtered_rating: `review_rating0` as ClassLabel, TextClassification task with `review_rating0` as label. Gets train/test split from respective json.gz files

This might be a bit tedious to reproduce, so I am sorry, but these are the steps:

- Clone datasets -> `datasets/` and install it
- Clone `https://huggingface.co/datasets/BramVanroy/hebban-reviews` into `datasets/datasets` so that you have a new folder `datasets/datasets/hebban-reviews/`.
- Replace the HebbanReviews class with this new one:

```python
class HebbanReviews(datasets.GeneratorBasedBuilder):
    """The Hebban book reviews dataset."""

    BUILDER_CONFIGS = [
        HebbanReviewsConfig(
            name="unfiltered",
            description=_HEBBAN_REVIEWS_UNFILTERED_DESCRIPTION,
            version=datasets.Version(_HEBBAN_VERSION)
        ),
        HebbanReviewsConfig(
            name="filtered_sentiment",
            description=f"This config has the negative, neutral, and positive sentiment scores as ClassLabel in the 'review_sentiment' column.\n{_HEBBAN_REVIEWS_FILTERED_DESCRIPTION}",
            version=datasets.Version(_HEBBAN_VERSION)
        ),
        HebbanReviewsConfig(
            name="filtered_rating",
            description=f"This config has the 5-class ratings as ClassLabel in the 'review_rating0' column (which is a variant of 'review_rating' that starts counting from 0 instead of 1).\n{_HEBBAN_REVIEWS_FILTERED_DESCRIPTION}",
            version=datasets.Version(_HEBBAN_VERSION)
        )
    ]

    DEFAULT_CONFIG_NAME = "filtered_sentiment"

    _URLS = {
        "train": "train.jsonl.gz",
        "test": "test.jsonl.gz",
        "unfiltered": "unfiltered.jsonl.gz",
    }

    def _info(self):
        features = {
            "review_title": datasets.Value("string"),
            "review_text": datasets.Value("string"),
            "review_text_without_quotes": datasets.Value("string"),
            "review_n_quotes": datasets.Value("int32"),
            "review_n_tokens": datasets.Value("int32"),
            "review_rating": datasets.Value("int32"),
            "review_rating0": datasets.Value("int32"),
            "review_author_url": datasets.Value("string"),
            "review_author_type": datasets.Value("string"),
            "review_n_likes": datasets.Value("int32"),
            "review_n_comments": datasets.Value("int32"),
            "review_url": datasets.Value("string"),
            "review_published_date": datasets.Value("string"),
            "review_crawl_date": datasets.Value("string"),
            "lid": datasets.Value("string"),
            "lid_probability": datasets.Value("float32"),
            "review_sentiment": datasets.features.ClassLabel(names=["negative", "neutral", "positive"]),
            "review_sentiment_label": datasets.Value("string"),
            "book_id": datasets.Value("int32"),
        }

        if self.config.name == "filtered_sentiment":
            task_templates = [datasets.TextClassification(text_column="review_text_without_quotes", label_column="review_sentiment")]
        elif self.config.name == "filtered_rating":
            # For CrossEntropy, our classes need to start at index 0 -- not 1
            features["review_rating0"] = datasets.features.ClassLabel(names=["1", "2", "3", "4", "5"])
            features["review_sentiment"] = datasets.Value("int32")
            task_templates = [datasets.TextClassification(text_column="review_text_without_quotes", label_column="review_rating0")]
        elif self.config.name == "unfiltered":  # no ClassLabels in unfiltered
            features["review_sentiment"] = datasets.Value("int32")
            task_templates = None
        else:
            raise ValueError(f"Unsupported config {self.config.name}. Expected one of 'filtered_sentiment' (default),"
                             f" 'filtered_rating', or 'unfiltered'")
        print("AT INFO", self.config.name, task_templates)
        return datasets.DatasetInfo(
            description=self.config.description,
            features=datasets.Features(features),
            homepage="https://huggingface.co/datasets/BramVanroy/hebban-reviews",
            citation=_HEBBAN_REVIEWS_CITATION,
            task_templates=task_templates,
            license="cc-by-4.0"
        )

    def _split_generators(self, dl_manager):
        if self.config.name.startswith("filtered"):
            files = dl_manager.download_and_extract({"train": "train.jsonl.gz",
                                                     "test": "test.jsonl.gz"})
            return [
                datasets.SplitGenerator(
                    name=datasets.Split.TRAIN,
                    gen_kwargs={
                        "data_file": files["train"]
                    },
                ),
                datasets.SplitGenerator(
                    name=datasets.Split.TEST,
                    gen_kwargs={
                        "data_file": files["test"]
                    },
                ),
            ]
        elif self.config.name == "unfiltered":
            files = dl_manager.download_and_extract({"train": "unfiltered.jsonl.gz"})
            return [
                datasets.SplitGenerator(
                    name=datasets.Split.TRAIN,
                    gen_kwargs={
                        "data_file": files["train"]
                    },
                ),
            ]
        else:
            raise ValueError(f"Unsupported config {self.config.name}. Expected one of 'filtered_sentiment' (default),"
                             f" 'filtered_rating', or 'unfiltered'")

    def _generate_examples(self, data_file):
        lines = Path(data_file).open(encoding="utf-8").readlines()
        for line_idx, line in enumerate(lines):
            row = json.loads(line)
            yield line_idx, row
```

- finally, run `datasets-cli test ./datasets/hebban-reviews/ --save_infos --all_configs` from within the topmost `datasets` directory

## Expected results
Succeeding tests for three different configs.

## Actual results

I printed out the values that are given to `DatasetInfo` for config name and task_templates, as you can see. There, as expected, I get `unfiltered None`. I also modified datasets/info.py and added this line [at L.170](https://github.com/huggingface/datasets/blob/f5847a304aa1b38b3a3c54a8318b4df60f1299bc/src/datasets/info.py#L170):

```python
print("INTERNALLY AT INFO.PY", self.config_name, self.task_templates)
```

to my surprise, here I get `unfiltered [TextClassification(task='text-classification', text_column='review_text_without_quotes', label_column='review_sentiment')]`. So one way or another, here I suddenly see that `unfiltered` now does have a task_template -- even though that is not what is written in the data loading script, as the first print statement correctly shows.

I do not quite understand how, but it seems that the config name and task_templates get mixed.

This ultimately leads to the following error, but this trace may not be very useful in itself:

```
Traceback (most recent call last):
  File "C:\Users\bramv\.virtualenvs\hebban-U6poXNQd\Scripts\datasets-cli-script.py", line 33, in <module>
    sys.exit(load_entry_point('datasets', 'console_scripts', 'datasets-cli')())
  File "c:\dev\python\hebban\datasets\src\datasets\commands\datasets_cli.py", line 39, in main
    service.run()
  File "c:\dev\python\hebban\datasets\src\datasets\commands\test.py", line 144, in run
    builder.as_dataset()
  File "c:\dev\python\hebban\datasets\src\datasets\builder.py", line 899, in as_dataset
    datasets = map_nested(
  File "c:\dev\python\hebban\datasets\src\datasets\utils\py_utils.py", line 393, in map_nested
    mapped = [
  File "c:\dev\python\hebban\datasets\src\datasets\utils\py_utils.py", line 394, in <listcomp>
    _single_map_nested((function, obj, types, None, True, None))
  File "c:\dev\python\hebban\datasets\src\datasets\utils\py_utils.py", line 330, in _single_map_nested
    return function(data_struct)
  File "c:\dev\python\hebban\datasets\src\datasets\builder.py", line 930, in _build_single_dataset
    ds = self._as_dataset(
  File "c:\dev\python\hebban\datasets\src\datasets\builder.py", line 1006, in _as_dataset
    return Dataset(fingerprint=fingerprint, **dataset_kwargs)
  File "c:\dev\python\hebban\datasets\src\datasets\arrow_dataset.py", line 661, in __init__
    info = info.copy() if info is not None else DatasetInfo()
  File "c:\dev\python\hebban\datasets\src\datasets\info.py", line 286, in copy
    return self.__class__(**{k: copy.deepcopy(v) for k, v in self.__dict__.items()})
  File "<string>", line 20, in __init__
  File "c:\dev\python\hebban\datasets\src\datasets\info.py", line 176, in __post_init__
    self.task_templates = [
  File "c:\dev\python\hebban\datasets\src\datasets\info.py", line 177, in <listcomp>
    template.align_with_features(self.features) for template in (self.task_templates)
  File "c:\dev\python\hebban\datasets\src\datasets\tasks\text_classification.py", line 22, in align_with_features
    raise ValueError(f"Column {self.label_column} is not a ClassLabel.")
ValueError: Column review_sentiment is not a ClassLabel.
```

## Environment info
- `datasets` version: 2.4.1.dev0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.8
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Added dataset information in clinic oos dataset card

This PR aims to add relevant information like the Description, Language and citation information of the clinic oos dataset card.


Easily create loading script for benchmark comprising multiple huggingface datasets

Hi ! I think the simplest is to copy paste the `_split_generators` code from the other datasets and do a bunch of if-else, as in the glue dataset: https://huggingface.co/datasets/glue/blob/main/glue.py#L467

Hi,

I would like to create a loading script for a benchmark comprising multiple huggingface datasets.
The function _split_generators needs to return the files for the respective dataset. However, the files are not always in the same location for each dataset. I want to just make a wrapper dataset that provides a single interface to all the underlying datasets.
I thought about downloading the files with the load_dataset function and then providing the link to the cached file. But this seems a bit inelegant to me. What approach would you propose to do this?

Please let me know if you have any questions.

Cheers,
Joel


Add image classification processing guide

This PR follows up on #4710 to separate the object detection and image classification guides. It expands a little more on the original guide to include a more complete example of loading and transforming a whole dataset.


Shard parquet in `download_and_prepare`

This is ready for review cc @mariosasko :) please let me know what you think !

Following https://github.com/huggingface/datasets/pull/4724 (needs to be merged first)

It's good practice to shard parquet files to enable parallelism with spark/dask/etc.

I added the `max_shard_size` parameter to `download_and_prepare` (default to 500MB for parquet, and None for arrow).

```python
from datasets import *

output_dir = "./output_dir"  # also supports "s3://..."
builder = load_dataset_builder("squad")
builder.download_and_prepare(output_dir, file_format="parquet", max_shard_size="5MB")
```

### Implementation details

The examples are written to a parquet file until `ParquetWriter._num_bytes > max_shard_size`. When this happens, a new writer is instantiated to start writing the next shard. At the end, all the shards are renamed to include the total number of shards in their names: `{builder.name}-{split}-{shard_id:05d}-of-{num_shards:05d}.parquet`

I also added the `MAX_SHARD_SIZE` config variable (default to 500MB)

TODO:
- [x] docstrings
- [x] docs
- [x] tests

cc @severo


Dataset Viewer issue for yanekyuk/wikikey

The dataset is empty, as far as I can tell: there are no files in the repository at https://huggingface.co/datasets/yanekyuk/wikikey/tree/main

Maybe the viewer can display a better message for empty datasets

OK. Closing as it's not an error. We will work on making the error message a lot clearer.

### Link

_No response_

### Description

_No response_

### Owner

_No response_


Allow `list_datasets` to include private datasets

enhancement

New feature or request

Thanks for opening this issue :)

If it can help, I think you can already use `huggingface_hub` to achieve this:
```python
>>> from huggingface_hub import HfApi
>>> [ds_info.id for ds_info in HfApi().list_datasets(use_auth_token=token) if ds_info.private]
['bigscience/xxxx', 'bigscience-catalogue-data/xxxxxxx', ... ]
```

---------

Though the latest versions of `huggingface_hub` that contain this feature are not available on python 3.6, so maybe we should first drop support for python 3.6 (see #4460) to update `list_datasets` in `datasets` as well (or we would have to copy/paste some `huggingface_hub` code)

Great, thanks @lhoestq the workaround works! I think it would be intuitive to have the support directly in `datasets` but it makes sense to wait given that the workaround exists :)

i also think that going forward we should replace more and more implementations inside datasets with the corresponding ones from `huggingface_hub` (same as we're doing in `transformers`)

`datasets.list_datasets` is now deprecated in favor of `huggingface_hub.list_datasets` (returns private datasets when `token` is present), so I'm closing this issue.

I am working with a large collection of private datasets, it would be convenient for me to be able to list them.

I would envision extending the convention of using `use_auth_token` keyword argument to `list_datasets` function, then calling:

```
list_datasets(use_auth_token="my_token")
```

would return the list of all datasets I have permissions to view, including private ones. The only current alternative I see is to use the hub website to manually obtain the list of dataset names - this is in the context of BigScience where respective private spaces contain hundreds of datasets, so not very convenient to list manually.


Remove instructions to generate dummy data from our docs

Note that for me personally, conceptually all the dummy data (even for "canonical" datasets) should be superseded by `datasets-server`, which performs some kind of CI/CD of datasets (including the canonical ones)

I totally agree: next step should be rethinking if dummy data makes sense for canonical datasets (once we have datasets-server) and eventually remove it.

But for now, we could at least start by removing the indication to generate dummy data from our docs.

In our docs, we indicate to generate the dummy data: https://huggingface.co/docs/datasets/dataset_script#testing-data-and-checksum-metadata

However:
- dummy data makes sense only for datasets in our GitHub repo: so that we can test their loading with our CI
- for datasets on the Hub:
  - they do not pass any CI test requiring dummy data
  - there are no instructions on how they can test their dataset locally using the dummy data
  - the generation of the dummy data assumes our GitHub directory structure:
    - the dummy data will be generated under `./datasets/<dataset_name>/dummy` even if locally there is no `./datasets` directory (which is the usual case). See issue:
      - #4742

CC: @stevhliu


Update map docs

This PR updates the `map` docs for processing text to include `return_tensors="np"` to make it run faster (see #4676).


Dummy data nowhere to be found

bug

Something isn't working

Hi @BramVanroy, thanks for reporting.

First of all, please note that you do not need the dummy data: this was the case when we were adding datasets to the `datasets` library (on this GitHub repo), so that we could test the correct loading of all datasets with our CI. However, this is no longer the case for datasets on the Hub.
- We should definitely update our docs.

Second, the dummy data is generated locally:
- in your case, the dummy data will be generated inside the directory:  `./datasets/hebban-reviews/dummy`
- please note the preceding `./datasets` directory: the reason for this is that the command to generate the dummy data was specifically created for our `datasets` library, and therefore assumes our directory structure: commands are run from the root directory of our GitHub repo, and datasets scripts are under `./datasets`

I have opened an Issue to update the instructions on dummy data generation:
- #4744

Dummy data generation is deprecated now, so I think we can close this issue.

## Describe the bug
To finalize my dataset, I wanted to create dummy data as per the guide and I ran

```shell
 datasets-cli dummy_data datasets/hebban-reviews --auto_generate
```

where hebban-reviews is [this repo](https://huggingface.co/datasets/BramVanroy/hebban-reviews). And even though the scripts runs and shows a message at the end that it succeeded, I cannot find the dummy data anywhere. Where is it?

## Expected results

To see the dummy data in the datasets' folder or in the folder where I ran the command.

## Actual results

I see the following message but I cannot find the dummy data anywhere.

```
Dummy data generation done and dummy data test succeeded for config 'filtered''.
Automatic dummy data generation succeeded for all configs of '.\datasets\hebban-reviews\'
```

## Environment info
- `datasets` version: 2.4.1.dev0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.8
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Fix to dict conversion of `DatasetInfo`/`Features`

_The documentation is not available anymore as the PR was closed or merged._

Fix #4681


Fix multiprocessing in map_nested

@lhoestq as a workaround to preserve previous behavior, the parameter `multiprocessing_min_length=16` is passed from `download` to `map_nested`, so that multiprocessing is only used if at least 16 files to be downloaded.

Note that there is a small breaking change (I think previously it was unintended behavior, so that I have fixed it):
- Before (with default `num_proc=16`) if there were 16 files to be downloaded, multiprocessing was not used
- Now (with default `num_proc=16`) if there are 16 files to be downloaded, multiprocessing is used

As previously discussed:

Before, multiprocessing was not used in `map_nested` if `num_proc` was greater than or equal to `len(iterable)`.
- Multiprocessing was not used e.g. when passing `num_proc=20` but having 19 files to download
- As by default, `DownloadManager` sets `num_proc=16`, before multiprocessing was only used when `len(iterable)>16` by default

Now, if `num_proc` is greater than or equal to ``len(iterable)``, `num_proc` is set to ``len(iterable)`` and multiprocessing is used.
- We pass the variable `parallel_min_length=16`, so that multiprocessing is only used if at least 16 files to be downloaded
- ~As by default, `DownloadManager` sets `num_proc=16`, now multiprocessing is used when `len(iterable)>1` by default~

See discussion below.

~After having had to fix some tests (87602ac), I am wondering:~
- ~do we want to have multiprocessing by default?~
  - ~please note that `DownloadManager.download` sets `num_proc=16` by default~
- ~or would it be better to ask the user to set it explicitly if they want multiprocessing (and default to `num_proc=1`)?~

Fix #4636.

CC: @nateraw


Deprecate metrics

I mark this as Draft because the deprecated version number needs being updated after the latest release.

Perhaps now is the time to also update the `inspect_metric` from `evaluate` with the changes introduced in https://github.com/huggingface/datasets/pull/4433 (cc @lvwerra)

What do you think of including what changes users have to do to switch to `evaluate` in the warning message ?
(basically replace `datasets.load_metric` by `evaluate.load`)

I think it can help users migrate to `evaluate` and silence the warnings

Deprecate metrics:
- deprecate public functions: `load_metric`, `list_metrics` and `inspect_metric`: docstring and warning
- test deprecation warnings are issues
- deprecate metrics in all docs
- remove mentions to metrics in docs and README
- deprecate internal functions/classes

Maybe we should also stop testing metrics?


Use CI unit/integration tests

I think this PR can be merged. Willing to see it in action.

CC: @lhoestq

This PR:
- Implements separate unit/integration tests
- A fail in integration tests does not cancel the rest of the jobs
  - We should implement more robust integration tests: work in progress in a subsequent PR
- For the moment, test involving network requests are marked as integration: to be evolved


Download error on scene_parse_150

Something isn't working

Hi! The server with the data seems to be down. I've reported this issue (https://github.com/CSAILVision/sceneparsing/issues/34) in the dataset repo.

The URL seems to work now, and therefore the script as well.

```
from datasets import load_dataset
dataset = load_dataset("scene_parse_150", "scene_parsing")

FileNotFoundError: Couldn't find file at http://data.csail.mit.edu/places/ADEchallenge/ADEChallengeData2016.zip
```


Dataset Viewer issue for deepklarity/huggingface-spaces-dataset

Thanks for reporting. You're right, workers were under-provisioned due to a manual error, and the job queue was full. It's fixed now.

### Link

https://huggingface.co/datasets/deepklarity/huggingface-spaces-dataset/viewer/deepklarity--huggingface-spaces-dataset/train

### Description

Hi Team,
I'm getting the following error on a uploaded dataset. I'm getting the same status for a couple of hours now. The dataset size is `<1MB` and the format is csv, so I'm not sure if it's supposed to take this much time or not.
```
Status code:   400
Exception:     Status400Error
Message:       The split is being processed. Retry later.
```

Is there any explicit step to be taken to get the viewer to work?

### Owner

Yes


Pin rouge_score test dependency

Temporarily pin `rouge_score` (to avoid latest version 0.7.0) until the issue is fixed.

Fix #4734


Package rouge-score cannot be imported

Something isn't working

We have added a comment on an existing issue opened in their repo: https://github.com/google-research/google-research/issues/1212#issuecomment-1192267130
- https://github.com/google-research/google-research/issues/1212

## Describe the bug
After the today release of `rouge_score-0.0.7` it seems no longer importable. Our CI fails: https://github.com/huggingface/datasets/runs/7463218591?check_suite_focus=true
```
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_builder_class_bigbench
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_builder_configs_bigbench
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_bigbench
FAILED tests/test_metric_common.py::LocalMetricTest::test_load_metric_rouge
```
with errors:
```
>   from rouge_score import rouge_scorer
E   ModuleNotFoundError: No module named 'rouge_score'
```
```
E           ImportError: To be able to use rouge, you need to install the following dependency: rouge_score.
E           Please install it using 'pip install rouge_score' for instance'
```


rouge metric

Something isn't working

## Describe the bug
A clear and concise description of what the bug is.
Loading  Rouge metric gives error after latest rouge-score==0.0.7 release.
Downgrading rougemetric==0.0.4 works fine.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
```

## Expected results
A clear and concise description of the expected results.
from rouge_score import rouge_scorer, scoring
should run

## Actual results
Specify the actual results or traceback.
File "/root/.cache/huggingface/modules/datasets_modules/metrics/rouge/0ffdb60f436bdb8884d5e4d608d53dbe108e82dac4f494a66f80ef3f647c104f/rouge.py", line 21, in <module>
    from rouge_score import rouge_scorer, scoring
ImportError: cannot import name 'rouge_scorer' from 'rouge_score' (unknown location)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform: Linux
- Python version:3.9
- PyArrow version:


Document better that loading a dataset passing its name does not use the local script

Thanks for the feedback!

I think since this issue is closely related to loading, I can add a clearer explanation under [Load > local loading script](https://huggingface.co/docs/datasets/main/en/loading#local-loading-script).

That makes sense but I think having a line about it under https://huggingface.co/docs/datasets/installation#source the "source" header here would be useful. My mental model of `pip install -e .` does not include the fact that the source files aren't actually being used.

Thanks for sharing your perspective. I think the `load_dataset` function is the only one that pulls from GitHub, and since this use-case is very specific, I don't think we need to include such a broad clarification in the Installation section.

Feel free to check out the linked PR and let me know if it needs any additional explanation 😊

As reported by @TrentBrick here https://github.com/huggingface/datasets/issues/4725#issuecomment-1191858596, it could be more clear that loading a dataset by passing its name does not use the (modified) local script of it.

What he did:
- he installed `datasets` from source
- he modified locally `datasets/the_pile/the_pile.py` loading script
- he tried to load it but using `load_dataset("the_pile")` instead of `load_dataset("datasets/the_pile")`
  - as explained here https://github.com/huggingface/datasets/issues/4725#issuecomment-1191040245:
    - the former does not use the local script, but instead it downloads a copy of `the_pile.py` from our GitHub, caches it locally (inside `~/.cache/huggingface/modules`) and uses that.

He suggests adding a more clear explanation about this. He suggests adding it maybe in [Installation > source](https://huggingface.co/docs/datasets/installation))

CC: @stevhliu


docs: ✏ fix TranslationVariableLanguages example


Loading imagenet-1k validation split takes much more RAM than expected

Something isn't working

My bad, `482 * 418 * 50000 * 3 / 1000000 = 30221 MB` ( https://stackoverflow.com/a/42979315 ).

Meanwhile `256 * 256 * 50000 * 3 / 1000000 = 9830 MB`. We are loading the non-cropped images and that is why we take so much RAM.

## Describe the bug

Loading into memory the validation split of imagenet-1k takes much more RAM than expected. Assuming ImageNet-1k is 150 GB, split is 50000 validation images and 1,281,167 train images, I would expect only about 6 GB loaded in RAM.

## Steps to reproduce the bug
```python
from datasets import  load_dataset

dataset = load_dataset("imagenet-1k", split="validation")

print(dataset)

"""prints
Dataset({
    features: ['image', 'label'],
    num_rows: 50000
})
"""

pipe_inputs = dataset["image"]
# and wait :-)
```

## Expected results
Use only < 10 GB RAM when loading the images.

## Actual results
![image](https://user-images.githubusercontent.com/9808326/180249183-62f75ca4-d127-402a-9330-f12825a22b0a.png)

```
Using custom data configuration default
Reusing dataset imagenet-1k (/home/fxmarty/.cache/huggingface/datasets/imagenet-1k/default/1.0.0/a1e9bfc56c3a7350165007d1176b15e9128fcaf9ab972147840529aed3ae52bc)
Killed
```

## Environment info
- `datasets` version: 2.3.3.dev0
- Platform: Linux-5.15.0-41-generic-x86_64-with-glibc2.35
- Python version: 3.9.12
- PyArrow version: 7.0.0
- Pandas version: 1.3.5
- datasets commit: 4e4222f1b6362c2788aec0dd2cd8cede6dd17b80


Refactor Hub tests

This PR refactors `test_upstream_hub` by removing unittests and using the following pytest Hub fixtures:
- `ci_hub_config`
- `set_ci_hub_access_token`: to replace setUp/tearDown
- `temporary_repo` context manager: to replace `try... finally`
- `cleanup_repo`: to delete repo accidentally created if one of the tests fails

This is a preliminary work done to manage unit/integration tests separately.


load_dataset gives "403" error when using Financial Phrasebank

Hi @rohitvincent, thanks for reporting.

Unfortunately I'm not able to reproduce your issue:
```python
In [2]: from datasets import load_dataset, DownloadMode
   ...: load_dataset(path='financial_phrasebank',name='sentences_allagree', download_mode="force_redownload")
Downloading builder script: 6.04kB [00:00, 2.87MB/s]
Downloading metadata: 13.7kB [00:00, 7.24MB/s]
Downloading and preparing dataset financial_phrasebank/sentences_allagree (download: 665.91 KiB, generated: 296.26 KiB, post-processed: Unknown size, total: 962.17 KiB) to .../.cache/huggingface/datasets/financial_phrasebank/sentences_allagree/1.0.0/550bde12e6c30e2674da973a55f57edde5181d53f5a5a34c1531c53f93b7e141...
Downloading data: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 682k/682k [00:00<00:00, 7.66MB/s]
Dataset financial_phrasebank downloaded and prepared to .../.cache/huggingface/datasets/financial_phrasebank/sentences_allagree/1.0.0/550bde12e6c30e2674da973a55f57edde5181d53f5a5a34c1531c53f93b7e141. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 918.80it/s]
Out[2]:
DatasetDict({
    train: Dataset({
        features: ['sentence', 'label'],
        num_rows: 2264
    })
})
```

Are you able to access the link? https://www.researchgate.net/profile/Pekka-Malo/publication/251231364_FinancialPhraseBank-v10/data/0c96051eee4fb1d56e000000/FinancialPhraseBank-v10.zip

Yes was able to download from the link manually. But still, get the same error when I use load_dataset.

Fixed once data files are hosted on the Hub:
- #4598

I tried both codes below to download the financial phrasebank dataset (https://huggingface.co/datasets/financial_phrasebank) with the sentences_allagree subset. However, the code gives a 403 error when executed from multiple machines locally or on the cloud.

```
from datasets import load_dataset, DownloadMode
load_dataset(path='financial_phrasebank',name='sentences_allagree',download_mode=DownloadMode.FORCE_REDOWNLOAD)
```

```
from datasets import load_dataset, DownloadMode
load_dataset(path='financial_phrasebank',name='sentences_allagree')
```

**Error**
ConnectionError: Couldn't reach https://www.researchgate.net/profile/Pekka_Malo/publication/251231364_FinancialPhraseBank-v10/data/0c96051eee4fb1d56e000000/FinancialPhraseBank-v10.zip (error 403)


Dataset Viewer issue for TheNoob3131/mosquito-data

The preview is working OK:

![Screenshot from 2022-07-21 09-46-09](https://user-images.githubusercontent.com/8515462/180158929-bd8faad4-6392-4fc1-8d9c-df38aa9f8438.png)

### Link

https://huggingface.co/datasets/TheNoob3131/mosquito-data/viewer/TheNoob3131--mosquito-data/test

### Description

Dataset preview not showing with large files. Says 'split cache is empty' even though there are train and test splits.

### Owner

_No response_


Fix broken link to the Hub

_The documentation is not available anymore as the PR was closed or merged._

The Markdown link fails to render if it is in the same line as the `<span>`. This PR implements @mishig25's fix by using `<a href=" ">` instead.

![Screen Shot 2022-07-20 at 3 53 05 PM](https://user-images.githubusercontent.com/59462357/180096412-7fbb33be-abb0-4e54-a52d-201b3b58e0f9.png)


the_pile datasets URL broken.

Something isn't working

Thanks for reporting, @TrentBrick. We are addressing the change with their data host server.

On the meantime, if you would like to work with your fixed local copy of the_pile script, you should use:
```python
load_dataset("path/to/your/local/the_pile/the_pile.py",...
```
instead of just `load_dataset("the_pile",...`.

The latter downloads a copy of `the_pile.py` from our GitHub, caches it locally (inside `~/.cache/huggingface/modules`) and uses that.

@TrentBrick, I have checked the URLs and both hosts work, the original (https://the-eye.eu/) and the mirror (https://mystic.the-eye.eu/). See e.g.:
- https://mystic.the-eye.eu/public/AI/pile/
- https://mystic.the-eye.eu/public/AI/pile_preliminary_components/

Please, let me know if you still find any issue loading this dataset by using current server URLs.

Great this is working now. Re the download from GitHub... I'm sure thought went into doing this but could it be made more clear maybe here? https://huggingface.co/docs/datasets/installation for example under installing from source? I spent over an hour questioning my sanity as I kept trying to edit this file, uninstall and reinstall the repo, git reset to previous versions of the file etc.

Thanks @TrentBrick for the suggestion about improving our docs: we should definitely do this if you find they are not clear enough.

Currently, our docs explain how to load a dataset from a local loading script here: [Load > Local loading script](https://huggingface.co/docs/datasets/loading#local-loading-script)

I've opened an issue here:
- #4732

Feel free to comment on it any additional explanation/suggestion/requirement related to this problem.

https://github.com/huggingface/datasets/pull/3627 changed the Eleuther AI Pile dataset URL from https://the-eye.eu/ to https://mystic.the-eye.eu/ but the latter is now broken and the former works again.

Note that when I git clone the repo and use `pip install -e .` and then edit the URL back the codebase doesn't seem to use this edit so the mystic URL is also cached somewhere else that I can't find?


Download and prepare as Parquet for cloud storage

Added some docs for dask and took your comments into account

cc @philschmid if you also want to take a look :)

Just noticed that it would be more convenient to pass the output dir to download_and_prepare directly, to bypass the caching logic which prepares the dataset at `<cache_dir>/<name>/<version>/<hash>/`. And this way the cache is only used for the downloaded files. What do you think ?

```python

builder = load_datadet_builder("squad")
# or with a custom cache
builder = load_datadet_builder("squad", cache_dir="path/to/local/cache/for/downloaded/files")

# download and prepare to s3
builder.download_and_prepare("s3://my_bucket/squad")
```

Might be of interest:
PyTorch and AWS introduced better support for S3 streaming in `torchtext`.
![image](https://user-images.githubusercontent.com/32632186/183354186-a7f005e3-4167-4d80-ad1a-c62dd51ad7b6.png)

Having thought about it a bit more, I also agree with @philschmid in that it's important to follow the existing APIs (pandas/dask), which means we should support the following at some point:

* remote data files resolution for the packaged modules to support `load_dataset("<format>", data_files="<fs_url>")`
* `to_<format>("<fs_url>")`
* `load_from_disk` and `save_to_disk` already expose the `fs` param, but it would be cool to support specifying `fsspec` URLs directly as the source/destination path (perhaps we can then deprecate `fs` to be fully aligned with pandas/dask)

IMO these are the two main issues with the current approach:
* relying on the builder API to generate the formatted files results in a non-friendly format due to how our caching works (a lot of nested subdirectories)
* this approach still downloads the files needed to generate a dataset locally. Considering one of our goals is to align the streaming API with the non-streaming one, this could be avoided by running `to_<format>` on streamed/iterable datasets

Alright I did the last change I wanted to do, here is the final API:

```python
builder = load_dataset_builder(...)
builder.download_and_prepare("s3://...", storage_options={"token": ...})
```

and it creates the arrow files directly in the specified directory, not in a nested subdirectory structure as we do in the cache !

> this approach still downloads the files needed to generate a dataset locally. Considering one of our goals is to align the streaming API with the non-streaming one, this could be avoided by running to_<format> on streamed/iterable datasets

Yup this can be explored in some future work I think. Though to keep things simple and clear I would keep the streaming behaviors only when you load a dataset in streaming mode, and not include it in `download_and_prepare` (because it wouldn't be aligned with the name of the function, which imply to 1. download and 2. prepare ^^). Maybe an API like that can make sense for those who need full streaming

```python
ds = load_dataset(..., streaming=True)
ds.to_parquet("s3://...")
```

totally agree with your comment on the meaning of "loading", I'll update the docs

I took your comments into account and reverted all the changes related to `cache_dir` to keep the support for remote `cache_dir` for beam datasets. I also updated the wording in the docs to not use "load" when it's not appropriate :)

Download a dataset as Parquet in a cloud storage can be useful for streaming mode and to use with spark/dask/ray.

This PR adds support for `fsspec` URIs like `s3://...`, `gcs://...` etc. and ads the `file_format` to save as parquet instead of arrow:
```python
from datasets import *

cache_dir = "s3://..."
builder = load_dataset_builder("crime_and_punish", cache_dir=cache_dir)
builder.download_and_prepare(file_format="parquet")
```

EDIT: actually changed the API to

```python
from datasets import *

builder = load_dataset_builder("crime_and_punish")
builder.download_and_prepare("s3://...", file_format="parquet")
```

credentials to cloud storage can be passed using the `storage_options` argument in

For consistency with the BeamBasedBuilder, I name the parquet files `{builder.name}-{split}-xxxxx-of-xxxxx.parquet`. I think this is fine since we'll need to implement parquet sharding after this PR, so that a dataset can be used efficiently with dask for example.

Note that images/audio files are not embedded yet in the parquet files, this will added in a subsequent PR

TODO:
- [x] docs
- [x] tests


Refactor conftest fixtures

Previously, fixture modules `hub_fixtures` and `s3_fixtures`:
- were both at the root test directory
- were imported using `import *`
  - as a side effect, the modules `os` and `pytest` were imported from `s3_fixtures` into `conftest`

This PR:
- puts both fixture modules in a dedicated directory `fixtures`
- renames both to: `fixtures.hub` and `fixtures.s3`
- imports them into `conftest` as plugins, using the `pytest_plugins`: this avoids the `import *`
- additionally creates a new fixture module `fixtures.files` with all file-related fixtures


Docs: Fix same-page haslinks

`href="/docs/datasets/quickstart#audio"` implicitly goes to `href="/docs/datasets/{$LATEST_STABLE_VERSION}/quickstart#audio"`. Therefore, https://huggingface.co/docs/datasets/quickstart#audio #audio hashlink does not work since the new docs were not added to v2.3.2 (LATEST_STABLE_VERSION)

to preserve the version, it should be just `href="#audio"`, which will implicilty go to curren_page + #audio element


PyArrow Dataset error when calling `load_dataset`

Something isn't working

Hi ! It looks like a bug in `pyarrow`. If you manage to end up with only one chunk per parquet file it should workaround this issue.

To achieve that you can try to lower the value of `max_shard_size` and also don't use `map` before `push_to_hub`.

Do you have a minimum reproducible example that we can share with the Arrow team for further debugging ?

> If you manage to end up with only one chunk per parquet file it should workaround this issue.

Yup, I did not encounter this bug when I was testing my script with a slice of <1000 samples for my dataset.

> Do you have a minimum reproducible example...

Not sure if I can get more minimal than the script I shared above. Are you asking for a sample json file?
Just generate a random manifest list, I can add that to the above script if that's what you mean?

Actually this is probably linked to this open issue: https://issues.apache.org/jira/browse/ARROW-5030.

setting `max_shard_size="2GB"` should do the job (or `max_shard_size="1GB"` if you want to be on the safe side, especially given that there can be some variance in the shard sizes if the dataset is not evenly distributed)

## Describe the bug

I am fine tuning a wav2vec2 model following the script here using my own dataset: https://github.com/huggingface/transformers/blob/main/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py

Loading my Audio dataset from the hub which was originally generated from disk results in the following PyArrow error:

```sh
File "/home/ubuntu/w2v2/run_speech_recognition_ctc.py", line 227, in main
  raw_datasets = load_dataset(
File "/home/ubuntu/.virtualenvs/meval/lib/python3.8/site-packages/datasets/load.py", line 1679, in load_dataset
  builder_instance.download_and_prepare(
File "/home/ubuntu/.virtualenvs/meval/lib/python3.8/site-packages/datasets/builder.py", line 704, in download_and_prepare
  self._download_and_prepare(
File "/home/ubuntu/.virtualenvs/meval/lib/python3.8/site-packages/datasets/builder.py", line 793, in _download_and_prepare
  self._prepare_split(split_generator, **prepare_split_kwargs)
File "/home/ubuntu/.virtualenvs/meval/lib/python3.8/site-packages/datasets/builder.py", line 1268, in _prepare_split
  for key, table in logging.tqdm(
File "/home/ubuntu/.virtualenvs/meval/lib/python3.8/site-packages/tqdm/std.py", line 1195, in __iter__
  for obj in iterable:
File "/home/ubuntu/.virtualenvs/meval/lib/python3.8/site-packages/datasets/packaged_modules/parquet/parquet.py", line 68, in _generate_tables
  for batch_idx, record_batch in enumerate(
File "pyarrow/_parquet.pyx", line 1309, in iter_batches
File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Nested data conversions not implemented for chunked array outputs
```

## Steps to reproduce the bug

I created a dataset from a JSON lines manifest of `audio_filepath`, `text`, and `duration`.

When creating the dataset, I do something like this:

```python
import json
from datasets import Dataset, Audio

# manifest_lines is a list of dicts w/ "audio_filepath", "duration", and "text
for line in manifest_lines:
    line = line.strip()
    if line:
        line_dict = json.loads(line)
        manifest_dict["audio"].append(f"{root_path}/{line_dict['audio_filepath']}")
        manifest_dict["duration"].append(line_dict["duration"])
        manifest_dict["transcription"].append(line_dict["text"])

# Create a HF dataset
dataset = Dataset.from_dict(manifest_dict).cast_column(
    "audio", Audio(sampling_rate=16_000),
)

# From the docs for saving to disk
# https://huggingface.co/docs/datasets/v2.3.2/en/package_reference/main_classes#datasets.Dataset.save_to_disk
def read_audio_file(example):
    with open(example["audio"]["path"], "rb") as f:
        return {"audio": {"bytes": f.read()}}

dataset = dataset.map(read_audio_file, num_proc=70)
dataset.save_to_disk(f"/audio-data/hf/{artifact_name}")
dataset.push_to_hub(f"{org-name}/{artifact_name}", max_shard_size="5GB", private=True)
```

Then when I call `load_dataset()` in my training script, with the same dataset I generated above, and download from the huggingface hub I get the above stack trace.
I am able to load the dataset fine if I use `load_from_disk()`.

## Expected results

`load_dataset()` should behave just like `load_from_disk()` and not cause any errors.

## Actual results

See above

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
I am using the `huggingface/transformers-pytorch-gpu:latest` image
- `datasets` version: 2.3.0
- Platform: Docker/Ubuntu 20.04
- Python version: 3.8
- PyArrow version: 8.0.0