The metric Meteor doesn't work for nltk ==3.6.4

bug

Something isn't working

Hi @zhaowei-wang98, thanks for reporting.

We are fixing it...

## Describe the bug
Using the metric Meteor with nltk == 3.6.4 gives a TypeError:
TypeError: descriptor 'lower' for 'str' objects doesn't apply to a 'list' object

## Steps to reproduce the bug
```python
import datasets
metric = datasets.load_metric("meteor")
predictions = ["hello world"]
references = ["hello world"]
metric.compute(predictions=predictions, references=references)
```

## Expected results
TypeError: descriptor 'lower' for 'str' objects doesn't apply to a 'list' object

I think this TypeError exists because input sentences are tokenized into lists of tokens and the str.lower() is applied to this list of tokens.

## Actual results
No error but a meteor score

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: linux
- Python version: 3.8.12
- PyArrow version: 7.0.0


Image process doc

documentation

Improvements or additions to documentation

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3882). All of your documentation changes will be reflected on that endpoint.

This PR is a first draft of how to process image data. It adds:

- Load an image dataset with `image` and `path` (adds tip about `decode=False` param to access the path and bytes, thanks to @mariosasko).
- Load an image using the `ImageFolder` builder. I know there is an [example](https://huggingface.co/docs/datasets/master/en/loading#image-folders) of this already, but I also wanted to add it here so users don't miss it. This doc seems important for centralizing all of the image-related things so far. Datasets has grown so quickly 🚀  now that I think maybe splitting up the How-to guides by modality may be better since working with vision/audio data is slightly different from what users have seen up until now. This way we can continue to scale the docs to better accommodate vision/audio things.
- Add a data augmentation with `set_transform`. There is only 1 example here so far, but we can certainly add more.

Todo:
- [x] Couldn't figure out why my augmentation function works with `set_transform` but not `map` 🥲. Working with @mariosasko on this!


How to use Image folder

question

Further information is requested

Even this from docs throw same error
```
dataset = load_dataset("imagefolder", data_files="https://download.microsoft.com/download/3/E/1/3E1C3F21-ECDB-4869-8368-6DEBA77B919F/kagglecatsanddogs_3367a.zip", split="train")

```

Hi @INF800,

Please note that the `imagefolder` feature enhancement was just recently merged to our master branch (https://github.com/huggingface/datasets/commit/207be676bffe9d164740a41a883af6125edef135), but has not yet been released.

We are planning to make the 2.0 release of our library in the coming days and then that feature will be available by updating your `datasets` library from PyPI.

In the meantime, you can incorporate that feature if you install our library from our GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```

Then:
```python
In [1]: from datasets import load_dataset
        ds = load_dataset("imagefolder", data_files="https://download.microsoft.com/download/3/E/1/3E1C3F21-ECDB-4869-8368-6DEBA77B919F/kagglecatsanddogs_3367a.zip", split="train")
Using custom data configuration default-7eb4e80d960deb18
Downloading and preparing dataset image_folder/default to .../.cache/huggingface/datasets/image_folder/default-7eb4e80d960deb18/0.0.0/8de8dc6d68ce3c81cc102b93cc82ede27162b5d30cd003094f935942c8294f60...
Downloading data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 690.19it/s]
Extracting data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 852.85it/s]
Dataset image_folder downloaded and prepared to .../.cache/huggingface/datasets/image_folder/default-7eb4e80d960deb18/0.0.0/8de8dc6d68ce3c81cc102b93cc82ede27162b5d30cd003094f935942c8294f60. Subsequent calls will reuse this data.

In [2]: ds
Out[2]:
Dataset({
    features: ['image', 'label'],
    num_rows: 25000
})
```

Hey @albertvillanova. Does this load entire dataset in memory? Because I am facing huge trouble with loading very big datasets (OOM errors)

Can you provide the error stack trace? The loader only stores the `data_files` dict, which can get big after globbing. Then, the OOM error would mean you don't have enough memory to keep all the paths to the image files. You can circumvent this by generating an archive and loading the dataset from there. Maybe we can optimize the globbing part in our data files resolution at some point, cc @lhoestq for visibility.

Hey, memory error is resolved. It was fluke.

But there is another issue. Currently `load_dataset("imagefolder", data_dir="./path/to/train",)` takes only `train` as arg to `split` parameter.

I am creating vaildation dataset using

```
ds_valid = datasets.DatasetDict(valid=load_dataset("imagefolder", data_dir="./path/to/valid",)['train'])
```

`data_dir="path/to/folder"` is a shorthand syntax fox `data_files={"train": "path/to/folder/**"}`, so use `data_files` in that case instead:
```python
ds = load_dataset("imagefolder", data_files={"train": "path/to/train/**", "test": "path/to/test/**", "valid": "path/to/valid/**"})
```

And there was another issue. I loaded black and white images (jpeg file). Using load dataset. It reads it as PIL jpeg data format. But instead of converting it into 3 channel tensor, input to collator function is coming as a single channel tensor.

We don't apply any additional preprocessing on top of `PIL.Image.open(image_file)`, so you need to do the conversion yourself:

```python
def to_rgb(batch):
    batch["image"] = [img.convert("RGB") for img in batch["image"]]
    return batch

ds_rgb = ds.map(to_rgb, batched=True)
```

Please use our Forum for questions of this kind in the future.

Ran this code
```
 load_dataset("imagefolder", data_dir="./my-dataset")
```

`https://raw.githubusercontent.com/huggingface/datasets/master/datasets/imagefolder/imagefolder.py` missing
```
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
/tmp/ipykernel_33/1648737256.py in <module>
----> 1 load_dataset("imagefolder", data_dir="./my-dataset")

/opt/conda/lib/python3.7/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, script_version, **config_kwargs)
   1684         revision=revision,
   1685         use_auth_token=use_auth_token,
-> 1686         **config_kwargs,
   1687     )
   1688

/opt/conda/lib/python3.7/site-packages/datasets/load.py in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, use_auth_token, script_version, **config_kwargs)
   1511         download_config.use_auth_token = use_auth_token
   1512     dataset_module = dataset_module_factory(
-> 1513         path, revision=revision, download_config=download_config, download_mode=download_mode, data_files=data_files
   1514     )
   1515

/opt/conda/lib/python3.7/site-packages/datasets/load.py in dataset_module_factory(path, revision, download_config, download_mode, force_local_path, dynamic_modules_path, data_files, **download_kwargs)
   1200                         f"Couldn't find a dataset script at {relative_to_absolute_path(combined_path)} or any data file in the same directory. "
   1201                         f"Couldn't find '{path}' on the Hugging Face Hub either: {type(e1).__name__}: {e1}"
-> 1202                     ) from None
   1203                 raise e1 from None
   1204     else:

FileNotFoundError: Couldn't find a dataset script at /kaggle/working/imagefolder/imagefolder.py or any data file in the same directory. Couldn't find 'imagefolder' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/imagefolder/imagefolder.py
```


Change the framework switches to the new syntax

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3880). All of your documentation changes will be reflected on that endpoint.

This PR updates the syntax of the framework-specific code samples. With this new syntax, you'll be able to:

- have paragraphs of text be framework-specific instead of just code samples
- have support for Flax code samples if you want.

This should be merged after https://github.com/huggingface/doc-builder/pull/63 and https://github.com/huggingface/doc-builder/pull/130


SQuAD v2 metric: create README.md

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3879). All of your documentation changes will be reflected on that endpoint.

Proposing SQuAD v2 metric card


Update cats_vs_dogs size

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3878). All of your documentation changes will be reflected on that endpoint.

Maybe `NonMatchingSplitsSizesError` errors should also tell the user to try using a more recent version of the dataset to get the fixes ?

@lhoestq Good idea. Will open a new PR to improve the error messages of NonMatchingSplitsSizesError, NonMatchingChecksumsError, ...

It seems there is still a problem. I am using datasets version 2.5.1.
I just typed `ds = load_dataset("cats_vs_dogs")` and get the error below.

```
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=3893603, num_examples=23422, dataset_name='cats_vs_dogs'), 'recorded': SplitInfo(name='train', num_bytes=3891612, num_examples=23410, dataset_name='cats_vs_dogs')}]
```
It looks like the dataset still only has 23,410 examples....

Thanks for reporting, I opened https://github.com/huggingface/datasets/pull/5047

It seems like 12 new examples have been added to the `cats_vs_dogs`. This PR updates the size in the card and the info file to avoid a verification error (reported by @stevhliu).


Align metadata to DCAT/DCAT-AP

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
Align to DCAT metadata to describe datasets

**Describe the solution you'd like**
Reuse terms and structure from DCAT in the metadata file, ideally generate a json-ld file dcat compliant

**Describe alternatives you've considered**

**Additional context**
DCAT is a W3C standard extended in Europe with DCAT-AP, an example is data.europa.eu publishing datasets metadata in DCAT-AP


Fix download_mode in dataset_module_factory

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3876). All of your documentation changes will be reflected on that endpoint.

Fix `download_mode` value set in `dataset_module_factory`.

Before the fix, it was set to `bool` (default to `False`).

Also set properly its default value in all public functions.


Module namespace cleanup for v2.0

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3875). All of your documentation changes will be reflected on that endpoint.

@severo No, this PR doesn't fix that issue in the current state. We can fix it by adding `__all__` to `datasets/__init__.py` and `datasets/formatting/__init__.py`. However, this would require updating `__all__` for each new function/class definition, which could become cumbersome, and we can't do this dynamically because `mypy` is a static type checker.

@lhoestq @albertvillanova WDYT?

Feel free to merge this one if it's good for you :)

This is an attempt to make the user-facing `datasets`' submodule namespace cleaner:

In particular, this PR does the following:
* removes the unused `zip_nested` and `flatten_nest_dict` and their accompanying tests
* removes `pyarrow` from the top-level namespace
* properly uses `__all__` and the `from  <module> import *` syntax to avoid importing the `<module>`'s submodules
* cleans up the `utils` namespace
* moves the `temp_seed` context manage from `datasets/utils/file_utils.py` to `datasets/utils/py_utils.py`


add MSE and MAE metrics - V2

@mariosasko New PR here. I'm not sure how to add you as a co-author here.  Also I see flake8 tests are failing, any inputs on how to resolve this ?
Also, let me know if any other changes are required. Thank you.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3874). All of your documentation changes will be reflected on that endpoint.

Created a new pull request to resolve unrelated changes in PR caused due to rebasing.
Ref Older PR :  [#3845](https://github.com/huggingface/datasets/pull/3845)

Feature request here : Add support for continuous metrics (RMSE, MAE) [#3608](https://github.com/huggingface/datasets/issues/3608)


Create SQuAD metric README.md

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3873). All of your documentation changes will be reflected on that endpoint.

Oh one last thing I almost forgot, I think I would add a section "Examples" with examples of inputs and outputs and in particular: an example giving maximal values, an examples giving minimal values and maybe a standard examples from SQuAD. What do you think?

Proposal for a metrics card structure (with an example based on the SQuAD metric).
@thomwolf @lhoestq @douwekiela @lewtun -- feel free to comment on structure or content (it's an initial draft, so I realize there's stuff missing!).


HTTP error 504 Server Error: Gateway Time-out

is pushing directly with git (and git-lfs) an option for you?

I have installed git-lfs and doing this push with that

yes but is there any way you could try pushing with `git` command line directly instead of `push_to_hub`?

Okay. I didnt saved the dataset to my local machine. So, I processed the dataset and pushed it directly to the hub. I think I should try saving those dataset to my local machine by `save_to_disk` and then push it with git command line

cc @lhoestq @albertvillanova @LysandreJik because maybe I'm giving dumb advice here 😅

`push_to_hub` is the preferred way of uploading a dataset to the Hub, which can then be reloaded with `load_dataset`. Feel free to try again and see if the server is working as expected now. Maybe we can add a retry mechanism in the meantime to workaround 504 errors.

Regarding `save_to_disk`, this must only be used for local serialization (because it's uncompressed and compatible with memory-mapping). If you upload a dataset saved with `save_to_disk` to the Hub, then to reload it you will have to download/clone the repository locally by yourself and use `load_from_disk`.

I am trying to push a large dataset(450000+) records with the help of `push_to_hub()`
While pushing, it gives some error like this.

```
Traceback (most recent call last):
  File "data_split_speech.py", line 159, in <module>
    data_new_2.push_to_hub("user-name/dataset-name",private=True)
  File "/opt/conda/lib/python3.8/site-packages/datasets/dataset_dict.py", line 951, in push_to_hub
    repo_id, split, uploaded_size, dataset_nbytes = self[split]._push_parquet_shards_to_hub(
  File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 3556, in _push_parquet_shards_to_hub
    api.upload_file(
  File "/opt/conda/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 1017, in upload_file
    raise err
  File "/opt/conda/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 1008, in upload_file
    r.raise_for_status()
  File "/opt/conda/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: https://huggingface.co/api/datasets/user-name/dataset-name/upload/main/data/train2-00041-of-00064.parquet

```
Can anyone help me to resolve this issue.


add pandas to env command

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3871). All of your documentation changes will be reflected on that endpoint.

Think failures are unrelated - feel free to merge whenever you want :-)

Pandas is a required packages and used quite a bit. I don't see any downside with adding its version to the `datasets-cli env` command.


Add wikitablequestions dataset

@lhoestq Would you mind reviewing it when you're available? Thanks!

> Awesome thanks for adding this dataset ! :) The dataset script and dataset cards look pretty good
>
> It looks like your `dummy_data.zip` files are quite big though (>1MB each), do you think we can reduce their sizes ? This way this git repository doesn't become too big

I have manually reduced the `dummy_data.zip` and its current size is about 54KB. Hope it is fine for you!

@lhoestq I think the dataset is ready to merge now. Any follow-up question is welcome :-D

> Thanks ! It looks all good now :)

Awesome! Thanks for your quick response!


Making the Hub the place for datasets in Portuguese

dataset request

Requesting to add a new dataset

Hi @omarespejel! I think the philosophy for `datasets` issues is to create concrete issues with proposals to add a specific, individual dataset rather than umbrella issues for things such as datasets for a language, since we could end up with hundreds of issues (one per language). I see NILC - USP has many datasets, I would suggest to either create an issue for their datasets, or even better, we are trying to push to upload datasets as community datasets instead of adding them to the core library as guided in https://huggingface.co/docs/datasets/share. That would have the additional benefit that the dataset would live under the NILC organization.

@lhoestq correct me if I'm wrong please 😄

Let's make Hugging Face Datasets the central hub for datasets in Portuguese :)

**Motivation**. Datasets are currently quite scattered and an open-source central point such as the Hugging Face Hub would be ideal to support the growth of the Portuguese speaking community.

What are some datasets in Portuguese worth integrating into the Hugging Face hub?

Special thanks to @augusnunes for his collaboration on identifying the first ones:

-  [NILC - USP](http://www.nilc.icmc.usp.br/nilc/index.php/tools-and-resources).

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

cc @osanseviero


Ignore duplicate keys if `ignore_verifications=True`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3868). All of your documentation changes will be reflected on that endpoint.

Cool thanks ! Could you add a test please ?

Currently, it's impossible to generate a dataset if some keys from `_generate_examples` are duplicated. This PR allows skipping the check for duplicate keys if `ignore_verifications` is set to `True`.


Update for the rename doc-builder -> hf-doc-utils

~~@julien-c there was a vote 🙂 https://huggingface.slack.com/archives/C021H1P1HKR/p1646405136644739~~

oh I see you already commeented in the thread as well

Thanks ! It looks all good to me (provided `hf-doc-utils` is the name we keep in the end). I'm fine with this name, and `hf-doc-builder` is also fine IMHO

ok, this is definitely not a hill I'll die on =) @mishig25 @sgugger

This PR adapts the job to the upcoming change of name of `doc-builder`.


Bring back imgs so that forsk dont get broken

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3866). All of your documentation changes will be reflected on that endpoint.

I think we just need to keep `datasets_logo_name.jpg` and `course_banner.png` because they appear in the README.md of the forks of `datasets`. The other images can be removed


Add logo img

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3865). All of your documentation changes will be reflected on that endpoint.


Update image dataset tags

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3864). All of your documentation changes will be reflected on that endpoint.

Align the existing image datasets' tags with new tags introduced in #3800.


Update code blocks

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3863). All of your documentation changes will be reflected on that endpoint.

Following https://github.com/huggingface/datasets/pull/3860#issuecomment-1061756712 and https://github.com/huggingface/datasets/pull/3690 we need to update the code blocks to use markdown instead of sphinx


Manipulate columns on IterableDataset (rename columns, cast, etc.)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3862). All of your documentation changes will be reflected on that endpoint.

> IIUC we check if columns are present/not present directly in the yielded examples and not in info.features because info.features can be None (after map, for instance)?

Yes exactly

> We should develop a solution that ensures info.features is never None. For example, one approach would be to infer them from examples in map and make them promotable from Value("null") to a specific type, in case of None values.

I agree this would be useful. Though inferring the type requires to start streaming some data, which takes a few seconds (compared to being instantaneous right now).

Let's discuss this in a new issue maybe ?

I added:
- add_column
- cast
- rename_column
- rename_columns

related to https://github.com/huggingface/datasets/issues/3444

TODO:
- [x] docs
- [x] tests


big_patent cased version

Requesting to add a new dataset

To follow up on this: the cased and uncased versions actually contain different content, and the cased one is easier since it contains a Summary of the Invention in the input.

See the paper describing the issue here:
https://aclanthology.org/2022.gem-1.34/

Thanks for proposing the addition of the cased version of this dataset and for pinging again recently.

I have just merged a PR that adds the cased version: https://huggingface.co/datasets/big_patent/discussions/3

The cased version (2.1.2) is the default one:
```python
ds = load_dataset("big_patent", "all")
```

To use the 1.0.0 version (lower cased tokenized words), pass both parameters `codes` and `version`:
```python
ds = load_dataset("big_patent", codes="all", version="1.0.0")
```

Closed by: https://huggingface.co/datasets/big_patent/discussions/3

Hi! I am interested in working with the big_patent dataset.

In Tensorflow, there are a number of versions of the dataset:

- 1.0.0 : lower cased tokenized words
- 2.0.0 : Update to use cased raw strings
- 2.1.2 (default): Fix update to cased raw strings.

The version in the huggingface `datasets` library is the 1.0.0. I would be very interested in using the 2.1.2 cased version (used more, recently, for example in the Pegasus paper), but it does not seem to be supported (I tried using the `revision` parameter in `load_datasets`). Is there a way to already load it, or would it be possible to add that version?


Small doc fixes

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3860). All of your documentation changes will be reflected on that endpoint.

There are still some `.. code-block:: python` (e.g. see [this](https://huggingface.co/docs/datasets/master/en/package_reference/main_classes#datasets.Dataset.align_labels_with_mapping)) directives in our codebase, so maybe we can remove those as well as part of this PR.


Unable to dowload big_patent (FileNotFoundError)

bug

Something isn't working

duplicate

This issue or pull request already exists

Hi @slvcsl, thanks for reporting.

Yesterday we just made a patch release of our `datasets` library that fixes this issue: version 1.18.4.
https://pypi.org/project/datasets/#history

Please, feel free to update `datasets` library to the latest version:
```shell
pip install -U datasets
```
And then you should force redownload of the data file to update your local cache:
```python
ds = load_dataset("big_patent", "g", split="validation", download_mode="force_redownload")
```
- Note that before the fix, you just downloaded and cached the Google Drive virus scan warning page, instead of the data file

This issue was already reported
- #3784

and its root cause is a change in the Google Drive service. See:
- #3786

We already fixed it. See:
- #3787

## Describe the bug
I am trying to download some splits of the big_patent dataset, using the following code:

`ds = load_dataset("big_patent", "g", split="validation", download_mode="force_redownload")
`
However, this leads to a FileNotFoundError.

FileNotFoundError                         Traceback (most recent call last)
[<ipython-input-3-8d8a745706a9>](https://localhost:8080/#) in <module>()
      1 from datasets import load_dataset
----> 2 ds = load_dataset("big_patent", "g", split="validation", download_mode="force_redownload")

8 frames
[/usr/local/lib/python3.7/dist-packages/datasets/load.py](https://localhost:8080/#) in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, script_version, **config_kwargs)
   1705         ignore_verifications=ignore_verifications,
   1706         try_from_hf_gcs=try_from_hf_gcs,
-> 1707         use_auth_token=use_auth_token,
   1708     )
   1709

[/usr/local/lib/python3.7/dist-packages/datasets/builder.py](https://localhost:8080/#) in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    593                     if not downloaded_from_gcs:
    594                         self._download_and_prepare(
--> 595                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    596                         )
    597                     # Sync info

[/usr/local/lib/python3.7/dist-packages/datasets/builder.py](https://localhost:8080/#) in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    659         split_dict = SplitDict(dataset_name=self.name)
    660         split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 661         split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    662
    663         # Checksums verification

[/root/.cache/huggingface/modules/datasets_modules/datasets/big_patent/bdefa7c0b39fba8bba1c6331b70b738e30d63c8ad4567f983ce315a5fef6131c/big_patent.py](https://localhost:8080/#) in _split_generators(self, dl_manager)
    123         split_types = ["train", "val", "test"]
    124         extract_paths = dl_manager.extract(
--> 125             {k: os.path.join(dl_path, "bigPatentData", k + ".tar.gz") for k in split_types}
    126         )
    127         extract_paths = {k: os.path.join(extract_paths[k], k) for k in split_types}

[/usr/local/lib/python3.7/dist-packages/datasets/utils/download_manager.py](https://localhost:8080/#) in extract(self, path_or_paths, num_proc)
    282         download_config.extract_compressed_file = True
    283         extracted_paths = map_nested(
--> 284             partial(cached_path, download_config=download_config), path_or_paths, num_proc=num_proc, disable_tqdm=False
    285         )
    286         path_or_paths = NestedDataStructure(path_or_paths)

[/usr/local/lib/python3.7/dist-packages/datasets/utils/py_utils.py](https://localhost:8080/#) in map_nested(function, data_struct, dict_only, map_list, map_tuple, map_numpy, num_proc, types, disable_tqdm)
    260         mapped = [
    261             _single_map_nested((function, obj, types, None, True))
--> 262             for obj in utils.tqdm(iterable, disable=disable_tqdm)
    263         ]
    264     else:

[/usr/local/lib/python3.7/dist-packages/datasets/utils/py_utils.py](https://localhost:8080/#) in <listcomp>(.0)
    260         mapped = [
    261             _single_map_nested((function, obj, types, None, True))
--> 262             for obj in utils.tqdm(iterable, disable=disable_tqdm)
    263         ]
    264     else:

[/usr/local/lib/python3.7/dist-packages/datasets/utils/py_utils.py](https://localhost:8080/#) in _single_map_nested(args)
    194     # Singleton first to spare some computation
    195     if not isinstance(data_struct, dict) and not isinstance(data_struct, types):
--> 196         return function(data_struct)
    197
    198     # Reduce logging to keep things readable in multiprocessing with tqdm

[/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py](https://localhost:8080/#) in cached_path(url_or_filename, download_config, **download_kwargs)
    314     elif is_local_path(url_or_filename):
    315         # File, but it doesn't exist.
--> 316         raise FileNotFoundError(f"Local file {url_or_filename} doesn't exist")
    317     else:
    318         # Something unknown

FileNotFoundError: Local file /root/.cache/huggingface/datasets/downloads/extracted/ad068abb3e11f9f2f5440b62e37eb2b03ee515df9de1637c55cd1793b68668b2/bigPatentData/train.tar.gz doesn't exist

I have tried this in a number of machines, including on Colab, so I think this is not environment dependent.

How do I load the bigPatent dataset?


Udpate index.mdx margins

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3858). All of your documentation changes will be reflected on that endpoint.


Order of dataset changes due to glob.glob.

generic discussion

Generic discussion on the library

I agree using `glob.glob` alone is bad practice because it's not deterministic. Using `sorted` is a nice solution.

Note that the `xglob` function you are referring to in the `streaming_download_manager.py` code just extends `glob.glob` for URLs - we don't change its behavior. That's why it has no `sorted()`

## Describe the bug

After discussion with @lhoestq, just want to mention here that `glob.glob(...)` should always be used in combination with `sorted(...)` to make sure the list of files returned by `glob.glob(...)` doesn't change depending on the OS system.

There are currently multiple datasets that use `glob.glob()` without making use of `sorted(...)` even the streaming download manager (if I'm not mistaken):
https://github.com/huggingface/datasets/blob/c14bfeb4af89da14f870de5ddaa584b08aa08eeb/src/datasets/utils/streaming_download_manager.py#L483


Fix push_to_hub with null images

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3856). All of your documentation changes will be reflected on that endpoint.

This code currently raises an error because of the null image:
```python
import datasets

dataset_dict = { 'name': ['image001.jpg', 'image002.jpg'], 'image': ['cat.jpg', None] }
features = datasets.Features({
    'name': datasets.Value('string'),
    'image': datasets.Image(),
})
dataset = datasets.Dataset.from_dict(dataset_dict, features)
dataset.push_to_hub("username/dataset") # this line produces an error: 'NoneType' object is not subscriptable
```
I fixed this in this PR

TODO:
- [x] add a test


Bad error message when loading private dataset

bug

Something isn't working

We raise the error “ FileNotFoundError: can’t find the dataset” mainly to follow best practice in security (otherwise users could be able to guess what private repositories users/orgs may have)

We can indeed reformulate this and add the "If this is a private repository,..." part !

## Describe the bug

A pretty common behavior of an interaction between the Hub and datasets is the following.
An organization adds a dataset in private mode and wants to load it afterward.

```python
from transformers import load_dataset

ds = load_dataset("NewT5/dummy_data", "dummy")
```

This command then fails with:

```bash
FileNotFoundError: Couldn't find a dataset script at /home/patrick/NewT5/dummy_data/dummy_data.py or any data file in the same directory. Couldn't find 'NewT5/dummy_data' on the Hugging Face Hub either: FileNotFoundError: Dataset 'NewT5/dummy_data' doesn't exist on the Hub
```

**even though** the user has access to the website `NewT5/dummy_data` since she/he is part of the org.

We need to improve the error message here similar to how @sgugger, @LysandreJik and @julien-c have done it for transformers IMO.

## Steps to reproduce the bug

E.g. execute the following code to see the different error messages between `transformes` and `datasets`.

1. Transformers
```python
from transformers import BertModel

BertModel.from_pretrained("NewT5/dummy_model")
```

The error message is clearer here - it gives:

```
OSError: patrickvonplaten/gpt2-xl is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with `use_auth_token` or log in with `huggingface-cli login` and pass `use_auth_token=True`.
```

Let's maybe do the same for datasets? The PR was introduced to `transformers` here:
https://github.com/huggingface/transformers/pull/15261

## Expected results

Better error message

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.4.dev0
- Platform: Linux-5.15.15-76051515-generic-x86_64-with-glibc2.34
- Python version: 3.9.7
- PyArrow version: 6.0.1


load only England English dataset from common voice english dataset

question

Further information is requested

Hi @amanjaiswal777,

First note that the dataset you are trying to load is deprecated: it was the Common Voice dataset release as of Dec 2020.

Currently, Common Voice dataset releases are directly hosted on the Hub, under the Mozilla Foundation organization: https://huggingface.co/mozilla-foundation

For example, to get their latest Common Voice relase (8.0):
- Go to the dataset page and request access permission (Mozilla Foundation requires this for people willing to use their datasets): https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0
- Looking at the dataset card, you can check that data instances have, among other fields, the ones you are interested in: "accent", "age",...
- Then you can load their "en" language dataset as usual, besides passing your authentication token (more info on auth token here: https://huggingface.co/docs/hub/security)
  ```python
  from datasets import load_dataset
  ds_en = load_dataset("mozilla-foundation/common_voice_8_0", "en", use_auth_token=True)
  ```
- Finally, you can filter only the data instances you are interested in (more info on `filter` here: https://huggingface.co/docs/datasets/process#select-and-filter):
  ```python
  ds_england_en = ds_en.filter(lambda item: item["accent"] == "England English")
  ```

Feel free to reopen this issue if you need further assistance.

Hey @albertvillanova trying the same approach as you with the common_voice_16_1 dataset. What I'm trying to do is to filter the valencian accent in the catalan subset. Gave me this error and I have everything it asks for decoding mp3:
![image](https://github.com/huggingface/datasets/assets/96977715/7ec02483-e728-4358-9372-ba74ec1b7fd4)

![image](https://github.com/huggingface/datasets/assets/96977715/c10fcf23-a141-4dba-a88d-89e293acfe67)

training_data = load_dataset("common_voice", "en",split='train[:250]+validation[:250]')
testing_data = load_dataset("common_voice", "en", split="test[:200]")

I'm trying to load only 8% of the English common voice data with accent == "England English." Can somebody assist me with this?

**Typical Voice Accent Proportions:**

- 24% United States English
- 8% England English
- 5% India and South Asia (India, Pakistan, Sri Lanka)
- 3% Australian English
- 3% Canadian English
- 2% Scottish English
- 1% Irish English
- 1% Southern African (South Africa, Zimbabwe, Namibia)
- 1% New Zealand English

Can we replicate this for Age as well?

**Age proportions of the common voice:-**

- 24% 19 - 29
- 14% 30 - 39
- 10% 40 - 49
- 6% < 19
- 4% 50 - 59
- 4% 60 - 69
- 1% 70 – 79


add ontonotes_conll dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3853). All of your documentation changes will be reflected on that endpoint.

The CI fail is unrelated to this dataset, merging :)

# Introduction of the dataset
OntoNotes v5.0 is the final version of OntoNotes corpus, and is a large-scale, multi-genre,
multilingual corpus manually annotated with syntactic, semantic and discourse information.

This dataset is the version of OntoNotes v5.0 extended and used in the CoNLL-2012 shared task
, includes v4 train/dev and v9 test data for English/Chinese/Arabic and corrected version v12 train/dev/test data (English only).

This dataset is widely used in name entity recognition, coreference resolution, and semantic role labeling.

In dataset loading script, I modify and use the code of [AllenNLP/Ontonotes](https://docs.allennlp.org/models/main/models/common/ontonotes/#ontonotes) to read the special conll files while don't get extra package dependency.

# Some workarounds I did

1. task ids
I add tasks that I can't find anywhere `semantic-role-labeling`, `lemmatization`, and `word-sense-disambiguation` to the task category `structure-prediction`, because they are related to "syntax". I feel there is another good name for the task category since some tasks mentioned aren't related to structure, but I have no good idea.

2. `dl_manage.extract`
Since we'll get another zip after unzip the downloaded zip data, I have to use `dl_manager.extract` directly inside `_generate_examples`. But when testing dummy data, `dl_manager.extract` do nothing. So I make a conditional such that it  manually  extract data when testing dummy data.

# Help
Don't know how to fix the building doc error.


Redundant add dataset information and dead link.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3852). All of your documentation changes will be reflected on that endpoint.

> Alternatively, you can follow the steps to [add a dataset](https://huggingface.co/docs/datasets/add_dataset.html) and [share a dataset](https://huggingface.co/docs/datasets/share_dataset.html) in the documentation.

The "add a dataset link" gives 404 Error, and the share_dataset link has changed. I feel this information is redundant/deprecated now since we have a more detailed guide for "How to add a dataset?".


Load audio dataset error

bug

Something isn't working

Hi @lemoner20, thanks for reporting.

I'm sorry but I cannot reproduce your problem:
```python
In [1]: from datasets import load_dataset, load_metric, Audio
   ...: raw_datasets = load_dataset("superb", "ks", split="train")
   ...: print(raw_datasets[0]["audio"])
Downloading builder script: 30.2kB [00:00, 13.0MB/s]
Downloading metadata: 38.0kB [00:00, 16.6MB/s]
Downloading and preparing dataset superb/ks (download: 1.45 GiB, generated: 9.64 MiB, post-processed: Unknown size, total: 1.46 GiB) to .../.cache/huggingface/datasets/superb/ks/1.9.0/fc1f59e1fa54262dfb42de99c326a806ef7de1263ece177b59359a1a3354a9c9...
Downloading data: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.49G/1.49G [00:37<00:00, 39.3MB/s]
Downloading data: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 71.3M/71.3M [00:01<00:00, 36.1MB/s]
Downloading data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:41<00:00, 20.67s/it]
Extracting data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:28<00:00, 14.24s/it]
Dataset superb downloaded and prepared to .../.cache/huggingface/datasets/superb/ks/1.9.0/fc1f59e1fa54262dfb42de99c326a806ef7de1263ece177b59359a1a3354a9c9. Subsequent calls will reuse this data.
{'path': '.../.cache/huggingface/datasets/downloads/extracted/8571921d3088b48f58f75b2e514815033e1ffbd06aa63fd4603691ac9f1c119f/_background_noise_/doing_the_dishes.wav', 'array': array([ 0.        ,  0.        ,  0.        , ..., -0.00592041,
       -0.00405884, -0.00253296], dtype=float32), 'sampling_rate': 16000}
```

Which version of `datasets` are you using? Could you please fill in the environment info requested in the bug report template? You can run the command `datasets-cli env` and copy-and-paste its output below
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:

@albertvillanova Thanks for your reply. The environment info below

## Environment info
- `datasets` version: 1.18.3
- Platform: Linux-4.19.91-007.ali4000.alios7.x86_64-x86_64-with-debian-buster-sid
- Python version: 3.6.12
- PyArrow version: 6.0.1

Thanks @lemoner20,

I cannot reproduce your issue in datasets version 1.18.3 either.

Maybe redownloading the data file may work if you had already cached this dataset previously. Could you please try passing "force_redownload"?
```python
raw_datasets = load_dataset("superb", "ks", split="train", download_mode="force_redownload")

Thanks, @albertvillanova,

I install the python package of **librosa=0.9.1** again, it works now!

@albertvillanova, you can actually reproduce the error if you reach the cell `common_voice_train[0]["path"]` of this [notebook](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Fine_Tune_XLSR_Wav2Vec2_on_Turkish_ASR_with_%F0%9F%A4%97_Transformers.ipynb#scrollTo=_0kRndSvqaKk). Error gets solved after updating the versions of the libraries used in there.

@jvel07, thanks for reporting and finding a solution.

Maybe we could tell @patrickvonplaten about the version pinning issue in his notebook.

Should I update the version of datasets @albertvillanova ?

## Load audio dataset error

Hi, when I load audio dataset following https://huggingface.co/docs/datasets/audio_process and https://github.com/huggingface/datasets/tree/master/datasets/superb,
```
from datasets import load_dataset, load_metric, Audio
raw_datasets = load_dataset("superb", "ks", split="train")
print(raw_datasets[0]["audio"])
```
following errors occur
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-169-3f8253239fa0> in <module>
----> 1 raw_datasets[0]["audio"]

/usr/lib/python3.6/site-packages/datasets/arrow_dataset.py in __getitem__(self, key)
   1924         """Can be used to index columns (by string names) or rows (by integer index or iterable of indices or bools)."""
   1925         return self._getitem(
-> 1926             key,
   1927         )
   1928

/usr/lib/python3.6/site-packages/datasets/arrow_dataset.py in _getitem(self, key, decoded, **kwargs)
   1909         pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
   1910         formatted_output = format_table(
-> 1911             pa_subtable, key, formatter=formatter, format_columns=format_columns, output_all_columns=output_all_columns
   1912         )
   1913         return formatted_output

/usr/lib/python3.6/site-packages/datasets/formatting/formatting.py in format_table(table, key, formatter, format_columns, output_all_columns)
    530     python_formatter = PythonFormatter(features=None)
    531     if format_columns is None:
--> 532         return formatter(pa_table, query_type=query_type)
    533     elif query_type == "column":
    534         if key in format_columns:

/usr/lib/python3.6/site-packages/datasets/formatting/formatting.py in __call__(self, pa_table, query_type)
    279     def __call__(self, pa_table: pa.Table, query_type: str) -> Union[RowFormat, ColumnFormat, BatchFormat]:
    280         if query_type == "row":
--> 281             return self.format_row(pa_table)
    282         elif query_type == "column":
    283             return self.format_column(pa_table)

/usr/lib/python3.6/site-packages/datasets/formatting/formatting.py in format_row(self, pa_table)
    310         row = self.python_arrow_extractor().extract_row(pa_table)
    311         if self.decoded:
--> 312             row = self.python_features_decoder.decode_row(row)
    313         return row
    314

/usr/lib/python3.6/site-packages/datasets/formatting/formatting.py in decode_row(self, row)
    219
    220     def decode_row(self, row: dict) -> dict:
--> 221         return self.features.decode_example(row) if self.features else row
    222
    223     def decode_column(self, column: list, column_name: str) -> list:

/usr/lib/python3.6/site-packages/datasets/features/features.py in decode_example(self, example)
   1320             else value
   1321             for column_name, (feature, value) in utils.zip_dict(
-> 1322                 {key: value for key, value in self.items() if key in example}, example
   1323             )
   1324         }

/usr/lib/python3.6/site-packages/datasets/features/features.py in <dictcomp>(.0)
   1319             if self._column_requires_decoding[column_name]
   1320             else value
-> 1321             for column_name, (feature, value) in utils.zip_dict(
   1322                 {key: value for key, value in self.items() if key in example}, example
   1323             )

/usr/lib/python3.6/site-packages/datasets/features/features.py in decode_nested_example(schema, obj)
   1053     # Object with special decoding:
   1054     elif isinstance(schema, (Audio, Image)):
-> 1055         return schema.decode_example(obj) if obj is not None else None
   1056     return obj
   1057

/usr/lib/python3.6/site-packages/datasets/features/audio.py in decode_example(self, value)
    100                 array, sampling_rate = self._decode_non_mp3_file_like(file)
    101             else:
--> 102                 array, sampling_rate = self._decode_non_mp3_path_like(path)
    103         return {"path": path, "array": array, "sampling_rate": sampling_rate}
    104

/usr/lib/python3.6/site-packages/datasets/features/audio.py in _decode_non_mp3_path_like(self, path)
    143
    144         with xopen(path, "rb") as f:
--> 145             array, sampling_rate = librosa.load(f, sr=self.sampling_rate, mono=self.mono)
    146         return array, sampling_rate
    147

/usr/lib/python3.6/site-packages/librosa/core/audio.py in load(path, sr, mono, offset, duration, dtype, res_type)
    110
    111     y = []
--> 112     with audioread.audio_open(os.path.realpath(path)) as input_file:
    113         sr_native = input_file.samplerate
    114         n_channels = input_file.channels

/usr/lib/python3.6/posixpath.py in realpath(filename)
    392     """Return the canonical path of the specified filename, eliminating any
    393 symbolic links encountered in the path."""
--> 394     filename = os.fspath(filename)
    395     path, ok = _joinrealpath(filename[:0], filename, {})
    396     return abspath(path)

TypeError: expected str, bytes or os.PathLike object, not _io.BufferedReader
```

## Expected results
```
>>> raw_datasets[0]["audio"]
{'array': array([-0.0005188 , -0.00109863,  0.00030518, ...,  0.01730347,
        0.01623535,  0.01724243]),
'path': '/root/.cache/huggingface/datasets/downloads/extracted/bb3a06b491a64aff422f307cd8116820b4f61d6f32fcadcfc554617e84383cb7/bed/026290a7_nohash_0.wav',
'sampling_rate': 16000}
```


[feat] Add tqdm arguments

In this PR, tqdm arguments can be passed to the map() function and such, in order to be more flexible.


Add "Adversarial GLUE" dataset to datasets library

_The documentation is not available anymore as the PR was closed or merged._

@lhoestq can you review when you have some time?

Hi @lhoestq -- thanks so much for your review! I just added the stuff you requested to the README.md, including an example from the dataset, the table of contents, and lots of section headers with "More Information Needed" below. Let me know if there's anything else I need to do!

Feel free to also merge `master` into your branch to get the latest updates for the tests ;)

thanks @lhoestq - just made all the updates you requested!

Adds the Adversarial GLUE dataset: https://adversarialglue.github.io/

```python
>>> import datasets
>>> >>> datasets.load_dataset('adv_glue')
Using the latest cached version of the module from /home/jxm3/.cache/huggingface/modules/datasets_modules/datasets/adv_glue/26709a83facad2830d72d4419dd179c0be092f4ad3303ad0ebe815d0cdba5cb4 (last modified on Mon Mar  7 19:19:48 2022) since it couldn't be found locally at adv_glue., or remotely on the Hugging Face Hub.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jxm3/random/datasets/src/datasets/load.py", line 1657, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/jxm3/random/datasets/src/datasets/load.py", line 1510, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/home/jxm3/random/datasets/src/datasets/builder.py", line 1021, in __init__
    super().__init__(*args, **kwargs)
  File "/home/jxm3/random/datasets/src/datasets/builder.py", line 258, in __init__
    self.config, self.config_id = self._create_builder_config(
  File "/home/jxm3/random/datasets/src/datasets/builder.py", line 337, in _create_builder_config
    raise ValueError(
ValueError: Config name is missing.
Please pick one among the available configs: ['adv_sst2', 'adv_qqp', 'adv_mnli', 'adv_mnli_mismatched', 'adv_qnli', 'adv_rte']
Example of usage:
	`load_dataset('adv_glue', 'adv_sst2')`
>>> datasets.load_dataset('adv_glue', 'adv_sst2')['validation'][0]
Reusing dataset adv_glue (/home/jxm3/.cache/huggingface/datasets/adv_glue/adv_sst2/1.0.0/3719a903f606f2c96654d87b421bc01114c37084057cdccae65cd7bc24b10933)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 604.11it/s]
{'sentence': "it 's an uneven treat that bores fun at the democratic exercise while also examining its significance for those who take part .", 'label': 1, 'idx': 0}
```


NonMatchingChecksumError when checksum is None

Something isn't working

Hi @jxmorris12, thanks for reporting.

The objective of `verify_checksums` is to check that both checksums are equal. Therefore if one is None and the other is non-None, they are not equal, and the function accordingly raises a NonMatchingChecksumError. That behavior is expected.

The question is: how did you generate the expected checksum? Normally, it should not be None. To properly generate it (it is contained in the `dataset_infos.json` file), you should have runned: https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md
```shell
datasets-cli test <your-dataset-folder> --save_infos --all_configs
```

On the other hand, you should take into account that the generation of this file is NOT mandatory for personal/community datasets (we only require it for "canonical" datasets, i.e., datasets added to our library GitHub repository: https://github.com/huggingface/datasets/tree/master/datasets). Therefore, other option would be just to delete the `dataset_infos.json` file. If that file is not present, the function `verify_checksums` is not executed.

Finally, you can circumvent the `verify_checksums` function by passing `ignore_verifications=True` to `load_dataset`:
```python
load_dataset(..., ignore_verifications=True)
```

Thanks @albertvillanova!

That's fine. I did run that command when I was adding a new dataset. Maybe because the command crashed in the middle, the checksum wasn't stored properly. I don't know where the bug is happening. But either (i) `verify_checksums` should properly handle this edge case, where the passed checksum is None or (ii) the `datasets-cli test` shouldn't generate a corrupted dataset_infos.json file.

Just a more high-level thing, I was trying to follow the instructions for adding a dataset in the CONTRIBUTING.md, so if running that command isn't even necessary, that should probably be mentioned in the document, right? But that's somewhat of a moot point, since something isn't working quite right internally if I was able to get into this corrupted state in the first place, just by following those instructions.

Hi @jxmorris12,

Definitely, your `dataset_infos.json` was corrupted (and wrongly contains expected None checksum).

While we further investigate how this can happen and fix it, feel free to delete your `dataset_infos.json` file and recreate it with:
```shell
datasets-cli test <your-dataset-folder> --save_infos --all_configs
```

Also note that `verify_checksum` is working as expected: if it receives a None and and a non-None checksums as input pair, it must raise an exception: they are not equal. That is not a bug.

At a higher level, also note that we are preparing the release of `datasets` version 2.0, and some docs are being updated...

In order to add a dataset, I think the most updated instructions are in our official documentation pages: https://huggingface.co/docs/datasets/share

Thanks for the info. Maybe you can update the contributing.md if it's not up-to-date.

Hi @jxmorris12, we have discovered the bug why `None` checksums wrongly appeared when generating the `dataset_infos.json` file:
- #3892

The fix will be accessible once this PR merged. And we are planning to do our 2.0 release today.

We are also working on updating all our docs for our release today.

Thanks @albertvillanova - congrats on the release!

I ran into the following error when adding a new dataset:

```bash
expected_checksums = {'https://adversarialglue.github.io/dataset/dev.zip': {'checksum': None, 'num_bytes': 40662}}
recorded_checksums = {'https://adversarialglue.github.io/dataset/dev.zip': {'checksum': 'efb4cbd3aa4a87bfaffc310ae951981cc0a36c6c71c6425dd74e5b55f2f325c9', 'num_bytes': 40662}}
verification_name = 'dataset source files'

    def verify_checksums(expected_checksums: Optional[dict], recorded_checksums: dict, verification_name=None):
        if expected_checksums is None:
            logger.info("Unable to verify checksums.")
            return
        if len(set(expected_checksums) - set(recorded_checksums)) > 0:
            raise ExpectedMoreDownloadedFiles(str(set(expected_checksums) - set(recorded_checksums)))
        if len(set(recorded_checksums) - set(expected_checksums)) > 0:
            raise UnexpectedDownloadedFile(str(set(recorded_checksums) - set(expected_checksums)))
        bad_urls = [url for url in expected_checksums if expected_checksums[url] != recorded_checksums[url]]
        for_verification_name = " for " + verification_name if verification_name is not None else ""
        if len(bad_urls) > 0:
            error_msg = "Checksums didn't match" + for_verification_name + ":\n"
>           raise NonMatchingChecksumError(error_msg + str(bad_urls))
E           datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
E           ['https://adversarialglue.github.io/dataset/dev.zip']

src/datasets/utils/info_utils.py:40: NonMatchingChecksumError
```

## Expected results
The dataset downloads correctly, and there is no error.

## Actual results
Datasets library is looking for a checksum of None, and it gets a non-None checksum, and throws an error. This is clearly a bug.


Datasets' cache not re-used

Something isn't working

<s>I think this is because the tokenizer is stateful and because the order in which the splits are processed is not deterministic. Because of that, the hash of the tokenizer may change for certain splits, which causes issues with caching.

To fix this we can try making the order of the splits deterministic for map.</s>

Actually this is not because of the order of the splits, but most likely because the tokenizer used to process the second split is in a state that has been modified by the first split.

Therefore after reloading the first split from the cache, then the second split can't be reloaded since the tokenizer hasn't seen the first split (and therefore is considered a different tokenizer).

This is a bit trickier to fix, we can explore fixing this next week maybe

Sorry didn't have the bandwidth to take care of this yet - will re-assign when I'm diving into it again !

I had this issue with `run_speech_recognition_ctc.py` for wa2vec2.0 fine-tuning. I made a small change and the hash for the function (which includes tokenisation) is now the same before and after pre-porocessing. With the hash being the same, the caching works as intended.

Before:
```
    def prepare_dataset(batch):
        # load audio
        sample = batch[audio_column_name]

        inputs = feature_extractor(sample["array"], sampling_rate=sample["sampling_rate"])
        batch["input_values"] = inputs.input_values[0]
        batch["input_length"] = len(batch["input_values"])

        # encode targets
        additional_kwargs = {}
        if phoneme_language is not None:
            additional_kwargs["phonemizer_lang"] = phoneme_language

        batch["labels"] = tokenizer(batch["target_text"], **additional_kwargs).input_ids

        return batch

    with training_args.main_process_first(desc="dataset map preprocessing"):
        vectorized_datasets = raw_datasets.map(
            prepare_dataset,
            remove_columns=next(iter(raw_datasets.values())).column_names,
            num_proc=num_workers,
            desc="preprocess datasets",
        )
```
After:
```
    def prepare_dataset(batch, feature_extractor, tokenizer):
        # load audio
        sample = batch[audio_column_name]

        inputs = feature_extractor(sample["array"], sampling_rate=sample["sampling_rate"])
        batch["input_values"] = inputs.input_values[0]
        batch["input_length"] = len(batch["input_values"])

        # encode targets
        additional_kwargs = {}
        if phoneme_language is not None:
            additional_kwargs["phonemizer_lang"] = phoneme_language

        batch["labels"] = tokenizer(batch["target_text"], **additional_kwargs).input_ids

        return batch

    pd = lambda batch: prepare_dataset(batch, feature_extractor, tokenizer)

    with training_args.main_process_first(desc="dataset map preprocessing"):
        vectorized_datasets = raw_datasets.map(
            pd,
            remove_columns=next(iter(raw_datasets.values())).column_names,
            num_proc=num_workers,
            desc="preprocess datasets",
        )
```

Not sure why the second one would work and not the first one - they're basically the same with respect to hashing. In both cases the function is hashed recursively, and therefore the feature_extractor and the tokenizer are hashed the same way.

With which tokenizer or feature extractor are you experiencing this behavior ?

Do you also experience this ?
> Tokenization for some subsets are repeated at the 2nd and 3rd run. Starting from the 4th run, everything are loaded from cache.

Thanks ! Hopefully this can be useful to others, and also to better understand and improve hashing/caching

`tokenizer.save_pretrained(training_args.output_dir)` produces a different tokenizer hash when loaded on restart of the script. When I was debugging before I was terminating the script prior to this command, then rerunning.

I compared the tokenizer items on the first and second runs, there are two different items:
1st:
```
('_additional_special_tokens', [AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True)])

...

('tokens_trie', <transformers.tokenization_utils.Trie object at 0x7f4d6d0ddb38>)
```

2nd:
```
('_additional_special_tokens', [AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True)])

...

('tokens_trie', <transformers.tokenization_utils.Trie object at 0x7efc23dcce80>)
```

 On every run of this the special tokens are being added on, and the hash is different on the `tokens_trie`. The increase in the special tokens category could be cleaned, but not sure about the hash for the `tokens_trie`. What might work is that the call for the tokenizer encoding can be translated into a function that strips any unnecessary information out, but that's a guess.

Thanks for investigating ! Does that mean that `save_pretrained`() produces non-deterministic tokenizers on disk ? Or is it `from_pretrained()` which is not deterministic given the same files on disk ?

I think one way to fix this would be to make save/from_pretrained deterministic, or make the pickling of `transformers.tokenization_utils.Trie` objects deterministic (this could be implemented in `transformers`, but maybe let's discuss in an issue in `transformers` before opening a PR)

Late to the party but everything should be deterministic (afaik at least).

But `Trie` is a simple class object, so afaik it's hash function is linked to its `id(self)` so basically where it's stored in memory, so super highly non deterministic. Could that be the issue ?

> But Trie is a simple class object, so afaik it's hash function is linked to its id(self) so basically where it's stored in memory, so super highly non deterministic. Could that be the issue ?

We're computing the hash of the pickle dump of the class so it should be fine, as long as the pickle dump is deterministic

I've ported wav2vec2.0 fine-tuning into Optimum-Graphcore which is where I found the issue. The majority of the script was copied from the Transformers version to keep it similar, [here is the tokenizer loading section from the source](https://github.com/huggingface/transformers/blob/f0982682bd6fd0b438dda79ec45f3a8fac83a985/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py#L531).

In the last comment I have two loaded tokenizers, one from run 'N' of the script and one from 'N+1'. I think what's happening is that when you add special tokens (e.g. PAD and UNK) another AddedToken object is appended when tokenizer is saved regardless of whether special tokens are there already.

If there is a AddedTokens cleanup at load/save this could solve the issue, but then is Trie going to cause hash to be different? I'm not sure.

Which Python version are you using ?

The trie is basically a big dict of dics, so deterministic nature depends on python version:
https://stackoverflow.com/questions/2053021/is-the-order-of-a-python-dictionary-guaranteed-over-iterations

Maybe the investigation is actually not finding the right culprit though (the memory id is changed, but `datasets` is not using that to compare, so maybe we need to be looking within  `datasets` so see where the comparison fails)

Similar issue found on `BartTokenizer`. You can bypass the bug by loading a fresh new tokenizer everytime.

```
    dataset = dataset.map(lambda x: tokenize_func(x, BartTokenizer.from_pretrained(xxx)),
                          num_proc=num_proc, desc='Tokenize')
```

Linking in https://github.com/huggingface/datasets/issues/6179#issuecomment-1701244673 with an explanation.

I got the same problem while using Wav2Vec2CTCTokenizer in a distributed experiment (many processes), and found that the problem was localized in the serialization (pickle dump) of the field `tokenizer.tokens_trie._tokens` (just a python set). I focussed into the set serialization and found it is not deterministic:

```
from datasets.fingerprint import Hasher
from pickle import dumps,loads

# used just once to get a serialized literal
#print(dumps(set("abc")))
serialized = b'\x80\x04\x95\x11\x00\x00\x00\x00\x00\x00\x00\x8f\x94(\x8c\x01a\x94\x8c\x01c\x94\x8c\x01b\x94\x90.'

myset = loads(serialized)
print(f'{myset=} {Hasher.hash(myset)}')
print(serialized == dumps(myset))
```

Every time you run the python script (different processes) you get a random result. @lhoestq does it make any sense?

OK, I assume python's set is just a hash table implementation that uses internally the hash() function. The problem is that python's hash() is not deterministic. I believe that setting the environment variable PYTHONHASHSEED to a fixed value, you can force it to be deterministic. I tried it (file `set_pickle_dump.py`):

```
#!/usr/bin/python3

from datasets.fingerprint import Hasher
from pickle import dumps,loads

# used just once to get a serialized literal (with environment variable PYTHONHASHSEED set to 42)
#print(dumps(set("abc")))
serialized = b'\x80\x04\x95\x11\x00\x00\x00\x00\x00\x00\x00\x8f\x94(\x8c\x01b\x94\x8c\x01c\x94\x8c\x01a\x94\x90.'

myset = loads(serialized)
print(f'{myset=} {Hasher.hash(myset)}')
print(serialized == dumps(myset))
```

and now every run (`PYTHONHASHSEED=42 ./set_pickle_dump.py`) gets tthe same result. I tried then to test it with the tokenizer (file `test_tokenizer.py`):

```
#!/usr/bin/python3
from transformers import AutoTokenizer
from datasets.fingerprint import Hasher

tokenizer = AutoTokenizer.from_pretrained('model')
print(f'{type(tokenizer)=}')
print(f'{Hasher.hash(tokenizer)=}')
```

executed as `PYTHONHASHSEED=42 ./test_tokenizer.py` and now the tokenizer fingerprint is allways the same!

Thanks for reporting. I opened a PR here to propose a fix: https://github.com/huggingface/datasets/pull/6318 and doesn't require setting `PYTHONHASHSEED`

Can you try to install `datasets` from this branch and tell me if it fixes the issue ?

I patched (*) the file `datasets/utils/py_utils.py` and cache is working propperly now. Thanks!

(*): I am running my experiments inside a docker container that depends on `huggingface/transformers-pytorch-gpu:latest`, so pattched the file instead of rebuilding the container from scratch

The OP issue hasn't been fixed, re-opening

I think the Trie()._tokens of PreTrainedTokenizer need to be a sorted set So that the results of `hash_bytes(dumps(tokenizer))` are consistent every time

I believe the issue may be linked to [tokenization_utils.py#L507](https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils.py#L507)，specifically in the line where self.tokens_trie.add(token.content) is called. The function _update_trie appears to modify an unordered set. Consequently, this line:
`value = hash_bytes(dumps(tokenizer.tokens_trie._tokens))`
can lead to inconsistencies when rerunning the code.

This, in turn, results in inconsistent outputs for both `hash_bytes(dumps(function))`  at  [arrow_dataset.py#L3053](https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py#L3053) and
`hasher.update(transform_args[key])` at  [fingerprint.py#L323](https://github.com/huggingface/datasets/blob/main/src/datasets/fingerprint.py#L323)

```
dataset_kwargs = {
    "shard": raw_datasets,
    "function": tokenize_function,
}
transform = format_transform_for_fingerprint(Dataset._map_single)
kwargs_for_fingerprint = format_kwargs_for_fingerprint(Dataset._map_single, (), dataset_kwargs)
kwargs_for_fingerprint["fingerprint_name"] = "new_fingerprint"
new_fingerprint = update_fingerprint(raw_datasets._fingerprint, transform, kwargs_for_fingerprint)
```

Alternatively, does the "dumps" function require separate processing for the set?

We did a fix that does sorting whenever we hash sets. The fix is available on `main` if you want to try it out. We'll do a new release soon :)

Is there a documentation chapter that discusses in which cases you should expect your dataset preprocessing to be cached. Including do's and don'ts for the preprocessing functions? I think Datasets team does amazing job at tacking this issue on their side, but it would be great to have some guidelines on the user side as well.

In our current project we have two cases (text-to-text classification and summarization) and in one of them the cache is sometimes reused when it's not supposed to be reused while in the other it's never used at all 😅

You can find some docs here :)
https://huggingface.co/docs/datasets/about_cache

## Describe the bug
For most tokenizers I have tested (e.g. the RoBERTa tokenizer), the data preprocessing cache are not fully reused in the first few runs, although their `.arrow` cache files are in the cache directory.

## Steps to reproduce the bug
Here is a reproducer. The GPT2 tokenizer works perfectly with caching, but not the RoBERTa tokenizer in this example.
```python
from datasets import load_dataset
from transformers import AutoTokenizer

raw_datasets = load_dataset("wikitext", "wikitext-2-raw-v1")
# tokenizer = AutoTokenizer.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("roberta-base")
text_column_name = "text"
column_names = raw_datasets["train"].column_names

def tokenize_function(examples):
    return tokenizer(examples[text_column_name], return_special_tokens_mask=True)

tokenized_datasets = raw_datasets.map(
    tokenize_function,
    batched=True,
    remove_columns=column_names,
    load_from_cache_file=True,
    desc="Running tokenizer on every text in dataset",
)
```

## Expected results
No tokenization would be required after the 1st run. Everything should be loaded from the cache.

## Actual results
Tokenization for some subsets are repeated at the 2nd and 3rd run. Starting from the 4th run, everything are loaded from cache.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Ubuntu 18.04.6 LTS
- Python version: 3.6.9
- PyArrow version: 6.0.1


Update faiss device docstring

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3846). All of your documentation changes will be reflected on that endpoint.

Following https://github.com/huggingface/datasets/pull/3721 I updated the docstring of the `device` argument of the FAISS related methods of `Dataset`


add RMSE and MAE metrics.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3845). All of your documentation changes will be reflected on that endpoint.

@mariosasko I've reopened it here. Please suggest any changes if required. Thank you.

Thanks for suggestions. :) I have added update the KWARGS_DESCRIPTION for the missing params and also changed RMSE to MSE.
While testing, I noticed that when the input is a list of lists, we get an error :
`TypeError: float() argument must be a string or a number, not 'list'`
Could you suggest the datasets.Value() attribute to support both list of floats and list of lists containing floats ?

Just add a new config to cover that case. You can do this by replacing the current `features` dict with:
```python
features=datasets.Features(
    {
        "predictions": datasets.Sequence(datasets.Value("float")),
        "references": datasets.Sequence(datasets.Value("float")),
    }
    if self.config_name == "multioutput"
    else {
        "predictions": datasets.Value("float"),
        "references": datasets.Value("float"),
     }
),
```
Feel free to suggest a better name for the config than `multioutput`

Also, could you please move the changes to a new branch and open a PR from there (for the 3rd time 😄) because the diff shows changes from unrelated PRs (maybe due to rebasing?).

Thanks for the input, I have added new config to support multi-dimensional lists and updated the examples as well.

Sure. Will do that and open a new PR for these changes.

This PR adds RMSE - Root Mean Squared Error and MAE - Mean Absolute Error to the metrics API.
Both implementations are based on usage of sciket-learn.

Feature request here : Add support for continuous metrics (RMSE, MAE) [#3608](https://github.com/huggingface/datasets/issues/3608)

Please suggest any changes if required. Thank you.


Add rmse and mae metrics.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3844). All of your documentation changes will be reflected on that endpoint.

@dnaveenr This PR is in pretty good shape, so feel free to reopen it.

This PR adds RMSE - Root Mean Squared Error and MAE - Mean Absolute Error to the metrics API.
Both implementations are based on usage of sciket-learn.

Feature request here : Add support for continuous metrics (RMSE, MAE) [#3608](https://github.com/huggingface/datasets/issues/3608)

Any suggestions and changes required will be helpful.


Fix Google Drive URL to avoid Virus scan warning in streaming mode

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3843). All of your documentation changes will be reflected on that endpoint.

Cool ! Looks like it breaks `test_streaming_gg_drive_gzipped` for some reason...

The streaming version of https://github.com/huggingface/datasets/pull/3787.

Fix #3835

CC: @albertvillanova


Align IterableDataset.shuffle with Dataset.shuffle

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3842). All of your documentation changes will be reflected on that endpoint.

We should also add `generator` as a param to `shuffle` to fully align the APIs, no?

I added the `generator` argument.

I had to make a few other adjustments to make it work. In particular when you call `set_epoch()` on a streaming dataset, it updates the underlying random generator by using a new effective seed. The effective seed is generated using the previous generator and the epoch number.

From #3444 , Dataset.shuffle can have the same API than IterableDataset.shuffle (i.e. in streaming mode).

Currently you can pass an optional seed to both if you want, BUT currently IterableDataset.shuffle always requires a buffer_size, used for approximate shuffling. I propose using a reasonable default value (maybe 1000) instead.

In this PR, I set the default `buffer_size` value to 1,000, and I reorder the `IterableDataset.shuffle` arguments to match `Dataset.shuffle`, i.e. making `seed` the first argument.


Pyright reportPrivateImportUsage when `from datasets import load_dataset`

Something isn't working

Hi!

This issue stems from `datasets` having `py.typed` defined (see https://github.com/microsoft/pyright/discussions/3764#discussioncomment-3282142) - to avoid it, we would either have to remove `py.typed` (added to be compliant with PEP-561) or export the names with `__all__`/`from .submodule import name as name`.

Transformers is fine as it no longer has `py.typed` (removed in https://github.com/huggingface/transformers/pull/18485)

WDYT @lhoestq @albertvillanova @polinaeterna

@sgugger's point makes sense - we should either be "properly typed" (have py.typed + mypy tests) or drop `py.typed` as Transformers did (I like this option better).

(cc @Wauplin since `huggingface_hub` has the same issue.)

I'm fine with dropping it, but autotrain people won't be happy @SBrandeis

> (cc @Wauplin since huggingface_hub has the same issue.)

Hmm maybe we have the same issue but I haven't been able to reproduce something similar to `"load_dataset" is not exported from module "datasets"`  message (using VSCode+Pylance -that is powered by Pyright). `huggingface_hub` contains a `py.typed` file but the package itself is actually typed. We are running `mypy` in our CI tests since ~3 months and so far it seems to be ok. But happy to change if it causes some issues with linters.

Also the top-level [`__init__.py`](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/__init__.py) is quite different in `hfh` than `datasets` (at first glance). We have a section at the bottom to import all high level methods/classes in a `if TYPE_CHECKING` block.

@Wauplin I only get the error if I use Pyright's CLI tool or the Pyright extension (not sure why, but Pylance also doesn't report this issue on my machine)

> Also the top-level [`__init__.py`](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/__init__.py) is quite different in `hfh` than `datasets` (at first glance). We have a section at the bottom to import all high level methods/classes in a `if TYPE_CHECKING` block.

I tried to fix the issue with `TYPE_CHECKING`, but it still fails if `py.typed` is present.

@mariosasko thank for the tip. I have been able to reproduce the issue as well. I would be up for including a (huge) static `__all__` variable in the `__init__.py` (since the file is already generated automatically in `hfh`) but honestly I don't think it's worth the hassle.

I'll delete the `py.typed` file in `huggingface_hub` to be consistent between HF libraries. I opened a PR here: https://github.com/huggingface/huggingface_hub/pull/1329

I am getting this error in google colab today:

![image](https://user-images.githubusercontent.com/3464445/219883967-c7193a23-0388-4ba3-b00c-a53883fb6512.png)

The code runs just fine too.

## Describe the bug
Pyright complains about module not exported.

## Steps to reproduce the bug
Use an editor/IDE with Pyright Language server with default configuration:
```python
from datasets import load_dataset
```

## Expected results
No complain from Pyright

## Actual results
Pyright complain below:
```
`load_dataset` is not exported from module "datasets"
 Import from "datasets.load" instead [reportPrivateImportUsage]
```
Importing from `datasets.load` does indeed solves the problem but I believe importing directly from top level `datasets` is the intended usage per the documentation.

## Environment info
- `datasets` version: 1.18.3
- Platform: macOS-12.2.1-arm64-arm-64bit
- Python version: 3.9.10
- PyArrow version: 7.0.0


Pin responses to fix CI for Windows

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3840). All of your documentation changes will be reflected on that endpoint.

Temporarily fix CI for Windows by pinning `responses`.

See: https://app.circleci.com/pipelines/github/huggingface/datasets/10292/workflows/83de4a55-bff7-43ec-96f7-0c335af5c050/jobs/63355

Fix: #3839


CI is broken for Windows

Something isn't working

## Describe the bug
See: https://app.circleci.com/pipelines/github/huggingface/datasets/10292/workflows/83de4a55-bff7-43ec-96f7-0c335af5c050/jobs/63355

```
___________________ test_datasetdict_from_text_split[test] ____________________
[gw0] win32 -- Python 3.7.11 C:\tools\miniconda3\envs\py37\python.exe

split = 'test'
text_path = 'C:\\Users\\circleci\\AppData\\Local\\Temp\\pytest-of-circleci\\pytest-0\\popen-gw0\\data6\\dataset.txt'
tmp_path = WindowsPath('C:/Users/circleci/AppData/Local/Temp/pytest-of-circleci/pytest-0/popen-gw0/test_datasetdict_from_text_spl7')

    @pytest.mark.parametrize("split", [None, NamedSplit("train"), "train", "test"])
    def test_datasetdict_from_text_split(split, text_path, tmp_path):
        if split:
            path = {split: text_path}
        else:
            split = "train"
            path = {"train": text_path, "test": text_path}
        cache_dir = tmp_path / "cache"
        expected_features = {"text": "string"}
>       dataset = TextDatasetReader(path, cache_dir=cache_dir).read()

tests\io\test_text.py:118:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\tools\miniconda3\envs\py37\lib\site-packages\datasets\io\text.py:43: in read
    use_auth_token=use_auth_token,
C:\tools\miniconda3\envs\py37\lib\site-packages\datasets\builder.py:588: in download_and_prepare
    self._download_prepared_from_hf_gcs(dl_manager.download_config)
C:\tools\miniconda3\envs\py37\lib\site-packages\datasets\builder.py:630: in _download_prepared_from_hf_gcs
    reader.download_from_hf_gcs(download_config, relative_data_dir)
C:\tools\miniconda3\envs\py37\lib\site-packages\datasets\arrow_reader.py:260: in download_from_hf_gcs
    downloaded_dataset_info = cached_path(remote_dataset_info.replace(os.sep, "/"))
C:\tools\miniconda3\envs\py37\lib\site-packages\datasets\utils\file_utils.py:301: in cached_path
    download_desc=download_config.download_desc,
C:\tools\miniconda3\envs\py37\lib\site-packages\datasets\utils\file_utils.py:560: in get_from_cache
    headers=headers,
C:\tools\miniconda3\envs\py37\lib\site-packages\datasets\utils\file_utils.py:476: in http_head
    max_retries=max_retries,
C:\tools\miniconda3\envs\py37\lib\site-packages\datasets\utils\file_utils.py:397: in _request_with_retry
    response = requests.request(method=method.upper(), url=url, timeout=timeout, **params)
C:\tools\miniconda3\envs\py37\lib\site-packages\requests\api.py:61: in request
    return session.request(method=method, url=url, **kwargs)
C:\tools\miniconda3\envs\py37\lib\site-packages\requests\sessions.py:529: in request
    resp = self.send(prep, **send_kwargs)
C:\tools\miniconda3\envs\py37\lib\site-packages\requests\sessions.py:645: in send
    r = adapter.send(request, **kwargs)
C:\tools\miniconda3\envs\py37\lib\site-packages\responses\__init__.py:840: in unbound_on_send
    return self._on_request(adapter, request, *a, **kwargs)
C:\tools\miniconda3\envs\py37\lib\site-packages\responses\__init__.py:780: in _on_request
    match, match_failed_reasons = self._find_match(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <responses.RequestsMock object at 0x000002048AD70588>
request = <PreparedRequest [HEAD]>

    def _find_first_match(self, request):
        match_failed_reasons = []
>       for i, match in enumerate(self._matches):
E       AttributeError: 'RequestsMock' object has no attribute '_matches'

C:\tools\miniconda3\envs\py37\lib\site-packages\moto\core\models.py:289: AttributeError
```


Add a data type for labeled images (image segmentation)

enhancement

New feature or request

It might be a mix of Image and ClassLabel, and the color palette might be generated automatically.

---

### Example

every pixel in the images of the annotation column (in https://huggingface.co/datasets/scene_parse_150) has a value that gives its class, and the dataset itself is associated with a color palette (eg https://github.com/open-mmlab/mmsegmentation/blob/98a353b674c6052d319e7de4e5bcd65d670fcf84/mmseg/datasets/ade.py#L47) that maps every class with a color.

So we might want to render the image as a colored image instead of a black and white one.

<img width="785" alt="156741519-fbae6844-2606-4c28-837e-279d83d00865" src="https://user-images.githubusercontent.com/1676121/157005263-7058c584-2b70-465a-ad94-8a982f726cf4.png">

See https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/core/features/labeled_image.py for reference in Tensorflow


Release: 1.18.4


Logo float left

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3836). All of your documentation changes will be reflected on that endpoint.

Weird, the logo doesn't seem to be floating on my side (using Chrome) at https://huggingface.co/docs/datasets/master/en/index

https://huggingface.co/docs/datasets/index

The needed css change from moon-landing just got deployed

<img width="1000" alt="Screenshot 2022-03-07 at 09 35 29" src="https://user-images.githubusercontent.com/11827707/156996422-339ba43e-932b-4849-babf-9321cb99c922.png">


The link given on the gigaword does not work

bug

Something isn't working

## Dataset viewer issue for '*name of the dataset*'

**Link:** *link to the dataset viewer page*

*short description of the issue*

Am I the one who added this dataset ? Yes-No


Fix dead dataset scripts creation link.

Previous link gives 404 error. Updated with a new dataset scripts creation link.


Small typos in How-to-train tutorial.


Making Hugging Face the place to go for Graph NNs datasets

dataset request

Requesting to add a new dataset

graph

Datasets for Graph Neural Networks

It will be indeed really great to add support to GNN datasets. Big :+1: for this initiative.

@napoles-uach identifies the [TUDatasets](https://chrsmrrs.github.io/datasets/) (A collection of benchmark datasets for graph classification and regression).

Added to the Tasks in the initial issue.

Thanks Omar, that is a great collection!

Great initiative! Let's keep this issue for these 3 datasets, but moving forward maybe let's create a new issue per dataset :rocket: great work @napoles-uach and @omarespejel!

Let's make Hugging Face Datasets the central hub for GNN datasets :)

**Motivation**. Datasets are currently quite scattered and an open-source central point such as the Hugging Face Hub would be ideal to support the growth of the GNN field.

What are some datasets worth integrating into the Hugging Face hub?

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Special thanks to @napoles-uach for his collaboration on identifying the first ones:

- [ ] [SNAP-Stanford OGB Datasets](https://github.com/snap-stanford/ogb).
- [ ] [SNAP-Stanford Pretrained GNNs Chemistry and Biology Datasets](https://github.com/snap-stanford/pretrain-gnns).
- [ ] [TUDatasets](https://chrsmrrs.github.io/datasets/) (A collection of benchmark datasets for graph classification and regression)

cc @osanseviero


when using to_tf_dataset with shuffle is true, not all completed batches are made

bug

Something isn't working

Hi @greenned, this is expected behaviour for `to_tf_dataset`. By default, we drop the smaller 'remainder' batch during training (i.e. when `shuffle=True`). If you really want to keep that batch, you can set `drop_remainder=False` when calling `to_tf_dataset()`.

@Rocketknight1 Oh, thank you. I didn't get **drop_remainder** Have a nice day!

## Describe the bug
when converting a dataset to tf_dataset by using to_tf_dataset with shuffle true, the remainder is not converted to one batch

## Steps to reproduce the bug
this is the sample code below
https://colab.research.google.com/drive/1_oRXWsR38ElO1EYF9ayFoCU7Ou1AAej4?usp=sharing

## Expected results
regardless of shuffle is true or not, 67 rows dataset should be 5 batches when batch size is 16.

## Actual results
4 batches

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 6.0.1


Got error when load cnn_dailymail dataset

Was able to reproduce the issue on Colab; full logs below.

```
---------------------------------------------------------------------------
NotADirectoryError                        Traceback (most recent call last)
[<ipython-input-2-39967739ba7f>](https://localhost:8080/#) in <module>()
      1 import datasets
      2
----> 3 train_data = datasets.load_dataset("cnn_dailymail", "3.0.0", split="train")

5 frames
[/usr/local/lib/python3.7/dist-packages/datasets/load.py](https://localhost:8080/#) in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, script_version, **config_kwargs)
   1705         ignore_verifications=ignore_verifications,
   1706         try_from_hf_gcs=try_from_hf_gcs,
-> 1707         use_auth_token=use_auth_token,
   1708     )
   1709

[/usr/local/lib/python3.7/dist-packages/datasets/builder.py](https://localhost:8080/#) in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    593                     if not downloaded_from_gcs:
    594                         self._download_and_prepare(
--> 595                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    596                         )
    597                     # Sync info

[/usr/local/lib/python3.7/dist-packages/datasets/builder.py](https://localhost:8080/#) in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    659         split_dict = SplitDict(dataset_name=self.name)
    660         split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 661         split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    662
    663         # Checksums verification

[/root/.cache/huggingface/modules/datasets_modules/datasets/cnn_dailymail/3cb851bf7cf5826e45d49db2863f627cba583cbc32342df7349dfe6c38060234/cnn_dailymail.py](https://localhost:8080/#) in _split_generators(self, dl_manager)
    253     def _split_generators(self, dl_manager):
    254         dl_paths = dl_manager.download_and_extract(_DL_URLS)
--> 255         train_files = _subset_filenames(dl_paths, datasets.Split.TRAIN)
    256         # Generate shared vocabulary
    257

[/root/.cache/huggingface/modules/datasets_modules/datasets/cnn_dailymail/3cb851bf7cf5826e45d49db2863f627cba583cbc32342df7349dfe6c38060234/cnn_dailymail.py](https://localhost:8080/#) in _subset_filenames(dl_paths, split)
    154     else:
    155         logger.fatal("Unsupported split: %s", split)
--> 156     cnn = _find_files(dl_paths, "cnn", urls)
    157     dm = _find_files(dl_paths, "dm", urls)
    158     return cnn + dm

[/root/.cache/huggingface/modules/datasets_modules/datasets/cnn_dailymail/3cb851bf7cf5826e45d49db2863f627cba583cbc32342df7349dfe6c38060234/cnn_dailymail.py](https://localhost:8080/#) in _find_files(dl_paths, publisher, url_dict)
    133     else:
    134         logger.fatal("Unsupported publisher: %s", publisher)
--> 135     files = sorted(os.listdir(top_dir))
    136
    137     ret_files = []

NotADirectoryError: [Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b/cnn/stories'
```

Hi @jon-tow, thanks for reporting. And hi @dynamicwebpaige, thanks for your investigation.

This issue was already reported
- #3784

and its root cause is a change in the Google Drive service. See:
- #3786

We have already fixed it. See:
- #3787

We are planning to make a patch release today (indeed, we were planning to do it last Friday).

In the meantime, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```

CC: @lhoestq

When using datasets.load_dataset method to load cnn_dailymail dataset,  got error as below:

- windows os: FileNotFoundError: [WinError 3] 系统找不到指定的路径。: 'D:\\SourceCode\\DataScience\\HuggingFace\\Data\\downloads\\1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b\\cnn\\stories'
- google colab: NotADirectoryError: [Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b/cnn/stories'

The code is to load dataset:
windows os:
```
from datasets import load_dataset

dataset = load_dataset("cnn_dailymail", "3.0.0", cache_dir="D:\\SourceCode\\DataScience\\HuggingFace\\Data")

```
google colab:
```
import datasets

train_data = datasets.load_dataset("cnn_dailymail", "3.0.0", split="train")
```


[📄 Docs] Create a `datasets` performance guide.

enhancement

New feature or request

Hi ! Yes this is definitely something we'll explore, since optimizing processing pipelines can be challenging and because performance is key here: we want anyone to be able to play with large-scale datasets more easily.

I think we'll start by documenting the performance of the dataset transforms we provide, and then we can have some tools to help debugging/optimizing them

## Brief Overview
Downloading, saving, and preprocessing large datasets from the `datasets` library can often result in [performance bottlenecks](https://github.com/huggingface/datasets/issues/3735). These performance snags can be challenging to identify and to debug, especially for users who are less experienced with building deep learning experiments.

## Feature Request
Could we create a performance guide for using `datasets`, similar to:
* [Better performance with the `tf.data` API](https://github.com/huggingface/datasets/issues/3735)
* [Analyze `tf.data` performance with the TF Profiler](https://www.tensorflow.org/guide/data_performance_analysis)

This performance guide should detail practical options for improving performance with `datasets`, and enumerate any common best practices. It should also show how to use tools like the PyTorch Profiler or the TF Profiler to identify any performance bottlenecks (example below).

![image](https://user-images.githubusercontent.com/3712347/156859152-a3cb9565-3ec6-4d39-8e77-56d0a75a4954.png)

## Related Issues
* [wiki_dpr pre-processing performance #1670](https://github.com/huggingface/datasets/issues/1670)
* [Adjusting chunk size for streaming datasets #3499](https://github.com/huggingface/datasets/issues/3499)
* [how large datasets are handled under the hood #1004](https://github.com/huggingface/datasets/issues/1004)
* [using map on loaded Tokenizer 10x - 100x slower than default Tokenizer? #1830](https://github.com/huggingface/datasets/issues/1830)
* [Best way to batch a large dataset? #315](https://github.com/huggingface/datasets/issues/315)
* [Saving processed dataset running infinitely #1911](https://github.com/huggingface/datasets/issues/1911)


The Pile's _FEATURE spec seems to be incorrect

bug

Something isn't working

Hi @dlwh, thanks for reporting.

Please note, that the source data files for "all" config are different from the other configurations.

The "all" config contains the official Pile data files, from https://mystic.the-eye.eu/public/AI/pile/
All data examples contain a "meta" dict with a single "pile_set_name" key:
```python
In [1]: from datasets import load_dataset
        ds = load_dataset("the_pile", "all", split="train", streaming=True)
        item = next(iter(ds))
Downloading builder script: 9.09kB [00:00, 4.42MB/s]

In [3]: item["meta"]
Out[3]: {'pile_set_name': 'Pile-CC'}
```

On the other hand, all the other subset configs data files come from the Pile preliminary components directory: https://mystic.the-eye.eu/public/AI/pile_preliminary_components/
For theses components, the "meta" field may have different keys depending on the subset: "id", "language", "pmid",... Because of that, if we had kept the `dict` data format for the "meta" field, we would have an error when trying to concatenate different subsets, whose "meta" keys are not identical. In order to avoid that, the "meta" field is cast to `str` in all these cases, so that there is no incompatibility in their "meta" data type when concatenating.

You can check, for example, that for "pubmed_central" the "meta" field is cast to `str`:
```python
In [4]: from datasets import load_dataset
        ds = load_dataset("the_pile", "pubmed_central", split="train", streaming=True)
        item = next(iter(ds))

In [5]: item["meta"]
Out[5]: "{'id': 'PMC6071596'}"
```

Feel free to reopen this issue if you have further questions.

## Describe the bug
If you look at https://huggingface.co/datasets/the_pile/blob/main/the_pile.py:

For "all"
*  the pile_set_name is never set for data
* there's actually an id field inside of "meta"

For subcorpora pubmed_central and hacker_news:
* the meta is specified to be a string, but it's actually a dict with an id field inside.

## Steps to reproduce the bug

## Expected results
Feature spec should match the data I'd think?

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version: