Source links broken

Something isn't working

Thanks for reporting @qqaatw.

@mishig25 @sgugger do you think this can be tweaked in the new doc framework?
- From: https://github.com/huggingface/datasets/blob/v2.0.0/
- To: https://github.com/huggingface/datasets/blob/2.0.0/

@qqaatw thanks a lot for notifying about this issue!

in comparison, transformers tags start with `v` like [this one](https://github.com/huggingface/transformers/blob/v4.17.0/src/transformers/models/bert/configuration_bert.py#L54).

Therefore, we have to do one of 2 options below:
1. Make necessary changes on doc-builder side
OR
2. Make [datasets tags](https://github.com/huggingface/datasets/tags) start with `v`, just like [transformers](https://github.com/huggingface/transformers/tags) (so that tag naming can be consistent amongst hf repos)

I'll let you decide @albertvillanova @lhoestq @sgugger

I think option 2 is the easiest and would provide harmony in the HF ecosystem but we can also add a doc config parameter to decide whether the default version has a v or not if `datasets` folks prefer their tags without a v :-)

For me it is OK to conform to the rest of libraries and tag/release with a preceding "v", rather than adding an extra argument to the doc builder just for `datasets`.

Let me know if it is also OK for you @lhoestq.

https://github.com/huggingface/doc-build/commit/f41c1e8ff900724213af4c75d287d8b61ecf6141

hotfix so that `datasets` docs source button works correctly on hf.co/docs/datasets

We could add a tag for each release without a 'v' but it could be confusing on github to see both tags `v2.0.0` and `2.0.0` IMO (not sure if many users check them though). Removing the tags without 'v' would break our versioning for github datasets: the library looks for dataset scripts at the URLs like `https://raw.githubusercontent.com/huggingface/datasets/{revision}/datasets/{path}/{name}` where `revision` is equal to `datasets.__version__` (which doesn't start with a 'v') for all released versions of `datasets`.

I think we could just have a parameter for the documentation - and having different URLs schemes for the source links that the users don't even see (they simply click on a button) is probably fine

This is done in #3943 to go along with [doc-builder#146](https://github.com/huggingface/doc-builder/pull/146).

Note that this will only work for future versions, so once those two are merged, the actual v2.0.0 doc should be fixed. The easiest is to cherry-pick this commit on the v2.0.0 release branch (or on a new branch created from the 2.0.0 tag, with a name that triggers the doc building job, for instance v2.0.0-release)

## Describe the bug

The source links of v2.0.0 docs are broken:

For exmaple, clicking the source button of this [class](https://huggingface.co/docs/datasets/v2.0.0/en/package_reference/main_classes#datasets.ClassLabel) will direct users to `https://github.com/huggingface/datasets/blob/v2.0.0/src/datasets/features/features.py#L747`

here, the `v2.0.0` should be `2.0.0`.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
```

## Expected results

Redirecting to this link: `https://github.com/huggingface/datasets/blob/2.0.0/src/datasets/features/features.py#L747`

## Actual results
Described above.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:


Avoid info log messages from transformers in FrugalScore metric

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3938). All of your documentation changes will be reflected on that endpoint.

Fix #3928.


Missing languages in lvwerra/github-code dataset

Dataset discussion

Discussions on the datasets

Thanks for contacting @Eytan-S.

I think @lvwerra could better answer this.

That seems to be an oversight - I originally planned to include them in the dataset and for some reason they were in the list of languages but not in the query. Since there is an issue with the deduplication step I'll rerun the pipeline anyway and will double check the query.

Thanks for reporting this @Eytan-S!

Hi,

I'm working with the github-code dataset. First of all, thank you for creating this amazing dataset!
I've noticed that two languages are missing from the dataset: TypeScript and Scala.
Looks like they're also omitted from the query you used to get the original code.
Are there any plans to add them in the future?

Thanks!


Fix Wikipedia version and re-add tests

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3936). All of your documentation changes will be reflected on that endpoint.

To keep backward compatibility when loading using "wikipedia" dataset ID (https://huggingface.co/datasets/wikipedia), we have created the pre-processed data for the same languages we were offering before, but with updated date "20220301":
- de
- en
- fr
- frr
- it
- simple

These pre-processed data can be accessed, e.g.:
```python
ds = load_dataset("wikipedia", "20220301.frr", split="train")
```

The next step will be to offer the pre-processed data for many other languages, but when loading using "wikimedia/wikipedia": https://huggingface.co/datasets/wikimedia/wikipedia


Create MAUVE metric card

_The documentation is not available anymore as the PR was closed or merged._

Proposing a MAUVE metric card


Update README.md

Fixing missing triple quote


Create SARI metric card

SARI metric card! (do we have an expert in text simplification to validate?.. :sweat_smile: )


Add align_labels_with_mapping docs

documentation

Improvements or additions to documentation

This PR documents the `align_labels_with_mapping` function to ensure predicted labels are aligned with the dataset, or to assign a different mapping of labels to ids (requested by @mariosasko 🎉 ).

For this specific code sample, the current dataset has a `mixed` label that the original [dataset](https://huggingface.co/datasets/poem_sentiment#data-fields) didn't. Is there a way to remove this label so it is completely aligned with the original dataset mappings? Otherwise, I'll just leave it as it is.


Create README.md

Creating a README for IndicGLUE

cc @mcmillanmajora for fact checking in terms of languages (also, are there any limitations of the dataset or eval metric that I'm not aware of?)


Load a local dataset twice

bug

Something isn't working

Hi @caush, thanks for reporting:

In order to load local CSV files, you can use our "csv" loading script: https://huggingface.co/docs/datasets/loading#csv
```python
dataset = load_dataset("csv", data_files=["data/file1.csv", "data/file2.csv"])
```
OR:
```python
dataset = load_dataset("csv", data_dir="data")
```

Alternatively, you may also use:
```python
dataset = load_dataset("data")

## Describe the bug
Load a local "dataset" composed of two csv files twice.

## Steps to reproduce the bug
Put the two joined files in a repository named "Data".
Then in python:

import datasets as ds
ds.load_dataset('Data', data_files = {'file1.csv', 'file2.csv'})

## Expected results
Should give something like (because files have only one data row):
Title, clicks
Truc et astuce, 123
Machin, 12

## Actual results
Gives
Title, clicks
Truc et astuce, 123
Machin, 12
Truc et astuce, 123
Machin, 12

## Environment info
[file1.csv](https://github.com/huggingface/datasets/files/8256322/file1.csv)
[file2.csv](https://github.com/huggingface/datasets/files/8256323/file2.csv)
- `datasets` version: 2.0.0
- Platform: Linux-5.4.0-65-generic-x86_64-with-glibc2.10
- Python version: 3.8.12
- PyArrow version: 7.0.0
- Pandas version: 1.4.1


Frugal score deprecations

Something isn't working

Hi @Ierezell, thanks for reporting.

I'm making a PR to suppress those logs from the terminal.

## Describe the bug
The frugal score returns a really verbose output with warnings that can be easily changed.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
from datasets.load import load_metric

frugal = load_metric("frugalscore")
frugal.compute(predictions=["Do you like spinachis"], references=["Do you like spinach"])
```

## Expected results
A clear and concise description of the expected results.
```
{'scores': [0.9946]}
```

## Actual results
Specify the actual results or traceback.
```
PyTorch: setting up devices
The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 864.09ba/s]
Using amp half precision backend
The following columns in the test set  don't have a corresponding argument in `BertForSequenceClassification.forward` and have been ignored: sentence2, sentence1. If sentence2, sentence1 are not expected by `BertForSequenceClassification.forward`,  you can safely ignore this message.
***** Running Prediction *****
  Num examples = 1
  Batch size = 64
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 4644.85it/s]
 {'scores': [0.9946]}
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.17.0
- Platform: Linux-5.13.0-30-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 7.0.0


Update main readme

The main readme was still focused on text datasets - I extended it by mentioning that we also support image and audio datasets


Doc maintenance

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3926). All of your documentation changes will be reflected on that endpoint.

This PR adds some minor maintenance to the docs. The main fix is properly linking to pages in the callouts because some of the links would just redirect to a non-existent section on the same page.


Fix main_classes docs index

Hmm it's still not good
![image](https://user-images.githubusercontent.com/42851186/158429361-e19ce25b-c259-4ded-8473-075deafdbb96.png)

any idea what could cause this ?

Currently the `main_classes` documentation has a wrong index. I believe this comes from issues in the examples of the Translation feature types

![image](https://user-images.githubusercontent.com/42851186/158426345-2ee1ceef-ddf3-4a6f-a93e-d1a8f38a44f5.png)


Document cases for github datasets

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3924). All of your documentation changes will be reflected on that endpoint.

In general we recommend adding the new dataset under a username or organization in the Hugging Face Hub at [hf.co/datasets](hf.co/datasets), but users can still add a dataset on github in some cases.

I added a paragraph in the documentation to explain in which cases it can make more sense to open a PR on github:

- when you need the dataset to be reviewed
- when you need long-term maintenance from the HF team
- when there’s no clear org name / namespace that you can put the dataset under


Add methods to IterableDatasetDict

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3923). All of your documentation changes will be reflected on that endpoint.

Is this feature stale or needs any help to it ? If so I can quickly send a PR. Thanks

CC : @lhoestq, @albertvillanova

These features have been merged and are already available, thanks :)

Hello @lhoestq, I see that `IterableDataset` doesn't allow features like `take`, `len`, `slice`  which can enable a lot of stuffs. Is it worth an addition ? Or is it intended that they didn't have those features ?

IterableDataset objects don't have `len` or `slice` because they can be possibly unbounded (you don't know in advance how many items they contain). Though IterableDataset.take and IterableDataset.skip do exist.

Following the new methods added in #3826 and https://github.com/huggingface/datasets/pull/3862 I added several methods to IterableDatasetDict:
    - map
    - filter
    - shuffle
    - with_format
    - cast
    - cast_column
    - remove_columns
    - rename_column
    - rename_columns


Fix NonMatchingChecksumError in MultiWOZ 2.2 dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3922). All of your documentation changes will be reflected on that endpoint.

Unrelated CI test failure. This PR can be merged.

Fix #2957


Fix NonMatchingChecksumError in CRD3 dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3921). All of your documentation changes will be reflected on that endpoint.

Unrelated test failure. This PR can be merged.

Fix #3051


'datasets.features' is not a package

Hi @Arij-Aladel,

You are using a very old version of our library `datasets`: 1.8.0
Current version is 2.0.0 (and the previous one was 1.18.4)

Please, try to update `datasets` library and check if the problem persists:
```shell
/env/bin/pip install -U datasets

The problem I can no I have build my project on this version and old version on transformers. I have preprocessed the data again to use it. Thank for your reply

@albertvillanova
python 3.9
os: ubuntu 20.04

In conda environment

torch installed by

```/env/bin/pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html```
datasets package is installed by
```
/env/bin/pip  install datasets==1.8.0

```
During runing the code I have this error

```
[6]<stderr>:  File "/home/arij/Memory-transformer-with-hierarchical-attention_MLM/env/lib/python3.9/site-packages/torch/serialization.py", line 875, in find_class
[6]<stderr>:    return super().find_class(mod_name, name)
[6]<stderr>:ModuleNotFoundError: No module named 'datasets.features.features'; 'datasets.features' is not a package

```
precisely this error appears when

torch.load('data_file.pt')
```

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/arij/Memory-transformer-with-hierarchical-attention_MLM/env/lib/python3.9/site-packages/torch/serialization.py", line 607, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "/home/arij/Memory-transformer-with-hierarchical-attention_MLM/env/lib/python3.9/site-packages/torch/serialization.py", line 882, in _load
    result = unpickler.load()
  File "/home/arij/Memory-transformer-with-hierarchical-attention_MLM/env/lib/python3.9/site-packages/torch/serialization.py", line 875, in find_class
    return super().find_class(mod_name, name)
ModuleNotFoundError: No module named 'datasets.features.features'; 'datasets.features' is not a package
```

Why I am getting this error?


Something isn't working

You are likely trying to get the `features` from a `DatasetDict`, a dictionary containing `Datasets`. You probably first want to index into a particular split from your `DatasetDict` i.e. `dataset['train'].features`.

For example

```python
ds = load_dataset('mnist')
ds.features
```
Returns
```python
---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

[<ipython-input-39-791c1f9df6c2>](https://localhost:8080/#) in <module>()
----> 1 ds.features

AttributeError: 'DatasetDict' object has no attribute 'features'
```

If we look at the dataset variable, we see it is a `DatasetDict`:

```python
print(ds)
DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 60000
    })
    test: Dataset({
        features: ['image', 'label'],
        num_rows: 10000
    })
})
```

We can grab the features from a split by indexing into `train`:
```python
ds['train'].features
{'image': Image(decode=True, id=None),
 'label': ClassLabel(num_classes=10, names=['0', '1', '2', '3', '4', '5', '6', '7', '8', '9'], id=None)}
```

Hope that helps

## Describe the bug
Receiving the error when trying to check for Dataset features

## Steps to reproduce the bug
from datasets import Dataset
dataset = Dataset.from_pandas(df[['id', 'words', 'bboxes', 'ner_tags', 'image_path']])

dataset.features

## Expected results
A clear and concise description of the expected results.

## Actual results
Getting the following errror

AttributeError: 'DatasetDict' object has no attribute 'features'

## Environment info
Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 1.18.4
- Platform: Linux-4.14.252-131.483.amzn1.x86_64-x86_64-with-glibc2.9
- Python version: 3.6.13
- PyArrow version: 6.0.1


Something isn't working

duplicate

This issue or pull request already exists

Hi @willowdong! These issues were fixed on master. We will have a new release of `datasets` later today. In the meantime, you can avoid these issues by installing `datasets` from master as follows:
```bash
pip install git+https://github.com/huggingface/datasets.git
```

You should force redownload:
```python
dataset = load_dataset("multi_news", download_mode="force_redownload")
dataset_2 = load_dataset("reddit_tifu", "long", download_mode="force_redownload")

## Describe the bug
Can't load the dataset

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
```
from datasets import load_dataset

dataset = load_dataset('multi_news')
dataset_2=load_dataset("reddit_tifu", "long")

## Actual results
 raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1ffWfITKFMJeqjT8loC8aiCLRNJpc_XnF']

## Environment info
- `datasets` version: 1.18.4
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.0
- PyArrow version: 6.0.1


Create README.md

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3917). All of your documentation changes will be reflected on that endpoint.

This follows the same structure as the GLUE metric card, hope that works for everyone :)


Create README.md for GLUE

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3916). All of your documentation changes will be reflected on that endpoint.

I still have a hesitation regarding the format of inputs -- whether it's a list or a list of lists? -- hopefully @lhoestq will be able to clarify.

Also tagging @yjernite for the Limitations section. Happy to hear your thoughts!


Metric card template

_The documentation is not available anymore as the PR was closed or merged._

Looks like a great start! I have a general comment and a few specific comments.

My general comment is I wonder if we need a post for this template and the data and model card templates (or a combined one?) explaining why this documentation is needed and how it serves both the writer and the audience.

Specific comments:
- Maybe we can add some more desiderata to the overview instructions like: what task was the metric originally developed for, what tasks is it used for now, what is the range of possible outputs?
- In the data card, we call the data instances inputs `fields`. It might be good to synchronize on that across the templates and change `input_name` to `input_field`? Also are the instructions for the `input_name` complete? It ends with 'In the *' and I'm not sure what that refers to.
- 'Values' seems ambiguous to me, maybe 'scores' would be more explicit? Also could add a request for the range of possible outputs.
- We could add a reference in the examples section to the overview section if that's where further explanation should go. Suggestion to add: 'Provide a range of examples that show both typical and atypical results' or something similar.
- I'm not sure if we'd want to add this to the example section or make a new section, but it would be good to prompt somewhere for links to specific use cases in HF
- In the limitations and bias section, add 'with links'

Looks like a great start! I have a general comment and a few specific comments.

My general comment is I wonder if we need a post for this template and the data and model card templates (or a combined one?) explaining why this documentation is needed and how it serves both the writer and the audience.

Specific comments:
- Maybe we can add some more desiderata to the overview instructions like: what task was the metric originally developed for, what tasks is it used for now, what is the range of possible outputs?
- In the data card, we call the data instances `fields`. It might be good to synchronize on that across the templates and change `input_name` to `input_field`? Also are the instructions for the `input_name` complete? It ends with 'In the *' and I'm not sure what that refers to.
- 'Values' seems ambiguous to me, maybe 'scores' would be more explicit? Also could add a request for the range of possible outputs.
- We could add a reference to the examples section to the overview section if that's where further explanation should go. Suggestion to add: 'Provide a range of examples that show both typical and atypical results' or something similar.
- I'm not sure if we'd want to add this to the example section or make a new section, but it would be good to prompt somewhere for links to specific use cases in HF
- In the limitations and bias section, add 'with links'

Thanks for your feedback, @mcmillanmajora ! I totally agree that we should write a post -- we were going to write one up when we are done with a good chunk of the metric cards, but we can also do that earlier :smile:

With regards to your more specific comments:

- It is our intention to put what the metric was developed for (whether it is a specific task or dataset, for example). You can see the [WER](https://github.com/huggingface/datasets/tree/master/metrics/wer) metric card for that.
- `input_field` works for me!
- the values aren't always scores, it's more like the values the metric can take. And it does include the range of possible values, including the max and min, that are outputted.
- I like the suggestion to add: 'Provide a range of examples that show both typical and atypical results' :hugs:
- I have been putting specific use cases in 'Further references', just because there isn't always something to put there, especially for less popular metrics

Oh cool! I was just looking at the template, it definitely helps seeing an example metric card. Based on just the instructions, I had assumed that examples meant research papers where the metric was used to evaluate a model, but I like the explicit coding examples!

Adding a metric card template, based on ideas and edits from @sashavor and I, as well as from comments from @lhoestq and others (thank you!).

All feedback is welcome, but am especially curious about feedback in terms of:
- things that should be included but aren't
- things that are included but should be changed or removed
- the instructions I included, and whether they should be added to, clarified, or deleted altogether


Use templates for doc-builidng jobs

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3914). All of your documentation changes will be reflected on that endpoint.

You can ignore the CI failures btw, they're unrelated to this PR

This PR updates the jobs for all doc-building related things by using the templates introduced on `doc-builder`. By putting those once there, we make sure every repo gets the latest fixes on the doc-building github actions :-)

Note: all libraries must share the same docker image for those doc-building jobs. For now, all the one used (`huggingface/transformers-doc-builder`) contains all extra steps of the datasets install for docbuling (mainly libsndfile) but if in the future some additional steps are necessary on top of `pip install -e .[dev]`, this docker image will need to be updated with the extra deps.


Deterministic split order in DatasetDict.map

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3913). All of your documentation changes will be reflected on that endpoint.

I'm surprised this is needed because the order of the `dict` keys is deterministic as of Python 3.6 (documented in 3.7). Is there a reproducer for this behavior? I wouldn't make this change unless it's absolutely needed because `sorted` modifies the initial order of the keys.

Indeed this doesn't fix the issue apparently. Actually this is probably because the tokenizer used to process the second split is in a state that has been modified by the first split.

Therefore after reloading the first split from the cache, then the second split can't be reloaded since the tokenizer hasn't seen the first split (and therefore is considered a different tokenizer).

The order in which the splits are processed by `map` is not deterministic in `DatasetDict.map`. This can cause caching issues when the processing function is stateful and sensible to the order in which examples are processed

Close https://github.com/huggingface/datasets/issues/3847


add draft of registering function for pandas

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3912). All of your documentation changes will be reflected on that endpoint.

That's cool ! Though I would expect such an integration to only require `huggingface_hub`, not the full `datasets` library.
 Indeed if users want to use the `datasets` lib they could just to `Dataset.from_pandas(df).push_to_hub()` already. Therefore I would explore something that doesn't not necessarily requires `datasets`.

For other could storage solutions (S3, GCS, etc.), pandas allows users to pass URIs like `s3://bucket-name/path/data.csv` to the `read_xxx` and `to_xxx` (for csv, parquet, json, etc). It also support passing the **root directory** like `s3://bucket-name/dataset-dir` instead of a single file name.

In the Hugging Face Hub case, we have one dataset = one repository. We can enter pandas' paradigm by saying one dataset = one repository = one root directory. Here is what we could have:

### push to Hub:
```python
"""
Demo script for writing a pandas data frame to a CSV file on HF using fsspec-supported pandas APIs
"""
import pandas as pd

HF_USER = os.getenv("HF_USER")
HF_TOKEN = os.getenv("HF_TOKEN")

books_df = pd.DataFrame(
    data={"Title": ["Book I", "Book II", "Book III"], "Price": [56.6, 59.87, 74.54]},
    columns=["Title", "Price"],
)

dataset_name = "books1"

books_df.to_csv(
    f"hf://{HF_USER}/{dataset_name}",
    index=False,
    storage_options={
        "repo_type": "dataset",
        "token": HF_TOKEN,
    },
)

```

### load from Hub:
```python
"""
Demo script for reading a CSV file from HF into a pandas data frame using fsspec-supported pandas
APIs
"""
import pandas as pd

HF_USER = os.getenv("HF_USER")
HF_TOKEN = os.getenv("HF_TOKEN")

dataset_name = "books1"

books_df = pd.read_csv(
    f"hf://{HF_USER}/{dataset_name}",
    storage_options={
        "repo_type": "dataset",
        "token": HF_TOKEN,
    },
)

print(books_df)
```

And you could do the same with Parquet data using `read/to_parquet` or other formats. Formats like CSV, Parquet or JSON Lines would work out of the box with `datasets`. This API would also allow anyone to use Dask with the Hugging Face Hub for example.

What do you think ?

I'm closing this PR as [`hffs`](https://github.com/huggingface/hffs) can now be used for reading/writing data frames from/to the Hub.

This PR adds a register function for `pandas`. It allows to directly push `DataFrame` objects to the hub and in return loading datasets on the hub from `DataFrame`. The motivation for this integration is to enable the vast number of `pandas` users to be able to easily push `DataFrames` to the hub.

Here is an example:

```python
import pandas as pd
from datasets import register_pandas

register_pandas()

# push to hub
df = pd.DataFrame.from_dict({"test": [1,2,3]})
df.push_to_hub("my_test")

# load from hub
df_retrieved = pd.DataFrame.load_from_hub("lvwerra/my_test")
```

It follows a similar philosophy as the `tqdm` [integration](https://github.com/tqdm/tqdm#pandas-integration). Also see [this issue](https://github.com/pandas-dev/pandas/issues/46000) on the `pandas` repository.

This is just a rough draft of what such integration could look like but I would like appreciate some feedback on this: is this something you would like to add the library and is this the way to go? cc @lhoestq @albertvillanova @julien-c


Create README.md for CER metric

Initial proposal for a CER metric card
cc @patrickvonplaten - wdyt this time around? :smile:


Fix text loader to split only on universal newlines

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3910). All of your documentation changes will be reflected on that endpoint.

Looks like the test needs to be updated for windows ^^'

I don't think this is the same issue as in https://github.com/oscar-corpus/corpus/issues/18, where the OSCAR metadata has line offsets that use only `\n` as the newline marker to count lines, not `\r\n` or `\r`.

It looks like the OSCAR data loader is opening the data files with `gzip.open` directly and I don't think this text loader is used, but I'm not familiar with a lot of `datasets` internals so I could be mistaken?

You are right @adrianeboyd.

This PR fixes #3729.

Additionally, this PR is somehow related to the OSCAR issue. However, the OSCAR issue have multiple root causes: one is the offset initialization (as you pointed out); other is similar to this case: Unicode newlines are not properly handled.

I will make a change proposal for OSCAR this afternoon.

@lhoestq I'm working on fixing the Windows tests on my Windows machine...

I finally changed the approach in order to avoid having "\r\n" and "\r" line breaks in Python `str` read from files on Windows/old Macintosh machines.

Currently, `text` loader breaks on a superset of universal newlines, which also contains Unicode line boundaries. See: https://docs.python.org/3/library/stdtypes.html#str.splitlines

However, the expected behavior is to get the lines splitted only on universal newlines: "\n", "\r\n" and "\r".

See: oscar-corpus/corpus#18

Fix #3729.


Error loading file audio when downloading the Common Voice dataset directly from the Hub

bug

Something isn't working

Hi ! It could an issue with torchaudio, which version of torchaudio are you using ? Can you also try updating `datasets` to 2.0.0 and see if it works ?

I _might_ have a similar issue. I'm trying to use the librispeech_asr dataset and read it with soundfile.

```python
from datasets import load_dataset, load_metric
from transformers import Speech2TextForConditionalGeneration, Speech2TextProcessor
import soundfile as sf

librispeech_eval = load_dataset("librispeech_asr", "clean", split="test")  # change to "other" for other test dataset
wer = load_metric("wer")

model = Speech2TextForConditionalGeneration.from_pretrained("facebook/s2t-small-librispeech-asr").to("cuda")
processor = Speech2TextProcessor.from_pretrained("facebook/s2t-small-librispeech-asr", do_upper_case=True)

def map_to_array(batch):
    speech, _ = sf.read(batch["file"])
    batch["speech"] = speech
    return batch

librispeech_eval = librispeech_eval.map(map_to_array)

def map_to_pred(batch):
    features = processor(batch["speech"], sampling_rate=16000, padding=True, return_tensors="pt")
    input_features = features.input_features.to("cuda")
    attention_mask = features.attention_mask.to("cuda")

    gen_tokens = model.generate(input_ids=input_features, attention_mask=attention_mask)
    batch["transcription"] = processor.batch_decode(gen_tokens, skip_special_tokens=True)
    return batch

result = librispeech_eval.map(map_to_pred, batched=True, batch_size=8, remove_columns=["speech"])

print("WER:", wer(predictions=result["transcription"], references=result["text"]))
```

The code is taken directly from "https://huggingface.co/facebook/s2t-small-librispeech-asr".

The short error code is "RuntimeError: Error opening '6930-75918-0000.flac': System error." (it can't find the first file), and I agree, I can't find the file either. The dataset has downloaded correctly (it says), but on the location, there are only ".arrow" files, no ".flac" files.

**Error message:**

```python
RuntimeError                              Traceback (most recent call last)
Input In [15], in <cell line: 16>()
     13     batch["speech"] = speech
     14     return batch
---> 16 librispeech_eval = librispeech_eval.map(map_to_array)
     18 def map_to_pred(batch):
     19     features = processor(batch["speech"], sampling_rate=16000, padding=True, return_tensors="pt")

File C:\ProgramData\Miniconda3\envs\noise_cancel\lib\site-packages\datasets\arrow_dataset.py:1953, in Dataset.map(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   1950 disable_tqdm = not logging.is_progress_bar_enabled()
   1952 if num_proc is None or num_proc == 1:
-> 1953     return self._map_single(
   1954         function=function,
   1955         with_indices=with_indices,
   1956         with_rank=with_rank,
   1957         input_columns=input_columns,
   1958         batched=batched,
   1959         batch_size=batch_size,
   1960         drop_last_batch=drop_last_batch,
   1961         remove_columns=remove_columns,
   1962         keep_in_memory=keep_in_memory,
   1963         load_from_cache_file=load_from_cache_file,
   1964         cache_file_name=cache_file_name,
   1965         writer_batch_size=writer_batch_size,
   1966         features=features,
   1967         disable_nullable=disable_nullable,
   1968         fn_kwargs=fn_kwargs,
   1969         new_fingerprint=new_fingerprint,
   1970         disable_tqdm=disable_tqdm,
   1971         desc=desc,
   1972     )
   1973 else:
   1975     def format_cache_file_name(cache_file_name, rank):

File C:\ProgramData\Miniconda3\envs\noise_cancel\lib\site-packages\datasets\arrow_dataset.py:519, in transmit_tasks.<locals>.wrapper(*args, **kwargs)
    517     self: "Dataset" = kwargs.pop("self")
    518 # apply actual function
--> 519 out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    520 datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    521 for dataset in datasets:
    522     # Remove task templates if a column mapping of the template is no longer valid

File C:\ProgramData\Miniconda3\envs\noise_cancel\lib\site-packages\datasets\arrow_dataset.py:486, in transmit_format.<locals>.wrapper(*args, **kwargs)
    479 self_format = {
    480     "type": self._format_type,
    481     "format_kwargs": self._format_kwargs,
    482     "columns": self._format_columns,
    483     "output_all_columns": self._output_all_columns,
    484 }
    485 # apply actual function
--> 486 out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    487 datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    488 # re-apply format to the output

File C:\ProgramData\Miniconda3\envs\noise_cancel\lib\site-packages\datasets\fingerprint.py:458, in fingerprint_transform.<locals>._fingerprint.<locals>.wrapper(*args, **kwargs)
    452             kwargs[fingerprint_name] = update_fingerprint(
    453                 self._fingerprint, transform, kwargs_for_fingerprint
    454             )
    456 # Call actual function
--> 458 out = func(self, *args, **kwargs)
    460 # Update fingerprint of in-place transforms + update in-place history of transforms
    462 if inplace:  # update after calling func so that the fingerprint doesn't change if the function fails

File C:\ProgramData\Miniconda3\envs\noise_cancel\lib\site-packages\datasets\arrow_dataset.py:2318, in Dataset._map_single(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, disable_tqdm, desc, cache_only)
   2316 if not batched:
   2317     for i, example in enumerate(pbar):
-> 2318         example = apply_function_on_filtered_inputs(example, i, offset=offset)
   2319         if update_data:
   2320             if i == 0:

File C:\ProgramData\Miniconda3\envs\noise_cancel\lib\site-packages\datasets\arrow_dataset.py:2218, in Dataset._map_single.<locals>.apply_function_on_filtered_inputs(inputs, indices, check_same_num_examples, offset)
   2216 if with_rank:
   2217     additional_args += (rank,)
-> 2218 processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
   2219 if update_data is None:
   2220     # Check if the function returns updated examples
   2221     update_data = isinstance(processed_inputs, (Mapping, pa.Table))

File C:\ProgramData\Miniconda3\envs\noise_cancel\lib\site-packages\datasets\arrow_dataset.py:1913, in Dataset.map.<locals>.decorate.<locals>.decorated(item, *args, **kwargs)
   1909 decorated_item = (
   1910     Example(item, features=self.features) if not batched else Batch(item, features=self.features)
   1911 )
   1912 # Use the LazyDict internally, while mapping the function
-> 1913 result = f(decorated_item, *args, **kwargs)
   1914 # Return a standard dict
   1915 return result.data if isinstance(result, LazyDict) else result

Input In [15], in map_to_array(batch)
     11 def map_to_array(batch):
---> 12     speech, _ = sf.read(batch["file"])
     13     batch["speech"] = speech
     14     return batch

File C:\ProgramData\Miniconda3\envs\noise_cancel\lib\site-packages\soundfile.py:256, in read(file, frames, start, stop, dtype, always_2d, fill_value, out, samplerate, channels, format, subtype, endian, closefd)
    170 def read(file, frames=-1, start=0, stop=None, dtype='float64', always_2d=False,
    171          fill_value=None, out=None, samplerate=None, channels=None,
    172          format=None, subtype=None, endian=None, closefd=True):
    173     """Provide audio data from a sound file as NumPy array.
    174
    175     By default, the whole file is read from the beginning, but the
   (...)
    254
    255     """
--> 256     with SoundFile(file, 'r', samplerate, channels,
    257                    subtype, endian, format, closefd) as f:
    258         frames = f._prepare_read(start, stop, frames)
    259         data = f.read(frames, dtype, always_2d, fill_value, out)

File C:\ProgramData\Miniconda3\envs\noise_cancel\lib\site-packages\soundfile.py:629, in SoundFile.__init__(self, file, mode, samplerate, channels, subtype, endian, format, closefd)
    626 self._mode = mode
    627 self._info = _create_info_struct(file, mode, samplerate, channels,
    628                                  format, subtype, endian)
--> 629 self._file = self._open(file, mode_int, closefd)
    630 if set(mode).issuperset('r+') and self.seekable():
    631     # Move write position to 0 (like in Python file objects)
    632     self.seek(0)

File C:\ProgramData\Miniconda3\envs\noise_cancel\lib\site-packages\soundfile.py:1183, in SoundFile._open(self, file, mode_int, closefd)
   1181 else:
   1182     raise TypeError("Invalid file: {0!r}".format(self.name))
-> 1183 _error_check(_snd.sf_error(file_ptr),
   1184              "Error opening {0!r}: ".format(self.name))
   1185 if mode_int == _snd.SFM_WRITE:
   1186     # Due to a bug in libsndfile version <= 1.0.25, frames != 0
   1187     # when opening a named pipe in SFM_WRITE mode.
   1188     # See http://github.com/erikd/libsndfile/issues/77.
   1189     self._info.frames = 0

File C:\ProgramData\Miniconda3\envs\noise_cancel\lib\site-packages\soundfile.py:1357, in _error_check(err, prefix)
   1355 if err != 0:
   1356     err_str = _snd.sf_error_number(err)
-> 1357     raise RuntimeError(prefix + _ffi.string(err_str).decode('utf-8', 'replace'))

RuntimeError: Error opening '6930-75918-0000.flac': System error.
```

**Package versions:**
```python
python: 3.9
transformers: 4.17.0
datasets: 2.0.0
SoundFile: 0.10.3.post1
```

Hi ! In `datasets` 2.0 can access the audio array with `librispeech_eval[0]["audio"]["array"]` already, no need to use `map_to_array`. See our documentation on [how to process audio data](https://huggingface.co/docs/datasets/audio_process) :)

cc @patrickvonplaten we will need to update the readme at [facebook/s2t-small-librispeech-asr](https://huggingface.co/facebook/s2t-small-librispeech-asr) as well as https://huggingface.co/docs/transformers/model_doc/speech_to_text

Thanks!

And sorry for posting this problem in what turned on to be an unrelated thread.

I rewrote the code, and the model works. The WER is 0.137 however, so I'm not sure if I have missed a step. I will look further into that at a later point. The transcriptions look good through manual inspection.

The rewritten code:
```python
from datasets import load_dataset, load_metric
from transformers import Speech2TextForConditionalGeneration, Speech2TextProcessor, Wav2Vec2Processor

librispeech_eval = load_dataset("librispeech_asr", "clean", split="test")  # change to "other" for other test dataset
wer = load_metric("wer")

model = Speech2TextForConditionalGeneration.from_pretrained("facebook/s2t-small-librispeech-asr").to("cuda")
processor = Speech2TextProcessor.from_pretrained("facebook/s2t-small-librispeech-asr", do_upper_case=True)

def map_to_pred(batch):
    audio = batch["audio"]
    features = processor(audio["array"], sampling_rate=audio["sampling_rate"], padding=True, return_tensors="pt")
    input_features = features.input_features.to("cuda")
    attention_mask = features.attention_mask.to("cuda")

    gen_tokens = model.generate(input_features=input_features, attention_mask=attention_mask)
    batch["transcription"] = processor.batch_decode(gen_tokens, skip_special_tokens=True)
    return batch

result = librispeech_eval.map(map_to_pred)#, batched=True, batch_size=8)

print("WER:", wer.compute(predictions=result["transcription"], references=result["text"]))
```

I think the issue comes from the fact that you set `batched=False` while `map_to_pred` still returns a list of strings for "transcription". You can fix it by adding `[0]` at the end of this line to get the string:
```python
batch["transcription"] = processor.batch_decode(gen_tokens, skip_special_tokens=True)[0]
```

Updating as many model cards now as I can find

We no longer use `torchaudio` for decoding MP3 files, and the problem with model cards has been addressed, so I'm closing this issue.

## Describe the bug
When loading the Common_Voice dataset, by downloading it directly from the Hugging Face hub, some files can not be opened.

## Steps to reproduce the bug
```python
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re

test_dataset = load_dataset("common_voice", "it", split="test")
#test_dataset = load_dataset('csv', data_files = {'test': '/workspace/Dataset/Common_Voice/cv-corpus80/it/test.csv'})
wer = load_metric("wer")

processor = Wav2Vec2Processor.from_pretrained("joorock12/wav2vec2-large-xlsr-italian")
model = Wav2Vec2ForCTC.from_pretrained("joorock12/wav2vec2-large-xlsr-italian")
model.to("cuda")

chars_to_ignore_regex = '[\,\?\.\!\-\;\:\"\“\'\�]'
resampler = torchaudio.transforms.Resample(48_000, 16_000)
```
## Expected results
The common voice dataset downloaded and correctly loaded whit the use of the hugging face datasets library.

## Actual results
The error is:
```python
0ex [00:00, ?ex/s]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-48-ef87f4129e6e> in <module>
      7     return batch
      8
----> 9 test_dataset = test_dataset.map(speech_file_to_array_fn)

/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py in map(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   2107
   2108         if num_proc is None or num_proc == 1:
-> 2109             return self._map_single(
   2110                 function=function,
   2111                 with_indices=with_indices,

/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    516             self: "Dataset" = kwargs.pop("self")
    517         # apply actual function
--> 518         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    519         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    520         for dataset in datasets:

/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    483         }
    484         # apply actual function
--> 485         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    486         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    487         # re-apply format to the output

/opt/conda/lib/python3.8/site-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    411             # Call actual function
    412
--> 413             out = func(self, *args, **kwargs)
    414
    415             # Update fingerprint of in-place transforms + update in-place history of transforms

/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py in _map_single(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, disable_tqdm, desc, cache_only)
   2465                 if not batched:
   2466                     for i, example in enumerate(pbar):
-> 2467                         example = apply_function_on_filtered_inputs(example, i, offset=offset)
   2468                         if update_data:
   2469                             if i == 0:

/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py in apply_function_on_filtered_inputs(inputs, indices, check_same_num_examples, offset)
   2372             if with_rank:
   2373                 additional_args += (rank,)
-> 2374             processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
   2375             if update_data is None:
   2376                 # Check if the function returns updated examples

/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py in decorated(item, *args, **kwargs)
   2067                 )
   2068                 # Use the LazyDict internally, while mapping the function
-> 2069                 result = f(decorated_item, *args, **kwargs)
   2070                 # Return a standard dict
   2071                 return result.data if isinstance(result, LazyDict) else result

<ipython-input-48-ef87f4129e6e> in speech_file_to_array_fn(batch)
      3 def speech_file_to_array_fn(batch):
      4     batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
----> 5     speech_array, sampling_rate = torchaudio.load(batch["path"])
      6     batch["speech"] = resampler(speech_array).squeeze().numpy()
      7     return batch

/opt/conda/lib/python3.8/site-packages/torchaudio/backend/sox_io_backend.py in load(filepath, frame_offset, num_frames, normalize, channels_first, format)
    150                 filepath, frame_offset, num_frames, normalize, channels_first, format)
    151         filepath = os.fspath(filepath)
--> 152     return torch.ops.torchaudio.sox_io_load_audio_file(
    153         filepath, frame_offset, num_frames, normalize, channels_first, format)
    154

RuntimeError: Error loading audio file: failed to open file common_voice_it_17415776.mp3  ```

## Environment info

- `datasets` version: 1.18.4
- Platform: Linux-5.4.0-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyArrow version: 7.0.0


Update README.md for SQuAD v2 metric

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3908). All of your documentation changes will be reflected on that endpoint.

Putting "Values from popular papers" as a subsection of "Output values"


Update README.md for SQuAD metric

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3907). All of your documentation changes will be reflected on that endpoint.


NonMatchingChecksumError on Spider dataset

Something isn't working

Hi @kolk, thanks for reporting.

Indeed, Google Drive service recently changed their service and we had to add a fix to our library to cope with that change:
- #3787

We just made patch release last week: 1.18.4 https://github.com/huggingface/datasets/releases/tag/1.18.4

Please, feel free to update your local `datasets` version, so that you get the fix:
```shell
pip install -U datasets
```

## Describe the bug
Failure to generate dataset ```spider``` because of checksums error for dataset source files.

## Steps to reproduce the bug
```
from datasets import load_dataset
spider = load_dataset("spider")
```

## Expected results
Checksums should match for files from url ['https://drive.google.com/uc?export=download&id=1_AckYkinAnhqmRQtGsQgUKAnTHxxX5J0']

## Actual results
```
>>> load_dataset("spider")
load_dataset("spider")
Downloading and preparing dataset spider/spider (download: 95.12 MiB, generated: 5.17 MiB, post-processed: Unknown size, total: 100.29 MiB) to /home/user/.cache/huggingface/datasets/spider/spider/1.0.0/79778ebea87c59b19411f1eb3eda317e9dd5f7788a556d837ef25c3ae6e5e8b7...
Traceback (most recent call last):
  File "/home/user/py3_env/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-5-d4cb54197348>", line 1, in <module>
    load_dataset("spider")
  File "/home/user/py3_env/lib/python3.8/site-packages/datasets/load.py", line 1702, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/user/py3_env/lib/python3.8/site-packages/datasets/builder.py", line 594, in download_and_prepare
    self._download_and_prepare(
  File "/home/user/py3_env/lib/python3.8/site-packages/datasets/builder.py", line 665, in _download_and_prepare
    verify_checksums(
  File "/home/user/py3_env/lib/python3.8/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1_AckYkinAnhqmRQtGsQgUKAnTHxxX5J0']
```

## Environment info
datasets version: 1.18.3
Platform: Ubuntu 20 LTS
Python version: 3.8.10
PyArrow version: 6.0.1


Perplexity Metric Card

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3905). All of your documentation changes will be reflected on that endpoint.

I'm wondering if we should add that perplexity can be used for analyzing datasets as well

Otherwise, looks good! Good job, @emibaylor !

Add Perplexity metric card

Note that it is currently still missing the citation, but I plan to add it later today.


CONLL2003 Dataset not available

dataset bug

A bug in a dataset script provided in the library

Thanks for reporting, @omarespejel.

I'm sorry but I can't reproduce the issue: the loading of the dataset works perfecto for me and I can reach the data URL: https://data.deepai.org/conll2003.zip

Might it be due to a temporary problem in the data owner site (https://data.deepai.org/) that is fixed now?
Could you please try loading the dataset again and tell if the problem persists?

@omarespejel I'm closing this issue. Feel free to reopen it if the problem persists.

getting same issue. Can't find any solution.

I am getting the same issue. I use google colab with CPU.
The code I used is exactly the same as described above.
```
from datasets import load_dataset
dataset = load_dataset("conll2003")
```

The produced error:
![image](https://github.com/huggingface/datasets/assets/9371628/d87f7fb0-ef58-4755-abb5-f8f92c51fe02)

Note: This error is different from what was initially described in this thread. This is because I use CPU. When I use GPU I reproduce the same initial error of the thread.

Moreover, I receive the following warning:
```
WARNING:urllib3.connection:Certificate did not match expected hostname: data.deepai.org. Certificate: {'subject': ((('commonName', '*.b-cdn.net'),),), 'issuer': ((('countryName', 'GB'),), (('stateOrProvinceName', 'Greater Manchester'),), (('localityName', 'Salford'),), (('organizationName', 'Sectigo Limited'),), (('commonName', 'Sectigo RSA Domain Validation Secure Server CA'),)), 'version': 3, 'serialNumber': 'DDED48B13E1EA03983E833AB2C35EF07', 'notBefore': 'Nov  7 00:00:00 2022 GMT', 'notAfter': 'Nov 11 23:59:59 2023 GMT', 'subjectAltName': (('DNS', '*.b-cdn.net'), ('DNS', 'b-cdn.net')), 'OCSP': ('http://ocsp.sectigo.com/',), 'caIssuers': ('http://crt.sectigo.com/SectigoRSADomainValidationSecureServerCA.crt',)}
Downloading and preparing dataset conll2003/conll2003 to /root/.cache/huggingface/datasets/conll2003/conll2003/1.0.0/9a4d16a94f8674ba3466315300359b0acd891b68b6c8743ddf60b9c702adce98...
WARNING:urllib3.connection:Certificate did not match expected hostname: data.deepai.org. Certificate: {'subject': ((('commonName', '*.b-cdn.net'),),), 'issuer': ((('countryName', 'GB'),), (('stateOrProvinceName', 'Greater Manchester'),), (('localityName', 'Salford'),), (('organizationName', 'Sectigo Limited'),), (('commonName', 'Sectigo RSA Domain Validation Secure Server CA'),)), 'version': 3, 'serialNumber': 'DDED48B13E1EA03983E833AB2C35EF07', 'notBefore': 'Nov  7 00:00:00 2022 GMT', 'notAfter': 'Nov 11 23:59:59 2023 GMT', 'subjectAltName': (('DNS', '*.b-cdn.net'), ('DNS', 'b-cdn.net')), 'OCSP': ('http://ocsp.sectigo.com/',), 'caIssuers': ('http://crt.sectigo.com/SectigoRSADomainValidationSecureServerCA.crt',)}
```

## Describe the bug
[CONLL2003](https://huggingface.co/datasets/conll2003) Dataset can no longer reach 'https://data.deepai.org/conll2003.zip'
![image](https://user-images.githubusercontent.com/4755430/158084483-ff83631c-5154-4823-892d-577bf1166db0.png)

## Steps to reproduce the bug
```python
from datasets import load_dataset
datasets = load_dataset("conll2003")
```

## Expected results
Download the conll2003 dataset.

## Actual results
Error: `ConnectionError: Couldn't reach https://data.deepai.org/conll2003.zip (error 502)`


Add Biwi Kinect Head Pose dataset.

_The documentation is not available anymore as the PR was closed or merged._

Thanks for the detailed explanation of the structure!

1. IMO it makes the most sense to yield one example for each person (so the total of 24 examples), so the features dict should be similar to this:

  ```python
    features = Features({
        "rgb": Sequence(Image()),  # for the png frames
        "rgb_cal": {"intrisic_mat": Array2D(shape=(3, 3), dtype="float32"), "extrinsic_mat": {"rotation": Array2D(shape=(3, 3), dtype="float32"), "translation": Sequence(Value("float32", length=3)}},
        "depth": Sequence(Value("string")), # for the depth frames
        "depth_cal": the same as "rgb_cal",
        "head_pose_gt": Sequence({"center": Sequence(Value("float32", length=3), "rotation": Array2D(shape=(3, 3), dtype="float32")}),
        "head_template": Value("string"),  # for the person's obj file

    })
  ```
   We can add a "Data Processing" section to the card to explain how to parse the files.

2. Yes, it's ok to parse the files as long as it doesn't take too much time/memory (e.g., it's ok to parse the `*_pose.txt` or `*.cal` files, but it's better to leave the `*_depth.bin` or `*.obj` files unprocessed and yield the paths to them)

Thanks for the suggestions @mariosasko, yielding one example for each person would make things much easier.
Okay. I'll look at parsing the files and then displaying the information.

Added the following :
- Features, I have included sequence_number and subject_id along with the features you had suggested.
- Tested loading of the dataset along with dummy_data and full_data tests.
- Created the dataset_infos.json file.

To-Do :
- [x] Update Dataset Cards with more details.
- [x] "Data Processing" section

Any inputs on what to include in the "Data Processing" section ?

@mariosasko  Please could you review this when you get time. Thank you.

In the Data Processing section, I've added example code for a compressed binary depth image file. Updated the Readme as well.

@mariosasko / @lhoestq , Please could you review this when you get time. Thank you.

Created an issue here: https://github.com/huggingface/datasets/issues/4152

Got it. Thanks for the comments. I've collapsed the C++ code in the readme and added the suggestions.

Hi ! The `AttributeError ` bug has been fixed, feel free to merge `master` into your branch ;)

I haven't been able to figure out why CI is failing, the error shown is :

```
E           ValueError: The following issues have been found in the dataset cards:
E           README Parsing:
E           list index out of range
E           The following issues have been found in the dataset cards:
E           README Validation:
E           list index out of range
```

Any inputs would be helpful.

I think it's because there are tabulations in the c++ code, can you replace them with regular spaces please ?

(then in another PR we can maybe fix the Readme parser to support text indented with tabulations)

@lhoestq , initially the idea was to have one example = one image with an additional field mentioning the frame_number. But each subject, we had a head template, calibration information for the depth and the color camera which was common to all the examples for that subject. Also, the images were continuous frames.
@mariosasko  suggested this structure and it made sense to group the images together for a particular subject.

> Don't you think it would be more practical to have one example = one image in this dataset ?

Having one example = one image would be good but since we have a head template, calibration information for the depth and the color camera which is common to all the images for that subject and the images being continuous frames, I think it makes sense to group the images together for each subject. This will make the feature representation easier.

Ok I see, sounds good then. Users can still separate the images if they want to

The CI fails are unrelated to this PR and fixed on master, merging !

Great. Thanks @lhoestq , I think we can close this issue now. ( #3822 )

This PR adds the Biwi Kinect Head Pose dataset.

Dataset Request : Add Biwi Kinect Head Pose Database [#3822](https://github.com/huggingface/datasets/issues/3822)

The Biwi Kinect Head Pose Database is acquired with the Microsoft Kinect sensor, a structured IR light device.It contains 15K images of 20 people with 6 females and 14 males where 4 people were recorded twice.

For each frame, there is :
- a depth image, (.bin file)
- a corresponding rgb image (both 640x480 pixels),
- annotation ( present inside a .txt file)
 The ground truth is the 3D location of the head and its rotation.

The dataset structure is as follows :
```
- 01.obj
- 01
     - frame_00003_depth.bin
     - frame_00003_pose.txt
     - frame_00003_rgb.png
     .
     .
     .
- 02.obj
 - 02
     - frame_00003_depth.bin
     - frame_00003_pose.txt
     - frame_00003_rgb.png
     .
     .
     .
```
Preview of frame_00003_pose.txt :
```
0.988397 0.0731349 0.133128
-0.0441539 0.976945 -0.208876
-0.145334 0.200575 0.968838

126.665 40.4515 876.198
```
I have used the following dataset features :
```
features=datasets.Features(
    {
        "person_id": datasets.Value("string"),
        "frame_number": datasets.Value("string"),
        "depth_image": datasets.Value("string"),
        "rgb_image": datasets.Image(),
        "3D_head_center": datasets.Array2D(shape=(3, 3), dtype="float"),
        "3D_head_rotation": datasets.Value("float"),
    }
```
I am giving the path to the depth_image here.

I need some inputs for the following :

1. For each person, the dataset has the following additional information :
```
For each sequence, the corresponding .obj file represents a head template deformed to match the neutral face of that specific person. [*.obj file]
In each folder, two .cal files contain calibration information for the depth and the color camera, e.g., the intrinsic camera matrix of the depth camera and the global rotation and translation to the rgb camera.
```
Wanted to know how we can represent these features ?

2. For _generate_examples , do I parse the directories and fetch the required information ? This would mean reading the .txt file to obtain the "3D_head_center" and "3D_head_rotation" details. We could precompute the features information  and have a metadata file and use the metadata file to yield information in _generate_examples ? Wanted your thoughts for the best approach for this ?


Can't import datasets: partially initialized module 'fsspec' has no attribute 'utils'

bug

Something isn't working

Update: `"python3 -c "from from datasets import Dataset, DatasetDict"` works, but not if I import without the `python3 -c`

Hi @arunasank, thanks for reporting.

It seems that this can be caused because you are using an old version of `fsspec`: the reason why it works if you run `python3` seems to be that `python3` runs in a Python virtual env (with an updated version of `fsspec`); whereas the error arises when you run the import from other Python virtual env (with an old version of `fsspec`).

In order to fix this, you should update `fsspec` from within the "problematic" Python virtual env:
```
pip install -U "fsspec[http]>=2021.05.0"

I'm closing this issue, @arunasank.

Feel free to re-open it if the problem persists.

from lightgbm import LGBMModel,LGBMClassifier, plot_importance
after importing lib getting (partially initialized module 'fsspec' has no attribute 'utils' (most likely due to a circular import) error, can help me

@deepakmahtha I think you are not using `datasets`: this is the GitHub repository of Hugging Face Datasets.

If you are using `lightgbm`, you should report the issue to their repository instead.

Anyway, we have proposed a possible fix just in a comment above: to update fsspec.
https://github.com/huggingface/datasets/issues/3902#issuecomment-1066517824

## Describe the bug
Unable to import datasets

## Steps to reproduce the bug
```python
from datasets import Dataset, DatasetDict
```

## Expected results
The import works without errors

## Actual results
```
AttributeError                            Traceback (most recent call last)
<ipython-input-37-c8cfcbe62127> in <module>
     11 # from tqdm import tqdm
     12 # import torch
---> 13 from datasets import Dataset
     14 # from transformers import Trainer, TrainingArguments, AutoModel, AutoTokenizer, AutoModelForMaskedLM, DataCollatorForLanguageModeling
     15 # from sentence_transformers import SentenceTransformer

~/.local/lib/python3.8/site-packages/datasets/__init__.py in <module>
     31     )
     32
---> 33 from .arrow_dataset import Dataset, concatenate_datasets
     34 from .arrow_reader import ArrowReader, ReadInstruction
     35 from .arrow_writer import ArrowWriter

~/.local/lib/python3.8/site-packages/datasets/arrow_dataset.py in <module>
     46 )
     47
---> 48 import fsspec
     49 import numpy as np
     50 import pandas as pd

~/.local/lib/python3.8/site-packages/fsspec/__init__.py in <module>
     10 from . import _version, caching
     11 from .callbacks import Callback
---> 12 from .core import get_fs_token_paths, open, open_files, open_local
     13 from .exceptions import FSTimeoutError
     14 from .mapping import FSMap, get_mapper

~/.local/lib/python3.8/site-packages/fsspec/core.py in <module>
     16     caches,
     17 )
---> 18 from .compression import compr
     19 from .registry import filesystem, get_filesystem_class
     20 from .utils import (

~/.local/lib/python3.8/site-packages/fsspec/compression.py in <module>
     68
     69
---> 70 register_compression("zip", unzip, "zip")
     71 register_compression("bz2", BZ2File, "bz2")
     72

~/.local/lib/python3.8/site-packages/fsspec/compression.py in register_compression(name, callback, extensions, force)
     44
     45     for ext in extensions:
---> 46         if ext in fsspec.utils.compressions and not force:
     47             raise ValueError(
     48                 "Duplicate compression file extension: %s (%s)" % (ext, name)

AttributeError: partially initialized module 'fsspec' has no attribute 'utils' (most likely due to a circular import)
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.4
- Platform: Jupyter notebook
- Python version: 3.8.10
- PyArrow version: 7.0.0


Dataset viewer issue for IndicParaphrase- the preview doesn't show

dataset-viewer

Related to the dataset viewer on huggingface.co

It seems to have been fixed:

<img width="1534" alt="Capture d’écran 2022-04-12 à 14 10 07" src="https://user-images.githubusercontent.com/1676121/162959599-6b7fef7c-8411-4e03-8f00-90040a658079.png">

## Dataset viewer issue for '*IndicParaphrase*'

**Link:** *[IndicParaphrase](https://huggingface.co/datasets/ai4bharat/IndicParaphrase/viewer/hi/validation)*

*The preview of the dataset doesn't come up.
The error on the console is:
Status code:   400
Exception:     FileNotFoundError
Message:       [Errno 2] No such file or directory: '/home/hf/datasets-preview-backend/hi_IndicParaphrase_v1.0.tar'*

Am I the one who added this dataset ? Yes


Add MetaShift dataset

_The documentation is not available anymore as the PR was closed or merged._

@lhoestq Please could you review this when you get time. Thank you.

Thanks a lot for your inputs @mariosasko .
> Maybe we can add the generated meta-graphs to the card as images (with attributions)?

Yes. We can do this for the default set of classes. Will add this.

> Would be cool if we could have them as additional configs. Also, maybe we could have configs that expose [image metadata](https://github.com/Weixin-Liang/MetaShift/tree/main/dataset/meta_data) from the https://nlp.stanford.edu/data/gqa/sceneGraphs.zip file (this file is downloaded in the script but not used).

I'll try adding the bonus section as additional config.
Regarding exposing the image metadata with a config parameter, how will we showcase/display this information ?

> Regarding exposing the image metadata with a config parameter, how will we showcase/display this information ?

Oh, I forgot to mention that. Let's add a `Dataset Usage` section to the card to document the params (similar to this: https://huggingface.co/datasets/electricity_load_diagrams#dataset-usage). Also, feel free to add the constants that can be tuned as config params (e.g. `IMAGE_SUBSET_SIZE_THRESHOLD` or the `5` in `len(subject_data) <= 5`).

Okay. Got it. Will add these and constants as config parameters.

The image metadata from scene graphs looks like this :
```json
{
    "2407890": {
        "width": 640,
        "height": 480,
        "location": "living room",
        "weather": none,
        "objects": {
            "271881": {
                "name": "chair",
                "x": 220,
                "y": 310,
                "w": 50,
                "h": 80,
                "attributes": ["brown", "wooden", "small"],
                "relations": {
                    "32452": {
                        "name": "on",
                        "object": "275312"
                    },
                    "32452": {
                        "name": "near",
                        "object": "279472"
                    }
                }
            }
        }
    }
}
```
``load_dataset("metashift", selected_classes=["cat", "dog", ...], image_metadata=True)``
How do we showcase/display the image metadata(json) information ?

> How do we showcase/display the image metadata(json) information ?

We can add the JSON fields as keys to the features dict:
```python
  if self.config.image_metadata:
      features.update({"width": Value("int"), "height": Value("int"), "location": Value("string"), ...})
```

P.S. Would rename `image_metadata`  to `with_image_metadata`

I have added the following :
- Added the meta-graphs to the card as images under the Section "Dataset Meta-Graphs".
- Generate the Attributes-Dataset using config parameter. [ [Link](https://github.com/Weixin-Liang/MetaShift#bonus-generate-the-metashift-attributes-dataset-subsets-defined-by-subject-attributes) ]
- Expose image metadata using config parameter.
Format of the image metadata is as follows : [Link](https://cs.stanford.edu/people/dorarad/gqa/download.html)
I have modified the "Objects" which is dict to a list of dicts with an additional parameter named object_id.
I have defined the structure as follows :
```
{
    "width": datasets.Value("int64"),
    "height": datasets.Value("int64"),
    "location": datasets.Value("string"),
    "weather": datasets.Value("string"),
    "objects": datasets.Sequence(
        {
            "object_id": datasets.Value("string"),
            "name": datasets.Value("string"),
            "x": datasets.Value("int64"),
            "y": datasets.Value("int64"),
            "w": datasets.Value("int64"),
            "h": datasets.Value("int64"),
            "attributes": datasets.Sequence(datasets.Value("string")),
            "relations": datasets.Sequence(
                {
                    "name": datasets.Value("string"),
                    "object": datasets.Value("string"),
                }
            ),
        }
    ),
}
```
Problem is that objects is not being shown as list of dicts. The output looks as follows :

> metashift_dataset['train'][0]

```json
{'image_id': '2338755', 'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x281 at 0x7F066C5A49D0>, 'label': 0, 'context': 'ground', 'width': 500, 'height': 281, 'location': None, 'weather': None, 'objects': {'object_id': ['3070704', '3070705', '3070706', '2416713', '3070702', '2790660', '3063157', '2354960', '2037127', '2392939', '2912743', '2125407', '2735257', '3260906', '2351018', '3288269', '3699852', '2734378', '3421201', '2863115'], 'name': ['bicycle', 'bicycle', 'bicycle', 'boot', 'bicycle', 'motorcycle', 'pepperoni', 'head', 'building', 'wall', 'shorts', 'people', 'wheel', 'bricks', 'man', 'cat', 'boot', 'door', 'ground', 'building'], 'x': [137, 371, 458, 215, 468, 399, 368, 245, 0, 140, 260, 284, 138, 451, 339, 187, 210, 26, 0, 313], 'y': [116, 86, 94, 150, 91, 80, 107, 22, 0, 44, 109, 69, 145, 226, 69, 22, 230, 0, 119, 0], 'w': [197, 27, 15, 73, 24, 53, 9, 37, 289, 46, 43, 30, 74, 28, 35, 116, 53, 107, 500, 55], 'h': [126, 25, 38, 128, 43, 50, 16, 44, 158, 73, 51, 52, 97, 15, 73, 252, 46, 147, 162, 77], 'attributes': [[], [], [], ['white'], [], [], [], [], [], [], [], [], [], [], [], ['white'], ['white'], ['large', 'black'], ['brick'], []], 'relations': [{'name': ['to the left of'], 'object': ['3260906']}, {'name': ['to the left of', 'to the right of', 'to the right of', 'to the left of', 'to the right of', 'to the left of', 'to the right of'], 'object': ['3070706', '2351018', '2125407', '2790660', '2037127', '3070702', '3288269']}, {'name': ['to the right of', 'to the right of', 'to the left of', 'to the right of', 'to the right of'], 'object': ['2351018', '3070705', '3070702', '2790660', '3063157']}, {'name': ['to the right of'], 'object': ['2735257']}, {'name': ['to the right of', 'to the right of', 'to the right of', 'to the right of', 'to the right of'], 'object': ['2351018', '2790660', '3070706', '3070705', '3063157']}, {'name': ['to the right of', 'to the right of', 'to the left of', 'to the left of', 'to the right of', 'to the right of', 'to the right of', 'to the right of'], 'object': ['3070705', '2351018', '3070702', '3070706', '3063157', '2125407', '2037127', '3288269']}, {'name': ['to the right of', 'to the left of', 'to the left of', 'to the right of', 'to the right of', 'to the left of', 'to the right of'], 'object': ['2037127', '3070706', '3070702', '2912743', '3288269', '2790660', '2125407']}, {'name': ['to the left of', 'to the right of'], 'object': ['2863115', '2734378']}, {'name': ['to the left of', 'to the left of', 'to the left of', 'to the left of', 'to the left of', 'to the left of'], 'object': ['3070705', '2351018', '3063157', '2125407', '2790660', '2863115']}, {'name': ['to the left of', 'to the right of', 'to the left of'], 'object': ['2125407', '2734378', '3288269']}, {'name': ['to the left of', 'on', 'to the left of'], 'object': ['2351018', '3288269', '3063157']}, {'name': ['to the left of', 'to the left of', 'to the right of', 'to the left of', 'to the right of', 'to the left of'], 'object': ['3063157', '2351018', '2037127', '3070705', '2392939', '2790660']}, {'name': ['to the left of', 'to the left of'], 'object': ['2416713', '3288269']}, {'name': ['to the right of'], 'object': ['3070704']}, {'name': ['to the right of', 'to the left of', 'to the right of', 'to the left of', 'to the left of', 'to the right of', 'to the left of', 'to the right of', 'walking down'], 'object': ['2037127', '2790660', '2125407', '3070705', '3070706', '2912743', '3070702', '3288269', '3421201']}, {'name': ['to the right of', 'to the right of', 'to the left of', 'to the right of', 'to the left of', 'to the left of', 'to the left of', 'to the left of'], 'object': ['2392939', '2734378', '2790660', '2735257', '3063157', '3070705', '2351018', '2863115']}, {'name': [], 'object': []}, {'name': ['of', 'to the left of', 'to the left of', 'to the left of'], 'object': ['2037127', '2354960', '3288269', '2392939']}, {'name': [], 'object': []}, {'name': ['to the right of', 'to the right of', 'to the right of'], 'object': ['2037127', '3288269', '2354960']}]}}
```
Expected output of image_metadata would be :
```
{'height': 281,
 'location': None,
 'objects': [{'attributes': [],
              'h': 126,
              'name': 'bicycle',
              'object_id': '3070704',
              'relations': [{'name': 'to the left of', 'object': '3260906'}],
              'w': 197,
              'x': 137,
              'y': 116},
             {'attributes': [],
              'h': 25,
              'name': 'bicycle',
              'object_id': '3070705',
              'relations': [{'name': 'to the left of', 'object': '3070706'},
                            {'name': 'to the right of', 'object': '2351018'},
                            {'name': 'to the right of', 'object': '2125407'},
                            {'name': 'to the left of', 'object': '2790660'},
                            {'name': 'to the right of', 'object': '2037127'},
                            {'name': 'to the left of', 'object': '3070702'},
                            {'name': 'to the right of', 'object': '3288269'}],
              'w': 27,
              'x': 371,
              'y': 86},
             {'attributes': ['white'],
              'h': 252,
              'name': 'cat',
              'object_id': '3288269',
              'relations': [{'name': 'to the right of', 'object': '2392939'},
                            {'name': 'to the right of', 'object': '2734378'},
                            {'name': 'to the left of', 'object': '2790660'},
                            {'name': 'to the right of', 'object': '2735257'},
                            {'name': 'to the left of', 'object': '3063157'},
                            {'name': 'to the left of', 'object': '3070705'},
                            {'name': 'to the left of', 'object': '2351018'},
                            {'name': 'to the left of', 'object': '2863115'}],
              'w': 116,
              'x': 187,
              'y': 22},
             {'attributes': ['white'],
              'h': 46,
              'name': 'boot',
              'object_id': '3699852',
              'relations': [],
              'w': 53,
              'x': 210,
              'y': 230},
             .
             .
             .
             {'attributes': ['large', 'black'],
              'h': 147,
              'name': 'door',
              'object_id': '2734378',
              'relations': [{'name': 'of', 'object': '2037127'},
                            {'name': 'to the left of', 'object': '2354960'},
                            {'name': 'to the left of', 'object': '3288269'},
                            {'name': 'to the left of', 'object': '2392939'}],
              'w': 107,
              'x': 26,
              'y': 0},
             {'attributes': ['brick'],
              'h': 162,
              'name': 'ground',
              'object_id': '3421201',
              'relations': [],
              'w': 500,
              'x': 0,
              'y': 119},
             {'attributes': [],
              'h': 77,
              'name': 'building',
              'object_id': '2863115',
              'relations': [{'name': 'to the right of', 'object': '2037127'},
                            {'name': 'to the right of', 'object': '3288269'},
                            {'name': 'to the right of', 'object': '2354960'}],
              'w': 55,
              'x': 313,
              'y': 0}],
 'weather': None,
 'width': 500}

```

May I know how to get the list of dicts representation correctly ?

---
To-Do :

- [x] Generate dataset_infos.json file.
- [x] Add “Dataset Usage” section in the cards and write about the config parameters.
- [x] Add the constants that can be tuned as config params.

> Problem is that objects is not being shown as list of dicts. The output looks as follows :

That's expected. We convert a sequence of dictionaries to a dictionary of sequences to keep the formatting aligned with Tensorflow Datasets. You could disable this behavior by replacing `"objects": datasets.Sequence(object_fields_dict)` with `"objects": [object_fields_dict]`, but that's not what we usually do, so let's keep it like that.

Also, to limit the size of the dataset repo, please remove the pushed images and pass URLs to the images instead under the `src` attribute (and specify `alt` in case the URLs go down).

I'll do a proper review again after you are finished with the dummy data.

> That's expected.

Okay. Got it. Thanks. I thought I was doing something wrong.

> Also, to limit the size of the dataset repo, please remove the pushed images and pass URLs to the images instead under the src attribute (and specify alt in case the URLs go down).

Sure. Where do we host these images ? Can I upload them to any free image hosting platform or is there any particular website you use ?

> I'll do a proper review again after you are finished with the dummy data.

Sure. Thanks. I'm working on this part. Will update you.

Update :
- I have generated the dataset_infos.json file.

> I suggest you try to generate the dataset_infos.json file first, and then I can help with the dummy data.

I am having issues creating the dummy data. I get the following which I use the command :

`datasets-cli dummy_data datasets/metashift`

```
Dataset metashift with config MetashiftConfig(name='metashift', version=1.0.0, data_dir=None, data_files=None, description=None) seems to already open files in the method `_split_generators(...)`. You might consider to instead only open files in the method `_generate_examples(...)` instead. If this is not possible the dummy data has to be created with less guidance. Make sure you create the file dummy_data/full-candidate-subsets.pkl.
Traceback (most recent call last):
  File "datasets-cli", line 33, in <module>
    sys.exit(load_entry_point('datasets', 'console_scripts', 'datasets-cli')())
  File "/datasets/commands/datasets_cli.py", line 33, in main
    service.run()
  File "/datasets/commands/dummy_data.py", line 324, in run
    dataset_builder=dataset_builder, mock_dl_manager=mock_dl_manager
  File "/datasets/commands/dummy_data.py", line 407, in _print_dummy_data_instructions
    for split in generator_splits:
UnboundLocalError: local variable 'generator_splits' referenced before assignment
```

> Feel free to host the images online (on imgur for example) :)

Sure. Will do that.

Thanks for the explanation regarding the dummy data zip files. I will try it out and let you know.

Instead of uploading the images to a hosting service, you can directly reference their GitHub URLs (open the image in the MetaShift repo -> click Download -> copy the image URL). For instance, this is the URL of one of the images:`https://raw.githubusercontent.com/Weixin-Liang/MetaShift/main/docs/figures/Cat-MetaGraph.jpg`. Also, feel free to replace `main` with the most recent commit hash in the copied URLs to make them more robust.

@mariosasko I've actually created metagraphs for all the default classes other than those present in the GitHub Repo and included all of them. :)  The Repo has them only for two classes.

In case we want to limit the no.of meta graphs included, we can stick to the github URLs from the repo itself.

Update :
- I could add the dummy data and get the dummy data test to work. Since we have a preprocessing step on the dataset, one of the .pkl file size is on the higher side. This was done for the tests to pass. I hope that is okay. The dummy.zip file size is about 273K.

To-Do :
- [x] Update Dataset Structure in the data cards to include Data Instances when config parameters are used.

Please could you review when you get time. Thank you.

Thanks a lot for your suggestions, Mario. The thing I learnt from the review is that I need to make better sentence formations. I will keep this in mind. :)

Thanks a lot for your support. @mariosasko  and @lhoestq .

> Super impressed by your work on this, congrats :)

Its my first dataset contribution to the 🤗 Datasets library, I'm super excited. Thank you. :)

Also, I think we can close this request issue now, [#3813](https://github.com/huggingface/datasets/issues/3813)

This PR adds the MetaShift dataset.

Dataset Request  : Add MetaShift dataset [#3813](https://github.com/huggingface/datasets/issues/3813)

@lhoestq As discussed,
- I have copied the preprocessing script and modified it as required to not create new directories and folders and instead yield the images.
-  I do the preprocessing in _split_generators to get the required data which is then passed to _generate_examples.
-  Beyond the generated MetaShift dataset, the original preprocess script also generates the meta-graphs for each class, I have currently not included this part. [ Ref : [Link](https://github.com/Weixin-Liang/MetaShift#generate-full-metashift) ]
-  There is a Bonus section, the authors share.  I have currently not included this part. [ Ref : [Link](https://github.com/Weixin-Liang/MetaShift#bonus-generate-the-metashift-attributes-dataset-subsets-defined-by-subject-attributes) ]
- I had a basic test script which downloaded the dataset and tested the basic functionality. Things seems fine.
For real data, I performed the following test :
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_metashift
============================================== test session starts ===============================================
platform linux -- Python 3.7.11, pytest-7.0.1, pluggy-1.0.0
rootdir: ./datasets
plugins: hydra-core-1.1.1, datadir-1.3.1, forked-1.4.0, xdist-2.5.0
collected 1 item

tests/test_dataset_common.py .                                                                             [100%]

========================================= 1 passed in 4821.25s (1:20:21) =========================================
```
- I couldn't get the dummy dataset. Need some inputs here.
Error as follows :
```
Using custom data configuration default
Dataset metashift with config None seems to already open files in the method `_split_generators(...)`. You might consider to instead only open files in the method `_generate_examples(...)` instead. If this is not possible the dummy data has to be created with less guidance. Make sure you create the file dummy_data/full-candidate-subsets.pkl.
    for split in generator_splits:
UnboundLocalError: local variable 'generator_splits' referenced before assignment
```

To-Do :

- [x] Currently I am using the default _SELECTED_CLASSES. I need to use config option here as suggested
- [x] Complete fields in the Dataset Card.
- [x] Tagging the dataset using the Datasets Tagging app.

Need your help and suggestions for improvement. Thank you


Add exact match metric

Adding the exact match metric and its metric card.

Note: Some of the tests have failed, but I wanted to make a PR anyway so that the rest of the code can be reviewed if anyone has time. I'll look into + work on fixing the failed tests when I'm back online after the weekend


Create README.md for WER metric

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3898). All of your documentation changes will be reflected on that endpoint.

For ASR you can probably ping @patrickvonplaten

Ah only noticed now that ` # Values from popular papers` is from a template. @lhoestq @sashavor - not really sure if this section is useful in general really.

IMO, it's more confusing/misleading than it helps. E.g. a value of 0.03 WER on a fake read-out audio dataset is not better than a WER of 0.3 on a real-world noisy, conversational audio dataset. I think the same holds true for other metrics no? I can think of very little metrics where a metric value is not dataset dependent. E.g. perplexity is super dataset dependent, summarization metrics like ROUGE as well, ...

Also, I don't really see what this section tries to achieve - is the idea here to give the reader some papers that use this metric to better understand in which context it is used? Should we maybe rename the section to `Popular papers making use of this metric` or something?

I put "Values from popular papers" as a subsection of "Output values" -- I hope that's a compromise that works for everyone :hugs:

Proposing a draft WER metric card, @lhoestq I'm not very certain about "Values from popular papers" -- I don't know ASR very well, what do you think of the examples I found?


Align tqdm control/cache control with Transformers

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3897). All of your documentation changes will be reflected on that endpoint.

This PR:
* aligns the `tqdm` logic with Transformers (follows https://github.com/huggingface/transformers/pull/15167) by moving the code to `utils/logging.py`, adding `enable_progres_bar`/`disable_progres_bar` and removing `set_progress_bar_enabled` (a note for @lhoestq: I'm not adding `logging.tqdm` to the public namespace in this PR to avoid the situation where `from datasets import *; tqdm` would overshadow the standard `tqdm`
* aligns the cache control with the new `tqdm` logic by adding `enable_caching`/`disable_caching` to the  public namespace and deprecating `set_caching_enabled` (not fully removing it because it's used more often than `set_progress_bar_enabled` and has a dedicated example in the old docs)

Fix #3586


Missing google file for `multi_news` dataset

`datasets` 1.18.4 fixes the issue when you load the dataset with `load_dataset`.

When loading in streaming mode, the fix is indeed on https://github.com/huggingface/datasets/pull/3843 which will be merged soon :)

That is. The PR #3843 was just opened a bit later we had made our 1.18.4 patch release...
Once merged, that will fix this issue.

OK. Should fix the viewer for 50 datasets

<img width="148" alt="Capture d’écran 2022-03-14 à 11 51 02" src="https://user-images.githubusercontent.com/1676121/158157853-6c544a47-2d6d-4ac4-964a-6f10951ec36b.png">

## Dataset viewer issue for '*multi_news*'

**Link:** https://huggingface.co/datasets/multi_news

```
Server error

Status code:   400
Exception:     FileNotFoundError
Message:       https://drive.google.com/uc?export=download&id=1vRY2wM6rlOZrf9exGTm5pXj5ExlVwJ0C/multi-news-original/train.src
```

Am I the one who added this dataset ? No


Fix code examples indentation

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3895). All of your documentation changes will be reflected on that endpoint.

My last commit should have fixed it, I don't know why the dev doc build is not showing my last changes

Let me merge this and we can see on `master` how it renders, until the dev doc build is fixed

Some code examples are currently not rendered correctly. I think this is because they are over-indented

cc @mariosasko


[docs] make dummy data creation optional

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3894). All of your documentation changes will be reflected on that endpoint.

The dev doc build rendering doesn't seem to be updated with my last commit for some reason

Merging it anyway since I'd like to share this page with users 🙃

Related to #3507 : dummy data for datasets created on the Hugging Face Hub are optional.

We can discuss later to make them optional for datasets in this repository as well


Add default branch for doc building

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3893). All of your documentation changes will be reflected on that endpoint.

Yes! And when we discovered on the Transformers side that this check fails on the GitHub actions, we added a config attribute to have a default. Setting in Transformers fixed the issue of the doc being deployed to main, so porting the fix here too :-)

Since other libraries use `main` as their default branch and it's now the standard default, you have to specify a different name in the doc config if you're using `master` like datasets (`doc-builder` tries to guess it, but in the job, we have weird checkout of merge commits so it doesn't always manage to get it right).

This PR makes sure it will always use master for the dev doc (until you decide to switchto main)


Fix CLI test checksums

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3892). All of your documentation changes will be reflected on that endpoint.

Feel free to merge if it's good for you :)

I've added a test @lhoestq. Once all green, I'll merge.

Last failing tests do not have nothing to do with this PR.

Previous PR:
- #3796

introduced a side effect: `datasets-cli test` generates `dataset_infos.json` with `None` checksum values.

See:
- #3805

This PR introduces a way for `datasets-cli test` to force to record infos, even if `verify_infos=False`

Close #3848.

CC: @craffel


Fix race condition in doc build

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3891). All of your documentation changes will be reflected on that endpoint.

Following https://github.com/huggingface/datasets/runs/5499386744 it seems that race conditions that create issues when updating the doc. I took the same approach as in `transformers` to fix race conditions


Update beans download urls

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3890). All of your documentation changes will be reflected on that endpoint.

@albertvillanova Thanks for investigating and fixing that issue. I regenerated the `dataset_infos.json` file.

Replace the old URLs with the Hub [URLs](https://huggingface.co/datasets/beans/tree/main/data).

Also reported by @stevhliu.

Fix #3889


Cannot load beans dataset (Couldn't reach the dataset)

Hi ! A pull request is open to fix the dataset, we'll release a patch soon with a new release of `datasets` :)

## Describe the bug
The beans dataset is unavailable to download.

## Steps to reproduce the bug
```python
from datasets import load_dataset
ds = load_dataset('beans')
```

## Expected results
The dataset would be downloaded with no issue.

## Actual results

```
ConnectionError: Couldn't reach https://storage.googleapis.com/ibeans/train.zip (error 403)
```

[It looks like the billing of this project has been disabled because it is associated with a delinquent account.](https://storage.googleapis.com/ibeans/train.zip )

## Environment info
Google Colab


IterableDataset columns and feature types

generic discussion

Generic discussion on the library

@alvarobartt I've assigned you the issue since I'm not actively working on it.

Cool thanks @mariosasko I'll try to fix it in the upcoming days, thanks!

@lhoestq so in order to address what’s not completed in this issue, do you think it makes sense to add a param `features` to `IterableDataset.map` so that the output features right after the `map` are defined there?

Yes that would be ideal IMO, thanks again for the help :)

@lhoestq cool then if you agree I can work on that! I’ll also update the docs accordingly once done, thanks!

I've already started with a PR as a draft @lhoestq, should we also try to look for a way to explicitly request pre-fetching right after a map operation is applied, so that the features are inferred if the user says explicitly so? Thanks!

> should we also try to look for a way to explicitly request pre-fetching right after a map operation is applied, so that the features are inferred if the user says explicitly so?

Right now one can use `ds = ds._resolve_features()` do to so. It can be used after `map` or `load_dataset` if the features are not known. Maybe we can make this method public ?

Right now, an IterableDataset (e.g. when streaming a dataset) doesn't require to know the list of columns it contains, nor their types: `my_iterable_dataset.features` may be `None`

However it's often interesting to know the column types and types. This helps knowing what's inside your dataset without having to manually check a few examples, and this is useful to prepare a processing pipeline or to train models.

Here are a few cases that lead to `features` being `None`:

1. when loading a dataset with `load_dataset` on CSV, JSON Lines, etc. files: type inference is only done when iterating over the dataset
2. when calling `map`, because we don't know in advance what's the output of the user's function passed to `map`
3. when calling `rename_columns`, `remove_columns`, etc. because they rely on `map`

Things we can consider, for each point above:

1.a infer the type automatically from the first samples on the dataset using prefetching, when the dataset builder doesn't provide the `features`
2.a allow the user to specify the `features` as an argument to `map` (this would be consistent with the non-streaming API)
2.b prefetch the first output value to infer the type
3.a don't rely on `map` directly and reuse the previous `features` and rename/remove the corresponding ones

The thing is that prefetching can take a few seconds, while the operations above are instantaneous since no data are downloaded. Therefore I'm not sure whether this solution may be worth it. Maybe prefetching could also be done when explicitly asked by the user

cc @mariosasko @albertvillanova


ImageFolder improvements

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3887). All of your documentation changes will be reflected on that endpoint.

This PR adds the following improvements to the `imagefolder` dataset:
* skip the extract step for image files (as discussed in https://github.com/huggingface/datasets/pull/2830#discussion_r816817919)
* option to drop labels by setting `drop_labels=True` (useful for image pretraining cc @NielsRogge). This is faster than loading a dataset and removing the `label` column because we don't need to iterate over the files to infer class labels.


Retry HfApi call inside push_to_hub when 504 error

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3886). All of your documentation changes will be reflected on that endpoint.

I made it more robust by increasing the wait time, and I also added some logs when a request is retried. Let me know if it's ok for you

At the end you did not set the agreed max value of 60s.

Moreover, with the new numbers, there is a slight contradiction: although you set max_retries=5, we will only make 4 retries at most because of the combined values of `base_wait_time` and `max_wait_time`.

Yea I thought that in total we could wait 1min, but if we have a max_wait_time of 20sec between each request it's fine IMO

> Moreover, with the new numbers, there is a slight contradiction: although you set max_retries=5, we will only make 4 retries at most because of the combined values of base_wait_time and max_wait_time.

What makes you think this ? If the exponential wait time becomes bigger than `max_wait_time` then it still does the retry, but after a wait time of `max_wait_time`

Sorry, I meant 4 retries **with exponential backoff**; the fifth one is with constant backoff.

OK, and one question: do you think that the retries do not affect the time the server needs to be operational again and able to process the request? I guess that if does not affect, then the cause are other users' requests, or others; not our specific request.

Just to be sure:
- Then 20s at most between consecutive requests do not impact the server.
- And we expect after a total of 5 retries (within a total 50s of wait time + request processing/uploading time), the server should be able to come back to normality.

> do you think that the retries do not affect the time the server needs for being able to process the request (I guess in this case the cause are other users' requests, or other causes; not our specific request).

Yes I don't think the retries would affect the server, I think the cause of the 504 errors is elsewhere

> Just to be sure:
>
> Then 20s at most between consecutive requests do not impact the server.
> And we expect after a total of 5 retries (within a total 50s of wait time + request processing/uploading time), the server should be able to come back to normality.

Yes I think it's fine for now, we can still adapt this later if needed

Will be curious to see the impact of this in terms of upload reliability! Don't forget to let us know when you have more data. cc @huggingface/moon-landing-back

Ass suggested by @lhoestq in #3872, this PR:
- Implements a retry function
- Retries HfApi call inside `push_to_hub` when 504 error. To be agreed:
  - max_retries = 2 (at 0.5 and 1 seconds)

Fix #3872.


Fix some shuffle docs

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3885). All of your documentation changes will be reflected on that endpoint.

Following #3842 some docs were still outdated (with `buffer_size` as the first argument)


Fix bug in METEOR metric due to nltk version

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3884). All of your documentation changes will be reflected on that endpoint.

Fix #3883.