Error loading file audio when downloading the Common Voice dataset directly from the Hub

Something isn't working

Hi ! It could an issue with torchaudio, which version of torchaudio are you using ? Can you also try updating `datasets` to 2.0.0 and see if it works ?

I _might_ have a similar issue. I'm trying to use the librispeech_asr dataset and read it with soundfile.

```python
from datasets import load_dataset, load_metric
from transformers import Speech2TextForConditionalGeneration, Speech2TextProcessor
import soundfile as sf

librispeech_eval = load_dataset("librispeech_asr", "clean", split="test")  # change to "other" for other test dataset
wer = load_metric("wer")

model = Speech2TextForConditionalGeneration.from_pretrained("facebook/s2t-small-librispeech-asr").to("cuda")
processor = Speech2TextProcessor.from_pretrained("facebook/s2t-small-librispeech-asr", do_upper_case=True)

def map_to_array(batch):
    speech, _ = sf.read(batch["file"])
    batch["speech"] = speech
    return batch

librispeech_eval = librispeech_eval.map(map_to_array)

def map_to_pred(batch):
    features = processor(batch["speech"], sampling_rate=16000, padding=True, return_tensors="pt")
    input_features = features.input_features.to("cuda")
    attention_mask = features.attention_mask.to("cuda")

    gen_tokens = model.generate(input_ids=input_features, attention_mask=attention_mask)
    batch["transcription"] = processor.batch_decode(gen_tokens, skip_special_tokens=True)
    return batch

result = librispeech_eval.map(map_to_pred, batched=True, batch_size=8, remove_columns=["speech"])

print("WER:", wer(predictions=result["transcription"], references=result["text"]))
```

The code is taken directly from "https://huggingface.co/facebook/s2t-small-librispeech-asr".

The short error code is "RuntimeError: Error opening '6930-75918-0000.flac': System error." (it can't find the first file), and I agree, I can't find the file either. The dataset has downloaded correctly (it says), but on the location, there are only ".arrow" files, no ".flac" files.

**Error message:**

```python
RuntimeError                              Traceback (most recent call last)
Input In [15], in <cell line: 16>()
     13     batch["speech"] = speech
     14     return batch
---> 16 librispeech_eval = librispeech_eval.map(map_to_array)
     18 def map_to_pred(batch):
     19     features = processor(batch["speech"], sampling_rate=16000, padding=True, return_tensors="pt")

File C:\ProgramData\Miniconda3\envs\noise_cancel\lib\site-packages\datasets\arrow_dataset.py:1953, in Dataset.map(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   1950 disable_tqdm = not logging.is_progress_bar_enabled()
   1952 if num_proc is None or num_proc == 1:
-> 1953     return self._map_single(
   1954         function=function,
   1955         with_indices=with_indices,
   1956         with_rank=with_rank,
   1957         input_columns=input_columns,
   1958         batched=batched,
   1959         batch_size=batch_size,
   1960         drop_last_batch=drop_last_batch,
   1961         remove_columns=remove_columns,
   1962         keep_in_memory=keep_in_memory,
   1963         load_from_cache_file=load_from_cache_file,
   1964         cache_file_name=cache_file_name,
   1965         writer_batch_size=writer_batch_size,
   1966         features=features,
   1967         disable_nullable=disable_nullable,
   1968         fn_kwargs=fn_kwargs,
   1969         new_fingerprint=new_fingerprint,
   1970         disable_tqdm=disable_tqdm,
   1971         desc=desc,
   1972     )
   1973 else:
   1975     def format_cache_file_name(cache_file_name, rank):

File C:\ProgramData\Miniconda3\envs\noise_cancel\lib\site-packages\datasets\arrow_dataset.py:519, in transmit_tasks.<locals>.wrapper(*args, **kwargs)
    517     self: "Dataset" = kwargs.pop("self")
    518 # apply actual function
--> 519 out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    520 datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    521 for dataset in datasets:
    522     # Remove task templates if a column mapping of the template is no longer valid

File C:\ProgramData\Miniconda3\envs\noise_cancel\lib\site-packages\datasets\arrow_dataset.py:486, in transmit_format.<locals>.wrapper(*args, **kwargs)
    479 self_format = {
    480     "type": self._format_type,
    481     "format_kwargs": self._format_kwargs,
    482     "columns": self._format_columns,
    483     "output_all_columns": self._output_all_columns,
    484 }
    485 # apply actual function
--> 486 out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    487 datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    488 # re-apply format to the output

File C:\ProgramData\Miniconda3\envs\noise_cancel\lib\site-packages\datasets\fingerprint.py:458, in fingerprint_transform.<locals>._fingerprint.<locals>.wrapper(*args, **kwargs)
    452             kwargs[fingerprint_name] = update_fingerprint(
    453                 self._fingerprint, transform, kwargs_for_fingerprint
    454             )
    456 # Call actual function
--> 458 out = func(self, *args, **kwargs)
    460 # Update fingerprint of in-place transforms + update in-place history of transforms
    462 if inplace:  # update after calling func so that the fingerprint doesn't change if the function fails

File C:\ProgramData\Miniconda3\envs\noise_cancel\lib\site-packages\datasets\arrow_dataset.py:2318, in Dataset._map_single(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, disable_tqdm, desc, cache_only)
   2316 if not batched:
   2317     for i, example in enumerate(pbar):
-> 2318         example = apply_function_on_filtered_inputs(example, i, offset=offset)
   2319         if update_data:
   2320             if i == 0:

File C:\ProgramData\Miniconda3\envs\noise_cancel\lib\site-packages\datasets\arrow_dataset.py:2218, in Dataset._map_single.<locals>.apply_function_on_filtered_inputs(inputs, indices, check_same_num_examples, offset)
   2216 if with_rank:
   2217     additional_args += (rank,)
-> 2218 processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
   2219 if update_data is None:
   2220     # Check if the function returns updated examples
   2221     update_data = isinstance(processed_inputs, (Mapping, pa.Table))

File C:\ProgramData\Miniconda3\envs\noise_cancel\lib\site-packages\datasets\arrow_dataset.py:1913, in Dataset.map.<locals>.decorate.<locals>.decorated(item, *args, **kwargs)
   1909 decorated_item = (
   1910     Example(item, features=self.features) if not batched else Batch(item, features=self.features)
   1911 )
   1912 # Use the LazyDict internally, while mapping the function
-> 1913 result = f(decorated_item, *args, **kwargs)
   1914 # Return a standard dict
   1915 return result.data if isinstance(result, LazyDict) else result

Input In [15], in map_to_array(batch)
     11 def map_to_array(batch):
---> 12     speech, _ = sf.read(batch["file"])
     13     batch["speech"] = speech
     14     return batch

File C:\ProgramData\Miniconda3\envs\noise_cancel\lib\site-packages\soundfile.py:256, in read(file, frames, start, stop, dtype, always_2d, fill_value, out, samplerate, channels, format, subtype, endian, closefd)
    170 def read(file, frames=-1, start=0, stop=None, dtype='float64', always_2d=False,
    171          fill_value=None, out=None, samplerate=None, channels=None,
    172          format=None, subtype=None, endian=None, closefd=True):
    173     """Provide audio data from a sound file as NumPy array.
    174
    175     By default, the whole file is read from the beginning, but the
   (...)
    254
    255     """
--> 256     with SoundFile(file, 'r', samplerate, channels,
    257                    subtype, endian, format, closefd) as f:
    258         frames = f._prepare_read(start, stop, frames)
    259         data = f.read(frames, dtype, always_2d, fill_value, out)

File C:\ProgramData\Miniconda3\envs\noise_cancel\lib\site-packages\soundfile.py:629, in SoundFile.__init__(self, file, mode, samplerate, channels, subtype, endian, format, closefd)
    626 self._mode = mode
    627 self._info = _create_info_struct(file, mode, samplerate, channels,
    628                                  format, subtype, endian)
--> 629 self._file = self._open(file, mode_int, closefd)
    630 if set(mode).issuperset('r+') and self.seekable():
    631     # Move write position to 0 (like in Python file objects)
    632     self.seek(0)

File C:\ProgramData\Miniconda3\envs\noise_cancel\lib\site-packages\soundfile.py:1183, in SoundFile._open(self, file, mode_int, closefd)
   1181 else:
   1182     raise TypeError("Invalid file: {0!r}".format(self.name))
-> 1183 _error_check(_snd.sf_error(file_ptr),
   1184              "Error opening {0!r}: ".format(self.name))
   1185 if mode_int == _snd.SFM_WRITE:
   1186     # Due to a bug in libsndfile version <= 1.0.25, frames != 0
   1187     # when opening a named pipe in SFM_WRITE mode.
   1188     # See http://github.com/erikd/libsndfile/issues/77.
   1189     self._info.frames = 0

File C:\ProgramData\Miniconda3\envs\noise_cancel\lib\site-packages\soundfile.py:1357, in _error_check(err, prefix)
   1355 if err != 0:
   1356     err_str = _snd.sf_error_number(err)
-> 1357     raise RuntimeError(prefix + _ffi.string(err_str).decode('utf-8', 'replace'))

RuntimeError: Error opening '6930-75918-0000.flac': System error.
```

**Package versions:**
```python
python: 3.9
transformers: 4.17.0
datasets: 2.0.0
SoundFile: 0.10.3.post1
```

Hi ! In `datasets` 2.0 can access the audio array with `librispeech_eval[0]["audio"]["array"]` already, no need to use `map_to_array`. See our documentation on [how to process audio data](https://huggingface.co/docs/datasets/audio_process) :)

cc @patrickvonplaten we will need to update the readme at [facebook/s2t-small-librispeech-asr](https://huggingface.co/facebook/s2t-small-librispeech-asr) as well as https://huggingface.co/docs/transformers/model_doc/speech_to_text

Thanks!

And sorry for posting this problem in what turned on to be an unrelated thread.

I rewrote the code, and the model works. The WER is 0.137 however, so I'm not sure if I have missed a step. I will look further into that at a later point. The transcriptions look good through manual inspection.

The rewritten code:
```python
from datasets import load_dataset, load_metric
from transformers import Speech2TextForConditionalGeneration, Speech2TextProcessor, Wav2Vec2Processor

librispeech_eval = load_dataset("librispeech_asr", "clean", split="test")  # change to "other" for other test dataset
wer = load_metric("wer")

model = Speech2TextForConditionalGeneration.from_pretrained("facebook/s2t-small-librispeech-asr").to("cuda")
processor = Speech2TextProcessor.from_pretrained("facebook/s2t-small-librispeech-asr", do_upper_case=True)

def map_to_pred(batch):
    audio = batch["audio"]
    features = processor(audio["array"], sampling_rate=audio["sampling_rate"], padding=True, return_tensors="pt")
    input_features = features.input_features.to("cuda")
    attention_mask = features.attention_mask.to("cuda")

    gen_tokens = model.generate(input_features=input_features, attention_mask=attention_mask)
    batch["transcription"] = processor.batch_decode(gen_tokens, skip_special_tokens=True)
    return batch

result = librispeech_eval.map(map_to_pred)#, batched=True, batch_size=8)

print("WER:", wer.compute(predictions=result["transcription"], references=result["text"]))
```

I think the issue comes from the fact that you set `batched=False` while `map_to_pred` still returns a list of strings for "transcription". You can fix it by adding `[0]` at the end of this line to get the string:
```python
batch["transcription"] = processor.batch_decode(gen_tokens, skip_special_tokens=True)[0]
```

Updating as many model cards now as I can find

## Describe the bug
When loading the Common_Voice dataset, by downloading it directly from the Hugging Face hub, some files can not be opened.

## Steps to reproduce the bug
```python
import torch
import torchaudio
from datasets import load_dataset, load_metric
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import re

test_dataset = load_dataset("common_voice", "it", split="test")
#test_dataset = load_dataset('csv', data_files = {'test': '/workspace/Dataset/Common_Voice/cv-corpus80/it/test.csv'})
wer = load_metric("wer")

processor = Wav2Vec2Processor.from_pretrained("joorock12/wav2vec2-large-xlsr-italian")
model = Wav2Vec2ForCTC.from_pretrained("joorock12/wav2vec2-large-xlsr-italian")
model.to("cuda")

chars_to_ignore_regex = '[\,\?\.\!\-\;\:\"\“\'\�]'
resampler = torchaudio.transforms.Resample(48_000, 16_000)
```
## Expected results
The common voice dataset downloaded and correctly loaded whit the use of the hugging face datasets library.

## Actual results
The error is:
```python
0ex [00:00, ?ex/s]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-48-ef87f4129e6e> in <module>
      7     return batch
      8
----> 9 test_dataset = test_dataset.map(speech_file_to_array_fn)

/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py in map(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   2107
   2108         if num_proc is None or num_proc == 1:
-> 2109             return self._map_single(
   2110                 function=function,
   2111                 with_indices=with_indices,

/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    516             self: "Dataset" = kwargs.pop("self")
    517         # apply actual function
--> 518         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    519         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    520         for dataset in datasets:

/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    483         }
    484         # apply actual function
--> 485         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    486         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    487         # re-apply format to the output

/opt/conda/lib/python3.8/site-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    411             # Call actual function
    412
--> 413             out = func(self, *args, **kwargs)
    414
    415             # Update fingerprint of in-place transforms + update in-place history of transforms

/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py in _map_single(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, disable_tqdm, desc, cache_only)
   2465                 if not batched:
   2466                     for i, example in enumerate(pbar):
-> 2467                         example = apply_function_on_filtered_inputs(example, i, offset=offset)
   2468                         if update_data:
   2469                             if i == 0:

/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py in apply_function_on_filtered_inputs(inputs, indices, check_same_num_examples, offset)
   2372             if with_rank:
   2373                 additional_args += (rank,)
-> 2374             processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
   2375             if update_data is None:
   2376                 # Check if the function returns updated examples

/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py in decorated(item, *args, **kwargs)
   2067                 )
   2068                 # Use the LazyDict internally, while mapping the function
-> 2069                 result = f(decorated_item, *args, **kwargs)
   2070                 # Return a standard dict
   2071                 return result.data if isinstance(result, LazyDict) else result

<ipython-input-48-ef87f4129e6e> in speech_file_to_array_fn(batch)
      3 def speech_file_to_array_fn(batch):
      4     batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
----> 5     speech_array, sampling_rate = torchaudio.load(batch["path"])
      6     batch["speech"] = resampler(speech_array).squeeze().numpy()
      7     return batch

/opt/conda/lib/python3.8/site-packages/torchaudio/backend/sox_io_backend.py in load(filepath, frame_offset, num_frames, normalize, channels_first, format)
    150                 filepath, frame_offset, num_frames, normalize, channels_first, format)
    151         filepath = os.fspath(filepath)
--> 152     return torch.ops.torchaudio.sox_io_load_audio_file(
    153         filepath, frame_offset, num_frames, normalize, channels_first, format)
    154

RuntimeError: Error loading audio file: failed to open file common_voice_it_17415776.mp3  ```

## Environment info

- `datasets` version: 1.18.4
- Platform: Linux-5.4.0-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyArrow version: 7.0.0


Update README.md for SQuAD v2 metric

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3908). All of your documentation changes will be reflected on that endpoint.

Putting "Values from popular papers" as a subsection of "Output values"


Update README.md for SQuAD metric

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3907). All of your documentation changes will be reflected on that endpoint.


NonMatchingChecksumError on Spider dataset

bug

Something isn't working

Hi @kolk, thanks for reporting.

Indeed, Google Drive service recently changed their service and we had to add a fix to our library to cope with that change:
- #3787

We just made patch release last week: 1.18.4 https://github.com/huggingface/datasets/releases/tag/1.18.4

Please, feel free to update your local `datasets` version, so that you get the fix:
```shell
pip install -U datasets
```

## Describe the bug
Failure to generate dataset ```spider``` because of checksums error for dataset source files.

## Steps to reproduce the bug
```
from datasets import load_dataset
spider = load_dataset("spider")
```

## Expected results
Checksums should match for files from url ['https://drive.google.com/uc?export=download&id=1_AckYkinAnhqmRQtGsQgUKAnTHxxX5J0']

## Actual results
```
>>> load_dataset("spider")
load_dataset("spider")
Downloading and preparing dataset spider/spider (download: 95.12 MiB, generated: 5.17 MiB, post-processed: Unknown size, total: 100.29 MiB) to /home/user/.cache/huggingface/datasets/spider/spider/1.0.0/79778ebea87c59b19411f1eb3eda317e9dd5f7788a556d837ef25c3ae6e5e8b7...
Traceback (most recent call last):
  File "/home/user/py3_env/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-5-d4cb54197348>", line 1, in <module>
    load_dataset("spider")
  File "/home/user/py3_env/lib/python3.8/site-packages/datasets/load.py", line 1702, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/user/py3_env/lib/python3.8/site-packages/datasets/builder.py", line 594, in download_and_prepare
    self._download_and_prepare(
  File "/home/user/py3_env/lib/python3.8/site-packages/datasets/builder.py", line 665, in _download_and_prepare
    verify_checksums(
  File "/home/user/py3_env/lib/python3.8/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1_AckYkinAnhqmRQtGsQgUKAnTHxxX5J0']
```

## Environment info
datasets version: 1.18.3
Platform: Ubuntu 20 LTS
Python version: 3.8.10
PyArrow version: 6.0.1


Perplexity Metric Card

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3905). All of your documentation changes will be reflected on that endpoint.

I'm wondering if we should add that perplexity can be used for analyzing datasets as well

Otherwise, looks good! Good job, @emibaylor !

Add Perplexity metric card

Note that it is currently still missing the citation, but I plan to add it later today.


CONLL2003 Dataset not available

dataset bug

A bug in a dataset script provided in the library

Thanks for reporting, @omarespejel.

I'm sorry but I can't reproduce the issue: the loading of the dataset works perfecto for me and I can reach the data URL: https://data.deepai.org/conll2003.zip

Might it be due to a temporary problem in the data owner site (https://data.deepai.org/) that is fixed now?
Could you please try loading the dataset again and tell if the problem persists?

@omarespejel I'm closing this issue. Feel free to reopen it if the problem persists.

## Describe the bug
[CONLL2003](https://huggingface.co/datasets/conll2003) Dataset can no longer reach 'https://data.deepai.org/conll2003.zip'
![image](https://user-images.githubusercontent.com/4755430/158084483-ff83631c-5154-4823-892d-577bf1166db0.png)

## Steps to reproduce the bug
```python
from datasets import load_dataset
datasets = load_dataset("conll2003")
```

## Expected results
Download the conll2003 dataset.

## Actual results
Error: `ConnectionError: Couldn't reach https://data.deepai.org/conll2003.zip (error 502)`


Add Biwi Kinect Head Pose dataset.

_The documentation is not available anymore as the PR was closed or merged._

Thanks for the detailed explanation of the structure!

1. IMO it makes the most sense to yield one example for each person (so the total of 24 examples), so the features dict should be similar to this:

  ```python
    features = Features({
        "rgb": Sequence(Image()),  # for the png frames
        "rgb_cal": {"intrisic_mat": Array2D(shape=(3, 3), dtype="float32"), "extrinsic_mat": {"rotation": Array2D(shape=(3, 3), dtype="float32"), "translation": Sequence(Value("float32", length=3)}},
        "depth": Sequence(Value("string")), # for the depth frames
        "depth_cal": the same as "rgb_cal",
        "head_pose_gt": Sequence({"center": Sequence(Value("float32", length=3), "rotation": Array2D(shape=(3, 3), dtype="float32")}),
        "head_template": Value("string"),  # for the person's obj file

    })
  ```
   We can add a "Data Processing" section to the card to explain how to parse the files.

2. Yes, it's ok to parse the files as long as it doesn't take too much time/memory (e.g., it's ok to parse the `*_pose.txt` or `*.cal` files, but it's better to leave the `*_depth.bin` or `*.obj` files unprocessed and yield the paths to them)

Thanks for the suggestions @mariosasko, yielding one example for each person would make things much easier.
Okay. I'll look at parsing the files and then displaying the information.

Added the following :
- Features, I have included sequence_number and subject_id along with the features you had suggested.
- Tested loading of the dataset along with dummy_data and full_data tests.
- Created the dataset_infos.json file.

To-Do :
- [x] Update Dataset Cards with more details.
- [x] "Data Processing" section

Any inputs on what to include in the "Data Processing" section ?

@mariosasko  Please could you review this when you get time. Thank you.

In the Data Processing section, I've added example code for a compressed binary depth image file. Updated the Readme as well.

@mariosasko / @lhoestq , Please could you review this when you get time. Thank you.

Created an issue here: https://github.com/huggingface/datasets/issues/4152

Got it. Thanks for the comments. I've collapsed the C++ code in the readme and added the suggestions.

Hi ! The `AttributeError ` bug has been fixed, feel free to merge `master` into your branch ;)

I haven't been able to figure out why CI is failing, the error shown is :

```
E           ValueError: The following issues have been found in the dataset cards:
E           README Parsing:
E           list index out of range
E           The following issues have been found in the dataset cards:
E           README Validation:
E           list index out of range
```

Any inputs would be helpful.

I think it's because there are tabulations in the c++ code, can you replace them with regular spaces please ?

(then in another PR we can maybe fix the Readme parser to support text indented with tabulations)

@lhoestq , initially the idea was to have one example = one image with an additional field mentioning the frame_number. But each subject, we had a head template, calibration information for the depth and the color camera which was common to all the examples for that subject. Also, the images were continuous frames.
@mariosasko  suggested this structure and it made sense to group the images together for a particular subject.

> Don't you think it would be more practical to have one example = one image in this dataset ?

Having one example = one image would be good but since we have a head template, calibration information for the depth and the color camera which is common to all the images for that subject and the images being continuous frames, I think it makes sense to group the images together for each subject. This will make the feature representation easier.

Ok I see, sounds good then. Users can still separate the images if they want to

The CI fails are unrelated to this PR and fixed on master, merging !

Great. Thanks @lhoestq , I think we can close this issue now. ( #3822 )

This PR adds the Biwi Kinect Head Pose dataset.

Dataset Request : Add Biwi Kinect Head Pose Database [#3822](https://github.com/huggingface/datasets/issues/3822)

The Biwi Kinect Head Pose Database is acquired with the Microsoft Kinect sensor, a structured IR light device.It contains 15K images of 20 people with 6 females and 14 males where 4 people were recorded twice.

For each frame, there is :
- a depth image, (.bin file)
- a corresponding rgb image (both 640x480 pixels),
- annotation ( present inside a .txt file)
 The ground truth is the 3D location of the head and its rotation.

The dataset structure is as follows :
```
- 01.obj
- 01
     - frame_00003_depth.bin
     - frame_00003_pose.txt
     - frame_00003_rgb.png
     .
     .
     .
- 02.obj
 - 02
     - frame_00003_depth.bin
     - frame_00003_pose.txt
     - frame_00003_rgb.png
     .
     .
     .
```
Preview of frame_00003_pose.txt :
```
0.988397 0.0731349 0.133128
-0.0441539 0.976945 -0.208876
-0.145334 0.200575 0.968838

126.665 40.4515 876.198
```
I have used the following dataset features :
```
features=datasets.Features(
    {
        "person_id": datasets.Value("string"),
        "frame_number": datasets.Value("string"),
        "depth_image": datasets.Value("string"),
        "rgb_image": datasets.Image(),
        "3D_head_center": datasets.Array2D(shape=(3, 3), dtype="float"),
        "3D_head_rotation": datasets.Value("float"),
    }
```
I am giving the path to the depth_image here.

I need some inputs for the following :

1. For each person, the dataset has the following additional information :
```
For each sequence, the corresponding .obj file represents a head template deformed to match the neutral face of that specific person. [*.obj file]
In each folder, two .cal files contain calibration information for the depth and the color camera, e.g., the intrinsic camera matrix of the depth camera and the global rotation and translation to the rgb camera.
```
Wanted to know how we can represent these features ?

2. For _generate_examples , do I parse the directories and fetch the required information ? This would mean reading the .txt file to obtain the "3D_head_center" and "3D_head_rotation" details. We could precompute the features information  and have a metadata file and use the metadata file to yield information in _generate_examples ? Wanted your thoughts for the best approach for this ?


Can't import datasets: partially initialized module 'fsspec' has no attribute 'utils'

bug

Something isn't working

Update: `"python3 -c "from from datasets import Dataset, DatasetDict"` works, but not if I import without the `python3 -c`

Hi @arunasank, thanks for reporting.

It seems that this can be caused because you are using an old version of `fsspec`: the reason why it works if you run `python3` seems to be that `python3` runs in a Python virtual env (with an updated version of `fsspec`); whereas the error arises when you run the import from other Python virtual env (with an old version of `fsspec`).

In order to fix this, you should update `fsspec` from within the "problematic" Python virtual env:
```
pip install -U "fsspec[http]>=2021.05.0"

I'm closing this issue, @arunasank.

Feel free to re-open it if the problem persists.

## Describe the bug
Unable to import datasets

## Steps to reproduce the bug
```python
from datasets import Dataset, DatasetDict
```

## Expected results
The import works without errors

## Actual results
```
AttributeError                            Traceback (most recent call last)
<ipython-input-37-c8cfcbe62127> in <module>
     11 # from tqdm import tqdm
     12 # import torch
---> 13 from datasets import Dataset
     14 # from transformers import Trainer, TrainingArguments, AutoModel, AutoTokenizer, AutoModelForMaskedLM, DataCollatorForLanguageModeling
     15 # from sentence_transformers import SentenceTransformer

~/.local/lib/python3.8/site-packages/datasets/__init__.py in <module>
     31     )
     32
---> 33 from .arrow_dataset import Dataset, concatenate_datasets
     34 from .arrow_reader import ArrowReader, ReadInstruction
     35 from .arrow_writer import ArrowWriter

~/.local/lib/python3.8/site-packages/datasets/arrow_dataset.py in <module>
     46 )
     47
---> 48 import fsspec
     49 import numpy as np
     50 import pandas as pd

~/.local/lib/python3.8/site-packages/fsspec/__init__.py in <module>
     10 from . import _version, caching
     11 from .callbacks import Callback
---> 12 from .core import get_fs_token_paths, open, open_files, open_local
     13 from .exceptions import FSTimeoutError
     14 from .mapping import FSMap, get_mapper

~/.local/lib/python3.8/site-packages/fsspec/core.py in <module>
     16     caches,
     17 )
---> 18 from .compression import compr
     19 from .registry import filesystem, get_filesystem_class
     20 from .utils import (

~/.local/lib/python3.8/site-packages/fsspec/compression.py in <module>
     68
     69
---> 70 register_compression("zip", unzip, "zip")
     71 register_compression("bz2", BZ2File, "bz2")
     72

~/.local/lib/python3.8/site-packages/fsspec/compression.py in register_compression(name, callback, extensions, force)
     44
     45     for ext in extensions:
---> 46         if ext in fsspec.utils.compressions and not force:
     47             raise ValueError(
     48                 "Duplicate compression file extension: %s (%s)" % (ext, name)

AttributeError: partially initialized module 'fsspec' has no attribute 'utils' (most likely due to a circular import)
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.4
- Platform: Jupyter notebook
- Python version: 3.8.10
- PyArrow version: 7.0.0


Dataset viewer issue for IndicParaphrase- the preview doesn't show

dataset-viewer

Related to the dataset viewer on huggingface.co

It seems to have been fixed:

<img width="1534" alt="Capture d’écran 2022-04-12 à 14 10 07" src="https://user-images.githubusercontent.com/1676121/162959599-6b7fef7c-8411-4e03-8f00-90040a658079.png">

## Dataset viewer issue for '*IndicParaphrase*'

**Link:** *[IndicParaphrase](https://huggingface.co/datasets/ai4bharat/IndicParaphrase/viewer/hi/validation)*

*The preview of the dataset doesn't come up.
The error on the console is:
Status code:   400
Exception:     FileNotFoundError
Message:       [Errno 2] No such file or directory: '/home/hf/datasets-preview-backend/hi_IndicParaphrase_v1.0.tar'*

Am I the one who added this dataset ? Yes


Add MetaShift dataset

_The documentation is not available anymore as the PR was closed or merged._

@lhoestq Please could you review this when you get time. Thank you.

Thanks a lot for your inputs @mariosasko .
> Maybe we can add the generated meta-graphs to the card as images (with attributions)?

Yes. We can do this for the default set of classes. Will add this.

> Would be cool if we could have them as additional configs. Also, maybe we could have configs that expose [image metadata](https://github.com/Weixin-Liang/MetaShift/tree/main/dataset/meta_data) from the https://nlp.stanford.edu/data/gqa/sceneGraphs.zip file (this file is downloaded in the script but not used).

I'll try adding the bonus section as additional config.
Regarding exposing the image metadata with a config parameter, how will we showcase/display this information ?

> Regarding exposing the image metadata with a config parameter, how will we showcase/display this information ?

Oh, I forgot to mention that. Let's add a `Dataset Usage` section to the card to document the params (similar to this: https://huggingface.co/datasets/electricity_load_diagrams#dataset-usage). Also, feel free to add the constants that can be tuned as config params (e.g. `IMAGE_SUBSET_SIZE_THRESHOLD` or the `5` in `len(subject_data) <= 5`).

Okay. Got it. Will add these and constants as config parameters.

The image metadata from scene graphs looks like this :
```json
{
    "2407890": {
        "width": 640,
        "height": 480,
        "location": "living room",
        "weather": none,
        "objects": {
            "271881": {
                "name": "chair",
                "x": 220,
                "y": 310,
                "w": 50,
                "h": 80,
                "attributes": ["brown", "wooden", "small"],
                "relations": {
                    "32452": {
                        "name": "on",
                        "object": "275312"
                    },
                    "32452": {
                        "name": "near",
                        "object": "279472"
                    }
                }
            }
        }
    }
}
```
``load_dataset("metashift", selected_classes=["cat", "dog", ...], image_metadata=True)``
How do we showcase/display the image metadata(json) information ?

> How do we showcase/display the image metadata(json) information ?

We can add the JSON fields as keys to the features dict:
```python
  if self.config.image_metadata:
      features.update({"width": Value("int"), "height": Value("int"), "location": Value("string"), ...})
```

P.S. Would rename `image_metadata`  to `with_image_metadata`

I have added the following :
- Added the meta-graphs to the card as images under the Section "Dataset Meta-Graphs".
- Generate the Attributes-Dataset using config parameter. [ [Link](https://github.com/Weixin-Liang/MetaShift#bonus-generate-the-metashift-attributes-dataset-subsets-defined-by-subject-attributes) ]
- Expose image metadata using config parameter.
Format of the image metadata is as follows : [Link](https://cs.stanford.edu/people/dorarad/gqa/download.html)
I have modified the "Objects" which is dict to a list of dicts with an additional parameter named object_id.
I have defined the structure as follows :
```
{
    "width": datasets.Value("int64"),
    "height": datasets.Value("int64"),
    "location": datasets.Value("string"),
    "weather": datasets.Value("string"),
    "objects": datasets.Sequence(
        {
            "object_id": datasets.Value("string"),
            "name": datasets.Value("string"),
            "x": datasets.Value("int64"),
            "y": datasets.Value("int64"),
            "w": datasets.Value("int64"),
            "h": datasets.Value("int64"),
            "attributes": datasets.Sequence(datasets.Value("string")),
            "relations": datasets.Sequence(
                {
                    "name": datasets.Value("string"),
                    "object": datasets.Value("string"),
                }
            ),
        }
    ),
}
```
Problem is that objects is not being shown as list of dicts. The output looks as follows :

> metashift_dataset['train'][0]

```json
{'image_id': '2338755', 'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x281 at 0x7F066C5A49D0>, 'label': 0, 'context': 'ground', 'width': 500, 'height': 281, 'location': None, 'weather': None, 'objects': {'object_id': ['3070704', '3070705', '3070706', '2416713', '3070702', '2790660', '3063157', '2354960', '2037127', '2392939', '2912743', '2125407', '2735257', '3260906', '2351018', '3288269', '3699852', '2734378', '3421201', '2863115'], 'name': ['bicycle', 'bicycle', 'bicycle', 'boot', 'bicycle', 'motorcycle', 'pepperoni', 'head', 'building', 'wall', 'shorts', 'people', 'wheel', 'bricks', 'man', 'cat', 'boot', 'door', 'ground', 'building'], 'x': [137, 371, 458, 215, 468, 399, 368, 245, 0, 140, 260, 284, 138, 451, 339, 187, 210, 26, 0, 313], 'y': [116, 86, 94, 150, 91, 80, 107, 22, 0, 44, 109, 69, 145, 226, 69, 22, 230, 0, 119, 0], 'w': [197, 27, 15, 73, 24, 53, 9, 37, 289, 46, 43, 30, 74, 28, 35, 116, 53, 107, 500, 55], 'h': [126, 25, 38, 128, 43, 50, 16, 44, 158, 73, 51, 52, 97, 15, 73, 252, 46, 147, 162, 77], 'attributes': [[], [], [], ['white'], [], [], [], [], [], [], [], [], [], [], [], ['white'], ['white'], ['large', 'black'], ['brick'], []], 'relations': [{'name': ['to the left of'], 'object': ['3260906']}, {'name': ['to the left of', 'to the right of', 'to the right of', 'to the left of', 'to the right of', 'to the left of', 'to the right of'], 'object': ['3070706', '2351018', '2125407', '2790660', '2037127', '3070702', '3288269']}, {'name': ['to the right of', 'to the right of', 'to the left of', 'to the right of', 'to the right of'], 'object': ['2351018', '3070705', '3070702', '2790660', '3063157']}, {'name': ['to the right of'], 'object': ['2735257']}, {'name': ['to the right of', 'to the right of', 'to the right of', 'to the right of', 'to the right of'], 'object': ['2351018', '2790660', '3070706', '3070705', '3063157']}, {'name': ['to the right of', 'to the right of', 'to the left of', 'to the left of', 'to the right of', 'to the right of', 'to the right of', 'to the right of'], 'object': ['3070705', '2351018', '3070702', '3070706', '3063157', '2125407', '2037127', '3288269']}, {'name': ['to the right of', 'to the left of', 'to the left of', 'to the right of', 'to the right of', 'to the left of', 'to the right of'], 'object': ['2037127', '3070706', '3070702', '2912743', '3288269', '2790660', '2125407']}, {'name': ['to the left of', 'to the right of'], 'object': ['2863115', '2734378']}, {'name': ['to the left of', 'to the left of', 'to the left of', 'to the left of', 'to the left of', 'to the left of'], 'object': ['3070705', '2351018', '3063157', '2125407', '2790660', '2863115']}, {'name': ['to the left of', 'to the right of', 'to the left of'], 'object': ['2125407', '2734378', '3288269']}, {'name': ['to the left of', 'on', 'to the left of'], 'object': ['2351018', '3288269', '3063157']}, {'name': ['to the left of', 'to the left of', 'to the right of', 'to the left of', 'to the right of', 'to the left of'], 'object': ['3063157', '2351018', '2037127', '3070705', '2392939', '2790660']}, {'name': ['to the left of', 'to the left of'], 'object': ['2416713', '3288269']}, {'name': ['to the right of'], 'object': ['3070704']}, {'name': ['to the right of', 'to the left of', 'to the right of', 'to the left of', 'to the left of', 'to the right of', 'to the left of', 'to the right of', 'walking down'], 'object': ['2037127', '2790660', '2125407', '3070705', '3070706', '2912743', '3070702', '3288269', '3421201']}, {'name': ['to the right of', 'to the right of', 'to the left of', 'to the right of', 'to the left of', 'to the left of', 'to the left of', 'to the left of'], 'object': ['2392939', '2734378', '2790660', '2735257', '3063157', '3070705', '2351018', '2863115']}, {'name': [], 'object': []}, {'name': ['of', 'to the left of', 'to the left of', 'to the left of'], 'object': ['2037127', '2354960', '3288269', '2392939']}, {'name': [], 'object': []}, {'name': ['to the right of', 'to the right of', 'to the right of'], 'object': ['2037127', '3288269', '2354960']}]}}
```
Expected output of image_metadata would be :
```
{'height': 281,
 'location': None,
 'objects': [{'attributes': [],
              'h': 126,
              'name': 'bicycle',
              'object_id': '3070704',
              'relations': [{'name': 'to the left of', 'object': '3260906'}],
              'w': 197,
              'x': 137,
              'y': 116},
             {'attributes': [],
              'h': 25,
              'name': 'bicycle',
              'object_id': '3070705',
              'relations': [{'name': 'to the left of', 'object': '3070706'},
                            {'name': 'to the right of', 'object': '2351018'},
                            {'name': 'to the right of', 'object': '2125407'},
                            {'name': 'to the left of', 'object': '2790660'},
                            {'name': 'to the right of', 'object': '2037127'},
                            {'name': 'to the left of', 'object': '3070702'},
                            {'name': 'to the right of', 'object': '3288269'}],
              'w': 27,
              'x': 371,
              'y': 86},
             {'attributes': ['white'],
              'h': 252,
              'name': 'cat',
              'object_id': '3288269',
              'relations': [{'name': 'to the right of', 'object': '2392939'},
                            {'name': 'to the right of', 'object': '2734378'},
                            {'name': 'to the left of', 'object': '2790660'},
                            {'name': 'to the right of', 'object': '2735257'},
                            {'name': 'to the left of', 'object': '3063157'},
                            {'name': 'to the left of', 'object': '3070705'},
                            {'name': 'to the left of', 'object': '2351018'},
                            {'name': 'to the left of', 'object': '2863115'}],
              'w': 116,
              'x': 187,
              'y': 22},
             {'attributes': ['white'],
              'h': 46,
              'name': 'boot',
              'object_id': '3699852',
              'relations': [],
              'w': 53,
              'x': 210,
              'y': 230},
             .
             .
             .
             {'attributes': ['large', 'black'],
              'h': 147,
              'name': 'door',
              'object_id': '2734378',
              'relations': [{'name': 'of', 'object': '2037127'},
                            {'name': 'to the left of', 'object': '2354960'},
                            {'name': 'to the left of', 'object': '3288269'},
                            {'name': 'to the left of', 'object': '2392939'}],
              'w': 107,
              'x': 26,
              'y': 0},
             {'attributes': ['brick'],
              'h': 162,
              'name': 'ground',
              'object_id': '3421201',
              'relations': [],
              'w': 500,
              'x': 0,
              'y': 119},
             {'attributes': [],
              'h': 77,
              'name': 'building',
              'object_id': '2863115',
              'relations': [{'name': 'to the right of', 'object': '2037127'},
                            {'name': 'to the right of', 'object': '3288269'},
                            {'name': 'to the right of', 'object': '2354960'}],
              'w': 55,
              'x': 313,
              'y': 0}],
 'weather': None,
 'width': 500}

```

May I know how to get the list of dicts representation correctly ?

---
To-Do :

- [x] Generate dataset_infos.json file.
- [x] Add “Dataset Usage” section in the cards and write about the config parameters.
- [x] Add the constants that can be tuned as config params.

> Problem is that objects is not being shown as list of dicts. The output looks as follows :

That's expected. We convert a sequence of dictionaries to a dictionary of sequences to keep the formatting aligned with Tensorflow Datasets. You could disable this behavior by replacing `"objects": datasets.Sequence(object_fields_dict)` with `"objects": [object_fields_dict]`, but that's not what we usually do, so let's keep it like that.

Also, to limit the size of the dataset repo, please remove the pushed images and pass URLs to the images instead under the `src` attribute (and specify `alt` in case the URLs go down).

I'll do a proper review again after you are finished with the dummy data.

> That's expected.

Okay. Got it. Thanks. I thought I was doing something wrong.

> Also, to limit the size of the dataset repo, please remove the pushed images and pass URLs to the images instead under the src attribute (and specify alt in case the URLs go down).

Sure. Where do we host these images ? Can I upload them to any free image hosting platform or is there any particular website you use ?

> I'll do a proper review again after you are finished with the dummy data.

Sure. Thanks. I'm working on this part. Will update you.

Update :
- I have generated the dataset_infos.json file.

> I suggest you try to generate the dataset_infos.json file first, and then I can help with the dummy data.

I am having issues creating the dummy data. I get the following which I use the command :

`datasets-cli dummy_data datasets/metashift`

```
Dataset metashift with config MetashiftConfig(name='metashift', version=1.0.0, data_dir=None, data_files=None, description=None) seems to already open files in the method `_split_generators(...)`. You might consider to instead only open files in the method `_generate_examples(...)` instead. If this is not possible the dummy data has to be created with less guidance. Make sure you create the file dummy_data/full-candidate-subsets.pkl.
Traceback (most recent call last):
  File "datasets-cli", line 33, in <module>
    sys.exit(load_entry_point('datasets', 'console_scripts', 'datasets-cli')())
  File "/datasets/commands/datasets_cli.py", line 33, in main
    service.run()
  File "/datasets/commands/dummy_data.py", line 324, in run
    dataset_builder=dataset_builder, mock_dl_manager=mock_dl_manager
  File "/datasets/commands/dummy_data.py", line 407, in _print_dummy_data_instructions
    for split in generator_splits:
UnboundLocalError: local variable 'generator_splits' referenced before assignment
```

> Feel free to host the images online (on imgur for example) :)

Sure. Will do that.

Thanks for the explanation regarding the dummy data zip files. I will try it out and let you know.

Instead of uploading the images to a hosting service, you can directly reference their GitHub URLs (open the image in the MetaShift repo -> click Download -> copy the image URL). For instance, this is the URL of one of the images:`https://raw.githubusercontent.com/Weixin-Liang/MetaShift/main/docs/figures/Cat-MetaGraph.jpg`. Also, feel free to replace `main` with the most recent commit hash in the copied URLs to make them more robust.

@mariosasko I've actually created metagraphs for all the default classes other than those present in the GitHub Repo and included all of them. :)  The Repo has them only for two classes.

In case we want to limit the no.of meta graphs included, we can stick to the github URLs from the repo itself.

Update :
- I could add the dummy data and get the dummy data test to work. Since we have a preprocessing step on the dataset, one of the .pkl file size is on the higher side. This was done for the tests to pass. I hope that is okay. The dummy.zip file size is about 273K.

To-Do :
- [x] Update Dataset Structure in the data cards to include Data Instances when config parameters are used.

Please could you review when you get time. Thank you.

Thanks a lot for your suggestions, Mario. The thing I learnt from the review is that I need to make better sentence formations. I will keep this in mind. :)

Thanks a lot for your support. @mariosasko  and @lhoestq .

> Super impressed by your work on this, congrats :)

Its my first dataset contribution to the 🤗 Datasets library, I'm super excited. Thank you. :)

Also, I think we can close this request issue now, [#3813](https://github.com/huggingface/datasets/issues/3813)

This PR adds the MetaShift dataset.

Dataset Request  : Add MetaShift dataset [#3813](https://github.com/huggingface/datasets/issues/3813)

@lhoestq As discussed,
- I have copied the preprocessing script and modified it as required to not create new directories and folders and instead yield the images.
-  I do the preprocessing in _split_generators to get the required data which is then passed to _generate_examples.
-  Beyond the generated MetaShift dataset, the original preprocess script also generates the meta-graphs for each class, I have currently not included this part. [ Ref : [Link](https://github.com/Weixin-Liang/MetaShift#generate-full-metashift) ]
-  There is a Bonus section, the authors share.  I have currently not included this part. [ Ref : [Link](https://github.com/Weixin-Liang/MetaShift#bonus-generate-the-metashift-attributes-dataset-subsets-defined-by-subject-attributes) ]
- I had a basic test script which downloaded the dataset and tested the basic functionality. Things seems fine.
For real data, I performed the following test :
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_metashift
============================================== test session starts ===============================================
platform linux -- Python 3.7.11, pytest-7.0.1, pluggy-1.0.0
rootdir: ./datasets
plugins: hydra-core-1.1.1, datadir-1.3.1, forked-1.4.0, xdist-2.5.0
collected 1 item

tests/test_dataset_common.py .                                                                             [100%]

========================================= 1 passed in 4821.25s (1:20:21) =========================================
```
- I couldn't get the dummy dataset. Need some inputs here.
Error as follows :
```
Using custom data configuration default
Dataset metashift with config None seems to already open files in the method `_split_generators(...)`. You might consider to instead only open files in the method `_generate_examples(...)` instead. If this is not possible the dummy data has to be created with less guidance. Make sure you create the file dummy_data/full-candidate-subsets.pkl.
    for split in generator_splits:
UnboundLocalError: local variable 'generator_splits' referenced before assignment
```

To-Do :

- [x] Currently I am using the default _SELECTED_CLASSES. I need to use config option here as suggested
- [x] Complete fields in the Dataset Card.
- [x] Tagging the dataset using the Datasets Tagging app.

Need your help and suggestions for improvement. Thank you


Add exact match metric

Adding the exact match metric and its metric card.

Note: Some of the tests have failed, but I wanted to make a PR anyway so that the rest of the code can be reviewed if anyone has time. I'll look into + work on fixing the failed tests when I'm back online after the weekend


Create README.md for WER metric

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3898). All of your documentation changes will be reflected on that endpoint.

For ASR you can probably ping @patrickvonplaten

Ah only noticed now that ` # Values from popular papers` is from a template. @lhoestq @sashavor - not really sure if this section is useful in general really.

IMO, it's more confusing/misleading than it helps. E.g. a value of 0.03 WER on a fake read-out audio dataset is not better than a WER of 0.3 on a real-world noisy, conversational audio dataset. I think the same holds true for other metrics no? I can think of very little metrics where a metric value is not dataset dependent. E.g. perplexity is super dataset dependent, summarization metrics like ROUGE as well, ...

Also, I don't really see what this section tries to achieve - is the idea here to give the reader some papers that use this metric to better understand in which context it is used? Should we maybe rename the section to `Popular papers making use of this metric` or something?

I put "Values from popular papers" as a subsection of "Output values" -- I hope that's a compromise that works for everyone :hugs:

Proposing a draft WER metric card, @lhoestq I'm not very certain about "Values from popular papers" -- I don't know ASR very well, what do you think of the examples I found?


Align tqdm control/cache control with Transformers

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3897). All of your documentation changes will be reflected on that endpoint.

This PR:
* aligns the `tqdm` logic with Transformers (follows https://github.com/huggingface/transformers/pull/15167) by moving the code to `utils/logging.py`, adding `enable_progres_bar`/`disable_progres_bar` and removing `set_progress_bar_enabled` (a note for @lhoestq: I'm not adding `logging.tqdm` to the public namespace in this PR to avoid the situation where `from datasets import *; tqdm` would overshadow the standard `tqdm`
* aligns the cache control with the new `tqdm` logic by adding `enable_caching`/`disable_caching` to the  public namespace and deprecating `set_caching_enabled` (not fully removing it because it's used more often than `set_progress_bar_enabled` and has a dedicated example in the old docs)

Fix #3586


Missing google file for `multi_news` dataset

`datasets` 1.18.4 fixes the issue when you load the dataset with `load_dataset`.

When loading in streaming mode, the fix is indeed on https://github.com/huggingface/datasets/pull/3843 which will be merged soon :)

That is. The PR #3843 was just opened a bit later we had made our 1.18.4 patch release...
Once merged, that will fix this issue.

OK. Should fix the viewer for 50 datasets

<img width="148" alt="Capture d’écran 2022-03-14 à 11 51 02" src="https://user-images.githubusercontent.com/1676121/158157853-6c544a47-2d6d-4ac4-964a-6f10951ec36b.png">

## Dataset viewer issue for '*multi_news*'

**Link:** https://huggingface.co/datasets/multi_news

```
Server error

Status code:   400
Exception:     FileNotFoundError
Message:       https://drive.google.com/uc?export=download&id=1vRY2wM6rlOZrf9exGTm5pXj5ExlVwJ0C/multi-news-original/train.src
```

Am I the one who added this dataset ? No


Fix code examples indentation

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3895). All of your documentation changes will be reflected on that endpoint.

My last commit should have fixed it, I don't know why the dev doc build is not showing my last changes

Let me merge this and we can see on `master` how it renders, until the dev doc build is fixed

Some code examples are currently not rendered correctly. I think this is because they are over-indented

cc @mariosasko


[docs] make dummy data creation optional

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3894). All of your documentation changes will be reflected on that endpoint.

The dev doc build rendering doesn't seem to be updated with my last commit for some reason

Merging it anyway since I'd like to share this page with users 🙃

Related to #3507 : dummy data for datasets created on the Hugging Face Hub are optional.

We can discuss later to make them optional for datasets in this repository as well


Add default branch for doc building

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3893). All of your documentation changes will be reflected on that endpoint.

Yes! And when we discovered on the Transformers side that this check fails on the GitHub actions, we added a config attribute to have a default. Setting in Transformers fixed the issue of the doc being deployed to main, so porting the fix here too :-)

Since other libraries use `main` as their default branch and it's now the standard default, you have to specify a different name in the doc config if you're using `master` like datasets (`doc-builder` tries to guess it, but in the job, we have weird checkout of merge commits so it doesn't always manage to get it right).

This PR makes sure it will always use master for the dev doc (until you decide to switchto main)


Fix CLI test checksums

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3892). All of your documentation changes will be reflected on that endpoint.

Feel free to merge if it's good for you :)

I've added a test @lhoestq. Once all green, I'll merge.

Last failing tests do not have nothing to do with this PR.

Previous PR:
- #3796

introduced a side effect: `datasets-cli test` generates `dataset_infos.json` with `None` checksum values.

See:
- #3805

This PR introduces a way for `datasets-cli test` to force to record infos, even if `verify_infos=False`

Close #3848.

CC: @craffel


Fix race condition in doc build

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3891). All of your documentation changes will be reflected on that endpoint.

Following https://github.com/huggingface/datasets/runs/5499386744 it seems that race conditions that create issues when updating the doc. I took the same approach as in `transformers` to fix race conditions


Update beans download urls

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3890). All of your documentation changes will be reflected on that endpoint.

@albertvillanova Thanks for investigating and fixing that issue. I regenerated the `dataset_infos.json` file.

Replace the old URLs with the Hub [URLs](https://huggingface.co/datasets/beans/tree/main/data).

Also reported by @stevhliu.

Fix #3889


Cannot load beans dataset (Couldn't reach the dataset)

Hi ! A pull request is open to fix the dataset, we'll release a patch soon with a new release of `datasets` :)

## Describe the bug
The beans dataset is unavailable to download.

## Steps to reproduce the bug
```python
from datasets import load_dataset
ds = load_dataset('beans')
```

## Expected results
The dataset would be downloaded with no issue.

## Actual results

```
ConnectionError: Couldn't reach https://storage.googleapis.com/ibeans/train.zip (error 403)
```

[It looks like the billing of this project has been disabled because it is associated with a delinquent account.](https://storage.googleapis.com/ibeans/train.zip )

## Environment info
Google Colab


IterableDataset columns and feature types

generic discussion

Generic discussion on the library

Right now, an IterableDataset (e.g. when streaming a dataset) doesn't require to know the list of columns it contains, nor their types: `my_iterable_dataset.features` may be `None`

However it's often interesting to know the column types and types. This helps knowing what's inside your dataset without having to manually check a few examples, and this is useful to prepare a processing pipeline or to train models.

Here are a few cases that lead to `features` being `None`:

1. when loading a dataset with `load_dataset` on CSV, JSON Lines, etc. files: type inference is only done when iterating over the dataset
2. when calling `map`, because we don't know in advance what's the output of the user's function passed to `map`
3. when calling `rename_columns`, `remove_columns`, etc. because they rely on `map`

Things we can consider, for each point above:

1.a infer the type automatically from the first samples on the dataset using prefetching, when the dataset builder doesn't provide the `features`
2.a allow the user to specify the `features` as an argument to `map` (this would be consistent with the non-streaming API)
2.b prefetch the first output value to infer the type
3.a don't rely on `map` directly and reuse the previous `features` and rename/remove the corresponding ones

The thing is that prefetching can take a few seconds, while the operations above are instantaneous since no data are downloaded. Therefore I'm not sure whether this solution may be worth it. Maybe prefetching could also be done when explicitly asked by the user

cc @mariosasko @albertvillanova


ImageFolder improvements

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3887). All of your documentation changes will be reflected on that endpoint.

This PR adds the following improvements to the `imagefolder` dataset:
* skip the extract step for image files (as discussed in https://github.com/huggingface/datasets/pull/2830#discussion_r816817919)
* option to drop labels by setting `drop_labels=True` (useful for image pretraining cc @NielsRogge). This is faster than loading a dataset and removing the `label` column because we don't need to iterate over the files to infer class labels.


Retry HfApi call inside push_to_hub when 504 error

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3886). All of your documentation changes will be reflected on that endpoint.

I made it more robust by increasing the wait time, and I also added some logs when a request is retried. Let me know if it's ok for you

At the end you did not set the agreed max value of 60s.

Moreover, with the new numbers, there is a slight contradiction: although you set max_retries=5, we will only make 4 retries at most because of the combined values of `base_wait_time` and `max_wait_time`.

Yea I thought that in total we could wait 1min, but if we have a max_wait_time of 20sec between each request it's fine IMO

> Moreover, with the new numbers, there is a slight contradiction: although you set max_retries=5, we will only make 4 retries at most because of the combined values of base_wait_time and max_wait_time.

What makes you think this ? If the exponential wait time becomes bigger than `max_wait_time` then it still does the retry, but after a wait time of `max_wait_time`

Sorry, I meant 4 retries **with exponential backoff**; the fifth one is with constant backoff.

OK, and one question: do you think that the retries do not affect the time the server needs to be operational again and able to process the request? I guess that if does not affect, then the cause are other users' requests, or others; not our specific request.

Just to be sure:
- Then 20s at most between consecutive requests do not impact the server.
- And we expect after a total of 5 retries (within a total 50s of wait time + request processing/uploading time), the server should be able to come back to normality.

> do you think that the retries do not affect the time the server needs for being able to process the request (I guess in this case the cause are other users' requests, or other causes; not our specific request).

Yes I don't think the retries would affect the server, I think the cause of the 504 errors is elsewhere

> Just to be sure:
>
> Then 20s at most between consecutive requests do not impact the server.
> And we expect after a total of 5 retries (within a total 50s of wait time + request processing/uploading time), the server should be able to come back to normality.

Yes I think it's fine for now, we can still adapt this later if needed

Will be curious to see the impact of this in terms of upload reliability! Don't forget to let us know when you have more data. cc @huggingface/moon-landing-back

Ass suggested by @lhoestq in #3872, this PR:
- Implements a retry function
- Retries HfApi call inside `push_to_hub` when 504 error. To be agreed:
  - max_retries = 2 (at 0.5 and 1 seconds)

Fix #3872.


Fix some shuffle docs

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3885). All of your documentation changes will be reflected on that endpoint.

Following #3842 some docs were still outdated (with `buffer_size` as the first argument)


Fix bug in METEOR metric due to nltk version

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3884). All of your documentation changes will be reflected on that endpoint.

Fix #3883.


The metric Meteor doesn't work for nltk ==3.6.4

bug

Something isn't working

Hi @zhaowei-wang98, thanks for reporting.

We are fixing it...

## Describe the bug
Using the metric Meteor with nltk == 3.6.4 gives a TypeError:
TypeError: descriptor 'lower' for 'str' objects doesn't apply to a 'list' object

## Steps to reproduce the bug
```python
import datasets
metric = datasets.load_metric("meteor")
predictions = ["hello world"]
references = ["hello world"]
metric.compute(predictions=predictions, references=references)
```

## Expected results
TypeError: descriptor 'lower' for 'str' objects doesn't apply to a 'list' object

I think this TypeError exists because input sentences are tokenized into lists of tokens and the str.lower() is applied to this list of tokens.

## Actual results
No error but a meteor score

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: linux
- Python version: 3.8.12
- PyArrow version: 7.0.0


Image process doc

documentation

Improvements or additions to documentation

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3882). All of your documentation changes will be reflected on that endpoint.

This PR is a first draft of how to process image data. It adds:

- Load an image dataset with `image` and `path` (adds tip about `decode=False` param to access the path and bytes, thanks to @mariosasko).
- Load an image using the `ImageFolder` builder. I know there is an [example](https://huggingface.co/docs/datasets/master/en/loading#image-folders) of this already, but I also wanted to add it here so users don't miss it. This doc seems important for centralizing all of the image-related things so far. Datasets has grown so quickly 🚀  now that I think maybe splitting up the How-to guides by modality may be better since working with vision/audio data is slightly different from what users have seen up until now. This way we can continue to scale the docs to better accommodate vision/audio things.
- Add a data augmentation with `set_transform`. There is only 1 example here so far, but we can certainly add more.

Todo:
- [x] Couldn't figure out why my augmentation function works with `set_transform` but not `map` 🥲. Working with @mariosasko on this!


How to use Image folder

question

Further information is requested

Even this from docs throw same error
```
dataset = load_dataset("imagefolder", data_files="https://download.microsoft.com/download/3/E/1/3E1C3F21-ECDB-4869-8368-6DEBA77B919F/kagglecatsanddogs_3367a.zip", split="train")

```

Hi @INF800,

Please note that the `imagefolder` feature enhancement was just recently merged to our master branch (https://github.com/huggingface/datasets/commit/207be676bffe9d164740a41a883af6125edef135), but has not yet been released.

We are planning to make the 2.0 release of our library in the coming days and then that feature will be available by updating your `datasets` library from PyPI.

In the meantime, you can incorporate that feature if you install our library from our GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```

Then:
```python
In [1]: from datasets import load_dataset
        ds = load_dataset("imagefolder", data_files="https://download.microsoft.com/download/3/E/1/3E1C3F21-ECDB-4869-8368-6DEBA77B919F/kagglecatsanddogs_3367a.zip", split="train")
Using custom data configuration default-7eb4e80d960deb18
Downloading and preparing dataset image_folder/default to .../.cache/huggingface/datasets/image_folder/default-7eb4e80d960deb18/0.0.0/8de8dc6d68ce3c81cc102b93cc82ede27162b5d30cd003094f935942c8294f60...
Downloading data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 690.19it/s]
Extracting data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 852.85it/s]
Dataset image_folder downloaded and prepared to .../.cache/huggingface/datasets/image_folder/default-7eb4e80d960deb18/0.0.0/8de8dc6d68ce3c81cc102b93cc82ede27162b5d30cd003094f935942c8294f60. Subsequent calls will reuse this data.

In [2]: ds
Out[2]:
Dataset({
    features: ['image', 'label'],
    num_rows: 25000
})
```

Hey @albertvillanova. Does this load entire dataset in memory? Because I am facing huge trouble with loading very big datasets (OOM errors)

Can you provide the error stack trace? The loader only stores the `data_files` dict, which can get big after globbing. Then, the OOM error would mean you don't have enough memory to keep all the paths to the image files. You can circumvent this by generating an archive and loading the dataset from there. Maybe we can optimize the globbing part in our data files resolution at some point, cc @lhoestq for visibility.

Hey, memory error is resolved. It was fluke.

But there is another issue. Currently `load_dataset("imagefolder", data_dir="./path/to/train",)` takes only `train` as arg to `split` parameter.

I am creating vaildation dataset using

```
ds_valid = datasets.DatasetDict(valid=load_dataset("imagefolder", data_dir="./path/to/valid",)['train'])
```

`data_dir="path/to/folder"` is a shorthand syntax fox `data_files={"train": "path/to/folder/**"}`, so use `data_files` in that case instead:
```python
ds = load_dataset("imagefolder", data_files={"train": "path/to/train/**", "test": "path/to/test/**", "valid": "path/to/valid/**"})
```

And there was another issue. I loaded black and white images (jpeg file). Using load dataset. It reads it as PIL jpeg data format. But instead of converting it into 3 channel tensor, input to collator function is coming as a single channel tensor.

We don't apply any additional preprocessing on top of `PIL.Image.open(image_file)`, so you need to do the conversion yourself:

```python
def to_rgb(batch):
    batch["image"] = [img.convert("RGB") for img in batch["image"]]
    return batch

ds_rgb = ds.map(to_rgb, batched=True)
```

Please use our Forum for questions of this kind in the future.

Ran this code
```
 load_dataset("imagefolder", data_dir="./my-dataset")
```

`https://raw.githubusercontent.com/huggingface/datasets/master/datasets/imagefolder/imagefolder.py` missing
```
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
/tmp/ipykernel_33/1648737256.py in <module>
----> 1 load_dataset("imagefolder", data_dir="./my-dataset")

/opt/conda/lib/python3.7/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, script_version, **config_kwargs)
   1684         revision=revision,
   1685         use_auth_token=use_auth_token,
-> 1686         **config_kwargs,
   1687     )
   1688

/opt/conda/lib/python3.7/site-packages/datasets/load.py in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, use_auth_token, script_version, **config_kwargs)
   1511         download_config.use_auth_token = use_auth_token
   1512     dataset_module = dataset_module_factory(
-> 1513         path, revision=revision, download_config=download_config, download_mode=download_mode, data_files=data_files
   1514     )
   1515

/opt/conda/lib/python3.7/site-packages/datasets/load.py in dataset_module_factory(path, revision, download_config, download_mode, force_local_path, dynamic_modules_path, data_files, **download_kwargs)
   1200                         f"Couldn't find a dataset script at {relative_to_absolute_path(combined_path)} or any data file in the same directory. "
   1201                         f"Couldn't find '{path}' on the Hugging Face Hub either: {type(e1).__name__}: {e1}"
-> 1202                     ) from None
   1203                 raise e1 from None
   1204     else:

FileNotFoundError: Couldn't find a dataset script at /kaggle/working/imagefolder/imagefolder.py or any data file in the same directory. Couldn't find 'imagefolder' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/imagefolder/imagefolder.py
```


Change the framework switches to the new syntax

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3880). All of your documentation changes will be reflected on that endpoint.

This PR updates the syntax of the framework-specific code samples. With this new syntax, you'll be able to:

- have paragraphs of text be framework-specific instead of just code samples
- have support for Flax code samples if you want.

This should be merged after https://github.com/huggingface/doc-builder/pull/63 and https://github.com/huggingface/doc-builder/pull/130


SQuAD v2 metric: create README.md

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3879). All of your documentation changes will be reflected on that endpoint.

Proposing SQuAD v2 metric card


Update cats_vs_dogs size

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3878). All of your documentation changes will be reflected on that endpoint.

Maybe `NonMatchingSplitsSizesError` errors should also tell the user to try using a more recent version of the dataset to get the fixes ?

@lhoestq Good idea. Will open a new PR to improve the error messages of NonMatchingSplitsSizesError, NonMatchingChecksumsError, ...

It seems like 12 new examples have been added to the `cats_vs_dogs`. This PR updates the size in the card and the info file to avoid a verification error (reported by @stevhliu).


Align metadata to DCAT/DCAT-AP

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
Align to DCAT metadata to describe datasets

**Describe the solution you'd like**
Reuse terms and structure from DCAT in the metadata file, ideally generate a json-ld file dcat compliant

**Describe alternatives you've considered**

**Additional context**
DCAT is a W3C standard extended in Europe with DCAT-AP, an example is data.europa.eu publishing datasets metadata in DCAT-AP


Fix download_mode in dataset_module_factory

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3876). All of your documentation changes will be reflected on that endpoint.

Fix `download_mode` value set in `dataset_module_factory`.

Before the fix, it was set to `bool` (default to `False`).

Also set properly its default value in all public functions.


Module namespace cleanup for v2.0

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3875). All of your documentation changes will be reflected on that endpoint.

@severo No, this PR doesn't fix that issue in the current state. We can fix it by adding `__all__` to `datasets/__init__.py` and `datasets/formatting/__init__.py`. However, this would require updating `__all__` for each new function/class definition, which could become cumbersome, and we can't do this dynamically because `mypy` is a static type checker.

@lhoestq @albertvillanova WDYT?

Feel free to merge this one if it's good for you :)

This is an attempt to make the user-facing `datasets`' submodule namespace cleaner:

In particular, this PR does the following:
* removes the unused `zip_nested` and `flatten_nest_dict` and their accompanying tests
* removes `pyarrow` from the top-level namespace
* properly uses `__all__` and the `from  <module> import *` syntax to avoid importing the `<module>`'s submodules
* cleans up the `utils` namespace
* moves the `temp_seed` context manage from `datasets/utils/file_utils.py` to `datasets/utils/py_utils.py`


add MSE and MAE metrics - V2

@mariosasko New PR here. I'm not sure how to add you as a co-author here.  Also I see flake8 tests are failing, any inputs on how to resolve this ?
Also, let me know if any other changes are required. Thank you.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3874). All of your documentation changes will be reflected on that endpoint.

Created a new pull request to resolve unrelated changes in PR caused due to rebasing.
Ref Older PR :  [#3845](https://github.com/huggingface/datasets/pull/3845)

Feature request here : Add support for continuous metrics (RMSE, MAE) [#3608](https://github.com/huggingface/datasets/issues/3608)


Create SQuAD metric README.md

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3873). All of your documentation changes will be reflected on that endpoint.

Oh one last thing I almost forgot, I think I would add a section "Examples" with examples of inputs and outputs and in particular: an example giving maximal values, an examples giving minimal values and maybe a standard examples from SQuAD. What do you think?

Proposal for a metrics card structure (with an example based on the SQuAD metric).
@thomwolf @lhoestq @douwekiela @lewtun -- feel free to comment on structure or content (it's an initial draft, so I realize there's stuff missing!).


HTTP error 504 Server Error: Gateway Time-out

is pushing directly with git (and git-lfs) an option for you?

I have installed git-lfs and doing this push with that

yes but is there any way you could try pushing with `git` command line directly instead of `push_to_hub`?

Okay. I didnt saved the dataset to my local machine. So, I processed the dataset and pushed it directly to the hub. I think I should try saving those dataset to my local machine by `save_to_disk` and then push it with git command line

cc @lhoestq @albertvillanova @LysandreJik because maybe I'm giving dumb advice here 😅

`push_to_hub` is the preferred way of uploading a dataset to the Hub, which can then be reloaded with `load_dataset`. Feel free to try again and see if the server is working as expected now. Maybe we can add a retry mechanism in the meantime to workaround 504 errors.

Regarding `save_to_disk`, this must only be used for local serialization (because it's uncompressed and compatible with memory-mapping). If you upload a dataset saved with `save_to_disk` to the Hub, then to reload it you will have to download/clone the repository locally by yourself and use `load_from_disk`.

I am trying to push a large dataset(450000+) records with the help of `push_to_hub()`
While pushing, it gives some error like this.

```
Traceback (most recent call last):
  File "data_split_speech.py", line 159, in <module>
    data_new_2.push_to_hub("user-name/dataset-name",private=True)
  File "/opt/conda/lib/python3.8/site-packages/datasets/dataset_dict.py", line 951, in push_to_hub
    repo_id, split, uploaded_size, dataset_nbytes = self[split]._push_parquet_shards_to_hub(
  File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 3556, in _push_parquet_shards_to_hub
    api.upload_file(
  File "/opt/conda/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 1017, in upload_file
    raise err
  File "/opt/conda/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 1008, in upload_file
    r.raise_for_status()
  File "/opt/conda/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: https://huggingface.co/api/datasets/user-name/dataset-name/upload/main/data/train2-00041-of-00064.parquet

```
Can anyone help me to resolve this issue.


add pandas to env command

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3871). All of your documentation changes will be reflected on that endpoint.

Think failures are unrelated - feel free to merge whenever you want :-)

Pandas is a required packages and used quite a bit. I don't see any downside with adding its version to the `datasets-cli env` command.


Add wikitablequestions dataset

@lhoestq Would you mind reviewing it when you're available? Thanks!

> Awesome thanks for adding this dataset ! :) The dataset script and dataset cards look pretty good
>
> It looks like your `dummy_data.zip` files are quite big though (>1MB each), do you think we can reduce their sizes ? This way this git repository doesn't become too big

I have manually reduced the `dummy_data.zip` and its current size is about 54KB. Hope it is fine for you!

@lhoestq I think the dataset is ready to merge now. Any follow-up question is welcome :-D

> Thanks ! It looks all good now :)

Awesome! Thanks for your quick response!


Making the Hub the place for datasets in Portuguese

dataset request

Requesting to add a new dataset

Hi @omarespejel! I think the philosophy for `datasets` issues is to create concrete issues with proposals to add a specific, individual dataset rather than umbrella issues for things such as datasets for a language, since we could end up with hundreds of issues (one per language). I see NILC - USP has many datasets, I would suggest to either create an issue for their datasets, or even better, we are trying to push to upload datasets as community datasets instead of adding them to the core library as guided in https://huggingface.co/docs/datasets/share. That would have the additional benefit that the dataset would live under the NILC organization.

@lhoestq correct me if I'm wrong please 😄

Let's make Hugging Face Datasets the central hub for datasets in Portuguese :)

**Motivation**. Datasets are currently quite scattered and an open-source central point such as the Hugging Face Hub would be ideal to support the growth of the Portuguese speaking community.

What are some datasets in Portuguese worth integrating into the Hugging Face hub?

Special thanks to @augusnunes for his collaboration on identifying the first ones:

-  [NILC - USP](http://www.nilc.icmc.usp.br/nilc/index.php/tools-and-resources).

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

cc @osanseviero


Ignore duplicate keys if `ignore_verifications=True`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3868). All of your documentation changes will be reflected on that endpoint.

Cool thanks ! Could you add a test please ?

Currently, it's impossible to generate a dataset if some keys from `_generate_examples` are duplicated. This PR allows skipping the check for duplicate keys if `ignore_verifications` is set to `True`.


Update for the rename doc-builder -> hf-doc-utils

~~@julien-c there was a vote 🙂 https://huggingface.slack.com/archives/C021H1P1HKR/p1646405136644739~~

oh I see you already commeented in the thread as well

Thanks ! It looks all good to me (provided `hf-doc-utils` is the name we keep in the end). I'm fine with this name, and `hf-doc-builder` is also fine IMHO

ok, this is definitely not a hill I'll die on =) @mishig25 @sgugger

This PR adapts the job to the upcoming change of name of `doc-builder`.


Bring back imgs so that forsk dont get broken

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3866). All of your documentation changes will be reflected on that endpoint.

I think we just need to keep `datasets_logo_name.jpg` and `course_banner.png` because they appear in the README.md of the forks of `datasets`. The other images can be removed


Add logo img

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3865). All of your documentation changes will be reflected on that endpoint.


Update image dataset tags

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3864). All of your documentation changes will be reflected on that endpoint.

Align the existing image datasets' tags with new tags introduced in #3800.


Update code blocks

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3863). All of your documentation changes will be reflected on that endpoint.

Following https://github.com/huggingface/datasets/pull/3860#issuecomment-1061756712 and https://github.com/huggingface/datasets/pull/3690 we need to update the code blocks to use markdown instead of sphinx


Manipulate columns on IterableDataset (rename columns, cast, etc.)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3862). All of your documentation changes will be reflected on that endpoint.

> IIUC we check if columns are present/not present directly in the yielded examples and not in info.features because info.features can be None (after map, for instance)?

Yes exactly

> We should develop a solution that ensures info.features is never None. For example, one approach would be to infer them from examples in map and make them promotable from Value("null") to a specific type, in case of None values.

I agree this would be useful. Though inferring the type requires to start streaming some data, which takes a few seconds (compared to being instantaneous right now).

Let's discuss this in a new issue maybe ?

I added:
- add_column
- cast
- rename_column
- rename_columns

related to https://github.com/huggingface/datasets/issues/3444

TODO:
- [x] docs
- [x] tests


big_patent cased version

Requesting to add a new dataset

Hi! I am interested in working with the big_patent dataset.

In Tensorflow, there are a number of versions of the dataset:

- 1.0.0 : lower cased tokenized words
- 2.0.0 : Update to use cased raw strings
- 2.1.2 (default): Fix update to cased raw strings.

The version in the huggingface `datasets` library is the 1.0.0. I would be very interested in using the 2.1.2 cased version (used more, recently, for example in the Pegasus paper), but it does not seem to be supported (I tried using the `revision` parameter in `load_datasets`). Is there a way to already load it, or would it be possible to add that version?


Small doc fixes

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3860). All of your documentation changes will be reflected on that endpoint.

There are still some `.. code-block:: python` (e.g. see [this](https://huggingface.co/docs/datasets/master/en/package_reference/main_classes#datasets.Dataset.align_labels_with_mapping)) directives in our codebase, so maybe we can remove those as well as part of this PR.


Unable to dowload big_patent (FileNotFoundError)

bug

Something isn't working

duplicate

This issue or pull request already exists

Hi @slvcsl, thanks for reporting.

Yesterday we just made a patch release of our `datasets` library that fixes this issue: version 1.18.4.
https://pypi.org/project/datasets/#history

Please, feel free to update `datasets` library to the latest version:
```shell
pip install -U datasets
```
And then you should force redownload of the data file to update your local cache:
```python
ds = load_dataset("big_patent", "g", split="validation", download_mode="force_redownload")
```
- Note that before the fix, you just downloaded and cached the Google Drive virus scan warning page, instead of the data file

This issue was already reported
- #3784

and its root cause is a change in the Google Drive service. See:
- #3786

We already fixed it. See:
- #3787

## Describe the bug
I am trying to download some splits of the big_patent dataset, using the following code:

`ds = load_dataset("big_patent", "g", split="validation", download_mode="force_redownload")
`
However, this leads to a FileNotFoundError.

FileNotFoundError                         Traceback (most recent call last)
[<ipython-input-3-8d8a745706a9>](https://localhost:8080/#) in <module>()
      1 from datasets import load_dataset
----> 2 ds = load_dataset("big_patent", "g", split="validation", download_mode="force_redownload")

8 frames
[/usr/local/lib/python3.7/dist-packages/datasets/load.py](https://localhost:8080/#) in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, script_version, **config_kwargs)
   1705         ignore_verifications=ignore_verifications,
   1706         try_from_hf_gcs=try_from_hf_gcs,
-> 1707         use_auth_token=use_auth_token,
   1708     )
   1709

[/usr/local/lib/python3.7/dist-packages/datasets/builder.py](https://localhost:8080/#) in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    593                     if not downloaded_from_gcs:
    594                         self._download_and_prepare(
--> 595                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    596                         )
    597                     # Sync info

[/usr/local/lib/python3.7/dist-packages/datasets/builder.py](https://localhost:8080/#) in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    659         split_dict = SplitDict(dataset_name=self.name)
    660         split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 661         split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    662
    663         # Checksums verification

[/root/.cache/huggingface/modules/datasets_modules/datasets/big_patent/bdefa7c0b39fba8bba1c6331b70b738e30d63c8ad4567f983ce315a5fef6131c/big_patent.py](https://localhost:8080/#) in _split_generators(self, dl_manager)
    123         split_types = ["train", "val", "test"]
    124         extract_paths = dl_manager.extract(
--> 125             {k: os.path.join(dl_path, "bigPatentData", k + ".tar.gz") for k in split_types}
    126         )
    127         extract_paths = {k: os.path.join(extract_paths[k], k) for k in split_types}

[/usr/local/lib/python3.7/dist-packages/datasets/utils/download_manager.py](https://localhost:8080/#) in extract(self, path_or_paths, num_proc)
    282         download_config.extract_compressed_file = True
    283         extracted_paths = map_nested(
--> 284             partial(cached_path, download_config=download_config), path_or_paths, num_proc=num_proc, disable_tqdm=False
    285         )
    286         path_or_paths = NestedDataStructure(path_or_paths)

[/usr/local/lib/python3.7/dist-packages/datasets/utils/py_utils.py](https://localhost:8080/#) in map_nested(function, data_struct, dict_only, map_list, map_tuple, map_numpy, num_proc, types, disable_tqdm)
    260         mapped = [
    261             _single_map_nested((function, obj, types, None, True))
--> 262             for obj in utils.tqdm(iterable, disable=disable_tqdm)
    263         ]
    264     else:

[/usr/local/lib/python3.7/dist-packages/datasets/utils/py_utils.py](https://localhost:8080/#) in <listcomp>(.0)
    260         mapped = [
    261             _single_map_nested((function, obj, types, None, True))
--> 262             for obj in utils.tqdm(iterable, disable=disable_tqdm)
    263         ]
    264     else:

[/usr/local/lib/python3.7/dist-packages/datasets/utils/py_utils.py](https://localhost:8080/#) in _single_map_nested(args)
    194     # Singleton first to spare some computation
    195     if not isinstance(data_struct, dict) and not isinstance(data_struct, types):
--> 196         return function(data_struct)
    197
    198     # Reduce logging to keep things readable in multiprocessing with tqdm

[/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py](https://localhost:8080/#) in cached_path(url_or_filename, download_config, **download_kwargs)
    314     elif is_local_path(url_or_filename):
    315         # File, but it doesn't exist.
--> 316         raise FileNotFoundError(f"Local file {url_or_filename} doesn't exist")
    317     else:
    318         # Something unknown

FileNotFoundError: Local file /root/.cache/huggingface/datasets/downloads/extracted/ad068abb3e11f9f2f5440b62e37eb2b03ee515df9de1637c55cd1793b68668b2/bigPatentData/train.tar.gz doesn't exist

I have tried this in a number of machines, including on Colab, so I think this is not environment dependent.

How do I load the bigPatent dataset?


Udpate index.mdx margins

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3858). All of your documentation changes will be reflected on that endpoint.


Order of dataset changes due to glob.glob.

generic discussion

Generic discussion on the library

I agree using `glob.glob` alone is bad practice because it's not deterministic. Using `sorted` is a nice solution.

Note that the `xglob` function you are referring to in the `streaming_download_manager.py` code just extends `glob.glob` for URLs - we don't change its behavior. That's why it has no `sorted()`

## Describe the bug

After discussion with @lhoestq, just want to mention here that `glob.glob(...)` should always be used in combination with `sorted(...)` to make sure the list of files returned by `glob.glob(...)` doesn't change depending on the OS system.

There are currently multiple datasets that use `glob.glob()` without making use of `sorted(...)` even the streaming download manager (if I'm not mistaken):
https://github.com/huggingface/datasets/blob/c14bfeb4af89da14f870de5ddaa584b08aa08eeb/src/datasets/utils/streaming_download_manager.py#L483


Fix push_to_hub with null images

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3856). All of your documentation changes will be reflected on that endpoint.

This code currently raises an error because of the null image:
```python
import datasets

dataset_dict = { 'name': ['image001.jpg', 'image002.jpg'], 'image': ['cat.jpg', None] }
features = datasets.Features({
    'name': datasets.Value('string'),
    'image': datasets.Image(),
})
dataset = datasets.Dataset.from_dict(dataset_dict, features)
dataset.push_to_hub("username/dataset") # this line produces an error: 'NoneType' object is not subscriptable
```
I fixed this in this PR

TODO:
- [x] add a test


Bad error message when loading private dataset

bug

Something isn't working

We raise the error “ FileNotFoundError: can’t find the dataset” mainly to follow best practice in security (otherwise users could be able to guess what private repositories users/orgs may have)

We can indeed reformulate this and add the "If this is a private repository,..." part !

## Describe the bug

A pretty common behavior of an interaction between the Hub and datasets is the following.
An organization adds a dataset in private mode and wants to load it afterward.

```python
from transformers import load_dataset

ds = load_dataset("NewT5/dummy_data", "dummy")
```

This command then fails with:

```bash
FileNotFoundError: Couldn't find a dataset script at /home/patrick/NewT5/dummy_data/dummy_data.py or any data file in the same directory. Couldn't find 'NewT5/dummy_data' on the Hugging Face Hub either: FileNotFoundError: Dataset 'NewT5/dummy_data' doesn't exist on the Hub
```

**even though** the user has access to the website `NewT5/dummy_data` since she/he is part of the org.

We need to improve the error message here similar to how @sgugger, @LysandreJik and @julien-c have done it for transformers IMO.

## Steps to reproduce the bug

E.g. execute the following code to see the different error messages between `transformes` and `datasets`.

1. Transformers
```python
from transformers import BertModel

BertModel.from_pretrained("NewT5/dummy_model")
```

The error message is clearer here - it gives:

```
OSError: patrickvonplaten/gpt2-xl is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with `use_auth_token` or log in with `huggingface-cli login` and pass `use_auth_token=True`.
```

Let's maybe do the same for datasets? The PR was introduced to `transformers` here:
https://github.com/huggingface/transformers/pull/15261

## Expected results

Better error message

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.4.dev0
- Platform: Linux-5.15.15-76051515-generic-x86_64-with-glibc2.34
- Python version: 3.9.7
- PyArrow version: 6.0.1