Can't get the text data when calling to_tf_dataset

You are correct that `to_tf_dataset` only handles numerical columns right now, yes, though this is a limitation we might remove in future! The main reason we do this is that our models mostly do not include the tokenizer as a model layer, because it's very difficult to compile some of them in TF. So the "normal" Huggingface workflow is to first tokenize your dataset, and then pass tokenized tensors to the model.

For your use case, would you prefer to pass strings to the model, and use some text processing layers instead of the built-in tokenizers?

Also tagging @gante just so he's aware, but I can handle this one!

Thanks for the quick follow-up to my issue.

For my use-case, instead of the built-in tokenizers I wanted to use the `TextVectorization` layer to map from strings to integers. To achieve this, I came up with the following solution:

```
from datasets import load_dataset
from transformers import DefaultDataCollator
import tensorflow as tf
import string
import re
from tensorflow.keras.layers.experimental.preprocessing import TextVectorization

#some hyper-parameters for the text-to-integer mapping
max_features = 20000
embedding_dim = 128
sequence_length = 210

data_collator = DefaultDataCollator(return_tensors="tf")
dataset = load_dataset("sst", "default")

#adapt the vectorization layer on train data only
vectorize_layer.adapt(dataset["train"].to_dict(batched=False)["sentence"])

def prepare_features(text, label):
    text = tf.expand_dims(text, -1)
    return {"vectorized_text": vectorize_layer(text)[0], "label": tf.expand_dims(label, axis=-1)}

encoded_dataset = dataset.map(lambda example: prepare_features(example["sentence"], example["label"]), batched=False)

def custom_standardization(input_data):
    lowercase = tf.strings.lower(input_data)
    return tf.strings.regex_replace(
        lowercase, f"[{re.escape(string.punctuation)}]", ""
    )

vectorize_layer = TextVectorization(
    standardize=custom_standardization,
    max_tokens=max_features,
    output_mode="int",
    output_sequence_length=sequence_length,
)

train_dataset = encoded_dataset["train"].to_tf_dataset(columns=['vectorized_text'], label_cols=["label"],
                                                       shuffle=True, batch_size=1, collate_fn=data_collator).unbatch()
#similar for the other sub-sets

```

Since the strings would have been mapped to integers or floats at some point, it's no drawback that this mapping is done early in the process.

For the future, however, it'd be more convenient to get the string data, since I am also inspecting the dataset (longest sentence, shortest sentence), which is more challenging when working with integer or float. For now, this can be done by calling `to_dict`.

> For the future, however, it'd be more convenient to get the string data, since I am also inspecting the dataset (longest sentence, shortest sentence), which is more challenging when working with integer or float.

Yes, I agree, so let's keep this issue open.

Going to close this now - methods like `to_tf_dataset` and `prepare_tf_dataset` now support string data, and have done for a while! If anyone sees this and is encountering issues with string data in those methods, please file a new issue!

I am working with the SST2 dataset, and am using TensorFlow 2.5
I'd like to convert it to a `tf.data.Dataset` by calling the `to_tf_dataset` method.
The following snippet is what I am using to achieve this:

```
from datasets import load_dataset
from transformers import DefaultDataCollator

data_collator = DefaultDataCollator(return_tensors="tf")
dataset = load_dataset("sst")

train_dataset = dataset["train"].to_tf_dataset(columns=['sentence'], label_cols="label", shuffle=True, batch_size=8,collate_fn=data_collator)
```
However, this only gets me the labels; the text--the most important part--is missing:

```
for s in train_dataset.take(1):
  print(s) #prints something like: ({}, <tf.Tensor: shape=(8,), ...>)
```

As you can see, it only returns the label part, not the data, as indicated by the empty dictionary, `{}`. So far, I've played with various settings of the method arguments, but to no avail; I do not want to perform any text processing at this time. On my quest to achieve what I want ( a `tf.data.Dataset`), I've consulted these resources:

[https://www.philschmid.de/huggingface-transformers-keras-tf](https://www.philschmid.de/huggingface-transformers-keras-tf)
[https://huggingface.co/docs/datasets/use_dataset.html?highlight=tensorflow](https://huggingface.co/docs/datasets/use_dataset.html?highlight=tensorflow)

I was surprised to not find more extensive examples on how to transform a Hugginface dataset to one compatible with TensorFlow.

If you could point me to where I am going wrong, please do so.
Thanks in advance for your support.

---
Edit: In the [docs](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.to_tf_dataset), I found the following description:

_In general, only columns that the model can use as input should be included here (numeric data only)._

Does this imply that no textual, i.e., `string` data can be loaded?


`Translation` features cannot be `flatten`ed

bug

Something isn't working

Thanks for reporting, @SBrandeis! Some additional feature types that don't behave as expected when flattened: `Audio`, `Image` and `TranslationVariableLanguages`

## Describe the bug

(`Dataset.flatten`)[https://github.com/huggingface/datasets/blob/master/src/datasets/arrow_dataset.py#L1265] fails for columns with feature (`Translation`)[https://github.com/huggingface/datasets/blob/3edbeb0ec6519b79f1119adc251a1a6b379a2c12/src/datasets/features/translation.py#L8]

## Steps to reproduce the bug

```python
from datasets import load_dataset

dataset = load_dataset("europa_ecdc_tm", "en2fr", split="train[:10]")
print(dataset.features)
# {'translation': Translation(languages=['en', 'fr'], id=None)}
print(dataset[0])
# {'translation': {'en': 'Vaccination against hepatitis C is not yet available.', 'fr': 'Aucune vaccination contre l’hépatite C n’est encore disponible.'}}

dataset.flatten()
```

## Expected results

`dataset.flatten` should flatten the `Translation` column as if it were a dict of `Value("string")`

```python
dataset[0]
# {'translation.en':  'Vaccination against hepatitis C is not yet available.', 'translation.fr': 'Aucune vaccination contre l’hépatite C n’est encore disponible.' }
dataset.features
# {'translation.en': Value("string"), 'translation.fr': Value("string")}
```

## Actual results

```python
In [31]: dset.flatten()
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-31-bb88eb5276ee> in <module>
----> 1 dset.flatten()

[...]\site-packages\datasets\fingerprint.py in wrapper(*args, **kwargs)
    411             # Call actual function
    412
--> 413             out = func(self, *args, **kwargs)
    414
    415             # Update fingerprint of in-place transforms + update in-place history of transforms

[...]\site-packages\datasets\arrow_dataset.py in flatten(self, new_fingerprint, max_depth)
   1294                 break
   1295         dataset.info.features = self.features.flatten(max_depth=max_depth)
-> 1296         dataset._data = update_metadata_with_features(dataset._data, dataset.features)
   1297         logger.info(f'Flattened dataset from depth {depth} to depth {1 if depth + 1 < max_depth else "unknown"}.')
   1298         dataset._fingerprint = new_fingerprint

[...]\site-packages\datasets\arrow_dataset.py in update_metadata_with_features(table, features)
    534 def update_metadata_with_features(table: Table, features: Features):
    535     """To be used in dataset transforms that modify the features of the dataset, in order to update the features stored in the metadata of its schema."""
--> 536     features = Features({col_name: features[col_name] for col_name in table.column_names})
    537     if table.schema.metadata is None or b"huggingface" not in table.schema.metadata:
    538         pa_metadata = ArrowWriter._build_metadata(DatasetInfo(features=features))

[...]\site-packages\datasets\arrow_dataset.py in <dictcomp>(.0)
    534 def update_metadata_with_features(table: Table, features: Features):
    535     """To be used in dataset transforms that modify the features of the dataset, in order to update the features stored in the metadata of its schema."""
--> 536     features = Features({col_name: features[col_name] for col_name in table.column_names})
    537     if table.schema.metadata is None or b"huggingface" not in table.schema.metadata:
    538         pa_metadata = ArrowWriter._build_metadata(DatasetInfo(features=features))

KeyError: 'translation.en'
```

## Environment info

- `datasets` version: 1.18.3
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.10
- PyArrow version: 3.0.0


Add support for `Audio` and `Image` feature in `push_to_hub`

> Cool thanks !
>
> Also cc @patrickvonplaten @anton-l it means that when calling push_to_hub, the audio bytes are embedded in the parquet files (we don't upload the audio files themselves)

Just to verify quickly the size of the dataset doesn't change in this case no? E.g. if a dataset has say 20GB in size when stored in `.mp3` format it could have up to 100GB when stored in WAV. But since we are just taking the bytes here a 20GB .mp3 dataset would also have 20GB when stored in parquet no?

@lhoestq  I've addressed your comments. Additionally, I've modified `cast_storage` to account for possible null (`None`) values.

@patrickvonplaten Yes, the dataset size stays the same (at least because Parquet files are compressed).

Feel free to merge if it's all good to you :)

Add support for the `Audio` and the `Image` feature in `push_to_hub`.

The idea is to remove local path information and store file content under "bytes" in the Arrow table before the push.

My initial approach (https://github.com/huggingface/datasets/commit/34c652afeff9686b6b8bf4e703c84d2205d670aa) was to use a map transform similar to [`decode_nested_example`](https://github.com/huggingface/datasets/blob/5e0f6068741464f833ff1802e24ecc2064aaea9f/src/datasets/features/features.py#L1023-L1056) while having decoding turned off, but I wasn't satisfied with the code quality, so I ended up using the `temporary_assignment` decorator to override `cast_storage`, which allows me to directly modify the underlying storage (the final op is similar to `Dataset.cast`) and results in a much simpler code.

Additionally, I added the `allow_cast` flag that can disable this behavior in the situations where it's not needed (e.g. the dataset is already in the correct format for the Hub, etc.)

EDIT:
`allow_cast` renamed to `embed_external_files`


[fix]: iwslt2017 download urls

dataset contribution

Contribution to a dataset script

Hi ! Thanks for the fix ! Do you know where this new URL comes from ?

Also we try to not use Google Drive if possible, since it has download quota limitations. Do you know if the data is available from another host than Google Drive ?

Oh, I found it just by following the link from the [IWSLT2017 homepage](https://wit3.fbk.eu/2017-01). Not sure if it's available from another host.

Ok cool ! I guess it's ok to use this URL for now, and we can see later if we need to change it.

Before merging, could you update the `dataset_infos.json` file by running this command please ?
```
datasets-cli test ./datasets/iwslt2017 --save_infos --all_configs
```

sure thing. lmk if there's anything else i can do to help.

just checking in. is there anything i can do to help on my end to get this merged? (the dummy data tests are failing due an incorrect path, i think)

Thanks ! I also fixed the dummy data :)

To fix the CI, feel free to merge the `master` branch into your PR.

If you have some time, feel free to also take a look at the missing YAML tags at the top of the README.md file of this dataset:
```
E           ValueError: The following issues have been found in the dataset cards:
E           YAML tags:
E           missing 9 required tags: 'annotations_creators', 'language_creators', 'languages', 'licenses', 'multilinguality', 'size_categories', 'source_datasets', 'task_categories', and 'task_ids'
```
you can use the dataset tagging app here: https://huggingface.co/spaces/huggingface/datasets-tagging

I guess this PR was superseded by this other:
- #4481

Thanks for your contribution anyway, @msarmi9.

Fixes #2076.


added told-br (brazilian hate speech) dataset

Amazing thank you ! Feel free to regenerate the `dataset_infos.json` to account for the feature type change, and then I think we'll be good to merge :)

Hey,

Adding ToLD-Br. Feel free to ask for modifications.

Thanks!!


adding told-br for toxic/abusive hatespeech detection

Sorry for using multiple github accounts, I didn't notice I was using my professional account to commit/push. Please consider this @JAugusto97 account as the correct one.

Will remake the PR with the correct github account.

Hey,

I'm adding our dataset from our paper published at AACL 2020. Feel free to ask for modifications.

Thanks!


Fix TestCommand to move dataset_infos instead of copying

All the datasets that are loaded normally with `load_dataset`, if `dataset_infos.json` exists, have this file in the importable directory. So it's fine if we copy the file instead of moving it but it's not a big deal.

Any reason to prefer moving it rather than copying it ?

@lvwerra reported than when generating the `dataset_infos.json` for multiple dataset directories containing only JSONL files, subsequent `dataset_infos.json` files contained all previous directories as configs:
- First generate metadata for dataset in dir `dir1`: dataset_infos.json contains one config for `dir1`
- Then generate metadata for dataset in dir `dir2`: dataset_infos.json contains 2 configs, for `dir1` and `dir2`

The reason is that all dataset_infos.json files are first created in the same dir (the one containing the json builder) and then **copied** to the user dir.

Subsequent calls of TestCommand don't replace the dataset_infos.json already present in the dir of the json builder, but append to it.

MAYBE: we should just move for this use case, and copy for the other use cases? See this use case here:
- #3680

@lhoestq aren't you mentioning the case in the else clause?
```python
else:  # in case of a remote dataset
    dataset_dir = None
```

In that case `dataset_infos.json` is not copied: `dataset_dir = None`

When using the JSON loader, calling `get_imported_module_dir()` returns a path inside the pip installed packages, so we shouldn't write files in it anyway, and the dataset_infos.json file should be written directly in the user's directory instead (some users don't have write access to the pip installed packages for example).

Maybe the packaged modules like `json` should override `_save_infos` to save them in the user's directory instead of next to the builder's script. What do you think ?

Anyway as a hotfix we can just add an exception for the `json` builder for now, if the issue has to be fixed soon

Why do we copy instead of moving the file?

CC: @lhoestq @lvwerra


Fix TestCommand to copy dataset_infos to local dir with only data files

Currently this case is missed.

CC: @lvwerra


Download datasets from a private hub

enhancement

New feature or request

Hi ! For information one can set the environment variable `HF_ENDPOINT` (default is `https://huggingface.co`) if they want to use a private hub.

We may need to coordinate with the other libraries to have a consistent way of changing the hub endpoint

Yes, I tested it successfully this morning. Thanks.

In the context of a private hub deployment, customers would like to use load_dataset() to load datasets from their hub, not from the public hub. This doesn't seem to be configurable at the moment and it would be nice to add this feature.

The obvious workaround is to clone the repo first and then load it from local storage, but this adds an extra step. It'd be great to have the same experience regardless of where the hub is hosted.

The same issue exists with the transformers library and the CLI. I'm going to create issues there as well, and I'll reference them below.


Add code example in wikipedia card

Close #3292.


Discovery cannot be streamed anymore

bug

Something isn't working

Seems like a regression from https://github.com/huggingface/datasets/pull/2843

Or maybe it's an issue with the hosting. I don't think so, though, because https://www.dropbox.com/s/aox84z90nyyuikz/discovery.zip seems to work as expected

Hi @severo, thanks for reporting.

Some servers do not support HTTP range requests, and those are required to stream some file formats (like ZIP in this case).

Let me try to propose a workaround.

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
from datasets import load_dataset
iterable_dataset = load_dataset("discovery", name="discovery", split="train", streaming=True)
list(iterable_dataset.take(1))
```

## Expected results

The first row of the train split.

## Actual results

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 365, in __iter__
    for key, example in self._iter():
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 362, in _iter
    yield from ex_iterable
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 272, in __iter__
    yield from islice(self.ex_iterable, self.n)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 79, in __iter__
    yield from self.generate_examples_fn(**self.kwargs)
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/discovery/542fab7a9ddc1d9726160355f7baa06a1ccc44c40bc8e12c09e9bc743aca43a2/discovery.py", line 333, in _generate_examples
    with open(data_file, encoding="utf8") as f:
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/streaming.py", line 64, in wrapper
    return function(*args, use_auth_token=use_auth_token, **kwargs)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/utils/streaming_download_manager.py", line 369, in xopen
    file_obj = fsspec.open(file, mode=mode, *args, **kwargs).open()
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/core.py", line 456, in open
    return open_files(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/core.py", line 288, in open_files
    fs, fs_token, paths = get_fs_token_paths(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/core.py", line 611, in get_fs_token_paths
    fs = filesystem(protocol, **inkwargs)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/registry.py", line 253, in filesystem
    return cls(**storage_options)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/spec.py", line 68, in __call__
    obj = super().__call__(*args, **kwargs)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/implementations/zip.py", line 57, in __init__
    self.zip = zipfile.ZipFile(self.fo)
  File "/home/slesage/.pyenv/versions/3.9.6/lib/python3.9/zipfile.py", line 1257, in __init__
    self._RealGetContents()
  File "/home/slesage/.pyenv/versions/3.9.6/lib/python3.9/zipfile.py", line 1320, in _RealGetContents
    endrec = _EndRecData(fp)
  File "/home/slesage/.pyenv/versions/3.9.6/lib/python3.9/zipfile.py", line 263, in _EndRecData
    fpin.seek(0, 2)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/implementations/http.py", line 676, in seek
    raise ValueError("Cannot seek streaming HTTP file")
ValueError: Cannot seek streaming HTTP file
```

## Environment info

- `datasets` version: 1.18.3
- Platform: Linux-5.11.0-1027-aws-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 6.0.1


`None` replaced by `[]` after first batch in map

It looks like this is because of this behavior in pyarrow:
```python
import pyarrow as pa

arr = pa.array([None, [0]])
reconstructed_arr = pa.ListArray.from_arrays(arr.offsets, arr.values)
print(reconstructed_arr.to_pylist())
# [[], [0]]
```

It seems that `arr.offsets` can reconstruct the array properly, but an offsets array with null values can:
```python
fixed_offsets = pa.array([None, 0, 1])
fixed_arr = pa.ListArray.from_arrays(fixed_offsets, arr.values)
print(fixed_arr.to_pylist())
# [None, [0]]

print(arr.offsets.to_pylist())
# [0, 0, 1]
print(fixed_offsets.to_pylist())
# [None, 0, 1]
```
EDIT: this is because `arr.offsets` is not enough to reconstruct the array, we also need the validity bitmap

The offsets don't have nulls because they don't include the validity bitmap from `arr.buffers()[0]`, which is used to say which values are null and which values are non-null.

Though the validity bitmap also seems to be wrong:
```python
bin(int(arr.buffers()[0].hex(), 16))
# '0b10'
# it should be 0b110 - 1 corresponds to non-null and 0 corresponds to null, if you take the bits in reverse order
```

So apparently I can't even create the fixed offsets array using this.

If I understand correctly it's always missing the 1 on the left, so I can add it manually as a hack to fix the issue until this is fixed in pyarrow EDIT: actually it may be more complicated than that

EDIT2: actuall it's right, it corresponds to the validity bitmap of the array of logical length 2. So if we use the offsets array, the values array, and this validity bitmap it should be possible to reconstruct the array properly

I created an issue on Apache Arrow's JIRA: https://issues.apache.org/jira/browse/ARROW-15837

And another one: https://issues.apache.org/jira/browse/ARROW-15839

FYI the behavior is the same with:
- `datasets` version: 1.18.3
- Platform: Linux-5.8.0-50-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.11
- PyArrow version: 6.0.1

but not with:
- `datasets` version: 1.8.0
- Platform: Linux-4.18.0-305.40.2.el8_4.x86_64-x86_64-with-redhat-8.4-Ootpa
- Python version: 3.7.11
- PyArrow version: 3.0.0

i.e. it outputs:
```py
0  [None, [0]]
1  [None, [0]]
2  [None, [0]]
3  [None, [0]]
```

Thanks for the insights @PaulLerner !

I found a way to workaround this issue for the code example presented in this issue.

Note that empty lists will still appear when you explicitly `cast` a list of lists that contain None values like [None, [0]] to a new feature type (e.g. to change the integer precision). In this case it will show a warning that it happened. If you don't cast anything, then the None values will be kept as expected.

Let me know what you think !

Hi! I feel like I’m missing something in your answer, *what* is the workaround? Is it fixed in some `datasets` version?

`pa.ListArray.from_arrays` returns empty lists instead of None values. The workaround I added inside `datasets` simply consists in not using `pa.ListArray.from_arrays` :)

Once this PR [here ](https://github.com/huggingface/datasets/pull/4282)is merged, we'll release a new version of `datasets` that currectly returns the None values in the case described in this issue

EDIT: released :) but let's keep this issue open because it might happen again if users change the integer precision for example

Sometimes `None` can be replaced by `[]` when running map:

```python
from datasets import Dataset

ds = Dataset.from_dict({"a": range(4)})
ds = ds.map(lambda x: {"b": [[None, [0]]]}, batched=True, batch_size=1, remove_columns=["a"])
print(ds.to_pandas())
#              b
# 0  [None, [0]]
# 1    [[], [0]]
# 2    [[], [0]]
# 3    [[], [0]]
```

This issue has been experienced when running the `run_qa.py` example from `transformers` (see issue https://github.com/huggingface/transformers/issues/15401)

This can be due to a bug in when casting `None` in nested lists. Casting only happens after the first batch, since the first batch is used to infer the feature types.

cc @sgugger


Add CodeContests dataset

dataset request

Requesting to add a new dataset

This dataset is now available here: https://huggingface.co/datasets/deepmind/code_contests.

## Adding a Dataset
- **Name:** CodeContests
- **Description:** CodeContests is a competitive programming dataset for machine-learning.
- **Paper:**
- **Data:** https://github.com/deepmind/code_contests
- **Motivation:** This dataset was used when training [AlphaCode](https://deepmind.com/blog/article/Competitive-programming-with-AlphaCode).

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Add FrugalScore metric

@lhoestq

The model used by default (`moussaKam/frugalscore_tiny_bert-base_bert-score`) is a tiny model.

I still want to make one modification before merging.
I would like to load the model checkpoint once. Do you think it's a good idea if I load it in `_download_and_prepare`? In this case should the model name be the `self.config_name` or another variable say `self.model_name` ?

OK, I added a commit that loads the checkpoint in `_download_and_prepare`. Please let me know if it looks good.

I triggered the CI and it's failing, can you merge the `master` branch into yours ? It should fix the issues.

Also the doctest apparently raises an error because it outputs `{'scores': [0.6307542, 0.6449357]}` instead of `{'scores': [0.631, 0.645]}` - feel free to edit the code example in the docstring to round the scores, that should fix it

This pull request add FrugalScore metric for NLG systems evaluation.

FrugalScore is a reference-based metric for NLG models evaluation. It is based on a distillation approach that allows to learn a fixed, low cost version of any expensive NLG metric, while retaining most of its original performance.

Paper: https://arxiv.org/abs/2110.08559?context=cs
Github: https://github.com/moussaKam/FrugalScore

@lhoestq


`load_dataset("snli")` is different from dataset viewer

bug

Something isn't working

dataset-viewer

Related to the dataset viewer on huggingface.co

Yes, we decided to replace the encoded label with the corresponding label when possible in the dataset viewer. But
1. maybe it's the wrong default
2. we could find a way to show both (with a switch, or showing both ie. `0 (neutral)`).

Hi @severo,

Thanks for clarifying.

I think this default is a bit counterintuitive for the user. However, this is a personal opinion that might not be general. I think it is nice to have the actual (non-encoded) labels in the viewer. On the other hand, it would be nice to match what the user sees with what they get when they download a dataset. I don't know - I can see the difficulty of choosing a default :)
Maybe having non-encoded labels as a default can be useful?

Anyway, I think the issue has been addressed. Thanks a lot for your super-quick answer!

Thanks for the 👍  in https://github.com/huggingface/datasets/issues/3673#issuecomment-1029008349 @mariosasko @gary149 @pietrolesci, but as I proposed various solutions, it's not clear to me which you prefer. Could you write your preferences as a comment?

_(note for myself: one idea per comment in the future)_

As I am working with seq2seq, I prefer having the label in string form rather than numeric. So the viewer is fine and the underlying dataset should be "decoded" (from int to str). In this way, the user does not have to search for a mapping `int -> original name` (even though is trivial to find, I reckon). Also, encoding labels is rather easy.

I hope this is useful

I like the idea of "0 (neutral)". The label name can even be greyed to make it clear that it's not part of the actual item in the dataset, it's just the meaning.

I like @lhoestq's idea of having grayed-out labels.

Proposals by @gary149. Which one do you prefer? Please vote with the thumbs

- 👍

    ![image](https://user-images.githubusercontent.com/1676121/152387949-883c7d7e-a9f3-48aa-bff9-11a691555e6e.png)

- 👎

    ![image (1)](https://user-images.githubusercontent.com/1676121/152388061-32d95e42-cade-4ae4-9a77-7365e7b72b8f.png)

I like Option 1 better as it shows clearly what the user is downloading

It's [live](https://huggingface.co/datasets/glue/viewer/cola/train):

<img width="1126" alt="Capture d’écran 2022-02-14 à 10 26 03" src="https://user-images.githubusercontent.com/1676121/153836716-25f6205b-96af-42d8-880a-7c09cb24c420.png">

Thanks all for the help to improve the UI!

## Describe the bug
The dataset that is downloaded from the Hub via `load_dataset("snli")` is different from what is available in the dataset viewer. In the viewer the labels are not encoded (i.e., "neutral", "entailment", "contradiction"), while the downloaded dataset shows the encoded labels (i.e., 0, 1, 2).

Is this expected?

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform: Ubuntu 20.4
- Python version: 3.7


Prioritize `module.builder_kwargs` over defaults in `TestCommand`

This fixes a bug in the `TestCommand` where multiple kwargs for `name` were passed if it was set in both default and `module.builder_kwargs`. Example error:

```Python
Traceback (most recent call last):
  File "create_metadata.py", line 96, in <module>
    main(**vars(args))
  File "create_metadata.py", line 86, in main
    metadata_command.run()
  File "/opt/conda/lib/python3.7/site-packages/datasets/commands/test.py", line 144, in run
    for j, builder in enumerate(get_builders()):
  File "/opt/conda/lib/python3.7/site-packages/datasets/commands/test.py", line 141, in get_builders
    name=name, cache_dir=self._cache_dir, data_dir=self._data_dir, **module.builder_kwargs
TypeError: type object got multiple values for keyword argument 'name'
```

Let me know what you think.


Give an estimate of the dataset size in DatasetInfo

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**

Currently, only part of the datasets provide `dataset_size`, `download_size`, `size_in_bytes` (and `num_bytes` and `num_examples` inside `splits`). I would want to get this information, or an estimation, for all the datasets.

**Describe the solution you'd like**

- get access to the git information for the dataset files hosted on the hub
- look at the [`Content-Length`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Length) for the files served by HTTP


feat: 🎸 generate info if dataset_infos.json does not exist

It's a first attempt at solving https://github.com/huggingface/datasets/issues/3013.

I only kept these ones:
```
    path: str,
    data_files: Optional[Union[Dict, List, str]] = None,
    download_config: Optional[DownloadConfig] = None,
    download_mode: Optional[GenerateMode] = None,
    revision: Optional[Union[str, Version]] = None,
    use_auth_token: Optional[Union[bool, str]] = None,
    **config_kwargs,
```

Let me know if it's better for you now !

(note that there's no breaking change since the ones that are removed can be passed as config_kwargs if you really want)

(https://github.com/huggingface/datasets/pull/3670/commits/5636911880ea4306c27c7f5825fa3f9427ccc2b6 and https://github.com/huggingface/datasets/pull/3670/commits/07c3f0800dd34dfebb9674ad46c67a907b08ded8 -> I has forgotten to update black in my venv)

in get_dataset_infos(). Also: add the `use_auth_token` parameter, and create get_dataset_config_info()

✅ Closes: #3013


Common voice validated partition

Hi @patrickvonplaten - could you please advise whether this would be a welcomed change, and if so, who I consult regarding the unit-tests?

I'd be happy with adding this change. @anton-l @lhoestq - what do you think?

Cool ! I just fixed the tests by adding a dummy `validated.tsv` file in the dummy data archive of common_voice

I wonder if you should separate the train/valid/test configuration from the validated/invalidated configuration of the splits ?
In particular having `validated` along with the train/valid/test splits could be a bit weird since it comprises them. We can do that if you think it makes more sense. Otherwise it's also good as it is right now :)

Thanks! I think that there are 2 cases for using the validated partition: 1) trainset = {validated - dev - test}, dev and test as they come; 2) train, dev, and test sampled from validated manually with the desired ratios.
In either case, I think that it's quite a big change on the HF interface part, so could as well be taken care of in the client code. Or is it not? (In which case, what's the most compact way to implement this?)

What's important IMO is to let the users as much flexibility as they need - so we try to not do too much regarding splits to not constrain users. So I guess the way it is right now is ok. Can you confirm that it's ok @patrickvonplaten and that it won't break some speech training script out there ?

@lhoestq all split names are explicit in our example scripts, so this shouldn't break anything, feel free to merge :)
I'll go ahead and add this to the official `mozilla-foundation` datasets as well

Good for me! This has no real down-sides IMO and surely won't break any training scripts.

This patch adds access to the 'validated' partitions of CommonVoice datasets (provided by the dataset creators but not available in the HuggingFace interface yet).
As 'validated' contains significantly more data than 'train' (although it contains both test and validation, so one needs to be careful there), it can be useful to train better models where no strict comparison with the previous work is intended.


Couldn't cast array of type string error with cast_column

bug

Something isn't working

Hi ! I wasn't able to reproduce the error, are you still experiencing this ? I tried calling `cast_column` on a string column containing paths.

If you manage to share a reproducible code example that would be perfect

Hi,

I think my team mate got this solved. Clolsing it for now and will reopen if I experience this again.
Thanks :)

Hi @R4ZZ3,

If it is not too much of a bother, can you please help me how to resolve this error? I am exactly getting the same error where I am going as per the documentation guideline:

`my_audio_dataset = my_audio_dataset.cast_column("audio_paths", Audio())`

where `"audio_paths"` is a dataset column (feature) having strings of absolute paths to mp3 files of the dataset.

I was having the same issue with this code:

```
dataset = dataset.map(
    lambda batch: {"full_path" : os.path.join(self.data_path, batch["path"])},
    num_procs = 4
)
my_audio_dataset = dataset.cast_column("full_path", Audio(sampling_rate=16_000))
```

Removing the "num_procs" argument fixed it somehow.
Using a mac with m1 chip

Hi @Hubert-Bonisseur, I think this will be fixed by https://github.com/huggingface/datasets/pull/4614

## Describe the bug

In OVH cloud during Huggingface Robust-speech-recognition event on a AI training notebook instance using jupyter lab and running jupyter notebook When using the dataset.cast_column("audio",Audio(sampling_rate=16_000))
method I get error
![image](https://user-images.githubusercontent.com/25264037/152214027-9c42a71a-dd24-463c-a346-57e0287e5a8f.png)

This was working with datasets version 1.17.1.dev0
but now with version 1.18.3 produces the error above.

## Steps to reproduce the bug

load dataset:
![image](https://user-images.githubusercontent.com/25264037/152216145-159553b6-cddc-4f0b-8607-7e76b600e22a.png)

remove columns:
![image](https://user-images.githubusercontent.com/25264037/152214707-7c7e89d1-87d8-4b4f-8cfc-5d7223d35644.png)

run my fix_path function.
This also creates the audio column that is referring to the absolute file path of the audio

![image](https://user-images.githubusercontent.com/25264037/152214773-51f71ccf-d31b-4449-b63a-1af56436e49f.png)

Then I concatenate few other datasets and finally try the cast_column method
![image](https://user-images.githubusercontent.com/25264037/152215032-f341ec86-9d6d-48c9-943b-e2efe37a4d98.png)

but get error:
![image](https://user-images.githubusercontent.com/25264037/152215073-b85bd057-98e8-413c-9b05-51e9805f2c24.png)

## Expected results
A clear and concise description of the expected results.

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform:
OVH Cloud, AI Training section, container for Huggingface Robust Speech Recognition event image(baaastijn/ovh_huggingface)
![image](https://user-images.githubusercontent.com/25264037/152215161-b4ff7bfb-2736-4afb-9223-761a3338d23c.png)

- Python version: 3.8.8
- PyArrow version:
![image](https://user-images.githubusercontent.com/25264037/152215936-4d365760-557e-456b-b5eb-ad1d15cf5073.png)


Process .opus files with torchaudio

> Note that torchaudio is maybe less practical to use for TF or JAX users. This is not in the scope of this PR, but in the future if we manage to find a way to let the user control the decoding it would be nice

@lhoestq so maybe don't do this PR? :) if it doesn't work anyway with an opened file, only with path

Yes as discussed offline there seems to be issues with torchaudio on opened files. Feel free to close this PR if it's better to stick with soundfile because of that

We should be able to remove torchaudio, which has torch as a hard dependency, soon and use only soundfile for decoding: https://github.com/bastibe/python-soundfile/issues/252#issuecomment-1000246773 (opus + mp3 support is on the way).

@anton-l suggested to proccess .opus files with `torchaudio` instead of `soundfile` as it's faster:
![opus](https://user-images.githubusercontent.com/16348744/152177816-2df6076c-f28b-4aef-a08d-b499b921414d.png)

(moreover, I didn't manage to load .opus files with `soundfile` / `librosa` locally on any my machine anyway for some reason, even with `ffmpeg` installed).

For now my current changes work with locally stored file:
```python
# download sample opus file (from MultilingualSpokenWords dataset)
!wget https://huggingface.co/datasets/polinaeterna/test_opus/resolve/main/common_voice_tt_17737010.opus

from datasets import Dataset, Audio

audio_path = "common_voice_tt_17737010.opus"
dataset = Dataset.from_dict({"audio": [audio_path]}).cast_column("audio", Audio(48000))
dataset[0]
# {'audio': {'path': 'common_voice_tt_17737010.opus',
#   'array': array([ 0.0000000e+00,  0.0000000e+00,  3.0517578e-05, ...,
#          -6.1035156e-05,  6.1035156e-05,  0.0000000e+00], dtype=float32),
#   'sampling_rate': 48000}}
```
But it doesn't work when loading inside s dataset from bytes (I checked on [MultilingualSpokenWords](https://github.com/huggingface/datasets/pull/3666), the PR is a draft now, maybe the bug is somewhere there  )

```python
import torchaudio
with open(audio_path, "rb") as b:
    print(torchaudio.load(b))
# RuntimeError: Error loading audio file: failed to open file <in memory buffer>
```


process .opus files (for Multilingual Spoken Words)

@lhoestq  I still have problems with processing `.opus` files with `soundfile` so I actually cannot fully check that it works but it should... Maybe this should be investigated in case of someone else would also have problems with that.

Also, as the data is in a private repo on the hub (before we come to a decision about audio data privacy), the needed checks cannot be done right now.

@lhoestq I check the data redownloading for configs sharing the same languages, you were right: the data is downloaded once for each language. But samples are generated from scratch each time. Is it a supposed behavior?

> But samples are generated from scratch each time. Is it a supposed behavior?

Yea that's the way it works right now, because we generate one arrow file per configuration. Since changing the languages creates a new configuration, then it generates a new arrow file.

Opus files requires `libsndfile>=1.0.30`. Add check for this version and tests.

**outdated:**
Add [Multillingual Spoken Words dataset](https://mlcommons.org/en/multilingual-spoken-words/)

You can specify multiple languages for downloading 😌:
```python
ds = load_dataset("datasets/ml_spoken_words", languages=["ar", "tt"])
```

1. I didn't take into account that each time you pass a set of languages the data for a specific language is downloaded even if it was downloaded before (since these are custom configs like `ar+tt` and `ar+tt+br`. Maybe that wasn't a good idea?
2. The script will have to be slightly changed after merge of https://github.com/huggingface/datasets/pull/3664
2. Just can't figure out what wrong with dummy files... 😞 Maybe we should get rid of them at some point 😁


Fix MP3 resampling when a dataset's audio files have different sampling rates

The resampler needs to be updated if the `orig_freq` doesn't match the audio file sampling rate

Fix https://github.com/huggingface/datasets/issues/3662


[WIP] Return local paths to Common Voice

Cool thanks for giving it a try @anton-l !

Would be very much in favor of having "real" paths to the audio files again for non-streaming use cases. At the same time it would be nice to make the audio data loading script as understandable as possible so that the community can easily add audio datasets in the future by looking at this one as an example. Think if it's clear for a contributor how to add an audio datasets script that works for the standard non-streaming case while it is easy to extend it afterwards to a streaming dataset script, then this would be perfect

> @anton-l @patrickvonplaten @lhoestq Is it possible somehow to provide this logic inside the library instead of a loading script so that we don't need to completely rewrite all the scripts for audio datasets and users don't have to care about two different loading approaches in the same script? thinking

Not sure @lhoestq - what do you think?

Now that we've corrected the previous resampling bug, think this one here is of high importance. @lhoestq - what do you think how we should proceed here?

> @anton-l @patrickvonplaten @lhoestq Is it possible somehow to provide this logic inside the library instead of a loading script so that we don't need to completely rewrite all the scripts for audio datasets and users don't have to care about two different loading approaches in the same script? 🤔

Yes let's do this :)

Maybe we can change the behavior of `DownloadManager.iter_archive` back to extracting the TAR archive locally, and return an iterable of (local path, file obj). And the `StreamingDownloadManager.iter_archive` can return an iterable of (relative path inside the archive, file obj) ?

In this case, a dataset would need to have something like this:
```python
for path, f in files:
    yield id_, {"audio": {"path": path, "bytes": f.read() if not is_local_file(path) else None}}
```

Alternatively, we can allow this if we consider that `Audio.encode_example` sets the "bytes" field to `None` automatically if `path` is a local path:
```python
for path, f in files:
    yield id_, {"audio": {"path": path, "bytes": f.read()}}
```
Note that in this case the file is read for nothing though (maybe it's not a big deal ?)

Let me know if it sounds good to you and what you'd prefer !

@lhoestq I'm very much in favor of your first aproach! With the full paths returned I think we won't even need to mess with `os.path.join` vs `"/".join()"` and other local/streaming differences 👍

@lhoestq I also like the idea and favor your first approach to avoid an unnecessary read and make yielding faster.

Looks cool - thanks for working on this. I just feel strongly about `path` being an absolute `path` that exist and can be inspected in the non-streaming case :-) For streaming=True IMO it's absolutely fine if we only have access to the bytes

Hi ! I started implementing this but I noticed that returning an absolute path is breaking for many datasets that do things like
```python
for path, f in files:
    if path.startswith(data_dir):
        ...
```
so I think I will have to add a parameter to `iter_archive` like `extract_locally=True` to avoid the breaking change, does that sound good to you ?

This makes me also think that in streaming mode it could also return a local path too, if we think that writing and deleting temporary files on-the-fly while iterating over the streaming dataset is ok.

@lhoestq I think it is a good idea to rollback to extracting the archives locally in non-streaming mode, as far as (as you mentioned) we do not store the bytes in the Arrow file for those cases to avoid "doubling" the disk space usage.

On the other hand, I don't like:
- neither the possibility to avoid extracting locally in non-streaming: the behavior should be consistent; thus we always extract in non-streaming
  - which could be the criterium to decide whether an archive should or should not be extracted? Just because I want to make a condition on path.startswith?
- nor the option to download/delete temporary files in streaming (see discussion here: https://github.com/huggingface/datasets/pull/3689#issuecomment-1032858345)

Unfortunately, in order to fix the datasets that are breaking after the rollback, I would suggest fixing their scripts so that the paths are handled more robustly (considering that they can be absolute or relative).

I agree with Albert, fixing all of the audio datasets isn't too big of a deal (yet). I can help with those if needed :)

Ok cool ! I'm completely rolling it back then

Alright I did the rollback and now you can get local paths :)
Feel free to try it out and let me know if it's good for you

Ok according to the CI there around 60+ datasets to fix

> fixing all of the audio datasets isn't too big of a deal (yet). I can help with those if needed :)

I can help with them too :)

Here is the full list to keep track of things:

- [x] air_dialogue
- [x] id_nergrit_corpus
- [ ] id_newspapers_2018
- [x] imdb
- [ ] indic_glue
- [ ] inquisitive_qg
- [x] klue
- [x] lama
- [x] lex_glue
- [ ] lm1b
- [x] amazon_polarity
- [ ] mac_morpho
- [ ] math_dataset
- [ ] md_gender_bias
- [ ] mdd
- [ ] assin
- [ ] atomic
- [ ] babi_qa
- [ ] mlqa
- [ ] mocha
- [ ] blended_skill_talk
- [ ] capes
- [ ] cbt
- [ ] newsgroup
- [ ] cifar10
- [ ] cifar100
- [ ] norec
- [ ] ohsumed
- [ ] code_x_glue_cc_clone_detection_poj104
- [x] openslr
- [ ] orange_sum
- [ ] paws
- [ ] paws-x
- [ ] cppe-5
- [ ] polyglot_ner
- [ ] dbrd
- [ ] empathetic_dialogues
- [ ] eraser_multi_rc
- [ ] flores
- [ ] flue
- [ ] food101
- [ ] py_ast
- [ ] qasc
- [ ] qasper
- [ ] race
- [ ] reuters21578
- [ ] ropes
- [ ] rotten_tomatoes
- [x] vivos
- [ ] wi_locness
- [ ] wiki_movies
- [ ] wikiann
- [ ] wmt20_mlqe_task1
- [ ] wmt20_mlqe_task2
- [ ] wmt20_mlqe_task3
- [ ] scicite
- [ ] xsum
- [ ] scielo
- [ ] scifact
- [ ] setimes
- [ ] social_bias_frames
- [ ] sogou_news
- [x] speech_commands
- [ ] ted_hrlr
- [ ] ted_multi
- [ ] tlc
- [ ] turku_ner_corpus

I'll do my best to fix as many as possible tomorrow :)

the audio datasets are fixed if I didn't forget anything :)

btw what are we gonna do with the community ones that would be broken after the fix?

Closing in favor of https://github.com/huggingface/datasets/pull/3736

Fixes https://github.com/huggingface/datasets/issues/3663

This is a proposed way of returning the old local file-based generator while keeping the new streaming generator intact.

TODO:
- [ ] brainstorm a bit more on https://github.com/huggingface/datasets/issues/3663 to see if we can do better
- [ ] refactor the heck out of this PR to avoid completely copying the logic between the two generators


[Audio] Path of Common Voice cannot be used for audio loading anymore

Something isn't working

Having talked to @lhoestq, I see that this feature is no longer supported.

I really don't think this was a good idea. It is a major breaking change and one for which we don't even have a working solution at the moment, which is bad for PyTorch as we don't want to force people to have `datasets` decode audio files automatically, but **really** bad for Tensorflow and Flax where we **currently cannot** even use `datasets` to load `.mp3` files - e.g. `common_voice` doesn't work anymore in a TF training script. Note this worked perfectly fine before making the change (think it was done [here](https://github.com/huggingface/datasets/pull/3290) no?)

IMO, it's really important to think about a solution here and I strongly favor to make a difference here between loading a dataset in streaming mode and in non-streaming mode, so that in non-streaming mode the actual downloaded file is displayed. It's really crucial for people to be able to analyse the original files IMO when the dataset is not downloaded in streaming mode.

There are the following reasons why it is paramount to have access to the **original** audio file in my opinion (in non-streaming mode):
- There are a wide variety of different libraries to load audio data with varying support on different platforms. For me it was quite clear that there is simply to single good library to load audio files for all platforms - so we have to leave the option to the user to decide which loading to use.
- We had support for audio datasets a long time before streaming audio was possible. There were quite some versions where  we advertised **everywhere** to load the audio from the path name (and there are many places where we still do even though it's not possible anymore). To give some examples:
     - Official example of TF Wav2Vec2: https://github.com/huggingface/transformers/blob/f427e750490b486944cc9be3c99834ad5cf78b57/src/transformers/models/wav2vec2/modeling_tf_wav2vec2.py#L1423 Wav2Vec2 is as important for speech as BERT is for NLP - so it's **very** important. The official example currently doesn't work and we don't even have a workaround for it for MP3 files at the moment. Same goes for Flax.
     - The most downloaded non-nlp checkpoint: https://huggingface.co/facebook/wav2vec2-base-960h#usage has a usage example which doesn't work anymore with the current datasets implementation. I'll update this now, but we have >1000 wav2vec2 checkpoints on the Hub and we can't update all the model cards.
    => This is a big breaking change with no current solution. For `transformers` breaking changes are one of the biggest complaints.
- Similar to this we also shouldn't assume that there is only one resampling method for Audio. I think it's good to have one offered automatically by `datasets`, but we have to leave the user the freedom to choose her/his own resampling as well. Resampling can take very different filtering windows and other parameters which are currently somewhat hardcoded in `datasets`, which users might very well want to change.

=> IMO, it's a **very** big priority to again have the correct absolute path in non-streaming mode. The other solution of providing a path-like object derived from the bytes stocked in the `.array` file is not nearly as user-friendly, but better than nothing.

Agree that we need to have access to the original sound files. Few days ago I was looking for these original files because I suspected there is bug in the audio resampling (confirmed in https://github.com/huggingface/datasets/issues/3662) and I want to do my own resampling to workaround the bug, which is now not possible anymore due to the unavailability of the original files.

@patrickvonplaten
> The other solution of providing a path-like object derived from the bytes stocked in the .array file is not nearly as user-friendly, but better than nothing

Just to clarify, here you describe the approach that uses the `Audio.decode` attribute to access the underlying bytes?

> The official example currently doesn't work and we don't even have a workaround for it for MP3 files at the moment

I'd assume this is because we use `sox_io` as a backend for decoding. However, soon we should be able to use `soundfile`, which supports path-like objects, for MP3 (https://github.com/huggingface/datasets/pull/3667#issuecomment-1030090627).

Your concern is reasonable, but there are situations where we can only serve bytes (see https://github.com/huggingface/datasets/pull/3685 for instance). IMO it makes sense to fix the affected datasets for now, but I don't think we should care too much whether we rely on local paths or bytes after soundfile adds support for MP3 as long as our examples work (shouldn't be too hard to update the `map_to_array` functions) and we properly document how to access the underlying path/bytes for custom decoding (via `ds.cast_column("audio", Audio(decode=False))`).

Related to this discussion: in https://github.com/huggingface/datasets/pull/3664#issuecomment-1031866858 I propose how we could change `iter_archive` to work for streaming and also return local paths (as it used too !). I'd love your opinions on this

> @patrickvonplaten
>
> > The other solution of providing a path-like object derived from the bytes stocked in the .array file is not nearly as user-friendly, but better than nothing
>
> Just to clarify, here you describe the approach that uses the `Audio.decode` attribute to access the underlying bytes?

Yes!

>
> > The official example currently doesn't work and we don't even have a workaround for it for MP3 files at the moment
>
> I'd assume this is because we use `sox_io` as a backend for decoding. However, soon we should be able to use `soundfile`, which supports path-like objects, for MP3 ([#3667 (comment)](https://github.com/huggingface/datasets/pull/3667#issuecomment-1030090627)).
> Your concern is reasonable, but there are situations where we can only serve bytes (see #3685 for instance). IMO it makes sense to fix the affected datasets for now, but I don't think we should care too much whether we rely on local paths or bytes after soundfile adds support for MP3 as long as our examples work (shouldn't be too hard to update the `map_to_array` functions) and we properly document how to access the underlying path/bytes for custom decoding (via `ds.cast_column("audio", Audio(decode=False))`).

Yes this might be, but I highly doubt that `soundfile` is the go-to library for audio then. @anton-l and I have tried out a bunch of different audio loading libraries (`soundfile`, `librosa`, `torchaudio`, pure `ffmpeg`, `audioread`, ...). One thing that was pretty clear  to me is that there is just no "de-facto standard" library and they all have pros and cons. None of the libraries really supports "batch"-ed audio loading. Some depend on PyTorch. `torchaudio` is 100x faster (really!) than `librosa's` fallback on MP3. `torchaudio` often has problems with multi-proessing, ... Also we should keep in mind that resampling is similarly not as simple as reading a text file. It's a pretty complex signal processing transform and people very well might want to use special filters, etc...at the moment we just hard-code `torchaudio's` or `librosa's` default filter when doing resampling.

=> All this to say that we **should definitely** care about whether we rely on local paths or bytes IMO. We don't want to loose all users that are forced to use `datasets` decoding or resampling or have to built a very much not intuitive way of loading bytes into a numpy array. It's much more intuitive to be able to inspect a local file. I feel pretty strongly about this and am happy to also jump on a call. Keeping libraries flexible and lean as well as exposing internals is very important IMO (this philosophy has worked quite well so far with Transformers).

Thanks a lot for the very detailed explanation. Now everything makes much more sense.

From https://github.com/huggingface/datasets/pull/3736 the Common Voice dataset now gives access to the local audio files as before

I understand the argument that it is bad to have a breaking change. How to deal with the introduction of breaking changes is a topic of its own and not sure how you want to deal with that (or is the policy this is never allowed, and there must be a `load_dataset_v2` or so if you really want to introduce a breaking change?).

Regardless of whether it is a breaking change, however, I don't see the other arguments.

> but **really** bad for Tensorflow and Flax where we **currently cannot** even use `datasets` to load `.mp3` files

I don't exactly understand this. Why not?

Why does the HF dataset on-the-fly decoding mechanism not work? Why is it anyway specific to PyTorch or TensorFlow? Isn't this independent?

But even if you just provide the raw bytes to TF, on TF you could just use sth like `tfio.audio.decode_mp3` or `tf.audio.decode_ogg` or `tfio.audio.decode_flac`?

> There are the following reasons why it is paramount to have access to the original audio file in my opinion  ...

I don't really understand the arguments (despite that it maybe breaks existing code). You anyway have the original audio files but it is just embedded in the dataset? I don't really know about any library which cannot also load the audio from memory (i.e. from the dataset).

Btw, on librosa being slow for decoding audio files, I saw that as well, so we have this comment RETURNN:

> Don't use librosa.load which internally uses audioread which would use Gstreamer as a backend which has multiple issues:
> https://github.com/beetbox/audioread/issues/62
> https://github.com/beetbox/audioread/issues/63
> Instead, use PySoundFile (soundfile), which is also faster. See here for discussions:
> https://github.com/beetbox/audioread/issues/64
> https://github.com/librosa/librosa/issues/681

Resampling is also a separate aspect, which is also less straightforward and with different compromises between speed and quality. So there the different tradeoffs and different implementations can make a difference.

However, I don't see how this is related to the question whether there should be the raw bytes inside the dataset or as separate local files.

Thanks for your comments here @albertz - cool to get your input!

Answering a bit here between the lines:

> I understand the argument that it is bad to have a breaking change. How to deal with the introduction of breaking changes is a topic of its own and not sure how you want to deal with that (or is the policy this is never allowed, and there must be a `load_dataset_v2` or so if you really want to introduce a breaking change?).
>
> Regardless of whether it is a breaking change, however, I don't see the other arguments.
>
> > but **really** bad for Tensorflow and Flax where we **currently cannot** even use `datasets` to load `.mp3` files
>
> I don't exactly understand this. Why not?

> Why does the HF dataset on-the-fly decoding mechanism not work? Why is it anyway specific to PyTorch or TensorFlow? Isn't this independent?

The problem with decoding on the fly is that we currently rely on `torchaudio` for this now which relies on `torch` which is not necessarily something people would like to install when using `tensorflow` or `flax`. Therefore we cannot just rely on people using the decoding on the fly method. We just didn't find a library that is ML framework independent and fast enough for all formats. `torchaudio` is currently in our opinion by far the best here.

So for TF and Flax it's important that users can load audio files or bytes they way the want to - this might become less important if we find (or make) a good library with few dependencies that is fast for all kinds of platforms / use cases.

Now the question is whether it's better to store audio data as a path to a file or as raw bytes I guess.\
My main arguments for storing the audio data as a path to a file is pretty much all about users experience - I don't really expect our users to understand the inner workings of datasets:

- 1. It's not straightforward to know which function to use to decode it - not all `load_audio(...)` or `read_audio(...)` work on raw bytes. E.g. Looking at https://pytorch.org/audio/stable/torchaudio.html?highlight=load#torchaudio.load one would not see directly how to load raw bytes . There are also some functions of other libraries which only work on files which would require the user to save the bytes as a file first before being able to load it.
- 2. It's difficult to see which format the bytes are coming from (mp3, ogg, ...) - guess this could be remedied by adding the format to each sample though
- 3. It is a bit scary IMO to see raw bytes for users. Overall, I think it's better to leave the data in it's raw form as this way it's much easier for people to play around with the audio files, less need to read docs because people don't worry about what happened to the audio files (are the bytes already resampled?)

But the argument that the audio should be loadable directly from memory is good - haven't thought about this too much.
I guess it's still very much possible for the user to do this:

```python
def save_as_bytes:
      batch["bytes"] = read_in_bytes_from_file(batch["file"])\
      os.remove(batch["file"])

ds = ds.map(save_as_bytes)

ds.save_to_disk(...)
```

Guess the question is more a bit about what should be the default case?

> The problem with decoding on the fly is that we currently rely on `torchaudio` for this now which relies on `torch` which is not necessarily something people would like to install when using `tensorflow` or `flax`. Therefore we cannot just rely on people using the decoding on the fly method. We just didn't find a library that is ML framework independent and fast enough for all formats. `torchaudio` is currently in our opinion by far the best here.

But how is this relevant for this issue here? I thought this issue here is about having the (correct) path in the dataset or having raw bytes in the dataset.

How did TF users use it at all then? Or they just do not use on-the-fly decoding? I did not even notice this problem (maybe because I had `torchaudio` installed). But what do they use instead?

But as I outlined before, they could just use `tfio.audio.decode_flac` and co, where it would be more natural if you already provide the raw bytes.

> Looking at https://pytorch.org/audio/stable/torchaudio.html?highlight=load#torchaudio.load one would not see directly how to load raw bytes

I was not really familiar with `torchaudio`. It seems that they really don't provide an easy/direct API to operate on raw bytes. Which is very strange and unfortunate because as far as I can see, all the underlying backend libraries (e.g. soundfile) easily allow that. So I would say that this is the fault of `torchaudio` then. But despite, if you anyway use `torchaudio` with `soundfile` backend, why not just use `soundfile` directly. It's very simple to use and crossplatform.

But ok, now we are just discussing how to handle the on-the-fly decoding. I still think this is a separate issue and having raw bytes in the dataset instead of local files should just be fine as well.

> It is a bit scary IMO to see raw bytes for users.

I think nobody who writes code is scared by seeing the raw bytes content of a binary file. :)

> I guess it's still very much possible for the user to do this:
>
> ```python
> def save_as_bytes:
>       batch["bytes"] = read_in_bytes_from_file(batch["file"])\
>       os.remove(batch["file"])
>
> ds = ds.map(save_as_bytes)
>
> ds.save_to_disk(...)
> ```

In https://github.com/huggingface/datasets/pull/4184#issuecomment-1105191639, you said/proposed that this `map` is not needed anymore and `save_to_disk` could do it automatically (maybe via some option)?

> Guess the question is more a bit about what should be the default case?

Yea this is up to you. I'm happy as long as we can get it the way we want easily and this is a well supported use case. :)

> In https://github.com/huggingface/datasets/pull/4184#issuecomment-1105191639, you said/proposed that this map is not needed anymore and save_to_disk could do it automatically (maybe via some option)?

Yes! Should be super easy now see discussion here: https://github.com/rwth-i6/i6_core/issues/257#issuecomment-1105494468

Thanks for the super useful input :-)

> Despite the comments that this has been fixed, I am finding the exact same problem is occurring again (with datasets version 2.3.2)

It appears downgrading to torchaudio 0.11.0 fixed this problem.

@DCNemesis, sorry which problem exactly is occuring again? Also cc @lhoestq @polinaeterna here

@patrickvonplaten @lhoestq @polinaeterna I was unable to load audio from Common Voice using 🤗 with the current version of torchaudio, but downgrading to torchaudio 0.11.0 fixed it. This is probably more of a torch problem than a Hugging Face problem.

@DCNemesis that's interesting, could you please share the error message if you still can access it?

@polinaeterna I believe it is the same exact error as above. It occurs on other .mp3 sources as well, but the problem is with torchaudio > 0.11.0. I've created a short colab notebook that reproduces the error, and the fix here: https://colab.research.google.com/drive/18wsuwdHwBPN3JkcnhEtk8MUYqF9swuWZ?usp=sharing

Hi @DCNemesis,

Your issue was slightly different from the original one in this issue page. Yours seems related to a change in the backend used by `torchaudio` (`ffmpeg` instead of `sox`). Refer to the issue page here:
- #4776

Normally, it should be circumvented with the patch made by @polinaeterna in:
- #4923

I think the original issue reported here was already fixed by:
- #3736

Otherwise, feel free to reopen.

## Describe the bug

## Steps to reproduce the bug
```python
from datasets import load_dataset
from torchaudio import load

ds = load_dataset("common_voice", "ab", split="train")

# both of the following commands fail at the moment
load(ds[0]["audio"]["path"])
load(ds[0]["path"])
```

## Expected results

The path should be the complete absolute path to the downloaded audio file not some relative path.

## Actual results

```bash
~/hugging_face/venv_3.9/lib/python3.9/site-packages/torchaudio/backend/sox_io_backend.py in load(filepath, frame_offset, num_frames, normalize, channels_first, format)
    150                 filepath, frame_offset, num_frames, normalize, channels_first, format)
    151         filepath = os.fspath(filepath)
--> 152     return torch.ops.torchaudio.sox_io_load_audio_file(
    153         filepath, frame_offset, num_frames, normalize, channels_first, format)
    154

RuntimeError: Error loading audio file: failed to open file cv-corpus-6.1-2020-12-11/ab/clips/common_voice_ab_19904194.mp3
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3.dev0
- Platform: Linux-5.4.0-96-generic-x86_64-with-glibc2.27
- Python version: 3.9.1
- PyArrow version: 3.0.0


[Audio] MP3 resampling is incorrect when dataset's audio files have different sampling rates

Thanks @lhoestq for finding the reason of incorrect resampling. This issue affects all languages which have sound files with different sampling rates such as Turkish and Luganda.

@cahya-wirawan - do you know how many languages have different sampling rates in Common Voice? I'm quite surprised to see this for multiple languages actually

@cahya-wirawan, I can reproduce the problem for Common Voice 7 for Turkish. Here a script you can use:

```python
#!/usr/bin/env python3
from datasets import load_dataset
import torchaudio
from io import BytesIO
from datasets import Audio
from collections import Counter
import sys

ds_name = str(sys.argv[1])
lang = str(sys.argv[2])

ds = load_dataset(ds_name, lang, split="train", use_auth_token=True)
ds = ds.cast_column("audio", Audio(decode=False))

all_sampling_rates = []

def print_sampling_rate(x):
    x, sr = torchaudio.load(BytesIO(x["audio"]["bytes"]), format="mp3")
    all_sampling_rates.append(sr)

ds.map(print_sampling_rate)

print(Counter(all_sampling_rates))
```

can be run with:

```bash
python run.py mozilla-foundation/common_voice_7_0 tr
```

For CV 6.1 all samples seem to have the same audio

It actually shows that many more samples are in 32kHz format than it 48kHz which is unexpected. Thanks a lot for flagging! Will contact Common Voice about this as well

I only checked the CV 7.0 for Turkish, Luganda and Indonesian, they have audio files with difference sampling rates, and all of them are affected by this issue. Percentage of incorrect resampling as follow, Turkish: 9.1%, Luganda: 88.2% and Indonesian: 64.1%.
I checked it using the original CV files. I check the original sampling rates and the length of audio array of each files and compare it with the length of audio array (and the sampling rate which is always 48kHz) from mozilla-foundation/common_voice_7_0 datasets. if the length of audio array from dataset is not equal to 48kHz/original sampling rate * length of audio array of the original audio file then it is affected,

Ok wow, thanks a lot for checking this - you've found a pretty big bug :sweat_smile: It seems like **a lot** more datasets are actually affected than I original thought. We'll try to solve this as soon as possible and make an announcement tomorrow.

The Audio feature resampler for MP3 gets stuck with the first original frequencies it meets, which leads to subsequent decoding to be incorrect.

Here is a code to reproduce the issue:

Let's first consider two audio files with different sampling rates 32000 and 16000:
```python
# first download a mp3 file with sampling_rate=32000
!wget https://file-examples-com.github.io/uploads/2017/11/file_example_MP3_700KB.mp3

import torchaudio

audio_path = "file_example_MP3_700KB.mp3"
audio_path2 = audio_path.replace(".mp3", "_resampled.mp3")
resample = torchaudio.transforms.Resample(32000, 16000)  # create a new file with sampling_rate=16000
torchaudio.save(audio_path2, resample(torchaudio.load(audio_path)[0]), 16000)
```

Then we can see an issue here when decoding:
```python
from datasets import Dataset, Audio

dataset = Dataset.from_dict({"audio": [audio_path, audio_path2]}).cast_column("audio", Audio(48000))
dataset[0]  # decode the first audio file sets the resampler orig_freq to 32000
print(dataset .features["audio"]._resampler.orig_freq)
# 32000
print(dataset[0]["audio"]["array"].shape)  # here decoding is fine
# (1308096,)

dataset = Dataset.from_dict({"audio": [audio_path, audio_path2]}).cast_column("audio", Audio(48000))
dataset[1]  # decode the second audio file sets the resampler orig_freq to 16000
print(dataset .features["audio"]._resampler.orig_freq)
# 16000
print(dataset[0]["audio"]["array"].shape)  # here decoding uses orig_freq=16000 instead of 32000
# (2616192,)
```

The value of `orig_freq` doesn't change no matter what file needs to be decoded

cc @patrickvonplaten @anton-l @cahya-wirawan @albertvillanova

The issue seems to be here in `Audio.decode_mp3`:
https://github.com/huggingface/datasets/blob/4c417d52def6e20359ca16c6723e0a2855e5c3fd/src/datasets/features/audio.py#L176-L180


Remove unnecessary 'r' arg in

The CI failure is only because of the datasets is missing some sections in their cards - we can ignore that since it's unrelated to this PR

Originally from #3489


Change HTTP links to HTTPS

I tested the links. I also fixed some typos.

Originally from #3489


push_to_hub but preview not working

Hi @thomas-happify, please note that the preview may take some time before rendering the data.

I've seen it is already working.

I close this issue. Please feel free to reopen it if the problem arises again.

## Dataset viewer issue for '*happifyhealth/twitter_pnn*'

**Link:** *[link to the dataset viewer page](https://huggingface.co/datasets/happifyhealth/twitter_pnn)*

I used
```
dataset.push_to_hub("happifyhealth/twitter_pnn")
```
but the preview is not working.

Am I the one who added this dataset ? Yes


Dataset viewer issue for *P3*

The error is now:

```
Status code:   400
Exception:     Status400Error
Message:       this dataset is not supported for now.
```

We've disabled the dataset viewer for several big datasets like this one. We hope being able to reenable it soon.

The list of splits cannot be obtained. cc @huggingface/datasets

```
Error code:   SplitsNamesError
Exception:    SplitsNotFoundError
Message:      The split names could not be parsed from the dataset config.
Traceback:    Traceback (most recent call last):
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 354, in get_dataset_config_info
                 for split_generator in builder._split_generators(
               File "/tmp/modules-cache/datasets_modules/datasets/bigscience--P3/12c0badfecad4564ecb8a6f81b5d0559656f269f08b13c59c93283f3a84134ba/P3.py", line 154, in _split_generators
                 data_dir = dl_manager.download_and_extract(_URLs)
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 944, in download_and_extract
                 return self.extract(self.download(url_or_urls))
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 907, in extract
                 urlpaths = map_nested(self._extract, path_or_paths, map_tuple=True)
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 393, in map_nested
                 mapped = [
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 394, in <listcomp>
                 _single_map_nested((function, obj, types, None, True, None))
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 346, in _single_map_nested
                 return {k: _single_map_nested((function, v, types, None, True, None)) for k, v in pbar}
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 346, in <dictcomp>
                 return {k: _single_map_nested((function, v, types, None, True, None)) for k, v in pbar}
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 346, in _single_map_nested
                 return {k: _single_map_nested((function, v, types, None, True, None)) for k, v in pbar}
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 346, in <dictcomp>
                 return {k: _single_map_nested((function, v, types, None, True, None)) for k, v in pbar}
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 330, in _single_map_nested
                 return function(data_struct)
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 912, in _extract
                 protocol = _get_extraction_protocol(urlpath, use_auth_token=self.download_config.use_auth_token)
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 402, in _get_extraction_protocol
                 return _get_extraction_protocol_with_magic_number(f)
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 367, in _get_extraction_protocol_with_magic_number
                 magic_number = f.read(MAGIC_NUMBER_MAX_LENGTH)
               File "/src/services/worker/.venv/lib/python3.9/site-packages/fsspec/implementations/http.py", line 574, in read
                 return super().read(length)
               File "/src/services/worker/.venv/lib/python3.9/site-packages/fsspec/spec.py", line 1575, in read
                 out = self.cache._fetch(self.loc, self.loc + length)
               File "/src/services/worker/.venv/lib/python3.9/site-packages/fsspec/caching.py", line 377, in _fetch
                 self.cache = self.fetcher(start, bend)
               File "/src/services/worker/.venv/lib/python3.9/site-packages/fsspec/asyn.py", line 111, in wrapper
                 return sync(self.loop, func, *args, **kwargs)
               File "/src/services/worker/.venv/lib/python3.9/site-packages/fsspec/asyn.py", line 96, in sync
                 raise return_result
               File "/src/services/worker/.venv/lib/python3.9/site-packages/fsspec/asyn.py", line 53, in _runner
                 result[0] = await coro
               File "/src/services/worker/.venv/lib/python3.9/site-packages/fsspec/implementations/http.py", line 616, in async_fetch_range
                 out = await r.read()
               File "/src/services/worker/.venv/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 1036, in read
                 self._body = await self.content.read()
               File "/src/services/worker/.venv/lib/python3.9/site-packages/aiohttp/streams.py", line 375, in read
                 block = await self.readany()
               File "/src/services/worker/.venv/lib/python3.9/site-packages/aiohttp/streams.py", line 397, in readany
                 await self._wait("readany")
               File "/src/services/worker/.venv/lib/python3.9/site-packages/aiohttp/streams.py", line 304, in _wait
                 await waiter
             aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed

             The above exception was the direct cause of the following exception:

             Traceback (most recent call last):
               File "/src/services/worker/src/worker/responses/splits.py", line 75, in get_splits_response
                 split_full_names = get_dataset_split_full_names(dataset, hf_token)
               File "/src/services/worker/src/worker/responses/splits.py", line 35, in get_dataset_split_full_names
                 return [
               File "/src/services/worker/src/worker/responses/splits.py", line 38, in <listcomp>
                 for split in get_dataset_split_names(dataset, config, use_auth_token=hf_token)
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 404, in get_dataset_split_names
                 info = get_dataset_config_info(
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 359, in get_dataset_config_info
                 raise SplitsNotFoundError("The split names could not be parsed from the dataset config.") from err
             datasets.inspect.SplitsNotFoundError: The split names could not be parsed from the dataset config.
```

Closing in favor of https://huggingface.co/datasets/bigscience/P3/discussions/6 and https://github.com/huggingface/datasets-server/issues/1689

## Dataset viewer issue for '*P3*'

**Link: https://huggingface.co/datasets/bigscience/P3**

```
Status code:   400
Exception:     SplitsNotFoundError
Message:       The split names could not be parsed from the dataset config.
```
Am I the one who added this dataset ? No


Extend dataset builder for streaming in `get_dataset_split_names`

I'm impatient to see if it has an impact on the number of valid datasets for the dataset viewer. For the record, today:

<img width="660" alt="Capture d’écran 2022-02-01 à 14 32 19" src="https://user-images.githubusercontent.com/1676121/151977579-b5a239d9-6662-4aeb-bfd1-eef6b8249991.png">

This is now available in `datasets` 1.18.3 :)

I'm on it https://github.com/huggingface/datasets-preview-backend/issues/130

The result:
<img width="671" alt="Capture d’écran 2022-02-03 à 23 45 55" src="https://user-images.githubusercontent.com/1676121/152442169-bfdac643-9a00-4901-bfa7-1d60a1679d4b.png">

Not very different. Maybe it fixed issues in the community datasets... But I'm not 100% the two states are comparable (datasets have been created, or updated, meanwhile)

Currently, `get_dataset_split_names` doesn't extend a builder module to support streaming, even though it uses `StreamingDownloadManager` to download data. This PR fixes that.

To test the change, run the following:
```bash
pip install git+https://github.com/huggingface/datasets.git@fix-get_dataset_split_names-streaming
python -c "from datasets import get_dataset_split_names; print(get_dataset_split_names('facebook/multilingual_librispeech', 'german', download_mode='force_redownload', revision='137923f945552c6afdd8b60e4a7b43e3088972c1'))"
```


checksum error subjqa dataset

Something isn't working

Hi @RensDimmendaal,

I'm sorry but I can't reproduce your bug:
```python
In [1]: from datasets import load_dataset
   ...: ds = load_dataset("subjqa", "electronics")
Downloading builder script: 9.15kB [00:00, 4.10MB/s]
Downloading metadata: 17.7kB [00:00, 8.51MB/s]
Downloading and preparing dataset subjqa/electronics (download: 10.86 MiB, generated: 3.01 MiB, post-processed: Unknown size, total: 13.86 MiB) to .../.cache/huggingface/datasets/subjqa/electronics/1.1.0/e5588f9298ff2d70686a00cc377e4bdccf4e32287459e3c6baf2dc5ab57fe7fd...
Downloading data: 11.4MB [00:03, 3.50MB/s]
Dataset subjqa downloaded and prepared to .../.cache/huggingface/datasets/subjqa/electronics/1.1.0/e5588f9298ff2d70686a00cc377e4bdccf4e32287459e3c6baf2dc5ab57fe7fd. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 605.09it/s]

In [2]: ds
Out[2]:
DatasetDict({
    train: Dataset({
        features: ['domain', 'nn_mod', 'nn_asp', 'query_mod', 'query_asp', 'q_reviews_id', 'question_subj_level', 'ques_subj_score', 'is_ques_subjective', 'review_id', 'id', 'title', 'context', 'question', 'answers'],
        num_rows: 1295
    })
    test: Dataset({
        features: ['domain', 'nn_mod', 'nn_asp', 'query_mod', 'query_asp', 'q_reviews_id', 'question_subj_level', 'ques_subj_score', 'is_ques_subjective', 'review_id', 'id', 'title', 'context', 'question', 'answers'],
        num_rows: 358
    })
    validation: Dataset({
        features: ['domain', 'nn_mod', 'nn_asp', 'query_mod', 'query_asp', 'q_reviews_id', 'question_subj_level', 'ques_subj_score', 'is_ques_subjective', 'review_id', 'id', 'title', 'context', 'question', 'answers'],
        num_rows: 255
    })
})
```

Could you please try again and see if the problem persists?

If that is the case, you can circumvent the issue by passing `ignore_verifications`:
```python
ds = load_dataset("subjqa", "electronics", ignore_verifications=True)

Thanks checking!

You're totally right. I don't know what's changed, but I'm glad it's working now!

## Describe the bug

I get a checksum error when loading the `subjqa` dataset (used in the transformers book).

## Steps to reproduce the bug
```python
from datasets import load_dataset
subjqa = load_dataset("subjqa","electronics")
```

## Expected results
Loading the dataset

## Actual results

```
---------------------------------------------------------------------------

NonMatchingChecksumError                  Traceback (most recent call last)

<ipython-input-2-d2857d460155> in <module>()
      2 from datasets import load_dataset
      3
----> 4 subjqa = load_dataset("subjqa","electronics")

3 frames

/usr/local/lib/python3.7/dist-packages/datasets/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     38     if len(bad_urls) > 0:
     39         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 40         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     41     logger.info("All the checksums matched successfully" + for_verification_name)
     42

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://github.com/lewtun/SubjQA/archive/refs/heads/master.zip']
```

## Environment info

Google colab

- `datasets` version: 1.18.2
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 3.0.0


Pubmed dataset not reachable

Something isn't working

Hi @abhi-mosaic, thanks for reporting.

I'm looking at it...

Hey @albertvillanova, sorry to reopen this... I can confirm that on `master` branch the dataset is downloadable now but it is still broken in streaming mode:

```python
 >>> import datasets
 >>> pubmed_train = datasets.load_dataset('pubmed', split='train', streaming=True)
 >>> next(iter(pubmed_train))
```
```
 No such file or directory: 'gzip://pubmed22n0001.xml::ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed22n0001.xml.gz'
```

Hi @abhi-mosaic, would you mind opening another issue for this new problem?

First issue (already solved) was a ConnectionError due to the yearly update release of PubMed: we fixed it by updating the URLs from year 2021 to year 2022.

However this is another problem: to make pubmed streamable. Please note that NOT all our datastes are streamable: we are making streamable more and more of them... but this is an on-going process...

Thanks.

@albertvillanova
When I tried below codes, I got the similar error

```

dataset=load_dataset("pubmed",split="train")

Couldn't reach ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed21n0601.xml.gz
```

@y-rok you need to update `datasets`:
```shell
pip install -U datasets
```

## Describe the bug
Trying to use the `pubmed` dataset fails to reach / download the source files.

## Steps to reproduce the bug
```python
pubmed_train = datasets.load_dataset('pubmed', split='train')
```

## Expected results
Should begin downloading the pubmed dataset.

## Actual results
```
ConnectionError: Couldn't reach ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed21n0865.xml.gz (InvalidSchema("No connection adapters were found for 'ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed21n0865.xml.gz'"))
```

## Environment info
- `datasets` version: 1.18.2
- Platform: macOS-11.4-x86_64-i386-64bit
- Python version: 3.8.2
- PyArrow version: 6.0.0


Better TQDM output

@lhoestq I've created a notebook for you to see the difference: https://colab.research.google.com/drive/1by3EqnoKvC2p-yKW4lPDGOFOZHyGVyeQ?usp=sharing.

Feel free to suggest better descriptions for the progress bars.

If everything looks good, think we can merge.

This PR does the following:
* if `dataset_infos.json` exists for a dataset, uses `num_examples` to print the total number of examples that needs to be generated (in `builder.py`)
* fixes `tqdm` + multiprocessing in Jupyter Notebook/Colab (the issue stems from this commit in the `tqdm` repo: https://github.com/tqdm/tqdm/commit/f7722edecc3010cb35cc1c923ac4850a76336f82)
* adds the missing `drop_last_batch` and `with_ranks` params to `DatasetDict.map`
* correctly computes the number of iterations in `map` and the CSV/JSON loader when `batched=True` to fix `tqdm` progress bars
* removes the `bool(logging.get_verbosity() == logging.NOTSET)` (or simplifies `bool(logging.get_verbosity() == logging.NOTSET) or not utils.is_progress_bar_enabled()` to `not utils.is_progress_bar_enabled()`)  condition and uses `utils.is_progress_bar_enabled` to check if `tqdm` output is enabled (this comment from @stas00 explains why  the  `bool(logging.get_verbosity() == logging.NOTSET)` check is problematic: https://github.com/huggingface/transformers/issues/14889#issue-1087318463)

Fix #2630


Something isn't working

## Describe the bug

`to_json` in multiprocessing fashion sometimes deadlock, instead of raising exceptions. Temporary solution is to see that it deadlocks, and then reduce the number of processes or batch size in order to reduce the memory footprint.

As @lhoestq pointed out, this might be related to https://bugs.python.org/issue22393#msg315684 where `multiprocessing` fails to raise the OOM exception. One suggested alternative is not use `concurrent.futures` instead.

## Steps to reproduce the bug

## Expected results

Script fails when one worker hits OOM, and raise appropriate error.

## Actual results

Deadlock

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.1
- Platform: Linux
- Python version: 3.8
- PyArrow version: 6.0.1


sp. Columbia => Colombia

The original openslr site mixed both names https://openslr.org/72/ :-)

Yeah, I filed the issue to have it fixed there last year, but it looks like they missed a few.

"Columbia" is various places in North America. The country is "Colombia".


Update link in wiki_bio dataset

> all the tests pass, but I'm still not able to import the dataset

Since it's not merged on `master` yet, you have to provide the path to your local `wiki_bio.py` to use it.
Indeed the library downloads the dataset files from `master` if you have a dev installation of the library.

I agree it would be nice to change that, and use the local dataset scripts from the `datasets` directory - it feels definitely more natural.

Fixes #3580 and makes the wiki_bio dataset work again. I changed the link and some documentation, and all the tests pass. Thanks @lhoestq for uploading the dataset to the HuggingFace data bucket.

@lhoestq -- all the tests pass, but I'm still not able to import the dataset, as the old Google Drive link is cached somewhere:
```python
>>> from datasets import load_dataset
load_dataset("wiki_bio>>> load_dataset("wiki_bio")
Using custom data configuration default
Downloading and preparing dataset wiki_bio/default (download: 318.53 MiB, generated: 736.94 MiB, post-processed: Unknown size, total: 1.03 GiB) to /home/jxm3/.cache/huggingface/datasets/wiki_bio/default/1.1.0/5293ce565954ba965dada626f1e79684e98172d950371d266bf3caaf87e911c9...
Traceback (most recent call last):
  ...
  File "/home/jxm3/random/datasets/src/datasets/utils/file_utils.py", line 612, in get_from_cache
    raise FileNotFoundError(f"Couldn't find file at {url}")
FileNotFoundError: Couldn't find file at https://drive.google.com/uc?export=download&id=1L7aoUXzHPzyzQ0ns4ApBbYepsjFOtXil
```

what do I have to do to invalidate the cache and actually import the dataset? It's clearly set up correctly, since the data is downloaded and processed by the tests.

As an aside, this caching-loading-scripts behavior makes for a really bad developer experience. I just wasted an hour trying to figure out where the caching was happening and how to disable it, and I don't know. All I wanted to do was update the link and submit a pull request! I recommend that you all either change this behavior (i.e. updating the link to a dataset should "just work") or document it, since I couldn't find any information about this in the contributing.md or readme or anywhere else! Thanks!


Allow 'to_json' to run in unordered fashion in order to lower memory footprint

Hi @thomasw21, I remember suggesting `imap_unordered` to @lhoestq at that time to speed up `to_json` further but after trying `pool_imap` on multiple datasets (>9GB) , memory utilisation was almost constant and we decided to go ahead with that only.

1. Did you try this without `gzip`? Because `gzip`  feature was introduced recently and I didn't check multi_proc thing with `gzip`. One thing I know is that `gzip` is slow in our implementation than `zip` (it's a WIP #3551)
2. You can try reducing your batch size, this can also help in avoiding OOM errors!

Thanks @bhavitvyamalik ! I see. I'm not sure this PR actually fixes things for me either (I ended up reducing the num_proc/batch_size to lower it). It does allow the process to run for longer, but I think the reason why it was waiting is that one of the process crashes .... Unfortunately I was working on a setup with a low RAM/cpu core ratio. I'm actually very surprised that it doesn't change memory utilization, otherwise I don't see the purpose of `imap_unordered` existing. I think it's main purpose are when you have high variance in samples (in terms of bytes), which causes unecessary accumulation in `imap`
  1. Did not try without `gzip`
  2. Yeah or `num_proc`

Can you please try without `gzip` to see how it performs? If it works fine then we can improve `gzip` from our side (I'm already working on it)

I'll be busy for next few weeks on another project, will do as soon as I have some bandwidth.

Yes we can close this PR if considered unneeded.

I'm using `to_json(..., num_proc=num_proc, compressiong='gzip')` with `num_proc>1`. I'm having an issue where things seem to deadlock at some point. Eventually I see OOM. I'm guessing it's an issue where one process starts to take a long time for a specific batch, and so other process keep accumulating their results in memory.

In order to flush memory, I propose we use optional `imap_unordered`. This will prevent one process to block the other ones. The logical thinking is that index are rarily relevant, and in one wants to keep an index, one can still create another column and reconstruct from there.


Add IGLUE dataset

dataset request

Requesting to add a new dataset

multimodal

Multimodal datasets

## Adding a Dataset
- **Name:** IGLUE
- **Description:** IGLUE brings together 4 vision-and-language tasks across 20 languages (Twitter [thread](https://twitter.com/ebugliarello/status/1487045497583976455?s=20&t=SB4LZGDhhkUW83ugcX_m5w))
- **Paper:** https://arxiv.org/abs/2201.11732
- **Data:** https://github.com/e-bug/iglue
- **Motivation:** This dataset would provide a nice example of combining the text and image features of `datasets` together for multimodal applications.

Note: the data / code are not yet visible on the GitHub repo, so I've pinged the authors for more information.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Fix Windows CI: bump python to 3.7

Python>=3.7 is needed to install `tokenizers` 0.11


Fix `add_column` on datasets with indices mapping

Sure, let's include this in today's release.

Cool ! The windows CI should be fixed on master now, feel free to merge :)

My initial idea was to avoid the `flatten_indices` call and reorder a new column instead, but in the end I decided to follow `concatenate_datasets` and use `flatten_indices` to avoid padding when `dataset._indices.num_rows != dataset._data.num_rows`.

Fix #3599


Fix streaming datasets that are not reset correctly

Works smoothly with the `transformers.Trainer` class now, thank you!

Streaming datasets that use `StreamingDownloadManager.iter_archive` and `StreamingDownloadManager.iter_files` had some issues. Indeed if you try to iterate over such dataset twice, then the second time it will be empty.

This is because the two methods above are generator functions. I fixed this by making them return iterables that are reset properly instead.

Close https://github.com/huggingface/datasets/issues/3645
cc @anton-l


Streaming dataset based on dl_manager.iter_archive/iter_files are not reset correctly

Hi ! When iterating over a streaming dataset once, it's not reset correctly because of some issues with `dl_manager.iter_archive` and `dl_manager.iter_files`. Indeed they are generator functions (so the iterator that is returned can be exhausted). They should be iterables instead, and be reset if we do a for loop again:

```python
from datasets import load_dataset
d = load_dataset("common_voice", "ab", split="test", streaming=True)

i = 0
for i, _ in enumerate(d):
    pass
print(i)  # 8
# let's do it again
i = 0
for i, _ in enumerate(d):
    pass
print(i)  # 0
```