"no-token" is passed to `huggingface_hub` when token is `None`

bug

Something isn't working

## Describe the bug

In the 2 lines listed below, a token is passed to `huggingface_hub` to get information from a dataset. If no token is provided, a "no-token" string is passed. What is the purpose of it ? If no real, I would prefer if the `None` value could be sent directly to be handle by `huggingface_hub`. I feel that here it is working because we assume the token will never be validated.

https://github.com/huggingface/datasets/blob/5b23f58535f14cc4dd7649485bce1ccc836e7bca/src/datasets/load.py#L753
https://github.com/huggingface/datasets/blob/5b23f58535f14cc4dd7649485bce1ccc836e7bca/src/datasets/load.py#L1121

## Expected results
Pass `token=None` to `huggingface_hub`.

## Actual results
`token="no-token"` is passed.

## Environment info
`huggingface_hub v0.10.0dev`


Running add_column() seems to corrupt existing sequence-type column info

Something isn't working

I have a dataset that contains a column ("foo") that is a sequence type of length 4. So when I run .to_pandas() on it, the resulting dataframe correctly contains 4 columns - foo_0, foo_1, foo_2, foo_3. So the 1st row of the dataframe might look like:

ds = load_dataset(...)
df = ds.to_pandas()

df:
foo_0 | foo_1 | foo_2 | foo_3
0.0 | 1.0 | 2.0 | 3.0

If I run .add_column("new_col", data) on the dataset, and then .to_pandas() on the resulting new dataset, the resulting dataframe contains only 2 columns - foo, new_col. The values in column foo are lists of length 4, the 4 elements that should have been split into separate columns. Dataframe 1st row would be:

ds = load_dataset(...)
new_ds = ds.add_column("new_col", data)
df = new_ds.to_pandas()

df:
foo | new_col
[0.0, 1.0, 2.0, 3.0] | new_val

I've explored the 2 datasets in a debugger and haven't noticed any changes to any attributes related to the foo column, but I can't determine why the dataframes are so different.


Add `IterableDataset.from_generator` to the API

enhancement

New feature or request

good first issue

Good for newcomers

We've just added `Dataset.from_generator` to the API. It would also be cool to add `IterableDataset.from_generator` to support creating an iterable dataset from a generator.

cc @lhoestq


Embed image/audio data in dl_and_prepare parquet

_The documentation is not available anymore as the PR was closed or merged._

Embed the bytes of the image or audio files in the Parquet files directly, instead of having a "path" that points to a local file.

Indeed Parquet files are often used to share data or to be used by workers that may not have access to the local files.


[doc] Fix broken snippet that had too many quotes

Spent the day familiarising myself with the huggingface line of products, and happened to run into some small issues here and there. Magically, I've found exactly one small issue in `transformers`, one in `accelerate` and now one in `datasets`, hah!

As for this PR, the issue seems solved according to the [new PR documentation](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4986/en/process#map):
![image](https://user-images.githubusercontent.com/37621491/190646405-6afa06fa-9eac-48f6-ab30-2677944fb7b6.png)

Hello!

### Pull request overview
* Fix broken snippet in https://huggingface.co/docs/datasets/main/en/process that has too many quotes

### Details
The snippet in question can be found here: https://huggingface.co/docs/datasets/main/en/process#map
This screenshot shows the issue, there is a quote too many, causing the snippet to be colored incorrectly:
![image](https://user-images.githubusercontent.com/37621491/190640627-f7587362-0e44-4464-a5d1-a0b98df6986f.png)

The change speaks for itself.

Thank you for the detailed documentation, by the way.

- Tom Aarsen


[WIP] Prefer split patterns from directories over split patterns from filenames

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4985). All of your documentation changes will be reflected on that endpoint.

related to https://github.com/huggingface/datasets/issues/4895

todo:

- [ ] test


docs: ✏ add links to the Datasets API

OK, thanks @lhoestq. I'll close this PR, and come back to it with @stevhliu once we work on https://github.com/huggingface/datasets-server/issues/568

I added some links to the Datasets API in the docs. See https://github.com/huggingface/datasets-server/pull/566 for a companion PR in the datasets-server. The idea is to improve the discovery of the API through the docs.

I'm a bit shy about pasting a lot of links to the API in the docs, so it's minimal for now. I'm interested in ideas to integrate the API better in these docs without being too much. cc @lhoestq @julien-c @albertvillanova @stevhliu.


How to convert  torch.utils.data.Dataset to huggingface dataset?

enhancement

New feature or request

Hi! I think you can use the newly-added `from_generator` method for that:
```python
from datasets import Dataset

def gen():
    for idx in len(torch_dataset):
        yield torch_dataset[idx]  # this has to be a dictionary
    ## or if it's an IterableDataset
    # for ex in torch_dataset:
    #     yield ex

dset = Dataset.from_generator(gen)
```

Maybe `Dataset.from_list` can work as well no ?
```python
from datasets import Dataset

dset = Dataset.from_list(torch_dataset)
```

I look through the huggingface dataset docs, and it seems that there is no offical support function to  convert  `torch.utils.data.Dataset`  to huggingface dataset. However, there is a way to convert huggingface dataset to  `torch.utils.data.Dataset`, like below:
```python
from datasets import Dataset
data = [[1, 2],[3, 4]]
ds = Dataset.from_dict({"data": data})
ds = ds.with_format("torch")
ds[0]
ds[:2]
```
So is there something I miss, or there IS no function to convert  `torch.utils.data.Dataset` to huggingface dataset. If so, is there any way to do this convert?
Thanks.


Create dataset_infos.json with VALIDATION and TEST splits

bug

Something isn't working

The problem is described in that [issue](https://github.com/huggingface/datasets/issues/4895#issuecomment-1247975569).

> When I try to create data_infos.json using datasets-cli test Peter.py --save_infos --all_configs I get an error:
> ValueError: Unknown split "test". Should be one of ['train'].
>
> The data_infos.json is created perfectly fine when I use only one split - datasets.Split.TRAIN
>
> You can find the code here: https://huggingface.co/datasets/sberbank-ai/Peter/tree/add_splits (add_splits branch)

I tried to clear the cache folder, than I got an another error. I run:

```
git clone https://huggingface.co/datasets/sberbank-ai/Peter
cd Peter
git checkout add_splits  # switch to a add_splits branch
rm dataset_infos.json  # remove local dataset_infos.json
rm -r ~/.cache/huggingface   # remove cached dataset_infos.json
datasets-cli test Peter.py --save_infos --all_configs  # trying to create new dataset_infos.json
```

The error message:
```
Using custom data configuration default
Testing builder 'default' (1/1)
Downloading and preparing dataset peter/default to /Users/kalinin/.cache/huggingface/datasets/peter/default/0.0.0/ef579519e140d6a40df2555996f26165f04c47557d7373709c8d7e7b4fd7465d...
Downloading data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 5160.63it/s]
Extracting data files:   0%|                                                                                | 0/4 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/usr/local/bin/datasets-cli", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/datasets/commands/datasets_cli.py", line 39, in main
    service.run()
  File "/usr/local/lib/python3.9/site-packages/datasets/commands/test.py", line 137, in run
    builder.download_and_prepare(
  File "/usr/local/lib/python3.9/site-packages/datasets/builder.py", line 704, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.9/site-packages/datasets/builder.py", line 1227, in _download_and_prepare
    super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
  File "/usr/local/lib/python3.9/site-packages/datasets/builder.py", line 771, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/Users/kalinin/.cache/huggingface/modules/datasets_modules/datasets/Peter/ef579519e140d6a40df2555996f26165f04c47557d7373709c8d7e7b4fd7465d/Peter.py", line 23, in _split_generators
    data_files = dl_manager.download_and_extract(_URLS)
  File "/usr/local/lib/python3.9/site-packages/datasets/download/download_manager.py", line 431, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/usr/local/lib/python3.9/site-packages/datasets/download/download_manager.py", line 403, in extract
    extracted_paths = map_nested(
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 393, in map_nested
    mapped = [
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 394, in <listcomp>
    _single_map_nested((function, obj, types, None, True, None))
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 330, in _single_map_nested
    return function(data_struct)
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/file_utils.py", line 213, in cached_path
    output_path = ExtractManager(cache_dir=download_config.cache_dir).extract(
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/extract.py", line 46, in extract
    self.extractor.extract(input_path, output_path, extractor_format)
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/extract.py", line 263, in extract
    with FileLock(lock_path):
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/filelock.py", line 399, in __init__
    max_filename_length = os.statvfs(os.path.dirname(lock_file)).f_namemax
FileNotFoundError: [Errno 2] No such file or directory: ''
Exception ignored in: <function BaseFileLock.__del__ at 0x11caeec10>
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/filelock.py", line 328, in __del__
    self.release(force=True)
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/filelock.py", line 303, in release
    with self._thread_lock:
AttributeError: 'UnixFileLock' object has no attribute '_thread_lock'
Extracting data files:   0%|                                                                                | 0/4 [00:00<?, ?it/s]
```

Can you help me please?

## Environment info
- `datasets` version: 2.4.0
- Platform: macOS-12.5.1-x86_64-i386-64bit
- Python version: 3.9.5
- PyArrow version: 9.0.0
- Pandas version: 1.2.4


Can't create a dataset with `float16` features

Something isn't working

Hi @dconathan, thanks for reporting.

We rely on Arrow as a backend, and as far as I know currently support for `float16` in Arrow is not fully implemented in Python (C++), hence the `ArrowNotImplementedError` you get.

See, e.g.: https://arrow.apache.org/docs/status.html?highlight=float16#data-types

Thanks for the link…. didn’t realize arrow didn’t support it yet.  Should it be removed from https://huggingface.co/docs/datasets/v2.4.0/en/package_reference/main_classes#datasets.Value until Arrow supports it?

Yes, you are right: maybe we should either remove it from our docs or add a comment explaining the issue.

The thing is that in Arrow it is partially supported: you can create `float16` values, but you can't cast them from/to other types. And current implementation of `Value` always tries to perform a cast from `float64` to `float16`.

## Describe the bug
I can't create a dataset with `float16` features.

I understand from the traceback that this is a `pyarrow` error, but I don't see anywhere in the `datasets` documentation about how to successfully do this.  Is it actually supported?  I've tried older versions of `pyarrow` as well with the same exact error.

The bug seems to arise from `datasets` casting the values to `double` and then `pyarrow` doesn't know how to convert those back to `float16`... does that sound right?  Is there a way to bypass this since it's not necessary in the `numpy` and `torch` cases?

Thanks!

## Steps to reproduce the bug
All of the following raise the following error with the same exact (as far as I can tell) traceback:
```python
ArrowNotImplementedError: Unsupported cast from double to halffloat using function cast_half_float
```
```python
from datasets import Dataset, Features, Value
Dataset.from_dict({"x": [0.0, 1.0, 2.0]}, features=Features(x=Value("float16")))

import numpy as np
Dataset.from_dict({"x": np.arange(3, dtype=np.float16)}, features=Features(x=Value("float16")))

import torch
Dataset.from_dict({"x": torch.arange(3).to(torch.float16)}, features=Features(x=Value("float16")))
```

## Expected results
A dataset with `float16` features is successfully created.

## Actual results

```python
---------------------------------------------------------------------------
ArrowNotImplementedError                  Traceback (most recent call last)
Cell In [14], line 1
----> 1 Dataset.from_dict({"x": [1.0, 2.0, 3.0]}, features=Features(x=Value("float16")))

File ~/scratch/scratch-env-39/.venv/lib/python3.9/site-packages/datasets/arrow_dataset.py:870, in Dataset.from_dict(cls, mapping, features, info, split)
    865     mapping = features.encode_batch(mapping)
    866 mapping = {
    867     col: OptimizedTypedSequence(data, type=features[col] if features is not None else None, col=col)
    868     for col, data in mapping.items()
    869 }
--> 870 pa_table = InMemoryTable.from_pydict(mapping=mapping)
    871 if info.features is None:
    872     info.features = Features({col: ts.get_inferred_type() for col, ts in mapping.items()})

File ~/scratch/scratch-env-39/.venv/lib/python3.9/site-packages/datasets/table.py:750, in InMemoryTable.from_pydict(cls, *args, **kwargs)
    734 @classmethod
    735 def from_pydict(cls, *args, **kwargs):
    736     """
    737     Construct a Table from Arrow arrays or columns
    738
   (...)
    748         :class:`datasets.table.Table`:
    749     """
--> 750     return cls(pa.Table.from_pydict(*args, **kwargs))

File ~/scratch/scratch-env-39/.venv/lib/python3.9/site-packages/pyarrow/table.pxi:3648, in pyarrow.lib.Table.from_pydict()

File ~/scratch/scratch-env-39/.venv/lib/python3.9/site-packages/pyarrow/table.pxi:5174, in pyarrow.lib._from_pydict()

File ~/scratch/scratch-env-39/.venv/lib/python3.9/site-packages/pyarrow/array.pxi:343, in pyarrow.lib.asarray()

File ~/scratch/scratch-env-39/.venv/lib/python3.9/site-packages/pyarrow/array.pxi:231, in pyarrow.lib.array()

File ~/scratch/scratch-env-39/.venv/lib/python3.9/site-packages/pyarrow/array.pxi:110, in pyarrow.lib._handle_arrow_array_protocol()

File ~/scratch/scratch-env-39/.venv/lib/python3.9/site-packages/datasets/arrow_writer.py:197, in TypedSequence.__arrow_array__(self, type)
    192     # otherwise we can finally use the user's type
    193     elif type is not None:
    194         # We use cast_array_to_feature to support casting to custom types like Audio and Image
    195         # Also, when trying type "string", we don't want to convert integers or floats to "string".
    196         # We only do it if trying_type is False - since this is what the user asks for.
--> 197         out = cast_array_to_feature(out, type, allow_number_to_str=not self.trying_type)
    198     return out
    199 except (TypeError, pa.lib.ArrowInvalid) as e:  # handle type errors and overflows

File ~/scratch/scratch-env-39/.venv/lib/python3.9/site-packages/datasets/table.py:1683, in _wrap_for_chunked_arrays.<locals>.wrapper(array, *args, **kwargs)
   1681     return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
   1682 else:
-> 1683     return func(array, *args, **kwargs)

File ~/scratch/scratch-env-39/.venv/lib/python3.9/site-packages/datasets/table.py:1853, in cast_array_to_feature(array, feature, allow_number_to_str)
   1851     return array_cast(array, get_nested_type(feature), allow_number_to_str=allow_number_to_str)
   1852 elif not isinstance(feature, (Sequence, dict, list, tuple)):
-> 1853     return array_cast(array, feature(), allow_number_to_str=allow_number_to_str)
   1854 raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{feature}")

File ~/scratch/scratch-env-39/.venv/lib/python3.9/site-packages/datasets/table.py:1683, in _wrap_for_chunked_arrays.<locals>.wrapper(array, *args, **kwargs)
   1681     return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
   1682 else:
-> 1683     return func(array, *args, **kwargs)

File ~/scratch/scratch-env-39/.venv/lib/python3.9/site-packages/datasets/table.py:1762, in array_cast(array, pa_type, allow_number_to_str)
   1760     if pa.types.is_null(pa_type) and not pa.types.is_null(array.type):
   1761         raise TypeError(f"Couldn't cast array of type {array.type} to {pa_type}")
-> 1762     return array.cast(pa_type)
   1763 raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{pa_type}")

File ~/scratch/scratch-env-39/.venv/lib/python3.9/site-packages/pyarrow/array.pxi:919, in pyarrow.lib.Array.cast()

File ~/scratch/scratch-env-39/.venv/lib/python3.9/site-packages/pyarrow/compute.py:389, in cast(arr, target_type, safe, options)
    387     else:
    388         options = CastOptions.safe(target_type)
--> 389 return call_function("cast", [arr], options)

File ~/scratch/scratch-env-39/.venv/lib/python3.9/site-packages/pyarrow/_compute.pyx:560, in pyarrow._compute.call_function()

File ~/scratch/scratch-env-39/.venv/lib/python3.9/site-packages/pyarrow/_compute.pyx:355, in pyarrow._compute.Function.call()

File ~/scratch/scratch-env-39/.venv/lib/python3.9/site-packages/pyarrow/error.pxi:144, in pyarrow.lib.pyarrow_internal_check_status()

File ~/scratch/scratch-env-39/.venv/lib/python3.9/site-packages/pyarrow/error.pxi:121, in pyarrow.lib.check_status()

ArrowNotImplementedError: Unsupported cast from double to halffloat using function cast_half_float
```

## Environment info

- `datasets` version: 2.4.0
- Platform: macOS-12.5.1-arm64-arm-64bit
- Python version: 3.9.13
- PyArrow version: 9.0.0
- Pandas version: 1.4.4


Make `pyarrow` optional

enhancement

New feature or request

The whole datasets library is pretty much a wrapper to pyarrow (just take a look at some of the source for a Dataset) https://github.com/huggingface/datasets/blob/51aef08ad7053c0bfe8f9a961207b26df15850d3/src/datasets/arrow_dataset.py#L639

I think removing the pyarrow dependency would involve a complete rewrite / a different library with minimal functionality (datasets-lite ?)

Thanks for the proposal, @KOLANICH. And also thanks for your answer, @dconathan.

Indeed, we are using `pyarrow` as the backend for our datasets, in order to cache them and also allow memory-mapping (using datasets larger than your RAM memory).

One way to avoid using `pyarrow` could be loading the datasets in streaming mode, by passing `streaming=True` to `load_dataset`. This way you basically get a generator for the dataset; nothing is downloaded, nor cached.

Thanks for the info. Could `datasets` then be made optional for `transformers` instead? I used `transformers` only to deal with pretrained models to deploy them (convert to ONNX, and then I use TVM), so I don't really need `pyarrow` and `datasets` by now.

**Is your feature request related to a problem? Please describe.**
Is `pyarrow` really needed for every dataset?

**Describe the solution you'd like**
It is made optional.

**Describe alternatives you've considered**
Likely, no.


Fix missing tags in dataset cards

Fix missing tags in dataset cards.

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896
- #4908
- #4921
- #4931


Update IndicGLUE download links


Providing dataset size

New feature or request

Hi @sashavor, thanks for your suggestion.

Until now we have the CLI command
```
datasets-cli test datasets/<your-dataset-folder> --save_infos --all_configs
```
that generates the `dataset_infos.json` with the size of the downloaded dataset, among other information.

We are currently in the middle of removing those JSON files and putting their information directly in the header of the `README.md` (as YAML tags). Normally, the CLI command should continue working but saving its output to the dataset card instead. See:
- #4926

Additionally, the download size can be inferred by doing HEAD requests to the files to be downloaded. And for files hosted on the hub you can even get the file sizes using the Hub API

Amazing @albertvillanova ! I think just having that information visible in the dataset info (without having to do any requests/additional coding) would be really useful :hugs:

**Is your feature request related to a problem? Please describe.**
Especially for big datasets like [LAION](https://huggingface.co/datasets/laion/laion2B-en/), it's hard to know exactly the downloaded size (because there are many files and you don't have their exact size when downloaded).

**Describe the solution you'd like**
Auto-populating the downloaded dataset size on the dataset page would be really useful, including that of each split (when there are some).

**Describe alternatives you've considered**
People should be adding this to dataset cards, but I don't think that is systematically the case :slightly_smiling_face:

**Additional context**
Mentioned to @lhoestq


Hope to adapt Python3.9 as soon as possible

New feature or request

Hi! `datasets` should work in Python 3.9. What kind of issue have you encountered?

There is this related issue already: https://github.com/huggingface/datasets/issues/4113
And I guess we need a CI job for 3.9 ^^

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context about the feature request here.


Add `fn_kwargs` param to `IterableDataset.map`

_The documentation is not available anymore as the PR was closed or merged._

Add the `fn_kwargs` parameter to `IterableDataset.map`.

("Resolves" https://discuss.huggingface.co/t/how-to-use-large-image-text-datasets-in-hugging-face-hub-without-downloading-for-free/22780/3)


[GH->HF] Part 2: Remove all dataset scripts from github

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4974). All of your documentation changes will be reflected on that endpoint.

So this means metrics will be deleted from this repo in favor of the "evaluate" library? Maybe you guys could just redirect metrics to that library.

We are deprecating the metrics in `datasets` indeed and suggest users to switch to `evaluate` (via a warning message)

We'll keep the current metrics as they are for now, but they'll be completely removed at one point

Now that all the datasets live on the Hub we can remove the /datasets directory that contains all the dataset scripts of this repository

Needs https://github.com/huggingface/datasets/pull/4973 to be merged first
and PR to be enabled on the Hub for non-namespaced datasets


[GH->HF] Load datasets from the Hub

Currently datasets with no namespace (e.g. squad, glue) are loaded from github.

In this PR I changed this logic to use the Hugging Face Hub instead.

This is the first step in removing all the dataset scripts in this repository

related to discussions in https://github.com/huggingface/datasets/pull/4059 (I should have continued from this PR actually)


Fix map batched with torch output

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4972). All of your documentation changes will be reflected on that endpoint.

Reported in https://discuss.huggingface.co/t/typeerror-when-applying-map-after-set-format-type-torch/23067/2

Currently it fails  if one uses batched `map` and the map function returns a torch tensor.

I fixed it for torch, tf, jax and pandas series.


Preserve non-`input_colums` in `Dataset.map` if `input_columns` are specified

Currently, if the `input_columns` list in `Dataset.map` is specified, the columns not in that list are dropped after the `map` transform.

This makes the behavior inconsistent with `IterableDataset.map`.

(It seems this issue was introduced by mistake in https://github.com/huggingface/datasets/pull/2246)

Fix https://github.com/huggingface/datasets/issues/4858


Support streaming nli_tr dataset

Support streaming nli_tr dataset.

This PR removes legacy `codecs.open` and replaces it with `open` that supports passing encoding.

Fix #3186.


Fix data URL and metadata of vivos dataset

After contacting the authors of the VIVOS dataset to report that their data server is down, we have received a reply from Hieu-Thi Luong that their data is now hosted on Zenodo: https://doi.org/10.5281/zenodo.7068130

This PR updates their data URL and some metadata (homepage, citation and license).

Fix #4936.


Support streaming compguesswhat dataset

Support streaming `compguesswhat` dataset.

Fix #3191.


Strip "/" in local dataset path to avoid empty dataset name error


[Apple M1] MemoryError: Cannot allocate write+execute memory for ffi.callback()

bug

Something isn't working

Hi! This seems like a bug in `soundfile`. Could you please open an issue in their repo? `soundfile` works without any issues on my M1, so I'm not sure we can help.

Hi @mariosasko, can you share how you installed `soundfile` on your mac M1?

## Describe the bug
I'm trying to run `cast_column("audio", Audio())` on Apple M1 Pro, but it seems that it doesn't work.

## Steps to reproduce the bug
```python
import datasets

dataset = load_dataset("csv", data_files="./train.csv")["train"]
dataset = dataset.map(lambda x: {"audio": str(DATA_DIR / "audio" / x["audio"])})
dataset = dataset.cast_column("audio", Audio())
dataset[0]
```

## Expected results
```
{'audio': {'bytes': None,
  'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~JOINT_ACCOUNT/602ba55abb1e6d0fbce92065.wav'},
 'english_transcription': 'I would like to set up a joint account with my partner',
 'intent_class': 11,
 'lang_id': 4,
 'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~JOINT_ACCOUNT/602ba55abb1e6d0fbce92065.wav',
 'transcription': 'I would like to set up a joint account with my partner'}
```

## Actual results
````---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
Input In [6], in <cell line: 1>()
----> 1 dataset[0]

File ~/miniconda3/envs/rodan/lib/python3.8/site-packages/datasets/arrow_dataset.py:2165, in Dataset.__getitem__(self, key)
   2163 def __getitem__(self, key):  # noqa: F811
   2164     """Can be used to index columns (by string names) or rows (by integer index or iterable of indices or bools)."""
-> 2165     return self._getitem(
   2166         key,
   2167     )

File ~/miniconda3/envs/rodan/lib/python3.8/site-packages/datasets/arrow_dataset.py:2150, in Dataset._getitem(self, key, decoded, **kwargs)
   2148 formatter = get_formatter(format_type, features=self.features, decoded=decoded, **format_kwargs)
   2149 pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
-> 2150 formatted_output = format_table(
   2151     pa_subtable, key, formatter=formatter, format_columns=format_columns, output_all_columns=output_all_columns
   2152 )
   2153 return formatted_output

File ~/miniconda3/envs/rodan/lib/python3.8/site-packages/datasets/formatting/formatting.py:532, in format_table(table, key, formatter, format_columns, output_all_columns)
    530 python_formatter = PythonFormatter(features=None)
    531 if format_columns is None:
--> 532     return formatter(pa_table, query_type=query_type)
    533 elif query_type == "column":
    534     if key in format_columns:

File ~/miniconda3/envs/rodan/lib/python3.8/site-packages/datasets/formatting/formatting.py:281, in Formatter.__call__(self, pa_table, query_type)
    279 def __call__(self, pa_table: pa.Table, query_type: str) -> Union[RowFormat, ColumnFormat, BatchFormat]:
    280     if query_type == "row":
--> 281         return self.format_row(pa_table)
    282     elif query_type == "column":
    283         return self.format_column(pa_table)

File ~/miniconda3/envs/rodan/lib/python3.8/site-packages/datasets/formatting/formatting.py:312, in PythonFormatter.format_row(self, pa_table)
    310 row = self.python_arrow_extractor().extract_row(pa_table)
    311 if self.decoded:
--> 312     row = self.python_features_decoder.decode_row(row)
    313 return row

File ~/miniconda3/envs/rodan/lib/python3.8/site-packages/datasets/formatting/formatting.py:221, in PythonFeaturesDecoder.decode_row(self, row)
    220 def decode_row(self, row: dict) -> dict:
--> 221     return self.features.decode_example(row) if self.features else row

File ~/miniconda3/envs/rodan/lib/python3.8/site-packages/datasets/features/features.py:1647, in Features.decode_example(self, example, token_per_repo_id)
   1634 def decode_example(self, example: dict, token_per_repo_id: Optional[Dict[str, Union[str, bool, None]]] = None):
   1635     """Decode example with custom feature decoding.
   1636
   1637     Args:
   (...)
   1644         :obj:`dict[str, Any]`
   1645     """
-> 1647     return {
   1648         column_name: decode_nested_example(feature, value, token_per_repo_id=token_per_repo_id)
   1649         if self._column_requires_decoding[column_name]
   1650         else value
   1651         for column_name, (feature, value) in zip_dict(
   1652             {key: value for key, value in self.items() if key in example}, example
   1653         )
   1654     }

File ~/miniconda3/envs/rodan/lib/python3.8/site-packages/datasets/features/features.py:1648, in <dictcomp>(.0)
   1634 def decode_example(self, example: dict, token_per_repo_id: Optional[Dict[str, Union[str, bool, None]]] = None):
   1635     """Decode example with custom feature decoding.
   1636
   1637     Args:
   (...)
   1644         :obj:`dict[str, Any]`
   1645     """
   1647     return {
-> 1648         column_name: decode_nested_example(feature, value, token_per_repo_id=token_per_repo_id)
   1649         if self._column_requires_decoding[column_name]
   1650         else value
   1651         for column_name, (feature, value) in zip_dict(
   1652             {key: value for key, value in self.items() if key in example}, example
   1653         )
   1654     }

File ~/miniconda3/envs/rodan/lib/python3.8/site-packages/datasets/features/features.py:1260, in decode_nested_example(schema, obj, token_per_repo_id)
   1257 # Object with special decoding:
   1258 elif isinstance(schema, (Audio, Image)):
   1259     # we pass the token to read and decode files from private repositories in streaming mode
-> 1260     return schema.decode_example(obj, token_per_repo_id=token_per_repo_id) if obj is not None else None
   1261 return obj

File ~/miniconda3/envs/rodan/lib/python3.8/site-packages/datasets/features/audio.py:156, in Audio.decode_example(self, value, token_per_repo_id)
    154         array, sampling_rate = self._decode_non_mp3_file_like(file)
    155     else:
--> 156         array, sampling_rate = self._decode_non_mp3_path_like(path, token_per_repo_id=token_per_repo_id)
    157 return {"path": path, "array": array, "sampling_rate": sampling_rate}

File ~/miniconda3/envs/rodan/lib/python3.8/site-packages/datasets/features/audio.py:257, in Audio._decode_non_mp3_path_like(self, path, format, token_per_repo_id)
    254     use_auth_token = None
    256 with xopen(path, "rb", use_auth_token=use_auth_token) as f:
--> 257     array, sampling_rate = librosa.load(f, sr=self.sampling_rate, mono=self.mono)
    258 return array, sampling_rate

File ~/miniconda3/envs/rodan/lib/python3.8/site-packages/librosa/util/decorators.py:88, in deprecate_positional_args.<locals>._inner_deprecate_positional_args.<locals>.inner_f(*args, **kwargs)
     86 extra_args = len(args) - len(all_args)
     87 if extra_args <= 0:
---> 88     return f(*args, **kwargs)
     90 # extra_args > 0
     91 args_msg = [
     92     "{}={}".format(name, arg)
     93     for name, arg in zip(kwonly_args[:extra_args], args[-extra_args:])
     94 ]

File ~/miniconda3/envs/rodan/lib/python3.8/site-packages/librosa/core/audio.py:164, in load(path, sr, mono, offset, duration, dtype, res_type)
    161 else:
    162     # Otherwise try soundfile first, and then fall back if necessary
    163     try:
--> 164         y, sr_native = __soundfile_load(path, offset, duration, dtype)
    166     except RuntimeError as exc:
    167         # If soundfile failed, try audioread instead
    168         if isinstance(path, (str, pathlib.PurePath)):

File ~/miniconda3/envs/rodan/lib/python3.8/site-packages/librosa/core/audio.py:195, in __soundfile_load(path, offset, duration, dtype)
    192     context = path
    193 else:
    194     # Otherwise, create the soundfile object
--> 195     context = sf.SoundFile(path)
    197 with context as sf_desc:
    198     sr_native = sf_desc.samplerate

File ~/miniconda3/envs/rodan/lib/python3.8/site-packages/soundfile.py:629, in SoundFile.__init__(self, file, mode, samplerate, channels, subtype, endian, format, closefd)
    626 self._mode = mode
    627 self._info = _create_info_struct(file, mode, samplerate, channels,
    628                                  format, subtype, endian)
--> 629 self._file = self._open(file, mode_int, closefd)
    630 if set(mode).issuperset('r+') and self.seekable():
    631     # Move write position to 0 (like in Python file objects)
    632     self.seek(0)

File ~/miniconda3/envs/rodan/lib/python3.8/site-packages/soundfile.py:1179, in SoundFile._open(self, file, mode_int, closefd)
   1177     file_ptr = _snd.sf_open_fd(file, mode_int, self._info, closefd)
   1178 elif _has_virtual_io_attrs(file, mode_int):
-> 1179     file_ptr = _snd.sf_open_virtual(self._init_virtual_io(file),
   1180                                     mode_int, self._info, _ffi.NULL)
   1181 else:
   1182     raise TypeError("Invalid file: {0!r}".format(self.name))

File ~/miniconda3/envs/rodan/lib/python3.8/site-packages/soundfile.py:1197, in SoundFile._init_virtual_io(self, file)
   1194 def _init_virtual_io(self, file):
   1195     """Initialize callback functions for sf_open_virtual()."""
   1196     @_ffi.callback("sf_vio_get_filelen")
-> 1197     def vio_get_filelen(user_data):
   1198         curr = file.tell()
   1199         file.seek(0, SEEK_END)

MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks
```

## Environment info
- `datasets` version: 2.4.0
- Platform: macOS-12.5.1-arm64-arm-64bit
- Python version: 3.8.13
- PyArrow version: 9.0.0
- Pandas version: 1.4.4


Column of arrays (2D+) are using unreasonably high memory

Something isn't working

note i have tried the same code with `datasets` version 2.4.0, the outcome is the very same as described above.

Seems related to issues #4623 and #4802 so it would appear this issue has been around for a few months.

## Describe the bug
When trying to store `Array2D, Array3D, etc` as column values in a dataset, accessing that column (or creating depending on how you create it, see code below) will cause more than 10 fold of memory usage.

## Steps to reproduce the bug
```python
from datasets import Dataset, Features, Array2D, Array3D
import numpy as np

column_name = "a"
array_shape = (64, 64, 3)

data = np.random.random((10000,) + array_shape)
dataset = Dataset.from_dict({column_name: data}, features=Features({column_name: Array3D(shape=array_shape, dtype="float64")}))
```

the code above will use about 10Gb of RAM while constructing the `dataset` object.

The code below will use roughly the same amount of memory (and time) when trying to actually access the data itself of that column.
```python
from datasets import Dataset
import numpy as np

column_name = "a"
array_shape = (64, 64, 3)

data = np.random.random((10000,) + array_shape)
dataset = Dataset.from_dict({column_name: data})
dataset[column_name]
```

## Expected results
Some memory overhead, but not like as it is now and certainly not an overhead of such runtime that is currently happening.

## Actual results
Enormous memory- and runtime overhead.

## Environment info
- `datasets` version: 2.3.2
- Platform: macOS-12.5.1-arm64-arm-64bit
- Python version: 3.8.13
- PyArrow version: 9.0.0
- Pandas version: 1.4.4


Dataset without script does not support regular JSON data file

Hi @julien-c,

Out of the box, we only support JSON lines (NDJSON) data files, but your data file is a regular JSON file. The reason is we use `pyarrow.json.read_json` and this only supports line-delimited JSON.

### Link

https://huggingface.co/datasets/julien-c/label-studio-my-dogs

### Description

<img width="1115" alt="image" src="https://user-images.githubusercontent.com/326577/189422048-7e9c390f-bea7-4521-a232-43f049ccbd1f.png">

### Owner

Yes


Update setup.py

Before addressing this PR, we should be sure about the issue. See my comment in:
- https://github.com/huggingface/datasets/issues/4961#issuecomment-1243376247

Once we know 2022.8.2 works, I'm closing this PR, as the corresponding issue.

exclude broken version of fsspec. See the [related issue](https://github.com/huggingface/datasets/issues/4961)


fsspec 2022.8.2 breaks xopen in streaming mode

Something isn't working

loading `fsspec==2022.7.1` fixes this issue, setup.py would need to be changed to prevent users from using the latest version of fsspec.

Opened [PR](https://github.com/huggingface/datasets/pull/4962) to address this.

Hi @DCNemesis, thanks for reporting.

That was a temporary issue in `fsspec` releases 2022.8.0 and 2022.8.1. But they fixed it in their patch release 2022.8.2 (and yanked both previous versions). See:
- https://github.com/huggingface/transformers/pull/18846

Are you sure you have version 2022.8.2 installed?
```shell
pip install -U fsspec
```

@albertvillanova I was using a temporary Google Colab instance, but checking it again today it seems it was loading 2022.8.1 rather than 2022.8.2. It's surprising that colab is using the version that was replaced the same day it was released. Testing with 2022.8.2 did work. It appears Colab [will be fixing it](https://github.com/googlecolab/colabtools/issues/3055) on their end too.

Thanks for the additional information.

Once we know 2022.8.2 works, I'm closing this issue. Feel free to reopen it if necessary.

Colab just upgraded their default `fsspec` version to 2022.8.2:
- https://github.com/googlecolab/colabtools/issues/3055#issuecomment-1244019010

## Describe the bug
When fsspec 2022.8.2 is installed in your environment, xopen will prematurely close files, making streaming mode inoperable.

## Steps to reproduce the bug
```python

import datasets

data = datasets.load_dataset('MLCommons/ml_spoken_words', 'id_wav', split='train', streaming=True)

```

## Expected results
Dataset should load as iterator.

## Actual results
```
[/usr/local/lib/python3.7/dist-packages/datasets/load.py](https://localhost:8080/#) in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1737     # Return iterable dataset in case of streaming
   1738     if streaming:
-> 1739         return builder_instance.as_streaming_dataset(split=split)
   1740
   1741     # Some datasets are already processed on the HF google storage

[/usr/local/lib/python3.7/dist-packages/datasets/builder.py](https://localhost:8080/#) in as_streaming_dataset(self, split, base_path)
   1023         )
   1024         self._check_manual_download(dl_manager)
-> 1025         splits_generators = {sg.name: sg for sg in self._split_generators(dl_manager)}
   1026         # By default, return all splits
   1027         if split is None:

[~/.cache/huggingface/modules/datasets_modules/datasets/MLCommons--ml_spoken_words/321ea853cf0a05abb7a2d7efea900692a3d8622af65a2f3ce98adb7800a5d57b/ml_spoken_words.py](https://localhost:8080/#) in _split_generators(self, dl_manager)
    182                 name=datasets.Split.TRAIN,
    183                 gen_kwargs={
--> 184                     "audio_archives": [download_audio(split="train", lang=lang) for lang in self.config.languages],
    185                     "local_audio_archives_paths": [download_extract_audio(split="train", lang=lang) for lang in
    186                                                    self.config.languages] if not dl_manager.is_streaming else None,

[~/.cache/huggingface/modules/datasets_modules/datasets/MLCommons--ml_spoken_words/321ea853cf0a05abb7a2d7efea900692a3d8622af65a2f3ce98adb7800a5d57b/ml_spoken_words.py](https://localhost:8080/#) in <listcomp>(.0)
    182                 name=datasets.Split.TRAIN,
    183                 gen_kwargs={
--> 184                     "audio_archives": [download_audio(split="train", lang=lang) for lang in self.config.languages],
    185                     "local_audio_archives_paths": [download_extract_audio(split="train", lang=lang) for lang in
    186                                                    self.config.languages] if not dl_manager.is_streaming else None,

[~/.cache/huggingface/modules/datasets_modules/datasets/MLCommons--ml_spoken_words/321ea853cf0a05abb7a2d7efea900692a3d8622af65a2f3ce98adb7800a5d57b/ml_spoken_words.py](https://localhost:8080/#) in _download_audio_archives(dl_manager, lang, format, split)
    267 # for streaming case
    268 def _download_audio_archives(dl_manager, lang, format, split):
--> 269     archives_paths = _download_audio_archives_paths(dl_manager, lang, format, split)
    270     return [dl_manager.iter_archive(archive_path) for archive_path in archives_paths]

[~/.cache/huggingface/modules/datasets_modules/datasets/MLCommons--ml_spoken_words/321ea853cf0a05abb7a2d7efea900692a3d8622af65a2f3ce98adb7800a5d57b/ml_spoken_words.py](https://localhost:8080/#) in _download_audio_archives_paths(dl_manager, lang, format, split)
    251     n_files_path = dl_manager.download(n_files_url)
    252
--> 253     with open(n_files_path, "r", encoding="utf-8") as file:
    254         n_files = int(file.read().strip())  # the file contains a number of archives
    255

ValueError: I/O operation on closed file.
```

## Environment info
- `datasets` version: 2.4.0
- Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5


BioASQ  AttributeError: 'BuilderConfig' object has no attribute 'schema'

dataset bug

A bug in a dataset script provided in the library

Following worked:

```
data_dir = "/Users/dlituiev/repos/datasets/bioasq/"
bioasq_task_b = load_dataset("aps/bioasq_task_b", data_dir=data_dir, name="bioasq_9b_source")
```

Would maintainers be open to one of the following:
- automating this with a latest default config (e.g.  `bioasq_9b_source`); how can this be generalized to other datasets?
- providing an actionable error message that lists available `name` values? I only got available `name` values once I've provided something there (`name="aps/bioasq_task_b"`), before it would not even mention that it requires `name` argument

Hi ! In general the list of available configurations is prompted. I think this is an issue with this specific dataset.

Feel free to open a new discussions at https://huggingface.co/datasets/aps/bioasq_task_b/discussions

cc @apsdehal

In particular it sounds like the `BUILDER_CONFIG_CLASS= BigBioConfig ` class attribute is missing and the _info should account for schema being None and raise an error

## Describe the bug
I am trying to load a dataset from drive and running into an error.

## Steps to reproduce the bug
```python
data_dir = "/Users/dlituiev/repos/datasets/bioasq/BioASQ-training9b"
bioasq_task_b = load_dataset("aps/bioasq_task_b", data_dir=data_dir)
```

## Actual results

`AttributeError: 'BuilderConfig' object has no attribute 'schema'`

<details>

```
Using custom data configuration default-a1ca3e05be5abf2f
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Input In [8], in <cell line: 2>()
      1 data_dir = "/Users/dlituiev/repos/datasets/bioasq/BioASQ-training9b"
----> 2 bioasq_task_b = load_dataset("aps/bioasq_task_b", data_dir=data_dir)

File ~/opt/anaconda3/envs/spacy3/lib/python3.10/site-packages/datasets/load.py:1723, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1720 ignore_verifications = ignore_verifications or save_infos
   1722 # Create a dataset builder
-> 1723 builder_instance = load_dataset_builder(
   1724     path=path,
   1725     name=name,
   1726     data_dir=data_dir,
   1727     data_files=data_files,
   1728     cache_dir=cache_dir,
   1729     features=features,
   1730     download_config=download_config,
   1731     download_mode=download_mode,
   1732     revision=revision,
   1733     use_auth_token=use_auth_token,
   1734     **config_kwargs,
   1735 )
   1737 # Return iterable dataset in case of streaming
   1738 if streaming:

File ~/opt/anaconda3/envs/spacy3/lib/python3.10/site-packages/datasets/load.py:1526, in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, use_auth_token, **config_kwargs)
   1523     raise ValueError(error_msg)
   1525 # Instantiate the dataset builder
-> 1526 builder_instance: DatasetBuilder = builder_cls(
   1527     cache_dir=cache_dir,
   1528     config_name=config_name,
   1529     data_dir=data_dir,
   1530     data_files=data_files,
   1531     hash=hash,
   1532     features=features,
   1533     use_auth_token=use_auth_token,
   1534     **builder_kwargs,
   1535     **config_kwargs,
   1536 )
   1538 return builder_instance

File ~/opt/anaconda3/envs/spacy3/lib/python3.10/site-packages/datasets/builder.py:1154, in GeneratorBasedBuilder.__init__(self, writer_batch_size, *args, **kwargs)
   1153 def __init__(self, *args, writer_batch_size=None, **kwargs):
-> 1154     super().__init__(*args, **kwargs)
   1155     # Batch size used by the ArrowWriter
   1156     # It defines the number of samples that are kept in memory before writing them
   1157     # and also the length of the arrow chunks
   1158     # None means that the ArrowWriter will use its default value
   1159     self._writer_batch_size = writer_batch_size or self.DEFAULT_WRITER_BATCH_SIZE

File ~/opt/anaconda3/envs/spacy3/lib/python3.10/site-packages/datasets/builder.py:307, in DatasetBuilder.__init__(self, cache_dir, config_name, hash, base_path, info, features, use_auth_token, repo_id, data_files, data_dir, name, **config_kwargs)
    305 if info is None:
    306     info = self.get_exported_dataset_info()
--> 307     info.update(self._info())
    308     info.builder_name = self.name
    309     info.config_name = self.config.name

File ~/.cache/huggingface/modules/datasets_modules/datasets/aps--bioasq_task_b/3d54b1213f7e8001eef755af92877f9efa44161ee83c2a70d5d649defa95759e/bioasq_task_b.py:477, in BioasqTaskBDataset._info(self)
    474 def _info(self):
    475
    476     # BioASQ Task B source schema
--> 477     if self.config.schema == "source":
    478         features = datasets.Features(
    479             {
    480                 "id": datasets.Value("string"),
   (...)
    504             }
    505         )
    506     # simplified schema for QA tasks

AttributeError: 'BuilderConfig' object has no attribute 'schema'
```

</details>

## Environment info

- `datasets` version: 2.4.0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.10.4
- PyArrow version: 9.0.0
- Pandas version: 1.4.3


Fix data URLs of compguesswhat dataset

_The documentation is not available anymore as the PR was closed or merged._

After we informed the `compguesswhat` dataset authors about an error with their data URLs, they have updated them:
- https://github.com/CompGuessWhat/compguesswhat.github.io/issues/1

This PR updates their data URLs in our loading script.

Related to:
- #3191


ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.4.0/datasets/jsonl/jsonl.py

bug

Something isn't working

I have solved this problem... The extension of the file should be `.json` not `.jsonl`

Hi,
When I use load_dataset from local jsonl files, below error happens, and I type the link into the browser prompting me `404: Not Found`. I download the other `.py` files using the same method and it works. It seems that the server is missing the appropriate file, or it is a problem with the code version.

```
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.3.0/datasets/jsonl/jsonl.py (ConnectionError(MaxRetryError("HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /huggingface/datasets/2.3.0/datasets/jsonl/jsonl.py (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x2b08342004c0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))")))

```


Add `Dataset.from_generator`

I restarted the builder PR job just in case

CI is now green. https://github.com/huggingface/doc-builder/pull/296 explains why it failed.

Add `Dataset.from_generator` to the API to allow creating datasets from data larger than RAM. The implementation relies on a packaged module not exposed in `load_dataset` to tie this method with `datasets`' caching mechanism.

Closes https://github.com/huggingface/datasets/issues/4417


Fix TF tests for 2.10

Fixes #4953


Raise a more precise error when the URL is unreachable in streaming mode

enhancement

New feature or request

See for example:

- https://github.com/huggingface/datasets/issues/3191
- https://github.com/huggingface/datasets/issues/3186

It would help provide clearer information on the Hub and help the dataset maintainer solve the issue by themselves quicker. Currently:

- https://huggingface.co/datasets/compguesswhat

- https://huggingface.co/datasets/nli_tr

cc @albertvillanova


Pin TensorFlow temporarily

Temporarily fix TensorFlow until a permanent solution is found.

Related to:
- #4953


CI test of TensorFlow is failing

bug

Something isn't working

## Describe the bug
The following CI test fails: https://github.com/huggingface/datasets/runs/8246722693?check_suite_focus=true
```
FAILED tests/test_py_utils.py::TempSeedTest::test_tensorflow - AssertionError:
```

Details:
```
_________________________ TempSeedTest.test_tensorflow _________________________
[gw0] linux -- Python 3.7.13 /opt/hostedtoolcache/Python/3.7.13/x64/bin/python

self = <tests.test_py_utils.TempSeedTest testMethod=test_tensorflow>

    @require_tf
    def test_tensorflow(self):
        import tensorflow as tf
        from tensorflow.keras import layers

        def gen_random_output():
            model = layers.Dense(2)
            x = tf.random.uniform((1, 3))
            return model(x).numpy()

        with temp_seed(42, set_tensorflow=True):
            out1 = gen_random_output()
        with temp_seed(42, set_tensorflow=True):
            out2 = gen_random_output()
        out3 = gen_random_output()

>       np.testing.assert_equal(out1, out2)
E       AssertionError:
E       Arrays are not equal
E
E       Mismatched elements: 2 / 2 (100%)
E       Max absolute difference: 0.84619296
E       Max relative difference: 16.083529
E        x: array([[-0.793581,  0.333286]], dtype=float32)
E        y: array([[0.052612, 0.539708]], dtype=float32)

tests/test_py_utils.py:149: AssertionError
```


Add test-datasets CI job

Closing this one since the dataset scripts will be removed in https://github.com/huggingface/datasets/pull/4974

To avoid having too many conflicts in the datasets and metrics dependencies I split the CI into test and test-catalog

test does the test of the core of the `datasets` lib, while test-catalog tests the datasets scripts and metrics scripts

This also makes `pip install -e .[dev]` much smaller for developers

WDYT @albertvillanova ?


Fix license information in qasc dataset card

This PR adds the license information to `qasc` dataset, once reported via GitHub by Tushar Khot, the dataset is licensed under CC BY 4.0:
- https://github.com/allenai/qasc/issues/5


Update Enwik8 broken link and information

The current enwik8 dataset link give a 502 bad gateway error which can be view on https://huggingface.co/datasets/enwik8 (click the dropdown to see the dataset preview, it will show the error). This corrects the links, and json metadata as well as adds a little bit more information about enwik8.


Update enwik8 fixing the broken link

Closing pull request to following contributing guidelines of making a new branch and will make a new pull request


Fix minor typo in error message for missing imports


Try to fix the Windows CI after TF update 2.10

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4947). All of your documentation changes will be reflected on that endpoint.


Introduce regex check when pushing as well

Let me take over this PR if you don't mind

Closes https://github.com/huggingface/datasets/issues/4945 by adding a regex check when pushing to hub.

Let me know if this is helpful and if it's the fix you would have in mind for the issue and I'm happy to contribute tests.


Push to hub can push splits that do not respect the regex

Something isn't working

## Describe the bug

The `push_to_hub` method can push splits that do not respect the regex check that is used for downloads. Therefore, splits may be pushed but never re-used, which can be painful if the split was done after runtime preprocessing.

## Steps to reproduce the bug
```python
>>> from datasets import Dataset, DatasetDict, load_dataset

>>> d = Dataset.from_dict({'x': [1,2,3], 'y': [1,2,3]})
>>> di = DatasetDict()
>>> di['identifier-with-column'] = d

>>> di.push_to_hub('open-source-metrics/test')
Pushing split identifier-with-column to the Hub.
Pushing dataset shards to the dataset hub: 100%|██████████| 1/1 [00:04<00:00,  4.40s/it]
```

Loading it afterwards:
```python
>>> load_dataset('open-source-metrics/test')
Downloading: 100%|██████████| 610/610 [00:00<00:00, 432kB/s]
Using custom data configuration open-source-metrics--test-28b63ec7cde80488
Downloading and preparing dataset None/None (download: 950 bytes, generated: 48 bytes, post-processed: Unknown size, total: 998 bytes) to /home/lysandre/.cache/huggingface/datasets/open-source-metrics___parquet/open-source-metrics--test-28b63ec7cde80488/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec...
Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]
Downloading data: 100%|██████████| 950/950 [00:00<00:00, 1.01MB/s]
Downloading data files: 100%|██████████| 1/1 [00:01<00:00,  1.48s/it]
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 2291.97it/s]
Traceback (most recent call last):
  File "/home/lysandre/.pyenv/versions/3.10.6/lib/python3.10/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "/home/lysandre/Workspaces/python/Metrics/GitHub-Metrics/.env/lib/python3.10/site-packages/datasets/load.py", line 1746, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/lysandre/Workspaces/python/Metrics/GitHub-Metrics/.env/lib/python3.10/site-packages/datasets/builder.py", line 704, in download_and_prepare
    self._download_and_prepare(
  File "/home/lysandre/Workspaces/python/Metrics/GitHub-Metrics/.env/lib/python3.10/site-packages/datasets/builder.py", line 771, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/lysandre/Workspaces/python/Metrics/GitHub-Metrics/.env/lib/python3.10/site-packages/datasets/packaged_modules/parquet/parquet.py", line 48, in _split_generators
    splits.append(datasets.SplitGenerator(name=split_name, gen_kwargs={"files": files}))
  File "<string>", line 5, in __init__
  File "/home/lysandre/Workspaces/python/Metrics/GitHub-Metrics/.env/lib/python3.10/site-packages/datasets/splits.py", line 599, in __post_init__
    NamedSplit(self.name)  # check that it's a valid split name
  File "/home/lysandre/Workspaces/python/Metrics/GitHub-Metrics/.env/lib/python3.10/site-packages/datasets/splits.py", line 346, in __init__
    raise ValueError(f"Split name should match '{_split_re}' but got '{split_name}'.")
ValueError: Split name should match '^\w+(\.\w+)*$' but got 'identifier-with-column'.
```

## Expected results

I would expect `push_to_hub` to stop me in my tracks if trying to upload a split that will not be working afterwards.

## Actual results

See above

## Environment info

- `datasets` version: 2.4.0
- Platform: Linux-5.15.64-1-lts-x86_64-with-glibc2.36
- Python version: 3.10.6
- PyArrow version: 9.0.0
- Pandas version: 1.4.4


larger dataset, larger GPU memory in the training phase? Is that correct?

Something isn't working

does the trainer save it in GPU? sooo curious... how to fix it

It's my bad. didn't limit the input length

    from datasets import set_caching_enabled
    set_caching_enabled(False)
    for ds_name in ["squad","newsqa","nqopen","narrativeqa"]:
            train_ds = load_from_disk("../../../dall/downstream/processedproqa/{}-train.hf".format(ds_name))

        break
    train_ds = concatenate_datasets([train_ds,train_ds,train_ds,train_ds]) #operation 1

   trainer = QuestionAnsweringTrainer( #huggingface trainer
        model=model,
        args=training_args,
        train_dataset=train_ds,
        eval_dataset= None,
        eval_examples=None,
        answer_column_name=answer_column,
        dataset_name="squad",
        tokenizer=tokenizer,
        data_collator=data_collator,
        compute_metrics=compute_metrics if training_args.predict_with_generate else None,
    )

with operation 1, the GPU memory increases from 16G to 23G


Add splits to MBPP dataset

Hi @cwarny ! Thanks for adding the correct splits :)

You can fix the CI error by running `make style` - this should reformat the dataset script

This PR addresses https://github.com/huggingface/datasets/issues/4795


Trec Dataset has incorrect labels

Something isn't working

Thanks for reporting, @wmpauli.

Indeed we recently fixed this issue:
- #4801

The fix will be accessible after our next library release. In the meantime, you can have it by passing `revision="main"` to `load_dataset`.

## Describe the bug
Both coarse and fine labels seem to be out of line.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = "trec"
raw_datasets = load_dataset(dataset)
df = pd.DataFrame(raw_datasets["test"])
df.head()
```

## Expected results
text (string) | coarse_label (class label) | fine_label (class label)
-- | -- | --
How far is it from Denver to Aspen ? | 5  	(NUM) | 40  	(NUM:dist)
What county is Modesto , California in ? | 4  	(LOC) | 32  	(LOC:city)
Who was Galileo ? | 3  	(HUM) | 31  	(HUM:desc)
What is an atom ? | 2  	(DESC) | 24  	(DESC:def)
When did Hawaii become a state ? | 5  	(NUM) | 39  	(NUM:date)

## Actual results
  index | label-coarse  |label-fine                  |                    text
-- |-- | -- | --
0    |         4     |     40     | How far is it from Denver to Aspen ?
1      |       5        |  21  | What county is Modesto , California in ?
2        |     3        |  12        |                 Who was Galileo ?
3         |    0       |    7         |                What is an atom ?
4         |    4       |    8       |   When did Hawaii become a state ?

## Environment info

- `datasets` version: 2.4.0
- Platform: Linux-5.4.0-1086-azure-x86_64-with-glibc2.27
- Python version: 3.9.13
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Add Papers with Code ID to scifact dataset

This PR:
- adds Papers with Code ID
- forces sync between GitHub and Hub, which previously failed due to Hub validation error of the license tag: https://github.com/huggingface/datasets/runs/8200223631?check_suite_focus=true


Fix multilinguality tag and missing sections in xquad_r dataset card

This PR fixes issue reported on the Hub:
- Label as multilingual: https://huggingface.co/datasets/xquad_r/discussions/1


Fix NonMatchingChecksumError in adv_glue dataset

Fix issue reported on the Hub: https://huggingface.co/datasets/adv_glue/discussions/1


Remove main branch rename notice

We added a notice in README.md to show that we renamed the master branch to main, but we can remove it now (it's been 2 months)

I also unpinned the github issue about the branch renaming


Remove deprecated identical_ok

`huggingface-hub` says that the `identical_ok` argument of `HfApi.upload_file` is now deprecated, and will be removed soon. It even has no effect at the moment when it's passed:

```python
Args:
...
    identical_ok (`bool`, *optional*, defaults to `True`):
        Deprecated: will be removed in 0.11.0.
        Changing this value has no effect.
...
```

There was only one occurence of `identical_ok=False` but it's maybe not worth adding a check ti verify if the files were the same.

cc @mariosasko


vivos (Vietnamese speech corpus) dataset not accessible

If you need an example of a small audio datasets, I just created few hours ago a speech dataset with only 300MB of compressed audio files https://huggingface.co/datasets/indonesian-nlp/librivox-indonesia. It works also with streaming (@albertvillanova helped me adding this functionality) :-)

@cahya-wirawan omg this is awesome!! thank you!

We have contacted the authors to ask them.

## Describe the bug
VIVOS data is not accessible anymore, neither of these links work (at least from France):
* https://ailab.hcmus.edu.vn/assets/vivos.tar.gz (data)
* https://ailab.hcmus.edu.vn/vivos (dataset page)

Therefore `load_dataset` doesn't work.

## Steps to reproduce the bug
```python
ds = load_dataset("vivos")
```

## Expected results
dataset loaded

## Actual results
```
ConnectionError: Couldn't reach https://ailab.hcmus.edu.vn/assets/vivos.tar.gz (ConnectionError(MaxRetryError("HTTPSConnectionPool(host='ailab.hcmus.edu.vn', port=443): Max retries exceeded with url: /assets/vivos.tar.gz (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9d8a27d190>: Failed to establish a new connection: [Errno -5] No address associated with hostname'))")))
```

Will try to contact the authors, as we wanted to use Vivos as an example in documentation on how to create scripts for audio datasets (https://github.com/huggingface/datasets/pull/4872), because it's small and straightforward and uses tar archives.


Dataset Viewer issue for ubuntu_dialogs_corpus

dataset-viewer

Related to the dataset viewer on huggingface.co

The dataset maintainers (https://huggingface.co/datasets/ubuntu_dialogs_corpus) decided to forbid the dataset from being downloaded automatically (https://huggingface.co/docs/datasets/v2.4.0/en/loading#manual-download), and the dataset viewer respects this.
We will try to improve the error display though. Thanks for reporting.

### Link

_No response_

### Description

_No response_

### Owner

_No response_


Dataset Viewer issue for indonesian-nlp/librivox-indonesia

The error is not related to the dataset viewer. I'm having a look...

Thanks @albertvillanova for checking the issue. Actually, I can use the dataset like following:
```
>>> from datasets import load_dataset
>>> ds=load_dataset("indonesian-nlp/librivox-indonesia")
No config specified, defaulting to: librivox-indonesia/all
Reusing dataset librivox-indonesia (/root/.cache/huggingface/datasets/indonesian-nlp___librivox-indonesia/all/1.0.0/9a934a42bfb53dc103003d191618443b8a786bea2bd7bb0bc2d9454b8494521e)
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 500.87it/s]
>>> ds
DatasetDict({
    train: Dataset({
        features: ['path', 'language', 'reader', 'sentence', 'audio'],
        num_rows: 7815
    })
})
>>> ds["train"][0]
{'path': '/root/.cache/huggingface/datasets/downloads/extracted/c8ead52370fa28feb64643ea9d05cd7d820192dc8a1700d665ec45ec7624f5a3/librivox-indonesia/sundanese/universal-declaration-of-human-rights/human_rights_un_sun_brc_0000.mp3', 'language': 'sun', 'reader': '3174', 'sentence': 'pernyataan umum ngeunaan hak hak asasi manusa sakabeh manusa', 'audio': {'path': '/root/.cache/huggingface/datasets/downloads/extracted/c8ead52370fa28feb64643ea9d05cd7d820192dc8a1700d665ec45ec7624f5a3/librivox-indonesia/sundanese/universal-declaration-of-human-rights/human_rights_un_sun_brc_0000.mp3', 'array': array([ 0.        ,  0.        ,  0.        , ..., -0.02419001,
       -0.01957154, -0.01502833], dtype=float32), 'sampling_rate': 44100}}

```
It would be just nice if I also can see it using dataset viewer.

Yes, the issue arises when streaming (that is used by the viewer): your script does not support streaming and to support it in this case there are some subtleties that we are explaining better in our docs in a work-in progress pull request:
- #4872

Just note that when streaming, `local_extracted_archive` is None, and this code line generates the error:
```python
filepath = local_extracted_archive + "/librivox-indonesia/audio_transcription.csv"
```

For a proper implementation, you could have a look at: https://huggingface.co/datasets/common_voice/blob/main/common_voice.py

You can test your script locally by passing `streaming=True` to `load_dataset`:
```python
ds = load_dataset("indonesian-nlp/librivox-indonesia", split="train", streaming=True); item = next(iter(ds)); item
```

Great, I will have a look and update the script. Thanks.

Hi @albertvillanova , I just add the streaming functionality and it works in the first try :-) Thanks a lot!

### Link

https://huggingface.co/datasets/indonesian-nlp/librivox-indonesia

### Description

I created a new speech dataset https://huggingface.co/datasets/indonesian-nlp/librivox-indonesia, but the dataset preview doesn't work with following error message:
```
Server error
Status code:   400
Exception:     TypeError
Message:       unsupported operand type(s) for +: 'NoneType' and 'str'
```
Please help, I am not sure what the problem here is. Thanks a lot.

### Owner

Yes


Dataset/DatasetDict.filter() cannot have `batched=True` due to `mask` (numpy array?) being non-iterable.

bug

Something isn't working

Hi ! When `batched=True`, you filter function must take a batch as input, and return a list of booleans.

In your case, something like
```python
from datasets import load_dataset

ds_mc4_ja = load_dataset("mc4", "ja")  # This will take 6+ hours... perhaps test it with a toy dataset instead?
ds_mc4_ja_2020 = ds_mc4_ja.filter(
  lambda batch: [timestamp[:4] == "2020" for timestamp in batch["timestamp"]],
  batched=True,
)
```

Let me know if it helps !

> Hi ! When `batched=True`, you filter function must take a batch as input, and return a list of booleans.
> [...]
> Let me know if it helps !

Hi @lhoestq,

Ah, my bad, I totally forgot that part...
Sorry for the trouble and thank you for the kind help!

## Describe the bug
`Dataset/DatasetDict.filter()` cannot have `batched=True` due to `mask` (numpy array?) being non-iterable.

## Steps to reproduce the bug
(In a python 3.7.12 env, I've tried 2.4.0 and 2.3.2 with both `pyarraw==9.0.0` and `pyarrow==8.0.0`.)

```python
from datasets import load_dataset

ds_mc4_ja = load_dataset("mc4", "ja")  # This will take 6+ hours... perhaps test it with a toy dataset instead?
ds_mc4_ja_2020 = ds_mc4_ja.filter(
  lambda example: example["timestamp"][:4] == "2020",
  batched=True,
)
```

## Expected results
No error

## Actual results
```python
---------------------------------------------------------------------------
RemoteTraceback                           Traceback (most recent call last)
RemoteTraceback:
"""
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/multiprocess/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 557, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 524, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/datasets/fingerprint.py", line 480, in wrapper
    out = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 2779, in _map_single
    offset=offset,
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 2655, in apply_function_on_filtered_inputs
    processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 2347, in decorated
    result = f(decorated_item, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 4946, in get_indices_from_mask_function
    indices_array = [i for i, to_keep in zip(indices, mask) if to_keep]
TypeError: zip argument #2 must support iteration
"""

The above exception was the direct cause of the following exception:

TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_51348/2345782281.py in <module>
      7     batched=True,
      8     # batch_size=10_000,
----> 9     num_proc=111,
     10 )
     11 # ds_mc4_ja_clean_2020 = ds_mc4_ja.filter(

/opt/conda/lib/python3.7/site-packages/datasets/dataset_dict.py in filter(self, function, with_indices, input_columns, batched, batch_size, keep_in_memory, load_from_cache_file, cache_file_names, writer_batch_size, fn_kwargs, num_proc, desc)
    878                     desc=desc,
    879                 )
--> 880                 for k, dataset in self.items()
    881             }
    882         )

/opt/conda/lib/python3.7/site-packages/datasets/dataset_dict.py in <dictcomp>(.0)
    878                     desc=desc,
    879                 )
--> 880                 for k, dataset in self.items()
    881             }
    882         )

/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    522         }
    523         # apply actual function
--> 524         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    525         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    526         # re-apply format to the output

/opt/conda/lib/python3.7/site-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    478             # Call actual function
    479
--> 480             out = func(self, *args, **kwargs)
    481
    482             # Update fingerprint of in-place transforms + update in-place history of transforms

/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py in filter(self, function, with_indices, input_columns, batched, batch_size, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   2920             new_fingerprint=new_fingerprint,
   2921             input_columns=input_columns,
-> 2922             desc=desc,
   2923         )
   2924         new_dataset = copy.deepcopy(self)

/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py in map(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   2498
   2499                     for index, async_result in results.items():
-> 2500                         transformed_shards[index] = async_result.get()
   2501
   2502             assert (

/opt/conda/lib/python3.7/site-packages/multiprocess/pool.py in get(self, timeout)
    655             return self._value
    656         else:
--> 657             raise self._value
    658
    659     def _set(self, i, obj):

TypeError: zip argument #2 must support iteration
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: Linux-4.19.0-21-cloud-amd64-x86_64-with-debian-10.12
- Python version: 3.7.12
- PyArrow version: 9.0.0
- Pandas version: 1.3.5

(I've tried 2.4.0 and 2.3.2 with both `pyarraw==9.0.0` and `pyarrow==8.0.0`.)


Dataset Viewer issue for bigscience-biomedical/biosses

Possibly not related to the dataset viewer in itself. cc @huggingface/datasets.

In particular, I think that the import of bigbiohub is not working here: https://huggingface.co/datasets/bigscience-biomedical/biosses/blob/main/biosses.py#L29 (requires a relative path?)

```python
>>> from datasets import get_dataset_config_names
>>> get_dataset_config_names('bigscience-biomedical/biosses')
Downloading builder script: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8.00k/8.00k [00:00<00:00, 7.47MB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 289, in get_dataset_config_names
    dataset_module = dataset_module_factory(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1247, in dataset_module_factory
    raise e1 from None
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1220, in dataset_module_factory
    return HubDatasetModuleFactoryWithScript(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 931, in get_module
    local_imports = _download_additional_modules(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 215, in _download_additional_modules
    raise ImportError(
ImportError: To be able to use bigscience-biomedical/biosses, you need to install the following dependency: bigbiohub.
Please install it using 'pip install bigbiohub' for instance'
```

Opened a PR here to (hopefully) fix the dataset script: https://huggingface.co/datasets/bigscience-biomedical/biosses/discussions/1/files

thanks for taking a look @severo . agree this isn't related to dataset viewer (sorry just clicked on the auto issue creator). also thanks @lhoestq , I see the format to use for relative imports. was a bit confused b/c it seems to be working here

https://huggingface.co/datasets/bigscience-biomedical/scitail/blob/main/scitail.py#L31

I'll try this PR a see what happens.

closing as I think the issue is relative imports and attempting to read json files directly in the repo (thanks again @lhoestq )

### Link

https://huggingface.co/datasets/bigscience-biomedical/biosses

### Description

I've just been working on adding the dataset loader script to this dataset and working with the relative imports. I'm not sure how to interpret the error below (show where the dataset preview used to be) .
```
Status code:   400
Exception:     ModuleNotFoundError
Message:       No module named 'datasets_modules.datasets.bigscience-biomedical--biosses.ddbd5893bf6c2f4db06f407665eaeac619520ba41f69d94ead28f7cc5b674056.bigbiohub'
```

### Owner

Yes


Fix missing tags in dataset cards

_The documentation is not available anymore as the PR was closed or merged._

Fix missing tags in dataset cards.

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896
- #4908
- #4921


Add cc-by-nc-2.0 to list of licenses

this list needs to be kept in sync with the ones in moon-landing and hub-docs :)

@julien-c don't you think it might be better to a have a single file (source of truth) in one of the repos and then use it in every other repo, instead of having 3 copies of the same file that must be kept in sync?

Also note that the licenses we are adding were all already present in our previous `licenses.json` file: are we regenerating it, step by step? Why don't we use a file with ALL the licenses we previously had in the list?

Licenses added:
- #4887
- #4930

Previous `licenses.json` file:
- https://github.com/huggingface/datasets/blob/b7612754928e0fd43b9e3c3becb906ec280ff5d4/src/datasets/utils/resources/licenses.json
- removed in this commit: https://github.com/huggingface/datasets/pull/4613/commits/9f7725412dac1089b3e057f9e3fcf39cc222bc26

Let me know what you think and I can take care of this.

> Let me know what you think and I can take care of this.

What I think is that we shouldn't add licenses that are just used in a couple of datasets, and just use `license_details` for this.

> don't you think it might be better to a have a single file (source of truth) in one of the repos and then use it in every other repo, instead of having 3 copies of the same file that must be kept in sync?

Yes, in my opinion we can just delete this file from `datasets`, the validation is happening hub-side anyways now?

Feel free to delete the license list in `datasets` @albertvillanova ;)

Also FYI in #4926 I also removed all the validation steps anyway (language, license, types etc.)

This PR adds the `cc-by-nc-2.0` to the list of licenses because it is required by `scifact` dataset: https://github.com/allenai/scifact/blob/master/LICENSE.md


Fixes a typo in loading documentation

As show in the [documentation page](https://huggingface.co/docs/datasets/loading) here the `"tr"in` should be `"train`.

![image](https://user-images.githubusercontent.com/7144772/188390445-e1f04d54-e3e3-4762-8686-63ecbe4087e5.png)


Add ability to read-write to SQL databases.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4928). All of your documentation changes will be reflected on that endpoint.

Ah CI runs with `pandas=1.3.5` which doesn't return the number of row inserted.

@lhoestq I'm getting error in integration tests, not sure if it's related to my PR. Any help would be appreciated :)

```
if not self._is_valid_token(token):
>               raise ValueError("Invalid token passed!")
E               ValueError: Invalid token passed!
```

I just relaunched the tests, it should be fixed now

Thanks a lot for working on this!

I have some concerns with the current design:
* Besides SQLite, the loader should also work with the other engines supported by SQLAlchemy. (A better name for it in the current state would be `sqlite` :))
* It should support arbitrary queries/table names - only the latter currently works.
* Exposing this loader as a packaged builder (`load_dataset("sql", ...)`) is not a good idea for the following reasons:
    * Considering the scenario where a table with the same name is present in multiple files is very unlikely, the data files resolution is not needed here. And if we remove that, what the name of the default split should be? "train"?
    * `load_dataset("sql", ...)` also implies that streaming should work, but that's not the case.  And I don't think we can change that, considering how hard it is to make SQLite files streamable.

All this makes me think we shouldn't expose this builder as a packaged module and, instead, limit the API to `Dataset.from_sql`/`Dataset.to_sql` (with the signatures matching the ones in pandas as much as possible; regarding this, note that SQLAlchemy connections are not hashable/picklable, which is required for caching, but I think it's OK only to allow URI strings as connections to bypass that (Dask has the same limitation).

WDYT?

Hi @mariosasko thank you for your review.

I agree that `load_dataset('sql',...)` is a bit weird and I would be happy to remove it. To be honest, I only added it when I saw that it was the preferred way in `loading.mdx`.

I agree that the `SELECT` should be a parameters as well. I'll add it.

So far, only `Dataset.to_sql` explicitly supports any SQLAlchemy Connexion, I'm pretty sure that `Dataset.from_sql` would work with a Connexion as well, but it would break the typing from the parent class which is `path_or_paths: NestedDataStructureLike[PathLike]`. I would prefer not to break this API Contract.

I will have time to work on this over the weekend. Please let me know what you think if I do the following:
* Remove `load_dataset('sql', ...)` and edit the documentation to use `to_sql, from_sql`.
* Tentatively make `Dataset.from_sql` typing work with SQLAlchemy Connexion.
* Add support for custom queries (Default would be `SELECT * FROM {table_name}`).

Cheers!

Perhaps after we merge https://github.com/huggingface/datasets/pull/4957 (**Done!**), you can subclass `AbstractDatasetInputStream` instead of `AbstractDatasetReader` to not break the contract with the connection object. Also, let's avoid having the default value for the query/table (you can set it to `None` in the builder and raise an error in the builder config's `__post_init__` if it's not provided). Other than that, sounds good!

Fixes #3094

Add ability to read/write to SQLite files and also read from any SQL database supported by SQLAlchemy.

I didn't add SQLAlchemy as a dependence as it is fairly big and it remains optional.

I also recorded a Loom to showcase the feature.

https://www.loom.com/share/f0e602c2de8a46f58bca4b43333d541f


fix BLEU metric card

I've fixed some typos in BLEU metric card.


Dataset infos in yaml

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4926). All of your documentation changes will be reflected on that endpoint.

Alright this is ready for review :)
I mostly would like your opinion on the YAML structure and what we can do in the docs (IMO we can add the docs about those fields in the Hub docs). Other than that let me know if the changes in info.py and features.py look good to you

To simplify the addition of new datasets, we'd like to have the dataset infos in the YAML and deprecate the dataset_infos.json file. YAML is readable and easy to edit, and the YAML metadata of the readme already contain dataset metadata so we would have everything in one place.

To be more specific, I moved these fields from DatasetInfo to the YAML:
- config_name (if there are several configs)
- download_size
- dataset_size
- features
- splits

Here is what I ended up with for `squad`:
```yaml
dataset_infos:
  features:
  - name: id
    dtype: string
  - name: title
    dtype: string
  - name: context
    dtype: string
  - name: question
    dtype: string
  - name: answers
    sequence:
    - name: text
      dtype: string
    - name: answer_start
      dtype: int32
  splits:
  - name: train
    num_bytes: 79346360
    num_examples: 87599
  - name: validation
    num_bytes: 10473040
    num_examples: 10570
  download_size: 35142551
  dataset_size: 89819400
```

and it can be a list if there are several configs

I already did the change for `conll2000` and `crime_and_punish` as an example.

## Implementation details

### Load/Read

This is done via `DatasetInfoDict.write_to_directory/from_directory`

I had to implement a custom the YAML export logic for `SplitDict`, `Version` and `Features`.
The first two are trivial, but the logic for `Features` is more complicated, because I added a simplification step (or the YAML would be too long and less readable): it's just a formatting step to remove unnecessary nesting of YAML data.

### Other changes

I had to update the DatasetModule factories to also download the README.md alongside the dataset scripts/data files, and not just the dataset_infos.json

## YAML validation

I removed the old validation code that was in metadata.py, now we can just use the Hub YAML validation

## Datasets-cli

The `datasets-cli test --save_infos` command now creates a README.md file with the dataset_infos in it, instead of a datasets_infos.json file

## Backward compatibility

`dataset_infos.json` files are still supported and loaded if they exist to have full backward compatibility.
Though I removed the unnecessary keys when the value is the default (like all the `id: null` from the Value feature types) to make them easier to read.

## TODO

- [x] add comments
- [x] tests
- [ ] document the new YAML fields (to be done in the Hub docs)
- [x] try to reload the new dataset_infos.json file content with an old version of `datasets`

## EDITS

- removed "config_name" when there's only one config
- removed "version" for now (?), because it's not useful in general

Fix https://github.com/huggingface/datasets/issues/4876


Add note about loading image / audio files to docs

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4925). All of your documentation changes will be reflected on that endpoint.

Thanks for the feedback @polinaeterna ! I've reworded the docs a bit to integrate your comments and this should be ready for another review :)

> I've just realized that there is another PR about audio documentation open: #4872
> and there the more detailed description on how to use `audiofolder` is moved to another section ("Create an audio dataset")

Ah yes, let's add a comment to #4872 - that will be simpler than the alternatives :)

@polinaeterna @lhoestq What do you think about adding support for the metadata format from Kaggle (one metadata file for each split with the name equal to the split name) to ImageFolder/AudioFolder? I also think we can relax some requirements a bit by:
* allowing `filename` as the name of the main metadata column (currently, only `file_path` is allowed)
* not requiring that the features of all the given metadata files are equal. Instead, we can have a soft check by using `_check_if_features_can_be_aligned` + `_align_features`. The rationale is that train/val metadata often has extra columns compared to test metadata.

These changes would allow us to load the Kaggle dataset linked in the forum thread without any "interventions".

PS: this metadata format for ImageFolder was also proposed by @abhishekkrthakur initially.

Can you give more details about the Kaggle format ? I'm down to discuss it in a separate issue if you don't mind.

> allowing filename as the name of the main metadata column (currently, only file_path is allowed)

`filename` refers to the name of the file, so there's no logic about relative path or directories. If I recall correctly this is what we're doing right now so why not

> not requiring that the features of all the given metadata files are equal. Instead, we can have a soft check by using _check_if_features_can_be_aligned + _align_features. The rationale is that train/val metadata often has extra columns compared to test metadata.

+1 and we can set to None the missing features

I'm not sure if this is worth opening a new issue :).

What I mean by the Kaggle format is the structure like this one (the name of a metadata file is equal to the directory it "references"):
```
- train
    - img1.jpeg
    - img2.jpeg
    - ...
- test
    - img1.jpeg
    - img2.jpeg
    - ...
- train.csv
- test.csv
```

This PR adds a small note about how to load image / audio datasets that have multiple splits in their dataset structure.

Related forum thread: https://discuss.huggingface.co/t/loading-train-and-test-splits-with-audiofolder/22447

cc @NielsRogge


Concatenate_datasets loads everything into RAM

Something isn't working

## Describe the bug
When loading the datasets seperately and saving them on disk, I want to concatenate them. But `concatenate_datasets` is filling up my RAM and the process gets killed. Is there a way to prevent this from happening or is this intended behaviour? Thanks in advance

## Steps to reproduce the bug
```python
gcs = gcsfs.GCSFileSystem(project='project')
datasets = [load_from_disk(f'path/to/slice/of/data/{i}', fs=gcs, keep_in_memory=False) for i in range(10)]

dataset = concatenate_datasets(datasets)
```

## Expected results
A concatenated dataset which is stored on my disk.

## Actual results
Concatenated dataset gets loaded into RAM and overflows it which gets the process killed.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: Linux-4.19.0-21-cloud-amd64-x86_64-with-glibc2.10
- Python version: 3.8.13
- PyArrow version: 8.0.1
- Pandas version: 1.4.3


WIP: decode mp3 with librosa if torchaudio is > 0.12 as a temporary workaround

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4923). All of your documentation changes will be reflected on that endpoint.

Thanks ! Should we still support torchaudio>0.12 if it works ? And if it doesn't we can explain that downgrading is the right solution, or alternatively use librosa

@lhoestq

> Should we still support torchaudio>0.12 if it works ? And if it doesn't we can explain that downgrading is the right solution, or alternatively use librosa

I'm not sure here, because from the one hand, if `torchaudio` works - it works 60 times faster then `librosa`.
But from the other hand, we will get inconsistent behavior (=different results of decoding) for users of `torchaudio>=0.12`.
I'd better go for using `librosa` only to avoid inconsistency then. wdyt?

It seems a bit too constraining to not allow users who have a working torchaudio 0.12 setup to not use it.

If the issue is about avoiding silent errors if the decoding changes, maybe we can log which back-end is used ? It can even be a warning with performance suggestions ("you're using librosa but torchaudio 0.xx is recommended").

Note that users can still have a requirements.txt or whatever in their projects if they really want full reproducibility (and it's the bare minimum imo)

There are multiple possible back-ends so it's maybe not reasonable to only allow one back-end, especially since each back-end has installation constrains and there's no "best" back-end.

`torchaudio>0.12` fails with decoding mp3 files if `ffmpeg<4`. currently we ask users to downgrade torchaudio, but sometimes it's not possible as torchaudio version is binded to torch version. as a temporary workaround we can decode mp3 with librosa (though it 60 times slower, at least it works)

another option would be to ask users to install the required version of `ffmpeg`, but is non-trivial on colab: it's not in apt packages in ubuntu 18 and `conda` is not preinstalled (with `conda` it would be easily installable)

- [x] decode with torchaudio anyway if the version of ffmpeg is correct? it's 60 times faster
- [x] tests
- [x] DO NOT FORGET to get back all the tests

see https://github.com/huggingface/datasets/issues/4776 and https://github.com/huggingface/datasets/issues/3663#issuecomment-1225797165 (there is a Colab notebook to reproduce the error)


I/O error on Google Colab in streaming mode

Something isn't working

## Describe the bug
When trying to load a streaming dataset in Google Colab the loading fails with an I/O error

## Steps to reproduce the bug
```python
import datasets
from datasets import load_dataset
hf_ds = load_dataset(path='wmt19', name='cs-en', streaming=True, split=datasets.Split.VALIDATION)
list(hf_ds.take(5))
```

## Expected results
It should load five data points

## Actual results
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-13-7b5b8b1e7e58>](https://localhost:8080/#) in <module>
      2 from datasets import load_dataset
      3 hf_ds = load_dataset(path='wmt19', name='cs-en', streaming=True, split=datasets.Split.VALIDATION)
----> 4 list(hf_ds.take(5))

6 frames
[/usr/local/lib/python3.7/dist-packages/datasets/iterable_dataset.py](https://localhost:8080/#) in __iter__(self)
    716
    717     def __iter__(self):
--> 718         for key, example in self._iter():
    719             if self.features:
    720                 # `IterableDataset` automatically fills missing columns with None.

[/usr/local/lib/python3.7/dist-packages/datasets/iterable_dataset.py](https://localhost:8080/#) in _iter(self)
    706         else:
    707             ex_iterable = self._ex_iterable
--> 708         yield from ex_iterable
    709
    710     def _iter_shard(self, shard_idx: int):

[/usr/local/lib/python3.7/dist-packages/datasets/iterable_dataset.py](https://localhost:8080/#) in __iter__(self)
    582
    583     def __iter__(self):
--> 584         yield from islice(self.ex_iterable, self.n)
    585
    586     def shuffle_data_sources(self, generator: np.random.Generator) -> "TakeExamplesIterable":

[/usr/local/lib/python3.7/dist-packages/datasets/iterable_dataset.py](https://localhost:8080/#) in __iter__(self)
    110
    111     def __iter__(self):
--> 112         yield from self.generate_examples_fn(**self.kwargs)
    113
    114     def shuffle_data_sources(self, generator: np.random.Generator) -> "ExamplesIterable":

[~/.cache/huggingface/modules/datasets_modules/datasets/wmt19/aeadcbe9f1cbf9969e603239d33d3e43670cf250c1158edf74f5f6e74d4f21d0/wmt_utils.py](https://localhost:8080/#) in _generate_examples(self, split_subsets, extraction_map, with_translation)
    845                 raise ValueError("Invalid number of files: %d" % len(files))
    846
--> 847             for sub_key, ex in sub_generator(*sub_generator_args):
    848                 if not all(ex.values()):
    849                     continue

[~/.cache/huggingface/modules/datasets_modules/datasets/wmt19/aeadcbe9f1cbf9969e603239d33d3e43670cf250c1158edf74f5f6e74d4f21d0/wmt_utils.py](https://localhost:8080/#) in _parse_parallel_sentences(f1, f2, filename1, filename2)
    923         l2_sentences, l2 = parse_file(f2_i, filename2)
    924
--> 925         for line_id, (s1, s2) in enumerate(zip(l1_sentences, l2_sentences)):
    926             key = f"{f_id}/{line_id}"
    927             yield key, {l1: s1, l2: s2}

[~/.cache/huggingface/modules/datasets_modules/datasets/wmt19/aeadcbe9f1cbf9969e603239d33d3e43670cf250c1158edf74f5f6e74d4f21d0/wmt_utils.py](https://localhost:8080/#) in gen()
    895
    896         def gen():
--> 897             with open(path, encoding="utf-8") as f:
    898                 for line in f:
    899                     seg_match = re.match(seg_re, line)

ValueError: I/O operation on closed file.
```

## Environment info
Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 2.4.0
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 9.0.0. (the same error happened with PyArrow version 6.0.0)
- Pandas version: 1.3.5


Fix missing tags in dataset cards

_The documentation is not available anymore as the PR was closed or merged._

Fix missing tags in dataset cards.

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896
- #4908


Something isn't working

Hi @DataNoob0723,

Under the hood, we use `pandas` to load CSV/TSV files. Therefore, you should use "csv" and pass `sep="\t"`, as explained in our docs: https://huggingface.co/docs/datasets/v2.4.0/en/package_reference/loading_methods#from-files
```python
ds = load_dataset('csv', sep="\t", data_files=data_files)
```

## Describe the bug
Unable to load local tsv files through load_dataset method.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
data_files = {
    'train': 'train.tsv',
    'test': 'test.tsv'
}
raw_datasets = load_dataset('tsv', data_files=data_files)

## Expected results
I am pretty sure the data files exist in the current directory. The above code should load them as Datasets, but threw exceptions.

## Actual results
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
[<ipython-input-9-24207899c1af>](https://localhost:8080/#) in <module>
----> 1 raw_datasets = load_dataset('tsv', data_files='train.tsv')

2 frames
[/usr/local/lib/python3.7/dist-packages/datasets/load.py](https://localhost:8080/#) in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1244                         f"Couldn't find a dataset script at {relative_to_absolute_path(combined_path)} or any data file in the same directory. "
   1245                         f"Couldn't find '{path}' on the Hugging Face Hub either: {type(e1).__name__}: {e1}"
-> 1246                     ) from None
   1247                 raise e1 from None
   1248     else:

FileNotFoundError: Couldn't find a dataset script at /content/tsv/tsv.py or any data file in the same directory. Couldn't find 'tsv' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/main/datasets/tsv/tsv.py

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5