Support remote file systems for `Audio`

enhancement

New feature or request

### Feature request

Hi there!

It would be super cool if `Audio()`, and potentially other features, could read files from a remote file system.

### Motivation

Large amounts of data is often stored in buckets. `load_from_disk` is able to retrieve data from cloud storage but to my knowledge actually copies the datasets across first, so if you're working off a system with smaller disk specs (like a VM), you can run out of space very quickly.

### Your contribution

Something like this (for Google Cloud Platform in this instance):

```python
from datasets import Dataset, Audio
import gcsfs

fs = gcsfs.GCSFileSystem()

list_of_audio_fp = {'audio': ['1', '2', '3']}
ds = Dataset.from_dict(list_of_audio_fp)

ds = ds.cast_column("audio", Audio(sampling_rate=16000, fs=fs))
```

Under the hood:
```python

import librosa
from io import BytesIO

def load_audio(fp, sampling_rate=None, fs=None):
    if fs is not None:
        with fs.open(fp, 'rb') as f:
            arr, sr = librosa.load(BytesIO(f), sr=sampling_rate)
    else:
        # Perform existing io operations
```

Written from memory so some things could be wrong.


Hi @J-shel, thanks for reporting.

I think the issue comes from your call to `load_dataset`. As first argument, you should pass:
- either the name of your dataset ("mrf") if this is already published on the Hub
- or the path to the loading script of your dataset ("path/to/your/local/mrf.py").

Hi, following your suggestion, I changed my call to load_dataset. Below is the latest:
reader = load_dataset('data/mrf.py',"default", input_size=1024, split=split, streaming=True, keep_in_memory=None)
However, I still got the same error.
I have one question that is if I only define input_size=2048 in BUILDER_CONFIGS, may I specify input_size=1024 when loading the dataset?  Cause I found that I could only specify name="default" since I only define  name="default" in BUILDER_CONFIGS.

### Describe the bug

I try to define a custom configuration with a input_size attribute following the instructions by "Specifying several dataset configurations" in https://huggingface.co/docs/datasets/v1.2.1/add_dataset.html

But when I load the dataset, I got an error "__init__() got an unexpected keyword argument 'input_size'"

### Steps to reproduce the bug

Following is the code to define the dataset:
class CsvConfig(datasets.BuilderConfig):
    """BuilderConfig for CSV."""
    input_size: int = 2048

class MRF(datasets.ArrowBasedBuilder):
    """Archival MRF data"""
    BUILDER_CONFIG_CLASS = CsvConfig
    VERSION = datasets.Version("1.0.0")
    BUILDER_CONFIGS = [
        CsvConfig(name="default", version=VERSION, description="MRF data", input_size=2048),
    ]

...

    def _generate_examples(self):
        input_size = self.config.input_size
        if input_size > 1000:
           numin = 10000
        else:
           numin = 15000

Below is the code to load the dataset:
reader = load_dataset("default", input_size=1024)

### Expected behavior

I hope to pass the "input_size" parameter to MRF datasets, and change "input_size" to any value when loading the datasets.

### Environment info

- `datasets` version: 2.5.1
- Platform: Linux-4.18.0-305.3.1.el8.x86_64-x86_64-with-glibc2.31
- Python version: 3.9.12
- PyArrow version: 9.0.0
- Pandas version: 1.5.0


Do we need to implement `_prepare_split`?

Hi! `DatasetBuilder` is a parent class for concrete builders: `GeneratorBasedBuilder`, `ArrowBasedBuilder` and `BeamBasedBuilder`. When writing a builder script, these classes are the ones you should inherit from. And since all of them implement `_prepare_split`, you only have to implement the three methods mentioned above.

Thanks so much @mariosasko for the fast response! I've been referencing [this page in the docs](https://huggingface.co/docs/datasets/v2.4.0/en/about_dataset_load) because it it pretty comprehensive in terms of what we have to do and I figured since we subclass the `BuilderConfig` the same pattern would hold, but I've also seen the page with those sub-classed builders as well, so that fills in a knowledge gap for me.

cc @stevhliu who may have some ideas on how to improve this part of the docs.

one more question for my understanding @mariosasko. the requirement of a loading script has always seemed counterintuitive to me. if i have to provide a script with every dataset, what is the point of using `datasets` if we're doing all the work of loading it, I can just do that in my code and skip the datasets integration (this of course discounts other potential benefits around metadata management, etc., my example is just simplest use case though for the sake of discussion).

so i figured I would implement my own `BuilderConfig` and `DatasetBuilder` to handle that portion of it and not have to make a script. i _thought_ this would result in `datasets` (via `download_and_prepare`) then making me something that I could load using `load_dataset` moving forward.

Concretely, i envisioned this pattern being possible:

 ```
class MyBuilderConfig(BuilderConfig):
    def __init__(self, name="my_named_dataset", ...):
        super().__init__(name, ...)

class MyDatasetBuilder(GeneratorBasedBuilder):
   BUILDER_CONFIG_CLASS = MyBuilderConfig
    ....

my_builder = MyDatasetBuilder(...)

# this doesn't exactly work like I thought; I don't get a dataset back, but NoneType instead
# though I can see it loading the files and it generates the cache, etc.
my_dataset = my_builder.download_and_prepare()

# load the dataset in the future by referencing it by name and loading from the cached arrow version
new_instance_of_my_dataset = load_dataset("my_named_dataset")
```

I've seen references to the `save_to_disk` method which might be the next step I need in order to load it by name, in which case, that makes sense, then i just need to debug why `download_and_prepare` isn't returning me a dataset, but I feel like I still have a larger conceptual knowledge gap on how to use the library correctly.

Thanks again in advance!

>  the requirement of a loading script has always seemed counterintuitive to me

This is a requirement only for datasets not stored in standard formats such as CSV, JSON, SQL, Parquet, ImageFolder, etc.

>  if i have to provide a script with every dataset, what is the point of using datasets if we're doing all the work of loading it, I can just do that in my code and skip the datasets integration (this of course discounts other potential benefits around metadata management, etc., my example is just simplest use case though for the sake of discussion)

Our README/documentation lists the main features...

One of the main ones is that our library makes it easy to work with datasets larger than RAM (thanks to Arrow and the caching mechanism), and this is not trivial to implement.

Regarding the step-by-step builder, this is the pattern:
```python
from datasets import load_dataset_builder
builder = load_dataset_builder("path/to/script") # or direct instantiation with MyDatasetBuilder(...)
builder.download_and_prepare()
dset = builder.as_dataset()
```

ok, that makes sense. thank you @mariosasko. I realized i'd never looked on the hub at any of the files associated with any datasets. just did that now and it appears that i'll need to have a script regardless _but_ that will just contain my custom config and builder classes, so without realizing it I was already making my script,  I just need to wrap that in a file that sits alongside my data (I looked at Glue and realized I was already doing what I thought didn't make sense to have to do, lol).

`download_and_prepare` isn't returning me a dataset though, but I'll look into that and open another issue if I can't figure it out.

`download_and_prepare` downloads and prepares the arrow files. You need to call `as_dataset` on the builder to get the dataset.

ok, I think I was assigning the output of `builder.download_and_prepare` but it's an inplace op, so that explains the `NoneType` i was getting back. Now I'm getting:

```
ArrowInvalid                              Traceback (most recent call last)
<ipython-input-7-3ed50fb87c70> in <module>
----> 1 ds = dataset_builder.as_dataset()

/databricks/python/lib/python3.8/site-packages/datasets/builder.py in as_dataset(self, split, run_post_process, ignore_verifications, in_memory)
   1020
   1021         # Create a dataset for each of the given splits
-> 1022         datasets = map_nested(
   1023             partial(
   1024                 self._build_single_dataset,

/databricks/python/lib/python3.8/site-packages/datasets/utils/py_utils.py in map_nested(function, data_struct, dict_only, map_list, map_tuple, map_numpy, num_proc, parallel_min_length, types, disable_tqdm, desc)
    442         num_proc = 1
    443     if num_proc <= 1 or len(iterable) < parallel_min_length:
--> 444         mapped = [
    445             _single_map_nested((function, obj, types, None, True, None))
    446             for obj in logging.tqdm(iterable, disable=disable_tqdm, desc=desc)

/databricks/python/lib/python3.8/site-packages/datasets/utils/py_utils.py in <listcomp>(.0)
    443     if num_proc <= 1 or len(iterable) < parallel_min_length:
    444         mapped = [
--> 445             _single_map_nested((function, obj, types, None, True, None))
    446             for obj in logging.tqdm(iterable, disable=disable_tqdm, desc=desc)
    447         ]

/databricks/python/lib/python3.8/site-packages/datasets/utils/py_utils.py in _single_map_nested(args)
    344     # Singleton first to spare some computation
    345     if not isinstance(data_struct, dict) and not isinstance(data_struct, types):
--> 346         return function(data_struct)
    347
    348     # Reduce logging to keep things readable in multiprocessing with tqdm

/databricks/python/lib/python3.8/site-packages/datasets/builder.py in _build_single_dataset(self, split, run_post_process, ignore_verifications, in_memory)
   1051
   1052         # Build base dataset
-> 1053         ds = self._as_dataset(
   1054             split=split,
   1055             in_memory=in_memory,

/databricks/python/lib/python3.8/site-packages/datasets/builder.py in _as_dataset(self, split, in_memory)
   1120         """
   1121         cache_dir = self._fs._strip_protocol(self._output_dir)
-> 1122         dataset_kwargs = ArrowReader(cache_dir, self.info).read(
   1123             name=self.name,
   1124             instructions=split,

/databricks/python/lib/python3.8/site-packages/datasets/arrow_reader.py in read(self, name, instructions, split_infos, in_memory)
    236             msg = f'Instruction "{instructions}" corresponds to no data!'
    237             raise ValueError(msg)
--> 238         return self.read_files(files=files, original_instructions=instructions, in_memory=in_memory)
    239
    240     def read_files(

/databricks/python/lib/python3.8/site-packages/datasets/arrow_reader.py in read_files(self, files, original_instructions, in_memory)
    257         """
    258         # Prepend path to filename
--> 259         pa_table = self._read_files(files, in_memory=in_memory)
    260         # If original_instructions is not None, convert it to a human-readable NamedSplit
    261         if original_instructions is not None:

/databricks/python/lib/python3.8/site-packages/datasets/arrow_reader.py in _read_files(self, files, in_memory)
    192             f["filename"] = os.path.join(self._path, f["filename"])
    193         for f_dict in files:
--> 194             pa_table: Table = self._get_table_from_filename(f_dict, in_memory=in_memory)
    195             pa_tables.append(pa_table)
    196         pa_tables = [t for t in pa_tables if len(t) > 0]

/databricks/python/lib/python3.8/site-packages/datasets/arrow_reader.py in _get_table_from_filename(self, filename_skip_take, in_memory)
    327             filename_skip_take["take"] if "take" in filename_skip_take else None,
    328         )
--> 329         table = ArrowReader.read_table(filename, in_memory=in_memory)
    330         if take == -1:
    331             take = len(table) - skip

/databricks/python/lib/python3.8/site-packages/datasets/arrow_reader.py in read_table(filename, in_memory)
    348         """
    349         table_cls = InMemoryTable if in_memory else MemoryMappedTable
--> 350         return table_cls.from_file(filename)
    351
    352

/databricks/python/lib/python3.8/site-packages/datasets/table.py in from_file(cls, filename, replays)
   1034     @classmethod
   1035     def from_file(cls, filename: str, replays=None):
-> 1036         table = _memory_mapped_arrow_table_from_file(filename)
   1037         table = cls._apply_replays(table, replays)
   1038         return cls(table, filename, replays)

/databricks/python/lib/python3.8/site-packages/datasets/table.py in _memory_mapped_arrow_table_from_file(filename)
     48 def _memory_mapped_arrow_table_from_file(filename: str) -> pa.Table:
     49     memory_mapped_stream = pa.memory_map(filename)
---> 50     opened_stream = pa.ipc.open_stream(memory_mapped_stream)
     51     pa_table = opened_stream.read_all()
     52     return pa_table

/databricks/python/lib/python3.8/site-packages/pyarrow/ipc.py in open_stream(source)
    152     reader : RecordBatchStreamReader
    153     """
--> 154     return RecordBatchStreamReader(source)
    155
    156

/databricks/python/lib/python3.8/site-packages/pyarrow/ipc.py in __init__(self, source)
     43
     44     def __init__(self, source):
---> 45         self._open(source)
     46
     47

/databricks/python/lib/python3.8/site-packages/pyarrow/ipc.pxi in pyarrow.lib._RecordBatchStreamReader._open()

/databricks/python/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

/databricks/python/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: Tried reading schema message, was null or length 0
```

looks like my arrow files are all empty @mariosasko

![image](https://user-images.githubusercontent.com/7530947/208179977-9ae62c9a-866c-472b-9a09-25d1191188fb.png)

i also see the `incomplete_info.lock` file a level up too. seems like the data isn't being persisted to disk when I call `download_and_prepare`. is there something else i need to do before then, perhaps?

quick update @mariosasko. i got it working! i had to downgrade to `datasets==2.4.0`. testing other versions now and will let you know the results.

I've tested with every version of `datasets>2.4.0` and i get the same error with all of them.

### Describe the bug

I'm not sure this is a bug or if it's just missing in the documentation, or i'm not doing something correctly, but I'm subclassing `DatasetBuilder` and getting the following error because on the `DatasetBuilder` class the `_prepare_split` method is abstract (as are the others we are required to implement, hence the genesis of my question):

```
Traceback (most recent call last):
  File "/home/jason/source/python/prism_machine_learning/examples/create_hf_datasets.py", line 28, in <module>
    dataset_builder.download_and_prepare()
  File "/home/jason/.virtualenvs/pml/lib/python3.8/site-packages/datasets/builder.py", line 704, in download_and_prepare
    self._download_and_prepare(
  File "/home/jason/.virtualenvs/pml/lib/python3.8/site-packages/datasets/builder.py", line 793, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/jason/.virtualenvs/pml/lib/python3.8/site-packages/datasets/builder.py", line 1124, in _prepare_split
    raise NotImplementedError()
NotImplementedError
```

### Steps to reproduce the bug

I will share implementation if it turns out that everything should be working (i.e. we only need to implement those 3 methods the docs mention), but I don't want to distract from the original question.

### Expected behavior

I just need to know if there are additional methods we need to implement when subclassing `DatasetBuilder` besides what the documentation specifies -> `_info`, `_split_generators` and `_generate_examples`

### Environment info

- `datasets` version: 2.4.0
- Platform: Linux-5.4.0-135-generic-x86_64-with-glibc2.2.5
- Python version: 3.8.12
- PyArrow version: 7.0.0
- Pandas version: 1.4.1


Clean up Loading methods docstrings

_The documentation is not available anymore as the PR was closed or merged._

Clean up for the docstrings in Loading methods!


Clean up remaining Main Classes docstrings

This PR cleans up the remaining docstrings in Main Classes (`IterableDataset`, `IterableDatasetDict`, and `Features`).


The data downloaded in the download folder of the cache does not respect `umask`

note, that `datasets` already did some of that umask fixing in the past and also at the hub - the recent work on the hub about the same: https://github.com/huggingface/huggingface_hub/pull/1220

Also I noticed that each file has a .json counterpart and the latter always has the correct perms:

```
-rw------- 1 uue59kq cnw 173M Dec  9 01:37 537596e64721e2ae3d98785b91d30fda0360c196a8224e29658ad629e7303a4d
-rw-rw---- 1 uue59kq cnw  101 Dec  9 01:37 537596e64721e2ae3d98785b91d30fda0360c196a8224e29658ad629e7303a4d.json
```

so perhaps cheating is possible and syncing the perms between the 2 will do the trick.

### Describe the bug

For a project on a cluster we are several users to share the same cache for the datasets library. And we have a problem with the permissions on the data downloaded in the cache.

Indeed, it seems that the data is downloaded by giving read and write permissions only to the user launching the command (and no permissions to the group). In our case, those permissions don't respect the `umask` of this user, which was `0007`.

Traceback:
```
Using custom data configuration default
Downloading and preparing dataset text_caps/default to /gpfswork/rech/cnw/commun/datasets/HuggingFaceM4___text_caps/default/1.0.0/2b9ad220cd90fcf2bfb454645bc54364711b83d6d39401ffdaf8cc40882e9141...
Downloading data files: 100%|████████████████████| 3/3 [00:00<00:00, 921.62it/s]
---------------------------------------------------------------------------
PermissionError                           Traceback (most recent call last)
Cell In [3], line 1
----> 1 ds = load_dataset(dataset_name)

File /gpfswork/rech/cnw/commun/conda/lucile-m4_3/lib/python3.8/site-packages/datasets/load.py:1746, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1743 try_from_hf_gcs = path not in _PACKAGED_DATASETS_MODULES
   1745 # Download and prepare data
-> 1746 builder_instance.download_and_prepare(
   1747     download_config=download_config,
   1748     download_mode=download_mode,
   1749     ignore_verifications=ignore_verifications,
   1750     try_from_hf_gcs=try_from_hf_gcs,
   1751     use_auth_token=use_auth_token,
   1752 )
   1754 # Build dataset for splits
   1755 keep_in_memory = (
   1756     keep_in_memory if keep_in_memory is not None else is_small_dataset(builder_instance.info.dataset_size)
   1757 )

File /gpfswork/rech/cnw/commun/conda/lucile-m4_3/lib/python3.8/site-packages/datasets/builder.py:704, in DatasetBuilder.download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    702         logger.warning("HF google storage unreachable. Downloading and preparing it from source")
    703 if not downloaded_from_gcs:
--> 704     self._download_and_prepare(
    705         dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    706     )
    707 # Sync info
    708 self.info.dataset_size = sum(split.num_bytes for split in self.info.splits.values())

File /gpfswork/rech/cnw/commun/conda/lucile-m4_3/lib/python3.8/site-packages/datasets/builder.py:1227, in GeneratorBasedBuilder._download_and_prepare(self, dl_manager, verify_infos)
   1226 def _download_and_prepare(self, dl_manager, verify_infos):
-> 1227     super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)

File /gpfswork/rech/cnw/commun/conda/lucile-m4_3/lib/python3.8/site-packages/datasets/builder.py:771, in DatasetBuilder._download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    769 split_dict = SplitDict(dataset_name=self.name)
    770 split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 771 split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    773 # Checksums verification
    774 if verify_infos and dl_manager.record_checksums:

File /gpfswork/rech/cnw/commun/modules/datasets_modules/datasets/HuggingFaceM4--TextCaps/2b9ad220cd90fcf2bfb454645bc54364711b83d6d39401ffdaf8cc40882e9141/TextCaps.py:125, in TextCapsDataset._split_generators(self, dl_manager)
    123 def _split_generators(self, dl_manager):
    124     # urls = _URLS[self.config.name] # TODO later
--> 125     data_dir = dl_manager.download_and_extract(_URLS)
    126     gen_kwargs = {
    127         split_name: {
    128             f"{dir_name}_path": Path(data_dir[dir_name][split_name])
   (...)
    133         for split_name in ["train", "val", "test"]
    134     }
    136     for split_name in ["train", "val", "test"]:

File /gpfswork/rech/cnw/commun/conda/lucile-m4_3/lib/python3.8/site-packages/datasets/download/download_manager.py:431, in DownloadManager.download_and_extract(self, url_or_urls)
    415 def download_and_extract(self, url_or_urls):
    416     """Download and extract given url_or_urls.
    417
    418     Is roughly equivalent to:
   (...)
    429         extracted_path(s): `str`, extracted paths of given URL(s).
    430     """
--> 431     return self.extract(self.download(url_or_urls))

File /gpfswork/rech/cnw/commun/conda/lucile-m4_3/lib/python3.8/site-packages/datasets/download/download_manager.py:324, in DownloadManager.download(self, url_or_urls)
    321 self.downloaded_paths.update(dict(zip(url_or_urls.flatten(), downloaded_path_or_paths.flatten())))
    323 start_time = datetime.now()
--> 324 self._record_sizes_checksums(url_or_urls, downloaded_path_or_paths)
    325 duration = datetime.now() - start_time
    326 logger.info(f"Checksum Computation took {duration.total_seconds() // 60} min")

File /gpfswork/rech/cnw/commun/conda/lucile-m4_3/lib/python3.8/site-packages/datasets/download/download_manager.py:229, in DownloadManager._record_sizes_checksums(self, url_or_urls, downloaded_path_or_paths)
    226 """Record size/checksum of downloaded files."""
    227 for url, path in zip(url_or_urls.flatten(), downloaded_path_or_paths.flatten()):
    228     # call str to support PathLike objects
--> 229     self._recorded_sizes_checksums[str(url)] = get_size_checksum_dict(
    230         path, record_checksum=self.record_checksums
    231     )

File /gpfswork/rech/cnw/commun/conda/lucile-m4_3/lib/python3.8/site-packages/datasets/utils/info_utils.py:82, in get_size_checksum_dict(path, record_checksum)
     80 if record_checksum:
     81     m = sha256()
---> 82     with open(path, "rb") as f:
     83         for chunk in iter(lambda: f.read(1 << 20), b""):
     84             m.update(chunk)

PermissionError: [Errno 13] Permission denied: '/gpfswork/rech/cnw/commun/datasets/downloads/1e6aa6d23190c30885194fabb193dce3874d902d7636b66315ee8aaa584e80d6'
```

### Steps to reproduce the bug

I think the following will reproduce the bug.

Given 2 users belonging to the same group with `umask` set to `0007`

- first run with User 1:
```python
from datasets import load_dataset
ds_name = "HuggingFaceM4/VQAv2"
ds = load_dataset(ds_name)
```

- then run with User 2:

```python
from datasets import load_dataset
ds_name = "HuggingFaceM4/TextCaps"
ds = load_dataset(ds_name)
```

### Expected behavior

No `PermissionError`

### Environment info

- `datasets` version: 2.4.0
- Platform: Linux-4.18.0-305.65.1.el8_4.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.13
- PyArrow version: 7.0.0
- Pandas version: 1.4.2


Force soundfile to return float32 instead of the default float64

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5347). All of your documentation changes will be reflected on that endpoint.

Cool ! Feel free to add a comment in the code to explain that and we can merge :)

I'm not sure if this is a good change since we plan to get rid of `torchaudio` in the next couple of months...

What do you think @polinaeterna @patrickvonplaten ? Models are usually using float32 (e.g. Wev2vec2 in `transformers`) IIRC

IMO we can safely assume that float32 is always good enough when using audio models in inference or training. Nevertheless there might be use cases for audio datasets in the future where float64 is needed.

=> I would by default always cast to float32, but then possible allow the user to cast to float64

> I'm not sure if this is a good change since we plan to get rid of torchaudio in the next couple of months...

@mariosasko I agree but who knows how long we will have to wait until we are really able to do so (https://github.com/bastibe/libsndfile-binaries/pull/17 is a draft. so as @patrickvonplaten is okay with float32, I'd merge.

@polinaeterna Can you comment on the linked PR to see why it's still a draft? Maybe we can help somehow to get this merged finally.

I think it's weird to align `soundfile` with `torchaudio` when the latter is only used for MP3 (and prob for not much longer).

(Fixes issue #5345)


[Quick poll] Give your opinion on the future of the Hugging Face Open Source ecosystem!

As the survey is finished, can we close this issue, @LysandreJik ?

Yes! I'll post a public summary on the forums shortly.

Is the summary available? I would be interested in reading your findings.

Thanks to all of you, Datasets is just about to pass 15k stars!

Since the last survey, a lot has happened: the [diffusers](https://github.com/huggingface/diffusers), [evaluate](https://github.com/huggingface/evaluate) and [skops](https://github.com/skops-dev/skops) libraries were born. `timm` joined the Hugging Face ecosystem. There were 25 new releases of `transformers`, 21 new releases of `datasets`, 13 new releases of `accelerate`.

If you have a couple of minutes and want to participate in shaping the future of the ecosystem, please share your thoughts:

[**hf.co/oss-survey**](https://docs.google.com/forms/d/e/1FAIpQLSf4xFQKtpjr6I_l7OfNofqiR8s-WG6tcNbkchDJJf5gYD72zQ/viewform?usp=sf_link)

(please reply in the above feedback form rather than to this thread)

Thank you all on behalf of the HuggingFace team! 🤗


Wrong dtype for array in audio features

After some more investigation, this is due to [this line of code](https://github.com/huggingface/datasets/blob/main/src/datasets/features/audio.py#L279). The function `sf.read(file)` should be updated to `sf.read(file, dtype="float32")`

Indeed, the default value in soundfile is `float64` ([see here](https://pysoundfile.readthedocs.io/en/latest/#soundfile.read)).

@qmeeus I agree, decoding of different audio formats should return the same dtypes indeed!

But note that here you are concatenating datasets with different sampling rates: 48000 for CommonVoice and 16000 for Voxpopuli. So you should cast them to the same sampling rate value before interleaving, for example:
```
cv = cv.cast_column("audio", Audio(sampling_rate=16000))
```
otherwise you would get the same error because features of the same column ("audio") are not the same.

Also, the error you get is unexpected. Could you please confirm that you use the latest main version of the `datasets`?  We had an issue that could lead to an error like this after using `rename_column` method, but it was fixed in https://github.com/huggingface/datasets/pull/5287

Hi Polina,
Sorry for the late answer
It is possible that the issue was due to a bug that is now fixed. I installed an editable version of datasets from github, but I don't recall whether I had updated it at the time of the issue. My research led me to other directions so I did not follow through on the interleave datasets.

### Describe the bug

When concatenating/interleaving different datasets, I stumble into an error because the features can't be aligned. After some investigation, I understood that the audio arrays had different dtypes, namely `float32` and `float64`. Consequently, the datasets cannot be merged.

### Steps to reproduce the bug

For example, for `facebook/voxpopuli` and `mozilla-foundation/common_voice_11_0`:

```
from datasets import load_dataset, interleave_datasets

covost = load_dataset("mozilla-foundation/common_voice_11_0", "en", split="train", streaming=True)
voxpopuli = datasets.load_dataset("facebook/voxpopuli", "nl", split="train", streaming=True)

sample_cv, = covost.take(1)
sample_vp, = voxpopuli.take(1)

assert sample_cv["audio"]["array"].dtype == sample_vp["audio"]["array"].dtype
# Fails

dataset = interleave_datasets([covost, voxpopuli])

# ValueError: The features can't be aligned because the key audio of features {'audio_id': Value(dtype='string', id=None), 'language': Value(dtype='int64', id=None), 'audio': {'array': Sequence(feature=Value(dtype='float64', id=None), length=-1, id=None), 'path': Value(dtype='string', id=None), 'sampling_rate': Value(dtype='int64', id=None)}, 'normalized_text': Value(dtype='string', id=None), 'gender': Value(dtype='string', id=None), 'speaker_id': Value(dtype='string', id=None), 'is_gold_transcript': Value(dtype='bool', id=None), 'accent': Value(dtype='string', id=None), 'sentence': Value(dtype='string', id=None)} has unexpected type - {'array': Sequence(feature=Value(dtype='float64', id=None), length=-1, id=None), 'path': Value(dtype='string', id=None), 'sampling_rate': Value(dtype='int64', id=None)} (expected either Audio(sampling_rate=16000, mono=True, decode=True, id=None) or Value("null").
```

### Expected behavior

The audio should be loaded to arrays with a unique dtype (I guess `float32`)

### Environment info

```
- `datasets` version: 2.7.1.dev0
- Platform: Linux-4.18.0-425.3.1.el8.x86_64-x86_64-with-glibc2.28
- Python version: 3.9.15
- PyArrow version: 10.0.1
- Pandas version: 1.5.2
```


Clean up Dataset and DatasetDict

This PR cleans up the docstrings for the other half of the methods in `Dataset` and finishes `DatasetDict`.


T5 for Q&A produces truncated sentence

Dear all, I am fine-tuning T5 for Q&A task using the MedQuAD ([GitHub - abachaa/MedQuAD: Medical Question Answering Dataset of 47,457 QA pairs created from 12 NIH websites](https://github.com/abachaa/MedQuAD)) dataset. In the dataset, there are many long answers with thousands of words. I have used pytorch_lightning to train the T5-large model. I have two questions.

For example, I set both the max_length, max_input_length, max_output_length to 128.

How to deal with those long answers? I just left them as is and the T5Tokenizer can automatically handle. I would assume the tokenizer just truncates an answer at the position of 128th word (or 127th). Is it possible that I manually split an answer into different parts, each part has 128 words; and then all these sub-answers serve as a separate answer to the same question?

Another question is that I get incomplete (truncated) answers when using the fine-tuned model in inference, even though the predicted answer is shorter than 128 words. I found a message posted 2 years ago saying that one should add at the end of texts when fine-tuning T5. I followed that but then got a warning message that duplicated were found. I am assuming that this is because the tokenizer truncates an answer text, thus is missing in the truncated answer, such that the end token is not produced in predicted answer. However, I am not sure. Can anybody point out how to address this issue?

Any suggestions are highly appreciated.

Below is some code snippet.

`
import pytorch_lightning as pl
from torch.utils.data import DataLoader
import torch
import numpy as np
import time
from pathlib import Path

from transformers import (
    Adafactor,
    T5ForConditionalGeneration,
    T5Tokenizer,
    get_linear_schedule_with_warmup
)
from torch.utils.data import RandomSampler
from question_answering.utils import *

class T5FineTuner(pl.LightningModule):
    def __init__(self, hyparams):
        super(T5FineTuner, self).__init__()
        self.hyparams = hyparams
        self.model = T5ForConditionalGeneration.from_pretrained(hyparams.model_name_or_path)
        self.tokenizer = T5Tokenizer.from_pretrained(hyparams.tokenizer_name_or_path)

        if self.hyparams.freeze_embeds:
            self.freeze_embeds()
        if self.hyparams.freeze_encoder:
            self.freeze_params(self.model.get_encoder())
            # assert_all_frozen()

        self.step_count = 0
        self.output_dir = Path(self.hyparams.output_dir)

        n_observations_per_split = {
            'train': self.hyparams.n_train,
            'validation': self.hyparams.n_val,
            'test': self.hyparams.n_test
        }
        self.n_obs = {k: v if v >= 0 else None for k, v in n_observations_per_split.items()}
        self.em_score_list = []
        self.subset_score_list = []

        data_folder = r'C:\Datasets\MedQuAD-master'
        self.train_data, self.val_data, self.test_data = load_medqa_data(data_folder)

    def freeze_params(self, model):
        for param in model.parameters():
            param.requires_grad = False

    def freeze_embeds(self):
        try:
            self.freeze_params(self.model.model.shared)
            for d in [self.model.model.encoder, self.model.model.decoder]:
                self.freeze_params(d.embed_positions)
                self.freeze_params(d.embed_tokens)
        except AttributeError:
            self.freeze_params(self.model.shared)
            for d in [self.model.encoder, self.model.decoder]:
                self.freeze_params(d.embed_tokens)

    def lmap(self, f, x):
        return list(map(f, x))

    def is_logger(self):
        return self.trainer.proc_rank <= 0

    def forward(self, input_ids, attention_mask=None, decoder_input_ids=None, decoder_attention_mask=None, labels=None):
        return self.model(
            input_ids,
            attention_mask=attention_mask,
            decoder_input_ids=decoder_input_ids,
            decoder_attention_mask=decoder_attention_mask,
            labels=labels
        )

    def _step(self, batch):
        labels = batch['target_ids']
        labels[labels[:, :] == self.tokenizer.pad_token_id] = -100

        outputs = self(
            input_ids = batch['source_ids'],
            attention_mask=batch['source_mask'],
            labels=labels,
            decoder_attention_mask=batch['target_mask']
        )

        loss = outputs[0]

        return loss

    def ids_to_clean_text(self, generated_ids):
        gen_text = self.tokenizer.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)
        return self.lmap(str.strip, gen_text)

    def _generative_step(self, batch):
        t0 = time.time()

        generated_ids = self.model.generate(
            batch["source_ids"],
            attention_mask=batch["source_mask"],
            use_cache=True,
            decoder_attention_mask=batch['target_mask'],
            max_length=128,
            num_beams=2,
            early_stopping=True
        )
        preds = self.ids_to_clean_text(generated_ids)
        targets = self.ids_to_clean_text(batch["target_ids"])

        gen_time = (time.time() - t0) / batch["source_ids"].shape[0]

        loss = self._step(batch)
        base_metrics = {'val_loss': loss}
        summ_len = np.mean(self.lmap(len, generated_ids))
        base_metrics.update(gen_time=gen_time, gen_len=summ_len, preds=preds, target=targets)
        em_score, subset_match_score = calculate_scores(preds, targets)

        self.em_score_list.append(em_score)
        self.subset_score_list.append(subset_match_score)

        em_score = torch.tensor(em_score, dtype=torch.float32)
        subset_match_score = torch.tensor(subset_match_score, dtype=torch.float32)

        base_metrics.update(em_score=em_score, subset_match_score=subset_match_score)

        #         rouge_results = self.rouge_metric.compute()
        #         rouge_dict = self.parse_score(rouge_results)

        return base_metrics

    def training_step(self, batch, batch_idx):
        loss = self._step(batch)
        tensorboard_logs = {'train_loss': loss}
        return {'loss': loss, 'log': tensorboard_logs}

    def training_epoch_end(self, outputs):
        avg_train_loss = torch.stack([x['loss'] for x in outputs]).mean()
        tensorboard_logs = {'avg_train_loss': avg_train_loss}
        # return {'avg_train_loss': avg_train_loss, 'log': tensorboard_logs, 'progress_bar': tensorboard_logs}

    def validation_step(self, batch, batch_idx):
        return self._generative_step(batch)

    def validation_epoch_end(self, outputs):
        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        tensorboard_logs = {'val_loss': avg_loss}

        if len(self.em_score_list) <= 2:
            average_em_score = sum(self.em_score_list) / len(self.em_score_list)
            average_subset_match_score = sum(self.subset_score_list) / len(self.subset_score_list)

        else:
            latest_em_score = self.em_score_list[:-2]
            latest_subset_score = self.subset_score_list[:-2]
            average_em_score = sum(latest_em_score) / len(latest_em_score)
            average_subset_match_score = sum(latest_subset_score) / len(latest_subset_score)

        average_em_score = torch.tensor(average_em_score, dtype=torch.float32)
        average_subset_match_score = torch.tensor(average_subset_match_score, dtype=torch.float32)
        tensorboard_logs.update(em_score=average_em_score, subset_match_score=average_subset_match_score)

        self.target_gen = []
        self.prediction_gen = []
        return {
            'avg_val_loss': avg_loss,
            'em_score': average_em_score,
            'subset_match_socre': average_subset_match_score,
            'log': tensorboard_logs,
            'progress_bar': tensorboard_logs
        }

    def configure_optimizers(self):
        model = self.model
        no_decay = ["bias", "LayerNorm.weight"]
        optimizer_grouped_parameters = [
            {
                "params": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],
                "weight_decay": self.hyparams.weight_decay,
            },
            {
                "params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],
                "weight_decay": 0.0,
            },
        ]

        optimizer = Adafactor(optimizer_grouped_parameters, lr=self.hyparams.learning_rate, scale_parameter=False,
                              relative_step=False)
        self.opt = optimizer
        return [optimizer]

    def optimizer_step(self, epoch, batch_idx, optimizer, optimizer_idx, optimizer_closure=None,
                       on_tpu=False, using_native_amp=False, using_lbfgs=False):
        optimizer.step(closure=optimizer_closure)
        optimizer.zero_grad()
        self.lr_scheduler.step()

    def get_tqdm_dict(self):
        tqdm_dict = {"loss": "{:.3f}".format(self.trainer.avg_loss), "lr": self.lr_scheduler.get_last_lr()[-1]}

        return tqdm_dict

    def train_dataloader(self):
        n_samples = self.n_obs['train']
        train_dataset = get_dataset(tokenizer=self.tokenizer, data=self.train_data, num_samples=n_samples,
                                    args=self.hyparams)
        sampler = RandomSampler(train_dataset)
        dataloader = DataLoader(train_dataset, sampler=sampler, batch_size=self.hyparams.train_batch_size,
                                drop_last=True, num_workers=4)
        # t_total = (
        #         (len(dataloader.dataset) // (self.hyparams.train_batch_size * max(1, self.hyparams.n_gpu)))
        #         // self.hyparams.gradient_accumulation_steps
        #         * float(self.hyparams.num_train_epochs)
        # )
        t_total = 100000
        scheduler = get_linear_schedule_with_warmup(
            self.opt, num_warmup_steps=self.hyparams.warmup_steps, num_training_steps=t_total
        )
        self.lr_scheduler = scheduler
        return dataloader

    def val_dataloader(self):
        n_samples = self.n_obs['validation']

        validation_dataset = get_dataset(tokenizer=self.tokenizer, data=self.val_data, num_samples=n_samples,
                                         args=self.hyparams)
        sampler = RandomSampler(validation_dataset)
        return DataLoader(validation_dataset, shuffle=False, batch_size=self.hyparams.eval_batch_size, sampler=sampler, num_workers=4)

    def test_dataloader(self):
        n_samples = self.n_obs['test']
        test_dataset = get_dataset(tokenizer=self.tokenizer, data=self.test_data, num_samples=n_samples, args=self.hyparams)

        return DataLoader(test_dataset, batch_size=self.hyparams.eval_batch_size, num_workers=4)

    def on_save_checkpoint(self, checkpoint):
        save_path = self.output_dir.joinpath("best_tfmr")
        self.model.config.save_step = self.step_count
        self.model.save_pretrained(save_path)
        self.tokenizer.save_pretrained(save_path)

import os
import argparse
import pytorch_lightning as pl

from question_answering.t5_closed_book import T5FineTuner

if __name__ == '__main__':
    args_dict = dict(
        output_dir="",  # path to save the checkpoints
        model_name_or_path='t5-large',
        tokenizer_name_or_path='t5-large',
        max_input_length=128,
        max_output_length=128,
        freeze_encoder=False,
        freeze_embeds=False,
        learning_rate=1e-5,
        weight_decay=0.0,
        adam_epsilon=1e-8,
        warmup_steps=0,
        train_batch_size=4,
        eval_batch_size=4,
        num_train_epochs=2,
        gradient_accumulation_steps=10,
        n_gpu=1,
        resume_from_checkpoint=None,
        val_check_interval=0.5,
        n_val=4000,
        n_train=-1,
        n_test=-1,
        early_stop_callback=False,
        fp_16=False,
        opt_level='O1',
        max_grad_norm=1.0,
        seed=101,
    )

    args_dict.update({'output_dir': 't5_large_MedQuAD_256', 'num_train_epochs': 100,
                      'train_batch_size': 16, 'eval_batch_size': 16, 'learning_rate': 1e-3})
    args = argparse.Namespace(**args_dict)

    checkpoint_callback = pl.callbacks.ModelCheckpoint(dirpath=args.output_dir, monitor="em_score", mode="max", save_top_k=1)

    ## If resuming from checkpoint, add an arg resume_from_checkpoint
    train_params = dict(
        accumulate_grad_batches=args.gradient_accumulation_steps,
        gpus=args.n_gpu,
        max_epochs=args.num_train_epochs,
        # early_stop_callback=False,
        precision=16 if args.fp_16 else 32,
        # amp_level=args.opt_level,
        # resume_from_checkpoint=args.resume_from_checkpoint,
        gradient_clip_val=args.max_grad_norm,
        checkpoint_callback=checkpoint_callback,
        val_check_interval=args.val_check_interval,
        # accelerator='dp'
        # logger=wandb_logger,
        # callbacks=[LoggingCallback()],
    )

    model = T5FineTuner(args)
    trainer = pl.Trainer(**train_params)

    trainer.fit(model)
`


Emotion dataset cannot be downloaded

duplicate

This issue or pull request already exists

Hi @cbarond there's already an open issue at https://github.com/dair-ai/emotion_dataset/issues/5, as the data seems to be missing now, so check that issue instead 👍🏻

Thanks @cbarond for reporting and @alvarobartt for pointing to the issue we opened in the author's repo.

Indeed, this issue was first raised in the "emotion" dataset Community tab: https://huggingface.co/datasets/emotion/discussions/3

I'm closing this issue and leave the issue above for the subsequent updates.

Duplicate of: https://huggingface.co/datasets/emotion/discussions/3

> try using "SetFit/emotion" instead

I' replaced "emotion" with "SetFit/Emotion", but the code is getting stuck at

`emotions = load_dataset("SetFit/emotion")`

I pause execution using the debugger, and it takes me to filelock.py:226

`with self._thread_lock:`

Do you know a way to get past this issue?

> try using "SetFit/emotion" instead

It really helps a lot, thank you!

The dataset loading script has been fixed: https://huggingface.co/datasets/emotion/discussions/4

### Describe the bug

The emotion dataset gives a FileNotFoundError. The full error is: `FileNotFoundError: Couldn't find file at https://www.dropbox.com/s/1pzkadrvffbqw6o/train.txt?dl=1`.

It was working yesterday (December 7, 2022), but stopped working today (December 8, 2022).

### Steps to reproduce the bug

```python
from datasets import load_dataset
dataset = load_dataset("emotion")
```

### Expected behavior

The dataset should load properly.

### Environment info

- `datasets` version: 2.7.1
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.9.13
- PyArrow version: 10.0.1
- Pandas version: 1.5.1


Remove tasks.json

_The documentation is not available anymore as the PR was closed or merged._

After discussions in https://github.com/huggingface/datasets/pull/5335 we should remove this file that is not used anymore. We should update https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Types.ts instead.


Clean up DatasetInfo and Dataset docstrings

This PR cleans up the docstrings for `DatasetInfo` and about half of the methods in `Dataset`.


Add Video feature, videofolder, and video-classification task

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5339). All of your documentation changes will be reflected on that endpoint.

@lhoestq I think I need some serious help with the tests 😅...I started this locally but it got too time consuming.

One issue I remember running into is with lossless audio encoding/decoding. I started thinking of using the underlying Audio feature instead of PyAV so I didn't have to rewrite similar logic here...but assumed that would turn into a mess w/ underlying logic

I'm closing this PR. Feel free to reopen it if necessary.

This PR does the following:

- Adds `Video` feature (Resolves #5225 )
- Adds `video-classification` task
- Adds `videofolder` packaged module for easy loading of local video classification datasets

TODO:
  - [ ] add tests
  - [ ] add docs


`map()` stops every 1000 steps

Hi !

> It starts using all the cores (I am not sure why because I did not pass num_proc)

The tokenizer uses Rust code that is multithreaded. And maybe the `feature_extractor` might run some things in parallel as well - but I'm not super familiar with its internals.

> then progress bar stops at every 1k steps. (starts using a single core)

Every 1000 examples we flush the processed examples to disk. It is this way because Arrow is a columnar format: you must write data chunk by chunk. The processing in on hold while writing right now - maybe this can be improved in the future.

Hi @lhoestq
Thanks for the explanation! it was so helpful! Let me check why `feature_extractor` is running on multiple cpus.

### Describe the bug

I am passing the following `prepare_dataset` function to `Dataset.map` (code is inspired from [here](https://github.com/huggingface/community-events/blob/main/whisper-fine-tuning-event/run_speech_recognition_seq2seq_streaming.py#L454))
```python3
def prepare_dataset(batch):
    # load and resample audio data from 48 to 16kHz
    audio = batch["audio"]

    # compute log-Mel input features from input audio array
    batch["input_features"] = feature_extractor(audio["array"], sampling_rate=audio["sampling_rate"]).input_features[0]

    # encode target text to label ids
    batch["labels"] = tokenizer(batch[text_column]).input_ids
    return batch
...
train_ds = train_ds.map(prepare_dataset)
```

Here is the exact code I am running https://github.com/bayartsogt-ya/whisper-multiple-hf-datasets/blob/main/train.py#L70-L71

It starts using all the cores (I am not sure why because I did not pass `num_proc`)
then progress bar stops at every 1k steps. (starts using a single core)
then come back to using all the cores again.

link to [screen record](https://youtu.be/jPQpQQGp6Gc)

Can someone explain this process and maybe provide a way to improve this pipeline? cc: @lhoestq

### Steps to reproduce the bug

1. load the dataset
2. create a Whisper processor
3. create a `prepare_dataset` function
4. pass the function to `dataset.map(prepare_dataset)`

### Expected behavior

- Use a single core per a function
- not to stop at some point?

### Environment info

- `datasets` version: 2.7.1.dev0
- Platform: Linux-5.4.0-109-generic-x86_64-with-glibc2.27
- Python version: 3.8.10
- PyArrow version: 10.0.1
- Pandas version: 1.5.2


Support webdataset format

I like the idea of having `webdataset` as an optional dependency to ensure our loader generates web datasets the same way as the main project.

Webdataset is the one of the most popular dataset formats for large scale computer vision tasks. Upvote for this issue.

We haven't had the bandwidth to implement it so far, but if someone wants to give it a shot please don't hesitate ^^

Webdataset is an efficient format for iterable datasets. It would be nice to support it in `datasets`, as discussed in https://github.com/rom1504/img2dataset/issues/234.

In particular it would be awesome to be able to load one using `load_dataset` in streaming mode (either from a local directory, or from a dataset on the Hugging Face Hub). Some datasets on the Hub are already in webdataset format.

It terms of implementation, we can have something similar to the Parquet loader.
I also think it's fine to have webdataset as an optional dependency.


Set `IterableDataset.map` param `batch_size` typing as optional

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5336). All of your documentation changes will be reflected on that endpoint.

Hi @mariosasko, @lhoestq I was wondering whether we should include `batched` as a `pytest.mark` param for the functions testing `IterableDataset.map` so as to ensure that the changes done in this PR work fine without breaking anything of the actual functionality.

I've pushed updated tests just for one of the unit testing functions to be run as `pytest tests/test_iterable_dataset.py::test_mapped_examples_iterable -s --durations 0`, but some are still missing `batched` param, it was just to ask you whether we're supposed to do this for the rest of the functions or not, if it's a yes I'll push the commit as it's ready, but didn't want to push extra stuff that may be discarded later!

Thanks :hugs:

Thanks for the feedback @lhoestq, I agree with keeping `Optional` instead of `Union[type, None]` for now 👍🏻

This PR solves #5325

~Indeed we're using the typing for optional values as `Union[type, None]` as it's similar to how Python 3.10 handles optional values as `type | None`, instead of using `Optional[type]`.~

~Do we want to start using `Union[type, None]` for type-hinting optional values or just keep on using `Optional`?~ -> Keeping `Optional` still for consistency with the rest of the code in `datasets`

Also we now allow `batch_size` to be `None` for `IterableDataset.map` and `IterableDataset.filter`e.g. `MappedExamplesIterable` as `map` is internally instantiating those and propagating the `batch_size` param so if it can be `None` for `map` it should also do so for `MappedExamplesIterable`, as well as for `FilteredExamplesIterable` when calling `IterableDataset.filter`.

## TODOs

- [x] Add integration tests
- [x] Handle scenario where `batched=True` and `batch_size=None` or `batch_size<=0`


Update tasks.json

I think the only place where we need to add it is here https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Types.ts

And I think we can remove tasks.json completely from this repo

Isn't tasks.json used anymore in this repo?

> I think the only place where we need to add it is here https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Types.ts
>
> And I think we can remove tasks.json completely from this repo

What about the warning I mentioned in https://github.com/huggingface/datasets/issues/5255#issuecomment-1339013527? Also, the depth estimation entry is already present in https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Types.ts.

The update is based on what I received in the output of the export job (c.f. https://github.com/huggingface/datasets/issues/5255#issuecomment-1339107195).

Edit: Oh, are you referring to the dataset card of NYU Depth V2?

Yes, my suggestion was for the dataset card: you got the error message because you tried to set `depth-estimation` in `class_ids` instead of `class_categories`.

> What about the warning I mentioned in https://github.com/huggingface/datasets/issues/5255#issuecomment-1339013527? Also, the depth estimation entry is already present in https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Types.ts.

if you place it in `task_categories` you should be good :)

yes i would suggest rm'ing tasks.json here for clarity

It's not clear if we can remove it btw, since old versions of `evaluate` rely on it (see https://github.com/huggingface/evaluate/pull/309)

cc @lvwerra

Actually it can be removed without incidence in old versions of evaluate since we kept an hardcoded `known_task_ids` that is marked "DEPRECATED"

Context:

* https://github.com/huggingface/datasets/issues/5255#issuecomment-1339107195

Cc: @osanseviero


Clean up docstrings

documentation

Improvements or additions to documentation

_The documentation is not available anymore as the PR was closed or merged._

Thanks ! Let us know if we can help :)

Small pref for having multiple PRs

Awesome, thanks! Sorry this one is a little big, I'll open some smaller ones next :)

As raised by @polinaeterna in #5324, some of the docstrings are a bit of a mess because it has both Markdown and Sphinx syntax. This PR fixes the docstring for `DatasetBuilder`.

I'll start working on cleaning up the rest of the docstrings and removing the old Sphinx syntax (let me know if you prefer one big PR with all the cleaned changes or multiple smaller ones)! 🧼


fix: 🐛 pass the token to get the list of config names

Otherwise, get_dataset_infos doesn't work on gated or private datasets, even with the correct token.


Passing numpy array to ClassLabel names causes ValueError

Should `datasets` allow `ClassLabel` input parameter to be an `np.array` even though internally we need to cast it to a Python list? @lhoestq @mariosasko

Hi! No, I don't think so. The `names` parameter is [annotated](https://github.com/huggingface/datasets/blob/582236640b9109988e5f7a16a8353696ffa09a16/src/datasets/features/features.py#L892) as `List[str]` (**NumPy arrays are not lists**), and considering that type checking is not a common practice in Python, I think we can leave the code as-is.

I appreciate it is the wrong type, and that type checking is not common, but I think there's a few circumstances that make it a good idea from a usability perspective.

It's quite a difficult error to debug because it comes from a utility function (so it's not immediately obvious which parameter caused it). What makes it even more difficult is the exception happens when the features instance is used to instantiate the dataset, **not** when when the wrong type is actually passed when the features is instantiated. When I was debugging the error, I didn't really consider it could be an issue with the features instance because it had instantiated fine. It's also not one of the more common exceptions caused by trying to use a non-list as a list.

It's also relatively easy to accidentally get a numpy array of class types (e.g. calling `unique()` on a pandas dataframe column). Additionally, passing in a `set` instead of the list (again, relatively easy because people may run `set(classes)` to generate uniques) causes an error when the features instance is used, albeit a slightly more obvious one.

The names list is already being processed and validated in the `__post_init__` method anyway, so it would not really be adding any complexity to check it is actually a list here too. I'm happy to contribute this change if you change your mind about whether it's worthwhile.

I agree that it's not easy to debug this issue, so perhaps we could add some basic type checking (e.g. `not isinstance(names, list)` -> error) to make debugging easier. Feel free to submit a PR.

> Additionally, passing in a set instead of the list (again, relatively easy because people may run set(classes) to generate uniques) causes an error when the features instance is used, albeit a slightly more obvious one.

`set` is an unordered structure (it's ordered in Python 3.6+, but this is CPython's implementation detail), and the order of ClassLabel `names` matters, so this doesn't require a fix.

What about checking for `Sequence` instead? I think users can pass a list or a tuple as well.

### Describe the bug

If a numpy array is passed to the names argument of ClassLabel, creating a dataset with those features causes an error.

### Steps to reproduce the bug

https://colab.research.google.com/drive/1cV_es1PWZiEuus17n-2C-w0KEoEZ68IX

TLDR:

If I define my classes as:
```
my_classes = np.array(['one', 'two', 'three'])
```

Then this errors:

```py
features = Features({'value': Value('string'), 'label': ClassLabel(names=my_classes)})
dataset = Dataset.from_list(my_data, features=features)
```
```
ValueError                                Traceback (most recent call last)
[<ipython-input-8-a8a9d53ec82f>](https://localhost:8080/#) in <module>
----> 1 dataset = Dataset.from_list(my_data, features=features)

11 frames
[/usr/local/lib/python3.8/dist-packages/datasets/utils/py_utils.py](https://localhost:8080/#) in _asdict_inner(obj)
    183             for f in fields(obj):
    184                 value = _asdict_inner(getattr(obj, f.name))
--> 185                 if not f.init or value != f.default or f.metadata.get("include_in_asdict_even_if_is_default", False):
    186                     result[f.name] = value
    187             return result

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
```

But this works:
```
features2 = Features({'value': Value('string'), 'label': ClassLabel(names=list(my_classes))})
dataset2 = Dataset.from_list(my_data, features=features2)
```

### Expected behavior

If I provide a numpy array of class names, I would expect either an error that the names list is the wrong type, or for it to be cast internally.

### Environment info

- `datasets` version: 2.7.1
- Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.10
- Python version: 3.8.15
- PyArrow version: 10.0.1
- Pandas version: 1.5.2

Additionally:
- Numpy version: 1.23.5


Support for multiple configs in packaged modules via metadata yaml info

feel free to merge `main` into your PR to fix the CI :)

Let me see if I can fix the pattern thing ^^'

Hmm I think it would be easier to specify the `data_files` in the end, because having a split pattern like `{split}-...` at the root of the repository can lead to unexpected behaviors IMO, and we probably don't want to have a different behavior for `data_files` depending if it's inside a `data_dir` or not

Maybe something like
```yaml
builder_config:
  data_dir: data_dir
  data_files:
  - split: train
    pattern: train-[0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*
```

 > Also, I'm not sure if it's a good idea to have this field in the YAML metadata - Transformers use this part of the card only for Hub-related stuff (widgets, tags, CO2 emission, etc.), and I think we should aim to do the same in Datasets. We could achieve this by having these kwargs in a special file (they can be seen as a faster way of defining a builder (builder script) that subclasses a packaged builder) and removing the dataset_info field (the only useful info there seem to be features and we can fetch those directly from a dataset script/Parquet files).

Something like `config.json`?

```json
{
    "data_dir": "data"
    "data_files":  {
        "train": "train-[0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*"
    }
}
```

we could also support lists for several configs

I opened a PR to this PR to add data_files in YAML: https://github.com/polinaeterna/datasets/pull/1

```yaml
builder_config:
  data_files:
  - split: train
    pattern: data/train-*
```

Let me open a PR to see if I can move the data files resolution outside of the MetadataConfigs to not modify it in-place

> I wonder if we can make the cache backward compatible: we could just check if the cache directory with the old path exists. It will be useful for the research team which has a big datasets cache

![image](https://github.com/huggingface/datasets/assets/16348744/90a96e79-2a0d-4d37-95bd-b75fa962c094)

In the next PR maybe? :D
It's possible but requires some additional logic to correctly pass old `config_kwargs` (which used to include `data_files` but now it's `None` for builders from metadata) to generate the hash which is used to create the path.

If we only consider datasets that were pushed to hub, it's just a matter of using `"{username}__parquet"` instead of `"{username}__{dataset_name}"` in the cache directory name. The hashes stay the same :)

EDIT: and the config name
EDIT2: and the arrow file names

Did a small PR for backward compatibility, it was easy to add in the end: https://github.com/polinaeterna/datasets/pull/3

Just created a branch [dev-3.0](https://github.com/huggingface/datasets/tree/dev-3.0) in which we can merge this one and the other datasets 3.0 related PRs

We can, it was just in case we had other things to merge after @mariosasko or @albertvillanova 's reviews

@lhoestq @albertvillanova @mariosasko we agreed on having `configs` (in plural) as a metadata field in readme but apparently Hub's yaml validation doesn't allow it to be not a list :D
![image](https://github.com/huggingface/datasets/assets/16348744/52131ee8-80e0-4f6e-90cd-8ff83caf4625)
(with `config` (in singular) it works)

edit: and now the tests for hub datasets with metadata configs are failing because I cannot change the yaml there...

> we agreed on having configs (in plural) as a metadata field in readme but apparently Hub's yaml validation doesn't allow it to be not a list :D

If the `configs` field is specified in the YAML, the Hub can use it to [improve](https://github.com/huggingface/moon-landing/blob/97aca4cac32fbb7d84ce5eba9b18afad87968c4a/server/views/components/DatasetLibraryModal/datasetLibrarySnippets.ts#L11) the `Use in dataset library` snippet by listing the possible config values in `load_dataset`. So I think this needs to be fixed on the Hub side.

PS: I couldn't find an instance of someone using this field on the Hub, so I think using it for this feature is OK.

> I couldn't find an instance of someone using this field on the Hub, so I think using it for this feature is OK.

@mariosasko I think it's because @lhoestq renamed `configs` to `config_names` in all canonical datasets :D so yes, `configs` field is now supposed to include custom configuration parameters introduced in this PR, and `config_names` is used (not really used lol) for list of strings of config names. It's being fixed on the Hub's side https://github.com/huggingface/moon-landing/pull/6490

after more thought I agree it's maybe overkill to do a major release for this one, since we have a good backward compatibility

There is one edge case I forgot to mention in the reviews - I think it's a good idea to support passing config params that are functions (Pandas uses them a lot) using this API (e.g. `converters` in the CSV config for converting a string column into a sequence). I see two solutions: string blocks with Python code in YAML or PyYAML [tags](https://pyyaml.org/wiki/PyYAMLDocumentation#yaml-tags-and-python-types).

But I think this can be addressed later.

I'm resolving the conflicts and writing some docs :) let's merge this soon !

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005868 / 0.011353 (-0.005485) | 0.003544 / 0.011008 (-0.007464) | 0.080329 / 0.038508 (0.041821) | 0.061072 / 0.023109 (0.037963) | 0.307802 / 0.275898 (0.031904) | 0.340353 / 0.323480 (0.016873) | 0.004665 / 0.007986 (-0.003321) | 0.002779 / 0.004328 (-0.001550) | 0.062065 / 0.004250 (0.057815) | 0.046350 / 0.037052 (0.009297) | 0.312045 / 0.258489 (0.053556) | 0.353524 / 0.293841 (0.059683) | 0.026965 / 0.128546 (-0.101581) | 0.007906 / 0.075646 (-0.067740) | 0.260678 / 0.419271 (-0.158593) | 0.044167 / 0.043533 (0.000634) | 0.309757 / 0.255139 (0.054618) | 0.340188 / 0.283200 (0.056988) | 0.020440 / 0.141683 (-0.121243) | 1.486886 / 1.452155 (0.034732) | 1.548330 / 1.492716 (0.055614) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.188658 / 0.018006 (0.170652) | 0.422204 / 0.000490 (0.421715) | 0.003508 / 0.000200 (0.003308) | 0.000068 / 0.000054 (0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025173 / 0.037411 (-0.012238) | 0.072868 / 0.014526 (0.058343) | 0.084817 / 0.176557 (-0.091739) | 0.151667 / 0.737135 (-0.585468) | 0.085632 / 0.296338 (-0.210706) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.400998 / 0.215209 (0.185789) | 4.022274 / 2.077655 (1.944619) | 2.025768 / 1.504120 (0.521648) | 1.874193 / 1.541195 (0.332998) | 2.006537 / 1.468490 (0.538047) | 0.501799 / 4.584777 (-4.082978) | 2.987487 / 3.745712 (-0.758225) | 4.552295 / 5.269862 (-0.717566) | 2.775859 / 4.565676 (-1.789817) | 0.057596 / 0.424275 (-0.366679) | 0.006449 / 0.007607 (-0.001158) | 0.470776 / 0.226044 (0.244732) | 4.725933 / 2.268929 (2.457005) | 2.480130 / 55.444624 (-52.964494) | 2.183919 / 6.876477 (-4.692558) | 2.408052 / 2.142072 (0.265979) | 0.584038 / 4.805227 (-4.221190) | 0.124964 / 6.500664 (-6.375701) | 0.060939 / 0.075469 (-0.014530) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.221263 / 1.841788 (-0.620524) | 18.326372 / 8.074308 (10.252064) | 13.398937 / 10.191392 (3.207545) | 0.149153 / 0.680424 (-0.531271) | 0.016941 / 0.534201 (-0.517260) | 0.332106 / 0.579283 (-0.247177) | 0.339958 / 0.434364 (-0.094406) | 0.378125 / 0.540337 (-0.162212) | 0.517787 / 1.386936 (-0.869149) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005927 / 0.011353 (-0.005426) | 0.003607 / 0.011008 (-0.007402) | 0.062925 / 0.038508 (0.024417) | 0.058676 / 0.023109 (0.035566) | 0.362129 / 0.275898 (0.086231) | 0.395864 / 0.323480 (0.072384) | 0.004652 / 0.007986 (-0.003334) | 0.002893 / 0.004328 (-0.001435) | 0.062696 / 0.004250 (0.058445) | 0.049988 / 0.037052 (0.012935) | 0.365366 / 0.258489 (0.106877) | 0.412326 / 0.293841 (0.118485) | 0.027118 / 0.128546 (-0.101429) | 0.008179 / 0.075646 (-0.067467) | 0.068048 / 0.419271 (-0.351223) | 0.041065 / 0.043533 (-0.002468) | 0.359858 / 0.255139 (0.104719) | 0.386589 / 0.283200 (0.103390) | 0.020467 / 0.141683 (-0.121216) | 1.438070 / 1.452155 (-0.014084) | 1.479617 / 1.492716 (-0.013099) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.231516 / 0.018006 (0.213510) | 0.413407 / 0.000490 (0.412917) | 0.000358 / 0.000200 (0.000158) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026071 / 0.037411 (-0.011340) | 0.076486 / 0.014526 (0.061960) | 0.085943 / 0.176557 (-0.090613) | 0.138087 / 0.737135 (-0.599048) | 0.087466 / 0.296338 (-0.208872) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.417711 / 0.215209 (0.202502) | 4.171915 / 2.077655 (2.094260) | 2.140677 / 1.504120 (0.636557) | 1.960164 / 1.541195 (0.418969) | 2.002134 / 1.468490 (0.533644) | 0.499699 / 4.584777 (-4.085078) | 2.991814 / 3.745712 (-0.753898) | 2.906589 / 5.269862 (-2.363272) | 1.842305 / 4.565676 (-2.723372) | 0.057633 / 0.424275 (-0.366642) | 0.006465 / 0.007607 (-0.001142) | 0.492874 / 0.226044 (0.266830) | 4.931613 / 2.268929 (2.662684) | 2.623161 / 55.444624 (-52.821463) | 2.310624 / 6.876477 (-4.565853) | 2.483146 / 2.142072 (0.341074) | 0.586910 / 4.805227 (-4.218317) | 0.124681 / 6.500664 (-6.375983) | 0.061561 / 0.075469 (-0.013908) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.319111 / 1.841788 (-0.522677) | 18.637326 / 8.074308 (10.563018) | 13.803912 / 10.191392 (3.612520) | 0.143989 / 0.680424 (-0.536435) | 0.017025 / 0.534201 (-0.517176) | 0.333156 / 0.579283 (-0.246127) | 0.342163 / 0.434364 (-0.092201) | 0.380357 / 0.540337 (-0.159981) | 0.512261 / 1.386936 (-0.874675) |

![](https://cml.dev/watermark.png#f49a16346dc35e5eabeec39778d0f2e4e850dfd7 "CML watermark")

will solve https://github.com/huggingface/datasets/issues/5209 and https://github.com/huggingface/datasets/issues/5151 and many other...

Config parameters for packaged builders are parsed from `“builder_config”` field in README.md file (separate firs-level field, not part of “dataset_info”), example:
```yaml
---
dataset_info:
...
configs:
  - config_name: v1
    data_dir: v1
    drop_labels: true
  - config_name: v2
    data_dir: v2
    drop_labels: false
```

I tried to align packaged builders with custom configs parsed from metadata with scripts dataset builder as much as possible. Their builders are created dynamically (see `configure_builder_class()` in load.py`) and have `BUILDER_CONFIGS` attribute filled with `BuilderConfig` objects in the same way as for datasets with script.

## load_dataset
1. If there is single config in meta and it doesn’t have a name, the name becomes “default” (as we do for “dataset_info”), [example](https://huggingface.co/datasets/polinaeterna/audiofolder_one_default_config_in_metadata/blob/main/README.md):
```python
load_dataset("ds") == load_dataset("ds", "default")   # load with the params provided in metadata

load_dataset("ds", "random name")  # ValueError: BuilderConfig 'random_name' not found. Available: ['default']
```
2. If there is single config in metadata with `config_name` provided, it becomes a default one (loaded when no `config_name` is specified, [example](https://huggingface.co/datasets/polinaeterna/audiofolder_one_nondefault_config_in_metadata)
```python

load_dataset("ds") == load_dataset("ds", "custom")   # load with the params provided in meta

load_dataset("ds", "random name")  # ValueError: BuilderConfig 'random_name' not found. Available: ['custom']
```
3. If there are several configs in metadata with names [example](https://huggingface.co/datasets/polinaeterna/audiofolder_two_configs_in_metadata/blob/main/README.md)
```python
load_dataset("ds", "v1")  # load with "v1" params

load_dataset("ds", "v2")  # load with "v2" params

load_dataset("ds")  # ValueError: BuilderConfig 'default' not found. Available: ['v1', 'v2']

```
Thanks to @lhoestq and [this change](https://github.com/polinaeterna/datasets/pull/1), it's possible to add `"default"` field in yaml and set it to True, to make the config a default one (loaded when no config is specified):
```yaml
configs:
  - config_name: v1
    drop_labels: true
    default: true
  - config_name: v2
...
```
then `load_dataset("ds") == load_dataset("ds", "v1")`.

## dataset_name and cache
I decided that it’s reasonable to add a `dataset_name` attribute to `DatasetBuilder` class which would be equal to `name` for scripts dataset but reflect a real dataset name for packaged builders (last part of path/name from hub). This is mostly to reorganize cache structure (I believe we can do this in the major release?) because otherwise, with custom configs for packaged builders which were all stored in the same directory, i it was becoming a mess. And in general it makes much more sense like this, from datasets server perspective too, though it’s a breaking change

So the cache dir has the following structure: `"{namespace__}<dataset_name>/<config_name>/<version>/<hash>/"` and arrow/parquet filenames are also `"<dataset_name>-<split>.arrow"`.
For example `polinaeterna___audiofolder_two_configs_in_metadata/v1-5532fac9443ea252/0.0.0/6cbdd16f8688354c63b4e2a36e1585d05de285023ee6443ffd71c4182055c0fc/` for `polinaeterna/audiofolder_two_configs_in_metadata` Hub dataset, train arrow file is `audiofolder_two_configs_in_metadata-train.arrow`.

For script datasets it remains unchanged.

## push_to_hub
To support custom configs with `push_to_hub`, the data is put under directory named either as `<config_name>` if `config_name` is **not** "default" or "data" if `config_name` is omitted or "default" (for backward compatibility). `"builder_config"` field is added to README.md, with `config_name` (optional) and `data_files` fields. for `"data_files"`, `"pattern"` parameter is introduced, to resolve data files correctly, see https://github.com/polinaeterna/datasets/pull/1.
 - `ds.push_to_hub("ds")` --> one config ("default"), put under "data" directory, [example](https://huggingface.co/datasets/polinaeterna/push_to_hub_single_config/blob/main/README.md)
```yaml
dataset_info:
...
configs:
  data_files:
  - split: train
    pattern: data/train-*
...
```
- `ds.push_to_hub("ds", "custom")` --> put under "custom" directory, [example](https://huggingface.co/datasets/polinaeterna/push_to_hub_singe_nondefault_config/blob/main/README.md)

```yaml
configs:
  config_name: custom
  data_files:
    - split: train
      path: custom/train-*
...
```

 - for many configs, [example](https://huggingface.co/datasets/polinaeterna/push_to_hub_many_configs/blob/main/README.md):
```yaml
configs:
  - config_name: v1
    data_files:
      - split: train
        path: v1/train-*
...
  - config_name: v2
    data_files:
      - split: train
        path: v2/train-*
...
```
Thanks to @lhoestq and https://github.com/polinaeterna/datasets/pull/1, when pushing to datasets created **before** this change, README.md is updated accordingly (config for old data is added along with the one that is being pushed).

`"dataset_info"` yaml field is updated accordingly (new configs are added).
This shouldn't break anything!

TODO in separate PRs:
- [x] docs
- [ ] probably update test cli util (make --save_info not rewrite `builder_config` in readme)


Clarify imagefolder is for small datasets

_The documentation is not available anymore as the PR was closed or merged._

I think it's also reasonable to add the same note to the AudioFolder decription

Thank you ! I think "regular" is more appropriate than "small". It can easily scale to a few thousands of images - just not millions x)

Replaced "small" with "several thousand" since what is considered "regular" and even "small" can be kind of vague!

Based on feedback from [here](https://github.com/huggingface/datasets/issues/5317#issuecomment-1334108824), this PR adds a note to the `imagefolder` loading and creating docs that `imagefolder` is designed for small scale image datasets.


Fix docs building for main

EDIT
At least the docs for ~~main~~ PR branch are now built:
- https://github.com/huggingface/datasets/actions/runs/3594847760/jobs/6053620813

Build documentation for main branch was triggered after this PR being merged: https://github.com/huggingface/datasets/actions/runs/3603370082/jobs/6071482470

This PR reverts the triggering event for building documentation introduced by:
- #5250

Fix #5326.


Avoid unwanted behaviour when splits from script and metadata are not matching because of outdated metadata

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5327). All of your documentation changes will be reflected on that endpoint.

will fix  #5315


No documentation for main branch is built

bug

Something isn't working

Since:
- #5250
  - Commit: 703b84311f4ead83c7f79639f2dfa739295f0be6

the docs for main branch are no longer built.

The change introduced only triggers the docs building for releases.


map(...batch_size=None) for IterableDataset

enhancement

New feature or request

good first issue

Good for newcomers

Hi! I agree it makes sense for `IterableDataset.map` to support the `batch_size=None` case. This should be super easy to fix.

@mariosasko as this is something simple maybe I can include it as part of https://github.com/huggingface/datasets/pull/5311? Let me know :+1:

Feel free to close this @lhoestq as part of https://github.com/huggingface/datasets/pull/5336 :hugs:

Thanks again :)

> For practical usages, an alternative to this would be to convert from an iterable dataset to a map-style dataset, but it is not obvious how to do this.

This is interesting as well, if anyone wants to explore

### Feature request

Dataset.map(...) allows batch_size to be None. It would be nice if IterableDataset did too.

### Motivation

Although it may seem a bit of a spurious request given that `IterableDataset` is meant for larger than memory datasets, but there are a couple of reasons why this might be nice.

One is that load_dataset(...) can return either IterableDataset or Dataset. mypy will then complain if batch_size=None even if we know it is Dataset. Of course we can do:

    assert isinstance(d, datasets.DatasetDict)

But it is a mild inconvenience. What's more annoying is that whenever we use something like e.g. `combine_datasets(...)`, we end up with the union again, and so have to do the assert again.

Another is that we could actually end up with an IterableDataset small enough for memory in normal/correct usage, e.g. by filtering a massive IterableDataset.

For practical usages, an alternative to this would be to convert from an iterable dataset to a map-style dataset, but it is not obvious how to do this.

### Your contribution

Not this time.


Fix docstrings and types in documentation that appears on the website

I agree we have a mess with docstrings...

Ok, I believe we've cleaned up most of the old syntax we were using for the user-facing docs! There are still a couple of `:obj:`'s and `:class:` floating around in the docstrings we don't expose that I'll track down :)

Hi @polinaeterna @albertvillanova @stevhliu,  I hope you all are doing well.

Is this issue still unresolved as I am interested in it?

It should be mostly fixed for the user-facing APIs, but there may be some Sphinx syntax still lurking around in the non-public APIs. Feel free to open a PR to fix those if you catch any! 🤗

Thanks for your reply @stevhliu  :)
Sure, I will try to find out the remaining and fix that.

While I was working on https://github.com/huggingface/datasets/pull/5313 I've noticed that we have a mess in how we annotate types and format args and return values in the code. And some of it is displayed in the [Reference section](https://huggingface.co/docs/datasets/package_reference/builder_classes) of the documentation on the website.

Would be nice someday, maybe before releasing datasets 3.0.0, to unify it......