Add video feature

New feature or request

help wanted

Extra attention is needed

vision

Vision datasets

@NielsRogge @rwightman may have additional requirements regarding this feature.

When adding a new (decodable) type, the hardest part is choosing the right decoding library. What I mean by "right" here is that it has all the features we need and is easy to install (with GPU support?).

Some candidates/options:
* [`decord`](https://github.com/dmlc/decord): no longer [maintained](https://github.com/dmlc/decord/issues/214), not trivial to install with GPU support
* [`pyAV`](https://github.com/PyAV-Org/PyAV):  used for CPU decoding in `torchvision`, GPU decoding not supported if I'm not mistaken, otherwise the best candidate probably
* [`video_reader`](https://github.com/pytorch/vision/blob/de350bc01ad2193ea2888f0ce8a6a346d3cba5a9/torchvision/csrc/io/video_reader/video_reader.cpp): used for GPU decoding in `torchvision`, depends on `torch'
* OpenCV: uses `ffmpeg` for video decoding under the hood
* ...

And the last resort is building our own library, which is the most flexible solution but also requires the most work.

PS: I'm adding a link to an article that compares various video decoding libraries: https://towardsdatascience.com/lightning-fast-video-reading-in-python-c1438771c4e6

@mariosasko is GPU decoding a hard requirement here? Do we really need it? (I don't know)

Something to consider with `decord` is that it doesn't (AFAIK) support writing videos, so you'd still need something else for that. also I've noticed [issues](https://github.com/dmlc/decord/issues/242) with decord's ability to decode stereo audio streams along side the video (which you don't run into with PyAV).

---

I think PyAV should be able to do the job just fine to start. If we write the video io utilities as their own functions, we can hot swap them later if we find/write a different solution that's faster/better.

Video is still a bit of a mess, but I'd say pyAV is likely the best approach (or supporting all three via pytorchvideo, but that adds a middle man dependency).

Being able to decode on the GPU, into memory that could be passed off to a Tensor in whatever framework is being used would be the dream, I don't think there is any interop of that nature working right now. Number of decoder instances per GPU is limited so it's not clear if balancing load btw GPU decoders and CPUs would be needed in say large scale video training.

Any of these solutions is less than ideal due to the nature of video, having a simple Python interface video / start -> end results in lots of extra memory (you need to decode whole range of the clips into a buffer before using anything). Any scalable video system would be streaming on the fly (issuing frames via callbacks as soon as the stream is far enough along to have re-ordered the frames and synced audio+video+other metadata (sensors, CC, etc).

For standalone usage, decoding on GPU could be ideal but isn't async processing of inputs on CPUs while letting the accelerator busy for training the de-facto? Of course, I am aware of other advanced mechanisms such as CPU offloading, but I think my point is conveyed.

Here's a minimal implementation of the helper functions we'd need from PyAV, a lot of which I borrowed from `pytorchvideo`, stripping out the `torch` specific stuff:

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/gist/nateraw/c327cb6ff6b074e6ddc8068d19c0367d/pyav-io.ipynb)

It's not too much code...@mariosasko we could probably just maintain these helper fns within the `datasets` library, right?

Also wanted to note I added a PR for video classification in `transformers` here, which uses `decord`. It's still open...should we make a decision now to align the libraries we are using between `datasets` and `transformers`? (CC @Narsil )

https://github.com/huggingface/transformers/pull/20151

Fully agree on at least trying to unite things.

Making clear function boundaries to help us change dependency if needed seems like a good idea since there doesn't seem to be a clear winner.

I also happen to like directly calling ffmpeg. For some reason it was a lot faster than pyav.

### Feature request

Add a `Video` feature to the library so folks can include videos in their datasets.

### Motivation

Being able to load Video data would be quite helpful. However, there are some challenges when it comes to videos:

1. Videos, unlike images, can end up being extremely large files
2. Often times when training video models, you need to do some very specific sampling. Videos might end up needing to be broken down into X number of clips used for training/inference
3. Videos have an additional audio stream, which must be accounted for
4. The feature needs to be able to encode/decode videos (with right video settings) from bytes.

### Your contribution

I did work on this a while back in [this (now closed) PR](https://github.com/huggingface/datasets/pull/4532). It used a library I made called [encoded_video](https://github.com/nateraw/encoded-video), which is basically the utils from [pytorchvideo](https://github.com/facebookresearch/pytorchvideo), but without the `torch` dep. It included the ability to read/write from bytes, as we need to do here. We don't want to be using a sketchy library that I made as a dependency in this repo, though.

Would love to use this issue as a place to:
- brainstorm ideas on how to do this right
- list ways/examples to work around it for now

CC @sayakpaul @mariosasko @fcakyon


Seems to freeze when loading audio dataset with wav files from local folder

I just tried to do the same but changing the `.wav` files to `.mp3` files and that doesn't fix it.

I don't know if anyone will ever read this but I've tried to upload the same dataset with google colab and the output seems more clarifying. I didn't specify the train/test split so the dataset wasn't fully uploaded (or that is what I understood, might be wrong!!).

Now, including the `drop_metadata` flag I can load the dataset normally (at least with colab notebook):

```python
from datasets import load_dataset

dataset = load_dataset("audiofolder", data_dir="../archive/Dataset", , drop_metadata=True)
```

I'll close the issue.

@uriii3 Hello, I understand correctly that you converted your wav files to mp3?

Yes but it didn't matter. I don't remember which of them I ended up working with.

### Describe the bug

I'm following the instructions in [https://huggingface.co/docs/datasets/audio_load#audiofolder-with-metadata](url) to be able to load a dataset from a local folder.

I have everything into a folder, into a train folder and then the audios and csv. When I try to load the dataset and run from terminal, seems to work but then freezes with no apparent reason.

The metadata.csv file contains a few columns but the important ones, `file_name` with the filename and `transcription` with the transcription are okay.

The audios are `.wav` files, I don't know if that might be the problem (I will proceed to try to change them all to `.mp3` and try again).

### Steps to reproduce the bug

The code I'm using:
```python
from datasets import load_dataset

dataset = load_dataset("audiofolder", data_dir="../archive/Dataset")
dataset[0]["audio"]
```

The output I obtain:
```
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 439/439 [00:00<00:00, 311135.43it/s]
Using custom data configuration default-38d4546ffd010f3e
Downloading and preparing dataset audiofolder/default to /Users/mine/.cache/huggingface/datasets/audiofolder/default-38d4546ffd010f3e/0.0.0/6cbdd16f8688354c63b4e2a36e1585d05de285023ee6443ffd71c4182055c0fc...
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 439/439 [00:00<00:00, 166467.72it/s]
Using custom data configuration default-38d4546ffd010f3e
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 439/439 [00:00<00:00, 187772.74it/s]
Using custom data configuration default-38d4546ffd010f3e
Resolving data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 439/439 [00:00<00:00, 59623.71it/s]
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 439/439 [00:00<00:00, 138090.55it/s]
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 439/439 [00:00<00:00, 106065.64it/s]
Resolving data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 439/439 [00:00<00:00, 56036.38it/s]
Resolving data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 439/439 [00:00<00:00, 74004.24it/s]
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 439/439 [00:00<00:00, 162343.45it/s]
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 439/439 [00:00<00:00, 101881.23it/s]
Using custom data configuration default-38d4546ffd010f3e
Resolving data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 439/439 [00:00<00:00, 60145.67it/s]
Resolving data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 439/439 [00:00<00:00, 80890.02it/s]
Resolving data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 439/439 [00:00<00:00, 54036.67it/s]
Resolving data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 439/439 [00:00<00:00, 95851.09it/s]
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 439/439 [00:00<00:00, 155897.00it/s]
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 439/439 [00:00<00:00, 137656.96it/s]
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 439/439 [00:00<00:00, 131230.81it/s]
Using custom data configuration default-38d4546ffd010f3e
Using custom data configuration default-38d4546ffd010f3e
Using custom data configuration default-38d4546ffd010f3e
Using custom data configuration default-38d4546ffd010f3e
Using custom data configuration default-38d4546ffd010f3e
Using custom data configuration default-38d4546ffd010f3e
Using custom data configuration default-38d4546ffd010f3e
Using custom data configuration default-38d4546ffd010f3e
Using custom data configuration default-38d4546ffd010f3e
Using custom data configuration default-38d4546ffd010f3e
Using custom data configuration default-38d4546ffd010f3e
Using custom data configuration default-38d4546ffd010f3e
Using custom data configuration default-38d4546ffd010f3e
```
And then here it just freezes and nothing more happens.

### Expected behavior

Load the dataset.

### Environment info

Datasets version:
datasets                  2.6.1                    pypi_0    pypi


Add SQL guide

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5223). All of your documentation changes will be reflected on that endpoint.

I think we may want more content on this page that's not SQL related. Some of that content probably already lives in the main `load` docs page, but might be bad to remove major things like csv/pandas from there...WDYT we should do @lhoestq ?

Maybe the main load page can only show one example and redirect to this page for more details ?

We can do the same for pandas stuff: have one example in load, and redirect to this page for more details

This PR adapts @nateraw's awesome SQL notebook as a guide for the docs!


HuggingFace website is incorrectly reporting that my datasets are pickled

cc @McPatate maybe you know what's happening ?

Yes I think I know what is happening. We check in zips for pickles, and the UI must display the pickle jar when a scan has an associated list of imports, even when empty.
~I'll fix ASAP !~

>  I'll fix ASAP !

Actually I'd rather leave it like that for now, as it indicates that we checked for pickles and nothing dangerous appeared :)

Closing the issue with the typical "feature not a bug"

### Describe the bug

HuggingFace is incorrectly reporting that my datasets are pickled. They are not picked, they are simple ZIP files containing PNG images.

Hopefully this is the right location to report this bug.

### Steps to reproduce the bug

Inspect my dataset respository here: https://huggingface.co/datasets/ProGamerGov/StableDiffusion-v1-5-Regularization-Images

### Expected behavior

They should not be reported as being pickled.

### Environment info

N/A


Cannot push

Did you run `huggingface-cli lfs-enable-largefiles` before committing or before adding ? Maybe you can try before adding

Anyway I'd encourage you to split your data into several TAR archives if possible, this way the dataset can loaded faster using multiprocessing (by giving each process a subset of shards to process)

@lhoestq
Thanks for the help!
> Maybe you can try before adding

It did not help

But I totally got your point about split into multiple TAR archives. It really helped!

### Describe the bug

I am facing the issue when I try to push the tar.gz file around 11G to HUB.

```
(venv) ╭─laptop@laptop ~/PersonalProjects/data/ulaanbal_v0 ‹main●›
╰─$ du -sh *
4.0K	README.md
 13G	data
516K	test.jsonl
 18M	train.jsonl
4.0K	ulaanbal_v0.py
 11G	ulaanbal_v0.tar.gz
452K	validation.jsonl

(venv) ╭─laptop@laptop~/PersonalProjects/data/ulaanbal_v0 ‹main●›
╰─$ git add ulaanbal_v0.tar.gz && git commit -m 'large version'

(venv) ╭─laptop@laptop ~/PersonalProjects/data/ulaanbal_v0 ‹main●›
╰─$ git push
EOFoading LFS objects:   0% (0/1), 0 B | 0 B/s
Uploading LFS objects:   0% (0/1), 0 B | 0 B/s, done.
error: failed to push some refs to 'https://huggingface.co/datasets/bayartsogt/ulaanbal_v0'
```

I have already tried pushing a small version of this and it was working fine. So my guess it is probably because of the big file.

Following I run before the commit:
```
╰─$ git lfs install
╰─$ huggingface-cli lfs-enable-largefiles .
```

### Steps to reproduce the bug

Create a private dataset on huggingface and push 12G tar.gz file

### Expected behavior

To be pushed with no issue

### Environment info

- `datasets` version: 2.6.1
- Platform: Darwin-21.6.0-x86_64-i386-64bit
- Python version: 3.7.11
- PyArrow version: 10.0.0
- Pandas version: 1.3.5


Implicit type conversion of lists in to_pandas

I think this behavior comes from PyArrow:
```python
import pyarrow as pa
t = pa.table({"a": [[0]]})
t.to_pandas().a.values[0]
# array([0])
```

I believe this has to do with zero-copy: you can get a pandas DataFrame without copying the buffers from arrow, and therefore end up with numpy arrays.

That's interesting, I guess not much to do here then.

### Describe the bug

```
ds = Dataset.from_list([{'a':[1,2,3]}])
ds.to_pandas().a.values[0]
```
Results in `array([1, 2, 3])`  -- a rather unexpected conversion of types which made downstream tools expecting lists not happy.

### Steps to reproduce the bug

See snippet

### Expected behavior

Keep the original type

### Environment info

datasets 2.6.1
python 3.8.10


Delta Tables usage using Datasets Library

enhancement

New feature or request

Hi ! Interesting :) Can you provide concrete examples of cases where it can be useful ?

Few example blogs and posts that might help on this -

1. https://hevodata.com/learn/databricks-delta-tables/
2. https://docs.databricks.com/delta/index.html

Basically, we are looking at utility of Datasets library with Delta Lake Tables.

`datasets` can already read/write from parquet from/to a cloud storage using fsspec, if I understand correctly it's should be possible to load parquet files as delat lake tables no ? :) Or is there someting missing ?

@lhoestq Per my understanding, delta lake table is a bunch of paruqet files together with the meta to support ACID. For example file 1 contains v0.1 of record A while file 2 contains v0.2 of record A. I am assuming the Hugging face dataset would delegate the read/write delta table to 3rd party lib, maybe pyarrow. Correct me if I was wrong @reichenbch

And I am assuming, people are asking the versioning of Hugging face datasets. But I am assuming Hugging face delegate this function to github and it is not the key requirement for Public Data set. It actually the key function of ML Ops, I am not sure whether hugging face would like expand to that area.

### Feature request

Adding compatibility of Datasets library with Delta Format. Elevating the utilities of Datasets library from Machine Learning Scope to Data Engineering Scope as well.

### Motivation

We know datasets library can absorb csv, json, parquet, etc. file formats but it would be great if Datasets library could work with Delta Tables (with delta format) as it has different features such as time travelling, layout optimization, query performance, aids in Data Engineering.

This will help and enhance Datasets library from Machine Learning utility to Data Engineering utilities and expand horizons thereafter. I am totally using Datasets library in all my usecases and as my role expands so does the work, compatibility with Datasets library is something I don't want to lose.

### Your contribution

Would love to work on this feature, even if this has to picked up from scratch, including design paradigms and patterns.
I have basic idea about Delta Live Tables, would brush it easily for this feature.


New feature or request


Reword E2E training and inference tips in the vision guides

_The documentation is not available anymore as the PR was closed or merged._

Reference: https://github.com/huggingface/datasets/pull/5188#discussion_r1012148730


save_elasticsearch_index

Hi ! I think there exist tools to dump and reload an index in your elastic search but I'm not super familiar with it.

Anyway after reloading an index in elastic search you can call `ds.load_elasticsearch_index` which will connect the index to the dataset without re-indexing

Hi,

I am new to Dataset and elasticsearch. I was wondering is there any equivalent approach to save elasticsearch index as of save_faiss_index locally for later use, to remove the need to re-index a dataset?


Update github pr docs actions

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5214). All of your documentation changes will be reflected on that endpoint.


Add support for different configs with `push_to_hub`

New feature or request

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5213). All of your documentation changes will be reflected on that endpoint.

Nice thanks !

Would it be possible to have the new folders at the same level as "data" ? This way they're all separated
```
├─ config-v1/
│  ├── train-00000-00002-...-.parquet
│  └── train-00001-00002-...-.parquet
└ config-v2/
    ├── train-00000-00002-...-.parquet
    └── train-00001-00002-...-.parquet
```
and if you don't provide a config name, it goes in a folder named "default" instead, that would be loaded by default.

We could also write in the YAML something like
```yaml
configs:
- name: config-v1
  data_dir: config-v1
- name: config-v2
  data_dir: config-v2
```
and loading `config-v1` would be equivalent to run `load_dataset(ds_name, "config-v1", data_dir="config-v1")`

Do you think it would make sense ?

For backward compatibility we can just keep the "data/*" pattern. It's ok to expect users to have an updated version of `datasets` to be able to load datasets with configurations.

@lhoestq thank you for the feedback! i'll reflect on this on Moday, my mind just melted because of the fever.

@mariosasko  @albertvillanova what do you think?

Thanks for addressing this, @polinaeterna. It is good:
- we support configs for datasets without scripts
- we align the behavior to datasets with scripts as much as possible

Maybe adding some tests will help clarify what is the expected behavior...

After some discussion with @lhoestq we decided that it's better to rely on metadata file than on data files patterns.

So we decided to introduce a new field to yaml (like `configs` or smth like that) that would contain arbitrary configs kwargs to be passed to loader, including `data_dir` and `data_files`.
This is more aligned with datasets with custom scripts where we explicitly write all the supported configs and config parameters in the code and is extendable to all packaged modules.
This would solve https://github.com/huggingface/datasets/issues/5209

(@lhoestq was right 21 days ago, this is a more general solution idk why i ignored this...)

closed in favor of https://github.com/huggingface/datasets/pull/5331

will solve  #5151

@lhoestq @albertvillanova @mariosasko
This is still a super draft so please ignore code issues but I want to discuss some conceptually important things.

I suggest a way to do `.push_to_hub("repo_id", "config_name")` with pushing parquet files to directories named as `config_name` (inside `data/` dir as it is now), for example:
```
data
   |__config-v1
        train-00000-00002-...-.parquet
        train-00001-00002-...-.parquet
        ...
   |__config-v2
        ....
```

When loading a dataset, I parse these configs from repository data files (only for `"data/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*"` pattern that is used for parquet datasets pushed with `.push_to_hub`).
Therefore,
- when user tries to load a dataset that has configs parsed from data files dir names without providing a config (like `load_dataset("repo")` instead of `load_dataset("repo", "config-v1")`) - raise error and asks for config - to be aligned with how it works in datasets with scripts.
- for backward compatibility: if user tries to `.push_to_hub(""repo", "config_name")` to an existing parquet repo with no configurations (all parquet files are directly in `data/` dir) - raise error. My initial idea was to raise a warning and move these files to another dir with name (config) like "default" or smth but in a PR and suggest user to merge it on the Hub. But there is no support for renaming (moving) files via `HfApi` yet so it would require deleting and pushing again if I understand it right.

This parsing approach can be extended to other Hub packaged modules, and to local packaged modules and other data files patterns
(except for cases when splits are in dir names `KEYWORDS_IN_DIR_NAME_BASE_PATTERNS` because we allow for arbitrary depth of directory hierarchy).
Do you think it's reasonable? Not sure how to provide flexibility (and backward compatibility) to not parsing configs and load all the data in a single config as it is now.

I also thought about getting information about configs from Readme.md `dataset_info` ([example](https://huggingface.co/datasets/polinaeterna/test_push_two_configs/blob/main/README.md)). But that way we
are dependent on if it exists. It is created automatically with `.push_to_hub` but what if it is
accidentally deleted or smth).

Also, what I don't like is that this parsing is a part of Module/DataFiles logic, not Builder's one, which is not aligned with datasets with custom scripts. But I don't know to implement the second approach in current library's logic.

 What do you think about this all? Am I missing smth?

TODO:

- [ ] save cache in the same dir for configs of the same datasets
- [ ] fix verification errors
- [ ] correctly update `dataset_infos.json` too
- [ ] ...


Fix CI require_beam maximum compatible dill version

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5212). All of your documentation changes will be reflected on that endpoint.

A previous commit to main branch introduced an additional requirement on maximum compatible `dill` version with `apache-beam` in our CI `require_beam`:
- d7c942228b8dcf4de64b00a3053dce59b335f618
- ec222b220b79f10c8d7b015769f0999b15959feb

This PR fixes the maximum compatible `dill` version with `apache-beam`, which is <0.3.2 (and not 0.3.6): https://github.com/apache/beam/blob/v2.42.0/sdks/python/setup.py#L219


Update Overview.ipynb google colab

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5211). All of your documentation changes will be reflected on that endpoint.

- removed metrics stuff
- added image example
- added audio example (with ffmpeg instructions)
- updated the "add a new dataset" section


Tweak readme

Nit: We should also update the `Disclaimers` section to let the dataset owners know they should use Hub discussions rather than GH issues for removal requests/updates

Updated the disclaimers section, thanks !

Does it sound good to you @albertvillanova ?

Tweaked some paragraphs mentioning the modalities we support + added a paragraph on security


Implement ability to define splits in metadata section of dataset card

New feature or request

@merveenoyan Do you want different files to be splits or configurations?

From [what you specified in `Readme.md`](https://huggingface.co/datasets/inria-soda/tabular-benchmark/commit/fb4575853772c62a20203bdd6cc0202f5db4ce4e) I hypothesize that you want to have 4 **configs** corresponding to directories: `"clf_cat", "clf_num", "reg_cat", "reg_num"`. And inside each config you require to have as many splits as there are `csv` files
so if you run
```python
load_dataset("inria-soda/tabular-benchmark", "clf_cat", split="compass")
```
you will generate the data only from `compass.csv` file.
In this case, running `load_dataset("inria-soda/tabular-benchmark", "clf_cat"`) without split parameter will return `DatasetDict` object with `"KDDCup09_upselling", "cat_compass", "cat_covertype", ... "road_safety"` keys (which values are splits - `Dataset` objects)

**or**
do you want each file to be a separate config? Like:
```python
load_dataset("inria-soda/tabular-benchmark", "clf_cat_compass")   # returns DatasetDict with a single "train" split
```
**or**
maybe smth completely different? :smile:

Anyway, now I have an impression that this is probably rather a matter of automatically inferring configs from repository structure rather than providing parameters in metadata yaml.

@polinaeterna I want the latter where you can think of every CSV file as a config, like MNLI from GLUE.

@merveenoyan @lhoestq I see two solutions to this case.
1. Parse configurations automatically from directories names. That is, if you have data structure like:
```
tabular-benchmark
  └─clf_cat_compass
      └─compass.csv
  └─clf_cat_cat_covertype
      └─covertype.csv
 ...
  └─reg_cat_house_sales
      └─house_sales.csv
```
you'll get  "clf_cat_compass", "clf_cat_cat_covertype", ... "reg_cat_house_sales" configurations that would contain **only files from corresponding directories**.
**\+** this is a requested change and needed in general and would solve other problems, see https://github.com/huggingface/datasets/issues/4578, would also help with https://github.com/huggingface/datasets/pull/5213 which I'm working on currently
**\+** would allow users to do just `load_dataset(“inria-soda/tabular-benchmark”, “clf_cat_compass”)`, no `data_files` param required
**\-** in this specific case it would require restructuring of the data - putting each file in a directory named as a config name (to me personally it doesn't seem to be a big deal)

2. More or less what we discussed before - add support for manually specifying parameters in the metadata. We can add new metadata yaml field (say, `"custom_configs_info"`), so that we can provide smth like:
```yaml
---
...
dataset_info:
 ...
custom_configs_info:
- config_name: reg_cat_house_sales
 data_files:
 - reg_cat/house_sales.csv
- config_name: clf_cat_compass
 data_files:
 - clf_cat/compass.csv
...
---
```
**\+** Would be useful not only for tabular data and not only for `data_files` parameter - any packaged dataset’s viewer can be customized to use specific, non-default parameters. @merveenoyan do you maybe have any other examples/use cases in mind where you want to provide any specific parameters to the viewer?
**\-** I'm not sure here but assume that it might require changes in interaction with the viewer on the hub side - to parse these configurations, as they not default configurations (not in `BUILDER_CONFIGS` list). cc @severo But probably this can be solved on the `datasets` side too.

Overall, I would start from implementing the first solution since it's related to what I'm doing now and is super useful for `datasets` in general. And then  if we agree that having more flexibility in providing parameters to the viewer is required, I can implement the second one. Let me know what you think :)

> We can add new metadata yaml field (say, "custom_configs_info"), so that we can provide smth like:

Love it ! Some other ideas to name the "custom_configs_info" field: "configs", "parameters", "config_args", "configurations"

> it might require changes in interaction with the viewer on the hub side - to parse these configurations, as they not default configurations (not in BUILDER_CONFIGS list)

If we update the `get_dataset_config_names()` function in `datasets` in inspect.py we should be fine - that's what the viewer is using

> Overall, I would start from implementing the first solution since it's related to what I'm doing now and is super useful for datasets in general. And then if we agree that having more flexibility in providing parameters to the viewer is required, I can implement the second one. Let me know what you think :)

Actually I feel like the second solution includes the first use case you mentioned. If you implement the second solution, then users would just have to add a few lines of YAML and their directories would be considered configurations no ? Maybe there's no need to implement two different logics to do the same thing

@merveenoyan I haven't started working on this yet, working on adding configs to packaged datasets instead: https://github.com/huggingface/datasets/pull/5213 because this both would allow you to solve your issue and is a frequently requested feature.

adding arbitrary parameters to yaml would be my next task i think!

@merveenoyan ignore my comment above, I'm switching to this task now :D

I want to be able to create folders in a model.

### Feature request

If you go here: https://huggingface.co/datasets/inria-soda/tabular-benchmark/tree/main you will see bunch of folders that has various CSV files. I’d like dataset viewer to show these files instead of only one dataset like it currently does. (and also people to be able to load them as splits instead of loading through `data_files`)
e.g GLUE has various splits on viewer but it’s too overkill to ask people to implement loading script, so it would be better to let them define these in the README file instead.

Also pinging @polinaeterna @lhoestq @adrinjalali


Refactor CI hub fixtures to use monkeypatch instead of patch

_The documentation is not available anymore as the PR was closed or merged._

Minor refactoring of CI to use `pytest` `monkeypatch` instead of `unittest` `patch`.


Connection error of the HuggingFace's dataset Hub due to SSLError with proxy

Thanks for your reply. Does this mean that I have to use the `do_dataset `function and the `requests `function to download the dataset from the company's proxy environment?

* Reference:
```bash
### How to load this dataset directly with the [datasets](https://github.com/huggingface/datasets) library

* https://huggingface.co/datasets/moyix/debian_csrc

* from datasets import load_dataset
dataset = load_dataset("moyix/debian_csrc")

### Or just clone the dataset repo

git lfs install
git clone https://huggingface.co/datasets/moyix/debian_csrc
# if you want to clone without large files – just their pointers
# prepend your git clone with the following env var:
GIT_LFS_SKIP_SMUDGE=1
```

You can use `requests` to see if downloading a file from the Hugging Face Hub works. If so, then `datasets` should work as well. If not, then you have to find another way using an internet connection that works

I resolved this issue by applying to "unblock websites" at https://huggingface.com in a corporate network environment with a firewall.

> Hi ! It looks like an issue with your python environment, can you make sure you're able to run GET requests to https://huggingface.co using `requests` in python ?

yes，but still not work

![image](https://github.com/huggingface/datasets/assets/40717349/a502e051-beff-4a5e-aef6-90b525add877)
![image](https://github.com/huggingface/datasets/assets/40717349/88b131fd-ee26-4a8e-9e72-6ecdc126404f)

I read https://github.com/huggingface/datasets/blob/main/src/datasets/load.py, it fail when get the dataset metadata, so download_config has not worked.
```python
            hf_api = HfApi(config.HF_ENDPOINT)
            try:
                dataset_info = hf_api.dataset_info(
                    repo_id=path,
                    revision=revision,
                    token=download_config.token,
                    timeout=100.0,
                )
            except Exception as e:  # noqa catch any exception of hf_hub and consider that the dataset doesn't exist
                if isinstance(
                    e,
                    (
                        OfflineModeIsEnabled,
                        requests.exceptions.ConnectTimeout,
                        requests.exceptions.ConnectionError,
                    ),
                ):
                    raise ConnectionError(f"Couldn't reach '{path}' on the Hub ({type(e).__name__})")
```
I configure the huggingface_hub api, use configure_http_backend
```python
from huggingface_hub import configure_http_backend
def backend_factory() -> requests.Session:
    session = requests.Session()
    session.proxies = proxy
    session.verify = False
    return session

configure_http_backend(backend_factory=backend_factory)
```
It works.

Even tough it does not look like a certificate error in the error message, I had the same error and adding following lines to my code solved my problem.

import os
os.environ['CURL_CA_BUNDLE'] = ''

@kuikuikuizzZ Could you please explain where the configuration code is added?

> Even tough it does not look like a certificate error in the error message, I had the same error and adding following lines to my code solved my problem.
>
> import os os.environ['CURL_CA_BUNDLE'] = ''

Worked for as well!
I faced the issue while submitting jobs through SLURM.

> Even tough it does not look like a certificate error in the error message, I had the same error and adding following lines to my code solved my problem.
>
> import os os.environ['CURL_CA_BUNDLE'] = ''

doesn't work , what does this code mean?

### Describe the bug

It's weird. I could not normally  connect  the dataset Hub  of  HuggingFace due to a SSLError in my office.
Even when I try to connect using my company's proxy address (e.g., http_proxy and https_proxy),
I'm getting the  SSLError issue. What should I do to download the datanet stored in HuggingFace normally?
I welcome any comments. I think those comments will be helpful to me.

* Dataset address - https://huggingface.co/datasets/moyix/debian_csrc/viewer/moyix--debian_csrc
* Log message
```
      ............ OMISSION ..............
Traceback (most recent call last):
  File "/data/home/geunsik-lim/qtlab/./transformers/examples/pytorch/language-modeling/run_clm.py", line 587, in <module>
    main()
  File "/data/home/geunsik-lim/qtlab/./transformers/examples/pytorch/language-modeling/run_clm.py", line 278, in main
    raw_datasets = load_dataset(
  File "/home/geunsik-lim/anaconda3/envs/deepspeed/lib/python3.10/site-packages/datasets/load.py", line 1719, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/geunsik-lim/anaconda3/envs/deepspeed/lib/python3.10/site-packages/datasets/load.py", line 1497, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/home/geunsik-lim/anaconda3/envs/deepspeed/lib/python3.10/site-packages/datasets/load.py", line 1222, in dataset_module_factory
    raise e1 from None
  File "/home/geunsik-lim/anaconda3/envs/deepspeed/lib/python3.10/site-packages/datasets/load.py", line 1179, in dataset_module_factory
    raise ConnectionError(f"Couldn't reach '{path}' on the Hub ({type(e).__name__})")
ConnectionError: Couldn't reach 'moyix/debian_csrc' on the Hub (SSLError)
[2022-11-07 15:23:38,476] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 6760
[2022-11-07 15:23:38,476] [ERROR] [launch.py:324:sigkill_handler] ['/home/geunsik-lim/anaconda3/envs/deepspeed/bin/python', '-u', './transformers/examples/pytorch/language-modeling/run_clm.py', '--local_rank=0', '--model_name_or_path=Salesforce/codegen-350M-multi', '--per_device_train_batch_size=1', '--learning_rate', '2e-5', '--num_train_epochs', '1', '--output_dir=./codegen-350M-finetuned', '--overwrite_output_dir', '--dataset_name', 'moyix/debian_csrc', '--cache_dir', '/data/home/geunsik-lim/.cache', '--tokenizer_name', 'Salesforce/codegen-350M-multi', '--block_size', '2048', '--gradient_accumulation_steps', '32', '--do_train', '--fp16', '--deepspeed', 'ds_config_zero2.json'] exits with return code = 1

real    0m7.742s
user    0m4.930s

```

### Steps to reproduce the bug

Steps to reproduce this behavior.

```
(deepspeed) geunsik-lim@ai02:~/qtlab$ ./test_debian_csrc_dataset.py
Traceback (most recent call last):
  File "/data/home/geunsik-lim/qtlab/./test_debian_csrc_dataset.py", line 6, in <module>
    dataset = load_dataset("moyix/debian_csrc")
  File "/home/geunsik-lim/anaconda3/envs/deepspeed/lib/python3.10/site-packages/datasets/load.py", line 1719, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/geunsik-lim/anaconda3/envs/deepspeed/lib/python3.10/site-packages/datasets/load.py", line 1497, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/home/geunsik-lim/anaconda3/envs/deepspeed/lib/python3.10/site-packages/datasets/load.py", line 1222, in dataset_module_factory
    raise e1 from None
  File "/home/geunsik-lim/anaconda3/envs/deepspeed/lib/python3.10/site-packages/datasets/load.py", line 1179, in dataset_module_factory
    raise ConnectionError(f"Couldn't reach '{path}' on the Hub ({type(e).__name__})")
ConnectionError: Couldn't reach 'moyix/debian_csrc' on the Hub (SSLError)
(deepspeed) geunsik-lim@ai02:~/qtlab$
(deepspeed) geunsik-lim@ai02:~/qtlab$
(deepspeed) geunsik-lim@ai02:~/qtlab$
(deepspeed) geunsik-lim@ai02:~/qtlab$ cat ./test_debian_csrc_dataset.py
#!/usr/bin/env python
from datasets import load_dataset
dataset = load_dataset("moyix/debian_csrc")

```

1.  Adde proxy address of a company in /etc/profile
2.  Download dataset with load_dataset() function of  datasets package that is provided by HuggingFace.
3.  In this case, the address would be "moyix--debian_csrc".
4.  I get the "`ConnectionError: Couldn't reach 'moyix/debian_csrc' on the Hub (SSLError`)" error message.

### Expected behavior

* error message:
ConnectionError: Couldn't reach 'moyix/debian_csrc' on the Hub (SSLError)

### Environment info

* software version information:

```
(deepspeed) geunsik-lim@ai02:~$
(deepspeed) geunsik-lim@ai02:~$ conda list -f pytorch
# packages in environment at /home/geunsik-lim/anaconda3/envs/deepspeed:
#
# Name                    Version                   Build  Channel
pytorch                   1.13.0          py3.10_cuda11.7_cudnn8.5.0_0    pytorch
(deepspeed) geunsik-lim@ai02:~$ conda list -f python
# packages in environment at /home/geunsik-lim/anaconda3/envs/deepspeed:
#
# Name                    Version                   Build  Channel
python                    3.10.6               haa1d7c7_1
(deepspeed) geunsik-lim@ai02:~$ conda list -f datasets
# packages in environment at /home/geunsik-lim/anaconda3/envs/deepspeed:
#
# Name                    Version                   Build  Channel
datasets                  2.6.1                      py_0    huggingface
(deepspeed) geunsik-lim@ai02:~$ uname -a
Linux ai02 5.4.0-131-generic #147-Ubuntu SMP Fri Oct 14 17:07:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
(deepspeed) geunsik-lim@ai02:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.5 LTS"

```


Use logging instead of printing to console

Actually upon closer inspection, it is documented in the code that this behavior is intentional, so I'll close this.

### Describe the bug

Some logs ([here](https://github.com/huggingface/datasets/blob/4a6e1fe2735505efc7e3a3dbd3e1835da0702575/src/datasets/builder.py#L778), [here](https://github.com/huggingface/datasets/blob/4a6e1fe2735505efc7e3a3dbd3e1835da0702575/src/datasets/builder.py#L786), and [here](https://github.com/huggingface/datasets/blob/4a6e1fe2735505efc7e3a3dbd3e1835da0702575/src/datasets/builder.py#L830)) generated by the `DatasetBuilder` are printed to the console instead of passed to `datasets` logger.

### Steps to reproduce the bug

```python
>> import datasets

>> datasets.load_dataset("some-dataset")

Downloading and preparing dataset csv/data to <path>...
Downloading data files: 100%|██████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 7729.06it/s]
Extracting data files: 100%|████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 527.23it/s]
Dataset csv downloaded and prepared to <path>. Subsequent calls will reuse this data.
```

### Expected behavior

The logs should not be printed to the console directly but passed to the logger so that the user can redirect them wherever he wants.

### Environment info

- `datasets` version: 2.6.1
- Platform: macOS-13.0-x86_64-i386-64bit
- Python version: 3.9.15
- PyArrow version: 10.0.0
- Pandas version: 1.5.1


Add missing `DownloadConfig.use_auth_token` value

This PR solves https://github.com/huggingface/datasets/issues/5204

Now the `token` is propagated so that `DownloadConfig.use_auth_token` value is set before trying to download private files from existing datasets in the Hub.


`push_to_hub` not propagating `token` through `DownloadConfig`

@lhoestq can you close this issue as part of the recent #5205 merge? Thanks 🤗

### Describe the bug

When trying to upload a new 🤗 Dataset to the Hub via Python, and providing the `token` as a parameter to the `Dataset.push_to_hub` function, it just works for the first time, assuming that the dataset didn't exist before.

But when trying to run `Dataset.push_to_hub` again over the same dataset, instead of updating it, it throws a `ConnectionError` when trying to retrieve the `README.md` that may contain some metadata about the dataset, so as to also update it, but since the `token` is not propagated, the `DownloadConfig` provided to the `datasets.utils.file_utils.get_from_cache` function doesn't contain the `use_auth_token` value set to `token`, it's just using the default one which is None/False.

So on, when uploading a dataset via Python with `push_to_hub` with the `token` as a parameter with the HuggingFace API Token as value, it can just be uploaded when the dataset is new, otherwise it fails with to `ConnectionError` due to the `token` not being propagated as `use_auth_token`.

### Steps to reproduce the bug

Let's create a new dataset in our HF account via Python as:

```python
from datasets import Dataset

data = {"a": [1, 2, 3], "b": [4, 5, 6]}
ds = Dataset.from_dict(data)
ds.push_to_hub(repo_id=<HF_USERNAME>/<HF_DATASET>, private=private, token=<HF_TOKEN_HERE>)
```

When we create the `Dataset` for the first time it works and there are no issues, but when trying to actually upload a new version of the same dataset (same name under the same username), we encounter the following issue:

```python
from datasets import Dataset

data = {"a": [1, 2, 3], "b": [4, 5, 6]}
ds = Dataset.from_dict(data)
ds.push_to_hub(repo_id=<HF_USERNAME>/<HF_DATASET>, private=private, token=<HF_TOKEN_HERE>)
>>> ConnectionError: Couldn't reach https://huggingface.co/datasets/alvarobartt/demo/resolve/main/README.md (ConnectionError('Unauthorized for URL https://huggingface.co/datasets/<HF_USERNAME>/<HF_DATASET>/resolve/main/README.md. Please use the parameter `use_auth_token=True` after logging in with `huggingface-cli login`'))
```

### Expected behavior

Ideally, the `token` parameter provided to `push_to_hub` should be propagated and used to download the `README.md` when trying to update a `Dataset`, instead of throwing that exception, so that the authentication can be done directly through code without running `huggingface-cli login`as mentioned at https://huggingface.co/docs/datasets/upload_dataset#upload-with-python.

### Environment info

- `datasets` version: 2.6.1
- Platform: macOS-13.0-arm64-arm-64bit
- Python version: 3.10.8
- PyArrow version: 10.0.0
- Pandas version: 1.5.1


Update canonical links to Hub links

This PR updates some of the canonical dataset links to their corresponding links on the Hub; closes #5200.


CI fails after bulk edit of canonical datasets

bug

Something isn't working

```
______ test_get_dataset_config_info[paws-labeled_final-expected_splits2] _______
[gw0] linux -- Python 3.7.15 /opt/hostedtoolcache/Python/3.7.15/x64/bin/python

path = 'paws', config_name = 'labeled_final'
expected_splits = ['train', 'test', 'validation']

    @pytest.mark.parametrize(
        "path, config_name, expected_splits",
        [
            ("squad", "plain_text", ["train", "validation"]),
            ("dalle-mini/wit", "dalle-mini--wit", ["train"]),
            ("paws", "labeled_final", ["train", "test", "validation"]),
        ],
    )
    def test_get_dataset_config_info(path, config_name, expected_splits):
        info = get_dataset_config_info(path, config_name=config_name)
        assert info.config_name == config_name
>       assert list(info.splits.keys()) == expected_splits
E       AssertionError: assert ['test', 'tra... 'validation'] == ['train', 'te... 'validation']
E         At index 0 diff: 'test' != 'train'
E         Full diff:
E         - ['train', 'test', 'validation']
E         + ['test', 'train', 'validation']

tests/test_inspect.py:45: AssertionError
_ test_get_dataset_info[paws-expected_configs2-expected_splits_in_first_config2] _
[gw0] linux -- Python 3.7.15 /opt/hostedtoolcache/Python/3.7.15/x64/bin/python

path = 'paws'
expected_configs = ['labeled_final', 'labeled_swap', 'unlabeled_final']
expected_splits_in_first_config = ['train', 'test', 'validation']

    @pytest.mark.parametrize(
        "path, expected_configs, expected_splits_in_first_config",
        [
            ("squad", ["plain_text"], ["train", "validation"]),
            ("dalle-mini/wit", ["dalle-mini--wit"], ["train"]),
            ("paws", ["labeled_final", "labeled_swap", "unlabeled_final"], ["train", "test", "validation"]),
        ],
    )
    def test_get_dataset_info(path, expected_configs, expected_splits_in_first_config):
        infos = get_dataset_infos(path)
        assert list(infos.keys()) == expected_configs
        expected_config = expected_configs[0]
        assert expected_config in infos
        info = infos[expected_config]
        assert info.config_name == expected_config
>       assert list(info.splits.keys()) == expected_splits_in_first_config
E       AssertionError: assert ['test', 'tra... 'validation'] == ['train', 'te... 'validation']
E         At index 0 diff: 'test' != 'train'
E         Full diff:
E         - ['train', 'test', 'validation']
E         + ['test', 'train', 'validation']

tests/test_inspect.py:90: AssertionError
______ test_get_dataset_split_names[paws-labeled_final-expected_splits2] _______
[gw0] linux -- Python 3.7.15 /opt/hostedtoolcache/Python/3.7.15/x64/bin/python

path = 'paws', expected_config = 'labeled_final'
expected_splits = ['train', 'test', 'validation']

    @pytest.mark.parametrize(
        "path, expected_config, expected_splits",
        [
            ("squad", "plain_text", ["train", "validation"]),
            ("dalle-mini/wit", "dalle-mini--wit", ["train"]),
            ("paws", "labeled_final", ["train", "test", "validation"]),
        ],
    )
    def test_get_dataset_split_names(path, expected_config, expected_splits):
        infos = get_dataset_infos(path)
        assert expected_config in infos
        info = infos[expected_config]
        assert info.config_name == expected_config
>       assert list(info.splits.keys()) == expected_splits
E       AssertionError: assert ['test', 'tra... 'validation'] == ['train', 'te... 'validation']
E         At index 0 diff: 'test' != 'train'
E         Full diff:
E         - ['train', 'test', 'validation']
E         + ['test', 'train', 'validation']
```


Do not sort splits in dataset info

It would be coherent with https://github.com/huggingface/datasets-server/issues/614#issuecomment-1290534153

I think we started working on this issue nearly at the same time... :sweat_smile:
- CI was fixed with this: https://huggingface.co/datasets/paws/discussions/1

Related issue:
- #5202

@albertvillanova yeah I noticed it right after the PR :smile: thank you! the fix of the dataset info yaml fixes tests on CI, but in general order of splits in yaml influences the order in which they are displayed in the viewer, if I understand it correctly. So I suggest not to sort splits in yaml initially to avoid this for other datasets in the future. I think [this change](https://github.com/huggingface/datasets/pull/5201/files#diff-198ba4fdf2f94cb3e1aba8a0170a43b08d4ab5636d682374321c5a383a8be24dR571) should work for it.

Changes to tests here maybe can be reverted considering that order in yaml now corresponds to the one in tests, thanks to your change in the dataset info.

Hehe, @polinaeterna, we make comments nearly at the same time as well... :laughing:

I suggest not to sort splits by their names in dataset_info in README so that they are displayed in the order specified in the loading script. Otherwise `test` split is displayed first, see this repo: https://huggingface.co/datasets/paws
What do you think?

But I added sorting in tests to fix CI (for the same dataset).


Some links to canonical datasets in the docs are outdated

documentation

Improvements or additions to documentation

Thanks for catching this, I can go through the docs and replace the links to their corresponding datasets on the Hub!

As we don't have canonical datasets in the github repo anymore, some old links to them doesn't work. I don't know how many of them are there, I found link to SuperGlue here: https://huggingface.co/docs/datasets/dataset_script#multiple-configurations, probably there are more of them. These links should be replaced by links to the corresponding datasets on the Hub.


Deprecate dummy data generation command

_The documentation is not available anymore as the PR was closed or merged._

Deprecate the `dummy_data` CLI command.


Add note about the name of a dataset script

Add note that a dataset script should has the same name as a repo/dir, a bit related to this issue https://github.com/huggingface/datasets/issues/5193

also fixed two minor issues in audio docs (broken links)


[zstd] Use max window log size

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5197). All of your documentation changes will be reflected on that endpoint.

ZstdDecompressor has a parameter `max_window_size` to limit max memory usage when decompressing zstd files. The default `max_window_size` is not enough when files are compressed by `zstd --ultra` flags.

Change `max_window_size` to the zstd's max window size. NOTE, the `zstd.WINDOWLOG_MAX` is the log_2 value of the max window size.


Use hfh hf_hub_url function

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5196). All of your documentation changes will be reflected on that endpoint.

@lhoestq I think we should first agree if `datasets` can introduce the breaking change of ignoring `config.HUB_DATASETS_URL`: some users may have override this.

If so, I then would suggest to initiate a deprecation cycle.

After a discussion with the rest of the datasets team, we agreed we can introduce the breaking change of ignoring `config.HUB_DATASETS_URL`: this will have minimal impact, only for **private Hubs**. We will address eventual possible impacts in the future.

Additionally, we also ignore `config.HUB_DEFAULT_VERSION`.

See explanation in this PR description: https://github.com/huggingface/datasets/pull/5196#issue-1434401646

I'm trying to upgrade datasets to 2.7.0 in https://github.com/huggingface/datasets-server, and the tests fail due to this change. I think it's a breaking change (that was not listed in https://github.com/huggingface/datasets/releases/tag/2.7.0) since code that previously worked (by setting `datasets.config.HUB_DATASETS_URL = CI_HUB_DATASETS_URL` for example) does not work anymore.

I'm not sure what is the correct way to set up the tests; besides setting the env var "HF_ENDPOINT" before launching the tests (which, I think, is not a good way to do: the tests should not depend on the environment).

OK, I re-read this thread, and https://github.com/huggingface/datasets/pull/5196#issuecomment-1307430175 explicitely states that `config.HUB_DATASETS_URL` (as well as `config.HUB_DEFAULT_VERSION`) is now ignored. I was expecting the breaking changes to be listed in the release notes: https://github.com/huggingface/datasets/releases/tag/2.7.0.

> I'm not sure what is the correct way to set up the tests; besides setting the env var "HF_ENDPOINT" before launching the tests (which, I think, is not a good way to do: the tests should not depend on the environment).

I think the current workaround of settings an env variable before launching the tests is "not so bad" when considering the fact that env variables are evaluated at import time in `huggingface_hub` (and most probable `datasets` as well). I think that when refactoring this in huggingface_hub (https://github.com/huggingface/huggingface_hub/issues/1172) I'll opt for instantiating a `Settings` object (or `Constants`) that contains all the settings variables. This way it will not be possible to import attributes individually + tests would be easier. As I see it, it would be similar to [what `Pydantic` does](https://pydantic-docs.helpmanual.io/usage/settings/) even though we most probably don't want Pydantic as a root dependency just for that.

You can use fixtures in your tests:
```python
CI_HUB_ENDPOINT = "https://hub-ci.huggingface.co"
CI_HUB_DATASETS_URL = CI_HUB_ENDPOINT + "/datasets/{repo_id}/resolve/{revision}/{path}"
CI_HFH_HUGGINGFACE_CO_URL_TEMPLATE = CI_HUB_ENDPOINT + "/{repo_id}/resolve/{revision}/{filename}"

@pytest.fixture
def ci_hfh_hf_hub_url(monkeypatch):
    monkeypatch.setattr(
        "huggingface_hub.file_download.HUGGINGFACE_CO_URL_TEMPLATE", CI_HFH_HUGGINGFACE_CO_URL_TEMPLATE
    )

@pytest.fixture
def ci_hub_config(monkeypatch):
    monkeypatch.setattr("datasets.config.HF_ENDPOINT", CI_HUB_ENDPOINT)
    monkeypatch.setattr("datasets.config.HUB_DATASETS_URL", CI_HUB_DATASETS_URL)
```

and use `@pytest.fixture(autouse=True)` if you want to always use the CI endpoints.

And when `huggingface-hub` and `datasets` change the way we can set the endpoint, we'll just need to update the fixtures.
I think ultimately you'll only have to change the `huggingface-hub` endpoint settings

OK.

In fact, in datasets-server we set `config.HUB_DATASETS_URL` (https://github.com/huggingface/datasets-server/blob/35a30dbcd687b26db1f02502ea8305f70c064473/workers/splits/src/splits/config.py#L26) at config time, before starting the workers. It's not an issue with how to launch the tests, but with the app in itself.

I understand that for now, the only way to fix this is to setup `HF_ENDPOINT` in the env when launching the app (currently, we set the endpoint with `COMMON_HF_ENDPOINT`, a custom env var I set to be sure not to have side-effects)

> You can use fixtures in your tests:

Thanks, used in https://github.com/huggingface/datasets-server/pull/644.

Small refactoring to use `hf_hub_url` function from `huggingface_hub`.

This PR also creates the `hub` module that will contain all `huggingface_hub` functionalities relevant to `datasets`.

This is a necessary stage before implementing the use of the `hfh` caching system (which uses its `hf_hub_url` under the hood).

EDIT:
~~Finally, we use our `config.HUB_DATASETS_URL` when using `hfh.hf_hub_url`~~
There is a breaking change:  the `hfh` `hf_hub_url` function uses
- `hfh` `HUGGINGFACE_CO_URL_TEMPLATE` URL template, different from the `datasets` `config.HUB_DATASETS_URL`
- also, `hfh` `DEFAULT_REVISION`, instead of `datasets` `config.HUB_DEFAULT_VERSION`


[wip testing docs]

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5195). All of your documentation changes will be reflected on that endpoint.


Fix docs about dataset_info in YAML

This PR fixes some misalignment in the docs after we transferred the dataset_info from `dataset_infos.json` to YAML in the dataset card:
- #4926

Related to:
- #5193


"One or several metadata. were found, but not in the same directory or in a parent directory"

Also unrelated but still: https://huggingface.co/docs/datasets/image_dataset#generate-the-dataset
```If your loading script passed the test, you should now have a dataset_infos.json file in your dataset folder.```
It's not the case anymore as it's now in the readme.md, it was confusing to me

And here is my data loader script: https://huggingface.co/datasets/corentinm7/MyoQuant-SDH-Data/blob/main/SDH_16k.py
I have one file archive to download that contains the images for all splits and one `metadata.jsonl` to download that contains the informations about what image goes into what split.

> Hi @lambda-science! It seems that your repo is recognized as a packaged module [ImageFolder](https://huggingface.co/docs/datasets/main/en/image_dataset#imagefolder), not as a dataset with the custom loading script, because loader looks for a script that has the same name as the dataset repo. So please try to rename your script to `MyoQuant-SDH-Data.py`, this should help.

Hi !

Thank you for your answer. That was... embarrassingly easy, sorry for this issue, everything is fixed now !

Have a nice day ! :)

@lambda-science that's not embarrassing at all! it's actually not clear from the documentation that the script should have the same name, so thank you for the issue, we'll add this information to the docs :)

### Describe the bug

When loading my own dataset, on loading it I get an error.
Here is my dataset link: https://huggingface.co/datasets/corentinm7/MyoQuant-SDH-Data
And the error after loading with:
```python
from datasets import load_dataset

load_dataset("corentinm7/MyoQuant-SDH-Data")
```
```python
Downloading readme: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.34k/3.34k [00:00<00:00, 4.45MB/s]
Using custom data configuration SDH_16k-53e7301a92ab0025
Downloading and preparing dataset None/SDH_16k to /home/corentin/.cache/huggingface/datasets/corentinm7___imagefolder/SDH_16k-53e7301a92ab0025/0.0.0/37fbb85cc714a338bea574ac6c7d0b5be5aff46c1862c1989b20e0771199e93f...
Downloading data: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.28M/3.28M [00:00<00:00, 4.31MB/s]
Downloading data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00,  1.75s/it]
Downloading data: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.13G/1.13G [00:15<00:00, 74.3MB/s]
Downloading data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:16<00:00, 16.09s/it]
Extracting data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:13<00:00, 13.16s/it]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/corentin/code-project/hugging_face_play/.venv/lib/python3.10/site-packages/datasets/load.py", line 1742, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/corentin/code-project/hugging_face_play/.venv/lib/python3.10/site-packages/datasets/builder.py", line 814, in download_and_prepare
    self._download_and_prepare(
  File "/home/corentin/code-project/hugging_face_play/.venv/lib/python3.10/site-packages/datasets/builder.py", line 1423, in _download_and_prepare
    super()._download_and_prepare(
  File "/home/corentin/code-project/hugging_face_play/.venv/lib/python3.10/site-packages/datasets/builder.py", line 905, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/corentin/code-project/hugging_face_play/.venv/lib/python3.10/site-packages/datasets/builder.py", line 1374, in _prepare_split
    for key, record in logging.tqdm(
  File "/home/corentin/code-project/hugging_face_play/.venv/lib/python3.10/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/home/corentin/code-project/hugging_face_play/.venv/lib/python3.10/site-packages/datasets/packaged_modules/folder_based_builder/folder_based_builder.py", line 394, in _generate_examples
    raise ValueError(
ValueError: One or several metadata. were found, but not in the same directory or in a parent directory of /home/corentin/.cache/huggingface/datasets/downloads/extracted/60c4aa8d4da3065bb3d310de4373dffd73bd4dc331aedcb4ee867febe4fdb7cd/validation/sick/2_CG_SDH_TAM_Bin1cKO_ko_pla_4_1640.tif.
```

However the test command is working fine. ```datasets-cli test hugging_face_play/ds_test/SDH_16k.py --save_info --all_configs --force_redownload```
```
Using custom data configuration SDH_16k
Testing builder 'SDH_16k' (1/1)
Downloading and preparing dataset sdh_16k/SDH_16k to /home/corentin/.cache/huggingface/datasets/sdh_16k/SDH_16k/1.0.0/21b584239a638aeeda33cba1ac2ca4869d48e4b4f20fb22274d5a5ddc487659d...
Downloading data: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.13G/1.13G [00:14<00:00, 76.5MB/s]
Downloading data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:15<00:00, 15.66s/it]
Downloading data: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.28M/3.28M [00:02<00:00, 1.44MB/s]
Downloading data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:03<00:00,  3.21s/it]
Downloading data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 11586.48it/s]
Extracting data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:13<00:00, 13.42s/it]
Dataset sdh_16k downloaded and prepared to /home/corentin/.cache/huggingface/datasets/sdh_16k/SDH_16k/1.0.0/21b584239a638aeeda33cba1ac2ca4869d48e4b4f20fb22274d5a5ddc487659d. Subsequent calls will reuse this data.
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 605.27it/s]
Dataset card saved at hugging_face_play/ds_test/README.md
Test successful.
```

### Steps to reproduce the bug

Simply run on python
```python
from datasets import load_dataset

load_dataset("corentinm7/MyoQuant-SDH-Data")
```

### Expected behavior

As the test command worked, this error should not appear

### Environment info

- `datasets` version: 2.6.1
- Platform: Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.31
- Python version: 3.10.6
- PyArrow version: 10.0.0
- Pandas version: 1.5.1


Drop labels in Image and Audio folders if files are on different levels in directory or if there is only one label

bug

Something isn't working

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5192). All of your documentation changes will be reflected on that endpoint.

> Nit: maybe we can use the count_path_segments function from this file for counting (updated with your logic to make it faster).

@mariosasko just to make sure I understood you correctly - are you okay with this change? (actually `os.path.normpath` is redundant here as paths from `data_files` should be already normalized but just in case)
https://github.com/huggingface/datasets/pull/5192/files#diff-1f09f7a178211f7539b1499b64b69793bd53b30c8b7b34cfcc5835e25d31929fR33
If you are, we can merge.

Will close https://github.com/huggingface/datasets/issues/5153

Drop labels by default (`drop_labels=None`) when:
* there are files on different levels of directory hierarchy by checking their path depth
* all files are in the same directory (=only one label was inferred)

First one fixes cases like this:
```
repo
   image3.jpg
   image4.jpg
   data
      image1.jpg
      image2.jpg
```
Second one fixes cases like this:
```
repo
   image1.jpg
   image2.jpg
   image3.jpg
```
This is mostly to fix the viewer for people who just drop images in the Hub interface into the root dir.

I added tests for both of the cases on local and remote files. **I also changed data files for old test on drop_labels** (`test_generate_examples_drop_labels`). The files I provide to `test_generate_examples_drop_labels` now has "canonical" classification structure (two dirs) in order not to change the logic of the test (=not to check these two cases addressed in the PR).


Make torch.Tensor and spacy models cacheable

_The documentation is not available anymore as the PR was closed or merged._

Override `Pickler.save` to implement deterministic reduction (lazily registered; inspired by https://github.com/uqfoundation/dill/blob/master/dill/_dill.py#L343) functions for `torch.Tensor` and spaCy models.

Fix https://github.com/huggingface/datasets/issues/5170, fix https://github.com/huggingface/datasets/issues/3178


`path` is `None` when downloading a custom audio dataset from the Hub

Hi! Yes, this is expected behavior - we do this as a security measure to not leak local paths (this info would be useless on other users' machines anyways) and only push audio bytes.

### Describe the bug

I've created an [audio dataset](https://huggingface.co/datasets/lewtun/audio-test-push) using the `audiofolder` feature desribed in the [docs](https://huggingface.co/docs/datasets/audio_dataset#audiofolder) and then pushed it to the Hub.

Locally, I can see the `audio.path` feature is of the expected form `path/to/data_dir`, but when I download the dataset from the Hub, I see `audio.path` is `None`

Here's an example:

```python
from datasets import load_dataset

ds = load_dataset("lewtun/audio-test-push")
ds["train"][0]
# {
#     "audio": {
#         "path": None,    <-- Is this expected?
#         "array": array(
#             [
#                 3.97140226e-07,
#                 7.30310290e-07,
#                 7.56406735e-07,
#                 ...,
#                 -1.19636677e-01,
#                 -1.16811886e-01,
#                 -1.12441722e-01,
#             ]
#         ),
#         "sampling_rate": 44100,
#     },
#     "song_id": 0,
#     "genre_id": 0,
#     "genre": "Electronic",
# }
```

Is this expected behaviour? If yes, feel free to close this issue as it's not a true bug then :)

### Steps to reproduce the bug

1. Create an audio dataset with the `audiofolder` feature
2. Push the dataset to the Hub with `push_to_hub()`
3. Download the Hub dataset and inspect the `audio.path` feature

### Expected behavior

`audio.path` points to the file associated with the audio data

### Environment info

- `datasets` version: 2.6.2.dev0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.13
- PyArrow version: 9.0.0
- Pandas version: 1.5.1


Reduce friction in tabular dataset workflow by eliminating having splits when dataset is loaded

enhancement

New feature or request

I have to admit I'm not a fan of this idea, as this would result in a non-consistent behavior between tabular and non-tabular datasets, which is confusing if done without the context you provided. Instead, we could consider returning a `Dataset` object rather than `DatasetDict` if there is only one split in the generated dataset. But then again, I think this lib is a bit too old to make such changes. @lhoestq @albertvillanova WDYT?

We can brainstorm here to see how we could make it happen ? And then depending on the options we see if it's a change we can do.

I'm starting with a first reasoning

Currently not passing `split=` in `load_dataset` means "return a dict with each split".

Now what would happen if a dataset has no split ? Ideally it should return one Dataset. And passing `split=` would have no sense. So depending on the dataset content, not passing `split=` should return a dict or a Dataset. In particular, those two cases should work:
```python
# case 1: dataset without split
ds = load_dataset("dataset_without_split")
ds[0], ds["column_name"], list(ds) # we want this

# case 2: dataset with splits
ds = load_dataset("dataset_with_splits")
ds["train"] # this works and can't be changed
ds = load_dataset("dataset_with_splits", split="train")
ds[0], ds["column_name"], list(ds) # this works and can't be changed
```

I can see several ideas:
1. allowing `load_dataset` to return a different object based on the dataset content - either a Dataset or a DatasetDict
    - we can update `get_dataset_split_names` to return None or a list if users want to know in advance what object will be returned. They can also use `isinstance` _a posteriori_
    - but in this case we expect users to be careful when loading datasets and always to extra steps to check if they got a Dataset or DatasetDict
2. merge Dataset and DatasetDict objects
    - they already share many functions: map, filter, push_to_hub etc.
    - we can define `ds[0]` to be the first item of the first split, and consider that the uses accesses rows from the full table of all the splits concatenated
    - however there is a collision when doing `ds["column_name"]` or `ds["train"]` that we need to address: the first returns a list, while the other returns a Dataset.

What are your opinions on those two ideas ? Do you have other ideas in mind ?

I like the first idea more (concatenating splits doesn't seem useful, no?). This is a significant breaking change, so I think we should do a poll (or something similar) to gather more info on the actual "expected behavior" and wait for Datasets 3.0 if we decide to implement it.

PS: @thomwolf also suggested the same thing a while ago (https://github.com/huggingface/datasets/issues/743#issuecomment-746074641).

I think it's an interesting improvement to the user experience for a case that comes often (no split) so I would definitively support it.

I would be more in favor of option 2 rather than returning various types of objects from load_dataset and handling carefully the possible collisions indeed

Related: if a dataset only has one split, we don't show the splits select control in the dataset viewer on the Hub, eg. compare  https://huggingface.co/datasets/hf-internal-testing/fixtures_image_utils/viewer/image/test with https://huggingface.co/datasets/glue/viewer/mnli/test.

See https://github.com/huggingface/moon-landing/pull/3858 for more details (internal)

I feel like the second idea is a bit more overkill.
@severo I would say it's a bit irrelevant to the problem we have but is a separate problem @polinaeterna is solving at the moment. 😅 (also discussed on slack)

OK, sorry for polluting the thread. The relation I saw with the dataset viewer is that from a UX point of view, we hide the concepts of split and configuration whenever possible -> this issue feels like doing the same in the datasets library.

I would agree that returning different types based on the content of the dataset might be confusing.

We can do something similar to what `fetch_*` or `load_*` from `sklearn.datasets` do, which is to have an arg which changes the type of the returned type. For instance, `load_iris` would return a dict, but `load_iris(..., return_X_y=True)` would return a tuple.

Here we can have a similar arg such as `return_X` which would then only return a single `DataSet` or an array.

> I feel like the second idea is a bit more overkill.

Overkill in what sense ?

> Here we can have a similar arg such as return_X which would then only return a single DataSet or an array.

Right now one can already pass `split="all"` to get one `Dataset` object with all the data in it (unsplit). We could also have something like `return_all=True` so make the API clearer.

> I would be more in favor of option 2 rather than returning various types of objects from load_dataset and handling carefully the possible collisions indeed

I think it would be ok to handle the collision by allowing both `ds["train"]` and `ds["column_name"]` (and maybe adding something like `ds.splits` for those who want to iterate over the splits or add new ones)

Would it make sense to remove the notion of "split" in `load_dataset`? I feel a lof of it comes from the want to have some sort of group of more or less similar dataset. "train"/"test"/"validation" are the traditional ones, but there are some datasets that have much more splits.

Would it make sense to force `load_dataset` to only load a single `Dataset` object, and fail if it doesn't point to one. And have another method that's like `load_dataset_group_info` that can return a very arbitrary info class (Dict, List whatever), but you need to pass individual infos to `load_dataset` to run anything? Typically I don't think `DatasetDict.map` is really that helpful, but that's my personal opinion. This would help make things more readable (typically knowing if an object is a `Dataset` or a `DatasetDict`)

> Would it make sense to remove the notion of "split" in load_dataset?

I think we need to keep it - though in practice people can name the splits whatever they want anyway.

> Would it make sense to force load_dataset to only load a single Dataset object, and fail if it doesn't point to one.

We need to keep backward compatibility ideally - in particular the load_dataset + ds["train"] one

> I think we need to keep it - though in practice people can name the splits whatever they want anyway.

It was my understanding that the whole issue was that `load_dataset` returned multiple types of objects.

> We need to keep backward compatibility ideally - in particular the load_dataset + ds["train"] one

Yeah sorry I meant ideally. One can always start developing `load_dataset_v2` can deprecate the first one and remove it in the longer term.

> It was my understanding that the whole issue was that load_dataset returned multiple types of objects.

Yes indeed, but we still want to keep a way to load the train/val/test/whatever splits alone ;)

@thomasw21's solution is good but it will break backwards compatibility. 😅

Started to experiment with merging Dataset and DatasetDict. My plan is to define the splits of a Dataset in Dataset.info.splits (already exists, but never used). A Dataset would then be the concatenation of its splits if they exist.

Not sure yet this is the way to go. My plan is to play with it and see and share it with you, so we can see if it makes sense from a UX point of view.

So just to make sure that I understand the current direction, people will have to be extra careful when handling splits right?
Imagine "potato" a dataset containing train/validation split:
```
load_dataset("potato") # returns the concatenation of all the splits
```
Previously the design would force you to choose a split (it would raise otherwise), or manually concat them if you really wanted to play with concatenated splits. Now it would potentially run without raising for a bit of time until you figure out that you've been training on both train and validation split.

Would it make sense to use a dataset specific default instead of using the concatenation, typically "potato" dataset's default would be train?
```
load_dataset("potato") # returns "train" split
load_dataset("potato", split="train") # returns "train" split
load_dataset("potato", split="validation") # returns "validation" split
concatenate_datasets([load_dataset("potato", split="train"), load_dataset("potato", split="validation")]) # returns concatenation
```

> load_dataset("potato") # returns "train" split

To avoid a breaking change we need to be able to do `load_dataset("potato")["validation"]` as well.

In that case I'd wonder where the validation split comes from, since the rows of the dataset wouldn't contain the validation split according to your example. That's why I'm more in favor of concatenating.

A dataset is one table, that optionally has some split info about subsets (e.g. for training an evaluation)

This also allows anyone to re-split the dataset the way they want if they're not happy with the default:

```python
ds = load_dataset("potato").train_test_split(test_size=0.2)
train_ds = ds["train"]
test_ds = ds["test"]
```

Just thinking about this, we could just have `to_dataframe()` as `load_dataset("blah").to_dataframe()` to get the whole dataset, and not change anything else.

I have a first implementation of option 2 (merging Dataset and DatasetDict) in this PR: https://github.com/huggingface/datasets/pull/5301/

Feel free to play with it if you're interested, and let me know what you think. In this PR, a dataset is one table that optionally has some split info about subsets.

@adrinjalali we already have [to_pandas](https://huggingface.co/docs/datasets/package_reference/main_classes#datasets.Dataset.to_pandas) AFAIK that essentially does the same thing (for a dataset, not for a dataset dict), I was wondering if it makes sense to have this as I don't know portion of people who load non-tabular datasets into dataframes. @lhoestq I saw your PR and it will break a lot of things imo, WDYT of this option?

> we already have [to_pandas](https://huggingface.co/docs/datasets/package_reference/main_classes#datasets.Dataset.to_pandas) AFAIK that essentially does the same thing (for a dataset, not for a dataset dict)

yes correct :)

> I saw your PR and it will break a lot of things imo

Do you have concrete examples you can share ?

> WDYT of this option?

The to_dataframe option ? I think it not enough, since you'd still get a `DatasetDict({"train": Dataset()})` if you load a dataset with no splits (e.g. one CSV), and this doesn't really make sense.

Note that in the PR I opened you can do
```python
ds = load_dataset("dataset_with_just_one_csv")  # Dataset type
df = load_dataset("dataset_with_just_one_csv").to_pandas()  # DataFrame type
```

@lhoestq no I think @adrinjalali and I meant when user calls `to_dataframe` if there's only train split in `DatasetDict` we could directly load that into dataframe. This might cause a confusion given there's to_pandas but I think it's more intuitive and least breaking change. (given people -who use `datasets` for tabular workflows- will eventually call `to_pandas` anyway)

So in that case it would be fine to still end up with a dataset dict with a "train" split ?

yeah what I mean is this:

```py
dataset = load_dataset("blah")

# deal with a split of the dataset
train = dataset["train"]
train_df = dataset["train"].to_dataframe()

# deal with the whole dataset
dataset_df = dataset.to_dataframe()
```

So we do two things to improve tabular experience:
- allow datasets to have a single split
- add `to_dataframe` to the root dict level so that users can simply call `df = load_dataset("blah").to_dataframe()` and have it in their `pandas.DataFrame` object.

Ok ! Note that we already have `Dataset.to_pandas()` so for consistency I'd call it `DatasetDict.to_pandas()` as well, does it sound good to you ? This is something we can add pretty easily

> So just to make sure that I understand the current direction, people will have to be extra careful when handling splits right?

We can raise an error if someone does `load_dataset(...)[0]` if the dataset is made of several splits, and return the first example if there's one or zero splits (i.e. when it's not ambiguous). Had this idea from the dicussions in #5312 WDYT @thomasw21 ?

> We can raise an error if someone does load_dataset(...)[0] if the dataset is made of several splits,

But then how is that different to have the distinction between DatasetDict and Dataset then? Is it just that "default behaviour when there are no splits or single split, it returns directly the split when there's no ambiguity".

Also I was wondering how the concatenation could have heavy impacts when running mapping functions/filtering in batch? Typically can batch be somehow mixed?

> But then how is that different to have the distinction between DatasetDict and Dataset then?

Because it doesn't make sense to be able to do `example = ds[0]` or `examples = list(ds)` on a class named `DatasetDict` of type `Dict[str, Dataset]`.

> Also I was wondering how the concatenation could have heavy impacts when running mapping functions/filtering in batch? Typically can batch be somehow mixed?

No, we run each function on each split separated

> Because it doesn't make sense to be able to do example = ds[0] or examples = list(ds) on a class named DatasetDict of type Dict[str, Dataset].

Hum but you're still going to raise an exception in both those cases with your current change no? (actually list(ds) would return the name of the splits no?)

> No, we run each function on each split separated

Nice!

### Feature request

Sorry for cryptic name but I'd like to explain using code itself. When I want to load a specific dataset from a repository (for instance, this: https://huggingface.co/datasets/inria-soda/tabular-benchmark)

```python
from datasets import load_dataset
dataset = load_dataset("inria-soda/tabular-benchmark", data_files=["reg_cat/house_sales.csv"], streaming=True)
print(next(iter(dataset["train"])))
```

`datasets` library is essentially designed for people who'd like to use benchmark datasets on various modalities to fine-tune their models, and these benchmark datasets usually have pre-defined train and test splits. However, for tabular workflows, having train and test splits usually ends up model overfitting to validation split so usually the users would like to do validation techniques like `StratifiedKFoldCrossValidation` or when they tune for hyperparameters they do `GridSearchCrossValidation` so often the behavior is to create their own splits. Even [in this paper](https://hal.archives-ouvertes.fr/hal-03723551) a benchmark is introduced but the split is done by authors.
It's a bit confusing for average tabular user to try and load a dataset and see `"train"` so it would be nice if we would not load dataset into a split called `train `by default.

```diff
from datasets import load_dataset
dataset = load_dataset("inria-soda/tabular-benchmark", data_files=["reg_cat/house_sales.csv"], streaming=True)
-print(next(iter(dataset["train"])))
+print(next(iter(dataset)))
```

### Motivation

I explained it above 😅

### Your contribution

I think this is quite a big change that seems small (e.g. how to determine datasets that will not be load to train split?), it's best if we discuss first!


add: segmentation guide.

_The documentation is not available anymore as the PR was closed or merged._

Thanks @osanseviero. Am I good to merge?

I would wait for a second approval just in case :)

Merging since the images have been pushed as LFS files ([PR](https://huggingface.co/datasets/huggingface/documentation-images/discussions/8)).

Closes #5181

I have opened a PR on Hub (https://huggingface.co/datasets/huggingface/documentation-images/discussions/5) to include the images in our central Hub repository. Once the PR is merged I will edit the image links.

I have also prepared a [Colab Notebook](https://colab.research.google.com/drive/1BMDCfOTBnyshoME5RSxn5iQy-TWeFbOA?usp=sharing) in case anyone wants to play.

- [x] Replace the image links


chore: add notebook links to img cls and obj det.

New feature or request

@nateraw I guess the failing test is unrelated.

Alright. Will wait for @osanseviero's take and then merge.

@osanseviero @stevhliu @nateraw thank you for your comments. Acted on them.

Thanks! Can I merge? Or should we wait for approvals from the others?

Since @stevhliu approved as well, I think you're good to go

Alright!

Merging as a Member for the first time 🫀

Closes https://github.com/huggingface/datasets/issues/5182


Incorrect error message when Dataset.from_sql fails and sqlalchemy not installed

Hi! The first `Dataset.from_sql` call also outputs the "ImportError: Using URI string without sqlalchemy installed." message, but you also get "During handling of the above exception another exception occurred: ..." after which the ValueError is printed. I agree that this behavior makes it easy to miss the original error.

I think we can improve this by not throwing the writer's ValueError if the error from a dataset script is already being handled to make debugging easier. @lhoestq @albertvillanova wdyt?

Yup ! Alternatively the error can be raised in sql.py before generating the examples ? In `_info` for example

yea @lhoestq that would probably be good. The 2nd error is useless if the 1st error is the real reason it failed.

### Describe the bug

When calling `Dataset.from_sql` (in my case, with sqlite3), it fails with a message ```ValueError: Please pass `features` or at least one example when writing data``` when I don't have `sqlalchemy` installed.

### Steps to reproduce the bug

Make a new sqlite db with `sqlite3` and `pandas` from a remote [URL](https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-states.csv).

```python
import sqlite3
import pandas as pd
from datasets import Dataset

conn = sqlite3.connect('us_covid_data.db')
df = pd.read_csv('https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-states.csv')
df.to_sql('states', conn, if_exists='replace')
```

Then if you try to query this DB like this:

```python
ds = Dataset.from_sql('''SELECT * from states WHERE state=="New York";''', "sqlite:///us_covid_data.db")
```

You run into the error I described above:

```ValueError: Please pass `features` or at least one example when writing data```

However, if you try to pass features, as the error suggests, then you get an error that tells you the underlying problem...

```python
from datasets import Dataset, Features, Value

features = Features({
    'date': Value('date32'),
    'label': Value('string'),
    'fips': Value('int32'),
    'cases': Value('int32'),
    'deaths': Value('int32')
})
ds = Dataset.from_sql(
    '''SELECT * from states WHERE state=="New York";''',
    "sqlite:///us_covid_data.db",
    features=features
)
```

Which results in the actual underlying error: `ImportError: Using URI string without sqlalchemy installed.`

### Expected behavior

Instead of `ValueError` about needing to pass features, we should provide the actual underlying error about not having SQLAlchemy installed when it isn't found in the environment.

### Environment info

- `datasets` version: 2.6.1
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.10
- PyArrow version: 10.0.0
- Pandas version: 1.2.5


Allow passing a subset of output features to Dataset.map

New feature or request

### Feature request

Currently, map does one of two things to the features (if I'm not mistaken):

* when you do not pass features, types are assumed to be equal to the input if they can be cast, and inferred otherwise
* when you pass a full specification of features, output features are set to this

However, sometimes you want to just pass some of the output types, particularly when the first of these modes makes an incorrect type. This currently crashes.

### Motivation

To give a little background: this problem appears in converting labels to ids, where the labels happen to be floats rather than strings

Consider the following use of map to convert from float to int
```python
data = Dataset.from_dict({'y':[1.0,2.0,3.0]})
mapped = data.map(lambda r: {'y': int(r['y'])})
mapped['y'] # is floats, not ints
```

The result is a float again, since after the mapping operation it forces the old datatypes back on the data.

Passing `features=Features({"y": Value(dtype="int64")})` to map works in principle, but then extending it a little to e.g.
```python
def format_data(r):
    return {**tokenizer(r["text"]), "y": int(r["y"])}

data = Dataset.from_dict({"y": [1.0, 2.0, 3.0], "text": ["one", "two", "three"]})
mapped = data.map(
    format_data,
    features=Features({'y': Value(dtype="int64")}),
    remove_columns=["text"],
)
```

Results in a crash in dataset internals, as it expects either all or no output features to be specified.
Of course one can pass a full feature specification, but this becomes tokenizer specific and very awkward.

### Your contribution

I've looked at `write_batch` and particularly `col_type = features[col] if features else None`, but checking for `col in features` here makes it fail elsewhere, but the structure makes it hard to understand how and why. I do not think I would have the time myself to get to the bottom of this anytime soon.


Loading an external dataset in a format similar to conll2003

I'm trying to load a custom dataset in a Dataset object, it's similar to conll2003 but with 2 columns only (word entity), I used the following script:

features = datasets.Features(
            {"tokens": datasets.Sequence(datasets.Value("string")),
        "ner_tags": datasets.Sequence(
            datasets.features.ClassLabel(
                names=["B-PER", .... etc.]))}
        )

from datasets import Dataset

INPUT_COLUMNS = "tokens ner_tags".split(" ")

def read_conll(file):
    #all_labels = []
    example = {col: [] for col in INPUT_COLUMNS}
    idx = 0
    with open(file) as f:
        for line in f:
          if line:
            if line.startswith("-DOCSTART-") and example["tokens"] != []:
                print(idx, example)
                yield idx, example
                idx += 1
                example = {col: [] for col in INPUT_COLUMNS}
            elif line == "\n" or (line.startswith("-DOCSTART-") and example["tokens"] == []):
              continue
            else:
                row_cols = line.split(" ")
                for i, col in enumerate(example):
                    example[col] = row_cols[i].rstrip()

dset = Dataset.from_generator(read_conll, gen_kwargs={"file": "/content/new_train.txt"}, features = features)

The following error happened:
[/usr/local/lib/python3.7/dist-packages/datasets/utils/py_utils.py](https://localhost:8080/#) in <genexpr>(.0)
    285     for key in unique_values(itertools.chain(*dicts)):  # set merge all keys
    286         # Will raise KeyError if the dict don't have the same keys
--> 287         yield key, tuple(d[key] for d in dicts)
    288

TypeError: tuple indices must be integers or slices, not str

What does this mean and what should I modify?


Add notebook / other resource links to the task-specific data loading guides

New feature or request

Yea this would be great! We would need an object detection tutorial notebook too if it doesn't already exist there.

There is one: https://huggingface.co/docs/datasets/object_detection.

I will start the work.

Does it make sense to include links to notebooks / scripts that show how to use a dataset for training / fine-tuning a model?

For example, here in [https://huggingface.co/docs/datasets/image_classification] we could include a mention of https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb.

Applies to https://huggingface.co/docs/datasets/object_detection as well.

Cc: @osanseviero @nateraw


Add a guide for semantic segmentation

Sure this sounds great! Would this be pure torchvision, albumentations, or something else?

I am considering `torchvision` and `albumentations`. Also [works with TensorFlow](https://github.com/deep-diver/segformer-tf-transformers/blob/main/notebooks/TFSegFormer_Finetune.ipynb).

I am assigning the issue to myself then.

Currently, we have these guides for object detection and image classification:

* https://huggingface.co/docs/datasets/object_detection
* https://huggingface.co/docs/datasets/image_classification

I am proposing adding a similar guide for semantic segmentation.

I am happy to contribute a PR for it.

Cc: @osanseviero @nateraw


An example or recommendations for creating large image datasets?

The beam utilities allow to prepare a dataset as parquet in your cloud storage. From my perspective this CLI is not super easy to use, but we've been working on a new python API to prepare a dataset in your cloud storage:
```python
from datasets import load_dataset_builder

builder = load_dataset_builder("c4", "en")
builder.download_and_prepapre("s3://my-bucket/c4", file_format="parquet")
```

And to use Beam you can do:
```python
beam_runner = ...  # one of "SparkRunner", "DataFlowRunner", "DirectRunner", etc.
beam_options = ...

builder.download_and_prepapre(
    "s3://my-bucket/c4",
    file_format="parquet",
    beam_runner=beam_runner,
    beam_options=beam_options
)
```

Though Beam can be used ONLY if there is a dataset script based on the `BeamBasedBuilder` right now - it doesn't work on an arbitrary dataset (see [wikipedia.py](https://huggingface.co/datasets/wikipedia/blob/main/wikipedia.py) for example).

Thanks!

Would be nice to have something similar for creating large image datasets.

I know that Apache Beam and `datasets` have [some connector utilities](https://huggingface.co/docs/datasets/beam). But it's a little unclear what we mean by "But if you want to run your own Beam pipeline with Dataflow, here is how:". What does that pipeline do?

As a user, I was wondering if we have this support for creating large image datasets. If so, we should mention that [here](https://huggingface.co/docs/datasets/image_dataset).

Cc @lhoestq