Support cloud storage in load_dataset

enhancement

New feature or request

good second issue

Issues a bit more difficult than "Good First" issues

Or for example an archive on GitHub releases!  Before I added support for JXL (locally only, PR still pending) I was considering hosting my files on GitHub instead...

+1 to this. I would like to use 'audiofolder' with a data_dir that's on S3, for example. I don't want to upload my dataset to the Hub, but I would find all the fingerprinting/caching features useful.

Adding to the conversation, Dask also uses `fsspec` for this feature.

[Dask: How to connect to remote data](https://docs.dask.org/en/stable/how-to/connect-to-remote-data.html)

Happy to help on this feature :D

+1 to this feature request since I think it also tackles my use-case. I am collaborating with a team, working with a loading script which takes some time to generate the dataset artifacts. It would be very handy to use this as a cloud cache to avoid duplicating the effort.

Currently we could use `builder.download_and_prepare(path_to_cloud_storage, storage_options, ...)`  to cache the artifacts to cloud storage, but then `builder.as_dataset()` yields `NotImplementedError: Loading a dataset cached in SomeCloudFileSystem is not supported`

Makes sense ! If you want to load locally a dataset that you download_and_prepared on a cloud storage, you would use `load_dataset(path_to_cloud_storage)` indeed. It would download the data from the cloud storage, cache them locally, and return a `Dataset`.

It seems currently the `cached_path` function handles all URLs by `get_from_cache` that only supports `ftp` and `http(s)` here:
https://github.com/huggingface/datasets/blob/b5672a956d5de864e6f5550e493527d962d6ae55/src/datasets/utils/file_utils.py#L181

I guess one can add another condition that handles `s3://` or `gs://` URLs via `fsspec` here.

I could use this functionality, so I put together a PR using @kyamagu's suggestion to use `fsspec` in `datasets.utils.file_utils`

https://github.com/huggingface/datasets/pull/5580

Thanks @dwyatte for adding support for fsspec urls

Let me just reopen this since the original issue is not resolved

I'm not yet understanding how to use https://github.com/huggingface/datasets/pull/5580 in order to use `load_dataset(data_files="s3://...")`. Any help/example would be much appreciated :) thanks!

It's still not officially supported x) But you can try to update `request_etag` in `file_utils.py` to use `fsspec_head` instead of `http_head`. It is responsible of getting the ETags of the remote files for caching. This change may do the trick for S3 urls

Thank you for your guys help on this and merging in #5580. I manually pulled the changes to my local datasets package (datasets.utils.file_utils.py) since it only seemed to be this file that was changed in the PR and I'm getting the error:
InvalidSchema: No connection adapters were found for 's3://bucket/folder/'. I'm calling load_dataset using the S3 URI. When I use the S3 URL I get HTTPError: 403 Client Error.
Am I not supposed to use the S3 URI? How do I pull in the changes from this merge? I'm running datasets 2.10.1.

The current implementation depends on gcsfs/s3fs being able to authenticate through some other means e.g., environmental variables. For AWS, it looks like you can set `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN`

Note that while testing this just now, I did note a discrepancy between gcsfs and  s3fs that we might want to address where gcsfs passes the timeout from `storage_options` [here](https://github.com/huggingface/datasets/blob/3e6269979fc80ae8939294d26298897f0db5b84d/src/datasets/utils/file_utils.py#L333) down into the `aiohttp.ClientSession.request`, but s3fs does not handle this (tries to pass to the `aiobotocore.session.AioSession` constructor raising `TypeError: __init__() got an unexpected keyword argument 'requests_timeout'`).

It seems like some work trying to unify kwargs across different fsspec implementations, so if the plan is to pass down `storage_options`, I wonder if we should just let users control the timeout (and other kwargs) using that and if not specified, use the default?

> Note that while testing this just now, I did note a discrepancy between gcsfs and s3fs that we might want to address where gcsfs passes the timeout from storage_options [here](https://github.com/huggingface/datasets/blob/3e6269979fc80ae8939294d26298897f0db5b84d/src/datasets/utils/file_utils.py#L333) down into the aiohttp.ClientSession.request, but s3fs does not handle this (tries to pass to the aiobotocore.session.AioSession constructor raising TypeError: __init__() got an unexpected keyword argument 'requests_timeout').

> It seems like some work trying to unify kwargs across different fsspec implementations, so if the plan is to pass down storage_options, I wonder if we should just let users control the timeout (and other kwargs) and if not specified, use the default?

@lhoestq here's a small PR for this: https://github.com/huggingface/datasets/pull/5673

@lhoestq sorry for being a little dense here but I am very keen to use fsspec / adlfs for for a larger image dataset I have for object detection. I have to keep it on Azure storage and would also like to avoid a full download or zipping (so use `load_dataset(..., streaming=True)`. So this development is godsend :) only... I am unable to make it work.

Would you expect the setup to work for:
- azure blob storage
- image files (not the standard formats `json`, `parquet`....

? I appreciate that you mostly focus on s3 but it seems that, similar to the [remaining cloud storage functionality](https://huggingface.co/docs/datasets/filesystems#cloud-storage), it should also work for Azure blob storage.

I would imagine that something like (Streaming true or false):
```python
d = load_dataset("new_dataset.py", storage_options=storage_options, split="train")
```
would work with
```python
# new_dataset.py
....
_URL="abfs://container/image_folder```

archive_path = dl_manager.download(_URL)
split_metadata_paths = dl_manager.download(_METADATA_URLS)
return [
    datasets.SplitGenerator(
        name=datasets.Split.TRAIN,
        gen_kwargs={
            "annotation_file_path": split_metadata_paths["train"],
            "files": dl_manager.iter_files(archive_path)
},
      ),
...
```
but I get
```
Traceback (most recent call last):
...        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/miniconda3/envs/hf/lib/python3.11/site-packages/datasets/load.py", line 1797, in load_dataset
    builder_instance.download_and_prepare(
  File "~/miniconda3/envs/hf/lib/python3.11/site-packages/datasets/builder.py", line 890, in download_and_prepare
    self._download_and_prepare(
  File "~/miniconda3/envs/hf/lib/python3.11/site-packages/datasets/builder.py", line 1649, in _download_and_prepare
    super()._download_and_prepare(
  File "~/miniconda3/envs/hf/lib/python3.11/site-packages/datasets/builder.py", line 963, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.cache/huggingface/modules/datasets_modules/datasets/new_dataset/dd26a081eab90074f41fa2c821b458424fde393cc73d3d8241aca956d1fb3aa0/new_dataset_script.py", line 56, in _split_generators
    archive_path = dl_manager.download(_URL)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/miniconda3/envs/hf/lib/python3.11/site-packages/datasets/download/download_manager.py", line 427, in download
    downloaded_path_or_paths = map_nested(
                               ^^^^^^^^^^^
  File "~/miniconda3/envs/hf/lib/python3.11/site-packages/datasets/utils/py_utils.py", line 435, in map_nested
    return function(data_struct)
           ^^^^^^^^^^^^^^^^^^^^^
  File "~/miniconda3/envs/hf/lib/python3.11/site-packages/datasets/download/download_manager.py", line 453, in _download
    return cached_path(url_or_filename, download_config=download_config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/miniconda3/envs/hf/lib/python3.11/site-packages/datasets/utils/file_utils.py", line 206, in cached_path
    raise ValueError(f"unable to parse {url_or_filename} as a URL or as a local path")
ValueError: unable to parse abfs://container/image_folder as a URL or as a local path

```

What version of `datasets` are you using ?

@lhoestq
hello, i still have problem with loading json from S3:

storage_options = {
    "key": xxxx,
    "secret": xxx,
    "endpoint_url": xxxx
}
path = 's3://xxx/xxxxxxx.json'
dataset = load_dataset("json", data_files=path, storage_options=storage_options)

and it throws an error:
TypeError: AioSession.__init__() got an unexpected keyword argument 'hf'
and I use the lastest 2.14.4_dev0 version

Hi @lhoestq, thanks for getting back to me :) you have been busy over the summer I see... I was on `2.12.0`. I have updated to `2.14.4`.

Now `d = load_dataset("new_dataset.py", storage_options=storage_options, split="train", streaming=True)` works for Azure blob storage (with a local data loader script) when I explicitly list all blobs  (I am struggling to make `fs.ls(<path>)` work in the script to make the list available to the download manager).

 Any chance that it could work out-of-the-box by supplying just the image folder, not the full list of image filenames? It seems that `dl_manager.download(_URL)` always wants one or more (possibly archived) files. In my situation, where I don't want to archive or download, it would be great to just supply the folder (seems reasonably doable with fsspec).

Let me know if there is anything I can do to help.

Thanks,

> Any chance that it could work out-of-the-box by supplying just the image folder, not the full list of image filenames? It seems that dl_manager.download(_URL) always wants one or more (possibly archived) files. In my situation, where I don't want to archive or download, it would be great to just supply the folder (seems reasonably doable with fsspec).

@mayorblock  This is not supported right now, you have to use archives or implement a way to get the list by yourself

> TypeError: AioSession.init() got an unexpected keyword argument 'hf'

@hjq133 Can you update `fsspec` and try again ?

```python
pip install -U fsspec
```

thanks for your suggestion，it works now !

I'm seeing same problem as @hjq133 with following versions:

```
datasets==2.15.0
(venv) ➜  finetuning-llama2 git:(main) ✗ pip freeze | grep s3fs
s3fs==2023.10.0
(venv) ➜  finetuning-llama2 git:(main) ✗ pip freeze | grep fsspec
fsspec==2023.10.0
```

> @lhoestq hello, i still have problem with loading json from S3:
>
> storage_options = { "key": xxxx, "secret": xxx, "endpoint_url": xxxx } path = 's3://xxx/xxxxxxx.json' dataset = load_dataset("json", data_files=path, storage_options=storage_options)
>
> and it throws an error: TypeError: AioSession.**init**() got an unexpected keyword argument 'hf' and I use the lastest 2.14.4_dev0 version

I am trying to do the same thing, but the loading is just hanging, without any error.
@lhoestq is there any documentation how to load from private s3 buckets?

Hi ! S3 support is still experimental. It seems like there is an extra `hf` field passed to the `s3fs` storage_options that causes this error. I just check the source code of `_prepare_single_hop_path_and_storage_options` and I think you can try passing explicitly your own `storage_options={"s3": {...}}`. Also note that it's generally better to load datasets from HF (we run extensive tests and benchmarks for speed and robustness)

That worked! Thanks
It seems thought that `data_dir=...` doesn't work on s3, only `data_files`.

@lhoestq Would this work either with an Azure Blob Storage Container or its respective Azure Machine Learning Datastore? If yes, what would that look like in code? I've tried a couple of combinations but no success so far, on the latest version of `datasets`. I need to migrate a dataset to the Azure cloud, `load_dataset("path_to_data")` worked perfectly while the files were local only. Thank you!

@mayorblock would you mind sharing how you got it to work? What did you pass as `storage_options`? Would it maybe work without a custom data loader script?

@lhoestq I've been using this feature for the last year on GCS without problem, but I think we need to fix an issue with S3 and then document the supported calling patterns to reduce confusion

It looks like `datasets` uses a default `DownloadConfig` which is where some potentially unintended storage options are getting passed to fsspec

```
DownloadConfig(
	cache_dir=None,
	force_download=False,
	resume_download=False,
	local_files_only=False,
	proxies=None,
	user_agent=None,
	extract_compressed_file=False,
	force_extract=False,
	delete_extracted=False,
	use_etag=True,
	num_proc=None,
	max_retries=1,
	token=None,
	ignore_url_params=False,
	storage_options={'hf': {'token': None, 'endpoint': 'https://huggingface.co'}},
	download_desc=None
)
```
(specifically the `storage_options={'hf': {'token': None, 'endpoint': 'https://huggingface.co'}}` part)

`gcsfs` is robust to the extra key in storage options for whatever reason, but `s3fs` is not (haven't dug into why). I'm unable to test `adlfs` but it looks like people here got it working

Is this an issue that needs fixed with s3fs? Or can we avoid passing these default storage options in some cases?

Update: I think probably https://github.com/huggingface/datasets/pull/6127 is where these default storage options were introduced

Hmm not sure, maybe it has to do with `_prepare_single_hop_path_and_storage_options` returning the "hf" storage options when it shouldn't

Also running into this issue downloading a parquet dataset from S3 (Upload worked fine using [current main branch](https://github.com/huggingface/datasets/commit/90b896193a0a315789022580eb4c80305d168d4d)).
`dataset = Dataset.from_parquet('s3://path-to-file')`
raises
`TypeError: AioSession.__init__() got an unexpected keyword argument 'hf'`
Found that the issue is introduced in [#6028](https://github.com/huggingface/datasets/commit/14f6edd9222e577dccb962ed5338b79b73502fa5#diff-8a868b8c1127c5cbe7870bbeec30bd2cbec6696a378e65b66782a6935e3f412e)

When commenting out the __post_init__ part to set 'hf', I am able to download the dataset.

Would be nice to be able to do

```python
data_files=["s3://..."]  # or gs:// or any cloud storage path
storage_options = {...}
load_dataset(..., data_files=data_files, storage_options=storage_options)
```

The idea would be to use `fsspec` as in `download_and_prepare` and `save_to_disk`.

This has been requested several times already. Some users want to use their data from private cloud storage to train models

related:

https://github.com/huggingface/datasets/issues/3490
https://github.com/huggingface/datasets/issues/5244
[forum](https://discuss.huggingface.co/t/how-to-use-s3-path-with-load-dataset-with-streaming-true/25739/2)


Import error

Hi ! Can you
```python
import platform
print(platform.python_version())
```
to see that it returns ?

Hi,

3.8.13

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Quentin Lhoest ***@***.***>
Sent: Tuesday, November 22, 2022 2:37:02 PM
To: huggingface/datasets ***@***.***>
Cc: feketedavid1012 ***@***.***>; Author ***@***.***>
Subject: Re: [huggingface/datasets] Import error (Issue #5280)

Hi ! Can you

import platform
print(platform.python_version())

to see that it returns ?

—
Reply to this email directly, view it on GitHub<https://github.com/huggingface/datasets/issues/5280#issuecomment-1323691385>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJW7F5YGG32W6WABYC25NJTWJTD75ANCNFSM6AAAAAASHZJ2AU>.
You are receiving this because you authored the thread.Message ID: ***@***.***>

Then it should work as expected if you use the same python when using `datasets`

Please make sure you're running your code in the right environment

It's the right environment. But in if statement I have
"3.8.13" < 3.7
And in the error message is Python>=3.7 which is true in my case (3.8.13 is greater then 3.7), so I don't understand my python should be below the 3.7 which case the if statement is right, but the message is wrong, or above 3.7 which case if statement is wrong, error message is right.

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Quentin Lhoest ***@***.***>
Sent: Tuesday, November 22, 2022 2:41:43 PM
To: huggingface/datasets ***@***.***>
Cc: feketedavid1012 ***@***.***>; Author ***@***.***>
Subject: Re: [huggingface/datasets] Import error (Issue #5280)

Then it should work as expected if you use the same python when using datasets

Please make sure you're running your code in the right environment

—
Reply to this email directly, view it on GitHub<https://github.com/huggingface/datasets/issues/5280#issuecomment-1323697094>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJW7F54JURTAJJWWDO2QGI3WJTERPANCNFSM6AAAAAASHZJ2AU>.
You are receiving this because you authored the thread.Message ID: ***@***.***>

If you're having an error then you're not running your code in the right environment.

https://github.com/huggingface/datasets/blob/cd3d8e637cfab62d352a3f4e5e60e96597b5f0e9/src/datasets/__init__.py#L28

Hy,
I have error at the above line. I have python version 3.8.13, the message says I need python>=3.7, which is True, but I think the if statement not working properly (or the message wrong)


Warn about checksums

_The documentation is not available anymore as the PR was closed or merged._

I'm also in favor of disabling this by default - it's kinda impractical

Great, thanks for the quick turnaround on this!

It takes a lot of time on big datasets to compute the checksums, we should at least add a warning to notify the user about this step. I also mentioned how to disable it, and added a tqdm bar (delay=5 seconds)

cc @ola13


load_dataset does not read jsonl metadata file properly

Can you try to remove "drop_labels=false" ? It may force the loader to infer the labels instead of reading the metadata

Hi, thanks for responding. I tried that, but it does not change anything.

Probably the issue, will report back asap!

Okay, now it seems to actually load the metadata and create the train_split, but it still says only returns "image" and "label", which is always 0 since all images are from same folder

> Can you try updating `datasets` ? Metadata support was added in `datasets` 2.4

Update: This was the issue.

### Describe the bug

Hi, I'm following [this page](https://huggingface.co/docs/datasets/image_dataset) to create a dataset of images and captions via an image folder and a metadata.json file, but I can't seem to get the dataloader to recognize the "text" column. It just spits out "image" and "label" as features.

Below is code to reproduce my exact example/problem.

### Steps to reproduce the bug

```ruby
dataset_link="19Unu89Ih_kP6zsE7f9Mkw8dy3NwHopRF"
id = dataset_link
output = 'Godardv01.zip'
gdown.download(id=id, output=output, quiet=False)

ds = load_dataset("imagefolder", data_dir="/kaggle/working/Volumes/TOSHIBA/Godard_imgs/Volumes/TOSHIBA/Godard_imgs/Full/train", split="train", drop_labels=False)
print(ds)
```

### Expected behavior

I would expect that it returned "image" and "text" columns from the code above.

### Environment info

- `datasets` version: 2.1.0
- Platform: Linux-5.15.65+-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- PyArrow version: 5.0.0
- Pandas version: 1.3.5


Remove YAML integer keys from class_label metadata

Also note that this approach is valid when metadata keys are str, but also if they are int.
- This will be helpful for any community dataset using old integer keys in their metadata

Fix partially #5275.


Well I just sharded the original commonVoice dataset and pushed a small chunk of it in a private rep

What did go wrong?

Holen Sie sich Outlook für iOS<https://aka.ms/o0ukef>
________________________________
Von: Quentin Lhoest ***@***.***>
Gesendet: Tuesday, November 22, 2022 3:03:40 PM
An: huggingface/datasets ***@***.***>
Cc: capsabogdan ***@***.***>; Author ***@***.***>
Betreff: Re: [huggingface/datasets] Bug in downloading common_voice data and snall chunk of it to one's own hub (Issue #5276)

Sounds like one of the file is not a valid one, can you make sure you uploaded valid mp3 files ?

—
Reply to this email directly, view it on GitHub<https://github.com/huggingface/datasets/issues/5276#issuecomment-1323727434>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALSIFOAPAL2V4TBJTSPMAULWJTHDZANCNFSM6AAAAAASHQJ63U>.
You are receiving this because you authored the thread.Message ID: ***@***.***>

It should be all good then !
Could you share a link to your repository for me to investigate what went wrong ?

https://huggingface.co/datasets/DTU54DL/common-voice-test16k

Am Di., 22. Nov. 2022 um 16:43 Uhr schrieb Quentin Lhoest <
***@***.***>:

> It should be all good then !
> Could you share a link to your repository for me to investigate what went
> wrong ?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/5276#issuecomment-1323876682>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ALSIFOEUJRZWXAM7DYA5VJDWJTS3NANCNFSM6AAAAAASHQJ63U>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>

I see ! This is a bug with MP3 files.

When we store audio data in parquet, we store the bytes and the file name. From the file name extension we know if it's a WAV, an MP3 or else. But here it looks like the paths are all None.

It looks like it comes from here:

https://github.com/huggingface/datasets/blob/7feeb5648a63b6135a8259dedc3b1e19185ee4c7/src/datasets/features/audio.py#L212

Cc @polinaeterna maybe we should simply put the file name instead of None values ?

@lhoestq I remember we wanted to avoid storing redundant data but maybe it's not that crucial indeed to store one more string value.
Or we can store paths only for mp3s, considering that for other formats we don't have such a problem with reading from bytes without format specified.

It doesn't cost much to always store the file name IMO

thanks for the help!

can I do anything on my side? we are doing a DL project and we need the
data really quick.

thanks
bogdan

> Message ID: ***@***.***>
>

I opened a pull requests here: https://github.com/huggingface/datasets/pull/5285, we'll do a new release soon with this fix.

Otherwise if you're really in a hurry you can install `datasets` from this PR

[image: image.png]

> Message ID: ***@***.***>
>

any idea on what's going wrong here?

thanks

Am So., 27. Nov. 2022 um 13:53 Uhr schrieb Bogdan Capsa <
***@***.***>:

> [image: image.png]
>
>> Message ID: ***@***.***>
>>
>

hi @capsabogdan!
could you please share more specifically what problem do you have now?

I have attached this screenshot above . can u pls help? So can not pip from pull request

![image](https://user-images.githubusercontent.com/48530104/204354027-6173e6d1-e3d4-4085-a363-e924cfe1a7f4.png)

The pull request has been merged on `main`.
You can install `datasets` from `main` using
```
pip install git+https://github.com/huggingface/datasets.git
```

I've tried to load this dataset DTU54DL/common-voice-test16k, but am
getting the same error.

So the bug fix will fix only if I upload a new dataset, or also loading
previously uploaded datasets?

thanks

Am Mo., 28. Nov. 2022 um 19:51 Uhr schrieb Quentin Lhoest <
***@***.***>:

> The pull request has been merged on main.
> You can install datasets from main using
>
> pip install git+https://github.com/huggingface/datasets.git
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/5276#issuecomment-1329587334>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ALSIFOCNYYIGHM2EX3ZIO6DWKT5MXANCNFSM6AAAAAASHQJ63U>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>

> So the bug fix will fix only if I upload a new dataset, or also loading
previously uploaded datasets?

You have to reupload the dataset, sorry for the inconvenience

thank you so much for the help! works like a charm!

Am Di., 29. Nov. 2022 um 12:15 Uhr schrieb Quentin Lhoest <
***@***.***>:

> So the bug fix will fix only if I upload a new dataset, or also loading
> previously uploaded datasets?
>
> You have to reupload the dataset, sorry for the inconvenience
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/5276#issuecomment-1330468393>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ALSIFOBKEFZO57BAKY4IGW3WKXQUZANCNFSM6AAAAAASHQJ63U>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>

### Describe the bug

I'm trying to load the common voice dataset. Currently there is no implementation to download just par tof the data, and I need just one part of it, without downloading the entire dataset

Help please?

![image](https://user-images.githubusercontent.com/48530104/203260511-26df766f-6013-4eaf-be26-8aa13794def2.png)

### Steps to reproduce the bug

So here is what I have done:
1. Download common_voice data
2. Trim part of it and publish it to my own repo.
3. Download data from my own repo, but am getting this error.

### Expected behavior

There shouldn't be an error in downloading part of the data and publishing it to one's own repo

### Environment info

common_voice 11


YAML integer keys are not preserved Hub server-side

bug

Something isn't working

@huggingface/datasets if you agree, I can make the bulk edit on the Hub to fix integer keys into strings.

Ok for me, and we can merge (internal) https://github.com/huggingface/moon-landing/pull/4609

FYI there are still 2k+ weekly users on `datasets` 2.6.1 which doesn't support the string label format for class labels. And among those, some are using datasets with class labels like imdb (60 users), conllpp (40), msra_ner (40), peoples_daily_enr (40), weibo_ner (30), conll2003 (20), etc. And renaming to string would break these users code.

but isn't `datasets 2.6.1` downloading files from the Hub with the corresponding tag? I thought we had something like this before

We're using `main` as models do. Some datasets need to be updated from time to time, e.g. when a link to download the data is dead.

But yea a year ago we had those tags, we just ended up not using them

I opened https://github.com/huggingface/datasets/issues/5406 to communicate on this. Let me know what you think, and if it sounds good to you I can pin this issue

So, is it OK to make the bulk edit on the Hub now or should we wait longer? If the latter, how long?

I think we can do it. If you want to be extra cautious you can do it for all datasets except imdb and conllpp for now which are actively used by 2.6.1 users. For those two we can keep the YAML like this for some more time, or alternatively use the old dataset_infos.json file

The bulk edit of canonical datasets (except imdb and conllpp) is running.

See e.g.: https://huggingface.co/datasets/acronym_identification/discussions/3

EDITED:
Done, except for "universal_morphologies", where I get
```
HTTPError: 413 Client Error: Payload Too Large for url: https://huggingface.co/api/validate-yaml
```

Also not done for the datasets missing matadata "dataset_info":
- mc4: https://huggingface.co/datasets/mc4/discussions/3
- the_pile: https://huggingface.co/datasets/the_pile/discussions/6
- timit_asr: https://huggingface.co/datasets/timit_asr/discussions/1

@lhoestq, there are 6 community datasets with YAML integer keys in their `dataset_info` `class_label`:
- indonlp/indonlu
- rcds/swiss_judgment_prediction
- Jean-Baptiste/wikiner_fr
- Bingsu/Cat_and_Dog
- taskydata/tasky_or_not
- RCC-MSU/collection3

Maybe we could open a PR on them as well?

Let's do this then:

- [x] [indonlp/indonlu](https://huggingface.co/datasets/indonlp/indonlu/discussions/3)
- [x] rcds/swiss_judgment_prediction
- [x] Jean-Baptiste/wikiner_fr
- [x] Bingsu/Cat_and_Dog -> merged
- [x] taskydata/tasky_or_not (was already using quotes)
- [x] RCC-MSU/collection3

EDIT: all done :)

@lhoestq I was not asking you to do it, but asking if you agree me to do it... :man_facepalming:
As I self-assigned this issue... :sweat_smile:

After an internal discussion (https://github.com/huggingface/moon-landing/issues/4563):
- YAML integer keys are not preserved server-side: they are transformed to strings
  - See for example this Hub PR: https://huggingface.co/datasets/acronym_identification/discussions/1/files
    - Original:
      ```yaml
      class_label:
        names:
          0: B-long
          1: B-short
      ```
    - Returned by the server:
      ```yaml
      class_label:
        names:
          '0': B-long
          '1': B-short
      ```
- They are planning to enforce only string keys
- Other projects already use interger-transformed-to string keys: e.g. `transformers` models `id2label`: https://huggingface.co/roberta-large-mnli/blob/main/config.json
  ```yaml
  "id2label": {
    "0": "CONTRADICTION",
    "1": "NEUTRAL",
    "2": "ENTAILMENT"
  }
  ```

On the other hand, at `datasets` we are currently using YAML integer keys for `dataset_info` `class_label`.

Please note (thanks @lhoestq for pointing out) that previous versions (2.6 and 2.7) of `datasets` need being patched:
```python
In [18]: Features._from_yaml_list([{'dtype': {'class_label': {'names': {'0': 'neg', '1': 'pos'}}}, 'name': 'label'}])
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-18-974f07eea526> in <module>
----> 1 Features._from_yaml_list(ry)

~/Desktop/hf/nlp/src/datasets/features/features.py in _from_yaml_list(cls, yaml_data)
   1743                 raise TypeError(f"Expected a dict or a list but got {type(obj)}: {obj}")
   1744
-> 1745         return cls.from_dict(from_yaml_inner(yaml_data))
   1746
   1747     def encode_example(self, example):

~/Desktop/hf/nlp/src/datasets/features/features.py in from_yaml_inner(obj)
   1739             elif isinstance(obj, list):
   1740                 names = [_feature.pop("name") for _feature in obj]
-> 1741                 return {name: from_yaml_inner(_feature) for name, _feature in zip(names, obj)}
   1742             else:
   1743                 raise TypeError(f"Expected a dict or a list but got {type(obj)}: {obj}")

~/Desktop/hf/nlp/src/datasets/features/features.py in <dictcomp>(.0)
   1739             elif isinstance(obj, list):
   1740                 names = [_feature.pop("name") for _feature in obj]
-> 1741                 return {name: from_yaml_inner(_feature) for name, _feature in zip(names, obj)}
   1742             else:
   1743                 raise TypeError(f"Expected a dict or a list but got {type(obj)}: {obj}")

~/Desktop/hf/nlp/src/datasets/features/features.py in from_yaml_inner(obj)
   1734                             return {"_type": snakecase_to_camelcase(obj["dtype"])}
   1735                     else:
-> 1736                         return from_yaml_inner(obj["dtype"])
   1737                 else:
   1738                     return {"_type": snakecase_to_camelcase(_type), **unsimplify(obj)[_type]}

~/Desktop/hf/nlp/src/datasets/features/features.py in from_yaml_inner(obj)
   1736                         return from_yaml_inner(obj["dtype"])
   1737                 else:
-> 1738                     return {"_type": snakecase_to_camelcase(_type), **unsimplify(obj)[_type]}
   1739             elif isinstance(obj, list):
   1740                 names = [_feature.pop("name") for _feature in obj]

~/Desktop/hf/nlp/src/datasets/features/features.py in unsimplify(feature)
   1704             if isinstance(feature.get("class_label"), dict) and isinstance(feature["class_label"].get("names"), dict):
   1705                 label_ids = sorted(feature["class_label"]["names"])
-> 1706                 if label_ids and label_ids != list(range(label_ids[-1] + 1)):
   1707                     raise ValueError(
   1708                         f"ClassLabel expected a value for all label ids [0:{label_ids[-1] + 1}] but some ids are missing."

TypeError: can only concatenate str (not "int") to str
```

TODO:
- [x] Remove YAML integer keys from `dataset_info` metadata
- [x] Make a patch release for affected `datasets` versions: 2.6 and 2.7
- [x] Communicate on the fix
- [x] Wait for adoption
- [x] Bulk edit the Hub to fix this in all canonical datasets


load_dataset possibly broken for gated datasets?

Btw, thanks very much for finding the hub rollback temporary fix and bringing the issue to our attention @KhoomeiK!

I see the same issue when calling `load_dataset('poloclub/diffusiondb', 'large_random_1k')` with `datasets==2.7.1` and `huggingface-hub=0.11.0`. No issue with `datasets=2.6.1` and `huggingface_hub==0.10.1`.

https://github.com/poloclub/diffusiondb/issues/7

I fixed my issue by specifying `repo_type` in `hf_hub_url()`. https://github.com/poloclub/diffusiondb/commit/9eb91c79aaca98b0515a0ce45778b8af65b84652

I opened a PR on the Winoground's repo: https://huggingface.co/datasets/facebook/winoground/discussions/2

This is a bug in the script, indeed. The most robust fix is to use a relative path instead of `hf_hub_url`, which does not depend on `huggingface_hub`'s version 🙂. I've opened a PR here: https://huggingface.co/datasets/facebook/winoground/discussions/3.

Awesome, big thanks to both @xiaohk and @mariosasko!

so, if i reproduce the bug, what should i do ? with huggingface_hub0.13.3  dataset2.6.1

huggingface_hub.utils._validators.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name':

    tokenizer = AutoTokenizer.from_pretrained(ARGS.model_path, trust_remote_code=True)

Please handle automatically for local path and repo name inside, otherwise users always get confused about this

I think I'm also hitting this error, trying to load a model from a local path.

### Describe the bug

When trying to download the [winoground dataset](https://huggingface.co/datasets/facebook/winoground), I get this error unless I roll back the version of huggingface-hub:

```
[/usr/local/lib/python3.7/dist-packages/huggingface_hub/utils/_validators.py](https://localhost:8080/#) in validate_repo_id(repo_id)
    165     if repo_id.count("/") > 1:
    166         raise HFValidationError(
--> 167             "Repo id must be in the form 'repo_name' or 'namespace/repo_name':"
    168             f" '{repo_id}'. Use `repo_type` argument if needed."
    169         )

HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': 'datasets/facebook/winoground'. Use `repo_type` argument if needed
```

### Steps to reproduce the bug

Install requirements:

```
pip install transformers
pip install datasets
# It works if you uncomment the following line, rolling back huggingface hub:
# pip install huggingface-hub==0.10.1
```

Then:

```
from datasets import load_dataset
auth_token = ""  # Replace with an auth token, which you can get from your huggingface account: Profile -> Settings -> Access Tokens -> New Token
winoground = load_dataset("facebook/winoground", use_auth_token=auth_token)["test"]
```

### Expected behavior

Downloading of the datset

### Environment info

Just a google colab; see here: https://colab.research.google.com/drive/15wwOSte2CjTazdnCWYUm2VPlFbk2NGc0?usp=sharing


download_mode="force_redownload" does not refresh cached dataset

### Describe the bug

`load_datasets` does not refresh dataset when features are imported from external file, even with `download_mode="force_redownload"`. The bug is not limited to nested fields, however it is more likely to occur with nested fields.

### Steps to reproduce the bug

To reproduce the bug 3 files are needed: `dataset.py` (contains dataset loading script), `schema.py` (contains features of dataset) and `main.py` (to run `load_datasets`)

`dataset.py`
```python
import datasets
from schema import features

class NewDataset(datasets.GeneratorBasedBuilder):

    def _info(self):
        return datasets.DatasetInfo(
            features=features
        )

    def _split_generators(self, dl_manager):
        return [
            datasets.SplitGenerator(
                name=datasets.Split.TRAIN
            )
        ]

    def _generate_examples(self):
        data = [
            {"id": 0, "nested": []},
            {"id": 1, "nested": []}
        ]
        for key, example in enumerate(data):
            yield key, example
```

`schema.py`
```python
import datasets
features = datasets.Features(
    {
        "id": datasets.Value("int32"),
        "nested": [
            {"text": datasets.Value("string")}
        ]
    }
)
```

`main.py`
```python
import datasets
a = datasets.load_dataset("dataset.py")
print(a["train"].info.features)
```

Now if `main.py` is run it prints the following correct output: `{'id': Value(dtype='int32', id=None), 'nested': [{'text': Value(dtype='string', id=None)}]}`. However, if f.e. the label of the feature "text" is changed to something else, f.e. to

`schema.py`
```python
import datasets
features = datasets.Features(
    {
        "id": datasets.Value("int32"),
        "nested": [
            {"textfoo": datasets.Value("string")}
        ]
    }
)
```

 `main.py` still prints `{'id': Value(dtype='int32', id=None), 'nested': [{'text': Value(dtype='string', id=None)}]}`, even if run with `download_mode="force_redownload"`. The only fix is to delete the folder in the cache.

### Expected behavior

The cached dataset is deleted and refreshed when using `load_datasets` with `download_mode="force_redownload"`.

### Environment info

- `datasets` version: 2.7.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.9
- PyArrow version: 10.0.0
- Pandas version: 1.3.5


Use pyarrow Tensor dtype

enhancement

New feature or request

Hi ! We're using the Arrow format for the datasets, and PyArrow tensors are not part of the Arrow format AFAIK:

> There is no direct support in the arrow columnar format to store Tensors as column values.

source: https://github.com/apache/arrow/issues/4802#issuecomment-508494694

@wesm @rok its been around three years. any updates, regarding dataset arrow tensor support? 🙏 I know you must be very busy, would appreciate to learn what is the state of art. I saw the PR is still open [#8510](https://github.com/apache/arrow/pull/8510)

Hey @franz101 & @lhoestq!
There is a plan and a PR to create an [ExtensionArray of Tensors](https://github.com/apache/arrow/pull/8510) of equal sizes as well as a plan to do the same for Tensors of different sizes [ARROW-8714](https://issues.apache.org/jira/browse/ARROW-8714).

The work stalled a little because it was not clear where TensorArray would live. However Arrow community recently agreed to make a [well-known-extension-type document](https://lists.apache.org/thread/sxd5fhc42hb6svs79t3fd79gkqj83pfh) and I would like https://github.com/apache/arrow/pull/8510 to land there and add an implementation to C++/Python + another language. Is that something you would find beneficial to you?

that is a great update, thank you.
it looks like this feature would benefit datasets implementation of [ArrayExtensionArray](https://github.com/huggingface/datasets/blob/9f2ff14673cac1f1ad56d80221a793f5938b68c7/src/datasets/features/features.py#L585-L641). Is that correct @eladsegal @lhoestq?

TensorArray sounds great ! Looking forward to it :)

We've had our own ExtensionArray for fixed shape tensors for a while now, hoping to see something more standardized by the arrow community.

Also super interested in the extension array for tensors of different sizes cc @mariosasko

[FixedShapeTensor ExtensionType](https://github.com/apache/arrow/pull/8510) was merged and will be in Arrow 12.0.0 (release is planned mid April).

@rok Thanks for keeping us updated! I think it's best to introduce a new feature type that would use this extension type under the hood. I'll create an issue to discuss the design with the community in the coming days.

Also, is there a tentative time frame for the variable-shape Tensor extension type?

@mariosasko please tag me in the discussion, perhaps I can contribute.

As for the [variable shape tensor array](https://github.com/apache/arrow/issues/24868) - I'd be interested in working on it but didn't see much interest in community yet. Are you saying `huggingface/datasets` could use it?

pyarrow 12 is out 🎉, will have a look if I can work on it for the ExtensionArray

I think these two issues need to be fixed first on the Arrow side before adding the tensor feature type here: https://github.com/apache/arrow/issues/35573 and https://github.com/apache/arrow/issues/35599.

@rok We've had a couple of requests for supporting variable-shape tensors on the forum/GH, but I did not manage to find the concrete issues using the search. TF/TFDS (and PyTorch with the `nested_tensor` API) support them, so it makes sense for us to do the same eventually (the Ray project has an [extension](https://github.com/ray-project/ray/blob/42a8d1489b37243f203120899a23d919dc85bf2a/python/ray/air/util/tensor_extensions/arrow.py#L634) type to support this case)

> @rok We've had a couple of requests for supporting variable-shape tensors on the forum/GH, but I did not manage to find the concrete issues using the search. TF/TFDS (and PyTorch with the `nested_tensor` API) support them, so it makes sense for us to do the same eventually (the Ray project has an [extension](https://github.com/ray-project/ray/blob/42a8d1489b37243f203120899a23d919dc85bf2a/python/ray/air/util/tensor_extensions/arrow.py#L634) type to support this case)

That does make sense indeed. We should probably also be careful about memory layout to enable zero-copy interface to TF/PyTorch.

So there is no way we can use [pyarrow.Tensor](https://arrow.apache.org/docs/python/generated/pyarrow.Tensor.html#pyarrow.Tensor) ?

Not with with the Arrow format, and therefore not in `datasets`. But they released a new [FixedShapeTensorArray](https://arrow.apache.org/docs/python/extending_types.html#fixed-size-tensor) to store tensors in Arrow format. We plan to support this in `datasets` at one point !

There is also an open issue to enable the conversion of `pyarrow.Tensor` to `pyarrow.FixedShapeTensorType`: https://github.com/apache/arrow/issues/35068. This way one could indirectly use `pyarrow.Tensor` in Arrow format.

We started a [mailing list discussion](https://lists.apache.org/thread/qc9qho0fg5ph1dns4hjq56hp4tj7rk1k) about potential `VariableShapeTensor` extension array, please check it out and give feedback. For more details here's also a PR https://github.com/apache/arrow/pull/37166.

### Feature request

I was going the discussion of converting tensors to lists.
Is there a way to leverage pyarrow's Tensors for nested arrays / embeddings?

For example:
```python
import pyarrow as pa
import numpy as np
x = np.array([[2, 2, 4], [4, 5, 100]], np.int32)
pa.Tensor.from_numpy(x, dim_names=["dim1","dim2"])
```

[Apache docs](https://arrow.apache.org/docs/python/generated/pyarrow.Tensor.html)

Maybe this belongs into the pyarrow features / repo.

### Motivation

Working with big data, we need to make sure to use the best data structures and IO out there

### Your contribution

Can try to a PR if code changes necessary


Fix #5269

```
$ datasets-cli convert --datasets_directory <TAB>
datasets_directory
benchmarks/  docs/        metrics/     notebooks/   src/         templates/   tests/       utils/
```


When len(_URLS) > 16, download will hang

It can fix the bug temporarily.
```python
from datasets import DownloadConfig
config = DownloadConfig(num_proc=8)
In [5]: dataset = load_dataset('Freed-Wu/kodak', split='test', download_config=config)
Downloading and preparing dataset kodak/default to /home/wzy/.cache/huggingface/datasets/Freed-Wu___kodak/default/0.0.1/6cf51f2b3d686d24a33fe86945f9e16802def212325f9345cf3cbb1b9f5f4a57...
Downloading data files #4: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:02<00:00,  1.39obj/s]
Downloading data files #2: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:02<00:00,  1.38obj/s]
Downloading data files #3: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:02<00:00,  1.13obj/s]
Downloading data files #7: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:02<00:00,  1.09obj/s]
Downloading data files #5: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:02<00:00,  1.08obj/s]
Downloading data files #0: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:02<00:00,  1.08obj/s]
Downloading data files #1: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:10<00:00,  3.36s/obj]
Downloading data: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 492k/492k [00:01<00:00, 253kB/s]
Downloading data files #6: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:13<00:00,  4.63s/obj]
Extracting data files #0: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1407.17obj/s]
Extracting data files #1: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1325.91obj/s]
Extracting data files #3: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1524.46obj/s]
Extracting data files #2: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1404.66obj/s]
Extracting data files #4: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1538.63obj/s]
Extracting data files #6: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1711.73obj/s]
Extracting data files #7: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 2144.33obj/s]
Extracting data files #5: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1964.85obj/s]
Dataset kodak downloaded and prepared to /home/wzy/.cache/huggingface/datasets/Freed-Wu___kodak/default/0.0.1/6cf51f2b3d686d24a33fe86945f9e16802def212325f9345cf3cbb1b9f5f4a57. Subsequent calls will reuse this data.
```

Thanks for reporting ! This sounds like an issue with python multiprocessing. If we switch to multithreading for the downloads it should be much more robust - let me know if this is something you'd like to contribute, I'd be happy to help and give you some pointers

> an issue with python multiprocessing

If it is an issue with multiprocessing, should we report it to upstream?

Debugging this would require quite some work in my opinion, and I've often failed to make reproducible examples, since it's pretty correlated to one's environment + hardware. So I wouldn't spend too much time on this unless we manage to reproduce this on another machine consistently.

Instead I'd encourage a more pragmatic fix that is: not create tons of processes (on regular machines it may slow things down anyway), and instead use multithreading by default.

I am not expert of python. I hear about python has GIL, which result in multi processing is worse than multi threading. So I am not sure if this change makes sense?

And if this is a bug of multi processing, why not report to upstream and let them fix? And even if change it to multi threading, how can we make sure it can truly fix this problem?

> Just my 2c. No offense.

sure np ^^

> I hear about python has GIL, which result in multi processing is worse than multi threading. So I am not sure if this change makes sense?

Here the bottleneck speed is the bandwidth used to download the files. When downloading, the GIL is released, so multithreading gives the same speed as multiprocessing.

> And if this is a bug of multi processing, why not report to upstream and let them fix?

Usually to fix a bug it's important to be able to reproduce it. This way you can share it, experiment with it, and then make sure it's fixed. Here I'm afraid it's not easy to reproduce. Though I think that spawning too many processes for your machine can lead to this kind of issues.

> And even if change it to multi threading, how can we make sure it can truly fix this problem?

Multithreading is more robust in python because IIRC there are less locks involved which are often the cause of code hanging for no reason.

### Describe the bug

```python
In [9]: dataset = load_dataset('Freed-Wu/kodak', split='test')
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.53k/2.53k [00:00<00:00, 1.88MB/s]
[11/19/22 22:16:21] WARNING  Using custom data configuration default                                                                                builder.py:379
Downloading and preparing dataset kodak/default to /home/wzy/.cache/huggingface/datasets/Freed-Wu___kodak/default/0.0.1/bd1cc3434212e3e654f7e16ad618f8a1470b5982b086c91b1d6bc7187183c6e9...
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 531k/531k [00:02<00:00, 239kB/s]
#10: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:04<00:00,  4.06s/obj]
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 534k/534k [00:02<00:00, 193kB/s]
#14: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:04<00:00,  4.37s/obj]
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 692k/692k [00:02<00:00, 269kB/s]
#12: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:04<00:00,  4.44s/obj]
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 566k/566k [00:02<00:00, 210kB/s]
#5: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:04<00:00,  4.53s/obj]
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 613k/613k [00:02<00:00, 235kB/s]
#13: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:04<00:00,  4.53s/obj]
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 786k/786k [00:02<00:00, 342kB/s]
#3: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:04<00:00,  4.60s/obj]
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 619k/619k [00:02<00:00, 254kB/s]
#4: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:04<00:00,  4.68s/obj]
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 737k/737k [00:02<00:00, 271kB/s]
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 788k/788k [00:02<00:00, 285kB/s]
#6: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:05<00:00,  5.04s/obj]
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 618k/618k [00:04<00:00, 153kB/s]
#0: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:11<00:00,  5.69s/obj]
^CProcess ForkPoolWorker-47:
Process ForkPoolWorker-46:
Process ForkPoolWorker-36:
Process ForkPoolWorker-38:██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:05<00:00,  5.04s/obj]
Process ForkPoolWorker-37:
Process ForkPoolWorker-45:
Process ForkPoolWorker-39:
Process ForkPoolWorker-43:
Process ForkPoolWorker-33:
Process ForkPoolWorker-18:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 364, in get
    with self._rlock:
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.10/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 364, in get
    with self._rlock:
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 364, in get
    with self._rlock:
KeyboardInterrupt
  File "/usr/lib/python3.10/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
KeyboardInterrupt
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 364, in get
    with self._rlock:
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 364, in get
    with self._rlock:
  File "/usr/lib/python3.10/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
  File "/usr/lib/python3.10/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
KeyboardInterrupt
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/usr/lib/python3.10/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 364, in get
    with self._rlock:
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 365, in get
    res = self._reader.recv_bytes()
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 364, in get
    with self._rlock:
  File "/usr/lib/python3.10/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
  File "/usr/lib/python3.10/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 221, in recv_bytes
    buf = self._recv_bytes(maxlength)
KeyboardInterrupt
KeyboardInterrupt
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 419, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 384, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 364, in get
    with self._rlock:
  File "/usr/lib/python3.10/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
Process ForkPoolWorker-20:
Process ForkPoolWorker-44:
Process ForkPoolWorker-22:
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/usr/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 215, in _single_map_nested
    mapped = [_single_map_nested((function, v, types, None, True)) for v in pbar]
  File "/usr/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 215, in <listcomp>
    mapped = [_single_map_nested((function, v, types, None, True)) for v in pbar]
  File "/usr/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 197, in _single_map_nested
    return function(data_struct)
  File "/usr/lib/python3.10/site-packages/datasets/utils/download_manager.py", line 217, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/usr/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 298, in cached_path
    output_path = get_from_cache(
  File "/usr/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 561, in get_from_cache
    response = http_head(
  File "/usr/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 476, in http_head
    response = _request_with_retry(
  File "/usr/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 405, in _request_with_retry
    response = requests.request(method=method.upper(), url=url, timeout=timeout, **params)
  File "/usr/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 358, in connect
    self.sock = conn = self._new_conn()
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
KeyboardInterrupt
#1:   0%|                                                                                | 0/2 [03:00<?, ?obj/s]
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/usr/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 215, in _single_map_nested
    mapped = [_single_map_nested((function, v, types, None, True)) for v in pbar]
  File "/usr/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 215, in <listcomp>
    mapped = [_single_map_nested((function, v, types, None, True)) for v in pbar]
  File "/usr/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 197, in _single_map_nested
    return function(data_struct)
  File "/usr/lib/python3.10/site-packages/datasets/utils/download_manager.py", line 217, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/usr/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 298, in cached_path
    output_path = get_from_cache(
  File "/usr/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 659, in get_from_cache
    http_get(
  File "/usr/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 442, in http_get
    response = _request_with_retry(
  File "/usr/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 405, in _request_with_retry
    response = requests.request(method=method.upper(), url=url, timeout=timeout, **params)
  File "/usr/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 358, in connect
    self.sock = conn = self._new_conn()
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/usr/lib/python3.10/site-packages/urllib3/util/connection.py", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 215, in _single_map_nested
    mapped = [_single_map_nested((function, v, types, None, True)) for v in pbar]
  File "/usr/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
  File "/usr/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 215, in <listcomp>
    mapped = [_single_map_nested((function, v, types, None, True)) for v in pbar]
  File "/usr/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 197, in _single_map_nested
    return function(data_struct)
  File "/usr/lib/python3.10/site-packages/datasets/utils/download_manager.py", line 217, in _download
    return cached_path(url_or_filename, download_config=download_config)
KeyboardInterrupt
  File "/usr/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 298, in cached_path
    output_path = get_from_cache(
  File "/usr/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 561, in get_from_cache
    response = http_head(
  File "/usr/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 476, in http_head
    response = _request_with_retry(
  File "/usr/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 405, in _request_with_retry
    response = requests.request(method=method.upper(), url=url, timeout=timeout, **params)
  File "/usr/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 358, in connect
    self.sock = conn = self._new_conn()
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/lib/python3.10/site-packages/urllib3/util/connection.py", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
KeyboardInterrupt
#3:   0%|                                                                                | 0/2 [03:00<?, ?obj/s]
#11:   0%|                                                                                | 0/1 [00:49<?, ?obj/s]
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/usr/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 215, in _single_map_nested
    mapped = [_single_map_nested((function, v, types, None, True)) for v in pbar]
  File "/usr/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 215, in <listcomp>
    mapped = [_single_map_nested((function, v, types, None, True)) for v in pbar]
  File "/usr/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 197, in _single_map_nested
    return function(data_struct)
  File "/usr/lib/python3.10/site-packages/datasets/utils/download_manager.py", line 217, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/usr/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 298, in cached_path
    output_path = get_from_cache(
  File "/usr/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 561, in get_from_cache
    response = http_head(
  File "/usr/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 476, in http_head
    response = _request_with_retry(
  File "/usr/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 405, in _request_with_retry
    response = requests.request(method=method.upper(), url=url, timeout=timeout, **params)
  File "/usr/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 723, in send
    history = [resp for resp in gen]
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 723, in <listcomp>
    history = [resp for resp in gen]
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 266, in resolve_redirects
    resp = self.send(
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 358, in connect
    self.sock = conn = self._new_conn()
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
KeyboardInterrupt
#5:   0%|                                                                                | 0/1 [03:00<?, ?obj/s]
KeyboardInterrupt

Process ForkPoolWorker-42:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/usr/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 215, in _single_map_nested
    mapped = [_single_map_nested((function, v, types, None, True)) for v in pbar]
  File "/usr/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 215, in <listcomp>
    mapped = [_single_map_nested((function, v, types, None, True)) for v in pbar]
  File "/usr/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 197, in _single_map_nested
    return function(data_struct)
  File "/usr/lib/python3.10/site-packages/datasets/utils/download_manager.py", line 217, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/usr/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 298, in cached_path
    output_path = get_from_cache(
  File "/usr/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 561, in get_from_cache
    response = http_head(
  File "/usr/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 476, in http_head
    response = _request_with_retry(
  File "/usr/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 405, in _request_with_retry
    response = requests.request(method=method.upper(), url=url, timeout=timeout, **params)
  File "/usr/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 358, in connect
    self.sock = conn = self._new_conn()
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/lib/python3.10/site-packages/urllib3/util/connection.py", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
KeyboardInterrupt
#9:   0%|                                                                                | 0/1 [00:51<?, ?obj/s]
```

### Steps to reproduce the bug

```python
"""Kodak.

Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
"""
import datasets

NUMBER = 17
_DESCRIPTION = """\
The pictures below link to lossless, true color (24 bits per pixel, aka "full
color") images. It is my understanding they have been released by the Eastman
Kodak Company for unrestricted usage. Many sites use them as a standard test
suite for compression testing, etc. Prior to this site, they were only
available in the Sun Raster format via ftp. This meant that the images could
not be previewed before downloading. Since their release, however, the lossless
PNG format has been incorporated into all the major browsers. Since PNG
supports 24-bit lossless color (which GIF and JPEG do not), it is now possible
to offer this browser-friendly access to the images.
"""
_HOMEPAGE = "https://r0k.us/graphics/kodak/"
_LICENSE = "GPLv3"
_URLS = [
    f"https://github.com/MohamedBakrAli/Kodak-Lossless-True-Color-Image-Suite/raw/master/PhotoCD_PCD0992/{i}.png"
    for i in range(1, 1 + NUMBER)
]

class Kodak(datasets.GeneratorBasedBuilder):
    """Kodak datasets."""

    VERSION = datasets.Version("0.0.1")

    def _info(self):
        features = datasets.Features(
            {
                "image": datasets.Image(),
            }
        )
        return datasets.DatasetInfo(
            description=_DESCRIPTION,
            features=features,
            homepage=_HOMEPAGE,
            license=_LICENSE,
        )

    def _split_generators(self, dl_manager):
        """Return SplitGenerators."""
        file_paths = dl_manager.download_and_extract(_URLS)
        return [
            datasets.SplitGenerator(
                name=datasets.Split.TEST,
                gen_kwargs={
                    "file_paths": file_paths,
                },
            ),
        ]

    def _generate_examples(self, file_paths):
        """Yield examples."""
        for file_path in file_paths:
            yield file_path, {"image": file_path}
```

### Expected behavior

When `len(_URLS) < 16`, it works.
```python
In [3]: dataset = load_dataset('Freed-Wu/kodak', split='test')
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.53k/2.53k [00:00<00:00, 3.02MB/s]
[11/19/22 22:04:28] WARNING  Using custom data configuration default                                                                                builder.py:379
Downloading and preparing dataset kodak/default to /home/wzy/.cache/huggingface/datasets/Freed-Wu___kodak/default/0.0.1/d26017602a592b5bfa7e008127cdf9dec5af220c9068005f1b4eda036031f475...
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 593k/593k [00:00<00:00, 2.88MB/s]
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 621k/621k [00:03<00:00, 166kB/s]
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 531k/531k [00:01<00:00, 366kB/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:13<00:00,  1.18it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:00<00:00, 3832.38it/s]
Dataset kodak downloaded and prepared to /home/wzy/.cache/huggingface/datasets/Freed-Wu___kodak/default/0.0.1/d26017602a592b5bfa7e008127cdf9dec5af220c9068005f1b4eda036031f475. Subsequent calls will reuse this data.
```

### Environment info

- `datasets` version: 2.7.0
- Platform: Linux-6.0.8-arch1-1-x86_64-with-glibc2.36
- Python version: 3.10.8
- PyArrow version: 9.0.0
- Pandas version: 1.4.4


Shell completions

New feature or request

I don't think we need completion on the datasets-cli, since we're mainly developing huggingface-cli

### Feature request

Like <https://github.com/huggingface/huggingface_hub/issues/1197>, datasets-cli maybe need it, too.

### Motivation

See above.

### Your contribution

Maybe.


Sharded save_to_disk + multiprocessing

_The documentation is not available anymore as the PR was closed or merged._

Added both num_shards and max_shard_size in push_to_hub/save_to_disk. Will take care of updating the tests later

It's ready for a final review @mariosasko and @albertvillanova, let me know what you think :)

Took your comments into account, and also changed `iflatmap_unordered` to take an iterable of kwargs to make the code more redable :)

Added `num_shards=` and `num_proc=` to `save_to_disk()`

EDIT: also added `max_shard_size=` to `save_to_disk()`, and also `num_shards=` to `push_to_hub`

I also:
- deprecated the fs parameter in favor of storage_options (for consistency with the rest of the lib) in save_to_disk and load_from_disk
- always embed the image/audio data in arrow when doing `save_to_disk`
- added a tqdm bar in `save_to_disk`
- Use the MockFileSystem in tests for `save_to_disk` and `load_from_disk`
- removed the unused integration tests with S3, since we can now test with `mockfs` instead of `s3fs`

TODO:
- [x] implem save_to_disk for dataset dict
- [x] save_to_disk for dataset dict tests
- [x] deprecate fs in dataset dict load_from_disk as well
- [x] update docs

Close #5263
Close https://github.com/huggingface/datasets/issues/4196
Close https://github.com/huggingface/datasets/issues/4351


Fix `max_shard_size` docs


Specify arguments as keywords in librosa.reshape to avoid future errors

Fixes a warning and future deprecation from `librosa.reshape`:
```
FutureWarning: Pass orig_sr=16000, target_sr=48000 as keyword args. From version 0.10 passing these as positional arguments will result in an error

   array = librosa.resample(array, sampling_rate, self.sampling_rate, res_type="kaiser_best")
```


Get an IterableDataset from a map-style Dataset

New feature or request

I think `stream` could be misleading since the data is not being streamed from remote endpoints (one could think that's the case when they see `load_dataset` followed by `stream`). Hence, I prefer the second option.

PS: When we resolve https://github.com/huggingface/datasets/issues/4542, we could add `as_tf_dataset` to the API for consistency and deprecate `to_tf_dataset`.

This is useful to leverage iterable datasets specific features like:
- fast approximate shuffling
- lazy map, filter etc.

Iterating over the resulting iterable dataset should be at least as fast at iterating over the map-style dataset.

Here are some ideas regarding the API:
```python
# 1.
# - consistency with load_dataset(..., streaming=True)
# - gives intuition that map/filter/etc. are done on-the-fly
ids = ds.stream()

# 2.
# - more explicit on the output type
# - but maybe sounds like a conversion tool rather than a step in a processing pipeline
ids = ds.as_iterable_dataset()
```