load_dataset gives "403" error when using Financial Phrasebank

Hi @rohitvincent, thanks for reporting.

Unfortunately I'm not able to reproduce your issue:
```python
In [2]: from datasets import load_dataset, DownloadMode
   ...: load_dataset(path='financial_phrasebank',name='sentences_allagree', download_mode="force_redownload")
Downloading builder script: 6.04kB [00:00, 2.87MB/s]
Downloading metadata: 13.7kB [00:00, 7.24MB/s]
Downloading and preparing dataset financial_phrasebank/sentences_allagree (download: 665.91 KiB, generated: 296.26 KiB, post-processed: Unknown size, total: 962.17 KiB) to .../.cache/huggingface/datasets/financial_phrasebank/sentences_allagree/1.0.0/550bde12e6c30e2674da973a55f57edde5181d53f5a5a34c1531c53f93b7e141...
Downloading data: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 682k/682k [00:00<00:00, 7.66MB/s]
Dataset financial_phrasebank downloaded and prepared to .../.cache/huggingface/datasets/financial_phrasebank/sentences_allagree/1.0.0/550bde12e6c30e2674da973a55f57edde5181d53f5a5a34c1531c53f93b7e141. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 918.80it/s]
Out[2]:
DatasetDict({
    train: Dataset({
        features: ['sentence', 'label'],
        num_rows: 2264
    })
})
```

Are you able to access the link? https://www.researchgate.net/profile/Pekka-Malo/publication/251231364_FinancialPhraseBank-v10/data/0c96051eee4fb1d56e000000/FinancialPhraseBank-v10.zip

Yes was able to download from the link manually. But still, get the same error when I use load_dataset.

Fixed once data files are hosted on the Hub:
- #4598

I tried both codes below to download the financial phrasebank dataset (https://huggingface.co/datasets/financial_phrasebank) with the sentences_allagree subset. However, the code gives a 403 error when executed from multiple machines locally or on the cloud.

```
from datasets import load_dataset, DownloadMode
load_dataset(path='financial_phrasebank',name='sentences_allagree',download_mode=DownloadMode.FORCE_REDOWNLOAD)
```

```
from datasets import load_dataset, DownloadMode
load_dataset(path='financial_phrasebank',name='sentences_allagree')
```

**Error**
ConnectionError: Couldn't reach https://www.researchgate.net/profile/Pekka_Malo/publication/251231364_FinancialPhraseBank-v10/data/0c96051eee4fb1d56e000000/FinancialPhraseBank-v10.zip (error 403)


Dataset Viewer issue for TheNoob3131/mosquito-data

dataset-viewer

Related to the dataset viewer on huggingface.co

The preview is working OK:

![Screenshot from 2022-07-21 09-46-09](https://user-images.githubusercontent.com/8515462/180158929-bd8faad4-6392-4fc1-8d9c-df38aa9f8438.png)

### Link

https://huggingface.co/datasets/TheNoob3131/mosquito-data/viewer/TheNoob3131--mosquito-data/test

### Description

Dataset preview not showing with large files. Says 'split cache is empty' even though there are train and test splits.

### Owner

_No response_


Fix broken link to the Hub

_The documentation is not available anymore as the PR was closed or merged._

The Markdown link fails to render if it is in the same line as the `<span>`. This PR implements @mishig25's fix by using `<a href=" ">` instead.

![Screen Shot 2022-07-20 at 3 53 05 PM](https://user-images.githubusercontent.com/59462357/180096412-7fbb33be-abb0-4e54-a52d-201b3b58e0f9.png)


the_pile datasets URL broken.

bug

Something isn't working

Thanks for reporting, @TrentBrick. We are addressing the change with their data host server.

On the meantime, if you would like to work with your fixed local copy of the_pile script, you should use:
```python
load_dataset("path/to/your/local/the_pile/the_pile.py",...
```
instead of just `load_dataset("the_pile",...`.

The latter downloads a copy of `the_pile.py` from our GitHub, caches it locally (inside `~/.cache/huggingface/modules`) and uses that.

@TrentBrick, I have checked the URLs and both hosts work, the original (https://the-eye.eu/) and the mirror (https://mystic.the-eye.eu/). See e.g.:
- https://mystic.the-eye.eu/public/AI/pile/
- https://mystic.the-eye.eu/public/AI/pile_preliminary_components/

Please, let me know if you still find any issue loading this dataset by using current server URLs.

Great this is working now. Re the download from GitHub... I'm sure thought went into doing this but could it be made more clear maybe here? https://huggingface.co/docs/datasets/installation for example under installing from source? I spent over an hour questioning my sanity as I kept trying to edit this file, uninstall and reinstall the repo, git reset to previous versions of the file etc.

Thanks @TrentBrick for the suggestion about improving our docs: we should definitely do this if you find they are not clear enough.

Currently, our docs explain how to load a dataset from a local loading script here: [Load > Local loading script](https://huggingface.co/docs/datasets/loading#local-loading-script)

I've opened an issue here:
- #4732

Feel free to comment on it any additional explanation/suggestion/requirement related to this problem.

https://github.com/huggingface/datasets/pull/3627 changed the Eleuther AI Pile dataset URL from https://the-eye.eu/ to https://mystic.the-eye.eu/ but the latter is now broken and the former works again.

Note that when I git clone the repo and use `pip install -e .` and then edit the URL back the codebase doesn't seem to use this edit so the mystic URL is also cached somewhere else that I can't find?


Download and prepare as Parquet for cloud storage

Added some docs for dask and took your comments into account

cc @philschmid if you also want to take a look :)

Just noticed that it would be more convenient to pass the output dir to download_and_prepare directly, to bypass the caching logic which prepares the dataset at `<cache_dir>/<name>/<version>/<hash>/`. And this way the cache is only used for the downloaded files. What do you think ?

```python

builder = load_datadet_builder("squad")
# or with a custom cache
builder = load_datadet_builder("squad", cache_dir="path/to/local/cache/for/downloaded/files")

# download and prepare to s3
builder.download_and_prepare("s3://my_bucket/squad")
```

Might be of interest:
PyTorch and AWS introduced better support for S3 streaming in `torchtext`.
![image](https://user-images.githubusercontent.com/32632186/183354186-a7f005e3-4167-4d80-ad1a-c62dd51ad7b6.png)

Having thought about it a bit more, I also agree with @philschmid in that it's important to follow the existing APIs (pandas/dask), which means we should support the following at some point:

* remote data files resolution for the packaged modules to support `load_dataset("<format>", data_files="<fs_url>")`
* `to_<format>("<fs_url>")`
* `load_from_disk` and `save_to_disk` already expose the `fs` param, but it would be cool to support specifying `fsspec` URLs directly as the source/destination path (perhaps we can then deprecate `fs` to be fully aligned with pandas/dask)

IMO these are the two main issues with the current approach:
* relying on the builder API to generate the formatted files results in a non-friendly format due to how our caching works (a lot of nested subdirectories)
* this approach still downloads the files needed to generate a dataset locally. Considering one of our goals is to align the streaming API with the non-streaming one, this could be avoided by running `to_<format>` on streamed/iterable datasets

Alright I did the last change I wanted to do, here is the final API:

```python
builder = load_dataset_builder(...)
builder.download_and_prepare("s3://...", storage_options={"token": ...})
```

and it creates the arrow files directly in the specified directory, not in a nested subdirectory structure as we do in the cache !

> this approach still downloads the files needed to generate a dataset locally. Considering one of our goals is to align the streaming API with the non-streaming one, this could be avoided by running to_<format> on streamed/iterable datasets

Yup this can be explored in some future work I think. Though to keep things simple and clear I would keep the streaming behaviors only when you load a dataset in streaming mode, and not include it in `download_and_prepare` (because it wouldn't be aligned with the name of the function, which imply to 1. download and 2. prepare ^^). Maybe an API like that can make sense for those who need full streaming

```python
ds = load_dataset(..., streaming=True)
ds.to_parquet("s3://...")
```

totally agree with your comment on the meaning of "loading", I'll update the docs

I took your comments into account and reverted all the changes related to `cache_dir` to keep the support for remote `cache_dir` for beam datasets. I also updated the wording in the docs to not use "load" when it's not appropriate :)

Download a dataset as Parquet in a cloud storage can be useful for streaming mode and to use with spark/dask/ray.

This PR adds support for `fsspec` URIs like `s3://...`, `gcs://...` etc. and ads the `file_format` to save as parquet instead of arrow:
```python
from datasets import *

cache_dir = "s3://..."
builder = load_dataset_builder("crime_and_punish", cache_dir=cache_dir)
builder.download_and_prepare(file_format="parquet")
```

EDIT: actually changed the API to

```python
from datasets import *

builder = load_dataset_builder("crime_and_punish")
builder.download_and_prepare("s3://...", file_format="parquet")
```

credentials to cloud storage can be passed using the `storage_options` argument in

For consistency with the BeamBasedBuilder, I name the parquet files `{builder.name}-{split}-xxxxx-of-xxxxx.parquet`. I think this is fine since we'll need to implement parquet sharding after this PR, so that a dataset can be used efficiently with dask for example.

Note that images/audio files are not embedded yet in the parquet files, this will added in a subsequent PR

TODO:
- [x] docs
- [x] tests


Refactor conftest fixtures

Previously, fixture modules `hub_fixtures` and `s3_fixtures`:
- were both at the root test directory
- were imported using `import *`
  - as a side effect, the modules `os` and `pytest` were imported from `s3_fixtures` into `conftest`

This PR:
- puts both fixture modules in a dedicated directory `fixtures`
- renames both to: `fixtures.hub` and `fixtures.s3`
- imports them into `conftest` as plugins, using the `pytest_plugins`: this avoids the `import *`
- additionally creates a new fixture module `fixtures.files` with all file-related fixtures


Docs: Fix same-page haslinks

`href="/docs/datasets/quickstart#audio"` implicitly goes to `href="/docs/datasets/{$LATEST_STABLE_VERSION}/quickstart#audio"`. Therefore, https://huggingface.co/docs/datasets/quickstart#audio #audio hashlink does not work since the new docs were not added to v2.3.2 (LATEST_STABLE_VERSION)

to preserve the version, it should be just `href="#audio"`, which will implicilty go to curren_page + #audio element


PyArrow Dataset error when calling `load_dataset`

Something isn't working

Hi ! It looks like a bug in `pyarrow`. If you manage to end up with only one chunk per parquet file it should workaround this issue.

To achieve that you can try to lower the value of `max_shard_size` and also don't use `map` before `push_to_hub`.

Do you have a minimum reproducible example that we can share with the Arrow team for further debugging ?

> If you manage to end up with only one chunk per parquet file it should workaround this issue.

Yup, I did not encounter this bug when I was testing my script with a slice of <1000 samples for my dataset.

> Do you have a minimum reproducible example...

Not sure if I can get more minimal than the script I shared above. Are you asking for a sample json file?
Just generate a random manifest list, I can add that to the above script if that's what you mean?

Actually this is probably linked to this open issue: https://issues.apache.org/jira/browse/ARROW-5030.

setting `max_shard_size="2GB"` should do the job (or `max_shard_size="1GB"` if you want to be on the safe side, especially given that there can be some variance in the shard sizes if the dataset is not evenly distributed)

## Describe the bug

I am fine tuning a wav2vec2 model following the script here using my own dataset: https://github.com/huggingface/transformers/blob/main/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py

Loading my Audio dataset from the hub which was originally generated from disk results in the following PyArrow error:

```sh
File "/home/ubuntu/w2v2/run_speech_recognition_ctc.py", line 227, in main
  raw_datasets = load_dataset(
File "/home/ubuntu/.virtualenvs/meval/lib/python3.8/site-packages/datasets/load.py", line 1679, in load_dataset
  builder_instance.download_and_prepare(
File "/home/ubuntu/.virtualenvs/meval/lib/python3.8/site-packages/datasets/builder.py", line 704, in download_and_prepare
  self._download_and_prepare(
File "/home/ubuntu/.virtualenvs/meval/lib/python3.8/site-packages/datasets/builder.py", line 793, in _download_and_prepare
  self._prepare_split(split_generator, **prepare_split_kwargs)
File "/home/ubuntu/.virtualenvs/meval/lib/python3.8/site-packages/datasets/builder.py", line 1268, in _prepare_split
  for key, table in logging.tqdm(
File "/home/ubuntu/.virtualenvs/meval/lib/python3.8/site-packages/tqdm/std.py", line 1195, in __iter__
  for obj in iterable:
File "/home/ubuntu/.virtualenvs/meval/lib/python3.8/site-packages/datasets/packaged_modules/parquet/parquet.py", line 68, in _generate_tables
  for batch_idx, record_batch in enumerate(
File "pyarrow/_parquet.pyx", line 1309, in iter_batches
File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Nested data conversions not implemented for chunked array outputs
```

## Steps to reproduce the bug

I created a dataset from a JSON lines manifest of `audio_filepath`, `text`, and `duration`.

When creating the dataset, I do something like this:

```python
import json
from datasets import Dataset, Audio

# manifest_lines is a list of dicts w/ "audio_filepath", "duration", and "text
for line in manifest_lines:
    line = line.strip()
    if line:
        line_dict = json.loads(line)
        manifest_dict["audio"].append(f"{root_path}/{line_dict['audio_filepath']}")
        manifest_dict["duration"].append(line_dict["duration"])
        manifest_dict["transcription"].append(line_dict["text"])

# Create a HF dataset
dataset = Dataset.from_dict(manifest_dict).cast_column(
    "audio", Audio(sampling_rate=16_000),
)

# From the docs for saving to disk
# https://huggingface.co/docs/datasets/v2.3.2/en/package_reference/main_classes#datasets.Dataset.save_to_disk
def read_audio_file(example):
    with open(example["audio"]["path"], "rb") as f:
        return {"audio": {"bytes": f.read()}}

dataset = dataset.map(read_audio_file, num_proc=70)
dataset.save_to_disk(f"/audio-data/hf/{artifact_name}")
dataset.push_to_hub(f"{org-name}/{artifact_name}", max_shard_size="5GB", private=True)
```

Then when I call `load_dataset()` in my training script, with the same dataset I generated above, and download from the huggingface hub I get the above stack trace.
I am able to load the dataset fine if I use `load_from_disk()`.

## Expected results

`load_dataset()` should behave just like `load_from_disk()` and not cause any errors.

## Actual results

See above

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
I am using the `huggingface/transformers-pytorch-gpu:latest` image
- `datasets` version: 2.3.0
- Platform: Docker/Ubuntu 20.04
- Python version: 3.8
- PyArrow version: 8.0.0


Dataset Viewer issue for shamikbose89/lancaster_newsbooks

It seems like the list of splits could not be obtained:

```python
>>> from datasets import get_dataset_split_names
>>> get_dataset_split_names("shamikbose89/lancaster_newsbooks", "default")
Using custom data configuration default
Traceback (most recent call last):
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 354, in get_dataset_config_info
    for split_generator in builder._split_generators(
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/shamikbose89--lancaster_newsbooks/2d1c63d269bf7b9342accce0a95960b1710ab4bc774248878bd80eb96c1afaf7/lancaster_newsbooks.py", line 73, in _split_generators
    data_dir = dl_manager.download_and_extract(_URL)
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 916, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 879, in extract
    urlpaths = map_nested(self._extract, path_or_paths, map_tuple=True)
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 348, in map_nested
    return function(data_struct)
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 884, in _extract
    protocol = _get_extraction_protocol(urlpath, use_auth_token=self.download_config.use_auth_token)
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 388, in _get_extraction_protocol
    return _get_extraction_protocol_with_magic_number(f)
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 354, in _get_extraction_protocol_with_magic_number
    f.seek(0)
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/fsspec/implementations/http.py", line 684, in seek
    raise ValueError("Cannot seek streaming HTTP file")
ValueError: Cannot seek streaming HTTP file

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 404, in get_dataset_split_names
    info = get_dataset_config_info(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 359, in get_dataset_config_info
    raise SplitsNotFoundError("The split names could not be parsed from the dataset config.") from err
datasets.inspect.SplitsNotFoundError: The split names could not be parsed from the dataset config.
```

ping @huggingface/datasets

Oh, I removed the 'split' key from `kwargs`. I put  it back in, but there's still the same error

It looks like the data host doesn't support http range requests, which is necessary to glob inside a ZIP archive in streaming mode. Can you try hosting the dataset elsewhere ? Or download each file separately from https://ota.bodleian.ox.ac.uk/repository/xmlui/handle/20.500.12024/2531 ?

@lhoestq Thanks! That seems to have solved it. I can get the splits with the `get_dataset_split_names()` function. The dataset viewer is still not loading properly, though. The new error is
```
Status code:   400
Exception:     BadZipFile
Message:       File is not a zip file
```

PS. The dataset loads properly and can be accessed

### Link

https://huggingface.co/datasets/shamikbose89/lancaster_newsbooks

### Description

Status code:   400
Exception:     ValueError
Message:       Cannot seek streaming HTTP file

I am able to use the dataset loading script locally and it also runs when I'm using the one from the hub, but the viewer still doesn't load

### Owner

Yes


Issue loading TheNoob3131/mosquito-data dataset

I am also getting a ValueError: 'Couldn't cast' at the bottom. Is this because of some delimiter issue? My dataset is on the Huggingface Hub. If you could look at it, that would be greatly appreciated.

Hi @thenerd31, thanks for reporting.

Please note that your issue is not caused by the Hugging Face Datasets library, but it has to do with the specific implementation of your dataset on the Hub.

Therefore, I'm transferring this discussion to your own dataset Community tab: https://huggingface.co/datasets/TheNoob3131/mosquito-data/discussions/1

![image](https://user-images.githubusercontent.com/53668030/179815591-d75fa7d3-3122-485f-a852-b06a68909066.png)

So my dataset is public in the Huggingface Hub, but when I try to load it using the load_dataset command, it shows that it is downloading the files, but throws a ValueError. When I went to my directory to see if the files were downloaded, the folder was blank.

Here is the error below:
ValueError                                Traceback (most recent call last)
Input In [8], in <cell line: 3>()
      1 from datasets import load_dataset
----> 3 dataset = load_dataset("TheNoob3131/mosquito-data", split="train")

File ~\Anaconda3\lib\site-packages\datasets\load.py:1679, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1676 try_from_hf_gcs = path not in _PACKAGED_DATASETS_MODULES
   1678 # Download and prepare data
-> 1679 builder_instance.download_and_prepare(
   1680     download_config=download_config,
   1681     download_mode=download_mode,
   1682     ignore_verifications=ignore_verifications,
   1683     try_from_hf_gcs=try_from_hf_gcs,
   1684     use_auth_token=use_auth_token,
   1685 )
   1687 # Build dataset for splits
   1688 keep_in_memory = (
   1689     keep_in_memory if keep_in_memory is not None else is_small_dataset(builder_instance.info.dataset_size)
   1690 )

Is the dataset in the wrong format or is there some security permission that I should enable?


Make Extractor accept Path as input

_The documentation is not available anymore as the PR was closed or merged._

This PR:
- Makes `Extractor` accept instance of `Path` as input
- Removes unnecessary castings of `Path` to `str`


Dataset Viewer issue for LawalAfeez/englishreview-ds-mini

It's currently working, as far as I understand

https://huggingface.co/datasets/LawalAfeez/englishreview-ds-mini/viewer/LawalAfeez--englishreview-ds-mini/train

<img width="1556" alt="Capture d’écran 2022-07-19 à 09 24 01" src="https://user-images.githubusercontent.com/1676121/179761130-2d7980b9-c0f6-4093-8b1d-f0a3872fef3f.png">

---

What was your issue?

### Link

_No response_

### Description

Unable to view the split data

### Owner

_No response_


Support "tags" yaml tag

IMO `DatasetMetadata` shouldn't crash with attributes that it doesn't know, btw

Yea this PR is mostly to have a validation that this field contains a list of strings.

Regarding unknown fields, the tagging app currently returns an error if a field is unknown using the `DatasetMetadata`. We can change that though

Added the "tags" YAML tag, so that users can specify data domain/topics keywords for dataset search


Fix POS tags

CI failures are about missing content in the dataset cards or bad tags, and this is unrelated to this PR. Merging :)

We're now using `part-of-speech` and not `part-of-speech-tagging`, see discussion here: https://github.com/huggingface/datasets/commit/114c09aff2fa1519597b46fbcd5a8e0c0d3ae020#r78794777


Fix named split sorting and remove unnecessary casting

hahaha what a timing, I added my comment right after you merged x)

you can ignore my (nit), it's fine

This PR:
- makes `NamedSplit` sortable: so that `sorted()` can be called on them
- removes unnecessary `sorted()` on `dict.keys()`: `dict_keys` view is already like a `set`
- removes unnecessary casting of `NamedSplit` to `str`


Document installation of sox OS dependency for audio

The `sox` OS package needs being installed manually using the distribution package manager.

This PR adds this explanation to the docs.


Highlight non-commercial license in amazon_reviews_multi dataset card

Highlight that the licence granted by Amazon only covers non-commercial research use.


Document how to create a dataset loading script for audio/vision

documentation

Improvements or additions to documentation

Currently, in our docs for Audio/Vision/Text, we explain how to:
- Load data
- Process data

However we only explain how to *Create a dataset loading script* for text data.

I think it would be useful that we add the same for Audio/Vision as these have some specificities different from Text.

See, for example:
- #4697
  - and comment there: https://github.com/huggingface/datasets/issues/4697#issuecomment-1191502492

CC: @stevhliu


Add object detection processing tutorial

Great idea! Now that we have more than one task, it makes sense to separate image classification and object detection so it'll be easier for users to follow.

@lhoestq do we want to do that in this PR, or should we merge it and let @stevhliu reorganize separately?

The following adds a quick guide on how to process object detection datasets with `albumentations`.


WMT21 & WMT22

good first issue

Good for newcomers

dataset request

Requesting to add a new dataset

Hi ! That would be awesome to have them indeed, thanks for opening this issue

I just added you to the WMT org on the HF Hub if you're interested in adding those datasets.

Feel free to create a dataset repository for each dataset and upload the data files there :) preferably in ZIP archives instead of TAR archives (the current WMT scripts don't support streaming TAR archives, so it would break the dataset preview). We've also had issues with the `statmt.org` host (data unavailable, slow download speed), that's why I think it's better if we re-host the files on the Hub.

`wmt21` (and wmt22) can be added in this GitHub repository I think, for consistency with the previous ones.
To add it, you can copy paste the code of the previous one (e.g. wmt19), and add the new data:
- in wmt_utils.py, add the new data subsets. You need to provide the download URLs, as well as the target and source languages
- in wmt21.py (renamed from wmt19.py), you can specify the subsets that WMT21 uses (i.e. the one you just added)
- in wmt_utils.py, define the python function that must be used to parse the subsets you added. To do so, you must go in `_generate_examples` and chose the proper `sub_generator` based on the subset name. For example, the `paracrawl_v3` subset uses the `_parse_tmx` function:

https://github.com/huggingface/datasets/blob/ede72d3f9796339701ec59899c7c31d2427046fb/datasets/wmt19/wmt_utils.py#L834-L835

Hopefully the data is in a format that is already supported and there's no need to write a new `_parse_*` function for the new subsets. Let me know if you have questions or if I can help :)

## Adding a Dataset
- **Name:** WMT21 & WMT22
- **Description:** We are going to have three tracks: two small tasks and a large task.
The small tracks evaluate translation between fairly related languages and English (all pairs). The large track uses 101 languages.
- **Paper:** /
- **Data:** https://statmt.org/wmt21/large-scale-multilingual-translation-task.html https://statmt.org/wmt22/large-scale-multilingual-translation-task.html
- **Motivation:** Many more languages than previous WMT versions - Could be very high impact

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md).

I could also tackle this. I saw the existing logic for WMT models is a bit complex (datasets are stored on the wmt account & retrieved in separate wmt datasets afaict). How long do you think it would take me? @lhoestq


Fix require torchaudio and refactor test requirements

Currently there is a bug in `require_torchaudio` (indeed it is requiring `sox` instead):
```python
def require_torchaudio(test_case):
    if find_spec("sox") is None:
...
```

The bug was introduced by:
- #3685
  - Commit: https://github.com/huggingface/datasets/pull/3685/commits/b5a3e7122d49c4dcc9333b1d8d18a833fc04b940

which moved
```python
require_sndfile = pytest.mark.skipif(
    # In Windows and OS X, soundfile installs sndfile
    (sys.platform != "linux" and find_spec("soundfile") is None)
    # In Linux, soundfile throws RuntimeError if sndfile not installed with distribution package manager
    or (sys.platform == "linux" and find_library("sndfile") is None),
    reason="Test requires 'sndfile': `pip install soundfile`; "
    "Linux requires sndfile installed with distribution package manager, e.g.: `sudo apt-get install libsndfile1`",
)
require_sox = pytest.mark.skipif(
    find_library("sox") is None,
    reason="Test requires 'sox'; only available in non-Windows, e.g.: `sudo apt-get install sox`",
)
require_torchaudio = pytest.mark.skipif(find_spec("torchaudio") is None, reason="Test requires 'torchaudio'")
```
to
```python
def require_sndfile(test_case):
    """
    Decorator marking a test that requires soundfile.
    These tests are skipped when soundfile isn't installed.
    """
    if (sys.platform != "linux" and find_spec("soundfile") is None) or (
        sys.platform == "linux" and find_library("sndfile") is None
    ):
        test_case = unittest.skip(
            "test requires 'sndfile': `pip install soundfile`; "
            "Linux requires sndfile installed with distribution package manager, e.g.: `sudo apt-get install libsndfile1`",
        )(test_case)
    return test_case

def require_sox(test_case):
    """
    Decorator marking a test that requires sox.
    These tests are skipped when sox isn't installed.
    """
    if find_library("sox") is None:
        return unittest.skip("test requires 'sox'; only available in non-Windows, e.g.: `sudo apt-get install sox`")(
            test_case
        )
    return test_case

def require_torchaudio(test_case):
    """
    Decorator marking a test that requires torchaudio.
    These tests are skipped when torchaudio isn't installed.
    """
    if find_spec("sox") is None:
        return unittest.skip("test requires 'torchaudio'")(test_case)
    return test_case
```

This PR;
- fixes the bug in `require_torchaudio`
- refactors the test requirements back to using `pytest` instead of `unittest`
  - the text in `pytest.skipif` `reason` can be used if needed in a test body: `require_torchaudio.kwargs["reason"]`


Thanks for reporting. I refreshed the dataset viewer and it now works as expected.

https://huggingface.co/datasets/TheNoob3131/mosquito-data

<img width="1135" alt="Capture d’écran 2022-07-18 à 13 15 22" src="https://user-images.githubusercontent.com/1676121/179566497-e47f1a27-fd84-4a8d-9d7f-2e0f2da803df.png">

We will investigate why it occurred in the first place

By chance, could you provide some details about the operations done on the dataset: was it private? gated?

Yes, it was a private dataset, and when I made it public, the Dataset Preview did not work.

However, now when I make the dataset private, it says that the Dataset Preview has been disabled. Why is this?

Thanks for the details. For now, the dataset viewer is always disabled on private datasets (see https://huggingface.co/docs/hub/datasets-viewer for more details)

Hi, it was working fine for a few hours, but then I can't see the dataset viewer again (public dataset). Why is this still happening?
It's the same error too:
![image](https://user-images.githubusercontent.com/53668030/179602465-f220f971-d3aa-49ba-a31b-60510f4c2a89.png)

OK? This is a bug, thanks for help spotting and reproducing it (it occurs when a dataset is switched to private, then to public). We will be working on it, meanwhile, I've restored the dataset viewer manually again.

### Link

_No response_

### Description

Getting this error when trying to view dataset preview:
Message:       401, message='Unauthorized', url=URL('https://huggingface.co/datasets/TheNoob3131/mosquito-data/resolve/8aceebd6c4a359d216d10ef020868bd9e8c986dd/0_Africa_train.csv')

### Owner

_No response_


Fix empty examples in xtreme dataset for bucc18 config

I guess the report link is this instead: https://huggingface.co/datasets/xtreme/discussions/1

As reported in https://huggingface.co/muibk, there are empty examples in xtreme/bucc18.de

I applied your fix @mustaszewski

I also used a dict to make the dataset generation much faster


Fix crd3

As reported in https://huggingface.co/datasets/crd3/discussions/1#62cc377073b2512b81662794, each split of the dataset was containing the same data. This issues comes from a bug in the dataset script

I fixed it and also uploaded the data to hf.co to make the dataset work in streaming mode


Skip tests only for lz4/zstd params if not installed

Currently, if `zstandard` or `lz4` are not installed, `test_compression_filesystems` and `test_streaming_dl_manager_extract_all_supported_single_file_compression_types` are skipped for all compression format parameters.

This PR fixes these tests, so that if `zstandard` or `lz4` are not installed, the tests are skipped only for the corresponding compression parameters (`zstd` or `lz4`), whereas the tests are not skipped for all the other compression parameters (`gzip`, `xz` and `bz2`).

Related to:
- #4688


Make cast in `from_pandas` more robust

Make the cast in `from_pandas` more robust (as it was done for the packaged modules in https://github.com/huggingface/datasets/pull/4364)

This should be useful in situations like [this one](https://discuss.huggingface.co/t/loading-custom-audio-dataset-and-fine-tuning-model/8836/4).


Domain specific dataset discovery on the Hugging Face hub

enhancement

New feature or request

> Hi! I added a link to this issue in our internal request for adding keywords/topics to the Hub, which is identical to the `topic tags` solution. The `collections` solution seems too complex (as you point out). Regarding the `domain tags` solution, we primarily focus on machine learning, so I'm not sure if it's a good idea to make our current taxonomy more complex.

Thanks, for letting me know. Will you allow the topic tags to be user-generated or only chosen from a list?

Thanks for opening this issue @davanstrien.

As we discussed last week, the tag approach would be in principle the simpler to be implemented, either the domain tag (with closed vocabulary: more reliable but also more rigid), or the topic tag (with open vocabulary: more flexible for user needs)

Hi @davanstrien If i remember correctly this was also discussed inside a hf.co Discussion, would you be able to link it here too?

(where i suggested using `tags: - foo - bar` IIRC.

Thanks a ton!

> Hi @davanstrien If i remember correctly this was also discussed inside a hf.co Discussion, would you be able to link it here too?
>
> (where i suggested using `tags: - foo - bar` IIRC.
>
> Thanks a ton!

This doesn't ring a bell - I did a quick search of https://discuss.huggingface.co but didn't find anything.

The `tags: ` approach sounds like a good option for this. It would be especially nice if these could suggest existing tags, but this probably won't be easily possible through the current interface.

I opened a PR to add "tags" to the YAML validator:
https://github.com/huggingface/datasets/pull/4716

I also added "tags" to the [tagging app](https://huggingface.co/spaces/huggingface/datasets-tagging), with suggestions like "bio" or "newspapers"

Thanks @lhoestq for the initiative.

Just one question: are "tags" already supported on the Hub?

I think they aren't. Thus, the Hub should support them so that they are properly displayed.

> I think they're not displayed, but at least it should enable users to filter by tag in using `huggingface_hub` or using the appropriate query params on the website (not sure if it's possible yet though)

I think this would already be a helpful start. I'm happy to try this out with the datasets added to https://huggingface.co/organizations/biglam and use the `huggingface_hub` to filter those datasets using the tags.

**Is your feature request related to a problem? Please describe.**

## The problem

The datasets hub currently has `8,239` datasets. These datasets span a wide range of different modalities and tasks (currently with a bias towards textual data).

There are various ways of identifying datasets that may be relevant for a particular use case:

- searching
- various filters

Currently, however, there isn't an easy way to identify datasets belonging to a specific domain. For example, I want to browse machine learning datasets related to 'social science' or 'climate change research'.

The ability to identify datasets relating to a specific domain has come up in discussions around the [BigLA](https://github.com/bigscience-workshop/lam/) datasets hackathon https://github.com/bigscience-workshop/lam/discussions/31#discussioncomment-3123610. As part of the hackathon, we're currently collecting datasets related to Libraries, Archives and Museums and making them available via the hub. We currently do this under a Hugging Face organization (https://huggingface.co/biglam). However, going forward, I can see some of these datasets being migrated to sit under an organization that is the custodian of the dataset (for example, a national library the data was originally from). At this point, it becomes more difficult to quickly identify datasets from this domain without relying on search.

This is also related to some existing issues on Github related to metadata on the hub:
- https://github.com/huggingface/datasets/issues/3625
- https://github.com/huggingface/datasets/issues/3877

**Describe the solution you'd like**

### Some possible solutions that may help with this:

#### Enable domain tags (from a controlled vocabulary)
- This would add metadata field to the YAML for the domain a dataset relates to
- Advantages:
	- the list is controlled, allowing it to be more easily integrated into the datasets tag app (https://huggingface.co/space/huggingface/datasets-tagging)
	- the controlled vocabulary could align with an existing controlled vocabulary
	- this additional metadata can be used to perform filtering by domain
- disadvantages
	- choosing the best controlled vocab may be difficult
	- there are many datasets that are likely to fit into the 'machine learning' domain (i.e. there is a long tail of datasets that aren't in more 'generic' machine learning domain

#### Enable topic tags (user-generated)

Enable 'free form' topic tags for datasets and models. This would be closer to GitHub's repository topics which can be chosen from a controlled list (https://github.com/topics/) but can also be more user/org specific. This could potentially be useful for organizations to also manage their own models and datasets as the number they hold in their org grows. For example, they may create 'topic tags' for a specific project, so it's clearer which datasets /models are related to that project.

#### Collections

This solution would likely be the biggest shift and may require significant changes in the hub fronted. Collections could work in several different ways but would include:

Users can curate particular datasets, models, spaces, etc., into a collection. For example, they may create a collection of 'historic newspapers suitable for training language models'. These collections would not be mutually exclusive, i.e. a dataset can belong to zero, one or many collections. Collections can also potentially be nested under other collections.

This is fairly common on other data reposotiores for example the following collections:
<img width="293" alt="Screenshot 2022-07-18 at 11 50 44" src="https://user-images.githubusercontent.com/8995957/179496445-963ed122-5e26-4574-96e8-41081bce3e2b.png">

all belong under a higher level collection (https://bl.iro.bl.uk/collections/353c908d-b495-4413-b047-87236d2573e3?locale=en).

There are different models one could use for how these collections could be created:

- only within an org
- for any dataset/model
- the owner or a dataset/model has to agree to be added to a collection
- a collection owner can have people suggest additions to their collection
- other models....

These collections could be thematic, related to particular training approaches, curate models with particular inference properties etc. Whilst some of these features may duplicate current/or future tag filters on the hub, they offer the advantage of being flexible and not having to predict what users will want to do upfront.

There is also potential for automating the creation of these collections based on existing metadata. For example, one could collect models trained on a collection of datasets so for example, if we had a collection of 'historic newspapers suitable for training language models' that contained 30 datasets, we could create another collection 'historic newspaper language models' that takes any model on the hub whose metadata says it used one or more of those 30 datasets.

There is also the option of exploring ML approaches to suggest models/datasets may be relevant to a particular collection.

This approach is likely to be quite difficult to implement well and would require significant thought. There is also likely to be a benefit in doing quite a bit of upfront work in curating useful collections to demonstrate the benefits of collections.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

It is possible to collate this information externally, i.e. one could link back to the relevant models/datasets from an external platform.

**Additional context**
Add any other context about the feature request here.

I'm cc'ing others involved in the BigLAM hackathon who may also have thoughts @cakiki @clancyoftheoverflow @albertvillanova


Added more information in the README about contributors of the Arabic Speech Corpus

Added more information in the README about contributors and encouraged reading the thesis for more infos


Support extract lz4 compressed data files

_The documentation is not available anymore as the PR was closed or merged._


Fix Authentification Error while streaming

Hi, thanks for working on this, but the fix for this has already been merged in https://github.com/huggingface/datasets/pull/4608.

I fixed a few errors when it occurs while streaming the private dataset on the Huggingface Hub.
```
from datasets import load_dataset

dataset = load_dataset(<repo_id>, use_auth_token=<private_token>, streaming=True)
for d in dataset['train']:
    print(d)
    break # this is for checking
```
This code is an example for streaming private datasets.
when the version of the datasets is 2.2.2, it works well but datasets>2.2.2 occurs error like this,
```
/usr/local/lib/python3.7/dist-packages/aiohttp/client_reqrep.py in raise_for_status(self)
1007 status=self.status,
1008 message=self.reason,
→ 1009 headers=self.headers,
1010 )
1011

ClientResponseError: 401, message='Unauthorized', url=URL('https://huggingface.co/datasets/.../train-00000-of-00001-168b451062c67c34.parquet')
```
(this is an example on the dataset has `parquet` extenstion)
It seems that the `xisfile `module in `download/streaming_download_manager.py` couldn't recognize the file on "https://huggingface.co/~".

so I add three lines.
With this change, there is no error anymore(but this code is ad-hoc).


Enable streaming dataset to use the "all" split

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4698). All of your documentation changes will be reflected on that endpoint.

@albertvillanova
Adding the validation split causes these two `assert_called_once` assertions to fail with `AssertionError: Expected 'ArrowWriter' to have been called once. Called 2 times`:

https://github.com/huggingface/datasets/blob/main/tests/test_builder.py#L548-L562

It might be better to create a new dummy generator for the streaming tests, WDYT? Alternatively we could test for `self.call_count` equalling 2.

@cakiki have you read my comment in the issue page?
https://github.com/huggingface/datasets/issues/4637#issuecomment-1175984812

Streaming with `split=all` seems to be working, will fix the failing test next

Not sure if marking the PR as "ready for review" actually notified you, so tagging @albertvillanova just in case :smiley_cat:

Fixes #4637


Trouble with streaming frgfm/imagenette vision dataset with TAR archive

Hi @frgfm, thanks for reporting.

As the error message says, streaming mode is not supported out of the box when the dataset contains TAR archive files.

To make the dataset streamable, you have to use `dl_manager.iter_archive`.

There are several examples in other datasets, e.g. food101: https://huggingface.co/datasets/food101/blob/main/food101.py

And yes, as the link you pointed out, for the streaming to be possible, the metadata file must be loaded before all of the images:
- either this is the case when iterating the archive (and you get the metadata file before the images)
- or you have to extract the metadata file by hand and upload it separately to the Hub

Hi @albertvillanova :wave:

Thanks! Yeah I saw that but since I didn't have any metadata, I wasn't sure whether I should create them myself.

So one last question:
What is the metadata supposed to be for archives? The relative path of all files in it?
_(Sorry I'm a bit confused since it's quite hard to debug using the single error message from the data preview :sweat_smile: )_

Hi @frgfm, streaming a dataset that contains a TAR file requires some tweaks because (contrary to ZIP files), tha TAR archive does not allow random access to any of the contained member files. Instead they have to be accessed sequentially (in the order in which they were put into the TAR file when created) and yielded.

So when iterating over the TAR file content, when an image file is found, we need to yield it (and not keeping it in memory, which will require huge RAM memory for large datasets). But when yielding an image file, we also need to yield with it what we call "metadata": the class label, and other textual information (for example, for audio files, sometimes we also add info such as the speaker ID, their sex, their age,...).

All this information usually is stored in what we call the metadata file: either a JSON or a CSV/TSV file.

But if this is also inside the TAR archive, we need to find this file in the first place when iterating the TAR archive, so that we already have this information when we find an image file and we can yield the image file and its metadata info.

Therefore:
- either the TAR archive contains the metadata file as the first member when iterating it (something we cannot change as it is done at the creation of the TAR file)
- or if not, then we need to have the metadata file elsewhere
  - in these cases, what we do (if the dataset license allows it) is:
    - we download the TAR file locally, we extract the metadata file and we host the metadata on the Hub
    - we modify the dataset loading script so that it first downloads the metadata file (and reads it) and only then starts iterating the content of the TAR archive file

See an example of this process we recently did for "google/fleurs" (their metadata files for "train" were at the end of the TAR archives, after all audio files): https://huggingface.co/datasets/google/fleurs/discussions/4
- we uploaded the metadata file to the Hub
- we adapted the loading script to use it

Hi @albertvillanova :wave:

Thanks, since my last message, I went through the repo of https://huggingface.co/datasets/food101/blob/main/food101.py and managed to get it to work in the end :pray:

Here it is: https://huggingface.co/datasets/frgfm/imagenette

I appreciate you opening an issue to document the process, it might help a few!

Great to see that you manage to make your dataset streamable. :rocket:

I'm closing this issue, as for the docs update there is another issue opened:
- #4711

### Link

https://huggingface.co/datasets/frgfm/imagenette

### Description

Hello there :wave:

Thanks for the amazing work you've done with HF Datasets! I've just started playing with it, and managed to upload my first dataset. But for the second one, I'm having trouble with the preview since there is some archive extraction involved :sweat_smile:

Basically, I get a:
```
Status code:   400
Exception:     NotImplementedError
Message:       Extraction protocol for TAR archives like 'https://s3.amazonaws.com/fast-ai-imageclas/imagenette2.tgz' is not implemented in streaming mode. Please use `dl_manager.iter_archive` instead.
```

I've tried several things and checked this issue https://github.com/huggingface/datasets/issues/4181 as well, but no luck so far!

Could you point me in the right direction please? :pray:

### Owner

Yes


Cannot load LinCE dataset

bug

Something isn't working

Hi @finiteautomata, thanks for reporting.

Unfortunately, I'm not able to reproduce your issue:
```python
In [1]: from datasets import load_dataset
   ...: dataset = load_dataset("lince", "ner_spaeng")
Downloading builder script: 20.8kB [00:00, 9.09MB/s]
Downloading metadata: 31.2kB [00:00, 13.5MB/s]
Downloading and preparing dataset lince/ner_spaeng (download: 2.93 MiB, generated: 18.45 MiB, post-processed: Unknown size, total: 21.38 MiB) to .../.cache/huggingface/datasets/lince/ner_spaeng/1.0.0/10d41747f55f0849fa84ac579ea1acfa7df49aa2015b60426bc459c111b3d589...
Downloading data: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.08M/3.08M [00:01<00:00, 2.73MB/s]
Dataset lince downloaded and prepared to .../.cache/huggingface/datasets/lince/ner_spaeng/1.0.0/10d41747f55f0849fa84ac579ea1acfa7df49aa2015b60426bc459c111b3d589. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 630.66it/s]

In [2]: dataset
Out[2]:
DatasetDict({
    train: Dataset({
        features: ['idx', 'words', 'lid', 'ner'],
        num_rows: 33611
    })
    validation: Dataset({
        features: ['idx', 'words', 'lid', 'ner'],
        num_rows: 10085
    })
    test: Dataset({
        features: ['idx', 'words', 'lid', 'ner'],
        num_rows: 23527
    })
})
```

Please note that for this dataset, the original data files are not hosted on the Hugging Face Hub, but on https://ritual.uh.edu
And sometimes, the server might be temporarily unavailable, as your error message said (trying to connect to the server timed out):
```
ConnectionError: Couldn't reach https://ritual.uh.edu/lince/libaccess/eyJ1c2VybmFtZSI6ICJodWdnaW5nZmFjZSBubHAiLCAidXNlcl9pZCI6IDExMSwgImVtYWlsIjogImR1bW15QGVtYWlsLmNvbSJ9/ner_spaeng.zip (ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='ritual.uh.edu', port=443): Max retries exceeded with url: /lince/libaccess/eyJ1c2VybmFtZSI6ICJodWdnaW5nZmFjZSBubHAiLCAidXNlcl9pZCI6IDExMSwgImVtYWlsIjogImR1bW15QGVtYWlsLmNvbSJ9/ner_spaeng.zip (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7feb1c45a690>, 'Connection to ritual.uh.edu timed out. (connect timeout=100)'))")))
```
In these cases you could:
- either contact the owners of the data server where the data is hosted to inform them about the issue in their server
- or re-try after waiting some time: usually these issues are just temporary

## Describe the bug

Cannot load LinCE dataset due to a connection error

## Steps to reproduce the bug
```python

from datasets import load_dataset

dataset = load_dataset("lince", "ner_spaeng")
```

A notebook with this code and corresponding error can be found at https://colab.research.google.com/drive/1pgX3bNB9amuUwAVfPFm-XuMV5fEg-cD2

## Expected results

It should load the dataset

## Actual results

```python
---------------------------------------------------------------------------
ConnectionError                           Traceback (most recent call last)
<ipython-input-2-fc551ddcebef> in <module>()
      1 from datasets import load_dataset
      2
----> 3 dataset = load_dataset("lince", "ner_spaeng")

10 frames
/usr/local/lib/python3.7/dist-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1682         ignore_verifications=ignore_verifications,
   1683         try_from_hf_gcs=try_from_hf_gcs,
-> 1684         use_auth_token=use_auth_token,
   1685     )
   1686

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    703                     if not downloaded_from_gcs:
    704                         self._download_and_prepare(
--> 705                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    706                         )
    707                     # Sync info

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos)
   1219
   1220     def _download_and_prepare(self, dl_manager, verify_infos):
-> 1221         super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
   1222
   1223     def _get_examples_iterable_for_split(self, split_generator: SplitGenerator) -> ExamplesIterable:

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    769         split_dict = SplitDict(dataset_name=self.name)
    770         split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 771         split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    772
    773         # Checksums verification

/root/.cache/huggingface/modules/datasets_modules/datasets/lince/10d41747f55f0849fa84ac579ea1acfa7df49aa2015b60426bc459c111b3d589/lince.py in _split_generators(self, dl_manager)
    481     def _split_generators(self, dl_manager):
    482         """Returns SplitGenerators."""
--> 483         lince_dir = dl_manager.download_and_extract(f"{_LINCE_URL}/{self.config.name}.zip")
    484         data_dir = os.path.join(lince_dir, self.config.data_dir)
    485         return [

/usr/local/lib/python3.7/dist-packages/datasets/download/download_manager.py in download_and_extract(self, url_or_urls)
    429             extracted_path(s): `str`, extracted paths of given URL(s).
    430         """
--> 431         return self.extract(self.download(url_or_urls))
    432
    433     def get_recorded_sizes_checksums(self):

/usr/local/lib/python3.7/dist-packages/datasets/download/download_manager.py in download(self, url_or_urls)
    313             num_proc=download_config.num_proc,
    314             disable_tqdm=not is_progress_bar_enabled(),
--> 315             desc="Downloading data files",
    316         )
    317         duration = datetime.now() - start_time

/usr/local/lib/python3.7/dist-packages/datasets/utils/py_utils.py in map_nested(function, data_struct, dict_only, map_list, map_tuple, map_numpy, num_proc, types, disable_tqdm, desc)
    346     # Singleton
    347     if not isinstance(data_struct, dict) and not isinstance(data_struct, types):
--> 348         return function(data_struct)
    349
    350     disable_tqdm = disable_tqdm or not logging.is_progress_bar_enabled()

/usr/local/lib/python3.7/dist-packages/datasets/download/download_manager.py in _download(self, url_or_filename, download_config)
    333             # append the relative path to the base_path
    334             url_or_filename = url_or_path_join(self._base_path, url_or_filename)
--> 335         return cached_path(url_or_filename, download_config=download_config)
    336
    337     def iter_archive(self, path_or_buf: Union[str, io.BufferedReader]):

/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py in cached_path(url_or_filename, download_config, **download_kwargs)
    195             use_auth_token=download_config.use_auth_token,
    196             ignore_url_params=download_config.ignore_url_params,
--> 197             download_desc=download_config.download_desc,
    198         )
    199     elif os.path.exists(url_or_filename):

/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag, max_retries, use_auth_token, ignore_url_params, download_desc)
    531         _raise_if_offline_mode_is_enabled(f"Tried to reach {url}")
    532         if head_error is not None:
--> 533             raise ConnectionError(f"Couldn't reach {url} ({repr(head_error)})")
    534         elif response is not None:
    535             raise ConnectionError(f"Couldn't reach {url} (error {response.status_code})")

ConnectionError: Couldn't reach https://ritual.uh.edu/lince/libaccess/eyJ1c2VybmFtZSI6ICJodWdnaW5nZmFjZSBubHAiLCAidXNlcl9pZCI6IDExMSwgImVtYWlsIjogImR1bW15QGVtYWlsLmNvbSJ9/ner_spaeng.zip (ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='ritual.uh.edu', port=443): Max retries exceeded with url: /lince/libaccess/eyJ1c2VybmFtZSI6ICJodWdnaW5nZmFjZSBubHAiLCAidXNlcl9pZCI6IDExMSwgImVtYWlsIjogImR1bW15QGVtYWlsLmNvbSJ9/ner_spaeng.zip (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7feb1c45a690>, 'Connection to ritual.uh.edu timed out. (connect timeout=100)'))")))
```

## Environment info

- `datasets` version: 2.3.2
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5


Add MANtIS dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4695). All of your documentation changes will be reflected on that endpoint.

This PR adds MANtIS dataset.
Arxiv: [https://arxiv.org/abs/1912.04639](https://arxiv.org/abs/1912.04639)
Github: [https://github.com/Guzpenha/MANtIS](https://github.com/Guzpenha/MANtIS)

README and dataset tags are WIP.


Distributed data parallel training for streaming datasets

enhancement

New feature or request

Hi ! According to https://huggingface.co/docs/datasets/use_with_pytorch#stream-data you can use the pytorch DataLoader with `num_workers>0` to distribute the shards across your workers (it uses `torch.utils.data.get_worker_info()` to get the worker ID and select the right subsets of shards to use)

### Feature request

Any documentations for the the `load_dataset(streaming=True)` for (multi-node multi-GPU) DDP training?

### Motivation

Given a bunch of data files, it is expected to split them onto different GPUs. Is there a guide or documentation?

### Your contribution

Does it requires manually split on data files for each worker in `DatasetBuilder._split_generator()`? What is`IterableDatasetShard` expected to do?


update `samsum` script

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4693). All of your documentation changes will be reflected on that endpoint.

update `samsum` script after #4672 was merged (citation is also updated)


Unable to cast a column with `Image()` by using the `cast_column()` feature

bug

Something isn't working

Hi, thanks for reporting! A PR (https://github.com/huggingface/datasets/pull/4614) has already been opened to address this issue.

## Describe the bug
A clear and concise description of what the bug is.

When I create a dataset, then add a column to the created dataset through the `dataset.add_column` feature and then try to cast a column of the dataset (this column contains image paths) with `Image()` by using the `cast_column()` feature, I get the following error - ``` TypeError: Couldn't cast array of type
string
to
{'bytes': Value(dtype='binary', id=None), 'path': Value(dtype='string', id=None)} ```

When I try and cast the same column, but without doing the `add_column` in the previous step, it works as expected.

## Steps to reproduce the bug
```python
from datasets import Dataset, Image

data_dict = {
    "img_path": ["https://picsum.photos/200/300"]
}

dataset = Dataset.from_dict(data_dict)

#NOTE Comment out this line and use cast_column and it works properly
dataset = dataset.add_column("yeet", [1])

#NOTE This line fails to execute properly if `add_column` is called before
dataset = dataset.cast_column("img_path", Image())

# #NOTE This is my current workaround. This seems to work fine with/without `add_column`. While
# # running this, make sure to comment out the `cast_column` line
# new_features = dataset.features.copy()
# new_features["img_path"] = Image()
# dataset = dataset.cast(new_features)

print(dataset)
print(dataset.features)
print(dataset[0])
```

## Expected results
A clear and concise description of the expected results.

Able to successfully use `cast_column` to cast a column containing img_paths to now be Image() features after modifying the dataset using `add_column` in a previous step

## Actual results
Specify the actual results or traceback.

```
Traceback (most recent call last):
  File "/home/surya/Desktop/hf_bug_test.py", line 14, in <module>
    dataset = dataset.cast_column("img_path", Image())
  File "/home/surya/anaconda3/envs/snap_test/lib/python3.9/site-packages/datasets/fingerprint.py", line 458, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/surya/anaconda3/envs/snap_test/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 1580, in cast_column
    dataset._data = dataset._data.cast(dataset.features.arrow_schema)
  File "/home/surya/anaconda3/envs/snap_test/lib/python3.9/site-packages/datasets/table.py", line 1487, in cast
    new_tables.append(subtable.cast(subschema, *args, **kwargs))
  File "/home/surya/anaconda3/envs/snap_test/lib/python3.9/site-packages/datasets/table.py", line 834, in cast
    return InMemoryTable(table_cast(self.table, *args, **kwargs))
  File "/home/surya/anaconda3/envs/snap_test/lib/python3.9/site-packages/datasets/table.py", line 1897, in table_cast
    return cast_table_to_schema(table, schema)
  File "/home/surya/anaconda3/envs/snap_test/lib/python3.9/site-packages/datasets/table.py", line 1880, in cast_table_to_schema
    arrays = [cast_array_to_feature(table[name], feature) for name, feature in features.items()]
  File "/home/surya/anaconda3/envs/snap_test/lib/python3.9/site-packages/datasets/table.py", line 1880, in <listcomp>
    arrays = [cast_array_to_feature(table[name], feature) for name, feature in features.items()]
  File "/home/surya/anaconda3/envs/snap_test/lib/python3.9/site-packages/datasets/table.py", line 1673, in wrapper
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/home/surya/anaconda3/envs/snap_test/lib/python3.9/site-packages/datasets/table.py", line 1673, in <listcomp>
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/home/surya/anaconda3/envs/snap_test/lib/python3.9/site-packages/datasets/table.py", line 1846, in cast_array_to_feature
    raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{feature}")
TypeError: Couldn't cast array of type
string
to
{'bytes': Value(dtype='binary', id=None), 'path': Value(dtype='string', id=None)}
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Ubuntu 20.04.3 LTS
- Python version: 3.9.7
- PyArrow version: 7.0.0


Dataset Viewer issue for rajistics/indian_food_images

Hi, thanks for reporting. I triggered a refresh of the preview for this dataset, and it works now. I'm not sure what occurred.
<img width="1019" alt="Capture d’écran 2022-07-18 à 11 01 52" src="https://user-images.githubusercontent.com/1676121/179541327-f62ecd5e-a18a-4d91-b316-9e2ebde77a28.png">

### Link

https://huggingface.co/datasets/rajistics/indian_food_images/viewer/rajistics--indian_food_images/train

### Description

I have a train/test split in my dataset
<img width="410" alt="Screen Shot 2022-07-15 at 11 44 42 AM" src="https://user-images.githubusercontent.com/6808012/179293215-7b419ec3-3527-46f2-8dad-adbc5568cfa0.png">
t
The dataset viewer works for the test split (images of indian food), but does not show my train split.  My guess is maybe there is some corrupt image file that is guessing this.  But I have no idea.

The original dataset was pulled from here: https://www.kaggle.com/datasets/l33tc0d3r/indian-food-classification?resource=download-directory

### Owner

Yes


Refactor base extractors

_The documentation is not available anymore as the PR was closed or merged._

This PR:
- Refactors base extractors as subclasses of `BaseExtractor`:
  - this is an abstract class defining the interface with:
    - `is_extractable`: abstract class method
    - `extract`: abstract static method
- Implements abstract `MagicNumberBaseExtractor` (as subclass of `BaseExtractor`):
  - this has a default implementation of `is_extractable`
  - this improves performance (reducing the number of file reads) by allowing passing already read `magic_number`
- Refactors `Extractor`:
  - reads magic number from file only once

This PR deprecates:
```python
is_extractable, extractor = self.extractor.is_extractable(input_path, return_extractor=True)
self.extractor.extract(input_path, output_path, extractor=extractor)
```
and uses more Pythonic instead:
```python
extractor_format = self.extractor.infer_extractor_format(input_path)
self.extractor.extract(input_path, output_path, extractor_format)
```


Test extractors for all compression formats

This PR:
- Adds all compression formats to `test_extractor`
- Tests each base extractor for all compression formats

Note that all compression formats are tested except "rar".


Skip test_extractor only for zstd param if zstandard not installed

Currently, if `zstandard` is not installed, `test_extractor` is skipped for all compression format parameters.

This PR fixes `test_extractor` so that if `zstandard` is not installed, `test_extractor` is skipped only for the `zstd` compression parameter, that is, it is not skipped for all the other compression parameters (`gzip`, `xz`,...).


Trigger CI also on push to main

Currently, new CI (on GitHub Actions) is only triggered on pull requests branches when the base branch is main.

This PR also triggers the CI when a PR is merged to main branch.


Align logging with Transformers (again)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4686). All of your documentation changes will be reflected on that endpoint.

I wasn't aware of https://github.com/huggingface/datasets/pull/1845 before opening this PR. This issue seems much more complex now ...

Fix #2832


Fix mock fsspec

This PR:
- Removes an unused method from `DummyTestFS`
- Refactors `mock_fsspec` to make it simpler


How to assign new values to Dataset?

enhancement

New feature or request

Hi! One option is use `map` with a function that overwrites the labels (`dset = dset.map(lamba _: {"label": 0}, features=dset.features`)). Or you can use the `remove_column` + `add_column` combination (`dset = dset.remove_columns("label").add_column("label", [0]*len(data)).cast(dset.features)`, but note that this approach creates an in-memory table for the added column instead of writing to disk, which could be problematic for large datasets.

![image](https://user-images.githubusercontent.com/37113676/179149159-bbbda0c8-a661-403c-87ed-dc2b4219cd68.png)

Hi, if I want to change some values of the dataset, or add new columns to it, how can I do it?

For example, I want to change all the labels of the SST2 dataset to `0`:
```python
from datasets import load_dataset
data = load_dataset('glue','sst2')

data['train']['label'] = [0]*len(data)
```

I will get the error:
```
TypeError: 'Dataset' object does not support item assignment
```


Update create dataset card docs

This PR proposes removing the [online dataset card creator](https://huggingface.co/datasets/card-creator/) in favor of simply copy/pasting a template and using the [Datasets Tagger app](https://huggingface.co/spaces/huggingface/datasets-tagging) to generate the tags. The Tagger app provides more guidance by showing all possible values a user can select in the dropdown menus, whereas the online dataset card creator doesn't, which can make it difficult to know what tag values to input.

Let me know what you think! :)


weird issue/bug with columns (dataset iterable/stream mode)

I have a dataset online (CloverSearch/cc-news-mutlilingual) that has a bunch of columns, two of which are "score_title_maintext" and "score_title_description". the original files are jsonl formatted. I was trying to iterate through via streaming mode and grab all "score_title_description" values, but I kept getting key not found after a certain point of iteration. I found that some json objects in the file don't have "score_title_description". And in SOME cases, this returns a NONE and in others it just gets a key error. Why is there an inconsistency here and how can I fix it?


IndexError when loading ImageFolder

bug

Something isn't working

Hi, thanks for reporting! If there are no examples in ImageFolder, the `label` column is of type `ClassLabel(names=[])`, which leads to an error in [this line](https://github.com/huggingface/datasets/blob/c15b391942764152f6060b59921b09cacc5f22a6/src/datasets/arrow_writer.py#L387) as `asdict(info)` calls `Features({..., "label": {'num_classes': 0, 'names': [], 'id': None, '_type': 'ClassLabel'}})`, which then calls `require_decoding` [here](https://github.com/huggingface/datasets/blob/c15b391942764152f6060b59921b09cacc5f22a6/src/datasets/features/features.py#L1516) on the dict value it does not expect.

I see two ways to fix this:
* custom `asdict` where `dict_factory` is also applied on the `dict` object itself besides dataclasses (the built-in implementation calls `type(dict_obj)` - this means we also need to fix `Features.to_dict` btw)
* implement `DatasetInfo.to_dict` (though adding `to_dict` to a data class is a bit weird IMO)

@lhoestq Which one of these approaches do you like more?

Small pref for the first option, it feels weird to know that `Features()` can be called with a dictionary of types defined as dictionaries instead of type instances.

## Describe the bug
Loading an image dataset with `imagefolder` throws `IndexError: list index out of range` when the given folder contains a non-image file (like a csv).

## Steps to reproduce the bug
Put a csv file in a folder with images and load it:
```python
import datasets
 datasets.load_dataset("imagefolder", data_dir=path/to/folder)
```

## Expected results
I would expect a better error message, like `Unsupported file` or even the dataset loader just ignoring every file that is not an image in that case.

## Actual results
Here is the whole traceback:

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Linux-5.11.0-051100-generic-x86_64-with-glibc2.27
- Python version: 3.9.9
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Dataset Viewer issue for codeparrot/xlcost-text-to-code

There seems to be an issue with the `C++-snippet-level` config:

```python
>>> from datasets import get_dataset_split_names
>>> get_dataset_split_names("codeparrot/xlcost-text-to-code", "C++-snippet-level")
Traceback (most recent call last):
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 352, in get_dataset_config_info
    info.splits = {
TypeError: 'NoneType' object is not iterable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 404, in get_dataset_split_names
    info = get_dataset_config_info(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 359, in get_dataset_config_info
    raise SplitsNotFoundError("The split names could not be parsed from the dataset config.") from err
datasets.inspect.SplitsNotFoundError: The split names could not be parsed from the dataset config.
```

I remove the dataset-viewer tag since it's not directly related.

Pinging @huggingface/datasets

Thanks I found that this subset wasn't properly defined the the config, I fixed it. Now I can see the subsets but I get this error for the viewer
````
Status code:   400
Exception:     Status400Error
Message:       The split cache is empty.
```

Yes, the cache is being refreshed, hopefully, it will work in some minutes for all the splits. Some are already here:

https://huggingface.co/datasets/codeparrot/xlcost-text-to-code/viewer/Python-snippet-level/train

<img width="1533" alt="Capture d’écran 2022-07-18 à 12 04 06" src="https://user-images.githubusercontent.com/1676121/179553933-64d874fa-ada9-4b82-900e-082619523c20.png">

I think all the splits are working as expected now

### Link

https://huggingface.co/datasets/codeparrot/xlcost-text-to-code

### Description

Error
```
Server Error
Status code:   400
Exception:     TypeError
Message:       'NoneType' object is not iterable
```
Before I did a minor change in the dataset script (removing some comments), the viewer was working but not properely, it wasn't showing the dataset subsets. But the data can be loaded successfully.

Thanks!

### Owner

Yes


Added method to remove excess nesting in a DatasetDict

Hi ! I think the issue you linked is closed and suggests to use `remove_columns`.

Moreover if you end up with a dataset with an unnecessarily nested data, please modify your processing functions to not output nested data, or use `map(..., batched=True)` if you function take batches as input

Hi @lhoestq , you are right about the issues this pull has steered beyond that issue. I created this [colab notebook](https://colab.research.google.com/drive/16aLu6QrDSV_aUYRdpufl5E4iS08qkUGj?usp=sharing) to present the error. I tried using batch and that won't resolve it either. I'm looking into that error right now.

I think you just need to pass one example at a time to your tokenizer, this way you don't end up with nested data:
```python

def preprocessFunction(row):
  collatedContext = tokenizer.eos_token.join([row["context"+str(i+1)] for i in range(int(AMT_OF_CONTEXT))])
  response = row["response"]
  tokenizedContext = tokenizer(
    collatedContext, max_length=max_context_length, truncation=True  # don't pass as a list here
  )
  with tokenizer.as_target_tokenizer():
    tokenized_response = tokenizer(
      response, max_length=max_response_length, truncation=True  # don't pass a a list here
    )
  tokenizedContext["labels"] = tokenized_response["input_ids"]
  return tokenizedContext
```

Yes that is correct, the purpose of this pull is to advise of a more general solution like with `def remove_excess_nesting(self)` or maybe automate the solution (stas00 advised not to automate it as it could "not be backwards compatible").

I'm not sure I understand how having `remove_excess_nesting` would make more sense than just fixing the preprocessFunction to simply not return nested samples, can you elaborate ?

Figuring out the issue can be a bit difficult to figure out. Only until I added batch does it make a little more sense with the error

> sequence item 0: expected str instance, list found

but batch was never intended.

When you run the colab you will notice that only until collating do you learn there is this error. So i figured it would be better to address it during at the `DatasetDict` level.
I think it would be ideal if the user could be notified at the preprocess function.

I'm not arguing that `remove_excess_nesting` is the right solution but what I aim to address is dealing with unnecessary nesting as early as possible.

> When you run the colab you will notice that only until collating do you learn there is this error.

I think users can just check the `dataset.features` and they would notice that the data are nested
```python
{
   'input_ids': Sequence(Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None), length=-1, id=None)
   ...
}
```

Sometime nested data are intentional, so you can't know in advance if it's a user's mistake or something planned.

Yes, I understand, it could be intentional and only the collator has problems with it. So, it is not worth handling it any differently in any other non-erroneous data.

That being said do you think there is any use for the `remove_excess_nesting` method? Or maybe it should be applied in a different way? If not feel free to close this PR.

I think users can write it and use `map` themselves if needed, it is pretty straightforward to implement.

I'm closing this PR if you don't mind, and thank you for the discussion :)

No problem @lhoestq , thanks for walking me through it.

Added the ability for a DatasetDict to remove additional nested layers within its features to avoid conflicts when collating. It is meant to accompany [this PR](https://github.com/huggingface/transformers/pull/18119) to resolve the same issue [#15505](https://github.com/huggingface/transformers/issues/15505).

@stas00 @lhoestq


Cant pass streaming dataset to dataloader after take()

Something isn't working

Hi! Calling `take` on an iterable/streamable dataset makes it not possible to shard the dataset, which in turn disables multi-process loading (attempts to split the workload over the shards), so to go past this limitation, you can either use single-process loading in `DataLoader` (`num_workers=None`) or fetch the first `50_000/batch_size` batches in the loop.

## Describe the bug
I am trying to pass a streaming version of c4 to a dataloader, but it can't be passed after I call `dataset.take(n)`. Some functions such as `shuffle()` can be applied without breaking the dataloader but not take.

## Steps to reproduce the bug
```python
import datasets
import torch

dset = datasets.load_dataset(path='c4', name='en', split="train", streaming=True)
dset = dset.take(50_000)
dset = dset.with_format("torch")

num_workers = 8
batch_size = 512

loader = torch.utils.data.DataLoader(dataset=dset,
                                     batch_size=batch_size,
                                     num_workers=num_workers)
for batch in loader:
    ...
```

## Expected results
No error thrown when iterating over the dataloader

## Actual results
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.9/dist-packages/torch/utils/data/_utils/fetch.py", line 32, in fetch
    data.append(next(self.dataset_iter))
  File "/root/.local/lib/python3.9/site-packages/datasets/formatting/dataset_wrappers/torch_iterable_dataset.py", line 48, in __iter__
    for key, example in self._iter_shard(shard_idx):
  File "/root/.local/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 586, in _iter_shard
    yield from ex_iterable.shard_data_sources(shard_idx)
  File "/root/.local/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 60, in shard_data_sources
    raise NotImplementedError(f"{type(self)} doesn't implement shard_data_sources yet")
NotImplementedError: <class 'datasets.iterable_dataset.TakeExamplesIterable'> doesn't implement shard_data_sources yet

## Environment info
- `datasets` version: 2.3.2
- Platform: Linux-5.4.0-120-generic-x86_64-with-glibc2.31
- Python version: 3.9.13
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Random 400 Client Error when pushing dataset

Something isn't working

## Describe the bug
When pushing a dataset, the client errors randomly with `Bad Request for url:...`.
At the next call, a new parquet file is created for each shard.
The client may fail at any random shard.

## Steps to reproduce the bug
```python
dataset.push_to_hub("ORG/DATASET", private=True, branch="main")
```

## Expected results
Push all the dataset to the Hub with no duplicates.
If it fails, it should retry or fail, but continue from the last failed shard.

## Actual results
```
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
testing.ipynb Cell 29 in <cell line: 1>()
----> [1](testing.ipynb?line=0) dataset.push_to_hub("ORG/DATASET", private=True, branch="main")

File ~/.local/lib/python3.9/site-packages/datasets/arrow_dataset.py:4297, in Dataset.push_to_hub(self, repo_id, split, private, token, branch, max_shard_size, shard_size, embed_external_files)
   4291     warnings.warn(
   4292         "'shard_size' was renamed to 'max_shard_size' in version 2.1.1 and will be removed in 2.4.0.",
   4293         FutureWarning,
   4294     )
   4295     max_shard_size = shard_size
-> 4297 repo_id, split, uploaded_size, dataset_nbytes, repo_files, deleted_size = self._push_parquet_shards_to_hub(
   4298     repo_id=repo_id,
   4299     split=split,
   4300     private=private,
   4301     token=token,
   4302     branch=branch,
   4303     max_shard_size=max_shard_size,
   4304     embed_external_files=embed_external_files,
   4305 )
   4306 organization, dataset_name = repo_id.split("/")
   4307 info_to_dump = self.info.copy()

File ~/.local/lib/python3.9/site-packages/datasets/arrow_dataset.py:4195, in Dataset._push_parquet_shards_to_hub(self, repo_id, split, private, token, branch, max_shard_size, embed_external_files)
   4193         shard.to_parquet(buffer)
   4194         uploaded_size += buffer.tell()
-> 4195         _retry(
   4196             api.upload_file,
   4197             func_kwargs=dict(
   4198                 path_or_fileobj=buffer.getvalue(),
   4199                 path_in_repo=shard_path_in_repo,
   4200                 repo_id=repo_id,
   4201                 token=token,
   4202                 repo_type="dataset",
   4203                 revision=branch,
   4204                 identical_ok=False,
   4205             ),
   4206             exceptions=HTTPError,
   4207             status_codes=[504],
   4208             base_wait_time=2.0,
   4209             max_retries=5,
   4210             max_wait_time=20.0,
   4211         )
   4212     shards_path_in_repo.append(shard_path_in_repo)
   4214 # Cleanup to remove unused files

File ~/.local/lib/python3.9/site-packages/datasets/utils/file_utils.py:284, in _retry(func, func_args, func_kwargs, exceptions, status_codes, max_retries, base_wait_time, max_wait_time)
    282 except exceptions as err:
    283     if retry >= max_retries or (status_codes and err.response.status_code not in status_codes):
--> 284         raise err
    285     else:
    286         sleep_time = min(max_wait_time, base_wait_time * 2**retry)  # Exponential backoff

File ~/.local/lib/python3.9/site-packages/datasets/utils/file_utils.py:281, in _retry(func, func_args, func_kwargs, exceptions, status_codes, max_retries, base_wait_time, max_wait_time)
    279 while True:
    280     try:
--> 281         return func(*func_args, **func_kwargs)
    282     except exceptions as err:
    283         if retry >= max_retries or (status_codes and err.response.status_code not in status_codes):

File ~/.local/lib/python3.9/site-packages/huggingface_hub/hf_api.py:1967, in HfApi.upload_file(self, path_or_fileobj, path_in_repo, repo_id, token, repo_type, revision, identical_ok, commit_message, commit_description, create_pr)
   1957 commit_message = (
   1958     commit_message
   1959     if commit_message is not None
   1960     else f"Upload {path_in_repo} with huggingface_hub"
   1961 )
   1962 operation = CommitOperationAdd(
   1963     path_or_fileobj=path_or_fileobj,
   1964     path_in_repo=path_in_repo,
   1965 )
-> 1967 pr_url = self.create_commit(
   1968     repo_id=repo_id,
   1969     repo_type=repo_type,
   1970     operations=[operation],
   1971     commit_message=commit_message,
   1972     commit_description=commit_description,
   1973     token=token,
   1974     revision=revision,
   1975     create_pr=create_pr,
   1976 )
   1977 if pr_url is not None:
   1978     re_match = re.match(REGEX_DISCUSSION_URL, pr_url)

File ~/.local/lib/python3.9/site-packages/huggingface_hub/hf_api.py:1844, in HfApi.create_commit(self, repo_id, operations, commit_message, commit_description, token, repo_type, revision, create_pr, num_threads)
   1836 commit_url = f"{self.endpoint}/api/{repo_type}s/{repo_id}/commit/{revision}"
   1838 commit_resp = requests.post(
   1839     url=commit_url,
   1840     headers={"Authorization": f"Bearer {token}"},
   1841     json=commit_payload,
   1842     params={"create_pr": 1} if create_pr else None,
   1843 )
-> 1844 _raise_for_status(commit_resp)
   1845 return commit_resp.json().get("pullRequestUrl", None)

File ~/.local/lib/python3.9/site-packages/huggingface_hub/utils/_errors.py:84, in _raise_for_status(request)
     76 if request.status_code == 401:
     77     # The repo was not found and the user is not Authenticated
     78     raise RepositoryNotFoundError(
     79         f"401 Client Error: Repository Not Found for url: {request.url}. If the"
     80         " repo is private, make sure you are authenticated. (Request ID:"
     81         f" {request_id})"
     82     )
---> 84 _raise_with_request_id(request)

File ~/.local/lib/python3.9/site-packages/huggingface_hub/utils/_errors.py:95, in _raise_with_request_id(request)
     92 if request_id is not None and len(e.args) > 0 and isinstance(e.args[0], str):
     93     e.args = (e.args[0] + f" (Request ID: {request_id})",) + e.args[1:]
---> 95 raise e

File ~/.local/lib/python3.9/site-packages/huggingface_hub/utils/_errors.py:90, in _raise_with_request_id(request)
     88 request_id = request.headers.get("X-Request-Id")
     89 try:
---> 90     request.raise_for_status()
     91 except Exception as e:
     92     if request_id is not None and len(e.args) > 0 and isinstance(e.args[0], str):

File ~/.local/lib/python3.9/site-packages/requests/models.py:1021, in Response.raise_for_status(self)
   1016     http_error_msg = (
   1017         f"{self.status_code} Server Error: {reason} for url: {self.url}"
   1018     )
   1020 if http_error_msg:
-> 1021     raise HTTPError(http_error_msg, response=self)

HTTPError: 400 Client Error: Bad Request for url: https://huggingface.co/api/datasets/ORG/DATASET/commit/main (Request ID: a_F0IQAHJdxGKVRYyu1cF)
```

## Environment info
- `datasets` version: 2.3.2
- Platform: Linux-5.13.0-1025-aws-x86_64-with-glibc2.31
- Python version: 3.9.4
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Dataset.map gets stuck on _cast_to_python_objects

Something isn't working

good first issue

Good for newcomers

Are you able to reproduce this? My example is small enough that it should be easy to try.

Hi! Thanks for reporting and providing a reproducible example. Indeed, by default, `datasets` performs an expensive cast on the values returned by `map` to convert them to one of the types supported by PyArrow (the underlying storage format used by `datasets`). This cast is not needed on NumPy arrays as PyArrow supports them natively, so one way to make this transform faster is to add `return_tensors="np"` to the tokenizer call.

I think we should mention this in the docs (cc @stevhliu)

I tested this tokenize function and indeed noticed a casting. However it seems to only concerns the `offset_mapping` field, which contains a list of tuples, that is converted to a list of lists. Since `pyarrow` also supports tuples, we actually don't need to convert the tuples to lists.

I think this can be changed here:

https://github.com/huggingface/datasets/blob/ede72d3f9796339701ec59899c7c31d2427046fb/src/datasets/features/features.py#L382-L383

```diff
- if isinstance(obj, list):
+ if isinstance(obj, (list, tuple)):
```

and here:

https://github.com/huggingface/datasets/blob/ede72d3f9796339701ec59899c7c31d2427046fb/src/datasets/features/features.py#L386-L387

```diff
- return obj if isinstance(obj, list) else [], isinstance(obj, tuple)
+ return obj, False
```

@srobertjames can you try applying these changes and let us know if it helps ? If so, feel free to open a Pull Request to contribute this improvement if you want :)

Wow, adding `return_tensors="np"` sped up my example by a **factor 17x** of and completely eliminated the casting! I'd recommend not only to document it, but to make that the default.

The code at https://github.com/huggingface/notebooks/blob/main/examples/question_answering.ipynb does not specify `return_tensors="np"` but yet avoids the casting penalty.  How does it do that?  (The ntbk seems to do `return_overflowing_tokens=True, return_offsets_mapping=True,`).

Also, surprisingly enough, using `return_tensors="pt"` (which is my eventual application) yields this error:
```
TypeError: Provided `function` which is applied to all elements of table returns a `dict` of types
[<class 'torch.Tensor'>, <class 'torch.Tensor'>, <class 'torch.Tensor'>, <class 'torch.Tensor'>].
When using `batched=True`, make sure provided `function` returns a `dict` of types like
`(<class 'list'>, <class 'numpy.ndarray'>)`.
```

Setting the output to `"np"` makes the whole pipeline fast because it moves the data buffers from rust to python to arrow using zero-copy, and also because it does eliminate the casting completely ;)

Have you had a chance to try eliminating the tuple casting using the trick above ?

@lhoestq I just benchmarked the two edits to `features.py` above, and they appear to solve the problem, bringing my original example to within 20% the speed of the output `"np"` example.  Nice!

For a pull request, do you suggest simply following https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md ?

Cool ! Sure feel free to follow these instructions to open a PR :) thanks !

## Describe the bug
`Dataset.map`, when fed a Huggingface Tokenizer as its map func, can sometimes spend huge amounts of time doing casts.  A minimal example follows.

Not all usages suffer from this.  For example, I profiled the preprocessor at https://github.com/huggingface/notebooks/blob/main/examples/question_answering.ipynb , and it did _not_ have this problem.  However, I'm at a loss to figure out how it avoids it, as the example below is simple and minimal and still has this problem.

This casting, where it occurs, causes the `Dataset.map` to run approximately 7x slower than it runs for code which does not cause this casting.

This may be related to https://github.com/huggingface/datasets/issues/1046 .  However, the tokenizer is _not_ set to return Tensors.

## Steps to reproduce the bug
A minimal, self-contained example to reproduce is below:
```python
import transformers
from transformers import AutoTokenizer
from datasets import load_dataset
import torch
import cProfile

pretrained = 'distilbert-base-uncased'
tokenizer = AutoTokenizer.from_pretrained(pretrained)

squad = load_dataset('squad')
squad_train = squad['train']
squad_tiny = squad_train.select(range(5000))

assert isinstance(tokenizer, transformers.PreTrainedTokenizerFast)

def tokenize(ds):
        tokens = tokenizer(text=ds['question'],
                                text_pair=ds['context'],
                                add_special_tokens=True,
                                padding='max_length',
                                truncation='only_second',
                                max_length=160,
                                stride=32,
                                return_overflowing_tokens=True,
                                return_offsets_mapping=True,
                                )
        return tokens

cmd = 'squad_tiny.map(tokenize, batched=True, remove_columns=squad_tiny.column_names)'
cProfile.run(cmd, sort='tottime')
```

## Actual results
The code works, but takes 10-25 sec per batch (about 7x slower than non-casting code), with the following profile.  Note that `_cast_to_python_objects` is the culprit.
```
    63524075 function calls (58206482 primitive calls) in 121.836 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
5274034/40   68.751    0.000  111.060    2.776 features.py:262(_cast_to_python_objects)
 42223832   24.077    0.000   33.310    0.000 {built-in method builtins.isinstance}
 16338/20    5.121    0.000  111.053    5.553 features.py:361(<listcomp>)
  5274135    4.747    0.000    4.749    0.000 {built-in method _abc._abc_instancecheck}
    80/40    4.731    0.059  116.292    2.907 {pyarrow.lib.array}
  5274135    4.485    0.000    9.234    0.000 abc.py:96(__instancecheck__)
2661564/2645196    2.959    0.000    4.298    0.000 features.py:1081(_check_non_null_non_empty_recursive)
        5    2.786    0.557    2.786    0.557 {method 'encode_batch' of 'tokenizers.Tokenizer' objects}
  2668052    0.930    0.000    0.930    0.000 {built-in method builtins.len}
     5000    0.930    0.000    0.938    0.000 tokenization_utils_fast.py:187(_convert_encoding)
        5    0.750    0.150    0.808    0.162 {method 'to_pydict' of 'pyarrow.lib.Table' objects}
        1    0.444    0.444  121.749  121.749 arrow_dataset.py:2501(_map_single)
       40    0.375    0.009  116.291    2.907 arrow_writer.py:151(__arrow_array__)
       10    0.066    0.007    0.066    0.007 {method 'write_batch' of 'pyarrow.lib._CRecordBatchWriter' objects}
        1    0.060    0.060  121.835  121.835 fingerprint.py:409(wrapper)
11387/5715    0.049    0.000    0.175    0.000 {built-in method builtins.getattr}
       36    0.049    0.001    0.049    0.001 {pyarrow._compute.call_function}
    15000    0.040    0.000    0.040    0.000 _collections_abc.py:719(__iter__)
        3    0.023    0.008    0.023    0.008 {built-in method _imp.create_dynamic}
       77    0.020    0.000    0.020    0.000 {built-in method builtins.dir}
       37    0.019    0.001    0.019    0.001 socket.py:543(send)
       15    0.017    0.001    0.017    0.001 tokenization_utils_fast.py:460(<listcomp>)
  432/421    0.015    0.000    0.024    0.000 traitlets.py:1388(_notify_observers)
     5000    0.015    0.000    0.018    0.000 _collections_abc.py:672(keys)
       51    0.014    0.000    0.042    0.001 traitlets.py:276(getmembers)
        5    0.014    0.003    3.775    0.755 tokenization_utils_fast.py:392(_batch_encode_plus)
      3/1    0.014    0.005    0.035    0.035 {built-in method _imp.exec_dynamic}
        5    0.012    0.002    0.950    0.190 tokenization_utils_fast.py:438(<listcomp>)
    31626    0.012    0.000    0.012    0.000 {method 'append' of 'list' objects}
1532/1001    0.011    0.000    0.189    0.000 traitlets.py:643(get)
        5    0.009    0.002    3.796    0.759 arrow_dataset.py:2631(apply_function_on_filtered_inputs)
       51    0.009    0.000    0.062    0.001 traitlets.py:1766(traits)
        5    0.008    0.002    3.784    0.757 tokenization_utils_base.py:2632(batch_encode_plus)
      368    0.007    0.000    0.044    0.000 traitlets.py:1715(_get_trait_default_generator)
       26    0.007    0.000    0.022    0.001 traitlets.py:1186(setup_instance)
       51    0.006    0.000    0.010    0.000 traitlets.py:1781(<listcomp>)
    80/32    0.006    0.000    0.052    0.002 table.py:1758(cast_array_to_feature)
      684    0.006    0.000    0.007    0.000 {method 'items' of 'dict' objects}
4344/1794    0.006    0.000    0.192    0.000 traitlets.py:675(__get__)
...
```
## Environment info
I observed this on both Google colab and my local workstation:

### Google colab
- `datasets` version: 2.3.2
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5

### Local
- `datasets` version: 2.3.2
- Platform: Windows-7-6.1.7601-SP1
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Unable to use dataset with PyTorch dataloader

bug

Something isn't working

Hi! `para_crawl` has a single column of type `Translation`, which stores translation dictionaries. These dictionaries can be stored in a NumPy array but not in a PyTorch tensor since PyTorch only supports numeric types. In `datasets`, the conversion to `torch` works as follows:
1. convert PyArrow table to NumPy arrays
2. convert NumPy arrays to Torch tensors.

The 2nd step is problematic for your case as `datasets` attempts to convert the array of dictionaries to a PyTorch tensor. One way to fix this is to use the [preprocessing logic](https://github.com/huggingface/transformers/blob/8581a798c0a48fca07b29ce2ca2ef55adcae8c7e/examples/pytorch/translation/run_translation.py#L440-L458) from the Transformers translation script. And on our side, I think we can replace a NumPy array of dicts with a dict of NumPy array if the feature type is `Translation`/`TranslationVariableLanguages` (one array for each language) to get the official PyTorch error message for strings in such case.

## Describe the bug

When using `.with_format("torch")`, an arrow table is returned and I am unable to use it by passing it to a PyTorch DataLoader: please see the code below.

## Steps to reproduce the bug
```python
from datasets import load_dataset
from torch.utils.data import DataLoader

ds = load_dataset(
    "para_crawl",
    name="enfr",
    cache_dir="/tmp/test/",
    split="train",
    keep_in_memory=True,
)

dataloader = DataLoader(ds.with_format("torch"), num_workers=32)
print(next(iter(dataloader)))
```

Is there something I am doing wrong? The documentation does not say much about the behavior of `.with_format()` so I feel like I am a bit stuck here :-/

Thanks in advance for your help!

## Expected results

The code should run with no error

## Actual results

```
AttributeError: 'str' object has no attribute 'dtype'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Linux-4.18.0-348.el8.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.4
- PyArrow version: 8.0.0
- Pandas version: 1.4.3