Push to hub can push splits that do not respect the regex

Something isn't working

## Describe the bug

The `push_to_hub` method can push splits that do not respect the regex check that is used for downloads. Therefore, splits may be pushed but never re-used, which can be painful if the split was done after runtime preprocessing.

## Steps to reproduce the bug
```python
>>> from datasets import Dataset, DatasetDict, load_dataset

>>> d = Dataset.from_dict({'x': [1,2,3], 'y': [1,2,3]})
>>> di = DatasetDict()
>>> di['identifier-with-column'] = d

>>> di.push_to_hub('open-source-metrics/test')
Pushing split identifier-with-column to the Hub.
Pushing dataset shards to the dataset hub: 100%|██████████| 1/1 [00:04<00:00,  4.40s/it]
```

Loading it afterwards:
```python
>>> load_dataset('open-source-metrics/test')
Downloading: 100%|██████████| 610/610 [00:00<00:00, 432kB/s]
Using custom data configuration open-source-metrics--test-28b63ec7cde80488
Downloading and preparing dataset None/None (download: 950 bytes, generated: 48 bytes, post-processed: Unknown size, total: 998 bytes) to /home/lysandre/.cache/huggingface/datasets/open-source-metrics___parquet/open-source-metrics--test-28b63ec7cde80488/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec...
Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]
Downloading data: 100%|██████████| 950/950 [00:00<00:00, 1.01MB/s]
Downloading data files: 100%|██████████| 1/1 [00:01<00:00,  1.48s/it]
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 2291.97it/s]
Traceback (most recent call last):
  File "/home/lysandre/.pyenv/versions/3.10.6/lib/python3.10/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "/home/lysandre/Workspaces/python/Metrics/GitHub-Metrics/.env/lib/python3.10/site-packages/datasets/load.py", line 1746, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/lysandre/Workspaces/python/Metrics/GitHub-Metrics/.env/lib/python3.10/site-packages/datasets/builder.py", line 704, in download_and_prepare
    self._download_and_prepare(
  File "/home/lysandre/Workspaces/python/Metrics/GitHub-Metrics/.env/lib/python3.10/site-packages/datasets/builder.py", line 771, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/lysandre/Workspaces/python/Metrics/GitHub-Metrics/.env/lib/python3.10/site-packages/datasets/packaged_modules/parquet/parquet.py", line 48, in _split_generators
    splits.append(datasets.SplitGenerator(name=split_name, gen_kwargs={"files": files}))
  File "<string>", line 5, in __init__
  File "/home/lysandre/Workspaces/python/Metrics/GitHub-Metrics/.env/lib/python3.10/site-packages/datasets/splits.py", line 599, in __post_init__
    NamedSplit(self.name)  # check that it's a valid split name
  File "/home/lysandre/Workspaces/python/Metrics/GitHub-Metrics/.env/lib/python3.10/site-packages/datasets/splits.py", line 346, in __init__
    raise ValueError(f"Split name should match '{_split_re}' but got '{split_name}'.")
ValueError: Split name should match '^\w+(\.\w+)*$' but got 'identifier-with-column'.
```

## Expected results

I would expect `push_to_hub` to stop me in my tracks if trying to upload a split that will not be working afterwards.

## Actual results

See above

## Environment info

- `datasets` version: 2.4.0
- Platform: Linux-5.15.64-1-lts-x86_64-with-glibc2.36
- Python version: 3.10.6
- PyArrow version: 9.0.0
- Pandas version: 1.4.4


larger dataset, larger GPU memory in the training phase? Is that correct?

bug

Something isn't working

does the trainer save it in GPU? sooo curious... how to fix it

It's my bad. didn't limit the input length

    from datasets import set_caching_enabled
    set_caching_enabled(False)
    for ds_name in ["squad","newsqa","nqopen","narrativeqa"]:
            train_ds = load_from_disk("../../../dall/downstream/processedproqa/{}-train.hf".format(ds_name))

        break
    train_ds = concatenate_datasets([train_ds,train_ds,train_ds,train_ds]) #operation 1

   trainer = QuestionAnsweringTrainer( #huggingface trainer
        model=model,
        args=training_args,
        train_dataset=train_ds,
        eval_dataset= None,
        eval_examples=None,
        answer_column_name=answer_column,
        dataset_name="squad",
        tokenizer=tokenizer,
        data_collator=data_collator,
        compute_metrics=compute_metrics if training_args.predict_with_generate else None,
    )

with operation 1, the GPU memory increases from 16G to 23G


Add splits to MBPP dataset

_The documentation is not available anymore as the PR was closed or merged._

Hi @cwarny ! Thanks for adding the correct splits :)

You can fix the CI error by running `make style` - this should reformat the dataset script

This PR addresses https://github.com/huggingface/datasets/issues/4795


Trec Dataset has incorrect labels

Something isn't working

Thanks for reporting, @wmpauli.

Indeed we recently fixed this issue:
- #4801

The fix will be accessible after our next library release. In the meantime, you can have it by passing `revision="main"` to `load_dataset`.

## Describe the bug
Both coarse and fine labels seem to be out of line.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = "trec"
raw_datasets = load_dataset(dataset)
df = pd.DataFrame(raw_datasets["test"])
df.head()
```

## Expected results
text (string) | coarse_label (class label) | fine_label (class label)
-- | -- | --
How far is it from Denver to Aspen ? | 5  	(NUM) | 40  	(NUM:dist)
What county is Modesto , California in ? | 4  	(LOC) | 32  	(LOC:city)
Who was Galileo ? | 3  	(HUM) | 31  	(HUM:desc)
What is an atom ? | 2  	(DESC) | 24  	(DESC:def)
When did Hawaii become a state ? | 5  	(NUM) | 39  	(NUM:date)

## Actual results
  index | label-coarse  |label-fine                  |                    text
-- |-- | -- | --
0    |         4     |     40     | How far is it from Denver to Aspen ?
1      |       5        |  21  | What county is Modesto , California in ?
2        |     3        |  12        |                 Who was Galileo ?
3         |    0       |    7         |                What is an atom ?
4         |    4       |    8       |   When did Hawaii become a state ?

## Environment info

- `datasets` version: 2.4.0
- Platform: Linux-5.4.0-1086-azure-x86_64-with-glibc2.27
- Python version: 3.9.13
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Add Papers with Code ID to scifact dataset

This PR:
- adds Papers with Code ID
- forces sync between GitHub and Hub, which previously failed due to Hub validation error of the license tag: https://github.com/huggingface/datasets/runs/8200223631?check_suite_focus=true


Fix multilinguality tag and missing sections in xquad_r dataset card

This PR fixes issue reported on the Hub:
- Label as multilingual: https://huggingface.co/datasets/xquad_r/discussions/1


Fix NonMatchingChecksumError in adv_glue dataset

Fix issue reported on the Hub: https://huggingface.co/datasets/adv_glue/discussions/1


Remove main branch rename notice

We added a notice in README.md to show that we renamed the master branch to main, but we can remove it now (it's been 2 months)

I also unpinned the github issue about the branch renaming


Remove deprecated identical_ok

`huggingface-hub` says that the `identical_ok` argument of `HfApi.upload_file` is now deprecated, and will be removed soon. It even has no effect at the moment when it's passed:

```python
Args:
...
    identical_ok (`bool`, *optional*, defaults to `True`):
        Deprecated: will be removed in 0.11.0.
        Changing this value has no effect.
...
```

There was only one occurence of `identical_ok=False` but it's maybe not worth adding a check ti verify if the files were the same.

cc @mariosasko


vivos (Vietnamese speech corpus) dataset not accessible

dataset bug

A bug in a dataset script provided in the library

If you need an example of a small audio datasets, I just created few hours ago a speech dataset with only 300MB of compressed audio files https://huggingface.co/datasets/indonesian-nlp/librivox-indonesia. It works also with streaming (@albertvillanova helped me adding this functionality) :-)

@cahya-wirawan omg this is awesome!! thank you!

We have contacted the authors to ask them.

## Describe the bug
VIVOS data is not accessible anymore, neither of these links work (at least from France):
* https://ailab.hcmus.edu.vn/assets/vivos.tar.gz (data)
* https://ailab.hcmus.edu.vn/vivos (dataset page)

Therefore `load_dataset` doesn't work.

## Steps to reproduce the bug
```python
ds = load_dataset("vivos")
```

## Expected results
dataset loaded

## Actual results
```
ConnectionError: Couldn't reach https://ailab.hcmus.edu.vn/assets/vivos.tar.gz (ConnectionError(MaxRetryError("HTTPSConnectionPool(host='ailab.hcmus.edu.vn', port=443): Max retries exceeded with url: /assets/vivos.tar.gz (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9d8a27d190>: Failed to establish a new connection: [Errno -5] No address associated with hostname'))")))
```

Will try to contact the authors, as we wanted to use Vivos as an example in documentation on how to create scripts for audio datasets (https://github.com/huggingface/datasets/pull/4872), because it's small and straightforward and uses tar archives.


Dataset Viewer issue for ubuntu_dialogs_corpus

dataset-viewer

Related to the dataset viewer on huggingface.co

The dataset maintainers (https://huggingface.co/datasets/ubuntu_dialogs_corpus) decided to forbid the dataset from being downloaded automatically (https://huggingface.co/docs/datasets/v2.4.0/en/loading#manual-download), and the dataset viewer respects this.
We will try to improve the error display though. Thanks for reporting.

### Link

_No response_

### Description

_No response_

### Owner

_No response_


Dataset Viewer issue for indonesian-nlp/librivox-indonesia

The error is not related to the dataset viewer. I'm having a look...

Thanks @albertvillanova for checking the issue. Actually, I can use the dataset like following:
```
>>> from datasets import load_dataset
>>> ds=load_dataset("indonesian-nlp/librivox-indonesia")
No config specified, defaulting to: librivox-indonesia/all
Reusing dataset librivox-indonesia (/root/.cache/huggingface/datasets/indonesian-nlp___librivox-indonesia/all/1.0.0/9a934a42bfb53dc103003d191618443b8a786bea2bd7bb0bc2d9454b8494521e)
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 500.87it/s]
>>> ds
DatasetDict({
    train: Dataset({
        features: ['path', 'language', 'reader', 'sentence', 'audio'],
        num_rows: 7815
    })
})
>>> ds["train"][0]
{'path': '/root/.cache/huggingface/datasets/downloads/extracted/c8ead52370fa28feb64643ea9d05cd7d820192dc8a1700d665ec45ec7624f5a3/librivox-indonesia/sundanese/universal-declaration-of-human-rights/human_rights_un_sun_brc_0000.mp3', 'language': 'sun', 'reader': '3174', 'sentence': 'pernyataan umum ngeunaan hak hak asasi manusa sakabeh manusa', 'audio': {'path': '/root/.cache/huggingface/datasets/downloads/extracted/c8ead52370fa28feb64643ea9d05cd7d820192dc8a1700d665ec45ec7624f5a3/librivox-indonesia/sundanese/universal-declaration-of-human-rights/human_rights_un_sun_brc_0000.mp3', 'array': array([ 0.        ,  0.        ,  0.        , ..., -0.02419001,
       -0.01957154, -0.01502833], dtype=float32), 'sampling_rate': 44100}}

```
It would be just nice if I also can see it using dataset viewer.

Yes, the issue arises when streaming (that is used by the viewer): your script does not support streaming and to support it in this case there are some subtleties that we are explaining better in our docs in a work-in progress pull request:
- #4872

Just note that when streaming, `local_extracted_archive` is None, and this code line generates the error:
```python
filepath = local_extracted_archive + "/librivox-indonesia/audio_transcription.csv"
```

For a proper implementation, you could have a look at: https://huggingface.co/datasets/common_voice/blob/main/common_voice.py

You can test your script locally by passing `streaming=True` to `load_dataset`:
```python
ds = load_dataset("indonesian-nlp/librivox-indonesia", split="train", streaming=True); item = next(iter(ds)); item
```

Great, I will have a look and update the script. Thanks.

Hi @albertvillanova , I just add the streaming functionality and it works in the first try :-) Thanks a lot!

### Link

https://huggingface.co/datasets/indonesian-nlp/librivox-indonesia

### Description

I created a new speech dataset https://huggingface.co/datasets/indonesian-nlp/librivox-indonesia, but the dataset preview doesn't work with following error message:
```
Server error
Status code:   400
Exception:     TypeError
Message:       unsupported operand type(s) for +: 'NoneType' and 'str'
```
Please help, I am not sure what the problem here is. Thanks a lot.

### Owner

Yes


Dataset/DatasetDict.filter() cannot have `batched=True` due to `mask` (numpy array?) being non-iterable.

bug

Something isn't working

Hi ! When `batched=True`, you filter function must take a batch as input, and return a list of booleans.

In your case, something like
```python
from datasets import load_dataset

ds_mc4_ja = load_dataset("mc4", "ja")  # This will take 6+ hours... perhaps test it with a toy dataset instead?
ds_mc4_ja_2020 = ds_mc4_ja.filter(
  lambda batch: [timestamp[:4] == "2020" for timestamp in batch["timestamp"]],
  batched=True,
)
```

Let me know if it helps !

> Hi ! When `batched=True`, you filter function must take a batch as input, and return a list of booleans.
> [...]
> Let me know if it helps !

Hi @lhoestq,

Ah, my bad, I totally forgot that part...
Sorry for the trouble and thank you for the kind help!

## Describe the bug
`Dataset/DatasetDict.filter()` cannot have `batched=True` due to `mask` (numpy array?) being non-iterable.

## Steps to reproduce the bug
(In a python 3.7.12 env, I've tried 2.4.0 and 2.3.2 with both `pyarraw==9.0.0` and `pyarrow==8.0.0`.)

```python
from datasets import load_dataset

ds_mc4_ja = load_dataset("mc4", "ja")  # This will take 6+ hours... perhaps test it with a toy dataset instead?
ds_mc4_ja_2020 = ds_mc4_ja.filter(
  lambda example: example["timestamp"][:4] == "2020",
  batched=True,
)
```

## Expected results
No error

## Actual results
```python
---------------------------------------------------------------------------
RemoteTraceback                           Traceback (most recent call last)
RemoteTraceback:
"""
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/multiprocess/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 557, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 524, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/datasets/fingerprint.py", line 480, in wrapper
    out = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 2779, in _map_single
    offset=offset,
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 2655, in apply_function_on_filtered_inputs
    processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 2347, in decorated
    result = f(decorated_item, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 4946, in get_indices_from_mask_function
    indices_array = [i for i, to_keep in zip(indices, mask) if to_keep]
TypeError: zip argument #2 must support iteration
"""

The above exception was the direct cause of the following exception:

TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_51348/2345782281.py in <module>
      7     batched=True,
      8     # batch_size=10_000,
----> 9     num_proc=111,
     10 )
     11 # ds_mc4_ja_clean_2020 = ds_mc4_ja.filter(

/opt/conda/lib/python3.7/site-packages/datasets/dataset_dict.py in filter(self, function, with_indices, input_columns, batched, batch_size, keep_in_memory, load_from_cache_file, cache_file_names, writer_batch_size, fn_kwargs, num_proc, desc)
    878                     desc=desc,
    879                 )
--> 880                 for k, dataset in self.items()
    881             }
    882         )

/opt/conda/lib/python3.7/site-packages/datasets/dataset_dict.py in <dictcomp>(.0)
    878                     desc=desc,
    879                 )
--> 880                 for k, dataset in self.items()
    881             }
    882         )

/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    522         }
    523         # apply actual function
--> 524         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    525         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    526         # re-apply format to the output

/opt/conda/lib/python3.7/site-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    478             # Call actual function
    479
--> 480             out = func(self, *args, **kwargs)
    481
    482             # Update fingerprint of in-place transforms + update in-place history of transforms

/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py in filter(self, function, with_indices, input_columns, batched, batch_size, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   2920             new_fingerprint=new_fingerprint,
   2921             input_columns=input_columns,
-> 2922             desc=desc,
   2923         )
   2924         new_dataset = copy.deepcopy(self)

/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py in map(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   2498
   2499                     for index, async_result in results.items():
-> 2500                         transformed_shards[index] = async_result.get()
   2501
   2502             assert (

/opt/conda/lib/python3.7/site-packages/multiprocess/pool.py in get(self, timeout)
    655             return self._value
    656         else:
--> 657             raise self._value
    658
    659     def _set(self, i, obj):

TypeError: zip argument #2 must support iteration
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: Linux-4.19.0-21-cloud-amd64-x86_64-with-debian-10.12
- Python version: 3.7.12
- PyArrow version: 9.0.0
- Pandas version: 1.3.5

(I've tried 2.4.0 and 2.3.2 with both `pyarraw==9.0.0` and `pyarrow==8.0.0`.)


Dataset Viewer issue for bigscience-biomedical/biosses

Possibly not related to the dataset viewer in itself. cc @huggingface/datasets.

In particular, I think that the import of bigbiohub is not working here: https://huggingface.co/datasets/bigscience-biomedical/biosses/blob/main/biosses.py#L29 (requires a relative path?)

```python
>>> from datasets import get_dataset_config_names
>>> get_dataset_config_names('bigscience-biomedical/biosses')
Downloading builder script: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8.00k/8.00k [00:00<00:00, 7.47MB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 289, in get_dataset_config_names
    dataset_module = dataset_module_factory(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1247, in dataset_module_factory
    raise e1 from None
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1220, in dataset_module_factory
    return HubDatasetModuleFactoryWithScript(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 931, in get_module
    local_imports = _download_additional_modules(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 215, in _download_additional_modules
    raise ImportError(
ImportError: To be able to use bigscience-biomedical/biosses, you need to install the following dependency: bigbiohub.
Please install it using 'pip install bigbiohub' for instance'
```

Opened a PR here to (hopefully) fix the dataset script: https://huggingface.co/datasets/bigscience-biomedical/biosses/discussions/1/files

thanks for taking a look @severo . agree this isn't related to dataset viewer (sorry just clicked on the auto issue creator). also thanks @lhoestq , I see the format to use for relative imports. was a bit confused b/c it seems to be working here

https://huggingface.co/datasets/bigscience-biomedical/scitail/blob/main/scitail.py#L31

I'll try this PR a see what happens.

closing as I think the issue is relative imports and attempting to read json files directly in the repo (thanks again @lhoestq )

### Link

https://huggingface.co/datasets/bigscience-biomedical/biosses

### Description

I've just been working on adding the dataset loader script to this dataset and working with the relative imports. I'm not sure how to interpret the error below (show where the dataset preview used to be) .
```
Status code:   400
Exception:     ModuleNotFoundError
Message:       No module named 'datasets_modules.datasets.bigscience-biomedical--biosses.ddbd5893bf6c2f4db06f407665eaeac619520ba41f69d94ead28f7cc5b674056.bigbiohub'
```

### Owner

Yes


Fix missing tags in dataset cards

_The documentation is not available anymore as the PR was closed or merged._

Fix missing tags in dataset cards:
- coqa
- hyperpartisan_news_detection
- opinosis
- scientific_papers
- scifact
- search_qa
- wiki_qa
- wiki_split
- wikisql

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896
- #4908
- #4921


Add cc-by-nc-2.0 to list of licenses

this list needs to be kept in sync with the ones in moon-landing and hub-docs :)

@julien-c don't you think it might be better to a have a single file (source of truth) in one of the repos and then use it in every other repo, instead of having 3 copies of the same file that must be kept in sync?

Also note that the licenses we are adding were all already present in our previous `licenses.json` file: are we regenerating it, step by step? Why don't we use a file with ALL the licenses we previously had in the list?

Licenses added:
- #4887
- #4930

Previous `licenses.json` file:
- https://github.com/huggingface/datasets/blob/b7612754928e0fd43b9e3c3becb906ec280ff5d4/src/datasets/utils/resources/licenses.json
- removed in this commit: https://github.com/huggingface/datasets/pull/4613/commits/9f7725412dac1089b3e057f9e3fcf39cc222bc26

Let me know what you think and I can take care of this.

> Let me know what you think and I can take care of this.

What I think is that we shouldn't add licenses that are just used in a couple of datasets, and just use `license_details` for this.

> don't you think it might be better to a have a single file (source of truth) in one of the repos and then use it in every other repo, instead of having 3 copies of the same file that must be kept in sync?

Yes, in my opinion we can just delete this file from `datasets`, the validation is happening hub-side anyways now?

Feel free to delete the license list in `datasets` @albertvillanova ;)

Also FYI in #4926 I also removed all the validation steps anyway (language, license, types etc.)

This PR adds the `cc-by-nc-2.0` to the list of licenses because it is required by `scifact` dataset: https://github.com/allenai/scifact/blob/master/LICENSE.md


Fixes a typo in loading documentation

As show in the [documentation page](https://huggingface.co/docs/datasets/loading) here the `"tr"in` should be `"train`.

![image](https://user-images.githubusercontent.com/7144772/188390445-e1f04d54-e3e3-4762-8686-63ecbe4087e5.png)


Add ability to read-write to SQL databases.

Ah CI runs with `pandas=1.3.5` which doesn't return the number of row inserted.

@lhoestq I'm getting error in integration tests, not sure if it's related to my PR. Any help would be appreciated :)

```
if not self._is_valid_token(token):
>               raise ValueError("Invalid token passed!")
E               ValueError: Invalid token passed!
```

I just relaunched the tests, it should be fixed now

Thanks a lot for working on this!

I have some concerns with the current design:
* Besides SQLite, the loader should also work with the other engines supported by SQLAlchemy. (A better name for it in the current state would be `sqlite` :))
* It should support arbitrary queries/table names - only the latter currently works.
* Exposing this loader as a packaged builder (`load_dataset("sql", ...)`) is not a good idea for the following reasons:
    * Considering the scenario where a table with the same name is present in multiple files is very unlikely, the data files resolution is not needed here. And if we remove that, what the name of the default split should be? "train"?
    * `load_dataset("sql", ...)` also implies that streaming should work, but that's not the case.  And I don't think we can change that, considering how hard it is to make SQLite files streamable.

All this makes me think we shouldn't expose this builder as a packaged module and, instead, limit the API to `Dataset.from_sql`/`Dataset.to_sql` (with the signatures matching the ones in pandas as much as possible; regarding this, note that SQLAlchemy connections are not hashable/picklable, which is required for caching, but I think it's OK only to allow URI strings as connections to bypass that (Dask has the same limitation).

WDYT?

Hi @mariosasko thank you for your review.

I agree that `load_dataset('sql',...)` is a bit weird and I would be happy to remove it. To be honest, I only added it when I saw that it was the preferred way in `loading.mdx`.

I agree that the `SELECT` should be a parameters as well. I'll add it.

So far, only `Dataset.to_sql` explicitly supports any SQLAlchemy Connexion, I'm pretty sure that `Dataset.from_sql` would work with a Connexion as well, but it would break the typing from the parent class which is `path_or_paths: NestedDataStructureLike[PathLike]`. I would prefer not to break this API Contract.

I will have time to work on this over the weekend. Please let me know what you think if I do the following:
* Remove `load_dataset('sql', ...)` and edit the documentation to use `to_sql, from_sql`.
* Tentatively make `Dataset.from_sql` typing work with SQLAlchemy Connexion.
* Add support for custom queries (Default would be `SELECT * FROM {table_name}`).

Cheers!

Perhaps after we merge https://github.com/huggingface/datasets/pull/4957 (**Done!**), you can subclass `AbstractDatasetInputStream` instead of `AbstractDatasetReader` to not break the contract with the connection object. Also, let's avoid having the default value for the query/table (you can set it to `None` in the builder and raise an error in the builder config's `__post_init__` if it's not provided). Other than that, sounds good!

@Dref360 I've made final changes/refinements to align the SQL API with Pandas/Dask. Let me know what you think.

Thank you so much! I was missing a lot of things sorry about that.
LGTM

I think we can merge if the tests pass.

One last thing I would like to get your opinion on - currently, if SQLAlchemy is not installed, the missing dependency error will be thrown inside `pandas.read_sql`. Do you think we should be the ones throwing this error, e.g. after the imports in `packaged_modules/sql/sql.py` if `SQLALCHEMY_AVAILABLE` is `False` (note that this would mean making `sqlalchemy` a required dependency for the docs to be able to add `SqlConfig` to the package reference)?

> One last thing I would like to get your opinion on - currently, if SQLAlchemy is not installed, the missing dependency error will be thrown inside pandas.read_sql

Is sqlalchemy always required for pd.read_sql ? If so, I think we can raise the error on our side.
But sqlalchemy should still be an optional dependency for `datasets` IMO

@lhoestq
> Is sqlalchemy always required for pd.read_sql ? If so, I think we can raise the error on our side.

In our case, it's always required as we only support database URIs.

> But sqlalchemy should still be an optional dependency for datasets IMO

Yes, it will remain optional for datasets but will be required for building the docs (as is`s3fs`, for instance).

Fixes #3094

Add ability to read/write to SQLite files and also read from any SQL database supported by SQLAlchemy.

I didn't add SQLAlchemy as a dependence as it is fairly big and it remains optional.

I also recorded a Loom to showcase the feature.

https://www.loom.com/share/f0e602c2de8a46f58bca4b43333d541f


fix BLEU metric card

I've fixed some typos in BLEU metric card.


Dataset infos in yaml

dataset contribution

Contribution to a dataset script

Alright this is ready for review :)
I mostly would like your opinion on the YAML structure and what we can do in the docs (IMO we can add the docs about those fields in the Hub docs). Other than that let me know if the changes in info.py and features.py look good to you

LGTM and looking forward to having this merged!! ❤

We plan to do a release today, we'll merge this after the release :)

EDIT: actually tomorrow

Created https://github.com/huggingface/datasets/pull/5018 where I added the YAML `dataset_info` of every single dataset in this repo

see other dataset cards: [imagenet-1k](https://github.com/huggingface/datasets/blob/040102f100964a33fd334e2695f1c493fa6b92db/datasets/imagenet-1k/README.md), [glue](https://github.com/huggingface/datasets/blob/040102f100964a33fd334e2695f1c493fa6b92db/datasets/glue/README.md), [flores](https://github.com/huggingface/datasets/blob/040102f100964a33fd334e2695f1c493fa6b92db/datasets/flores/README.md), [gem](https://github.com/huggingface/datasets/blob/040102f100964a33fd334e2695f1c493fa6b92db/datasets/gem/README.md)

Took your comments into account and updated `push_to_hub` to push the dataset_info to the README.md instead of json :) Let me know if it sounds good to you now !

To simplify the addition of new datasets, we'd like to have the dataset infos in the YAML and deprecate the dataset_infos.json file. YAML is readable and easy to edit, and the YAML metadata of the readme already contain dataset metadata so we would have everything in one place.

To be more specific, I moved these fields from DatasetInfo to the YAML:
- config_name (if there are several configs)
- download_size
- dataset_size
- features
- splits

Here is what I ended up with for `squad`:
```yaml
dataset_info:
  features:
  - name: id
    dtype: string
  - name: title
    dtype: string
  - name: context
    dtype: string
  - name: question
    dtype: string
  - name: answers
    sequence:
    - name: text
      dtype: string
    - name: answer_start
      dtype: int32
  splits:
  - name: train
    num_bytes: 79346360
    num_examples: 87599
  - name: validation
    num_bytes: 10473040
    num_examples: 10570
  config_name: plain_text
  download_size: 35142551
  dataset_size: 89819400
```

and it can be a list if there are several configs

I already did the change for `conll2000` and `crime_and_punish` as an example.

## Implementation details

### Load/Read

This is done via `DatasetInfosDict.write_to_directory/from_directory`

I had to implement a custom the YAML export logic for `SplitDict`, `Version` and `Features`.
The first two are trivial, but the logic for `Features` is more complicated, because I added a simplification step (or the YAML would be too long and less readable): it's just a formatting step to remove unnecessary nesting of YAML data.

### Other changes

I had to update the DatasetModule factories to also download the README.md alongside the dataset scripts/data files, and not just the dataset_infos.json

## YAML validation

I removed the old validation code that was in metadata.py, now we can just use the Hub YAML validation

## Datasets-cli

The `datasets-cli test --save_infos` command now creates a README.md file with the dataset_infos in it, instead of a datasets_infos.json file

## Backward compatibility

`dataset_infos.json` files are still supported and loaded if they exist to have full backward compatibility.
Though I removed the unnecessary keys when the value is the default (like all the `id: null` from the Value feature types) to make them easier to read.

## TODO

- [x] add comments
- [x] tests
- [x] document the new YAML fields
- [x] try to reload the new dataset_infos.json file content with an old version of `datasets`

## EDITS

- removed "config_name" when there's only one config
- removed "version" for now (?), because it's not useful in general
- renamed the yaml field "dataset_info" instead of "dataset_infos", since it only has one by default (and because "infos" is not english)

Fix https://github.com/huggingface/datasets/issues/4876 and fix #2773


Add note about loading image / audio files to docs

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4925). All of your documentation changes will be reflected on that endpoint.

Thanks for the feedback @polinaeterna ! I've reworded the docs a bit to integrate your comments and this should be ready for another review :)

> I've just realized that there is another PR about audio documentation open: #4872
> and there the more detailed description on how to use `audiofolder` is moved to another section ("Create an audio dataset")

Ah yes, let's add a comment to #4872 - that will be simpler than the alternatives :)

@polinaeterna @lhoestq What do you think about adding support for the metadata format from Kaggle (one metadata file for each split with the name equal to the split name) to ImageFolder/AudioFolder? I also think we can relax some requirements a bit by:
* allowing `filename` as the name of the main metadata column (currently, only `file_path` is allowed)
* not requiring that the features of all the given metadata files are equal. Instead, we can have a soft check by using `_check_if_features_can_be_aligned` + `_align_features`. The rationale is that train/val metadata often has extra columns compared to test metadata.

These changes would allow us to load the Kaggle dataset linked in the forum thread without any "interventions".

PS: this metadata format for ImageFolder was also proposed by @abhishekkrthakur initially.

Can you give more details about the Kaggle format ? I'm down to discuss it in a separate issue if you don't mind.

> allowing filename as the name of the main metadata column (currently, only file_path is allowed)

`filename` refers to the name of the file, so there's no logic about relative path or directories. If I recall correctly this is what we're doing right now so why not

> not requiring that the features of all the given metadata files are equal. Instead, we can have a soft check by using _check_if_features_can_be_aligned + _align_features. The rationale is that train/val metadata often has extra columns compared to test metadata.

+1 and we can set to None the missing features

I'm not sure if this is worth opening a new issue :).

What I mean by the Kaggle format is the structure like this one (the name of a metadata file is equal to the directory it "references"):
```
- train
    - img1.jpeg
    - img2.jpeg
    - ...
- test
    - img1.jpeg
    - img2.jpeg
    - ...
- train.csv
- test.csv
```

@mariosasko +1 to allowing different features set and metadata filenames corresponding to split names

Considering filename column - right now it's even called `file_name` now, which is not nice because in fact it's a relative file path indeed, so I think it should be `file_path` (and I don't know why I haven't thought about it before the release...)

@lewtun don't you mind if I close this pull request as I've integrated your changes in https://github.com/huggingface/datasets/pull/4872 ? (it doesn't have a link to a kaggle example though)

This PR adds a small note about how to load image / audio datasets that have multiple splits in their dataset structure.

Related forum thread: https://discuss.huggingface.co/t/loading-train-and-test-splits-with-audiofolder/22447

cc @NielsRogge


Concatenate_datasets loads everything into RAM

bug

Something isn't working

## Describe the bug
When loading the datasets seperately and saving them on disk, I want to concatenate them. But `concatenate_datasets` is filling up my RAM and the process gets killed. Is there a way to prevent this from happening or is this intended behaviour? Thanks in advance

## Steps to reproduce the bug
```python
gcs = gcsfs.GCSFileSystem(project='project')
datasets = [load_from_disk(f'path/to/slice/of/data/{i}', fs=gcs, keep_in_memory=False) for i in range(10)]

dataset = concatenate_datasets(datasets)
```

## Expected results
A concatenated dataset which is stored on my disk.

## Actual results
Concatenated dataset gets loaded into RAM and overflows it which gets the process killed.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: Linux-4.19.0-21-cloud-amd64-x86_64-with-glibc2.10
- Python version: 3.8.13
- PyArrow version: 8.0.1
- Pandas version: 1.4.3


decode mp3 with librosa if torchaudio is > 0.12 as a temporary workaround

_The documentation is not available anymore as the PR was closed or merged._

Thanks ! Should we still support torchaudio>0.12 if it works ? And if it doesn't we can explain that downgrading is the right solution, or alternatively use librosa

@lhoestq

> Should we still support torchaudio>0.12 if it works ? And if it doesn't we can explain that downgrading is the right solution, or alternatively use librosa

I'm not sure here, because from the one hand, if `torchaudio` works - it works 60 times faster then `librosa`.
But from the other hand, we will get inconsistent behavior (=different results of decoding) for users of `torchaudio>=0.12`.
I'd better go for using `librosa` only to avoid inconsistency then. wdyt?

It seems a bit too constraining to not allow users who have a working torchaudio 0.12 setup to not use it.

If the issue is about avoiding silent errors if the decoding changes, maybe we can log which back-end is used ? It can even be a warning with performance suggestions ("you're using librosa but torchaudio 0.xx is recommended").

Note that users can still have a requirements.txt or whatever in their projects if they really want full reproducibility (and it's the bare minimum imo)

There are multiple possible back-ends so it's maybe not reasonable to only allow one back-end, especially since each back-end has installation constrains and there's no "best" back-end.

Woohoo all green ! Feel free to merge if it's all good for you :)

`torchaudio>0.12` fails with decoding mp3 files if `ffmpeg<4`. currently we ask users to downgrade torchaudio, but sometimes it's not possible as torchaudio version is binded to torch version. as a temporary workaround we can decode mp3 with librosa (though it 60 times slower, at least it works)

another option would be to ask users to install the required version of `ffmpeg`, but is non-trivial on colab: it's not in apt packages in ubuntu 18 and `conda` is not preinstalled (with `conda` it would be easily installable)

- [x] decode with torchaudio anyway if the version of ffmpeg is correct? it's 60 times faster
- [x] tests
- [x] DO NOT FORGET to get back all the tests

see https://github.com/huggingface/datasets/issues/4776 and https://github.com/huggingface/datasets/issues/3663#issuecomment-1225797165 (there is a Colab notebook to reproduce the error)


I/O error on Google Colab in streaming mode

Something isn't working

## Describe the bug
When trying to load a streaming dataset in Google Colab the loading fails with an I/O error

## Steps to reproduce the bug
```python
import datasets
from datasets import load_dataset
hf_ds = load_dataset(path='wmt19', name='cs-en', streaming=True, split=datasets.Split.VALIDATION)
list(hf_ds.take(5))
```

## Expected results
It should load five data points

## Actual results
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-13-7b5b8b1e7e58>](https://localhost:8080/#) in <module>
      2 from datasets import load_dataset
      3 hf_ds = load_dataset(path='wmt19', name='cs-en', streaming=True, split=datasets.Split.VALIDATION)
----> 4 list(hf_ds.take(5))

6 frames
[/usr/local/lib/python3.7/dist-packages/datasets/iterable_dataset.py](https://localhost:8080/#) in __iter__(self)
    716
    717     def __iter__(self):
--> 718         for key, example in self._iter():
    719             if self.features:
    720                 # `IterableDataset` automatically fills missing columns with None.

[/usr/local/lib/python3.7/dist-packages/datasets/iterable_dataset.py](https://localhost:8080/#) in _iter(self)
    706         else:
    707             ex_iterable = self._ex_iterable
--> 708         yield from ex_iterable
    709
    710     def _iter_shard(self, shard_idx: int):

[/usr/local/lib/python3.7/dist-packages/datasets/iterable_dataset.py](https://localhost:8080/#) in __iter__(self)
    582
    583     def __iter__(self):
--> 584         yield from islice(self.ex_iterable, self.n)
    585
    586     def shuffle_data_sources(self, generator: np.random.Generator) -> "TakeExamplesIterable":

[/usr/local/lib/python3.7/dist-packages/datasets/iterable_dataset.py](https://localhost:8080/#) in __iter__(self)
    110
    111     def __iter__(self):
--> 112         yield from self.generate_examples_fn(**self.kwargs)
    113
    114     def shuffle_data_sources(self, generator: np.random.Generator) -> "ExamplesIterable":

[~/.cache/huggingface/modules/datasets_modules/datasets/wmt19/aeadcbe9f1cbf9969e603239d33d3e43670cf250c1158edf74f5f6e74d4f21d0/wmt_utils.py](https://localhost:8080/#) in _generate_examples(self, split_subsets, extraction_map, with_translation)
    845                 raise ValueError("Invalid number of files: %d" % len(files))
    846
--> 847             for sub_key, ex in sub_generator(*sub_generator_args):
    848                 if not all(ex.values()):
    849                     continue

[~/.cache/huggingface/modules/datasets_modules/datasets/wmt19/aeadcbe9f1cbf9969e603239d33d3e43670cf250c1158edf74f5f6e74d4f21d0/wmt_utils.py](https://localhost:8080/#) in _parse_parallel_sentences(f1, f2, filename1, filename2)
    923         l2_sentences, l2 = parse_file(f2_i, filename2)
    924
--> 925         for line_id, (s1, s2) in enumerate(zip(l1_sentences, l2_sentences)):
    926             key = f"{f_id}/{line_id}"
    927             yield key, {l1: s1, l2: s2}

[~/.cache/huggingface/modules/datasets_modules/datasets/wmt19/aeadcbe9f1cbf9969e603239d33d3e43670cf250c1158edf74f5f6e74d4f21d0/wmt_utils.py](https://localhost:8080/#) in gen()
    895
    896         def gen():
--> 897             with open(path, encoding="utf-8") as f:
    898                 for line in f:
    899                     seg_match = re.match(seg_re, line)

ValueError: I/O operation on closed file.
```

## Environment info
Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 2.4.0
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 9.0.0. (the same error happened with PyArrow version 6.0.0)
- Pandas version: 1.3.5


Fix missing tags in dataset cards

Fix missing tags in dataset cards:
- eraser_multi_rc
- hotpot_qa
- metooma
- movie_rationales
- qanta
- quora
- quoref
- race
- ted_hrlr
- ted_talks_iwslt

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896
- #4908


Something isn't working

Hi @DataNoob0723,

Under the hood, we use `pandas` to load CSV/TSV files. Therefore, you should use "csv" and pass `sep="\t"`, as explained in our docs: https://huggingface.co/docs/datasets/v2.4.0/en/package_reference/loading_methods#from-files
```python
ds = load_dataset('csv', sep="\t", data_files=data_files)
```

## Describe the bug
Unable to load local tsv files through load_dataset method.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
data_files = {
    'train': 'train.tsv',
    'test': 'test.tsv'
}
raw_datasets = load_dataset('tsv', data_files=data_files)

## Expected results
I am pretty sure the data files exist in the current directory. The above code should load them as Datasets, but threw exceptions.

## Actual results
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
[<ipython-input-9-24207899c1af>](https://localhost:8080/#) in <module>
----> 1 raw_datasets = load_dataset('tsv', data_files='train.tsv')

2 frames
[/usr/local/lib/python3.7/dist-packages/datasets/load.py](https://localhost:8080/#) in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1244                         f"Couldn't find a dataset script at {relative_to_absolute_path(combined_path)} or any data file in the same directory. "
   1245                         f"Couldn't find '{path}' on the Hugging Face Hub either: {type(e1).__name__}: {e1}"
-> 1246                     ) from None
   1247                 raise e1 from None
   1248     else:

FileNotFoundError: Couldn't find a dataset script at /content/tsv/tsv.py or any data file in the same directory. Couldn't find 'tsv' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/main/datasets/tsv/tsv.py

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5


feat: improve error message on Keys mismatch. closes #4917

We are having an unrelated issue that makes several tests fail. We are working on that. Once fixed, you will be able to merge the main branch into this, so that you get the fix and the tests pass...

Hi @lhoestq what do you think?

Let me give you a code sample:
```py
>>> import datasets
>>> foo = datasets.Dataset.from_dict({'foo':[0,1], 'bar':[2,3]})
>>> foo.save_to_disk('foo')
# edit foo/dataset_info.json e.g. rename the 'foo' feature to 'baz'
>>> datasets.load_from_disk('foo')
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-4863e606b330> in <module>
----> 1 datasets.load_from_disk('foo')

~/code/datasets/src/datasets/load.py in load_from_disk(dataset_path, fs, keep_in_memory)
   1851         raise FileNotFoundError(f"Directory {dataset_path} not found")
   1852     if fs.isfile(Path(dest_dataset_path, config.DATASET_INFO_FILENAME).as_posix()):
-> 1853         return Dataset.load_from_disk(dataset_path, fs, keep_in_memory=keep_in_memory)
   1854     elif fs.isfile(Path(dest_dataset_path, config.DATASETDICT_JSON_FILENAME).as_posix()):
   1855         return DatasetDict.load_from_disk(dataset_path, fs, keep_in_memory=keep_in_memory)

~/code/datasets/src/datasets/arrow_dataset.py in load_from_disk(dataset_path, fs, keep_in_memory)
   1230             info=dataset_info,
   1231             split=split,
-> 1232             fingerprint=state["_fingerprint"],
   1233         )
   1234

~/code/datasets/src/datasets/arrow_dataset.py in __init__(self, arrow_table, info, split, indices_table, fingerprint)
    687             self.info.features = inferred_features
    688         else:  # make sure the nested columns are in the right order
--> 689             self.info.features = self.info.features.reorder_fields_as(inferred_features)
    690
    691         # Infer fingerprint if None

~/code/datasets/src/datasets/features/features.py in reorder_fields_as(self, other)
   1771                 return source
   1772
-> 1773         return Features(recursive_reorder(self, other))
   1774
   1775     def flatten(self, max_depth=16) -> "Features":

~/code/datasets/src/datasets/features/features.py in recursive_reorder(source, target, stack)
   1760                                f"{source.keys()-target.keys()} are missing from dataset.arrow "
   1761                                f"and {target.keys()-source.keys()} are missing from dataset_info.json"+stack_position)
-> 1762                     raise ValueError(message)
   1763                 return {key: recursive_reorder(source[key], target[key], stack + f".{key}") for key in target}
   1764             elif isinstance(source, list):

ValueError: Keys mismatch: between {'baz': Value(dtype='int64', id=None), 'bar': Value(dtype='int64', id=None)} (dataset_info.json) and {'foo': Value(dtype='int64', id=None), 'bar': Value(dtype='int64', id=None)} (inferred from dataset.arrow).
{'baz'} are missing from dataset.arrow and {'foo'} are missing from dataset_info.json
```


Dataset Viewer issue for pysentimiento/spanish-targeted-sentiment-headlines

Thanks for reporting, it's fixed now (I refreshed it manually). It's a known issue; we hope it will be fixed permanently in a few days.

<img width="1508" alt="Capture d’écran 2022-09-05 à 18 31 22" src="https://user-images.githubusercontent.com/1676121/188489762-0ed86a7e-dfb3-46e8-a125-43b815a2c6f4.png">

### Link

https://huggingface.co/datasets/pysentimiento/spanish-targeted-sentiment-headlines

### Description

After moving the dataset from my user (`finiteautomata`) to the `pysentimiento` organization, the dataset viewer says that it doesn't exist.

### Owner

_No response_


Keys mismatch: make error message more informative

enhancement

New feature or request

good first issue

Good for newcomers

Good idea ! I think this can be improved in `Features.reorder_fields_as()` indeed at

https://github.com/huggingface/datasets/blob/7feeb5648a63b6135a8259dedc3b1e19185ee4c7/src/datasets/features/features.py#L1739-L1740

Is it something you would be interested in contributing ?

Is this open to work on? I'd love to take on this as my first issue.

Hi @daspartho I’ve opened a PR #4919
I don’t think there’s much left to do

**Is your feature request related to a problem? Please describe.**
When loading a dataset from disk with a defect in its `dataset_info.json` describing its features (I don’t know when/why/how this happens but it deserves its own issue), you will get an error message like:
`ValueError: Keys mismatch: between {'bar': Value(dtype='int64', id=None)} and {'foo': Value(dtype='int64', id=None)}`

Which is fine when you have only a few features like in the example but it gets very hard to read when you have a lot of features in your dataset.

**Describe the solution you'd like**
The error message should give the difference between the features (what keys are in A but missing in B and vice-versa). It should also tell which keys are inferred from `dataset.arrow` and which come from `dataset_info.json`.

Willing to help :)


Apache Beam unable to write the downloaded wikipedia dataset

bug

Something isn't working

## Describe the bug
Hi, I am currently trying to download wikipedia dataset using
load_dataset("wikipedia", language="aa", date="20220401", split="train",beam_runner='DirectRunner'). However, I end up in getting filenotfound error. I get this error for any language I try to download. It downloads the file but while saving it in hugging face cache it fails to write. This happens for any available date of any language in wikipedia dump. I had raised another issue earlier #4915 but probably was not that clear and the solution provider misunderstood my problem. Hence raising one more issue. Any help is appreciated.

## Steps to reproduce the bug
```python
from datasets import load_dataset
load_dataset("wikipedia", language="aa", date="20220401", split="train",beam_runner='DirectRunner')
```

## Expected results
to load the dataset

## Actual results
I am pasting the error trace here:
Downloading builder script: 35.9kB [00:00, ?B/s]
Downloading metadata: 30.4kB [00:00, 1.94MB/s]
Using custom data configuration 20220401.aa-date=20220401,language=aa
Downloading and preparing dataset wikipedia/20220401.aa to C:\Users\Shilpa.cache\huggingface\datasets\wikipedia\20220401.aa-date=20220401,language=aa\2.0.0\aa542ed919df55cc5d3347f42dd4521d05ca68751f50dbc32bae2a7f1e167559...
Downloading data: 100%|████████████████████████████████████████████████████████████| 11.1k/11.1k [00:00<00:00, 712kB/s]
Downloading data files: 100%|████████████████████████████████████████████████████████████| 1/1 [00:02<00:00, 2.82s/it]
Extracting data files: 100%|█████████████████████████████████████████████████████████████████████| 1/1 [00:00<?, ?it/s]
Downloading data: 100%|███████████████████████████████████████████████████████████| 35.6k/35.6k [00:00<00:00, 84.3kB/s]
Downloading data files: 100%|████████████████████████████████████████████████████████████| 1/1 [00:02<00:00, 2.93s/it]
Traceback (most recent call last):
File "apache_beam\runners\common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam\runners\common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
File "apache_beam\runners\common.py", line 981, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
File "apache_beam\runners\common.py", line 1571, in apache_beam.runners.common._OutputHandler.handle_process_outputs
File "G:\Python3.7\lib\site-packages\apache_beam\io\iobase.py", line 1193, in process
self.writer = self.sink.open_writer(init_result, str(uuid.uuid4()))
File "G:\Python3.7\lib\site-packages\apache_beam\options\value_provider.py", line 193, in _f
return fnc(self, *args, **kwargs)
File "G:\Python3.7\lib\site-packages\apache_beam\io\filebasedsink.py", line 202, in open_writer
return FileBasedSinkWriter(self, writer_path)
File "G:\Python3.7\lib\site-packages\apache_beam\io\filebasedsink.py", line 419, in init
self.temp_handle = self.sink.open(temp_shard_path)
File "G:\Python3.7\lib\site-packages\apache_beam\io\parquetio.py", line 553, in open
self._file_handle = super().open(temp_path)
File "G:\Python3.7\lib\site-packages\apache_beam\options\value_provider.py", line 193, in _f
return fnc(self, *args, **kwargs)
File "G:\Python3.7\lib\site-packages\apache_beam\io\filebasedsink.py", line 139, in open
temp_path, self.mime_type, self.compression_type)
File "G:\Python3.7\lib\site-packages\apache_beam\io\filesystems.py", line 224, in create
return filesystem.create(path, mime_type, compression_type)
File "G:\Python3.7\lib\site-packages\apache_beam\io\localfilesystem.py", line 163, in create
return self._path_open(path, 'wb', mime_type, compression_type)
File "G:\Python3.7\lib\site-packages\apache_beam\io\localfilesystem.py", line 140, in _path_open
raw_file = io.open(path, mode)
FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\Shilpa\.cache\huggingface\datasets\wikipedia\20220401.aa-date=20220401,language=aa\2.0.0\aa542ed919df55cc5d3347f42dd4521d05ca68751f50dbc32bae2a7f1e167559.incomplete\beam-temp-wikipedia-train-880233e8287e11edaf9d3ca067f2714e\20a05238-6106-4420-a713-4eca6dd5959a.wikipedia-train'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "G:/abc/temp.py", line 32, in
beam_runner='DirectRunner')
File "G:\Python3.7\lib\site-packages\datasets\load.py", line 1751, in load_dataset
use_auth_token=use_auth_token,
File "G:\Python3.7\lib\site-packages\datasets\builder.py", line 705, in download_and_prepare
dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
File "G:\Python3.7\lib\site-packages\datasets\builder.py", line 1394, in _download_and_prepare
pipeline_results = pipeline.run()
File "G:\Python3.7\lib\site-packages\apache_beam\pipeline.py", line 574, in run
return self.runner.run_pipeline(self, self._options)
File "G:\Python3.7\lib\site-packages\apache_beam\runners\direct\direct_runner.py", line 131, in run_pipeline
return runner.run_pipeline(pipeline, options)
File "G:\Python3.7\lib\site-packages\apache_beam\runners\portability\fn_api_runner\fn_runner.py", line 201, in run_pipeline
options)
File "G:\Python3.7\lib\site-packages\apache_beam\runners\portability\fn_api_runner\fn_runner.py", line 212, in run_via_runner_api
return self.run_stages(stage_context, stages)
File "G:\Python3.7\lib\site-packages\apache_beam\runners\portability\fn_api_runner\fn_runner.py", line 443, in run_stages
runner_execution_context, bundle_context_manager, bundle_input)
File "G:\Python3.7\lib\site-packages\apache_beam\runners\portability\fn_api_runner\fn_runner.py", line 776, in _execute_bundle
bundle_manager))
File "G:\Python3.7\lib\site-packages\apache_beam\runners\portability\fn_api_runner\fn_runner.py", line 1000, in _run_bundle
data_input, data_output, input_timers, expected_timer_output)
File "G:\Python3.7\lib\site-packages\apache_beam\runners\portability\fn_api_runner\fn_runner.py", line 1309, in process_bundle
result_future = self._worker_handler.control_conn.push(process_bundle_req)
File "G:\Python3.7\lib\site-packages\apache_beam\runners\portability\fn_api_runner\worker_handlers.py", line 380, in push
response = self.worker.do_instruction(request)
File "G:\Python3.7\lib\site-packages\apache_beam\runners\worker\sdk_worker.py", line 598, in do_instruction
getattr(request, request_type), request.instruction_id)
File "G:\Python3.7\lib\site-packages\apache_beam\runners\worker\sdk_worker.py", line 635, in process_bundle
bundle_processor.process_bundle(instruction_id))
File "G:\Python3.7\lib\site-packages\apache_beam\runners\worker\bundle_processor.py", line 1004, in process_bundle
element.data)
File "G:\Python3.7\lib\site-packages\apache_beam\runners\worker\bundle_processor.py", line 227, in process_encoded
self.output(decoded_value)
File "apache_beam\runners\worker\operations.py", line 526, in apache_beam.runners.worker.operations.Operation.output
File "apache_beam\runners\worker\operations.py", line 528, in apache_beam.runners.worker.operations.Operation.output
File "apache_beam\runners\worker\operations.py", line 237, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
File "apache_beam\runners\worker\operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
File "apache_beam\runners\worker\operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam\runners\worker\operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam\runners\common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam\runners\common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam\runners\common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam\runners\common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "apache_beam\runners\common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
File "apache_beam\runners\common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
File "apache_beam\runners\worker\operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
File "apache_beam\runners\worker\operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam\runners\worker\operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam\runners\common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam\runners\common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam\runners\common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam\runners\common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "apache_beam\runners\common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
File "apache_beam\runners\common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
File "apache_beam\runners\worker\operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
File "apache_beam\runners\worker\operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam\runners\worker\operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam\runners\common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam\runners\common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam\runners\common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam\runners\common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
File "apache_beam\runners\common.py", line 981, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
File "apache_beam\runners\common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
File "apache_beam\runners\common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
File "apache_beam\runners\worker\operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
File "apache_beam\runners\worker\operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam\runners\worker\operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam\runners\common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam\runners\common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam\runners\common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam\runners\common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "apache_beam\runners\common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
File "apache_beam\runners\common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
File "apache_beam\runners\worker\operations.py", line 324, in apache_beam.runners.worker.operations.GeneralPurposeConsumerSet.receive
File "apache_beam\runners\worker\operations.py", line 905, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam\runners\worker\operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam\runners\worker\operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam\runners\common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam\runners\common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam\runners\common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam\runners\common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "apache_beam\runners\common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
File "apache_beam\runners\common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
File "apache_beam\runners\worker\operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
File "apache_beam\runners\worker\operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam\runners\worker\operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam\runners\common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam\runners\common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam\runners\common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam\runners\common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
File "apache_beam\runners\common.py", line 981, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
File "apache_beam\runners\common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
File "apache_beam\runners\common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
File "apache_beam\runners\worker\operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
File "apache_beam\runners\worker\operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam\runners\worker\operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam\runners\common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam\runners\common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam\runners\common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam\runners\common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
File "apache_beam\runners\common.py", line 981, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
File "apache_beam\runners\common.py", line 1571, in apache_beam.runners.common._OutputHandler.handle_process_outputs
File "G:\Python3.7\lib\site-packages\apache_beam\io\iobase.py", line 1193, in process
self.writer = self.sink.open_writer(init_result, str(uuid.uuid4()))
File "G:\Python3.7\lib\site-packages\apache_beam\options\value_provider.py", line 193, in _f
return fnc(self, *args, **kwargs)
File "G:\Python3.7\lib\site-packages\apache_beam\io\filebasedsink.py", line 202, in open_writer
return FileBasedSinkWriter(self, writer_path)
File "G:\Python3.7\lib\site-packages\apache_beam\io\filebasedsink.py", line 419, in init
self.temp_handle = self.sink.open(temp_shard_path)
File "G:\Python3.7\lib\site-packages\apache_beam\io\parquetio.py", line 553, in open
self._file_handle = super().open(temp_path)
File "G:\Python3.7\lib\site-packages\apache_beam\options\value_provider.py", line 193, in _f
return fnc(self, *args, **kwargs)
File "G:\Python3.7\lib\site-packages\apache_beam\io\filebasedsink.py", line 139, in open
temp_path, self.mime_type, self.compression_type)
File "G:\Python3.7\lib\site-packages\apache_beam\io\filesystems.py", line 224, in create
return filesystem.create(path, mime_type, compression_type)
File "G:\Python3.7\lib\site-packages\apache_beam\io\localfilesystem.py", line 163, in create
return self._path_open(path, 'wb', mime_type, compression_type)
File "G:\Python3.7\lib\site-packages\apache_beam\io\localfilesystem.py", line 140, in _path_open
raw_file = io.open(path, mode)
RuntimeError: FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\Shilpa\.cache\huggingface\datasets\wikipedia\20220401.aa-date=20220401,language=aa\2.0.0\aa542ed919df55cc5d3347f42dd4521d05ca68751f50dbc32bae2a7f1e167559.incomplete\beam-temp-wikipedia-train-880233e8287e11edaf9d3ca067f2714e\20a05238-6106-4420-a713-4eca6dd5959a.wikipedia-train' [while running 'train/Save to parquet/Write/WriteImpl/WriteBundles']

## Environment info
Python: 3.7.6
Windows 10 Pro
datasets :2.4.0
apache_beam: 2.41.0
mwparserfromhell: 0.6.4


FileNotFoundError while downloading wikipedia dataset for any language

Something isn't working

Hi @Shilpac20,

As explained in the Wikipedia dataset card: https://huggingface.co/datasets/wikipedia
> You can find the full list of languages and dates [here](https://dumps.wikimedia.org/backup-index.html).

This means that, before passing a specific date, you should first make sure it is available online, as Wikimedia only keeps last X months (depending on the size of the corresponding language dump)): e.g. to see which dates "aa" Wikipedia is available online, see https://dumps.wikimedia.org/aawiki/ (as of today 2022-08-31, the available dates are from [20220401](https://dumps.wikimedia.org/aawiki/20220401/) to [20220820](https://dumps.wikimedia.org/aawiki/20220820/)).

Hi, the date that I have specified "20220401" is available for the language "aa". The error persists for any other available dates as present in https://dumps.wikimedia.org/aawiki/. The error is mainly due to apache beam not able to write the downloaded files. Any help on this?

I see, sorry, I misread your issue.

We are investigating this.

I am struggling with basically the same issue. I am trying to download the German Wikipedia dump.

As per the [documentation](https://huggingface.co/datasets/wikipedia), `"20220301.de"` should be available as a pre-processed dataset.

Issuing the command mentioned in the documentation cited above

    from datasets import load_dataset
    load_dataset("wikipedia", "20220301.de")

raises the following `FileNotFound` error

    FileNotFoundError: Couldn't find file at https://dumps.wikimedia.org/dewiki/20220301/dumpstatus.json

Using the ([undocumented](https://huggingface.co/docs/datasets/v1.2.1/package_reference/loading_methods.html#datasets.load_dataset)?) call to `load_dataset()` with `language` and `date` parameters

    load_dataset("wikipedia", language="de", date="20220301", beam_runner="DirectRunner")

produces the same error.

EDIT: as I am using `datasets` v2.7.1, I should be looking at [that version's documentation](https://huggingface.co/docs/datasets/v2.7.1/en/package_reference/loading_methods#datasets.load_dataset)! It is mentioned there, that additional `kwargs` are "passed to the [BuilderConfig](https://huggingface.co/docs/datasets/v2.7.1/en/package_reference/builder_classes#datasets.BuilderConfig) and used in the [DatasetBuilder](https://huggingface.co/docs/datasets/v2.7.1/en/package_reference/builder_classes#datasets.DatasetBuilder)". So I guess that is how `language` and `date` are used.

As I can see a folder `20221130` on `https://dumps.wikimedia.org/dewiki/`, I also tried

    from datasets import load_dataset
    load_dataset("wikipedia", "20221130.de")

which throws another error:

    ValueError: BuilderConfig 20221120.de not found. Available: ['20220301.aa', ... '20220301.de', ...

basically telling me that the dataset I originally requested (`'20220301.de'`) is available...

It seems that `load_dataset` is not handling the vanishing older dumps for Wikipedia correctly?

I am able to start downloading the dataset when trying anything with the recent dumps for 20221201. But obviously, those are the big wiki dumps and I need the smaller preloaded version.

I am now getting some error when the files show up in my cache but it will say FileNotFoundError at the end of the download for some reason. The cache directory to the datasets\wikipedia\date.bn\ had something in it, then when the error came up it disappeared.

It is easy to test with the langauge "bn" because the amount of files is low.

dataset = load_dataset('wikipedia', date="20221201", language="bn", split='train', beam_runner='DirectRunner')

## Describe the bug
Hi, I am currently trying to download wikipedia dataset using
load_dataset("wikipedia", language="aa", date="20220401", split="train",beam_runner='DirectRunner'). However, I end up in getting filenotfound error.  I get this error for any language I try to download.

Environment:

## Steps to reproduce the bug
```python
from datasets import load_dataset
load_dataset("wikipedia", language="aa", date="20220401", split="train",beam_runner='DirectRunner')
```

## Expected results
to load the dataset

## Actual results
I am pasting the error trace here:
Downloading builder script: 35.9kB [00:00, ?B/s]
Downloading metadata: 30.4kB [00:00, 1.94MB/s]
Using custom data configuration 20220401.aa-date=20220401,language=aa
Downloading and preparing dataset wikipedia/20220401.aa to C:\Users\Shilpa\.cache\huggingface\datasets\wikipedia\20220401.aa-date=20220401,language=aa\2.0.0\aa542ed919df55cc5d3347f42dd4521d05ca68751f50dbc32bae2a7f1e167559...
Downloading data: 100%|████████████████████████████████████████████████████████████| 11.1k/11.1k [00:00<00:00, 712kB/s]
Downloading data files: 100%|████████████████████████████████████████████████████████████| 1/1 [00:02<00:00,  2.82s/it]
Extracting data files: 100%|█████████████████████████████████████████████████████████████████████| 1/1 [00:00<?, ?it/s]
Downloading data: 100%|███████████████████████████████████████████████████████████| 35.6k/35.6k [00:00<00:00, 84.3kB/s]
Downloading data files: 100%|████████████████████████████████████████████████████████████| 1/1 [00:02<00:00,  2.93s/it]
Traceback (most recent call last):
  File "apache_beam\runners\common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam\runners\common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam\runners\common.py", line 981, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File "apache_beam\runners\common.py", line 1571, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File "G:\Python3.7\lib\site-packages\apache_beam\io\iobase.py", line 1193, in process
    self.writer = self.sink.open_writer(init_result, str(uuid.uuid4()))
  File "G:\Python3.7\lib\site-packages\apache_beam\options\value_provider.py", line 193, in _f
    return fnc(self, *args, **kwargs)
  File "G:\Python3.7\lib\site-packages\apache_beam\io\filebasedsink.py", line 202, in open_writer
    return FileBasedSinkWriter(self, writer_path)
  File "G:\Python3.7\lib\site-packages\apache_beam\io\filebasedsink.py", line 419, in __init__
    self.temp_handle = self.sink.open(temp_shard_path)
  File "G:\Python3.7\lib\site-packages\apache_beam\io\parquetio.py", line 553, in open
    self._file_handle = super().open(temp_path)
  File "G:\Python3.7\lib\site-packages\apache_beam\options\value_provider.py", line 193, in _f
    return fnc(self, *args, **kwargs)
  File "G:\Python3.7\lib\site-packages\apache_beam\io\filebasedsink.py", line 139, in open
    temp_path, self.mime_type, self.compression_type)
  File "G:\Python3.7\lib\site-packages\apache_beam\io\filesystems.py", line 224, in create
    return filesystem.create(path, mime_type, compression_type)
  File "G:\Python3.7\lib\site-packages\apache_beam\io\localfilesystem.py", line 163, in create
    return self._path_open(path, 'wb', mime_type, compression_type)
  File "G:\Python3.7\lib\site-packages\apache_beam\io\localfilesystem.py", line 140, in _path_open
    raw_file = io.open(path, mode)
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\Shilpa\\.cache\\huggingface\\datasets\\wikipedia\\20220401.aa-date=20220401,language=aa\\2.0.0\\aa542ed919df55cc5d3347f42dd4521d05ca68751f50dbc32bae2a7f1e167559.incomplete\\beam-temp-wikipedia-train-880233e8287e11edaf9d3ca067f2714e\\20a05238-6106-4420-a713-4eca6dd5959a.wikipedia-train'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "G:/abc/temp.py", line 32, in <module>
    beam_runner='DirectRunner')
  File "G:\Python3.7\lib\site-packages\datasets\load.py", line 1751, in load_dataset
    use_auth_token=use_auth_token,
  File "G:\Python3.7\lib\site-packages\datasets\builder.py", line 705, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "G:\Python3.7\lib\site-packages\datasets\builder.py", line 1394, in _download_and_prepare
    pipeline_results = pipeline.run()
  File "G:\Python3.7\lib\site-packages\apache_beam\pipeline.py", line 574, in run
    return self.runner.run_pipeline(self, self._options)
  File "G:\Python3.7\lib\site-packages\apache_beam\runners\direct\direct_runner.py", line 131, in run_pipeline
    return runner.run_pipeline(pipeline, options)
  File "G:\Python3.7\lib\site-packages\apache_beam\runners\portability\fn_api_runner\fn_runner.py", line 201, in run_pipeline
    options)
  File "G:\Python3.7\lib\site-packages\apache_beam\runners\portability\fn_api_runner\fn_runner.py", line 212, in run_via_runner_api
    return self.run_stages(stage_context, stages)
  File "G:\Python3.7\lib\site-packages\apache_beam\runners\portability\fn_api_runner\fn_runner.py", line 443, in run_stages
    runner_execution_context, bundle_context_manager, bundle_input)
  File "G:\Python3.7\lib\site-packages\apache_beam\runners\portability\fn_api_runner\fn_runner.py", line 776, in _execute_bundle
    bundle_manager))
  File "G:\Python3.7\lib\site-packages\apache_beam\runners\portability\fn_api_runner\fn_runner.py", line 1000, in _run_bundle
    data_input, data_output, input_timers, expected_timer_output)
  File "G:\Python3.7\lib\site-packages\apache_beam\runners\portability\fn_api_runner\fn_runner.py", line 1309, in process_bundle
    result_future = self._worker_handler.control_conn.push(process_bundle_req)
  File "G:\Python3.7\lib\site-packages\apache_beam\runners\portability\fn_api_runner\worker_handlers.py", line 380, in push
    response = self.worker.do_instruction(request)
  File "G:\Python3.7\lib\site-packages\apache_beam\runners\worker\sdk_worker.py", line 598, in do_instruction
    getattr(request, request_type), request.instruction_id)
  File "G:\Python3.7\lib\site-packages\apache_beam\runners\worker\sdk_worker.py", line 635, in process_bundle
    bundle_processor.process_bundle(instruction_id))
  File "G:\Python3.7\lib\site-packages\apache_beam\runners\worker\bundle_processor.py", line 1004, in process_bundle
    element.data)
  File "G:\Python3.7\lib\site-packages\apache_beam\runners\worker\bundle_processor.py", line 227, in process_encoded
    self.output(decoded_value)
  File "apache_beam\runners\worker\operations.py", line 526, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam\runners\worker\operations.py", line 528, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam\runners\worker\operations.py", line 237, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam\runners\worker\operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam\runners\worker\operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam\runners\worker\operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam\runners\common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam\runners\common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam\runners\common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam\runners\common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "apache_beam\runners\common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File "apache_beam\runners\common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
  File "apache_beam\runners\worker\operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam\runners\worker\operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam\runners\worker\operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam\runners\common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam\runners\common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam\runners\common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam\runners\common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "apache_beam\runners\common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File "apache_beam\runners\common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
  File "apache_beam\runners\worker\operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam\runners\worker\operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam\runners\worker\operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam\runners\common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam\runners\common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam\runners\common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam\runners\common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam\runners\common.py", line 981, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File "apache_beam\runners\common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File "apache_beam\runners\common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
  File "apache_beam\runners\worker\operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam\runners\worker\operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam\runners\worker\operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam\runners\common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam\runners\common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam\runners\common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam\runners\common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "apache_beam\runners\common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File "apache_beam\runners\common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
  File "apache_beam\runners\worker\operations.py", line 324, in apache_beam.runners.worker.operations.GeneralPurposeConsumerSet.receive
  File "apache_beam\runners\worker\operations.py", line 905, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam\runners\worker\operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam\runners\worker\operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam\runners\common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam\runners\common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam\runners\common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam\runners\common.py", line 623, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "apache_beam\runners\common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File "apache_beam\runners\common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
  File "apache_beam\runners\worker\operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam\runners\worker\operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam\runners\worker\operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam\runners\common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam\runners\common.py", line 1491, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam\runners\common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam\runners\common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam\runners\common.py", line 981, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File "apache_beam\runners\common.py", line 1581, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File "apache_beam\runners\common.py", line 1694, in apache_beam.runners.common._OutputHandler._write_value_to_tag
  File "apache_beam\runners\worker\operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam\runners\worker\operations.py", line 907, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam\runners\worker\operations.py", line 908, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam\runners\common.py", line 1419, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam\runners\common.py", line 1507, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam\runners\common.py", line 1417, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam\runners\common.py", line 837, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam\runners\common.py", line 981, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File "apache_beam\runners\common.py", line 1571, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File "G:\Python3.7\lib\site-packages\apache_beam\io\iobase.py", line 1193, in process
    self.writer = self.sink.open_writer(init_result, str(uuid.uuid4()))
  File "G:\Python3.7\lib\site-packages\apache_beam\options\value_provider.py", line 193, in _f
    return fnc(self, *args, **kwargs)
  File "G:\Python3.7\lib\site-packages\apache_beam\io\filebasedsink.py", line 202, in open_writer
    return FileBasedSinkWriter(self, writer_path)
  File "G:\Python3.7\lib\site-packages\apache_beam\io\filebasedsink.py", line 419, in __init__
    self.temp_handle = self.sink.open(temp_shard_path)
  File "G:\Python3.7\lib\site-packages\apache_beam\io\parquetio.py", line 553, in open
    self._file_handle = super().open(temp_path)
  File "G:\Python3.7\lib\site-packages\apache_beam\options\value_provider.py", line 193, in _f
    return fnc(self, *args, **kwargs)
  File "G:\Python3.7\lib\site-packages\apache_beam\io\filebasedsink.py", line 139, in open
    temp_path, self.mime_type, self.compression_type)
  File "G:\Python3.7\lib\site-packages\apache_beam\io\filesystems.py", line 224, in create
    return filesystem.create(path, mime_type, compression_type)
  File "G:\Python3.7\lib\site-packages\apache_beam\io\localfilesystem.py", line 163, in create
    return self._path_open(path, 'wb', mime_type, compression_type)
  File "G:\Python3.7\lib\site-packages\apache_beam\io\localfilesystem.py", line 140, in _path_open
    raw_file = io.open(path, mode)
RuntimeError: FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\Shilpa\\.cache\\huggingface\\datasets\\wikipedia\\20220401.aa-date=20220401,language=aa\\2.0.0\\aa542ed919df55cc5d3347f42dd4521d05ca68751f50dbc32bae2a7f1e167559.incomplete\\beam-temp-wikipedia-train-880233e8287e11edaf9d3ca067f2714e\\20a05238-6106-4420-a713-4eca6dd5959a.wikipedia-train' [while running 'train/Save to parquet/Write/WriteImpl/WriteBundles']

## Environment info
Python: 3.7.6
Windows 10 Pro
datasets :2.4.0
apache_beam: 2.41.0
mwparserfromhell: 0.6.4


Support streaming swda dataset

Support streaming swda dataset.


Add license and citation information to cosmos_qa dataset

This PR adds the license information to `cosmos_qa` dataset, once reported via email by Yejin Choi, the dataset is licensed under CC BY 4.0.

This PR also updates the citation information.


datasets map() handles all data at a stroke and takes long time

Hi ! Interesting question ;)

> Which is better? Process in map() or in data-collator

As you said, both can be used in practice: map() if you want to preprocess before training, or a data-collator (or the equivalent `dataset.set_transform`) if you want to preprocess on-the-fly during training. Both options are great and really depend on your case.

To choose between the two, here are IMO the main caveats of each approach:
- if your preprocessing takes too much CPU for example, using a data-collator may slow down your training and your GPUs may not work at full speed
- on the other hand, map() may take a lot of time and disk space to run if your dataset is too big.

> Why huggingface advises map() function? There should be some advantages to using map()

To get the best throughput when training a model, it is often recommended to preprocess your dataset before training. Note that preprocessing may include other steps before tokenization such as data filtering, cleaning, chunking etc. which are often done before training.

Thanks for your clear explanation @lhoestq !
> *  if your preprocessing takes too much CPU for example, using a data-collator may slow down your training and your GPUs may not work at full speed
> * on the other hand, map() may take a lot of time and disk space to run if your dataset is too big.

I really agree with you. There should be some trade-off between processing before and during the train loop.
Besides, I find `map()` function can cache the results once it has been executed. Very useful!

I'm closing this issue if you don't mind, feel free to reopen if needed ;)

@lhoestq How to preprocess on-the-fly during training？my data is about 1w hours, when I use map to  preprocess, and It's not finished yet， but all disk space(2T) is full.

Hi ! You can do that using `set_transform`, see https://huggingface.co/docs/datasets/process#format-transform for more info :)

**1. Background**

Huggingface datasets package advises using `map()`  to process data in batches. In the example code on pretraining masked language model, they use `map()` to tokenize all data at a stroke before the train loop.

The corresponding code:
```
with accelerator.main_process_first():
    tokenized_datasets = raw_datasets.map(
        tokenize_function,
        batched=True,
        num_proc=args.preprocessing_num_workers,
        remove_columns=column_names,
        load_from_cache_file=not args.overwrite_cache,
        desc="Running tokenizer on every text in dataset"
    )
```

**2. The problem**

Thus, when I try the same pertaining code with a much larger corpus, it takes quite a long time to tokenize.

Also, we can choose to tokenize data in `data-collator`. In this way, the program only tokenizes one batch in the next training step and avoids getting stuck in tokenization.

**3. My question**

As described above, my questions are:
* **Which is better? Process in `map()` or in `data-collator`**
* **Why huggingface advises `map()` function?** There should be some advantages to using `map()`

Thanks for your answers!


[Tests] Ensure `datasets` supports renamed repositories

good second issue

Issues a bit more difficult than "Good First" issues

You could also switch to using `huggingface_hub` more directly, where such a guarantee is already tested =)

cc @Wauplin

On https://hf.co/datasets you can rename a dataset (or sometimes move it to another user/org). The website handles redirections correctly and AFAIK `datasets` does as well.

However it would be nice to have an integration test to make sure we don't break support for renamed datasets.

To implement this we can use the /api/repos/move endpoint on hub-ci to rename/move a repo (it is documented at https://huggingface.co/docs/hub/api)


Identical keywords in build_kwargs and config_kwargs lead to TypeError in load_dataset_builder()

bug

Something isn't working

good first issue

Good for newcomers

I am getting similar error - `TypeError: type object got multiple values for keyword argument 'name'` while following this [tutorial](https://huggingface.co/docs/datasets/dataset_script#create-a-dataset-loading-script). I am getting this error with the `dataset-cli test` command.

`datasets` version: 2.4.0

In my case, this was happening because I defined multiple `BuilderConfig` for multiple types, but didn't had all the data files that are requierd by those configs.

I think this is different than the original issue by @bablf .

Hi ! I think this can be fixed by letting the config_kwargs take over the builder kwargs here:

https://github.com/huggingface/datasets/blob/7feeb5648a63b6135a8259dedc3b1e19185ee4c7/src/datasets/load.py#L1533-L1534

maybe something like this ?
```python
    **{**builder_kwargs, **config_kwargs}
```

Let me know if you'd like to contribute and fix this bug, so I can assign you :)

> In my case, this was happening because I defined multiple BuilderConfig for multiple types, but didn't had all the data files that are requierd by those configs.
>
> I think this is different than the original issue by @bablf .

Feel free to to open an new issue, I'd be happy to help

@lhoestq Yeah, I want to, please assign.

Cool thank you ! Let me know if you have questions or if I can help

@lhoestq On second thoughts, I think this might be expected behavior; although a better error message might help.

Reasoning: Given n configs, if no data file is provided for any config, then it should be an error. Then why it should not be the case if out of n configs, for some data files are provided but not for others. Also, I was using `--all_configs` flag with `dataset-cli test`.

Ok I see - maybe we should check the values of builder_kwargs raise an error if any key in config_kwargs tries to overwrite it ? The builder kwargs are determined from the builder's type and location (in some cases it forces the base_path, data_files and config name for example)

## Describe the bug
In `load_dataset_builder()`, `build_kwargs` and `config_kwargs` can contain the same keywords leading to a TypeError("type object got multiple values for keyword argument "xyz").

I ran into this problem with the keyword: `base_path`. It might happen with other kwargs as well. I think a quickfix would be
```python
builder_cls = import_main_class(dataset_module.module_path)
builder_kwargs = dataset_module.builder_kwargs
data_files = builder_kwargs.pop("data_files", data_files)
config_name = builder_kwargs.pop("config_name", name)
hash = builder_kwargs.pop("hash")
base_path = builder_kwargs.pop("base_path")
```
and then pass base_path into `builder_cls`.

## Steps to reproduce the bug
```python
from datasets import load_dataset
load_dataset("rotten_tomatoes", base_path="./sample_data")
```

## Expected results
The docs state: `**config_kwargs` — Keyword arguments to be passed to the [BuilderConfig](https://huggingface.co/docs/datasets/v2.4.0/en/package_reference/builder_classes#datasets.BuilderConfig) and used in the [DatasetBuilder](https://huggingface.co/docs/datasets/v2.4.0/en/package_reference/builder_classes#datasets.DatasetBuilder).

So I would expect to be able to pass the base_path into `load_dataset()`.
## Actual results
TypeError("type object got multiple values for keyword argument "base_path").

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: macOS-12.5-arm64-arm-64bit
- Python version: 3.8.9
- PyArrow version: 9.0.0


Update GLUE evaluation metadata

_The documentation is not available anymore as the PR was closed or merged._

This PR updates the evaluation metadata for GLUE to:

* Include defaults for all configs except `ax` (which only has a `test` split with no known labels)
* Fix the default split from `test` to `validation` since `test` splits in GLUE have no labels (they're private)
* Fix the `task_id` for some existing defaults

cc @sashavor @douwekiela


Fix missing tags in dataset cards

Fix missing tags in dataset cards:
- asnq
- clue
- common_gen
- cosmos_qa
- guardian_authorship
- hindi_discourse
- py_ast
- x_stance

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896


None Type error for swda datasets

bug

Something isn't working

Thanks for reporting @hannan72 ! I couldn't reproduce the error on my side, can you share the full stack trace please ?

Thanks a lot for your response @lhoestq
The problem is solved accidentally today and I don't know exactly why it was happened yesterday.
The issue can be closed.

Ok, let us know if you encounter the issue again ;)

## Describe the bug
I got `'NoneType' object is not callable` error while calling the swda datasets.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("swda")
```

## Expected results
Run without error

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Python version: 3.8.10