Remove deprecated `remove_columns` param in `filter`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3827). All of your documentation changes will be reflected on that endpoint.

A leftover from #3803.


Add IterableDataset.filter

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3826). All of your documentation changes will be reflected on that endpoint.

Indeed ! If `batch_size` is `None` or `<=0` then the full dataset should be passed. It's been mentioned in the docs for a while but never actually implemented. We can fix that later

_Needs https://github.com/huggingface/datasets/pull/3801 to be merged first_

I added `IterableDataset.filter` with an API that is a subset of `Dataset.filter`:
```python
def filter(self, function, batched=False, batch_size=1000, with_indices=false, input_columns=None):
```

TODO:
- [x] tests
- [x] docs

related to https://github.com/huggingface/datasets/issues/3444 and https://github.com/huggingface/datasets/issues/3753


Update version and date in Wikipedia dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3825). All of your documentation changes will be reflected on that endpoint.

CC: @geohci


Allow not specifying feature cols other than `predictions`/`references` in `Metric.compute`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3824). All of your documentation changes will be reflected on that endpoint.

Fix #3818


500 internal server error when trying to open a dataset composed of Zarr stores

bug

Something isn't working

Hi @jacobbieker, thanks for reporting!

I have transferred this issue to our Hub team and they are investigating it. I keep you informed.

Hi @jacobbieker, we are investigating this issue on our side and we'll see if we can fix it, but please note that your repo is considered problematic for git. Here are the results of running https://github.com/github/git-sizer on it:

```
Processing blobs: 147448
Processing trees: 27
Processing commits: 4
Matching commits to trees: 4
Processing annotated tags: 0
Processing references: 3
| Name                         | Value     | Level of concern               |
| ---------------------------- | --------- | ------------------------------ |
| Biggest objects              |           |                                |
| * Trees                      |           |                                |
|   * Maximum entries      [1] |   167 k   | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! |
|                              |           |                                |
| Biggest checkouts            |           |                                |
| * Number of files        [2] |   189 k   | ***                            |

[1]  aa057d2667c34c70c6146efc631f5c9917ff326e (refs/heads/main:2016.zarr/unknown)
[2]  6897b7bf6440fdd16b2c39d08085a669e7eaa59d (refs/heads/main^{tree})
```

You can check https://github.com/github/git-sizer for more information on how to avoid such pathological structures.

Hi, thanks for getting back to me so quick! And yeah, I figured that was probably the problem. I was going to try to delete the repo, but couldn't through the website, so if that's the easiest way to solve it, I can regenerate the dataset in a different format with less tiny files, and you guys can delete the repo as it is. Zarr just saves everything as lots of small files to make chunks easy to load, which is why I was preferring that format, but maybne that just doesn't work well for HF datasets.

Hi @jacobbieker,

For future use cases, our Hub team is still pondering whether to limit the maximum number of files per repo to avoid technical issues...

On the meantime, they have made a fix and your dataset is working: https://huggingface.co/datasets/openclimatefix/mrms

## Describe the bug
The dataset [openclimatefix/mrms](https://huggingface.co/datasets/openclimatefix/mrms) gives a 500 server error when trying to open it on the website, or through code.

The dataset doesn't have a loading script yet, and I did push two [xarray](https://docs.xarray.dev/en/stable/) Zarr stores of data there recentlyish. The Zarr stores are composed of lots of small files, which I am guessing is probably the problem, as we have another [OCF dataset](https://huggingface.co/datasets/openclimatefix/eumetsat_uk_hrv) using xarray and Zarr, but with the Zarr stored on GCP public datasets instead of directly in HF datasets, and that one opens fine.

In general, we were hoping to use HF datasets to release some more public geospatial datasets as benchmarks, which are commonly stored as Zarr stores as they can be compressed well and deal with the multi-dimensional data and coordinates fairly easily compared to other formats, but with this error, I'm assuming we should try a different format?

For context, we are trying to have complete public model+data reimplementations of some SOTA weather and solar nowcasting models, like [MetNet, MetNet-2,](https://github.com/openclimatefix/metnet) [DGMR](https://github.com/openclimatefix/skillful_nowcasting), and [others](https://github.com/openclimatefix/graph_weather), which all have large, complex datasets.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("openclimatefix/mrms")
```

## Expected results
The dataset should be downloaded or open up

## Actual results
A 500 internal server error

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.15.25-1-MANJARO-x86_64-with-glibc2.35
- Python version: 3.9.10
- PyArrow version: 7.0.0


Add Biwi Kinect Head Pose Database

dataset request

Requesting to add a new dataset

vision

Vision datasets

Official dataset location : https://icu.ee.ethz.ch/research/datsets.html
In the "Biwi Kinect Head Pose Database" section, I do not find any information regarding "Downloading the dataset." . Do we mail the authors regarding this ?

I found the dataset on Kaggle : [Link](https://www.kaggle.com/kmader/biwi-kinect-head-pose-database) , but since 🤗 does not host any of the datasets, this would require the user to provide their Kaggle username and API key to download.

Any inputs on how we could proceed ? Thank you.
[ Need your inputs here, @lhoestq or @mariosasko ]

Hi @dnaveenr! Thanks for tackling this issue. This link should work: https://data.vision.ee.ethz.ch/cvl/gfanelli/kinect_head_pose_db.tgz

Added in https://github.com/huggingface/datasets/pull/3903, thanks @dnaveenr !

## Adding a Dataset
- **Name:** Biwi Kinect Head Pose Database
- **Description:** Over 15K images of 20 people recorded with a Kinect while turning their heads around freely. For each frame, depth and rgb images are provided, together with ground in the form of the 3D location of the head and its rotation angles.
- **Data:** [*link to the Github repository or current dataset location*](https://icu.ee.ethz.ch/research/datsets.html)
- **Motivation:** Useful pose estimation dataset

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Update Wikipedia dataset

_The documentation is not available anymore as the PR was closed or merged._

I'm starting to generate the pre-processed data for some of the languages (for backward compatibility).

Once this merged, we will create the pre-processed data on the Hub under the Wikimedia namespace.

All steps have been properly done.

I'm merging all these commits into master.

This PR combines all updates to Wikipedia dataset.

Once approved, this will be used to generate the pre-processed Wikipedia datasets.

Finally, this PR will be able to be merged into master:
- NOT using squash
- BUT a regular MERGE (or REBASE+MERGE), so that all commits are preserved

TODO:
- [x] #3435
- [x] #3789
- [x] #3825
- [x] Run to get the pre-processed data for big languages (backward compatibility)
- [x] #3958

CC: @geohci


`pubmed_qa` checksum mismatch

bug

Something isn't working

duplicate

This issue or pull request already exists

Hi @jon-tow, thanks for reporting.

This issue was already reported and its root cause is a change in the Google Drive service. See:
- #3786

We have already fixed it. See:
- #3787

We are planning to make a patch release today.

In the meantime, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```

## Describe the bug
Loading [`pubmed_qa`](https://huggingface.co/datasets/pubmed_qa) results in a mismatched checksum error.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
import datasets
try:
    datasets.load_dataset("pubmed_qa", "pqa_labeled")
except Exception as e:
    print(e)

try:
    datasets.load_dataset("pubmed_qa", "pqa_unlabeled")
except Exception as e:
    print(e)

try:
    datasets.load_dataset("pubmed_qa", "pqa_artificial")
except Exception as e:
    print(e)
```

## Expected results
Successful download.

## Actual results
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/datasets/load.py", line 1702, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.9/site-packages/datasets/builder.py", line 594, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.9/site-packages/datasets/builder.py", line 665, in _download_and_prepare
    verify_checksums(
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1RsGLINVce-0GsDkCLDuLZmoLuzfmoCuQ', 'https://drive.google.com/uc?export=download&id=15v1x6aQDlZymaHGP7cZJZZYFfeJt2NdS']

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: macOS
- Python version: 3.8.1
- PyArrow version: 3.0.0


Fix typo in doc build yml

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3819). All of your documentation changes will be reflected on that endpoint.

cc: @lhoestq


Support for "sources" parameter in the add() and add_batch() methods in datasets.metric - SARI

enhancement

New feature or request

Hi, thanks for reporting! We can add a `sources: datasets.Value("string")` feature to the `Features` dict in the `SARI` script to fix this. Would you be interested in submitting a PR?

Hi Mario,

Thanks for your message. I did try to add `sources` into the `Features` dict using a script for the metric:
```
            features=datasets.Features(
                {
                    "sources": datasets.Value("string", id="sequence"),
                    "predictions": datasets.Value("string", id="sequence"),
                    "references": datasets.Sequence(datasets.Value("string", id="sequence"), id="references"),
                }
            ),
```

But that only avoids a failure in `encode_batch` in the `add_batch` method:
```
        batch = {"predictions": predictions, "references": references}
        batch = self.info.features.encode_batch(batch)
```

The real problem is that `add_batch()`, `add()` and `compute()` does not receive a `sources` param:
```
def add_batch(self, *, predictions=None, references=None):
def add(self, *, prediction=None, reference=None):
def compute(self, *, predictions=None, references=None, **kwargs)
```

And then, it fails:
`TypeError: add_batch() got an unexpected keyword argument sources`

I need this for adding any metric based on SARI or alike, not only for sari.py :)

Let me know if I understood correctly the proposed solution.

The `Metric` class has been modified recently to support this use-case, but the `add_batch` + `compute` pattern still doesn't work correctly. I'll open a PR.

**Is your feature request related to a problem? Please describe.**
The methods `add_batch` and `add` from the `Metric` [class](https://github.com/huggingface/datasets/blob/1675ad6a958435b675a849eafa8a7f10fe0f43bc/src/datasets/metric.py) does not work with [SARI](https://github.com/huggingface/datasets/blob/master/metrics/sari/sari.py) metric. This metric not only relies on the predictions and references, but also in the input.

For example, when the `add_batch` method is used, then the `compute()` method fails:
```
metric = load_metric("sari")
metric.add_batch(
    predictions=["About 95 you now get in ."],
    references=[["About 95 species are currently known .","About 95 species are now accepted .","95 species are now accepted ."]])
metric.compute()
> TypeError: _compute() missing 1 required positional argument: 'sources'
```

Therefore, the `compute() `method can only be used standalone:
```
metric = load_metric("sari")
result = metric.compute(
    sources=["About 95 species are currently accepted ."],
    predictions=["About 95 you now get in ."],
    references=[["About 95 species are currently known .","About 95 species are now accepted .","95 species are now accepted ."]])
> {'sari': 26.953601953601954}
```

**Describe the solution you'd like**
Support for an additional parameter `sources` in the `add_batch` and `add` of the `Metric` class.
```
add_batch(*, sources=None, predictions=None, references=None, **kwargs)
add(*, sources=None, predictions=None, references=None, **kwargs)
compute()
```

**Describe alternatives you've considered**
I've tried to override the `add_batch` and `add`, however, these are highly dependent to the `Metric` class. We could also write a simple function that compute the scores of a sentences list, but then we lose the functionality from the original [add](https://huggingface.co/docs/datasets/_modules/datasets/metric.html#Metric.add) and [add_batch method](https://huggingface.co/docs/datasets/_modules/datasets/metric.html#Metric.add_batch).

**Additional context**
These methods are used in the transformers [pytorch examples](https://github.com/huggingface/transformers/blob/master/examples/pytorch/summarization/run_summarization_no_trainer.py).


Simplify Common Voice code

I think the script looks pretty clean and readable now! cool!

In #3736 we introduced one method to generate examples when streaming, that is different from the one when not streaming.

In this PR I propose a new implementation which is simpler: it only has one function, based on `iter_archive`. And you still have access to local audio files when loading the dataset in non-streaming mode.

cc @patrickvonplaten @polinaeterna @anton-l @albertvillanova since this will become the template for many audio datasets to come.

This change can also trivially be applied to the other audio datasets that already exist. Using this line, you can get access to local files in non-streaming mode:
```python
local_extracted_archive = dl_manager.extract(archive_path) if not dl_manager.is_streaming else None
```


Doc new UI test workflows2

<img src="https://www.bikevillastravel.com/cms/static/images/loading.gif" alt="Girl in a jacket" width="50" >


Fix iter_archive getting reset

The `DownloadManager.iter_archive` method currently returns an iterator - which is  **empty** once you iter over it once. This means you can't pass the same archive iterator to several splits.

To fix that, I changed the ouput of `DownloadManager.iter_archive` to be an iterable that you can iterate over several times, instead of a one-time-use iterator.

The `StreamingDownloadManager.iter_archive` already returns an appropriate iterable, and the code added in this PR is inspired from the one in `streaming_download_manager.py`


Handle Nones in PyArrow struct

Looks like I added my comments while you were editing - sorry about that

This PR fixes an issue introduced by #3575 where `None` values stored in PyArrow arrays/structs would get ignored by `cast_storage` or by the `pa.array(cast_to_python_objects(..))` pattern. To fix the former, it also bumps the minimal PyArrow version to v5.0.0 to use the `mask` param in `pa.SturctArray`.


Add MetaShift dataset

dataset request

Requesting to add a new dataset

vision

Vision datasets

I would like to take this up and give it a shot. Any image specific - dataset guidelines to keep in mind ? Thank you.

I've started working on adding this dataset. I require some inputs on the following :

Ref for the initial draft [here](https://github.com/dnaveenr/datasets/blob/add_metashift_dataset/datasets/metashift/metashift.py)
1. The dataset does not have a typical - train/test/val split. What do we do for the _split_generators() function ? How do we go about this ?
2. This dataset builds on the Visual Genome dataset, using a metadata file. The dataset is generated using generate_full_MetaShift.py script. By default, the authors choose to generate the dataset only for a SELECTED_CLASSES. The following script is used :
Code : https://github.com/Weixin-Liang/MetaShift/blob/main/dataset/generate_full_MetaShift.py
Info : https://metashift.readthedocs.io/en/latest/sub_pages/download_MetaShift.html#generate-the-full-metashift-dataset
Can I just copy over the required functions into the metashift.py to generate the dataset ?
3. How do we complete the _generate_examples for this dataset ?

The user has the ability to use default selected classes, get the complete dataset or add more specific additional classes. I think config would be a good option here.

Inputs, suggestions would be helpful. Thank you.

I think @mariosasko and @lhoestq should be able to help here 😄

Hi ! Thanks for adding this dataset :) Let me answer your questions:

1. in this case you can put everything in the "train" split
2. Yes you can copy the script (provided you also include the MIT license of the code in the file header for example). Though we ideally try to not create new directories nor files when generating dataset, so if possible this script should be adapted to not create the file structure they mentioned, but instead yield the images one by one in `_generate_examples`. Let me know if you think this is feasible
3. see point 2 haha

> The user has the ability to use default selected classes, get the complete dataset or add more specific additional classes. I think config would be a good option here.

Yup ! We can also define a `selected_classes` parameter such that users can do
```python
load_dataset("metashift", selected_classes=["cat", "dog", ...])
```

Great. This is helpful. Thanks @lhoestq .
Regarding Point 2, I'll try using yield instead of creating the directories and see if its feasible. selected_classes config sounds good.

## Adding a Dataset
- **Name:** MetaShift
- **Description:**  collection of 12,868 sets of natural images across 410 classes-
- **Paper:** https://arxiv.org/abs/2202.06523v1
- **Data:** https://github.com/weixin-liang/metashift

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


benchmark streaming speed with tar vs zip archives

I'm closing the PR since we're not going to merge it

# do not merge
## Hypothesis
packing data into a single zip archive could allow us not to care about splitting data into several tar archives for efficient streaming which is annoying (since data creators usually host the data in a single tar)

## Data

I host it [here](https://huggingface.co/datasets/polinaeterna/benchmark_dataset/)

## I checked three configurations:
1. All data in one zip archive, streaming only those files that exist in split metadata file (we can access them directrly with no need to iterate over full archive), see [this func](https://github.com/huggingface/datasets/compare/master...polinaeterna:benchmark-tar-zip?expand=1#diff-4f5200d4586aec5b2a89fcf34441c5f92156f9e9d408acc7e50666f9a1921ddcR196)
2.  All data in three splits, the standart way to make streaming efficient, see [this func](https://github.com/huggingface/datasets/compare/master...polinaeterna:benchmark-tar-zip?expand=1#diff-4f5200d4586aec5b2a89fcf34441c5f92156f9e9d408acc7e50666f9a1921ddcR174)
3. All data in single tar, iterate over the full archive and take only files existing in split metadata file, see [this func](https://github.com/huggingface/datasets/compare/master...polinaeterna:benchmark-tar-zip?expand=1#diff-4f5200d4586aec5b2a89fcf34441c5f92156f9e9d408acc7e50666f9a1921ddcR150)

## Results
1. one zip
![image](https://user-images.githubusercontent.com/16348744/156567611-e3652087-7147-4cf0-9047-9cbc00ec71f5.png)
2. three tars
![image](https://user-images.githubusercontent.com/16348744/156567688-2a462107-f83e-4722-8ea3-71a13b56c998.png)
3. one tar
![image](https://user-images.githubusercontent.com/16348744/156567772-1bceb5f7-e7d9-4fa3-b31b-17fec5f9a5a7.png)

didn't check on the full data as it's time consuming but anyway it's pretty obvious that one-zip-way is not a good idea. here it's even worse than full iteration over tar containing all three splits (but that would depend on the case).


Update dev doc gh workflows

Reflect changes from https://github.com/huggingface/transformers/pull/15891


Update version of xcopa dataset

Note that there was a version update of the `xcopa` dataset: https://github.com/cambridgeltl/xcopa/releases

We updated our loading script, but we did not bump a new version number:
- #3254

This PR updates our loading script version from `1.0.0` to `1.1.0`.


Checksums didn't match for datasets on Google Drive

bug

Something isn't working

Hi @muelletm, thanks for reporting.

This issue was already reported and its root cause is a change in the Google Drive service. See:
- #3786

We have already fixed it. See:
- #3787

Until our next `datasets` library release, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```

## Describe the bug

Datasets hosted on Google Drive do not seem to work right now.
Loading them fails with a checksum error.

## Steps to reproduce the bug
```python
from datasets import load_dataset
for dataset in ["head_qa", "yelp_review_full"]:
  try:
    load_dataset(dataset)
  except Exception as exception:
    print("Error", dataset, exception)
```

Here is a [colab](https://colab.research.google.com/drive/1wOtHBmL8I65NmUYakzPV5zhVCtHhi7uQ#scrollTo=cDzdCLlk-Bo4).

## Expected results

The datasets should be loaded.

## Actual results

```
Downloading and preparing dataset head_qa/es (download: 75.69 MiB, generated: 2.86 MiB, post-processed: Unknown size, total: 78.55 MiB) to /root/.cache/huggingface/datasets/head_qa/es/1.1.0/583ab408e8baf54aab378c93715fadc4d8aa51b393e27c3484a877e2ac0278e9...
Error head_qa Checksums didn't match for dataset source files:
['https://drive.google.com/u/0/uc?export=download&id=1a_95N5zQQoUCq8IBNVZgziHbeM-QxG2t']
Downloading and preparing dataset yelp_review_full/yelp_review_full (download: 187.06 MiB, generated: 496.94 MiB, post-processed: Unknown size, total: 684.00 MiB) to /root/.cache/huggingface/datasets/yelp_review_full/yelp_review_full/1.0.0/13c31a618ba62568ec8572a222a283dfc29a6517776a3ac5945fb508877dde43...
Error yelp_review_full Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=0Bz8a_Dbh9QhbZlU4dXhHTFhZQU0']
```

## Environment info
- `datasets` version: 1.18.3
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 6.0.1


Pre-Processing Cache Fails when using a Factory pattern

Something isn't working

Ok - this is still an issue but I believe the root cause is different than I originally thought. I'm now able to get caching to work consistently with the above example as long as I fix the python hash seed `export PYTHONHASHSEED=1234`

Hi!

Yes, our hasher should work with decorators. For instance, this dummy example:
```python
def f(arg):
    def f1(ex):
        return {"a": ex["col1"] + arg}
    return f1
```
gives the same hash across different Python sessions (`datasets.fingerprint.Hasher.hash(f("string1")` returns `"408c9059f89dbd6c"` on my machine).

Could you please make the example self-contained? This way, we can reproduce the bug. Additionally, you can try to find the problematic object yourself by testing their hash with `datasets.fingerprint.Hasher.hash(obj)`

This could be related to https://github.com/huggingface/datasets/issues/3638.

## Describe the bug
If you utilize a pre-processing function which is created using a factory pattern, the function hash changes on each run (even if the function is identical) and therefore the data will be reproduced each time.

## Steps to reproduce the bug
```python
    def preprocess_function_factory(augmentation=None):
        def preprocess_function(examples):
            # Tokenize the texts
            if augmentation:
                conversions1 = [
                    augmentation(example)
                    for example in examples[sentence1_key]
                ]
                if sentence2_key is None:
                    args = (conversions1,)
                else:
                    conversions2 = [
                        augmentation(example)
                        for example in examples[sentence2_key]
                    ]
                    args = (conversions1, conversions2)
            else:
                args = (
                    (examples[sentence1_key],)
                    if sentence2_key is None
                    else (examples[sentence1_key], examples[sentence2_key])
                )
            result = tokenizer(
                *args, padding=padding, max_length=max_seq_length, truncation=True
            )

            # Map labels to IDs (not necessary for GLUE tasks)
            if label_to_id is not None and "label" in examples:
                result["label"] = [
                    (label_to_id[l] if l != -1 else -1) for l in examples["label"]
                ]
            return result

        return preprocess_function

capitalize = lambda x: x.capitalize()
preprocess_function = preprocess_function_factory(augmentation=capitalize)
print(hash(preprocess_function)) # This will change on each run
raw_datasets = raw_datasets.map(
        preprocess_function,
        batched=True,
        load_from_cache_file=True,
        desc="Running transformation and tokenizer on dataset",
)
```

## Expected results
Running the code twice will cause the cache to be re-used.

## Actual results
Running the code twice causes the whole dataset to be re-processed


NonMatchingChecksumError in xcopa dataset

Something isn't working

@albertvillanova here's a separate issue for a bug similar to #3792

Hi @afcruzs-ms, thanks for opening this separate issue for your problem.

The root problem in the other issue (#3792) was a change in the service of Google Drive.

But in your case, the `xcopa` dataset is not hosted on Google Drive. Therefore, the root cause should be a different one.

Let me look at it...

@afcruzs-ms, I'm not able to reproduce the issue you reported:
```python
In [1]: from datasets import load_dataset
   ...: dataset = load_dataset("xcopa", "it")
Downloading builder script: 5.21kB [00:00, 2.75MB/s]
Downloading metadata: 28.6kB [00:00, 14.5MB/s]
Downloading and preparing dataset xcopa/it (download: 627.09 KiB, generated: 76.43 KiB, post-processed: Unknown size, total: 703.52 KiB) to .../.cache/huggingface/datasets/xcopa/it/1.0.0/e1fab65f984b24c8b66bcf7ac27a26a1182f84adfb2e74035861be65e214b9e6...
Downloading data: 642kB [00:00, 5.42MB/s]
Dataset xcopa downloaded and prepared to .../.cache/huggingface/datasets/xcopa/it/1.0.0/e1fab65f984b24c8b66bcf7ac27a26a1182f84adfb2e74035861be65e214b9e6. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 733.27it/s]

In [2]: dataset
Out[2]:
DatasetDict({
    test: Dataset({
        features: ['premise', 'choice1', 'choice2', 'question', 'label', 'idx', 'changed'],
        num_rows: 500
    })
    validation: Dataset({
        features: ['premise', 'choice1', 'choice2', 'question', 'label', 'idx', 'changed'],
        num_rows: 100
    })
})
```

Maybe you have some issue with your cached data... Could you please try to force the redownload of the data?
```python
dataset = load_dataset("xcopa", "it", download_mode="force_redownload")
```

unfortunately, i am having a similar problem with the irc_disentaglement dataset :/
my code:
```
from datasets import load_dataset

dataset = load_dataset("irc_disentangle", download_mode="force_redownload")
```

however, it produces the same error as @afcruzs-ms
```
[38](file:///Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/datasets/utils/info_utils.py?line=37) if len(bad_urls) > 0:
     [39](file:///Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/datasets/utils/info_utils.py?line=38)     error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> [40](file:///Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/datasets/utils/info_utils.py?line=39)     raise NonMatchingChecksumError(error_msg + str(bad_urls))
     [41](file:///Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/datasets/utils/info_utils.py?line=40) logger.info("All the checksums matched successfully" + for_verification_name)

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://github.com/jkkummerfeld/irc-disentanglement/tarball/master']
```

I attempted to use the `ignore_verifications' as such:
```
ds = datasets.load_dataset('irc_disentangle', download_mode="force_redownload", ignore_verifications=True)

```
```
Downloading builder script: 12.0kB [00:00, 5.92MB/s]
Downloading metadata: 7.58kB [00:00, 3.48MB/s]
No config specified, defaulting to: irc_disentangle/ubuntu
Downloading and preparing dataset irc_disentangle/ubuntu (download: 112.98 MiB, generated: 60.05 MiB, post-processed: Unknown size, total: 173.03 MiB) to /Users/laylabouzoubaa/.cache/huggingface/datasets/irc_disentangle/ubuntu/1.0.0/0f24ab262a21d8c1d989fa53ed20caa928f5880be26c162bfbc02445dbade7e5...
Downloading data: 118MB [00:09, 12.1MB/s]

Dataset irc_disentangle downloaded and prepared to /Users/laylabouzoubaa/.cache/huggingface/datasets/irc_disentangle/ubuntu/1.0.0/0f24ab262a21d8c1d989fa53ed20caa928f5880be26c162bfbc02445dbade7e5. Subsequent calls will reuse this data.
100%|██████████| 3/3 [00:00<00:00, 675.38it/s]
```
but, this returns an empty set?

```
DatasetDict({
    train: Dataset({
        features: ['id', 'raw', 'ascii', 'tokenized', 'date', 'connections'],
        num_rows: 0
    })
    test: Dataset({
        features: ['id', 'raw', 'ascii', 'tokenized', 'date', 'connections'],
        num_rows: 0
    })
    validation: Dataset({
        features: ['id', 'raw', 'ascii', 'tokenized', 'date', 'connections'],
        num_rows: 0
    })
})
```

not sure what else to try at this point?
Thanks in advanced🤗

Thanks @labouz for reporting: yes, better opening a new GitHub issue as you did. I'm addressing it:
- #4376

## Describe the bug
Loading the xcopa dataset doesn't work, it fails due to a mismatch in the checksum.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("xcopa", "it")
```

## Expected results
The dataset should be loaded correctly.

## Actual results
Fails with:
```python
in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     38     if len(bad_urls) > 0:
     39         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 40         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     41     logger.info("All the checksums matched successfully" + for_verification_name)
     42

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://github.com/cambridgeltl/xcopa/archive/master.zip']
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3, and 1.18.4.dev0
- Platform:
- Python version: 3.8
- PyArrow version:


Fix Spanish data file URL in wiki_lingua dataset

This PR fixes the URL for Spanish data file.

Previously, Spanish had the same URL as Vietnamese data file.


Remove decode: true for image feature in head_qa

This was erroneously added in https://github.com/huggingface/datasets/commit/701f128de2594e8dc06c0b0427c0ba1e08be3054. This PR removes it.


Text builder with custom separator line boundaries

enhancement

New feature or request

Hi ! Interresting :)

Could you give more details on what kind of separators you would like to use instead ?

In my case, I just want to use `\n` but not `U+2028`.

Ok I see, maybe there can be a `sep` parameter to allow users to specify what line/paragraph separator they'd like to use

Thanks for requesting this enhancement. We have recently found a somehow related issue with another dataset:
- #3704

Let me make a PR proposal.

**Is your feature request related to a problem? Please describe.**
The current [Text](https://github.com/huggingface/datasets/blob/207be676bffe9d164740a41a883af6125edef135/src/datasets/packaged_modules/text/text.py#L23) builder implementation splits texts with `splitlines()` which splits the text on several line boundaries. Not all of them are always wanted.

**Describe the solution you'd like**
```python
                if self.config.sample_by == "line":
                    batch_idx = 0
                    while True:
                        batch = f.read(self.config.chunksize)
                        if not batch:
                            break
                        batch += f.readline()  # finish current line
                        if self.config.custom_newline is None:
                            batch = batch.splitlines(keepends=self.config.keep_linebreaks)
                        else:
                            batch = batch.split(self.config.custom_newline)[:-1]
                        pa_table = pa.Table.from_arrays([pa.array(batch)], schema=schema)
                        # Uncomment for debugging (will print the Arrow table size and elements)
                        # logger.warning(f"pa_table: {pa_table} num rows: {pa_table.num_rows}")
                        # logger.warning('\n'.join(str(pa_table.slice(i, 1).to_pydict()) for i in range(pa_table.num_rows)))
                        yield (file_idx, batch_idx), pa_table
                        batch_idx += 1
```

**A clear and concise description of what you want to happen.**
Creating the dataset rows with a subset of the `splitlines()` line boundaries.


Remove deprecated methods/params (preparation for v2.0)

This PR removes the following deprecated methos/params:
* `Dataset.cast_`/`DatasetDict.cast_`
* `Dataset.dictionary_encode_column_`/`DatasetDict.dictionary_encode_column_`
* `Dataset.remove_columns_`/`DatasetDict.remove_columns_`
* `Dataset.rename_columns_`/`DatasetDict.rename_columns_`
* `prepare_module`
* param `script_version` in `load_dataset`/`load_metric`
* param `version` in `hf_github_url`


Release of FairLex dataset

This is awesome ! The dataset card and the dataset script look amazing :)

I wanted to ask you if you'd be interested to have this dataset under the namespace of you research group at https://huggingface.co/coastalcph ? If yes, then you can actually create a dataset repository under your research group name and upload the files from this PR there

Hi @lhoestq,

Yeah, I could do that. I see that people do that a lot of models, but not for datasets.

Is there any good reason to have it under the organization domain instead of the general domain?

 Thanks!

It's nice to have it under your namespace:
- it will appear on your research group page, along with your models
- you can edit or create datasets at any time - you don't need to open PRs on GitHub

All the datasets that are not under a namespace are this way because we started adding datasets from GitHub. Now we encourage users to upload them directly to make things simpler, and aligned with the workflow for models

(the documentation will be updated in the following days)

Note that we will keep accepting PRs here though when there is no clear namespace under which a dataset should be, or for users that want a review from us

Ok, I'll do that. So, I'll just have to upload all the files under the `/fairlex` directory in my PR, right?

Ok, I uploaded most of them from the UI environment (https://huggingface.co/datasets/coastalcph/fairlex). Can I possibly upload the dummy data as well from the UI environment. I really want to avoid the CLI right now 😄

Yea sure, feel free to use the UI of the website, even for the dummy data ^^

Did you upload them yourself? Because I see the data preview, and I'm pretty sure, I didn't do that 😄

The preview is computed from the real data ;)

The dummy data are used for testing only

Haha, ok I was shocked! Cool, I close this PR, then. Thanks, again!

**FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing**

We present a benchmark suite of four datasets for evaluating the fairness of pre-trained legal language models and the techniques used to fine-tune them for downstream tasks. Our benchmarks cover four jurisdictions (European Council, USA, Swiss, and Chinese), five languages (English, German, French, Italian, and Chinese), and fairness across five attributes (gender, age, nationality/region, language, and legal area). In our experiments, we evaluate pre-trained language models using several group-robust fine-tuning techniques and show that performance group disparities are vibrant in many cases, while none of these techniques guarantee fairness, nor consistently mitigate group disparities. Furthermore, we provide a quantitative and qualitative analysis of our results, highlighting open challenges in the development of robustness methods in legal NLP.

*Ilias Chalkidis, Tommaso Pasini, Sheng Zhang, Letizia Tomada, Letizia, Sebastian Felix Schwemer, Anders Søgaard. FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing. 2022. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Dublin, Ireland.*

Note: Please review this initial commit, and I'll update the publication link, once I'll have the ArXived version. Thanks!


[Breaking] Align `map` when streaming: update instead of overwrite + add missing parameters

Currently the datasets in streaming mode and in non-streaming mode have two distinct API for `map` processing.

In this PR I'm aligning the two by changing `map` in streamign mode. This includes a **major breaking change** and will require a major release of the library: **Datasets 2.0**

In particular, `Dataset.map` adds new columns (with dict.update) BUT `IterableDataset.map` used to discard previous columns (it overwrites the dict). In this PR I'm chaning the `IterableDataset.map` to behave the same way as `Dataset.map`: it will update the examples instead of overwriting them.

I'm also adding those missing parameters to streaming `map`: with_indices, input_columns, remove_columns

### TODO
- [x] tests
- [x] docs

Related to https://github.com/huggingface/datasets/issues/3444


Added computer vision tasks

Previous PR was in my fork so thought it'd be easier if I do it from a branch. Added computer vision task datasets according to HF tasks.


Xtreme-S Metrics

@lhoestq - if you could take a final review here this would be great (if you have 5min :-) )

Don't think the failures are related but not 100% sure

Yes the CI fail is unrelated - you can ignore it

**Added datasets (TODO)**:

- [x] MLS
- [x] Covost2
- [x] Minds-14
- [x] Voxpopuli
- [x] FLoRes (need data)

**Metrics**: Done


Fix error message in CSV loader for newer Pandas versions

Fix the error message in the CSV loader for `Pandas >= 1.4`. To fix this, I directly print the current file name in the for-loop. An alternative would be to use a check similar to this:
```python
csv_file_reader.handle.handle if datasets.config.PANDAS_VERSION >= version.parse("1.4") else csv_file_reader.f
```
CC: @SBrandeis


Reddit dataset card contribution

Description tags for webis-tldr-17 added.


Skip checksum computation if `ignore_verifications` is `True`

This will speed up the loading of the datasets where the number of data files is large (can easily happen with `imagefoler`, for instance)


can not flatten natural_questions dataset

bug

Something isn't working

same issue. downgrade it to a lower version.

Thanks for reporting, I'll take a look tomorrow :)

## Describe the bug
after downloading the natural_questions dataset, can not flatten the dataset considering there are `long answer` and `short answer` in `annotations`.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset('natural_questions',cache_dir = 'data/dataset_cache_dir')
dataset['train'].flatten()
```

## Expected results
a dataset with `long_answer` as features

## Actual results
Traceback (most recent call last):
  File "temp.py", line 5, in <module>
    dataset['train'].flatten()
  File "/Users/hannibal046/anaconda3/lib/python3.8/site-packages/datasets/fingerprint.py", line 413, in wrapper
    out = func(self, *args, **kwargs)
  File "/Users/hannibal046/anaconda3/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1296, in flatten
    dataset._data = update_metadata_with_features(dataset._data, dataset.features)
  File "/Users/hannibal046/anaconda3/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 536, in update_metadata_with_features
    features = Features({col_name: features[col_name] for col_name in table.column_names})
  File "/Users/hannibal046/anaconda3/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 536, in <dictcomp>
    features = Features({col_name: features[col_name] for col_name in table.column_names})
KeyError: 'annotations.long_answer'
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.13
- Platform: MBP
- Python version: 3.8
- PyArrow version: 6.0.1


Add Mahalanobis distance metric

Mahalanobis distance is a very useful metric to measure the distance from one datapoint X to a distribution P.
In this PR I implement the metric in a simple way with the help of numpy only.

Similar to the [MAUVE implementation](https://github.com/huggingface/datasets/blob/master/metrics/mauve/mauve.py), we can make this metric accept texts as input and encode them with a featurize model, if that is desirable.


Docs new UI actions no self hosted

It seems like the doc can't be compiled right now because of the following:

```
Traceback (most recent call last):
  File "/usr/local/bin/doc-builder", line 33, in <module>
    sys.exit(load_entry_point('doc-builder', 'console_scripts', 'doc-builder')())
  File "/__w/datasets/datasets/doc-builder/src/doc_builder/commands/doc_builder_cli.py", line 39, in main
    args.func(args)
  File "/__w/datasets/datasets/doc-builder/src/doc_builder/commands/build.py", line 95, in build_command
    build_doc(
  File "/__w/datasets/datasets/doc-builder/src/doc_builder/build_doc.py", line 361, in build_doc
    anchors_mapping = build_mdx_files(package, doc_folder, output_dir, page_info)
  File "/__w/datasets/datasets/doc-builder/src/doc_builder/build_doc.py", line 200, in build_mdx_files
    raise type(e)(f"There was an error when converting {file} to the MDX format.\n" + e.args[0]) from e
TypeError: There was an error when converting datasets/docs/source/package_reference/table_classes.mdx to the MDX format.
expected string or bytes-like object
```

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3793). All of your documentation changes will be reflected on that endpoint.

> It seems like the doc can't be compiled right now because of the following:

it is expected since there is something I need to change on doc-builder side.

> This is due to the injection of docstrings from PyArrow. I think I can fix that by moving all the docstrings and fix them manually.

@lhoestq I will let you know if we need to change it manually.

@LysandreJik thanks a lot for this PR! I only had one question [here](https://github.com/huggingface/datasets/pull/3793#discussion_r816100194)

> @lhoestq I will let you know if we need to change it manually.

It would be simpler to change it manually anyway - I don't want our documentation to break if PyArrow has documentation issues

For some reason it fails when `Installing node dependencies` when running `npm ci` from the `kit` directory, any idea why @mishig25 ?

It's very likely linked to an OOM error: https://github.com/huggingface/transformers/pull/15710#issuecomment-1051737337

Removes the need to have a self-hosted runner for the dev documentation


dataset-viewer

Related to the dataset viewer on huggingface.co

Same issue with `dataset = load_dataset("dbpedia_14")`
```
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=0Bz8a_Dbh9QhbQ2Vic1kxMmZZQ1k']

I think this is a side-effect of #3787. The checksums won't match because the URLs have changed. @rafikg @Y0mingZhang, while this is fixed, maybe you can load the datasets as such:

`data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]", ignore_verifications=True)`
`dataset = load_dataset("dbpedia_14", ignore_verifications=True)`

This will, most probably, skip the verifications and integrity checks listed [here](https://huggingface.co/docs/datasets/loading_datasets.html#integrity-verifications)

Hi! Installing the `datasets` package from master (`pip install git+https://github.com/huggingface/datasets.git`) and then redownloading the datasets with `download_mode` set to `force_redownload` (e.g. `dataset = load_dataset("dbpedia_14", download_mode="force_redownload")`) should fix the issue.

Hi @rafikg and @Y0mingZhang, thanks for reporting.

Indeed it seems that Google Drive changed their way to access their data files. We have recently handled that change:
- #3787

but it will be accessible to users only in our next release of the `datasets` version.
- Note that our latest release (version 1.18.3) was made before this fix: https://github.com/huggingface/datasets/releases/tag/1.18.3

In the meantime, as @mariosasko explained, you can incorporate this "fix" by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, you should force the redownload of the data (before the fix, you are just downloading/caching the virus scan warning page, instead of the data file):
```shell
data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]", download_mode="force_redownload")

@albertvillanova  by running:
```
pip install git+https://github.com/huggingface/datasets#egg=datasets
data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]", download_mode="force_redownload", ignore_verifications=True)
```

I had a pickle error **UnpicklingError: invalid load key, '<'** in this part of code  both `locally and on google colab`:

```
"""Yields examples."""
with open(filepath, "rb") as f:
         data = pickle.load(f)
for id_, row in enumerate(data.items()):
         yield id_, {"url": row[0], "article": self._process_article(row[1])}
```

This issue impacts many more datasets than the ones mention in this thread.  Can we post # of downloads for each dataset by day (by successes and failures)?  If so, it should be obvious which ones are failing.

I can see this problem too in xcopa, unfortunately installing the latest master (1.18.4.dev0) doesn't work, @albertvillanova .

```
from datasets import load_dataset
dataset = load_dataset("xcopa", "it")
```

Throws

```
in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     38     if len(bad_urls) > 0:
     39         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 40         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     41     logger.info("All the checksums matched successfully" + for_verification_name)
     42

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://github.com/cambridgeltl/xcopa/archive/master.zip']
```

Hi @rafikg, I think that is another different issue. Let me check it...

I guess maybe you are using a different Python version that the one the dataset owner used to create the pickle file...

@kwchurch the datasets impacted for this specific issue are the ones which are hosted at Google Drive.

@afcruzs-ms I think your issue is a different one, because that dataset is not hosted at Google Drive. Would you mind open another issue for that other problem, please? Thanks! :)

@albertvillanova just to let you know that I tried it locally and on colab and it is the same error

There are many many datasets on HugggingFace that are receiving this checksum error.  Some of these datasets are very popular.  There must be a way to track these errors, or to do regression testing.  We don't want to catch each of these errors on each dataset, one at a time.

@rafikg I am sorry, but I can't reproduce your issue. For me it works OK for all languages. See: https://colab.research.google.com/drive/1yIcLw1it118-TYE3ZlFmV7gJcsF6UCsH?usp=sharing

@kwchurch the PR #3787 fixes this issue (generated by a change in Google Drive service) for ALL datasets with this issue. Once we make our next library release (in a couple of days), the fix will be accessible to all users that update our library from PyPI.

By the way, @rafikg, I discovered the URL for Spanish was wrong. I've created a PR to fix it:
- #3806

I have the same problem with "wider_face" dataset. It seems that "load_dataset" function can not download the dataset from google drive.

still getting this issue with datasets==2.2.2 for
dataset_fever_original_dev = load_dataset('fever', "v1.0", split="labelled_dev")
(this one seems to be hosted by aws though)

update: also tried to install from source to get the latest 2.2.3.dev0, but still get the error below (and also force-redownloaded)

update2: Seems like this issues is linked to a change in the links in the specific fever datasets: https://fever.ai/
"28/04/2022
Dataset download URLs have changed
Download URLs for shared task data for FEVER, FEVER2.0 and FEVEROUS have been updated. New URLS begin with https://fever.ai/download/[task name]/[filename]. All resource pages have been updated with the new URLs. Previous dataset URLs may not work and should be updated if you require these in your scripts. "

=> I don't know how to update the links for HF datasets - would be great if someone could update them :)

```

Downloading and preparing dataset fever/v1.0 (download: 42.78 MiB, generated: 38.39 MiB, post-processed: Unknown size, total: 81.17 MiB) to /root/.cache/huggingface/datasets/fever/v1.0/1.0.0/956b0a9c4b05e126fd956be73e09da5710992b5c85c30f0e5e1c500bc6051d0a...

Downloading data files: 100%
6/6 [00:07<00:00, 1.21s/it]
Downloading data:
278/? [00:00<00:00, 2.34kB/s]
Downloading data:
278/? [00:00<00:00, 1.53kB/s]
Downloading data:
278/? [00:00<00:00, 7.43kB/s]
Downloading data:
278/? [00:00<00:00, 5.54kB/s]
Downloading data:
278/? [00:00<00:00, 6.19kB/s]
Downloading data:
278/? [00:00<00:00, 7.51kB/s]
Extracting data files: 100%
6/6 [00:00<00:00, 108.05it/s]

---------------------------------------------------------------------------

NonMatchingChecksumError                  Traceback (most recent call last)

[<ipython-input-20-92ec5c728ecf>](https://localhost:8080/#) in <module>()
     27 # get labels for fever-nli-dev from original fever - only works for dev
     28 # "(The labels for both dev and test are hidden but you can retrieve the label for dev using the cid and the original FEVER data.)"" https://github.com/easonnie/combine-FEVER-NSMN/blob/master/other_resources/nli_fever.md
---> 29 dataset_fever_original_dev = load_dataset('fever', "v1.0", split="labelled_dev")
     30 df_fever_original_dev = pd.DataFrame(data={"id": dataset_fever_original_dev["id"], "label": dataset_fever_original_dev["label"], "claim": dataset_fever_original_dev["claim"], "evidence_id": dataset_fever_original_dev["evidence_id"]})
     31 df_fever_dev = pd.merge(df_fever_dev, df_fever_original_dev, how="left", left_on="cid", right_on="id")

4 frames

[/usr/local/lib/python3.7/dist-packages/datasets/utils/info_utils.py](https://localhost:8080/#) in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     38     if len(bad_urls) > 0:
     39         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 40         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     41     logger.info("All the checksums matched successfully" + for_verification_name)
     42

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://s3-eu-west-1.amazonaws.com/fever.public/train.jsonl', 'https://s3-eu-west-1.amazonaws.com/fever.public/shared_task_dev.jsonl', 'https://s3-eu-west-1.amazonaws.com/fever.public/shared_task_dev_public.jsonl', 'https://s3-eu-west-1.amazonaws.com/fever.public/shared_task_test.jsonl', 'https://s3-eu-west-1.amazonaws.com/fever.public/paper_dev.jsonl', 'https://s3-eu-west-1.amazonaws.com/fever.public/paper_test.jsonl']
```

I think this has to be fixed on the google drive side, but you also have to delete the bad stuff from your local cache.  This is not a great design, but it is what it is.

We have fixed the issues with the datasets:
- wider_face: by hosting their data files on the HuggingFace Hub (CC: @HosseynGT)
- fever: by updating to their new data URLs (CC: @MoritzLaurer)

The yelp_review_full datasets has this problem as well and can't be fixed with the suggestion.

This is a super-common failure mode.  We really need to find a better workaround.  My solution was to wait until the owner of the dataset in question did the right thing, and then I had to delete my cached versions of the datasets with the bad checksums.  I don't understand why this happens.  Would it be possible to maintain a copy of the most recent version that was known to work, and roll back to that automatically if the checksums fail?  And if the checksums fail, couldn't the system automatically flush the cached versions with the bad checksums?  It feels like we are blaming the provider of the dataset, when in fact, there are things that the system could do to ease the pain.  Let's take these error messages seriously.  There are too many of them involving too many different datasets.

the [exams](https://huggingface.co/datasets/exams) dataset also has this issue and the provided fix above doesn't work

Same for [DART dataset](https://huggingface.co/datasets/dart):
```
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://raw.githubusercontent.com/Yale-LILY/dart/master/data/v1.1.1/dart-v1.1.1-full-train.json', 'https://raw.githubusercontent.com/Yale-LILY/dart/master/data/v1.1.1/dart-v1.1.1-full-dev.json', 'https://raw.githubusercontent.com/Yale-LILY/dart/master/data/v1.1.1/dart-v1.1.1-full-test.json']
```

- @thesofakillers the issue with `exams` was fixed on 16 Aug by this PR:
  - #4853
- @Aktsvigun the issue with `dart` has been transferred to the Hub: https://huggingface.co/datasets/dart/discussions/1
  - and fixed by PR: https://huggingface.co/datasets/dart/discussions/2
- @Carol-gutianle the issue with `multi_news` have been transferred to the Hub as well: https://huggingface.co/datasets/multi_news/discussions/1
  - not reproducible: maybe you should try to update `datasets`

For information to everybody, we are removing the checksum verifications (that were creating a bad user experience). This will be in place in the following weeks.

auto_gptq is required for real quantization
['/home/sam/Doctorproject/OmniQuant-main/main.py', '--model', '/home/sam/Doctorproject/OmniQuant-main/PATH/TO/LLaMA/llama-7b/', '--epochs', '20', '--output_dir', '/home/sam/Doctorproject/OmniQuant-main/outdir/llama-7b-w3a16/', '--eval_ppl', '--wbits', '3', '--abits', '16', '--lwc', '--net', 'llama-7b', '--aug_loss']
[2024-03-13 17:58:48 root](main.py 262): INFO Namespace(model='/home/sam/Doctorproject/OmniQuant-main/PATH/TO/LLaMA/llama-7b/', cache_dir='./cache', output_dir='/home/sam/Doctorproject/OmniQuant-main/outdir/llama-7b-w3a16/', save_dir=None, resume=None, real_quant=False, calib_dataset='wikitext2', nsamples=128, batch_size=1, seed=2, tasks='', eval_ppl=True, num_fewshot=0, wbits=3, abits=16, group_size=None, alpha=0.5, let_lr=0.005, lwc_lr=0.01, wd=0, epochs=20, let=False, lwc=True, aug_loss=True, symmetric=False, disable_zero_point=False, a_dynamic_method='per_token', w_dynamic_method='per_channel', limit=-1, multigpu=False, deactive_amp=False, attn_implementation='eager', net='llama-7b', act_scales=None, act_shifts=None)
Loading checkpoint shards:   0%|          | 0/33 [00:00<?, ?it/s]/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/torch/_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  return self.fget.__get__(instance, owner)()
Loading checkpoint shards: 100%|██████████| 33/33 [00:11<00:00,  2.98it/s]
vocab size:  32000
[2024-03-13 17:58:59 root](main.py 331): INFO === start quantization ===
get_wikitext2
[2024-03-13 18:02:20 datasets.load](load.py 1586): WARNING Using the latest cached version of the module from /home/sam/.cache/huggingface/modules/datasets_modules/datasets/wikitext/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126 (last modified on Wed Mar 13 16:54:26 2024) since it couldn't be found locally at wikitext, or remotely on the Hugging Face Hub.
Using the latest cached version of the module from /home/sam/.cache/huggingface/modules/datasets_modules/datasets/wikitext/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126 (last modified on Wed Mar 13 16:54:26 2024) since it couldn't be found locally at wikitext, or remotely on the Hugging Face Hub.
Downloading data: 243B [00:00, 877kB/s]
Generating test split:   0%|          | 0/4358 [00:00<?, ? examples/s]
Traceback (most recent call last):
  File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/builder.py", line 1742, in _prepare_split_single
    for key, record in generator:
  File "/home/sam/.cache/huggingface/modules/datasets_modules/datasets/wikitext/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126/wikitext.py", line 187, in _generate_examples
    with open(data_file, encoding="utf-8") as f:
  File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/streaming.py", line 75, in wrapper
    return function(*args, download_config=download_config, **kwargs)
  File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/download/streaming_download_manager.py", line 507, in xopen
    return open(main_hop, mode, *args, **kwargs)
NotADirectoryError: [Errno 20] Not a directory: '/home/sam/.cache/huggingface/datasets/downloads/94be2a7b3fff32ae7379658c8d3821035b666baddad3a06d29b55ab3a4ab3115/wikitext-2-raw/wiki.test.raw'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/sam/Doctorproject/OmniQuant-main/main.py", line 382, in <module>
    main()
  File "/home/sam/Doctorproject/OmniQuant-main/main.py", line 339, in main
    dataloader, _ = get_loaders(
  File "/home/sam/Doctorproject/OmniQuant-main/datautils.py", line 178, in get_loaders
    return get_wikitext2(nsamples, seed, seqlen, model)
  File "/home/sam/Doctorproject/OmniQuant-main/datautils.py", line 37, in get_wikitext2
    traindata = load_dataset(path='wikitext', name='wikitext-2-raw-v1', split='train', download_mode="force_redownload")
  File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/load.py", line 2598, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/builder.py", line 1021, in download_and_prepare
    self._download_and_prepare(
  File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/builder.py", line 1783, in _download_and_prepare
    super()._download_and_prepare(
  File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/builder.py", line 1116, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/builder.py", line 1621, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/home/sam/anaconda3/envs/omniquant/lib/python3.10/site-packages/datasets/builder.py", line 1778, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset

@albertvillanova @Y0mingZhang @kwchurch @HosseynGT @rafikg I tried the solutions you provided above, but none of them worked. Could you please give me some guidance

## Dataset viewer issue for 'wiki_lingua*'

**Link:** *link to the dataset viewer page*

`data = datasets.load_dataset("wiki_lingua", name=language, split="train[:2000]")
`
*short description of the issue*
```
[NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=11wMGqNVSwwk6zUnDaJEgm3qT71kAHeff']]()
```
Am I the one who added this dataset ? No


Add `data_dir` to `data_files` resolution and misc improvements to HfFileSystem

As discussed in https://github.com/huggingface/datasets/pull/2830#issuecomment-1048989764, this PR adds a QOL improvement to easily reference the files inside a directory in `load_dataset` using the `data_dir` param (very handy for ImageFolder because it avoids globbing, but also useful for the other loaders). Additionally, it fixes the issue with `HfFileSystem.isdir`, which would previously always return `False`, and aligns the path-handling logic in `HfFileSystem` with `fsspec.GitHubFileSystem`.


Add doc builder scripts

I think we're only missing the hosted runner to be configured for this repository and we should be good

Regarding the self-hosted runner, I actually encourage using the approach defined here: https://github.com/huggingface/transformers/pull/15710, which doesn't leverage a self-hosted runner. This prevents queuing jobs, which is important when we expect several concurrent jobs.

Opened a PR for that on your branch here: https://github.com/huggingface/datasets/pull/3793

I added the three scripts:
- build_dev_documentation.yml
- build_documentation.yml
- delete_dev_documentation.yml

I got them from `transformers` and did a few changes:
- I removed the `transformers`-specific dependencies
- I changed all the paths to be "datasets" instead of "transformers"
- I passed the `--library_name datasets` arg to the `doc-builder build` command (according to https://github.com/huggingface/doc-builder/pull/94/files#diff-bcc33cf7c223511e498776684a9a433810b527a0a38f483b1487e8a42b6575d3R26)

cc @LysandreJik @mishig25


Add URL and ID fields to Wikipedia dataset

Do you think we have a dedicated branch for all the changes we want to do to wikipedia ? Then once everything looks good + we have preprocessed the main languages, we can merge it on the `master` branch

Yes, @lhoestq, I agree with you.

I have just created the dedicated branch [`update-wikipedia`](https://github.com/huggingface/datasets/tree/update-wikipedia). We can merge every PR (once validated) to that branch; once all changes are merged to that branch, we could create the preprocessed datasets and then merge the branch to master.

This PR adds the URL field, so that we conform to proper attribution, required by their license: provide credit to the authors by including a hyperlink (where possible) or URL to the page or pages you are re-using.

About the conversion from title to URL, I found that apart from replacing blanks with underscores, some other special character must also be percent-encoded (e.g. `"` to `%22`): https://meta.wikimedia.org/wiki/Help:URL

Therefore, I have finally used `urllib.parse.quote` function. This additionally percent-encodes non-ASCII characters, but Wikimedia docs say these are equivalent:
> For the other characters either the code or the character can be used in internal and external links, they are equivalent. The system does a conversion when needed.
> [[%C3%80_propos_de_M%C3%A9ta]]
> is rendered as [À_propos_de_Méta](https://meta.wikimedia.org/wiki/%C3%80_propos_de_M%C3%A9ta), almost like [À propos de Méta](https://meta.wikimedia.org/wiki/%C3%80_propos_de_M%C3%A9ta), which leads to this page on Meta with in the address bar the URL
> [http://meta.wikipedia.org/wiki/%C3%80_propos_de_M%C3%A9ta](https://meta.wikipedia.org/wiki/%C3%80_propos_de_M%C3%A9ta)
> while [http://meta.wikipedia.org/wiki/À_propos_de_Méta](https://meta.wikipedia.org/wiki/%C3%80_propos_de_M%C3%A9ta) leads to the same.

Fix #3398.
CC: @geohci


Only-data dataset loaded unexpectedly as validation split

bug

Something isn't working

I see two options:
1. drop the "dev" keyword since it can be considered too generic
2. improve the pattern to something more reasonable, e.g. asking for a separator before and after "dev"
```python
["*[ ._-]dev[ ._-]*", "dev[ ._-]*"]
```

I think 2. is nice. If we agree on this one we can even decide to require the separation for the other split keywords "train", "test" etc.

Yes, I had something like that on mind: "dev" not being part of a word.
```
"[^a-zA-Z]dev[^a-zA-Z]"

Is there a reason why we want that regex? It feels like something that'll still be an issue for some weird case. "my_dataset_dev" doesn't match your regex, "my_dataset_validation" doesn't either ... Why not always "train" unless specified?

The regex is needed as part of our effort to make datasets configurable without code. In particular we define some generic dataset repository structures that users can follow

> ```
> "[^a-zA-Z]*dev[^a-zA-Z]*"
> ```

unfortunately our glob doesn't support "^":

https://github.com/fsspec/filesystem_spec/blob/3e739db7e53f5b408319dcc9d11e92bc1f938902/fsspec/spec.py#L465-L479

> "my_dataset_dev" doesn't match your regex, "my_dataset_validation" doesn't either ... Why not always "train" unless specified?

And `my_dataset_dev.foo` would match the pattern, and we also have the same pattern but for the "validation" keyword so `my_dataset_validation.foo` would work too

> The regex is needed as part of our effort to make datasets configurable without code

This feels like coding with the filename ^^'

This is still much easier than having to write a full dataset script right ? :p

## Describe the bug
As reported by @thomasw21 and @lhoestq, a dataset containing only a data file whose name matches the pattern `*dev*` will be returned as VALIDATION split, even if this is not the desired behavior, e.g. a file named `datosdevision.jsonl.gz`.


Fix Google Drive URL to avoid Virus scan warning

Once this PR merged into master and until our next `datasets` library release, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```

Thanks, that solved a bunch of problems we had downstream!
cf. https://github.com/ElementAI/picard/issues/61

This PR fixes, in the datasets library instead of in every specific dataset, the issue of downloading the Virus scan warning page instead of the actual data file for Google Drive URLs.

Fix #3786, fix #3784.


Bug downloading Virus scan warning page from Google Drive URLs

Something isn't working

Once the PR merged into master and until our next `datasets` library release, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```

## Describe the bug
Recently, some issues were reported with URLs from Google Drive, where we were downloading the Virus scan warning page instead of the data file itself.

See:
- #3758
- #3773
- #3784


Fix: Bypass Virus Checks in Google Drive Links (CNN-DM dataset)

Got it. Thanks for explaining this, @albertvillanova!

> On the other hand, the tests are not passing because the dummy data should also be fixed. Once done, this PR will be able to be merged into master.

Will do this 👍

Hi ! I think we need to fix the issue for every dataset. This can be done simply by fixing how we handle Google Drive links, see my comment https://github.com/huggingface/datasets/pull/3775#issuecomment-1050970157

Hi @lhoestq! I think @albertvillanova has already fixed this in #3787

Hi, @AngadSethi, I think that once:
- #3787

was merged, issue:
- #3784

was also fixed.

Therefore, I think this PR is no longer necessary. I'm closing it. Let me know if you agree.

Yes, absolutely @albertvillanova! I agree :)

This commit fixes the issue described in #3784. By adding an extra parameter to the end of Google Drive links, we are able to bypass the virus check and download the datasets.

So, if the original link looked like https://drive.google.com/uc?export=download&id=0BwmD_VLjROrfTHk4NFg2SndKcjQ

The new link now looks like https://drive.google.com/uc?export=download&id=0BwmD_VLjROrfTHk4NFg2SndKcjQ&confirm=t

Fixes #3784


Unable to Download CNN-Dailymail Dataset

Something isn't working

@AngadSethi thanks for reporting and thanks for your PR!

Glad to help @albertvillanova! Just fine-tuning the PR, will comment once I am able to get it up and running 😀

## Describe the bug
I am unable to download the CNN-Dailymail dataset. Upon closer investigation, I realised why this was happening:
- The dataset sits in Google Drive, and both the CNN and DM datasets are large.
- Google is unable to scan the folder for viruses, **so the link which would originally download the dataset, now downloads the source code of this web page:**
![image](https://user-images.githubusercontent.com/58678541/155658435-c2f497d7-7601-4332-94b1-18a62dd96422.png)
- **This leads to the following error**:
```python
NotADirectoryError: [Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b/cnn/stories'
```

## Steps to reproduce the bug
```python
import datasets
dataset = datasets.load_dataset("cnn_dailymail", "3.0.0", split="train")
```

## Expected results
That the dataset is downloaded and processed just like other datasets.

## Actual results
Hit with this error:
```python
NotADirectoryError: [Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b/cnn/stories'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 6.0.1


Support passing str to iter_files

@mariosasko it was indeed while reading that PR, that I remembered this change I wanted to do long ago... 😉


Error of writing with different schema, due to nonpreservation of nullability

Hi ! Thanks for reporting, indeed `disable_nullable` doesn't seem to be supported in this case. Maybe at one point we can have `disable_nullable` as a parameter of certain feature types

## 1. Case
```
dataset.map(
  batched=True,
  disable_nullable=True,
)
```
will get the following error at here https://github.com/huggingface/datasets/blob/c9967f55626931f8059dc416526c791444cdfdf7/src/datasets/arrow_writer.py#L516

`pyarrow.lib.ArrowInvalid: Tried to write record batch with different schema`

## 2. Debugging
### 2.1 tracing
During `_map_single`, the following are called
https://github.com/huggingface/datasets/blob/c9967f55626931f8059dc416526c791444cdfdf7/src/datasets/arrow_dataset.py#L2523
https://github.com/huggingface/datasets/blob/c9967f55626931f8059dc416526c791444cdfdf7/src/datasets/arrow_writer.py#L511
### 2.2. Observation
The problem is, even after `table_cast`, `pa_table.schema != self._schema`
`pa_table.schema` (before/after `table_cast`)
```
input_ids: list<item: int32>
  child 0, item: int32
```
`self._schema`
```
input_ids: list<item: int32> not null
  child 0, item: int32
```
### 2.3. Reason
https://github.com/huggingface/datasets/blob/c9967f55626931f8059dc416526c791444cdfdf7/src/datasets/table.py#L1121
Here we lose nullability stored in `schema` because it seems that `Features` is always nullable and don't store nullability.
https://github.com/huggingface/datasets/blob/c9967f55626931f8059dc416526c791444cdfdf7/src/datasets/table.py#L1103
So, casting to schema from such `Features` loses nullability, and eventually causes error of writing with different schema

## 3. Solution
1. Let `Features` stores nullability.
2. Directly cast table with original schema but not schema from converted `Features`. (this PR)
3. Don't `cast_table` when `write_table`


Reddit dataset card additions

Hello! I added the tags and created a PR. Just to note, regarding the paperswithcode_id tag, that currently has the value "reddit"; the dataset described as reddit in paperswithcode is https://paperswithcode.com/dataset/reddit and it isn't the Webis-tldr-17. I could not find Webis-tldr-17 in paperswithcode neither in the Summarization category nor using the keywords reddit, webis, & tldr. I didn't change this tag.

The changes proposed are based on the "TL;DR: Mining Reddit to Learn Automatic Summarization" paper & https://zenodo.org/record/1043504#.YhaKHpbQC38
It is a Reddit dataset indeed, but the name given to the dataset by the authors is Webis-TLDR-17 (corpus), so perhaps it should be modified as well.
The task at which the dataset is aimed is abstractive summarization.


Add ElkarHizketak v1.0 dataset

I also filled some missing sections in the dataset card


Update manual download URL in newsroom dataset

Fix #3778.


Not be able to download dataset - "Newsroom"

dataset bug

A bug in a dataset script provided in the library

Hi @Darshan2104, thanks for reporting.

Please note that at Hugging Face we do not host the data of this dataset, but just a loading script pointing to the host of the data owners.

Apparently the data owners changed their data host server. After googling it, I found their new website at: https://lil.nlp.cornell.edu/newsroom/index.html
- Download page: https://lil.nlp.cornell.edu/newsroom/download/index.html

I'm fixing the link in our Datasets library.

@albertvillanova Thanks for the solution and link you made my day!

Hello,

I tried to download the **newsroom** dataset but it didn't work out for me. it said me to **download it manually**!

For manually, Link is also didn't work! It is sawing some ad or something!

If anybody has solved this issue please help me out or if somebody has this dataset please share your google drive link, it would be a great help!

Thanks
Darshan Tank


Start removing canonical datasets logic

> I'm not sure if the documentation explains why the dataset identifiers might have a namespace or not (the user/org): 'glue' vs 'severo/glue'. Do you think we should explain it, and relate it to the GitHub/Hub distinction?

I added an explanation, let me know if it sounds good to you:

```
Datasets used to be hosted on our GitHub repository, but all datasets have now been migrated to the Hugging Face Hub.
The legacy GitHub datasets were added originally on our GitHub repository and therefore don't have a namespace: "squad", "glue", etc. unlike the other datasets that are named "username/dataset_name" or "org/dataset_name".
```

Thanks for the feedbacks ! Merging this now - if you have some comments I can take care of them in a subsequent PR

I'll also take care of resolving the conflicts with https://github.com/huggingface/datasets/pull/3690

I updated the source code and the documentation to start removing the "canonical datasets" logic.

Indeed this makes the documentation confusing and we don't want this distinction anymore in the future. Ideally users should share their datasets on the Hub directly.

### Changes

- the documentation about dataset loading mentions the datasets on the Hub (no difference between canonical and community, since they all have their own repository now)
- the documentation about adding a dataset doesn't explain the technical differences between canonical and community anymore, and only presents how to add a community dataset. There is still a small section at the bottom that mentions the datasets that are still on GitHub and redirects to the `ADD_NEW_DATASET.md` guide on GitHub about how to contribute a dataset to the `datasets` library
- the code source doesn't mention "canonical" anymore anywhere. There is still a `GitHubDatasetModuleFactory` class that is left, but I updated the docstring to say that it will be eventually removed in favor of the `HubDatasetModuleFactory` classes that already exist

Would love to have your feedbacks on this !

cc @julien-c @thomwolf @SBrandeis


Allow download only some files from the Wikipedia dataset

enhancement

New feature or request

Hi @jvanz, thank you for your proposal.

In fact, we are aware that it is very common the problem you mention. Because of that, we are currently working in implementing a new version of wikipedia on the Hub, with all data preprocessed (no need to use Apache Beam), from where you will be able to use `data_files` to load only a specific subset of the data files.

See:
- #3401

**Is your feature request related to a problem? Please describe.**
The Wikipedia dataset can be really big. This is a problem if you want to use it locally in a laptop with the Apache Beam `DirectRunner`. Even if your laptop have a considerable amount of memory (e.g. 32gb).

**Describe the solution you'd like**
I would like to use the `data_files` argument in the `load_dataset` function to define which file in the wikipedia dataset I would like to download. Thus, I can work with the dataset in a smaller machine using the Apache Beam `DirectRunner`.

**Describe alternatives you've considered**
I've tried to use the `simple` Wikipedia dataset. But it's in English and I would like to use Portuguese texts in my model.


Update gigaword card and info

I think it actually comes from an issue here:

https://github.com/huggingface/datasets/blob/810b12f763f5cf02f2e43565b8890d278b7398cd/src/datasets/utils/file_utils.py#L575-L579

and

https://github.com/huggingface/datasets/blob/810b12f763f5cf02f2e43565b8890d278b7398cd/src/datasets/utils/streaming_download_manager.py#L386-L389

This code doesn't seem to work anymore. This can probably be fixed with

```python
if url.startswith("https://drive.google.com/"):
    url += "&confirm=t"
    cookies = response.cookies
```

because Google Drive doesn't return the `download_warning` cookie anymore.

Actually it seems that is has been fixed already in https://github.com/huggingface/datasets/pull/3787 :)

I think it should have fixed the gigaword dataset loading

@lhoestq The linked PR indeed fixes the issue. This PR is still worth merging IMO to update `gigaword`'s card.

Reported on the forum: https://discuss.huggingface.co/t/error-loading-dataset/14999


Fix reddit_tifu data URL

Fix #3773.


Checksum mismatch for the reddit_tifu dataset

bug

Something isn't working

Thanks for reporting, @anna-kay. We are fixing it.

@albertvillanova Thank you for the fast response! However I am still getting the same error:

Downloading: 2.23kB [00:00, ?B/s]
Traceback (most recent call last):
  File "C:\Users\Anna\PycharmProjects\summarization\main.py", line 17, in <module>
    dataset = load_dataset('reddit_tifu', 'long')
  File "C:\Users\Anna\Desktop\summarization\summarization_env\lib\site-packages\datasets\load.py", line 1702, in load_dataset
    builder_instance.download_and_prepare(
  File "C:\Users\Anna\Desktop\summarization\summarization_env\lib\site-packages\datasets\builder.py", line 594, in download_and_prepare
    self._download_and_prepare(
  File "C:\Users\Anna\Desktop\summarization\summarization_env\lib\site-packages\datasets\builder.py", line 665, in _download_and_prepare
    verify_checksums(
  File "C:\Users\Anna\Desktop\summarization\summarization_env\lib\site-packages\datasets\utils\info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1ffWfITKFMJeqjT8loC8aiCLRNJpc_XnF']

I have cleaned the cache/huggingface/datasets & cache/huggingface/modules files and also tried on another machine with a fresh installation of trasnformers & datasets.
The reddit_tifu.py that gets downloaded still has the previous url on line 51, _URL = "https://drive.google.com/uc?export=download&id=1ffWfITKFMJeqjT8loC8aiCLRNJpc_XnF"

Hi @anna-kay, I'm sorry I didn't clearly explain the details to you:
- the error has been fixed in our `master` branch on GitHub: https://github.com/huggingface/datasets/commit/8ae21bf6a77175dc803ce2f1b93d18b8fbf45586
- the fix will not be accessible to users in PyPI until our next release of the `datasets` library
  - our latest release (version 1.18.3) was made 23 days ago: https://github.com/huggingface/datasets/releases/tag/1.18.3
- in the meantime, you can get the fix if you install datasets from our GitHub `master` branch:
  ```
  pip install git+https://github.com/huggingface/datasets#egg=datasets
  ```

@albertvillanova  Ok great, makes sence. Thank you very much for the explanation!

## Describe the bug
A checksum occurs when downloading the reddit_tifu data (both long & short).

## Steps to reproduce the bug
reddit_tifu_dataset = load_dataset('reddit_tifu', 'long')

## Expected results
The expected result is for the dataset to be downloaded and cached locally.

## Actual results
  File "/.../lib/python3.9/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1ffWfITKFMJeqjT8loC8aiCLRNJpc_XnF']

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.13.0-30-generic-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyArrow version: 7.0.0


Fix: dataset name is stored in keys


Fix DuplicatedKeysError on msr_sqa dataset

Fix #3770.


DuplicatedKeysError on msr_sqa dataset

Thanks for reporting, @kolk.

We are fixing it.

### Describe the bug
Failure to generate dataset msr_sqa because of duplicate keys.

### Steps to reproduce the bug
```
from datasets import load_dataset
load_dataset("msr_sqa")
```
### Expected results
The examples keys should be unique.

**Actual results**
```
>>> load_dataset("msr_sqa")
Downloading:
6.72k/? [00:00<00:00, 148kB/s]
Downloading:
2.93k/? [00:00<00:00, 53.8kB/s]
Using custom data configuration default
Downloading and preparing dataset msr_sqa/default (download: 4.57 MiB, generated: 26.25 MiB, post-processed: Unknown size, total: 30.83 MiB) to /root/.cache/huggingface/datasets/msr_sqa/default/0.0.0/70b2a497bd3cc8fc960a3557d2bad1eac5edde824505e15c9c8ebe4c260fd4d1...
Downloading: 100%
4.80M/4.80M [00:00<00:00, 7.49MB/s]
---------------------------------------------------------------------------
DuplicatedKeysError                       Traceback (most recent call last)
[/usr/local/lib/python3.7/dist-packages/datasets/builder.py](https://localhost:8080/#) in _prepare_split(self, split_generator)
   1080                     example = self.info.features.encode_example(record)
-> 1081                     writer.write(example, key)
   1082             finally:

8 frames
DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: nt-639
Keys should be unique and deterministic in nature

During handling of the above exception, another exception occurred:

DuplicatedKeysError                       Traceback (most recent call last)
[/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py](https://localhost:8080/#) in check_duplicate_keys(self)
    449         for hash, key in self.hkey_record:
    450             if hash in tmp_record:
--> 451                 raise DuplicatedKeysError(key)
    452             else:
    453                 tmp_record.add(hash)

DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: nt-639
Keys should be unique and deterministic in nature
```

### Environment info
datasets version: 1.18.3
Platform: Google colab notebook
Python version: 3.7
PyArrow version: 6.0.1


`dataset = dataset.map()` causes faiss index lost

Something isn't working

Hi ! Indeed `map` is dropping the index right now, because one can create a dataset with more or fewer rows using `map` (and therefore the index might not be relevant anymore)

I guess we could check the resulting dataset length, and if the user hasn't changed the dataset size we could keep the index, what do you think ?

doing `.add_column("x",x_data)` also removes the index. the new column might be irrelevant to the index so I don't think it should drop.

Minimal example

```python
from datasets import load_dataset
import numpy as np

data=load_dataset("ceyda/cats_vs_dogs_sample") #just a test dataset
data=data["train"]
embd_data=data.map(lambda x: {"emb":np.random.uniform(-1,0,50).astype(np.float32)})
embd_data.add_faiss_index(column="emb")
print(embd_data.list_indexes())
embd_data=embd_data.add_column("x",[0]*data.num_rows)
print(embd_data.list_indexes())
```

I agree `add_column` shouldn't drop the index indeed ! Is it something you'd like to contribute ? I think it's just a matter of copying the `self._indexes` dictionary to the output dataset

## Describe the bug
assigning the resulted dataset to original dataset causes lost of the faiss index

## Steps to reproduce the bug
`my_dataset` is a regular loaded dataset. It's a part of a customed dataset structure
```python
self.dataset.add_faiss_index('embeddings')
self.dataset.list_indexes()
# ['embeddings']

dataset2 = my_dataset.map(
    lambda x: self._get_nearest_examples_batch(x['text']), batch=True
)

# the unexpected result:
dataset2.list_indexes()
# []

self.dataset.list_indexes()
# ['embeddings']
```

in case something wrong with my `_get_nearest_examples_batch()`, it's like this
```python
def _get_nearest_examples_batch(self, examples, k=5):
    queries = embed(examples)
    scores_batch, retrievals_batch = self.dataset.get_nearest_examples_batch(self.faiss_column, queries, k)
    return {
        'neighbors': [batch['text'] for batch in retrievals_batch],
        'scores': scores_batch
    }
```

## Expected results
`map` shouldn't drop the indexes, in another word, indexes should be carried to the generated dataset

## Actual results
map drops the indexes

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Ubuntu 20.04.3 LTS
- Python version: 3.8.12
- PyArrow version: 7.0.0


Fix HfFileSystem docstring


Expose method and fix param

A fix + expose a new method, following https://github.com/huggingface/datasets/pull/3670


Fix head_qa data URL

Fix #3758.


Update URL for tagging app

Oh, this URL shouldn't be updated to the tagging app as it's actually used for creating the README - closing this.

This PR updates the URL for the tagging app to be the one on Spaces.


## Dataset viewer issue for '*name of the dataset*'

**Link:** *link to the dataset viewer page*

*short description of the issue*

Am I the one who added this dataset ? Yes-No


It's not possible download `20200501.pt` dataset

Something isn't working

Hi @jvanz, thanks for reporting.

Please note that Wikimedia website does not longer host Wikipedia dumps for so old dates.

For a list of accessible dump dates of `pt` Wikipedia, please see: https://dumps.wikimedia.org/ptwiki/

You can load for example `20220220` `pt` Wikipedia:
```python
dataset = load_dataset("wikipedia", language="pt", date="20220220", beam_runner="DirectRunner")
```

> ```python
> dataset = load_dataset("wikipedia", language="pt", date="20220220", beam_runner="DirectRunner")
> ```

Thank you! I did not know that I can do this. I was following the example in the error message when I do not define which language dataset I'm trying to download.

I've tried something similar changing the date in the `load_dataset` call that I've shared in the bug description. Obviously, it did not work. I need to read the docs more carefully next time. My bad!

Thanks again and sorry for the noise.

## Describe the bug
The dataset `20200501.pt` is broken.

The available datasets: https://dumps.wikimedia.org/ptwiki/

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("wikipedia", "20200501.pt", beam_runner='DirectRunner')
```

## Expected results
I expect to download the dataset locally.

## Actual results
```
>>> from datasets import load_dataset
>>> dataset = load_dataset("wikipedia", "20200501.pt", beam_runner='DirectRunner')
Downloading and preparing dataset wikipedia/20200501.pt to /home/jvanz/.cache/huggingface/datasets/wikipedia/20200501.pt/1.0.0/009f923d9b6dd00c00c8cdc7f408f2b47f45dd4f5fb7982a21f9448f4afbe475...
/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/apache_beam/__init__.py:79: UserWarning: This version of Apache Beam has not been sufficiently tested on Python 3.9. You may encounter bugs or missing features.
  warnings.warn(
  0%|                                                                                | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/load.py", line 1702, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/builder.py", line 594, in download_and_prepare
    self._download_and_prepare(
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/builder.py", line 1245, in _download_and_prepare
    super()._download_and_prepare(
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/builder.py", line 661, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/jvanz/.cache/huggingface/modules/datasets_modules/datasets/wikipedia/009f923d9b6dd00c00c8cdc7f408f2b47f45dd4f5fb7982a21f9448f4afbe475/wikipedia.py", line 420, in _split_generators
    downloaded_files = dl_manager.download_and_extract({"info": info_url})
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/utils/download_manager.py", line 307, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/utils/download_manager.py", line 195, in download
    downloaded_path_or_paths = map_nested(
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 260, in map_nested
    mapped = [
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 261, in <listcomp>
    _single_map_nested((function, obj, types, None, True))
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 196, in _single_map_nested
    return function(data_struct)
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/utils/download_manager.py", line 216, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/utils/file_utils.py", line 298, in cached_path
    output_path = get_from_cache(
  File "/home/jvanz/anaconda3/envs/tf-gpu/lib/python3.9/site-packages/datasets/utils/file_utils.py", line 612, in get_from_cache
    raise FileNotFoundError(f"Couldn't find file at {url}")
FileNotFoundError: Couldn't find file at https://dumps.wikimedia.org/ptwiki/20200501/dumpstatus.json

```

## Environment info
```
- `datasets` version: 1.18.3
- Platform: Linux-5.3.18-150300.59.49-default-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyArrow version: 6.0.1
```


`Dataset.class_encode` should support custom class names

enhancement

New feature or request

Hi @Dref360, thanks a lot for your proposal.

It totally makes sense to have more flexibility when class encoding, I agree.

You could even further customize the class encoding by passing an instance of `ClassLabel` itself (instead of replicating `ClassLabel` instantiation arguments as `Dataset.class_encode_column` arguments).

And the latter made me think of `Dataset.cast_column`...

Maybe better to have some others' opinions @lhoestq @mariosasko

Hi @Dref360! You can use [`Dataset.align_labels_with_mapping`](https://huggingface.co/docs/datasets/master/package_reference/main_classes.html#datasets.Dataset.align_labels_with_mapping) after `Dataset.class_encode_column` to assign a different mapping of labels to ids.

@albertvillanova I'd like to avoid adding more complexity to the API where it's not (absolutely) needed, so I don't think introducing a new param in `Dataset.class_encode_column` is a good idea.

I wasn't aware that it existed thank you for the link.

Closing then!

I can make a PR, just wanted approval before starting.

**Is your feature request related to a problem? Please describe.**

It is often the case that classes are not ordered in alphabetical order. Current `class_encode_column` sort the classes before indexing.
https://github.com/huggingface/datasets/blob/master/src/datasets/arrow_dataset.py#L1235

**Describe the solution you'd like**
I would like to add a **optional** parameter `class_names` to `class_encode_column` that would  be used for the mapping instead of sorting the unique values.

**Describe alternatives you've considered**

One can use map instead. I find it harder to read.
```python
CLASS_NAMES = ['apple', 'orange', 'potato']
ds = ds.map(lambda item: CLASS_NAMES.index(item[label_column]))

# Proposition
ds = ds.class_encode_column(label_column, CLASS_NAMES)
```

**Additional context**

I can make the PR if this feature is accepted.


Know your data for HF hub

New feature or request

Hi @Muhtasham you should take a look at https://huggingface.co/blog/data-measurements-tool and accompanying demo app at https://huggingface.co/spaces/huggingface/data-measurements-tool

We would be interested in your feedback. cc @meg-huggingface @sashavor @yjernite

**Is your feature request related to a problem? Please describe.**
Would be great to see be able to understand datasets with the goal of improving data quality, and helping mitigate fairness and bias issues.

**Describe the solution you'd like**
Something like https://knowyourdata.withgoogle.com/ for HF hub


Unable to view the Gradio flagged call back dataset

Hi @kingabzpro.

I think you need to create a loading script that creates the dataset from the CSV file and the image paths.

As example, you could have a look at the Food-101 dataset: https://huggingface.co/datasets/food101
- Loading script: https://huggingface.co/datasets/food101/blob/main/food101.py

Once the loading script is created, the viewer will show a previsualization of your dataset.

@albertvillanova I don't think this is the issue. I have created another dataset with similar files and format and it works. https://huggingface.co/datasets/kingabzpro/savtadepth-flags-V2

Yes, you are right, that was not the issue.

Just take into account that sometimes the viewer can take some time until it shows the preview of the dataset.
After some time, yours is finally properly shown: https://huggingface.co/datasets/kingabzpro/savtadepth-flags

The problem was resolved by deleted the dataset and creating new one with similar name and then clicking on flag button.

I think if you make manual changes to dataset the whole system breaks.

## Dataset viewer issue for '*savtadepth-flags*'

**Link:** *[savtadepth-flags](https://huggingface.co/datasets/kingabzpro/savtadepth-flags)*

*with the Gradio 2.8.1 the dataset viers stopped working. I tried to add values manually but its not working. The dataset is also not showing the link with the app https://huggingface.co/spaces/kingabzpro/savtadepth.*

Am I the one who added this dataset ? Yes


Rename GenerateMode to DownloadMode

Thanks! Used here: https://github.com/huggingface/datasets-preview-backend/blob/main/src/datasets_preview_backend/models/dataset.py#L26 :)

This PR:
- Renames `GenerateMode` to `DownloadMode`
- Implements `DeprecatedEnum`
- Deprecates `GenerateMode`

Close #769.