Add Fake News Detection in Filipino dataset

Hi! I'm the author of this paper (surprised to see our datasets have been added already).

That paper link only leads to the conference index, here's a link to the actual paper: https://www.aclweb.org/anthology/2020.lrec-1.316/

Would it be fine if I also edited your gsheet entry to reflect this change?

Hi Jan, please go ahead and update. I see you are also in the sprint slack channel. Let me know if what else needs updating. Thanks.

This PR adds the Fake News Filipino Dataset, a low-resource fake news detection corpora in Filipino. Contains 3,206 expertly-labeled news samples, half of which are real and half of which are fake.

Link to the paper: http://www.lrec-conf.org/proceedings/lrec2020/index.html

Link to the dataset/repo: https://github.com/jcblaisecruz02/Tagalog-fake-news


Add Dengue dataset in Filipino

This PR adds the Dengue Dataset, a benchmark dataset for low-resource multiclass classification, with 4,015 training, 500 testing, and 500 validation examples, each labeled as part of five classes. Each sample can be a part of multiple classes. Collected as tweets.

Link to the paper: https://ieeexplore.ieee.org/document/8459963

Link to the dataset/repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks


adding hind_encorp dataset

extension of the file is .plaintext so dummy data generation is failing

you can add the `--match_text_file "*.plaintext"` flag when generating the dummy data

also it looks like the PR is empty, is this expected ?

yes it is expected because I  made all my changes in  PR  #1186  then I again run code and open PR #1188 to see if this time test passes or not only so there is no code change from #1186 to #1188
i tried  --match_text_file "*.plaintext" this time it is also not generating dummy data don't know why

well this PR includes no code change at all, can you make sure you added your changes in this one ?

feel free to ping me when you have added the files so I can take a look and help you with the dummy data

how to do that i dont know did i have to open new PR

actually all my changes are visible in  #1186 but  don't know how to show same changes here

these are a the which i did in #1186 and same in #1188
![1](https://user-images.githubusercontent.com/56379013/101646577-b4864500-3a5d-11eb-8a5a-91b1b441040a.png)
![2](https://user-images.githubusercontent.com/56379013/101646965-32e2e700-3a5e-11eb-94d9-276e602c6ded.png)
![4](https://user-images.githubusercontent.com/56379013/101646989-38d8c800-3a5e-11eb-92bb-d9c4cb2c3595.png)
![5](https://user-images.githubusercontent.com/56379013/101647017-41c99980-3a5e-11eb-87cf-5268e79df19d.png)
![6](https://user-images.githubusercontent.com/56379013/101647038-48581100-3a5e-11eb-8d05-f67834fcaa7b.png)

![8](https://user-images.githubusercontent.com/56379013/101647080-55750000-3a5e-11eb-8455-8936a35b35c2.png)
![9](https://user-images.githubusercontent.com/56379013/101647084-55750000-3a5e-11eb-988e-ae87f0b252a0.png)
![10](https://user-images.githubusercontent.com/56379013/101647182-6f164780-3a5e-11eb-8af3-f0b0186483c9.png)
![11](https://user-images.githubusercontent.com/56379013/101647230-7c333680-3a5e-11eb-9aeb-2b4ce65965e0.png)
![13](https://user-images.githubusercontent.com/56379013/101647257-848b7180-3a5e-11eb-871c-2fd77b047320.png)
![14](https://user-images.githubusercontent.com/56379013/101647268-89502580-3a5e-11eb-9e2a-b9f7ff1fc95e.png)
these same codes are in both #1186 and #1188 so because it is already present from PR #1186 because of that it is showing zeor code change in #1188 because it is already present from #1186 how i can show or highlight those changes

well for me https://github.com/huggingface/datasets/pull/1188/files is blank

This PR tries to merge the master branch of you fork into this repo, however I can't find changes with your files inside your master branch.

Maybe you can fork again the repo and try to create another PR ?

@lhoestq  i  opened a new pr #1438 but this time it fails many circl ci tests

Closing this one since a new PR was created

adding Hindi_Encorp05 dataset


Added AQUA-RAT (Algebra Question Answering with Rationales) Dataset


all test passed

looks like this PR includes changes to 5000 files
could you create a new branch and a new PR ?

need help creating dummy data


Add Hate Speech Dataset in Filipino

This PR adds the Hate Speech Dataset, a text classification dataset in Filipino, consisting 10k tweets (training set) that are labeled as hate speech or non-hate speech. Released with 4,232 validation and 4,232 testing samples. Collected during the 2016 Philippine Presidential Elections.

Link to the paper: https://pcj.csp.org.ph/index.php/pcj/issue/download/29/PCJ%20V14%20N1%20pp1-14%202019

Link to the dataset/repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks


Add Adversarial SQuAD dataset

the CI error was just a connection error due to all the activity on the repo this week ^^'
I re-ran it so it should be good now

I hadn't realized the problem with the dummies since it had passed without errors.
Suggestion: maybe we can show the user a warning based on the generated dummy size.

Thanks for changing to configs ! Looks all good now :)

Before we merge, can you re-lighten the dummy data please if you don't mind ? The idea is to have them weigh only a few KB (currently it's 50KB each). Feel free to remove any unnecessary files or chunk of text

(also you can ignore the `RemoteDatasetTest ` CI errors, they're fixed on master )

# Adversarial SQuAD

Adding the Adversarial [SQuAD](https://github.com/robinjia/adversarial-squad) dataset as part of the sprint 🎉
This dataset adds adversarial sentences to a subset of the SQuAD dataset's dev examples. How to get the original squad example id is explained in readme->Data Instances. The whole data is intended for use in evaluation. (Which could of course be also used for training if one wants). So there is no classical train/val/test split, but a split based on the number of adversaries added.

There are 2 splits of this dataset:

- AddSent: Has up to five candidate adversarial sentences that don't answer the question, but have a lot of words in common with the question. This adversary is does not query the model in any way.
- AddOneSent: Similar to AddSent, but just one candidate sentences was picked at random. This adversary is does not query the model in any way.

(The AddAny and AddCommon datasets mentioned in the paper are dynamically generated based on model's output distribution thus are not included here)

The failing test look like some unrelated timeout thing, will probably clear if rerun.
- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)


add mkb dataset

Could you update the languages tags before we merge @VasudevGupta7 ?

This PR will add Mann Ki Baat dataset (parallel data for Indian languages).


ADD COVID-QA dataset

Wow, thanks for including this dataset from my side as well!

This PR adds the COVID-QA dataset, a question answering dataset consisting of 2,019 question/answer pairs annotated by volunteer biomedical experts on scientific articles related to COVID-19

Link to the paper: https://openreview.net/forum?id=JENSKEEzsoU
Link to the dataset/repo: https://github.com/deepset-ai/COVID-QA


added emotions detection in arabic dataset

Hi @abdulelahsm did you manage to fix your issue ?
Feel free to ping me if you have questions or if you're ready for a review

@lhoestq fixed it! ready to merge. I hope haha

Dataset for Emotions detection in Arabic text

more info: https://github.com/AmrMehasseb/Emotional-Tone


Add  KorQuAD v2 Dataset

looks like this PR also includes the changes for the V1
Could you only include the files of the V2 ?

hmm I have made the dummy data lighter retested on local and it passed not sure why it fails here?

# The Korean Question Answering Dataset v2
Adding the [KorQuAD](https://korquad.github.io/) v2 dataset as part of the sprint 🎉
This dataset is very similar to SQuAD and is an extension of [squad_kor_v1](https://github.com/huggingface/datasets/pull/1178) which is why I added it as `squad_kor_v2`.

- Crowd generated questions and answer (1-answer per question) for Wikipedia articles. Differently from V1 it includes the html structure and markup, which makes it a different enough dataset. (doesn't share ids between v1 and v2 either)

- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)

Edit: 🤦 looks like squad_kor_v1 commit sneaked in here too


Small update to the doc: add flatten_indices in doc


Add  KorQuAD v1 Dataset

# The Korean Question Answering Dataset
Adding the [KorQuAD](https://korquad.github.io/KorQuad%201.0/) v1 dataset as part of the sprint 🎉
This dataset is very similar to SQuAD which is why I added it as `squad_kor_v1`. There is also a v2 which I added [here](https://github.com/huggingface/datasets/pull/1180).

- Crowd generated questions and answer (1-answer per question) for Wikipedia articles.

- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)


Add Korean NER dataset

This PR adds the [Korean named entity recognition dataset](https://github.com/kmounlp/NER). This dataset has been used in many downstream tasks, such as training [KoBERT](https://github.com/SKTBrain/KoBERT) for NER, as seen in this [KoBERT-CRF implementation](https://github.com/eagle705/pytorch-bert-crf-ner).


Add OpenPI Dataset

dataset contribution

Contribution to a dataset script

Hi @Bharat123rox ! It looks like some of the dummy data is broken or missing. Did you auto-generate it? Does the local test pass for you?

@yjernite requesting you to have a look as to why the tests are failing only on Windows, there seems to be a backslash error somewhere, could it be the result of `os.path.join` and what should be the fix for this?

This is the `black` output locally:
```
(datasets_env) datasets (openpi) > black --check --line-length 119 --target-version py36 datasets/openpi/
All done! ✨ 🍰 ✨
1 file would be left unchanged.
```

Can you check your version of black (should be `20.8b1`) and run `make style again`? (And don't forget to rebase before pushing ;) )

The other test was a time-out error so should be good on the next commit

Thanks @yjernite the CI tests finally passed!!

Hi @Bharat123rox did you manage to join the different config into one using the IDs ?

Feel free to ping me when you're ready for the next review :)

> Hi @Bharat123rox did you manage to join the different config into one using the IDs ?
>
> Feel free to ping me when you're ready for the next review :)

Not yet @lhoestq still working on this! Meanwhile please review #1507 where I added the SelQA dataset :)

Ok ! Let me review SelQA then :)
Thanks for your help !

Apologies for the very late response. Here is the openpi dataset file with a single file per partition after merging `id_answers, answers.jsonl, question.jsonl , question_metadata.jsonl`

https://github.com/allenai/openpi-dataset/blob/main/data/gold-v1.1/dev.jsonl

Hi @Bharat123rox , when you get a chance, please feel free to use the dataset from the repo ( [Link](https://github.com/allenai/openpi-dataset/blob/main/data/gold-v1.1/dev.jsonl) ) . Please let me know if any file is missing! Thank you

Hi @Bharat123rox are you working on this?

@nikett Sorry I'm no longer working on this as I'm out of time for it, please feel free to raise a new PR for this

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest to create this dataset there. Please, feel free to tell us if you need some help.

Add the OpenPI Dataset by AI2 (AllenAI)


added ReDial dataset

Updating README
Dataset link: https://redialdata.github.io/website/datasheet


Add Universal Morphologies

Sorry for the delay, changed the default language to "ady" (first alphabetical) and only downloading the relevant files for each config (dataset_infos is till 918KB though)

Thanks for merging it ! Looks all good

Looks like I didn't reply to your last message, sorry about that.
Feel free to ping me when this happens :)

Adding unimorph universal morphology annotations for 110 languages, pfew!!!

one lemma per row with all possible forms and annotations

https://unimorph.github.io/


add wikipedia biography dataset

Does anyone know why am I getting this "Some checks were not successful" message? For the _code_quality_ one, I have successfully run the flake8 command.

Ok, I need to update the README.md, but don't know if that will fix the errors

Hi @ACR0S , thanks for adding the dataset!

It looks like `black` is throwing the code quality error: you need to run `make style` with the latest version of `black` (`black --version` should return 20.8b1)

We also added a requirement to specify encodings when using the python `open` function (line 163 in the current version of your script)

Finally, you will need to add the tags and field descriptions to the README as described here https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#tag-the-dataset-and-write-the-dataset-card

Let us know if you have any further questions!

Also, please leave the full template of the readme  with the `[More Information Needed]` paragraphs: you don't have to fill them out now but it will make it easier for us to go back to later :)

Thank you for your help, @yjernite! I have updated everything (finally run the _make style_, added the tags, the ecoding to the _open_ function and put back the empty fields in the README). Hope it works now! :)

My first PR containing the Wikipedia biographies dataset. I have followed all the steps in the [guide](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md). It passes all the tests.


Add proto_qa dataset

Added dataset tags as required.


Add imdb Urdu Reviews dataset.

Added the imdb Urdu reviews dataset. More info about the dataset over here.


Fix path handling for Windows


Add Opus fiskmo dataset for Finnish and Swedish for MT task

Adding fiskmo, a massive parallel corpus for Finnish and Swedish.
for more info : http://opus.nlpl.eu/fiskmo.php


Add Naver sentiment movie corpus

This PR adds the [Naver sentiment movie corpus](https://github.com/e9t/nsmc), a dataset containing Korean movie reviews from Naver, the most commonly used search engine in Korea. This dataset is often used to benchmark models on Korean NLP tasks, as seen in [this paper](https://www.aclweb.org/anthology/2020.lrec-1.199.pdf).


❓ On-the-fly tokenization with datasets, tokenizers, and torch Datasets and Dataloaders

question

Further information is requested

generic discussion

Generic discussion on the library

We're working on adding on-the-fly transforms in datasets.
Currently the only on-the-fly functions that can be applied are in `set_format` in which we transform the data in either numpy/torch/tf tensors or pandas.
For example
```python
dataset.set_format("torch")
```
applies `torch.Tensor` to the dataset entries on-the-fly.

We plan to extend this to user-defined formatting transforms.
For example
```python
dataset.set_format(transform=tokenize)
```

What do you think ?

You can now use `set_transform` to define custom formatting transforms.

Hi there,

I have a question regarding "on-the-fly" tokenization. This question was elicited by reading the "How to train a new language model from scratch using Transformers and Tokenizers" [here](https://huggingface.co/blog/how-to-train). Towards the end there is this sentence: "If your dataset is very large, you can opt to load and tokenize examples on the fly, rather than as a preprocessing step". I've tried coming up with a solution that would combine both `datasets` and `tokenizers`, but did not manage to find a good pattern.

I guess the solution would entail wrapping a dataset into a Pytorch dataset.

As a concrete example from the [docs](https://huggingface.co/transformers/custom_datasets.html)

```python
import torch

class SquadDataset(torch.utils.data.Dataset):
    def __init__(self, encodings):
        # instead of doing this beforehand, I'd like to do tokenization on the fly
        self.encodings = encodings

    def __getitem__(self, idx):
        return {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}

    def __len__(self):
        return len(self.encodings.input_ids)

train_dataset = SquadDataset(train_encodings)
```

How would one implement this with "on-the-fly" tokenization exploiting the vectorized capabilities of tokenizers?

----

Edit: I have come up with this solution. It does what I want, but I feel it's not very elegant

```python
class CustomPytorchDataset(Dataset):
    def __init__(self):
        self.dataset = some_hf_dataset(...)
        self.tokenizer = BertTokenizerFast.from_pretrained("bert-base-uncased")

    def __getitem__(self, batch_idx):
        instance = self.dataset[text_col][batch_idx]
        tokenized_text = self.tokenizer(instance, truncation=True, padding=True)
        return tokenized_text

    def __len__(self):
        return len(self.dataset)

    @staticmethod
    def collate_fn(batch):
        # batch is a list, however it will always contain 1 item because we should not use the
        # batch_size argument as batch_size is controlled by the sampler
        return {k: torch.tensor(v) for k, v in batch[0].items()}

torch_ds = CustomPytorchDataset()

# NOTE: batch_sampler returns list of integers and since here we have SequentialSampler
# it returns: [1, 2, 3], [4, 5, 6], etc. - check calling `list(batch_sampler)`
batch_sampler = BatchSampler(SequentialSampler(torch_ds), batch_size=3, drop_last=True)

# NOTE: no `batch_size` as now the it is controlled by the sampler!
dl = DataLoader(dataset=torch_ds, sampler=batch_sampler, collate_fn=torch_ds.collate_fn)
```


Opus montenegrinsubs

Opus montenegrinsubs - language pair en-me
more info : http://opus.nlpl.eu/MontenegrinSubs.php


Add ar rest reviews

Copy-pasted from the Slack discussion:
the annotation and language creators should be found , not unknown
the example should go under the "Data Instances" paragraph, not "Data fields"
can you remove the abstract from the citation and add it to the dataset description? More people will see that

@lhoestq not sure why it's failing tests now, I only changed cosmetics

You can ignores these errors
```

=========================== short test summary info ===========================
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_ajgt_twitter_ar
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_chr_en
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_ajgt_twitter_ar
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_chr_en
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_great_code
```

they're fixed on master

Feel free to ping me for the final review once you managed to change to ClassLabel :)

Hey @lhoestq I was able to fix it !! I think the same errors appeared on circleCI and now it's hopefully ready to be merged?

added restaurants reviews in Arabic for sentiment analysis tasks


Add DaNe dataset

Thanks, this looks great!

For the code quality test, it looks like `flake8` is throwing the error, so you can tun `flake8 datasets` locally and fix the errors it points out until it passes


The `RemoteDatasetTest` CI fail is fixed on master so it's fine

Added  memat : Xhosa-English parallel corpora
for more info : http://opus.nlpl.eu/memat.php


Add Mocha dataset

More information: https://allennlp.org/mocha


Linguisticprobing

dataset contribution

Contribution to a dataset script

Thanks for your contribution, @sileod.

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

As you already created this dataset under your organization namespace (https://huggingface.co/datasets/metaeval/linguisticprobing), I think we can safely close this PR.

We would suggest you add a dataset card with the YAML tags, to make it searchable and discoverable.

Please, feel free to tell us if you need some help.

Adding Linguistic probing datasets from
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties
 https://www.aclweb.org/anthology/P18-1198/


adding TabFact dataset

FYI you guys are on GitHub's homepage 😍

<img width="1589" alt="Screenshot 2020-12-09 at 12 34 28" src="https://user-images.githubusercontent.com/326577/101624883-a0ecc700-39e8-11eb-8a97-11af0d036536.png">

Adding TabFact: A Large-scale Dataset for Table-based Fact Verification.

https://github.com/wenhuchen/Table-Fact-Checking

- The tables are stored as individual csv files, so need to download 16,573 🤯  csv files. As a result the `datasets_infos.json` file is huge (6.62 MB).
- Original dataset has nested structure where, where table is one example and each table has multiple statements,
flattening the structure here so that each statement is one example.


Add Roman Urdu dataset

This PR adds the [Roman Urdu dataset](https://archive.ics.uci.edu/ml/datasets/Roman+Urdu+Data+Set#).


Add BBC Hindi NLI Dataset

Hi @avinsit123 !
Did you manage to rename the dataset and apply the suggestion I mentioned for the data fields ?
Feel free to ping me when you're ready for a review :)

Hi @avinsit123 ! Have you had a chance to take a look at my suggestions ?
Let me know if you have questions or if I can help

@lhoestq sorry I completely forgot about this pr. I will complete it ASAP.

@lhoestq I have fixed the code to resolve all your comments. Pls do check. I also don't seem to know why the CI tests are failing as I ran all the tests in CONTRIBUTING.md on my local pc and they passed.

@lhoestq thanks for ur patient review :) . I also wish to add similar 3 more NLI hindi datasets. Hope to do within this week.

@lhoestq would this be merged to master?

# Dataset Card for BBC Hindi NLI Dataset

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)

## Dataset Description

- HomePage : https://github.com/midas-research/hindi-nli-data
- Paper : "https://www.aclweb.org/anthology/2020.aacl-main.71"
- Point of Contact : https://github.com/midas-research/hindi-nli-data

### Dataset Summary

- Dataset for Natural Language Inference in Hindi Language. BBC Hindi Dataset consists of textual-entailment pairs.
- Each row of the Datasets if made up of 4 columns - Premise, Hypothesis, Label and Topic.
- Context and Hypothesis is written in Hindi while Entailment_Label is in English.
- Entailment_label is of 2 types - entailed and not-entailed.
- Dataset can be used to train models for Natural Language Inference tasks in Hindi Language.
[More Information Needed]

### Supported Tasks and Leaderboards

- Natural Language Inference for Hindi

### Languages

Dataset is in Hindi

## Dataset Structure

- Data is structured in TSV format.
- Train and Test files are in seperate files

### Dataset Instances

An example of 'train' looks as follows.

```
{'hypothesis': 'यह खबर की सूचना है|', 'label': 'entailed', 'premise': 'गोपनीयता की नीति', 'topic': '1'}

```
### Data Fields

- Each row contatins 4 columns - Premise, Hypothesis, Label and Topic.

### Data Splits

- Train : 15553
- Valid : 2581
- Test : 2593

## Dataset Creation

- We employ a recasting technique from Poliak et al. (2018a,b) to convert publicly available BBC Hindi news text classification datasets in Hindi and pose them as TE problems
- In this recasting process, we build template hypotheses for each class in the label taxonomy
- Then, we pair the original annotated sentence with each of the template hypotheses to create TE samples.
- For more information on the recasting process, refer to paper "https://www.aclweb.org/anthology/2020.aacl-main.71"

### Source Data

Source Dataset for the recasting process is the BBC Hindi Headlines Dataset(https://github.com/NirantK/hindi2vec/releases/tag/bbc-hindi-v0.1)

#### Initial Data Collection and Normalization

-  BBC Hindi News Classification Dataset contains 4, 335 Hindi news headlines tagged across 14 categories: India, Pakistan,news, International, entertainment, sport, science, China, learning english, social, southasia, business, institutional, multimedia
-  We processed this dataset to combine two sets of relevant but low prevalence classes.
- Namely, we merged the samples from Pakistan, China, international, and southasia as one class called international.
- Likewise, we also merged samples from news, business, social, learning english, and institutional as news.
- Lastly, we also removed the class multimedia because there were very few samples.

#### Who are the source language producers?

Pls refer to this paper: "https://www.aclweb.org/anthology/2020.aacl-main.71"

### Annotations

#### Annotation process

Annotation process has been described in Dataset Creation Section.

#### Who are the annotators?

Annotation is done automatically.

### Personal and Sensitive Information

No Personal and Sensitive Information is mentioned in the Datasets.

## Considerations for Using the Data

Pls refer to this paper: https://www.aclweb.org/anthology/2020.aacl-main.71

### Discussion of Biases

Pls refer to this paper: https://www.aclweb.org/anthology/2020.aacl-main.71

### Other Known Limitations

No other known limitations

## Additional Information

Pls refer to this link: https://github.com/midas-research/hindi-nli-data

### Dataset Curators

It is written in the repo : https://github.com/avinsit123/hindi-nli-data that
- This corpus can be used freely for research purposes.
- The paper listed below provide details of the creation and use of the corpus. If you use the corpus, then please cite the paper.
- If interested in commercial use of the corpus, send email to midas@iiitd.ac.in.
- If you use the corpus in a product or application, then please credit the authors and Multimodal Digital Media Analysis Lab - Indraprastha Institute of Information Technology, New Delhi appropriately. Also, if you send us an email, we will be thrilled to know about how you have used the corpus.
- Multimodal Digital Media Analysis Lab - Indraprastha Institute of Information Technology, New Delhi, India disclaims any responsibility for the use of the corpus and does not provide technical support. However, the contact listed above will be happy to respond to queries and clarifications.
- Rather than redistributing the corpus, please direct interested parties to this page
- Please feel free to send us an email:
  - with feedback regarding the corpus.
  - with information on how you have used the corpus.
  - if interested in having us analyze your data for natural language inference.
  - if interested in a collaborative research project.

### Licensing Information

Copyright (C) 2019 Multimodal Digital Media Analysis Lab - Indraprastha Institute of Information Technology, New Delhi (MIDAS, IIIT-Delhi).
Pls contact authors for any information on the dataset.

### Citation Information

```
    @inproceedings{uppal-etal-2020-two,
    title = "Two-Step Classification using Recasted Data for Low Resource Settings",
    author = "Uppal, Shagun  and
      Gupta, Vivek  and
      Swaminathan, Avinash  and
      Zhang, Haimin  and
      Mahata, Debanjan  and
      Gosangi, Rakesh  and
      Shah, Rajiv Ratn  and
      Stent, Amanda",
    booktitle = "Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing",
    month = dec,
    year = "2020",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.aacl-main.71",
    pages = "706--719",
    abstract = "An NLP model{'}s ability to reason should be independent of language. Previous works utilize Natural Language Inference (NLI) to understand the reasoning ability of models, mostly focusing on high resource languages like English. To address scarcity of data in low-resource languages such as Hindi, we use data recasting to create NLI datasets for four existing text classification datasets. Through experiments, we show that our recasted dataset is devoid of statistical irregularities and spurious patterns. We further study the consistency in predictions of the textual entailment models and propose a consistency regulariser to remove pairwise-inconsistencies in predictions. We propose a novel two-step classification method which uses textual-entailment predictions for classification task. We further improve the performance by using a joint-objective for classification and textual entailment. We therefore highlight the benefits of data recasting and improvements on classification performance using our approach with supporting experimental results.",
}
```


Add dataset XhosaNavy English -Xhosa

Add dataset XhosaNavy English -Xhosa
More info : http://opus.nlpl.eu/XhosaNavy.php


add telugu-news corpus

Adding Telugu News Corpus to datasets.


Add BSD

Glad to have more Japanese data! Couple of comments:
- the abbreviation might confuse some people as there is also an OPUS BSD corpus, would you mind renaming it as `bsd_ja_en`?
- `flake8` is throwing some errors, you can run it locally (`flake8 datasets`) and fix what it tells you until it's happy :)
- We're not using `os.path.join` for URLs as it's unstable across systems (introduces backslashes on Windows). Can you write the URLs explicitly instead?

Thanks!

> Fantastic, looks great!

Thanks for your help @yjernite, really appreciate it!

The RemoteDatasetTest is fixed on master so it's fine

This PR adds BSD, the Japanese-English business dialogue corpus by
[Rikters et al., 2020](https://www.aclweb.org/anthology/D19-5204.pdf).


Opus sardware

Added Opus sardware dataset for machine translation English to Sardinian.
for more info : http://opus.nlpl.eu/sardware.php


Adding dataset for proto_qa in huggingface datasets library

Added dataset for ProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoning
Followed all steps for adding a new dataset.


hindi discourse analysis dataset commit

That's a great dataset to have! We need a couple more things to be good to go:
- you should `make style` and `flake8 datasets` before pushing to make the code quality check happy :)
- the dataset will need some dummy data which you should be able to auto-generate and test locally: https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#automatically-add-code-metadata
- there's some good information in your current README, but we need the format to follow the template [here](https://github.com/huggingface/datasets/blob/master/templates/README.md) and to have YAML tags at the top, as described in the guide: https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#tag-the-dataset-and-write-the-dataset-card

LEt us know if you need any help!

Hi @yjernite
I was successfully able to generate the dataset_info.json file using the command
python datasets-cli test datasets/<your-dataset-folder> --save_infos --all_configs

But unfortunately, could not generate the dummy data

While running the command
python datasets-cli dummy_data datasets/<your-dataset-folder> --auto_generate
I got an error as

ValueError: Couldn't parse columns ['0', '1', '2', '3', '4', ......, '9982']. Maybe specify which json field must be used to read the data with --json_field <my_field>.

The thing is the dataset I am trying to upload is of the format
{
   '0': {'Story_no': 15, 'Sentence': ' गाँठ से साढ़े तीन रुपये लग गये, जो अब पेट में जाकर खनकते भी नहीं! जो तेरी करनी मालिक! ” “इसमें मालिक की क्या करनी है? ”', 'Discourse Mode': 'Dialogue'},
  '1': {'Story_no': story_no, 'Sentence': sentence, 'Discourse Mode': discourse_mode},
   .......,
  '9982': {'Story_no': story_no, 'Sentence': sentence, 'Discourse Mode': discourse_mode}
}

Can you please suggest any errors I am making in the  _generate_examples method?

Thanks!

The dummy data generator doesn't support this kind of json format yet.
Can you create the dummy data manually please ? You can get the instructions by running the
```
datasets-cli dummy_data ./datasets/dataset_name
```
command.

Hi, I created the dummy data manually but the tests are still failing it seems.
Can you suggest the format of JSON which is supported by dummy data generator?
I will have to modify my _generate_examples method accordingly.
Please advice on the same.
Thanks much.

Can you run `make style` to format the code for the CI please ?

Also about the dummy data, here is how to generate them:

We need a dummy_data.zip file in ./datasets/hindiDiscourse/dummy/1.0.0 (or replace hindiDiscourse by hindi_discourse since we have to rename the folder anyway)
To create the zip file, first go in this directory and create a folder named dummy_data.
Then inside the dummy_data folder create a file `discourse_dataset.json` and fill it with something like 5 examples.
Finally zip the dummy_data folder to end up with the dummy_data.zip file

Once it's done you can check if the dummy data test passes with
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_hindi_discourse
```

If it passes you can then remove the dummy_data folder to keep only the dummy_data.zip file

Hi @duttahritwik did you manage to make the dummy data ?
Feel free to ping me if you have questions or if we can help

The error `tests/test_file_utils.py::TempSeedTest::test_tensorflow` just appeared because of tensorflow's update.
Once it's fixed on master we'll be free to merge this one


adding psc dataset


adding dyk dataset


Fix typo in the comment in _info function


adding polemo2 dataset


Vinay/add/telugu books

Real data tests are failing as this dataset needs to be manually downloaded


Add LINNAEUS


Add Species-800

thanks @lhoestq ! I probably need to do the same change in the `SplitGenerator`s (lines 107, 110 and 113). I'll open a new PR for that

Yes indeed ! Good catch 👍
Feel free to open a PR and ping me

Hi , theres a issue pulling species_800 dataset , throws  google drive  error

error:

```
raise ConnectionError(f"Couldn't reach {url} ({repr(head_error)})")
ConnectionError: Couldn't reach https://drive.google.com/u/0/uc?id=1OletxmPYNkz2ltOr9pyT0b0iBtUWxslh&export=download/ (ReadTimeout(ReadTimeoutError("HTTPSConnectionPool(host='drive.google.com', port=443): Read timed out. (read timeout=10)")))
```
code:
```
from datasets import load_dataset

dataset = load_dataset("species_800")
```

Hi @obonyojimmy! I am running the same commands and they work for me. Did you check your internet connection?


Add JFLEG

Hi @j-chim ! You're right it does feel redundant: your option works better, but I'd even suggest having the references in a Sequence feature, which you can declare as:
```
	    features=datasets.Features(
                {
                    "sentence": datasets.Value("string"),
                    "corrections": datasets.Sequence(datasets.Value("string")),
                }
            ),
```

To create the dummy data, you just need to tell the generator which files it should use, which you can do with:
`python datasets-cli dummy_data datasets/<your-dataset-folder> --auto_generate --match_text_files "train*,dev*,test*"`

Many thanks for this @yjernite! I've incorporated your feedback and sorted out the dummy data.

This PR adds [JFLEG ](https://www.aclweb.org/anthology/E17-2037/), an English grammatical error correction benchmark.

The tests were successful on real data, although it would be great if I can get some guidance on the **dummy data**. Basically, **for each source sentence there are 4 possible gold standard target sentences**. The original dataset comprise files in a flat structure, labelled by split then by source/target (e.g., dev.src, dev.ref0, ..., dev.ref3). Not sure what is the best way of adding this.

I imagine I can treat each distinct source-target pair as its own split? But having so many copies of the source sentence feels redundant, and it would make it less convenient to end-users who might want to access multiple gold standard targets simultaneously.


Add the Winograd Schema Challenge

Adds the Winograd Schema Challenge, including configs for the more canonical wsc273 as well as wsc285 with 12 new examples.

- https://cs.nyu.edu/faculty/davise/papers/WinogradSchemas/WS.html

The data format was a bit of a nightmare but I think I got it to a workable format.


Fix PerSenT

New PR for dataset PerSenT


Add GitHub version of ETH Py150 Corpus

The `RemoteDatasetTest` is fixed on master so it's fine

thanks for rebasing :)

CI is green now, merging

Add the redistributable version of **ETH Py150 Corpus**


Add Urdu Sentiment Corpus (USC).

@lhoestq have made the suggested changes in the README file.

@lhoestq Created a new PR #1231 with only the relevant files.
closing this one :)

Added Urdu Sentiment Corpus. More details about the dataset over here.


Add ReFreSD dataset

Cool dataset! Replying in-line:

> This PR adds the **ReFreSD dataset**.
> The original data is hosted [on this github repo](https://github.com/Elbria/xling-SemDiv) and we use the `REFreSD_rationale` to expose all the data.
>
> Need feedback on:
>
> * I couldn't generate the dummy data. The file we download is a tsv file, but without extension, I suppose this is the problem. I'm sure there is a simple trick to make this work.

you can use `--match_text_files` in the dummy data generation:
`python datasets-cli dummy_data datasets/refresd --auto_generate --match_text_files "REFreSD_rationale"`

> * The feature names.
>
>   * I don't know if it's better to stick to the classic `sentence1`, `sentence2` or to `sentence_en`, `sentence_fr` to be more explicit.

It would actually be even better to use the `Translation` feature here to replace best:
`"sentence_pair": datasets.Translation(languages=['en', 'fr']),`

Then during `_generate_examples` this filed should look like"
`{"sentence_pair": {"fr": french, "en": english}}`

>   * There is a binary label (called `label`, no problem here), and a 3-class label called `#3_labels` in the original tsv. I changed it to `all_labels` but I'm sure there is better.
Looks good!

> * The rationales are lists of integers, extracted as a string at first. I wonder what's the best way to treat them, any idea? Also, I couldn't manage to make a `Sequence` of `int8` but I'm sure I've missed something simple.

Having the feature declared as `"rationale_en": datasets.Sequence(datasets.Value("int32"))` should work

>
> Thanks in advance

Hope that helps you out! Don't forget to `make style`, rebase from master, and run all the tests before pushing again! You will also need to add a `README.md` as described in the guide:
https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#tag-the-dataset-and-write-the-dataset-card

Thanks a lot for the answer, that does help a lot !
I opened a PR for a License in the original repo so I was waiting for that for the model card. If there is no news on Monday, I'll add it without License.

Looks good! It looks like it might need a rebase to pass the tests. Once you do that, should be good to go!

This PR adds the **ReFreSD dataset**.
The original data is hosted [on this github repo](https://github.com/Elbria/xling-SemDiv) and we use the `REFreSD_rationale` to expose all the data.

Need feedback on:
- I couldn't generate the dummy data. The file we download is a tsv file, but without extension, I suppose this is the problem. I'm sure there is a simple trick to make this work.
- The feature names.
  - I don't know if it's better to stick to the classic `sentence1`, `sentence2` or to `sentence_en`, `sentence_fr` to be more explicit.
  - There is a binary label (called `label`, no problem here), and a 3-class label called `#3_labels` in the original tsv. I changed it to `all_labels` but I'm sure there is better.
- The rationales are lists of integers, extracted as a string at first. I wonder what's the best way to treat them, any idea? Also, I couldn't manage to make a `Sequence` of `int8` but I'm sure I've missed something simple.

Thanks in advance


updated after the class name update

@lhoestq <---


add wmt mlqe 2020 shared task

re-created in #1218 because this was too messy

First commit for Shared task 1 (wmt_mlqw_task1) of WMT20 MLQE (quality estimation of machine translation)
Note that I copied the tags in the README for only one (of the 7 configurations): `en-de`.
There is one configuration for each pair of languages.


minor change in description in paws-x.py  and updated dataset_infos


added paws

Updating README and tags for dataset card in a while


adding xquad-r dataset


Adding XQUAD-R Dataset


Add Urdu Sentiment Corpus (USC).


adding discovery


Adding initial version of cord-19 dataset

Hi @ggdupont !
Have you had a chance to take a look at my suggestions ?
Feel free to ping me if you have questions or when you're ready for a review

> Hi @ggdupont !
> Have you had a chance to take a look at my suggestions ?
> Feel free to ping me if you have questions or when you're ready for a review

Yes I did, just busy period (and no time on weekend right now ;-) )

With some delay, reduced the dummy data and had t rebase

Thanks !

It looks like the rebase messed up the github diff for this PR (2.000+ files changed)
Could you create another branch and another PR please ?

Initial version only reading the metadata in CSV.

### Checklist:
- [x] Create the dataset script /datasets/my_dataset/my_dataset.py using the template
- [x] Fill the _DESCRIPTION and _CITATION variables
- [x] Implement _infos(), _split_generators() and _generate_examples()
- [x] Make sure that the BUILDER_CONFIGS class attribute is filled with the different configurations of the dataset and that the BUILDER_CONFIG_CLASS is specified if there is a custom config class.
- [x] Generate the metadata file dataset_infos.json for all configurations
- [x] Generate the dummy data dummy_data.zip files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card README.md using the template and at least fill the tags
- [x] Both tests for the real data and the dummy data pass.

### TODO:
- [x] add more metadata
- [x] add full text
- [x] add pre-computed document embedding


Add xquad-r dataset


Add wikiqaar dataset

Arabic Wiki Question Answering Corpus.


Adding babi dataset

This is ok now @lhoestq

I've included the tweak to `dummy_data` to only use the data transmitted to `_generate_examples` by default (it only do that if it can find at least one path to an existing file in the `gen_kwargs` and this can be unactivated with a flag).

Should I extract it in another PR or is it ok like this?

Nice !
Could you add the dummy data generation trick in another PR ?
I think we can also extend it to make it work not only with data files paths but also with data directories (sometimes it's one of the parent directory that is passed to gen_kwargs, not the actual path to the file).
This will help a lot to make the dummy data lighter !

This PR can be closed due to #2053 @lhoestq

Adding the English version of bAbI.

Samples are taken from ParlAI for consistency with the main users at the moment.

Supersede #945 (problem with the rebase) and adresses the issues mentioned in the review (dummy data are smaller now and code comments are fixed).


Add Urdu fake news dataset.

@lhoestq looks like a lot of files were updated... shall I create a new PR?

Hi @chaitnayabasava ! you can try rebasing and see if that fixes the number of files changed, otherwise please do open a new PR with only the relevant files and close this one :)

Created a new PR #1230.
closing this one :)

Added Urdu fake news dataset. More information about the dataset can be found here.


Add Xitsonga Ner

looks like this PR includes changes about many files other than the ones related to xitsonga NER

could you create another branch and another PR please ?

Clean Xitsonga Ner PR


adding cdt dataset

the `ms_terms` formatting CI fails is fixed on master


Add Urdu fake news.


adding cdt dataset


Add conda environment activation

Added activation of Conda environment before installing.


Add Google Great Code Dataset


Add Tashkeela dataset

> Sorry @lhoestq for the trouble, sometime I forget to change the names :/

haha it's ok ;)

Arabic Vocalized Words Dataset.


Fix incorrect MRQA train+SQuAD URL

Thanks ! could you regenerate the dataset_infos.json file ?

```
datasets-cli test ./datasets/mrqa --save_infos --all_configs --ignore_verifications
```

also cc @VictorSanh

> Thanks ! could you regenerate the dataset_infos.json file ?
>
> ```
> datasets-cli test ./datasets/mrqa --save_infos --all_configs --ignore_verifications
> ```
>
> also cc @VictorSanh

Updated the `dataset_infos.json` file.

Fix issue #1115


add dbpedia_14 dataset

Hi @hfawaz, this week we are doing the 🤗 `datasets` sprint (see some details [here](https://discuss.huggingface.co/t/open-to-the-community-one-week-team-effort-to-reach-v2-0-of-hf-datasets-library/2176)).

Nothing more to do on your side but it means that if you register on the thread I linked above, you can have some goodies for the present dataset that you have already added (and a special goodie if you want to spend more time and add 2 other datasets as well).

If you want to join, just tell me (or post on the thread on the HuggingFace forum: https://discuss.huggingface.co/t/open-to-the-community-one-week-team-effort-to-reach-v2-0-of-hf-datasets-library/2176)

Hello @thomwolf
Thanks for the feedback and for this invitation, indeed I would be glad to join you guys (you can add me).
I will see if I have the time to implement a couple of datasets.
Cheers!

@hfawaz invited you to the slack with your uha email.

Check your spam folder if you can't find the invitation :)

Oh thanks, but can you invite me on my gmail: hassanismailfawaz@gmail.com
UHA is my old organization, I haven't had the time to update my online profiles yet.
Thank you

This dataset corresponds to the DBpedia dataset requested in https://github.com/huggingface/datasets/issues/353.


Incorrect URL for MRQA SQuAD train subset

https://github.com/huggingface/datasets/blob/4ef4c8f8b7a60e35c6fa21115fca9faae91c9f74/datasets/mrqa/mrqa.py#L53

The URL for `train+SQuAD` subset of MRQA points to the dev set instead of train set. It should be `https://s3.us-east-2.amazonaws.com/mrqa/release/v2/train/SQuAD.jsonl.gz`.


Add sesotho ner corpus

Clean Sesotho PR


add qed

adding QED: Dataset for Explanations in Question Answering
https://github.com/google-research-datasets/QED
https://arxiv.org/abs/2009.06354


Initial version of cord-19 dataset from AllenAI with only the abstract

Initial version only reading the metadata in CSV.

### Checklist:
- [x] Create the dataset script /datasets/my_dataset/my_dataset.py using the template
- [x] Fill the _DESCRIPTION and _CITATION variables
- [x] Implement _infos(), _split_generators() and _generate_examples()
- [x] Make sure that the BUILDER_CONFIGS class attribute is filled with the different configurations of the dataset and that the BUILDER_CONFIG_CLASS is specified if there is a custom config class.
- [x] Generate the metadata file dataset_infos.json for all configurations
- [x] Generate the dummy data dummy_data.zip files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card README.md using the template and at least fill the tags
- [ ] Both tests for the real data and the dummy data pass.

### TODO:
- [ ] add more metadata
- [ ] add full text
- [ ] add pre-computed document embedding


Add Siswati Ner corpus

Clean Siswati PR


Using a feature named "_type" fails with certain operations

Thanks for reporting !

Indeed this is a keyword in the library that is used to encode/decode features to a python dictionary that we can save/load to json.
We can probably change `_type` to something that is less likely to collide with user feature names.
In this case we would want something backward compatible though.

Feel free to try a fix and open a PR, and to ping me if I can help :)

A column named `_type` leads to a `TypeError: unhashable type: 'dict'` for certain operations:
```python
from datasets import Dataset, concatenate_datasets

ds = Dataset.from_dict({"_type": ["whatever"]}).map()
concatenate_datasets([ds])
# or simply
Dataset(ds._data)
```
Context: We are using datasets to persist data coming from elasticsearch to feed to our pipeline, and elasticsearch has a `_type` field, hence the strange name of the column.

Not sure if you wish to support this specific column name, but if you do i would be happy to try a fix and provide a PR. I already had a look into it and i think the culprit is the `datasets.features.generate_from_dict` function. It uses the hard coded `_type` string to figure out if it reached the end of the nested feature object from a serialized dict.

Best wishes and keep up the awesome work!


add woz_dialogue

Adding Wizard-of-Oz task oriented dialogue dataset
https://github.com/nmrksic/neural-belief-tracker/tree/master/data/woz
https://arxiv.org/abs/1604.04562


Add Sepedi NER corpus

Finally a clean PR for Sepedi


Add arsentd_lev dataset

thanks ! can you also regenerate the dataset_infos.json file please ?

Add The Arabic Sentiment Twitter Dataset for Levantine dialect (ArSenTD-LEV)

Paper: [ArSentD-LEV: A Multi-Topic Corpus for Target-based Sentiment Analysis in Arabic Levantine Tweets](https://arxiv.org/abs/1906.01830)
Homepage: http://oma-project.com/


Add Urdu fake news


add xquad_r dataset

looks like this PR includes changes in many files than the ones for xquad_r, could you create a new branch and a new PR ?


add TLC

Added TLC dataset


Add support to download kaggle datasets

enhancement

New feature or request

Hey, I think this is great idea. Any plan to integrate kaggle private datasets loading to `datasets`?

The workflow for downloading a Kaggle dataset and turning it into an HF dataset is pretty simple:
```python
!kaggle datasets download -p path
ds = load_dataset(path)
```

Native support would make our download logic even more complex, and I don't think this is a good idea considering this particular feature is not requested often.

PS: Kaggle should integrate their API with `fsspec` to allow us to use a common interface if they are interested in tighter integrations

We can use API key


Add retries to download manager

New feature or request


Add Wikicorpus dataset

Add dataset.


Urdu fake news

Added Bend the Truth urdu fake news dataset. More inforation here.


Add tamilmixsentiment data


Add ToTTo Dataset

Adds a brand new table to text dataset: https://github.com/google-research-datasets/ToTTo


Add MSRA NER labels

Fixes #940


FIX matinf link in ADD_NEW_DATASET.md


Add TupleInf Open IE Dataset

Errors are in the CI are not related to this PR (RemoteDatasetError)
the CI is fixed on master so it's fine

@lhoestq Added the dataset card. Please let me know if more information needs to be added.

For more information: https://allenai.org/data/tuple-ie


add urdu fake news dataset

Added Urdu fake news dataset. The dataset can be found here.


Add NCBI Disease Corpus dataset


Add Coached Conversation Preference Dataset

Adding [Coached Conversation Preference Dataset](https://research.google/tools/datasets/coached-conversational-preference-elicitation/)


Add Google wellformed query dataset

This pull request will add Google wellformed_query dataset. Link of dataset is https://github.com/google-research-datasets/query-wellformedness


add thaisum

ThaiSum, a large-scale corpus for Thai text summarization obtained from several online news websites namely Thairath, ThaiPBS, Prachathai, and The Standard. This dataset consists of over 350,000 article and summary pairs written by journalists. We evaluate the performance of various existing summarization models on ThaiSum dataset and analyse the characteristic of the dataset to present its difficulties.


add sharc_modified

Adding modified ShARC dataset https://github.com/nikhilweee/neural-conv-qa


add xquad_r dataset


Add Big Patent dataset

@lhoestq reduced the dummy data size to around 19MB in total and added the dataset card.

@lhoestq so I ended up removing all the nested JSON objects in the gz datafile and keep only one object with minimal content: `{"publication_number": "US-8230922-B2", "abstract": "dummy abstract", "application_number": "US-201113163519-A", "description": "dummy description"}`.

They're reduced to 35KB in total (2.5KB per domain and 17.5KB for all domains), hopefully, they're small enough.

* More info on the dataset: https://evasharma.github.io/bigpatent/
* There's another raw version of the dataset available from tfds. However, they're quite large so I don't have the resources to fully test all the configs for that version yet. We'll try to add it later.


adding cdt dataset

> Thanks for adding this one !
>
> I left a few comments
>
> after the change you'll need to regenerate the dataset_infos.json file as well

dataset_infos.json regenerated

looks like this PR includes changes to many files other that the ones for CDT
could you create another branch and another PR please ?

- **Name:** *Cyberbullying Detection Task*
- **Description:** *The Cyberbullying Detection task was part of 2019 edition of PolEval competition. The goal is to predict if a given Twitter message contains a cyberbullying (harmful) content.*
- **Data:** *https://github.com/ptaszynski/cyberbullying-Polish*
- **Motivation:** *The KLEJ benchmark (Kompleksowa Lista Ewaluacji Językowych) is a set of nine evaluation tasks for the Polish language understanding.*


add mutual friends conversational dataset

Mutual friends dataset
WIP

TODO:
- scenario_kbs (bug with pyarrow conversion)
- download from codalab checksums bug


adding cdsc dataset

- **Name**: *cdsc (domains: cdsc-e & cdsc-r)*
- **Description**: *Polish CDSCorpus consists of 10K Polish sentence pairs which are human-annotated for semantic relatedness and entailment. The dataset may be used for the evaluation of compositional distributional semantics models of Polish. The dataset was presented at ACL 2017. Please refer to the Wróblewska and Krasnowska-Kieraś (2017) for a detailed description of the resource.*
- **Data**: *http://2019.poleval.pl/index.php/tasks/*
- **Motivation**: *The KLEJ benchmark (Kompleksowa Lista Ewaluacji Językowych) is a set of nine evaluation tasks for the Polish language understanding.*


Add the multilingual Exams dataset

Will slim down the dummy files in the morning

https://github.com/mhardalov/exams-qa

`multilingual` configs have all languages mixed together

`crosslingual` mixes the languages for test but separates them for train and dec, so I've made one config per language for train/dev data and one config with the joint test set


Myanmar news dataset

Add news topic classification dataset in Myanmar / Burmese languagess

This data was collected in 2017 by Aye Hninn Khine , and published on GitHub with a GPL license
https://github.com/ayehninnkhine/MyanmarNewsClassificationSystem


Add Knowledge-Enhanced Language Model Pre-training (KELM)

Adds the KELM dataset.

- Webpage/repo: https://github.com/google-research-datasets/KELM-corpus
- Paper: https://arxiv.org/pdf/2010.12688.pdf


Add WikiANN NER dataset

Dataset card added, so ready for review!

This PR adds the full set of 176 languages from the balanced train/dev/test splits of WikiANN / PAN-X from: https://github.com/afshinrahimi/mmner

Until now, only 40 of these languages were available in `datasets` as part of the XTREME benchmark

Courtesy of the dataset author, we can now download this dataset from a Dropbox URL without needing a manual download anymore 🥳, so at some point it would be worth updating the PAN-X subset of XTREME as well 😄

Link to gist with some snippets for producing dummy data: https://gist.github.com/lewtun/5b93294ab6dbcf59d1493dbe2cfd6bb9

P.S. @yjernite I think I was confused about needing to generate a set of YAML tags per config, so ended up just adding a single one in the README.


nkjp-ner

- **Name:** *nkjp-ner*
- **Description:** *The NKJP-NER is based on a human-annotated part of NKJP. We extracted sentences with named entities of exactly one type. The task is to predict the type of the named entity.*
- **Data:** *https://klejbenchmark.com/tasks/*
- **Motivation:** *The KLEJ benchmark (Kompleksowa Lista Ewaluacji Językowych) is a set of nine evaluation tasks for the Polish language understanding.*


add AJGT dataset

Arabic Jordanian General Tweets.


Added glucose dataset

This PR adds the [Glucose](https://github.com/ElementalCognition/glucose) dataset.


quac quac / coin coin

Add QUAC (Question Answering in Context)
I linearized most of the dictionnaries to lists.
Referenced to the authors' datasheet for the dataset card.
🦆🦆🦆
Coin coin


adding cleaned verion of E2E NLG

Found at: https://github.com/tuetschek/e2e-cleaning


Swedish MT STS-B

Added a Swedish machine translated version of the well known STS-B Corpus


Add DialogRE dataset

Adding the [DialogRE](https://github.com/nlpdata/dialogre) dataset Version 2.

- All tests passed successfully.


actually uses the previously declared VERSION on the configs in the template


add xlrd to test package requirements

Adds `xlrd` package to the test requirements to handle scripts that use `pandas` to load excel files


add conv_ai

This one will make @thomwolf reminisce ;)

Adding ConvAI dataset https://github.com/DeepPavlov/convai/tree/master/2017


Test


Add Pubmed (citation + abstract) dataset (2020).

LGTM! ftp addition looks fine but maybe have a look @thomwolf ?

It's not finished yet, I need to run the tests on the full dataset (it was running this weekend, there is an error somewhere deep)

@yjernite Ready for review !
@thomwolf

So I tried to follow closely the original format that means I still had to drop information (namely tags on elements are simply discarded for now but they don't seem to carry critical information).
Some elements are also discarded they tend to not come up often:
 - The most notable is Author affiliation, which seems to be all over the place in terms of what it look meaning it's hard to actually get a consistent format.
 - Journal is the same, all the elements in there can be wildly different so I decided to drop it for now instead of trying to figure out a way to have a common presentation. (the DOI and medline ID are present so it can be recovered).

I think this PR could go as it but we probably should add a way to get easier information to use with a config.
For instance `{"title": "string", "abstract": "string", "authors": List[str], "substances": List[str]}` maybe ? (substances for instance is a tricky one, some substances only have an international identifier, some have simply a common name, some both)

It's relatively easy to do I think it's mostly discarding other fields and renaming some deep structure into a flat one.

Look ok to me but @lhoestq is the master on the Download Manager side


add xquad-r dataset


Add ChrEn

> I just saw your PR actually ^^

Somehow that still doesn't work, lmk if you have any ideas.

Adding the Cherokee English machine translation dataset of https://github.com/ZhangShiyue/ChrEn


Not support links with 302 redirect

bug

Something isn't working

enhancement

New feature or request

Hi !
This kind of links is now supported by the library since #1316

> Hi !
> This kind of links is now supported by the library since #1316

I updated links in TLC datasets to be the github links in this pull request
 https://github.com/huggingface/datasets/pull/1737

Everything works now. Thank you.

I have an issue adding this download link https://github.com/jitkapat/thailitcorpus/releases/download/v.2.0/tlc_v.2.0.tar.gz

it might be because it is not a direct link (it returns 302 and redirects to aws that returns 403 for head requests).

```
r.head("https://github.com/jitkapat/thailitcorpus/releases/download/v.2.0/tlc_v.2.0.tar.gz", allow_redirects=True)
# <Response [403]>
```


Add the Ud treebank

This PR adds the 183 datasets in 104 languages of the UD Treebank.


Add KorNLU dataset

Nice thank you !
Can you regenerate the dataset_infos.json file ? Since we changed the features we must update it

Then I think we'll be good to merge :)

Added Korean NLU datasets. The link to the dataset can be found [here](https://github.com/kakaobrain/KorNLUDatasets) and the paper can be found [here](https://arxiv.org/abs/2004.03289)

**Note**: The MNLI tsv file is broken, so this code currently excludes the file. Please suggest other alternative if any @lhoestq

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data


add labr dataset

Arabic Book Reviews dataset.


Fix squad V2 metric script

The script with changes is used and tested in [#8924](https://github.com/huggingface/transformers/pull/8924). It gives the same results as the old `evaluate_squad` function when used on the same predictions.

The current squad v2 metric doesn't work with the squad (v1 or v2) datasets. The script is copied from `squad_evaluate` in transformers that requires the labels (with multiple answers) to be like this:
```
references = [{'id': 'a', 'answers': [
    {'text': 'Denver Broncos', 'answer_start': 177},
    {'text': 'Denver Broncos', 'answer_start': 177}
]}]
```
while the dataset had references like this:
```
references = [{'id': 'a', 'answers':
    {'text': ['Denver Broncos' 'Denver Broncos'], 'answer_start': [177, 177]}
}]
```

Using one or the other format fails with the current squad v2 metric:
```
from datasets import load_metric
metric = load_metric("squad_v2")
predictions = [{'id': 'a', 'prediction_text': 'Denver Broncos', 'no_answer_probability': 0.0}]
references = [{'id': 'a', 'answers': [
    {'text': 'Denver Broncos', 'answer_start': 177},
    {'text': 'Denver Broncos', 'answer_start': 177}
]}]
metric.compute(predictions=predictions, references=references)
```
fails as well as
```
from datasets import load_metric
metric = load_metric("squad_v2")
predictions = [{'id': 'a', 'prediction_text': 'Denver Broncos', 'no_answer_probability': 0.0}]
references = [{'id': 'a', 'answers':
    {'text': ['Denver Broncos' 'Denver Broncos'], 'answer_start': [177, 177]}
}]
metric.compute(predictions=predictions, references=references)
```

This is because arrow reformats the references behind the scene.

With this PR (tested locally), both the snippets up there work and return proper results.


Add TLC

I have reduced the size of the dummy file and added README sections as you suggested.

I have a little issue to run the test. It seems there is no failed case in my machine.

Thanks !
It looks like the PR includes changes to many other files than the ones of `tlc`, can you create another branch and another PR ?

Added TLC dataset


added paws-x dataset

Added paws-x dataset. Updating README and tags in the dataset card in a while


Adding TamilMixSentiment

looks like this pr incldues changes about many other files than the ones for tamilMixSentiment, could you create another branch and another PR ?


Add deal_or_no_dialog

Add deal_or_no_dialog Dataset

github: https://github.com/facebookresearch/end-to-end-negotiator
Paper: [Deal or No Deal? End-to-End Learning for Negotiation Dialogues](https://arxiv.org/abs/1706.05125)


Add hebrew-sentiment

@elronbandel it looks like something went wrong with the renaming, as the old files are still in the PR. Can you `git rm datasets/hebrew-sentiment` ?

This is the old version of the data.
Here is the fixed version.
https://github.com/OnlpLab/Hebrew-Sentiment-Data

I hope I would find time to open a PR. I think it supposed to be only to change the data path

Cool ! Sure feel free to open a PR if you have some time :) and feel free to ping me for review or if you have questions

hebrew-sentiment dataset is ready! (including tests, tags etc)


Add dataset - SemEval 2014 - Task 1

Added the dataset card.
Requesting another review.

Adding the dataset of SemEval 2014 Task 1

Found the dataset under the shared Google Sheet > Recurring Task Datasets
Task Homepage - https://alt.qcri.org/semeval2014/task1

Thank you!


Fix dataset URL and file names, and add column name in "Social Bias Frames" dataset

# Why I did
When I use "social_bias_frames" datasets in this library, I got 404 Errors.
So, I fixed this error and another some problems that I faced to use the dataset.

# What I did
* Modify this dataset URL
* Modify this dataset file names
* Add a "dataSource" column

Thank you!


add sharc dataset

This PR adds the ShARC dataset.

More info:
https://sharc-data.github.io/index.html


Add Facebook SimpleQuestionV2

I think @thomwolf may also be working on this one as part of the Babi benchmark in #945

Add simple questions v2: https://research.fb.com/downloads/babi/


Add GoEmotions

Whoops, didn't mean for that to be merged yet (my bad). I'm reaching out to the authors since we'd like their feedback on the best way to have the `author` field anonymized or removed. Will send a patch once they get back to me.

Adds the GoEmotions dataset, a nice emotion classification dataset with 27 (multi-)label annotations on reddit comments. Includes both a large raw version and a narrowed version with predefined train/test/val splits, which I've included as separate configs with the latter as a default.

- Webpage/repo: https://github.com/google-research/google-research/tree/master/goemotions
- Paper: https://arxiv.org/abs/2005.00547


Add siswati ner corpus


Adding NCHLT dataset


Add KorNLU

the CI error about `social_bias_frames` is fixed on master so it's fine

looks like this PR includes many changes to other files that the ones related to KorNLU
Could you create another branch and another PR please ?


Dataset.map() turns tensors into lists?

A solution is to have the tokenizer return a list instead of a tensor, and then use `dataset_tok.set_format(type = 'torch')` to convert that list into a tensor. Still not sure if bug.

It is expected behavior, you should set the format to `"torch"` as you mentioned to get pytorch tensors back.
By default datasets returns pure python objects.

I apply `Dataset.map()` to a function that returns a dict of torch tensors (like a tokenizer from the repo transformers). However, in the mapped dataset, these tensors have turned to lists!

```import datasets
import torch
from datasets import load_dataset
print("version datasets", datasets.__version__)

dataset = load_dataset("snli", split='train[0:50]')

def tokenizer_fn(example):
    # actually uses a tokenizer which does something like:
    return {'input_ids': torch.tensor([[0, 1, 2]])}

print("First item in dataset:\n", dataset[0])
tokenized = tokenizer_fn(dataset[0])
print("Tokenized hyp:\n", tokenized)
dataset_tok = dataset.map(tokenizer_fn, batched=False,
        remove_columns=['label', 'premise', 'hypothesis'])
print("Tokenized using map:\n", dataset_tok[0])
print(type(tokenized['input_ids']), type(dataset_tok[0]['input_ids']))
dataset_tok = dataset.map(tokenizer_fn, batched=False,
                          remove_columns=['label', 'premise', 'hypothesis'])
print("Tokenized using map:\n", dataset_tok[0])
print(type(tokenized['input_ids']), type(dataset_tok[0]['input_ids']))
```

The output is:

```
version datasets 1.1.3
Reusing dataset snli (/home/tom/.cache/huggingface/datasets/snli/plain_text/1.0.0/bb1102591c6230bd78813e229d5dd4c7fbf4fc478cec28f298761eb69e5b537c)
First item in dataset:
 {'premise': 'A person on a horse jumps over a broken down airplane.', 'hypothesis': 'A person is training his horse for a competition.', 'label': 1}
Tokenized hyp:
 {'input_ids': tensor([[0, 1, 2]])}
Loading cached processed dataset at /home/tom/.cache/huggingface/datasets/snli/plain_text/1.0.0/bb1102591c6230bd78813e229d5dd4c7fbf4fc478cec28f298761eb69e5b537c/cache-fe38f449fe9ac46f.arrow
Tokenized using map:
 {'input_ids': [[0, 1, 2]]}
<class 'torch.Tensor'> <class 'list'>
```

Or am I doing something wrong?


Add xitsonga ner corpus

Look like this PR includes changes to many other files than the ones related to xitsonga NER.
Could you create another branch and another PR please ?


Add AMTTL Chinese Word Segmentation Dataset


Add TSAC: Tunisian Sentiment Analysis Corpus

github: https://github.com/fbougares/TSAC

paper: https://www.aclweb.org/anthology/W17-1307/


Add Big Patent dataset

Looks like this PR include changes about many other files than the ones related to big patent.
Could you create another branch and another PR ?

@lhoestq Just created a new PR here: https://github.com/huggingface/datasets/pull/1087

- More info on the dataset: https://evasharma.github.io/bigpatent/
- There's another raw version of the dataset available from tfds. However, they're quite large so I don't have the resources to fully test all the configs for that version yet. We'll try to add it later.
- ~Currently, there are no dummy data for this dataset yet as I'm facing some problems with generating them. I'm trying to add them later.~


Add SuperGLUE metric

Adds a new metric for the SuperGLUE benchmark (similar to the GLUE benchmark metric).


Add UN Universal Declaration of Human Rights (UDHR)

Universal declaration of human rights with translations in 464 languages and dialects.

- UN page: https://www.ohchr.org/EN/UDHR/Pages/UDHRIndex.aspx
- Raw data source: https://unicode.org/udhr/index.html

Each instance of the dataset corresponds to one translation of the document. Since there's only one instance per language (and because there are 500 languages so the dummy data would be messy), I opted to just include them all under the same single config. I wasn't able to find any kind of license so I just copied the copyright notice.

I was pretty careful careful generating the language tags so they _should_ all be correct & consistent BCP-47 codes per the docs.


Update ADD NEW DATASET

This PR adds a couple of detail on cloning/rebasing the repo.


add med_hop

This PR adds the MedHop dataset from the QAngaroo multi hop reading comprehension datasets

More info:
http://qangaroo.cs.ucl.ac.uk/index.html


Fix docs indentation issues

Yes @lhoestq, look at the docs site. For example, in https://huggingface.co/docs/datasets/add_dataset.html, look at the indentation in the code block under the sentence:
> Here are the features of the SQuAD dataset for instance, which is taken from the squad dataset loading script:

Replace tabs with spaces.


Add PerSenT

looks like this PR contains changes in many other files than the ones for PerSenT
can you create another branch and another PR ?

Added [Person's SentimenT](https://stonybrooknlp.github.io/PerSenT/) dataset.


add wiki_hop

Also the dummy data files are quite big (500KB)
If you could reduce that that would be nice (just look at the files inside and remove unecessary chunks of texts)
in general dummy data are just a few KB and we suggest to not get higher than 50KB

Having light dummy data makes the repo faster to clone

This PR adds the WikiHop dataset from the QAngaroo multi hop reading comprehension datasets

More info:
http://qangaroo.cs.ucl.ac.uk/index.html


add scb_mt_enth_2020

## scb-mt-en-th-2020: A Large English-Thai Parallel Corpus

The primary objective of our work is to build a large-scale English-Thai dataset for machine translation.
We construct an English-Thai machine translation dataset with over 1 million segment pairs, curated from various sources,
namely news, Wikipedia articles, SMS messages, task-based dialogs, web-crawled data and government documents.
Methodology for gathering data, building parallel texts and removing noisy sentence pairs are presented in a reproducible manner.
We train machine translation models based on this dataset. Our models' performance are comparable to that of
Google Translation API (as of May 2020) for Thai-English and outperform Google when the Open Parallel Corpus (OPUS) is
included in the training data for both Thai-English and English-Thai translation.
The dataset, pre-trained models, and source code to reproduce our work are available for public use.


Add support for ".txm" format

Neat! Looks like you need a rebase and then should be good to go :)

Hi ! yes sure :) can you just merge master into this branch before we merge ?

In dummy data generation, add support for XML-like ".txm" file format.

Also support filenames with additional compression extension: ".txm.gz".


IIT B English to Hindi machine translation dataset

Please note that this dataset is actually behind a form that one needs to fill. However, the link is direct. I'm not sure what should the approach be in this case.

also pinging @thomwolf
The dataset webpage returns a form when trying to download the dataset (form here : http://www.cfilt.iitb.ac.in/iitb_parallel/dataset.html).
However the url we get with the form can be used for the dataset script.
Should we ask the authors or use the urls this way ?

> also pinging @thomwolf
> The dataset webpage returns a form when trying to download the dataset (form here : http://www.cfilt.iitb.ac.in/iitb_parallel/dataset.html).
> However the url we get with the form can be used for the dataset script.
> Should we ask the authors or use the urls this way ?

I had discussion on this with @thomwolf . We have already sent email to author of this dataset.

Hi @spatil6 !
Any news from the authors ?

IIT B folks will add this dataset to repo.

Adding IIT Bombay English-Hindi Corpus dataset
more info : http://www.cfilt.iitb.ac.in/iitb_parallel/


add crows_pairs

Looks good to merge for me, can edit the dataset card later if required. Merging

This PR adds CrowS-Pairs datasets.

More info:
https://github.com/nyu-mll/crows-pairs/
https://arxiv.org/pdf/2010.00133.pdf


allegro_reviews dataset

- **Name:** *allegro_reviews*
- **Description:** *Allegro Reviews is a sentiment analysis dataset, consisting of 11,588 product reviews written in Polish and extracted from Allegro.pl - a popular e-commerce marketplace. Each review contains at least 50 words and has a rating on a scale from one (negative review) to five (positive review).*
- **Data:** *https://github.com/allegro/klejbenchmark-allegroreviews*
- **Motivation:** *The KLEJ benchmark (Kompleksowa Lista Ewaluacji Językowych) is a set of nine evaluation tasks for the Polish language understanding.*


Add PEC

I'm a bit frustrated now to get this right.

Hey @zhongpeixiang!
Really nice addition here!

Did you officially joined the sprint by posting [on the forum thread](https://discuss.huggingface.co/t/open-to-the-community-one-week-team-effort-to-reach-v2-0-of-hf-datasets-library/2176) and joining our slack?
I can't seem to find you there! Should I add you directly with your gmail address?

> Hey @zhongpeixiang!
> Really nice addition here!
>
> Did you officially joined the sprint by posting [on the forum thread](https://discuss.huggingface.co/t/open-to-the-community-one-week-team-effort-to-reach-v2-0-of-hf-datasets-library/2176) and joining our slack?
> I can't seem to find you there! Should I add you directly with your gmail address?

Thank you for the invitation. This initiative is awesome. Sadly I’m occupied by my thesis writing this month. Good luck 🤗

> As you want @zhongpeixiang (I was maybe not clear but that just mean that by posting on the forum thread that you participated in the current event you will get a special gift (a tee-shirt) for the contribution that you have already done here :-) Nothing more to do)

Oh, I misunderstood the post. I'm glad to join.

A persona-based empathetic conversation dataset.


Add ASSET dataset for text simplification evaluation

Adding the ASSET dataset from https://github.com/facebookresearch/asset

One config for the simplification data, one for the human ratings of quality.

The README.md borrows from that written by @juand-r


Hi

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Lío o

````l`````````

```
O
```
`````
Ño
```
````

```


Add Sesotho Ner

looks like this PR include changes to other files (sepedi)
could you try to only include the files related to the addition of sesotho ner ?

I think i need to clean up my local repo. I am committing everything a fresh after sepedi

Feel free to ping me when yuo have a clean PR and it's ready to review :)


Add ZEST: ZEroShot learning from Task descriptions

Looks good to me, we can ping the authors for more info later. And yes apply `other-task` labels liberally, we can sort them out later :)

Looks ready to merge when you're ready @joeddav

Adds the ZEST dataset on zero-shot learning from task descriptions from AI2.

- Webpage: https://allenai.org/data/zest
- Paper: https://arxiv.org/abs/2011.08115

The nature of this dataset made the supported task tags tricky if you wouldn't mind giving any feedback @yjernite. Also let me know if you think we should have a `other-task-generalization` or something like that...


Add Schema Guided Dialogue dataset

This PR adds the Schema Guided Dialogue dataset created for the DSTC8 challenge
- https://github.com/google-research-datasets/dstc8-schema-guided-dialogue

A bit simpler than MultiWOZ, the only tricky thing is the sequence of dictionaries that had to be linearized. There is a config for the data proper, and a config for the schemas.


add MRQA

MRQA (shared task 2019)
out of distribution generalization
Framed as extractive question answering
Dataset is the concatenation (of subsets) of existing QA datasets processed to match the SQuAD format


Add Gutenberg time references dataset

Description: "A clean data resource containing all explicit time references in a dataset of 52,183 novels whose full text is available via Project Gutenberg and the Hathi Trust Digital Library 2." > This is just the Gutenberg part.

Also, the paragraph at the top of the file would make a good Dataset Summary in the README :)

This PR adds the gutenberg_time dataset: https://arxiv.org/abs/2011.04124


Add Setswana NER


Add caWaC dataset

Add dataset.


Add Sepedi NER

Sorry for this. I deleted sepedi_ner_corpus as per your earlier advise. Let me check.

This is a new branch created for this dataset


Specify file encoding

If not specified, Python uses system default, which for Windows is not "utf-8".


Add CLINC150 dataset

Added CLINC150 Dataset. The link to the dataset can be found [here](https://github.com/clinc/oos-eval) and the paper can be found [here](https://www.aclweb.org/anthology/D19-1131.pdf)

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data


add hard dataset

Hotel Reviews in Arabic language.


Add SciTLDR Dataset (Take 2)

@lhoestq please review this PR when you get free

> If the CI fails just because of `RemoteDatasetTest` errors it's fine, they're fixed on master

The same 3 tests are failing again :(
```
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_class_norwegian_ner
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_configs_norwegian_ner
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_norwegian_ner
```

> One trick if you want to add more datasets to avoid these errors : you can just rebase the master branch of your fork from the master branch of the repo. Then each time you make a new branch from master on your fork, it will include the fix for these errors

Yes, I almost always do that, but somehow seems even this branch got old 😓
I also do the following if I directly create a new branch locally: `git checkout -b <branchname> upstream/master` so it stays up-to date irrespective of my fork, still don't know how this crept in again

Merging this one since the CI is fixed on master

Adds the SciTLDR Dataset by AI2
Added the `README.md` card with tags to the best of my knowledge

Multi-target summaries or TLDRs of Scientific Documents

Continued from #986


Adding CS restaurants dataset

This PR adds the CS restaurants dataset; this is a re-opening of a previous PR with a chaotic commit history.


Adding  Evidence Inference Data:

http://evidence-inference.ebm-nlp.com/download/
https://arxiv.org/pdf/2005.04177.pdf


Add Bilingual Corpus of Arabic-English Parallel Tweets

IMO, the problem with this dataset is that it is not really a text/nlp dataset. These are just collections of tweet ids. So, ultimately, one needs to crawl twitter to get the actual text.

at least it's clear in the description that one needs to collect the tweets :
```
This resource is a result of a generic method for collecting parallel tweets.
```

Looks like this is failing for other datasets. Should I rebase it and push again?
Also rebasing and pushing is reflecting changes in many other files (ultimately forcing me to open a new branch and a new PR) any way to avoid this?

No let me merge this one directly, it's fine

Added Bilingual Corpus of Arabic-English Parallel Tweets. The link to the dataset can be found [here](https://alt.qcri.org/wp-content/uploads/2020/08/Bilingual-Corpus-of-Arabic-English-Parallel-Tweets.zip) and the paper can be found [here](https://www.aclweb.org/anthology/2020.bucc-1.3.pdf)

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data


Add NoReC: Norwegian Review Corpus


Adding C3 dataset: the first free-form multiple-Choice Chinese machine reading Comprehension dataset.

https://github.com/nlpdata/c3
https://arxiv.org/abs/1904.09679


Adding C3 dataset: the first free-form multiple-Choice Chinese machine reading Comprehension dataset.      https://github.com/nlpdata/c3 https://arxiv.org/abs/1904.09679


Include license file in source distribution

It would be helpful to include the license file in the source distribution.


add yahoo_answers_topics

feel free to merge/ping me to merge if there're no more changes to do

This PR adds yahoo answers topic classification dataset.

More info:
https://github.com/LC-John/Yahoo-Answers-Topic-Classification-Dataset

cc @joeddav, @yjernite


Adding Autshumato South african langages:


how large datasets are handled under the hood

This library uses Apache Arrow under the hood to store datasets on disk.
The advantage of Apache Arrow is that it allows to memory map the dataset. This allows to load datasets bigger than memory and with almost no RAM usage. It also offers excellent I/O speed.

For example when you access one element or one batch
```python
from datasets import load_dataset

squad = load_dataset("squad", split="train")
first_element = squad[0]
one_batch = squad[:8]
```

then only this element/batch is loaded in memory, while the rest of the dataset is memory mapped.

How can we change how much data is loaded to memory with Arrow? I think that I am having some performance issue with it. When Arrow loads the data from disk it does it in multiprocess? It's almost twice slower training with arrow than in memory.

EDIT:
My fault! I had not seen the `dataloader_num_workers` in `TrainingArguments` ! Now I can parallelize and go fast! Sorry, and thanks.

> How can we change how much data is loaded to memory with Arrow? I think that I am having some performance issue with it. When Arrow loads the data from disk it does it in multiprocess? It's almost twice slower training with arrow than in memory.

Loading arrow data from disk is done with memory-mapping. This allows to load huge datasets without filling your RAM.
Memory mapping is almost instantaneous and is done within one process.

Then, the speed of querying examples from the dataset is I/O bounded depending on your disk. If it's an SSD then fetching examples from the dataset will be very fast.
But since the I/O speed of an SSD is lower than the one of RAM it's expected to be slower to fetch data from disk than from memory.
Still, if you load the dataset in different processes then it can be faster but there will still be the I/O bottleneck of the disk.

> EDIT:
> My fault! I had not seen the `dataloader_num_workers` in `TrainingArguments` ! Now I can parallelize and go fast! Sorry, and thanks.

Ok let me know if that helps !

Hi
I want to use multiple large datasets with a mapping style dataloader, where they cannot fit into memory, could you tell me how you handled the datasets under the hood? is this you bring all in memory in case of mapping style ones? or is this some sharding under the hood and you bring in memory when necessary, thanks


Add multi_x_science_sum

Add Multi-XScience Dataset.

github repo: https://github.com/yaolu/Multi-XScience
paper: [Multi-XScience: A Large-scale Dataset for Extreme Multi-document Summarization of Scientific Articles](https://arxiv.org/abs/2010.14235)


Adding Medal: MeDAL: Medical Abbreviation Disambiguation Dataset for Natural Language Understanding Pretraining

Could you fix the dummy data before we merge ?
Looks like the dummy `train.csv` is missing

Thanks @Narsil @lhoestq for adding MeDAL :)


UM005: Urdu <> English Translation Dataset

Adds Urdu-English dataset for machine translation: http://ufal.ms.mff.cuni.cz/umc/005-en-ur/


add generated_reviews_enth

`generated_reviews_enth` is created as part of [scb-mt-en-th-2020](https://arxiv.org/pdf/2007.03541.pdf) for machine translation task. This dataset (referred to as `generated_reviews_yn` in [scb-mt-en-th-2020](https://arxiv.org/pdf/2007.03541.pdf)) are English product reviews generated by [CTRL](https://arxiv.org/abs/1909.05858), translated by Google Translate API and annotated as accepted or rejected (`correct`) based on fluency and adequacy of the translation by human annotators. This allows it to be used for English-to-Thai translation quality esitmation (binary label), machine translation, and sentiment analysis.


adding yahoo_answers_qa

Adding Yahoo Answers QA dataset.

More info:
https://ciir.cs.umass.edu/downloads/nfL6/


Microsoft CodeXGlue

#978 is working on adding code refinement

maybe we should keep the CodeXGlue benchmark (as glue) and don't merge the code_refinement dataset proposed in #978 ?

cc @reshinthadithyan

Hi @madlag and @lhoestq , I am extremely interested in getting this dataset into HF's library as I research in this area a lot. I see that it hasn't been updated in a while, but it is very close to being finished. If no one is currently working on this, I'd be happy to do any final touches that might be needed to get this merged.

Hi @ncoop57 ! Thanks for your interest and sorry for the inactivity on this PR.
Sure feel free to create another PR to continue this one ! This one was really close to being merged so I think it won't require that much changes. In addition to my previous comments, there should also be a "Contributions" subsection (see the template of the README [here](https://github.com/huggingface/datasets/blob/master/templates/README.md))

Superseded by https://github.com/huggingface/datasets/pull/2357 .

Datasets from https://github.com/microsoft/CodeXGLUE

This contains 13 datasets:

code_x_glue_cc_clone_detection_big_clone_bench
code_x_glue_cc_clone_detection_poj_104
code_x_glue_cc_cloze_testing_all
code_x_glue_cc_cloze_testing_maxmin
code_x_glue_cc_code_completion_line
code_x_glue_cc_code_completion_token
code_x_glue_cc_code_refinement
code_x_glue_cc_code_to_code_trans
code_x_glue_cc_defect_detection
code_x_glue_ct_code_to_text
code_x_glue_tc_nl_code_search_adv
code_x_glue_tc_text_to_code
code_x_glue_tt_text_to_text


NotADirectoryError while loading the CNN/Dailymail dataset

Looks like the google drive download failed.
I'm getting a `Google Drive - Quota exceeded` error while looking at the downloaded file.

We should consider finding a better host than google drive for this dataset imo
related : #873 #864

It is working now, thank you.

Should I leave this issue open to address the Quota-exceeded error?

Yes please. It's been happening several times, we definitely need to address it

Any updates on this one? I'm facing a similar issue trying to add CelebA.

I've looked into it and couldn't find a solution. This looks like a Google Drive limitation..
Please try to use other hosts when possible

The original links are google drive links.  Would it be feasible for HF to maintain their own servers for this? Also, I think the same issue must also exist with TFDS.

It's possible to host data on our side but we should ask the authors. TFDS has the same issue and doesn't have a solution either afaik.
Otherwise you can use the google drive link, but it it's not that convenient because of this quota issue.

Okay. I imagine asking every author who shares their dataset on Google Drive will also be cumbersome.

I am getting this error as well. Is there a fix?

Not as long as the data is stored on GG drive unfortunately.
Maybe we can ask if there's a mirror ?

Hi @JafferWilson is there a download link to get cnn dailymail from another host than GG drive ?

To give you some context, this library provides tools to download and process datasets. For CNN DailyMail the data are downloaded from the link you provide on your github repository. Unfortunately because of GG drive quotas, many users are not able to load this dataset.

The following copy of CNN/DM dataset, fixed the problem for me:
https://huggingface.co/datasets/ccdv/cnn_dailymail

Thanks for the link @mrazizi !

Apparently the original authors don't host the dataset themselves ("for legal reasons", source [here](https://github.com/abisee/cnn-dailymail/issues/9)).

Downloading and preparing dataset cnn_dailymail/3.0.0 (download: 558.32 MiB, generated: 1.28 GiB, post-processed: Unknown size, total: 1.82 GiB) to /root/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0/0128610a44e10f25b4af6689441c72af86205282d26399642f7db38fa7535602...

---------------------------------------------------------------------------

NotADirectoryError                        Traceback (most recent call last)

<ipython-input-9-cd4bf8bea840> in <module>()
     22
     23
---> 24 train = load_dataset('cnn_dailymail', '3.0.0', split='train')
     25 validation = load_dataset('cnn_dailymail', '3.0.0', split='validation')
     26 test = load_dataset('cnn_dailymail', '3.0.0', split='test')

5 frames

/root/.cache/huggingface/modules/datasets_modules/datasets/cnn_dailymail/0128610a44e10f25b4af6689441c72af86205282d26399642f7db38fa7535602/cnn_dailymail.py in _find_files(dl_paths, publisher, url_dict)
    132     else:
    133         logging.fatal("Unsupported publisher: %s", publisher)
--> 134     files = sorted(os.listdir(top_dir))
    135
    136     ret_files = []

NotADirectoryError: [Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b/cnn/stories'


added dataset circa

Dataset Circa added. Only README.md and dataset card left


Add Sepedi ner corpus

Looks like the PR includes commits about many other files.
Could you create a clean branch from master, and create another PR ?


Problem downloading amazon_reviews_multi

Hi @hfawaz ! This is working fine for me. Is it a repeated occurence? Have you tried from the latest verion?

Hi, it seems a connection problem.
Now it says:
`ConnectionError: Couldn't reach https://amazon-reviews-ml.s3-us-west-2.amazonaws.com/json/train/dataset_ja_train.json`

Thanks for adding the dataset.
After trying to load the dataset, I am getting the following error:
`ConnectionError: Couldn't reach https://amazon-reviews-ml.s3-us-west-2.amazonaws.com/json/train/dataset_fr_train.json
`
I used the following code to load the dataset:
`load_dataset(
            dataset_name,
            "all_languages",
            cache_dir=".data"
        )`

I am using version 1.1.3 of `datasets`

Note that I can perform a successfull `wget https://amazon-reviews-ml.s3-us-west-2.amazonaws.com/json/train/dataset_fr_train.json`


Add CAIL 2018 dataset


Adding farsi_news dataset (https://github.com/sci2lab/Farsi-datasets)


Add E2E NLG

Adding the E2E NLG dataset.

More info here : http://www.macs.hw.ac.uk/InteractionLab/E2E/

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template and at least fill the tags
- [x] Both tests for the real data and the dummy data pass.


Fix SV -> NO

This PR fixes the small typo as seen in #956


making sure datasets are not loaded in memory and distributed training of them

my implementation of sharding per TPU core: https://github.com/google-research/ruse/blob/d4dd58a2d8efe0ffb1a9e9e77e3228d6824d3c3c/seq2seq/trainers/t5_trainer.py#L316
my implementation of dataloader for this case https://github.com/google-research/ruse/blob/d4dd58a2d8efe0ffb1a9e9e77e3228d6824d3c3c/seq2seq/tasks/tasks.py#L496

Hi!  You can use the `assert not bool(dataset.cache_files)` assertion to ensure your data is in memory. And I suggest using `accelerate` for distributed training.

Hi
I am dealing with large-scale datasets which I need to train distributedly, I used the shard function to divide the dataset across the cores, without any sampler, this does not work for distributed training and does not become any faster than 1 TPU core. 1) how I can make sure data is not loaded in memory 2) in case of distributed training with iterative datasets which measures needs to be taken? Is this all sharding the data only. I was wondering if there can be possibility for me to discuss this with someone with distributed training with iterative datasets using dataset library. thanks


Add OPUS DOGC dataset


Add SciTLDR Dataset

CI failures seem to be unrelated (related to `norwegian_ner`)
```
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_class_norwegian_ner
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_configs_norwegian_ner
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_norwegian_ner
```

you can just rebase from master to fix the CI :)

can you just rebase from master before we merge ?

Sorry, the rebase from master went horribly wrong, I guess I'll just open another PR

Closing this one due to a mistake in rebasing :(

Adds the SciTLDR Dataset by AI2
Added README card with tags to the best of my knowledge

Multi-target summaries or TLDRs of Scientific Documents


Add GAP dataset

This dataset already exists apparently, sorry :/
see
https://github.com/huggingface/datasets/blob/master/datasets/gap/gap.py

Feel free to re-use the dataset card you did for `/datasets/gap`

I think you should also delete this branch.

GAP dataset
Gender bias coreference resolution


committing Whoa file

can't find the Whoa file since there' nothing left


add mc taco

MC-TACO
Temporal commonsense knowledge


add prachathai67k take2

I decided it will be faster to create a new pull request instead of fixing the rebase issues.
continuing from https://github.com/huggingface/datasets/pull/954


add wisesight_sentiment take2

Take 2 since last time the rebase issues were taking me too much time to fix as opposed to just open a new one.


Wongnai - Thai reviews dataset

Thank you for contributing a Thai dataset, @mapmeld ! I'm super hyped.
One comment I may add is that wongnai-corpus has two datasets: review classification (this) and word tokenization (https://github.com/wongnai/wongnai-corpus/blob/master/search/labeled_queries_by_judges.txt).
Would it be possible for you to rename this one something along the line of `wongnai-reviews` so that when/if we include the word tokenization dataset, we will know which is which.

This helps solve my check_code_quality issue.
```
make style
black --line-length 119 --target-version py36 datasets/wongnai
flake8 datasets/wongnai
isort datasets/wongnai/wongnai.py
```

@cstorm125 thanks! following your suggestions on formatting and on naming the dataset

I am writing a blog post about Thai NLP and transformers (example: mBERT does 1-2 character tokens instead of doing word segmentation), started adding this dataset to use as an example, and then saw you were adding other datasets. Great work! And if you know any Thai BERT models beyond https://github.com/ThAIKeras/bert we should maybe talk over email!

40,000 reviews, previously released on GitHub ( https://github.com/wongnai/wongnai-corpus ) with an LGPL license, and on a closed Kaggle competition ( https://www.kaggle.com/c/wongnai-challenge-review-rating-prediction/ )


[WIP] Add multi woz

This PR adds version 2.2 of the Multi-domain Wizard of OZ dataset: https://github.com/budzianowski/multiwoz/tree/master/data/MultiWOZ_2.2

It was a pretty big chunk of work to figure out the structure, so I stil have tol add the description to the README.md

On the plus side the structure is broadly similar to that of the Google Schema Guided dialogue [dataset](https://github.com/google-research-datasets/dstc8-schema-guided-dialogue), so will take care of that one next.