fix squad metric format

One thing for SQuAD is that I wanted to be able to use the SQuAD dataset directly in the metrics and I'm not sure it will be possible with this format.

(maybe it's not really possible in general though)

This is kinda related to one thing I had in mind which is that we may want to be able to dump our model predictions in a `Dataset` as well so that we don't keep them in memory (and we can export them in a nice format later as well when we will have a serialization formats).

Maybe this is overkill though, I haven't fully wraped my head around this.

I'm also perfectly fine with merging this PR in the current state and working on a larger scope later.

This is the format needed to run the official script directly. The format of the squad dataset is different from the input of the metric.

> One thing for SQuAD is that I wanted to be able to use the SQuAD dataset directly in the metrics and I'm not sure it will be possible with this format.
>
> (maybe it's not really possible in general though)

Ok I see. I'll try to use the same format

Ok with this update I changed the format to fit the squad dataset format.
Now you can do:
```python
squad_dset = nlp.load_dataset("squad")
squad_metric = nlp.load_metric("/Users/quentinlhoest/Desktop/hf/nlp-bis/metrics/squad")
predictions = [
    {"id": v["id"], "prediction_text": v["answers"]["text"][0]}  # take first possible answer
    for v in squad_dset["validation"]
]
squad_metric.compute(predictions, squad_dset["validation"])
```

The format of the squad metric was wrong.
This should fix #143

I tested with
```python3
predictions = [
    {'id': '56be4db0acb8001400a502ec', 'prediction_text': 'Denver Broncos'}
]
references = [
    {'answers': [{'text': 'Denver Broncos'}], 'id': '56be4db0acb8001400a502ec'}
]
```


Rename anli dataset

What we have now as the `anli` dataset is actually the αNLI dataset from the ART challenge dataset. This name is confusing because `anli` is also the name of adversarial NLI (see [https://github.com/facebookresearch/anli](https://github.com/facebookresearch/anli)).

I renamed the current `anli` dataset by `art`.


Adding Qanta (Quizbowl) Dataset

Hi @EntilZha - sorry for waiting so long until taking action here. We created a new command and a new recipe of how to add dummy_data. Can you maybe rebase to `master` as explained in 7. of https://github.com/huggingface/nlp/blob/master/CONTRIBUTING.md#how-to-contribute-to-nlp and check that your dummy data is correct following the instructions here: https://github.com/huggingface/nlp/blob/master/CONTRIBUTING.md#how-to-add-a-dataset ?

If the tests described in 5. of https://github.com/huggingface/nlp/blob/master/CONTRIBUTING.md#how-to-add-a-dataset pass we can merge the PR :-)

I updated to the most recent master and followed the steps, but still having the similar error where it can't find the correct file since the path to the directory is given, rather than the individual files within them. This still something wrong about how I'm inputting the data or how the tests are reading it?

It's the dummy_data structure. You actually have to call the dummy data file name `dummy_data` (not .json anything). So there should not be a `dummy_data` folder but for each config only a `dummy_data` which contains your json dummy data. Can you maybe try once more - if it doesn't work I do it for you :-).

Would that work if there are multiple files? In my case, I'm including something similar to squad 1.0/2.0 where we have the main dataset + an additional challenge set in different files. Would I have the zip decompress to two files in that case?

This dataset was actually a special case. It helped us improve the  dummy data instructions :-), see #195 .Close this PR and merge #194.

This PR adds the qanta question answering datasets from [Quizbowl: The Case for Incremental Question Answering](https://arxiv.org/abs/1904.04792) and [Trick Me If You Can: Human-in-the-loop Generation of Adversarial Question Answering Examples](https://www.aclweb.org/anthology/Q19-1029/) (adversarial fold)

This partially continues a discussion around fixing dummy data from https://github.com/huggingface/nlp/issues/161

I ran the following code to double check that it works and did some sanity checks on the output. The majority of the code itself is from our `allennlp` version of the dataset reader.

```python
import nlp
# Default is full question
data = nlp.load_dataset('./datasets/qanta')
# Four configs
# Primarily useful for training
data = nlp.load_dataset('./datasets/qanta', 'mode=sentences,char_skip=25')
# Primarily used in evaluation
data = nlp.load_dataset('./datasets/qanta', 'mode=first,char_skip=25')
data = nlp.load_dataset('./datasets/qanta', 'mode=full,char_skip=25')
# Primarily useful in evaluation and "live" play
data = nlp.load_dataset('./datasets/qanta', 'mode=runs,char_skip=25')
```


Loading 'wikitext' dataset fails

Hi, make sure you have a recent version of pyarrow.

Are you using it in Google Colab? In this case, this error is probably the same as #128

Thanks!

Yes I'm using Google Colab, it seems like a duplicate then.

Hi,
The squad bug seems to be fixed, but the loading of the 'wikitext' still suffers from this problem (on Colab with pyarrow=0.17.1).

When you install `nlp` for the first time on a Colab runtime, it updates the `pyarrow` library that was already on colab. This update shows this message on colab:
```
WARNING: The following packages were previously imported in this runtime:
  [pyarrow]
You must restart the runtime in order to use newly installed versions.
```
You just have to restart the runtime and it should be fine.

Loading the 'wikitext' dataset fails with Attribute error:

Code to reproduce (From example notebook):

import nlp
wikitext_dataset = nlp.load_dataset('wikitext')

Error:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-17-d5d9df94b13c> in <module>()
     11
     12 # Load a dataset and print the first examples in the training set
---> 13 wikitext_dataset = nlp.load_dataset('wikitext')
     14 print(wikitext_dataset['train'][0])

6 frames
/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    518         download_mode=download_mode,
    519         ignore_verifications=ignore_verifications,
--> 520         save_infos=save_infos,
    521     )
    522

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, dl_manager, **download_and_prepare_kwargs)
    363                 verify_infos = not save_infos and not ignore_verifications
    364                 self._download_and_prepare(
--> 365                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    366                 )
    367                 # Sync info

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    416             try:
    417                 # Prepare split will record examples associated to the split
--> 418                 self._prepare_split(split_generator, **prepare_split_kwargs)
    419             except OSError:
    420                 raise OSError("Cannot find data file. " + (self.MANUAL_DOWNLOAD_INSTRUCTIONS or ""))

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _prepare_split(self, split_generator)
    594             example = self.info.features.encode_example(record)
    595             writer.write(example)
--> 596         num_examples, num_bytes = writer.finalize()
    597
    598         assert num_examples == num_examples, f"Expected to write {split_info.num_examples} but wrote {num_examples}"

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in finalize(self, close_stream)
    173     def finalize(self, close_stream=True):
    174         if self.pa_writer is not None:
--> 175             self.write_on_file()
    176             self.pa_writer.close()
    177         if close_stream:

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in write_on_file(self)
    124             else:
    125                 # All good
--> 126                 self._write_array_on_file(pa_array)
    127             self.current_rows = []
    128

/usr/local/lib/python3.6/dist-packages/nlp/arrow_writer.py in _write_array_on_file(self, pa_array)
     93     def _write_array_on_file(self, pa_array):
     94         """Write a PyArrow Array"""
---> 95         pa_batch = pa.RecordBatch.from_struct_array(pa_array)
     96         self._num_bytes += pa_array.nbytes
     97         self.pa_writer.write_batch(pa_batch)

AttributeError: type object 'pyarrow.lib.RecordBatch' has no attribute 'from_struct_array'


[Tests] refactor tests

This PR separates AWS and Local tests to remove these ugly statements in the script:
```python
        if "/" not in dataset_name:
            logging.info("Skip {} because it is a canonical dataset")
            return
```

To run a `aws` test, one should now run the following command:

```python
pytest -s tests/test_dataset_common.py::AWSDatasetTest::test_builder_class_wmt14
```

The same `local` test, can be run with:
```python
pytest -s tests/test_dataset_common.py::LocalDatasetTest::test_builder_class_wmt14
```


Add a method to shuffle a dataset

generic discussion

Generic discussion on the library

+1 for the naming convention

About the `shuffle` method, from my understanding it should be done in `Dataloader` (better separation between dataset processing - usage)

+1 for shuffle in `Dataloader`.
Some `Dataloader` just store idxs of dataset and just shuffle those idxs, which might(?) be faster than do shuffle in dataset, especially when doing shuffle every epoch.

Also +1 for the naming convention.

As you might already know the issue of dataset shuffling came up in the nlp code [walkthrough](https://youtu.be/G3pOvrKkFuk?t=3204) by Yannic Kilcher

We added the `.shuffle` method :)

Closing this one.

Could maybe be a `dataset.shuffle(generator=None, seed=None)` signature method.

Also, we could maybe have a clear indication of which method modify in-place and which methods return/cache a modified dataset. I kinda like torch conversion of having an underscore suffix for all the methods which modify a dataset in-place. What do you think?


ANLI

Can I recommend the following:

For ANLI, use https://github.com/facebookresearch/anli. As that paper says, "Our dataset is not
to be confused with abductive NLI (Bhagavatula et al., 2019), which calls itself αNLI, or ART.".

Indeed, the paper cited under what is currently called anli says in the abstract "We introduce a challenge dataset, ART".

The current naming will confuse people :)


Add Spanish POR and NER Datasets

dataset request

Requesting to add a new dataset

Hello @mrm8488, are these datasets official datasets published in an NLP/CL/ML venue?

What about this one: https://github.com/ccasimiro88/TranslateAlignRetrieve?

Hi guys,
In order to cover multilingual support a little step could be adding standard Datasets used for Spanish NER and POS tasks.
I can provide it in raw and preprocessed formats.


[Feature request] Add cos-e v1.0

Requesting to add a new dataset

Sounds good, @mariamabarham do you want to give a look?
I think we should have two configurations so we can allow either version of the dataset to be loaded with the `1.0` version being the default maybe.

Cc some authors of the great cos-e: @nazneenrajani @bmccann

cos_e v1.0 is related to CQA v1.0 but only CQA v1.11 dataset is available on their website. Indeed their is lots of ids in cos_e v1, which are  not in CQA v1.11  or the other way around.
@sarahwie, @thomwolf, @nazneenrajani,  @bmccann  do you know where I can  find CQA v1.0

@mariamabarham I'm also not sure where to find CQA 1.0. Perhaps it's not possible to include this version of the dataset. I'll close the issue if that's the case.

I do have a copy of the dataset. I can upload it to our repo.

Great @nazneenrajani. let me know once done.
Thanks

@mariamabarham @sarahwie I added them to the cos-e repo https://github.com/salesforce/cos-e/tree/master/data/v1.0

You can now do
```python
from nlp import load_dataset
cos_e = load_dataset("cos_e", "v1.0")
```
Thanks @mariamabarham !

> @mariamabarham Just wanted to note that default behavior `cos_e = load_dataset("cos_e")` now loads `v1.0`. Not sure if this is intentional (but the flag specification does work as intended).

In the new version of `nlp`, if you try `cos_e = load_dataset("cos_e")` it throws this error:
```
ValueError: Config name is missing.
Please pick one among the available configs: ['v1.0', 'v1.11']
Example of usage:
	`load_dataset('cos_e', 'v1.0')`
```
For datasets with at least two configurations, we now force the user to pick one (no default)

I noticed the second release of cos-e (v1.11) is included in this repo. I wanted to request inclusion of v1.0, since this is the version on which results are reported on in [the paper](https://www.aclweb.org/anthology/P19-1487/), and v1.11 has noted [annotation](https://github.com/salesforce/cos-e/issues/2) [issues](https://arxiv.org/pdf/2004.14546.pdf).


fix prev files hash in map

Hi, yes, this seems to fix #160 -- I cloned the branch locally and verified

Fix the `.map` issue in #160.
This makes sure it takes the previous files when computing the hash.


Discussion on version identifier & MockDataLoaderManager for test data

generic discussion

Generic discussion on the library

usually you can replace `download` in your dataset script with `download_and_prepare()` - could you share the code for your dataset here? :-)

I have an initial version here: https://github.com/EntilZha/nlp/tree/master/datasets/qanta Thats pretty close to what I'll do as a PR, but still want to do some more sanity checks/tests (just got tests passing).

I figured out how to get all tests passing by adding a download command and some finagling with the data zip https://github.com/EntilZha/nlp/blob/master/tests/utils.py#L127

I'm quite positive that you can just replace the `dl_manager.download()` statements here: https://github.com/EntilZha/nlp/blob/4d46443b65f1f756921db8275594e6af008a1de7/datasets/qanta/qanta.py#L194 with `dl_manager.download_and_extract()` even though you don't extract anything. I would prefer to avoid adding more functions to the MockDataLoadManager and keep it as simple as possible (It's already to complex now IMO).

Could you check if you can replace the `download()` function?

I might be doing something wrong, but swapping those two gives this error:
```
>       with open(path) as f:
E       IsADirectoryError: [Errno 21] Is a directory: 'datasets/qanta/dummy/mode=first,char_skip=25/2018.4.18/dummy_data-zip-extracted/dummy_data'

src/nlp/datasets/qanta/3d965403133687b819905ead4b69af7bcee365865279b2f797c79f809b4490c3/qanta.py:280: IsADirectoryError

During handling of the above exception, another exception occurred:
```

So it seems like the directory name is getting passed. Is this not functioning as expected, or is there some caching happening maybe? I deleted the dummy files and re-ran the import script with no changes. I'm digging a bit in with a debugger, but no clear reason yet

From what I can tell here: https://github.com/huggingface/nlp/blob/master/tests/utils.py#L115

1. `data_url` is the correct http link
2. `path_to_dummy_data` is a directory, which is causing the issue

That path comes from `download_dummy_data`, which I think assumes that the data comes from the zip file, but isn't aware of individual files. So it seems like it data manager needs to be aware if the url its getting is for a file or a zip/directory, and pass this information along. This might happen in `download_dummy_data`, but probably better to happen in `download_and_extract`? Maybe a simple check to see if `os.path.basename` returns the dummy data zip filename, if not then join paths with the basename of the url?

I think the dataset script works correctly. Just the dummy data structure seems to be wrong. I will soon add more commands that should make the create of the dummy data easier.

I'd recommend that you won't concentrate too much on the dummy data.
If you manage to load the dataset correctly via:

```python
# use local path to qanta
nlp.load_dataset("./datasets/qanta")
```

then feel free to open a PR and we will look into the dummy data problem together :-)

Also please make sure that the Version is in the format 1.0.0 (three numbers separated by two points) - not a date.

The script loading seems to work fine so I'll work on getting a PR open after a few sanity checks on the data.

On version, we currently have it versioned with YYYY.MM.DD scheme so it would be nice to not change that, but will it cause issues?

> The script loading seems to work fine so I'll work on getting a PR open after a few sanity checks on the data.
>
> On version, we currently have it versioned with YYYY.MM.DD scheme so it would be nice to not change that, but will it cause issues?

It would cause issues for sure for the tests....not sure if it would also cause issues otherwise.

I would prefer to keep the same version style as we have for other models. You could for example simply add version 1.0.0 and add a comment with the date you currently use for the versioning.

 What is your opinion regarding the version here @lhoestq @mariamabarham @thomwolf ?

> Maybe use the YYYY.MM.DD as the config name ? That's what we are doing for wikipedia

I'm not sure if this will work because the name should be unique and it seems that he has multiple config name in his data with the same version.
As @patrickvonplaten  suggested, I think  you can add a comment about the version in the data description.

Actually maybe our versioning format (inherited from tfds) is too strong for what we use it for?
We could allow any string maybe?

I see it more and more like an identifier for the user that we will back with a serious hashing/versioning system.- so we could let the user quite free on it.

I'm good with either putting it in description, adding it to the config, or loosening version formatting. I mostly don't have a full conceptual grasp of what each identifier ends up meaning in the datasets code so hard to evaluate the best approach.

For background, the multiple formats is a consequence of:

1. Each example is one multi-sentence trivia question
2. For training, its better to treat each sentence as an example
3. For evaluation, should test on: (1) first sentence, (2) full question, and (3) partial questions (does the model get the question right having seen the first half)

We use the date format for version since: (1) we expect some degree of updates since new questions come in every year and (2) the timestamp itself matches the Wikipedia dump that it is dependent on (so similar to the Wikipedia dataset).

perhaps this is better discussed in https://github.com/huggingface/nlp/pull/169 or update title?

Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done.


caching in map causes same result to be returned for train, validation and test

dataset bug

A bug in a dataset script provided in the library

Hi @dpressel,

thanks for posting your issue! Can you maybe add a complete code snippet that we can copy paste to reproduce the error? For example, I'm not sure where the variable `train_set` comes from in your code and it seems like you are loading multiple datasets at once?

Hi, the full example was listed in the PR above, but here is the exact link:

https://github.com/dpressel/mead-baseline/blob/3c1aa3ca062cb23f303ca98ac40b6652b37ee971/api-examples/layers-classify-hf-datasets.py

The problem is coming from
```
            if cache_file_name is None:
                # we create a unique hash from the function, current dataset file and the mapping args
                cache_kwargs = {
                    "with_indices": with_indices,
                    "batched": batched,
                    "batch_size": batch_size,
                    "remove_columns": remove_columns,
                    "keep_in_memory": keep_in_memory,
                    "load_from_cache_file": load_from_cache_file,
                    "cache_file_name": cache_file_name,
                    "writer_batch_size": writer_batch_size,
                    "arrow_schema": arrow_schema,
                    "disable_nullable": disable_nullable,
                }
                cache_file_name = self._get_cache_file_path(function, cache_kwargs)
```
The cached value is always the same, but I was able to change that by just renaming the function each time which seems to fix the issue.

This fixed my issue (I think)

https://github.com/dpressel/mead-baseline/commit/48aa8ecde4b307bd3e7dde5fe71e43a1d4769ee1

> Ok, I think @lhoestq has already found a solution :-) Maybe you can chime in @lhoestq

Oh, awesome!  I see the PR, Ill check it out

The PR should prevent the cache from losing track of the of the dataset type (based on the location of its data). Not sure about your second problem though (cache off).

Yes, with caching on, it seems to work without the function renaming hack, I mentioned this also in the PR. Thanks!

hello,

I am working on a program that uses the `nlp` library with the `SST2` dataset.

The rough outline of the program is:

```
import nlp as nlp_datasets
...
parser.add_argument('--dataset', help='HuggingFace Datasets id', default=['glue', 'sst2'], nargs='+')
...
dataset = nlp_datasets.load_dataset(*args.dataset)
...
# Create feature vocabs
vocabs = create_vocabs(dataset.values(), vectorizers)
...
# Create a function to vectorize based on vectorizers and vocabs:

print('TS', train_set.num_rows)
print('VS', valid_set.num_rows)
print('ES', test_set.num_rows)

# factory method to create a `convert_to_features` function based on vocabs
convert_to_features = create_featurizer(vectorizers, vocabs)
train_set = train_set.map(convert_to_features, batched=True)
train_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
train_loader = torch.utils.data.DataLoader(train_set, batch_size=args.batchsz)

valid_set = valid_set.map(convert_to_features, batched=True)
valid_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
valid_loader = torch.utils.data.DataLoader(valid_set, batch_size=args.batchsz)

test_set = test_set.map(convert_to_features, batched=True)
test_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
test_loader = torch.utils.data.DataLoader(test_set, batch_size=args.batchsz)

print('TS', train_set.num_rows)
print('VS', valid_set.num_rows)
print('ES', test_set.num_rows)

```
Im not sure if Im using it incorrectly, but the results are not what I expect.  Namely, the `.map()`  seems to grab the datset from the cache and then loses track of what the specific dataset is, instead using my training data for all datasets:

```
TS 67349
VS 872
ES 1821
TS 67349
VS 67349
ES 67349
```

The behavior changes if I turn off the caching but then the results fail:

```
train_set = train_set.map(convert_to_features, batched=True, load_from_cache_file=False)
...
valid_set = valid_set.map(convert_to_features, batched=True, load_from_cache_file=False)
...
test_set = test_set.map(convert_to_features, batched=True, load_from_cache_file=False)
```

Now I get the right set of features back...
```
TS 67349
VS 872
ES 1821
100%|██████████| 68/68 [00:00<00:00, 92.78it/s]
100%|██████████| 1/1 [00:00<00:00, 75.47it/s]
  0%|          | 0/2 [00:00<?, ?it/s]TS 67349
VS 872
ES 1821
100%|██████████| 2/2 [00:00<00:00, 77.19it/s]
```
but I think its losing track of the original training set:

```
Traceback (most recent call last):
  File "/home/dpressel/dev/work/baseline/api-examples/layers-classify-hf-datasets.py", line 148, in <module>
    for x in train_loader:
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 338, in __getitem__
    output_all_columns=self._output_all_columns,
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 294, in _getitem
    outputs = self._unnest(self._data.slice(key, 1).to_pydict())
  File "pyarrow/table.pxi", line 1211, in pyarrow.lib.Table.slice
  File "pyarrow/public-api.pxi", line 390, in pyarrow.lib.pyarrow_wrap_table
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 3: In chunk 0: Invalid: Length spanned by list offsets (15859698) larger than values array (length 100000)

Process finished with exit code 1
```

The full-example program (minus the print stmts) is here:
https://github.com/dpressel/mead-baseline/pull/620/files


How can we add more datasets to nlp library?


add Toronto Books Corpus

This PR adds the Toronto Books Corpus.
.
It on consider TMX and plain text files (Moses) defined in the table **Statistics and TMX/Moses Downloads** [here](http://opus.nlpl.eu/Books.php )


nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"

You can just run:
`val = nlp.load_dataset('squad')`

if you want to have just the validation script you can also do:

`val = nlp.load_dataset('squad', split="validation")`

If you want to load a local dataset, make sure you include a `./` before the folder name.

This happens by just doing run all cells on colab ... I assumed the colab example is broken.

Oh I see you might have a wrong version of pyarrow install on the colab -> could you try the following. Add these lines to the beginning of your notebook, restart the runtime and run it again:
```
!pip uninstall -y -qq pyarrow
!pip uninstall -y -qq nlp
!pip install -qq git+https://github.com/huggingface/nlp.git
```

> Oh I see you might have a wrong version of pyarrow install on the colab -> could you try the following. Add these lines to the beginning of your notebook, restart the runtime and run it again:
>
> ```
> !pip uninstall -y -qq pyarrow
> !pip uninstall -y -qq nlp
> !pip install -qq git+https://github.com/huggingface/nlp.git
> ```

Tried, having the same error.

Can you post a link here of your colab? I'll make a copy of it and see what's wrong

This should be fixed in the current version of the notebook. You can try it again

I am getting this error when running this command
```
val = nlp.load_dataset('squad', split="validation")
```

FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/huggingface/datasets/squad/plain_text/1.0.0/dataset_info.json'

Can anybody help?

It seems like your download was corrupted :-/ Can you run the following command:

```
rm -r /root/.cache/huggingface/datasets
```

to delete the cache completely and rerun the download?

I tried the notebook again today and it worked without barfing. 👌

I'm trying to load datasets from nlp but there seems to have error saying
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a


SyntaxError with WMT datasets

Jeez - don't know what happened there :D Should be fixed now!

Thanks a lot for reporting this @tomhosking !

Hi @patrickvonplaten!

I'm now getting the below error:

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-28-3206959998b9> in <module>
      1 import nlp
      2
----> 3 dataset = nlp.load_dataset('wmt14')
      4 print(dataset['train'][0])

~/.local/lib/python3.6/site-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    507     # Instantiate the dataset builder
    508     builder_instance = builder_cls(
--> 509         cache_dir=cache_dir, name=name, version=version, data_dir=data_dir, data_files=data_files, **config_kwargs,
    510     )
    511

TypeError: Can't instantiate abstract class Wmt with abstract methods _subsets
```

To correct this error I think you need the master branch of `nlp`. Can you try to install `nlp` with. `WMT` was not included at the beta release of the library.

Can you try:
`pip install git+https://github.com/huggingface/nlp.git`

and check again?

That works, thanks :)

The WMT datasets are listed in by `list_datasets()` in the beta release on pypi - it would be good to only show datasets that are actually supported by that version?

Usually, the idea is that a dataset can be added without releasing a new version. The problem in the case of `WMT` was that some "core" code of the library had to be changed as well.

@thomwolf @lhoestq @julien-c - How should we go about this. If we add a dataset that also requires "core" code changes, how do we handle the versioning? The moment a dataset is on AWS it will actually be listed with `list_datasets()` in all earlier versions...

Is there a way to somehow insert the `pip version` to the HfApi() and get only the datasets that were available for this version (at the date of the release of the version) @julien-c ?

We plan to have something like a `requirements.txt` per dataset to prevent user from loading dataset with old version of `nlp` or any other libraries. Right now the solution is just to keep `nlp` up to date when you want to load a dataset that leverages the latests features of `nlp`.

For datasets that are on AWS but that use features that are not released yet we should be able to filter those from the `list_dataset` as soon as we have the `requirements.txt` feature on (filter datasets that need a future version of `nlp`).

Shall we rename this issue to be more explicit about the problem ?
Something like `Specify the minimum version of the nlp library required for each dataset` ?

Closing this one.
Feel free to re-open if you have other questions :)

The following snippet produces a syntax error:

```
import nlp

dataset = nlp.load_dataset('wmt14')
print(dataset['train'][0])
```

```
Traceback (most recent call last):

  File "/home/tom/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-8-3206959998b9>", line 3, in <module>
    dataset = nlp.load_dataset('wmt14')

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 505, in load_dataset
    builder_cls = import_main_class(module_path, dataset=True)

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 56, in import_main_class
    module = importlib.import_module(module_path)

  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 994, in _gcd_import

  File "<frozen importlib._bootstrap>", line 971, in _find_and_load

  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 678, in exec_module

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt14.py", line 21, in <module>
    from .wmt_utils import Wmt, WmtConfig

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt_utils.py", line 659
    <<<<<<< HEAD
     ^
SyntaxError: invalid syntax
```

Python version:
`3.6.9 (default, Apr 18 2020, 01:56:04)  [GCC 8.4.0]`
Running on Ubuntu 18.04, via a Jupyter notebook


Include more links in README, fix typos

Include more links and fix typos in README


add Ubuntu Dialogs Corpus datasets

This PR adds the Ubuntu Dialog Corpus datasets version 2.0.


Meta-datasets (GLUE/XTREME/...) – Special care to attributions and citations

generic discussion

Generic discussion on the library

As @yoavgo suggested, there should be the possibility to call a function like  nlp.bib that outputs all bibtex ref from the datasets and models actually used and eventually nlp.bib.forreadme that would output the same info + versions numbers so they can be included in a readme.md file.

Actually, double checking with @mariamabarham, we already have this feature I think.

It's like this currently:
```python
>>> from nlp import load_dataset
>>>
>>> dataset = load_dataset('glue', 'cola', split='train')
>>> print(dataset.info.citation)
@article{warstadt2018neural,
  title={Neural Network Acceptability Judgments},
  author={Warstadt, Alex and Singh, Amanpreet and Bowman, Samuel R},
  journal={arXiv preprint arXiv:1805.12471},
  year={2018}
}
@inproceedings{wang2019glue,
  title={{GLUE}: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding},
  author={Wang, Alex and Singh, Amanpreet and Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel R.},
  note={In the Proceedings of ICLR.},
  year={2019}
}

Note that each GLUE dataset has its own citation. Please see the source to see
the correct citation for each contained dataset.
```

What do you think @dseddah?

Looks good but why would there be a difference between the ref in the source and the one to be printed?

Yes, I think we should remove this warning @mariamabarham.

It's probably a relic of tfds which didn't have the same way to access citations.

Meta-datasets are interesting in terms of standardized benchmarks but they also have specific behaviors, in particular in terms of attribution and authorship. It's very important that each specific dataset inside a meta dataset is properly referenced and the citation/specific homepage/etc are very visible and accessible and not only the generic citation of the meta-dataset itself.

Let's take GLUE as an example:

The configuration has the citation for each dataset included (e.g. [here](https://github.com/huggingface/nlp/blob/master/datasets/glue/glue.py#L154-L161)) but it should be copied inside the dataset info so that, when people access `dataset.info.citation` they get both the citation for GLUE and the citation for the specific datasets inside GLUE that they have loaded.


Add GLUE config name check

If tests are being added, any guidance on where to add tests would be helpful!

Tagging @thomwolf for review

Looks good to me. Is this compatible with the way we are doing tests right now @patrickvonplaten ?

If the tests pass it should be fine :-)

@Bharat123rox could you check whether the tests pass locally via:
`pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_glue`

The test fails with an `AssertionError` because the name is not being passed to kwargs, however I'm not sure how to do that, because only the config file is being passed to the tests of all datasets?

I'm guessing this is the corresponding code:
https://github.com/huggingface/nlp/blob/2b3621bb5c78caf02c5a969b8e67fa0c145da4e6/tests/test_dataset_common.py#L141-L143

And these are the logs:
```
___________________ DatasetTest.test_load_dataset_local_glue ___________________

self = <tests.test_dataset_common.DatasetTest testMethod=test_load_dataset_local_glue>
dataset_name = 'glue'

    @local
    def test_load_dataset_local(self, dataset_name):
        # test only first config
        if "/" in dataset_name:
            logging.info("Skip {} because it is not a canonical dataset")
            return

        configs = self.dataset_tester.load_all_configs(dataset_name, is_local=True)[:1]
>       self.dataset_tester.check_load_dataset(dataset_name, configs, is_local=True)

tests/test_dataset_common.py:200:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_dataset_common.py:74: in check_load_dataset
    dataset_builder = dataset_builder_cls(config=config, cache_dir=processed_temp_dir)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <nlp.datasets.glue.fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597.glue.Glue object at 0x135c0ea90>
args = ()
kwargs = {'cache_dir': '/var/folders/r6/mnw5ntvn5y72j7d4s1fm273m0000gn/T/tmpa9rpq3tl', 'config': GlueConfig(name='cola', versio...linguistic theory. Each example is a sequence of words annotated\nwith whether it is a grammatical English sentence.')}

    def __init__(self, *args, **kwargs):
>       assert ('name' in kwargs and kwargs['name'] is not None), "Glue has to be called with a configuration name"
E       AssertionError: Glue has to be called with a configuration name

/usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/glue.py:139: AssertionError
----------------------------- Captured stderr call -----------------------------
INFO:nlp.load:Checking ./datasets/glue/glue.py for additional imports.
INFO:filelock:Lock 5209998288 acquired on ./datasets/glue/glue.py.lock
INFO:nlp.load:Found main folder for dataset ./datasets/glue/glue.py at /usr/local/lib/python3.7/site-packages/nlp/datasets/glue
INFO:nlp.load:Found specific version folder for dataset ./datasets/glue/glue.py at /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597
INFO:nlp.load:Found script file from ./datasets/glue/glue.py to /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/glue.py
INFO:nlp.load:Found dataset infos file from ./datasets/glue/dataset_infos.json to /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/dataset_infos.json
INFO:nlp.load:Found metadata file for dataset ./datasets/glue/glue.py at /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/glue.json
INFO:filelock:Lock 5209998288 released on ./datasets/glue/glue.py.lock
INFO:nlp.load:Checking ./datasets/glue/glue.py for additional imports.
INFO:filelock:Lock 5196802640 acquired on ./datasets/glue/glue.py.lock
INFO:nlp.load:Found main folder for dataset ./datasets/glue/glue.py at /usr/local/lib/python3.7/site-packages/nlp/datasets/glue
INFO:nlp.load:Found specific version folder for dataset ./datasets/glue/glue.py at /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597
INFO:nlp.load:Found script file from ./datasets/glue/glue.py to /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/glue.py
INFO:nlp.load:Found dataset infos file from ./datasets/glue/dataset_infos.json to /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/dataset_infos.json
INFO:nlp.load:Found metadata file for dataset ./datasets/glue/glue.py at /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/glue.json
INFO:filelock:Lock 5196802640 released on ./datasets/glue/glue.py.lock
------------------------------ Captured log call -------------------------------
INFO     nlp.load:load.py:157 Checking ./datasets/glue/glue.py for additional imports.
INFO     filelock:filelock.py:274 Lock 5209998288 acquired on ./datasets/glue/glue.py.lock
INFO     nlp.load:load.py:320 Found main folder for dataset ./datasets/glue/glue.py at /usr/local/lib/python3.7/site-packages/nlp/datasets/glue
INFO     nlp.load:load.py:333 Found specific version folder for dataset ./datasets/glue/glue.py at /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597
INFO     nlp.load:load.py:346 Found script file from ./datasets/glue/glue.py to /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/glue.py
INFO     nlp.load:load.py:356 Found dataset infos file from ./datasets/glue/dataset_infos.json to /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/dataset_infos.json
INFO     nlp.load:load.py:367 Found metadata file for dataset ./datasets/glue/glue.py at /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/glue.json
INFO     filelock:filelock.py:318 Lock 5209998288 released on ./datasets/glue/glue.py.lock
INFO     nlp.load:load.py:157 Checking ./datasets/glue/glue.py for additional imports.
INFO     filelock:filelock.py:274 Lock 5196802640 acquired on ./datasets/glue/glue.py.lock
INFO     nlp.load:load.py:320 Found main folder for dataset ./datasets/glue/glue.py at /usr/local/lib/python3.7/site-packages/nlp/datasets/glue
INFO     nlp.load:load.py:333 Found specific version folder for dataset ./datasets/glue/glue.py at /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597
INFO     nlp.load:load.py:346 Found script file from ./datasets/glue/glue.py to /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/glue.py
INFO     nlp.load:load.py:356 Found dataset infos file from ./datasets/glue/dataset_infos.json to /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/dataset_infos.json
INFO     nlp.load:load.py:367 Found metadata file for dataset ./datasets/glue/glue.py at /usr/local/lib/python3.7/site-packages/nlp/datasets/glue/fa7c9f982200144186b2831060b54199cf028e4bbdc5f40acd339ee343342597/glue.json
INFO     filelock:filelock.py:318 Lock 5196802640 released on ./datasets/glue/glue.py.lock
```

Fixes #130 by adding a name check to the Glue class


Fix JSON tests.


Add WNUT 17 NER dataset

The PR looks awesome!
Since you have already added a dataset I imagine the tests as described in 5. of https://github.com/huggingface/nlp/blob/master/CONTRIBUTING.md#how-to-add-a-dataset all pass, right @stefan-it ?

I think we are then good to merge this :-) @lhoestq

Nice !

One thing though: I saw that you copied the `dataset_info.json` (one split info), which is different from the `dataset_infos.json` (split infos of all configs) that we expect.

Could you generate the `dataset_infos.json` file using this command please ?
```
python nlp-cli test datasets/wnut_17 --save_infos --all_configs
```

Hi @patrickvonplaten I just rebased onto latest `master` version and executed the commands. All tests passed then :)

@lhoestq thanks for that hint! I've generated and added the `dataset_infos.json` and deleted `dataset_info.json`.

Awesome ! I guess it's ready to be merged now :)

Hi,

this PR adds the WNUT 17 dataset to `nlp`.

> Emerging and Rare entity recognition
> This shared task focuses on identifying unusual, previously-unseen entities in the context of emerging discussions. Named entities form the basis of many modern approaches to other tasks (like event clustering and summarisation), but recall on them is a real problem in noisy text - even among annotators. This drop tends to be due to novel entities and surface forms. Take for example the tweet “so.. kktny in 30 mins?” - even human experts find entity kktny hard to detect and resolve. This task will evaluate the ability to detect and classify novel, emerging, singleton named entities in noisy text.
>
> The goal of this task is to provide a definition of emerging and of rare entities, and based on that, also datasets for detecting these entities.

More information about the dataset can be found on the [shared task page](https://noisy-text.github.io/2017/emerging-rare-entities.html).

Dataset is taken is taken from their [GitHub repository](https://github.com/leondz/emerging_entities_17), because the data provided in this repository contains minor fixes in the dataset format.

## Usage

Then the WNUT 17 dataset can be used in `nlp` like this:

```python
import nlp

wnut_17 = nlp.load_dataset("./datasets/wnut_17/wnut_17.py")

print(wnut_17)
```

This outputs:

```txt
'train': Dataset(schema: {'id': 'string', 'tokens': 'list<item: string>', 'labels': 'list<item: string>'}, num_rows: 3394)
'validation': Dataset(schema: {'id': 'string', 'tokens': 'list<item: string>', 'labels': 'list<item: string>'}, num_rows: 1009)
'test': Dataset(schema: {'id': 'string', 'tokens': 'list<item: string>', 'labels': 'list<item: string>'}, num_rows: 1287)
```

Number are identical with the ones in [this paper](https://www.ijcai.org/Proceedings/2019/0702.pdf) and are the same as using the `dataset` reader in Flair.

## Features

The following feature format is used to represent a sentence in the WNUT 17 dataset:

| Feature | Example | Description
| ---- | ---- | -----------------
| `id` | `0` | Number (id) of current sentence
| `tokens` | `["AHFA", "extends", "deadline"]` | List of tokens (strings) for a sentence
| `labels` | `["B-group", "O", "O"]` | List of labels (outer span)

The following labels are used in WNUT 17:

```txt
O
B-corporation
I-corporation
B-location
I-location
B-product
I-product
B-person
I-person
B-group
I-group
B-creative-work
I-creative-work
```


[Feature request] Add Ubuntu Dialogue Corpus dataset

dataset request

Requesting to add a new dataset

@AlphaMycelium the Ubuntu Dialogue Corpus [version 2]( https://github.com/rkadlec/ubuntu-ranking-dataset-creator) is added. Note that it requires a manual download by following the download instructions in the [repos]( https://github.com/rkadlec/ubuntu-ranking-dataset-creator).
Maybe we can close this issue for now?

https://github.com/rkadlec/ubuntu-ranking-dataset-creator or http://dataset.cs.mcgill.ca/ubuntu-corpus-1.0/


Same error for dataset 'wiki40b'
Should be fixed on master :)

# Reproduce
In Colab,
```
%pip install -q  nlp
%pip install -q apache_beam mwparserfromhell

dataset = nlp.load_dataset('wikipedia')
```
get
```
Downloading and preparing dataset wikipedia/20200501.aa (download: Unknown size, generated: Unknown size, total: Unknown size) to /root/.cache/huggingface/datasets/wikipedia/20200501.aa/1.0.0...

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-6-52471d2a0088> in <module>()
----> 1 dataset = nlp.load_dataset('wikipedia')

1 frames

/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    515         download_mode=download_mode,
    516         ignore_verifications=ignore_verifications,
--> 517         save_infos=save_infos,
    518     )
    519

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, dl_manager, **download_and_prepare_kwargs)
    361                 verify_infos = not save_infos and not ignore_verifications
    362                 self._download_and_prepare(
--> 363                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    364                 )
    365                 # Sync info

TypeError: _download_and_prepare() got an unexpected keyword argument 'verify_infos'
```


Error with sklearn train_test_split

enhancement

New feature or request

Indeed. Probably we will want to have a similar method directly in the library

It would be nice if we could use sklearn `train_test_split` to quickly generate subsets from the dataset objects returned by `nlp.load_dataset`. At the moment the code:

```python
data = nlp.load_dataset('imdb', cache_dir=data_cache)
f_half, s_half = train_test_split(data['train'], test_size=0.5, random_state=seed)
```
throws:
```
ValueError: Can only get row(s) (int or slice) or columns (string).
```
It's not a big deal, since there are other ways to split the data, but it would be a cool thing to have.


Add BERTScore to metrics

This PR adds [BERTScore](https://arxiv.org/abs/1904.09675) to metrics.
Here is an example of how to use it.

```sh
import nlp
bertscore = nlp.load_metric('metrics/bertscore') # or simply nlp.load_metric('bertscore') after this is added to huggingface's s3 bucket
predictions = ['example', 'fruit']
references = [['this is an example.', 'this is one example.'], ['apple']]
results = bertscore.compute(predictions, references, lang='en')
print(results)
```


[AWS Tests] Follow-up PR from #144

I forgot to add this line in PR #145 .


[AWS tests] AWS test should not run for canonical datasets

AWS tests should in general not run for canonical datasets. Only local tests will run in this case. This way a PR is able to pass when adding a new dataset.

This PR changes to logic to the following:

1) All datasets that are present in `nlp/datasets` are tested only locally. This way when one adds a canonical dataset, the PR includes his dataset in the tests.

2) All datasets that are only present on AWS, such as `webis/tl_dr` atm are tested only on AWS.

I think the testing structure might need a bigger refactoring and better documentation very soon.

Merging for now to unblock new PRs @thomwolf @mariamabarham .


ArrowTypeError in squad metrics

metric bug

A bug in a metric script

There was an issue in the format, thanks.
Now you can do
```python3
squad_dset = nlp.load_dataset("squad")
squad_metric = nlp.load_metric("/Users/quentinlhoest/Desktop/hf/nlp-bis/metrics/squad")
predictions = [
    {"id": v["id"], "prediction_text": v["answers"]["text"][0]}  # take first possible answer
    for v in squad_dset["validation"]
]
squad_metric.compute(predictions, squad_dset["validation"])
```

and the expected format is
```
Args:
    predictions: List of question-answers dictionaries with the following key-values:
        - 'id': id of the question-answer pair as given in the references (see below)
        - 'prediction_text': the text of the answer
    references: List of question-answers dictionaries with the following key-values:
        - 'id': id of the question-answer pair (see above),
        - 'answers': a Dict {'text': list of possible texts for the answer, as a list of strings}
```

`squad_metric.compute` is giving following error
```
ArrowTypeError: Could not convert [{'text': 'Denver Broncos'}, {'text': 'Denver Broncos'}, {'text': 'Denver Broncos'}] with type list: was not a dict, tuple, or recognized null value for conversion to struct type
```

This is how my predictions and references look like
```
predictions[0]
# {'id': '56be4db0acb8001400a502ec', 'prediction_text': 'Denver Broncos'}
```

```
references[0]
# {'answers': [{'text': 'Denver Broncos'},
  {'text': 'Denver Broncos'},
  {'text': 'Denver Broncos'}],
 'id': '56be4db0acb8001400a502ec'}
```

These are structured as per the `squad_metric.compute` help string.


[WMT] Add all wmt

This PR adds all wmt datasets scripts. At the moment the script is **not** functional for the language pairs "cs-en", "ru-en", "hi-en" because apparently it takes up to a week to get the manual data for these datasets: see http://ufal.mff.cuni.cz/czeng.

The datasets are fully functional though for the "big" language pairs "de-en" and "fr-en".

Overall I think the scripts are very messy and might need a big refactoring at some point.

For now I think there are good to merge (most dataset configs can be used). I will add "cs", "ru" and "hi" when the manual data is available.


[Clean up] remove bogus folder

Same for the dataset_infos.json at the project root no ?

Sorry guys, I haven't noticed. Thank  you for mentioning it.

@mariamabarham  - I think you accidentally placed it there.


[Tests] run local tests as default

You are right and I think those are usual best practice :) I'm 100% fine with this^^

Merging this for now to unblock other PRs.

This PR also enables local tests by default

I think it's safer for now to enable both local and aws tests for every commit. The problem currently is that when we do a PR to add a dataset, the dataset is not yet on AWS on therefore not tested on the PR itself. Thus the PR will always be green even if the datasets are not correct. This PR aims at fixing this.

## Suggestion on how to commit to the repo from now on:
Now since the repo is "online", I think we should adopt a couple of best practices:
1) - No direct committing to the repo anymore. Every change should be opened in a PR and be well documented so that we can find it later
2) - Every PR has to be reviewed by at least x people (I guess @thomwolf you should decide here) because we now have to be much more careful when doing changes to the API for backward compatibility, etc...


Add GermEval 2014 NER dataset

Had really fun playing around with this new library :heart:

That's awesome - thanks @stefan-it :-)

Could you maybe rebase to master and check if all dummy data tests are fine. I should have included the local tests directly in the test suite so that all PRs are fully checked: #140 - sorry :D

@patrickvonplaten Rebased it 😅

How can it test 🤔 I used:

```bash
RUN_SLOW=1 RUN_LOCAL=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_local_germeval_14
# and
RUN_SLOW=1 RUN_LOCAL=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_dataset_all_configs_local_germeval_14
```

and the tests still pass :)

Perfect, if these tests pass that's great - I'll merge the PR then :-) Was it very difficult to create the dummy data structure?

Hi,

this PR adds the GermEval 2014 NER dataset 😃

> The GermEval 2014 NER Shared Task builds on a new dataset with German Named Entity annotation [1] with the following properties:

> - The data was sampled from German Wikipedia and News Corpora as a collection of citations.
> - The dataset covers over 31,000 sentences corresponding to over 590,000 tokens.
> - The NER annotation uses the NoSta-D guidelines, which extend the Tübingen Treebank guidelines, using four main NER categories with sub-structure, and annotating embeddings among NEs such as [ORG FC Kickers [LOC Darmstadt]].

Dataset will be downloaded from the [official GermEval 2014 website](https://sites.google.com/site/germeval2014ner/data).

## Dataset format

Here's an example of the dataset format from the original dataset:

```tsv
# http://de.wikipedia.org/wiki/Manfred_Korfmann [2009-10-17]
1 Aufgrund O O
2 seiner O O
3 Initiative O O
4 fand O O
5 2001/2002 O O
6 in O O
7 Stuttgart B-LOC O
8 , O O
9 Braunschweig B-LOC O
10 und O O
11 Bonn B-LOC O
12 eine O O
13 große O O
14 und O O
15 publizistisch O O
16 vielbeachtete O O
17 Troia-Ausstellung B-LOCpart O
18 statt O O
19 , O O
20 „ O O
21 Troia B-OTH B-LOC
22 - I-OTH O
23 Traum I-OTH O
24 und I-OTH O
25 Wirklichkeit I-OTH O
26 “ O O
27 . O O
```

The sentence is encoded as one token per line (tab separated columns.

The first column contains either a `#`, which signals the source the sentence is cited from and the date it was retrieved, or the token number within the sentence.

The second column contains the token.

Column three and four contain the named entity (in IOB2 scheme).
Outer spans are encoded in the third column, embedded/nested spans in the fourth column.

## Features

I decided to keep most information from the dataset. That means the so called "source" information (where the sentences come from + date information) is also returned for each sentence in the feature vector.

For each sentence in the dataset, one feature vector (`nlp.Features` definition) will be returned:

| Feature | Example | Description
| ---- | ---- | -----------------
| `id` | `0` | Number (id) of current sentence
| `source` | `http://de.wikipedia.org/wiki/Manfred_Korfmann [2009-10-17]` | URL and retrieval date as string
| `tokens` | `["Schwartau", "sagte", ":"]` | List of tokens (strings) for a sentence
| `labels` | `["B-PER", "O", "O"]` | List of labels (outer span)
| `nested-labels` | `["O", "O", "O"]` | List of labels for nested span

## Example

The following command downloads the dataset from the official GermEval 2014 page and pre-processed it:

```bash
python nlp-cli test datasets/germeval_14 --all_configs
```

It then outputs the number for training, development and testset. The training set consists of 24,000 sentences, the development set of 2,200 and the test of 5,100 sentences.

Now it can be imported and used with `nlp`:

```python
import nlp

germeval = nlp.load_dataset("./datasets/germeval_14/germeval_14.py")
assert len(germeval["train"]) == 24000

# Show first sentence of training set:
germeval["train"][0]
```


Consider renaming to nld

generic discussion

Generic discussion on the library

I would suggest `nlds`. NLP is a very general, broad and ambiguous term, the library is not about NLP (as in processing) per se, it is about accessing Natural Language related datasets. So the name should reflect its purpose.

Chiming in to second everything @honnibal said, and to add that I think the current name is going to impact the discoverability of this library. People who are looking for "NLP Datasets" through a search engine are going to see a library called `nlp` and think it's too broad. People who are looking to do NLP in python are going to search "Python NLP" and end up here, confused that this is a collection of datasets.

The names of the other huggingface libraries work because they're the only game in town: there are not very many robust, distinct libraries for `tokenizers` or `transformers` in python, for example. But there are several options for NLP in python, and adding this as a possible search result for "python nlp" when datasets are likely not what someone is searching for adds noise and frustrates potential users.

I'm also not sure whether the naming of `nlp` is the problem itself, as long as it comes with the appropriate identifier, so maybe something like `huggingface_nlp`? This is analogous to what @honnibal and spacy are doing for `spacy-transformers`. Of course, this is a "step back" from the recent changes/renaming of transformers, but may be some middle ground between a complete rebranding, and keeping it identifiable.

Interesting, thanks for sharing your thoughts.

As we’ll move toward a first non-beta release, we will pool the community of contributors/users of the library for their opinions on a good final name (like when we renamed the beautifully (?) named `pytorch-pretrained-bert`)

In the meantime, using `from nlp import load_dataset, load_metric` should work 😉

I feel like we are conflating two distinct subjects here:

1. @honnibal's point is that using `nlp` as a package name might break existing code and bring developer usability issues in the future
2. @pmbaumgartner's point is that the `nlp` package name is too broad and shouldn't be used by a package that exposes only datasets and metrics

(let me know if I mischaracterize your point)

I'll chime in to say that the first point is a bit silly IMO. As Python developers due to the limitations of the import system we already have to share:
- a single flat namespace for packages
- which also conflicts with local modules i.e. local files

If we add the constraint that this flat namespace also be shared with variable names this gets untractable pretty fast :)

I also think all Python software developers/ML engineers/scientists are capable of at least a subset of:
- importing only the methods that they need like @thomwolf suggested
- aliasing their import
- renaming a local variable

By the way, `nlp` will very likely not be only about datasets, and not even just about datasets and metrics.

I see it as a laboratory for testing several long-term ideas about how we could do NLP in terms of research as well as open-source and community sharing, most of these ideas being too experimental/big to fit in `transformers`.

Some of the directions we would like to explore are about sharing, traceability and more experimental models, as well as seeing a model as the community-based process of creating a composite entity from data, optimization, and code.

We'll see how these ideas end up being implemented and we'll better know how we should define the library when we start to dive into these topics. I'll try to get the `nlp` team to draft a roadmap on these topics at some point.

> If we add the constraint that this flat namespace also be shared with variable names this gets untractable pretty fast :)

I'm sort of confused by your point here. The namespace *is* shared by variable names. You should not use local variables that are named the same as modules, because then you cannot use the module within the scope of your function.

For instance,

```python

import nlp
import transformers

nlp = transformers.pipeline("sentiment-analysis")
```

This is a bug: you've just overwritten the module, so now you can't use it. Or instead:

```python

import transformers

nlp = transformers.pipeline("sentiment-analysis")
# (Later, e.g. in a notebook)
import nlp
```

This is also a bug: you've overwritten your variable with an import.

If you have a module named `nlp`, you should avoid using `nlp` as a variable, or you'll have bugs in some contexts and inconsistencies in other contexts. You'll have situations where you need to import differently in one module vs another, or name variables differently in one context vs another, which is bad.

> importing only the methods that they need like @thomwolf suggested

Okay but the same logic applies to naming the module *literally anything else*. There's absolutely no point in having a module name that's 3 letters if you always plan to do `import from`! It would be entirely better to name it `nlp_datasets` if you don't want people to do `import nlp`.

And finally:

> By the way, nlp will very likely not be only about datasets, and not even just about datasets and metrics.

So...it isn't a datasets library? https://twitter.com/Thom_Wolf/status/1261282491622731781

I'm confused 😕

Dropping by as I noticed that the library has been renamed `datasets` so I wonder if the conversation above is settled (`nlp` not used anymore) :)

I'd argue that `datasets` is worse than `nlp`. Datasets should be a user specific decision and not encapsulate all of python (`pip install datasets`). If this package contained every dataset in the world (NLP / vision / etc) then it would make sense =/

I can't speak for the HF team @jramapuram, but as member of the community it looks to me that HF wanted to avoid the past path of changing names as scope broadened over time:

Remember
https://github.com/huggingface/pytorch-openai-transformer-lm
https://github.com/huggingface/pytorch-pretrained-BERT
https://github.com/huggingface/pytorch-transformers
and now
https://github.com/huggingface/transformers

;)

Jokes aside, seems that the library is growing in a multi-modal direction (https://github.com/huggingface/datasets/pull/363) so the current name is not that implausible. Possibly HF ambition is really to grow its community and bring here a large chunk of datasets of the world (including tabular / vision / audio?).

Yea I see your point. However, wouldn't scoping solve the entire problem?

```python
import huggingface.datasets as D
import huggingface.transformers as T
```

Calling something `datasets` is akin to saying I'm going to name my package `python` --> `import python`

Sorry to reply to an old thread, but the name issue really makes troubles recently in my project.

I'd never known in advance there's a package called "datasets". My first thought is that such a general term may be safe to arbitrarily use. Avoiding such a common name because of its ambiguity is quite weird.

As we know in python it's not easy to differentiate system-wide and project-wide import like in C and C++.

On the contrary I fully understand the challenge to rename a popular library. So it seems to provide a "huggingface" wrapper library as suggested above by @jramapuram may be a happy medium for both developers and users.

Best Regards.

Hey :)

Just making a thread here recording what I said on Twitter, as it's impossible to follow discussion there. It's also just really not a good way to talk about this sort of thing.

The issue is that modules go into the global namespace, so you shouldn't use variable names that conflict with module names. This means the package makes `nlp` a bad variable name everywhere in the codebase. I've always used `nlp` as the canonical variable name of spaCy's `Language` objects, and this is a convention that a lot of other code has followed (Stanza, flair, etc). And actually, your `transformers` library uses `nlp` as the name for its `Pipeline` instance in your readme.

If you stick with the `nlp` name for this package, if anyone uses it then they should rewrite all of that code. If `nlp` is a bad choice of variable anywhere, it's a bad choice of variable everywhere --- because you shouldn't have to notice whether some other function uses a module when you're naming variables within a function. You want to have one convention that you can stick to everywhere.

If people use your `nlp` package and continue to use the `nlp` variable name, they'll find themselves with confusing bugs. There will be many many bits of code cut-and-paste from tutorials that give confusing results when combined with the data loading from the `nlp` library. The problem will be especially bad for shadowed modules (people might reasonably have a module named `nlp.py` within their codebase) and notebooks, as people might run notebook cells for data loading out-of-order.

I don't think it's an exaggeration to say that if your library becomes popular, we'll all be answering issues around this about once a week for the next few years. That seems pretty unideal, so I do hope you'll reconsider.

I suggest `nld` as a better name. It more accurately represents what the package actually does. It's pretty unideal to have a package named `nlp` that doesn't do any processing, and contains data about natural language generation or other non-NLP tasks. The name is equally short, and is sort of a visual pun on `nlp`, since a d is a rotated p.


Update README.md

small typo


Fix print statement in READ.md

print statement was throwing generator object instead of printing names of available datasets/metrics


Update README.md

the readme got removed, closing this one


[Question] Using/adding a local dataset

Hi @zphang,

So you can just give the local path to a dataset script file and it should work.

Here is an example:
- you can download one of the scripts in the `datasets` folder of the present repo (or clone the repo)
- then you can load it with `load_dataset('PATH/TO/YOUR/LOCAL/SCRIPT.py')`

Does it make sense?

Could you give a more concrete example, please?

I looked up wikitext dataset script from the repo. Should I just overwrite the `data_file` on line 98 to point to the local dataset directory? Would it work for different configurations of wikitext (wikitext2, wikitext103 etc.)?

Or maybe we can use DownloadManager to specify local dataset location? In that case, where do we use DownloadManager instance?

Thanks

Hi @MaveriQ , although what I am doing is to commit a new dataset, but I think looking at imdb script might help.
You may want to use `dl_manager.download_custom`, give it a url(arbitrary string), a custom_download(arbitrary function) and return a path, and finally use _get sample to fetch a sample.

The download manager supports local directories. You can specify a local directory instead of a url and it should work.

Closing this one.
Feel free to re-open if you have other questions :)

Users may want to either create/modify a local copy of a dataset, or use a custom-built dataset with the same `Dataset` API as externally downloaded datasets.

It appears to be possible to point to a local dataset path rather than downloading the external ones, but I'm not exactly sure how to go about doing this.

A notebook/example script demonstrating this would be very helpful.


[Feature Request] Add the OpenWebText dataset

dataset request

Requesting to add a new dataset

We're experimenting with hosting the OpenWebText corpus on Zenodo for easier downloading. https://zenodo.org/record/3834942#.Xs1w8i-z2J8

The OpenWebText dataset is an open clone of OpenAI's WebText dataset. It can be used to train ELECTRA as is specified in the [README](https://www.github.com/google-research/electra).

More information and the download link are available [here](https://skylion007.github.io/OpenWebTextCorpus/).


[Feature request] Add Toronto BookCorpus dataset

Requesting to add a new dataset

As far as I understand, `wikitext` is refer to `WikiText-103` and `WikiText-2` that created by researchers in Salesforce, and mostly used in traditional language modeling.

You might want to say `wikipedia`, a dump from wikimedia foundation.

Also I would like to have Toronto BookCorpus too ! Though it involves copyright problem...

Hi, @lhoestq, just a reminder that this is solved by #248 .😉

I know the copyright/distribution of this one is complex, but it would be great to have! That, combined with the existing `wikitext`, would provide a complete dataset for pretraining models like BERT.


Loading GLUE dataset loads CoLA by default

As a follow-up to this: It looks like the actual GLUE task name is supplied as the `name` argument. Is there a way to check what `name`s/sub-datasets are available under a grouping like GLUE? That information doesn't seem to be readily available in info from `nlp.list_datasets()`.

Edit: I found the info under `Glue.BUILDER_CONFIGS`

Yes so the first config is loaded by default when no `name` is supplied but for GLUE this should probably throw an error indeed.

We can probably just add an `__init__` at the top of the `class Glue(nlp.GeneratorBasedBuilder)` in the `glue.py` script which does this check:
```
class Glue(nlp.GeneratorBasedBuilder):
    def __init__(self, *args, **kwargs):
        assert 'name' in kwargs and kwargs[name] is not None, "Glue has to be called with a configuration name"
        super(Glue, self).__init__(*args, **kwargs)
```

An error is raised if the sub-dataset is not specified :)
```
ValueError: Config name is missing.
Please pick one among the available configs: ['cola', 'sst2', 'mrpc', 'qqp', 'stsb', 'mnli', 'mnli_mismatched', 'mnli_matched', 'qnli', 'rte', 'wnli', 'ax']
Example of usage:
	`load_dataset('glue', 'cola')`
```

If I run:

```python
dataset = nlp.load_dataset('glue')
```
The resultant dataset seems to be CoLA be default, without throwing any error. This is in contrast to calling:

```python
metric = nlp.load_metric("glue")
```
which throws an error telling the user that they need to specify a task in GLUE. Should the same apply for loading datasets?


[Feature request] Add Google Natural Question dataset

Requesting to add a new dataset

Indeed, I think this one is almost ready cc @lhoestq

I'm doing the latest adjustments to make the processing of the dataset run on Dataflow

Is there an update to this? It will be very beneficial for the QA community!

Still work in progress :)
The idea is to have the dataset already processed somewhere so that the user only have to download the processed files. I'm also doing it for wikipedia.

Super appreciate your hard work !!
I'll cross my fingers and hope easily loadable wikipedia dataset will come soon.

Quick update on NQ: due to some limitations I met using apache beam + parquet I was not able to use the dataset in a nested parquet structure in python to convert it to our Apache Arrow format yet.
However we had planned to change this conversion step anyways so we'll make just sure that it enables to process and convert the NQ dataset to arrow.

Would be great to have https://github.com/google-research-datasets/natural-questions as an alternative to SQuAD.


Some error inside nlp.load_dataset()

Google colab has an old version of Apache Arrow built-in.
Be sure you execute the "pip install" cell and restart the notebook environment if the colab asks for it.

First of all, nice work!

I am going through [this overview notebook](https://colab.research.google.com/github/huggingface/nlp/blob/master/notebooks/Overview.ipynb)

In simple step `dataset = nlp.load_dataset('squad', split='validation[:10%]')`

I get an error, which is connected with some inner code, I think:
```
---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-8-d848d3a99b8c> in <module>()
      1 # Downloading and loading a dataset
      2
----> 3 dataset = nlp.load_dataset('squad', split='validation[:10%]')

8 frames

/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    515         download_mode=download_mode,
    516         ignore_verifications=ignore_verifications,
--> 517         save_infos=save_infos,
    518     )
    519

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, dl_manager, **download_and_prepare_kwargs)
    361                 verify_infos = not save_infos and not ignore_verifications
    362                 self._download_and_prepare(
--> 363                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    364                 )
    365                 # Sync info

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    414             try:
    415                 # Prepare split will record examples associated to the split
--> 416                 self._prepare_split(split_generator, **prepare_split_kwargs)
    417             except OSError:
    418                 raise OSError("Cannot find data file. " + (self.MANUAL_DOWNLOAD_INSTRUCTIONS or ""))

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _prepare_split(self, split_generator)
    585         fname = "{}-{}.arrow".format(self.name, split_generator.name)
    586         fpath = os.path.join(self._cache_dir, fname)
--> 587         examples_type = self.info.features.type
    588         writer = ArrowWriter(data_type=examples_type, path=fpath, writer_batch_size=self._writer_batch_size)
    589

/usr/local/lib/python3.6/dist-packages/nlp/features.py in type(self)
    460     @property
    461     def type(self):
--> 462         return get_nested_type(self)
    463
    464     @classmethod

/usr/local/lib/python3.6/dist-packages/nlp/features.py in get_nested_type(schema)
    370     # Nested structures: we allow dict, list/tuples, sequences
    371     if isinstance(schema, dict):
--> 372         return pa.struct({key: get_nested_type(value) for key, value in schema.items()})
    373     elif isinstance(schema, (list, tuple)):
    374         assert len(schema) == 1, "We defining list feature, you should just provide one example of the inner type"

/usr/local/lib/python3.6/dist-packages/nlp/features.py in <dictcomp>(.0)
    370     # Nested structures: we allow dict, list/tuples, sequences
    371     if isinstance(schema, dict):
--> 372         return pa.struct({key: get_nested_type(value) for key, value in schema.items()})
    373     elif isinstance(schema, (list, tuple)):
    374         assert len(schema) == 1, "We defining list feature, you should just provide one example of the inner type"

/usr/local/lib/python3.6/dist-packages/nlp/features.py in get_nested_type(schema)
    379         # We allow to reverse list of dict => dict of list for compatiblity with tfds
    380         if isinstance(inner_type, pa.StructType):
--> 381             return pa.struct(dict((f.name, pa.list_(f.type, schema.length)) for f in inner_type))
    382         return pa.list_(inner_type, schema.length)
    383

/usr/local/lib/python3.6/dist-packages/nlp/features.py in <genexpr>(.0)
    379         # We allow to reverse list of dict => dict of list for compatiblity with tfds
    380         if isinstance(inner_type, pa.StructType):
--> 381             return pa.struct(dict((f.name, pa.list_(f.type, schema.length)) for f in inner_type))
    382         return pa.list_(inner_type, schema.length)
    383

TypeError: list_() takes exactly one argument (2 given)
```


Update Overview.ipynb

update notebook


remove webis

Remove webis from dataset folder.

Our first dataset script that only lives on AWS :-) https://s3.console.aws.amazon.com/s3/buckets/datasets.huggingface.co/nlp/datasets/webis/tl_dr/?region=us-east-1 @julien-c @jplu


[Newsroom] add newsroom

I checked it with the data link of the mail you forwarded @thomwolf => works well!


Xsum, require manual download of some files


[Tests] Local => aws

For each dataset, If there exist a `dataset_info.json`, then the command `nlp-cli test path/to/my/dataset --al_configs` is successful only if the `dataset_infos.json` is correct. The infos are correct if the size and checksums of the downloaded file are correct, and if the number of examples in each split are correct.

Note: the `test` command is supposed to test the script, that's why it runs the script even if the cached files already exist. Let me know if it's good to you.

> For each dataset, If there exist a `dataset_info.json`, then the command `nlp-cli test path/to/my/dataset --al_configs` is successful only if the `dataset_infos.json` is correct. The infos are correct if the size and checksums of the downloaded file are correct, and if the number of examples in each split are correct.
>
> Note: the `test` command is supposed to test the script, that's why it runs the script even if the cached files already exist. Let me know if it's good to you.

Does it have to download the whole data to check if the checksums are correct? I guess so no?

> > For each dataset, If there exist a `dataset_info.json`, then the command `nlp-cli test path/to/my/dataset --al_configs` is successful only if the `dataset_infos.json` is correct. The infos are correct if the size and checksums of the downloaded file are correct, and if the number of examples in each split are correct.
> > Note: the `test` command is supposed to test the script, that's why it runs the script even if the cached files already exist. Let me know if it's good to you.
>
> Does it have to download the whole data to check if the checksums are correct? I guess so no?

Yes it has to download them all (unless they were already downloaded in which case it just uses the cached downloaded files).

## Change default Test from local => aws

As a default we set` aws=True`, `Local=False`, `slow=False`

### 1. RUN_AWS=1 (default)
This runs 4 tests per dataset script.

a) Does the dataset script have a valid etag / Can it be reached on AWS?
b) Can we load its `builder_class`?
c) Can we load **all** dataset configs?
d) _Most importantly_: Can we load the dataset?

Important - we currently only test the first config of each dataset to reduce test time. Total test time is around 1min20s.

### 2. RUN_LOCAL=1 RUN_AWS=0

***This should be done when debugging dataset scripts of the ./datasets folder***

This only runs 1 test per dataset test, which is equivalent to aws d) - Can we load the dataset from the local `datasets` directory?

### 3. RUN_SLOW=1

We should set up to run these tests maybe 1 time per week ? @thomwolf

The `slow` tests include two more important tests.

e) Can we load the dataset with all possible configs? This test will probably fail at the moment because a lot of dummy data is missing. We should add the dummy data step by step to be sure that all configs work.

f) Test that the actual dataset can be loaded. This will take quite some time to run, but is important to make sure that the "real" data can be loaded. It will also test whether the dataset script has the correct checksums file which is currently not tested with `aws=True`. @lhoestq - is there an easy way to check cheaply whether the `dataset_info.json` is correct for each dataset script?


Final cleanup of readme and metrics


make style


🐛 `map` not working

I didn't assign the output 🤦♂

```python
dataset.map(test)
```

should be :

```python
dataset = dataset.map(test)
```

I'm trying to run a basic example (mapping function to add a prefix).
[Here is the colab notebook I'm using.](https://colab.research.google.com/drive/1YH4JCAy0R1MMSc-k_Vlik_s1LEzP_t1h?usp=sharing)

```python
import nlp

dataset = nlp.load_dataset('squad', split='validation[:10%]')

def test(sample):
    sample['title'] = "test prefix @@@ " + sample["title"]
    return sample

print(dataset[0]['title'])
dataset.map(test)
print(dataset[0]['title'])
```
Output :
> Super_Bowl_50
Super_Bowl_50

Expected output :
> Super_Bowl_50
test prefix @@@ Super_Bowl_50


🐛 Colab : type object 'pyarrow.lib.RecordBatch' has no attribute 'from_struct_array'

It's strange, after installing `nlp` on Colab, the `pyarrow` version seems fine from `pip` but not from python :

```python
import pyarrow

!pip show pyarrow
print("version = {}".format(pyarrow.__version__))
```

> Name: pyarrow
Version: 0.17.0
Summary: Python library for Apache Arrow
Home-page: https://arrow.apache.org/
Author: None
Author-email: None
License: Apache License, Version 2.0
Location: /usr/local/lib/python3.6/dist-packages
Requires: numpy
Required-by: nlp, feather-format
>
> version = 0.14.1

Ok I just had to restart the runtime after installing `nlp`. After restarting, the version of `pyarrow` is fine.

I'm trying to load CNN/DM dataset on Colab.

[Colab notebook](https://colab.research.google.com/drive/11Mf7iNhIyt6GpgA1dBEtg3cyMHmMhtZS?usp=sharing)

But I meet this error :

> AttributeError: type object 'pyarrow.lib.RecordBatch' has no attribute 'from_struct_array'


❓ How to apply a map to all subsets ?

I'm working with CNN/DM dataset, where I have 3 subsets : `train`, `test`, `validation`.

Should I apply my map function on the subsets one by one ?

```python
import nlp

cnn_dm = nlp.load_dataset('cnn_dailymail')
for corpus in ['train', 'test', 'validation']:
         cnn_dm[corpus] = cnn_dm[corpus].map(my_func)
```

Or is there a better way to do this ?


❓ How to remove specific rows of a dataset ?

Can you do it by now? Coz it would be awfully helpful!

you can convert dataset object to pandas and remove a feature and convert back to dataset .

That's what I ended up doing too. but it feels like a workaround to a feature that should be added to the datasets class.

I saw on the [example notebook](https://colab.research.google.com/github/huggingface/nlp/blob/master/notebooks/Overview.ipynb#scrollTo=efFhDWhlvSVC) how to remove a specific column :

```python
dataset.drop('id')
```

But I didn't find how to remove a specific row.

**For example, how can I remove all sample with `id` < 10 ?**


🐛 Trying to use ROUGE metric : pyarrow.lib.ArrowInvalid: Column 1 named references expected length 534 but got length 323

metric bug

A bug in a metric script

Can you share your data files or a minimally reproducible example?

Sure, [here is a Colab notebook](https://colab.research.google.com/drive/1uiS89fnHMG7HV_cYxp3r-_LqJQvNNKs9?usp=sharing) reproducing the error.

> ArrowInvalid: Column 1 named references expected length 36 but got length 56

This is because `add` takes as input a batch of elements and you provided only one. I think we should have `add` for one prediction/reference and `add_batch` for a batch of predictions/references. This would make it more coherent with the way we use Arrow.

Let me do this change

Thanks for noticing though. I was mainly used to do `.compute` directly ^^

I'm trying to use rouge metric.

I have to files : `test.pred.tokenized` and `test.gold.tokenized` with each line containing a sentence.
I tried :

```python
import nlp

rouge = nlp.load_metric('rouge')
with open("test.pred.tokenized") as p, open("test.gold.tokenized") as g:
    for lp, lg in zip(p, g):
            rouge.add(lp, lg)
```

But I meet following error :

> pyarrow.lib.ArrowInvalid: Column 1 named references expected length 534 but got length 323

---

Full stack-trace :

```
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/home/me/.venv/transformers/lib/python3.6/site-packages/nlp/metric.py", line 224, in add
    self.writer.write_batch(batch)
  File "/home/me/.venv/transformers/lib/python3.6/site-packages/nlp/arrow_writer.py", line 148, in write_batch
    pa_table: pa.Table = pa.Table.from_pydict(batch_examples, schema=self._schema)
  File "pyarrow/table.pxi", line 1550, in pyarrow.lib.Table.from_pydict
  File "pyarrow/table.pxi", line 1503, in pyarrow.lib.Table.from_arrays
  File "pyarrow/public-api.pxi", line 390, in pyarrow.lib.pyarrow_wrap_table
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 1 named references expected length 534 but got length 323
```

(`nlp` installed from source)


I could access the info by first accessing the different splits :

```python
import nlp

cnn_dm = nlp.load_dataset('cnn_dailymail')
print(cnn_dm['train'].info)
```

Information seems to be duplicated between the subsets :

```python
print(cnn_dm["train"].info == cnn_dm["test"].info == cnn_dm["validation"].info)
# True
```

Is it expected ?

Good point @Colanim ! What happens under the hood when running:

```python
import nlp

cnn_dm = nlp.load_dataset('cnn_dailymail')
```

is that for every split in `cnn_dailymail`, a different dataset object (which all holds the same info) is created. This has the advantages that the datasets are easily separable in a training setup.
Also note that you can load e.g. only the `train` split of the dataset via:

```python
cnn_dm_train = nlp.load_dataset('cnn_dailymail', split="train")
print(cnn_dm_train.info)
```

I think we should make the `info` object slightly different when creating the dataset for each split - at the moment it contains for example the variable `splits` which should maybe be renamed to `split` and contain only one `SplitInfo` object ...

I'm trying to access the information of CNN/DM dataset :

```python
cnn_dm = nlp.load_dataset('cnn_dailymail')
print(cnn_dm.info)
```

returns :

> AttributeError: 'dict' object has no attribute 'info'


Couldn't reach CNN/DM dataset

Installing from source (instead of Pypi package) solved the problem.

I can't get CNN / DailyMail dataset.

```python
import nlp

assert "cnn_dailymail" in [dataset.id for dataset in nlp.list_datasets()]
cnn_dm = nlp.load_dataset('cnn_dailymail')
```

[Colab notebook](https://colab.research.google.com/drive/1zQ3bYAVzm1h0mw0yWPqKAg_4EUlSx5Ex?usp=sharing)

gives following error :

```
ConnectionError: Couldn't reach https://s3.amazonaws.com/datasets.huggingface.co/nlp/cnn_dailymail/cnn_dailymail.py
```


Adding docstrings and some doc

Some doc


Qa4mre - add dataset

Added dummy data test only for the first config. Will do the rest later.
I had to do add some minor hacks to an important function to make it work.
There might be a cleaner way to handle it - can you take a look @thomwolf ?


[Clean-up] remove under construction datastes


fix reddit tifu dummy data


[Reclor] fix reclor

- That's probably one me. Could have made the manual data test more flexible. @mariamabarham


convert can use manual dir as second argument

@mariamabarham


add writer_batch_size to GeneratorBasedBuilder

You can now specify `writer_batch_size` in the builder arguments or directly in `load_dataset`


Add data dir test command

Nice - I think we can merge this. I will update the checksums for `wikihow` then as well


[New structure on AWS] Adapt paths

Some small changes so that we have the correct paths. @julien-c


Add trivia_q

Currently tested only for one config to pass tests. Needs to add more dummy data later.


[Manual downloads] add logic proposal for manual downloads and add wikihow

> Wikihow is an example that needs to manually download two files as stated in: https://github.com/mahnazkoupaee/WikiHow-Dataset.
>
> The user can then store these files under a hard-coded name: `wikihowAll.csv` and `wikihowSep.csv` in this case in a directory of his choice, e.g. `~/wikihow/manual_dir`.
>
> The dataset can then be loaded via:
>
> ```python
> import nlp
> nlp.load_dataset("wikihow", data_dir="~/wikihow/manual_dir")
> ```
>
> I added/changed so that there are explicit error messages when using manually downloaded files.

wouldn't be nicer if we can have `manual_dir/wikihow`?

> > Wikihow is an example that needs to manually download two files as stated in: https://github.com/mahnazkoupaee/WikiHow-Dataset.
> > The user can then store these files under a hard-coded name: `wikihowAll.csv` and `wikihowSep.csv` in this case in a directory of his choice, e.g. `~/wikihow/manual_dir`.
> > The dataset can then be loaded via:
> > ```python
> > import nlp
> > nlp.load_dataset("wikihow", data_dir="~/wikihow/manual_dir")
> > ```
> >
> >
> > I added/changed so that there are explicit error messages when using manually downloaded files.
>
> wouldn't be nicer if we can have `manual_dir/wikihow`?

Sure, I mean the user can decide whatever he likes best :-) The path one puts in `data_dir` will be used as the path to the manual dir. `nlp.load_dataset("wikihow", data_dir="~/manual_dir/wikihow")` would work as well as any other path ;-)

Wikihow is an example that needs to manually download two files as stated in: https://github.com/mahnazkoupaee/WikiHow-Dataset.

The user can then store these files under a hard-coded name: `wikihowAll.csv` and `wikihowSep.csv` in this case in a directory of his choice, e.g. `~/wikihow/manual_dir`.

The dataset can then be loaded via:

```python
import nlp
nlp.load_dataset("wikihow", data_dir="~/wikihow/manual_dir")
```

I added/changed so that there are explicit error messages when using manually downloaded files.


Run save infos

Haha that cornell dialogue dataset - that ran for 3h on my computer as well. The `generate_examples` method in this script is one of the most inefficient code samples I've ever seen :D

Indeed it's been 3 hours already
```73111 examples [3:07:48,  2.40 examples/s]```

I replaced the old checksum file with the new `dataset_infos.json` by running the script on almost all the datasets we have. The only one that is still running on my side is the cornell dialog


[Reddit] add reddit

- Everything worked fine @mariamabarham. Made my computer nearly crash, but all seems to be working :-)


Add per type scores in seqeval metric

LGTM :-) Some small suggestions to shorten the code a bit :-)

Can you put the kwargs as normal kwargs instead of a dict? (And add them to the kwargs description As well)

Yes and there is a dynamically generated doc string in the metric script KWARGS DESCRIPTION

This PR add a bit more detail in the seqeval metric. Now the usage and output are:

```python
import nlp
met = nlp.load_metric('metrics/seqeval')
references = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
predictions =  [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
met.compute(predictions, references)

#Output: {'PER': {'precision': 1.0, 'recall': 1.0, 'f1': 1.0, 'number': 1}, 'MISC': {'precision': 0.0, 'recall': 0.0, 'f1': 0, 'number': 1}, 'overall_precision': 0.5, 'overall_recall': 0.5, 'overall_f1': 0.5, 'overall_accuracy': 0.8}
```

It is also possible to compute scores for non IOB notations, POS tagging for example hasn't this kind of notation. Add `suffix` parameter:

```python
import nlp
met = nlp.load_metric('metrics/seqeval')
references = [['O', 'O', 'O', 'MISC', 'MISC', 'MISC', 'O'], ['PER', 'PER', 'O']]
predictions =  [['O', 'O', 'MISC', 'MISC', 'MISC', 'MISC', 'O'], ['PER', 'PER', 'O']]
met.compute(predictions, references, metrics_kwargs={"suffix": True})

#Output: {'PER': {'precision': 1.0, 'recall': 1.0, 'f1': 1.0, 'number': 1}, 'MISC': {'precision': 0.0, 'recall': 0.0, 'f1': 0, 'number': 1}, 'overall_precision': 0.5, 'overall_recall': 0.5, 'overall_f1': 0.5, 'overall_accuracy': 0.9}
```


[Cmrc 2018] fix cmrc2018


Webis tl-dr

> Should that rather be in an organization scope, @thomwolf @patrickvonplaten ?

I'm a bit indifferent - both would be fine for me!

@jplu - if creating the dummy_data is too tedious, I can do it as well :-)

Yeah I think naming it webis/tl_dr would be best @jplu if that works for you

> There is dummy_data here, no ?

Some paths were wrong - the structure is really confusing and the error messages don't really help either - I have to think about how to make this easier to understand!

Hope it was ok that I fiddled with your PR !

> Some paths were wrong - the structure is really confusing and the error message don't really help either - I have to think about how to make this easier to understand!

Oh ok! I haven't noticed that sorry :(

> Hope it was ok that I fiddled with your PR !

Of course it was ok :)

@julien-c Looks like what you have in mind?

```python
import nlp
nlp.load_dataset("datasets/webis", "tl_dr")

#Output: Downloading and preparing dataset webis/tl_dr (download: Unknown size, generated: Unknown size, total: Unknown size) to /home/jplu/.cache/huggingface/datasets/webis/tl_dr/1.0.0...
```

Merging this for now. Maybe we can see whether to rename it in a different PR @julien-c ?

Hi,
Author here of the webis-tldr corpus. Any plans on integrating this dataset into the hub? I remember we could access it in the previous versions of the library. If there is a particular issue that I can help with, do let me know.

Thanks!

Hi @shahbazsyed, this dataset _is_ inside the hub but it's namespaced by the organization name `webis`.

You can load it following the steps described in https://huggingface.co/datasets/webis/tl_dr

Here's a Colab showcasing that it works: https://colab.research.google.com/drive/11IrzRVpnMLJZ8_UFFHLR8FhiajjAHRUU?usp=sharing

The reason the code is in S3 and not in this repo is that the dataset is namespaced under the `webis` organization. We don't have a lot of namespaced datasets yet but this should become the main way we add more datasets in the future.
Let us know if that's an issue for you. Thank you!

Add the Webid TL:DR dataset.


[Csv] add tests for csv dataset script

@thomwolf - can you check and merge if ok?

Adds dummy data tests for csv.


lm1b

I might have a different version of `isort` than others. It seems like I'm always reordering the imports of others. But isn't really a problem...

Add lm1b dataset.


Replace checksums files by Dataset infos json

> Ok, really clean!
> I like the logic (not a huge fan of using `_asdict_inner` but it makes sense).
> I think it's a nice improvement!
>
> How should we update the files in the repo? Run a big job on a server or on somebody's computer who has most of the datasets already downloaded?

Maybe we can split the updates among us...IMO most datasets run very quickly.
I think I've downloaded 50 datasets and 80% are loaded in <5min,  15% in <1h and then `wmt` which is still downloading (since 12h).
I deleted my cache because the `wmt` downloads require quite a lot of space, so I only have parts of the `wmt` datasets on my computer.

@mariamabarham I guess you have downloaded most of the datasets no?

### Better verifications when loading a dataset

I replaced the `urls_checksums` directory that used to contain `checksums.txt` and `cached_sizes.txt`, by a single file `dataset_infos.json`. It's just a dict `config_name` -> `DatasetInfo`.

It simplifies and improves how verifications of checksums and splits sizes are done, as they're all stored in `DatasetInfo` (one per config). Also, having already access to `DatasetInfo` enables to check disk space before running `download_and_prepare` for a given config.

The dataset infos json file is user readable, you can take a look at the squad one that I generated in this PR.

### Renaming

According to these changes, I did some renaming:
`save_checksums` -> `save_infos`
`ignore_checksums` -> `ignore_verifications`

for example, when you are creating a dataset you have to run
```nlp-cli test path/to/my/dataset --save_infos --all_configs```
instead of
```nlp-cli test path/to/my/dataset --save_checksums --all_configs```

### And now, the fun part

We'll have to rerun the `nlp-cli test ... --save_infos --all_configs` for all the datasets

-----------------

feedback appreciated !


Librispeech

@jplu - I changed this weird archieve - iter method to something simpler. It's only one file to download anyways so I don't see the point of using weird iter methods...It's a huge file though :D 30 million lines of text. Took me quite some time to download :D

Add librispeech dataset and remove some useless content.


Cleanup notebooks and various fixes

Fixes on dataset (more flexible) metrics (fix) and general clean ups


[WIP] add wmt14

WMT14 takes forever to download :-/

- WMT is the first dataset that uses an abstract class IMO, so I had to modify the `load_dataset_module` a bit.


[Paracrawl] add paracrawl

- Huge dataset - took ~1h to download
- Also this PR reformats all dataset scripts and adds `datasets` to `make style`


Add download gg drive

awesome - so no manual downloaded needed here?

Yes exactly. It works like a standard download

We can now add datasets that download from google drive


Add list and inspect methods - cleanup hf_api

Add a bunch of methods to easily list and inspect the processing scripts up-loaded on S3:
```python
nlp.list_datasets()
nlp.list_metrics()
# Copy and prepare the scripts at `local_path` for easy inspection/modification.
nlp.inspect_dataset(path, local_path)
# Copy and prepare the scripts at `local_path` for easy inspection/modification.
nlp.inspect_metric(path, local_path)
```

Also clean up the `HfAPI` to use `dataclasses` for better user-experience


Add wiki40b

Looks good to me. I have not really looked too much into the Beam Datasets yet though - so I think you can merge whenever you think is good for Beam datasets :-)

This one is a beam dataset that downloads files using tensorflow.
I tested it on a small config and it works fine


Add Flores

Beautiful language for sure!


[Load => load_dataset] change naming

Rename leftovers @thomwolf