Correct data structure for PAN-X task in XTREME dataset?

Thanks for noticing ! This looks more reasonable indeed.
Feel free to open a PR

Hi @lhoestq
I made the proposed changes to the `xtreme.py` script. I noticed that I also need to change the schema in the `dataset_infos.json` file.  More specifically the `"features"` part of the PAN-X.LANG dataset:

```json
"features":{
   "word":{
      "dtype":"string",
      "id":null,
      "_type":"Value"
   },
   "ner_tag":{
      "dtype":"string",
      "id":null,
      "_type":"Value"
   },
   "lang":{
      "dtype":"string",
      "id":null,
      "_type":"Value"
   }
}
```
To fit the code above the fields `"word"`, `"ner_tag"`, and `"lang"` would become `"words"`, `ner_tags"` and `"langs"`. In addition the `dtype` should be changed from `"string"` to `"list"`.

 I made this changes but when trying to test this locally with `dataset = load_dataset("xtreme", "PAN-X.en", data_dir='./data')` I face the issue that the `dataset_info.json` file is always overwritten by a downloaded version with the old settings, which then throws an error because the schema does not match. This makes it hard to test the changes locally. Do you have any suggestions on how to deal with that?

Hi !

You have to point to your local script.
First clone the repo and then:

```python
dataset = load_dataset("./datasets/xtreme", "PAN-X.en")
```
The "xtreme" directory contains "xtreme.py".

You also have to change the features definition in the `_info` method. You could use:

```python
features = nlp.Features({
    "words": [nlp.Value("string")],
    "ner_tags": [nlp.Value("string")],
    "langs": [nlp.Value("string")],
})
```

Hope this helps !
Let me know if you have other questions.

Thanks, I am making progress. I got a new error `NonMatchingSplitsSizesError ` (see traceback below), which I suspect is due to the fact that number of rows in the dataset changed (one row per word --> one row per sentence) as well as the number of bytes due to the slightly updated data structure.

```python
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='validation', num_bytes=1756492, num_examples=80536, dataset_name='xtreme'), 'recorded': SplitInfo(name='validation', num_bytes=1837109, num_examples=10000, dataset_name='xtreme')}, {'expected': SplitInfo(name='test', num_bytes=1752572, num_examples=80326, dataset_name='xtreme'), 'recorded': SplitInfo(name='test', num_bytes=1833214, num_examples=10000, dataset_name='xtreme')}, {'expected': SplitInfo(name='train', num_bytes=3496832, num_examples=160394, dataset_name='xtreme'), 'recorded': SplitInfo(name='train', num_bytes=3658428, num_examples=20000, dataset_name='xtreme')}]
```
I can fix the error by replacing the values in the `datasets_infos.json` file, which I tested for English. However, to update this for all 40 datasets manually is slightly painful. Is there a better way to update the expected values for all datasets?

You can update the json file by calling
```
nlp-cli test ./datasets/xtreme --save_infos --all_configs
```

One more thing about features. I mentioned

```python
features = nlp.Features({
    "words": [nlp.Value("string")],
    "ner_tags": [nlp.Value("string")],
    "langs": [nlp.Value("string")],
})
```

but it's actually not consistent with the way we write datasets. Something like this is simpler to read and more consistent with the way we define datasets:

```python
features = nlp.Features({
    "words": nlp.Sequence(nlp.Value("string")),
    "ner_tags": nlp.Sequence(nlp.Value("string")),
    "langs": nlp.Sequence(nlp.Value("string")),
})
```

Sorry about that

Closing this since PR #437 fixed the problem and has been merged to `master`.

Hi 🤗  team!

## Description of the problem
Thanks to the fix from #416 I am now able to load the NER task in the XTREME dataset as follows:

```python
from nlp import load_dataset
# AmazonPhotos.zip is located in data/
dataset = load_dataset("xtreme", "PAN-X.en", data_dir='./data')
dataset_train = dataset['train']
```

However, I am not sure that `load_dataset()` is returning the correct data structure for NER.

Currently, every row in `dataset_train` is of the form
```python
{'word': str, 'ner_tag': str, 'lang': str}
```
but I think we actually want something like
```python
{'words': List[str], 'ner_tags': List[str], 'langs': List[str]}
```
so that each row corresponds to a _sequence_ of words associated with each example. With the current data structure I do not think it is possible to transform `dataset_train` into a form suitable for training because we do not know the boundaries between examples.

Indeed, [this line](https://github.com/google-research/xtreme/blob/522434d1aece34131d997a97ce7e9242a51a688a/third_party/utils_tag.py#L58) in the XTREME repo, processes the texts as lists of sentences, tags, and languages.

## Proposed solution
Replace
```python
with open(filepath) as f:
    data = csv.reader(f, delimiter="\t", quoting=csv.QUOTE_NONE)
    for id_, row in enumerate(data):
        if row:
            lang, word = row[0].split(":")[0], row[0].split(":")[1]
            tag = row[1]
            yield id_, {"word": word, "ner_tag": tag, "lang": lang}
```
from  [these lines](https://github.com/huggingface/nlp/blob/ce7d3a1d630b78fe27188d1706f3ea980e8eec43/datasets/xtreme/xtreme.py#L881-L887) of the `_generate_examples()` function with something like

```python
guid_index = 1
with open(filepath, encoding="utf-8") as f:
    words = []
    ner_tags = []
    langs = []
    for line in f:
        if line.startswith("-DOCSTART-") or line == "" or line == "\n":
            if words:
                yield guid_index, {"words": words, "ner_tags": ner_tags, "langs": langs}
                guid_index += 1
                words = []
                ner_tags = []
        else:
            # pan-x data is tab separated
            splits = line.split("\t")
            # strip out en: prefix
            langs.append(splits[0][:2])
            words.append(splits[0][3:])
            if len(splits) > 1:
                labels.append(splits[-1].replace("\n", ""))
            else:
                # examples have no label in test set
                labels.append("O")
```
If you agree, me or @lvwerra would be happy to implement this and create a PR.


Web of science

this PR adds the WebofScience dataset
#353


Change features vs schema logic

I had to make `SplitDict` serializable to be able to copy `DatasetInfo` objects properly.
Serialization was also asked in #389

One thing I forgot to say here, is that we also want to use the features arguments of `load_dataset` (which goes in the builder’s config) to override the default features of a dataset script.

## New logic for `nlp.Features` in datasets

Previously, it was confusing to have `features` and pyarrow's `schema` in `nlp.Dataset`.
However `features` is supposed to be the front-facing object to define the different fields of a dataset, while `schema` is only used to write arrow files.

Changes:
- Remove `schema` field in `nlp.Dataset`
- Make `features` the source of truth to read/write examples
- `features` can no longer be `None` in `nlp.Dataset`
- Update `features` after each dataset transform such as `nlp.Dataset.map`

Todo: change the tests to take these changes into account


- Corrected encoding for IMDB.

The preparation phase (after the download phase) crashed on windows because of charmap encoding not being able to decode certain characters. This change suggested in Issue #347 fixes it for the IMDB dataset.


Style change

Oh this is the PR where I ran make quality and make style and some previous files from master were changed

Oh right ! Let me fix the style myself if you don't mind

make quality and make style ran on scripts


Better handle nested features

Changes:
- added arrow schema to features conversion (it's going to be useful to fix #342 )
- make flatten handle deep features (useful for tfrecords conversion in #339 )
- add tests for flatten and features conversions
- the reader now returns the kwargs to instantiate a Dataset (fix circular dependencies)


EmoContext dataset add

EmoContext Dataset add

Signed-off-by: lordtt13 <thakurtanmay72@yahoo.com>


Addition of google drive links to dl_manager

I think the problem is the way you wrote your urls. Try the following structure to see `https://drive.google.com/uc?export=download&id=your_file_id` .

@lhoestq

Oh sorry, I think `_get_drive_url` is doing that.

Have you tried to use `dl_manager.download_and_extract(_get_drive_url(_TRAIN_URL)`? it should work with google drive links.

Hello there, I followed the template to create a download script of my own, which works fine for me, although I had to shun the dl_manager because it was downloading nothing from the drive links and instead use gdown.

This is the script for me:

```python
class EmoConfig(nlp.BuilderConfig):
    """BuilderConfig for SQUAD."""

    def __init__(self, **kwargs):
        """BuilderConfig for EmoContext.
    Args:
      **kwargs: keyword arguments forwarded to super.
    """
        super(EmoConfig, self).__init__(**kwargs)

_TEST_URL = "https://drive.google.com/file/d/1Hn5ytHSSoGOC4sjm3wYy0Dh0oY_oXBbb/view?usp=sharing"
_TRAIN_URL = "https://drive.google.com/file/d/12Uz59TYg_NtxOy7SXraYeXPMRT7oaO7X/view?usp=sharing"

class EmoDataset(nlp.GeneratorBasedBuilder):
    """ SemEval-2019 Task 3: EmoContext Contextual Emotion Detection in Text. Version 1.0.0 """

    VERSION = nlp.Version("1.0.0")
    force = False

    def _info(self):
        return nlp.DatasetInfo(
            description=_DESCRIPTION,
            features=nlp.Features(
                {
                    "text": nlp.Value("string"),
                    "label": nlp.features.ClassLabel(names=["others", "happy", "sad", "angry"]),
                }
            ),
            supervised_keys=None,
            homepage="https://www.aclweb.org/anthology/S19-2005/",
            citation=_CITATION,
        )

    def _get_drive_url(self, url):
        base_url = 'https://drive.google.com/uc?id='
        split_url = url.split('/')
        return base_url + split_url[5]

    def _split_generators(self, dl_manager):
        """Returns SplitGenerators."""
        if(not os.path.exists("emo-train.json") or self.force):
            gdown.download(self._get_drive_url(_TRAIN_URL), "emo-train.json", quiet = True)
        if(not os.path.exists("emo-test.json") or self.force):
            gdown.download(self._get_drive_url(_TEST_URL), "emo-test.json", quiet = True)
        return [
            nlp.SplitGenerator(
                name=nlp.Split.TRAIN,
                gen_kwargs={
                    "filepath": "emo-train.json",
                    "split": "train",
                },
            ),
            nlp.SplitGenerator(
                name=nlp.Split.TEST,
                gen_kwargs={"filepath": "emo-test.json", "split": "test"},
            ),
        ]

    def _generate_examples(self, filepath, split):
        """ Yields examples. """
        with open(filepath, 'rb') as f:
            data = json.load(f)
            for id_, text, label in zip(data["text"].keys(), data["text"].values(), data["Label"].values()):
                yield id_, {
                    "text": text,
                    "label": label,
                }
```

Can someone help me in adding gdrive links to be used with default dl_manager or adding gdown as another dl_manager, because I'd like to add this dataset to nlp's official database.


Fix docstrins multiple metrics instances

We change the docstrings of `nlp.Metric.compute`, `nlp.Metric.add` and `nlp.Metric.add_batch` depending on which metric is instantiated. However we had issues when instantiating multiple metrics (docstrings were duplicated).

This should fix #304


Fix xtreme panx directory

great, I think I did not download the data the way you do, but yours is more reasonable.

Fix #412


Something is wrong with WMT 19 kk-en dataset

dataset bug

A bug in a dataset script provided in the library

The translation in the `train` set does not look right:

```
>>>import nlp
>>>from nlp import load_dataset
>>>dataset = load_dataset('wmt19', 'kk-en')
>>>dataset["train"]["translation"][0]
{'kk': 'Trumpian Uncertainty', 'en': 'Трамптық белгісіздік'}
>>>dataset["validation"]["translation"][0]
{'kk': 'Ақша-несие саясатының сценарийін қайта жазсақ', 'en': 'Rewriting the Monetary-Policy Script'}
```


from_dict delete?

`from_dict` was added in #350 that was unfortunately not included in the 0.3.0 release. It's going to be included in the next release that will be out pretty soon though.
Right now if you want to use `from_dict` you have to install the package from the master branch
```
pip install git+https://github.com/huggingface/nlp.git
```

> `from_dict` was added in #350 that was unfortunately not included in the 0.3.0 release. It's going to be included in the next release that will be out pretty soon though.
> Right now if you want to use `from_dict` you have to install the package from the master branch
>
> ```
> pip install git+https://github.com/huggingface/nlp.git
> ```
OK, thank you.

AttributeError: type object 'Dataset' has no attribute 'from_dict'


Is there a way to download only NQ dev?

Unfortunately it's not possible to download only the dev set of NQ.

I think we could add a way to download only the test set by adding a custom configuration to the processing script though.

Ok, got it. I think this could be a valuable feature - especially for large datasets like NQ, but potentially also others.
For us, it will in this case make the difference of using the library or keeping the old downloads of the raw dev datasets.
However, I don't know if that fits into your plans with the library and can also understand if you don't want to support this.

I don't think we could force this behavior generally since the dataset script authors are free to organize the file download as they want (sometimes the mapping between split and files can be very much nontrivial) but we can add an additional configuration for Natural Question indeed as @lhoestq indicate.

Maybe I missed that in the docs, but is there a way to only download the dev set of natural questions (~1 GB)?
As we want to benchmark QA models on different datasets, I would like to avoid downloading the 41GB of training data.

I tried
```
dataset = nlp.load_dataset('natural_questions', split="validation", beam_runner="DirectRunner")
```
But this still triggered a big download of presumably the whole dataset. Is there any way of doing this or are splits / slicing options only available after downloading?

Thanks!


Unable to load XTREME dataset from disk

Hi @lewtun, you have to provide the full path to the downloaded file for example `/home/lewtum/..`

I was able to repro. Opening a PR to fix that.
Thanks for reporting this issue !

Hi 🤗  team!

## Description of the problem
Following the [docs](https://huggingface.co/nlp/loading_datasets.html?highlight=xtreme#manually-downloading-files) I'm trying to load the `PAN-X.fr` dataset from the [XTREME](https://github.com/google-research/xtreme) benchmark.

I have manually downloaded the `AmazonPhotos.zip` file from [here](https://www.amazon.com/clouddrive/share/d3KGCRCIYwhKJF0H3eWA26hjg2ZCRhjpEQtDL70FSBN?_encoding=UTF8&%2AVersion%2A=1&%2Aentries%2A=0&mgh=1) and am running into a `FileNotFoundError` when I point to the location of the dataset.

As far as I can tell, the problem is that `AmazonPhotos.zip` decompresses to `panx_dataset` and `load_dataset()` is not looking in the correct path:

```
# path where load_dataset is looking for fr.tar.gz
/root/.cache/huggingface/datasets/9b8c4f1578e45cb2539332c79738beb3b54afbcd842b079cabfd79e3ed6704f6/
# path where it actually exists
/root/.cache/huggingface/datasets/9b8c4f1578e45cb2539332c79738beb3b54afbcd842b079cabfd79e3ed6704f6/panx_dataset/
```

## Steps to reproduce the problem

1. Manually download the XTREME benchmark from [here](https://www.amazon.com/clouddrive/share/d3KGCRCIYwhKJF0H3eWA26hjg2ZCRhjpEQtDL70FSBN?_encoding=UTF8&%2AVersion%2A=1&%2Aentries%2A=0&mgh=1)

2. Run the following code snippet
```python
from nlp import load_dataset
# AmazonPhotos.zip is in the root of the folder
dataset = load_dataset("xtreme", "PAN-X.fr", data_dir='./')
```

3. Here is the stack trace
```
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-4-26786bb5fa93> in <module>
----> 1 dataset = load_dataset("xtreme", "PAN-X.fr", data_dir='./')

/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    522         download_mode=download_mode,
    523         ignore_verifications=ignore_verifications,
--> 524         save_infos=save_infos,
    525     )
    526

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    430                 verify_infos = not save_infos and not ignore_verifications
    431                 self._download_and_prepare(
--> 432                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    433                 )
    434                 # Sync info

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    464         split_dict = SplitDict(dataset_name=self.name)
    465         split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 466         split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    467         # Checksums verification
    468         if verify_infos:

/usr/local/lib/python3.6/dist-packages/nlp/datasets/xtreme/b8c2ed3583a7a7ac60b503576dfed3271ac86757628897e945bd329c43b8a746/xtreme.py in _split_generators(self, dl_manager)
    725             panx_dl_dir = dl_manager.extract(panx_path)
    726             lang = self.config.name.split(".")[1]
--> 727             lang_folder = dl_manager.extract(os.path.join(panx_dl_dir, lang + ".tar.gz"))
    728             return [
    729                 nlp.SplitGenerator(

/usr/local/lib/python3.6/dist-packages/nlp/utils/download_manager.py in extract(self, path_or_paths)
    196         """
    197         return map_nested(
--> 198             lambda path: cached_path(path, extract_compressed_file=True, force_extract=False), path_or_paths,
    199         )
    200

/usr/local/lib/python3.6/dist-packages/nlp/utils/py_utils.py in map_nested(function, data_struct, dict_only, map_tuple)
    170                 return tuple(mapped)
    171     # Singleton
--> 172     return function(data_struct)
    173
    174

/usr/local/lib/python3.6/dist-packages/nlp/utils/download_manager.py in <lambda>(path)
    196         """
    197         return map_nested(
--> 198             lambda path: cached_path(path, extract_compressed_file=True, force_extract=False), path_or_paths,
    199         )
    200

/usr/local/lib/python3.6/dist-packages/nlp/utils/file_utils.py in cached_path(url_or_filename, download_config, **download_kwargs)
    203     elif urlparse(url_or_filename).scheme == "":
    204         # File, but it doesn't exist.
--> 205         raise FileNotFoundError("Local file {} doesn't exist".format(url_or_filename))
    206     else:
    207         # Something unknown

FileNotFoundError: Local file /root/.cache/huggingface/datasets/9b8c4f1578e45cb2539332c79738beb3b54afbcd842b079cabfd79e3ed6704f6/fr.tar.gz doesn't exist
```

## OS and hardware
```
- `nlp` version: 0.3.0
- Platform: Linux-4.15.0-72-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.1.0 (True)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```


Sbf

This PR adds the Social Bias Frames Dataset (ACL 2020) .
dataset homepage: https://homes.cs.washington.edu/~msap/social-bias-frames/


20newsgroup

Add 20Newsgroup dataset.
#353


train_test_split error: 'dict' object has no attribute 'deepcopy'

It was fixed in 2ddd18d139d3047c9c3abe96e1e7d05bb360132c.
Could you pull the latest changes from master @morganmcg1 ?

`train_test_split` is giving me an error when I try and call it:

`'dict' object has no attribute 'deepcopy'`

## To reproduce

```
dataset = load_dataset('glue', 'mrpc', split='train')
dataset = dataset.train_test_split(test_size=0.2)
```

## Full Stacktrace
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-12-feb740dbec9a> in <module>
      1 dataset = load_dataset('glue', 'mrpc', split='train')
----> 2 dataset = dataset.train_test_split(test_size=0.2)

~/anaconda3/envs/fastai2_me/lib/python3.7/site-packages/nlp/arrow_dataset.py in train_test_split(self, test_size, train_size, shuffle, seed, generator, keep_in_memory, load_from_cache_file, train_cache_file_name, test_cache_file_name, writer_batch_size)
   1032                     "writer_batch_size": writer_batch_size,
   1033                 }
-> 1034                 train_kwargs = cache_kwargs.deepcopy()
   1035                 train_kwargs["split"] = "train"
   1036                 test_kwargs = cache_kwargs.deepcopy()

AttributeError: 'dict' object has no attribute 'deepcopy'
```


Add tests datasets gcp

Some datasets are available on our google cloud storage in arrow format, so that the users don't need to process the data.
These tests make sure that they're always available. It also makes sure that their scripts are in sync between S3 and the repo.
This should avoid future issues like #407


MissingBeamOptions for Wikipedia 20200501.en

Fixed. Could you try again @mitchellgordon95 ?
It was due a file not being updated on S3.

We need to make sure all the datasets scripts get updated properly @julien-c

I found the same issue with almost any language other than English. (For English, it works). Will someone need to update the file on S3 again?

This is because only some languages are already preprocessed (en, de, fr, it) and stored on our google storage.
We plan to have a systematic way to preprocess more wikipedia languages in the future.

For the other languages you have to process them on your side using apache beam. That's why the lib asks for a Beam runner.

There may or may not be a regression for the pre-processed Wikipedia dataset. This was working fine 10 commits ago (without having Apache Beam available):

```
nlp.load_dataset('wikipedia', "20200501.en", split='train')
```

And now, having pulled master, I get:

```
Downloading and preparing dataset wikipedia/20200501.en (download: 16.99 GiB, generated: 17.07 GiB, total: 34.06 GiB) to /home/hltcoe/mgordon/.cache/huggingface/datasets/wikipedia/20200501.en/1.0.0/76b0b2747b679bb0ee7a1621e50e5a6378477add0c662668a324a5bc07d516dd...
Traceback (most recent call last):
  File "scripts/download.py", line 11, in <module>
    fire.Fire(download_pretrain)
  File "/home/hltcoe/mgordon/.conda/envs/huggingface/lib/python3.6/site-packages/fire/core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/hltcoe/mgordon/.conda/envs/huggingface/lib/python3.6/site-packages/fire/core.py", line 468, in _Fire
    target=component.__name__)
  File "/home/hltcoe/mgordon/.conda/envs/huggingface/lib/python3.6/site-packages/fire/core.py", line 672, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "scripts/download.py", line 6, in download_pretrain
    nlp.load_dataset('wikipedia', "20200501.en", split='train')
  File "/exp/mgordon/nlp/src/nlp/load.py", line 534, in load_dataset
    save_infos=save_infos,
  File "/exp/mgordon/nlp/src/nlp/builder.py", line 460, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/exp/mgordon/nlp/src/nlp/builder.py", line 870, in _download_and_prepare
    "\n\t`{}`".format(usage_example)
nlp.builder.MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, S
park, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory).
Example of usage:
        `load_dataset('wikipedia', '20200501.en', beam_runner='DirectRunner')`
```


Faster Shuffling?

I think the slowness here probably come from the fact that we are copying from and to python.

@lhoestq for all the `select`-based methods I think we should stay in Arrow format and update the writer so that it can accept Arrow tables or batches as well. What do you think?

> @lhoestq for all the `select`-based methods I think we should stay in Arrow format and update the writer so that it can accept Arrow tables or batches as well. What do you think?

I just tried with `writer.write_table` with tables of 1000 elements and it's slower that the solution in #405

On my side (select 10 000 examples):
- Original implementation: 12s
- Batched solution: 100ms
- solution using arrow tables: 350ms

I'll try with arrays and record batches to see if we can make it work.

I tried using `.take` from pyarrow recordbatches but it doesn't improve the speed that much:
```python
import nlp
import numpy as np

dset = nlp.Dataset.from_file("dummy_test_select.arrow")  # dummy dataset with 100000 examples like {"a": "h"*512}
indices = np.random.randint(0, 100_000, 1000_000)
```

```python
%%time
batch_size = 10_000
writer = ArrowWriter(schema=dset.schema, path="dummy_path",
                     writer_batch_size=1000, disable_nullable=False)
for i in tqdm(range(0, len(indices), batch_size)):
    table = pa.concat_tables(dset._data.slice(int(i), 1) for i in indices[i : min(len(indices), i + batch_size)])
    batch = table.to_pydict()
    writer.write_batch(batch)
writer.finalize()
# 9.12s
```

```python
%%time
batch_size = 10_000
writer = ArrowWriter(schema=dset.schema, path="dummy_path",
                     writer_batch_size=1000, disable_nullable=False)
for i in tqdm(range(0, len(indices), batch_size)):
    batch_indices = indices[i : min(len(indices), i + batch_size)]
    # First, extract only the indices that we need with a mask
    mask = [False] * len(dset)
    for k in batch_indices:
        mask[k] = True
    t_batch = dset._data.filter(pa.array(mask))
    # Second, build the list of indices for the filtered table, and taking care of duplicates
    rev_positions = {}
    duplicates = 0
    for i, j in enumerate(sorted(batch_indices)):
        if j in rev_positions:
            duplicates += 1
        else:
            rev_positions[j] = i - duplicates
    rev_map = [rev_positions[j] for j in batch_indices]
    # Third, use `.take` from the combined recordbatch
    t_combined = t_batch.combine_chunks()  # load in memory
    recordbatch = t_combined.to_batches()[0]
    table = pa.Table.from_arrays(
        [recordbatch[c].take(pa.array(rev_map)) for c in range(len(dset._data.column_names))],
        schema=writer.schema
    )
    writer.write_table(table)
writer.finalize()
# 3.2s
```

Shuffling is now significantly faster thanks to #513
Feel free to play with it now :)

Closing this one, but feel free to re-open if you have other questions

> Shuffling is now significantly faster thanks to #513 Feel free to play with it now :)
>
> Closing this one, but feel free to re-open if you have other questions

I have a similar issue. My code is

```
    for batch_num in range(num_batches):
        print(f'--> {batch_num=}\n') if verbose else None
        # - Get batch
        shuffled_dataset = dataset.shuffle(buffer_size=buffer_size, seed=seed)
        raw_text_batch = shuffled_dataset.take(batch_size)
        tokenized_batch = map(raw_text_batch)
        if verbose:
            time_start = time.time()
            print(f'{raw_text_batch=}')
            print(f'{tokenized_batch=}')
            print(f'{next(iter(raw_text_batch))=}')
            print(f'{next(iter(tokenized_batch))=}')
            print(f'Time it took: {time.time() - time_start} seconds \a\n')
   ```

without the suffle it takes 4.5 secs with it takes 87.1 secs. Is this difference expected? my dataset version is:
```
(beyond_scale) brando9@ampere1:~/beyond-scale-language-data-diversity$ pip list | grep dataset
datasets                             2.14.3
```

@lhoestq thoughts?

Shuffling leads to doing random access in many different locations on disk which is slower than reading contiguous data.

There is a super fast approximate shuffling algorithm implemented for iterable datasets though:

```python
iterable_dataset = dataset.to_iterable_dataset(num_shards=1024)
shuffled_dataset = iterable_dataset.shuffle(buffer_size=1000)
```

(the first batch might be a bit slow to get because the algorithm first fills a buffer before returning the first batch, see the [docs](https://huggingface.co/docs/datasets/v2.14.4/en/stream#shuffle) for more info)

Consider shuffling bookcorpus:

```
dataset = nlp.load_dataset('bookcorpus', split='train')
dataset.shuffle()
```
According to tqdm, this will take around 2.5 hours on my machine to complete (even with the faster version of select from #405). I've also tried with `keep_in_memory=True` and `writer_batch_size=1000`.

But I can also just write the lines to a text file:

```
batch_size = 100000
with open('tmp.txt', 'w+') as out_f:
    for i in tqdm(range(0, len(dataset), batch_size)):
        batch = dataset[i:i+batch_size]['text']
        print("\n".join(batch), file=out_f)
```

Which completes in a couple minutes, followed by `shuf tmp.txt > tmp2.txt` which completes in under a minute. And finally,

```
dataset = nlp.load_dataset('text', data_files='tmp2.txt')
```

Which completes in under 10 minutes. I read up on Apache Arrow this morning, and it seems like the columnar data format is not especially well-suited to shuffling rows, since moving items around requires a lot of book-keeping.

Is shuffle inherently slow, or am I just using it wrong? And if it is slow, would it make sense to try converting the data to a row-based format on disk and then shuffling? (Instead of calling select with a random permutation, as is currently done.)


Make select() faster by batching reads

Here's a benchmark:

```
dataset = nlp.load_dataset('bookcorpus', split='train')

start = time.time()
dataset.select(np.arange(1000), reader_batch_size=1, load_from_cache_file=False)
end = time.time()
print(f'{end - start}')

start = time.time()
dataset.select(np.arange(1000), reader_batch_size=1000, load_from_cache_file=False)
end = time.time()
print(f'{end - start}')
```

Without batching, select takes around 1.27 seconds. With batching, it takes around 0.01 seconds. The slowness was upsetting me because dataset.shuffle() was supposed to take ~27 hours for bookcorpus. Now with the fix it takes ~2.5 hours (which still is pretty slow, but I'll open a separate issue for that).


Add seed in metrics

With #361 we noticed that some metrics were not deterministic.
In this PR I allow the user to specify numpy's seed when instantiating a metric with `load_metric`.
The seed is set only when `compute` is called, and reset afterwards.

Moreover when calling `compute` with the same metric instance (i.e. same experiment_id), the metric will always return the same results given the same inputs. This is the case even if the seed is was not specified by the user, as the previous seed is going to be reused.

However, instantiating twice a metric (two different experiments) without specifying a seed can create different results.


return python objects instead of arrays by default

We were using to_pandas() to convert from arrow types, however it returns numpy arrays instead of python lists.
I fixed it by using to_pydict/to_pylist instead.

Fix #387
It was mentioned in https://github.com/huggingface/transformers/issues/5729


Search qa

add SearchQA dataset

#336


add web_questions

What does the `nlp-cli dummy_data` command returns ?

`test.json` -> `test`
and
`train.json` -> `train`

as shown by the `nlp-cli dummy_data` command ;-)

LGTM for merge @lhoestq - I let you merge if you want to.

add Web Question dataset
#336

Maybe @patrickvonplaten you can help with the dummy_data structure? it still broken


Web questions

add the WebQuestion dataset
#336


Spelling mistake

In "Formatting the dataset" part, "The two toehr modifications..." should be "The two other modifications..." ,the word "other" wrong spelled as "toehr".


Add inline links

Do you mind adding a link to the much more extended pages on adding and sharing a dataset in the new documentation?

Add inline links to `Contributing.md`


Add contiguous sharding

This makes dset.shard() play nice with nlp.concatenate_datasets(). When I originally wrote the shard() method, I was thinking about a distributed training scenario, but https://github.com/huggingface/nlp/pull/389 also uses it for splitting the dataset for distributed preprocessing.

Usage:
```
nlp.concatenate_datasets([dset.shard(n, i, contiguous=True) for i in range(n)])
```


Fix memory issue when doing select

We were passing the `nlp.Dataset` object to get the hash for the new dataset's file name.
Fix #395


Memory issue when doing select

As noticed in #389, the following code loads the entire wikipedia in memory.

```python
import nlp
w = nlp.load_dataset("wikipedia", "20200501.en", split="train")
w.select([0])
```

This is caused by [this line](https://github.com/huggingface/nlp/blob/master/src/nlp/arrow_dataset.py#L626) for some reason, that tries to serialize the function with all the wikipedia data with it.

It's not the case with `.map` or `.filter`.
However functions that are based on `.select` like `.shuffle`, `.shard`, `.train_test_split`, `.sort` are affected.


Remove remaining nested dict

This PR deletes the remaining unnecessary nested dict
#378


Fix extracted files directory for the DownloadManager

The cache dir was often cluttered by extracted files because of the download manager.

For downloaded files, we are using the `downloads` directory to make things easier to navigate, but extracted files were still placed at the root of the cache directory. To fix that I changed the directory for extracted files to cache_dir/downloads/extracted.


Style change detection

Another [PAN task](https://pan.webis.de/clef20/pan20-web/style-change-detection.html). This time about identifying when the style/author changes in documents.

- There's the possibility of adding the [PAN19](https://zenodo.org/record/3577602) and PAN18 style change detection tasks too (these are datasets whose labels are a subset of PAN20's). These would probably make more sense as separate datasets (like wmt is now)
- I've converted the integer 0,1 values to a boolean
- Using manually downloaded data again. This might be changed at some point following the discussion in https://github.com/huggingface/nlp/pull/349.


Concatenate datasets

Looks cool :)

I feel like
```python
concatenated_dataset = dataset1.concatenate(dataset2)
```
could be more natural. What do you think ?

Also could you also concatenate the `nlp.Dataset._data_files` ?
```python
return cls(table, info=info, split=split, data_files=self._data_files + other_dataset._data_files)
```

I feel like "WikiBooks" would be a multi task dataset that could fit in the #217 discussion.
Not sure concatenate should be the solution for a multi task dataset.

Thanks for the suggestion! `dset1.concatenate(dset2)` does feel more natural. Although this seems to be a different "class" of transformation function than map() or filter(), acting on two datasets rather than on one. I would prefer the function signature treat both datasets symmetrically.

Python lists have `list1 + list2` or `list1.extend(list2)`.
NumPy has `np.concatenate((arr1, arr2))`.
Pandas has `pd.join((df1, df2))`.
PyTorch has `ConcatDataset((dset1, dset2))`.

Given the symmetrical treatment and clear communication that this creates a new object, rather than a simple chaining on the first, my preference is now for `nlp.concatenate((dset1, dset2))`. This would place the function in the same API class as `nlp.load_dataset`. Does that work?

The multi-task discussion is interesting, thanks for pointing me to that! I'll be focusing on T5 in a few weeks, so I'm sure I'll have many opinions then :). For now, I think a simple concatenate feature is important and orthogonal to that discussion. For example, a user may want to create a custom dataset that joins Wikipedia with their own custom text.

> Given the symmetrical treatment and clear communication that this creates a new object, rather than a simple chaining on the first, my preference is now for `nlp.concatenate((dset1, dset2))`. This would place the function in the same API class as `nlp.load_dataset`. Does that work?

Yep I like this idea. Maybe `nlp.concatenate_datasets()` ?

> For now, I think a simple concatenate feature is important and orthogonal to that discussion. For example, a user may want to create a custom dataset that joins Wikipedia with their own custom text.

I agree :)

I'm constructing the "WikiBooks" dataset, which is a concatenation of Wikipedia & BookCorpus. So I implemented the `Dataset.from_concat()` method, which concatenates two datasets with the same schema.

This would also be useful if someone wants to pretrain on a large generic dataset + their own custom dataset. Not in love with the method name, so would love to hear suggestions.

Usage:
```python
from nlp import Dataset, load_dataset

data1, data2 = {"id": [0, 1, 2]}, {"id": [3, 4, 5]}
dset1, dset2 = Dataset.from_dict(data1), Dataset.from_dict(data2)
dset_concat = Dataset.from_concat([dset1, dset2])
print(dset_concat)
# Dataset(schema: {'id': 'int64'}, num_rows: 6)
```


Fix pickling of SplitDict

By the way, the reason this is an issue for me is because I want to be able to "save" changes made to a dataset by writing something to disk. In this case, I would like to pre-process my dataset once, and then train multiple models on the dataset later without having to re-process the data.

Is pickling/unpickling the Dataset object the "sanctioned" way of doing this? Or is there a better way that I'm missing?

I've had success with saving datasets to disk via:

```python
cache_file = "/my/dset.cache"
dset = dset.map(whatever, cache_file_name=cache_file)
# then, later
dset = nlp.Dataset.from_file(cache_file)
```

This restores the dataset with all the attributes I need.

Thanks @jarednielsen, that makes sense. I'm a little wary of messing with the cache files, since I still don't really understand what's going on under the hood with Apache Arrow.

Related question: I'd like to do parallel pre-processing of the dataset. I know how to break the dataset up via sharding, but is there any way to combine the shards back together again once the processing is done? Right now I'm probably just going to iterate over each shard, write the contexts to a txt file, and then cat the txt files, but it feels like there ought to be a nicer way to concatenate datasets.

Haha, opened a PR for that functionality about an hour ago: https://github.com/huggingface/nlp/pull/390. Glad we're on the same page :)

Datasets are not supposed to be pickled as pickle tries to put all the dataset in memory if I'm not wrong (and write all the data on disk).
The concatenate method however is a very cool feature, looking forward to having it merged :)

Ah, yes, you are correct. The pickle file contains the whole dataset, not just the cache names, which is not quite what I expected.

I tried adding a warning when pickling a Dataset, to prevent others like me from trying it. Interestingly, however, the warning is raised whenever any function on the dataset is called (select, shard, etc.).

```
import nlp
wiki = nlp.load_dataset('wikipedia', split='train')
wiki = wiki.shard(16, 0) # Triggers pickling of dataset
```

I believe this is because [this line](https://github.com/huggingface/nlp/blob/master/src/nlp/arrow_dataset.py#L626), which gets the function signature, is actually pickling the whole dataset (and thereby serializing all the data to text). I checked by printing that string, and sure enough it was full of Wikipedia articles.

I don't think the whole pickling thing is worth the effort, so I'll close the PR. But I did want to mention this serialization behavior in case it's not intended.

Thanks for reporting. Indeed this line shouldn't serialize the data but only the function itself.

Keeping this open because I would like to keep brainstorming a bit on this.

One note on this is that we should have a clean serialization workflow, probably one that could serialize to a few formats (arrow, parquet and tfrecords come to mind).

This PR could be useful. My specific use case is `multiprocessing.Pool` for parallel preprocessing (because of the Python tokenization bottleneck at https://github.com/huggingface/transformers/issues/5729). I shard a large dataset, run map on each shard within a multiprocessing pool, and then concatenate them back together. This is only possible if a dataset can be pickled, otherwise the logic is much more complex. There's no reason to make it un-picklable, even if it's not the recommended usage.

```python
import nlp
import multiprocessing

def func(ex):
    return {"text": "Prefix: " + ex["text"]}

def map_helper(dset):
    return dset.map(func)

n_shards = 16
dset = nlp.load_dataset("wikitext-2-raw-v1", split="train")
with multiprocessing.Pool(processes=n_shards) as pool:
    shards = pool.map(map_helper, [dset.shard(n_shards, i, contiguous=True) for i in range(n_shards)])
dset = nlp.concatenate_datasets(shards)
```

Yes I agree.
#423 just got merged and should allow serialization of `SplitDict`. Could you try it and see if it'ok on your side now ?

Closing this, assuming it was fixed in #423.

It would be nice to pickle and unpickle Datasets, as done in [this tutorial](https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb). Example:

```
wiki = nlp.load_dataset('wikipedia', split='train')
def sentencize(examples):
    ...

wiki = wiki.map(sentencize, batched=True)
torch.save(wiki, 'sentencized_wiki_dataset.pt')
```

However, upon unpickling the dataset via torch.load(...), this error is raised:

```
ValueError("Cannot add elem. Use .add() instead.")
```
On line [492 of splits.py](https://github.com/huggingface/nlp/blob/master/src/nlp/splits.py#L492). This is because SplitDict subclasses dict, and pickle treats [dicts specially](https://github.com/huggingface/nlp/blob/master/src/nlp/splits.py#L492). Pickle expects access to `dict.__setitem__`, but this is disallowed by the class.

The workaround is to provide an explicit interface for pickle to call when pickling and unpickling, thereby avoiding the use of `__setitem__`.

Testing:
- Manually pickled and unpickled a modified wikipedia dataset.
- Ran `make style`

I would be happy to run any other tests, but I couldn't find any in the contributing guidelines.


🐛 [Dataset] Cannot download wmt14, wmt15 and wmt17

similar slow download speed here for nlp.load_dataset('wmt14', 'fr-en')
`
Downloading: 100%|██████████████████████████████████████████████████████████| 658M/658M [1:00:42<00:00, 181kB/s]
Downloading: 100%|██████████████████████████████████████████████████████████| 918M/918M [1:39:38<00:00, 154kB/s]
Downloading:   2%|▉                                                       | 40.9M/2.37G [04:48<5:03:06, 128kB/s]
`
Could we just download a specific subdataset in 'wmt14', such as 'newstest14'?

> The code runs but the download speed is extremely slow, the same behaviour is not observed on wmt16 and wmt18

The original source for the files may provide slow download speeds.
We can probably host these files ourselves.

> When trying to download wmt17 zh-en, I got the following error:
> ConnectionError: Couldn't reach https://storage.googleapis.com/tfdataset-data/downloadataset/uncorpus/UNv1.0.en-zh.tar.gz

Looks like the file`UNv1.0.en-zh.tar.gz` is missing, or the url changed. We need to fix that

> Could we just download a specific subdataset in 'wmt14', such as 'newstest14'?

Right now I don't think it's possible. Maybe @patrickvonplaten knows more about it

Yeah, the download speed is sadly always extremely slow :-/.
I will try to check out the `wmt17 zh-en` bug :-)

Maybe this can be used - https://stuncorpusprod.blob.core.windows.net/corpusfiles/UNv1.0.en-zh.tar.gz.00

1. I try downloading `wmt14`, `wmt15`, `wmt17`, `wmt19` with the following code:
```
nlp.load_dataset('wmt14','de-en')
nlp.load_dataset('wmt15','de-en')
nlp.load_dataset('wmt17','de-en')
nlp.load_dataset('wmt19','de-en')
```
The code runs but the download speed is **extremely slow**, the same behaviour is not observed on `wmt16` and `wmt18`

2. When trying to download `wmt17 zh-en`, I got the following error:
> ConnectionError: Couldn't reach https://storage.googleapis.com/tfdataset-data/downloadataset/uncorpus/UNv1.0.en-zh.tar.gz


Conversion through to_pandas output numpy arrays for lists instead of python objects

To convert from arrow type we have three options: to_numpy, to_pandas and to_pydict/to_pylist.

- to_numpy and to_pandas return numpy arrays instead of lists but are very fast.
- to_pydict/to_pylist can be 100x slower and become the bottleneck for reading data, but at least they return lists.

Maybe we can have to_pydict/to_pylist as the default and use to_numpy or to_pandas when the format (set by `set_format`) is 'numpy' or 'pandas'

In a related question, the conversion through to_pandas output numpy arrays for the lists instead of python objects.

Here is an example:
```python
>>> dataset._data.slice(key, 1).to_pandas().to_dict("list")
{'sentence1': ['Amrozi accused his brother , whom he called " the witness " , of deliberately distorting his evidence .'], 'sentence2': ['Referring to him as only " the witness " , Amrozi accused his brother of deliberately distorting his evidence .'], 'label': [1], 'idx': [0], 'input_ids': [array([  101,  7277,  2180,  5303,  4806,  1117,  1711,   117,  2292,
        1119,  1270,   107,  1103,  7737,   107,   117,  1104,  9938,
        4267, 12223, 21811,  1117,  2554,   119,   102])], 'token_type_ids': [array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0])], 'attention_mask': [array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1])]}
>>> type(dataset._data.slice(key, 1).to_pandas().to_dict("list")['input_ids'][0])
<class 'numpy.ndarray'>
>>> dataset._data.slice(key, 1).to_pydict()
{'sentence1': ['Amrozi accused his brother , whom he called " the witness " , of deliberately distorting his evidence .'], 'sentence2': ['Referring to him as only " the witness " , Amrozi accused his brother of deliberately distorting his evidence .'], 'label': [1], 'idx': [0], 'input_ids': [[101, 7277, 2180, 5303, 4806, 1117, 1711, 117, 2292, 1119, 1270, 107, 1103, 7737, 107, 117, 1104, 9938, 4267, 12223, 21811, 1117, 2554, 119, 102]], 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]}
```


Update dataset loading and features - Add TREC dataset

I just copied the files that are on google storage to follow the new `_relative_data_dir ` format. It should be good to merge now :)

Well actually it seems there are some merge conflicts to fix first

This PR:
- add a template for a new dataset script
- update the caching structure so that the path to the cached data files is also a function of the dataset loading script hash. This way when you update a loading script the data will be automatically updated instead of falling back to the previous version (which is usually a outdated). This makes it in particular easier to iterate when writing a new dataset loading script.
- fix a bug in the `ClassLabel` feature and make it more flexible so that its methods `str2int` and `int2str` can also accept list, numpy arrays and PyTorch/TensorFlow tensors.
- add the TREC-6 dataset


Remove unnecessary nested dict

We can probably scan the dataset scripts with a regexpr to try to identify this pattern cc @patrickvonplaten maybe

@mariamabarham This script should work. I tested it for a couple of datasets. There might be exceptions where the script breaks - did not test everything.

```python
#!/usr/bin/env python3

from nlp import prepare_module, DownloadConfig, import_main_class, hf_api
import tempfile

def scan_for_nested_unnecessary_dict(dataset_name):

    def load_builder_class(dataset_name):
        module_path = prepare_module(dataset_name, download_config=DownloadConfig(force_download=True))
        return import_main_class(module_path)

    def load_configs(dataset_name):
        builder_cls = load_builder_class(dataset_name)
        if len(builder_cls.BUILDER_CONFIGS) == 0:
            return [None]
        return builder_cls.BUILDER_CONFIGS

    def scan_features_for_nested_dict(features):
        is_sequence = False
        if hasattr(features, "_type"):
            if features._type != 'Sequence':
                return False
            else:
                is_sequence = True
                features = features.feature

        if isinstance(features, list):
            for value in features:
                if scan_features_for_nested_dict(value):
                    return True
            return False

        elif isinstance(features, dict):
            for key, value in features.items():
                if is_sequence and len(features.keys()) == 1 and hasattr(features[key], "_type") and features[key]._type != "Sequence":
                    return True
                if scan_features_for_nested_dict(value):
                    return True
            return False
        elif hasattr(features, "_type"):
            return False
        else:
            raise ValueError(f"{features} should be either a list, a dict or a feature")

    configs = load_configs(dataset_name)

    for config in configs:
        with tempfile.TemporaryDirectory() as processed_temp_dir:
            # create config and dataset
            dataset_builder_cls = load_builder_class(dataset_name)
            name = config.name if config is not None else None
            dataset_builder = dataset_builder_cls(name=name, cache_dir=processed_temp_dir)

        is_nested_dict_in_dataset = scan_features_for_nested_dict(dataset_builder._info().features)
        if is_nested_dict_in_dataset:
            print(f"{dataset_name} with {name} needs refactoring")

if __name__ == "__main__":
    scan_for_nested_unnecessary_dict("race")  # prints True
    scan_for_nested_unnecessary_dict("mlqa")  # prints True
    scan_for_nested_unnecessary_dict("squad")  # prints Nothing

    # ran the following lines for 1min and seems to work -> didn't check for all datasets though
#    api = hf_api.HfApi()
#    all_datasets = [x.id for x in api.dataset_list(with_community_datasets=False)]
#    for dataset in all_datasets:
#        scan_for_nested_unnecessary_dict(dataset)
```

> @mariamabarham This script should work. I tested it for a couple of datasets. There might be exceptions where the script breaks - did not test everything.
>
> ```python
> #!/usr/bin/env python3
>
> from nlp import prepare_module, DownloadConfig, import_main_class, hf_api
> import tempfile
>
>
> def scan_for_nested_unnecessary_dict(dataset_name):
>
>     def load_builder_class(dataset_name):
>         module_path = prepare_module(dataset_name, download_config=DownloadConfig(force_download=True))
>         return import_main_class(module_path)
>
>     def load_configs(dataset_name):
>         builder_cls = load_builder_class(dataset_name)
>         if len(builder_cls.BUILDER_CONFIGS) == 0:
>             return [None]
>         return builder_cls.BUILDER_CONFIGS
>
>     def scan_features_for_nested_dict(features):
>         is_sequence = False
>         if hasattr(features, "_type"):
>             if features._type != 'Sequence':
>                 return False
>             else:
>                 is_sequence = True
>                 features = features.feature
>
>         if isinstance(features, list):
>             for value in features:
>                 if scan_features_for_nested_dict(value):
>                     return True
>             return False
>
>         elif isinstance(features, dict):
>             for key, value in features.items():
>                 if is_sequence and len(features.keys()) == 1 and hasattr(features[key], "_type") and features[key]._type != "Sequence":
>                     return True
>                 if scan_features_for_nested_dict(value):
>                     return True
>             return False
>         else:
>             raise ValueError(f"{features} should be either a list of a dict")
>
>     configs = load_configs(dataset_name)
>
>     for config in configs:
>         with tempfile.TemporaryDirectory() as processed_temp_dir:
>             # create config and dataset
>             dataset_builder_cls = load_builder_class(dataset_name)
>             name = config.name if config is not None else None
>             dataset_builder = dataset_builder_cls(name=name, cache_dir=processed_temp_dir)
>
>         is_nested_dict_in_dataset = scan_features_for_nested_dict(dataset_builder._info().features)
>         if is_nested_dict_in_dataset:
>             print(f"{dataset_name} with {name} needs refactoring")
>
>
> if __name__ == "__main__":
>     scan_for_nested_unnecessary_dict("race")  # prints True
>     scan_for_nested_unnecessary_dict("mlqa")  # prints True
>     scan_for_nested_unnecessary_dict("squad")  # prints Nothing
>
>     # ran the following lines for 1min and seems to work -> didn't check for all datasets though
> #    api = hf_api.HfApi()
> #    all_datasets = [x.id for x in api.dataset_list(with_community_datasets=False)]
> #    for dataset in all_datasets:
> #        scan_for_nested_unnecessary_dict(dataset)
> ```

Great, I will try it

I'm not sure the work on this PR was finished @lhoestq  cc @mariamabarham @patrickvonplaten

Sorry for that, apparently there are other datasets that could have unnecessary nested dicts.
We can have another PR to scan and fix the other datasets.

This PR is removing unnecessary nested dictionary used in some datasets. For now the following datasets are updated:

- MLQA

- RACE

Will be adding more if necessary.

#378


Adding the Linguistic Code-switching Evaluation (LinCE) benchmark

I am checking the details of the CI log for the failed test, but I don't see how the error relates to the code I added; the error is coming from a config builder different than the `LinceConfig`, and it crashes when `self.config.data_files` because is self.config is None. I would appreciate if someone could help me find out where I could have messed things up :)

Also, the real and dummy data tests passed before committing and pushing my changes.

Thanks a lot in advance!

```
=================================== FAILURES ===================================
____________________ AWSDatasetTest.test_load_dataset_text _____________________

self = <tests.test_dataset_common.AWSDatasetTest testMethod=test_load_dataset_text>
dataset_name = 'text'

    def test_load_dataset(self, dataset_name):
        configs = self.dataset_tester.load_all_configs(dataset_name)[:1]
>       self.dataset_tester.check_load_dataset(dataset_name, configs)

tests/test_dataset_common.py:243:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_dataset_common.py:137: in check_load_dataset
    try_from_hf_gcs=False,
../.local/lib/python3.6/site-packages/nlp/builder.py:432: in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
../.local/lib/python3.6/site-packages/nlp/builder.py:466: in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <nlp.datasets.text.bf5568367c6707640e5601a44ed0af98f40a8483db81a7db99b85fab6606fc8b.text.Text object at 0x7efa744ffb70>
dl_manager = <nlp.utils.mock_download_manager.MockDownloadManager object at 0x7efb304c52b0>

    def _split_generators(self, dl_manager):
        """ The `datafiles` kwarg in load_dataset() can be a str, List[str], Dict[str,str], or Dict[str,List[str]].

            If str or List[str], then the dataset returns only the 'train' split.
            If dict, then keys should be from the `nlp.Split` enum.
        """
        if isinstance(self.config.data_files, (str, list, tuple)):
            # Handle case with only one split
            files = self.config.data_files
            if isinstance(files, str):
                files = [files]
            return [nlp.SplitGenerator(name=nlp.Split.TRAIN, gen_kwargs={"files": files})]
        else:
            # Handle case with several splits and a dict mapping
            splits = []
            for split_name in [nlp.Split.TRAIN, nlp.Split.VALIDATION, nlp.Split.TEST]:
>               if split_name in self.config.data_files:
E               TypeError: argument of type 'NoneType' is not iterable

../.local/lib/python3.6/site-packages/nlp/datasets/text/bf5568367c6707640e5601a44ed0af98f40a8483db81a7db99b85fab6606fc8b/text.py:24: TypeError
=============================== warnings summary ===============================
...
=========================== short test summary info ============================
FAILED tests/test_dataset_common.py::AWSDatasetTest::test_load_dataset_text
====== 1 failed, 963 passed, 532 skipped, 5 warnings in 166.33s (0:02:46) ======

Exited with code exit status 1
```

@lhoestq Hi Quentin, I was wondering if you could give some feedback on this error from the `run_dataset_script_tests` script. It seems that's coming from a different config builder than the one I added, so I am not sure why this error would occur. Thanks in advance!

Awesome! Thank you for all your comments! 👌  I will update the PR in a bit with all the required changes 🙂

Let me just provide a bit of context for my changes:

I was referring to the GLUE, XTREME and WNUT_17 dataset scripts to build mine (not sure if the new documentation was available last week). This is where I took the naming convention for the citation and description variables. Also, these scripts didn't have the `BUILDER_CONFIG_CLASS = LinceConfig` line so I commented this out thinking I didn't need that; I tried this line in my attempts to make the real and dummy data tests pass but it was not helping.

The problem I was facing was that the tests were passing a default `BuilderConfig` (i.e., `self.config.name` property was set to `'default'` and my custom properties were not available). This means, for example, that within the `def _info(...)`  method, I was not able to access the specific fields of my `LinceConfig` class (which is why I have now a global variable `_LINCE_CITATIONS`, to detach the individual citations from the corresponding LinceConfig objects, as well as I am constructing manually the feature infos). This default `BuilderConfig` is why I added the `if not isinstance(self.config, LinceConfig): return []` statement. Otherwise, accessing custom properties like `self.config.colnames` was failing the test because such properties did not exist in the default config (i.e., it was not a `LinceConfig`).

I will update the PR and see if these problems happen in the CI tests.

Thanks again for the follow-up! @lhoestq

Ok I see !

To give you more details: the line `BUILDER_CONFIG_CLASS = LinceConfig` tells the tests how to instantiate a config for this dataset. Therefore if you have this line you should have all the fields of your config available.

To fix the errors you get you'll have to, first, have the `BUILDER_CONFIG_CLASS = LinceConfig` line, and second, add default values for the parameters of your config (or the tests functions will be unable to instantiate it by calling `LinceConfig()`.

An example of dataset with a custom config with additional filed like this one is [biomrc](https://github.com/huggingface/nlp/blob/master/datasets/biomrc/biomrc.py).
Feel free to give a look at it if you want.

Thanks for the reference!

I just updated the PR with the suggested changes. It seems the CI failed on the same test you said we could ignore, so I guess it's okay :)

Please let me know if there is something else I may need to change.

Hi,

First of all, this library is really cool! Thanks for putting all of this together!

This PR contains the [Linguistic Code-switching Evaluation (LinCE) benchmark](https://ritual.uh.edu/lince). As described in the official website (FAQ):

> 1. Why do we need LinCE?
>LinCE brings 10 code-switching datasets together for 4 tasks and 4 language pairs with 5 leaderboards in a single evaluation platform. We examined each dataset and fixed major issues on the partitions (or even define official partitions) with a comprehensive stratification method (see our paper for more details).
>Besides, we believe that online benchmarks like LinCE bring steady research progress and allow to compare state-of-the-art models at the pace of the progress in NLP. We expect to benefit greatly the code-switching community with this benchmark.

The data comes from social media and here's the summary table of tasks per language pair:

| Language Pairs                         | LID | POS | NER | SA |
|----------------------------------------|-----|-----|-----|----|
| Spanish-English                        | ✅  | ✅  | ✅  | ✅ |
| Hindi-English                          | ✅  | ✅  | ✅  |    |
| Modern Standard Arabic-Egyptian Arabic | ✅  |     | ✅  |    |
| Nepali-English                         | ✅  |     |     |    |

The tasks are as follows:
* LID: token-level language identification
* POS: part-of-speech tagging
* NER: named entity recognition
* SA: sentiment analysis

With the exception of MSA-EA, the rest of the datasets contain token-level LID labels.

## Usage

For Spanish-English LID, we can load the data as follows:
```
import nlp

data = nlp.load_dataset('./datasets/lince/lince.py', 'lid_spaeng')

for split in data:
    print(data[split])
```

Here's the output:
```
Dataset(schema: {'idx': 'int32', 'tokens': 'list<item: string>', 'lid': 'list<item: string>'}, num_rows: 21030)
Dataset(schema: {'idx': 'int32', 'tokens': 'list<item: string>', 'lid': 'list<item: string>'}, num_rows: 3332)
Dataset(schema: {'idx': 'int32', 'tokens': 'list<item: string>', 'lid': 'list<item: string>'}, num_rows: 8289)
```

Here's the list of shortcut names for every dataset available in LinCE:
* `lid_spaeng`
* `lid_hineng`
* `lid_nepeng`
* `lid_msaea`
* `pos_spaeng`
* `pos_hineng`
* `ner_spaeng`
* `ner_hineng`
* `ner_msaea`
* `sa_spaeng`

All the numbers match with Table 3 in the LinCE [paper](https://www.aclweb.org/anthology/2020.lrec-1.223.pdf). Also, note that the MSA-EA datasets use the Persian script while the other datasets use the Roman script.

## Features

Here is how the features look in the case of language identification (LID) tasks:

| LID Feature          | Type          | Description                               |
|----------------------|---------------|-------------------------------------------|
| `idx`                | `int`       | Dataset index of current sentence         |
| `tokens`             | `list<str>` | List of tokens (string) of a sentence     |
| `lid`                | `list<str>` | List of LID labels (string) of a sentence |

For part-of-speech (POS) tagging:

| POS Feature          | Type          | Description                               |
|----------------------|---------------|-------------------------------------------|
| `idx`                | `int`       | Dataset index of current sentence         |
| `tokens`             | `list<str>` | List of tokens (string) of a sentence     |
| `lid`                | `list<str>` | List of LID labels (string) of a sentence |
| `pos`                | `list<str>` | List of POS tags (string) of a sentence   |

For named entity recognition (NER):

| NER Feature          | Type          | Description                               |
|----------------------|---------------|-------------------------------------------|
| `idx`                | `int`       | Dataset index of current sentence         |
| `tokens`             | `list<str>` | List of tokens (string) of a sentence     |
| `lid`                | `list<str>` | List of LID labels (string) of a sentence |
| `ner`                | `list<str>` | List of NER labels (string) of a sentence |

**NOTE**: the MSA-EA NER dataset does not contain the `lid` feature.

For sentiment analysis (SA):

| SA Feature          | Type        | Description                               |
|---------------------|-------------|-------------------------------------------|
| `idx`               | `int`       | Dataset index of current sentence         |
| `tokens`            | `list<str>` | List of tokens (string) of a sentence     |
| `lid`               | `list<str>` | List of LID labels (string) of a sentence |
| `sa`                | `str`       | Sentiment label (string) of a sentence    |


1080


NLp


[dataset] Structure of MLQA seems unecessary nested

Same for the RACE dataset: https://github.com/huggingface/nlp/blob/master/datasets/race/race.py

Should we scan all the datasets to remove this pattern of un-necessary nesting?

You're right, I think we don't need to use the nested dictionary.

The features of the MLQA dataset comprise several nested dictionaries with a single element inside (for `questions` and `ids`): https://github.com/huggingface/nlp/blob/master/datasets/mlqa/mlqa.py#L90-L97

Should we keep this @mariamabarham @patrickvonplaten? Was this added for compatibility with tfds?

```python
            features=nlp.Features(
                {
                    "context": nlp.Value("string"),
                    "questions": nlp.features.Sequence({"question": nlp.Value("string")}),
                    "answers": nlp.features.Sequence(
                        {"text": nlp.Value("string"), "answer_start": nlp.Value("int32"),}
                    ),
                    "ids": nlp.features.Sequence({"idx": nlp.Value("string")})
```


Iyy!!!


to_pandas conversion doesn't always work

**Edit**: other topic previously in this message moved to a new issue: https://github.com/huggingface/nlp/issues/387

Could you try to update pyarrow to >=0.17.0 ? It should fix the `to_pandas` bug

Also I'm not sure that structures like list<struct> are fully supported in the lib (none of the datasets use that).
It can cause issues when using dataset transforms like `filter` for example

For some complex nested types, the conversion from Arrow to python dict through pandas doesn't seem to be possible.

Here is an example using the official SQUAD v2 JSON file.

This example was found while investigating #373.

```python
>>> squad = load_dataset('json', data_files={nlp.Split.TRAIN: ["./train-v2.0.json"]}, download_mode=nlp.GenerateMode.FORCE_REDOWNLOAD, version="1.0.0", field='data')
>>> squad['train']
Dataset(schema: {'title': 'string', 'paragraphs': 'list<item: struct<qas: list<item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>>, context: string>>'}, num_rows: 442)
>>> squad['train'][0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/thomwolf/Documents/GitHub/datasets/src/nlp/arrow_dataset.py", line 589, in __getitem__
    format_kwargs=self._format_kwargs,
  File "/Users/thomwolf/Documents/GitHub/datasets/src/nlp/arrow_dataset.py", line 529, in _getitem
    outputs = self._unnest(self._data.slice(key, 1).to_pandas().to_dict("list"))
  File "pyarrow/array.pxi", line 559, in pyarrow.lib._PandasConvertible.to_pandas
  File "pyarrow/table.pxi", line 1367, in pyarrow.lib.Table._to_pandas
  File "/Users/thomwolf/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/pandas_compat.py", line 766, in table_to_blockmanager
    blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)
  File "/Users/thomwolf/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/pandas_compat.py", line 1101, in _table_to_blocks
    list(extension_columns.keys()))
  File "pyarrow/table.pxi", line 881, in pyarrow.lib.table_to_blocks
  File "pyarrow/error.pxi", line 105, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Not implemented type for Arrow list to pandas: struct<qas: list<item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>>, context: string>
```

cc @lhoestq would we have a way to detect this from the schema maybe?

Here is the schema for this pretty complex JSON:
```python
>>> squad['train'].schema
title: string
paragraphs: list<item: struct<qas: list<item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>>, context: string>>
  child 0, item: struct<qas: list<item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>>, context: string>
      child 0, qas: list<item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>>
          child 0, item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>
              child 0, question: string
              child 1, id: string
              child 2, answers: list<item: struct<text: string, answer_start: int64>>
                  child 0, item: struct<text: string, answer_start: int64>
                      child 0, text: string
                      child 1, answer_start: int64
              child 3, is_impossible: bool
              child 4, plausible_answers: list<item: struct<text: string, answer_start: int64>>
                  child 0, item: struct<text: string, answer_start: int64>
                      child 0, text: string
                      child 1, answer_start: int64
      child 1, context: string
```


TypeError when computing bertscore

I am not able to reproduce this issue on my side.
Could you give us more details about the inputs you used ?

I do get another error though:
```
~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/bert_score/utils.py in bert_cos_score_idf(model, refs, hyps, tokenizer, idf_dict, verbose, batch_size, device, all_layers)
    371         return sorted(list(set(l)), key=lambda x: len(x.split(" ")))
    372
--> 373     sentences = dedup_and_sort(refs + hyps)
    374     embs = []
    375     iter_range = range(0, len(sentences), batch_size)

ValueError: operands could not be broadcast together with shapes (0,) (2,)
```
That's because it gets numpy arrays as input and not lists. See #387

The other issue was fixed by #403

Do you still get this issue @willywsm1013 ?

Hi,

I installed nlp 0.3.0 via pip, and my python version is 3.7.
When I tried to compute bertscore with the code:
```
import nlp
bertscore = nlp.load_metric('bertscore')
# load hyps and refs
...
print (bertscore.compute(hyps, refs, lang='en'))
```

I got the following error.
```
Traceback (most recent call last):
  File "bert_score_evaluate.py", line 16, in <module>
    print (bertscore.compute(hyps, refs, lang='en'))
  File "/home/willywsm/anaconda3/envs/torcher/lib/python3.7/site-packages/nlp/metric.py", line 200, in compute
    output = self._compute(predictions=predictions, references=references, **metrics_kwargs)
  File "/home/willywsm/anaconda3/envs/torcher/lib/python3.7/site-packages/nlp/metrics/bertscore/fb176889831bf0ce995ed197edc94b2e9a83f647a869bb8c9477dbb2d04d0f08/bertscore.py", line 105, in _compute
    hashcode = bert_score.utils.get_hash(model_type, num_layers, idf, rescale_with_baseline)
TypeError: get_hash() takes 3 positional arguments but 4 were given
```

It seems like there is something wrong with get_hash() function?


Add dataset post processing for faiss indexes

I changed the `wiki_dpr` script to ignore the last 24 examples for now. Hopefully we'll have the full version soon.
The datasets_infos.json and the data on GCS are updated.

And I also added a check to make sure we don't have post processing resources in sub-directories.

I added a dummy config that can be loaded with:
```python
wiki = load_dataset("wiki_dpr", "dummy_psgs_w100_no_embeddings", with_index=True, split="train")
```
It's only 6MB of arrow files and 30MB of index

# Post processing of datasets for faiss indexes

Now that we can have datasets with embeddings (see `wiki_pr` for example), we can allow users to load the dataset + get the Faiss index that comes with it to do nearest neighbors queries.

## Implementation proposition

- Faiss indexes have to be added to the `nlp.Dataset` object, and therefore it's in a different scope that what are doing the `_split_generators` and `_generate_examples` methods of `nlp.DatasetBuilder`. Therefore I added a new method for post processing of the `nlp.Dataset` object called `_post_process` (name could change)
- The role of `_post_process` is to apply dataset transforms (filter/map etc.) or indexing functions (add_faiss_index) to modify/enrich the `nlp.Dataset` object. It is not part of the `download_and_prepare` process (that is focused on arrow files creation) so the post processing is run inside the `as_dataset` method.
- `_post_process` can generate new files (cached files from dataset transforms or serialized faiss indexes) and their names are defined by `_post_processing_resources`
- as we know what are the post processing resources, we can download them automatically from google storage instead of computing them if they're available (as we do for arrow files)

I'd happy to discuss these choices !

## The `wiki_dpr` index

It takes 1h20 and ~7GB of memory to compute. The final index is 1.42GB and takes ~1.5GB of memory.
This is pretty cool given that a naive flat index would take 170GB of memory to store the 21M vectors of dim 768.
I couldn't use directly the Faiss `index_factory` as I needed to set the metric to inner product.

## Example of usage

```python
import nlp
dset = nlp.load_dataset(
    "wiki_dpr",
    "psgs_w100_with_nq_embeddings",
    split="train",
    with_index=True
)
print(len(dset), dset.list_indexes())  # (21015300, ['embeddings'])
```

(it also works with the dataset configuration without the embeddings because I added the index file in google storage for this one too)

## Demo

You can also check a demo on google colab that shows how to use it with the DPRQuestionEncoder from transformers:
https://colab.research.google.com/drive/1FakNU8W5EPMcWff7iP1H6REg3XSS0YLp?usp=sharing


Segmentation fault when loading local JSON dataset as of #372

I've seen this sort of thing before -- it might help to delete the directory -- I've also noticed that there is an error with the json Dataloader for any data I've tried to load. I've replaced it with this, which skips over the data feature population step:

```python
import os

import pyarrow.json as paj

import nlp as hf_nlp

from nlp import DatasetInfo, BuilderConfig, SplitGenerator, Split, utils
from nlp.arrow_writer import ArrowWriter

class JSONDatasetBuilder(hf_nlp.ArrowBasedBuilder):
    BUILDER_CONFIG_CLASS = BuilderConfig

    def _info(self):
        return DatasetInfo()

    def _split_generators(self, dl_manager):
        """ We handle string, list and dicts in datafiles
        """
        if isinstance(self.config.data_files, (str, list, tuple)):
            files = self.config.data_files
            if isinstance(files, str):
                files = [files]
            return [SplitGenerator(name=Split.TRAIN, gen_kwargs={"files": files})]
        splits = []
        for split_name in [Split.TRAIN, Split.VALIDATION, Split.TEST]:
            if split_name in self.config.data_files:
                files = self.config.data_files[split_name]
                if isinstance(files, str):
                    files = [files]
                splits.append(SplitGenerator(name=split_name, gen_kwargs={"files": files}))
        return splits

    def _prepare_split(self, split_generator):
        fname = "{}-{}.arrow".format(self.name, split_generator.name)
        fpath = os.path.join(self._cache_dir, fname)

        writer = ArrowWriter(path=fpath)

        generator = self._generate_tables(**split_generator.gen_kwargs)
        for key, table in utils.tqdm(generator, unit=" tables", leave=False):
            writer.write_table(table)
        num_examples, num_bytes = writer.finalize()

        split_generator.split_info.num_examples = num_examples
        split_generator.split_info.num_bytes = num_bytes

    def _generate_tables(self, files):
        for i, file in enumerate(files):
            pa_table = paj.read_json(
                file
            )
            yield i, pa_table

```

Yes, deleting the directory solves the error whenever I try to rerun.

By replacing the json-loader, you mean the cached file in my `site-packages` directory? e.g. `/home/XXX/.cache/lib/python3.7/site-packages/nlp/datasets/json/(...)/json.py`

When I was testing this out before the #372 PR was merged I had issues installing it properly locally. Since the `json.py` script was downloaded instead of actually using the one provided in the local install. Manually updating that file seemed to solve it, but it didn't seem like a proper solution. Especially when having to run this on a remote compute cluster with no access to that directory.

I see, diving in the JSON file for SQuAD it's a pretty complex structure.

The best solution for you, if you have a dataset really similar to SQuAD would be to copy and modify the SQuAD data processing script. We will probably add soon an option to be able to specify file path to use instead of the automatic URL encoded in the script but in the meantime you can:
- copy the [squad script](https://github.com/huggingface/nlp/blob/master/datasets/squad/squad.py)  in a new script for your dataset
- in the new script replace [these `urls_to_download `](https://github.com/huggingface/nlp/blob/master/datasets/squad/squad.py#L99-L102) by `urls_to_download=self.config.data_files`
- load the dataset with `dataset = load_dataset('path/to/your/new/script', data_files={nlp.Split.TRAIN: "./datasets/train-v2.0.json"})`

This way you can reuse all the processing logic of the SQuAD loading script.

This seems like a more sensible solution! Thanks, @thomwolf. It's been a little daunting to understand what these scripts actually do, due to the level of abstraction and central documentation.

Am I correct in assuming that the `_generate_examples()` function is the actual procedure for how the data is loaded from file? Meaning that essentially with a file containing another format, that is the only function that requires re-implementation? I'm working with a lot of datasets that, due to licensing and privacy, cannot be published. As this library is so neatly integrated with the transformers library and gives easy access to public sets such as SQUAD and increased performance, it is very neat to be able to load my private sets as well. As of now, I have just been working on scripts for translating all my data into the SQUAD-format before using the json script, but I see that it might not be necessary after all.

Yes `_generate_examples()` is the main entry point. If you change the shape of the returned dictionary you also need to update the `features` in the `_info`.

I'm currently writing the doc so it should be easier soon to use the library and know how to add your datasets.

Could you try to update pyarrow to >=0.17.0 @vegarab ?
I don't have any segmentation fault with my version of pyarrow (0.17.1)

I tested with
```python
import nlp
s = nlp.load_dataset("json", data_files="train-v2.0.json", field="data", split="train")
s[0]
# {'title': 'Normans', 'paragraphs': [{'qas': [{'question': 'In what country is Normandy located?', 'id':...
```

Also if you want to have your own dataset script, we now have a new documentation !
See here:
https://huggingface.co/nlp/add_dataset.html

@lhoestq
For some reason, I am not able to reproduce the segmentation fault, on pyarrow==0.16.0. Using the exact same environment and file.

Anyhow, I discovered that pyarrow>=0.17.0 is required to read in a JSON file where the pandas structs contain lists. Otherwise, pyarrow complains when attempting to cast the struct:
```py
import nlp
>>> s = nlp.load_dataset("json", data_files="datasets/train-v2.0.json", field="data", split="train")
Using custom data configuration default
>>> s[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vegarab/.conda/envs/torch/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 558, in __getitem__
    format_kwargs=self._format_kwargs,
  File "/home/vegarab/.conda/envs/torch/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 498, in _getitem
    outputs = self._unnest(self._data.slice(key, 1).to_pandas().to_dict("list"))
  File "pyarrow/array.pxi", line 559, in pyarrow.lib._PandasConvertible.to_pandas
  File "pyarrow/table.pxi", line 1367, in pyarrow.lib.Table._to_pandas
  File "/home/vegarab/.conda/envs/torch/lib/python3.7/site-packages/pyarrow/pandas_compat.py", line 766, in table_to_blockmanager
    blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)
  File "/home/vegarab/.conda/envs/torch/lib/python3.7/site-packages/pyarrow/pandas_compat.py", line 1101, in _table_to_blocks
    list(extension_columns.keys()))
  File "pyarrow/table.pxi", line 881, in pyarrow.lib.table_to_blocks
  File "pyarrow/error.pxi", line 105, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Not implemented type for Arrow list to pandas: struct<qas: list<item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>>, context: string>
>>> s
Dataset(schema: {'title': 'string', 'paragraphs': 'list<item: struct<qas: list<item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>>, context: string>>'}, num_rows: 35)
```

Upgrading to >=0.17.0 provides the same dataset structure, but accessing the records is possible without the same exception.

Very happy to see some extended documentation!

#376 seems to be reporting the same issue as mentioned above.

This issue helped me a lot, thanks.
Hope this issue will be fixed soon.

The last issue was closed (#369) once the #372 update was merged. However, I'm still not able to load a SQuAD formatted JSON file. Instead of the previously recorded pyarrow error, I now get a segmentation fault.

```
dataset = nlp.load_dataset('json', data_files={nlp.Split.TRAIN: ["./datasets/train-v2.0.json"]}, field='data')
```
causes
```
Using custom data configuration default
Downloading and preparing dataset json/default (download: Unknown size, generated: Unknown size, total: Unknown size) to /home/XXX/.cache/huggingface/datasets/json/default/0.0.0...
0 tables [00:00, ? tables/s]Segmentation fault (core dumped)
```
where `./datasets/train-v2.0.json` is downloaded directly from https://rajpurkar.github.io/SQuAD-explorer/.
This is consistent with other SQuAD-formatted JSON files.

When attempting to load the dataset again, I get the following:
```
Using custom data configuration default
Traceback (most recent call last):
  File "dataloader.py", line 6, in <module>
    'json', data_files={nlp.Split.TRAIN: ["./datasets/train-v2.0.json"]}, field='data')
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/load.py", line 524, in load_dataset
    save_infos=save_infos,
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 382, in download_and_prepare
    with incomplete_dir(self._cache_dir) as tmp_data_dir:
  File "/home/XXX/.conda/envs/torch/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 368, in incomplete_dir
    os.makedirs(tmp_dir)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/home/XXX/.cache/huggingface/datasets/json/default/0.0.0.incomplete'
```

(Not sure if you wanted this in the previous issue #369 or not as it was closed.)


Make the json script more flexible

Fix https://github.com/huggingface/nlp/issues/359
Fix https://github.com/huggingface/nlp/issues/369

JSON script now can accept JSON files containing a single dict with the records as a list in one attribute to the dict (previously it only accepted JSON files containing records as rows of dicts in the file).

In this case, you should indicate using `field=XXX` the name of the field in the JSON structure which contains the records you want to load. The records can be a dict of lists or a list of dicts.

E.g. to load the SQuAD dataset JSON (without using the `squad` specific dataset loading script), in which the data rows are in the `data` field of the JSON dict, you can do:
```python
from nlp import load_dataset
dataset = load_dataset('json', data_files='/PATH/TO/JSON', field='data')
```


Fix cached file path for metrics with different config names

The config name was not taken into account to build the cached file path.
It should fix #368


Allow indexing Dataset via np.ndarray

Looks like a flaky CI, failed download from S3.


can't load local dataset: pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries

I am able to reproduce this with the official SQuAD `train-v2.0.json` file downloaded directly from https://rajpurkar.github.io/SQuAD-explorer/

I am facing this issue in transformers library 3.0.2 while reading a csv using datasets.
Is this fixed in latest version?
I updated the latest version 4.0.1 but still getting this error. What could cause this error?

Trying to load a local SQuAD-formatted dataset (from a JSON file, about 60MB):
```
dataset = nlp.load_dataset(path='json', data_files={nlp.Split.TRAIN: ["./path/to/file.json"]})
```
causes
```
Traceback (most recent call last):
  File "dataloader.py", line 9, in <module>
    ["./path/to/file.json"]})
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/load.py", line 524, in load_dataset
    save_infos=save_infos,
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 432, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 483, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 719, in _prepare_split
    for key, table in utils.tqdm(generator, unit=" tables", leave=False):
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/tqdm/std.py", line 1129, in __iter__
    for obj in iterable:
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/datasets/json/88c1bc5c68489f7eda549ed05a5a738527c613b3e7a4ee3524d9d233353a949b/json.py", line 53, in _generate_tables
    file, read_options=self.config.pa_read_options, parse_options=self.config.pa_parse_options,
  File "pyarrow/_json.pyx", line 191, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries (try to increase block size?)
```

I haven't been able to find any reports of this specific pyarrow error here or elsewhere.


load_metric can't acquire lock anymore

I found that, in the same process (or the same interactive session), if I do

import nlp

m1 = nlp.load_metric('glue', 'mrpc')
m2 = nlp.load_metric('glue', 'sst2')

I will get the same error `ValueError: Cannot acquire lock, caching file might be used by another process, you should setup a unique 'experiment_id'`.

I can't load metric (glue) anymore after an error in a previous run. I even removed the whole cache folder `/home/XXX/.cache/huggingface/`, and the issue persisted. What are the steps to fix this?

    Traceback (most recent call last):
      File "/home/XXX/miniconda3/envs/ML-DL-py-3.7/lib/python3.7/site-packages/nlp/metric.py", line 101, in __init__
        self.filelock.acquire(timeout=1)
      File "/home/XXX/miniconda3/envs/ML-DL-py-3.7/lib/python3.7/site-packages/filelock.py", line 278, in acquire
        raise Timeout(self._lock_file)
    filelock.Timeout: The file lock '/home/XXX/.cache/huggingface/metrics/glue/1.0.0/1-glue-0.arrow.lock' could not be acquired.

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "examples_huggingface_nlp.py", line 268, in <module>
        main()
      File "examples_huggingface_nlp.py", line 242, in main
        dataset, metric = get_dataset_metric(glue_task)
      File "examples_huggingface_nlp.py", line 77, in get_dataset_metric
        metric = nlp.load_metric('glue', glue_config, experiment_id=1)
      File "/home/XXX/miniconda3/envs/ML-DL-py-3.7/lib/python3.7/site-packages/nlp/load.py", line 440, in load_metric
        **metric_init_kwargs,
      File "/home/XXX/miniconda3/envs/ML-DL-py-3.7/lib/python3.7/site-packages/nlp/metric.py", line 104, in __init__
        "Cannot acquire lock, caching file might be used by another process, "
    ValueError: Cannot acquire lock, caching file might be used by another process, you should setup a unique 'experiment_id' for this run.
    I0709 15:54:41.008838 139854118430464 filelock.py:318] Lock 139852058030936 released on /home/XXX/.cache/huggingface/metrics/glue/1.0.0/1-glue-0.arrow.lock


Update Xtreme to add PAWS-X es

This PR adds the `PAWS-X.es` in the Xtreme dataset #362


Add quora dataset

Tests seem to be failing because of pandas

Kaggle needs authentification to download datasets. We don't have a way to handle that in the lib for now

Added the [Quora question pairs dataset](https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs).

Implementation Notes:
- I used the original version provided on the quora website. There's also a [Kaggle competition](https://www.kaggle.com/c/quora-question-pairs) which has a nice train/test split but I can't find an easy way to download it.
- I've made the questions into a list:
    ```python
    {
        "questions": [
            {"id":0, "text": "Is this an example question?"},
            {"id":1, "text": "Is this a sample question?"},
        ],
    ...
    }
    ```
    rather than:
    ```python
    {
        "question1": "Is this an example question?",
        "question2": "Is this a sample question?"
        "qid0": 0
        "qid1": 1
    ...
    }
    ```
    Not sure if this was the right call.
- Can't find a good citation for this dataset


How to augment data ?

Using batched map is probably the easiest way at the moment.
What kind of augmentation would you like to do ?

Some samples in the dataset are too long, I want to divide them in several samples.

Using batched map is the way to go then.
We'll make it clearer in the docs that map could be used for augmentation.

Let me know if you think there should be another way to do it. Or feel free to close the issue otherwise.

It just feels awkward to use map to augment data. Also it means it's not possible to augment data in a non-batched way.

But to be honest I have no idea of a good API...

Or for non-batched samples, how about returning a tuple ?

```python
def aug(sample):
    # Simply copy the existing data to have x2 amount of data
    return sample, sample

dataset = dataset.map(aug)
```

It feels really natural and easy, but :

* it means the behavior with batched data is different
* I don't know how doable it is backend-wise

@lhoestq

As we're working with arrow's columnar format we prefer to play with batches that are dictionaries instead of tuples.
If we have tuple it implies to re-format the data each time we want to write to arrow, which can lower the speed of map for example.

It's also a matter of coherence, as we don't want users to be confused whether they have to return dictionaries for some functions and tuples for others when they're doing batches.

Is there any clean way to augment data ?

For now my work-around is to use batched map, like this :

```python
def aug(samples):
    # Simply copy the existing data to have x2 amount of data
    for k, v in samples.items():
        samples[k].extend(v)
    return samples

dataset = dataset.map(aug, batched=True)
```


add MS MARCO dataset

The dummy data for v2.1 is missing as far as I can see. I think running the  dummy data command should work correctly here.

Also, it might be that the structure of the dummy data is wrong - looking at `generate_examples` the structure does not look too easy.

The fact that the dummy data for v2.1 is missing shouldn't make the test fails I think. But as you mention the dummy data structure of v1.1 is wrong. I tried to rename files but it does not solve the issue.

> Is MS mARCO added to nlp library?I am not able to view it?

Hi @parthplc ,the PR is  not merged yet. The dummy data structure is still failing. Maybe @patrickvonplaten can help with it.

> Dataset is fixed and should be ready for use. @mariamabarham @lhoestq feel free to merge whenever!

thanks

This PR adds the MS MARCO dataset as requested in this issue #336. MS mARCO has multiple task including:

- Passage and Document Retrieval

- Keyphrase Extraction

- QA and NLG

This PR only adds the 2 versions of the  QA and NLG task dataset which was realeased with the original paper here https://arxiv.org/pdf/1611.09268.pdf

Tests are failing because of the dummy data. I tried to fix it without success. Can you please have a look at it? @patrickvonplaten , @lhoestq