Add LCCC dataset

_The documentation is not available anymore as the PR was closed or merged._

Thank you very much for your help @albertvillanova .

I think I have fixed all the comments.

Please let me know if this PR need further modification ;)

@albertvillanova Thank you very much for your kind help.
These suggestions make the code looks more pythonic.

I have commited these changes

> Hi ! The dataset seems to be a duplicate of https://huggingface.co/datasets/silver/lccc - next time no need to add it on github if it's already available on huggingface.co ;)

OK, sorry for the inconvenience. I have closed another two PRs since these datasets are already available on huggingface.co

It's fine, thanks @silverriver for adding these datasets !

Hi, I am trying to add a new dataset lccc.
All tests are passed.


Update `dataset_infos.json` with new split info in `Dataset.push_to_hub` to avoid verification error

Update `dataset_infos.json` when pushing splits one by one via `Dataset.push_to_hub` to avoid the splits verification error.

TODO:
~~- [ ] handle token + `{Audio, Image}.embed_storage`~~
- [x] tests


Rename DatasetBuilder config_name

This PR renames the DatasetBuilder keyword argument `name` to `config_name` so that:
- it avoids confusion with the attribute `DatasetBuilder.name`, which is different
- it aligns with the Dataset property name `config_name`, defined in `DatasetInfoMixin.config_name`

Other simpler possibility could be to rename it to just `config` instead.

Please note I have only renamed this argument of DatasetBuilder because I think this refactoring has a low impact on users: we can assume this is not a public facing parameter, but private or related to the inners of our library.

It would have a  major impact to rename it also in:
- load_dataset
- load_dataset_builder: although this could also be assumed as inners...
- in our CLI commands

Besides the naming of `name`, I also find really confusing the naming of `path` in `load_dataset`. IMHO, they should have a more simpler and precise meaning (currently, they are too vague). I would propose (maybe for next major release):
```
load_dataset(dataset, config,...
```
instead of
```
load_dataset(path, name,...
```


Dataset Viewer issue for ett

dataset-viewer

Related to the dataset viewer on huggingface.co

Thanks for reporting @dgcnz.

I have checked that the dataset works fine in streaming mode.

Additionally, other datasets containing timestamps are properly rendered by the viewer: https://huggingface.co/datasets/blbooks

I have tried to force the refresh of the preview, but the endpoint is not responsive: Connection timed out

CC: @severo

I've just resent the refresh of the preview to the new endpoint, without success.

CC: @severo

Fixed!

https://huggingface.co/datasets/ett/viewer/h1/test

<img width="982" alt="Capture d’écran 2022-06-15 à 09 30 22" src="https://user-images.githubusercontent.com/1676121/173769035-a075d753-ecfc-4a43-b54b-973105d464d3.png">

### Link

https://huggingface.co/datasets/ett

### Description

Timestamp is not JSON serializable.

```
Status code:   500
Exception:     Status500Error
Message:       Type is not JSON serializable: Timestamp
```

### Owner

No


Skip hidden files/directories in data files resolution and `iter_files`

This PR (via new release) broke many transformers tests.

I will try to post a summary shortly.

cc: @ydshieh

So now it can't handle a local path via: `--train_file tests/deepspeed/../fixtures/tests_samples/wmt_en_ro/train.json` even though it's there. it works just fine if I change the path to not have `..`

You can reproduce the original problem with:

```
$ cd transformers
$ python examples/pytorch/translation/run_translation.py --model_name_or_path t5-small --train_file tests/fixtures/tests_samples/wmt_en_ro/train.json --validation_file tests/deepspeed/../fixtures/tests_samples/wmt_en_ro/val.json --output_dir /tmp/tmp5o5to4k0 --overwrite_output_dir --max_source_length 32 --max_target_length 32 --val_max_target_length 32 --warmup_steps 8 --predict_with_generate --save_steps 0 --eval_steps 1 --group_by_length --label_smoothing_factor 0.1 --source_lang en --target_lang ro --report_to none --source_prefix "translate English to Romanian: " --fp16 --do_train --num_train_epochs 1 --max_train_samples 16 --per_device_train_batch_size 2 --learning_rate 3e-3
[...]
Traceback (most recent call last):
  File "examples/pytorch/translation/run_translation.py", line 656, in <module>
    main()
  File "examples/pytorch/translation/run_translation.py", line 346, in main
    raw_datasets = load_dataset(
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/load.py", line 1656, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/load.py", line 1439, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/load.py", line 1097, in dataset_module_factory
    return PackagedDatasetModuleFactory(
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/load.py", line 743, in get_module
    data_files = DataFilesDict.from_local_or_remote(
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/data_files.py", line 588, in from_local_or_remote
    DataFilesList.from_local_or_remote(
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/data_files.py", line 556, in from_local_or_remote
    data_files = resolve_patterns_locally_or_by_urls(base_path, patterns, allowed_extensions)
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/data_files.py", line 194, in resolve_patterns_locally_or_by_urls
    for path in _resolve_single_pattern_locally(base_path, pattern, allowed_extensions):
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/data_files.py", line 144, in _resolve_single_pattern_locally
    raise FileNotFoundError(error_msg)
FileNotFoundError: Unable to find '/mnt/nvme0/code/huggingface/transformers-master/tests/deepspeed/../fixtures/tests_samples/wmt_en_ro/val.json' at /mnt/nvme0/code/huggingface/transformers-master
```

will apply a workaround to `transformers` tests here https://github.com/huggingface/transformers/pull/17721

This has been fixed with https://github.com/huggingface/datasets/pull/4505, will do a patch release tomorrow for `datasets` ;)

Fix #4115


Update `_format_columns` in `remove_columns`

🤗 This PR closes https://github.com/huggingface/datasets/issues/4398

Hi! Thanks for reporting and providing a fix. I made a small change to make the fix easier to understand.

Hi, @mariosasko thanks! It makes sense, sorry I'm not that familiar with `datasets` code 😩

Sure @albertvillanova I'll do that later today and ping you once done, thanks! :hugs:

Hi again @albertvillanova! Let me know if those tests are fine 🤗

Hi @alvarobartt,

I think your tests are failing. I don't know why previously, after your last commit, the CI tests were not triggered.

In order to force the re-running of the CI tests, I had to edit your file using the GitHub UI.

First I tried to do it using my terminal, but I don't have push right to your PR branch: I would ask you next time you open a PR, please mark the checkbox "Allow edits from maintainers": https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork#enabling-repository-maintainer-permissions-on-existing-pull-requests

Hi @albertvillanova, let me check those again! And regarding that checkbox I thought it was already checked so my bad there 😩

@albertvillanova  again it seems that the tests were not automatically triggered, but I tested those locally and now they work, as previously those were failing as I created an assertion as `self.assertEqual` over an empty list that was compared as `None` while the value was `[]` so I updated it to be `self.assertListEqual` and changed the comparison value to `[]`.

@lhoestq any idea why the CI is not triggered?

@alvarobartt I have tested locally and the tests continue failing.

I think there is a basis error: `new_dset._format_columns` is always `None` in those cases.

You're right @albertvillanova I was indeed running the tests with `datasets==2.2.0` rather than with the branch version, I'll check it again! Sorry for the inconvenience...

> @alvarobartt I have tested locally and the tests continue failing.
>
> I think there is a basis error: `new_dset._format_columns` is always `None` in those cases.

In order to have some regressions tests for the fixed scenario, I've manually updated the value of `_format_columns` in the `ArrowDataset` so as to check whether it's updated or not right after calling `remove_columns`, and it does behave as expected, so with the latest version of this branch the reported issue doesn't occur anymore.

Hi again @albertvillanova sorry I was on leave! I'll do that ASAP :hugs:

@albertvillanova, does it make sense to add regression tests for `DatasetDict`? As `DatasetDict` doesn't have the attribute `_format_columns`, when we call `remove_columns` over a `DatasetDict` it removes the columns and updates the attributes of each split which is an `ArrowDataset`.

So on, we can either:
- Update first the `_format_columns` attribute of each split and then remove the columns over the `DatasetDict`
- Loop over the splits of `DatasetDict` and remove the columns right after updating `_format_columns` of each `ArrowDataset`.

I assume that the best regression test is the one implemented (mentioned first above), let me know if there's a better way to do that 👍🏻

> I think there's already a decorator to support transmitting the right `_format_columns`: `@transmit_format`, have you tried adding this decorator to `remove_columns` ?

Hi @lhoestq I can check now!

It worked indeed @lhoestq, thanks for the proposal and the review! 🤗

Oops, I forgot about `@transmit_format`'s existence. From what I see, we should also use this decorator in `flatten`, `rename_column` and  `rename_columns`.

@alvarobartt Let me know if you'd like to work on this (in a subsequent PR).

Sure @mariosasko I can prepare another PR to add those too, thanks!

As explained at #4398, when calling `dataset.add_faiss_index` under certain conditions when calling a sequence of operations `cast_column`, `map`, and `remove_columns`, it fails as it's trying to look for already removed columns.

So on, after testing some possible fixes, it seems that setting the dataset format right after removing the columns seems to be working fine, so I had to add a call to `.set_format` in the `remove_columns` function.

Hope this helps!


Remove Google Drive URL in spider dataset

_The documentation is not available anymore as the PR was closed or merged._

The `spider` dataset is distributed under the [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/legalcode) license.

Fix #4401.


Update: add using pcm bytes (#4323)

@lhoestq Maybe I'm missing something, but what's the reason to read and encode PCM files to WAV in `Audio.encode_example`. Isn't the whole purpose of the decodable types to operate on raw files whenever possible? IMO this PR should only modify `Audio.decode_example` to support PCM files/bytes decoding.

Because the PCM file is not enough, we also need the `sampling_rate` associated to it. Therefore the two alternatives are either:
- convert to WAV
- add a `sampling_rate` field to the Audio arrow storage (not sure how it would behave for backward compatibility though)

How does it get the sampling rate of a PCM file then ? According to [SO](https://stackoverflow.com/a/57027667/17517845) it's not possible to infer it from the file alone

> Awesome thanks ! Could you also add tests in `tests/features/test_audio.py` ?
>
> Maybe add a small pcm file in `tests/features/data` and check that everything works as expected in tests cases like `test_audio_encode_example_pcm` and `test_audio_decode_example_pcm` for example.

@lhoestq  how can i test test_audio.py? where is "__main__" func?
do you have some example or guideline?

> But [`scipy.io.wavfile.read`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.wavfile.read.html), which is used for reading such files, returns a file's sampling rate. The only tricky part is [resampling](https://stackoverflow.com/questions/33682490/how-to-read-a-wav-file-using-scipy-at-a-different-sampling-rate) to a different sampling rate than the default one.

@mariosasko @lhoestq
thanks for comment!

First of all, "PCM file" can not read alone to any audio library.
"PCM file" has not any audio META information header. (it just purely audio byte data. therefore, we don't have to encoding and decoding)
but, "PCM file" is audio extension, so we can use `datasets.Audio`

if you want to read "PCM file" to audio file likely, it have to needs additional parameter. (channel, sampling_rate, else....)
but, in many situation, we only know sampling_rate for PCM

and, if we want to use `datasets.Audio` for "PCM file", we must process encode_example.
therefore, i have to use sampling_rate for encoding for making wav-style byte. (we only know sampling_rate)

In my source code, I don't compare sampling rate(`datasets.Audio's self.sampling_rate` and `read pcm sampling_rate(value["sampling_rate"])`) and checking mono
@mariosasko ! do you want to process resampling and making mono? then i can modify my source

There is no "main" function in test scripts :) To run a test script you must use the `pytest` command:
```
pytest tests/features/test_audio.py
```

to run only one function you can also do
```
pytest tests/features/test_audio.py::test_audio_feature_type_to_arrow
```
for example

@lhoestq
maybe, if i write test code, i have to commit test_audio.py and send pr?
because, we need to keep `test_audio_encode_example_pcm` and `test_audio_decode_example_pcm` method after my pr merged?

You can add your tests in this PR with the other changes you did

@lhoestq
test complete & commit my test_audio.py

AND, some change in my code.

audio.py
i think "sampling_rate" is already Audio object initial variable. so, we don`t have to use input parameter.

test_audio.py
we can check "PCM" file to path (exactly, extenstion)
so, test case has to know `path`. if only have `bytes`, we don`t know that is "PCM" or not

@lhoestq
and, why circleci raised exception?
maybe, [repo](https://huggingface.co/api/datasets/lhoestq/_dummy?full=true) url is not found!
PLZ, CHK!

@lhoestq
test_audio.py
if we don`t use path in pcm, test-case need to be changed
so, we check path just None

i'm merge branch already and `multiprocess` in `setup.py` but circleci error only win version
![image](https://user-images.githubusercontent.com/34292279/175461714-c7d2e741-3b7b-40a3-bba9-13ce2af0055c.png)
how can i fixed it?

@lhoestq thx for comment!
test_audio.py test complete. it runs sucessfully
and, self.get("sampling_rate") -> value.get("sampling_rate") changed

and, some comment is not agreed to me, plz check my sub comment!

first of all, please look #4323

why i can not use {"path","array","sampling_rate"}
because sf.write(format="wav") and sf.read(BytesIO) is changed my pcm data value
maybe, i think wav got header but, pcm is not.
and variable naming, pcm data is "byte" type. so, "array" name is not fair i think

so, i use scipy lib and numpy (that is huggingface dependency)
and refer to @lhoestq answered,
1. encode -> using sampling_rate and pcm byte -> wav style byte (scipy.wavfile.write to byte)
2. byte converting using fairseq style pcm audio read  [FileAudioDataset](https://github.com/facebookresearch/fairseq/blob/main/fairseq/data/audio/raw_audio_dataset.py)
4. decode -> read wavfile.read

that way is not screw up my pcm byte to float data, and another audio type(wav) safety

please check!


Update imagenet gate


Dataset Viewer issue for conll2012_ontonotesv5

Thanks for reporting, @jiangwy99.

I guess this could be addressed only once we fix our issue with irresponsive backend endpoint.

CC: @severo

I've just sent the forcing of the refresh of the preview to the new endpoint.

Fixed, thanks for the patience. The issue was the amount of RAM allowed to extract the first rows of the dataset was not sufficient.

### Link

https://huggingface.co/datasets/conll2012_ontonotesv5

### Description

Dataset viewer outage.

### Owner

No


Improve language tag for PIAF dataset

Hi,

As pointed out by @lhoestq in this discussion (https://huggingface.co/datasets/asi/wikitext_fr/discussions/1), it is not yet possible to edit datasets outside of a namespace with the Hub PR feature and that you have to go through GitHub.

This modification should allow better referencing since only the xx language tags are currently taken into account and not the xx-xx.


[TypeError: Couldn't cast array of type] Cannot process dataset in v2.2.2

bug

Something isn't working

And if the problem is that the way I am to construct the {Entity Type: list of spans} makes entity types without any spans hard to handle, is there a better way to meet the demand? Although I have verified that to make entity types without any spans to behave like `entity_chunk[label] = [[""]]` can perform normally, I still wonder if there is a more elegant way?

## Describe the bug
I am trying to process the [conll2012_ontonotesv5](https://huggingface.co/datasets/conll2012_ontonotesv5) dataset in `datasets` v2.2.2 and am running into a type error when casting the features.

## Steps to reproduce the bug
```python
import os
from typing import (
    List,
    Dict,
)
from collections import (
    defaultdict,
)
from dataclasses import (
    dataclass,
)
from datasets import (
    load_dataset,
)

@dataclass
class ConllConverter:

    path: str
    name: str
    cache_dir: str

    def __post_init__(
        self,
    ):
        self.dataset = load_dataset(
            path=self.path,
            name=self.name,
            cache_dir=self.cache_dir,
        )

    def convert(
        self,
    ):

        class_label = self.dataset["train"].features["sentences"][0]["named_entities"].feature
        # label_set = list(set([
        #     label.split("-")[1] if label != "O" else label for label in class_label.names
        # ]))

        def prepare_chunk(token, entity):
            assert len(token) == len(entity)
            # Sequence length
            length = len(token)
            # Variable used
            entity_chunk = defaultdict(list)
            idx = flag = 0
            # While loop
            while idx < length:
                if entity[idx] == "O":
                    flag += 1
                    idx += 1
                else:
                    iob_tp, lab_tp = entity[idx].split("-")
                    assert iob_tp == "B"
                    idx += 1
                    while idx < length and entity[idx].startswith("I-"):
                        idx += 1
                    entity_chunk[lab_tp].append(token[flag: idx])
                    flag = idx
            entity_chunk = dict(entity_chunk)
            # for label in label_set:
            #     if label != "O" and label not in entity_chunk.keys():
            #         entity_chunk[label] = None
            return entity_chunk

        def prepare_features(
            batch: Dict[str, List],
        ) -> Dict[str, List]:
            sentence = [
                sent for doc_sent in batch["sentences"] for sent in doc_sent
            ]
            feature = {
                "sentence": list(),
            }
            for sent in sentence:
                token = sent["words"]
                entity = class_label.int2str(sent["named_entities"])
                entity_chunk = prepare_chunk(token, entity)
                sent_feat = {
                    "token": token,
                    "entity": entity,
                    "entity_chunk": entity_chunk,
                }
                feature["sentence"].append(sent_feat)

            return feature

        column_names = self.dataset.column_names["train"]
        dataset = self.dataset.map(
            function=prepare_features,
            with_indices=False,
            batched=True,
            batch_size=3,
            remove_columns=column_names,
            num_proc=1,
        )
        dataset.save_to_disk(
            dataset_dict_path=os.path.join("data", self.path, self.name)
        )

if __name__ == "__main__":
    converter = ConllConverter(
        path="conll2012_ontonotesv5",
        name="english_v4",
        cache_dir="cache",
    )
    converter.convert()

```

## Expected results
I want to use the dataset to perform NER task and to change the label list into a {Entity Type: list of spans} format.

## Actual results
<details>
<summary>Traceback</summary>

```python
Traceback (most recent call last):                                                                                | 0/81 [00:00<?, ?ba/s]
  File "/home2/jiangwangyi/miniconda3/lib/python3.9/site-packages/multiprocess/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home2/jiangwangyi/miniconda3/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 532, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home2/jiangwangyi/miniconda3/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 499, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home2/jiangwangyi/miniconda3/lib/python3.9/site-packages/datasets/fingerprint.py", line 458, in wrapper
    out = func(self, *args, **kwargs)
  File "/home2/jiangwangyi/miniconda3/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 2751, in _map_single
    writer.write_batch(batch)
  File "/home2/jiangwangyi/miniconda3/lib/python3.9/site-packages/datasets/arrow_writer.py", line 503, in write_batch
    arrays.append(pa.array(typed_sequence))
  File "pyarrow/array.pxi", line 230, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/home2/jiangwangyi/miniconda3/lib/python3.9/site-packages/datasets/arrow_writer.py", line 198, in __arrow_array__
    out = cast_array_to_feature(out, type, allow_number_to_str=not self.trying_type)
  File "/home2/jiangwangyi/miniconda3/lib/python3.9/site-packages/datasets/table.py", line 1675, in wrapper
    return func(array, *args, **kwargs)
  File "/home2/jiangwangyi/miniconda3/lib/python3.9/site-packages/datasets/table.py", line 1793, in cast_array_to_feature
    arrays = [_c(array.field(name), subfeature) for name, subfeature in feature.items()]
  File "/home2/jiangwangyi/miniconda3/lib/python3.9/site-packages/datasets/table.py", line 1793, in <listcomp>
    arrays = [_c(array.field(name), subfeature) for name, subfeature in feature.items()]
  File "/home2/jiangwangyi/miniconda3/lib/python3.9/site-packages/datasets/table.py", line 1675, in wrapper
    return func(array, *args, **kwargs)
  File "/home2/jiangwangyi/miniconda3/lib/python3.9/site-packages/datasets/table.py", line 1844, in cast_array_to_feature
    raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{feature}")
TypeError: Couldn't cast array of type
struct<CARDINAL: list<item: list<item: string>>, DATE: list<item: list<item: string>>, EVENT: list<item: list<item: string>>, FAC: list<item: list<item: string>>, GPE: list<item: list<item: string>>, LANGUAGE: list<item: list<item: string>>, LAW: list<item: list<item: string>>, LOC: list<item: list<item: string>>, MONEY: list<item: list<item: string>>, NORP: list<item: list<item: string>>, ORDINAL: list<item: list<item: string>>, ORG: list<item: list<item: string>>, PERCENT: list<item: list<item: string>>, PERSON: list<item: list<item: string>>, QUANTITY: list<item: list<item: string>>, TIME: list<item: list<item: string>>, WORK_OF_ART: list<item: list<item: string>>>
to
{'CARDINAL': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'DATE': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'EVENT': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'FAC': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'GPE': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'LAW': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'LOC': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'MONEY': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'NORP': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'ORDINAL': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'ORG': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'PERCENT': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'PERSON': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'PRODUCT': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'QUANTITY': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'TIME': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'WORK_OF_ART': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None)}
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home2/jiangwangyi/workspace/work/Entity/dataconverter.py", line 110, in <module>
    converter.convert()
  File "/home2/jiangwangyi/workspace/work/Entity/dataconverter.py", line 91, in convert
    dataset = self.dataset.map(
  File "/home2/jiangwangyi/miniconda3/lib/python3.9/site-packages/datasets/dataset_dict.py", line 770, in map
    {
  File "/home2/jiangwangyi/miniconda3/lib/python3.9/site-packages/datasets/dataset_dict.py", line 771, in <dictcomp>
    k: dataset.map(
  File "/home2/jiangwangyi/miniconda3/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 2459, in map
    transformed_shards[index] = async_result.get()
  File "/home2/jiangwangyi/miniconda3/lib/python3.9/site-packages/multiprocess/pool.py", line 771, in get
    raise self._value
TypeError: Couldn't cast array of type
struct<CARDINAL: list<item: list<item: string>>, DATE: list<item: list<item: string>>, EVENT: list<item: list<item: string>>, FAC: list<item: list<item: string>>, GPE: list<item: list<item: string>>, LANGUAGE: list<item: list<item: string>>, LAW: list<item: list<item: string>>, LOC: list<item: list<item: string>>, MONEY: list<item: list<item: string>>, NORP: list<item: list<item: string>>, ORDINAL: list<item: list<item: string>>, ORG: list<item: list<item: string>>, PERCENT: list<item: list<item: string>>, PERSON: list<item: list<item: string>>, QUANTITY: list<item: list<item: string>>, TIME: list<item: list<item: string>>, WORK_OF_ART: list<item: list<item: string>>>
to
{'CARDINAL': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'DATE': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'EVENT': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'FAC': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'GPE': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'LAW': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'LOC': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'MONEY': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'NORP': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'ORDINAL': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'ORG': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'PERCENT': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'PERSON': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'PRODUCT': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'QUANTITY': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'TIME': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None), 'WORK_OF_ART': Sequence(feature=Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), length=-1, id=None)}
```

</details>

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.2
- Platform: Ubuntu 18.04
- Python version: 3.9.7
- PyArrow version: 7.0.0


Dataset should have a `.name` field

enhancement

New feature or request

Hi! You can already use `dset.builder_name` and `dset.config_name` for that purpose. And when it comes to versioning, it's better to use `dset._fingerprint` than the `version` attribute as the former represents a deterministic hash that encodes all the mutable ops executed on a dataset, and the latter stays the same unless it's manually updated after each op.

@mariosasko  Can we make ._fingerprint not private? seems a critical component for tracking how a model was generated to ensure reproducibility.

**Is your feature request related to a problem? Please describe.**
If building pipelines that can evaluate on more than one dataset, it would be nice to be able to log results of things like `Evaluating on {dataset.name}` or `results for {dataset.name} are: {results}`

Without some way of concisely identifying a dataset from the dataset object, tools which might run on more than one dataset must be passed the dataset object _and_ the name/id of the dataset being used.

**Describe the solution you'd like**
The DatasetInfo class should have a `name` field which is the name of a dataset. then for a given dataset if it evolves in time the `version` can be updated but its different versions of the same dataset with a unique `name`. The name could then all be accessed by `dataset.name`

**Describe alternatives you've considered**
For my own purposes I am considering making `NamedDataset[Dataset]` where the subclass just has a .name field.

**Additional context**
My guess is that most usecases are not working with more than one dataset in a given pipeline so a name is not really needed. This has surprised me though as one of the advantages of a standard dataset interface is to be able to build pipelines which can be passed in a dataset and separate responsibilities of the dataset loading from the train or eval pipeline.


Uncomment logging deactivation for ArrowBasedBuilder

_The documentation is not available anymore as the PR was closed or merged._


Skip identical files in `push_to_hub` instead of overwriting

Skip identical files instead of overwriting them to save bandwidth and circumvent (user-side/server-side) errors, which can arise when working with large datasets due to long-lasting HTTP connections, by repeating calls to `push_to_hub` to resume an upload.

To be able to check if an upload can be resumed, this PR modifies the shard naming scheme from:
```
data/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9].parquet
```
to:
```
data/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]-<SHARD_FINGERPRINT>.parquet
```
cc @LysandreJik


"NonMatchingChecksumError" when importing 'spider' dataset

Thanks for reporting, @OmarAlaaeldein.

Datasets hosted at Google Drive give problems quite often due to a change in their service:
- #3786

Related to:
- #3906

I'm having a look.

We have made a Pull Request to replace the Google Drive URL. This fix will be accessible in our next `datasets` library release.

In the meantime, once the PR merged into master, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```

## Describe the bug
When importing 'spider' dataset [https://huggingface.co/datasets/spider] an error occurs

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset('spider')
```

## Expected results
Dataset object

## Actual results
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1_AckYkinAnhqmRQtGsQgUKAnTHxxX5J0']

## Environment info
- `datasets` version: 2.2.2
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.11
- PyArrow version: 6.0.1
- Pandas version: 1.3.5


load dataset wikitext-2-raw-v1 failed. Could not reach wikitext-2-raw-v1.py.

bug

Something isn't working

## Describe the bug
Could not reach wikitext-2-raw-v1.py

## Steps to reproduce the bug
```python
from datasets import load_dataset
load_dataset("wikitext-2-raw-v1")
```

## Expected results
Download `wikitext-2-raw-v1` dataset successfully.

## Actual results
```
  File "load_datasets.py", line 13, in <module>
    load_dataset("wikitext-2-raw-v1")
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 1715, in load_dataset
    **config_kwargs,
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 1536, in load_dataset_builder
    data_files=data_files,
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 1282, in dataset_module_factory
    raise e1 from None
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 1224, in dataset_module_factory
    dynamic_modules_path=dynamic_modules_path,
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 559, in get_module
    local_path = self.download_loading_script(revision)
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/load.py", line 539, in download_loading_script
    return cached_path(file_path, download_config=download_config)
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 246, in cached_path
    download_desc=download_config.download_desc,
  File "/root/miniconda3/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 582, in get_from_cache
    raise ConnectionError(f"Couldn't reach {url} ({repr(head_error)})")
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.2.2/datasets/wikitext-2-raw-v1/wikitext-2-raw-v1.py (ReadTimeout(ReadTimeoutError("HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Read timed out. (read timeout=100)",),))
```
I tried to download wikitext-2-raw-v1.py by chrome and got:
![image](https://user-images.githubusercontent.com/20658907/170171595-0ca9f1da-c05a-4b57-861e-9530bfa3bdb9.png)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.2
- Platform: CentOS 7
- Python version: 3.6
- PyArrow version: 3.0.0


LocalDatasetModuleFactoryWithoutScript extracts invalid builder name

Something isn't working

good first issue

Good for newcomers

Ok, so
```
os.path.basename("/home/user/")
```
gives `''` while
```
os.path.basename("/home/user")
```
gives `user`.
The code should check if the last char is a slash.

The fix is:
```
"name": os.path.basename(self.path[:-1] if self.path[-1] == "/" else self.path)
```

I came through the same issue , just removing the last slash in the dataset path fixed it for me, may be this repo moderators could accept this as an accepted answer atleast if this could not be integrated

> The fix is:
>
> ```
> "name": os.path.basename(self.path[:-1] if self.path[-1] == "/" else self.path)
> ```

@apohllo consider making a pull request on this

Thanks for the amazing contributions from huggingface people !!

@apohllo Would you be interested in submitting a PR with the fix?

@mariosasko here we go:

https://github.com/huggingface/datasets/pull/4967

TBH I haven't tested it yet, but should work, since this is a basic change.

## Describe the bug
Trying to load a local dataset raises an error indicating that the config builder has to have a name.
No error should be reported, since the call is completly valid.

## Steps to reproduce the bug
```python
load_dataset("./data/some-dataset/", name="some-name")
```

## Expected results
The dataset should be loaded.

## Actual results
```
Traceback (most recent call last):
  File "train_lquad.py", line 19, in <module>
    load(tokenize_target_function, tokenize_target_function, {}, tokenizer)
  File "train_lquad.py", line 14, in load
    dataset = load_dataset("./data/lquad/", name="lquad")
  File "/net/pr2/scratch/people/plgapohl/python-3.8.6/lib/python3.8/site-packages/datasets/load.py", line 1708, in load_dataset
    builder_instance = load_dataset_builder(
  File "/net/pr2/scratch/people/plgapohl/python-3.8.6/lib/python3.8/site-packages/datasets/load.py", line 1560, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/net/pr2/scratch/people/plgapohl/python-3.8.6/lib/python3.8/site-packages/datasets/builder.py", line 269, in __init__
    self.config, self.config_id = self._create_builder_config(
  File "/net/pr2/scratch/people/plgapohl/python-3.8.6/lib/python3.8/site-packages/datasets/builder.py", line 403, in _create_builder_config
    raise ValueError(f"BuilderConfig must have a name, got {builder_config.name}")
ValueError: BuilderConfig must have a name, got
```

## Environment info
- `datasets` version: 2.2.2
- Platform: Linux-4.18.0-348.20.1.el8_5.x86_64-x86_64-with-glibc2.2.5
- Python version: 3.8.6
- PyArrow version: 8.0.0
- Pandas version: 1.4.2

The error is probably in line 795 in load.py:

```
 builder_kwargs = {
     "hash": hash,
     "data_files": data_files,
     "name": os.path.basename(self.path),
     "base_path": self.path,
     **builder_kwargs,
 }
```

`os.path.basename` for a directory returns an empty string, rather than the name of the directory.


Calling `cast_column`/`remove_columns` and a sequence of `map` operations ends up making `faiss` fail with `ValueError`

bug

Something isn't working

It works if we either remove the `ds = ds.cast_column("id", Value("int32"))` line from the code above, or if instead calling `ds.remove_columns()` we remove the columns inside each mapping as `ds.map(..., remove_columns=[...])` instead of right after the mapping.

Both of those solutions seem to fix the issue, so the root cause of it may be around that. Sorry I cannot provide you more insights, in case I get to fix it I'll submit a PR, in the meanwhile the code that I'm using as a workaround is the following:

```python
from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
import torch

torch.set_grad_enabled(False)
ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")

from datasets import load_dataset, Value

ds = load_dataset("csv", data_files=["sample.csv"], split="train")
ds = ds.cast_column("id", Value("int32"))
ds = ds.map(lambda x: {"inputs": f"{ctx_tokenizer.sep_token}".join(["title", "summary"])}, remove_columns=["title", "summary"])

def generate_embeddings(x):
    return {"embeddings": ctx_encoder(**ctx_tokenizer(x["inputs"], return_tensors="pt"))[0][0].numpy()}

ds = ds.map(generate_embeddings, remove_columns=["inputs"])
ds.add_faiss_index(column="embeddings")
```

FYI the main reason I want to use `dataset.remove_columns` rather than the function inside `dataset.map` is because according to the 🤗 Datasets documentation, it's faster.

"🤗 Datasets also has a [Dataset.remove_columns()](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/main_classes#datasets.Dataset.remove_columns) method that is functionally identical, but faster, because it doesn’t copy the data of the remaining columns."

More information at https://huggingface.co/docs/datasets/process#map

Here I'm presenting all the scenarios so that you can further investigate the issue:

- ✅ `cast_column` -> `map` with `remove_columns` -> `map` with `remove_columns` -> `add_faiss_index`

  ```python
  from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
  import torch

  torch.set_grad_enabled(False)
  ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
  ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")

  from datasets import load_dataset, Value

  ds = load_dataset("csv", data_files=["sample.csv"], split="train")
  ds = ds.cast_column("id", Value("int32"))
  ds = ds.map(lambda x: {"inputs": f"{ctx_tokenizer.sep_token}".join(["title", "summary"])}, remove_columns=["title", "summary"])

  def generate_embeddings(x):
      return {"embeddings": ctx_encoder(**ctx_tokenizer(x["inputs"], return_tensors="pt"))[0][0].numpy()}

  ds = ds.map(generate_embeddings, remove_columns=["inputs"])
  ds.add_faiss_index(column="embeddings")
  ```

- ❌ `cast_column` -> `map` -> `remove_columns` -> `map` -> `remove_columns` -> `add_faiss_index`

  ```python
  from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
  import torch

  torch.set_grad_enabled(False)
  ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
  ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")

  from datasets import load_dataset, Value

  ds = load_dataset("csv", data_files=["sample.csv"], split="train")
  ds = ds.cast_column("id", Value("int32"))
  ds = ds.map(lambda x: {"inputs": f"{ctx_tokenizer.sep_token}".join(["title", "summary"])})
  ds = ds.remove_columns(["title", "summary"])

  def generate_embeddings(x):
      return {"embeddings": ctx_encoder(**ctx_tokenizer(x["inputs"], return_tensors="pt"))[0][0].numpy()}

  ds = ds.map(generate_embeddings)
  ds = ds.remove_columns(["inputs"])
  ds.add_faiss_index(column="embeddings")
  ```

- ❌ `cast_column` -> `map` with `remove_columns` -> `map` -> `remove_columns` -> `add_faiss_index`

  ```python
  from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
  import torch

  torch.set_grad_enabled(False)
  ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
  ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")

  from datasets import load_dataset, Value

  ds = load_dataset("csv", data_files=["sample.csv"], split="train")
  ds = ds.cast_column("id", Value("int32"))
  ds = ds.map(lambda x: {"inputs": f"{ctx_tokenizer.sep_token}".join(["title", "summary"])}, remove_columns=["title", "summary"])

  def generate_embeddings(x):
      return {"embeddings": ctx_encoder(**ctx_tokenizer(x["inputs"], return_tensors="pt"))[0][0].numpy()}

  ds = ds.map(generate_embeddings)
  ds = ds.remove_columns(["inputs"])
  ds.add_faiss_index(column="embeddings")
  ```

- ✅ `cast_column` -> `map` -> `remove_columns` -> `map` with `remove_columns` -> `add_faiss_index`

  ```python
  from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
  import torch

  torch.set_grad_enabled(False)
  ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
  ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")

  from datasets import load_dataset, Value

  ds = load_dataset("csv", data_files=["sample.csv"], split="train")
  ds = ds.cast_column("id", Value("int32"))
  ds = ds.map(lambda x: {"inputs": f"{ctx_tokenizer.sep_token}".join(["title", "summary"])})
  ds = ds.remove_columns(["title", "summary"])

  def generate_embeddings(x):
      return {"embeddings": ctx_encoder(**ctx_tokenizer(x["inputs"], return_tensors="pt"))[0][0].numpy()}

  ds = ds.map(generate_embeddings, remove_columns=["inputs"])
  ds.add_faiss_index(column="embeddings")
  ```

- ✅ `map` -> `remove_columns` -> `map` -> `remove_columns` -> `add_faiss_index`

  ```python
  from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
  import torch

  torch.set_grad_enabled(False)
  ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
  ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")

  from datasets import load_dataset, Value

  ds = load_dataset("csv", data_files=["sample.csv"], split="train")
  ds = ds.map(lambda x: {"inputs": f"{ctx_tokenizer.sep_token}".join(["title", "summary"])})
  ds = ds.remove_columns(["title", "summary"])

  def generate_embeddings(x):
      return {"embeddings": ctx_encoder(**ctx_tokenizer(x["inputs"], return_tensors="pt"))[0][0].numpy()}

  ds = ds.map(generate_embeddings)
  ds = ds.remove_columns(["inputs"])
  ds.add_faiss_index(column="embeddings")
  ```

So on, I've created #4411 so as to fix the bug with `remove_columns` under certain conditions before `add_faiss_index`, which means that the scenarios not working above are already working fine.

First of all, sorry in advance for the unclear title, but this bug is weird to explain (at least for me), so I tried my best to summarize all the information in this issue.

## Describe the bug

Calling a certain combination of operations over a 🤗 `Dataset` and then trying to calculate the `faiss` index with `.add_faiss_index` ends up throwing an exception while trying to set the format back of a previously removed column. But this just happens over certain conditions... I'll present some scenarios below!

## Steps to reproduce the bug

Assuming the following dataset named `sample.csv` with some IMDb data:

```csv
id,title,summary
1877830,"The Batman","When a sadistic serial killer begins murdering key political figures in Gotham, Batman is forced to investigate the city's hidden corruption and question his family's involvement."
9419884,"Doctor Strange in the Multiverse of Madness","Doctor Strange teams up with a mysterious teenage girl from his dreams who can travel across multiverses, to battle multiple threats, including other-universe versions of himself, which threaten to wipe out millions across the multiverse. They seek help from Wanda the Scarlet Witch, Wong and others."
11138512,"The Northman","From visionary director Robert Eggers comes The Northman, an action-filled epic that follows a young Viking prince on his quest to avenge his father's murder."
1745960,"Top Gun: Maverick","After more than thirty years of service as one of the Navy's top aviators, Pete Mitchell is where he belongs, pushing the envelope as a courageous test pilot and dodging the advancement in rank that would ground him."
```

We'll be able to reproduce the bug using the following piece of code:

```python
# Sample code to reproduce the bug
from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
import torch

torch.set_grad_enabled(False)
ctx_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")

from datasets import load_dataset, Value

ds = load_dataset("csv", data_files=["sample.csv"], split="train")
ds = ds.cast_column("id", Value("int32")) # from `int64` to `int32`
ds = ds.map(lambda x: {"inputs": f"{ctx_tokenizer.sep_token}".join(["title", "summary"])})
ds = ds.remove_columns(["title", "summary"])

def generate_embeddings(x):
    return {"embeddings": ctx_encoder(**ctx_tokenizer(x["inputs"], return_tensors="pt"))[0][0].numpy()}

ds = ds.map(generate_embeddings)
ds = ds.remove_columns("inputs")
ds.add_faiss_index(column="embeddings") # It fails here!
```

The code above is an adaptation of https://huggingface.co/docs/datasets/faiss_es, for the sake of presenting the bug with a simple example.

## Expected results

Ideally, the `faiss` index should be calculated over the 🤗 `Dataset` and no exception should be triggered.

## Actual results

But what happens instead is that a `ValueError: Columns ['inputs'] not in the dataset. Current columns in the dataset: ['id', 'embeddings']`, which makes no sense as that column has been previously dropped.

## Environment info

<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.2
- Platform: Linux-5.4.0-1074-azure-x86_64-with-glibc2.31
- Python version: 3.9.5
- PyArrow version: 8.0.0
- Pandas version: 1.4.2


Fix dependency on dill version

We had to make a hotfix by pinning dill:
- #4380

because from version 0.3.5, our custom `save_function` pickling function was raising an exception:
- #4379

This PR fixes this by implementing our custom `save_function` depending on the version of dill.

CC: @anivegesana

This PR needs first being merged:
- [x] #4384
  - so that a circular import is fixed

It is also convenient to merge first:
- [x] #4385


Fix URL in gem dataset for totto config

As commented in:
- https://github.com/huggingface/datasets/issues/4386#issuecomment-1134902372

CC: @StevenTang1998


Add Pascal VOC dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4395). All of your documentation changes will be reflected on that endpoint.

Some CI fails are unrelated to your PR and fixed on master, feel free to merge master into your branch :)

This PR adds the Pascal VOC dataset in the same way TFDS has it added. I believe we can iterate on this dataset and in future versions include more data, such as segmentation masks, but for now I think it is a good idea to just add it the same way as TFDS to get a solid first version out there.


trainer became extremely slow after reload dataset by `load_from_disk`

Something isn't working

I tried to make the dataset much more smaller (100000 rows) , then the speed became `33.88it/s` from`8.62s/it`.  It's nearly 200 times... Do you have any idea? Thank you!

Similar issue: https://github.com/huggingface/transformers/issues/8818

I changed `RandomSampler` to  `SequentialSampler` in the `trainer.py`, but the speed didn't become faster.

I changed
```
tokenized_datasets = load_from_disk(
        "/pathto/dataset"
    )
```
to
```
tokenized_datasets = load_from_disk(
        "/pathto/dataset", keep_in_memory=True
    )
```
and obtained normal speed. It's seems that the problem is on the os's speed limit.

Hi ! Currently `save_to_disk` saves one big Arrow file, which causes some slow downs. This has been discussed in #3735 and we'll implement sharding pretty soon to solve this

For now you can try splitting and saving your dataset in several Arrow files. Then you can load them one by one and use `concatenate_datasets` to have your big dataset again and hopefully with a better speed

## Describe the bug
Due to memory problem, I need to save my tokenized datasets locally by CPU and reload it by multi GPU for running training script. However, after I reload it by `load_from_disk` and start training, the speed is extremely slow. It says I need about 1500 hours with 8 A100 cards. Before this, I can run the whole script in one day with a single A100 card.
Since I am try to pre-train a BERT, **my dataset is very large(29058165 rows)**

## Steps to reproduce the bug
```python
tokenized_datasets.save_to_disk(
        "/pathto/dataset"
    )
tokenized_datasets = load_from_disk(
        "/pathto/dataset"
    )
trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=tokenized_datasets["train"] if training_args.do_train else None,
        eval_dataset=tokenized_datasets["validation"]
        if training_args.do_eval
        else None,
        tokenizer=tokenizer,
        data_collator=data_collator,
    )
 train_result = trainer.train(resume_from_checkpoint=checkpoint)
```

## Expected results
Without the save and reload process, I only need about one day to run the whole script with one A100 card.

## Actual results
```
[INFO|trainer.py:1290] 2022-05-23 22:49:46,266 >> ***** Running training *****
[INFO|trainer.py:1291] 2022-05-23 22:49:46,266 >>   Num examples = 29058165
[INFO|trainer.py:1292] 2022-05-23 22:49:46,266 >>   Num Epochs = 5
[INFO|trainer.py:1293] 2022-05-23 22:49:46,266 >>   Instantaneous batch size per device = 16
[INFO|trainer.py:1294] 2022-05-23 22:49:46,266 >>   Total train batch size (w. parallel, distributed & accumulation) = 256
[INFO|trainer.py:1295] 2022-05-23 22:49:46,266 >>   Gradient Accumulation steps = 2
[INFO|trainer.py:1296] 2022-05-23 22:49:46,266 >>   Total optimization steps = 567540
  0%|          | 1/567540 [00:09<1544:49:04,  9.80s/it]
  0%|          | 2/567540 [00:17<1320:00:17,  8.37s/it]
  0%|          | 3/567540 [00:26<1393:10:17,  8.84s/it]
  0%|          | 4/567540 [00:34<1344:56:33,  8.53s/it]
  0%|          | 5/567540 [00:43<1359:36:12,  8.62s/it]
```

## Environment info
```
torch              1.11.0+cu113
torchaudio         0.11.0+cu113
torchvision        0.12.0+cu113
transformers       4.18.0
datasets           2.2.2
```


Update CI deprecated legacy image

_The documentation is not available anymore as the PR was closed or merged._

Now our CI still uses a deprecated legacy image:
> You’re using a [deprecated Docker convenience image.](https://discuss.circleci.com/t/legacy-convenience-image-deprecation/41034) Upgrade to a next-gen Docker convenience image.

This PR updates to next-generation convenience image.

Related to:
- #2955


remove int documentation from logging docs

Removes the `int` documentation from the [logging section](https://huggingface.co/docs/datasets/package_reference/logging_methods#levels) of the docs.


Refactor column mappings for question answering datasets

> Thanks.
>
> I have no visibility about this, but if you say it is more useful for AutoTrain this way...

Thanks for the review @albertvillanova ! Yes, I need some way to reconstruct the original column names with a period because that's how they appear after we flatten the nested columns. In any case, we can adjust this later if needed :)

> Does that mean that we need to change the metadata?

Yes, but this PR takes care of it :)

This PR tweaks the keys in the metadata that are used to define the column mapping for question answering datasets. This is needed in order to faithfully reconstruct column names like `answers.text` and `answers.answer_start` from the keys in AutoTrain.

As observed in https://github.com/huggingface/datasets/pull/4367 we cannot use periods `.` in the keys of the YAML tags, so a decision was made to use a flat mapping with underscores. For QA datasets, however, it's handy to be able to reconstruct the nesting -- hence this PR.

cc @sashavor


Fix metadata validation

Since Python 3.8, the typing module:
- raises an AttributeError when trying to access `__args__` on any type, e.g.: `List.__args__`
- provides the `get_args` function instead: `get_args(List)`

This PR implements a fix for Python >=3.8 whereas maintaining backward compatibility.


Fix bug in gem dataset for wiki_auto_asset_turk config

This PR fixes some URLs.

Fix #4386.


Set builder name from module instead of class

Now the builder name attribute is set from from the builder class name.

This PR sets the builder name attribute from the module name instead. Some motivating reasons:
- The dataset ID is relevant and unique among all datasets and this is directly related to the repository name, i.e., the name of the directory containing the dataset
- The name of the module (i.e. the file containing the loading loading script) is already relevant for loading: it must have the same name as its containing directory (related to the dataset ID), as we search for it using its directory name
- On the other hand, the name of the builder class is not relevant for loading: in our code, we just search for a class which is subclass of `DatasetBuilder` (independently of its name). We do not put any constraint on the naming of the builder class and indeed it can have a name completely different from its module/direcotry/dataset_id

IMO it makes more sense to align the caching directory name with the dataset_id/directory/module name instead of the builder class name.

Fix #4381.


device/google/accessory/adk2012 - Git at Google

"git clone https://android.googlesource.com/device/google/accessory/adk2012"
 https://android.googlesource.com/device/google/accessory/adk2012/#:~:text=git%20clone%20https%3A//android.googlesource.com/device/google/accessory/adk2012


Bug for wiki_auto_asset_turk from GEM

Something isn't working

Thanks for reporting, @StevenTang1998.

I'm looking into it.

Hi @StevenTang1998,

We have fixed the issue:
- #4389

The fix will be available in our next `datasets` library release. In the meantime, you can incorporate that fix by installing `datasets` from our GitHub repo:
```
pip install git+https://github.com/huggingface/datasets#egg=datasets
```

Thanks for your reply!!
And the totto dataset has the same problem. The url should be change to [https://storage.googleapis.com/totto-public/totto_data.zip](https://storage.googleapis.com/totto-public/totto_data.zip).

Hi again @StevenTang1998,

I don't see any problem when loading `totto` dataset:
```python
In [4]: import datasets
   ...: ds = datasets.load_dataset("totto")
Downloading builder script: 5.58kB [00:00, 5.33MB/s]
Downloading metadata: 2.78kB [00:00, 2.96MB/s]
Using custom data configuration default
Downloading and preparing dataset totto/default (download: 179.03 MiB, generated: 706.59 MiB, post-processed: Unknown size, total: 885.62 MiB) to .../.cache/huggingface/datasets/totto/default/1.0.0/263c85871e5451bc892c65ca0306c0629eb7beb161e0eb998f56231562335dd2...
Downloading data: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 188M/188M [00:32<00:00, 5.77MB/s]
Dataset totto downloaded and prepared to .../.cache/huggingface/datasets/totto/default/1.0.0/263c85871e5451bc892c65ca0306c0629eb7beb161e0eb998f56231562335dd2. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 147.95it/s]

In [5]: ds
Out[5]:
DatasetDict({
    train: Dataset({
        features: ['id', 'table_page_title', 'table_webpage_url', 'table_section_title', 'table_section_text', 'table', 'highlighted_cells', 'example_id', 'sentence_annotations', 'overlap_subset'],
        num_rows: 120761
    })
    validation: Dataset({
        features: ['id', 'table_page_title', 'table_webpage_url', 'table_section_title', 'table_section_text', 'table', 'highlighted_cells', 'example_id', 'sentence_annotations', 'overlap_subset'],
        num_rows: 7700
    })
    test: Dataset({
        features: ['id', 'table_page_title', 'table_webpage_url', 'table_section_title', 'table_section_text', 'table', 'highlighted_cells', 'example_id', 'sentence_annotations', 'overlap_subset'],
        num_rows: 7700
    })
})
```

Sorry, I didn't express it clearly. It's the totto dataset from gem.
datasets.load_dataset('gem', 'totto')

## Describe the bug
The script of wiki_auto_asset_turk for GEM may be out of date.

## Steps to reproduce the bug
```python
import datasets
datasets.load_dataset('gem', 'wiki_auto_asset_turk')
```

## Actual results
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/load.py", line 1731, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/builder.py", line 640, in download_and_prepare
    self._download_and_prepare(
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/builder.py", line 1158, in _download_and_prepare
    super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/builder.py", line 707, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/tangtianyi/.cache/huggingface/modules/datasets_modules/datasets/gem/982a54473b12c6a6e40d4356e025fb7172a5bb2065e655e2c1af51f2b3cf4ca1/gem.py", line 538, in _split_generators
    dl_dir = dl_manager.download_and_extract(_URLs[self.config.name])
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 416, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 294, in download
    downloaded_path_or_paths = map_nested(
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 351, in map_nested
    mapped = [
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 352, in <listcomp>
    _single_map_nested((function, obj, types, None, True, None))
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 288, in _single_map_nested
    return function(data_struct)
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 320, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 234, in cached_path
    output_path = get_from_cache(
  File "/home/tangtianyi/miniconda3/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 579, in get_from_cache
    raise FileNotFoundError(f"Couldn't find file at {url}")
FileNotFoundError: Couldn't find file at https://github.com/facebookresearch/asset/raw/master/dataset/asset.test.orig
```


Test dill

I should point out that the hash will be the same if computed twice with the same code on the same version of dill (after adding huggingface's code that removes line numbers and file names, and sorts globals.) My changes in dill 0.3.5 and ones that I will make in 0.3.6 will result in different pickles than the ones dill 0.3.4 was making. This should still be fine for caching.

Just some comments @lhoestq:

The best practice for testing is to have a `test_<filename>.py` for each `<filename>.py`. Therefore in order to have the filenames aligned, I would propose:
- either renaming `fingerprint.py` to `caching.py`
- or renaming `test_caching.py` to `test_fingerprint.py`

On the other hand, my idea when implementing this test was not to test all the functionalities of the `Hasher`, but just to have a regression test that fails if dill version is > 0.3.4 and the pin in our `setup.py` is not present. Just recall that we had no failing test in our CI when the issue with dill was found on `transformers`.

The objective of this PR is just to have a regression test for that case: I tested and I got `AttributeError: module 'dill._dill' has no attribute 'stack'`

For this regression test, I took into account this comment by @gugarosa: https://github.com/huggingface/datasets/issues/4379#issuecomment-1133131825

There is no equivalent test in `test_caching.py` because our CI did not fail before pinning dill.

Ok I see, renaming it to `test_fingerprint.py` sounds like a good idea :)

Regression test for future releases of `dill`.

Related to #4379.


Refactor download

This looks like a breaking change no ?
Also could you explain why it would be better this way ?

The might be only there to help type checkers, but I am not too familiar with the code base to know for sure. I think this might be useful:

https://docs.python.org/3/library/typing.html#typing.TYPE_CHECKING

> This looks like a breaking change no ?
> Also could you explain why it would be better this way ?

Sorry, @lhoestq, I naively thought it was obvious. I have tried to give some arguments in the motivation of this PR (see above). I can give additional arguments if needed.

This PR performs a refactoring of the download functionalities, by proposing a modular solution and moving them to their own package "download". Some motivating arguments:
- understandability: from a logical partitioning of the library, it makes sense to have all download functionalities grouped together instead of scattered in a much larger directory containing many more different functionalities
- abstraction: the level of abstraction of "download" (higher) is not the same as "utils" (lower); putting different levels of abstraction together, makes dependencies more intricate (potential circular dependencies) and the system more tightly coupled; when the levels of abstraction are clearly separated, the dependencies flow in a neat direction from higher to lower
- architectural: "download" is a domain-specific functionality of our library/application (a dataset builder performs several actions: download, generate dataset and cache it); these functionalities are at the core of our library; on the other hand, "utils" are always a low-level set of functionalities, not directly related to our domain/business core logic (all libraries have "utils"), thus at the periphery of our lib architecture

Also note that when a library is not architecturally designed following simple, neat, clean principles, this has a negative impact on extensibility, making more and more difficult to make enhancements.

As a concrete example in this case, please see: https://app.circleci.com/pipelines/github/huggingface/datasets/12185/workflows/ff25a790-8e3f-45a1-aadd-9d79dfb73c4d/jobs/72860
- After an extension, a circular import is found
- Diving into the cause of this circular import, see the dependency flow, which should be from higher to lower levels of abstraction:
```
ImportError while loading conftest '/home/circleci/datasets/tests/conftest.py'.
tests/conftest.py:12: in <module>
    import datasets
../.pyenv/versions/3.6.15/lib/python3.6/site-packages/datasets/__init__.py:37: in <module>
    from .arrow_dataset import Dataset, concatenate_datasets
../.pyenv/versions/3.6.15/lib/python3.6/site-packages/datasets/arrow_dataset.py:59: in <module>
    from . import config
../.pyenv/versions/3.6.15/lib/python3.6/site-packages/datasets/config.py:8: in <module>
    from .utils.logging import get_logger
../.pyenv/versions/3.6.15/lib/python3.6/site-packages/datasets/utils/__init__.py:30: in <module>
    from .download_manager import DownloadConfig, DownloadManager, DownloadMode
../.pyenv/versions/3.6.15/lib/python3.6/site-packages/datasets/utils/download_manager.py:39: in <module>
    from .py_utils import NestedDataStructure, map_nested, size_str
../.pyenv/versions/3.6.15/lib/python3.6/site-packages/datasets/utils/py_utils.py:608: in <module>
    if config.DILL_VERSION < version.parse("0.3.5"):
E   AttributeError: module 'datasets.config' has no attribute 'DILL_VERSION'
```

Imports:
- datasets
  - Dataset: lower level than datasets
    - config: lower level than Dataset
      - logger: lower level than config
        - DownloadManager: !!! HIGHER level of abstraction than logger!!

Why when importing logger we require importing DownloadManager?!?
- Logically, it does not make sense
- This is due to an error in the design/architecture of our library:
  - To import the logger, we need to import it from `.utils.logging`
  - To import `.utils.logging` we need to import `.utils`
  - The import of `.utils` require the import of all its submodules defined in `utils.__init__.py`, among them: `.utils.download_manager`!

When putting `logging` and `download_manager` both inside `utils`, in order to import `logging` we need to import `download_manager` first: this is a strong coupling between modules and moreover between modules at different levels of abstraction (to import a lower level module, we require to import a higher level module). Additionally, it is clear that is makes no sense that in order to import `logging` we require to import `download_manager` first.


L

Something isn't working

## Describe the   L
L
## Expected   L
A clear and concise lmll
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:


First time trying

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Bug in caching 2 datasets both with the same builder class name

bug

Something isn't working

Hi @NouamaneTazi, thanks for reporting.

Please note that both datasets are cached in the same directory because their loading builder classes have the same name: `class MTOP(datasets.GeneratorBasedBuilder)`.

You should name their builder classes differently, e.g.:
- `MtopDomain`
- `MtopIntent`

Hi @NouamaneTazi, please note that after our fix:
- #4388

we do not consider the class name anymore, but the name of the file where the loading builder class is implemented.

## Describe the bug
The two datasets `mteb/mtop_intent` and `mteb/mtop_domain `use both the same cache folder `.cache/huggingface/datasets/mteb___mtop`. So if you first load `mteb/mtop_intent` then datasets will not load `mteb/mtop_domain`.
If you delete this cache folder and flip the order how you load the two datasets , you will get the opposite datasets loaded (difference is here in terms of the label and label_text).

## Steps to reproduce the bug
```python
import datasets

dataset = datasets.load_dataset("mteb/mtop_intent", "en")
print(dataset['train'][0])
dataset = datasets.load_dataset("mteb/mtop_domain", "en")
print(dataset['train'][0])
```

## Expected results
```
Reusing dataset mtop (/home/nouamane/.cache/huggingface/datasets/mteb___mtop_intent/en/0.0.0/f930e32a294fed424f70263d8802390e350fff17862266e5fc156175c07d9c35)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 920.14it/s]
{'id': 3232343436343136, 'text': 'Has Angelika Kratzer video messaged me?', 'label': 1, 'label_text': 'GET_MESSAGE'}
Reusing dataset mtop (/home/nouamane/.cache/huggingface/datasets/mteb___mtop_domain/en/0.0.0/f930e32a294fed424f70263d8802390e350fff17862266e5fc156175c07d9c35)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1307.59it/s]
{'id': 3232343436343136, 'text': 'Has Angelika Kratzer video messaged me?', 'label': 0, 'label_text': 'messaging'}
```

## Actual results
```
Reusing dataset mtop (/home/nouamane/.cache/huggingface/datasets/mteb___mtop/en/0.0.0/f930e32a294fed424f70263d8802390e350fff17862266e5fc156175c07d9c35)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 920.14it/s]
{'id': 3232343436343136, 'text': 'Has Angelika Kratzer video messaged me?', 'label': 1, 'label_text': 'GET_MESSAGE'}
Reusing dataset mtop (/home/nouamane/.cache/huggingface/datasets/mteb___mtop/en/0.0.0/f930e32a294fed424f70263d8802390e350fff17862266e5fc156175c07d9c35)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1307.59it/s]
{'id': 3232343436343136, 'text': 'Has Angelika Kratzer video messaged me?', 'label': 1, 'label_text': 'GET_MESSAGE'}
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.1
- Platform: macOS-12.1-arm64-arm-64bit
- Python version: 3.9.12
- PyArrow version: 8.0.0
- Pandas version: 1.4.2


Pin dill

_The documentation is not available anymore as the PR was closed or merged._

Hotfix #4379.

CC: @sgugger


Latest dill release raises exception

Something isn't working

Just an additional insight, the latest dill (either 0.3.5 or 0.3.5.1) also broke the hashing/fingerprinting of any mapping function.

For example:
```
from datasets import load_dataset

d = load_dataset("rotten_tomatoes")
d.map(lambda x: x)
```

Returns the standard non-dillable error:
```
Parameter 'function'=<function <lambda> at 0x7fe7d18c9560> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly....
```

@albertvillanova ExamplesTests.test_run_speech_recognition_seq2seq is in which file?

Thanks a lot @gugarosa for the insight: we will incorporate it in our CI as regression testing for future dill releases.

Hi @anivegesana, that test is in `transformers` library:
- https://github.com/huggingface/transformers/blob/main/examples/pytorch/test_pytorch_examples.py#L449
- https://github.com/huggingface/transformers/blob/main/examples/pytorch/speech-recognition/run_speech_recognition_seq2seq.py

@albertvillanova

I did a deep dive into @gugarosa's problem and found the issue and it might be related to the one @sgugger discovered. In dill 0.3.5(.1), I created a new `save_function` that fixes a bug in dill that prevented the pickling of recursive inner functions. It was a more complete solution to the problem that `dill._dill.stack` tried to solve in the internal API of dill. Since `dill._dill.stack` was no longer needed, I removed it. Since datasets copies the `save_function` directly from the dill API, it stops working with the new dill version since `dill._dill.stack` is no longer present and the `save_function` has been updated with new code.

https://github.com/huggingface/datasets/blob/95193ae61e92aa537d0c65d37a1fd9d2393aae89/src/datasets/utils/py_utils.py#L607-L678

~If the dill version is below 0.3.5, you should keep this function. If it is after, you would need to update your copy of `save_function` to use the code I introduced, or manually add a `stack` variable to `dill._dill` if it doesn't exist. Fortunately, in any version of Python 3.7+, dictionaries are always in insertion order and dill no longer supports Python 3.6 or older. So, any globals dictionary saved by dill 0.3.5+ will be deterministic given that the version of dill is held constant and this save_function is unnecessary for newer versions of dill.~

Ah. I see what is happening. I guess a different copy of the function code is needed that sorts the global variables by name.

```py
if dill.__version__.split('.') < ['0', '3', '5']:
    # current save_function code inside here
else:
    # new save_function code inside here with the following line inserted after creating the globals
    globs = {k: globs[k] for k in sorted(globs.keys())}
```

Will look into the test case @sgugger pointed out after that and verify if this is causing the problem.

I am actually looking into rewriting the global variables code in uqfoundation/dill#466 and will keep this in mind and will try to create an easy way to modify the global variables in dill 0.3.6 (for example, sort them by key like datasets does).

Thanks a lot for your investigation @anivegesana.

Yes, we copied-pasted the old `save_function` function from `dill`, just adding a line to make deterministic the order of global variables `globs`.

However, this function has changed a lot from version 0.3.5, after your PR (thank you for the fix in recursiveness, indeed):
- uqfoundation/dill#443

We have to address this change.

If finally your PR to sort global variables is merged into dill 0.3.6, that will make our life easier, as the tweak will no longer be necessary. ;)

I have included a regression test so that we are sure future releases of dill do not break `datasets`:
- #4385

I should note that because Python 3.6 and older are now deprecated and Python 3.7 has insertion order dictionaries, the globals in dill will have a deterministic order, just not sorted. I would still keep it sorted like you have it to help with stability (for example, if someone reorders variables in a file, then sorting the globals would not invalidate the cache.)

It seems that the order is not quite deterministic in IPython. Huggingface datasets seems to do well in Jupyter regardless, so it is not a good idea to remove the sorting. uqfoundation/dill#19

## Describe the bug
As reported by @sgugger, latest dill release is breaking things with Datasets.

```
______________ ExamplesTests.test_run_speech_recognition_seq2seq _______________

self = <multiprocess.pool.ApplyResult object at 0x7fa5981a1cd0>, timeout = None

    def get(self, timeout=None):
        self.wait(timeout)
        if not self.ready():
            raise TimeoutError
        if self._success:
            return self._value
        else:
>           raise self._value
E           TypeError: '>' not supported between instances of 'NoneType' and 'float'
```


Tidy up license metadata for google_wellformed_query, newspop, sick

Amend three licenses on datasets to fit naming convention (lower case, cc licenses include sub-version number). I think that's it - everything else on datasets looks great & super-searchable now!


Fix checksum and bug in irc_disentangle dataset

There was a bug in filepath segment:
- wrong: `jkkummerfeld-irc-disentanglement-fd379e9`
- right: `jkkummerfeld-irc-disentanglement-35f0a40`

Also there was a bug in the checksum of the downloaded file.

This PR fixes these issues.

Fix partially #4376.


irc_disentagle viewer error

DUPLICATED comment from https://github.com/huggingface/datasets/issues/3807:

my code:
```
from datasets import load_dataset

dataset = load_dataset("irc_disentangle", download_mode="force_redownload")
```
however, it produces the same error
```
[38](file:///Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/datasets/utils/info_utils.py?line=37) if len(bad_urls) > 0:
     [39](file:///Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/datasets/utils/info_utils.py?line=38)     error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> [40](file:///Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/datasets/utils/info_utils.py?line=39)     raise NonMatchingChecksumError(error_msg + str(bad_urls))
     [41](file:///Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/datasets/utils/info_utils.py?line=40) logger.info("All the checksums matched successfully" + for_verification_name)

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://github.com/jkkummerfeld/irc-disentanglement/tarball/master']
```
I attempted to use the `ignore_verifications' as such:

```
ds = datasets.load_dataset('irc_disentangle', download_mode="force_redownload", ignore_verifications=True)

Downloading builder script: 12.0kB [00:00, 5.92MB/s]
Downloading metadata: 7.58kB [00:00, 3.48MB/s]
No config specified, defaulting to: irc_disentangle/ubuntu
Downloading and preparing dataset irc_disentangle/ubuntu (download: 112.98 MiB, generated: 60.05 MiB, post-processed: Unknown size, total: 173.03 MiB) to /Users/laylabouzoubaa/.cache/huggingface/datasets/irc_disentangle/ubuntu/1.0.0/0f24ab262a21d8c1d989fa53ed20caa928f5880be26c162bfbc02445dbade7e5...
Downloading data: 118MB [00:09, 12.1MB/s]

Dataset irc_disentangle downloaded and prepared to /Users/laylabouzoubaa/.cache/huggingface/datasets/irc_disentangle/ubuntu/1.0.0/0f24ab262a21d8c1d989fa53ed20caa928f5880be26c162bfbc02445dbade7e5. Subsequent calls will reuse this data.
100%|██████████| 3/3 [00:00<00:00, 675.38it/s]
```
but, this returns an empty set?
```
DatasetDict({
    train: Dataset({
        features: ['id', 'raw', 'ascii', 'tokenized', 'date', 'connections'],
        num_rows: 0
    })
    test: Dataset({
        features: ['id', 'raw', 'ascii', 'tokenized', 'date', 'connections'],
        num_rows: 0
    })
    validation: Dataset({
        features: ['id', 'raw', 'ascii', 'tokenized', 'date', 'connections'],
        num_rows: 0
    })
})
```
not sure what else to try at this point?
Thanks in advanced🤗

Thanks for reporting, @labouz. I'm addressing it.

The issue with checksum and empty dataset has been fixed by:
- #4377

To load the dataset, you should force the re-generation of the dataset from the downloaded file by passing `download_mode="reuse_cache_if_exists"` to `load_dataset`.

In relation with the issue with the dataset viewer, first the dataset should be refactored to support streaming.

the dataviewer shows this message for "ubuntu" - "train", "test", and "validation" splits:
```
Server error
Status code:   400
Exception:     ValueError
Message:       Cannot seek streaming HTTP file

```
it appears to give the same message for the "channel_two" data as well.

I get a Checksums error when using `load_data()` with this dataset. Even with the `download_mode` and `ignore_verifications` options set. i referenced the issue here: https://github.com/huggingface/datasets/issues/3807


Support DataLoader with num_workers > 0 in streaming mode

Alright this is finally ready for review ! It's quite long I'm sorry, but it's not easy to disentangle everything ^^'

The main additions are in
- src/datasets/formatting/dataset_wrappers/torch_iterable_dataset.py
- src/datasets/iterable_dataset.py
- src/datasets/utils/patching.py

Added some comments and an error when lists have different lengths for sharding :)

Let's resolve the merge conflict and the CI error (if it's related to the changes), and I can review the PR again.

Feel free to review again :) The CI fail is unrelated to this PR and will be fixed by https://github.com/huggingface/datasets/pull/4472 (the hub now returns 401 instead of 404 for unauthenticated requests to non-existing repos)

CI failures are unrelated to this PR - merging :)

(CI fails are a mix of pip install fails and Hub fails)

### Issue

It's currently not possible to properly stream a dataset using multiple `torch.utils.data.DataLoader` workers:

- the `TorchIterableDataset` can't be pickled and passed to the subprocesses: https://github.com/huggingface/datasets/issues/3950
- streaming extension is failing: https://github.com/huggingface/datasets/issues/3951
- `fsspec` doesn't work out of the box in subprocesses

### Solution in this PR

I fixed these to enable passing an `IterableDataset` to a  `torch.utils.data.DataLoader` with `num_workers > 0`.

I also had to shard the `IterableDataset` to give each worker a shard, otherwise data would be duplicated. This is implemented in `TorchIterableDataset.__iter__` and uses the new `IterableDataset._iter_shard(shard_idx)` method

I also had to do a few changes the patching that enable streaming in dataset scripts:
- the patches are now always applied - not just for streaming mode. They're applied when a builder is instantiated
- I improved it to also check for renamed modules or attributes (ex: pandas vs pd)
- I grouped all the patches of pathlib.Path into a class `xPath`, so that `Path` outside of dataset scripts stay unchanged - otherwise I didn't change the content of the extended Path methods for streaming
- I fixed a bug with the `pd.read_csv` patch, opening the file in "rb" mode was missing and causing some datasets to not work in streaming mode, and compression inference was missing

### A few details regarding `fsspec` in multiprocessing

From https://github.com/fsspec/filesystem_spec/pull/963#issuecomment-1131709948 :
> Non-async instances might be safe in the forked child, if they hold no open files/sockets etc.; I'm not sure any implementations pass this test!
> If any async instance has been created, the newly forked processes must:
> 1. discard references to locks, threads and event loops and make new ones
> 2. not use any async fsspec instances from the parent process
> 3. clear all class instance caches

Therefore in a DataLoader's worker, I clear the reference to the loop and thread (1). We should be fine for 2 and 3 already since we don't use fsspec class instances from the parent process.

Fix https://github.com/huggingface/datasets/issues/3950
Fix https://github.com/huggingface/datasets/issues/3951

TODO:
- [x] fix tests


extremely slow processing when using a custom dataset

Something isn't working

question

Further information is requested

Hi !

My guess is that some examples in your dataset are bigger than your RAM, and therefore loading them in RAM to pass them to `remove_non_indic_sentences` takes forever because it might use SWAP memory.

Maybe several examples in your dataset are grouped together, can you check `len(lang_dataset["train"])` and `lang_dataset["train"].data.nbytes` of both datasets please ? It can also be helpful to check the distribution of lengths of each examples in your dataset.

## processing a custom dataset loaded as .txt file  is extremely slow,  compared to a dataset of similar volume from the hub

I have a large .txt file of 22 GB which i load into HF dataset

`lang_dataset = datasets.load_dataset("text", data_files="hi.txt")`

further i use a pre-processing function to clean the dataset

 `lang_dataset["train"] = lang_dataset["train"].map(
            remove_non_indic_sentences, num_proc=12, batched=True,  remove_columns=lang_dataset['train'].column_names), batch_size=64)`

the following processing takes astronomical time to process, while hoging all the ram.

similar dataset of same size that's available in the huggingface hub works completely fine. which runs the same processing function and has the same amount of data.
`lang_dataset = datasets.load_dataset("oscar-corpus/OSCAR-2109", "hi",  use_auth_token=True)`

the hours predicted to preprocess are as follows:

huggingface hub dataset:  6.5 hrs
custom loaded dataset: 7000 hrs

note: both the datasets are almost actually same, just provided by different sources with has +/- some samples, only one is hosted on the HF hub and the other is downloaded in a text format.

## Steps to reproduce the bug
```
import datasets
import psutil
import sys
import glob
from fastcore.utils import listify
import re
import gc

def remove_non_indic_sentences(example):
    tmp_ls = []
    eng_regex = r'[. a-zA-Z0-9ÖÄÅöäå _.,!"\'\/$]*'
    for e in listify(example['text']):
        matches = re.findall(eng_regex, e)
        for match in (str(match).strip() for match in matches if match not in [""," ", "  ", ",", " ,", ", ", " , "]):
            if len(list(match.split(" "))) > 2:
                e = re.sub(match," ",e,count=1)
        tmp_ls.append(e)
        gc.collect()
    example['clean_text'] = tmp_ls
    return example

lang_dataset = datasets.load_dataset("text", data_files="hi.txt")

lang_dataset["train"] = lang_dataset["train"].map(
            remove_non_indic_sentences, num_proc=12, batched=True,  remove_columns=lang_dataset['train'].column_names), batch_size=64)

## same thing work much faster when loading similar dataset from hub

lang_dataset = datasets.load_dataset("oscar-corpus/OSCAR-2109", "hi", split="train", use_auth_token=True)

lang_dataset["train"] = lang_dataset["train"].map(
            remove_non_indic_sentences, num_proc=12, batched=True,  remove_columns=lang_dataset['train'].column_names), batch_size=64)

```

## Actual results

similar dataset of same size that's available in the huggingface hub works completely fine. which runs the same processing function and has the same amount of data.
`lang_dataset = datasets.load_dataset("oscar-corpus/OSCAR-2109", "hi", use_auth_token=True)

**the hours predicted to preprocess are as follows:**
huggingface hub dataset:  6.5 hrs
custom loaded dataset: 7000 hrs

**i even tried the following:**

-  sharding the large 22gb text files into smaller files and loading
- saving the file to disk and then loading
- using lesser num_proc
- using smaller batch size
- processing without batches ie :  without `batched=True`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.2.dev0
- Platform: Ubuntu 20.04 LTS
- Python version: 3.9.7
- PyArrow version:8.0.0


Remove links in docs to old dataset viewer

_The documentation is not available anymore as the PR was closed or merged._

Remove the links in the docs to the no longer maintained dataset viewer.


Check if dataset features match before push in `DatasetDict.push_to_hub`

Fix #4211


Add missing language tags for udhr dataset

Related to #4362.


Add redirect to dataset script in the repo structure page

Following https://github.com/huggingface/hub-docs/pull/146 I added a redirection to the dataset scripts documentation in the repository structure page.


Add long answer candidates to natural questions dataset

Once we have added `long_answer_candidates` maybe it would be worth to also add the missing `candidate_index` (inside `long_answer`). What do you think, @seirasto ?

Also note the "Data Fields" section in the README is missing the `long_answer` field.

Moreover, there is no instance example in "Data Instances" section.

We could either make these fixes in this PR or in a subsequent PR.

@albertvillanova I've added the missing fields and updated the README to include a data instance and some other things.

Great! I've made the updates to align the README. Please let me know if I missed anything.

As there were many minor little fixes, I thought it would be easier to fix them directly.

I think the loading script is OK now. If it is also validated by another datasets maintainer, I could run the generation of the pre-processed data and then merge this PR into master (once all the tests are green).

CC: @lhoestq

I have merged the master branch, so that we include all the fixes on Apache Beam + Google Dataflow.

Pre-processing is running!

Already finished for "dev" config:
```python
In [2]: ds = load_dataset("datasets/natural_questions", "dev")

In [3]: ds
Out[3]:
DatasetDict({
    validation: Dataset({
        features: ['id', 'document', 'question', 'long_answer_candidates', 'annotations'],
        num_rows: 7830
    })
})
```

There is an issue while running the preprocessing for the "default" (train+dev) config. Train data files are larger than than dev ones and workers run out of memory.

I'm opening a separate issue to handle this problem: #4525

@seirasto is proposing uploading their preprocessed data files to our Datasets bucket.

I think @lhoestq can give a more informed answer about authentication requirements.

Now that the data fiels are uploaded, can you merge the `main` branch into yours to re-trigger the CI @seirasto please ? :) Then I think we can merge if it's good for you @albertvillanova

Merge is done! I think someone needs to approve the CI to run :)

Can you run `make style` to fix the code formatting required by the CI please ?

Thanks @albertvillanova! I've committed all your suggestions.

This is a modification of the Natural Questions dataset to include missing information specifically related to long answer candidates. (See here: https://github.com/google-research-datasets/natural-questions#long-answer-candidates). This information is important to ensure consistent comparison with prior work. It does not disturb the rest of the format . @lhoestq @albertvillanova


Remove config names as yaml keys

I included the change from https://github.com/huggingface/datasets/pull/4302 directly in this PR, this way the datasets will be updated right away in the CI (the CI is only triggered when a dataset card is changed)

_The documentation is not available anymore as the PR was closed or merged._

Alright it's ready now :)

Here is an example for the `ade_corpus_v2` dataset card. Notice the new `configs` key:

https://github.com/huggingface/datasets/blob/76d9a141740a03f6836feb251f6059894b8d8046/datasets/ade_corpus_v2/README.md#L1-L78

CI failures are only related to dataset cards missing some content.

Many datasets have dots in their config names. However it causes issues with the YAML tags of the dataset cards since we can't have dots in YAML keys.

I fix this, I removed the tags separations per config name completely, and have a single flat YAML for all configurations. Dataset search doesn't use this info anyway. I removed all the config names used as YAML keys, and I moved them in under a new `config:` key.

This is related to https://github.com/huggingface/datasets/pull/2362 (internal https://github.com/huggingface/moon-landing/issues/946).

Also removing the dots in the YAML keys would allow us to do as in https://github.com/huggingface/datasets/pull/4302 which removes a hack that replaces all the dots by underscores in the YAML tags.

I also added a test in the CI that checks that all the YAML tags to make sure that:

- they can be parsed using a YAML parser
- they contain only valid YAML tags like languages or task_ids


duplicate

This issue or pull request already exists

Duplicate of:
- #3956

I think you should report that issue to `elasticsearch` library: https://github.com/elastic/elasticsearch-py

  "name" : "node-1",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "",
  "version" : {
    "number" : "7.5.0",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "",
    "build_date" : "2019-11-26T01:06:52.518245Z",
    "build_snapshot" : false,
    "lucene_version" : "8.3.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"

when I run the order:
nohup python3 custom_service.pyc > service.log 2>&1&

the log:
nohup: 忽略输入
Traceback (most recent call last):
  File "/home/xfz/p3_custom_test/custom_service.py", line 55, in <module>
  File "/home/xfz/p3_custom_test/custom_service.py", line 48, in doInitialize
  File "custom_impl.py", line 286, in custom_setup
  File "custom_impl.py", line 127, in create_es_index
  File "/usr/local/lib/python3.7/site-packages/elasticsearch/_sync/client/__init__.py", line 345, in __init__
    ssl_show_warn=ssl_show_warn,
  File "/usr/local/lib/python3.7/site-packages/elasticsearch/_sync/client/utils.py", line 105, in client_node_configs
    node_configs = hosts_to_node_configs(hosts)
  File "/usr/local/lib/python3.7/site-packages/elasticsearch/_sync/client/utils.py", line 154, in hosts_to_node_configs
    node_configs.append(host_mapping_to_node_config(host))
  File "/usr/local/lib/python3.7/site-packages/elasticsearch/_sync/client/utils.py", line 221, in host_mapping_to_node_config
    return NodeConfig(**options)  # type: ignore
TypeError: __init__() missing 1 required positional argument: 'scheme'
[1]+  退出 1                nohup python3 custom_service.pyc > service.log 2>&1

custom_service_pyc can't running


Remove dots in config names

Closing in favor of https://github.com/huggingface/datasets/pull/4367

20+ datasets have dots in their config names. However it causes issues with the YAML tags of the dataset cards since we can't have dots in YAML keys.

This is related to https://github.com/huggingface/datasets/pull/2362 (internal https://github.com/huggingface/moon-landing/issues/946).

Also removing the dots in the config names would allow us to merge https://github.com/huggingface/datasets/pull/4302 which removes a hack that replaces all the dots by underscores in the YAML tags.

I also added a test in the CI that checks that all the YAML tags to make sure that:
- they can be parsed using a YAML parser
- they contain only valid YAML tags like `languages` or `task_ids`
- they contain valid config names (no invalid characters `<>:/\|?*.`)


Support complex feature types as `features` in packaged loaders

This PR adds `table_cast` to the packaged loaders to fix casting to the `Image`/`Audio`, `ArrayND` and `ClassLabel` types. If these types are not present in the `builder.config.features` dictionary, the built-in `pa.Table.cast` is used for better performance. Additionally, this PR adds `cast_storage` to `ClassLabel` to support the string to int conversion in `table_cast` and ensure that integer labels are in a valid range.

Fix https://github.com/huggingface/datasets/issues/4210

This PR is also a solution for these (popular) discussions: https://discuss.huggingface.co/t/converting-string-label-to-int/2816 and https://discuss.huggingface.co/t/class-labels-for-custom-datasets/15130/2

TODO:
* [x] tests


The dataset preview is not available for this split.

Hi!  A dataset has to be streamable to work with the viewer. I did a quick test, and yours is, so this might be a bug in the viewer. cc @severo

Looking at it. The message is now:

```
Message:       cannot cache function '__shear_dense': no locator available for file '/src/services/worker/.venv/lib/python3.9/site-packages/librosa/util/utils.py'
```

so possibly it's related to the libraries versions?

Maybe this SO thread can help: https://stackoverflow.com/questions/59290386/runtimeerror-at-cannot-cache-function-shear-dense-no-locator-available-fo

Same error for https://huggingface.co/datasets/LIUM/tedlium/viewer/release1/test. cc @sanchit-gandhi . I'm on it

Fixed in the datasets viewer, by setting the `NUMBA_CACHE_DIR` env var to a writable directory.

https://huggingface.co/datasets/Roh/ryanspeech/viewer/male/train

<img width="1538" alt="Capture d’écran 2022-06-08 à 11 30 08" src="https://user-images.githubusercontent.com/1676121/172583285-4cd49a0f-5715-423b-95dd-5f6ace3b2416.png">

https://huggingface.co/datasets/LIUM/tedlium/viewer/

<img width="1538" alt="Capture d’écran 2022-06-08 à 14 31 52" src="https://user-images.githubusercontent.com/1676121/172616897-fbcb7df7-0308-4d09-a17d-48826bc91374.png">

I have uploaded the corpus developed by our lab in the speech domain to huggingface [datasets](https://huggingface.co/datasets/Roh/ryanspeech). You can read about the companion paper accepted in interspeech 2021 [here](https://arxiv.org/abs/2106.08468). The dataset works fine but I can't make the dataset preview work. It gives me the following error that I don't understand. Can you help me to begin debugging it?

```
Status code:   400
Exception:     AttributeError
Message:       'NoneType' object has no attribute 'split'
```