head_qa file missing

bug

Something isn't working

We usually find issues with files hosted at Google Drive...

In this case we download the Google Drive Virus scan warning instead of the data file.

Fixed: https://huggingface.co/datasets/head_qa/viewer/en/train. Thanks

<img width="1551" alt="Capture d’écran 2022-02-28 à 15 29 04" src="https://user-images.githubusercontent.com/1676121/156000224-fd3f62c6-8b54-4df1-8911-bdcb0bac3f1a.png">

## Describe the bug

A file for the `head_qa` dataset is missing (https://drive.google.com/u/0/uc?export=download&id=1a_95N5zQQoUCq8IBNVZgziHbeM-QxG2t/HEAD_EN/train_HEAD_EN.json)

## Steps to reproduce the bug
```python
>>> from datasets import load_dataset
>>> load_dataset("head_qa", name="en")
```

## Expected results

The dataset should be loaded

## Actual results

```
Downloading and preparing dataset head_qa/en (download: 75.69 MiB, generated: 2.69 MiB, post-processed: Unknown size, total: 78.38 MiB) to /home/slesage/.cache/huggingface/datasets/head_qa/en/1.1.0/583ab408e8baf54aab378c93715fadc4d8aa51b393e27c3484a877e2ac0278e9...
Downloading data: 2.21kB [00:00, 2.05MB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/load.py", line 1729, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/builder.py", line 594, in download_and_prepare
    self._download_and_prepare(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/builder.py", line 665, in _download_and_prepare
    verify_checksums(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/u/0/uc?export=download&id=1a_95N5zQQoUCq8IBNVZgziHbeM-QxG2t']
```

## Environment info
- `datasets` version: 1.18.4.dev0
- Platform: Linux-5.11.0-1028-aws-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 6.0.1


Add perplexity to metrics

Awesome thank you ! The implementation of the parent `Metric` class was assuming that all metrics were supposed to have references/predictions pairs - I just changed that so you don't have to override `compute()`. I took the liberty of doing the changes directly inside this PR to make sure it works as expected with perplexity.

Other than that it looks in pretty good shape :) I just did minor changes like remove a remaining `print` as well as fixing the `Features` defined in `_info()`. I also renamed `input_text` to `input_texts` since it makes it more obvious that it's a list of strings - let me know if it sounds good to you.

Let me know if you'd like to make other changes or if it's all good for you !

The test with the full test set seems to take too much time in the CI - maybe we can just select `split="test[:few_examples]"` (around 10 maybe ?)

Adding perplexity metric

This code differs from the code in [this](https://huggingface.co/docs/transformers/perplexity) HF blog post because the blogpost code fails in at least the following circumstances:
- returns nans whenever the stride = 1
- hits a runtime error when the stride is significantly larger than the max model length (e.g. if max_model_length = 512 and stride = 1024)

Note that:
- As it is, it only works for causal models. Pseudoperplexity can be added later as another metric to work with masked language models.
- It takes in a list of strings so that it can be dataset independent. This does mean that it doesn't currently batch inputs, and is therefore relatively slow.
- It overwrites the metrics compute() function for a specific perplexity compute() function. This is because the current general metrics compute() function requires model-generated predictions, which doesn't make sense in the context of perplexity


Images get decoded when using `map()` with `input_columns` argument on a dataset

Something isn't working

Hi! If I'm not mistaken, this behavior is intentional, but I agree it could be more intuitive.

@albertvillanova Do you remember why you decided not to decode columns in the `Audio` feature PR when `input_columns` is not `None`? IMO we should decode those columns, and we don't even have to use lazy structures here because the user explicitly requires them in the map transform.

cc @lhoestq for visibility

I think I excluded to decorate the function when `input_columns` were passed as a quick fix for some non-passing tests:
- https://github.com/huggingface/datasets/pull/2324/commits/9d7c3e8fa53e23ec636859b4407eeec904b1b3f9

That PR was quite complex and I decided to focus on the main feature requests, leaving refinements for subsequent PRs.

Note that when `input_columns` are passed, the signature of the function is effectively changed, while the decorated function expects an item (whether an example or a batch) as first arg (which is not the case when passing `input_columns`.

I agree we should consider supporting the case when `input_columns` are passed.

## Describe the bug

The `datasets.features.Image` feature class decodes image data by default. Expectedly, when indexing a dataset or using the `map()` method, images are returned as PIL Image instances.

However, when calling `map()` and setting a specific data column with the `input_columns` argument, the image data is passed as raw byte representation to the mapping function.

## Steps to reproduce the bug
```python
from datasets import load_dataset
from torchvision import transforms
from PIL.Image import Image

dataset = load_dataset('mnist', split='train')

def transform_all_columns(example):
    # example['image'] is encoded as PIL Image
    assert isinstance(example['image'], Image)
    return example

def transform_image_column(image):
    # image is decoded here and represented as raw bytes
    assert isinstance(image, Image)
    return image

# single-sample dataset for debugging purposes
dev = dataset.select([0])

dev.map(transform_all_columns)
dev.map(transform_image_column, input_columns='image')
```

## Expected results

Image data should be passed in decoded form, i.e. as PIL Image objects to the mapping function unless the `decode` attribute on the image feature is set to `False`.

## Actual results

The mapping function receives images as raw byte data.

## Environment info
- `datasets` version: 1.18.3
- Platform: Linux-5.11.0-49-generic-x86_64-with-glibc2.32
- Python version: 3.8.0b4
- PyArrow version: 7.0.0


Cannot preview dataset

dataset-viewer

Related to the dataset viewer on huggingface.co

Thanks for reporting. The dataset viewer depends on some backend treatments, and for now, they might take some hours to get processed. We're working on improving it.

It has finally been processed. Thanks for the patience.

## Dataset viewer issue for '*rubrix/news*'

**Link:https://huggingface.co/datasets/rubrix/news** *link to the dataset viewer page*

Cannot see the dataset preview:

```
Status code:   400
Exception:     Status400Error
Message:       Not found. Cache is waiting to be refreshed.
```
Am I the one who added this dataset ? No


Overflowing indices in `select`

bug

Something isn't working

Fixed on master (see https://github.com/huggingface/datasets/pull/3719).

Awesome, I did not find that one! Thanks.

## Describe the bug
The `Dataset.select` function seems to accept indices that are larger than the dataset size and seems to effectively use `index %len(ds)`.

## Steps to reproduce the bug
```python
from datasets import Dataset

ds = Dataset.from_dict({"test": [1,2,3]})
ds = ds.select(range(5))

print(ds)
print()
print(ds["test"])
```
Result:
```python
Dataset({
    features: ['test'],
    num_rows: 5
})

[1, 2, 3, 1, 2]
```

This behaviour is not documented and can lead to unexpected behaviour when for example taking a sample larger than the dataset and thus creating a lot of duplicates.

## Expected results
It think this should throw an error or at least a very big warning:
```python
IndexError: Invalid key: 5 is out of bounds for size 3
```

## Environment info
- `datasets` version: 1.18.3
- Platform: macOS-12.0.1-x86_64-i386-64bit
- Python version: 3.9.10
- PyArrow version: 7.0.0


Expanding streaming capabilities

enhancement

New feature or request

Cool ! `filter` will be very useful. There can be a filter that you can apply on a streaming dataset:
```python
load_dataset(..., streaming=True).filter(lambda x: x["lang"] == "sw")
```

Otherwise if you want to apply a filter on the source files that are going to be used for streaming, the logic has to be impIemented directly in the dataset script, or if there's no dataset script this can be done with pattern matching
```python
load_dataset(..., lang="sw")                    # if the dataset script supports this parameter
load_dataset(..., data_files="data/lang=sw/*")  # if there's no dataset script, but only data files
```

--------------

Here are also some additional ideas of API to convert from iterable to map-style dataset:
```python
on_disk_dataset = streaming_dataset.to_disk()
on_disk_dataset = streaming_dataset.to_disk(path="path/to/my/dataset/dir")

in_memory_dataset = streaming_dataset.take(100).to_memory()  # to experiment without having to write files
```
--------------

Finally regarding `push_to_hub`, we can replace `batch_size` by `shard_size` (same API as for on-disk datasets). The default is 500MB per file

Let me know what you think !

Regarding conversion, I'd also ask for some kind of equivalent to `save_to_disk` for an `IterableDataset`.

Similarly to the streaming to hub idea, my use case would be to define a sequence of dataset transforms via `.map()`, using an `IterableDataset` as the input (so processing could start without doing whole download up-front), but streaming the resultant processed dataset just to disk.

That makes sense @athewsey , thanks for the suggestion :)

Maybe instead of the `to_disk` we could simply have `save_to_disk` instead:
```python
streaming_dataset.save_to_disk("path/to/my/dataset/dir")
on_disk_dataset = load_from_disk("path/to/my/dataset/dir")

in_memory_dataset = Dataset.from_list(list(streaming_dataset.take(100)))  # to experiment without having to write files
```

So far are implemented: `IterableDataset.filter()` and `Dataset.to_iterable_dataset()`.

Still missing: `IterableDataset.push_to_hub()` - though there is a hack to write on disk and then push to hub using

```python
ds_on_disk = Dataset.from_generator(streaming_ds.__iter__)  # stream to disk
ds_on_disk.push_to_hub(...)
```

Some ideas for a few features that could be useful when working with large datasets in streaming mode.

## `filter` for `IterableDataset`
Adding filtering to streaming datasets would be useful in several scenarios:
- filter a dataset with many languages for a subset of languages
- filter a dataset for specific licenses
- other custom logic to get a subset
The only way to achieve this at the moment is I think through writing a custom loading script and implementing filters there.

## `IterableDataset` to `Dataset` conversion
In combination with the above filter a functionality to "play" the whole stream would be useful. The motivation is that often one might filter the dataset to get a manageable size for experimentation. In that case streaming mode is no longer necessary as the filtered dataset is small enough and it would be useful to be able to play through the whole stream to create a normal `Dataset` with all its benefits.

```python
ds = load_dataset("some_large_dataset", streaming=True)
ds_filter = ds.filter(lambda x: x["lang"]="fr")
ds_filter = ds_filter.stream() # here the `IterableDataset` is converted to a `Dataset`
```
Naturally, this could be expanded with `stream(n=1000)` which creates a `Dataset` with the first `n` elements similar to `take`.

## Stream to the Hub
While streaming allows to use a dataset as is without saving the whole dataset on the local machine it is currently not possible to process a dataset and add it to the hub. The only way to do this is by downloading the full dataset and saving the processed dataset again before pushing them to the hub. The API could looks something like:

```python
ds = load_dataset("some_large_dataset", streaming=True)
ds_filter = ds.filter(some_filter_func)
ds_processed = ds_filter.map(some_processing_func)
ds_processed.push_to_hub("new_better_dataset", batch_size=100_000)
```

Under the hood this could be done by processing and aggregating `batch_size` elements and then pushing that batch as a single file to the hub. With this functionality one could process and create TB scale datasets while only requiring size of `batch_size` local disk space.

cc @lhoestq @albertvillanova


Update metadata JSON for cats_vs_dogs dataset

Note that the number of examples in the train split was already fixed in the dataset card.

Fix #3750.


Fix typo in train split name

In the README guide (and consequently in many datasets) there was a typo in the train split name:
```
| Tain   | Valid | Test |
```

This PR:
- fixes the typo in the train split name
- fixes the column alignment of the split tables

in the README guide and in all datasets.


`NonMatchingSplitsSizesError` for cats_vs_dogs dataset

bug

Something isn't working

Thnaks for reporting @jaketae. We are fixing it.

## Describe the bug

Cannot download cats_vs_dogs dataset due to `NonMatchingSplitsSizesError`.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("cats_vs_dogs")
```

## Expected results

Loading is successful.

## Actual results

```
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=7503250, num_examples=23422, dataset_name='cats_vs_dogs'), 'recorded': SplitInfo(name='train', num_bytes=7262410, num_examples=23410, dataset_name='cats_vs_dogs')}]
```

## Environment info

Reproduced on a fresh [Colab notebook](https://colab.research.google.com/drive/13GTvrSJbBGvL2ybDdXCBZwATd6FOkMub?usp=sharing).

## Additional Context

Originally reported in https://github.com/huggingface/transformers/issues/15698.

cc @mariosasko


Add tqdm arguments

Hi ! Thanks this will be very useful :)

It looks like there are some changes in the github diff that are not related to your contribution, can you try fixing this by merging `master` into your PR, or create a new PR from an updated version of `master` ?

I have already solved the conflict on this latest version. This is my first time sending PR, if there's anything I need to adjust just let me know~

Thanks, most changes are gone :)
It still seems to include changes though - do you mind try creating a new branch from upstream/master and create a new PR please ?

Yeah sure, I'll try to send a new PR today!

Please forward to [#3850](https://github.com/huggingface/datasets/pull/3850)

Thanks ! Closing this one in favor of https://github.com/huggingface/datasets/pull/3850/files

In this PR, tqdm arguments can be passed to the map() function and such, in order to be more flexible.


Add tqdm arguments

In this PR, there are two changes.

1. It is able to show the progress bar by adding the length of the iterator.
2. Pass in tqdm_kwargs so that can enable more feasibility for the control of tqdm library.


Passing invalid subset should throw an error

Something isn't working

## Describe the bug
Only some datasets have a subset (as in `load_dataset(name, subset)`). If you pass an invalid subset, an error should be thrown.

## Steps to reproduce the bug
```python
import datasets
datasets.load_dataset('rotten_tomatoes', 'asdfasdfa')
```

## Expected results
This should break, since `'asdfasdfa'` isn't a subset of the `rotten_tomatoes` dataset.

## Actual results
This API call silently succeeds.


Use the same seed to shuffle shards and metadata in streaming mode

When shuffling in streaming mode, those two entangled lists are shuffled independently. In this PR I changed this to shuffle the lists of same length with the exact same seed, in order for the files and metadata to still be aligned.
```python
gen_kwargs = {
    "files": [os.path.join(data_dir, filename) for filename in all_files],
    "metadata_files": [all_metadata[filename] for filename in all_files],
}
```
IMO this is important to avoid big but silent issues.

Fix https://github.com/huggingface/datasets/issues/3744


Add mIoU metric

Hmm the doctest failed again - maybe the full result needs to be on one single line

Cool ! Feel free to merge if it's all good for you

This PR adds the mean Intersection-over-Union metric to the library, useful for tasks like semantic segmentation.

It is entirely based on mmseg's [implementation](https://github.com/open-mmlab/mmsegmentation/blob/master/mmseg/core/evaluation/metrics.py).

I've removed any PyTorch dependency, and rely on Numpy only.


Better shards shuffling in streaming mode

enhancement

New feature or request

Sometimes a dataset script has a `_split_generators` that returns several files as well as the corresponding metadata of each file. It often happens that they end up in two separate lists in the `gen_kwargs`:
```python
gen_kwargs = {
    "files": [os.path.join(data_dir, filename) for filename in all_files],
    "metadata_files": [all_metadata[filename] for filename in all_files],
}
```
It happened for Multilingual Spoken Words for example in #3666

However currently **the two lists are shuffled independently** when shuffling the shards in streaming mode. This leads to `_generate_examples` not having the right metadata for each file.

To prevent this issue I suggest that we always shuffle lists of the same length the exact same way to avoid such a big but silent issue.

cc @polinaeterna


initial monash time series forecasting repository

_The documentation is not available anymore as the PR was closed or merged._

The CI fails are unrelated to this PR, merging !


Fix ValueError message formatting in int2str

Hi!

I bumped into this particular `ValueError` during my work (because an instance of `np.int64` was passed instead of regular Python `int`), and so I had to `print(type(values))` myself. Apparently, it's just the missing `f` to make message an f-string.

It ain't much for a contribution, but it's honest work. Hope it spares someone else a few seconds in the future 😃


Rm sphinx doc

Checklist
- [x] Update circle ci yaml
- [x] Delete sphinx static & python files in docs dir
- [x] Update readme in docs dir
- [ ] Update docs config in setup.py


Support streaming for pubmed

@albertvillanova just FYI, since you were so helpful with the previous pubmed issue :)

IIRC streaming from FTP is not fully tested yet, so I'm fine with switching to HTTPS for now, as long as the download speed/availability is great

@albertvillanova Thanks for pointing me to the `ET` module replacement. It should look a lot cleaner now.

Unfortunately I tried keeping the `ftp://` protocol but was seeing timeout errors? in streaming mode (below). I think the `https://` performance is not an issue, when I was profiling the `open(..) -> f.read() -> etree.fromstring(xml_str)` codepath, most of the time was spent in the XML parsing rather than the data download.

Error when using `ftp://`:

```
Traceback (most recent call last):
  File "/Users/abhinav/Documents/mosaicml/hf_datasets/venv/lib/python3.8/site-packages/fsspec/implementations/ftp.py", line 301, in _fetch_range
    self.fs.ftp.retrbinary(
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ftplib.py", line 430, in retrbinary
    callback(data)
  File "/Users/abhinav/Documents/mosaicml/hf_datasets/venv/lib/python3.8/site-packages/fsspec/implementations/ftp.py", line 293, in callback
    raise TransferDone
fsspec.implementations.ftp.TransferDone

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test_pubmed_streaming.py", line 9, in <module>
    print (next(iter(pubmed_train_streaming)))
  File "/Users/abhinav/Documents/mosaicml/hf_datasets/abhi-datasets/src/datasets/iterable_dataset.py", line 365, in __iter__
    for key, example in self._iter():
  File "/Users/abhinav/Documents/mosaicml/hf_datasets/abhi-datasets/src/datasets/iterable_dataset.py", line 362, in _iter
    yield from ex_iterable
  File "/Users/abhinav/Documents/mosaicml/hf_datasets/abhi-datasets/src/datasets/iterable_dataset.py", line 79, in __iter__
    yield from self.generate_examples_fn(**self.kwargs)
  File "/Users/abhinav/.cache/huggingface/modules/datasets_modules/datasets/pubmed/af552ed918e2841e8427203530e3cfed3a8bc3213041d7853bea1ca67eec683d/pubmed.py", line 362, in _generate_examples
    tree = ET.parse(filename)
  File "/Users/abhinav/Documents/mosaicml/hf_datasets/abhi-datasets/src/datasets/streaming.py", line 65, in wrapper
    return function(*args, use_auth_token=use_auth_token, **kwargs)
  File "/Users/abhinav/Documents/mosaicml/hf_datasets/abhi-datasets/src/datasets/utils/streaming_download_manager.py", line 636, in xet_parse
    return ET.parse(f, parser=parser)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/xml/etree/ElementTree.py", line 1202, in parse
    tree.parse(source, parser)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/xml/etree/ElementTree.py", line 595, in parse
    self._root = parser._parse_whole(source)
  File "/Users/abhinav/Documents/mosaicml/hf_datasets/abhi-datasets/src/datasets/utils/streaming_download_manager.py", line 293, in read_with_retries
    out = read(*args, **kwargs)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/gzip.py", line 292, in read
    return self._buffer.read(size)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/_compression.py", line 68, in readinto
    data = self.read(len(byte_view))
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/gzip.py", line 479, in read
    if not self._read_gzip_header():
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/gzip.py", line 422, in _read_gzip_header
    magic = self._fp.read(2)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/gzip.py", line 96, in read
    self.file.read(size-self._length+read)
  File "/Users/abhinav/Documents/mosaicml/hf_datasets/venv/lib/python3.8/site-packages/fsspec/spec.py", line 1485, in read
    out = self.cache._fetch(self.loc, self.loc + length)
  File "/Users/abhinav/Documents/mosaicml/hf_datasets/venv/lib/python3.8/site-packages/fsspec/caching.py", line 153, in _fetch
    self.cache = self.fetcher(start, end)  # new block replaces old
  File "/Users/abhinav/Documents/mosaicml/hf_datasets/venv/lib/python3.8/site-packages/fsspec/implementations/ftp.py", line 311, in _fetch_range
    self.fs.ftp.getmultiline()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ftplib.py", line 224, in getmultiline
    line = self.getline()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ftplib.py", line 206, in getline
    line = self.file.readline(self.maxline + 1)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out
```

This PR makes some minor changes to the `pubmed` dataset to allow for `streaming=True`. Fixes #3739.

Basically, I followed the C4 dataset which works in streaming mode as an example, and made the following changes:

* Change URL prefix from `ftp://` to `https://`
* Explicilty `open` the filename and pass the XML contents to `etree.fromstring(xml_str)`

The Github diff tool makes it look like the changes are larger than they are, sorry about that.

I tested locally and the `pubmed` dataset now works in both normal and streaming modes. There is some overhead at the start of each shard in streaming mode as building the XML tree online is quite slow (each pubmed .xml.gz file is ~20MB), but the overhead gets amortized over all the samples in the shard. On my laptop with a single CPU worker I am able to stream at about ~600 samples/s.


Pubmed dataset does not work in streaming mode

bug

Something isn't working

Thanks for reporting, @abhi-mosaic (related to #3655).

Please note that `xml.etree.ElementTree.parse` already supports streaming:
- #3476

No need to refactor to use `open`/`xopen`. Is is enough with importing the package `as ET` (instead of `as etree`).

## Describe the bug
Trying to use the `pubmed` dataset with `streaming=True` fails.

## Steps to reproduce the bug
```python
import datasets
pubmed_train = datasets.load_dataset('pubmed', split='train', streaming=True)
print (next(iter(pubmed_train)))
```

## Expected results
I would expect to see the first training sample from the pubmed dataset.

## Actual results
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/abhinav/Documents/mosaicml/mosaicml_venv/lib/python3.8/site-packages/datasets/iterable_dataset.py", line 367, in __iter__
    for key, example in self._iter():
  File "/Users/abhinav/Documents/mosaicml/mosaicml_venv/lib/python3.8/site-packages/datasets/iterable_dataset.py", line 364, in _iter
    yield from ex_iterable
  File "/Users/abhinav/Documents/mosaicml/mosaicml_venv/lib/python3.8/site-packages/datasets/iterable_dataset.py", line 79, in __iter__
    for key, example in self.generate_examples_fn(**self.kwargs):
  File "/Users/abhinav/.cache/huggingface/modules/datasets_modules/datasets/pubmed/9715addf10c42a7877a2149ae0c5f2fddabefc775cd1bd9b03ac3f012b86ce46/pubmed.py", line 373, in _generate_examples
    tree = etree.parse(filename)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/xml/etree/ElementTree.py", line 1202, in parse
    tree.parse(source, parser)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/xml/etree/ElementTree.py", line 584, in parse
    source = open(source, "rb")
FileNotFoundError: [Errno 2] No such file or directory: 'gzip://pubmed21n0001.xml::ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed21n0001.xml.gz'
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.2
- Platform: macOS-11.4-x86_64-i386-64bit
- Python version: 3.8.2
- PyArrow version: 6.0.0

## Comments
The error looks like an issue with `open` vs. `xopen` inside the `xml` package. It looks like it's trying to open the remote source URL, which has been edited with prefix `gzip://...`.

Maybe there can be an explicit `xopen` before passing the raw data to `etree`, something like:

```python
# Before
tree = etree.parse(filename)
root = tree.getroot()

# After
with xopen(filename) as f:
  data_str = f.read()
root = etree.fromstring(data_str)
```


For data-only datasets, streaming and non-streaming don't behave the same

Something isn't working

Note that we might change the heuristic and create a different config per file, at least in that case.

Hi @severo, thanks for reporting.

Yes, this happens because when non-streaming, a cast of all data is done in order to "concatenate" it all into a single dataset (thus the error), while this casting is not done while yielding item by item in streaming mode.

Maybe in streaming mode we should keep the schema (inferred from the first item) and throw an exception if a subsequent item does not conform to the inferred schema?

Why do we want to concatenate the files? Is it the expected behavior for most datasets that lack a script and dataset info?

These files are two different dataset configurations since they don't share the same schema.

IMO the streaming mode should fail in this case, as @albertvillanova said.

There is one challenge though: inferring the schema from the first example is not robust enough in the general case - especially if some fields are nullable. I guess we can at least make sure that no new columns are added

OK. So, if we make the streaming also fail, the dataset https://huggingface.co/datasets/huggingface/transformers-metadata will never be [viewable](https://github.com/huggingface/datasets-preview-backend/issues/144) (be it using streaming or fallback to downloading the files), right?

Yes, until we have a way for the user to specify explicitly that those two files are different configurations.

We can maybe have some rule to detect this automatically, maybe checking the first line of each file ? That would mean that for dataset of 10,000+ files we would have to verify every single one of them just to know if there is one ore more configurations, so I'm not sure if this is a good idea

i think requiring the user to specify that those two files are different configurations is in that case perfectly reasonable.

(Maybe at some point we could however detect this type of case and prompt them to define a config mapping etc)

OK, so, before closing the issue, what do you think should be done?

> Maybe in streaming mode we should keep the schema (inferred from the first item) and throw an exception if a subsequent item does not conform to the inferred schema?

or nothing?

We should at least raise an error if a new sample has column names that are missing, or if it has extra columns. No need to check for the type for now.

I'm in favor of having an error especially because we want to avoid silent issues as much as possible - i.e. when something goes wrong (when schemas don't match or some data are missing) and no errors/warnings are raised.

Consistency between streaming and non-streaming is also important.

See https://huggingface.co/datasets/huggingface/transformers-metadata: it only contains two JSON files.

In streaming mode, the files are concatenated, and thus the rows might be dictionaries with different keys:

```python
import datasets as ds
iterable_dataset = ds.load_dataset("huggingface/transformers-metadata", split="train", streaming=True);
rows = list(iterable_dataset.take(100))
rows[0]
# {'model_type': 'albert', 'pytorch': True, 'tensorflow': True, 'flax': True, 'processor': 'AutoTokenizer'}
rows[99]
# {'model_class': 'BartModel', 'pipeline_tag': 'feature-extraction', 'auto_class': 'AutoModel'}
```

In normal mode, an exception is thrown:

```python
import datasets as ds
dataset = ds.load_dataset("huggingface/transformers-metadata", split="train");
```

```
ValueError: Couldn't cast
model_class: string
pipeline_tag: string
auto_class: string
to
{'model_type': Value(dtype='string', id=None), 'pytorch': Value(dtype='bool', id=None), 'tensorflow': Value(dtype='bool', id=None), 'flax': Value(dtype='bool', id=None), 'processor': Value(dtype='string', id=None)}
because column names don't match
```


Make RedCaps streamable

Make RedCaps streamable.

@lhoestq Using `data/redcaps_v1.0_annotations.zip` as a download URL gives an error locally when running `datasets-cli test` (will investigate this another time)


Local paths in common voice

I just changed to `dl_manager.is_streaming` rather than an additional parameter `streaming` that has to be handled by the DatasetBuilder class - this way the streaming logic doesn't interfere with the base builder's code.

I think it's better this way, but let me know if you preferred the previous way and I can revert

> But on the other hand, IMHO, I think this specific solution adds complexity to handling streaming/non-streaming, and moves this complexity to the loading script and thus to the contributors/users who want to create the loading script for their canonical/community datasets (instead of keeping it hidden form the end users).

I'm down to discuss this more in the future !

@lhoestq good idea: much cleaner this way! That way each class has its own responsibilities without mixing around...

Continuation of https://github.com/huggingface/datasets/pull/3664:

- pass the `streaming` parameter to _split_generator
- update @anton-l's code to use this parameter for `common_voice`
- add a comment to explain why we use `download_and_extract` in non-streaming and `iter_archive` in streaming

Now the `common_voice` dataset has a local path back in `ds["path"]`, and this field is `None` in streaming mode.

cc @patrickvonplaten @anton-l @albertvillanova

Fix #3663.


Performance of `datasets` at scale

> using command line git-lfs - [...] 300MB/s!

which server location did you upload from?

The most surprising part to me is the saving time. Wondering if it could be due to compression (`ParquetWriter` uses SNAPPY  compression by default; it can be turned off with `to_parquet(..., compression=None)`).

+1 to what @mariosasko mentioned. Also, @lvwerra did you parallelize `to_parquet` using similar approach in #2747? (we used multiprocessing at the shard level). I'm working on a similar PR to add multi_proc in `to_parquet` which might give you further speed up.
Stas benchmarked his approach and mine in this [gist](https://gist.github.com/stas00/dc1597a1e245c5915cfeefa0eee6902c) for `lama` dataset when we were working on adding multi_proc support for `to_json`.

@mariosasko I did not turn it off but I can try the next time - I have to run the pipeline again, anyway.

@bhavitvyamalik Yes, I also sharded the dataset and used multiprocessing to save each shard. I'll have a closer look at your approach, too.

# Performance of `datasets` at 1TB scale

## What is this?
During the processing of a large dataset I monitored the performance of the `datasets` library to see if there are any bottlenecks. The insights of this analysis could guide the decision making to improve the performance of the library.

## Dataset
The dataset is a 1.1TB extract from GitHub with 120M code files and is stored as 5000 `.json.gz` files. The goal of the preprocessing is to remove duplicates and filter files based on their stats. While the calculating of the hashes for deduplication and stats for filtering can be parallelized the filtering itself is run with a single process. After processing the files are pushed to the hub.

## Machine
The experiment was run on a `m1` machine on GCP with 96 CPU cores and 1.3TB RAM.

## Performance breakdown
- Loading the data **3.5h** (_30sec_ from cache)
    - **1h57min** single core loading (not sure what is going on here, corresponds to second progress bar)
    - **1h10min** multi core json reading
    - **20min** remaining time before and after the two main processes mentioned above
- Process the data **2h** (_20min_ from cache)
    - **20min** Getting reading for processing
    - **40min** Hashing and files stats (96 workers)
    - **58min** Deduplication filtering (single worker)
- Save parquet files **5h**
    -  Saving 1000 parquet files (16 workers)
- Push to hub **37min**
    - **34min** git add
    - **3min** git push (several hours with `Repository.git_push()`)

## Conclusion
It appears that loading and saving the data is the main bottleneck at that scale (**8.5h**) whereas processing (**2h**) and pushing the data to the hub (**0.5h**) is relatively fast. To optimize the performance at this scale it would make sense to consider such an end-to-end example and target the bottlenecks which seem to be loading from and saving to disk. The processing itself seems to run relatively fast.

## Notes
- map operation on a 1TB dataset with 96 workers requires >1TB RAM
- map operation does not maintain 100% CPU utilization with 96 workers
- sometimes when the script crashes all the data files have a corresponding `*.lock` file in the data folder (or multiple e.g. `*.lock.lock` when it happened a several times). This causes the cache **not** to be triggered (which is significant at that scale) - i guess because there are new data files
- parallelizing `to_parquet` decreased the saving time from 17h to 5h, however adding more workers at this point had almost no effect. not sure if this is:
    a) a bug in my parallelization logic,
    b) i/o limit to load data form disk to memory or
    c) i/o limit to write from memory to disk.
- Using `Repository.git_push()` was much slower than using command line `git-lfs` - 10-20MB/s vs. 300MB/s! The `Dataset.push_to_hub()` function is even slower as it only uploads one file at a time with only a few MB/s, whereas `Repository.git_push()` pushes files in parallel (each at a similar speed).

cc @lhoestq @julien-c @LysandreJik @SBrandeis


Fix bugs in NewsQA dataset

Fix #3733.


Bugs in NewsQA dataset

Something isn't working

## Describe the bug
NewsQA dataset has the following bugs:
- the field `validated_answers` is an exact copy of the field `answers` but with the addition of `'count': [0]` to each dict
- the field `badQuestion` does not appear in `answers` nor `validated_answers`

## Steps to reproduce the bug
By inspecting the dataset script we can see that:
- the parsing of `validated_answers` is a copy-paste of the one for `answers`
- the `badQuestion` field is ignored in the parsing of both `answers` and `validated_answers`


Support streaming in size estimation function in `push_to_hub`

would this allow to include the size in the dataset info without downloading the files? related to https://github.com/huggingface/datasets/pull/3670

@severo I don't think so. We could use this to get `info.download_checksums["num_bytes"]`, but we must process the files to get the rest of the size info.

This PR adds the streamable version of `os.path.getsize` (`fsspec` can return `None`, so we fall back to `fs.open` to make it more robust) to account for possible streamable paths in the nested `extra_nbytes_visitor` function inside `push_to_hub`.


Fix Multi-News dataset metadata and card

Fix #3730.


Checksum Error when loading multi-news dataset

Something isn't working

Thanks for reporting @byw2.
We are fixing it.
In the meantime, you can load the dataset by passing `ignore_verifications=True`:
 ```python
dataset = load_dataset("multi_news", ignore_verifications=True)

## Describe the bug
When using the load_dataset function from datasets module to load the Multi-News dataset, does not load the dataset but throws Checksum Error instead.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("multi_news")
```

## Expected results
Should download and load Multi-News dataset.

## Actual results
Throws the following error and cannot load data successfully:

```
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1vRY2wM6rlOZrf9exGTm5pXj5ExlVwJ0C']
```

Could this issue please be looked at? Thanks!


Wrong number of examples when loading a text dataset

Something isn't working

Hi @kg-nlp, thanks for reporting.

That is weird... I guess we would need some sample data file where this behavior appears to reproduce the bug for further investigation...

ok, I found the reason why that two results are not same.
there is /u2029 in the text, the datasets will split sentence according to the /u2029,but when I use open function will not do that .
so I want to know which function shell do that
thanks

## Describe the bug
when I use load_dataset to read a txt file  I find  that the number of the samples is incorrect

## Steps to reproduce the bug
```
fr = open('train.txt','r',encoding='utf-8').readlines()
print(len(fr))  # 1199637

datasets = load_dataset('text', data_files={'train': ['train.txt']}, streaming=False)
print(len(datasets['train']))  # 1199649
```
I also use command line operation to verify it
```
$ wc -l train.txt
1199637 train.txt
```

## Expected results
please fix that issue

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.3
- Platform:windows&linux
- Python version:3.7
- PyArrow version:6.0.1


VoxPopuli

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** VoxPopuli
- **Description:** A Large-Scale Multilingual Speech Corpus
- **Paper:** https://arxiv.org/pdf/2101.00390.pdf
- **Data:** https://github.com/facebookresearch/voxpopuli
- **Motivation:** one of the largest (if not the largest) multilingual speech corpus: 400K hours of multilingual unlabeled speech + 17k hours of labeled speech

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

👀 @kahne @Molugan


Patch all module attributes in its namespace

When patching module attributes, only those defined in its `__all__` variable were considered by default (only falling back to `__dict__` if `__all__` was None).

However those are only a subset of all the module attributes in its namespace (`__dict__` variable).

This PR fixes the problem of modules that have non-None `__all__` variable, but try to access an attribute present in `__dict__` (and not in `__all__`).

For example, `pandas` has attribute `__version__` only present in `__dict__`.
- Before version 1.4, pandas `__all__` was None, thus all attributes in `__dict__` were patched
- From version 1.4, pandas `__all__` is not None, thus attributes in `__dict__` not present in `__all__` are ignored

Fix #3724.

CC: @severo @lvwerra


Use config pandas version in CSV dataset builder

Fix #3724.


Pin pandas to avoid bug in streaming mode

Temporarily pin pandas version to avoid bug in streaming mode (patching no longer works).

Related to #3724.


Bug while streaming CSV dataset with pandas 1.4

bug

Something isn't working

## Describe the bug
If we upgrade to pandas `1.4`, the patching of the pandas module is no longer working
```
AttributeError: '_PatchedModuleObj' object has no attribute '__version__'
```

## Steps to reproduce the bug
```
pip install pandas==1.4
```
```python
from datasets import load_dataset
ds = load_dataset("lvwerra/red-wine", split="train", streaming=True)
item = next(iter(ds))
item
```


Fix flatten of complex feature types

Apparently the merge brought back some tests that use `flatten_()` that we removed recently

_The documentation is not available anymore as the PR was closed or merged._

Fix `flatten` for the following feature types: Image/Audio, Translation, and  TranslationVariableLanguages.

Inspired by `cast`/`table_cast`, I've introduced a `table_flatten` function to handle the Image/Audio types.

CC: @SBrandeis

Fix #3686.


added electricity load diagram dataset

Initial Electricity Load Diagram time series dataset.


Multi-GPU support for `FaissIndex`

@albertvillanova Sorry for bothering you again, quick follow up: is there anything else you want me to add / modify?

Hi @rentruewang , we updated the documentation on `master`, could you merge `master` into your branch please ?

@lhoestq I've merge `huggingface/datasets/master` into this PR. Please review. Thanks! 🤗

Edit: Umm... I was experimenting with what renaming a branch would do to a pull request. Please ignore the `closed this PR` down below. 🤗

Per #3716 , current implementation does not take into consideration that `faiss` can run on multiple GPUs.

In this commit, I provided multi-GPU support for `FaissIndex` by modifying the device management in `IndexableMixin.add_faiss_index` and `FaissIndex.load`.

Now users are able to pass in
1. a positive integer (as usual) to use 1 GPU
2. a negative integer `-1` to use all GPUs
3. a list of integers e.g. `[0, 1]` to run only on those GPUs
4. Of course, passing in nothing still runs on CPU.

This closes: #3716


Builder Configuration Update Required on Common Voice Dataset

Something isn't working

Hi @aasem, thanks for reporting.

Please note that currently Commom Voice is hosted on our Hub as a community dataset by the Mozilla Foundation. See all Common Voice versions here: https://huggingface.co/mozilla-foundation

Maybe we should add an explaining note in our "legacy" Common Voice canonical script? What do you think @lhoestq @mariosasko ?

Thank you, @albertvillanova, for the quick response. I am not sure about the exact flow but I guess adding the following lines under the `_Languages` dictionary definition in [common_voice.py](https://github.com/huggingface/datasets/blob/master/datasets/common_voice/common_voice.py) might resolve the issue. I guess the dataset is recently made available so the file needs updating.

```
"ur": {
        "Language": "Urdu",
        "Date": "2022-01-19",
        "Size": "68 MB",
        "Version": "ur_3h_2022-01-19",
        "Validated_Hr_Total": 1,
        "Overall_Hr_Total": 3,
        "Number_Of_Voice": 48,
    },
```

@aasem for compliance reasons, we are no longer updating the `common_voice.py` script.

We agreed with Mozilla Foundation to use their community datasets instead, which will ask you to accept their terms of use:
```
You need to share your contact information to access this dataset.

This repository is publicly accessible, but you have to register to access its content — don't worry, it's just one click!

By clicking on “Access repository” below, you accept that your contact information (email address and username) can be shared with the repository authors. This will let the authors get in touch for instance if some parts of the repository's contents need to be taken down for licensing reasons.

By clicking on “Access repository” below, you also agree to not attempt to determine the identity of speakers in the Common Voice dataset.

You will immediately be granted access to the contents of the dataset.
```

In order to use e.g. their Common Voice dataset version 8.0, please:
- First visit their dataset page: https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0
- Accept their term of use by clicking "Access repository"
- You can then load their dataset with:
  ```python
  load_dataset("mozilla-foundation/common_voice_8_0", "ur", split="train+validation")
  ```

@albertvillanova
>Maybe we should add an explaining note in our "legacy" Common Voice canonical script?

Yes, I agree we should have a deprecation notice in the canonical script to redirect users to the new script.

@albertvillanova,
I now get the following error after downloading my access token from the huggingface and passing it to `load_dataset` call:

`AttributeError: 'DownloadManager' object has no attribute 'download_config'`

Any quick pointer on how it might be resolved?

@aasem What version of `datasets` are you using? We renamed that attribute from `_download_config` to `download_conig` fairly recently, so updating to the newest version should resolve the issue:
```
pip install -U datasets
```

Thanks a lot, @mariosasko. That completely resolved the issue.

Missing language in Common Voice dataset

**Link:** https://huggingface.co/datasets/common_voice

I tried to call the Urdu dataset using `load_dataset("common_voice", "ur", split="train+validation")` but couldn't due to builder configuration not found. I checked the source file here for the languages support:

https://github.com/huggingface/datasets/blob/master/datasets/common_voice/common_voice.py

and Urdu isn't included there. I assume a quick update will fix the issue as Urdu speech is now available at the Common Voice dataset.

Am I the one who added this dataset? No


Check if indices values in `Dataset.select` are within bounds

Fix #3707

Instead of reusing `_check_valid_index_key` from `datasets.formatting`, I defined a new function to provide a more meaningful error message.


Fix Evidence Infer Treatment dataset

This PR:
- fixes a bug in the script, by removing an unnamed column with the row index: fix KeyError
- fix the metadata JSON, by adding both configurations (1.1 and 2.0): fix ExpectedMoreDownloadedFiles
- updates the dataset card

Fix #3515.


wrong condition in `Features ClassLabel encode_example`

Something isn't working

Hi @Tudyx,

Please note that in Python, the boolean NOT operator (`not`) has lower precedence than comparison operators (`<=`, `<`), thus the expression you mention is equivalent to:
```python
 not (-1 <= example_data < self.num_classes)
```

Also note that as expected, the exception is raised if:
- `example_data < -1`
- or `example_data >= self.num_classes`

The raise of the exception is expected when `example_data` equals 4 and `self.num_classes` equals 4 too.

## Describe the bug

The `encode_example` function in *features.py* seems to have a wrong condition.

```python
if not -1 <= example_data < self.num_classes:
    raise ValueError(f"Class label {example_data:d} greater than configured num_classes {self.num_classes}")
```

## Expected results

The `not - 1` condition change the result of the condition. For instance, if  `example_data`  equals 4 and ` self.num_classes` equals 4 too, `example_data < self.num_classes` will give `False` as expected  . But if i add the  `not - 1`  condition, `not -1 <= example_data < self.num_classes` will give `True` and raise an exception.

## Environment info

- `datasets` version: 1.18.3
- Python version: 3.8.10
- PyArrow version: 7.00


`FaissIndex` to support multiple GPU and `custom_index`

enhancement

New feature or request

Hi @rentruewang, thansk for reporting and for your PR!!! We should definitely support this.

**Is your feature request related to a problem? Please describe.**
Currently, because `device` is of the type `int | None`, to leverage `faiss-gpu`'s multi-gpu support, you need to create a `custom_index`. However, if using a `custom_index` created by e.g. `faiss.index_cpu_to_all_gpus`, then `FaissIndex.save` does not work properly because it checks the device id (which is an int, so no multiple GPUs).

**Describe the solution you'd like**
I would like `FaissIndex` to support multiple GPUs, by passing in a list to `add_faiss_index`.

**Describe alternatives you've considered**
Alternatively, I would like it to at least provide a warning cause it wasn't the behavior that I expected.

**Additional context**
Relavent source code here:
 https://github.com/huggingface/datasets/blob/6ed6ac9448311930557810383d2cfd4fe6aae269/src/datasets/search.py#L340-L349

Device management needs changing to support multiple GPUs, probably by `isinstance` calls.

I can provide a PR if you like :)

Thanks for reading!


Fix bugs in msr_sqa dataset

dataset contribution

Contribution to a dataset script

It shows below when I run test:

FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_msr_sqa - ValueError: Unknown split "validation". Should be one of ['train', 'test'].

It make no sense for me😂.

@albertvillanova Does this PR has some additional fixes compared to https://github.com/huggingface/datasets/pull/3771 or we can close it?

@mariosasko besides the fix of the DuplicatedKeysError, this PR:
- changes the reading of one of the files: use pandas instead of splitting by comma
- changes the splits: modifying train and adding validation
- adds some extra logic in the processing of the data: adding a new field "question_and_history"

We should decide whether validating these additional changes.
- for example, if we accept as pertinent the addition of the field "question_and_history", this should be added as feature to the info, and the matadata should be regenerated...

Hi guys, anything we can do to fix that bug👀? @mariosasko @albertvillanova @lhoestq

_The documentation is not available anymore as the PR was closed or merged._

The last version has many problems,

1) Errors in table load-in. Split by a single comma instead of using pandas is wrong.
2) id reduplicated in _generate_examples function.
3) Missing information of history questions which make it hard to use.

I fix it refer to https://github.com/HKUNLP/UnifiedSKG. And we test it to perform normally.


tatoeba_mt: File not found error and key error

## Dataset viewer issue for 'tatoeba_mt'

**Link:** https://huggingface.co/datasets/Helsinki-NLP/tatoeba_mt

My data loader script does not seem to work.
The files are part of the local repository but cannot be found. An example where it should work is the subset for "afr-eng".

Another problem is that I do not have validation data for all subsets and I don't know how to properly check whether validation exists in the configuration before I try to download it. An example is the subset for "afr-deu".

Am I the one who added this dataset ? Yes


Rm sphinx doc

Thanks for pushing this :)
One minor comment regarding the PR itself - I noticed that some changes are coming from the upstream master, this might be due to a rebase. Would be nice if this PR doesn't include them for readabily, feel free to open a new one if necessary

Closing in favour https://github.com/huggingface/datasets/pull/3741

Checklist
- [x] Update circle ci yaml
- [x] Delete sphinx static & python files in docs dir
- [x] Update readme in docs dir
- [ ] Update docs config in setup.py


Fix the error of msr_sqa dataset

Fix the error of _load_table_data function in msr_sqa dataset, it is wrong to use comma to split each row.


Fix the error of _load_table_data function in msr_sqa dataset

The _load_table_data function from the last version is wrong, it is wrong to use comma to split each row.


Fix CI code quality issue

Fix CI code quality issue introduced by #3695.


Set base path to hub url for canonical datasets

If we agree to have data files in a dedicated directory "data/" then we should be fine. You're right we should not try to edit a dataset script from the repository directly, but from github, in order to avoid conflicts

This should allow canonical datasets to use relative paths to download data files from the Hub

cc @polinaeterna this will be useful if we have audio datasets that are canonical and for which you'd like to host data files


Loading JSON gets stuck with many workers/threads

bug

Something isn't working

Hi ! Note that it does `block_size *= 2` until `block_size > len(batch)`, so it doesn't loop indefinitely. What do you mean by "get stuck indefinitely" then ? Is this the actual call to `paj.read_json` that hangs ?

> increasing the `chunksize` argument decreases the chance of getting stuck

Could you share the values of chunksize that you're using to observe this ? And maybe the order of magnitude of number of bytes per line of JSON ?

To clarify, I don't think it loops indefinitely but the `paj.read_json` gets stuck after the first try. That's why I think it could be an issue with a lock somewhere.

Using `load_dataset(..., chunksize=40<<20)` worked without errors.

@lhoestq  I encountered another related issue. I use load_dataset() for my json data and set_transform() for preprocessing. But it hangs at the end of the epoch if `dataloader_num_workers>=1`.  It appears to be working fine with num_worker=0, but it's slow.
```
train_dataset = datasets.load_dataset("json",
                                      data_files=corpus_jsonl_path,
                                      keep_in_memory=False,
                                      cache_dir=model_args.cache_dir,
                                      streaming=False)
train_dataset.set_transform(psg_parse_fn)
```

I couldn't I think your problem is unrelated to this issue @memray
Indeed this issue discusses a bug when doing `load_dataset`, while your case has to do with the dataloader in a multiprocessing setup. Can you open a new issue and provide more details (share your env and what psg_parse_fn does) ?

I also encountered a similar issue when loading a 190GB dataset of jsonl files (255 files with less than 1Gb) where it got stuck for over 20h at tables generation (fig below), increasing the `chunksize` with `load_dataset(..., chunksize=40<<20)` fixed the issue

<img width="560" alt="image" src="https://user-images.githubusercontent.com/44069155/195605603-548a106e-7ad3-4269-8cdd-2ad3e975bf16.png">

> @lhoestq I encountered another related issue. I use load_dataset() for my json data and set_transform() for preprocessing. But it hangs at the end of the epoch if `dataloader_num_workers>=1`. It appears to be working fine with num_worker=0, but it's slow.
>
> ```
> train_dataset = datasets.load_dataset("json",
>                                       data_files=corpus_jsonl_path,
>                                       keep_in_memory=False,
>                                       cache_dir=model_args.cache_dir,
>                                       streaming=False)
> train_dataset.set_transform(psg_parse_fn)
> ```

In case people also get this problem, I found a way to fix it by adding `persistent_workers=True` when initializing DataLoader, like:
`train_loader = DataLoader(
            train_dataset,
            batch_size=self._train_batch_size,
            sampler=train_sampler,
            collate_fn=data_collator,
            num_workers=self.args.dataloader_num_workers,
            persistent_workers=True
        )`

The error was `CUDA error: initialization error Exception raised from insert_events at ../c10/cuda/CUDACachingAllocator.cpp:1266` after the 1st epoch, I guess it's because the data_loader worker is killed after each epoch and the data supply is cut off. This error only occurs when num_workers>1.

I can confirm the issue using datasets (2.12.0) with the following code and Accelerate (0.20.3) env:

````
trainDataloader = DataLoader(trainSplit, batch_size=args.train_batch_size, shuffle=True)
evalDataloader = DataLoader(validSplit, batch_size=args.valid_batch_size) // Here is where it gets stuck.
````
````
- `Accelerate` version: 0.20.3
- Platform: Linux-5.4.0-150-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Numpy version: 1.24.3
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- PyTorch XPU available: False
- System RAM: 503.28 GB
- GPU type: Tesla V100-SXM2-32GB
- `Accelerate` default config:
	- compute_environment: LOCAL_MACHINE
	- distributed_type: MULTI_GPU
	- mixed_precision: fp16
	- use_cpu: False
	- num_processes: 2
	- machine_rank: 0
	- num_machines: 1
	- gpu_ids: 0,1
	- rdzv_backend: static
	- same_network: True
	- main_training_function: main
	- downcast_bf16: no
	- tpu_use_cluster: False
	- tpu_use_sudo: False
	- tpu_env: []
````

Notable that with Accelerate configured for one GPU only, **it doesn't get stuck.**

The suggestion made by @memray worked in my case. This is how it was applied:
````
trainDataloader = DataLoader(trainSplit, batch_size=args.train_batch_size, shuffle=True, num_workers=2, persistent_workers=True)
evalDataloader = DataLoader(validSplit, batch_size=args.valid_batch_size, num_workers=2, persistent_workers=True)
````

I think your issue is related to `accelerate`, feel free to open an issue there: https://github.com/huggingface/accelerate/issues

`Dataset` objects generally work fine with the torch DataLoader, idk what `accelerate` does that could make it get stuck.

## Describe the bug
Loading a JSON dataset with `load_dataset` can get stuck when running on a machine with many CPUs. This is especially an issue when loading a large dataset on a large machine.

## Steps to reproduce the bug
I originally created the following script to reproduce the issue:
```python
from datasets import load_dataset
from multiprocessing import Process
from tqdm import tqdm
import datasets
from transformers import set_seed

def run_tasks_in_parallel(tasks, ds_list):
    for _ in tqdm(range(1000)):
        print('new batch')
        running_tasks = [Process(target=task, args=(ds, i)) for i, (task, ds) in enumerate(zip(tasks, ds_list))]
        for running_task in running_tasks:
            running_task.start()
        for running_task in running_tasks:
            running_task.join()

def get_dataset():
    dataset_name = 'transformersbook/codeparrot'
    ds = load_dataset(dataset_name+'-train', split="train", streaming=True)
    ds = ds.shuffle(buffer_size=1000, seed=1)
    return iter(ds)

def get_next_element(ds, process_id,  N=10000):
    for _ in range(N):
        _ = next(ds)['content']
    print(f'process {process_id} done')
    return

set_seed(1)
datasets.utils.logging.set_verbosity_debug()

n_processes = 8
tasks = [get_next_element for _ in range(n_processes)]
args = [get_dataset() for _ in range(n_processes)]
run_tasks_in_parallel(tasks, args)
```

Today I noticed that it can happen when running it on a single process on a machine with many cores without streaming. So just `load_dataset("transformersbook/codeparrot-train")` alone might cause the issue after waiting long enough or trying many times. It's a slightly random process which makes it especially hard to track down. When I encountered it today it had already processed 17GB of data (the size of the cache folder when it got stuck) before getting stuck.

Here's my current understanding of the error. As far as I can tell it happens in the following block: https://github.com/huggingface/datasets/blob/be701e9e89ab38022612c7263edc015bc7feaff9/src/datasets/packaged_modules/json/json.py#L119-L139

When the try on line 121 fails and the `block_size` is increased it can happen that it can't read the JSON again and gets stuck indefinitely. A hint that points in that direction is that increasing the `chunksize` argument decreases the chance of getting stuck and vice versa. Maybe it is an issue with a lock on the file that is not properly released.

## Expected results
Read a JSON before the end of the universe.

## Actual results
Read a JSON not before the end of the universe.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-4.19.0-18-cloud-amd64-x86_64-with-glibc2.28
- Python version: 3.9.10
- PyArrow version: 7.0.0

@lhoestq we dicsussed this a while ago. @albertvillanova we discussed this today :)


`.select`: unexpected behavior with `indices`

Something isn't working

Hi! Currently, we compute the final index as `index % len(dset)`. I agree this behavior is somewhat unexpected and that it would be more appropriate to raise an error instead (this is what `df.iloc` in Pandas does, for instance).

@albertvillanova @lhoestq wdyt?

I agree. I think `index % len(dset)` was used to support negative indices.

I think this needs to be fixed in `datasets.formatting.formatting._check_valid_index_key` if I'm not mistaken

## Describe the bug
The `.select` method will not throw when sending `indices` bigger than the dataset length; `indices` will be wrapped instead. This behavior is not documented anywhere, and is not intuitive.

## Steps to reproduce the bug
```python
from datasets import Dataset
ds = Dataset.from_dict({"text": ["d", "e", "f"], "label": [4, 5, 6]})
res1 = ds.select([1, 2, 3])['text']
res2 = ds.select([1000])['text']
```

## Expected results
Both results should throw an `Error`.

## Actual results
`res1` will give `['e', 'f', 'd']`
`res2` will give `['e']`

## Environment info
Bug found from this environment:
- `datasets` version: 1.16.1
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.7
- PyArrow version: 6.0.1

It was also replicated on `master`.


Unable to load dataset 'big_patent'

Something isn't working

Hi @ankitk2109,

Have you tried passing the split name with the keyword `split=`? See e.g. an example in our Quick Start docs: https://huggingface.co/docs/datasets/quickstart.html#load-the-dataset-and-model
```python
 ds = load_dataset("big_patent", "d", split="validation")

Hi @albertvillanova,

Thanks for your response.

Yes, I tried the `split='validation'` as well. But getting the same issue.

I'm sorry, but I can't reproduce your problem:
```python
In [5]: ds = load_dataset("big_patent", "d", split="validation")
Downloading and preparing dataset big_patent/d (download: 6.01 GiB, generated: 169.61 MiB, post-processed: Unknown size, total: 6.17 GiB) to .../.cache/big_patent/d/1.0.0/bdefa7c0b39fba8bba1c6331b70b738e30d63c8ad4567f983ce315a5fef6131c...
Downloading data: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6.45G/6.45G [27:36<00:00, 3.89MB/s]
Extracting data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [03:18<00:00, 66.08s/it]
Dataset big_patent downloaded and prepared to .../.cache/big_patent/d/1.0.0/bdefa7c0b39fba8bba1c6331b70b738e30d63c8ad4567f983ce315a5fef6131c. Subsequent calls will reuse this data.

In [6]: ds
Out[6]:
Dataset({
    features: ['description', 'abstract'],
    num_rows: 565
})

Maybe you had a connection issue while downloading the file and this was corrupted?
Our cache system uses the file you downloaded first time.
If so, you could try forcing redownload of the file with:
```python
ds = load_dataset("big_patent", "d", split="validation", download_mode="force_redownload")

I am able to download the dataset with ``` download_mode="force_redownload"```. As you mentioned it was an issue with the cached version which was failed earlier due to a network issue. I am closing the issue now, once again thank you.

## Describe the bug
Unable to load the "big_patent" dataset

## Steps to reproduce the bug
```python
load_dataset('big_patent', 'd', 'validation')
```

## Expected results
Download big_patents' validation split from the 'd' subset

## Getting an error saying:
{FileNotFoundError}Local file ..\huggingface\datasets\downloads\6159313604f4f2c01e7d1cac52139343b6c07f73f6de348d09be6213478455c5\bigPatentData\train.tar.gz doesn't exist

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:1.18.3
- Platform: Windows
- Python version:3.8
- PyArrow version:7.0.0


Raise informative error when loading a save_to_disk dataset

People recurrently report error when trying to load a dataset (using `load_dataset`) that was previously saved using `save_to_disk`.

This PR raises an informative error message telling them they should use `load_from_disk` instead.

Close #3700.


OSCAR-2109 datasets are misaligned and truncated

Something isn't working

Hi @adrianeboyd, thanks for reporting.

There is indeed a bug in that community dataset:
Line:
```python
metadata_and_text_files = list(zip(metadata_files, text_files))
```
should be replaced with
```python
metadata_and_text_files = list(zip(sorted(metadata_files), sorted(text_files)))
```

I am going to contact their owners (https://huggingface.co/oscar-corpus) in order to inform them about the bug.

I keep you informed.

That fix is part of it, but it's clearly not the only issue.

I also already contacted the OSCAR creators, but I reported it here because it looked like huggingface members were the main authors in the git history. Is there a better place to have reported this?

Hello,

We've had an issue that could be linked to this one here: https://github.com/oscar-corpus/corpus/issues/15.

I have been spot checking the source (`.txt`/`.jsonl`) files for a while, and have not found issues, especially in the start/end of corpora (but I conceed that more integration testing would be necessary on our side).

The text and metadata files are designed to be used in sync (with `lang_part_n.txt` and `lang_meta_part_n.jsonl` working together), while staying independent from part to part, so that anyone could randomly choose a part and work with it.

The fix @albertvillanova proposed should fix the problem, as the parts will be in sync again.

Let me know if you need help or more details, I'd be glad to help!

I'm happy to move the discussion to the other repo!

Merely sorting the files only **maybe** fixes the processing of the first part. If the first part contains non-unix newlines, it will still be misaligned/truncated, and all the following parts will be truncated with incorrect text offsets and metadata due the offset and newline bugs.

Hi @Uinelj, This is a total noobs question but how can I integrate that bugfix into my code? I reinstalled the datasets library this time from source. Should that have fixed the issue? I am still facing the misalignment issue. Do I need to download the dataset from scratch?

Hi, I re-downloaded the dataset and still have the problem. See: https://github.com/oscar-corpus/corpus/issues/18

Sorry @norakassner for the late reply.

There are indeed several issues creating the misalignment, as @adrianeboyd cleverly pointed out:
-  https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/commit/3cd7e95aa1799b73c5ea8afc3989635f3e19b86b fixed one of them
- but there are still others to be fixed

Normally, the issues should be fixed now:
- Fix offset initialization for each file: https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/commit/1ad9b7bfe00798a9258a923b887bb1c8d732b833
- Disable default universal newline support: https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/commit/0c2f307d3167f03632f502af361ac6c3c393f510

Feel free to reopen if you find additional misalignments/truncations.

CC: @adrianeboyd @norakassner @Uinelj

Thanks for the updates!

The purist in me would still like to have the rstrip not strip additional characters from the original text (unicode whitespace mainly in practice, I think), but the differences are extremely small in practice and it doesn't actually matter for my current task:

```python
text = "".join([text_f.readline() for _ in range(meta["nb_sentences"])]).rstrip("\n")
```

## Describe the bug

The `oscar-corpus/OSCAR-2109` data appears to be misaligned and truncated by the dataset builder for subsets that contain more than one part and for cases where the texts contain non-unix newlines.

## Steps to reproduce the bug

A few examples, although I'm not sure how deterministic the particular (mis)alignment is in various configurations:

```python
from datasets import load_dataset
dataset = load_dataset("oscar-corpus/OSCAR-2109", "deduplicated_fi", split="train", use_auth_token=True)
entry = dataset[0]
# entry["text"] is from fi_part_3.txt.gz
# entry["meta"] is from fi_meta_part_2.jsonl.gz

dataset = load_dataset("oscar-corpus/OSCAR-2109", "deduplicated_no", split="train", use_auth_token=True)
entry = dataset[900000]
# entry["text"] is from no_part_3.txt.gz and contains a blank line
# entry["meta"] is from no_meta_part_1.jsonl.gz

dataset = load_dataset("oscar-corpus/OSCAR-2109", "deduplicated_mk", split="train", streaming=True, use_auth_token=True)
# 9088 texts in the dataset are empty
```

For `deduplicated_fi`, all exported raw texts from the dataset are 17GB rather than 20GB as reported in the data splits overview table. The token count with `wc -w` for the raw texts is 2,067,556,874 rather than the expected 2,357,264,196 from the data splits table.

For `deduplicated_no` all exported raw texts contain 624,040,887 rather than the expected 776,354,517 tokens.

For `deduplicated_mk` it is 122,236,936 rather than 134,544,934 tokens.

I'm not expecting the `wc -w` counts to line up exactly with the data splits table, but for comparison the `wc -w` count for `deduplicated_mk` on the raw texts is 134,545,424.

## Issues

* The meta / text files are not paired correctly when loading, so the extracted texts do not have the right offsets, the metadata is not associated with the correct text, and the text files may not be processed to the end or may be processed beyond the end (empty texts).
* The line count offset is not reset per file so the texts aren't aligned to the right offsets in any parts beyond the first part, leading to truncation when in effect blank lines are not skipped.
* Non-unix newline characters are treated as newlines when reading the text files while the metadata only counts unix newlines for its line offsets, leading to further misalignments between the metadata and the extracted texts, and which also results in truncation.

## Expected results

All texts from the OSCAR release are extracted according to the metadata and aligned with the correct metadata.

## Fixes

Not necessarily the exact fixes/checks you may want to use (I didn't test all languages or do any cross-platform testing, I'm not sure all the details are compatible with streaming), however to highlight the issues:

```diff
diff --git a/OSCAR-2109.py b/OSCAR-2109.py
index bbac1076..5eee8de7 100644
--- a/OSCAR-2109.py
+++ b/OSCAR-2109.py
@@ -20,6 +20,7 @@
 import collections
 import gzip
 import json
+import os

 import datasets

@@ -387,9 +388,20 @@ class Oscar2109(datasets.GeneratorBasedBuilder):
         with open(checksum_file, encoding="utf-8") as f:
             data_filenames = [line.split()[1] for line in f if line]
             data_urls = [self.config.base_data_path + data_filename for data_filename in data_filenames]
-        text_files = dl_manager.download([url for url in data_urls if url.endswith(".txt.gz")])
-        metadata_files = dl_manager.download([url for url in data_urls if url.endswith(".jsonl.gz")])
+        # sort filenames so corresponding parts are aligned
+        text_files = sorted(dl_manager.download([url for url in data_urls if url.endswith(".txt.gz")]))
+        metadata_files = sorted(dl_manager.download([url for url in data_urls if url.endswith(".jsonl.gz")]))
+        assert len(text_files) == len(metadata_files)
         metadata_and_text_files = list(zip(metadata_files, text_files))
+        for meta_path, text_path in metadata_and_text_files:
+            # check that meta/text part numbers are the same
+            if "part" in os.path.basename(text_path):
+                assert (
+                    os.path.basename(text_path).replace(".txt.gz", "").split("_")[-1]
+                    == os.path.basename(meta_path).replace(".jsonl.gz", "").split("_")[-1]
+                )
+            else:
+                assert len(metadata_and_text_files) == 1
         return [
             datasets.SplitGenerator(name=datasets.Split.TRAIN, gen_kwargs={"metadata_and_text_files": metadata_and_text_files}),
         ]
@@ -397,10 +409,14 @@ class Oscar2109(datasets.GeneratorBasedBuilder):
     def _generate_examples(self, metadata_and_text_files):
         """This function returns the examples in the raw (text) form by iterating on all the files."""
         id_ = 0
-        offset = 0
         for meta_path, text_path in metadata_and_text_files:
+            # line offsets are per text file
+            offset = 0
             logger.info("generating examples from = %s", text_path)
-            with gzip.open(open(text_path, "rb"), "rt", encoding="utf-8") as text_f:
+            # some texts contain non-Unix newlines that should not be
+            # interpreted as line breaks for the line counts in the metadata
+            # with readline()
+            with gzip.open(open(text_path, "rb"), "rt", encoding="utf-8", newline="\n") as text_f:
                 with gzip.open(open(meta_path, "rb"), "rt", encoding="utf-8") as meta_f:
                     for line in meta_f:
                         # read meta
@@ -411,7 +427,12 @@ class Oscar2109(datasets.GeneratorBasedBuilder):
                             offset += 1
                             text_f.readline()
                         # read text
-                        text = "".join([text_f.readline() for _ in range(meta["nb_sentences"])]).rstrip()
+                        text_lines = [text_f.readline() for _ in range(meta["nb_sentences"])]
+                        # all lines contain text (no blank lines or EOF)
+                        assert all(text_lines)
+                        assert "\n" not in text_lines
                         offset += meta["nb_sentences"]
+                        # only strip the trailing newline
+                        text = "".join(text_lines).rstrip("\n")
                         yield id_, {"id": id_, "text": text, "meta": meta}
                         id_ += 1
```

I've tested this with a number of smaller deduplicated languages with 1-20 parts and the resulting datasets looked correct in terms of word count and size when compared to the data splits table and raw texts, and the text/metadata alignments were correct in all my spot checks. However, there are many many languages I didn't test and I'm not sure that there aren't any texts containing blank lines in the corpus, for instance. For the cases I tested, the assertions related to blank lines and EOF made it easier to verify that the text and metadata were aligned as intended, since there would be little chance of spurious alignments of variable-length texts across so much data.


> Hi! Some of our metrics require additional dependencies to work. In your case, simply installing the `seqeval` package with `pip install seqeval` should resolve the issue.
I installed seqeval, but still reported the same error. That's too bad.

> > Hi! Some of our metrics require additional dependencies to work. In your case, simply installing the `seqeval` package with `pip install seqeval` should resolve the issue.
> > I installed seqeval, but still reported the same error. That's too bad.

Same issue here. What should I do to fix this error? Please help! Thank you.

I tried to install **seqeval** package through anaconda instead of pip:
`conda install -c conda-forge seqeval`
It worked for me!

I can run it through the following steps:
![image](https://user-images.githubusercontent.com/69563759/159264511-1e252a4e-c8c8-44ab-b7bc-b4aac609bd9e.png)
Thank you for answering for me!

just change the file name seqeval.py to myseqeval.py

Metrics are deprecated in `datasets` and `evaluate` should be used instead: https://github.com/huggingface/evaluate

hi :
I want to use the seqeval indicator because of direct load_ When metric ('seqeval '), it will prompt that the network connection fails. So I downloaded the seqeval Py to load locally. Loading code: metric = load_ metric(path='mymetric/seqeval/seqeval.py')

But tips:

Traceback (most recent call last):
  File "/home/ubuntu/Python3.6_project/zyf_project/transformers/examples/pytorch/token-classification/run_ner.py", line 604, in <module>
    main()
  File "/home/ubuntu/Python3.6_project/zyf_project/transformers/examples/pytorch/token-classification/run_ner.py", line 481, in main
    metric = load_metric(path='mymetric/seqeval/seqeval.py')
  File "/home/ubuntu/Python3.6_project/zyf_project/transformers_venv_0209/lib/python3.7/site-packages/datasets/load.py", line 610, in load_metric
    dataset=False,
  File "/home/ubuntu/Python3.6_project/zyf_project/transformers_venv_0209/lib/python3.7/site-packages/datasets/load.py", line 450, in prepare_module
    f"To be able to use this {module_type}, you need to install the following dependencies"
ImportError: To be able to use this metric, you need to install the following dependencies['seqeval'] using 'pip install seqeval' for instance'

**What should I do? Please help me, thank you**


Update data URL of lm1b dataset

dataset contribution

Contribution to a dataset script

Hi ! I'm getting some 503 from both the http and https addresses. Do you think we could host this data somewhere else ? (please check if there is a license and if it allows redistribution)

Both HTTP and HTTPS links are working now.

We are closing this PR.

The http address doesn't work anymore


Pin ElasticSearch

Until we manage to support ES 8.0, I'm setting the version to `<8.0.0`

Currently we're getting this error on 8.0:
```python
ValueError: Either 'hosts' or 'cloud_id' must be specified
```
When instantiating a `Elasticsearch()` object


Unable to load a dataset

bug

Something isn't working

Hi! `load_dataset` is intended to be used to load a canonical dataset (`wikipedia`), a packaged dataset (`csv`, `json`, ...) or a dataset hosted on the Hub. For local datasets saved with `save_to_disk("path/to/dataset")`, use `load_from_disk("path/to/dataset")`.

Maybe we should raise an informative error message in this case...

## Describe the bug
Unable to load a dataset from Huggingface that I have just saved.

## Steps to reproduce the bug
On Google colab
`! pip install datasets `
`from datasets import load_dataset`
`my_path = "wiki_dataset"`
`dataset = load_dataset('wikipedia', "20200501.fr")`
`dataset.save_to_disk(my_path)`
`dataset = load_dataset(my_path)`

## Expected results
Loading the dataset

## Actual results
ValueError: Couldn't cast
_data_files: list<item: struct<filename: string>>
  child 0, item: struct<filename: string>
      child 0, filename: string
_fingerprint: string
_format_columns: null
_format_kwargs: struct<>
_format_type: null
_indexes: struct<>
_output_all_columns: bool
_split: string
to
{'builder_name': Value(dtype='string', id=None), 'citation': Value(dtype='string', id=None), 'config_name': Value(dtype='string', id=None), 'dataset_size': Value(dtype='int64', id=None), 'description': Value(dtype='string', id=None), 'download_checksums': {}, 'download_size': Value(dtype='int64', id=None), 'features': {'title': {'dtype': Value(dtype='string', id=None), 'id': Value(dtype='null', id=None), '_type': Value(dtype='string', id=None)}, 'text': {'dtype': Value(dtype='string', id=None), 'id': Value(dtype='null', id=None), '_type': Value(dtype='string', id=None)}}, 'homepage': Value(dtype='string', id=None), 'license': Value(dtype='string', id=None), 'post_processed': Value(dtype='null', id=None), 'post_processing_size': Value(dtype='null', id=None), 'size_in_bytes': Value(dtype='int64', id=None), 'splits': {'train': {'name': Value(dtype='string', id=None), 'num_bytes': Value(dtype='int64', id=None), 'num_examples': Value(dtype='int64', id=None), 'dataset_name': Value(dtype='string', id=None)}}, 'supervised_keys': Value(dtype='null', id=None), 'task_templates': Value(dtype='null', id=None), 'version': {'version_str': Value(dtype='string', id=None), 'description': Value(dtype='string', id=None), 'major': Value(dtype='int64', id=None), 'minor': Value(dtype='int64', id=None), 'patch': Value(dtype='int64', id=None)}}
because column names don't match

## Environment info
- `datasets` version: 1.18.3
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 6.0.1


Add dev-only config to Natural Questions dataset

Great thanks ! I think we can fix the CI by copying the NQ folder on gcs to 0.0.3. Does that sound good ?

I've copied the 0.0.2 folder content to 0.0.3, as suggested.

I'm updating the dataset card...

As suggested by @lhoestq and @thomwolf, a new config has been added to Natural Questions dataset, so that only dev split can be downloaded.

Fix #413.


Add finetune-data CodeFill

dataset contribution

Contribution to a dataset script

Thanks for your contribution, @rgismondi. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.


Add code-fill datasets for pretraining/finetuning/evaluating

Hi ! Thanks for adding this dataset :)

It looks like your PR contains many changes in files that are unrelated to your changes, I think it might come from running `make style` with an outdated version of `black`. Could you try opening a new PR that only contains your additions ? (or force push to this PR)


Force unique keys in newsqa dataset

Currently, it may raise `DuplicatedKeysError`.

Fix #3630.


Fix ClassLabel to/from dict when passed names_file

Currently, `names_file` is a field of the data class `ClassLabel`, thus appearing when transforming it to dict (when saving infos). Afterwards, when trying to read it from infos, it conflicts with the other field `names`.

This PR, removes `names_file` as a field of the data class `ClassLabel`.
- it is only used at instantiation to generate the `labels` field

Fix #3631.


Standardize to `Example::`


Update data URL in pubmed dataset

- I updated the previous dummy data: I just had to rename the file and its directory
  - the dummy data zip contains only a single file: `pubmed22n0001.xml.gz`

Then I discover it fails: https://app.circleci.com/pipelines/github/huggingface/datasets/9800/workflows/173a4433-8feb-4fc6-ab9e-59762084e3e1/jobs/60437
```
No such file or directory: '.../dummy_data/pubmed22n0002.xml.gz'
```
- it needs dummy data for all the 1114 files:
  `_URLs = [f"ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed22n{i:04d}.xml.gz" for i in range(1, 1115)]`
- this confirms me that it never passed the test: these dummy data files were not present before my PR
- therefore, is it really useful the data test if we just ignore it when it does not pass?

In relation with JSON metadata, I was generating the file for `pubmed` (see above) in a GCP instance: after running during ~12h without finishing, I decided to stop the process.

Hi ! Yes I remembered we hardcoded an exception for this one:
https://github.com/huggingface/datasets/blob/36db39c75179a0a491c69a4491f7ae7e4615e66f/src/datasets/utils/mock_download_manager.py#L174-L176

The exception was used to only require one dummy data file, feel free to update it if you want

Fix #3655.


Upgrade black to version ~=22.0

Upgrades the `datasets` library quality tool `black` to use the first stable release of `black`, version 22.0.


Update docs to new frontend/UI

We can have the docstrings of the properties that are missing docstrings (from discussion [here](https://github.com/huggingface/doc-builder/pull/96)) here by using your new `inject_arrow_table_documentation` onthem as well ?

@sgugger & @lhoestq could you help me with what should the `docs` section in setup.py be changed to [here](https://github.com/huggingface/datasets/blob/master/setup.py#L212-L227) ?

as a reference, here is a transformers setup.py docs [section](https://github.com/huggingface/transformers/blob/master/setup.py#L304-L308)

For now, you can put an empty list. Once the `doc-builder` is in a PyPi package (with the bug we fixed on Datasets but still waiting on the standing PR with the code switch) we can put it there.

None of those dependencies are needed from this list?

```py
    "docs": [
        "docutils==0.16.0",
        "recommonmark",
        "sphinx==3.1.2",
        "sphinx-markdown-tables",
        "sphinx-rtd-theme==0.4.3",
        "sphinxext-opengraph==0.4.1",
        "sphinx-copybutton",
        "fsspec<2021.9.0",
        "s3fs",
        "sphinx-panels",
        "sphinx-inline-tabs",
        "myst-parser",
        "Markdown!=3.3.5",
    ],
```

No, that was all for sphinx. The only thing needed to build the doc is a pip install of `doc-builder` (only from git right now).

@lhoestq feel free to request reviews from other maintainers 😊

Thanks ! @mariosasko and @albertvillanova feel free to take a look :)
I can do a thorough review this afternoon

Cool thanks ! Feel free to merge master into this branch and run `make style` to fix the python code formatting

Love the colorful vibes here!
![Screen Shot 2022-02-22 at 9 54 17 AM](https://user-images.githubusercontent.com/59462357/155193444-45e639dc-79cd-463c-98ad-1d44a6d6d385.png)

I just fixed the conflicts with the `master` branch :)

Could you update preprod please ? Or is there a preview somewhere I can check to make sure everything is ok ?

> Could you update preprod please ? Or is there a preview somewhere I can check to make sure everything is ok ?

I'll let you know once preprod gets updated

@lhoestq @stevhliu updated [preprod](https://moon-preprod.huggingface.co/docs/datasets/index) with the latest; please let e know if you see any errors

One more tiny error that doesn't seem specific to Datasets (Transformers example [here](https://huggingface.co/docs/transformers/multilingual#xlm-language-embeddings)), but apostrophes and symbols aren't properly displayed in the right navbar:

![Screen Shot 2022-03-02 at 8 39 10 AM](https://user-images.githubusercontent.com/59462357/156406988-27e79533-b02a-4fc2-af32-8ad84657488f.png)

In the latest commit https://github.com/huggingface/datasets/pull/3690/commits/20bddf28b22798c309e6eb1198a716f055889e1b, I tried to reflect changes from https://github.com/huggingface/transformers/pull/15903 , however, the gh workflow is not being triggered. @lhoestq do you know why it might be the case?

eve though, we have
https://github.com/huggingface/datasets/blob/20bddf28b22798c309e6eb1198a716f055889e1b/.github/workflows/build_dev_documentation.yml#L3-L7

I removed this line to trigger the job
```
  pull_request:
```

but got this error
```
[Error: .github#L1](https://github.com/huggingface/datasets/commit/033fe623c556b9dbc964708b672ff9bb4896c906#annotation_2897984435)
a step cannot have both the `uses` and `run` keys
```

It seems to be running again, and I re-added the line I removed.

Now the error is
```
> Run cd doc-build-dev && ...
README.md
datasets
transformers
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean
Error: Process completed with exit code 1.
```

@lhoestq if the CI passes, Im gonna merge this PR
please let me know if that sounds good

### TLDR: Update `datasets` `docs` to the new syntax (markdown and mdx files) & frontend (as how it looks on [hf.co/transformers](https://huggingface.co/docs/transformers/index))

| Light mode                                                                                | Dark mode                                                             |
|-----------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------|
|     |                                      |

## Checklist

- [x] update datasets docs to new syntax (should call `doc-builder convert`) (this PR)
- [x] discuss `@property` methods frontend https://github.com/huggingface/doc-builder/pull/87
- [x] discuss `inject_arrow_table_documentation` (this PR) https://github.com/huggingface/datasets/pull/3690#discussion_r801847860
- [x] update datasets docs path on moon-landing https://github.com/huggingface/moon-landing/pull/2089
- [x] convert pyarrow docstring from Numpydoc style to groups style  https://github.com/huggingface/doc-builder/pull/89(https://stackoverflow.com/a/24385103/6558628)
- [x] handle `Raises` section on frontend and doc-builder https://github.com/huggingface/doc-builder/pull/86
- [x] check imgs path (this PR) (nothing to update here)
- [x] doc exaples block has to follow format `Examples::` https://github.com/huggingface/datasets/pull/3693
- [x] fix [this docstring](https://github.com/huggingface/datasets/blob/6ed6ac9448311930557810383d2cfd4fe6aae269/src/datasets/arrow_dataset.py#L3339) (causing svelte compilation error)
- [x] Delete sphinx related files
- [x] Delete sphinx CI
- [x]  Update docs config in setup.py
- [x] add `versions.yml` in doc-build  https://github.com/huggingface/doc-build/pull/1
- [x] add `versions.yml` in doc-build-dev https://github.com/huggingface/doc-build-dev/pull/1
- [x] https://github.com/huggingface/moon-landing/pull/2089
- [x] format docstrings for example `datasets.DatasetBuilder.download_and_prepare` args format look wrong
- [x] create new github actions. (can probably be in a separate PR) (see the transformers equivalents below)
1. [build_dev_documentation.yml](https://github.com/huggingface/transformers/blob/master/.github/workflows/build_dev_documentation.yml)
2. [build_documentation.yml](https://github.com/huggingface/transformers/blob/master/.github/workflows/build_documentation.yml)
3. [delete_dev_documentation.yml](https://github.com/huggingface/transformers/blob/master/.github/workflows/delete_dev_documentation.yml)

## Note to reviewers

The number of changed files is a lot (100+) because I've converted all `.rst` files to `.mdx` files & they are compiling fine on the svelte side (also, moved all the imgs to to [doc-imgs repo](https://huggingface.co/datasets/huggingface/documentation-images/tree/main/datasets)). Moreover, you should just review them on preprod and see if the rendering look fine.
_Therefore, I'd suggest to focus on the changed_ **`.py`** and **CI files** (github workflows, etc. you can use [this filter here](https://github.com/huggingface/datasets/pull/3690/files?file-filters%5B%5D=.py&file-filters%5B%5D=.yml&show-deleted-files=true&show-viewed-files=true)) during the review & ignore `.mdx` files. (if there's a bug in `.mdx` files, we can always handle it in a separate PR afterwards).


Fix streaming for servers not supporting HTTP range requests

Does it mean that huge files might end up being downloaded? It would go against the purpose of streaming, I think. At least, this fallback should be an option that could be disabled

Yes, it is against the purpose of streaming, but streaming is not possible if the server does not allow HTTP range requests.

We have two options: either we download the file or we throw an error.

I think we simply cannot fallback to downloading the file if streaming fails without the user being aware of it. Some options:
- make the fallback optional (using an env var? or a function param)
- use the fallback only if the dataset size is under some threshold (provided we have the data in the DatasetInfo) -> it's the option I use in `datasets-preview-backend` ([here](https://github.com/huggingface/datasets-preview-backend/blob/48ac19e49c19809763e8d640986bf2c3d792faed/src/datasets_preview_backend/models/typed_row.py#L40) and [here](https://github.com/huggingface/datasets-preview-backend/blob/aa86c5493b275c9e2dbae7dab7bd469da5773a41/src/datasets_preview_backend/models/split.py#L31-L37))
- throw an exception and let the user decide what to do

IMO in general we should throw an exception and ask the user to not use streaming mode in that case.

Your second point is also interesting but I feel like it could be confusing for users sometimes: it doesn't feel natural that the streaming-ability should depend on the size of the file.

Sure, I think we should just throw an exception

Current behavior is already throwing an Exception:
```
ValueError: Cannot seek streaming HTTP file
```

We could customize the exception class and/or the exception message.

I'm not sure we really need to change anything. I opened the issue https://github.com/huggingface/datasets/issues/3677  because discovery was streamable and is not anymore (according to my test suite in https://github.com/huggingface/datasets-preview-backend): I was not sure if it was due to some regression in the library, or to some change in the dataset itself.

I'm wondering why it worked before and it is no longer working...

> We could customize the exception class and/or the exception message.

Yup a message that says that the host doesn't support streaming because it doesn't support HTTP Range requests would be useful !

Some servers do not support HTTP range requests, whereas this is required to stream some file formats (like ZIP).

~~This PR implements a workaround for those cases, by download the files locally in a temporary directory (cleaned up by the OS once the process is finished).~~

This PR raises custom error explaining that streaming is not possible because data host server does not support HTTP range requests.

Fix #3677.


Pyarrow version error

bug

Something isn't working

Hi @Zaker237, thanks for reporting.

This is weird: the error you get is only thrown if the installed pyarrow version is less than 3.0.0.

Could you please check that you install pyarrow in the same Python virtual environment where you installed datasets?

From the Python command line (or terminal) where you get the error, please type:
```
import pyarrow
print(pyarrow.__version__)
import datasets
print(datasets.__version__)
```

hi @albertvillanova  i try yesterday to create a new python environement with python 7 and try it on the environement and it worked. so i think that the error was not the package but may be jupyter notebook on conda. still yet i'm not yet sure but it worked in an environment created with venv

OK, thanks @Zaker237 for your feedback.

I close this issue then. Please, feel free to reopen it if the problem arises again.

## Describe the bug
I installed datasets(version 1.17.0, 1.18.0, 1.18.3) but i'm right now nor able to import it because of pyarrow. when i try to import it, i get the following error:
`To use datasets, the module pyarrow>=3.0.0 is required, and the current version of pyarrow doesn't match this condition`.
i tryed with all version of pyarrow execpt `4.0.0` but still get the same error.

## Steps to reproduce the bug
```python
import datasets
```

## Expected results
A clear and concise description of the expected results.

## Actual results
AttributeError                            Traceback (most recent call last)
<ipython-input-19-652e886d387f> in <module>
----> 1 import datasets

~\AppData\Local\Continuum\anaconda3\lib\site-packages\datasets\__init__.py in <module>
     26
     27
---> 28 if _version.parse(pyarrow.__version__).major < 3:
     29     raise ImportWarning(
     30         "To use `datasets`, the module `pyarrow>=3.0.0` is required, and the current version of `pyarrow` doesn't match this condition.\n"

AttributeError: 'Version' object has no attribute 'major'

## Environment info
Traceback (most recent call last):
  File "c:\users\alex\appdata\local\continuum\anaconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\alex\appdata\local\continuum\anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Alex\AppData\Local\Continuum\anaconda3\Scripts\datasets-cli.exe\__main__.py", line 5, in <module>
  File "c:\users\alex\appdata\local\continuum\anaconda3\lib\site-packages\datasets\__init__.py", line 28, in <module>
    if _version.parse(pyarrow.__version__).major < 3:
AttributeError: 'Version' object has no attribute 'major'

- `datasets` version:
- Platform: Linux(Ubuntu) and Windows: conda on the both
- Python version: 3.7
- PyArrow version: 7.0.0