Something isn't working

Hi @arymbe, thanks for reporting.

Unfortunately, I'm not able to reproduce your problem.

Could you please write the complete stack trace? That way we will be able to see which package originates the exception.

Hello, thank you for your fast replied. this is the complete error that I got

---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

Input In [27], in <module>
----> 1 from datasets import load_dataset

venv/lib/python3.8/site-packages/datasets/__init__.py:39, in <module>
     37 from .arrow_dataset import Dataset, concatenate_datasets
     38 from .arrow_reader import ReadInstruction
---> 39 from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
     40 from .combine import interleave_datasets
     41 from .dataset_dict import DatasetDict, IterableDatasetDict

venv/lib/python3.8/site-packages/datasets/builder.py:40, in <module>
     32 from .arrow_reader import (
     33     HF_GCP_BASE_URL,
     34     ArrowReader,
   (...)
     37     ReadInstruction,
     38 )
     39 from .arrow_writer import ArrowWriter, BeamWriter
---> 40 from .data_files import DataFilesDict, sanitize_patterns
     41 from .dataset_dict import DatasetDict, IterableDatasetDict
     42 from .features import Features

venv/lib/python3.8/site-packages/datasets/data_files.py:297, in <module>
    292     except FileNotFoundError:
    293         raise FileNotFoundError(f"The directory at {base_path} doesn't contain any data file") from None
    296 def _resolve_single_pattern_in_dataset_repository(
--> 297     dataset_info: huggingface_hub.hf_api.DatasetInfo,
    298     pattern: str,
    299     allowed_extensions: Optional[list] = None,
    300 ) -> List[PurePath]:
    301     data_files_ignore = FILES_TO_IGNORE
    302     fs = HfFileSystem(repo_info=dataset_info)

AttributeError: module 'huggingface_hub' has no attribute 'hf_api'

This is weird... It is long ago that the package `huggingface_hub` has a submodule called `hf_api`.

Maybe you have a problem with your installed `huggingface_hub`...

Could you please try to update it?
```shell
pip install -U huggingface_hub
```

Yap, I've updated several times. Then, I've tried numeral combination of datasets and huggingface_hub versions. However, I think your point is right that there is a problem with my huggingface_hub installation. I'll try another way to find the solution. I'll update it later when I get the solution. Thank you :)

I'm sorry I can't reproduce your problem.

Maybe you could try to create a new Python virtual environment and install all dependencies there from scratch. You can use either:
- Python venv: https://docs.python.org/3/library/venv.html
- or conda venv (if you are using conda): https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

Facing the same issue.

Response from `pip show datasets`
```
Name: datasets
Version: 1.15.1
Summary: HuggingFace community-driven open-source library of datasets
Home-page: https://github.com/huggingface/datasets
Author: HuggingFace Inc.
Author-email: thomas@huggingface.co
License: Apache 2.0
Location: /usr/local/lib/python3.8/dist-packages
Requires: aiohttp, dill, fsspec, huggingface-hub, multiprocess, numpy, packaging, pandas, pyarrow, requests, tqdm, xxhash
Required-by: lm-eval
```

Response from `pip show huggingface_hub`

```
Name: huggingface-hub
Version: 0.8.1
Summary: Client library to download and publish models, datasets and other repos on the huggingface.co hub
Home-page: https://github.com/huggingface/huggingface_hub
Author: Hugging Face, Inc.
Author-email: julien@huggingface.co
License: Apache
Location: /usr/local/lib/python3.8/dist-packages
Requires: filelock, packaging, pyyaml, requests, tqdm, typing-extensions
Required-by: datasets
```

response from `datasets-cli env`

```
Traceback (most recent call last):
  File "/usr/local/bin/datasets-cli", line 5, in <module>
    from datasets.commands.datasets_cli import main
  File "/usr/local/lib/python3.8/dist-packages/datasets/__init__.py", line 37, in <module>
    from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
  File "/usr/local/lib/python3.8/dist-packages/datasets/builder.py", line 44, in <module>
    from .data_files import DataFilesDict, _sanitize_patterns
  File "/usr/local/lib/python3.8/dist-packages/datasets/data_files.py", line 120, in <module>
    dataset_info: huggingface_hub.hf_api.DatasetInfo,
  File "/usr/local/lib/python3.8/dist-packages/huggingface_hub/__init__.py", line 105, in __getattr__
    raise AttributeError(f"No {package_name} attribute {name}")
AttributeError: No huggingface_hub attribute hf_api
```

A workaround:
I changed lines around Line 125 in `__init__.py` of `huggingface_hub` to something like
```
__getattr__, __dir__, __all__ = _attach(
    __name__,
    submodules=['hf_api'],
```
and it works ( which gives `datasets` direct access to `huggingface_hub.hf_api` ).

I was getting the same issue. After trying a few versions, following combination worked for me.
dataset==2.3.2
huggingface_hub==0.7.0

In another environment, I just installed latest repos from pip through `pip install -U transformers datasets tokenizers evaluate`, resulting in following versions. This also worked. Hope it helps someone.

datasets-2.3.2 evaluate-0.1.2 huggingface-hub-0.8.1 responses-0.18.0 tokenizers-0.12.1 transformers-4.20.1

(For layoutlmv3 fine-tuning) In my case, modifying `requirements.txt` as below worked.

- python = 3.7

```
datasets==2.3.2
evaluate==0.1.2
huggingface-hub==0.8.1
response==0.5.0
tokenizers==0.10.1
transformers==4.12.5
seqeval==1.2.2
deepspeed==0.5.7
tensorboard==2.7.0
seqeval==1.2.2
sentencepiece
timm==0.4.12
Pillow
einops
textdistance
shapely
```

> For layoutlm_v3 finetune datasets-2.3.2 evaluate-0.1.2 huggingface-hub-0.8.1 responses-0.18.0 tokenizers-0.12.1 transformers-4.12.5

GOOD!! Thanks!

## Describe the bug
Could you help me please. I got this following error.

AttributeError: module 'huggingface_hub' has no attribute 'hf_api'

## Steps to reproduce the bug
when I imported the datasets

# Sample code to reproduce the bug
from datasets import list_datasets, load_dataset, list_metrics, load_metric

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: macOS-12.3-x86_64-i386-64bit
- Python version: 3.8.9
- PyArrow version: 7.0.0
- Pandas version: 1.3.5
- Huggingface-hub: 0.5.0
- Transformers: 4.18.0

Thank you in advance.


Pretty print dataset info files

maybe just do it from now on no? (i.e. not for existing `dataset_infos.json` files)

_The documentation is not available anymore as the PR was closed or merged._

> maybe just do it from now on no? (i.e. not for existing dataset_infos.json files)

Yes, or do this only for datasets created with `push_to_hub` to (always) keep the GH datasets small?

I reverted the change to avoid the size increase and added the `pretty_print` flag, which pretty-prints the JSON, and that flag is only True for datasets created with `push_to_hub`.

Adds indentation to the `dataset_infos.json` file when saving for nicer diffs.

(suggested by @julien-c)

This PR also updates the info files of the GH datasets. Note that this change adds more than **10 MB** to the repo size (the total file size before the change: 29.672298 MB, after: 41.666475 MB), so I'm not sure this change is a good idea.

`src/datasets/info.py` is the only relevant file for reviewers.


ImageFolder add option to ignore some folders like '.ipynb_checkpoints'

enhancement

New feature or request

Maybe it would be nice to ignore private dirs like this one (ones starting with `.`) by default.

CC @mariosasko

Maybe we can add a `ignore_hidden_files` flag to the builder configs of our packaged loaders (to be consistent across all of them), wdyt @lhoestq @albertvillanova?

I think they should always ignore them actually ! Not sure if adding a flag would be helpful

@lhoestq But what if the user explicitly requests those files via regex?

`glob.glob` ignores hidden files (files starting with ".") by default unless they are explicitly requested, but fsspec's `glob` doesn't follow this behavior, which is probably a bug, so maybe we can raise an issue or open a PR in their repo?

> @lhoestq But what if the user explicitly requests those files via regex?

Usually hidden files are meant to be ignored. If they are data files, they must be placed outside a hidden directory in the first place right ? I think it's more sensible to explain this than adding a flag.

> glob.glob ignores hidden files (files starting with ".") by default unless they are explicitly requested, but fsspec's glob doesn't follow this behavior, which is probably a bug, so maybe we can raise an issue or open a PR in their repo?

After globbing using `fsspec`, we already ignore files that start with a `.` in `_resolve_single_pattern_locally` and `_resolve_single_pattern_in_dataset_repository`, I guess we can just account for parent directories as well ?

We could open an issue on `fsspec` but I think they won't change this since it's an important breaking change for them.

**Is your feature request related to a problem? Please describe.**
I sometimes like to peek at the dataset images from jupyterlab. thus '.ipynb_checkpoints' folder appears where my dataset is and (just realized) leads to accidental duplicate image additions. I think this is an easy enough thing to miss especially if the dataset is very large.

**Describe the solution you'd like**
maybe have an option `ignore` or something .gitignore style
`dataset = load_dataset("imagefolder", data_dir="./data/original", ignore="regex?")`

**Describe alternatives you've considered**
Could filter out manually


Allow downloading just some columns of a dataset

New feature or request

In the general case you can’t always reduce the quantity of data to download, since you can’t parse CSV or JSON data without downloading the whole files right ? ^^ However we could explore this case-by-case I guess

Actually for csv pandas has `usecols` which allows loading a subset of columns in a more efficient way afaik, but yes, you're right this might be more complex than I thought.

**Is your feature request related to a problem? Please describe.**
Some people are interested in doing label analysis of a CV dataset without downloading all the images. Downloading the whole dataset does not always makes sense for this kind of use case

**Describe the solution you'd like**
Be able to just download some columns of a dataset, such as doing
```python
load_dataset("huggan/wikiart",columns=["artist", "genre"])
```

Although this might make things a bit complicated in terms of local caching of datasets.


Multiprocessing with FileLock fails in python 3.9

bug

Something isn't working

On python 3.9, this code hangs:
```python
from multiprocessing import Pool
from filelock import FileLock

def run(i):
    print(f"got the lock in multi process [{i}]")

with FileLock("tmp.lock"):
    with Pool(2) as pool:
        pool.map(run, range(2))

```

This is because the subprocesses try to acquire the lock from the main process for some reason. This is not the case in older versions of python.

This can cause many issues in python 3.9. In particular, we use multiprocessing to fetch data files when you load a dataset (as long as there are >16 data files). Therefore `imagefolder` hangs, and I expect any dataset that needs to download >16 files to hang as well.

Let's see if we can fix this and have a CI that runs on 3.9.

cc @mariosasko @julien-c


ImageFolder with Grayscale images dataset

Hi! Replacing:
```python
transformed_dataset = dataset.with_transform(transforms)
transformed_dataset.set_format(type="torch", device="cuda")
```

with:
```python
def transform_func(examples):
    examples["image"] = [transforms(img).to("cuda") for img in examples["image"]]
    return examples

transformed_dataset = dataset.with_transform(transform_func)
```
should fix the issue. `datasets` doesn't support chaining of transforms (you can think of `set_format`/`with_format` as a predefined transform func for `set_transform`/`with_transforms`), so the last transform (in your case, `set_format`) takes precedence over the previous ones (in your case `with_format`). And the PyTorch formatter is not supported by the Image feature, hence the error (adding support for that is on our short-term roadmap).

Ok thanks a lot for the code snippet!

I love the way `datasets` is easy to use but it made it really long to pre-process all the images (400.000 in my case) before training anything. `ImageFolder` from pytorch is faster in my case but force me to have the images on my local machine.

I don't know how to speed up the process without switching to `ImageFolder` :smile:

You can pass `ignore_verifications=True` in `load_dataset` to skip checksum verification, which takes a lot of time if the number of files is large. We will consider making this the default behavior.

Hi, I'm facing a problem with a grayscale images dataset I have uploaded [here](https://huggingface.co/datasets/ChainYo/rvl-cdip) (RVL-CDIP)

I'm getting an error while I want to use images for training a model with PyTorch DataLoader. Here is the full traceback:

```bash
AttributeError: Caught AttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1765, in __getitem__
    return self._getitem(
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1750, in _getitem
    formatted_output = format_table(
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/formatting/formatting.py", line 532, in format_table
    return formatter(pa_table, query_type=query_type)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/formatting/formatting.py", line 281, in __call__
    return self.format_row(pa_table)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/formatting/torch_formatter.py", line 58, in format_row
    return self.recursive_tensorize(row)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/formatting/torch_formatter.py", line 54, in recursive_tensorize
    return map_nested(self._recursive_tensorize, data_struct, map_list=False)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 314, in map_nested
    mapped = [
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 315, in <listcomp>
    _single_map_nested((function, obj, types, None, True, None))
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 267, in _single_map_nested
    return {k: _single_map_nested((function, v, types, None, True, None)) for k, v in pbar}
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 267, in <dictcomp>
    return {k: _single_map_nested((function, v, types, None, True, None)) for k, v in pbar}
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 251, in _single_map_nested
    return function(data_struct)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/formatting/torch_formatter.py", line 51, in _recursive_tensorize
    return self._tensorize(data_struct)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/formatting/torch_formatter.py", line 38, in _tensorize
    if np.issubdtype(value.dtype, np.integer):
AttributeError: 'bytes' object has no attribute 'dtype'
```

I don't really understand why the image is still a bytes object while I used transformations on it. Here the code I used to upload the dataset (and it worked well):

```python
train_dataset = load_dataset("imagefolder", data_dir="data/train")
train_dataset = train_dataset["train"]
test_dataset = load_dataset("imagefolder", data_dir="data/test")
test_dataset = test_dataset["train"]
val_dataset = load_dataset("imagefolder", data_dir="data/val")
val_dataset = val_dataset["train"]

dataset = DatasetDict({
    "train": train_dataset,
    "val": val_dataset,
    "test": test_dataset
})
dataset.push_to_hub("ChainYo/rvl-cdip")
```

Now here is the code I am using to get the dataset and prepare it for training:

```python
img_size = 512
batch_size = 128
normalize = [(0.5), (0.5)]
data_dir = "ChainYo/rvl-cdip"

dataset = load_dataset(data_dir, split="train")

transforms = transforms.Compose([
        transforms.Resize(img_size),
        transforms.CenterCrop(img_size),
        transforms.ToTensor(),
        transforms.Normalize(*normalize)
])

transformed_dataset = dataset.with_transform(transforms)
transformed_dataset.set_format(type="torch", device="cuda")

train_dataloader = torch.utils.data.DataLoader(
    transformed_dataset, batch_size=batch_size, shuffle=True, num_workers=4, pin_memory=True
)
```

But this get me the error above. I don't understand why it's doing this kind of weird thing?
Do I need to map something on the dataset? Something like this:

```python
labels = dataset.features["label"].names
num_labels = dataset.features["label"].num_classes

def preprocess_data(examples):
    images = [ex.convert("RGB") for ex in examples["image"]]
    labels = [ex for ex in examples["label"]]
    return {"images": images, "labels": labels}

features = Features({
    "images": Image(decode=True, id=None),
    "labels": ClassLabel(num_classes=num_labels, names=labels)
})

decoded_dataset = dataset.map(preprocess_data, remove_columns=dataset.column_names, features=features, batched=True, batch_size=100)
```


Update security policy


Matthews Correlation Metric Card


Add Spearmanr Metric Card

changes made! @lhoestq let me know what you think

The CI fail is unrelated to this PR and fixed on master, feel free to merge :)


Perplexity Speedup

WRT the high values, can you add some unit tests with some [string, model] pairs and their resulting perplexity code, and @TristanThrush can run the same pairs through his version of the code?

> I thought that the perplexity metric should output the average perplexity value of all the strings that it gets as input (not a perplexity value per string, as the new version does). @lhoestq , @TristanThrush thoughts?

I support this change from Emi. If we have a perplexity function that loads GPT2 and then returns an average over all of the strings, then it is impossible to get multiple perplexities of a batch of strings efficiently. If we have this new perplexity function that is built for batching, then it is possible to get a batch of perplexities efficiently and you can still compute the average efficiently afterwards.

Thanks a lot for working on this @emibaylor @TristanThrush :)

For consistency with the other metrics, I think it's nice if we return the mean perplexity. Though I agree that having the separate perplexities per sample can also be useful. What do you think about returning both ?
```python
return {"perplexities": ppls, "mean_perplexity": np.mean(ppls)}
```
we're also doing this for the COMET metric.

> Thanks a lot for working on this @emibaylor @TristanThrush :)
>
> For consistency with the other metrics, I think it's nice if we return the mean perplexity. Though I agree that having the separate perplexities per sample can also be useful. What do you think about returning both ?
>
> ```python
> return {"perplexities": ppls, "mean_perplexity": np.mean(ppls)}
> ```
>
> we're also doing this for the COMET metric.

Thanks! Sounds great to me.

The CI fail is unrelated to your PR and has been fixed on master, feel free to merge the master branch into your PR to fix the CI ;)

This PR makes necessary changes to perplexity such that:
- it runs much faster (via batching)
- it throws an error when input is empty, or when input is one word without <BOS> token
- it adds the option to add a <BOS> token

Issues:
- The values returned are extremely high, and I'm worried they aren't correct. Even if they are correct, they are sometimes returned as `inf`, which is not very useful (see [comment below](https://github.com/huggingface/datasets/pull/4108#discussion_r843931094) for some of the output values).
  - If the values are not correct, can you help me find the error?
  - If the values are correct, it might be worth it to measure something like perplexity per word, which would allow us to get actual values for the larger perplexities, instead of just `inf`

Future:
- `stride` is not currently implemented here. I have some thoughts on how to make it happen with batching, but I think it would be better to get another set of eyes to look at any possible errors causing such large values now rather than later.


Unable to view the dataset and loading the same dataset throws the error - ArrowInvalid: Exceeded maximum rows

Something isn't working

  It's not related to the dataset viewer in itself. I can replicate the error with:

```
>>> import datasets as ds
>>> d = ds.load_dataset('Pavithree/explainLikeImFive')
Using custom data configuration Pavithree--explainLikeImFive-b68b6d8112cd8a51
Downloading and preparing dataset json/Pavithree--explainLikeImFive to /home/slesage/.cache/huggingface/datasets/json/Pavithree--explainLikeImFive-b68b6d8112cd8a51/0.0.0/ac0ca5f5289a6cf108e706efcf040422dbbfa8e658dee6a819f20d76bb84d26b...
Downloading data: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 305M/305M [00:03<00:00, 98.6MB/s]
Downloading data: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 17.9M/17.9M [00:00<00:00, 75.7MB/s]
Downloading data: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11.9M/11.9M [00:00<00:00, 70.6MB/s]
Downloading data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:05<00:00,  1.92s/it]
Extracting data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1948.42it/s]
Failed to read file '/home/slesage/.cache/huggingface/datasets/downloads/5fee9c8819754df277aee6f252e4db6897d785231c21938407b8862ca871d246' with error <class 'pyarrow.lib.ArrowInvalid'>: Exceeded maximum rows
Traceback (most recent call last):
  File "/home/slesage/hf/datasets/src/datasets/packaged_modules/json/json.py", line 144, in _generate_tables
    dataset = json.load(f)
  File "/home/slesage/.pyenv/versions/3.8.11/lib/python3.8/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/home/slesage/.pyenv/versions/3.8.11/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/home/slesage/.pyenv/versions/3.8.11/lib/python3.8/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 916 (char 915)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets/src/datasets/load.py", line 1691, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/slesage/hf/datasets/src/datasets/builder.py", line 605, in download_and_prepare
    self._download_and_prepare(
  File "/home/slesage/hf/datasets/src/datasets/builder.py", line 694, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/slesage/hf/datasets/src/datasets/builder.py", line 1151, in _prepare_split
    for key, table in logging.tqdm(
  File "/home/slesage/.pyenv/versions/datasets/lib/python3.8/site-packages/tqdm/std.py", line 1168, in __iter__
    for obj in iterable:
  File "/home/slesage/hf/datasets/src/datasets/packaged_modules/json/json.py", line 146, in _generate_tables
    raise e
  File "/home/slesage/hf/datasets/src/datasets/packaged_modules/json/json.py", line 122, in _generate_tables
    pa_table = paj.read_json(
  File "pyarrow/_json.pyx", line 246, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Exceeded maximum rows
```

cc @lhoestq @albertvillanova @mariosasko

It seems that train.json is not a valid JSON Lines file: it has several JSON objects in the first line (the 915th character in the first line starts a new object, and there's no "\n")

You need to have one JSON object per line

I'm closing this issue.

@Pavithree, please, feel free to re-open it if fixing the JSON file does not solve it.

## Dataset viewer issue - -ArrowInvalid: Exceeded maximum rows

**Link:** *https://huggingface.co/datasets/Pavithree/explainLikeImFive*

*This is the subset of original eli5 dataset  https://huggingface.co/datasets/vblagoje/lfqa. I just filtered the data samples which belongs to one particular subreddit thread. However, the dataset preview for train split  returns  the below mentioned error:
Status code:   400
Exception:     ArrowInvalid
Message:       Exceeded maximum rows
When I try to load the same dataset it returns ArrowInvalid: Exceeded maximum rows error*

Am I the one who added this dataset ? Yes


Support huggingface_hub 0.5

Looks like GH actions is not able to resolve `huggingface_hub` 0.5.0, I'm investivating

_The documentation is not available anymore as the PR was closed or merged._

I'm glad to see changes in `huggingface_hub` are simplifying code here.

seems to supersede #4102, feel free to close mine :)

maybe just cherry-pick the docstring fix

I think I've found the issue:
- https://github.com/huggingface/huggingface_hub/pull/790

Good catch, `huggingface_hub` doesn't support python 3.6 anymore indeed, therefore we should keep support for 0.4.0. I'm reverting the requirement version bump for now.

We can update the requirement once we drop support for python 3.6 in `datasets`

@lhoestq, I've opened this PR on `huggingface_hub`:
- https://github.com/huggingface/huggingface_hub/pull/823

Is there any strong reason why `huggingface_hub` no longer supports Python 3.6?

I think `datasets` can drop support for 3.6 soon. But for now maybe let's keep support for 0.4.0, python 3.6 users are not affected by https://github.com/huggingface/datasets/issues/4105 anyway.

`huggingface_hub` doesn't not have to support 3.6 again just for the CI IMO

@lhoestq I commented on the PR, that IMO it is not a good practice to drop support for Python 3.6 without a previous deprecation cycle.

Re-added support for older versions. I ended up checking `huggingface_hub` version to use the old, deprecated API for <0.5.0

I find it good practice to have all dependency version related code in a single file so that when you decide to remove support for an old version of a dependency it's easy to find and remove them, hence suggesting `utils/_fixes.py` in https://github.com/huggingface/datasets/issues/4105#issuecomment-1090041204

I used your suggestion @adrinjalali , I just replace the try/except with a check on the version of `huggingface_hub`

Following https://github.com/huggingface/datasets/issues/4105

`huggingface_hub` deprecated some parameters in `HfApi` in 0.5. This PR updates all the calls to HfApi to remove all the deprecations, <s>and I set the `hugginface_hub` requirement to `>=0.5.0`</s>

cc @adrinjalali @LysandreJik


push to hub fails with huggingface-hub 0.5.0

Something isn't working

Hi ! Indeed there was a breaking change in `huggingface_hub` 0.5.0 in `HfApi.create_repo`, which is called here in `datasets` by passing the org name in both the `repo_id` and the `organization` arguments:

https://github.com/huggingface/datasets/blob/2230f7f7d7fbaf102cff356f5a8f3bd1561bea43/src/datasets/arrow_dataset.py#L3363-L3369

I think we should fix that in `huggingface_hub`, will keep you posted. In the meantime please use `huggingface_hub` 0.4.0

I'll be sending a fix for this later today on the `huggingface_hub` side.

The error would be converted to a `FutureWarning` if `datasets` uses kwargs instead of positional, for example here:

https://github.com/huggingface/datasets/blob/2230f7f7d7fbaf102cff356f5a8f3bd1561bea43/src/datasets/arrow_dataset.py#L3363-L3369

to be:

``` python
            api.create_repo(
                name=dataset_name,
                token=token,
                repo_type="dataset",
                organization=organization,
                private=private,
            )
```

But `name` and `organization` are deprecated in `huggingface_hub=0.5`, and people should pass `repo_id='org/name` instead. Note that `repo_id` was introduced in 0.5 and if `datasets` wants to support older `huggingface_hub` versions (which I encourage it to do), there needs to be a helper function to do that. It can be something like:

```python
def create_repo(
    client,
    name: str,
    token: Optional[str] = None,
    organization: Optional[str] = None,
    private: Optional[bool] = None,
    repo_type: Optional[str] = None,
    exist_ok: Optional[bool] = False,
    space_sdk: Optional[str] = None,
) -> str:
    try:
        return client.create_repo(
            repo_id=f"{organization}/{name}",
            token=token,
            private=private,
            repo_type=repo_type,
            exist_ok=exist_ok,
            space_sdk=space_sdk,
        )
    except TypeError:
        return client.create_repo(
            name=name,
            organization=organization,
            token=token,
            private=private,
            repo_type=repo_type,
            exist_ok=exist_ok,
            space_sdk=space_sdk,
        )
```

in a `utils/_fixes.py` kinda file and and be used internally.

I'll be sending a patch to `huggingface_hub` to convert the error reported in this issue to a `FutureWarning`.

PR with the hotfix on the `huggingface_hub` side: https://github.com/huggingface/huggingface_hub/pull/822

We can definitely change `push_to_hub` to use `repo_id` in `datasets` and require `huggingface_hub>=0.5.0`.

Let me open a PR :)

`huggingface_hub` 0.5.1 just got released with a fix, feel free to update `huggingface_hub` ;)

## Describe the bug
`ds.push_to_hub` is failing when updating a dataset in the form "org_id/repo_id"

## Steps to reproduce the bug
```python
from datasets import load_dataset
ds = load_dataset("rubrix/news_test")
ds.push_to_hub("<your-user>/news_test", token="<your-token>")
```

## Expected results
The dataset is successfully uploaded

## Actual results
An error validation is raised:

```bash
if repo_id and (name or organization):
>           raise ValueError(
                "Only pass `repo_id` and leave deprecated `name` and "
                "`organization` to be None."
E               ValueError: Only pass `repo_id` and leave deprecated `name` and `organization` to be None.
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.1
- `huggingface-hub`: 0.5
- Platform: macOS
- Python version: 3.8.12
- PyArrow version: 6.0.0

cc @adrinjalali


Add time series data - stock market

dataset request

Requesting to add a new dataset

Can I use instructions present in below link for time series dataset as well?
https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md

cc'ing @kashif and @NielsRogge for visibility!

@INF800 happy to add this dataset! I will try to set a PR by the end of the day... if you can kindly point me to the dataset? Also, note we have a bunch of time series datasets checked in e.g. `electricity_load_diagrams` or `monash_tsf`, and ideally this dataset could also be in a similar format.

Thankyou. This is how raw data looks like before cleaning for an individual stocks:

1. https://github.com/INF800/marktech/tree/raw-data/f/data/raw
2. https://github.com/INF800/marktech/tree/raw-data/t/data/raw
3. https://github.com/INF800/marktech/tree/raw-data/rdfn/data/raw
4. https://github.com/INF800/marktech/tree/raw-data/irbt/data/raw
5. https://github.com/INF800/marktech/tree/raw-data/hll/data/raw
6. https://github.com/INF800/marktech/tree/raw-data/infy/data/raw
7. https://github.com/INF800/marktech/tree/raw-data/reli/data/raw
8. https://github.com/INF800/marktech/tree/raw-data/hdbk/data/raw

> Scraping is automated using GitHub Actions. So, everyday we will see a new file added in the above links.

I can rewrite the cleaning scripts to make sure it fits HF dataset standards. (P.S I am very much new to HF dataset)

The data set above can be converted into univariate regression / multivariate regression / sequence to sequence generation dataset etc. So, do we have some kind of transformation modules that will read the dataset as some type of dataset (`GenericTimeData`) and convert it to other possible dataset relating to a specific ML task. **By having this kind of transformation module, I only have to add data once** and use transformation module whenever necessary

Additionally, having some kind of versioning for the dataset will be really helpful because it will keep on updating - especially time series datasets

thanks @INF800 I'll have a look. I believe it should be possible to incorporate this into the time-series format.

@INF800 yes I am aware of the review repository and paper which is more or less a collection of abstracts etc. I am working on a unified library of implementations of these papers together with datasets to be then able to compare/contrast and build upon the research etc. but I am not ready to share them publicly just yet.

In any case regarding your dataset at the moment its seems from looking at the csv files, its mixture of textual and numerical data, sometimes in the same column etc. As you know, for time series models we would need just numeric data so I would need your help in disambiguating the dataset you have collected and also perhaps starting with just numerical data to start with...

Do you think you can make a version with just numerical data?

> @INF800 yes I am aware of the review repository and paper which is more or less a collection of abstracts etc. I am working on a unified library of implementations of these papers together with datasets to be then able to compare/contrast and build upon the research etc. but I am not ready to share them publicly just yet.
>
> In any case regarding your dataset at the moment its seems from looking at the csv files, its mixture of textual and numerical data, sometimes in the same column etc. As you know, for time series models we would need just numeric data so I would need your help in disambiguating the dataset you have collected and also perhaps starting with just numerical data to start with...
>
> Do you think you can make a version with just numerical data?

Will share the numeric data and conversion script within end of this week.

I am on a business trip currently - it is in my desktop.

## Adding a Time Series Dataset
- **Name:** 2min ticker data for stock market
- **Description:** 8 stocks' data collected for 1month post ukraine-russia war. 4 NSE stocks and 4 NASDAQ stocks. Along with technical indicators (additional features) as shown in below image
- **Data:** Collected by myself from investing.com
- **Motivation:** Test applicability of transformer based model on stock market / time series problem

![image](https://user-images.githubusercontent.com/45640029/161904077-52fe97cb-3720-4e3f-98ee-7f6720a056e2.png)


Add the `GSM8K` dataset

_The documentation is not available anymore as the PR was closed or merged._

The CI is failing because it's outdated, but the task tags are updated on `master`, merging :)


[hub] Fix `api.create_repo` call?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4102). All of your documentation changes will be reflected on that endpoint.

Closing in favor of https://github.com/huggingface/datasets/pull/4106


enhancement

New feature or request

Hi! Can you please specify the full name of the dataset? IIRC `full_numbers` is one of the configs of the `svhn` dataset, and its generation is slow due to data being stored in binary Matlab files. Even if you specify a specific split, `datasets` downloads all of them, but we plan to fix that soon and only download the requested split.

If you are in a hurry, download the `svhn` script [here](`https://huggingface.co/datasets/svhn/blob/main/svhn.py`), remove [this code](https://huggingface.co/datasets/svhn/blob/main/svhn.py#L155-L162), and run:
```python
from datasets import load_dataset
dset = load_dataset("path/to/your/local/script.py", "full_numbers")
```

And to make loading easier in Colab, you can create a dataset repo on the Hub and upload the script there. Or push the script to Google Drive and mount the drive in Colab.

How can I download only the train and test split for full numbers using load_dataset()?

I do not need the extra split and it will take 40 mins just to download in Colab. I have very short time in hand. Please help.


Improve RedCaps dataset card

I find this preprocessing a bit too specific to add it as a method to `datasets` as it's only useful in the context of CV (and we support multiple modalities). However, I agree it would be great to move this code to another lib to avoid code duplication. Maybe we should create a package with preprocessing functions/transforms for this purpose?

This PR modifies the RedCaps card to:
* fix the formatting of the Point of Contact fields on the Hub
* speed up the image fetching logic (aligns it with the [img2dataset](https://github.com/rom1504/img2dataset) tool) and make it more robust (return None if **any** exception is thrown)


bug

Something isn't working

Hi @andreybond, thanks for reporting.

Unfortunately, I'm not able to able to reproduce your issue:
```python
In [4]: from datasets import load_dataset
   ...: datasets = load_dataset("nielsr/XFUN", "xfun.ja")

In [5]: datasets
Out[5]:
DatasetDict({
    train: Dataset({
        features: ['id', 'input_ids', 'bbox', 'labels', 'image', 'entities', 'relations'],
        num_rows: 194
    })
    validation: Dataset({
        features: ['id', 'input_ids', 'bbox', 'labels', 'image', 'entities', 'relations'],
        num_rows: 71
    })
})
```

The only reason I can imagine this issue may arise is if your default encoding is not "UTF-8" (and it is ASCII instead). This is usually the case on Windows machines; but you say your environment is a Linux machine. Maybe you change your machine default encoding?

Could you please check this?
```python
In [6]: import sys

In [7]: sys.getdefaultencoding()
Out[7]: 'utf-8'
```

I opened a PR in the original dataset loading script:
-  microsoft/unilm#677

and fixed the corresponding dataset script on the Hub:
- https://huggingface.co/datasets/nielsr/XFUN/commit/73ba5e026621e05fb756ae0f267eb49971f70ebd

import sys
sys.getdefaultencoding()

returned: 'utf-8'

---------------------

I've just cloned master branch - your fix works! Thank you!

## Describe the bug
Error "UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 213: ordinal not in range(128)" is thrown when downloading dataset.

## Steps to reproduce the bug
```python
from datasets import load_dataset
datasets = load_dataset("nielsr/XFUN", "xfun.ja")
```

## Expected results
Dataset should be downloaded without exceptions

## Actual results
Stack trace (for the second-time execution):
Downloading and preparing dataset xfun/xfun.ja to /root/.cache/huggingface/datasets/nielsr___xfun/xfun.ja/0.0.0/e06e948b673d1be9a390a83c05c10e49438bf03dd85ae9a4fe06f8747a724477...
Downloading data files: 100%
2/2 [00:00<00:00, 88.48it/s]
Extracting data files: 100%
2/2 [00:00<00:00, 79.60it/s]

UnicodeDecodeErrorTraceback (most recent call last)
<ipython-input-31-79c26bd1109c> in <module>
      1 from datasets import load_dataset
      2
----> 3 datasets = load_dataset("nielsr/XFUN", "xfun.ja")

/usr/local/lib/python3.6/dist-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)

/usr/local/lib/python3.6/dist-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    604         )
    605
--> 606         # By default, return all splits
    607         if split is None:
    608             split = {s: s for s in self.info.splits}

/usr/local/lib/python3.6/dist-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos)

/usr/local/lib/python3.6/dist-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    692         Args:
    693             split: `datasets.Split` which subset of the data to read.
--> 694
    695         Returns:
    696             `Dataset`

/usr/local/lib/python3.6/dist-packages/datasets/builder.py in _prepare_split(self, split_generator, check_duplicate_keys)

/usr/local/lib/python3.6/dist-packages/tqdm/notebook.py in __iter__(self)
    252         if not self.disable:
    253             self.display(check_delay=False)
--> 254
    255     def __iter__(self):
    256         try:

/usr/local/lib/python3.6/dist-packages/tqdm/std.py in __iter__(self)
   1183             for obj in iterable:
   1184                 yield obj
-> 1185             return
   1186
   1187         mininterval = self.mininterval

~/.cache/huggingface/modules/datasets_modules/datasets/nielsr--XFUN/e06e948b673d1be9a390a83c05c10e49438bf03dd85ae9a4fe06f8747a724477/XFUN.py in _generate_examples(self, filepaths)
    140             logger.info("Generating examples from = %s", filepath)
    141             with open(filepath[0], "r") as f:
--> 142                 data = json.load(f)
    143
    144             for doc in data["documents"]:

/usr/lib/python3.6/json/__init__.py in load(fp, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    294
    295     """
--> 296     return loads(fp.read(),
    297         cls=cls, object_hook=object_hook,
    298         parse_float=parse_float, parse_int=parse_int,

/usr/lib/python3.6/encodings/ascii.py in decode(self, input, final)
     24 class IncrementalDecoder(codecs.IncrementalDecoder):
     25     def decode(self, input, final=False):
---> 26         return codecs.ascii_decode(input, self.errors)[0]
     27
     28 class StreamWriter(Codec,codecs.StreamWriter):

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 213: ordinal not in range(128)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0 (but reproduced with many previous versions)
- Platform: Docker: Linux da5b74136d6b 5.3.0-1031-azure #32~18.04.1-Ubuntu SMP Mon Jun 22 15:27:23 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux ; Base docker image is : huggingface/transformers-pytorch-cpu
- Python version: 3.6.9
- PyArrow version:   6.0.1


Proposing WikiSplit metric card

A quick Github tip ;) To avoid running N times the CI, you can push all the changes at once: go to Files Changed tab, and on each suggestion there's a "add to commit batch" and then you can do one commit for all the suggestions you want to approve ;)

Pinging @lhoestq to ensure that my distinction between the dataset and the metric are clear :sweat_smile:


Updating FrugalScore metric card

removing duplicate paragraph


Add support for streaming Zarr stores for hosted datasets

enhancement

New feature or request

Hi @jacobbieker, thanks for your request and study of possible alternatives.

We are very interested in finding a way to make `datasets` useful to you.

Looking at the Zarr docs, I saw that among its storage alternatives, there is the ZIP file format: https://zarr.readthedocs.io/en/stable/api/storage.html#zarr.storage.ZipStore

This might be convenient for many reasons:
- On the one hand, we avoid the Git issue with huge number of small files: chunks files are compressed into a single ZIP file
- On the other hand, the ZIP file format is specially suited for streaming data because it allows random access to its component files (i.e. it supports random access to its chunks)

Anyway, I think that a Python loading script will be necessary: you need to implement additional logic to select certain chunks (based on date or other criteria).

Please, let me know if this makes sense to you.

Ah okay, I missed the option of zip files for zarr, I'll try that with our repos and see if it works! Thanks a lot!

Hi @jacobbieker, does the Zarr ZipStore work for your use case?

Hi,

Yes, it seems to! I got it working for https://huggingface.co/datasets/openclimatefix/mrms thanks for the help!

On behalf of the Zarr developers, let me say THANK YOU for working to support Zarr on HF! 🙏  Zarr is a 100% open-source and community driven project (fiscally sponsored by NumFocus). We see it as an ideal format for ML training datasets, particularly in scientific domains.

I think the solution of zipping the Zarr store is a reasonable way to balance the constraints of Git LFS with the structure of Zarr.

It would be amazing to get something on the [Hugging Face Datasets Docs](https://huggingface.co/docs/datasets/index) about how to best work with Zarr. Let me know if there's a way I could help with that effort.

Also just noting here that I was able to lazily open @jacobbieker's dataset over the internet from HF hub 🚀 !

```python
import xarray as xr
url = "https://huggingface.co/datasets/openclimatefix/mrms/resolve/main/data/2016_001.zarr.zip"
zip_url = 'zip:///::' + url
ds = xr.open_dataset(zip_url, engine='zarr', chunks={})
```

<img width="740" alt="image" src="https://user-images.githubusercontent.com/1197350/164508663-bc75cdc0-734d-44f4-9562-2877ecfdf433.png">

However, I wasn't able to get streaming working using the Datasets api:

```python
from datasets import load_dataset
ds = load_dataset("openclimatefix/mrms", streaming=True, split='train')
item = next(iter(ds))
```

<details>
<summary>FileNotFoundError traceback</summary>

```
No config specified, defaulting to: mrms/2021
zip://::https://huggingface.co/datasets/openclimatefix/mrms/resolve/main/data/2016_001.zarr.zip
data/2016_001.zarr.zip
zip://2016_001.zarr.zip::https://huggingface.co/datasets/openclimatefix/mrms/resolve/main/data/2016_001.zarr.zip
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Input In [1], in <cell line: 3>()
      1 from datasets import load_dataset
      2 ds = load_dataset("openclimatefix/mrms", streaming=True, split='train')
----> 3 item = next(iter(ds))

File /opt/miniconda3/envs/hugginface/lib/python3.9/site-packages/datasets/iterable_dataset.py:497, in IterableDataset.__iter__(self)
    496 def __iter__(self):
--> 497     for key, example in self._iter():
    498         if self.features:
    499             # we encode the example for ClassLabel feature types for example
    500             encoded_example = self.features.encode_example(example)

File /opt/miniconda3/envs/hugginface/lib/python3.9/site-packages/datasets/iterable_dataset.py:494, in IterableDataset._iter(self)
    492 else:
    493     ex_iterable = self._ex_iterable
--> 494 yield from ex_iterable

File /opt/miniconda3/envs/hugginface/lib/python3.9/site-packages/datasets/iterable_dataset.py:87, in ExamplesIterable.__iter__(self)
     86 def __iter__(self):
---> 87     yield from self.generate_examples_fn(**self.kwargs)

File ~/.cache/huggingface/modules/datasets_modules/datasets/openclimatefix--mrms/2a6f697014d7eb3caf586ca137d47ca38785ae2fe36248611b021f8248b59936/mrms.py:150, in MRMS._generate_examples(self, filepath, split)
    147 filepath = "[https://huggingface.co/datasets/openclimatefix/mrms/resolve/main/data/2016_001.zarr.zip](https://huggingface.co/datasets/openclimatefix/mrms/resolve/main/data/2016_001.zarr.zip%3C/span%3E%3Cspan) style="color:rgb(175,0,0)">"
    148 # TODO: This method handles input defined in _split_generators to yield (key, example) tuples from the dataset.
    149 # The `key` is for legacy reasons (tfds) and is not important in itself, but must be unique for each example.
--> 150 with zarr.storage.FSStore(fsspec.open("zip::" + filepath, mode='r'), mode='r') as store:
    151     data = xr.open_zarr(store)
    152     for key, row in enumerate(data["time"].values):

File /opt/miniconda3/envs/hugginface/lib/python3.9/site-packages/zarr/storage.py:1120, in FSStore.__init__(self, url, normalize_keys, key_separator, mode, exceptions, dimension_separator, **storage_options)
   1117 import fsspec
   1118 self.normalize_keys = normalize_keys
-> 1120 protocol, _ = fsspec.core.split_protocol(url)
   1121 # set auto_mkdir to True for local file system
   1122 if protocol in (None, "file") and not storage_options.get("auto_mkdir"):

File /opt/miniconda3/envs/hugginface/lib/python3.9/site-packages/fsspec/core.py:514, in split_protocol(urlpath)
    512 def split_protocol(urlpath):
    513     """Return protocol, path pair"""
--> 514     urlpath = stringify_path(urlpath)
    515     if "://" in urlpath:
    516         protocol, path = urlpath.split("://", 1)

File /opt/miniconda3/envs/hugginface/lib/python3.9/site-packages/fsspec/utils.py:315, in stringify_path(filepath)
    313     return filepath
    314 elif hasattr(filepath, "__fspath__"):
--> 315     return filepath.__fspath__()
    316 elif isinstance(filepath, pathlib.Path):
    317     return str(filepath)

File /opt/miniconda3/envs/hugginface/lib/python3.9/site-packages/fsspec/core.py:98, in OpenFile.__fspath__(self)
     96 def __fspath__(self):
     97     # may raise if cannot be resolved to local file
---> 98     return self.open().__fspath__()

File /opt/miniconda3/envs/hugginface/lib/python3.9/site-packages/fsspec/core.py:140, in OpenFile.open(self)
    132 def open(self):
    133     """Materialise this as a real open file without context
    134
    135     The file should be explicitly closed to avoid enclosed file
   (...)
    138     been deleted; but a with-context is better style.
    139     """
--> 140     out = self.__enter__()
    141     closer = out.close
    142     fobjects = self.fobjects.copy()[:-1]

File /opt/miniconda3/envs/hugginface/lib/python3.9/site-packages/fsspec/core.py:103, in OpenFile.__enter__(self)
    100 def __enter__(self):
    101     mode = self.mode.replace("t", "").replace("b", "") + "b"
--> 103     f = self.fs.open(self.path, mode=mode)
    105     self.fobjects = [f]
    107     if self.compression is not None:

File /opt/miniconda3/envs/hugginface/lib/python3.9/site-packages/fsspec/spec.py:1009, in AbstractFileSystem.open(self, path, mode, block_size, cache_options, compression, **kwargs)
   1007 else:
   1008     ac = kwargs.pop("autocommit", not self._intrans)
-> 1009     f = self._open(
   1010         path,
   1011         mode=mode,
   1012         block_size=block_size,
   1013         autocommit=ac,
   1014         cache_options=cache_options,
   1015         **kwargs,
   1016     )
   1017     if compression is not None:
   1018         from fsspec.compression import compr

File /opt/miniconda3/envs/hugginface/lib/python3.9/site-packages/fsspec/implementations/zip.py:96, in ZipFileSystem._open(self, path, mode, block_size, autocommit, cache_options, **kwargs)
     94 if mode != "rb":
     95     raise NotImplementedError
---> 96 info = self.info(path)
     97 out = self.zip.open(path, "r")
     98 out.size = info["size"]

File /opt/miniconda3/envs/hugginface/lib/python3.9/site-packages/fsspec/archive.py:42, in AbstractArchiveFileSystem.info(self, path, **kwargs)
     40     return self.dir_cache[path + "/"]
     41 else:
---> 42     raise FileNotFoundError(path)

FileNotFoundError:
```

</details>

Is this a bug? Or am I just doing it wrong...

I'm still messing around with that dataset, so the data might have moved. I currently have each year of MRMS precipitation rate data as it's own zarr, but as they are quite large (on order of 100GB each) I'm working to split them into single days, and as such they are still being moved around, I was just trying to get a proof of concept working originally.

I've mostly finished rearranging the data now and uploading some more, so this works now:
```python
import datasets
ds = datasets.load_dataset("openclimatefix/mrms", streaming=True, split="train")
item = next(iter(ds))
print(item.keys())
print(item["timestamp"])
```

The MRMS data now goes most of 2016-2022, with quite a few gaps I'm working on filling in

**Is your feature request related to a problem? Please describe.**
Lots of geospatial data is stored in the Zarr format. This format works well for n-dimensional data and coordinates, and can have good compression. Unfortunately, HF datasets doesn't support streaming in data in Zarr format as far as I can tell. Zarr stores are designed to be easily streamed in from cloud storage, especially with xarray and fsspec. Since geospatial data tends to be very large, and on the order of TBs of data or 10's of TBs of data for a single dataset, it can be difficult to store the dataset locally for users. Just adding Zarr stores with HF git doesn't work well (see https://github.com/huggingface/datasets/issues/3823) as Zarr splits the data into lots of small chunks for fast loading, and that doesn't work well with git. I've somewhat gotten around that issue by tarring each Zarr store and uploading them as a single file, which seems to be working (see https://huggingface.co/datasets/openclimatefix/gfs-reforecast for example data files, although the script isn't written yet). This does mean that streaming doesn't quite work though. On the other hand, in https://huggingface.co/datasets/openclimatefix/eumetsat_uk_hrv we stream in a Zarr store from a public GCP bucket quite easily.

**Describe the solution you'd like**
A way to upload Zarr stores for hosted datasets so that we can stream it with xarray and fsspec.

**Describe alternatives you've considered**
Tarring each Zarr store individually and just extracting them in the dataset script -> Downside this is a lot of data that probably doesn't fit locally for a lot of potential users.
Pre-prepare examples in a format like Parquet -> Would use a lot more storage, and a lot less flexibility, in the eumetsat_uk_hrv, we use the one Zarr store for multiple different configurations.


fix typo in rename_column error message

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4095). All of your documentation changes will be reflected on that endpoint.

I feel bad submitting such a tiny change as a PR but it confused me today 😄


Helo Mayfrends

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


elena-soare/crawled-ecommerce: missing dataset

dataset-viewer

Related to the dataset viewer on huggingface.co

It's a bug! Thanks for reporting, I'm looking at it.

By the way, the error on our part is due to the huge size of every row (~90MB). The dataset viewer does not support such big dataset rows for the moment.
Anyway, we're working to give a hint about this in the dataset viewer.

Fixed. See https://huggingface.co/datasets/elena-soare/crawled-ecommerce/viewer/elena-soare--crawled-ecommerce/train.

<img width="1552" alt="Capture d’écran 2022-04-12 à 11 23 51" src="https://user-images.githubusercontent.com/1676121/162929722-2e2b80e2-154a-4b61-87bd-e341bd6c46e6.png">

Thanks for reporting!

elena-soare/crawled-ecommerce

**Link:** *link to the dataset viewer page*

*short description of the issue*

Am I the one who added this dataset ? Yes-No


Fix dataset `amazon_us_reviews` metadata - 4/4/2022

_The documentation is not available anymore as the PR was closed or merged._

Fixes #4048 by running `dataset-cli test` to reprocess data and regenerate metadata. Additionally I've updated the README to include up-to-date counts for the subsets.


Build a Dataset One Example at a Time Without Loading All Data Into Memory

enhancement

New feature or request

Hi! Yes, the problem with `add_item` is that it keeps examples in memory, so you are left with these options:
* writing a dataset loading script in which you iterate over `custom_example_dict_streamer` and yield the examples (in `_generate examples`)
* storing the data in a JSON/CSV/Parquet/TXT file and using `Dataset.from_{format}`
* using `add_item` + `save_to_disk` on smaller chunks:
    ```python
    from datasets import Dataset, concatenate_datasets
    MAX_SAMPLES_IN_MEMORY = 1000
    samples_in_dset = 0
    dset = Dataset.from_dict({"col1": [], "col2": []})  # empty dataset
    path_to_save_dir = "path/to/save/dir"
    num_chunks = 0
    for example_dict in custom_example_dict_streamer("/path/to/raw/data"):
        dset = dset.add_item(example_dict)
        samples_in_dset += 1
        if samples_in_dset == MAX_SAMPLES_IN_MEMORY:
            samples_in_dset = 0
            dset.save_to_disk(f"{path_to_save_dir}{num_chunks}")
            num_chunks =+ 1
            dset = Dataset.from_dict({"col1": [], "col2": []})  # empty dataset
    if samples_in_dset > 0:
        dset.save_to_disk(f"{path_to_save_dir}{num_chunks}")
        num_chunks =+ 1
    loaded_dsets = []  # memory-mapped
    for chunk_num in range(num_chunks):
        dset = Dataset.load_from_disk(f"{path_to_save_dir}{chunk_num}")
        loaded_dsets.append(dset)
    final_dset = concatenate_datasets(dset)
    ```
    If you still have issues with this approach, you can try to delete unused datasets with `gc.collect()` to free some memory.

This is really elegant, thank you @mariosasko!  I will try this.

**Is your feature request related to a problem? Please describe.**
I have a very large dataset stored on disk in a custom format.  I have some custom code that reads one data example at a time and yields it in the form of a dictionary.  I want to construct a `Dataset` with all examples, and then save it to disk.  I later want to load the saved `Dataset` and use it like any other HuggingFace dataset, get splits, wrap it in a PyTorch `DataLoader`, etc.  **Crucially, I do not ever want to materialize all the data in memory while building the dataset.**

**Describe the solution you'd like**
I would like to be able to do something like the following.  Notice how each example is read and then immediately added to the dataset.  We do not store all the data in memory when constructing the `Dataset`.  If it helps, I will know the schema of my dataset before hand.
```

# Initialize an empty Dataset, possibly from a known schema.
dataset = Dataset()

# Read in examples one by one using a custom data streamer.
for example_dict in custom_example_dict_streamer("/path/to/raw/data"):

    # Add this example to the dict but do not store it in memory.
    dataset.add_item(example_dict)

# Save the final dataset to disk as an Arrow-backed dataset.
dataset.save_to_disk("/path/to/dataset")

...

# I'd like to be able to later `load_from_disk` and use the loaded Dataset
# just like any other memory-mapped pyarrow-backed HuggingFace dataset...
loaded_dataset = Dataset.load_from_disk("/path/to/dataset")
loaded_dataset.set_format(type="torch", columnns=["foo", "bar", "baz"])
dataloader = torch.utils.data.DataLoader(loaded_dataset, batch_size=16)
...

```

**Describe alternatives you've considered**
I initially tried to read all the data into memory, construct a Pandas DataFrame and then call `Dataset.from_pandas`.  This would not work as it requires storing all the data in memory.  It seems that there is an `add_item` method already -- I tried to implement something like the desired API written above, but I've not been able to initialize an empty `Dataset` (this seems to require several layers of constructing `datasets.table.Table` which requires constructing a `pyarrow.lib.Table`, etc).  I also considered writing my data to multiple sharded CSV files or JSON files and then using `from_csv` or `from_json`.  I'd prefer not to do this because (1) I'd prefer to avoid the intermediate step of creating these temp CSV/JSON files and (2) I'm not sure if `from_csv` and `from_json` use memory-mapping.

Do you have any suggestions on how I'd be able to achieve this use case?  Does something already exist to support this?  Thank you very much in advance!


Avoid writing empty license files

This PR avoids the creation of empty `LICENSE` files.


Create metric card for Frugal Score

Proposing metric card for Frugal Score.

@albertvillanova or @lhoestq  -- there are certain aspects that I'm not 100% sure on (such as how exactly the distillation between BertScore and FrugalScore is done) -- so if you find that something isn't clear, please let me know!


Remove unused legacy Beam utils

This PR removes unused legacy custom `WriteToParquet`, once official Apache Beam includes the patch since version 2.22.0:
- Patch PR: https://github.com/apache/beam/pull/11699
- Issue: https://issues.apache.org/jira/browse/BEAM-10022

In relation with:
- #204


Fix BeamWriter output Parquet file

Since now, the `BeamWriter` saved a Parquet file with a simplified schema, where each field value was serialized to JSON. That resulted in Parquet files larger than Arrow files.

This PR:
- writes Parquet file preserving original schema and without serialization, thus avoiding serialization overhead and resulting in a smaller output file size.
- fixes `parquet_to_arrow` function


Dataset viewer issue for McGill-NLP/feedbackQA

Hi @cslizc, thanks for reporting.

I have just forced the refresh of the corresponding cache and the preview is working now.

## Dataset viewer issue for '*McGill-NLP/feedbackQA*'

**Link:** *[link to the dataset viewer page](https://huggingface.co/datasets/McGill-NLP/feedbackQA)*

*short description of the issue*
The dataset can be loaded correctly with `load_dataset` but the preview doesn't work. Error message:

```
Status code:   400
Exception:     Status400Error
Message:       Not found. Maybe the cache is missing, or maybe the dataset does not exist.
```

Am I the one who added this dataset ? Yes


bug

Something isn't working

Now, I can't find any reference to set_progress_bar_enabled in the code.

I think it have been deleted

Hi @virilo,

Please note that since `datasets` version 2.0.0, we have aligned with `transformers` the management of the progress bar (among other things):
- #3897

Now, you should update your code to use `datasets.logging.disable_progress_bar`.

You have more info in our docs: [Logging methods](https://huggingface.co/docs/datasets/package_reference/logging_methods)

One important thing for beginner like me is: from datasets.utils.logging import disable_progress_bar
Do not forget the 'utils' or you will waste a long time like me....

## Describe the bug

datasets.set_progress_bar_enabled(False) not working in datasets v2

## Steps to reproduce the bug
```python
datasets.set_progress_bar_enabled(False)
```

## Expected results
datasets not using any progress bar

## Actual results

AttributeError: module 'datasets' has no attribute 'set_progress_bar_enabled

## Environment info

datasets version 2


Errors in `Train with Datasets` Tensorflow code section on Huggingface.co

Something isn't working

Hi @blackhat-coder, thanks for reporting.

Please note that the `transformers` library updated their data collators API last year (version 4.10.0):
- huggingface/transformers#13105

now requiring to pass `return_tensors` argument at Data Collator instantiation.

And therefore, we also updated in the `datasets` library documentation all the examples using `transformers` data collators.

If you would like to follow our examples, please update your installed `transformers` version:
```
pip install -U transformers
```

## Describe the bug
Hi
### Error 1
Running the Tensforlow code on [Huggingface](https://huggingface.co/docs/datasets/use_dataset) gives a TypeError: __init__() got an unexpected keyword argument 'return_tensors'
### Error 2
`DataCollatorWithPadding` isn't imported

## Steps to reproduce the bug
```python
import tensorflow as tf
from datasets import load_dataset
from transformers import AutoTokenizer
dataset = load_dataset('glue', 'mrpc', split='train')
tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')
dataset = dataset.map(lambda e: tokenizer(e['sentence1'], truncation=True, padding='max_length'), batched=True)
data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="tf")
train_dataset = dataset["train"].to_tf_dataset(
  columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'],
  shuffle=True,
  batch_size=16,
  collate_fn=data_collator,
)
```
This is the same code on Huggingface.co

## Actual results
TypeError: __init__() got an unexpected keyword argument 'return_tensors'

## Environment info
- `datasets` version: 2.0.0
- Platform: Windows-10-10.0.19044-SP0
- Python version: 3.9.7
- PyArrow version: 6.0.0
- Pandas version: 1.4.1
>


Add SacreBLEU Metric Card


Add chrF(++) Metric Card


Close parquet writer properly in `push_to_hub`

@lhoestq / @albertvillanova / @mariosasko
I am facing the same scenario. Let me explain the situation point. I have a glue ETL job

1--> My files are in parquet format and stored in AWS s3.
2--> I am iterating a  loop for a data set where the same file name can occur with diffrent other data.
3--> I read the parquet and saved it in a pandas data frame.
4--> Done some operation on that data frame
5--> upload the updated data frame into the S3 parquet file.  Below are code snippet what I am using to save the updated
         data frame into parquet format and load into S3
         `header_name_column_list = dict(data_frame)
          header_list = []
            for col_id, col_type in header_name_column_list.items():
                header_list.append(pyarrow.field(col_id, pyarrow.string()))
            table_schema = pyarrow.schema(header_list)
            table = pyarrow.Table.from_pandas(data_frame, schema=table_schema, preserve_index=False)
            writer = parquet.ParquetWriter(b_buffer, table.schema)
            writer.write_table(table)
            writer.close()
            b_buffer.seek(0)
            .....
            ....
           self.s3_client.upload_fileobj(
                b_buffer,
                self.bucket,
                file_key,
                ExtraArgs=extra_args)`

But when I executed the glue etl job, the first time it works properly and but in the next iteration, when I try to open the same file got that error.

<html>
<body>
<!--StartFragment-->

INFO:Iot-dsip-de-duplication-job:Dataframe uploaded: s3://abc/2022/07/12/file1_ft_20220714122108.3065_12345.parquet INFO:Iot-dsip-de-duplication-job:Sleep for 60 sec
INFO:Iot-dsip-de-duplication-job:start after sleep
.......................
..........................
..........................
ERROR:Iot-dsip-de-duplication-job:Failed to read data from parquet file s3://abc/2022/07/12/file1_ft_20220714122108.3065_12345.parquet, error is : Invalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.INFO:Iot-dsip-de-duplication-job:Empty dataframe found

<!--EndFragment-->
</body>
</html>

Any clue will be really helpful..I got stuck with this problem.

We don’t call writer.close(), which causes https://github.com/huggingface/datasets/issues/4077. It can happen that we upload the file before the writer is garbage collected and writes the footer.

I fixed this by explicitly closing the parquet writer.

Close https://github.com/huggingface/datasets/issues/4077.


NonMatchingChecksumError for downloading conll2012_ontonotesv5 dataset

duplicate

This issue or pull request already exists

dataset bug

A bug in a dataset script provided in the library

Hi @richarddwang,

Indeed, we have recently updated the loading script of that dataset (and fixed that bug as well):
- #4002

That fix will be available in our next `datasets` library release. In the meantime, you can incorporate that fix by:
- installing `datasets` from our GitHub repo:
```bash
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
- forcing the data files to be redownloaded
```python
ds = load_dataset('conll2012_ontonotesv5', 'english_v4', split="test", download_mode="force_redownload")
```

Feel free to re-open this issue if the problem persists.

Duplicate of:
- #4031

## Steps to reproduce the bug
```python
datasets.load_dataset("conll2012_ontonotesv5",  "english_v12")
```

## Actual results
```
Downloading builder script: 32.2kB [00:00, 9.72MB/s]
Downloading metadata: 20.0kB [00:00, 10.4MB/s]
Downloading and preparing dataset conll2012_ontonotesv5/english_v12 (download: 174.83 MiB, generated: 204.29 MiB, post-processed: Unknown size
, total: 379.12 MiB) to ...
Traceback (most recent call last):                                                                                [315/390]
  File "/home/yisiang/lgtn/conll2012/run.py", line 86, in <module>
    train()
  File "/home/yisiang/lgtn/conll2012/run.py", line 65, in train
    trainer.fit(model, datamodule=dm)
  File "/home/yisiang/miniconda3/envs/ai/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 740, in fit
    self._call_and_handle_interrupt(
  File "/home/yisiang/miniconda3/envs/ai/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 685, in _call_and_handle_inte
rrupt
    return trainer_fn(*args, **kwargs)
  File "/home/yisiang/miniconda3/envs/ai/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 777, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/yisiang/miniconda3/envs/ai/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1131, in _run
    self._data_connector.prepare_data()
  File "/home/yisiang/miniconda3/envs/ai/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 154, in pre
pare_data
    self.trainer.datamodule.prepare_data()
  File "/home/yisiang/miniconda3/envs/ai/lib/python3.9/site-packages/pytorch_lightning/core/datamodule.py", line 474, in wrapped_fn
    fn(*args, **kwargs)
  File "/home/yisiang/lgtn/_abstract_task/data.py", line 43, in prepare_data
    raw_dsets = datasets.load_dataset(**load_dataset_kwargs)
  File "/home/yisiang/miniconda3/envs/ai/lib/python3.9/site-packages/datasets/load.py", line 1687, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/yisiang/miniconda3/envs/ai/lib/python3.9/site-packages/datasets/builder.py", line 605, in download_and_prepare
    self._download_and_prepare(
  File "/home/yisiang/miniconda3/envs/ai/lib/python3.9/site-packages/datasets/builder.py", line 1104, in _download_and_prepare
    super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
  File "/home/yisiang/miniconda3/envs/ai/lib/python3.9/site-packages/datasets/builder.py", line 676, in _download_and_prepare
    verify_checksums(
  File "/home/yisiang/miniconda3/envs/ai/lib/python3.9/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://md-datasets-cache-zipfiles-prod.s3.eu-west-1.amazonaws.com/zmycy7t9h9-1.zip']
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0


Increase max retries for GitHub datasets

As GitHub recurrently raises connectivity issues, this PR increases the number of max retries to request GitHub datasets, as previously done for GitHub metrics:
- #4063

Note that this is a temporary solution, while we decide when and how to load GitHub datasets from the Hub:
- #4059

Fix #2048

Related to:
- #4051
- #3210
- #2787
- #2075
- #2036

CC: @lhoestq


Fix GithubMetricModuleFactory instantiation with None download_config

Recent PR:
- #4063

introduced a potential bug if `GithubMetricModuleFactory` is instantiated with None `download_config`.

This PR add instantiation tests and fix that potential issue.

CC: @lhoestq


bug

Something isn't working

## Describe the bug

When uploading a relatively large image dataset of > 1GB, reloading doesn't work for me, even though pushing to the hub went just fine.

Basically, I do:

```
from datasets import load_dataset

dataset = load_dataset("imagefolder", data_files="path_to_my_files")

dataset.push_to_hub("dataset_name") # works fine, no errors

reloaded_dataset = load_dataset("dataset_name")
```

and it returns:

```
/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.
```

I created a Colab notebook to reproduce my error: https://colab.research.google.com/drive/141LJCcM2XyqprPY83nIQ-Zk3BbxWeahq?usp=sharing


Add ROUGE Metric Card

Add ROUGE metric card.

I've left the 'Values from popular papers' section empty for the time being because I don't know the summarization literature very well and am therefore not sure which paper(s) to pull from (note that the original rouge paper does not seem to present specific values, just correlations with human judgements). Any suggestions on which paper(s) to pull from would be helpful! :)


Add CCAgT dataset

dataset request

Requesting to add a new dataset

vision

Vision datasets

Awesome ! Let us know if you have questions or if we can help ;) I'm assigning you

PS: if possible, please try to not use Google Drive links in your dataset script, since Google Drive has download quotas and is not always reliable.

HI, I was waiting to come out in the second version to do the implementation.

- Paper: https://dx.doi.org/10.2139/ssrn.4126881
- Data: [Data mendelay](http://doi.org/10.17632/wg4bpm33hj.2)

The link of CCAgT dataset is: https://huggingface.co/datasets/lapix/CCAgT

## Adding a Dataset
- **Name:** CCAgT dataset: Images of Cervical Cells with AgNOR Stain Technique
- **Description:** The dataset contains 2540 images (1600x1200 where each pixel is 0.111μm×0.111μm) from three different slides, having at least one nucleus per image. These images are from fields belonging to a sample cervical slide, colored with silver-stained, a method known as Argyrophilic Nucleolar Organizer Regions (AgNOR).
- **Paper:** https://doi.org/10.1109/cbms49503.2020.00110
- **Data:** https://arquivos.ufsc.br/d/373be2177a33426a9e6c/ or https://drive.google.com/drive/u/4/folders/1TBpYCv6S1ydASLauSzcsvO7Wc5O-WUw0
- **Motivation:** This is a unique dataset (because of the stain), for a major health problem, cervical cancer, with real data.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Hi, this is a public version of the dataset that I have been working on, soon we will have another version of this dataset. But until this new version goes out, I thought I would add this dataset here, if it makes sense for the repository. You can assign the task to me if possible


Error in google/xtreme_s dataset card

documentation

Improvements or additions to documentation

Hi @wranai, thanks for reporting.

Please note that the information about language families and groups is taken form the original paper: [XTREME-S: Evaluating Cross-lingual Speech Representations](https://arxiv.org/abs/2203.10752).

If that information is wrong, feel free to contact the paper's authors to suggest that correction.

Just note that Hungarian language (contrary to their geographically surrounding neighbor languages) belongs to the Uralic (languages) family, together with (among others) Finnish, Estonian, some other languages in northern regions of Scandinavia...

**Link:** https://huggingface.co/datasets/google/xtreme_s

Not a big deal but Hungarian is considered an Eastern European language, together with Serbian, Slovak, Slovenian (all correctly categorized; Slovenia is mostly to the West of Hungary, by the way).


Create a metric card for Competition MATH

Proposing metric card for Competition MATH


Add installation instructions to image_process doc

This PR adds the installation instructions for the Image feature to the image process doc.


Loading issue for xuyeliu/notebookCDG dataset

Hi @Jun-jie-Huang,

As the error message says, ".pkl" data files are not supported.

If you would like to share your dataset on the Hub, you would need:
- either to create a Python loading script, that loads the data in any format
- or to transform your data files to one of the supported formats (listed in the error message above: CSV, JSON, Parquet, TXT,...)

You can find the details in our docs:
- How to share a dataset: https://huggingface.co/docs/datasets/share
- How to create a dataset loading script: https://huggingface.co/docs/datasets/dataset_script

Feel free to re-open this issue and ping us if you need further assistance.

## Dataset viewer issue for '*xuyeliu/notebookCDG*'

**Link:** *[link to the dataset viewer page](https://huggingface.co/datasets/xuyeliu/notebookCDG)*

*Couldn't load the xuyeliu/notebookCDG with provided scripts: *
```
from datasets import load_dataset

dataset = load_dataset("xuyeliu/notebookCDG/dataset_notebook.pkl")
```
I get an error message as follows:

FileNotFoundError: Couldn't find a dataset script at /home/code_documentation/code/xuyeliu/notebookCDG/notebookCDG.py or any data file in the same directory. Couldn't find 'xuyeliu/notebookCDG' on the Hugging Face Hub either: FileNotFoundError: Unable to resolve any data file that matches ['**train*'] in dataset repository xuyeliu/notebookCDG with any supported extension ['csv', 'tsv', 'json', 'jsonl', 'parquet', 'txt', 'blp', 'bmp', 'dib', 'bufr', 'cur', 'pcx', 'dcx', 'dds', 'ps', 'eps', 'fit', 'fits', 'fli', 'flc', 'ftc', 'ftu', 'gbr', 'gif', 'grib', 'h5', 'hdf', 'png', 'apng', 'jp2', 'j2k', 'jpc', 'jpf', 'jpx', 'j2c', 'icns', 'ico', 'im', 'iim', 'tif', 'tiff', 'jfif', 'jpe', 'jpg', 'jpeg', 'mpg', 'mpeg', 'msp', 'pcd', 'pxr', 'pbm', 'pgm', 'ppm', 'pnm', 'psd', 'bw', 'rgb', 'rgba', 'sgi', 'ras', 'tga', 'icb', 'vda', 'vst', 'webp', 'wmf', 'emf', 'xbm', 'xpm', 'zip']

Am I the one who added this dataset ? No


Create metric card for seqeval

Proposing metric card for seqeval. Not sure which values to report for Popular papers though.


Add support for metadata files to `imagefolder`

Love it !

+1 to using JSON Lines rather than CSV. I've also seen image datasets for which JSON Lines was used.

A `file_name` column sounds good as well, and it means we could reuse the same name for audio. And ok to check the metadata file by default :)

You suggested to name the file infos.json - since we already have a datasets_infos.json file, maybe it would be nice to have a name for the metadata/annotations that doesn't contain "info" ? (e.g. metadata.json, annotations.json, labels.json)

@lhoestq I've addressed your comments and my TODOs. Additionally, I've updated `encode_nested_example`/`decode_nested_example` to support null values in place of a dictionary (if it's not top-level) since JSON Lines also supports this.

@lhoestq Sure, feel free to add more tests if you have the time.

I created a dedicated test file for `imagefolder`, moved some existing tests there from `test_packaged_modules.py`, and added an end-to-end test of `imagefolder` with metadata. I tested for train split only, and for two splits train and test.

Let me know if the test looks ok to you. I'll add similar tests but with the other structures we support on tuesday

Thanks a lot for working on this! The test looks great :).

Added a test for archives. Will also add a test when the metadata file is not named correctly, and see if we can raise an informative error

This PR adds support for metadata files to `imagefolder` to add an ability to specify image fields other than `image` and `label`, which are inferred from the directory structure in the loaded dataset.

To be parsed as an image metadata file, a file should be named `"info.csv"` and should have the following structure:
```
image_id,some_col1_name,some_col2_name
rel/path/to/image1.jpg,image1_col1_value,image1_col2_value
rel/path/to/image2.jpg,image2_col1_value,image2_col2_value
...
```

This is how the resolution works:
```
- path/to/imagefolder/directory
  - info.csv
  - 10.jpg # referenced as 10.jpg in "info.csv"
  - Cat
    - 0.jpg  # referenced as Cat/0.jpg in "info.csv"
    - 1.jpg  # referenced as Cat/1.jpg in "info.csv"
  - Dog
    - 0.jpg  # referenced as Dog/0.jpg in "info.csv"
    - 1.jpg  # referenced as Dog/1.jpg in "info.csv"
```

Open questions:
1. IMO it makes more sense to store image metadata as JSON Lines than CSV. CSV is sufficient for textual metadata but not the best for representing bounding boxes, for instance. Also, JSON Lines is more strict, which is good in this case (CSV supports various delimiters, the header line is optional, etc., so it's easier to enforce rules on JSON Lines that it's on CSV)
2. A better name for the `image_id` column, which contains image identifiers? Maybe `image_file` or `image_filename`?
3. WDYT about making `with_metadata=True` the default behavior if the loaded repo/directory contains an `info.csv` file?

An example repository: https://huggingface.co/datasets/mariosasko/PetImages. Can be loaded by installing `datasets` from the PR branch and running `load_dataset("mariosasko/PetImages", with_metadata=True)`.

cc: @abhishekkrthakur (this PR should address https://huggingface.slack.com/archives/C02JB9L6JKF/p1645450017434029?thread_ts=1645157416.389499&cid=C02JB9L6JKF)

TODOs:
- [x] Test
- [x] Metadata file nesting
   ```
    - path/to/imagefolder/directory
      - info.csv
      - 10.jpg
      - Cat
        - info.csv  # should have higher precedence in this directory than the top-level info.csv, but we choose the first "eligible" metadata file currently
        - 0.jpg
        - 1.jpg
   ```


Improve out of bounds error message

_The documentation is not available anymore as the PR was closed or merged._

In 1.18.4 with https://github.com/huggingface/datasets/pull/3719 we introduced an error message for users using `select` with out of bounds indices. The message ended up being confusing for some users because it mentioned negative indices, which is not the main use case.

I replaced it with a message that is very similar to the one you get with you try to access a list with an out-of-range index.


Update datasets task tags to align tags with models

Looks good, but I think we are missing some scripts with outdated tags (RedCaps, MNIST, ...).

Just updated the tags of vision datasets :)
We can figure out one for image datasets without labels like PASS - not sure how to name the task for this, maybe `image-fill-mask` (for consistency with language modeling for pretraining) / `masked-auto-encoding` (from ViT). Let's see that in another PR later

**Requires https://github.com/huggingface/datasets/pull/4066 to be merged first**

Following https://github.com/huggingface/datasets/pull/4066 we need to update many dataset tags to use the new ones. This PR takes case of this and is quite big - feel free to review only certain tags if you don't want to spend too much time on it.

Note that the CI will never be green for this PR, because many dataset cards have missing tags or sections, and fixing them is out of scope of this PR (the CI on master will be green anyway)


Tasks alignment with models

Yay! This is exciting! Note that we would probably be able to generate this JSON directly from `huggingface/hub-docs`' `Types.ts` file (cc @osanseviero)

The following issue should make this much easier :smile: https://github.com/huggingface/hub-docs/issues/83

So far I think I've addressed all the comments that I got on slack, but feel free to do a review @osanseviero and let me know if it sounds good to you

It just occurred to me that we should probably restart the `datasets-tagging` space once this is merged to update all the task categories there: https://huggingface.co/spaces/huggingface/datasets-tagging

current automated export is visible at #4154

I updated our `tasks.json` file with the new task taxonomy that is aligned with models.

The rule that defines a task is the following:

**Two tasks are different if and only if the steps of their pipelines** are different, i.e. if they can’t reasonably be implemented using the same coherent code (level of granularity/complexity of the code to be defined - ideally I’d like to say “HF user’s level”) - this is the same definition in `transformers`

I will update the tags of all the datasets in this repository [in another PR](https://github.com/huggingface/datasets/pull/4067) for readability.

Main changes:
- conditional-text-generation is split between summarization, translation, text-generation and text2text-generation
- speech-processing is split into automatic-speech-recognition, audio-classification, etc.
- structure-prediction is renamed token-classification
- abstractive-qa now belongs to text2text-generation

Here is just a simplified YAML dump of `tasks.json`:

```yaml
audio-classification:
- keyword-spotting
- speaker-identification
- speaker-intent-classification
- emotion-recognition
- speaker-language-identification
audio-to-audio: []
automatic-speech-recognition: []
conversational:
- dialogue-generation
feature-extraction: []
fill-mask:
- slot-filling
- masked-language-modeling
image-classification:
- multi-label-image-classification
- multi-class-image-classification
image-segmentation:
- instance-segmentation
- semantic-segmentation
- panoptic-segmentation
image-to-text:
- image-captioning
multiple-choice:
- multiple-choice-qa
- multiple-choice-coreference-resolution
object-detection:
- face-detection
- vehicle-detection
question-answering:
- extractive-qa
- open-domain-qa
- closed-domain-qa
sentence-similarity: []
tabular-classification: []
tabular-to-text:
- rdf-to-text
summarization:
- news-articles-summarization
- news-articles-headline-generation
table-to-text: []
table-question-answering: []
text-classification:
- acceptability-classification
- entity-linking-classification
- fact-checking
- intent-classification
- multi-class-classification
- multi-label-classification
- natural-language-inference
- semantic-similarity-classification
- sentiment-classification
- topic-classification
- semantic-similarity-scoring
- sentiment-scoring
- sentiment-analysis
- hate-speech-detection
- text-scoring
text-generation:
- dialogue-modeling
- language-modeling
text-retrieval:
- document-retrieval
- utterance-retrieval
- entity-linking-retrieval
- fact-checking-retrieval
text-to-image: []
text-to-tabular:
- relation-extraction
- semantic-role-labeling
text-to-speech: []
text2text-generation:
- text-simplification
- explanation-generation
- abstractive-qa
- open-domain-abstractive-qa
- closed-domain-qa
- open-book-qa
- closed-book-qa
time-series-forecasting:
- univariate-time-series-forecasting
- multivariate-time-series-forecasting
token-classification:
- named-entity-recognition
- part-of-speech-tagging
- parsing
- lemmatization
- word-sense-disambiguation
- coreference-resolution
translation: []
visual-question-answering: []
voice-activity-detection: []
zero-shot-classification: []
zero-shot-image-classification: []
reinforcement-learning: []
other: []
```

Feel free to comment and give suggestions, especially if you think we can also align this list with other projects

cc @julien-c @osanseviero @severo @lewtun @yjernite @albertvillanova @mariosasko @polinaeterna


Create metric card for METEOR

Proposing a metric card for METEOR


Contributing MedMCQA dataset

@lhoestq Could you please take a look?
Thank you!!

Hi, thank you for the modifications and suggestions. Please check the changes.

Can you run `make style` to fix the code formatting please ?

Oh and was wrong with the dummy_data.zip file, it must actually be placed at `datasets/medmcqa/dummy/1.1.0/dummy_data.zip` - sorry about that

Can you also set the class label names to `names=["a", "b", "c", "d"]` to make it explicit which label corresponds to each answer ? You might have to regenerate `dataset_infos.json` after that

Hi,

1) Changed the dummy data folder

2) The labels are not ['a', 'b', 'c', 'd'] rather the labels are [1,2,3,4] where 1 represents the 1'st option, 2nd represents 2nd option so on, and its int.

I tried changing to ['a','b','c','d'] and while generating `dataset_infos.json` getting this error :

`ValueError: Class label 4 greater than configured num_classes 4`
Please check.

@lhoestq [lhoestq](https://github.com/lhoestq) Please check

You have this error because we expect the labels to start at 0, not 1. I think you just need to pass `int(data["cop"]) - 1` when generating the examples.

Sorry for the delay in responding btw

@lhoestq I corrected that but here is another issue I am facing while generating `dataset_infos.json`

I am using `" "` if it's test set and otherwise it's the correct option

https://github.com/monk1337/datasets/blob/179f81d48cdd3093302e498babce04c0bf1e33b3/datasets/medmcqa/medmcqa.py#L111
`                    "cop": "" if split == "test" else int(data["cop"]) -1,
`

but while running this command :

`datasets-cli test datasets/medmcqa --save_infos --all_configs
`

giving this error:

```
/content/datasets# datasets-cli test datasets/medmcqa --save_infos --all_configs
Using custom data configuration default
Testing builder 'default' (1/1)
Downloading and preparing dataset med_mcqa/default (download: 52.72 MiB, generated: 128.73 MiB, post-processed: Unknown size, total: 181.46 MiB) to /root/.cache/huggingface/datasets/med_mcqa/default/1.1.0/4c8e418778967b6d9603f79bbfc4fdfbcfffc389664d9aeb85e102cfde418043...
Traceback (most recent call last):
  File "/usr/local/bin/datasets-cli", line 33, in <module>
    sys.exit(load_entry_point('datasets', 'console_scripts', 'datasets-cli')())
  File "/content/datasets/src/datasets/commands/datasets_cli.py", line 33, in main
    service.run()
  File "/content/datasets/src/datasets/commands/test.py", line 162, in run
    try_from_hf_gcs=False,
  File "/content/datasets/src/datasets/builder.py", line 606, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/content/datasets/src/datasets/builder.py", line 1104, in _download_and_prepare
    super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
  File "/content/datasets/src/datasets/builder.py", line 694, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/content/datasets/src/datasets/builder.py", line 1095, in _prepare_split
    example = self.info.features.encode_example(record)
  File "/content/datasets/src/datasets/features/features.py", line 1356, in encode_example
    return encode_nested_example(self, example)
  File "/content/datasets/src/datasets/features/features.py", line 1007, in encode_nested_example
    return {k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in zip_dict(schema, obj)}
  File "/content/datasets/src/datasets/features/features.py", line 1007, in <dictcomp>
    return {k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in zip_dict(schema, obj)}
  File "/content/datasets/src/datasets/features/features.py", line 1052, in encode_nested_example
    return schema.encode_example(obj) if obj is not None else None
  File "/content/datasets/src/datasets/features/features.py", line 897, in encode_example
    example_data = self.str2int(example_data)
  File "/content/datasets/src/datasets/features/features.py", line 854, in str2int
    output.append(self._str2int[str(value)])
KeyError: ''
```

Hey ! You can use this instead:
`"cop": -1 if split == "test" else int(data["cop"]) -1`

@lhoestq Thank you for your assistance, and I have updated the `dataset_infos.json` without any error. All the issues are resolved. Please review and approve if it's ready to merge.

Thanks ! There are two things to fic the CI:
1. run `make style` to fix code formatting
2. fix the dummy_data.zip file. Currently it's created from a directory called "dummy" that contains the JSON file, but it should be called "dummy_data" instead

@lhoestq Please check if anything else needs to be done :)

Let me gently remind you that you can check the CI before pinging reviewers, this way you can know if something needs to be fixed right away.

Right now, if you check the CI, you will see that you didn't fix the code formatting, and that you didn't fix the dummy data.

Let me take a look

_The documentation is not available anymore as the PR was closed or merged._

Hi @lhoestq, I am sorry if I pinged multiple times; I have already corrected the dummy_data file issues and format issue before pinging for the merge request, as you commented last time

_fix the dummy_data.zip file. Currently, it's created from a directory called "dummy" that contains the JSON file, but it should be called "dummy_data" instead._

I fixed the file name and location.

And I also ran the commands last time.

```
make style
flake8 datasets
```
Please let me know if anything else needs to be changed.

Adding MedMCQA dataset ( https://paperswithcode.com/dataset/medmcqa )

**Name**: MedMCQA

**Description**: MedMCQA is a large-scale, Multiple-Choice Question Answering (MCQA) dataset designed to address real-world medical entrance exam questions.

MedMCQA has more than 194k high-quality AIIMS & NEET PG entrance exam MCQs covering 2.4k healthcare topics and 21 medical subjects are collected with an average token length of 12.77 and high topical diversity.

The dataset contains questions about the following topics: Anesthesia, Anatomy, Biochemistry, Dental, ENT, Forensic Medicine (FM), Obstetrics and Gynecology (O&G), Medicine, Microbiology, Ophthalmology, Orthopedics Pathology, Pediatrics, Pharmacology, Physiology,
Psychiatry, Radiology Skin, Preventive & Social Medicine (PSM), and Surgery

**Code**: https://github.com/medmcqa/medmcqa

All files are at place :

**a dataset script**  : medmcqa.py
**a dataset card with tags and information**  : README.md.
**a metadata file** : dataset_infos.json
**a dummy-data file**  : Please help to generate this file, I was facing

` raise JSONDecodeError("Extra data", s, end)`  error


Increase max retries for GitHub metrics

As GitHub recurrently raises connectivity issues, this PR increases the number of max retries to request GitHub metrics.

Related to:
- #3134

Also related to:
- #4059