Duplicated Keys in Taskmaster-2 Dataset

Thanks for reporting, @liaeh.

We are having a look at  it.

I have transferred the discussion to the Community tab of the dataset: https://huggingface.co/datasets/taskmaster2/discussions/1

### Describe the bug

Loading certain splits () of the taskmaster-2 dataset fails because of a DuplicatedKeysError. This occurs for the following domains: `'hotels', 'movies', 'music', 'sports'`. The domains `'flights', 'food-ordering', 'restaurant-search'` load fine.

Output:

### Steps to reproduce the bug

```
from datasets import load_dataset
dataset = load_dataset("taskmaster2", "music")
```

Output:
```
---------------------------------------------------------------------------
DuplicatedKeysError                       Traceback (most recent call last)
File ~/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py:1532, in GeneratorBasedBuilder._prepare_split_single(self, arg)
   [1531](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=1530) example = self.info.features.encode_example(record) if self.info.features is not None else record
-> [1532](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=1531) writer.write(example, key)
   [1533](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=1532) num_examples_progress_update += 1

File ~/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/arrow_writer.py:475, in ArrowWriter.write(self, example, key, writer_batch_size)
    [474](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/arrow_writer.py?line=473) if self._check_duplicates:
--> [475](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/arrow_writer.py?line=474)     self.check_duplicate_keys()
    [476](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/arrow_writer.py?line=475)     # Re-intializing to empty list for next batch

File ~/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/arrow_writer.py:492, in ArrowWriter.check_duplicate_keys(self)
    [486](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/arrow_writer.py?line=485)     duplicate_key_indices = [
    [487](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/arrow_writer.py?line=486)         str(self._num_examples + index)
    [488](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/arrow_writer.py?line=487)         for index, (duplicate_hash, _) in enumerate(self.hkey_record)
    [489](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/arrow_writer.py?line=488)         if duplicate_hash == hash
    [490](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/arrow_writer.py?line=489)     ]
--> [492](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/arrow_writer.py?line=491)     raise DuplicatedKeysError(key, duplicate_key_indices)
    [493](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/arrow_writer.py?line=492) else:

DuplicatedKeysError: Found multiple examples generated with the same key
The examples at index 858, 859 have the key dlg-89174425-d57a-4db7-a92b-165c3bff6735

During handling of the above exception, another exception occurred:

DuplicatedKeysError                       Traceback (most recent call last)
File ~/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py:1541, in GeneratorBasedBuilder._prepare_split_single(self, arg)
   [1540](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=1539) num_shards = shard_id + 1
-> [1541](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=1540) num_examples, num_bytes = writer.finalize()
   [1542](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=1541) writer.close()

File ~/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/arrow_writer.py:563, in ArrowWriter.finalize(self, close_stream)
    [562](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/arrow_writer.py?line=561) if self._check_duplicates:
--> [563](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/arrow_writer.py?line=562)     self.check_duplicate_keys()
    [564](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/arrow_writer.py?line=563)     # Re-intializing to empty list for next batch

File ~/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/arrow_writer.py:492, in ArrowWriter.check_duplicate_keys(self)
    [486](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/arrow_writer.py?line=485)     duplicate_key_indices = [
    [487](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/arrow_writer.py?line=486)         str(self._num_examples + index)
    [488](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/arrow_writer.py?line=487)         for index, (duplicate_hash, _) in enumerate(self.hkey_record)
    [489](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/arrow_writer.py?line=488)         if duplicate_hash == hash
    [490](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/arrow_writer.py?line=489)     ]
--> [492](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/arrow_writer.py?line=491)     raise DuplicatedKeysError(key, duplicate_key_indices)
    [493](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/arrow_writer.py?line=492) else:

DuplicatedKeysError: Found multiple examples generated with the same key
The examples at index 858, 859 have the key dlg-89174425-d57a-4db7-a92b-165c3bff6735

The above exception was the direct cause of the following exception:

DatasetGenerationError                    Traceback (most recent call last)
Cell In[23], line 1
----> 1 dataset = load_dataset("taskmaster2", "music")

File ~/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/load.py:1741, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, num_proc, **config_kwargs)
   [1738](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/load.py?line=1737) try_from_hf_gcs = path not in _PACKAGED_DATASETS_MODULES
   [1740](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/load.py?line=1739) # Download and prepare data
-> [1741](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/load.py?line=1740) builder_instance.download_and_prepare(
   [1742](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/load.py?line=1741)     download_config=download_config,
   [1743](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/load.py?line=1742)     download_mode=download_mode,
   [1744](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/load.py?line=1743)     ignore_verifications=ignore_verifications,
   [1745](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/load.py?line=1744)     try_from_hf_gcs=try_from_hf_gcs,
   [1746](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/load.py?line=1745)     use_auth_token=use_auth_token,
   [1747](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/load.py?line=1746)     num_proc=num_proc,
   [1748](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/load.py?line=1747) )
   [1750](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/load.py?line=1749) # Build dataset for splits
   [1751](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/load.py?line=1750) keep_in_memory = (
   [1752](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/load.py?line=1751)     keep_in_memory if keep_in_memory is not None else is_small_dataset(builder_instance.info.dataset_size)
   [1753](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/load.py?line=1752) )

File ~/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py:822, in DatasetBuilder.download_and_prepare(self, output_dir, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, file_format, max_shard_size, num_proc, storage_options, **download_and_prepare_kwargs)
    [820](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=819)     if num_proc is not None:
    [821](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=820)         prepare_split_kwargs["num_proc"] = num_proc
--> [822](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=821)     self._download_and_prepare(
    [823](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=822)         dl_manager=dl_manager,
    [824](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=823)         verify_infos=verify_infos,
    [825](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=824)         **prepare_split_kwargs,
    [826](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=825)         **download_and_prepare_kwargs,
    [827](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=826)     )
    [828](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=827) # Sync info
    [829](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=828) self.info.dataset_size = sum(split.num_bytes for split in self.info.splits.values())

File ~/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py:1555, in GeneratorBasedBuilder._download_and_prepare(self, dl_manager, verify_infos, **prepare_splits_kwargs)
   [1554](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=1553) def _download_and_prepare(self, dl_manager, verify_infos, **prepare_splits_kwargs):
-> [1555](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=1554)     super()._download_and_prepare(
   [1556](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=1555)         dl_manager, verify_infos, check_duplicate_keys=verify_infos, **prepare_splits_kwargs
   [1557](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=1556)     )

File ~/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py:913, in DatasetBuilder._download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    [909](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=908) split_dict.add(split_generator.split_info)
    [911](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=910) try:
    [912](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=911)     # Prepare split will record examples associated to the split
--> [913](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=912)     self._prepare_split(split_generator, **prepare_split_kwargs)
    [914](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=913) except OSError as e:
    [915](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=914)     raise OSError(
    [916](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=915)         "Cannot find data file. "
    [917](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=916)         + (self.manual_download_instructions or "")
    [918](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=917)         + "\nOriginal error:\n"
    [919](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=918)         + str(e)
    [920](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=919)     ) from None

File ~/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py:1396, in GeneratorBasedBuilder._prepare_split(self, split_generator, check_duplicate_keys, file_format, num_proc, max_shard_size)
   [1394](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=1393) gen_kwargs = split_generator.gen_kwargs
   [1395](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=1394) job_id = 0
-> [1396](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=1395) for job_id, done, content in self._prepare_split_single(
   [1397](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=1396)     {"gen_kwargs": gen_kwargs, "job_id": job_id, **_prepare_split_args}
   [1398](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=1397) ):
   [1399](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=1398)     if done:
   [1400](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=1399)         result = content

File ~/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py:1550, in GeneratorBasedBuilder._prepare_split_single(self, arg)
   [1548](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=1547)     if isinstance(e, SchemaInferenceError) and e.__context__ is not None:
   [1549](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=1548)         e = e.__context__
-> [1550](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=1549)     raise DatasetGenerationError("An error occurred while generating the dataset") from e
   [1552](file:///home/user/repos/tts-dataset/tts-dataset/venv/lib/python3.9/site-packages/datasets/builder.py?line=1551) yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_shards, shard_lengths)

DatasetGenerationError: An error occurred while generating the dataset
```

### Expected behavior

Loads the dataset

### Environment info

- `datasets` version: 2.7.1
- Platform: Linux-5.13.0-40-generic-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyArrow version: 10.0.1
- Pandas version: 1.5.2


Raise error for `.tar` archives in the same way as for `.tar.gz` and `.tgz` in `_get_extraction_protocol`

_The documentation is not available anymore as the PR was closed or merged._

Currently `download_and_extract` doesn't throw an error when it is used with files with `.tar` extension in streaming mode because `_get_extraction_protocol` doesn't do it (like it does for  `tar.gz` and `tgz`). `_get_extraction_protocol` returns formatted url as if we support tar protocol but we don't.
That means that in dataset scripts `.tar` files would be attempted to load and fail during examples generation (after `download_and_extract` execution). So this PR raises error for `tar` files too.


Fix loading from HF GCP cache

> Do you know why this stopped working?

It comes from the changes in https://github.com/huggingface/datasets/pull/5107/files#diff-355ae5c229f95f86895404b72378ecd6e966c41cbeebb674af6fe6e9611bc126

As reported in https://discuss.huggingface.co/t/error-loading-wikipedia-dataset/26599/4 it's not possible to download a cached version of Wikipedia from the HF GCP cache

I fixed it and added an integration test (runs in 10sec)


[Extract] Place the lock file next to the destination directory

Previously it was placed next to the archive to extract, but the archive can be in a read-only directory as noticed in https://github.com/huggingface/datasets/issues/5295

Therefore I moved the lock location to be next to the destination directory, which is required to have write permissions


Fix Text sample_by paragraph

Fix #5316.


Origin/fix missing features error

please review :) @lhoestq @ola13 thankoo

Thanks :) I just updated the test to make sure it works even when there's a column missing, and did a minor change to json.py to add the missing columns for the other kinds of JSON files as well (I moved the code to`self._cast_table`)

Thanks Unso! If @lhoestq is happy then I'm also happy :D

When I noticed the ping, this PR had already been merged...

Luckily, PyArrow's `read_json` behaves the same when `explicit_schema` is given via `ParseOptions`, so I'm okay with this change (our JSON loader doesn't use `read_json` for decoding JSON in some scenarios, so this manual approach is the right one).

This fixes the problem of when the dataset_load function reads a function with "features" provided but some read batches don't have columns that later show up. For instance, the provided "features" requires columns A,B,C but only columns B,C show. This fixes this by adding the column A with nulls.


`ImageFolder` performs poorly with large datasets

Hi ! ImageFolder is made for small scale datasets indeed. For large scale image datasets you better group your images in TAR archives or Arrow/Parquet files. This is true not just for ImageFolder loading performance, but also because having millions of files is not ideal for your filesystem or when moving the data around.

Option 1. use TAR archives

I'd suggest you to take a look at how we load [Imagenet](https://huggingface.co/datasets/imagenet-1k/tree/main) for example. The dataset is sharded in multiple TAR archives and there is a [script](https://huggingface.co/datasets/imagenet-1k/blob/main/imagenet-1k.py) that iterates over the archives to load the images.

Option 2. use Arrow/Parquet

You can load your images as an Arrow Dataset with
```python
from datasets import Dataset, Image, load_from_disk, load_dataset

ds = Dataset.from_dict({"image": list(glob.glob("path/to/dir/**/*.jpg"))})

def add_metadata(example):
    ...

ds = ds.map(add_metadata, num_proc=16)  # num_proc for multiprocessing
ds = ds.cast_column("image", Image())

# save as Arrow locally
ds.save_to_disk("output_dir")
reloaded = load_from_disk("output_dir")

# OR save as Parquet on the HF Hub
ds.push_to_hub("username/dataset_name")
reloaded = load_dataset("username/dataset_name")
# reloaded = load_dataset("username/dataset_name", num_proc=16)  # to use multiprocessing
```

PS: maybe we can actually have something similar to ImageFolder but for image archives at one point ?

@lhoestq Thanks!

Perhaps it'd be worth adding a note on the documentation that `ImageFolder` is not intended for large datasets? This limitation is not intuitively obvious to someone who has not used it before, I think.

Thanks for the feedback @salieri! I opened #5329 to make it clear `ImageFolder` is not intended for large datasets. Please feel free to comment if you have any other feedback! 🙂

### Describe the bug

While testing image dataset creation, I'm seeing significant performance bottlenecks with imagefolders when scanning a directory structure with large number of images.

## Setup
* Nested directories (5 levels deep)
* 3M+ images
* 1 `metadata.jsonl` file

## Performance Degradation Point 1

Degradation occurs because [`get_data_files_patterns`](https://github.com/huggingface/datasets/blob/main/src/datasets/data_files.py#L231-L243) runs the exact same scan for many different types of patterns, and there doesn't seem to be a way to easily limit this. It's controlled by the definition of [`ALL_DEFAULT_PATTERNS`](https://github.com/huggingface/datasets/blob/main/src/datasets/data_files.py#L82-L85).

One scan with 3M+ files takes about 10-15 minutes to complete on my setup, so having those extra scans really slows things down – from 10 minutes to 60+. Most of the scans return no matches, but they still take a significant amount of time to complete – hence the poor performance.

As a side effect, when this scan is run on 3M+ image files, Python also consumes up to 12 GB of RAM, which is not ideal.

## Performance Degradation Point 2

The second performance bottleneck is in [`PackagedDatasetModuleFactory.get_module`](https://github.com/huggingface/datasets/blob/d7dfbc83d68e87ba002c5eb2555f7a932e59038a/src/datasets/load.py#L707-L711), which calls `DataFilesDict.from_local_or_remote`.

It runs for a long time (60min+), consuming significant amounts of RAM – even more than the point 1 above. Based on `iostat -d 2`, it performs **zero** disk operations, which to me suggests that there is a code based bottleneck there that could be sorted out.

### Steps to reproduce the bug

```python
from datasets import load_dataset
import os
import huggingface_hub

dataset = load_dataset(
  'imagefolder',
  data_dir='/some/path',
  # just to spell it out:
  split=None,
  drop_labels=True,
  keep_in_memory=False
)

dataset.push_to_hub('account/dataset', private=True)
```

### Expected behavior

While it's certainly possible to write a custom loader to replace `ImageFolder` with, it'd be great if the off-the-shelf `ImageFolder` would by default have a setup that can scale to large datasets.

Or perhaps there could be a dedicated loader just for large datasets that trades off flexibility for performance? As in, maybe you have to define explicitly how you want it to work rather than it trying to guess your data structure like `_get_data_files_patterns()` does?

### Environment info

- `datasets` version: 2.7.1
- Platform: Linux-4.14.296-222.539.amzn2.x86_64-x86_64-with-glibc2.2.5
- Python version: 3.7.10
- PyArrow version: 10.0.1
- Pandas version: 1.3.5


Bug in sample_by="paragraph"

Thanks for reporting, @adampauls.

We are having a look at it.

### Describe the bug

I think [this line](https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/text/text.py#L96) is wrong and should be `batch = f.read(self.config.chunksize)`. Otherwise it will never terminate because even when `f` is finished reading, `batch` will still be truthy from the last iteration.

### Steps to reproduce the bug

```
> cat test.txt
a b c

d e f
````

```python
>>> import datasets
>>> datasets.load_dataset("text", data_files={"train":"test.txt"}, sample_by="paragraph")
```

This will go on forever.

### Expected behavior

Terminates very quickly.

### Environment info

`version = "2.6.1"` but I think the bug is still there on main.


Adding new splits to a dataset script with existing old splits info in metadata's `dataset_info` fails

bug

Something isn't working

EDIT:
I think in this case, the metadata files (either README or JSON) should not be read (i.e. `self.info.splits` should be None).

One idea:
- I think ideally we should set this behavior when we pass `--save_info` to the CLI `test`
- However, currently, the builder is unaware of this: `save_info` arg is not passed to it

> I think in this case

@albertvillanova  You mean in cases when the script was changed?

I suggest that we:
* add a check on the slice (like 'split_name[n%]) kind of format here: https://github.com/huggingface/datasets/blob/main/src/datasets/splits.py#L523 to catch things like this.
* Error here happens before splits verification, but in `_prepare_split`, and `_prepare_split` doesn't perform any verification and don't know about it. so we can pass this parameter and take splits from `split_generator`, not from `split.info` in case when `verify_infos` is False
* we can check if split **names** from split_generators and self.info.splits are the same **before** preparing splits (if `verify_info=True`) so that we don't spend time on generating unwanted data.
* provide some user-friendly warnings about `ignore_verifications` parameter so that users know that if something is not matching they can ignore it

I started it here: https://github.com/huggingface/datasets/pull/5327/files

What do you think @albertvillanova ?

I edited my previous comment:
- First I proposed setting `self.info.splits` to None when `ignore_verifications=True`
  - I thought it was the easiest implementation because `ignore_verifications` is passed to `DatasetBuilder.download_and_prepare`
  - However, afterwards, I realized this might not be a good idea for this use case:
    - A user wants to optimize the loading of the dataset, and passes `ignore_verifications=False` to avoid all the verifications
      - In this case, we want `self.info.splits` to be read from metadata file
- Then, I thought that it might be better to set `self.info.splits` to None when we pass `--save_info` to the CLI test: if we are going to save the info to the metadata file, it makes no sense to read the info from the metadata file
  - This implementation is not so easy because the Builder knows nothing about `--save_info`

I agree with you there are 2 things to be addressed here:
- One is what I have just commented: `self.info.splits` should be None in this case
- The other, a validation should be implemented when calling `make_file_instructions` and/or `SplitDict.__getitem__`, so that when passing "training" to it, we get a more descriptive error other than `TypeError: expected str, bytes or os.PathLike object, not NoneType`

### Describe the bug

If you first create a custom dataset with a specific set of splits, generate metadata with `datasets-cli test ... --save_info`, then change your script to include more splits, it fails.

That's what happened in https://huggingface.co/datasets/mrdbourke/food_vision_199_classes/discussions/2#6385fd1269634850f8ddff48.

### Steps to reproduce the bug

1. create a dataset with a custom split that returns, for example, only `"train"` split in `_splits_generators'`. specifically, if really want to reproduce, copy `https://huggingface.co/datasets/mrdbourke/food_vision_199_classes/blob/main/food_vision_199_classes.py
2. run `datasets-cli test dataset_script.py --save_info --all_configs` - this would generate metadata yaml in `README.md` that would contain info about splits, for example, like this:
```
  splits:
  - name: train
    num_bytes: 2973286
    num_examples: 19747
```
3. make changes to your script so that it returns another set of splits, for example, `"train"` and `"test"` (uncomment [these lines](https://huggingface.co/datasets/mrdbourke/food_vision_199_classes/blob/main/food_vision_199_classes.py#L271))
4. run `load_dataset` and get the following error:
```python
Traceback (most recent call last):
  File "/home/daniel/code/pytorch/env/bin/datasets-cli", line 8, in <module>
    sys.exit(main())
  File "/home/daniel/code/pytorch/env/lib/python3.8/site-packages/datasets/commands/datasets_cli.py", line 39, in main
    service.run()
  File "/home/daniel/code/pytorch/env/lib/python3.8/site-packages/datasets/commands/test.py", line 141, in run
    builder.download_and_prepare(
  File "/home/daniel/code/pytorch/env/lib/python3.8/site-packages/datasets/builder.py", line 822, in download_and_prepare
    self._download_and_prepare(
  File "/home/daniel/code/pytorch/env/lib/python3.8/site-packages/datasets/builder.py", line 1555, in _download_and_prepare
    super()._download_and_prepare(
  File "/home/daniel/code/pytorch/env/lib/python3.8/site-packages/datasets/builder.py", line 913, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/daniel/code/pytorch/env/lib/python3.8/site-packages/datasets/builder.py", line 1356, in _prepare_split
    split_info = self.info.splits[split_generator.name]
  File "/home/daniel/code/pytorch/env/lib/python3.8/site-packages/datasets/splits.py", line 525, in __getitem__
    instructions = make_file_instructions(
  File "/home/daniel/code/pytorch/env/lib/python3.8/site-packages/datasets/arrow_reader.py", line 111, in make_file_instructions
    name2filenames = {
  File "/home/daniel/code/pytorch/env/lib/python3.8/site-packages/datasets/arrow_reader.py", line 112, in <dictcomp>
    info.name: filenames_for_dataset_split(
  File "/home/daniel/code/pytorch/env/lib/python3.8/site-packages/datasets/naming.py", line 78, in filenames_for_dataset_split
    prefix = filename_prefix_for_split(dataset_name, split)
  File "/home/daniel/code/pytorch/env/lib/python3.8/site-packages/datasets/naming.py", line 57, in filename_prefix_for_split
    if os.path.basename(name) != name:
  File "/home/daniel/code/pytorch/env/lib/python3.8/posixpath.py", line 143, in basename
    p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType
```
5. bonus: try to regenerate metadata in `README.md` with `datasets-cli` as in step 2 and get the same error.

This is because `dataset.info.splits` contains only `"train"` split so when we are doing `self.info.splits[split_generator.name]` it tries to infer smth like `info.splits['train[50%]']` and that's not the case and it fails.

### Expected behavior

to be discussed?

This can be solved by removing splits information from metadata file first. But I wonder if there is a better way.

### Environment info

- Datasets version: 2.7.1
- Python version: 3.8.13


Datasets: classification_report() got an unexpected keyword argument 'suffix'

This seems similar to https://github.com/huggingface/datasets/issues/2512 Can you try to update seqeval ?

@JonathanAlis also note that the metrics are deprecated in our `datasets` library.

Please, use the new library 🤗 Evaluate instead: https://huggingface.co/docs/evaluate

https://github.com/huggingface/datasets/blob/main/metrics/seqeval/seqeval.py

> import datasets
predictions = [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
references = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
seqeval = datasets.load_metric("seqeval")
results = seqeval.compute(predictions=predictions, references=references)
print(list(results.keys()))
print(results["overall_f1"])
print(results["PER"]["f1"])

It raises the error:

> TypeError: classification_report() got an unexpected keyword argument 'suffix'

For context, versions on my pip list -v

> datasets                 1.12.1
seqeval                  1.2.2


Fix description of streaming in the docs

We say that "the data is being downloaded progressively" which is not true, it's just streamed, so I fixed it. Probably I missed some other places where it is written?

Also changed docstrings for `StreamingDownloadManager`'s `download` and `extract` to reflect the same, as these docstrings are displayed in the documentation cc @lhoestq


Add DatasetDict.to_pandas

The current implementation is what I had in mind, i.e. concatenate all splits by default.

However, I think most tabular datasets would come as a single split. So for that usecase, it wouldn't change UX if we raise when there are more than one splits.

And for multiple splits, the user either passes a list, or they can pass `splits="all"` to have all splits concatenated.

I think it's better to raise an error in cases when there are multiple splits but no split is specified so that users know for sure with which data they are working. I imagine a case when a user loads a dataset that they don't know much about (like what splits it has), and if they get a concatenation of everything, it might lead to incorrect processing or interpretations and it would be hard to notice it.
("explicit is better than implicit")

I just changed to raise an error if there are multiple splits. The error shows an example of how to choose a split to convert.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5312). All of your documentation changes will be reflected on that endpoint.

Thanks for the review, I've updated the type hint and added a line to raise an error on bad splits :)

Merging https://github.com/huggingface/datasets/pull/5301 would eliminate the need for this PR, no?

In the meantime, I find the current API cleaner.

This solution is simpler than https://github.com/huggingface/datasets/pull/5301 and covers most cases for tabular datasets, so I'm in favor of merging this one and put https://github.com/huggingface/datasets/pull/5301 on stand by

Let me know if it sounds good to you @mariosasko @albertvillanova :)

I'm still not convinced. If `DatasetDict` needs this method and there is no other way, then IMO it would make more sense to return a dictionary with the splits converted to `pd.DataFrame`.

@mariosasko the issue we're dealing with is that in tabular scenarios, we often don't have splits in the dataset, and imposing that concept to people dealing with the library hampers adoption.

@adrinjalali This PR proposes a solution inconsistent with the existing API (in other words, a solution that clutters our API 🙂). Moreover, our library primarily focuses on larger-than-RAM datasets, and tabular datasets don't (directly) fall into this group.

Instead of the temporary "fix" proposed here, it makes much more sense to align `load_dataset` with both tabular and DL workflows "in a consistent way", so I suggest we continue our discussion from https://github.com/huggingface/datasets/issues/5189 to have this resolved by version 3.0.

From discussions in https://github.com/huggingface/datasets/issues/5189, for tabular data it doesn't really make sense to have to do
```python
df = load_dataset(...)["train"].to_pandas()
```
because many datasets are not split.

In this PR I added `to_pandas` to `DatasetDict` which returns the DataFrame:

If there's only one split, you don't need to specify the split name:
```python
df = load_dataset(...).to_pandas()
```
EDIT: and if a dataset has multiple splits:
```python
df = load_dataset(...).to_pandas(splits=["train", "test"])
# or
df = load_dataset(...).to_pandas(splits="all")

# raises an error because you need to select the split(s) to convert
load_dataset(...).to_pandas()
```

I do have one question though @merveenoyan @adrinjalali @mariosasko:

Should we raise an error if there are multiple splits and ask the user to choose one explicitly ?


Add `features` param to `IterableDataset.map`

_The documentation is not available anymore as the PR was closed or merged._

## Description

As suggested by @lhoestq in #3888, we should be adding the param `features` to `IterableDataset.map` so that the features can be preserved (not turned into `None` as that's the default behavior) whenever the user passes those as param, so as to be consistent with `Dataset.map`, as it provides the `features` param so that those are not inferred by default, but specified by the user, and later validated by `ArrowWriter`.

This is internally handled already by the functions relying on `IterableDataset.map` such as `rename_column`, `rename_columns`, and `remove_columns` as described in #5287.

## Usage Example

```python
from datasets import load_dataset, Features

ds = load_dataset("rotten_tomatoes", split="validation", streaming=True)
print(ds.info.features)
ds = ds.map(
    lambda x: {"target": x["label"]},
    features=Features(
        {"target": ds.info.features["label"], "label": ds.info.features["label"], "text": ds.info.features["text"]}
    ),
)
print(ds.info.features)
```


Support xPath for Windows pathnames

This PR implements a string representation of `xPath`, which is valid for local paths (also windows) and remote URLs.

Additionally, some `os.path` methods are fixed for remote URLs on Windows machines.

Now, on Windows machines:
```python
In [2]: str(xPath("C:\\dir\\file.txt"))
Out[2]: 'C:\\dir\\file.txt'
In [3]: str(xPath("http://domain.com/file.txt"))
Out[3]: 'http://domain.com/file.txt'
```


Close stream in `ArrowWriter.finalize` before inference error

Ensure the file stream is closed in `ArrowWriter.finalize` before raising the `SchemaInferenceError` to avoid the `PermissionError` on Windows in `incomplete_dir`'s `shutil.rmtree`.


Support `topdown` parameter in `xwalk`

Add support for the `topdown` parameter in `xwalk` when `fsspec>=2022.11.0` is installed.


Use correct dataset type in `from_generator` docs

Use the correct dataset type in the `from_generator` docs (example with sharding).


Can't use custom feature description when loading a dataset

Forgot to actually convert the feature dict to a Feature object. Closing.

### Describe the bug

I have created a feature dictionary to describe my datasets' column types, to use when loading the dataset, following [the doc](https://huggingface.co/docs/datasets/main/en/about_dataset_features). It crashes at dataset load.

### Steps to reproduce the bug

```python
# Creating features
task_list = [f"motif_G{i}" for i in range(19, 53)]

features = {t: Sequence(feature=Value(dtype="float64")) for t in task_list}
for col_name in ["class_label"]:
    features[col_name] = Sequence(feature=Value(dtype="int64"))
for col_name in ["num_nodes"]:
    features[col_name] = Value(dtype="int64")
for col_name in ["num_bridges", "num_cycles", "avg_shortest_path_len"]:
    features[col_name] = Sequence(feature=Value(dtype="float64"))
for col_name in ["edge_attr", "node_feat", "edge_index"]:
    features[col_name] = Sequence(feature=Sequence(feature=Value(dtype="int64")))
print(features)

dataset = load_dataset(path=f"graphs-datasets/unbalanced-motifs-500K", split="train", features=features)
```

Last line will crash and say 'TypeError: argument of type 'Sequence' is not iterable'.

Full stack:
```
Traceback (most recent call last):
  File "pretrain_tokengt.py", line 131, in <module>
    main(output_folder = "../workspace/pretraining",
  File "pretrain_tokengt.py", line 52, in main
    dataset = load_dataset(path=f"graphs-datasets/{dataset_name}", split="train", features=features)
  File "huggingface_env/lib/python3.8/site-packages/datasets/load.py", line 1718, in load_dataset
    builder_instance = load_dataset_builder(
  File "huggingface_env/lib/python3.8/site-packages/datasets/load.py", line 1514, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "huggingface_env/lib/python3.8/site-packages/datasets/builder.py", line 321, in __init__
    info.update(self._info())
  File "huggingface_env/lib/python3.8/site-packages/datasets/packaged_modules/json/json.py", line 62, in _info
    return datasets.DatasetInfo(features=self.config.features)
  File "<string>", line 20, in __init__
  File "huggingface_env/lib/python3.8/site-packages/datasets/info.py", line 155, in __post_init__
    self.features = Features.from_dict(self.features)
  File "huggingface_env/lib/python3.8/site-packages/datasets/features/features.py", line 1599, in from_dict
    obj = generate_from_dict(dic)
  File "huggingface_env/lib/python3.8/site-packages/datasets/features/features.py", line 1282, in generate_from_dict
    return {key: generate_from_dict(value) for key, value in obj.items()}
  File "huggingface_env/lib/python3.8/site-packages/datasets/features/features.py", line 1282, in <dictcomp>
    return {key: generate_from_dict(value) for key, value in obj.items()}
  File "huggingface_env/lib/python3.8/site-packages/datasets/features/features.py", line 1281, in generate_from_dict
    if "_type" not in obj or isinstance(obj["_type"], dict):
TypeError: argument of type 'Sequence' is not iterable
```

### Expected behavior

For it not to crash.

### Environment info

- `datasets` version: 2.7.1
- Platform: Linux-5.14.0-1054-oem-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Dataset joelito/mc4_legal does not work with multiple files

Thanks for reporting @JoelNiklaus.

Please note that since we moved all dataset loading scripts to the Hub, the issues and pull requests relative to specific datasets are directly handled on the Hub, in their Community tab. I'm transferring this issue there: https://huggingface.co/datasets/joelito/mc4_legal/discussions

I am also having a look at the bug in your script.

Issue transferred to: https://huggingface.co/datasets/joelito/mc4_legal/discussions/1

### Describe the bug

The dataset https://huggingface.co/datasets/joelito/mc4_legal works for languages like bg with a single data file, but not for languages with multiple files like de. It shows zero rows for the de dataset.

joelniklaus@Joels-MacBook-Pro ~/N/P/C/L/p/m/mc4_legal (main) [1]> python test_mc4_legal.py                                                                                (debug)
Found cached dataset mc4_legal (/Users/joelniklaus/.cache/huggingface/datasets/mc4_legal/de/0.0.0/fb6952a097180f8c936e2a7605525ff670354a344fc1a2c70107684d3f7cb02f)
Dataset({
    features: ['index', 'url', 'timestamp', 'matches', 'text'],
    num_rows: 0
})
joelniklaus@Joels-MacBook-Pro ~/N/P/C/L/p/m/mc4_legal (main)> python test_mc4_legal.py                                                                                (debug)
Downloading and preparing dataset mc4_legal/bg to /Users/joelniklaus/.cache/huggingface/datasets/mc4_legal/bg/0.0.0/fb6952a097180f8c936e2a7605525ff670354a344fc1a2c70107684d3f7cb02f...
Downloading data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1240.55it/s]
Dataset mc4_legal downloaded and prepared to /Users/joelniklaus/.cache/huggingface/datasets/mc4_legal/bg/0.0.0/fb6952a097180f8c936e2a7605525ff670354a344fc1a2c70107684d3f7cb02f. Subsequent calls will reuse this data.
Dataset({
    features: ['index', 'url', 'timestamp', 'matches', 'text'],
    num_rows: 204
})

### Steps to reproduce the bug

import datasets
from datasets import load_dataset, get_dataset_config_names

language = "bg"

test = load_dataset("joelito/mc4_legal", language, split='train')

### Expected behavior

It should display the correct number of rows for the de dataset which should be a large number (thousands or more).

### Environment info

Package                  Version
------------------------ --------------
absl-py                  1.3.0
aiohttp                  3.8.1
aiosignal                1.2.0
astunparse               1.6.3
async-timeout            4.0.2
attrs                    22.1.0
beautifulsoup4           4.11.1
blinker                  1.4
blis                     0.7.8
Bottleneck               1.3.4
brotlipy                 0.7.0
cachetools               5.2.0
catalogue                2.0.7
certifi                  2022.5.18.1
cffi                     1.15.1
chardet                  4.0.0
charset-normalizer       2.1.0
click                    8.0.4
conllu                   4.5.2
cryptography             38.0.1
cymem                    2.0.6
datasets                 2.6.1
dill                     0.3.5.1
docker-pycreds           0.4.0
fasttext                 0.9.2
fasttext-langdetect      1.0.3
filelock                 3.0.12
flatbuffers              20210226132247
frozenlist               1.3.0
fsspec                   2022.5.0
gast                     0.4.0
gcloud                   0.18.3
gitdb                    4.0.9
GitPython                3.1.27
google-auth              2.9.0
google-auth-oauthlib     0.4.6
google-pasta             0.2.0
googleapis-common-protos 1.57.0
grpcio                   1.47.0
h5py                     3.7.0
httplib2                 0.21.0
huggingface-hub          0.8.1
idna                     3.4
importlib-metadata       4.12.0
Jinja2                   3.1.2
joblib                   1.0.1
keras                    2.9.0
Keras-Preprocessing      1.1.2
langcodes                3.3.0
lxml                     4.9.1
Markdown                 3.3.7
MarkupSafe               2.1.1
mkl-fft                  1.3.1
mkl-random               1.2.2
mkl-service              2.4.0
multidict                6.0.2
multiprocess             0.70.13
murmurhash               1.0.7
numexpr                  2.8.1
numpy                    1.22.3
oauth2client             4.1.3
oauthlib                 3.2.1
opt-einsum               3.3.0
packaging                21.3
pandas                   1.4.2
pathtools                0.1.2
pathy                    0.6.1
pip                      21.1.2
preshed                  3.0.6
promise                  2.3
protobuf                 4.21.9
psutil                   5.9.1
pyarrow                  8.0.0
pyasn1                   0.4.8
pyasn1-modules           0.2.8
pybind11                 2.9.2
pycountry                22.3.5
pycparser                2.21
pydantic                 1.8.2
PyJWT                    2.4.0
pylzma                   0.5.0
pyOpenSSL                22.0.0
pyparsing                3.0.4
PySocks                  1.7.1
python-dateutil          2.8.2
pytz                     2021.3
PyYAML                   6.0
regex                    2021.4.4
requests                 2.28.1
requests-oauthlib        1.3.1
responses                0.18.0
rsa                      4.8
sacremoses               0.0.45
scikit-learn             1.1.1
scipy                    1.8.1
sentencepiece            0.1.96
sentry-sdk               1.6.0
setproctitle             1.2.3
setuptools               65.5.0
shortuuid                1.0.9
six                      1.16.0
smart-open               5.2.1
smmap                    5.0.0
soupsieve                2.3.2.post1
spacy                    3.3.1
spacy-legacy             3.0.9
spacy-loggers            1.0.2
srsly                    2.4.3
tabulate                 0.8.9
tensorboard              2.9.1
tensorboard-data-server  0.6.1
tensorboard-plugin-wit   1.8.1
tensorflow               2.9.1
tensorflow-estimator     2.9.0
termcolor                2.1.0
thinc                    8.0.17
threadpoolctl            3.1.0
tokenizers               0.12.1
torch                    1.13.0
tqdm                     4.64.0
transformers             4.20.1
typer                    0.4.1
typing-extensions        4.3.0
Unidecode                1.3.6
urllib3                  1.26.12
wandb                    0.12.20
wasabi                   0.9.1
web-anno-tsv             0.0.1
Werkzeug                 2.1.2
wget                     3.2
wheel                    0.35.1
wrapt                    1.14.1
xxhash                   3.0.0
yarl                     1.8.1
zipp                     3.8.0

Python 3.8.10


timit_asr doesn't load the test split.

The [timit_asr.py](https://huggingface.co/datasets/timit_asr/blob/main/timit_asr.py) script iterates over the WAV files per split directory using this:
```python
wav_paths = sorted(Path(data_dir).glob(f"**/{split}/**/*.wav"))
wav_paths = wav_paths if wav_paths else sorted(Path(data_dir).glob(f"**/{split.upper()}/**/*.WAV"))
```

Can you check that there is a directory named "test" somewhere in your timit data directory ?

### Describe the bug

When I use the function ```timit = load_dataset('timit_asr', data_dir=data_dir)```, it only loads train split, not test split.

I tried to change the directory and filename to lower case to upper case for the test split, but it does not work at all.

```python
DatasetDict({
    train: Dataset({
        features: ['file', 'audio', 'text', 'phonetic_detail', 'word_detail', 'dialect_region', 'sentence_type', 'speaker_id', 'id'],
        num_rows: 4620
    })
    test: Dataset({
        features: ['file', 'audio', 'text', 'phonetic_detail', 'word_detail', 'dialect_region', 'sentence_type', 'speaker_id', 'id'],
        num_rows: 0
    })
})
```

The directory structure of both splits are same. (DIALECT_REGION / SPEAKER_CODE / DATA_FILES)

### Steps to reproduce the bug

1. just use ```timit = load_dataset('timit_asr', data_dir=data_dir)```

### Expected behavior

```python
DatasetDict({
    train: Dataset({
        features: ['file', 'audio', 'text', 'phonetic_detail', 'word_detail', 'dialect_region', 'sentence_type', 'speaker_id', 'id'],
        num_rows: 4620
    })
    test: Dataset({
        features: ['file', 'audio', 'text', 'phonetic_detail', 'word_detail', 'dialect_region', 'sentence_type', 'speaker_id', 'id'],
        num_rows: 1680
    })
})
```

### Environment info

- ubuntu 20.04
- python 3.9.13
- datasets 2.7.1