Write token isn't working while pushing to datasets

### Describe the bug

<img width="1001" alt="Screenshot 2024-05-01 at 3 37 06 AM" src="https://github.com/huggingface/datasets/assets/130903099/00fcf12c-fcc1-4749-8592-d263d4efcbcc">

As you can see I logged in to my account and the write token is valid.

But I can't upload on my main account and I am getting that error. It was okay on my test account at first try.

(I refreshed the token, tried a new token but still doesn't work)

### Steps to reproduce the bug

1. I loaded a dataset.

2. I logged in using both cli and huggingface_hub

3. I pushed to my down dataset
(It went well without any issues on my test account)

### Expected behavior

It should have gone smoothly and this is not even my first time uploading to huggingface datasets

### Environment info

colab, dataset (tried multiple versions)


load_dataset('emotion')    UnicodeDecodeError

### Describe the bug

**emotions = load_dataset('emotion')**

_UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte_

### Steps to reproduce the bug

load_dataset('emotion')

### Expected behavior

succese

### Environment info

py3.10
transformers  4.41.0.dev0
datasets     2.19.0


Problem loading voxpopuli dataset

@Namangarg110  @mohsen-goodarzi The bug appears because the number of urls is less than 16 and the algorithm is meant to work on the previously created mode for a single url as stated on line 314: https://github.com/huggingface/datasets/blob/1bf8a46cc7b096d5c547ea3794f6a4b6c31ea762/src/datasets/download/download_manager.py#L314

In addition, previously `map_nested` function was supported without batching and it is meant to be the default performance.

One of the shortest walk-arounds would be changing the part of the manager with the current setting:
```
        if len(url_or_urls) >= 16:
            download_func = partial(self._download_batched, download_config=download_config)
        else:
            download_func = partial(self._download_single, download_config=download_config)

        start_time = datetime.now()
        with stack_multiprocessing_download_progress_bars():
            downloaded_path_or_paths = map_nested(
                download_func,
                url_or_urls,
                map_tuple=True,
                num_proc=download_config.num_proc,
                desc="Downloading data files",
                batched=True if len(url_or_urls) >= 16 else False,
                batch_size=-1,
            )
```

I would suggest to consider other datasets for similar issues and make a pull-request.

Thanks for reporting @Namangarg110 and thanks for the investigation @MilanaShhanukova.

Apparently, there is an issue with the download functionality.
I am proposing a fix.

### Describe the bug

```
Exception has occurred: FileNotFoundError
Couldn't find file at https://huggingface.co/datasets/facebook/voxpopuli/resolve/main/{'en': 'data/en/asr_train.tsv'}
```

Error in logic for link url creation. The link should be  https://huggingface.co/datasets/facebook/voxpopuli/resolve/main/data/en/asr_train.tsv

Basically there should be links directly under ```metadata["train"]```, not under ```metadata["train"][self.config.languages[0]]```

same for audio urls

### Steps to reproduce the bug

```
from datasets import load_dataset

dataset = load_dataset("facebook/voxpopuli","en")
```

### Expected behavior

Dataset should be loaded successfully.

### Environment info

- `datasets` version: 2.19.0
- Platform: Linux-5.15.0-1041-aws-x86_64-with-glibc2.31
- Python version: 3.10.13
- `huggingface_hub` version: 0.22.2
- PyArrow version: 16.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.12.2


fix webdataset filename split

Hi ! This was fixed recently in https://github.com/huggingface/datasets/pull/6904 and https://github.com/huggingface/datasets/pull/6931

use `os.path.splitext` to parse field_name.

fix filename which has dot. like:

```
a.b.jpeg
a.b.txt
```


Cant Downlaod Common Voice 17.0 hy-AM

### Describe the bug

I want to download Common Voice 17.0 hy-AM but it returns an error.
```

The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
  @hydra.main(config_name='hfds_config', config_path=None)
/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
  ret = run_job(
/usr/local/lib/python3.10/dist-packages/datasets/load.py:1429: FutureWarning: The repository for mozilla-foundation/common_voice_17_0 contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/mozilla-foundation/common_voice_17_0
You can avoid this message in future by passing the argument `trust_remote_code=True`.
Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.
  warnings.warn(
Reading metadata...: 6180it [00:00, 133224.37it/s]les/s]
Generating train split: 0 examples [00:00, ? examples/s]
HuggingFace datasets failed due to some reason (stack trace below).
For certain datasets (eg: MCV), it may be necessary to login to the huggingface-cli (via `huggingface-cli login`).
Once logged in, you need to set `use_auth_token=True` when calling this script.

Traceback error for reference :

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1743, in _prepare_split_single
    example = self.info.features.encode_example(record) if self.info.features is not None else record
  File "/usr/local/lib/python3.10/dist-packages/datasets/features/features.py", line 1878, in encode_example
    return encode_nested_example(self, example)
  File "/usr/local/lib/python3.10/dist-packages/datasets/features/features.py", line 1243, in encode_nested_example
    {
  File "/usr/local/lib/python3.10/dist-packages/datasets/features/features.py", line 1243, in <dictcomp>
    {
  File "/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py", line 326, in zip_dict
    yield key, tuple(d[key] for d in dicts)
  File "/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py", line 326, in <genexpr>
    yield key, tuple(d[key] for d in dicts)
KeyError: 'sentence_id'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/nemo/scripts/speech_recognition/convert_hf_dataset_to_nemo.py", line 358, in main
    dataset = load_dataset(
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2549, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1005, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1767, in _download_and_prepare
    super()._download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1100, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1605, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1762, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset
```

### Steps to reproduce the bug

```
from datasets import load_dataset

cv_17 = load_dataset("mozilla-foundation/common_voice_17_0", "hy-AM")
```

### Expected behavior

It works fine with common_voice_16_1

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.15.0-1042-nvidia-x86_64-with-glibc2.35
- Python version: 3.11.6
- `huggingface_hub` version: 0.22.2
- PyArrow version: 15.0.2
- Pandas version: 2.2.2
- `fsspec` version: 2024.2.0


[Streaming] Only load requested splits without resolving files for the other splits

This should help fixing this issue: https://github.com/huggingface/datasets/pull/6832

I'm having a similar issue when using splices:

It seems to be downloading, loading, and generating splits using the entire dataset.

e.g. [thangvip](https://huggingface.co/thangvip)/[cosmopedia_vi_math](https://huggingface.co/datasets/thangvip/cosmopedia_vi_math) has 300 splits and it takes a very long time to load only one split.

This is due to `load_dataset()` resolving the files of all the splits even if only one is needed.

In `dataset-viewer` the splits are loaded in different jobs so it results in 300 jobs that resolve 300 splits -> 90k calls to `/paths-info`


Unimaginable super slow iteration

In every iteration you load the full "random_input" column in memory, only then to access it's i-th element.

You can try using this instead

a,b=dataset[i]['random_input'],dataset[i]['random_output']

### Describe the bug

Assuming there is a dataset with 52000 sentences, each with a length of 500, it takes 20 seconds to extract a sentence from the dataset……？Is there something wrong with my iteration?

### Steps to reproduce the bug

```python
import datasets
import time
import random

num_rows = 52000
num_cols = 500

random_input = [[random.randint(1, 100) for _ in range(num_cols)] for _ in range(num_rows)]
random_output = [[random.randint(1, 100) for _ in range(num_cols)] for _ in range(num_rows)]

s=time.time()
d={'random_input':random_input,'random_output':random_output}
dataset=datasets.Dataset.from_dict(d)
print('from dict',time.time()-s)
print(dataset)

for i in range(len(dataset)):
    aa=time.time()
    a,b=dataset['random_input'][i],dataset['random_output'][i]
    print(time.time()-aa)

```

corresponding output
```bash
from dict 9.215498685836792
Dataset({
    features: ['random_input', 'random_output'],
    num_rows: 52000
})
19.129778146743774
19.329464197158813
19.27668261528015
19.28557538986206
19.247620582580566
19.624247074127197
19.28673791885376
19.301053047180176
19.290496110916138
19.291821718215942
19.357765197753906

```

### Expected behavior

Under normal circumstances, iteration should be very rapid as it does not involve the main tasks other than getting items

### Environment info

- `datasets` version: 2.19.0
- Platform: Linux-3.10.0-1160.71.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.10.13
- `huggingface_hub` version: 0.21.4
- PyArrow version: 15.0.0
- Pandas version: 2.2.1
- `fsspec` version: 2024.2.0


load_dataset doesn't support list column

I encountered this same issue when loading a customized dataset for ORPO training, in which there were three columns and two of them were lists.
I debugged and found that it might be caused by the type-infer mechanism and because in some chunks one of the columns is always an empty list ([]), it was regarded as ```list<item: null>```, however in some other chunk it was ```list<item: string>```. This triggered a TypeError running the function ```table_cast()```.

I temporarily fixed this by re-dumping the file into a regular JSON format instead of lines of JSON dict. I didn't dig deeper for the lack of knowledge and programming ability but I do hope some developer of this repo will find and fix it.

### Describe the bug

dataset = load_dataset("Doraemon-AI/text-to-neo4j-cypher-chinese")

got exception:
Generating train split: 1834 examples [00:00, 5227.98 examples/s]
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/datasets/builder.py", line 2011, in _prepare_split_single
    writer.write_table(table)
  File "/usr/local/lib/python3.11/dist-packages/datasets/arrow_writer.py", line 585, in write_table
    pa_table = table_cast(pa_table, self._schema)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/table.py", line 2295, in table_cast
    return cast_table_to_schema(table, schema)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/table.py", line 2254, in cast_table_to_schema
    arrays = [cast_array_to_feature(table[name], feature) for name, feature in features.items()]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/table.py", line 2254, in <listcomp>
    arrays = [cast_array_to_feature(table[name], feature) for name, feature in features.items()]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/table.py", line 1802, in wrapper
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/table.py", line 1802, in <listcomp>
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/table.py", line 2018, in cast_array_to_feature
    casted_array_values = _c(array.values, feature[0])
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/table.py", line 1804, in wrapper
    return func(array, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/table.py", line 2115, in cast_array_to_feature
    raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{feature}")
TypeError: Couldn't cast array of type
struct<m.name: string, x.name: string, p.name: string, n.name: string, h.name: string, name: string, c: int64, collect(r.name): list<item: string>, q.name: string, rel.name: string, count(p): int64, 1: int64, p.location: string, max(n.name): null, mn.name: string, p.time: int64, min(q.name): string>
to
{'q.name': Value(dtype='string', id=None), 'mn.name': Value(dtype='string', id=None), 'x.name': Value(dtype='string', id=None), 'p.name': Value(dtype='string', id=None), 'n.name': Value(dtype='string', id=None), 'name': Value(dtype='string', id=None), 'm.name': Value(dtype='string', id=None), 'h.name': Value(dtype='string', id=None), 'count(p)': Value(dtype='int64', id=None), 'rel.name': Value(dtype='string', id=None), 'c': Value(dtype='int64', id=None), 'collect(r.name)': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), '1': Value(dtype='int64', id=None), 'p.location': Value(dtype='string', id=None), 'substring(h.name,0,5)': Value(dtype='string', id=None), 'p.time': Value(dtype='int64', id=None)}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/llm/train-2.py", line 150, in <module>
    dataset = load_dataset("Doraemon-AI/text-to-neo4j-cypher-chinese")
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/load.py", line 2609, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.11/dist-packages/datasets/builder.py", line 1027, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.11/dist-packages/datasets/builder.py", line 1122, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/usr/local/lib/python3.11/dist-packages/datasets/builder.py", line 1882, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/usr/local/lib/python3.11/dist-packages/datasets/builder.py", line 2038, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset

### Steps to reproduce the bug

dataset = load_dataset("Doraemon-AI/text-to-neo4j-cypher-chinese")

### Expected behavior

no exception

### Environment info

python 3.11
datasets 2.19.0


Retry on HF Hub error when streaming

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6844). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@Wauplin This PR is indeed not needed as explained in https://github.com/huggingface/datasets/issues/6843#issuecomment-2079630389.

So, I'm closing it.

Retry on the `huggingface_hub`'s `HfHubHTTPError` in the streaming mode.

Fix #6843


IterableDataset raises exception instead of retrying

Thanks for reporting! I've opened a PR with a fix.

Thanks, @mariosasko! Related question (although I guess this is a feature request): could we have some kind of exponential back-off for these retries? Here's my reasoning:
- If a one-time accidental error happens, you should retry immediately and will succeed immediately.
- If the Hub has a small outage on the order of minutes, you don't want to retry on the order of hours.
- If the Hub has a prologned outage of several hours, we don't want to keep retrying on the order of minutes.

There actually already exists an implementation for (clipped) exponential backoff in the HuggingFace suite ([here](https://github.com/huggingface/huggingface_hub/blob/61b156a4f2e5fe1a492ed8712b26803e2122bde0/src/huggingface_hub/utils/_http.py#L306)), but I don't think it is used here.

The requirements are basically that you have an initial minimum waiting time and a maximum waiting time, and with each retry, the waiting time is doubled. We don't want to overload your servers with needless retries, especially when they're down :sweat_smile:

Oh, I've just remembered that we added retries to the `HfFileSystem` in `huggingface_hub` 0.21.0 (see [this](https://github.com/huggingface/huggingface_hub/blob/61b156a4f2e5fe1a492ed8712b26803e2122bde0/src/huggingface_hub/hf_file_system.py#L703)), so I'll close the linked PR as we don't want to retry the retries :).

I agree with the exponential backoff suggestion, so I'll open another PR.

@mariosasko The call you linked indeed points to the implementation I linked in my previous comment, yes, but it has no configurability. Arguably, you want to have this hidden backoff under the hood that catches small network disturbances on the time scale of seconds -- perhaps even with hardcoded limits as is the case currently -- but you also still want to have a separate backoff on top of that with the configurability as suggested by @lhoestq in [the comment I linked](https://github.com/huggingface/datasets/issues/6172#issuecomment-1794876229).

My particular use-case is that I'm streaming a dataset while training on a university cluster with a very long scheduling queue. This means that when the backoff runs out of retries (which happens in under 30 seconds with the call you linked), I lose my spot on the cluster and have to queue for a whole day or more. Ideally, I should be able to specify that I want to retry for 2 to 3 hours but with more and more time between requests, so that I can smooth over hours-long outages without a setback of days.

I also have my runs crash a surprising amount due to the dataloader crashing because of the hub, some way to address this would be nice.

### Describe the bug

In light of the recent server outages, I decided to look into whether I could somehow wrap my IterableDataset streams to retry rather than error out immediately. To my surprise, `datasets` [already supports retries](https://github.com/huggingface/datasets/issues/6172#issuecomment-1794876229). Since a commit by @lhoestq [last week](https://github.com/huggingface/datasets/commit/a188022dc43a76a119d90c03832d51d6e4a94d91), that code lives here:

https://github.com/huggingface/datasets/blob/fe2bea6a4b09b180bd23b88fe96dfd1a11191a4f/src/datasets/utils/file_utils.py#L1097C1-L1111C19

If GitHub code snippets still aren't working, here's a copy:
```python
    def read_with_retries(*args, **kwargs):
        disconnect_err = None
        for retry in range(1, max_retries + 1):
            try:
                out = read(*args, **kwargs)
                break
            except (ClientError, TimeoutError) as err:
                disconnect_err = err
                logger.warning(
                    f"Got disconnected from remote data host. Retrying in {config.STREAMING_READ_RETRY_INTERVAL}sec [{retry}/{max_retries}]"
                )
                time.sleep(config.STREAMING_READ_RETRY_INTERVAL)
        else:
            raise ConnectionError("Server Disconnected") from disconnect_err
        return out
```

With the latest outage, the end of my stack trace looked like this:
```
...
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/datasets/download/streaming_download_manager.py", line 342, in read_with_retries
    out = read(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/draft/lib/python3.11/gzip.py", line 301, in read
    return self._buffer.read(size)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/draft/lib/python3.11/_compression.py", line 68, in readinto
    data = self.read(len(byte_view))
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/draft/lib/python3.11/gzip.py", line 505, in read
    buf = self._fp.read(io.DEFAULT_BUFFER_SIZE)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/draft/lib/python3.11/gzip.py", line 88, in read
    return self.file.read(size)
           ^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/fsspec/spec.py", line 1856, in read
    out = self.cache._fetch(self.loc, self.loc + length)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/fsspec/caching.py", line 189, in _fetch
    self.cache = self.fetcher(start, end)  # new block replaces old
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/huggingface_hub/hf_file_system.py", line 626, in _fetch_range
    hf_raise_for_status(r)
  File "/miniconda3/envs/draft/lib/python3.11/site-packages/huggingface_hub/utils/_errors.py", line 333, in hf_raise_for_status
    raise HfHubHTTPError(str(e), response=response) from e
huggingface_hub.utils._errors.HfHubHTTPError: 504 Server Error: Gateway Time-out for url: https://huggingface.co/datasets/allenai/c4/resolve/1588ec454efa1a09f29cd18ddd04fe05fc8653a2/en/c4-train.00346-of-01024.json.gz
```

Indeed, the code for retries only catches `ClientError`s and `TimeoutError`s, and all other exceptions, *including HuggingFace's own custom HTTP error class*, **are not caught. Nothing is retried,** and instead the exception is propagated upwards immediately.

### Steps to reproduce the bug

Not sure how you reproduce this. Maybe unplug your Ethernet cable while streaming a dataset; the issue is pretty clear from the stack trace.

### Expected behavior

All HTTP errors while iterating a streamable dataset should cause retries.

### Environment info

Output from `datasets-cli env`:
- `datasets` version: 2.18.0
- Platform: Linux-4.18.0-513.24.1.el8_9.x86_64-x86_64-with-glibc2.28
- Python version: 3.11.7
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0


Datasets with files with colon : in filenames cannot be used on Windows

### Describe the bug

Datasets (such as https://huggingface.co/datasets/MLCommons/peoples_speech) cannot be used on Windows due to the fact that windows does not allow colons ":" in filenames. These should be converted into alternative strings.

### Steps to reproduce the bug

1. Attempt to run load_dataset on MLCommons/peoples_speech

### Expected behavior

Does not crash during extraction

### Environment info

Windows 11, NTFS filesystem, Python 3.12


Unable to load wiki_auto_asset_turk from GEM

Hi! I've opened a [PR](https://huggingface.co/datasets/GEM/wiki_auto_asset_turk/discussions/5) with a fix. While waiting for it to be merged, you can load the dataset from the PR branch with `datasets.load_dataset("GEM/wiki_auto_asset_turk", revision="refs/pr/5")`

Thanks Mario. Still getting the same issue though with the suggested fix

#cat gem_sari.py
import datasets
print (datasets.__version__)
dataset =datasets.load_dataset("GEM/wiki_auto_asset_turk", revision="refs/pr/5")

End up with
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/load.py", line 2582, in load_dataset
    builder_instance.download_and_prepare(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/builder.py", line 1005, in download_and_prepare
    self._download_and_prepare(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/builder.py", line 1767, in _download_and_prepare
    super()._download_and_prepare(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/builder.py", line 1100, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/builder.py", line 1565, in _prepare_split
    split_info = self.info.splits[split_generator.name]
                 ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/splits.py", line 532, in __getitem__
    instructions = make_file_instructions(
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/arrow_reader.py", line 121, in make_file_instructions
    info.name: filenames_for_dataset_split(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/naming.py", line 72, in filenames_for_dataset_split
    prefix = os.path.join(path, prefix)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen posixpath>", line 76, in join
TypeError: expected str, bytes or os.PathLike object, not NoneType

Hmm, that's weird. Maybe try deleting the cache with `!rm -rf ~/.cache/huggingface/datasets` and then re-download.

Tried that a couple of time. It does download the data fresh but end up with same error. Is there a way to see if its using the right version ?

You can check the version with `python -c "import datasets; print(datasets.__version__)"`

the datasets version is 2.18.

I wanted to see if the command datasets.load_dataset("GEM/wiki_auto_asset_turk", revision="refs/pr/5") is using the right revision (refs/pr/5).

The issue is fixed once the fixing PR has been merged and the dataset has been converted to Parquet.

If the problem persists on your side, you should update your `datasets` library:
```shell
pip install -U datasets
```
And if you have already the latest version of `datasets`, then you need to delete the old version of this dataset in your cache:
```shell
rm -fr ~/.cache/huggingface/datasets/GEM___wiki_auto_asset_turk
rm -fr ~/.cache/huggingface/modules/datasets_modules/datasets/GEM--wiki_auto_asset_turk
```

### Describe the bug

I am unable to load the wiki_auto_asset_turk dataset. I get a fatal error while trying to access wiki_auto_asset_turk and load it with datasets.load_dataset. The error (TypeError: expected str, bytes or os.PathLike object, not NoneType) is from filenames_for_dataset_split in a os.path.join call

>>import datasets
>>print (datasets.__version__)
>>dataset = datasets.load_dataset("GEM/wiki_auto_asset_turk")

System output:

Generating train split: 100%|█| 483801/483801 [00:03<00:00, 127164.26 examples/s
Generating validation split: 100%|█| 20000/20000 [00:00<00:00, 116052.94 example
Generating test_asset split: 100%|██| 359/359 [00:00<00:00, 76155.93 examples/s]
Generating test_turk split: 100%|███| 359/359 [00:00<00:00, 87691.76 examples/s]
Traceback (most recent call last):
  File "/Users/abhinav.sethy/Code/openai_evals/evals/evals/grammarly_tasks/gem_sari.py", line 3, in <module>
    dataset = datasets.load_dataset("GEM/wiki_auto_asset_turk")
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/load.py", line 2582, in load_dataset
    builder_instance.download_and_prepare(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/builder.py", line 1005, in download_and_prepare
    self._download_and_prepare(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/builder.py", line 1767, in _download_and_prepare
    super()._download_and_prepare(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/builder.py", line 1100, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/builder.py", line 1565, in _prepare_split
    split_info = self.info.splits[split_generator.name]
                 ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/splits.py", line 532, in __getitem__
    instructions = make_file_instructions(
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/arrow_reader.py", line 121, in make_file_instructions
    info.name: filenames_for_dataset_split(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/naming.py", line 72, in filenames_for_dataset_split
    prefix = os.path.join(path, prefix)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen posixpath>", line 76, in join
TypeError: expected str, bytes or os.PathLike object, not NoneType

### Steps to reproduce the bug

import datasets
print (datasets.__version__)
dataset = datasets.load_dataset("GEM/wiki_auto_asset_turk")

### Expected behavior

Should be able to load the dataset without any issues

### Environment info

datasets version 2.18.0 (was able to reproduce bug with older versions 2.16 and 2.14 also)
Python 3.12.0


Delete uploaded files from the UI

enhancement

New feature or request

### Feature request

Once a file is uploaded and the commit is made, I am unable to delete individual files without completely deleting the whole dataset via the website UI.

### Motivation

Would be a useful addition

### Your contribution

Would love to help out with some guidance


Remove token arg from CLI examples

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6839). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005311 / 0.011353 (-0.006042) | 0.003691 / 0.011008 (-0.007317) | 0.063714 / 0.038508 (0.025206) | 0.030875 / 0.023109 (0.007766) | 0.251210 / 0.275898 (-0.024688) | 0.280539 / 0.323480 (-0.042941) | 0.004262 / 0.007986 (-0.003724) | 0.002723 / 0.004328 (-0.001606) | 0.049487 / 0.004250 (0.045237) | 0.045655 / 0.037052 (0.008603) | 0.264399 / 0.258489 (0.005910) | 0.306613 / 0.293841 (0.012772) | 0.028513 / 0.128546 (-0.100033) | 0.010726 / 0.075646 (-0.064921) | 0.210601 / 0.419271 (-0.208670) | 0.036918 / 0.043533 (-0.006614) | 0.257872 / 0.255139 (0.002733) | 0.278951 / 0.283200 (-0.004249) | 0.017900 / 0.141683 (-0.123783) | 1.096749 / 1.452155 (-0.355406) | 1.152603 / 1.492716 (-0.340113) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095193 / 0.018006 (0.077187) | 0.303919 / 0.000490 (0.303429) | 0.000226 / 0.000200 (0.000026) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018558 / 0.037411 (-0.018853) | 0.061106 / 0.014526 (0.046580) | 0.076233 / 0.176557 (-0.100323) | 0.122402 / 0.737135 (-0.614734) | 0.075579 / 0.296338 (-0.220760) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.283586 / 0.215209 (0.068377) | 2.766179 / 2.077655 (0.688524) | 1.481069 / 1.504120 (-0.023051) | 1.355004 / 1.541195 (-0.186191) | 1.392940 / 1.468490 (-0.075550) | 0.578878 / 4.584777 (-4.005899) | 2.432890 / 3.745712 (-1.312822) | 2.837912 / 5.269862 (-2.431949) | 1.762803 / 4.565676 (-2.802873) | 0.063339 / 0.424275 (-0.360937) | 0.005392 / 0.007607 (-0.002215) | 0.340271 / 0.226044 (0.114227) | 3.388371 / 2.268929 (1.119443) | 1.862622 / 55.444624 (-53.582002) | 1.543209 / 6.876477 (-5.333268) | 1.569858 / 2.142072 (-0.572215) | 0.651487 / 4.805227 (-4.153740) | 0.119048 / 6.500664 (-6.381616) | 0.042309 / 0.075469 (-0.033160) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.991161 / 1.841788 (-0.850627) | 11.778857 / 8.074308 (3.704549) | 9.586019 / 10.191392 (-0.605373) | 0.148093 / 0.680424 (-0.532331) | 0.014301 / 0.534201 (-0.519900) | 0.287983 / 0.579283 (-0.291301) | 0.266070 / 0.434364 (-0.168293) | 0.328261 / 0.540337 (-0.212076) | 0.417908 / 1.386936 (-0.969028) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005252 / 0.011353 (-0.006100) | 0.003740 / 0.011008 (-0.007268) | 0.049622 / 0.038508 (0.011114) | 0.030040 / 0.023109 (0.006931) | 0.262224 / 0.275898 (-0.013674) | 0.312216 / 0.323480 (-0.011264) | 0.004213 / 0.007986 (-0.003773) | 0.002737 / 0.004328 (-0.001592) | 0.049159 / 0.004250 (0.044908) | 0.041060 / 0.037052 (0.004008) | 0.275826 / 0.258489 (0.017337) | 0.301879 / 0.293841 (0.008038) | 0.029364 / 0.128546 (-0.099182) | 0.010453 / 0.075646 (-0.065193) | 0.058095 / 0.419271 (-0.361176) | 0.032898 / 0.043533 (-0.010635) | 0.263876 / 0.255139 (0.008737) | 0.281686 / 0.283200 (-0.001514) | 0.018711 / 0.141683 (-0.122971) | 1.126056 / 1.452155 (-0.326098) | 1.185125 / 1.492716 (-0.307591) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094153 / 0.018006 (0.076147) | 0.300719 / 0.000490 (0.300229) | 0.000207 / 0.000200 (0.000007) | 0.000048 / 0.000054 (-0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022610 / 0.037411 (-0.014801) | 0.075502 / 0.014526 (0.060977) | 0.088858 / 0.176557 (-0.087699) | 0.129421 / 0.737135 (-0.607714) | 0.089331 / 0.296338 (-0.207007) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.291595 / 0.215209 (0.076386) | 2.864377 / 2.077655 (0.786722) | 1.543387 / 1.504120 (0.039267) | 1.404273 / 1.541195 (-0.136922) | 1.421964 / 1.468490 (-0.046526) | 0.579275 / 4.584777 (-4.005502) | 0.979212 / 3.745712 (-2.766500) | 2.822043 / 5.269862 (-2.447818) | 1.745015 / 4.565676 (-2.820661) | 0.064626 / 0.424275 (-0.359649) | 0.005006 / 0.007607 (-0.002601) | 0.345509 / 0.226044 (0.119464) | 3.410369 / 2.268929 (1.141440) | 1.875930 / 55.444624 (-53.568694) | 1.600841 / 6.876477 (-5.275636) | 1.611818 / 2.142072 (-0.530254) | 0.662277 / 4.805227 (-4.142950) | 0.117861 / 6.500664 (-6.382803) | 0.041061 / 0.075469 (-0.034408) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.007834 / 1.841788 (-0.833954) | 12.345653 / 8.074308 (4.271345) | 9.775237 / 10.191392 (-0.416155) | 0.135166 / 0.680424 (-0.545258) | 0.016799 / 0.534201 (-0.517402) | 0.289235 / 0.579283 (-0.290048) | 0.126196 / 0.434364 (-0.308168) | 0.382905 / 0.540337 (-0.157432) | 0.435248 / 1.386936 (-0.951688) |

![](https://cml.dev/watermark.png#22bf5388748611a9255d8e17218d36d2f799f182 "CML watermark")

Remove token arg from CLI examples.

Fix #6838.

CC: @Wauplin


Remove token arg from CLI examples

As suggested by @Wauplin, see: https://github.com/huggingface/datasets/pull/6831#discussion_r1579492603
> I would not advertise the --token arg in the example as this shouldn't be the recommended way (best to login with env variable or huggingface-cli login)


Cannot use cached dataset without Internet connection (or when servers are down)

There are 2 workarounds, tho:
1. Download datasets from web and just load them locally
2. Use metadata directly (temporal solution, since metadata can change)
```
import datasets
from datasets.data_files import DataFilesDict, DataFilesList

data_files_list = DataFilesList(
    [
        "hf://datasets/allenai/c4@1588ec454efa1a09f29cd18ddd04fe05fc8653a2/en/c4-train.00000-of-01024.json.gz"
    ],
    [("allenai/c4", "1588ec454efa1a09f29cd18ddd04fe05fc8653a2")],
)
data_files = DataFilesDict({"train": data_files_list})
c4_dataset = datasets.load_dataset(
    path="allenai/c4",
    data_files=data_files,
    split="train",
    cache_dir="/datesets/cache",
    download_mode="reuse_cache_if_exists",
    token=False,
)
```
Second solution also shows where to find the bug. I suggest that the hashing functions should always use only original parameter `data_files`, and not the one they get after connecting to the server and creating `DataFilesDict`

Hi! You need to set the `HF_DATASETS_OFFLINE` env variable to `1` to load cached datasets offline, as explained in the docs [here](https://huggingface.co/docs/datasets/v2.19.0/en/loading#offline).

Just tested. It doesn't work, because of the exact problem I described above: hash of dataset config is different.
The only error difference is the reason why it cannot connect to HuggingFace (now it's 'offline mode is enabled')
![image](https://github.com/huggingface/datasets/assets/112088378/1a7e1720-d711-46e3-9c90-53d52c441e68)

### Describe the bug

I want to be able to use cached dataset from HuggingFace even when I have no Internet connection (or when HuggingFace servers are down, or my company has network issues).
The problem why I can't use it:
`data_files` argument from `datasets.load_dataset()` function get it updates from the server before calculating hash for caching. As a result, when I run the same code with and without Internet I get different dataset configuration directory name.

### Steps to reproduce the bug

```
import datasets

c4_dataset = datasets.load_dataset(
    path="allenai/c4",
    data_files={"train": "en/c4-train.00000-of-01024.json.gz"},
    split="train",
    cache_dir="/datesets/cache",
    download_mode="reuse_cache_if_exists",
    token=False,
)
```
1. Run this code with the Internet.
2. Run the same code without the Internet.

### Expected behavior

When running without the Internet connection, the loader should be able to get dataset from cache

### Environment info

- `datasets` version: 2.19.0
- Platform: Windows-10-10.0.19044-SP0
- Python version: 3.10.13
- `huggingface_hub` version: 0.22.2
- PyArrow version: 16.0.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.12.2


ExpectedMoreSplits error on load_dataset when upgrading to 2.19.0

Get same error on same datasets too.
+1
same error

### Describe the bug

Hi there, thanks for the great library! We have been using it a lot in torchtune and it's been a huge help for us.

Regarding the bug: the same call to `load_dataset` errors with `ExpectedMoreSplits` in 2.19.0 after working fine in 2.18.0. Full details given in the repro below.

### Steps to reproduce the bug

On 2.18.0, things work fine:
```
# First clear the locally cached dataset
rm -r ~/.cache/huggingface/datasets/lvwerra___stack-exchange-paired
pip install "datasets==2.18.0"
python3
>>> from datasets import load_dataset
>>> dataset = load_dataset('lvwerra/stack-exchange-paired', split='train', data_dir='data/rl')
```

On 2.19.0, they do not:
```
# First clear the locally cached dataset
rm -r ~/.cache/huggingface/datasets/lvwerra___stack-exchange-paired
pip install "datasets==2.19.0"
python3
>>> from datasets import load_dataset
>>> dataset = load_dataset('lvwerra/stack-exchange-paired', split='train', data_dir='data/rl')
```

The stack trace I see from the 2.19.0 version of load_dataset can be seen [here](https://gist.github.com/ebsmothers/f9b1f1949bee7030a8d7bb8a491550d2).

(Maybe unsurprising but) notably if I do not delete the cache first I am able to load the dataset successfully. So based on this I suspect the cause is somewhere in the download logic.

### Expected behavior

Download the dataset successfully :)

### Environment info

- `datasets` version: 2.19.0
- Platform: Linux-5.12.0-0_fbk16_zion_7661_geb00762ce6d2-x86_64-with-glibc2.34
- Python version: 3.11.9
- `huggingface_hub` version: 0.22.2
- PyArrow version: 16.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.3.1


Support pyarrow LargeListType

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6835). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Fixed the conversion from `pyarrow` to `python` `Sequence` features.

There is still an issue that if `features` are passed the `Sequence` always forces conversion to `ListArray`.
This probably causes issues if the `LargeListArray` is actually needed.

There doesn't seem to be a great solution since this list is created solely on the `schema` for `Sequence`.
One solution would be to always use `LargeListArray` instead.

Fixes #6834


largelisttype not supported (.from_polars())

### Describe the bug

The following code fails because LargeListType is not supported.
This is especially a problem for .from_polars since polars uses LargeListType.

### Steps to reproduce the bug

```python
import datasets
import polars as pl

df = pl.DataFrame({"list": [[]]})
datasets.Dataset.from_polars(df)
```

### Expected behavior

Convert LargeListType to list.

### Environment info

- `datasets` version: 2.19.1.dev0
- Platform: Linux-6.8.7-200.fc39.x86_64-x86_64-with-glibc2.38
- Python version: 3.12.2
- `huggingface_hub` version: 0.22.2
- PyArrow version: 16.0.0
- Pandas version: 2.1.4
- `fsspec` version: 2024.3.1


Super slow iteration with trivial custom transform

Similar issue in text process

```python

tokenizer=AutoTokenizer.from_pretrained(model_dir[args.model])
train_dataset=datasets.load_from_disk(dataset_dir[args.dataset],keep_in_memory=True)['train']
train_dataset=train_dataset.map(partial(dname2func[args.dataset],tokenizer=tokenizer),batched=True,num_proc =50,remove_columns=train_dataset.features.keys(),desc='tokenize',keep_in_memory=True)

```
After this train_dataset will be like
```python
Dataset({
    features: ['input_ids', 'labels'],
    num_rows: 51760
})
```
In which input_ids and labels are both List[int]
However, per iter on dataset cost 7.412479639053345s ……？
```python
for j in tqdm(range(len(train_dataset)),desc='first stage'):
    input_id,label=train_dataset['input_ids'][j],train_dataset['labels'][j]

```

The transform currently replaces the numpy formatting.

So you're back to copying data to long python lists which is super slow.

It would be cool for the transform to not remove the formatting in this case, but this requires a few changes in the lib

### Describe the bug

Dataset is 10X slower when applying trivial transforms:

```
import time
import numpy as np
from datasets import Dataset, Features, Array2D

a = np.zeros((800, 800))
a = np.stack([a] * 1000)
features = Features({"a": Array2D(shape=(800, 800), dtype="uint8")})

ds1 = Dataset.from_dict({"a": a}, features=features).with_format('numpy')

def transform(batch):
	return batch

ds2 = ds1.with_transform(transform)

%time sum(1 for _ in ds1)
%time sum(1 for _ in ds2)
```

```
CPU times: user 472 ms, sys: 319 ms, total: 791 ms
Wall time: 794 ms
CPU times: user 9.32 s, sys: 443 ms, total: 9.76 s
Wall time: 9.78 s
```

In my real code I'm using set_transform to apply some post-processing on-the-fly for the 2d array, but it significantly slows down the dataset even if the transform itself is trivial.

Related issue: https://github.com/huggingface/datasets/issues/5841

### Steps to reproduce the bug

Use code in the description to reproduce.

### Expected behavior

Trivial custom transform in the example should not slowdown the dataset iteration.

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.15.0-79-generic-x86_64-with-glibc2.35
- Python version: 3.11.4
- `huggingface_hub` version: 0.20.2
- PyArrow version: 15.0.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.12.2


Support downloading specific splits in `load_dataset`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6832). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This PR builds on https://github.com/huggingface/datasets/pull/6639 to support downloading only the specified splits in `load_dataset`. For this to work, a builder's `_split_generators` need to be able to accept the requested splits (as a list) via a `splits` argument to avoid processing the non-requested ones. Also, the builder has to define a `_available_splits` method that lists all the possible `splits` values.

Close https://github.com/huggingface/datasets/issues/4101, close https://github.com/huggingface/datasets/issues/2538 (I'm probably missing some)

Should also make it possible to address https://github.com/huggingface/datasets/issues/6793


Add docs about the CLI

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6831). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Concretely, the docs about convert_to_parquet are here: https://moon-ci-docs.huggingface.co/docs/datasets/pr_6831/en/cli#convert-to-parquet

There is an issue with the example snippet when copy/pasting it: the leading shell dollar sign is also copied. I guess they will not like to fix it in the backend: currently they only support Python code snippets (with leading `>>>` or `...`), as they appear in the IPython interactive console.

What do you suggest, @severo?

Add docs about the CLI.

Close #6830.

CC: @severo


Add a doc page for the convert_to_parquet CLI

documentation

Improvements or additions to documentation

Follow-up to https://github.com/huggingface/datasets/pull/6795. Useful for https://github.com/huggingface/dataset-viewer/issues/2742. cc @albertvillanova


Load and save from/to disk no longer accept pathlib.Path

bug

Something isn't working

Reported by @vttrifonov at https://github.com/huggingface/datasets/pull/6704#issuecomment-2071168296:
> This change is breaking in
> https://github.com/huggingface/datasets/blob/f96e74d5c633cd5435dd526adb4a74631eb05c43/src/datasets/arrow_dataset.py#L1515
> when the input is `pathlib.Path`. The issue is that `url_to_fs` expects a `str` and cannot deal with `Path`. `get_fs_token_paths` converts to `str` so it is not a problem

This change was introduced in:
- #6704


Support PathLike input in save_to_disk / load_from_disk

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6828). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


Loading a remote dataset fails in the last release (v2.19.0)

While loading a dataset with multiple splits I get an error saying `Couldn't find file at <URL>`

I am loading the dataset like so, nothing out of the ordinary.
This dataset needs a token to access it.
```
token="hf_myhftoken-sdhbdsjgkhbd"
load_dataset("speechcolab/gigaspeech", "test", cache_dir=f"gigaspeech/test", token=token)
```

I get the following error
![Screenshot 2024-04-19 at 11 03 07 PM](https://github.com/huggingface/datasets/assets/35369637/8dce757f-08ff-45dd-85b5-890fced7c5bc)

Now you can see that the URL that it is trying to reach has the JSON object of the dataset split appended to the base URL. I think this may be due to a newly introduced issue.

I did not have this issue with the previous version of the datasets. Everything was fine for me yesterday and after the release 12 hours ago, this seems to have broken. Also, the dataset in question runs custom code and I checked and there have been no commits to the dataset on Huggingface in 6 months.

### Steps to reproduce the bug

Since this happened with one particular dataset for me, I am listing steps to use that dataset.
1. Open https://huggingface.co/datasets/speechcolab/gigaspeech and fill the form to get access.
2. Create a token on your huggingface account with read access.
3. Run the following line, substituing `<your_token_here>` with your token.
```
load_dataset("speechcolab/gigaspeech", "test", cache_dir=f"gigaspeech/test", token="<your_token_here>")
```

### Expected behavior

Be able to load the dataset in question.

### Environment info

datasets == 2.19.0
python == 3.10
kernel == Linux 6.1.58+


Set dev version

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6826). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004893 / 0.011353 (-0.006460) | 0.003238 / 0.011008 (-0.007771) | 0.063143 / 0.038508 (0.024635) | 0.029770 / 0.023109 (0.006661) | 0.229052 / 0.275898 (-0.046846) | 0.254534 / 0.323480 (-0.068945) | 0.003083 / 0.007986 (-0.004903) | 0.002615 / 0.004328 (-0.001714) | 0.049684 / 0.004250 (0.045434) | 0.043745 / 0.037052 (0.006693) | 0.248985 / 0.258489 (-0.009504) | 0.275957 / 0.293841 (-0.017884) | 0.027323 / 0.128546 (-0.101223) | 0.010372 / 0.075646 (-0.065275) | 0.206494 / 0.419271 (-0.212778) | 0.035230 / 0.043533 (-0.008303) | 0.234235 / 0.255139 (-0.020904) | 0.252395 / 0.283200 (-0.030805) | 0.019442 / 0.141683 (-0.122240) | 1.130677 / 1.452155 (-0.321478) | 1.161721 / 1.492716 (-0.330996) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091659 / 0.018006 (0.073653) | 0.301323 / 0.000490 (0.300833) | 0.000212 / 0.000200 (0.000012) | 0.000049 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018360 / 0.037411 (-0.019051) | 0.061101 / 0.014526 (0.046575) | 0.072383 / 0.176557 (-0.104174) | 0.117656 / 0.737135 (-0.619479) | 0.073903 / 0.296338 (-0.222436) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.272768 / 0.215209 (0.057558) | 2.655714 / 2.077655 (0.578059) | 1.446254 / 1.504120 (-0.057866) | 1.330543 / 1.541195 (-0.210652) | 1.352527 / 1.468490 (-0.115964) | 0.561428 / 4.584777 (-4.023349) | 2.368182 / 3.745712 (-1.377530) | 2.746508 / 5.269862 (-2.523353) | 1.713972 / 4.565676 (-2.851705) | 0.062046 / 0.424275 (-0.362229) | 0.005427 / 0.007607 (-0.002180) | 0.321652 / 0.226044 (0.095607) | 3.181812 / 2.268929 (0.912883) | 1.766778 / 55.444624 (-53.677846) | 1.492502 / 6.876477 (-5.383975) | 1.534658 / 2.142072 (-0.607415) | 0.640372 / 4.805227 (-4.164856) | 0.118180 / 6.500664 (-6.382484) | 0.042698 / 0.075469 (-0.032771) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.993262 / 1.841788 (-0.848525) | 11.512827 / 8.074308 (3.438518) | 9.602140 / 10.191392 (-0.589252) | 0.144723 / 0.680424 (-0.535701) | 0.014122 / 0.534201 (-0.520079) | 0.302211 / 0.579283 (-0.277072) | 0.268026 / 0.434364 (-0.166338) | 0.326524 / 0.540337 (-0.213813) | 0.423781 / 1.386936 (-0.963155) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005388 / 0.011353 (-0.005965) | 0.003535 / 0.011008 (-0.007473) | 0.050139 / 0.038508 (0.011631) | 0.031813 / 0.023109 (0.008704) | 0.269501 / 0.275898 (-0.006397) | 0.294355 / 0.323480 (-0.029125) | 0.004128 / 0.007986 (-0.003858) | 0.002684 / 0.004328 (-0.001644) | 0.049295 / 0.004250 (0.045045) | 0.040129 / 0.037052 (0.003077) | 0.282406 / 0.258489 (0.023917) | 0.309822 / 0.293841 (0.015981) | 0.028506 / 0.128546 (-0.100040) | 0.010434 / 0.075646 (-0.065213) | 0.057890 / 0.419271 (-0.361382) | 0.032487 / 0.043533 (-0.011046) | 0.270631 / 0.255139 (0.015492) | 0.288734 / 0.283200 (0.005534) | 0.018710 / 0.141683 (-0.122973) | 1.151571 / 1.452155 (-0.300583) | 1.195222 / 1.492716 (-0.297494) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090939 / 0.018006 (0.072932) | 0.300278 / 0.000490 (0.299788) | 0.000202 / 0.000200 (0.000002) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022036 / 0.037411 (-0.015376) | 0.075131 / 0.014526 (0.060605) | 0.087775 / 0.176557 (-0.088782) | 0.125719 / 0.737135 (-0.611416) | 0.088491 / 0.296338 (-0.207848) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.300363 / 0.215209 (0.085154) | 2.931852 / 2.077655 (0.854197) | 1.633688 / 1.504120 (0.129568) | 1.512641 / 1.541195 (-0.028554) | 1.527703 / 1.468490 (0.059213) | 0.572781 / 4.584777 (-4.011996) | 2.445950 / 3.745712 (-1.299762) | 2.883667 / 5.269862 (-2.386195) | 1.761396 / 4.565676 (-2.804280) | 0.064422 / 0.424275 (-0.359853) | 0.005332 / 0.007607 (-0.002275) | 0.346730 / 0.226044 (0.120686) | 3.443815 / 2.268929 (1.174886) | 1.988677 / 55.444624 (-53.455948) | 1.707688 / 6.876477 (-5.168789) | 1.694216 / 2.142072 (-0.447856) | 0.634834 / 4.805227 (-4.170393) | 0.115044 / 6.500664 (-6.385620) | 0.040853 / 0.075469 (-0.034616) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.009382 / 1.841788 (-0.832405) | 12.327511 / 8.074308 (4.253203) | 10.123296 / 10.191392 (-0.068097) | 0.130770 / 0.680424 (-0.549654) | 0.015548 / 0.534201 (-0.518653) | 0.286650 / 0.579283 (-0.292633) | 0.270267 / 0.434364 (-0.164097) | 0.333485 / 0.540337 (-0.206852) | 0.428288 / 1.386936 (-0.958648) |

![](https://cml.dev/watermark.png#f96e74d5c633cd5435dd526adb4a74631eb05c43 "CML watermark")


Release: 2.19.0

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6825). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004945 / 0.011353 (-0.006407) | 0.003290 / 0.011008 (-0.007718) | 0.062404 / 0.038508 (0.023896) | 0.040056 / 0.023109 (0.016946) | 0.246574 / 0.275898 (-0.029324) | 0.275074 / 0.323480 (-0.048406) | 0.004118 / 0.007986 (-0.003867) | 0.002604 / 0.004328 (-0.001724) | 0.048618 / 0.004250 (0.044367) | 0.044088 / 0.037052 (0.007035) | 0.263059 / 0.258489 (0.004570) | 0.294602 / 0.293841 (0.000761) | 0.027425 / 0.128546 (-0.101121) | 0.010263 / 0.075646 (-0.065383) | 0.205925 / 0.419271 (-0.213346) | 0.048917 / 0.043533 (0.005384) | 0.264227 / 0.255139 (0.009088) | 0.273339 / 0.283200 (-0.009860) | 0.017783 / 0.141683 (-0.123900) | 1.137526 / 1.452155 (-0.314629) | 1.179551 / 1.492716 (-0.313165) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096809 / 0.018006 (0.078802) | 0.303854 / 0.000490 (0.303364) | 0.000207 / 0.000200 (0.000007) | 0.000042 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.017756 / 0.037411 (-0.019655) | 0.061005 / 0.014526 (0.046479) | 0.072986 / 0.176557 (-0.103571) | 0.119851 / 0.737135 (-0.617284) | 0.074733 / 0.296338 (-0.221605) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.278270 / 0.215209 (0.063061) | 2.737874 / 2.077655 (0.660219) | 1.460658 / 1.504120 (-0.043462) | 1.337695 / 1.541195 (-0.203499) | 1.364376 / 1.468490 (-0.104114) | 0.565622 / 4.584777 (-4.019155) | 2.365167 / 3.745712 (-1.380546) | 2.694544 / 5.269862 (-2.575317) | 1.699689 / 4.565676 (-2.865987) | 0.062564 / 0.424275 (-0.361712) | 0.005296 / 0.007607 (-0.002311) | 0.340122 / 0.226044 (0.114077) | 3.382133 / 2.268929 (1.113204) | 1.816907 / 55.444624 (-53.627718) | 1.530825 / 6.876477 (-5.345652) | 1.533266 / 2.142072 (-0.608807) | 0.638215 / 4.805227 (-4.167012) | 0.116227 / 6.500664 (-6.384437) | 0.041548 / 0.075469 (-0.033921) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.971031 / 1.841788 (-0.870757) | 11.117905 / 8.074308 (3.043597) | 9.358159 / 10.191392 (-0.833233) | 0.127954 / 0.680424 (-0.552470) | 0.013634 / 0.534201 (-0.520567) | 0.285399 / 0.579283 (-0.293885) | 0.267980 / 0.434364 (-0.166383) | 0.320219 / 0.540337 (-0.220119) | 0.416035 / 1.386936 (-0.970901) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005177 / 0.011353 (-0.006176) | 0.003078 / 0.011008 (-0.007930) | 0.049650 / 0.038508 (0.011142) | 0.030897 / 0.023109 (0.007787) | 0.271186 / 0.275898 (-0.004712) | 0.296050 / 0.323480 (-0.027430) | 0.004204 / 0.007986 (-0.003781) | 0.002755 / 0.004328 (-0.001574) | 0.049550 / 0.004250 (0.045300) | 0.039801 / 0.037052 (0.002749) | 0.283243 / 0.258489 (0.024753) | 0.310932 / 0.293841 (0.017091) | 0.029136 / 0.128546 (-0.099410) | 0.010278 / 0.075646 (-0.065368) | 0.059300 / 0.419271 (-0.359971) | 0.032965 / 0.043533 (-0.010568) | 0.272646 / 0.255139 (0.017507) | 0.293697 / 0.283200 (0.010497) | 0.018330 / 0.141683 (-0.123353) | 1.144251 / 1.452155 (-0.307904) | 1.209660 / 1.492716 (-0.283056) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091020 / 0.018006 (0.073014) | 0.298294 / 0.000490 (0.297804) | 0.000214 / 0.000200 (0.000014) | 0.000053 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021879 / 0.037411 (-0.015532) | 0.074728 / 0.014526 (0.060202) | 0.085499 / 0.176557 (-0.091057) | 0.125743 / 0.737135 (-0.611392) | 0.086130 / 0.296338 (-0.210208) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.292311 / 0.215209 (0.077102) | 2.861240 / 2.077655 (0.783585) | 1.590426 / 1.504120 (0.086306) | 1.472288 / 1.541195 (-0.068907) | 1.472901 / 1.468490 (0.004411) | 0.574924 / 4.584777 (-4.009853) | 2.450817 / 3.745712 (-1.294895) | 2.781903 / 5.269862 (-2.487959) | 1.747110 / 4.565676 (-2.818566) | 0.064680 / 0.424275 (-0.359595) | 0.005376 / 0.007607 (-0.002231) | 0.356846 / 0.226044 (0.130802) | 3.457851 / 2.268929 (1.188922) | 1.952678 / 55.444624 (-53.491946) | 1.670824 / 6.876477 (-5.205653) | 1.655872 / 2.142072 (-0.486200) | 0.655874 / 4.805227 (-4.149353) | 0.117098 / 6.500664 (-6.383566) | 0.040230 / 0.075469 (-0.035239) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.007423 / 1.841788 (-0.834365) | 11.818228 / 8.074308 (3.743920) | 10.153699 / 10.191392 (-0.037693) | 0.132073 / 0.680424 (-0.548351) | 0.015101 / 0.534201 (-0.519100) | 0.286555 / 0.579283 (-0.292728) | 0.281953 / 0.434364 (-0.152411) | 0.323647 / 0.540337 (-0.216691) | 0.418698 / 1.386936 (-0.968238) |

![](https://cml.dev/watermark.png#0d3c7462bc67407c42d3ad102b7f9d5914219d9d "CML watermark")


Winogrande does not seem to be compatible with datasets version of 1.18.0

Hi ! Do you mean 2.18 ? Can you try to update `fsspec` and `huggingface_hub` ?

```
pip install -U fsspec huggingface_hub
```

Yes I meant 2.18, and it works after updating `fsspec` and `huggingface_hub`. Thanks!

### Describe the bug

I get the following error when simply running `load_dataset('winogrande','winogrande_xl')`.

I do not have such an issue in the 1.17.0 version.

```Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2556, in load_dataset
    builder_instance = load_dataset_builder(
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2265, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 371, in __init__
    self.config, self.config_id = self._create_builder_config(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 620, in _create_builder_config
    builder_config._resolve_data_files(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 211, in _resolve_data_files
    self.data_files = self.data_files.resolve(base_path, download_config)
  File "/usr/local/lib/python3.10/dist-packages/datasets/data_files.py", line 799, in resolve
    out[key] = data_files_patterns_list.resolve(base_path, download_config)
  File "/usr/local/lib/python3.10/dist-packages/datasets/data_files.py", line 752, in resolve
    resolve_pattern(
  File "/usr/local/lib/python3.10/dist-packages/datasets/data_files.py", line 393, in resolve_pattern
    raise FileNotFoundError(error_msg)
FileNotFoundError: Unable to find 'hf://datasets/winogrande@ebf71e3c7b5880d019ecf6099c0b09311b1084f5/winogrande_xl/train/0000.parquet' with any supported extension ['.csv', '.tsv', '.json', '.jsonl', '.parquet', '.geoparquet', '.gpq', '.arrow', '.txt', '.tar', '.blp', '.bmp', '.dib', '.bufr', '.cur', '.pcx', '.dcx', '.dds', '.ps', '.eps', '.fit', '.fits', '.fli', '.flc', '.ftc', '.ftu', '.gbr', '.gif', '.grib', '.h5', '.hdf', '.png', '.apng', '.jp2', '.j2k', '.jpc', '.jpf', '.jpx', '.j2c', '.icns', '.ico', '.im', '.iim', '.tif', '.tiff', '.jfif', '.jpe', '.jpg', '.jpeg', '.mpg', '.mpeg', '.msp', '.pcd', '.pxr', '.pbm', '.pgm', '.ppm', '.pnm', '.psd', '.bw', '.rgb', '.rgba', '.sgi', '.ras', '.tga', '.icb', '.vda', '.vst', '.webp', '.wmf', '.emf', '.xbm', '.xpm', '.BLP', '.BMP', '.DIB', '.BUFR', '.CUR', '.PCX', '.DCX', '.DDS', '.PS', '.EPS', '.FIT', '.FITS', '.FLI', '.FLC', '.FTC', '.FTU', '.GBR', '.GIF', '.GRIB', '.H5', '.HDF', '.PNG', '.APNG', '.JP2', '.J2K', '.JPC', '.JPF', '.JPX', '.J2C', '.ICNS', '.ICO', '.IM', '.IIM', '.TIF', '.TIFF', '.JFIF', '.JPE', '.JPG', '.JPEG', '.MPG', '.MPEG', '.MSP', '.PCD', '.PXR', '.PBM', '.PGM', '.PPM', '.PNM', '.PSD', '.BW', '.RGB', '.RGBA', '.SGI', '.RAS', '.TGA', '.ICB', '.VDA', '.VST', '.WEBP', '.WMF', '.EMF', '.XBM', '.XPM', '.aiff', '.au', '.avr', '.caf', '.flac', '.htk', '.svx', '.mat4', '.mat5', '.mpc2k', '.ogg', '.paf', '.pvf', '.raw', '.rf64', '.sd2', '.sds', '.ircam', '.voc', '.w64', '.wav', '.nist', '.wavex', '.wve', '.xi', '.mp3', '.opus', '.AIFF', '.AU', '.AVR', '.CAF', '.FLAC', '.HTK', '.SVX', '.MAT4', '.MAT5', '.MPC2K', '.OGG', '.PAF', '.PVF', '.RAW', '.RF64', '.SD2', '.SDS', '.IRCAM', '.VOC', '.W64', '.WAV', '.NIST', '.WAVEX', '.WVE', '.XI', '.MP3', '.OPUS', '.zip']```

### Steps to reproduce the bug

from datasets import load_dataset
datasets = load_dataset('winogrande','winogrande_xl')

### Expected behavior

```Downloading data: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.06M/2.06M [00:00<00:00, 5.16MB/s]
Downloading data: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 118k/118k [00:00<00:00, 360kB/s]
Downloading data: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 85.9k/85.9k [00:00<00:00, 242kB/s]
Generating train split: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 40398/40398 [00:00<00:00, 845491.12 examples/s]
Generating test split: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1767/1767 [00:00<00:00, 362501.11 examples/s]
Generating validation split: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 1267/1267 [00:00<00:00, 318768.11 examples/s]```

### Environment info

datasets version: 1.18.0


Loading problems of Datasets with a single shard

### Describe the bug

When saving a dataset on disk and it has a single shard it is not loaded as when it is saved in multiple shards. I installed the latest version of datasets via pip.

### Steps to reproduce the bug

The code below reproduces the behavior. All works well when the range of the loop is 10000 but it fails when it is 1000.

```
from PIL import Image
import numpy as np
from datasets import Dataset, DatasetDict, load_dataset

def load_image():
    # Generate random noise image
    noise = np.random.randint(0, 256, (256, 256, 3), dtype=np.uint8)
    return Image.fromarray(noise)

def create_dataset():
    input_images = []
    output_images = []
    text_prompts = []

    for _ in range(10000):  # this is the problematic parameter
        input_images.append(load_image())
        output_images.append(load_image())
        text_prompts.append('test prompt')

    data = {'input_image': input_images, 'output_image': output_images, 'text_prompt': text_prompts}
    dataset = Dataset.from_dict(data)

    return DatasetDict({'train': dataset})

dataset = create_dataset()

print('dataset before saving')
print(dataset)
print(dataset['train'].column_names)
dataset.save_to_disk('test_ds')
print('dataset after loading')

dataset_loaded = load_dataset('test_ds')
print(dataset_loaded)
print(dataset_loaded['train'].column_names)
```

The output for 1000 iterations is:

```
dataset before saving
DatasetDict({
    train: Dataset({
        features: ['input_image', 'output_image', 'text_prompt'],
        num_rows: 1000
    })
})
['input_image', 'output_image', 'text_prompt']
Saving the dataset (1/1 shards): 100%|█| 1000/1000 [00:00<00:00, 5156.00 example
dataset after loading
Generating train split: 1 examples [00:00, 230.52 examples/s]
DatasetDict({
    train: Dataset({
        features: ['_data_files', '_fingerprint', '_format_columns', '_format_kwargs', '_format_type', '_output_all_columns', '_split'],
        num_rows: 1
    })
})
['_data_files', '_fingerprint', '_format_columns', '_format_kwargs', '_format_type', '_output_all_columns', '_split']
```

For 10000 iteration (8 shards) it is correct:

```
dataset before saving
DatasetDict({
    train: Dataset({
        features: ['input_image', 'output_image', 'text_prompt'],
        num_rows: 10000
    })
})
['input_image', 'output_image', 'text_prompt']
Saving the dataset (8/8 shards): 100%|█| 10000/10000 [00:01<00:00, 6237.68 examp
dataset after loading
Generating train split: 10000 examples [00:00, 10773.16 examples/s]
DatasetDict({
    train: Dataset({
        features: ['input_image', 'output_image', 'text_prompt'],
        num_rows: 10000
    })
})
['input_image', 'output_image', 'text_prompt']
```

### Expected behavior

The procedure should work for a dataset with one shrad the same as for one with multiple shards

### Environment info

- `datasets` version: 2.18.0
- Platform: macOS-14.1-arm64-arm-64bit
- Python version: 3.11.8
- `huggingface_hub` version: 0.22.2
- PyArrow version: 15.0.2
- Pandas version: 2.2.2
- `fsspec` version: 2024.2.0

Edit: I looked in the source code of load.py in datasets. I should have used "load_from_disk" and it indeed works that way. But ideally load_dataset would have raisen an error the same way as if I call a path:

```
    if Path(path, config.DATASET_STATE_JSON_FILENAME).exists():
        raise ValueError(
            "You are trying to load a dataset that was saved using `save_to_disk`. "
            "Please use `load_from_disk` instead."
        )
```
nevertheless I find it interesting that it works just well and without a warning if there are multiple shards.


Fix parquet export infos

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6822). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005084 / 0.011353 (-0.006269) | 0.003658 / 0.011008 (-0.007351) | 0.063369 / 0.038508 (0.024860) | 0.030739 / 0.023109 (0.007630) | 0.244335 / 0.275898 (-0.031564) | 0.271731 / 0.323480 (-0.051749) | 0.004133 / 0.007986 (-0.003853) | 0.002798 / 0.004328 (-0.001530) | 0.048790 / 0.004250 (0.044540) | 0.044054 / 0.037052 (0.007002) | 0.261514 / 0.258489 (0.003025) | 0.292155 / 0.293841 (-0.001686) | 0.027971 / 0.128546 (-0.100575) | 0.010723 / 0.075646 (-0.064923) | 0.207328 / 0.419271 (-0.211944) | 0.035928 / 0.043533 (-0.007605) | 0.245320 / 0.255139 (-0.009819) | 0.268774 / 0.283200 (-0.014426) | 0.017119 / 0.141683 (-0.124564) | 1.107052 / 1.452155 (-0.345103) | 1.151752 / 1.492716 (-0.340965) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.089941 / 0.018006 (0.071935) | 0.299788 / 0.000490 (0.299298) | 0.000211 / 0.000200 (0.000012) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018159 / 0.037411 (-0.019252) | 0.061876 / 0.014526 (0.047350) | 0.074733 / 0.176557 (-0.101824) | 0.122070 / 0.737135 (-0.615065) | 0.076100 / 0.296338 (-0.220238) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282209 / 0.215209 (0.067000) | 2.758098 / 2.077655 (0.680444) | 1.482454 / 1.504120 (-0.021666) | 1.372649 / 1.541195 (-0.168546) | 1.373171 / 1.468490 (-0.095319) | 0.563606 / 4.584777 (-4.021171) | 2.406760 / 3.745712 (-1.338952) | 2.796322 / 5.269862 (-2.473540) | 1.732327 / 4.565676 (-2.833350) | 0.063623 / 0.424275 (-0.360652) | 0.005338 / 0.007607 (-0.002269) | 0.337562 / 0.226044 (0.111518) | 3.345225 / 2.268929 (1.076296) | 1.844353 / 55.444624 (-53.600271) | 1.551003 / 6.876477 (-5.325474) | 1.570623 / 2.142072 (-0.571449) | 0.644843 / 4.805227 (-4.160385) | 0.118811 / 6.500664 (-6.381853) | 0.041731 / 0.075469 (-0.033738) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.970469 / 1.841788 (-0.871319) | 11.775531 / 8.074308 (3.701222) | 9.757852 / 10.191392 (-0.433540) | 0.130187 / 0.680424 (-0.550237) | 0.013654 / 0.534201 (-0.520547) | 0.328387 / 0.579283 (-0.250896) | 0.268181 / 0.434364 (-0.166183) | 0.325230 / 0.540337 (-0.215107) | 0.421055 / 1.386936 (-0.965881) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005846 / 0.011353 (-0.005507) | 0.003606 / 0.011008 (-0.007402) | 0.050787 / 0.038508 (0.012279) | 0.031635 / 0.023109 (0.008526) | 0.277040 / 0.275898 (0.001142) | 0.300544 / 0.323480 (-0.022936) | 0.004200 / 0.007986 (-0.003786) | 0.002749 / 0.004328 (-0.001580) | 0.049449 / 0.004250 (0.045198) | 0.041616 / 0.037052 (0.004564) | 0.289570 / 0.258489 (0.031081) | 0.316138 / 0.293841 (0.022297) | 0.029578 / 0.128546 (-0.098969) | 0.010582 / 0.075646 (-0.065064) | 0.058284 / 0.419271 (-0.360988) | 0.033078 / 0.043533 (-0.010455) | 0.277964 / 0.255139 (0.022825) | 0.295008 / 0.283200 (0.011808) | 0.017753 / 0.141683 (-0.123930) | 1.128635 / 1.452155 (-0.323519) | 1.190142 / 1.492716 (-0.302575) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091504 / 0.018006 (0.073498) | 0.303875 / 0.000490 (0.303385) | 0.000221 / 0.000200 (0.000021) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021413 / 0.037411 (-0.015998) | 0.074825 / 0.014526 (0.060299) | 0.086329 / 0.176557 (-0.090228) | 0.125632 / 0.737135 (-0.611503) | 0.087918 / 0.296338 (-0.208420) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.297914 / 0.215209 (0.082705) | 2.922885 / 2.077655 (0.845230) | 1.625758 / 1.504120 (0.121638) | 1.500174 / 1.541195 (-0.041021) | 1.517162 / 1.468490 (0.048672) | 0.576885 / 4.584777 (-4.007892) | 2.458723 / 3.745712 (-1.286989) | 2.798471 / 5.269862 (-2.471391) | 1.762499 / 4.565676 (-2.803178) | 0.064736 / 0.424275 (-0.359539) | 0.005325 / 0.007607 (-0.002282) | 0.351697 / 0.226044 (0.125652) | 3.496223 / 2.268929 (1.227294) | 1.977535 / 55.444624 (-53.467090) | 1.695223 / 6.876477 (-5.181254) | 1.689692 / 2.142072 (-0.452381) | 0.656404 / 4.805227 (-4.148823) | 0.123106 / 6.500664 (-6.377558) | 0.040980 / 0.075469 (-0.034489) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.036972 / 1.841788 (-0.804816) | 12.163931 / 8.074308 (4.089623) | 10.297927 / 10.191392 (0.106535) | 0.144087 / 0.680424 (-0.536337) | 0.015553 / 0.534201 (-0.518648) | 0.286225 / 0.579283 (-0.293058) | 0.275567 / 0.434364 (-0.158797) | 0.332717 / 0.540337 (-0.207620) | 0.423804 / 1.386936 (-0.963132) |

![](https://cml.dev/watermark.png#0bc709af303c8dc64c973a17016bd5aa5db2f3d5 "CML watermark")

Don't use the parquet export infos when USE_PARQUET_EXPORT is False.

Otherwise the `datasets-server` might reuse erroneous data when re-running a job

this follows https://github.com/huggingface/datasets/pull/6714