Fix docs phrasing about supported formats when sharing a dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6486). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005042 / 0.011353 (-0.006311) | 0.003452 / 0.011008 (-0.007557) | 0.061845 / 0.038508 (0.023337) | 0.052042 / 0.023109 (0.028933) | 0.241791 / 0.275898 (-0.034107) | 0.264639 / 0.323480 (-0.058841) | 0.003940 / 0.007986 (-0.004045) | 0.002768 / 0.004328 (-0.001560) | 0.047851 / 0.004250 (0.043600) | 0.037599 / 0.037052 (0.000547) | 0.251462 / 0.258489 (-0.007028) | 0.274737 / 0.293841 (-0.019104) | 0.027723 / 0.128546 (-0.100823) | 0.010510 / 0.075646 (-0.065137) | 0.205581 / 0.419271 (-0.213691) | 0.035504 / 0.043533 (-0.008029) | 0.242380 / 0.255139 (-0.012759) | 0.259791 / 0.283200 (-0.023409) | 0.017752 / 0.141683 (-0.123931) | 1.089289 / 1.452155 (-0.362865) | 1.161958 / 1.492716 (-0.330759) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094288 / 0.018006 (0.076282) | 0.303253 / 0.000490 (0.302763) | 0.000216 / 0.000200 (0.000016) | 0.000042 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018496 / 0.037411 (-0.018915) | 0.060411 / 0.014526 (0.045885) | 0.074294 / 0.176557 (-0.102262) | 0.122934 / 0.737135 (-0.614201) | 0.074710 / 0.296338 (-0.221629) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.286394 / 0.215209 (0.071185) | 2.806145 / 2.077655 (0.728490) | 1.497071 / 1.504120 (-0.007049) | 1.362254 / 1.541195 (-0.178940) | 1.389642 / 1.468490 (-0.078848) | 0.554503 / 4.584777 (-4.030274) | 2.348029 / 3.745712 (-1.397684) | 2.780862 / 5.269862 (-2.489000) | 1.728058 / 4.565676 (-2.837619) | 0.062617 / 0.424275 (-0.361658) | 0.004901 / 0.007607 (-0.002707) | 0.346267 / 0.226044 (0.120223) | 3.363744 / 2.268929 (1.094815) | 1.826994 / 55.444624 (-53.617630) | 1.560656 / 6.876477 (-5.315820) | 1.561083 / 2.142072 (-0.580990) | 0.643395 / 4.805227 (-4.161832) | 0.116206 / 6.500664 (-6.384458) | 0.042008 / 0.075469 (-0.033461) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.953416 / 1.841788 (-0.888371) | 11.461665 / 8.074308 (3.387357) | 10.623865 / 10.191392 (0.432473) | 0.128071 / 0.680424 (-0.552353) | 0.014277 / 0.534201 (-0.519924) | 0.288810 / 0.579283 (-0.290474) | 0.267575 / 0.434364 (-0.166788) | 0.327422 / 0.540337 (-0.212916) | 0.435151 / 1.386936 (-0.951785) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005242 / 0.011353 (-0.006111) | 0.003515 / 0.011008 (-0.007493) | 0.048483 / 0.038508 (0.009975) | 0.051684 / 0.023109 (0.028575) | 0.276564 / 0.275898 (0.000666) | 0.297582 / 0.323480 (-0.025898) | 0.004117 / 0.007986 (-0.003869) | 0.002610 / 0.004328 (-0.001719) | 0.047811 / 0.004250 (0.043561) | 0.040622 / 0.037052 (0.003569) | 0.280265 / 0.258489 (0.021776) | 0.311719 / 0.293841 (0.017878) | 0.028811 / 0.128546 (-0.099735) | 0.010600 / 0.075646 (-0.065047) | 0.056660 / 0.419271 (-0.362611) | 0.032638 / 0.043533 (-0.010894) | 0.276434 / 0.255139 (0.021295) | 0.299095 / 0.283200 (0.015896) | 0.018483 / 0.141683 (-0.123200) | 1.156382 / 1.452155 (-0.295773) | 1.252205 / 1.492716 (-0.240511) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.097868 / 0.018006 (0.079862) | 0.309438 / 0.000490 (0.308948) | 0.000229 / 0.000200 (0.000029) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021838 / 0.037411 (-0.015573) | 0.068358 / 0.014526 (0.053832) | 0.080432 / 0.176557 (-0.096125) | 0.119788 / 0.737135 (-0.617348) | 0.081742 / 0.296338 (-0.214597) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.301239 / 0.215209 (0.086030) | 2.962242 / 2.077655 (0.884587) | 1.693918 / 1.504120 (0.189798) | 1.573663 / 1.541195 (0.032468) | 1.583125 / 1.468490 (0.114635) | 0.557267 / 4.584777 (-4.027510) | 2.440048 / 3.745712 (-1.305664) | 2.727572 / 5.269862 (-2.542290) | 1.713557 / 4.565676 (-2.852120) | 0.062526 / 0.424275 (-0.361749) | 0.004982 / 0.007607 (-0.002625) | 0.353850 / 0.226044 (0.127806) | 3.530887 / 2.268929 (1.261958) | 2.047864 / 55.444624 (-53.396761) | 1.770776 / 6.876477 (-5.105701) | 1.757621 / 2.142072 (-0.384451) | 0.633847 / 4.805227 (-4.171381) | 0.114055 / 6.500664 (-6.386609) | 0.040078 / 0.075469 (-0.035391) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.983721 / 1.841788 (-0.858066) | 11.896537 / 8.074308 (3.822229) | 10.529883 / 10.191392 (0.338491) | 0.129593 / 0.680424 (-0.550831) | 0.016213 / 0.534201 (-0.517988) | 0.289623 / 0.579283 (-0.289660) | 0.280073 / 0.434364 (-0.154291) | 0.327446 / 0.540337 (-0.212892) | 0.574847 / 1.386936 (-0.812089) |

![](https://cml.dev/watermark.png#2684a98fe38e0c87bb11e050586004108e32b79d "CML watermark")

Fix docs phrasing.


Hi! It seems like the problem is your environment. Maybe this issue can help: https://github.com/pytest-dev/pytest/issues/9519.

### Describe the bug

it seems that sth wrong with my terrible "bug body" life, When i run this code, "import datasets"
i meet this error FileNotFoundError: [Errno 2] No such file or directory: 'nul'
![image](https://github.com/huggingface/datasets/assets/73683903/3973c120-ebb1-42b7-bede-b9de053e861d)
![image](https://github.com/huggingface/datasets/assets/73683903/0496adff-a7a7-4dcb-929e-ec11ede71f04)

### Steps to reproduce the bug

1.import datasets

### Expected behavior

i just run a single line code and stuct in this bug

### Environment info

OS: Windows10
Datasets==2.15.0
python=3.10


Iterable Dataset: rename column clashes with remove column

Column "text" doesn't exist anymore so you can't remove it

You can get the expected result by fixing typos in the snippet :)
```python
from datasets import load_dataset

# load LS in streaming mode
dataset = load_dataset("librispeech_asr", "clean", split="validation", streaming=True)

# check original features
dataset_features = dataset.features.keys()
print("Original features: ", dataset_features)

# rename "text" -> "sentence"
dataset = dataset.rename_column("text", "sentence")

# remove unwanted columns
COLUMNS_TO_KEEP = {"audio", "sentence"}
dataset = dataset.remove_columns(set(dataset.features) - COLUMNS_TO_KEEP)

# stream first sample, should return "audio" and "sentence" columns
print(next(iter(dataset)))
```

Fixed code:

```python
from datasets import load_dataset

# load LS in streaming mode
dataset = load_dataset("librispeech_asr", "clean", split="validation", streaming=True)

# check original features
dataset_features = dataset.features.keys()
print("Original features: ", dataset_features)

# rename "text" -> "sentence"
dataset = dataset.rename_column("text", "sentence")
dataset_features = dataset.features.keys()

# remove unwanted columns
COLUMNS_TO_KEEP = {"audio", "sentence"}
dataset = dataset.remove_columns(set(dataset_features - COLUMNS_TO_KEEP))

# stream first sample, should return "audio" and "sentence" columns
print(next(iter(dataset)))
```

Whoops 😅 Thanks for the swift reply both! Works like a charm!

### Describe the bug

Suppose I have a two iterable datasets, one with the features:
* `{"audio", "text", "column_a"}`

And the other with the features:
* `{"audio", "sentence", "column_b"}`

I want to combine both datasets using `interleave_datasets`, which requires me to unify the column names. I would typically do this by:
1. Renaming the common columns to the same name (e.g. `"text"` -> `"sentence"`)
2. Removing the unwanted columns (e.g. `"column_a"`, `"column_b"`)

However, the process of renaming and removing columns in an iterable dataset doesn't work, since we need to preserve the original text column, meaning we can't combine the datasets.

### Steps to reproduce the bug

```python
from datasets import load_dataset

# load LS in streaming mode
dataset = load_dataset("librispeech_asr", "clean", split="validation", streaming=True)

# check original features
dataset_features = dataset.features.keys()
print("Original features: ", dataset_features)

# rename "text" -> "sentence"
dataset = dataset.rename_column("text", "sentence")

# remove unwanted columns
COLUMNS_TO_KEEP = {"audio", "sentence"}
dataset = dataset.remove_columns(set(dataset_features - COLUMNS_TO_KEEP))

# stream first sample, should return "audio" and "sentence" columns
print(next(iter(dataset)))
```
Traceback:
```python
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[5], line 17
     14 COLUMNS_TO_KEEP = {"audio", "sentence"}
     15 dataset = dataset.remove_columns(set(dataset_features - COLUMNS_TO_KEEP))
---> 17 print(next(iter(dataset)))

File ~/datasets/src/datasets/iterable_dataset.py:1353, in IterableDataset.__iter__(self)
   1350         yield formatter.format_row(pa_table)
   1351     return
-> 1353 for key, example in ex_iterable:
   1354     if self.features:
   1355         # `IterableDataset` automatically fills missing columns with None.
   1356         # This is done with `_apply_feature_types_on_example`.
   1357         example = _apply_feature_types_on_example(
   1358             example, self.features, token_per_repo_id=self._token_per_repo_id
   1359         )

File ~/datasets/src/datasets/iterable_dataset.py:652, in MappedExamplesIterable.__iter__(self)
    650     yield from ArrowExamplesIterable(self._iter_arrow, {})
    651 else:
--> 652     yield from self._iter()

File ~/datasets/src/datasets/iterable_dataset.py:729, in MappedExamplesIterable._iter(self)
    727 if self.remove_columns:
    728     for c in self.remove_columns:
--> 729         del transformed_example[c]
    730 yield key, transformed_example
    731 current_idx += 1

KeyError: 'text'
```
=> we see that `datasets` is looking for the column "text", even though we've renamed this to "sentence" and then removed the un-wanted "text" column from our dataset.

### Expected behavior

Should be able to rename and remove columns from iterable dataset.

### Environment info

- `datasets` version: 2.15.1.dev0
- Platform: macOS-13.5.1-arm64-arm-64bit
- Python version: 3.11.6
- `huggingface_hub` version: 0.19.4
- PyArrow version: 14.0.1
- Pandas version: 2.1.2
- `fsspec` version: 2023.9.2


[Feature Request] Dataset versioning

Hello @kenfus, this is meant to be possible to do yes. Let me ping @lhoestq or @mariosasko from the `datasets` team (`huggingface_hub` is only the underlying library to download files from the Hub but here it looks more like a `datasets` problem).

Hi! https://github.com/huggingface/datasets/pull/6459 will fix this.

**Is your feature request related to a problem? Please describe.**
I am working on a project, where I would like to test different preprocessing methods for my ML-data. Thus, I would like to work a lot with revisions and compare them. Currently, I was not able to make it work with the revision keyword because it was not redownloading the data, it was reading in some cached data, until I put  `download_mode="force_redownload"`, even though the reversion was different.
Of course, I may have done something wrong or missed a setting somewhere!

**Describe the solution you'd like**
The solution would allow me to easily work with revisions:
- create a new dataset (by combining things, different preprocessing, ..) and give it a new revision (v.1.2.3), maybe like this:
`dataset_audio.push_to_hub('kenfus/xy', revision='v1.0.2')`

- then, get the current revision as follows:
```
dataset = load_dataset(
            'kenfus/xy', revision='v1.0.2',
        )
```
this downloads the new version and does not load in a different revision and all future map, filter, .. operations are done on this dataset and not loaded from cache produced from a different revision.
- if I rerun the run, the caching should be smart enough in every step to not reuse a mapping operation on a different revision.

**Describe alternatives you've considered**
I created my own caching, putting `download_mode="force_redownload"` and `load_from_cache_file=False,` everywhere.

**Additional context**
Thanks a lot for your great work! Creating NLP datasets and training a model with them is really easy and straightforward with huggingface.

This is the data loading in my script:

```
    ## CREATE PATHS
    prepared_dataset_path = os.path.join(
        DATA_FOLDER, str(DATA_VERSION), "prepared_dataset"
    )
    os.makedirs(os.path.join(DATA_FOLDER, str(DATA_VERSION)), exist_ok=True)

    ## LOAD DATASET
    if os.path.exists(prepared_dataset_path):
        print("Loading prepared dataset from disk...")
        dataset_prepared = load_from_disk(prepared_dataset_path)
    else:
        print("Loading dataset from HuggingFace Datasets...")
        dataset = load_dataset(
            PATH_TO_DATASET, revision=DATA_VERSION, download_mode="force_redownload"
        )

        print("Preparing dataset...")
        dataset_prepared = dataset.map(
            prepare_dataset,
            remove_columns=["audio", "transcription"],
            num_proc=os.cpu_count(),
            load_from_cache_file=False,
        )
        dataset_prepared.save_to_disk(prepared_dataset_path)
        del dataset

    if CHECK_DATASET:
        ## CHECK DATASET
        dataset_prepared = dataset_prepared.map(
            check_dimensions, num_proc=os.cpu_count(), load_from_cache_file=False
        )
        dataset_filtered = dataset_prepared.filter(
            lambda example: not example["incorrect_dimension"],
            load_from_cache_file=False,
        )

        for example in dataset_prepared.filter(
            lambda example: example["incorrect_dimension"], load_from_cache_file=False
        ):
            print(example["path"])

        print(
            f"Number of examples with incorrect dimension:  {len(dataset_prepared) - len(dataset_filtered)}"
        )

        print("Number of examples train: ", len(dataset_filtered["train"]))
        print("Number of examples test: ", len(dataset_filtered["test"]))

```


Fix max lock length on unix

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6482). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I'm getting `AttributeError: module 'os' has no attribute 'statvfs'` on windows - reverting

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005294 / 0.011353 (-0.006059) | 0.003562 / 0.011008 (-0.007446) | 0.062030 / 0.038508 (0.023522) | 0.053335 / 0.023109 (0.030226) | 0.233303 / 0.275898 (-0.042595) | 0.252029 / 0.323480 (-0.071451) | 0.002835 / 0.007986 (-0.005151) | 0.002732 / 0.004328 (-0.001597) | 0.047973 / 0.004250 (0.043723) | 0.038380 / 0.037052 (0.001328) | 0.235028 / 0.258489 (-0.023461) | 0.265555 / 0.293841 (-0.028286) | 0.027136 / 0.128546 (-0.101410) | 0.010806 / 0.075646 (-0.064840) | 0.205040 / 0.419271 (-0.214231) | 0.035063 / 0.043533 (-0.008470) | 0.236351 / 0.255139 (-0.018788) | 0.254556 / 0.283200 (-0.028643) | 0.019528 / 0.141683 (-0.122155) | 1.099012 / 1.452155 (-0.353142) | 1.156250 / 1.492716 (-0.336466) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093952 / 0.018006 (0.075946) | 0.304181 / 0.000490 (0.303692) | 0.000227 / 0.000200 (0.000027) | 0.000045 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018568 / 0.037411 (-0.018844) | 0.060323 / 0.014526 (0.045798) | 0.073010 / 0.176557 (-0.103546) | 0.121723 / 0.737135 (-0.615412) | 0.075668 / 0.296338 (-0.220670) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.288429 / 0.215209 (0.073220) | 2.797834 / 2.077655 (0.720180) | 1.480957 / 1.504120 (-0.023163) | 1.360872 / 1.541195 (-0.180323) | 1.406828 / 1.468490 (-0.061663) | 0.587596 / 4.584777 (-3.997181) | 2.533997 / 3.745712 (-1.211715) | 2.906697 / 5.269862 (-2.363164) | 1.801753 / 4.565676 (-2.763923) | 0.064360 / 0.424275 (-0.359915) | 0.005016 / 0.007607 (-0.002591) | 0.347334 / 0.226044 (0.121290) | 3.426344 / 2.268929 (1.157416) | 1.856014 / 55.444624 (-53.588610) | 1.581774 / 6.876477 (-5.294703) | 1.640036 / 2.142072 (-0.502037) | 0.656096 / 4.805227 (-4.149131) | 0.120212 / 6.500664 (-6.380452) | 0.044003 / 0.075469 (-0.031466) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.943933 / 1.841788 (-0.897855) | 11.846572 / 8.074308 (3.772263) | 10.330705 / 10.191392 (0.139313) | 0.129767 / 0.680424 (-0.550657) | 0.013508 / 0.534201 (-0.520693) | 0.289672 / 0.579283 (-0.289611) | 0.266427 / 0.434364 (-0.167937) | 0.342766 / 0.540337 (-0.197571) | 0.452068 / 1.386936 (-0.934868) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005308 / 0.011353 (-0.006045) | 0.003712 / 0.011008 (-0.007296) | 0.048848 / 0.038508 (0.010340) | 0.055156 / 0.023109 (0.032047) | 0.271942 / 0.275898 (-0.003956) | 0.293166 / 0.323480 (-0.030314) | 0.004056 / 0.007986 (-0.003930) | 0.002722 / 0.004328 (-0.001606) | 0.048418 / 0.004250 (0.044167) | 0.039320 / 0.037052 (0.002268) | 0.277184 / 0.258489 (0.018695) | 0.312398 / 0.293841 (0.018557) | 0.029392 / 0.128546 (-0.099155) | 0.011314 / 0.075646 (-0.064332) | 0.057883 / 0.419271 (-0.361389) | 0.032603 / 0.043533 (-0.010930) | 0.273025 / 0.255139 (0.017886) | 0.289265 / 0.283200 (0.006065) | 0.017553 / 0.141683 (-0.124129) | 1.127725 / 1.452155 (-0.324430) | 1.202293 / 1.492716 (-0.290423) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.097179 / 0.018006 (0.079173) | 0.309712 / 0.000490 (0.309222) | 0.000269 / 0.000200 (0.000069) | 0.000055 / 0.000054 (0.000000) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024742 / 0.037411 (-0.012670) | 0.070097 / 0.014526 (0.055571) | 0.082273 / 0.176557 (-0.094283) | 0.121696 / 0.737135 (-0.615439) | 0.082983 / 0.296338 (-0.213355) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.292688 / 0.215209 (0.077479) | 2.853436 / 2.077655 (0.775781) | 1.588999 / 1.504120 (0.084879) | 1.454547 / 1.541195 (-0.086648) | 1.476342 / 1.468490 (0.007852) | 0.559464 / 4.584777 (-4.025313) | 2.564597 / 3.745712 (-1.181115) | 2.900460 / 5.269862 (-2.369402) | 1.782156 / 4.565676 (-2.783520) | 0.061768 / 0.424275 (-0.362507) | 0.005042 / 0.007607 (-0.002565) | 0.345168 / 0.226044 (0.119124) | 3.412273 / 2.268929 (1.143344) | 1.953154 / 55.444624 (-53.491470) | 1.667347 / 6.876477 (-5.209130) | 1.685138 / 2.142072 (-0.456934) | 0.643270 / 4.805227 (-4.161958) | 0.115955 / 6.500664 (-6.384709) | 0.041090 / 0.075469 (-0.034379) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.976324 / 1.841788 (-0.865464) | 12.252294 / 8.074308 (4.177986) | 10.598062 / 10.191392 (0.406670) | 0.129779 / 0.680424 (-0.550644) | 0.015697 / 0.534201 (-0.518504) | 0.287241 / 0.579283 (-0.292042) | 0.287331 / 0.434364 (-0.147033) | 0.331710 / 0.540337 (-0.208628) | 0.574571 / 1.386936 (-0.812365) |

![](https://cml.dev/watermark.png#702344140461b7a111139860c944d3dd0a2689e3 "CML watermark")

reported in https://github.com/huggingface/datasets/pull/6482


using torchrun, save_to_disk suddenly shows SIGTERM

### Describe the bug

When I run my code using the "torchrun" command, when the code reaches the "save_to_disk" part, suddenly I get the following warning and error messages:
Because the dataset is too large, the "save_to_disk" function splits it into 70 parts for saving. However, an error occurs suddenly when it reaches the 14th shard.
WARNING: torch.distributed.elastic.multiprocessing.api: Sending process 2224968 closing signal SIGTERM
ERROR: torch.distributed.elastic.multiprocessing.api: failed (exitcode: -7). traceback: Signal 7 (SIGBUS) received by PID 2224967.

### Steps to reproduce the bug

ds_shard = ds_shard.map(map_fn, *args, **kwargs)
ds_shard.save_to_disk(ds_shard_filepaths[rank])

Saving the dataset (14/70 shards):  20%|██        | 875350/4376702 [00:19<01:53, 30863.15 examples/s]
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 2224968 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -7) local_rank: 0 (pid: 2224967) of binary: /home/bingxing2/home/scx6964/.conda/envs/ariya235/bin/python
Traceback (most recent call last):
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
    run(args)
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
==========================================================
run.py FAILED
----------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
----------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-12-08_20:09:04
  rank      : 0 (local_rank: 0)
  exitcode  : -7 (pid: 2224967)
  error_file: <N/A>
  traceback : Signal 7 (SIGBUS) received by PID 2224967

### Expected behavior

I hope it can save successfully without any issues, but it seems there is a problem.

### Environment info

`datasets` version: 2.14.6
- Platform: Linux-4.19.90-24.4.v2101.ky10.aarch64-aarch64-with-glibc2.28
- Python version: 3.10.11
- Huggingface_hub version: 0.17.3
- PyArrow version: 14.0.0
- Pandas version: 2.1.2


Add IterableDataset `__repr__`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6480). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005392 / 0.011353 (-0.005960) | 0.003120 / 0.011008 (-0.007888) | 0.062017 / 0.038508 (0.023509) | 0.048824 / 0.023109 (0.025715) | 0.232300 / 0.275898 (-0.043598) | 0.262045 / 0.323480 (-0.061435) | 0.002909 / 0.007986 (-0.005077) | 0.003916 / 0.004328 (-0.000413) | 0.049469 / 0.004250 (0.045218) | 0.038965 / 0.037052 (0.001913) | 0.247841 / 0.258489 (-0.010648) | 0.268259 / 0.293841 (-0.025582) | 0.027588 / 0.128546 (-0.100958) | 0.010334 / 0.075646 (-0.065312) | 0.205811 / 0.419271 (-0.213460) | 0.035456 / 0.043533 (-0.008077) | 0.242774 / 0.255139 (-0.012365) | 0.260377 / 0.283200 (-0.022823) | 0.017469 / 0.141683 (-0.124214) | 1.199665 / 1.452155 (-0.252489) | 1.259316 / 1.492716 (-0.233400) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092357 / 0.018006 (0.074350) | 0.303745 / 0.000490 (0.303255) | 0.000212 / 0.000200 (0.000012) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018820 / 0.037411 (-0.018592) | 0.061548 / 0.014526 (0.047022) | 0.072527 / 0.176557 (-0.104030) | 0.119696 / 0.737135 (-0.617440) | 0.074153 / 0.296338 (-0.222185) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.283952 / 0.215209 (0.068743) | 2.769844 / 2.077655 (0.692189) | 1.526100 / 1.504120 (0.021980) | 1.417584 / 1.541195 (-0.123611) | 1.440523 / 1.468490 (-0.027967) | 0.556994 / 4.584777 (-4.027783) | 2.400392 / 3.745712 (-1.345320) | 2.727794 / 5.269862 (-2.542068) | 1.724671 / 4.565676 (-2.841006) | 0.062111 / 0.424275 (-0.362164) | 0.004925 / 0.007607 (-0.002682) | 0.342748 / 0.226044 (0.116704) | 3.376790 / 2.268929 (1.107862) | 1.856498 / 55.444624 (-53.588127) | 1.574143 / 6.876477 (-5.302334) | 1.591828 / 2.142072 (-0.550245) | 0.644416 / 4.805227 (-4.160811) | 0.116862 / 6.500664 (-6.383802) | 0.041484 / 0.075469 (-0.033985) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.975704 / 1.841788 (-0.866084) | 11.196447 / 8.074308 (3.122139) | 10.567518 / 10.191392 (0.376126) | 0.126786 / 0.680424 (-0.553638) | 0.013768 / 0.534201 (-0.520433) | 0.284531 / 0.579283 (-0.294752) | 0.260855 / 0.434364 (-0.173509) | 0.328888 / 0.540337 (-0.211450) | 0.439911 / 1.386936 (-0.947025) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005108 / 0.011353 (-0.006245) | 0.003006 / 0.011008 (-0.008003) | 0.048673 / 0.038508 (0.010165) | 0.051066 / 0.023109 (0.027957) | 0.279578 / 0.275898 (0.003680) | 0.298356 / 0.323480 (-0.025123) | 0.003965 / 0.007986 (-0.004020) | 0.002662 / 0.004328 (-0.001667) | 0.049037 / 0.004250 (0.044786) | 0.039385 / 0.037052 (0.002333) | 0.284545 / 0.258489 (0.026055) | 0.314240 / 0.293841 (0.020399) | 0.028493 / 0.128546 (-0.100053) | 0.010400 / 0.075646 (-0.065247) | 0.057375 / 0.419271 (-0.361896) | 0.032382 / 0.043533 (-0.011151) | 0.283163 / 0.255139 (0.028024) | 0.298967 / 0.283200 (0.015768) | 0.017564 / 0.141683 (-0.124119) | 1.172425 / 1.452155 (-0.279730) | 1.219975 / 1.492716 (-0.272742) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090664 / 0.018006 (0.072658) | 0.298419 / 0.000490 (0.297929) | 0.000211 / 0.000200 (0.000011) | 0.000045 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021739 / 0.037411 (-0.015672) | 0.068274 / 0.014526 (0.053748) | 0.080820 / 0.176557 (-0.095736) | 0.119809 / 0.737135 (-0.617326) | 0.081612 / 0.296338 (-0.214727) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.303346 / 0.215209 (0.088137) | 2.971648 / 2.077655 (0.893993) | 1.634828 / 1.504120 (0.130708) | 1.510851 / 1.541195 (-0.030344) | 1.515236 / 1.468490 (0.046745) | 0.558487 / 4.584777 (-4.026289) | 2.436263 / 3.745712 (-1.309449) | 2.718525 / 5.269862 (-2.551336) | 1.727421 / 4.565676 (-2.838255) | 0.061396 / 0.424275 (-0.362879) | 0.004951 / 0.007607 (-0.002656) | 0.352950 / 0.226044 (0.126906) | 3.473766 / 2.268929 (1.204838) | 1.971299 / 55.444624 (-53.473325) | 1.712173 / 6.876477 (-5.164304) | 1.711334 / 2.142072 (-0.430738) | 0.627291 / 4.805227 (-4.177936) | 0.113779 / 6.500664 (-6.386885) | 0.046561 / 0.075469 (-0.028908) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.989507 / 1.841788 (-0.852280) | 11.777883 / 8.074308 (3.703575) | 10.525453 / 10.191392 (0.334061) | 0.129118 / 0.680424 (-0.551306) | 0.014989 / 0.534201 (-0.519212) | 0.282324 / 0.579283 (-0.296959) | 0.280688 / 0.434364 (-0.153676) | 0.322579 / 0.540337 (-0.217758) | 0.554327 / 1.386936 (-0.832609) |

![](https://cml.dev/watermark.png#79e94fcdf3d4378ddcdf7e130bb1ae23d99c6fce "CML watermark")

Example for glue sst2:

Dataset

```
DatasetDict({
    test: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 1821
    })
    train: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 67349
    })
    validation: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 872
    })
})
```

IterableDataset (new)

```
IterableDatasetDict({
    test: IterableDataset({
        features: ['sentence', 'label', 'idx'],
        n_shards: 1
    })
    train: IterableDataset({
        features: ['sentence', 'label', 'idx'],
        n_shards: 1
    })
    validation: IterableDataset({
        features: ['sentence', 'label', 'idx'],
        n_shards: 1
    })
})
```

IterableDataset (before)

```
{'test': <datasets.iterable_dataset.IterableDataset object at 0x130d421f0>, 'train': <datasets.iterable_dataset.IterableDataset object at 0x136f3aaf0>, 'validation': <datasets.iterable_dataset.IterableDataset object at 0x136f4b100>}
{'sentence': 'hide new secretions from the parental units ', 'label': 0, 'idx': 0}
```


More robust preupload retry mechanism

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6479). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005669 / 0.011353 (-0.005683) | 0.003684 / 0.011008 (-0.007324) | 0.063477 / 0.038508 (0.024969) | 0.068760 / 0.023109 (0.045651) | 0.252741 / 0.275898 (-0.023157) | 0.286499 / 0.323480 (-0.036981) | 0.003311 / 0.007986 (-0.004674) | 0.003487 / 0.004328 (-0.000842) | 0.049636 / 0.004250 (0.045385) | 0.040983 / 0.037052 (0.003931) | 0.262230 / 0.258489 (0.003740) | 0.292131 / 0.293841 (-0.001710) | 0.028231 / 0.128546 (-0.100315) | 0.010912 / 0.075646 (-0.064734) | 0.211248 / 0.419271 (-0.208023) | 0.036679 / 0.043533 (-0.006854) | 0.258139 / 0.255139 (0.003000) | 0.277568 / 0.283200 (-0.005631) | 0.019576 / 0.141683 (-0.122107) | 1.102588 / 1.452155 (-0.349567) | 1.178587 / 1.492716 (-0.314130) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.098968 / 0.018006 (0.080962) | 0.298777 / 0.000490 (0.298287) | 0.000220 / 0.000200 (0.000020) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.020408 / 0.037411 (-0.017003) | 0.062832 / 0.014526 (0.048306) | 0.076047 / 0.176557 (-0.100509) | 0.125209 / 0.737135 (-0.611926) | 0.079098 / 0.296338 (-0.217240) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.285603 / 0.215209 (0.070394) | 2.811530 / 2.077655 (0.733875) | 1.481012 / 1.504120 (-0.023108) | 1.362740 / 1.541195 (-0.178455) | 1.448999 / 1.468490 (-0.019491) | 0.557740 / 4.584777 (-4.027037) | 2.391377 / 3.745712 (-1.354335) | 2.973181 / 5.269862 (-2.296681) | 1.837147 / 4.565676 (-2.728530) | 0.064445 / 0.424275 (-0.359831) | 0.004992 / 0.007607 (-0.002615) | 0.339207 / 0.226044 (0.113162) | 3.378508 / 2.268929 (1.109580) | 1.843969 / 55.444624 (-53.600655) | 1.597794 / 6.876477 (-5.278682) | 1.657665 / 2.142072 (-0.484407) | 0.654267 / 4.805227 (-4.150961) | 0.120408 / 6.500664 (-6.380256) | 0.045298 / 0.075469 (-0.030171) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.949030 / 1.841788 (-0.892758) | 12.922161 / 8.074308 (4.847852) | 11.115660 / 10.191392 (0.924268) | 0.130556 / 0.680424 (-0.549868) | 0.016278 / 0.534201 (-0.517923) | 0.288137 / 0.579283 (-0.291146) | 0.265978 / 0.434364 (-0.168386) | 0.331491 / 0.540337 (-0.208847) | 0.437782 / 1.386936 (-0.949154) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005342 / 0.011353 (-0.006010) | 0.003636 / 0.011008 (-0.007373) | 0.049527 / 0.038508 (0.011019) | 0.054856 / 0.023109 (0.031746) | 0.271922 / 0.275898 (-0.003976) | 0.295654 / 0.323480 (-0.027826) | 0.004023 / 0.007986 (-0.003963) | 0.002814 / 0.004328 (-0.001515) | 0.048963 / 0.004250 (0.044712) | 0.039936 / 0.037052 (0.002884) | 0.274336 / 0.258489 (0.015847) | 0.310100 / 0.293841 (0.016259) | 0.030006 / 0.128546 (-0.098540) | 0.010750 / 0.075646 (-0.064896) | 0.057989 / 0.419271 (-0.361283) | 0.033692 / 0.043533 (-0.009841) | 0.274084 / 0.255139 (0.018945) | 0.289428 / 0.283200 (0.006229) | 0.018739 / 0.141683 (-0.122944) | 1.126224 / 1.452155 (-0.325931) | 1.171595 / 1.492716 (-0.321121) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093983 / 0.018006 (0.075977) | 0.298516 / 0.000490 (0.298026) | 0.000221 / 0.000200 (0.000022) | 0.000053 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022498 / 0.037411 (-0.014914) | 0.071909 / 0.014526 (0.057383) | 0.083940 / 0.176557 (-0.092617) | 0.121059 / 0.737135 (-0.616076) | 0.084141 / 0.296338 (-0.212198) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.301792 / 0.215209 (0.086583) | 2.971971 / 2.077655 (0.894317) | 1.618718 / 1.504120 (0.114598) | 1.495816 / 1.541195 (-0.045379) | 1.546709 / 1.468490 (0.078219) | 0.571448 / 4.584777 (-4.013329) | 2.459182 / 3.745712 (-1.286531) | 2.937584 / 5.269862 (-2.332278) | 1.804670 / 4.565676 (-2.761007) | 0.062264 / 0.424275 (-0.362011) | 0.004915 / 0.007607 (-0.002692) | 0.355054 / 0.226044 (0.129009) | 3.490468 / 2.268929 (1.221539) | 1.978948 / 55.444624 (-53.465677) | 1.701020 / 6.876477 (-5.175457) | 1.744684 / 2.142072 (-0.397388) | 0.635880 / 4.805227 (-4.169347) | 0.115933 / 6.500664 (-6.384732) | 0.042646 / 0.075469 (-0.032823) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.999486 / 1.841788 (-0.842302) | 13.373854 / 8.074308 (5.299546) | 10.959784 / 10.191392 (0.768392) | 0.131032 / 0.680424 (-0.549392) | 0.015059 / 0.534201 (-0.519142) | 0.289892 / 0.579283 (-0.289391) | 0.279383 / 0.434364 (-0.154981) | 0.337670 / 0.540337 (-0.202668) | 0.597102 / 1.386936 (-0.789834) |

![](https://cml.dev/watermark.png#dd9044cdaabc1f9abce02c1b71bdb48fd3525d4e "CML watermark")


How to load data from lakefs

You can create a `pandas` DataFrame following [this](https://lakefs.io/data-version-control/dvc-using-python/) tutorial, and then convert this DataFrame to a `Dataset` with `datasets.Dataset.from_pandas`. For larger datasets (to memory map them), you can use `Dataset.from_generator` with a generator function that reads lakeFS files with `s3fs`.

@mariosasko  hello,
This can achieve and https://huggingface.co/datasets Does the same effect apply to the dataset? For example, downloading while using

My dataset is stored on the company's lakefs server. How can I write code to load the dataset? It would be great if I could provide code examples or provide some references


Fix PermissionError on Windows CI

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6477). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005383 / 0.011353 (-0.005969) | 0.003644 / 0.011008 (-0.007364) | 0.063375 / 0.038508 (0.024866) | 0.055567 / 0.023109 (0.032457) | 0.261376 / 0.275898 (-0.014522) | 0.283731 / 0.323480 (-0.039749) | 0.004022 / 0.007986 (-0.003964) | 0.002780 / 0.004328 (-0.001549) | 0.049407 / 0.004250 (0.045156) | 0.038208 / 0.037052 (0.001156) | 0.256275 / 0.258489 (-0.002214) | 0.293203 / 0.293841 (-0.000638) | 0.028411 / 0.128546 (-0.100135) | 0.010753 / 0.075646 (-0.064894) | 0.210420 / 0.419271 (-0.208851) | 0.036062 / 0.043533 (-0.007471) | 0.260455 / 0.255139 (0.005317) | 0.294991 / 0.283200 (0.011791) | 0.019020 / 0.141683 (-0.122662) | 1.118334 / 1.452155 (-0.333821) | 1.227391 / 1.492716 (-0.265325) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094700 / 0.018006 (0.076694) | 0.302378 / 0.000490 (0.301888) | 0.000215 / 0.000200 (0.000015) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018745 / 0.037411 (-0.018667) | 0.061103 / 0.014526 (0.046578) | 0.075369 / 0.176557 (-0.101188) | 0.121573 / 0.737135 (-0.615563) | 0.076898 / 0.296338 (-0.219440) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.284143 / 0.215209 (0.068934) | 2.774298 / 2.077655 (0.696644) | 1.483557 / 1.504120 (-0.020563) | 1.365091 / 1.541195 (-0.176104) | 1.390170 / 1.468490 (-0.078320) | 0.561179 / 4.584777 (-4.023598) | 2.401654 / 3.745712 (-1.344058) | 2.782628 / 5.269862 (-2.487233) | 1.731497 / 4.565676 (-2.834179) | 0.061798 / 0.424275 (-0.362477) | 0.004998 / 0.007607 (-0.002609) | 0.336920 / 0.226044 (0.110875) | 3.371891 / 2.268929 (1.102963) | 1.832173 / 55.444624 (-53.612452) | 1.573515 / 6.876477 (-5.302962) | 1.595609 / 2.142072 (-0.546463) | 0.647652 / 4.805227 (-4.157575) | 0.118501 / 6.500664 (-6.382164) | 0.042521 / 0.075469 (-0.032948) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.939310 / 1.841788 (-0.902478) | 11.459855 / 8.074308 (3.385547) | 10.677954 / 10.191392 (0.486562) | 0.141029 / 0.680424 (-0.539395) | 0.014321 / 0.534201 (-0.519880) | 0.306679 / 0.579283 (-0.272604) | 0.262303 / 0.434364 (-0.172061) | 0.327422 / 0.540337 (-0.212915) | 0.436159 / 1.386936 (-0.950777) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005430 / 0.011353 (-0.005923) | 0.003646 / 0.011008 (-0.007362) | 0.049272 / 0.038508 (0.010764) | 0.075367 / 0.023109 (0.052257) | 0.275959 / 0.275898 (0.000061) | 0.296317 / 0.323480 (-0.027163) | 0.004129 / 0.007986 (-0.003857) | 0.002731 / 0.004328 (-0.001597) | 0.048475 / 0.004250 (0.044225) | 0.041571 / 0.037052 (0.004518) | 0.277993 / 0.258489 (0.019504) | 0.298709 / 0.293841 (0.004868) | 0.033117 / 0.128546 (-0.095429) | 0.010914 / 0.075646 (-0.064732) | 0.057599 / 0.419271 (-0.361673) | 0.033354 / 0.043533 (-0.010179) | 0.275669 / 0.255139 (0.020530) | 0.288451 / 0.283200 (0.005251) | 0.019953 / 0.141683 (-0.121729) | 1.148608 / 1.452155 (-0.303547) | 1.184818 / 1.492716 (-0.307898) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.099566 / 0.018006 (0.081560) | 0.344935 / 0.000490 (0.344445) | 0.000221 / 0.000200 (0.000021) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021925 / 0.037411 (-0.015486) | 0.068623 / 0.014526 (0.054097) | 0.081533 / 0.176557 (-0.095024) | 0.120996 / 0.737135 (-0.616139) | 0.082495 / 0.296338 (-0.213844) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.294990 / 0.215209 (0.079781) | 2.892344 / 2.077655 (0.814690) | 1.611090 / 1.504120 (0.106970) | 1.496072 / 1.541195 (-0.045123) | 1.486069 / 1.468490 (0.017579) | 0.569769 / 4.584777 (-4.015008) | 2.477623 / 3.745712 (-1.268089) | 2.819576 / 5.269862 (-2.450286) | 1.745717 / 4.565676 (-2.819959) | 0.063763 / 0.424275 (-0.360512) | 0.004970 / 0.007607 (-0.002637) | 0.344879 / 0.226044 (0.118834) | 3.452795 / 2.268929 (1.183867) | 1.964468 / 55.444624 (-53.480156) | 1.674526 / 6.876477 (-5.201951) | 1.679716 / 2.142072 (-0.462356) | 0.650005 / 4.805227 (-4.155222) | 0.117019 / 6.500664 (-6.383646) | 0.048297 / 0.075469 (-0.027172) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.965422 / 1.841788 (-0.876366) | 11.989414 / 8.074308 (3.915106) | 10.938462 / 10.191392 (0.747070) | 0.140089 / 0.680424 (-0.540334) | 0.015533 / 0.534201 (-0.518668) | 0.292188 / 0.579283 (-0.287095) | 0.277903 / 0.434364 (-0.156461) | 0.326164 / 0.540337 (-0.214173) | 0.565674 / 1.386936 (-0.821262) |

![](https://cml.dev/watermark.png#d78f07091bc42c41bea068bf1b6116e2bde46a6f "CML watermark")

Fix #6476.


CI on windows is broken: PermissionError

bug

Something isn't working

See: https://github.com/huggingface/datasets/actions/runs/7104781624/job/19340572394
```
FAILED tests/test_load.py::test_loading_from_the_datasets_hub - NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpfcnps56i\\hf-internal-testing___dataset_with_script\\default\\0.0.0\\c240e2be3370bdbd\\dataset_with_script-train.arrow'
```


laion2B-en failed to load on Windows with PrefetchVirtualMemory failed

~~You will see this error if the cache dir filepath contains relative `..` paths. Use `os.path.realpath(_CACHE_DIR)` before passing it to the `load_dataset` function.~~

This is a real issue and not related to paths.

Based on the StackOverflow answer, this causes the error to go away:
```diff
diff --git a/table.py b/table.py
--- a/table.py
+++ b/table.py	(date 1701824849806)
@@ -47,7 +47,7 @@

 def _memory_mapped_record_batch_reader_from_file(filename: str) -> pa.RecordBatchStreamReader:
-    memory_mapped_stream = pa.memory_map(filename)
+    memory_mapped_stream = pa.memory_map(filename, "r+")
     return pa.ipc.open_stream(memory_mapped_stream)
```
But now loading the dataset goes very, very slowly, which is unexpected.

Hi!

Instead of generating one (potentially large) Arrow file, we shard the generated data into 500 MB shards because memory-mapping large Arrow files can be problematic on some systems. Maybe deleting the dataset's cache and increasing the shard size (controlled with the `datasets.config.MAX_SHARD_SIZE` variable; e.g. to "4GB") can fix the issue for you.

> I don't really comprehend what it is that `datasets` gave me when it downloaded the laion2B-en dataset, because nothing can seemingly read these 1024 .arrow files it is retrieving. Not `polars`, not `pyarrow`, it's not an `ipc` file, it's not a `parquet` file...

Our `.arrow` files are in the [Arrow streaming format](https://arrow.apache.org/docs/python/ipc.html#using-streams). To load them as a `polars` DataFrame, do the following:
```python
df = pl.from_arrow(Dataset.from_from(path_to_arrow_file).data.table)
```

We plan to switch to the IPC version eventually.

Hmm, I have a feeling this works fine on Linux, and is a real bug for however `datasets` is doing the sharding on Windows. I will follow up, but I think this is a real bug.

### Describe the bug

I have downloaded laion2B-en, and I'm receiving the following error trying to load it:
```
Resolving data files: 100%|██████████| 128/128 [00:00<00:00, 1173.79it/s]
Traceback (most recent call last):
  File "D:\Art-Workspace\src\artworkspace\tokeneval\compute_frequencies.py", line 31, in <module>
    count = compute_frequencies()
            ^^^^^^^^^^^^^^^^^^^^^
  File "D:\Art-Workspace\src\artworkspace\tokeneval\compute_frequencies.py", line 17, in compute_frequencies
    laion2b_dataset = load_dataset("laion/laion2B-en", split="train", cache_dir=_CACHE_DIR, keep_in_memory=False)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bberman\Documents\Art-Workspace\venv\Lib\site-packages\datasets\load.py", line 2165, in load_dataset
    ds = builder_instance.as_dataset(split=split, verification_mode=verification_mode, in_memory=keep_in_memory)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bberman\Documents\Art-Workspace\venv\Lib\site-packages\datasets\builder.py", line 1187, in as_dataset
    datasets = map_nested(
               ^^^^^^^^^^^
  File "C:\Users\bberman\Documents\Art-Workspace\venv\Lib\site-packages\datasets\utils\py_utils.py", line 456, in map_nested
    return function(data_struct)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bberman\Documents\Art-Workspace\venv\Lib\site-packages\datasets\builder.py", line 1217, in _build_single_dataset
    ds = self._as_dataset(
         ^^^^^^^^^^^^^^^^^
  File "C:\Users\bberman\Documents\Art-Workspace\venv\Lib\site-packages\datasets\builder.py", line 1291, in _as_dataset
    dataset_kwargs = ArrowReader(cache_dir, self.info).read(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bberman\Documents\Art-Workspace\venv\Lib\site-packages\datasets\arrow_reader.py", line 244, in read
    return self.read_files(files=files, original_instructions=instructions, in_memory=in_memory)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bberman\Documents\Art-Workspace\venv\Lib\site-packages\datasets\arrow_reader.py", line 265, in read_files
    pa_table = self._read_files(files, in_memory=in_memory)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bberman\Documents\Art-Workspace\venv\Lib\site-packages\datasets\arrow_reader.py", line 200, in _read_files
    pa_table: Table = self._get_table_from_filename(f_dict, in_memory=in_memory)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bberman\Documents\Art-Workspace\venv\Lib\site-packages\datasets\arrow_reader.py", line 336, in _get_table_from_filename
    table = ArrowReader.read_table(filename, in_memory=in_memory)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bberman\Documents\Art-Workspace\venv\Lib\site-packages\datasets\arrow_reader.py", line 357, in read_table
    return table_cls.from_file(filename)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bberman\Documents\Art-Workspace\venv\Lib\site-packages\datasets\table.py", line 1059, in from_file
    table = _memory_mapped_arrow_table_from_file(filename)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bberman\Documents\Art-Workspace\venv\Lib\site-packages\datasets\table.py", line 66, in _memory_mapped_arrow_table_from_file
    pa_table = opened_stream.read_all()
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow\ipc.pxi", line 757, in pyarrow.lib.RecordBatchReader.read_all
  File "pyarrow\error.pxi", line 91, in pyarrow.lib.check_status
OSError: [WinError 8] PrefetchVirtualMemory failed. Detail: [Windows error 8] Not enough memory resources are available to process this command.
```

This error is probably a red herring: https://stackoverflow.com/questions/50263929/numpy-memmap-returns-not-enough-memory-while-there-are-plenty-available In other words, the issue is related to asking for a memory mapping of length N > M the length of the file on Windows. This gracefully succeeds on Linux.

I have 1024 arrow files in my cache instead of 128 like in the repository for it. Probably related. I don't know why `datasets` reorganized/rewrote the dataset in my cache to be 1024 slices instead of the original 128.

### Steps to reproduce the bug

```
# as a huggingface developer, you may already have laion2B-en somewhere
_CACHE_DIR = "."

from datasets import load_dataset
load_dataset("laion/laion2B-en", split="train", cache_dir=_CACHE_DIR, keep_in_memory=False)
```

### Expected behavior

This should correctly load as a memory mapped Arrow dataset.

### Environment info

- `datasets` version: 2.15.0
- Platform: Windows-10-10.0.20348-SP0 (this is windows 2022)
- Python version: 3.11.4
- `huggingface_hub` version: 0.19.4
- PyArrow version: 14.0.1
- Pandas version: 2.1.2
- `fsspec` version: 2023.10.0


Deprecate Beam API and download from HF GCS bucket

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6474). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005280 / 0.011353 (-0.006073) | 0.003770 / 0.011008 (-0.007238) | 0.064320 / 0.038508 (0.025812) | 0.031250 / 0.023109 (0.008141) | 0.245113 / 0.275898 (-0.030785) | 0.268646 / 0.323480 (-0.054834) | 0.003147 / 0.007986 (-0.004839) | 0.002736 / 0.004328 (-0.001592) | 0.050802 / 0.004250 (0.046551) | 0.044539 / 0.037052 (0.007487) | 0.261702 / 0.258489 (0.003213) | 0.304696 / 0.293841 (0.010855) | 0.028601 / 0.128546 (-0.099945) | 0.010731 / 0.075646 (-0.064915) | 0.208183 / 0.419271 (-0.211089) | 0.036597 / 0.043533 (-0.006936) | 0.245107 / 0.255139 (-0.010032) | 0.265936 / 0.283200 (-0.017264) | 0.019430 / 0.141683 (-0.122253) | 1.184697 / 1.452155 (-0.267458) | 1.206073 / 1.492716 (-0.286643) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.110242 / 0.018006 (0.092236) | 0.304185 / 0.000490 (0.303695) | 0.000220 / 0.000200 (0.000020) | 0.000047 / 0.000054 (-0.000008) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018839 / 0.037411 (-0.018573) | 0.062840 / 0.014526 (0.048314) | 0.075849 / 0.176557 (-0.100708) | 0.122983 / 0.737135 (-0.614153) | 0.075352 / 0.296338 (-0.220987) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.283617 / 0.215209 (0.068408) | 2.801903 / 2.077655 (0.724248) | 1.447678 / 1.504120 (-0.056442) | 1.327561 / 1.541195 (-0.213633) | 1.348714 / 1.468490 (-0.119776) | 0.574283 / 4.584777 (-4.010493) | 2.449396 / 3.745712 (-1.296316) | 2.908659 / 5.269862 (-2.361203) | 1.813935 / 4.565676 (-2.751742) | 0.063141 / 0.424275 (-0.361134) | 0.005024 / 0.007607 (-0.002583) | 0.338854 / 0.226044 (0.112810) | 3.346680 / 2.268929 (1.077752) | 1.833792 / 55.444624 (-53.610832) | 1.553680 / 6.876477 (-5.322796) | 1.590403 / 2.142072 (-0.551670) | 0.657442 / 4.805227 (-4.147785) | 0.119961 / 6.500664 (-6.380703) | 0.042602 / 0.075469 (-0.032867) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.977510 / 1.841788 (-0.864278) | 12.105448 / 8.074308 (4.031140) | 9.635282 / 10.191392 (-0.556110) | 0.132453 / 0.680424 (-0.547971) | 0.014225 / 0.534201 (-0.519976) | 0.292847 / 0.579283 (-0.286436) | 0.271498 / 0.434364 (-0.162866) | 0.329785 / 0.540337 (-0.210552) | 0.435058 / 1.386936 (-0.951879) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005656 / 0.011353 (-0.005697) | 0.003903 / 0.011008 (-0.007106) | 0.050670 / 0.038508 (0.012162) | 0.032405 / 0.023109 (0.009296) | 0.271443 / 0.275898 (-0.004455) | 0.298989 / 0.323480 (-0.024491) | 0.004397 / 0.007986 (-0.003589) | 0.003027 / 0.004328 (-0.001301) | 0.050227 / 0.004250 (0.045977) | 0.047798 / 0.037052 (0.010745) | 0.287610 / 0.258489 (0.029120) | 0.314084 / 0.293841 (0.020243) | 0.030406 / 0.128546 (-0.098140) | 0.011159 / 0.075646 (-0.064487) | 0.059510 / 0.419271 (-0.359762) | 0.056842 / 0.043533 (0.013309) | 0.274653 / 0.255139 (0.019514) | 0.290175 / 0.283200 (0.006976) | 0.019822 / 0.141683 (-0.121861) | 1.168894 / 1.452155 (-0.283261) | 1.205867 / 1.492716 (-0.286850) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094907 / 0.018006 (0.076901) | 0.304920 / 0.000490 (0.304431) | 0.000218 / 0.000200 (0.000018) | 0.000048 / 0.000054 (-0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023015 / 0.037411 (-0.014396) | 0.076868 / 0.014526 (0.062342) | 0.089099 / 0.176557 (-0.087458) | 0.128786 / 0.737135 (-0.608350) | 0.090836 / 0.296338 (-0.205503) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.284552 / 0.215209 (0.069343) | 2.800870 / 2.077655 (0.723215) | 1.577167 / 1.504120 (0.073047) | 1.448788 / 1.541195 (-0.092406) | 1.475493 / 1.468490 (0.007003) | 0.566639 / 4.584777 (-4.018138) | 2.503671 / 3.745712 (-1.242041) | 2.796769 / 5.269862 (-2.473092) | 1.829344 / 4.565676 (-2.736332) | 0.064200 / 0.424275 (-0.360075) | 0.005066 / 0.007607 (-0.002541) | 0.335390 / 0.226044 (0.109346) | 3.421265 / 2.268929 (1.152336) | 1.942245 / 55.444624 (-53.502379) | 1.641401 / 6.876477 (-5.235076) | 1.824574 / 2.142072 (-0.317499) | 0.661379 / 4.805227 (-4.143849) | 0.117207 / 6.500664 (-6.383457) | 0.041630 / 0.075469 (-0.033839) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.009636 / 1.841788 (-0.832152) | 12.998379 / 8.074308 (4.924071) | 10.609013 / 10.191392 (0.417621) | 0.142794 / 0.680424 (-0.537630) | 0.015648 / 0.534201 (-0.518553) | 0.292825 / 0.579283 (-0.286458) | 0.280790 / 0.434364 (-0.153574) | 0.329016 / 0.540337 (-0.211322) | 0.442440 / 1.386936 (-0.944496) |

![](https://cml.dev/watermark.png#f90b65d6ccaceaa532b851a1010ba3dcc77b8b42 "CML watermark")

Deprecate the Beam API and download from the HF GCS bucked.

TODO:
- [x] Convert the Beam-based [`wikipedia`](https://huggingface.co/datasets/wikipedia) to an Arrow-based dataset ([Hub PR](https://huggingface.co/datasets/wikipedia/discussions/19))
- [x] Make [`natural_questions`](https://huggingface.co/datasets/natural_questions) a no-code dataset ([Hub PR](https://huggingface.co/datasets/natural_questions/discussions/7))
- [x] Make [`wiki40b`](https://huggingface.co/datasets/wiki40b) a no-code dataset ([Hub PR](https://huggingface.co/datasets/wiki40b/discussions/5))
- [x] Make [`wiki_dpr`](https://huggingface.co/datasets/wiki_dpr) an Arrow-based dataset ([Hub PR](https://huggingface.co/datasets/wiki_dpr/discussions/14))


Fix CI quality

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6473). All of your documentation changes will be reflected on that endpoint.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005270 / 0.011353 (-0.006083) | 0.003471 / 0.011008 (-0.007537) | 0.061942 / 0.038508 (0.023434) | 0.052671 / 0.023109 (0.029562) | 0.250541 / 0.275898 (-0.025357) | 0.270677 / 0.323480 (-0.052803) | 0.002933 / 0.007986 (-0.005053) | 0.003264 / 0.004328 (-0.001064) | 0.048055 / 0.004250 (0.043804) | 0.037459 / 0.037052 (0.000407) | 0.254926 / 0.258489 (-0.003563) | 0.292547 / 0.293841 (-0.001294) | 0.027959 / 0.128546 (-0.100587) | 0.010762 / 0.075646 (-0.064884) | 0.204961 / 0.419271 (-0.214310) | 0.035488 / 0.043533 (-0.008045) | 0.254102 / 0.255139 (-0.001037) | 0.273654 / 0.283200 (-0.009546) | 0.018126 / 0.141683 (-0.123556) | 1.082330 / 1.452155 (-0.369825) | 1.147179 / 1.492716 (-0.345538) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093223 / 0.018006 (0.075217) | 0.301912 / 0.000490 (0.301422) | 0.000219 / 0.000200 (0.000019) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018407 / 0.037411 (-0.019004) | 0.060412 / 0.014526 (0.045886) | 0.074063 / 0.176557 (-0.102494) | 0.118743 / 0.737135 (-0.618392) | 0.076484 / 0.296338 (-0.219854) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.289929 / 0.215209 (0.074720) | 2.825096 / 2.077655 (0.747442) | 1.511444 / 1.504120 (0.007324) | 1.394812 / 1.541195 (-0.146383) | 1.419751 / 1.468490 (-0.048739) | 0.569995 / 4.584777 (-4.014782) | 2.402586 / 3.745712 (-1.343126) | 2.826223 / 5.269862 (-2.443639) | 1.751554 / 4.565676 (-2.814123) | 0.064266 / 0.424275 (-0.360009) | 0.005047 / 0.007607 (-0.002561) | 0.341513 / 0.226044 (0.115469) | 3.372106 / 2.268929 (1.103177) | 1.872693 / 55.444624 (-53.571931) | 1.588200 / 6.876477 (-5.288276) | 1.630800 / 2.142072 (-0.511272) | 0.654266 / 4.805227 (-4.150961) | 0.124292 / 6.500664 (-6.376372) | 0.042876 / 0.075469 (-0.032593) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.948406 / 1.841788 (-0.893382) | 11.652947 / 8.074308 (3.578639) | 10.218195 / 10.191392 (0.026803) | 0.128447 / 0.680424 (-0.551976) | 0.014092 / 0.534201 (-0.520109) | 0.287631 / 0.579283 (-0.291652) | 0.264843 / 0.434364 (-0.169521) | 0.329997 / 0.540337 (-0.210340) | 0.439597 / 1.386936 (-0.947339) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005418 / 0.011353 (-0.005935) | 0.003589 / 0.011008 (-0.007419) | 0.050074 / 0.038508 (0.011566) | 0.052566 / 0.023109 (0.029456) | 0.293447 / 0.275898 (0.017549) | 0.320518 / 0.323480 (-0.002962) | 0.004094 / 0.007986 (-0.003892) | 0.002690 / 0.004328 (-0.001639) | 0.048200 / 0.004250 (0.043949) | 0.040692 / 0.037052 (0.003640) | 0.297086 / 0.258489 (0.038597) | 0.323827 / 0.293841 (0.029986) | 0.029511 / 0.128546 (-0.099035) | 0.011079 / 0.075646 (-0.064568) | 0.058562 / 0.419271 (-0.360709) | 0.032897 / 0.043533 (-0.010636) | 0.297244 / 0.255139 (0.042105) | 0.316812 / 0.283200 (0.033612) | 0.018468 / 0.141683 (-0.123215) | 1.140948 / 1.452155 (-0.311207) | 1.195453 / 1.492716 (-0.297263) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092677 / 0.018006 (0.074671) | 0.300775 / 0.000490 (0.300285) | 0.000225 / 0.000200 (0.000025) | 0.000054 / 0.000054 (0.000000) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021617 / 0.037411 (-0.015794) | 0.077135 / 0.014526 (0.062610) | 0.079848 / 0.176557 (-0.096709) | 0.118475 / 0.737135 (-0.618661) | 0.081174 / 0.296338 (-0.215164) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.294424 / 0.215209 (0.079215) | 2.863989 / 2.077655 (0.786334) | 1.590604 / 1.504120 (0.086484) | 1.474345 / 1.541195 (-0.066849) | 1.482120 / 1.468490 (0.013630) | 0.567829 / 4.584777 (-4.016948) | 2.493782 / 3.745712 (-1.251930) | 2.823460 / 5.269862 (-2.446402) | 1.732677 / 4.565676 (-2.833000) | 0.065518 / 0.424275 (-0.358757) | 0.004923 / 0.007607 (-0.002684) | 0.349313 / 0.226044 (0.123268) | 3.428618 / 2.268929 (1.159689) | 1.970641 / 55.444624 (-53.473983) | 1.655884 / 6.876477 (-5.220593) | 1.657151 / 2.142072 (-0.484921) | 0.661208 / 4.805227 (-4.144019) | 0.119129 / 6.500664 (-6.381535) | 0.040770 / 0.075469 (-0.034699) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.964865 / 1.841788 (-0.876923) | 12.050218 / 8.074308 (3.975910) | 10.458749 / 10.191392 (0.267357) | 0.141856 / 0.680424 (-0.538568) | 0.015091 / 0.534201 (-0.519109) | 0.288897 / 0.579283 (-0.290387) | 0.275343 / 0.434364 (-0.159021) | 0.328363 / 0.540337 (-0.211975) | 0.579243 / 1.386936 (-0.807693) |

![](https://cml.dev/watermark.png#f7721021e284859ea0952444bae6300a0d00794f "CML watermark")

Fix #6472.


CI quality is broken

Something isn't working

maintenance

Maintenance tasks

See: https://github.com/huggingface/datasets/actions/runs/7100835633/job/19327734359
```
Would reformat: src/datasets/features/image.py
1 file would be reformatted, 253 files left unchanged
```


Remove delete doc CI

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6471). All of your documentation changes will be reflected on that endpoint.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005573 / 0.011353 (-0.005780) | 0.003449 / 0.011008 (-0.007559) | 0.063323 / 0.038508 (0.024815) | 0.049369 / 0.023109 (0.026260) | 0.254280 / 0.275898 (-0.021618) | 0.267721 / 0.323480 (-0.055759) | 0.002894 / 0.007986 (-0.005092) | 0.002646 / 0.004328 (-0.001683) | 0.049284 / 0.004250 (0.045033) | 0.037947 / 0.037052 (0.000895) | 0.251654 / 0.258489 (-0.006836) | 0.279729 / 0.293841 (-0.014112) | 0.028022 / 0.128546 (-0.100525) | 0.010653 / 0.075646 (-0.064993) | 0.208567 / 0.419271 (-0.210704) | 0.035863 / 0.043533 (-0.007670) | 0.248522 / 0.255139 (-0.006617) | 0.270274 / 0.283200 (-0.012925) | 0.019683 / 0.141683 (-0.122000) | 1.136342 / 1.452155 (-0.315812) | 1.206757 / 1.492716 (-0.285960) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094682 / 0.018006 (0.076676) | 0.304092 / 0.000490 (0.303602) | 0.000220 / 0.000200 (0.000020) | 0.000051 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018606 / 0.037411 (-0.018805) | 0.060568 / 0.014526 (0.046042) | 0.074067 / 0.176557 (-0.102490) | 0.118979 / 0.737135 (-0.618156) | 0.075676 / 0.296338 (-0.220663) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.290452 / 0.215209 (0.075243) | 2.848868 / 2.077655 (0.771213) | 1.534932 / 1.504120 (0.030812) | 1.386717 / 1.541195 (-0.154478) | 1.416645 / 1.468490 (-0.051845) | 0.569020 / 4.584777 (-4.015757) | 2.421168 / 3.745712 (-1.324545) | 2.781358 / 5.269862 (-2.488503) | 1.758495 / 4.565676 (-2.807182) | 0.063851 / 0.424275 (-0.360424) | 0.004968 / 0.007607 (-0.002639) | 0.339198 / 0.226044 (0.113154) | 3.356392 / 2.268929 (1.087464) | 1.858145 / 55.444624 (-53.586479) | 1.589000 / 6.876477 (-5.287477) | 1.569175 / 2.142072 (-0.572897) | 0.650571 / 4.805227 (-4.154657) | 0.120288 / 6.500664 (-6.380376) | 0.042489 / 0.075469 (-0.032980) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.939963 / 1.841788 (-0.901824) | 11.493612 / 8.074308 (3.419304) | 10.353780 / 10.191392 (0.162388) | 0.141945 / 0.680424 (-0.538479) | 0.014397 / 0.534201 (-0.519804) | 0.286971 / 0.579283 (-0.292312) | 0.266787 / 0.434364 (-0.167577) | 0.330385 / 0.540337 (-0.209952) | 0.438542 / 1.386936 (-0.948394) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005360 / 0.011353 (-0.005993) | 0.003720 / 0.011008 (-0.007288) | 0.048790 / 0.038508 (0.010282) | 0.050256 / 0.023109 (0.027147) | 0.275445 / 0.275898 (-0.000453) | 0.297725 / 0.323480 (-0.025755) | 0.004077 / 0.007986 (-0.003909) | 0.002759 / 0.004328 (-0.001569) | 0.047653 / 0.004250 (0.043403) | 0.040205 / 0.037052 (0.003153) | 0.281028 / 0.258489 (0.022539) | 0.304682 / 0.293841 (0.010841) | 0.030158 / 0.128546 (-0.098388) | 0.010957 / 0.075646 (-0.064689) | 0.058193 / 0.419271 (-0.361079) | 0.033277 / 0.043533 (-0.010256) | 0.279501 / 0.255139 (0.024362) | 0.295381 / 0.283200 (0.012181) | 0.017889 / 0.141683 (-0.123794) | 1.121354 / 1.452155 (-0.330801) | 1.225702 / 1.492716 (-0.267014) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093385 / 0.018006 (0.075378) | 0.304642 / 0.000490 (0.304152) | 0.000219 / 0.000200 (0.000019) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021456 / 0.037411 (-0.015955) | 0.068536 / 0.014526 (0.054010) | 0.080867 / 0.176557 (-0.095689) | 0.119093 / 0.737135 (-0.618042) | 0.081875 / 0.296338 (-0.214464) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.304434 / 0.215209 (0.089225) | 2.990303 / 2.077655 (0.912649) | 1.616959 / 1.504120 (0.112839) | 1.493256 / 1.541195 (-0.047939) | 1.542857 / 1.468490 (0.074367) | 0.575517 / 4.584777 (-4.009260) | 2.455165 / 3.745712 (-1.290547) | 2.810089 / 5.269862 (-2.459773) | 1.756502 / 4.565676 (-2.809175) | 0.064801 / 0.424275 (-0.359475) | 0.004969 / 0.007607 (-0.002638) | 0.360227 / 0.226044 (0.134183) | 3.575029 / 2.268929 (1.306100) | 1.989955 / 55.444624 (-53.454669) | 1.705306 / 6.876477 (-5.171171) | 1.688523 / 2.142072 (-0.453550) | 0.663266 / 4.805227 (-4.141962) | 0.121852 / 6.500664 (-6.378812) | 0.041853 / 0.075469 (-0.033616) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.983535 / 1.841788 (-0.858252) | 11.827656 / 8.074308 (3.753348) | 10.663265 / 10.191392 (0.471873) | 0.145942 / 0.680424 (-0.534482) | 0.016004 / 0.534201 (-0.518197) | 0.288907 / 0.579283 (-0.290376) | 0.279100 / 0.434364 (-0.155264) | 0.328061 / 0.540337 (-0.212276) | 0.570253 / 1.386936 (-0.816683) |

![](https://cml.dev/watermark.png#b52cbc18919869460557e15028e7f489eae8afc7 "CML watermark")


If an image in a dataset is corrupted, we get unescapable error

### Describe the bug

Example discussed in detail here: https://huggingface.co/datasets/sasha/birdsnap/discussions/1

### Steps to reproduce the bug

```
from datasets import load_dataset, VerificationMode
dataset = load_dataset(
    'sasha/birdsnap',
    split="train",
    verification_mode=VerificationMode.ALL_CHECKS,
    streaming=True # I recommend using streaming=True when reproducing, as this dataset is large
)
for idx, row in enumerate(dataset):
    # Iterating to 9287 took 7 minutes for me
    # If you already have the data locally cached and set streaming=False, you see the same error just by with dataset[9287]
    pass
    # error at 9287 OSError: image file is truncated (45 bytes not processed)
    # note that we can't avoid the error using a try/except + continue inside the loop
```

### Expected behavior

Able to escape errors in casting to Image() without killing the whole loop

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-5.15.0-84-generic-x86_64-with-glibc2.31
- Python version: 3.11.5
- `huggingface_hub` version: 0.19.4
- PyArrow version: 14.0.1
- Pandas version: 2.1.3
- `fsspec` version: 2023.10.0