get_dataset_default_config_name docstring

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6723). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005658 / 0.011353 (-0.005694) | 0.003883 / 0.011008 (-0.007125) | 0.064007 / 0.038508 (0.025499) | 0.030370 / 0.023109 (0.007261) | 0.246677 / 0.275898 (-0.029221) | 0.270846 / 0.323480 (-0.052634) | 0.003102 / 0.007986 (-0.004884) | 0.002931 / 0.004328 (-0.001397) | 0.049446 / 0.004250 (0.045196) | 0.043555 / 0.037052 (0.006503) | 0.261810 / 0.258489 (0.003321) | 0.289705 / 0.293841 (-0.004136) | 0.028676 / 0.128546 (-0.099870) | 0.010778 / 0.075646 (-0.064868) | 0.210604 / 0.419271 (-0.208667) | 0.035987 / 0.043533 (-0.007546) | 0.248034 / 0.255139 (-0.007105) | 0.265019 / 0.283200 (-0.018181) | 0.018522 / 0.141683 (-0.123161) | 1.096364 / 1.452155 (-0.355791) | 1.152750 / 1.492716 (-0.339966) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093987 / 0.018006 (0.075981) | 0.306143 / 0.000490 (0.305653) | 0.000218 / 0.000200 (0.000018) | 0.000045 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018727 / 0.037411 (-0.018685) | 0.061983 / 0.014526 (0.047457) | 0.074254 / 0.176557 (-0.102303) | 0.121256 / 0.737135 (-0.615880) | 0.076756 / 0.296338 (-0.219582) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.278824 / 0.215209 (0.063615) | 2.815960 / 2.077655 (0.738305) | 1.472946 / 1.504120 (-0.031174) | 1.349722 / 1.541195 (-0.191473) | 1.327844 / 1.468490 (-0.140646) | 0.574964 / 4.584777 (-4.009813) | 2.403458 / 3.745712 (-1.342254) | 2.769293 / 5.269862 (-2.500569) | 1.736970 / 4.565676 (-2.828706) | 0.063144 / 0.424275 (-0.361131) | 0.004983 / 0.007607 (-0.002625) | 0.331212 / 0.226044 (0.105168) | 3.231496 / 2.268929 (0.962567) | 1.798487 / 55.444624 (-53.646138) | 1.523010 / 6.876477 (-5.353467) | 1.559973 / 2.142072 (-0.582099) | 0.657036 / 4.805227 (-4.148191) | 0.119084 / 6.500664 (-6.381580) | 0.042982 / 0.075469 (-0.032487) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.976433 / 1.841788 (-0.865355) | 11.475946 / 8.074308 (3.401638) | 9.339369 / 10.191392 (-0.852023) | 0.141761 / 0.680424 (-0.538662) | 0.014506 / 0.534201 (-0.519695) | 0.289944 / 0.579283 (-0.289340) | 0.273667 / 0.434364 (-0.160697) | 0.326682 / 0.540337 (-0.213655) | 0.458946 / 1.386936 (-0.927990) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005194 / 0.011353 (-0.006159) | 0.003713 / 0.011008 (-0.007295) | 0.049297 / 0.038508 (0.010789) | 0.029723 / 0.023109 (0.006614) | 0.278664 / 0.275898 (0.002766) | 0.296387 / 0.323480 (-0.027093) | 0.004215 / 0.007986 (-0.003771) | 0.002680 / 0.004328 (-0.001648) | 0.048276 / 0.004250 (0.044025) | 0.044454 / 0.037052 (0.007402) | 0.290510 / 0.258489 (0.032021) | 0.319028 / 0.293841 (0.025187) | 0.029177 / 0.128546 (-0.099369) | 0.010361 / 0.075646 (-0.065285) | 0.056993 / 0.419271 (-0.362279) | 0.050765 / 0.043533 (0.007232) | 0.278234 / 0.255139 (0.023095) | 0.295848 / 0.283200 (0.012649) | 0.018776 / 0.141683 (-0.122906) | 1.134866 / 1.452155 (-0.317288) | 1.204083 / 1.492716 (-0.288634) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094397 / 0.018006 (0.076391) | 0.304693 / 0.000490 (0.304203) | 0.000207 / 0.000200 (0.000007) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021322 / 0.037411 (-0.016090) | 0.075384 / 0.014526 (0.060859) | 0.086961 / 0.176557 (-0.089596) | 0.124424 / 0.737135 (-0.612711) | 0.087802 / 0.296338 (-0.208536) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.305542 / 0.215209 (0.090333) | 2.980678 / 2.077655 (0.903023) | 1.632348 / 1.504120 (0.128228) | 1.501466 / 1.541195 (-0.039728) | 1.517681 / 1.468490 (0.049191) | 0.579318 / 4.584777 (-4.005459) | 2.460734 / 3.745712 (-1.284978) | 2.650164 / 5.269862 (-2.619697) | 1.752061 / 4.565676 (-2.813615) | 0.064561 / 0.424275 (-0.359714) | 0.005097 / 0.007607 (-0.002510) | 0.359613 / 0.226044 (0.133569) | 3.518549 / 2.268929 (1.249620) | 1.962575 / 55.444624 (-53.482050) | 1.686108 / 6.876477 (-5.190369) | 1.787873 / 2.142072 (-0.354199) | 0.653715 / 4.805227 (-4.151512) | 0.117617 / 6.500664 (-6.383048) | 0.040359 / 0.075469 (-0.035110) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.021533 / 1.841788 (-0.820255) | 11.974817 / 8.074308 (3.900509) | 10.073530 / 10.191392 (-0.117862) | 0.141477 / 0.680424 (-0.538947) | 0.015081 / 0.534201 (-0.519120) | 0.292622 / 0.579283 (-0.286661) | 0.291043 / 0.434364 (-0.143321) | 0.347822 / 0.540337 (-0.192516) | 0.443647 / 1.386936 (-0.943289) |

![](https://cml.dev/watermark.png#6fb6c834f008996c994b0a86c3808d0a33d44525 "CML watermark")

fix https://github.com/huggingface/datasets/pull/6722


Add details in docstring

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6722). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

see https://github.com/huggingface/datasets-server/pull/2554#discussion_r1516516867


Hi,do you know how to load the dataset from local file now?

@Gera001
# Loading Dataset from Local Files Using 🤗Hugging Face.

To load a dataset from local files using the Hugging Face datasets library, you can use the `load_dataset` function.

```
from datasets import load_dataset
dataset = load_dataset('csv', data_files={'train': 'path/to/train.csv',
                                         'test': 'path/to/test.csv'})
```

Reference to [HF Datasets docs for loading from local](https://huggingface.co/docs/datasets/en/loading#csv).

@albertvillanova
this issue can be closed here.

@ge00009
> like this: from datasets import load_from_disk dataset = load_from_disk(data_path)

Loads a dataset that was previously saved using `save_to_disk()`.

Reference link:
https://huggingface.co/docs/datasets/en/package_reference/loading_methods#datasets.load_from_disk.example

              Hi, if I want to load the dataset from local file, then how to specify the configuration name?

_Originally posted by @WHU-gentle in https://github.com/huggingface/datasets/issues/2976#issuecomment-1333455222_


TypeError: 'str' object is not callable

Hi ! I opened a PR to fix an issue in the Features defined in your code

Basically changing
```python
Sequence("float32")
```

to
```python
Sequence(Value("float32"))
```

https://huggingface.co/datasets/BramVanroy/hplt_mono_v1_2/discussions/1

D'oh! Was wondering why the `str() is not callable` was in there. Glad the error is my end though, and not related to zstandard (which I had not used in the past).

Thanks a lot!

### Describe the bug

I am trying to get the HPLT datasets on the hub. Downloading/re-uploading would be too time- and resource consuming so I wrote [a dataset loader script](https://huggingface.co/datasets/BramVanroy/hplt_mono_v1_2/blob/main/hplt_mono_v1_2.py). I think I am very close but for some reason I always get the error below. It happens during the clean-up phase where the directory cannot be removed because it is not empty.

My only guess would be that this may have to do with zstandard

```
Traceback (most recent call last):
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/builder.py", line 1744, in _prepare_split_single
    writer.write(example, key)
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/arrow_writer.py", line 492, in write
    self.write_examples_on_file()
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/arrow_writer.py", line 434, in write_examples_on_file
    if self.schema
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/arrow_writer.py", line 409, in schema
    else (pa.schema(self._features.type) if self._features is not None else None)
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/features/features.py", line 1643, in type
    return get_nested_type(self)
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/features/features.py", line 1209, in get_nested_type
    {key: get_nested_type(schema[key]) for key in schema}
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/features/features.py", line 1209, in <dictcomp>
    {key: get_nested_type(schema[key]) for key in schema}
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/features/features.py", line 1221, in get_nested_type
    value_type = get_nested_type(schema.feature)
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/features/features.py", line 1228, in get_nested_type
    return schema()
TypeError: 'str' object is not callable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/builder.py", line 1753, in _prepare_split_single
    num_examples, num_bytes = writer.finalize()
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/arrow_writer.py", line 588, in finalize
    self.write_examples_on_file()
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/arrow_writer.py", line 434, in write_examples_on_file
    if self.schema
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/arrow_writer.py", line 409, in schema
    else (pa.schema(self._features.type) if self._features is not None else None)
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/features/features.py", line 1643, in type
    return get_nested_type(self)
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/features/features.py", line 1209, in get_nested_type
    {key: get_nested_type(schema[key]) for key in schema}
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/features/features.py", line 1209, in <dictcomp>
    {key: get_nested_type(schema[key]) for key in schema}
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/features/features.py", line 1221, in get_nested_type
    value_type = get_nested_type(schema.feature)
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/features/features.py", line 1228, in get_nested_type
    return schema()
TypeError: 'str' object is not callable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/builder.py", line 959, in incomplete_dir
    yield tmp_dir
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/builder.py", line 1005, in download_and_prepare
    self._download_and_prepare(
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/builder.py", line 1767, in _download_and_prepare
    super()._download_and_prepare(
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/builder.py", line 1100, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/builder.py", line 1605, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/builder.py", line 1762, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pricie/vanroy/.config/JetBrains/PyCharm2023.3/scratches/scratch_5.py", line 4, in <module>
    ds = load_dataset(
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/load.py", line 2549, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/builder.py", line 985, in download_and_prepare
    with incomplete_dir(self._output_dir) as tmp_output_dir:
  File "/home/pricie/vanroy/.pyenv/versions/3.10.13/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/local/vanroy/dutch-instruction-datasets/.venv/lib/python3.10/site-packages/datasets/builder.py", line 966, in incomplete_dir
    shutil.rmtree(tmp_dir)
  File "/home/pricie/vanroy/.pyenv/versions/3.10.13/lib/python3.10/shutil.py", line 731, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/home/pricie/vanroy/.pyenv/versions/3.10.13/lib/python3.10/shutil.py", line 729, in rmtree
    os.rmdir(path)
OSError: [Errno 39] Directory not empty: '/home/pricie/vanroy/.cache/huggingface/datasets/BramVanroy___hplt_mono_v1_2/ky/1.2.0/7ab138629fe7e9e29fe93ce63d809d5ef9d963273b829f61ab538e012dc9cc47.incomplete'
```

Interestingly, though, this directory _does_ appear to be empty:

```shell
> cd /home/pricie/vanroy/.cache/huggingface/datasets/BramVanroy___hplt_mono_v1_2/ky/1.2.0/7ab138629fe7e9e29fe93ce63d809d5ef9d963273b829f61ab538e012dc9cc47.incomplete

> ls -lah
total 0
drwxr-xr-x. 1 vanroy vanroy   0 Mar  7 12:01 .
drwxr-xr-x. 1 vanroy vanroy 304 Mar  7 11:52 ..

> cd ..
> ls
7ab138629fe7e9e29fe93ce63d809d5ef9d963273b829f61ab538e012dc9cc47_builder.lock  7ab138629fe7e9e29fe93ce63d809d5ef9d963273b829f61ab538e012dc9cc47.incomplete
```

### Steps to reproduce the bug

```python
from datasets import load_dataset

ds = load_dataset(
    "BramVanroy/hplt_mono_v1_2",
    "ky",
    trust_remote_code=True
)

```

### Expected behavior

No error.

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-5.14.0-284.25.1.el9_2.x86_64-x86_64-with-glibc2.34
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.1
- Pandas version: 2.1.3
- `fsspec` version: 2023.10.0


Is there any way to solve hanging of IterableDataset using split by node + filtering during inference

### Describe the bug

I am using an iterable dataset in a multi-node setup, trying to do training/inference while filtering the data on the fly. I usually do not use `split_dataset_by_node` but it is very slow using the IterableDatasetShard in `accelerate` and `transformers`. When I filter after applying `split_dataset_by_node`, it results in shards that are not equal sizes due to unequal samples filtered from each one.

The distributed process hangs when trying to accomplish this. Is there any way to resolve this or is it impossible to implement?

### Steps to reproduce the bug

Here is a toy example of what I am trying to do that reproduces the behavior

```
# torchrun --nproc-per-node 2 file.py

import os

import pandas as pd
import torch
from accelerate import Accelerator
from datasets import Features, Value, load_dataset
from datasets.distributed import split_dataset_by_node
from torch.utils.data import DataLoader

accelerator = Accelerator(device_placement=True, dispatch_batches=False)
if accelerator.is_main_process:
    if not os.path.exists("scratch_data"):
        os.mkdir("scratch_data")

    n_shards = 4
    for i in range(n_shards):
        df = pd.DataFrame({"id": list(range(10 * i, 10 * (i + 1)))})
        df.to_parquet(f"scratch_data/shard_{i}.parquet")

world_size = accelerator.num_processes
local_rank = accelerator.process_index

def collate_fn(examples):
    input_ids = []
    for example in examples:
        input_ids.append(example["id"])
    return torch.LongTensor(input_ids)

dataset = load_dataset(
    "parquet", data_dir="scratch_data", split="train", streaming=True
)
dataset = (
    split_dataset_by_node(dataset, rank=local_rank, world_size=world_size)
    .filter(lambda x: x["id"] < 35)
    .shuffle(seed=42, buffer_size=100)
)

batch_size = 2
train_dataloader = DataLoader(
    dataset,
    batch_size=batch_size,
    collate_fn=collate_fn,
    num_workers=2
)

for x in train_dataloader:
    x = x.to(accelerator.device)
    print({"rank": local_rank, "id": x})

    y = accelerator.gather_for_metrics(x)
    if accelerator.is_main_process:
        print("gathered", y)

```

### Expected behavior

Is there any way to continue training/inference on the GPUs that have remaining data left without waiting for the others? Is it impossible to filter when

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.10.209-198.812.amzn2.x86_64-x86_64-with-glibc2.31
- Python version: 3.10.13
- `huggingface_hub` version: 0.21.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.1
- `fsspec` version: 2023.6.0


Fix concurrent script loading with force_redownload

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6718). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005074 / 0.011353 (-0.006279) | 0.003505 / 0.011008 (-0.007503) | 0.063683 / 0.038508 (0.025175) | 0.029308 / 0.023109 (0.006199) | 0.246648 / 0.275898 (-0.029250) | 0.265546 / 0.323480 (-0.057933) | 0.004108 / 0.007986 (-0.003878) | 0.002683 / 0.004328 (-0.001646) | 0.048634 / 0.004250 (0.044383) | 0.043786 / 0.037052 (0.006733) | 0.262197 / 0.258489 (0.003708) | 0.291582 / 0.293841 (-0.002259) | 0.027472 / 0.128546 (-0.101074) | 0.010213 / 0.075646 (-0.065434) | 0.206744 / 0.419271 (-0.212527) | 0.036195 / 0.043533 (-0.007337) | 0.249090 / 0.255139 (-0.006049) | 0.280002 / 0.283200 (-0.003198) | 0.018568 / 0.141683 (-0.123115) | 1.124844 / 1.452155 (-0.327311) | 1.159358 / 1.492716 (-0.333359) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093186 / 0.018006 (0.075180) | 0.302331 / 0.000490 (0.301842) | 0.000217 / 0.000200 (0.000017) | 0.000046 / 0.000054 (-0.000008) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018727 / 0.037411 (-0.018684) | 0.061730 / 0.014526 (0.047204) | 0.074330 / 0.176557 (-0.102226) | 0.119769 / 0.737135 (-0.617366) | 0.075611 / 0.296338 (-0.220727) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.285063 / 0.215209 (0.069854) | 2.824809 / 2.077655 (0.747155) | 1.481858 / 1.504120 (-0.022262) | 1.350193 / 1.541195 (-0.191002) | 1.358012 / 1.468490 (-0.110478) | 0.557842 / 4.584777 (-4.026935) | 2.380729 / 3.745712 (-1.364983) | 2.798891 / 5.269862 (-2.470970) | 1.719288 / 4.565676 (-2.846388) | 0.061705 / 0.424275 (-0.362570) | 0.005431 / 0.007607 (-0.002176) | 0.343233 / 0.226044 (0.117189) | 3.375223 / 2.268929 (1.106295) | 1.838188 / 55.444624 (-53.606436) | 1.570015 / 6.876477 (-5.306461) | 1.573157 / 2.142072 (-0.568915) | 0.650678 / 4.805227 (-4.154549) | 0.116412 / 6.500664 (-6.384252) | 0.041754 / 0.075469 (-0.033715) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.970431 / 1.841788 (-0.871357) | 11.317128 / 8.074308 (3.242819) | 9.691240 / 10.191392 (-0.500152) | 0.142260 / 0.680424 (-0.538164) | 0.014131 / 0.534201 (-0.520070) | 0.289910 / 0.579283 (-0.289373) | 0.265648 / 0.434364 (-0.168715) | 0.323130 / 0.540337 (-0.217208) | 0.447005 / 1.386936 (-0.939931) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005322 / 0.011353 (-0.006031) | 0.003755 / 0.011008 (-0.007253) | 0.049646 / 0.038508 (0.011138) | 0.029669 / 0.023109 (0.006560) | 0.284151 / 0.275898 (0.008253) | 0.298351 / 0.323480 (-0.025128) | 0.004183 / 0.007986 (-0.003803) | 0.002683 / 0.004328 (-0.001645) | 0.048814 / 0.004250 (0.044563) | 0.045017 / 0.037052 (0.007965) | 0.287358 / 0.258489 (0.028869) | 0.317394 / 0.293841 (0.023553) | 0.030025 / 0.128546 (-0.098521) | 0.010854 / 0.075646 (-0.064793) | 0.058694 / 0.419271 (-0.360578) | 0.052287 / 0.043533 (0.008754) | 0.279038 / 0.255139 (0.023899) | 0.295442 / 0.283200 (0.012242) | 0.019413 / 0.141683 (-0.122270) | 1.146106 / 1.452155 (-0.306048) | 1.197777 / 1.492716 (-0.294939) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092191 / 0.018006 (0.074184) | 0.302672 / 0.000490 (0.302182) | 0.000623 / 0.000200 (0.000423) | 0.000048 / 0.000054 (-0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022067 / 0.037411 (-0.015345) | 0.081760 / 0.014526 (0.067235) | 0.087548 / 0.176557 (-0.089009) | 0.126405 / 0.737135 (-0.610730) | 0.089331 / 0.296338 (-0.207008) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295821 / 0.215209 (0.080612) | 2.897930 / 2.077655 (0.820276) | 1.604500 / 1.504120 (0.100380) | 1.471502 / 1.541195 (-0.069692) | 1.497918 / 1.468490 (0.029428) | 0.576179 / 4.584777 (-4.008598) | 2.452103 / 3.745712 (-1.293609) | 2.668043 / 5.269862 (-2.601818) | 1.753544 / 4.565676 (-2.812133) | 0.064410 / 0.424275 (-0.359865) | 0.005027 / 0.007607 (-0.002580) | 0.351509 / 0.226044 (0.125465) | 3.479208 / 2.268929 (1.210280) | 1.990356 / 55.444624 (-53.454269) | 1.684920 / 6.876477 (-5.191556) | 1.794251 / 2.142072 (-0.347821) | 0.662692 / 4.805227 (-4.142535) | 0.118589 / 6.500664 (-6.382076) | 0.040813 / 0.075469 (-0.034656) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.002390 / 1.841788 (-0.839398) | 12.004617 / 8.074308 (3.930309) | 10.216005 / 10.191392 (0.024613) | 0.154354 / 0.680424 (-0.526070) | 0.015554 / 0.534201 (-0.518647) | 0.288741 / 0.579283 (-0.290542) | 0.276774 / 0.434364 (-0.157590) | 0.327055 / 0.540337 (-0.213282) | 0.435121 / 1.386936 (-0.951815) |

![](https://cml.dev/watermark.png#f45bc6caa25115a04c41b278671a5a89457eb66c "CML watermark")

I added `lock_importable_file` in `get_dataset_builder_class` and `extend_dataset_builder_for_streaming` to fix the issue, and I also added a test

cc @clefourrier


`remove_columns` method used with a streaming enable dataset mode produces a LibsndfileError on multichannel audio

And it also works well with `dataset = dataset.select_columns(["audio"])`

### Describe the bug

When loading a HF dataset in streaming mode and removing some columns, it is impossible to load a sample if the audio contains more than one channel. I have the impression that the time axis and channels are swapped or concatenated.

### Steps to reproduce the bug

Minimal error code:
```python
from datasets import load_dataset

dataset_name = "zinc75/Vibravox_dummy"
config_name = "BWE_Larynx_microphone"
# if we use "ASR_Larynx_microphone" subset which is a monochannel audio, no error is thrown.

dataset = load_dataset(
    path=dataset_name, name=config_name, split="train", streaming=True
)

dataset = dataset.remove_columns(["sensor_id"])
#  dataset = dataset.map(lambda x:x, remove_columns=["sensor_id"])
# The commented version does not produce an error, but loses the dataset features.
sample = next(iter(dataset))

```

Error:

```
Traceback (most recent call last):
  File "/home/julien/Bureau/github/vibravox/tmp.py", line 15, in <module>
    sample = next(iter(dataset))
             ^^^^^^^^^^^^^^^^^^^
  File "/home/julien/.pyenv/versions/vibravox/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1392, in __iter__
    example = _apply_feature_types_on_example(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/julien/.pyenv/versions/vibravox/lib/python3.11/site-packages/datasets/iterable_dataset.py", line 1080, in _apply_feature_types_on_example
    encoded_example = features.encode_example(example)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/julien/.pyenv/versions/vibravox/lib/python3.11/site-packages/datasets/features/features.py", line 1889, in encode_example
    return encode_nested_example(self, example)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/julien/.pyenv/versions/vibravox/lib/python3.11/site-packages/datasets/features/features.py", line 1244, in encode_nested_example
    {k: encode_nested_example(schema[k], obj.get(k), level=level + 1) for k in schema}
  File "/home/julien/.pyenv/versions/vibravox/lib/python3.11/site-packages/datasets/features/features.py", line 1244, in <dictcomp>
    {k: encode_nested_example(schema[k], obj.get(k), level=level + 1) for k in schema}
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/julien/.pyenv/versions/vibravox/lib/python3.11/site-packages/datasets/features/features.py", line 1300, in encode_nested_example
    return schema.encode_example(obj) if obj is not None else None
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/julien/.pyenv/versions/vibravox/lib/python3.11/site-packages/datasets/features/audio.py", line 98, in encode_example
    sf.write(buffer, value["array"], value["sampling_rate"], format="wav")
  File "/home/julien/.pyenv/versions/vibravox/lib/python3.11/site-packages/soundfile.py", line 343, in write
    with SoundFile(file, 'w', samplerate, channels,
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/julien/.pyenv/versions/vibravox/lib/python3.11/site-packages/soundfile.py", line 658, in __init__
    self._file = self._open(file, mode_int, closefd)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/julien/.pyenv/versions/vibravox/lib/python3.11/site-packages/soundfile.py", line 1216, in _open
    raise LibsndfileError(err, prefix="Error opening {0!r}: ".format(self.name))
soundfile.LibsndfileError: Error opening <_io.BytesIO object at 0x7fd795d24680>: Format not recognised.

Process finished with exit code 1
```

### Expected behavior

I would expect this code to run without error.

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-6.5.0-21-generic-x86_64-with-glibc2.35
- Python version: 3.11.0
- `huggingface_hub` version: 0.21.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.1
- `fsspec` version: 2023.10.0


Non-deterministic `Dataset.builder_name` value

When `rotten_tomatoes` is printed out, the following warning message is also printed out:

```
You can avoid this message in future by passing the argument `trust_remote_code=True`.
Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.
```

Hi ! This behavior happens because the dataset was originakky created using a dataset script [rotten_tomatoes.py](https://huggingface.co/datasets/rotten_tomatoes/blob/26f40d324d7b281d8b3fb1c47f30f8b9957f206b/rotten_tomatoes.py) and because we added features recently allowing to download the dataset directly from Parquet files (parquet builder) without running the dataset script (rotten_tomatoes). The flakiness must come from the availability of the Parquet files (we automatically export them in the refs/convert/parquet branch and we recently had to move some files).

Anyway the easy fix on our side is to remove the dataset script completely, let me open a PR at https://huggingface.co/datasets/rotten_tomatoes

EDIT: opened https://huggingface.co/datasets/rotten_tomatoes/discussions/6, feel free to comment there if you're ok with that change

@lhoestq Thanks for the comment, explanation, and patch!

> we automatically export them in the refs/convert/parquet branch

When this operation is in progress, the parquet files become temporarily unavailable?

> When this operation is in progress, the parquet files become temporarily unavailable?

Yes correct. I just merged the patch btw :)

@lhoestq Thanks for merging the PR! I think this issue can be closed.

### Describe the bug

I'm not sure if this is a bug, but `print(ds.builder_name)` in the following code sometimes prints out `rotten_tomatoes` instead of `parquet`:

```python
import datasets

for _ in range(100):
    ds = datasets.load_dataset("rotten_tomatoes", split="train")
    print(ds.builder_name)  # prints out "rotten_tomatoes" sometimes instead of "parquet"
```

Output:

```
...
parquet
parquet
parquet
rotten_tomatoes
parquet
parquet
parquet
...
```

Here's a reproduction using GitHub Actions:
https://github.com/mlflow/mlflow/actions/runs/8153247984/job/22284263613?pr=11329#step:12:241

One of our tests is flaky because `builder_name` is not deterministic.

### Steps to reproduce the bug

1. Run the code above.

### Expected behavior

Always prints out `parquet`?

### Environment info

```
Copy-and-paste the text below in your GitHub issue.
- `datasets` version: 2.18.0
- Platform: Linux-6.5.0-1015-azure-x86_64-with-glibc2.34
- Python version: 3.8.18
- `huggingface_hub` version: 0.21.3
- PyArrow version: 15.0.0
- Pandas version: 2.0.3
- `fsspec` version: 2024.2.0
```


Fix sliced ConcatenationTable pickling with mixed schemas vertically

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6715). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005294 / 0.011353 (-0.006059) | 0.003598 / 0.011008 (-0.007411) | 0.062798 / 0.038508 (0.024290) | 0.027479 / 0.023109 (0.004370) | 0.247146 / 0.275898 (-0.028752) | 0.272103 / 0.323480 (-0.051377) | 0.002979 / 0.007986 (-0.005007) | 0.002701 / 0.004328 (-0.001628) | 0.049384 / 0.004250 (0.045134) | 0.041562 / 0.037052 (0.004510) | 0.269924 / 0.258489 (0.011435) | 0.290749 / 0.293841 (-0.003092) | 0.028285 / 0.128546 (-0.100261) | 0.010464 / 0.075646 (-0.065183) | 0.207000 / 0.419271 (-0.212272) | 0.036186 / 0.043533 (-0.007347) | 0.254524 / 0.255139 (-0.000615) | 0.274843 / 0.283200 (-0.008356) | 0.020044 / 0.141683 (-0.121638) | 1.119223 / 1.452155 (-0.332931) | 1.156557 / 1.492716 (-0.336159) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092014 / 0.018006 (0.074008) | 0.297349 / 0.000490 (0.296859) | 0.000205 / 0.000200 (0.000005) | 0.000048 / 0.000054 (-0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018617 / 0.037411 (-0.018794) | 0.061879 / 0.014526 (0.047354) | 0.072877 / 0.176557 (-0.103680) | 0.121850 / 0.737135 (-0.615286) | 0.074686 / 0.296338 (-0.221653) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.281204 / 0.215209 (0.065995) | 2.728688 / 2.077655 (0.651033) | 1.469659 / 1.504120 (-0.034461) | 1.355306 / 1.541195 (-0.185889) | 1.350598 / 1.468490 (-0.117892) | 0.563669 / 4.584777 (-4.021108) | 2.377177 / 3.745712 (-1.368535) | 2.767402 / 5.269862 (-2.502460) | 1.720188 / 4.565676 (-2.845489) | 0.062594 / 0.424275 (-0.361681) | 0.005004 / 0.007607 (-0.002603) | 0.333017 / 0.226044 (0.106972) | 3.354543 / 2.268929 (1.085615) | 1.840031 / 55.444624 (-53.604593) | 1.545548 / 6.876477 (-5.330929) | 1.569858 / 2.142072 (-0.572214) | 0.642680 / 4.805227 (-4.162547) | 0.117463 / 6.500664 (-6.383201) | 0.042472 / 0.075469 (-0.032997) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.977436 / 1.841788 (-0.864351) | 11.285982 / 8.074308 (3.211673) | 9.441848 / 10.191392 (-0.749544) | 0.140773 / 0.680424 (-0.539650) | 0.013783 / 0.534201 (-0.520418) | 0.292304 / 0.579283 (-0.286979) | 0.275011 / 0.434364 (-0.159353) | 0.339094 / 0.540337 (-0.201244) | 0.447593 / 1.386936 (-0.939343) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005258 / 0.011353 (-0.006095) | 0.003539 / 0.011008 (-0.007469) | 0.049920 / 0.038508 (0.011412) | 0.029789 / 0.023109 (0.006680) | 0.277187 / 0.275898 (0.001288) | 0.296817 / 0.323480 (-0.026663) | 0.004133 / 0.007986 (-0.003852) | 0.002679 / 0.004328 (-0.001649) | 0.048999 / 0.004250 (0.044749) | 0.044087 / 0.037052 (0.007034) | 0.290359 / 0.258489 (0.031870) | 0.319572 / 0.293841 (0.025731) | 0.030248 / 0.128546 (-0.098298) | 0.010453 / 0.075646 (-0.065194) | 0.058734 / 0.419271 (-0.360537) | 0.051216 / 0.043533 (0.007683) | 0.278667 / 0.255139 (0.023528) | 0.298792 / 0.283200 (0.015592) | 0.019131 / 0.141683 (-0.122552) | 1.131814 / 1.452155 (-0.320340) | 1.167208 / 1.492716 (-0.325508) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.088316 / 0.018006 (0.070309) | 0.297143 / 0.000490 (0.296653) | 0.000207 / 0.000200 (0.000007) | 0.000048 / 0.000054 (-0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022457 / 0.037411 (-0.014954) | 0.075251 / 0.014526 (0.060726) | 0.086747 / 0.176557 (-0.089809) | 0.124975 / 0.737135 (-0.612161) | 0.087320 / 0.296338 (-0.209019) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.292339 / 0.215209 (0.077130) | 2.860196 / 2.077655 (0.782541) | 1.599058 / 1.504120 (0.094938) | 1.476104 / 1.541195 (-0.065091) | 1.509109 / 1.468490 (0.040619) | 0.564056 / 4.584777 (-4.020721) | 2.388870 / 3.745712 (-1.356842) | 2.582356 / 5.269862 (-2.687506) | 1.726033 / 4.565676 (-2.839644) | 0.061788 / 0.424275 (-0.362487) | 0.005021 / 0.007607 (-0.002586) | 0.345644 / 0.226044 (0.119600) | 3.384000 / 2.268929 (1.115071) | 1.946591 / 55.444624 (-53.498033) | 1.693485 / 6.876477 (-5.182992) | 1.790300 / 2.142072 (-0.351773) | 0.654637 / 4.805227 (-4.150590) | 0.116271 / 6.500664 (-6.384393) | 0.040710 / 0.075469 (-0.034759) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.007367 / 1.841788 (-0.834421) | 11.868065 / 8.074308 (3.793757) | 10.146212 / 10.191392 (-0.045180) | 0.128902 / 0.680424 (-0.551522) | 0.015259 / 0.534201 (-0.518942) | 0.288087 / 0.579283 (-0.291196) | 0.281516 / 0.434364 (-0.152848) | 0.325755 / 0.540337 (-0.214583) | 0.424814 / 1.386936 (-0.962122) |

![](https://cml.dev/watermark.png#8247202a7ed1c3164c88f8f183513c5f003aa2af "CML watermark")

A sliced + pickled ConcatenationTable could end up with a different schema than the original schema, if the slice only contains blocks with only a subset of the columns.

This can lead to issues when saving datasets from a concatenation of datasets with mixed schemas

Reported in https://discuss.huggingface.co/t/datasetdict-save-to-disk-with-num-proc-1-seems-to-hang-with-error/75595


Expand no-code dataset info with datasets-server info

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6714). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005237 / 0.011353 (-0.006116) | 0.003614 / 0.011008 (-0.007394) | 0.063349 / 0.038508 (0.024841) | 0.027297 / 0.023109 (0.004187) | 0.236203 / 0.275898 (-0.039695) | 0.260029 / 0.323480 (-0.063451) | 0.003096 / 0.007986 (-0.004889) | 0.003342 / 0.004328 (-0.000987) | 0.048703 / 0.004250 (0.044453) | 0.043121 / 0.037052 (0.006069) | 0.257491 / 0.258489 (-0.000998) | 0.282861 / 0.293841 (-0.010980) | 0.027701 / 0.128546 (-0.100845) | 0.010634 / 0.075646 (-0.065012) | 0.207369 / 0.419271 (-0.211903) | 0.035799 / 0.043533 (-0.007734) | 0.240445 / 0.255139 (-0.014694) | 0.261977 / 0.283200 (-0.021223) | 0.018175 / 0.141683 (-0.123508) | 1.143964 / 1.452155 (-0.308191) | 1.230057 / 1.492716 (-0.262659) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096656 / 0.018006 (0.078650) | 0.303434 / 0.000490 (0.302944) | 0.000225 / 0.000200 (0.000025) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018454 / 0.037411 (-0.018957) | 0.061792 / 0.014526 (0.047266) | 0.073384 / 0.176557 (-0.103172) | 0.120148 / 0.737135 (-0.616988) | 0.074221 / 0.296338 (-0.222118) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.290291 / 0.215209 (0.075082) | 2.822908 / 2.077655 (0.745254) | 1.483139 / 1.504120 (-0.020981) | 1.349619 / 1.541195 (-0.191576) | 1.356588 / 1.468490 (-0.111902) | 0.571723 / 4.584777 (-4.013054) | 2.402696 / 3.745712 (-1.343016) | 2.832215 / 5.269862 (-2.437647) | 1.794962 / 4.565676 (-2.770714) | 0.062707 / 0.424275 (-0.361568) | 0.004997 / 0.007607 (-0.002610) | 0.343093 / 0.226044 (0.117049) | 3.383028 / 2.268929 (1.114100) | 1.818624 / 55.444624 (-53.626000) | 1.549859 / 6.876477 (-5.326618) | 1.667838 / 2.142072 (-0.474235) | 0.648574 / 4.805227 (-4.156653) | 0.119181 / 6.500664 (-6.381484) | 0.042074 / 0.075469 (-0.033395) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.982039 / 1.841788 (-0.859748) | 11.411759 / 8.074308 (3.337451) | 9.783405 / 10.191392 (-0.407987) | 0.129577 / 0.680424 (-0.550847) | 0.014091 / 0.534201 (-0.520110) | 0.297925 / 0.579283 (-0.281358) | 0.263884 / 0.434364 (-0.170480) | 0.346032 / 0.540337 (-0.194305) | 0.444806 / 1.386936 (-0.942130) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005527 / 0.011353 (-0.005826) | 0.003677 / 0.011008 (-0.007332) | 0.050245 / 0.038508 (0.011737) | 0.030070 / 0.023109 (0.006961) | 0.272640 / 0.275898 (-0.003258) | 0.296555 / 0.323480 (-0.026925) | 0.004247 / 0.007986 (-0.003738) | 0.003833 / 0.004328 (-0.000495) | 0.049341 / 0.004250 (0.045091) | 0.046604 / 0.037052 (0.009552) | 0.282765 / 0.258489 (0.024276) | 0.314924 / 0.293841 (0.021084) | 0.029749 / 0.128546 (-0.098797) | 0.010524 / 0.075646 (-0.065122) | 0.057859 / 0.419271 (-0.361412) | 0.053172 / 0.043533 (0.009640) | 0.274906 / 0.255139 (0.019767) | 0.290566 / 0.283200 (0.007366) | 0.019299 / 0.141683 (-0.122384) | 1.164092 / 1.452155 (-0.288062) | 1.205074 / 1.492716 (-0.287642) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093943 / 0.018006 (0.075936) | 0.298746 / 0.000490 (0.298256) | 0.000232 / 0.000200 (0.000032) | 0.000054 / 0.000054 (-0.000000) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022098 / 0.037411 (-0.015313) | 0.075523 / 0.014526 (0.060997) | 0.086784 / 0.176557 (-0.089773) | 0.124610 / 0.737135 (-0.612525) | 0.087743 / 0.296338 (-0.208595) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.298555 / 0.215209 (0.083346) | 2.951493 / 2.077655 (0.873838) | 1.611448 / 1.504120 (0.107328) | 1.481503 / 1.541195 (-0.059692) | 1.497937 / 1.468490 (0.029447) | 0.580402 / 4.584777 (-4.004375) | 2.433308 / 3.745712 (-1.312404) | 2.712717 / 5.269862 (-2.557145) | 1.766286 / 4.565676 (-2.799391) | 0.063973 / 0.424275 (-0.360303) | 0.005006 / 0.007607 (-0.002601) | 0.354541 / 0.226044 (0.128497) | 3.486448 / 2.268929 (1.217519) | 1.972779 / 55.444624 (-53.471846) | 1.709018 / 6.876477 (-5.167458) | 1.864242 / 2.142072 (-0.277831) | 0.678213 / 4.805227 (-4.127014) | 0.119525 / 6.500664 (-6.381140) | 0.041387 / 0.075469 (-0.034082) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.021337 / 1.841788 (-0.820451) | 12.049563 / 8.074308 (3.975255) | 10.424701 / 10.191392 (0.233309) | 0.131444 / 0.680424 (-0.548980) | 0.015644 / 0.534201 (-0.518557) | 0.293712 / 0.579283 (-0.285571) | 0.279160 / 0.434364 (-0.155204) | 0.327991 / 0.540337 (-0.212346) | 0.435455 / 1.386936 (-0.951481) |

![](https://cml.dev/watermark.png#1fe9483acc1ccaf19f3c199b99391921a8526215 "CML watermark")

E.g., to have info about a dataset's number of examples for more informative TQDM bars.


Bump huggingface-hub lower version to 0.21.2

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6713). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq if you agree, I could make a patch release tomorrow morning.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005086 / 0.011353 (-0.006267) | 0.003695 / 0.011008 (-0.007313) | 0.063430 / 0.038508 (0.024922) | 0.026798 / 0.023109 (0.003689) | 0.253761 / 0.275898 (-0.022138) | 0.301301 / 0.323480 (-0.022179) | 0.004160 / 0.007986 (-0.003825) | 0.002783 / 0.004328 (-0.001545) | 0.050698 / 0.004250 (0.046448) | 0.040899 / 0.037052 (0.003846) | 0.269024 / 0.258489 (0.010535) | 0.323467 / 0.293841 (0.029626) | 0.027756 / 0.128546 (-0.100791) | 0.010684 / 0.075646 (-0.064963) | 0.207128 / 0.419271 (-0.212144) | 0.035874 / 0.043533 (-0.007659) | 0.251620 / 0.255139 (-0.003519) | 0.268668 / 0.283200 (-0.014532) | 0.017387 / 0.141683 (-0.124296) | 1.139230 / 1.452155 (-0.312925) | 1.183613 / 1.492716 (-0.309103) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096337 / 0.018006 (0.078331) | 0.305014 / 0.000490 (0.304524) | 0.000219 / 0.000200 (0.000019) | 0.000050 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018086 / 0.037411 (-0.019325) | 0.061626 / 0.014526 (0.047100) | 0.072598 / 0.176557 (-0.103959) | 0.119944 / 0.737135 (-0.617192) | 0.074549 / 0.296338 (-0.221789) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282661 / 0.215209 (0.067452) | 2.804473 / 2.077655 (0.726818) | 1.444602 / 1.504120 (-0.059517) | 1.313977 / 1.541195 (-0.227217) | 1.319426 / 1.468490 (-0.149064) | 0.570176 / 4.584777 (-4.014601) | 2.397895 / 3.745712 (-1.347818) | 2.760208 / 5.269862 (-2.509654) | 1.732457 / 4.565676 (-2.833220) | 0.062743 / 0.424275 (-0.361533) | 0.004950 / 0.007607 (-0.002657) | 0.338500 / 0.226044 (0.112456) | 3.287249 / 2.268929 (1.018320) | 1.777495 / 55.444624 (-53.667130) | 1.521255 / 6.876477 (-5.355222) | 1.517317 / 2.142072 (-0.624756) | 0.642202 / 4.805227 (-4.163025) | 0.116501 / 6.500664 (-6.384163) | 0.042418 / 0.075469 (-0.033052) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.968966 / 1.841788 (-0.872822) | 11.490531 / 8.074308 (3.416223) | 9.507803 / 10.191392 (-0.683589) | 0.141570 / 0.680424 (-0.538854) | 0.014000 / 0.534201 (-0.520201) | 0.284237 / 0.579283 (-0.295046) | 0.269341 / 0.434364 (-0.165022) | 0.321654 / 0.540337 (-0.218683) | 0.446914 / 1.386936 (-0.940022) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005280 / 0.011353 (-0.006072) | 0.003794 / 0.011008 (-0.007214) | 0.050328 / 0.038508 (0.011820) | 0.029756 / 0.023109 (0.006647) | 0.273403 / 0.275898 (-0.002495) | 0.297346 / 0.323480 (-0.026133) | 0.004310 / 0.007986 (-0.003676) | 0.002858 / 0.004328 (-0.001470) | 0.048833 / 0.004250 (0.044583) | 0.045696 / 0.037052 (0.008644) | 0.291034 / 0.258489 (0.032545) | 0.318899 / 0.293841 (0.025058) | 0.029809 / 0.128546 (-0.098737) | 0.010710 / 0.075646 (-0.064936) | 0.058183 / 0.419271 (-0.361089) | 0.051761 / 0.043533 (0.008228) | 0.275022 / 0.255139 (0.019883) | 0.291614 / 0.283200 (0.008414) | 0.017975 / 0.141683 (-0.123708) | 1.148489 / 1.452155 (-0.303666) | 1.218111 / 1.492716 (-0.274605) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091806 / 0.018006 (0.073799) | 0.299413 / 0.000490 (0.298923) | 0.000219 / 0.000200 (0.000019) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021506 / 0.037411 (-0.015905) | 0.075537 / 0.014526 (0.061011) | 0.087020 / 0.176557 (-0.089536) | 0.125270 / 0.737135 (-0.611865) | 0.088038 / 0.296338 (-0.208300) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.300401 / 0.215209 (0.085192) | 2.932571 / 2.077655 (0.854916) | 1.609502 / 1.504120 (0.105383) | 1.480078 / 1.541195 (-0.061117) | 1.514902 / 1.468490 (0.046412) | 0.575591 / 4.584777 (-4.009186) | 2.461873 / 3.745712 (-1.283839) | 2.728099 / 5.269862 (-2.541762) | 1.760054 / 4.565676 (-2.805622) | 0.064371 / 0.424275 (-0.359904) | 0.004990 / 0.007607 (-0.002617) | 0.350134 / 0.226044 (0.124090) | 3.453249 / 2.268929 (1.184321) | 1.979760 / 55.444624 (-53.464865) | 1.741128 / 6.876477 (-5.135348) | 1.825734 / 2.142072 (-0.316339) | 0.654902 / 4.805227 (-4.150325) | 0.116989 / 6.500664 (-6.383676) | 0.040800 / 0.075469 (-0.034669) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.033352 / 1.841788 (-0.808436) | 12.196711 / 8.074308 (4.122403) | 10.315114 / 10.191392 (0.123722) | 0.132541 / 0.680424 (-0.547882) | 0.016455 / 0.534201 (-0.517746) | 0.289025 / 0.579283 (-0.290258) | 0.281464 / 0.434364 (-0.152900) | 0.325302 / 0.540337 (-0.215036) | 0.428469 / 1.386936 (-0.958467) |

![](https://cml.dev/watermark.png#7093b4b1a69f413e452119c87669af9e8ceaf749 "CML watermark")

This should fix the version compatibility issue when using `huggingface_hub` < 0.21.2 and latest fsspec (>=2023.12.0).

See my comment: https://github.com/huggingface/datasets/pull/6687#issuecomment-1976493336
>> EDIT: the fix has been released in `huggingface_hub` 0.21.2 - I removed my commits that were using `huggingface_hub@main`
>
>Please note that people using `huggingface_hub` < 0.21.2 and latest `fsspec` will have issues when using `datasets`:
>- https://github.com/huggingface/lighteval/actions/runs/8139147047/job/22241658122?pr=86
>- https://github.com/huggingface/lighteval/pull/84

CC: @clefourrier


fix CastError pickling

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6712). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005151 / 0.011353 (-0.006202) | 0.003813 / 0.011008 (-0.007196) | 0.062957 / 0.038508 (0.024449) | 0.028282 / 0.023109 (0.005173) | 0.246036 / 0.275898 (-0.029862) | 0.290024 / 0.323480 (-0.033456) | 0.004009 / 0.007986 (-0.003977) | 0.002749 / 0.004328 (-0.001580) | 0.049351 / 0.004250 (0.045101) | 0.041143 / 0.037052 (0.004090) | 0.264782 / 0.258489 (0.006293) | 0.290711 / 0.293841 (-0.003130) | 0.027248 / 0.128546 (-0.101298) | 0.010691 / 0.075646 (-0.064955) | 0.205926 / 0.419271 (-0.213345) | 0.035652 / 0.043533 (-0.007880) | 0.246357 / 0.255139 (-0.008782) | 0.267851 / 0.283200 (-0.015348) | 0.018498 / 0.141683 (-0.123185) | 1.135996 / 1.452155 (-0.316159) | 1.181841 / 1.492716 (-0.310875) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094054 / 0.018006 (0.076048) | 0.305470 / 0.000490 (0.304980) | 0.000225 / 0.000200 (0.000025) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018842 / 0.037411 (-0.018569) | 0.061532 / 0.014526 (0.047006) | 0.073483 / 0.176557 (-0.103073) | 0.119426 / 0.737135 (-0.617709) | 0.075385 / 0.296338 (-0.220954) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.285544 / 0.215209 (0.070335) | 2.774256 / 2.077655 (0.696601) | 1.475719 / 1.504120 (-0.028401) | 1.353841 / 1.541195 (-0.187353) | 1.381891 / 1.468490 (-0.086599) | 0.570619 / 4.584777 (-4.014158) | 2.380300 / 3.745712 (-1.365412) | 2.788767 / 5.269862 (-2.481095) | 1.741790 / 4.565676 (-2.823886) | 0.061810 / 0.424275 (-0.362465) | 0.005004 / 0.007607 (-0.002603) | 0.334963 / 0.226044 (0.108918) | 3.286388 / 2.268929 (1.017459) | 1.831669 / 55.444624 (-53.612955) | 1.523372 / 6.876477 (-5.353105) | 1.581551 / 2.142072 (-0.560521) | 0.639642 / 4.805227 (-4.165585) | 0.117356 / 6.500664 (-6.383308) | 0.043277 / 0.075469 (-0.032192) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.973005 / 1.841788 (-0.868782) | 11.590148 / 8.074308 (3.515839) | 9.521262 / 10.191392 (-0.670130) | 0.143243 / 0.680424 (-0.537181) | 0.013529 / 0.534201 (-0.520672) | 0.285724 / 0.579283 (-0.293559) | 0.265642 / 0.434364 (-0.168721) | 0.366098 / 0.540337 (-0.174239) | 0.444410 / 1.386936 (-0.942526) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005347 / 0.011353 (-0.006006) | 0.003797 / 0.011008 (-0.007212) | 0.050441 / 0.038508 (0.011933) | 0.032812 / 0.023109 (0.009703) | 0.281278 / 0.275898 (0.005379) | 0.304524 / 0.323480 (-0.018956) | 0.005039 / 0.007986 (-0.002946) | 0.002735 / 0.004328 (-0.001594) | 0.049184 / 0.004250 (0.044933) | 0.046751 / 0.037052 (0.009698) | 0.292093 / 0.258489 (0.033604) | 0.322087 / 0.293841 (0.028246) | 0.029775 / 0.128546 (-0.098771) | 0.010540 / 0.075646 (-0.065106) | 0.057927 / 0.419271 (-0.361345) | 0.054240 / 0.043533 (0.010707) | 0.281537 / 0.255139 (0.026398) | 0.298386 / 0.283200 (0.015186) | 0.019773 / 0.141683 (-0.121910) | 1.157161 / 1.452155 (-0.294994) | 1.210395 / 1.492716 (-0.282321) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095098 / 0.018006 (0.077091) | 0.306952 / 0.000490 (0.306462) | 0.000211 / 0.000200 (0.000011) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022602 / 0.037411 (-0.014809) | 0.075242 / 0.014526 (0.060716) | 0.087134 / 0.176557 (-0.089422) | 0.127923 / 0.737135 (-0.609212) | 0.088645 / 0.296338 (-0.207693) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.304187 / 0.215209 (0.088978) | 2.977120 / 2.077655 (0.899465) | 1.663592 / 1.504120 (0.159473) | 1.527601 / 1.541195 (-0.013594) | 1.540121 / 1.468490 (0.071631) | 0.562492 / 4.584777 (-4.022285) | 2.473836 / 3.745712 (-1.271876) | 2.656782 / 5.269862 (-2.613080) | 1.754212 / 4.565676 (-2.811464) | 0.062330 / 0.424275 (-0.361945) | 0.005149 / 0.007607 (-0.002459) | 0.354905 / 0.226044 (0.128860) | 3.503587 / 2.268929 (1.234659) | 2.015682 / 55.444624 (-53.428943) | 1.744421 / 6.876477 (-5.132056) | 1.923120 / 2.142072 (-0.218952) | 0.652209 / 4.805227 (-4.153018) | 0.119406 / 6.500664 (-6.381258) | 0.042840 / 0.075469 (-0.032630) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.009164 / 1.841788 (-0.832624) | 12.379654 / 8.074308 (4.305346) | 10.408696 / 10.191392 (0.217304) | 0.141674 / 0.680424 (-0.538750) | 0.016815 / 0.534201 (-0.517386) | 0.292453 / 0.579283 (-0.286830) | 0.277577 / 0.434364 (-0.156787) | 0.325024 / 0.540337 (-0.215313) | 0.433181 / 1.386936 (-0.953755) |

![](https://cml.dev/watermark.png#b7a16a08c940e65397305aec5f1b484d91cee75a "CML watermark")

reported in https://discuss.huggingface.co/t/datasetdict-save-to-disk-with-num-proc-1-seems-to-hang-with-error/75595


3x Faster Text Preprocessing

Unfortunately, that won't improve the performance. StringZilla repository has extensive benchmarks comparing against different built-in functionality of several programming languages. Using `re.finditer` for tokenization is practically the slowest anti-pattern I've encountered in any language. The gap between that and a SIMD-accelerated kernel can be as big as 10 MB/s vs 10 GB/s.

I understand the need to keep the dependencies minimal. It helps the package remain small and portable. At this point, StringZilla provides 105 binaries for different OS and hardware versions (more portable than NumPy) and the [binary size generally ranges from 50 KB to 250 KB](https://pypi.org/project/stringzilla/), smaller than a single JPEG.

The `text` builder is not very popular, so I'm also not a fan of introducing a dependency for it.

Moreover, I couldn't find any projects of this size/usage depending on StringZilla (with GitHub search), so we should at least wait for its greater adoption to merge this PR.

I was preparing some datasets for AI training and noticed that `datasets` by HuggingFace uses the conventional `open` mechanism to read the file and split it into chunks. I thought it can be significantly accelerated, and [started with a benchmark](https://gist.github.com/ashvardanian/55c2052e9f78b05b8d614aa90cb12347):

```sh
$ pip install --upgrade --force-reinstall datasets
$ python benchmark_huggingface_datasets.py xlsum.csv
Generating train split: 1004598 examples [00:47, 21116.16 examples/s]
Time taken to load the dataset: 48.66838526725769 seconds
Time taken to chunk the dataset into parts of size 10000: 0.11466407775878906 seconds
Total time taken: 48.78304934501648 seconds
```

For benchmarks I've used a [large CSV file with mixed UTF-8 content](https://github.com/ashvardanian/StringZilla/blob/main/CONTRIBUTING.md#benchmarking-datasets), most common in modern large-scale pre-training pipelines. I've later patched the `datasets` library to use `stringzilla`, which resulted in significantly lower memory consumption and in 2.9x  throughput improvement on the AWS `r7iz` instances. That's using slow SSDs mounted over the network. Performance on local SSDs on something like a DGX-H100 should be even higher:

```sh
$ pip install -e .
$ python benchmark_huggingface_datasets.py xlsum.csv
Generating train split: 1004598 examples [00:15, 64529.90 examples/s]
Time taken to load the dataset: 16.45028805732727 seconds
Time taken to chunk the dataset into parts of size 10000: 0.1291060447692871 seconds
Total time taken: 16.579394102096558 seconds
```

I've already [pushed the patches to my fork](https://github.com/ashvardanian/datasets/tree/faster-text-parsers), and would love to contribute them to the upstream repository.

---

All the tests pass, but they leave a couple of important questions open. The default Python `open(..., newline=None)` uses universal newlines, where `\n`, `\r`, and `\r\n` are all converted to `\n` on the fly. I am not sure if its a good idea for a general purpose dataset preparation pipeline?

I can simulate the same behavior (which I don't yet do) for `"line"` splitter. Adjusting it for `"paragraph"`-splitter would be harder. Should we stick exactly to the old Pythonic behavior or stay closer to how C and other programming languages do that?


Persist IterableDataset epoch in workers

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6710). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Use shared memory for the IterableDataset epoch.

This way calling `ds.set_epoch()` in the main process will update the epoch in the DataLoader workers as well.
This is useful especially because the epoch is used to compute the `effective_seed` used for shuffling.

I used torch's shared memory in case users want to send dataset copies without shared memory using pickle. I also find it easier to use than using `multiprocessing.shared_memory` than requires unlinking only in the main process, or `mp.Value` that is not picklable.

close https://github.com/huggingface/datasets/issues/6673

cc @rwightman


set dev version

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6709). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005081 / 0.011353 (-0.006272) | 0.004182 / 0.011008 (-0.006826) | 0.063377 / 0.038508 (0.024869) | 0.027880 / 0.023109 (0.004770) | 0.247260 / 0.275898 (-0.028638) | 0.273580 / 0.323480 (-0.049900) | 0.002995 / 0.007986 (-0.004991) | 0.002804 / 0.004328 (-0.001524) | 0.049669 / 0.004250 (0.045418) | 0.042469 / 0.037052 (0.005417) | 0.268606 / 0.258489 (0.010117) | 0.292867 / 0.293841 (-0.000973) | 0.028077 / 0.128546 (-0.100469) | 0.011031 / 0.075646 (-0.064615) | 0.210225 / 0.419271 (-0.209047) | 0.035723 / 0.043533 (-0.007810) | 0.252131 / 0.255139 (-0.003008) | 0.272895 / 0.283200 (-0.010304) | 0.019809 / 0.141683 (-0.121874) | 1.138500 / 1.452155 (-0.313655) | 1.167752 / 1.492716 (-0.324964) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094881 / 0.018006 (0.076875) | 0.300168 / 0.000490 (0.299678) | 0.000207 / 0.000200 (0.000007) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.017917 / 0.037411 (-0.019494) | 0.061854 / 0.014526 (0.047328) | 0.074481 / 0.176557 (-0.102075) | 0.120075 / 0.737135 (-0.617061) | 0.074627 / 0.296338 (-0.221711) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.287888 / 0.215209 (0.072679) | 2.770165 / 2.077655 (0.692510) | 1.500071 / 1.504120 (-0.004049) | 1.374857 / 1.541195 (-0.166338) | 1.427291 / 1.468490 (-0.041200) | 0.558431 / 4.584777 (-4.026346) | 2.439352 / 3.745712 (-1.306361) | 2.787471 / 5.269862 (-2.482391) | 1.742636 / 4.565676 (-2.823041) | 0.061716 / 0.424275 (-0.362559) | 0.004961 / 0.007607 (-0.002646) | 0.345209 / 0.226044 (0.119164) | 3.360253 / 2.268929 (1.091325) | 1.847945 / 55.444624 (-53.596680) | 1.595733 / 6.876477 (-5.280744) | 1.642350 / 2.142072 (-0.499723) | 0.638639 / 4.805227 (-4.166588) | 0.116918 / 6.500664 (-6.383746) | 0.042132 / 0.075469 (-0.033338) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.980602 / 1.841788 (-0.861185) | 11.545402 / 8.074308 (3.471094) | 9.452471 / 10.191392 (-0.738921) | 0.129930 / 0.680424 (-0.550494) | 0.014143 / 0.534201 (-0.520058) | 0.290302 / 0.579283 (-0.288981) | 0.263785 / 0.434364 (-0.170579) | 0.339580 / 0.540337 (-0.200758) | 0.450355 / 1.386936 (-0.936581) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005565 / 0.011353 (-0.005788) | 0.003764 / 0.011008 (-0.007244) | 0.050082 / 0.038508 (0.011574) | 0.030354 / 0.023109 (0.007245) | 0.250609 / 0.275898 (-0.025289) | 0.277200 / 0.323480 (-0.046280) | 0.004276 / 0.007986 (-0.003710) | 0.002805 / 0.004328 (-0.001523) | 0.048765 / 0.004250 (0.044514) | 0.045477 / 0.037052 (0.008425) | 0.267704 / 0.258489 (0.009215) | 0.303214 / 0.293841 (0.009373) | 0.029393 / 0.128546 (-0.099153) | 0.010623 / 0.075646 (-0.065023) | 0.058201 / 0.419271 (-0.361070) | 0.053131 / 0.043533 (0.009599) | 0.258682 / 0.255139 (0.003543) | 0.276069 / 0.283200 (-0.007131) | 0.018260 / 0.141683 (-0.123423) | 1.141542 / 1.452155 (-0.310613) | 1.185780 / 1.492716 (-0.306936) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096857 / 0.018006 (0.078850) | 0.300656 / 0.000490 (0.300167) | 0.000450 / 0.000200 (0.000250) | 0.000059 / 0.000054 (0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022416 / 0.037411 (-0.014995) | 0.074781 / 0.014526 (0.060255) | 0.087299 / 0.176557 (-0.089257) | 0.127616 / 0.737135 (-0.609519) | 0.088382 / 0.296338 (-0.207957) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.298639 / 0.215209 (0.083430) | 2.940002 / 2.077655 (0.862347) | 1.709707 / 1.504120 (0.205587) | 1.556502 / 1.541195 (0.015307) | 1.592841 / 1.468490 (0.124351) | 0.570237 / 4.584777 (-4.014539) | 2.467576 / 3.745712 (-1.278137) | 2.741021 / 5.269862 (-2.528840) | 1.776526 / 4.565676 (-2.789151) | 0.063999 / 0.424275 (-0.360276) | 0.005068 / 0.007607 (-0.002539) | 0.360727 / 0.226044 (0.134682) | 3.535404 / 2.268929 (1.266476) | 2.035345 / 55.444624 (-53.409279) | 1.755916 / 6.876477 (-5.120561) | 1.889281 / 2.142072 (-0.252791) | 0.649025 / 4.805227 (-4.156202) | 0.118210 / 6.500664 (-6.382454) | 0.040815 / 0.075469 (-0.034654) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.005650 / 1.841788 (-0.836138) | 12.228314 / 8.074308 (4.154006) | 10.147363 / 10.191392 (-0.044029) | 0.159258 / 0.680424 (-0.521166) | 0.015288 / 0.534201 (-0.518913) | 0.288144 / 0.579283 (-0.291139) | 0.281319 / 0.434364 (-0.153045) | 0.323380 / 0.540337 (-0.216958) | 0.426887 / 1.386936 (-0.960049) |

![](https://cml.dev/watermark.png#8b04ccb486f3831b4b0d2474119823efa3815709 "CML watermark")