Add parallel module using joblib for Spark

Hi @lhoestq, I added the `parallel` part according to the discussion we had. Could you take a look to see if this is aligned with your proposal?

Meanwhile I'm working on adding a `parallel_backend` parameter to `load_datasets` so that it can be used like:
```python
with parallel_backend('spark', steps=['downloading']) as backend:
  ds = load_dataset(..., parallel_backend=backend)
```
where `parallel_backend` is a `ParallelBackend` class.

_The documentation is not available anymore as the PR was closed or merged._

@lhoestq Thanks for the comments!
With your suggestion, no changes made to `load_dataset` and I validated that downloading with spark is working now  with this:
```py
with parallel_backend('spark', steps=["download"]):
      dataset = load_dataset(..., num_proc=2)
```

@lhoestq Can a maintainer help trigger the tests again?
> One idea is to decorate the download method to set the current global step to "download", and then only use joblib if the current step is one of the steps provided in parallel_backend.

Yes I think this is doable in a subsequent PR.
For throwing `NotImplementedError` I also think it can be done in a subsequent PR, because I'm not sure if `Dataset.map` is the only function that a user would expect to run using `with parallel_backend`.

Just triggered the tests :)

> Yes I think this is doable in a subsequent PR.
For throwing NotImplementedError I also think it can be done in a subsequent PR, because I'm not sure if Dataset.map is the only function that a user would expect to run using with parallel_backend.

I think any Dataset method that has a `num_proc` argument: Dataset.map (the other methods like filter or cast or based on map), and later we can see for the to_xxx methods (to_csv, to_parquet, etc.)

Hi maintainers, I've just addressed most of the comments, please take another look, thank you.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008422 / 0.011353 (-0.002931) | 0.005658 / 0.011008 (-0.005350) | 0.135372 / 0.038508 (0.096864) | 0.044766 / 0.023109 (0.021657) | 0.417876 / 0.275898 (0.141978) | 0.462785 / 0.323480 (0.139305) | 0.005485 / 0.007986 (-0.002501) | 0.005640 / 0.004328 (0.001311) | 0.105020 / 0.004250 (0.100770) | 0.049114 / 0.037052 (0.012062) | 0.490450 / 0.258489 (0.231961) | 0.467693 / 0.293841 (0.173852) | 0.050929 / 0.128546 (-0.077617) | 0.014644 / 0.075646 (-0.061002) | 0.452373 / 0.419271 (0.033101) | 0.074897 / 0.043533 (0.031364) | 0.425816 / 0.255139 (0.170677) | 0.420415 / 0.283200 (0.137215) | 0.134121 / 0.141683 (-0.007561) | 1.927744 / 1.452155 (0.475589) | 2.014417 / 1.492716 (0.521701) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.254811 / 0.018006 (0.236805) | 0.550011 / 0.000490 (0.549521) | 0.004913 / 0.000200 (0.004714) | 0.000117 / 0.000054 (0.000062) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032644 / 0.037411 (-0.004768) | 0.135672 / 0.014526 (0.121146) | 0.158984 / 0.176557 (-0.017572) | 0.218267 / 0.737135 (-0.518869) | 0.150348 / 0.296338 (-0.145991) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.625723 / 0.215209 (0.410514) | 6.247559 / 2.077655 (4.169905) | 2.626785 / 1.504120 (1.122666) | 2.195224 / 1.541195 (0.654030) | 2.232140 / 1.468490 (0.763650) | 0.943082 / 4.584777 (-3.641695) | 5.799262 / 3.745712 (2.053550) | 2.849411 / 5.269862 (-2.420450) | 1.744160 / 4.565676 (-2.821516) | 0.119056 / 0.424275 (-0.305219) | 0.014233 / 0.007607 (0.006626) | 0.795238 / 0.226044 (0.569194) | 7.569586 / 2.268929 (5.300657) | 3.179481 / 55.444624 (-52.265143) | 2.519772 / 6.876477 (-4.356704) | 2.714570 / 2.142072 (0.572498) | 1.107197 / 4.805227 (-3.698030) | 0.229986 / 6.500664 (-6.270678) | 0.087993 / 0.075469 (0.012524) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.535610 / 1.841788 (-0.306178) | 18.639369 / 8.074308 (10.565061) | 21.081844 / 10.191392 (10.890452) | 0.253247 / 0.680424 (-0.427177) | 0.026711 / 0.534201 (-0.507490) | 0.503790 / 0.579283 (-0.075493) | 0.600124 / 0.434364 (0.165760) | 0.617944 / 0.540337 (0.077607) | 0.766947 / 1.386936 (-0.619989) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007885 / 0.011353 (-0.003468) | 0.004761 / 0.011008 (-0.006248) | 0.097995 / 0.038508 (0.059487) | 0.033624 / 0.023109 (0.010515) | 0.504307 / 0.275898 (0.228409) | 0.534803 / 0.323480 (0.211323) | 0.006048 / 0.007986 (-0.001937) | 0.005042 / 0.004328 (0.000714) | 0.102288 / 0.004250 (0.098038) | 0.048695 / 0.037052 (0.011643) | 0.559086 / 0.258489 (0.300597) | 0.553233 / 0.293841 (0.259392) | 0.044596 / 0.128546 (-0.083950) | 0.013696 / 0.075646 (-0.061950) | 0.109875 / 0.419271 (-0.309397) | 0.059993 / 0.043533 (0.016460) | 0.485579 / 0.255139 (0.230440) | 0.519835 / 0.283200 (0.236635) | 0.123504 / 0.141683 (-0.018179) | 1.820506 / 1.452155 (0.368351) | 1.963448 / 1.492716 (0.470732) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.292663 / 0.018006 (0.274656) | 0.557783 / 0.000490 (0.557293) | 0.001330 / 0.000200 (0.001130) | 0.000112 / 0.000054 (0.000057) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036890 / 0.037411 (-0.000522) | 0.140373 / 0.014526 (0.125847) | 0.140176 / 0.176557 (-0.036381) | 0.237378 / 0.737135 (-0.499757) | 0.160186 / 0.296338 (-0.136152) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.673599 / 0.215209 (0.458390) | 6.510280 / 2.077655 (4.432625) | 2.981617 / 1.504120 (1.477497) | 2.684664 / 1.541195 (1.143469) | 2.760471 / 1.468490 (1.291981) | 0.975413 / 4.584777 (-3.609364) | 5.708933 / 3.745712 (1.963220) | 2.772069 / 5.269862 (-2.497793) | 1.763627 / 4.565676 (-2.802049) | 0.111632 / 0.424275 (-0.312643) | 0.013223 / 0.007607 (0.005616) | 0.791545 / 0.226044 (0.565500) | 8.063287 / 2.268929 (5.794359) | 3.671920 / 55.444624 (-51.772704) | 3.057248 / 6.876477 (-3.819229) | 3.083569 / 2.142072 (0.941497) | 1.118136 / 4.805227 (-3.687092) | 0.214655 / 6.500664 (-6.286009) | 0.083074 / 0.075469 (0.007605) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.761731 / 1.841788 (-0.080056) | 18.874200 / 8.074308 (10.799892) | 22.383693 / 10.191392 (12.192301) | 0.240292 / 0.680424 (-0.440132) | 0.028850 / 0.534201 (-0.505351) | 0.557334 / 0.579283 (-0.021949) | 0.627732 / 0.434364 (0.193369) | 0.634484 / 0.540337 (0.094146) | 0.767372 / 1.386936 (-0.619564) |

![](https://cml.dev/watermark.png#accaaf2e69fbb5dc5e50229d2eb1591b8ad982b6 "CML watermark")

Discussion in https://github.com/huggingface/datasets/issues/5798


Cannot import datasets - ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility

Based on https://github.com/rapidsai/cudf/issues/10187, this probably means your `pyarrow` installation is not compatible with `datasets`.

Can you please execute the following commands in the terminal and paste the output here?
```
conda list | grep arrow
```
```
python -c "import pyarrow; print(pyarrow.__file__)"
```

> Based on [rapidsai/cudf#10187](https://github.com/rapidsai/cudf/issues/10187), this probably means your `pyarrow` installation is not compatible with `datasets`.
>
> Can you please execute the following commands in the terminal and paste the output here?
>
> ```
> conda list | grep arrow
> ```
>
> ```
> python -c "import pyarrow; print(pyarrow.__file__)"
> ```

Here is the output to the first command:
```
arrow-cpp                 11.0.0           py39h7f74497_0
pyarrow                   12.0.0                   pypi_0    pypi
```
and the second:
```
/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/pyarrow/__init__.py
```
Thanks!

after installing pytesseract 0.3.10, I got the above error. FYI

RuntimeError: Failed to import transformers.trainer because of the following error (look up to see its traceback):
pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject

I got the same error, pyarrow 12.0.0 released May/2023 (https://pypi.org/project/pyarrow/) is not compatible, running `pip install pyarrow==11.0.0` to force install the previous version solved the problem.

Do we need to update dependencies?

For conda with python3.8.16 this solved my problem! thanks!

> I got the same error, pyarrow 12.0.0 released May/2023 (https://pypi.org/project/pyarrow/) is not compatible, running `pip install pyarrow==11.0.0` to force install the previous version solved the problem.
>
> Do we need to update dependencies? I can work on that if no one else is working on it.

Thanks for replying. I am not sure about those environments but it seems like pyarrow-12.0.0 does not work for conda with python 3.8.16.

> Please note that our CI properly passes all tests with `pyarrow-12.0.0`, for Python 3.7 and Python 3.10, for Ubuntu and Windows: see for example https://github.com/huggingface/datasets/actions/runs/5157324334/jobs/9289582291

Got the same error with:

```
arrow-cpp                 11.0.0          py310h7516544_0
pyarrow                   12.0.0                   pypi_0    pypi

python                    3.10.11              h7a1cb2a_2

datasets                  2.13.0             pyhd8ed1ab_0    conda-forge
```

> I got the same error, pyarrow 12.0.0 released May/2023 (https://pypi.org/project/pyarrow/) is not compatible, running `pip install pyarrow==11.0.0` to force install the previous version solved the problem.
>
> Do we need to update dependencies?

This solved the issue for me as well.

> I got the same error, pyarrow 12.0.0 released May/2023 (https://pypi.org/project/pyarrow/) is not compatible, running `pip install pyarrow==11.0.0` to force install the previous version solved the problem.
>
> Do we need to update dependencies?

Solved it for me also

> 基于 [rapidsai/cudf#10187](https://github.com/rapidsai/cudf/issues/10187)，这可能意味着您的安装与 不兼容。`pyarrow``datasets`
>
> 您能否在终端中执行以下命令并将输出粘贴到此处？
>
> ```
> conda list | grep arrow
> ```
>
> ```
> python -c "import pyarrow; print(pyarrow.__file__)"
> ```

arrow-cpp                 11.0.0          py310h7516544_0
pyarrow                   12.0.1                   pypi_0    pypi

/root/miniconda3/lib/python3.10/site-packages/pyarrow/__init__.py

Got the same problem with

arrow-cpp                 11.0.0          py310h1fc3239_0
pyarrow                   12.0.1                   pypi_0    pypi

miniforge3/envs/mlp/lib/python3.10/site-packages/pyarrow/__init__.py

Reverting back to pyarrow 11 solved the problem.

Solved with `pip install pyarrow==11.0.0`

I got different. Solved with
pip install pyarrow==12.0.1
pip install cchardet

env:
Python 3.9.16
transformers  4.32.1

> I got the same error, pyarrow 12.0.0 released May/2023 (https://pypi.org/project/pyarrow/) is not compatible, running `pip install pyarrow==11.0.0` to force install the previous version solved the problem.
>
> Do we need to update dependencies?

This works for me as well

> I got different. Solved with pip install pyarrow==12.0.1 pip install cchardet
>
> env: Python 3.9.16 transformers 4.32.1

I guess it also depends on the Python version. I got Python 3.11.5 and pyarrow==12.0.0.
It works!

> Hi, if this helps anyone, pip install pyarrow==11.0.0 did not work for me (I'm using Colab) but this worked: !pip install --extra-index-url=https://pypi.nvidia.com cudf-cu11

thanks! I met the same problem and your suggestion solved it.

(I was doing quiet install so I didn't notice it initially)
I've been loading the same dataset for months on Colab, just now I got this error as well. I think Colab has changed their image recently (I had some errors regarding CUDA previously as well). beware of this and restart runtime if you're doing quite pip installs.
moreover installing stable version of datasets on pypi gives this:

```
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
ibis-framework 7.1.0 requires pyarrow<15,>=2, but you have pyarrow 15.0.0 which is incompatible.
Successfully installed datasets-2.17.0 dill-0.3.8 multiprocess-0.70.16 pyarrow-15.0.0
WARNING: The following packages were previously imported in this runtime:
  [pyarrow]
You must restart the runtime in order to use newly installed versions.
```

The above methods didn't help me. So I installed an older version: `!pip install datasets==2.16.1`
and `import datasets` worked!!

@rasith1998 @PennlaineChu You can avoid this issue by restarting the session after the `datasets` installation (see https://github.com/huggingface/datasets/issues/6661 for more info)

Also, we've contacted Google Colab folks to update the default PyArrow installation, so the issue should soon be "officially" resolved on their side.

> Also, we've contacted Google Colab folks to update the default PyArrow installation, so the issue should soon be "officially" resolved on their side.

This has been done! Google Colab now pre-installs PyArrow 14.0.2, which makes this issue unlikely to happen, so I'm closing it.

### Describe the bug

When trying to import datasets, I get a pyarrow ValueError:

Traceback (most recent call last):
  File "/Users/edward/test/test.py", line 1, in <module>
    import datasets
  File "/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/datasets/__init__.py", line 43, in <module>
    from .arrow_dataset import Dataset
  File "/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 65, in <module>
    from .arrow_reader import ArrowReader
  File "/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/datasets/arrow_reader.py", line 28, in <module>
    import pyarrow.parquet as pq
  File "/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/pyarrow/parquet/__init__.py", line 20, in <module>
    from .core import *
  File "/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/pyarrow/parquet/core.py", line 45, in <module>
    from pyarrow.fs import (LocalFileSystem, FileSystem, FileType,
  File "/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/pyarrow/fs.py", line 49, in <module>
    from pyarrow._gcsfs import GcsFileSystem  # noqa
  File "pyarrow/_gcsfs.pyx", line 1, in init pyarrow._gcsfs
ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject

### Steps to reproduce the bug

`import datasets`

### Expected behavior

Successful import

### Environment info

Conda environment, MacOS
python 3.9.12
datasets 2.12.0


Length of table does not accurately reflect the split

wontfix

This will not be worked on

As already replied by @lhoestq (private channel):
> `.train_test_split` (as well as `.shard`, `.select`) doesn't create a new arrow table to save time and disk space. Instead, it uses an indices mapping on top of the table that locate which examples are part of train or test.

This is an optimization that we don't plan to "fix", so I'm closing this issue.

### Describe the bug

I load a Huggingface Dataset and do `train_test_split`. I'm expecting the underlying table for the dataset to also be split, but it's not.

### Steps to reproduce the bug

![image](https://github.com/huggingface/datasets/assets/8068268/83e5768f-8b4c-422a-945c-832a7585afff)

### Expected behavior

The expected behavior is when `len(hf_dataset["train"].data)` should match the length of the train split, and not be the entire unsplit dataset.

### Environment info

datasets 2.10.1
python 3.10.11


Fix streaming parquet with image feature in schema

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007088 / 0.011353 (-0.004265) | 0.005216 / 0.011008 (-0.005793) | 0.097572 / 0.038508 (0.059064) | 0.036510 / 0.023109 (0.013401) | 0.316885 / 0.275898 (0.040987) | 0.348541 / 0.323480 (0.025061) | 0.006513 / 0.007986 (-0.001473) | 0.004579 / 0.004328 (0.000251) | 0.073779 / 0.004250 (0.069529) | 0.057500 / 0.037052 (0.020448) | 0.329840 / 0.258489 (0.071351) | 0.357530 / 0.293841 (0.063690) | 0.028515 / 0.128546 (-0.100031) | 0.009156 / 0.075646 (-0.066491) | 0.328340 / 0.419271 (-0.090932) | 0.068400 / 0.043533 (0.024867) | 0.313692 / 0.255139 (0.058553) | 0.329170 / 0.283200 (0.045971) | 0.111969 / 0.141683 (-0.029714) | 1.422096 / 1.452155 (-0.030059) | 1.550042 / 1.492716 (0.057326) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.285113 / 0.018006 (0.267107) | 0.546788 / 0.000490 (0.546298) | 0.006992 / 0.000200 (0.006792) | 0.000097 / 0.000054 (0.000043) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026841 / 0.037411 (-0.010570) | 0.108413 / 0.014526 (0.093887) | 0.118375 / 0.176557 (-0.058181) | 0.174889 / 0.737135 (-0.562246) | 0.122781 / 0.296338 (-0.173558) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.404187 / 0.215209 (0.188978) | 4.039673 / 2.077655 (1.962019) | 1.894616 / 1.504120 (0.390496) | 1.729182 / 1.541195 (0.187987) | 1.772917 / 1.468490 (0.304427) | 0.524046 / 4.584777 (-4.060731) | 3.628111 / 3.745712 (-0.117601) | 1.866075 / 5.269862 (-3.403787) | 1.026435 / 4.565676 (-3.539242) | 0.065328 / 0.424275 (-0.358947) | 0.012717 / 0.007607 (0.005110) | 0.505821 / 0.226044 (0.279777) | 5.049518 / 2.268929 (2.780589) | 2.338486 / 55.444624 (-53.106139) | 2.002874 / 6.876477 (-4.873602) | 2.193049 / 2.142072 (0.050976) | 0.664638 / 4.805227 (-4.140589) | 0.151323 / 6.500664 (-6.349341) | 0.063774 / 0.075469 (-0.011695) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.168168 / 1.841788 (-0.673620) | 15.289200 / 8.074308 (7.214891) | 13.614249 / 10.191392 (3.422857) | 0.167950 / 0.680424 (-0.512474) | 0.017522 / 0.534201 (-0.516679) | 0.393480 / 0.579283 (-0.185803) | 0.420549 / 0.434364 (-0.013815) | 0.461425 / 0.540337 (-0.078912) | 0.563583 / 1.386936 (-0.823353) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006859 / 0.011353 (-0.004493) | 0.004864 / 0.011008 (-0.006144) | 0.075084 / 0.038508 (0.036576) | 0.033989 / 0.023109 (0.010880) | 0.372512 / 0.275898 (0.096614) | 0.394725 / 0.323480 (0.071246) | 0.006382 / 0.007986 (-0.001604) | 0.004521 / 0.004328 (0.000193) | 0.076422 / 0.004250 (0.072172) | 0.055383 / 0.037052 (0.018331) | 0.400974 / 0.258489 (0.142485) | 0.411570 / 0.293841 (0.117729) | 0.028264 / 0.128546 (-0.100282) | 0.009123 / 0.075646 (-0.066523) | 0.081257 / 0.419271 (-0.338015) | 0.048147 / 0.043533 (0.004614) | 0.390735 / 0.255139 (0.135596) | 0.376426 / 0.283200 (0.093226) | 0.108164 / 0.141683 (-0.033518) | 1.429667 / 1.452155 (-0.022488) | 1.556291 / 1.492716 (0.063575) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.289514 / 0.018006 (0.271508) | 0.532860 / 0.000490 (0.532370) | 0.003810 / 0.000200 (0.003611) | 0.000121 / 0.000054 (0.000066) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031292 / 0.037411 (-0.006119) | 0.116530 / 0.014526 (0.102005) | 0.127624 / 0.176557 (-0.048932) | 0.178276 / 0.737135 (-0.558859) | 0.133742 / 0.296338 (-0.162597) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.431505 / 0.215209 (0.216296) | 4.309206 / 2.077655 (2.231551) | 2.174779 / 1.504120 (0.670659) | 1.998122 / 1.541195 (0.456927) | 2.126478 / 1.468490 (0.657988) | 0.528971 / 4.584777 (-4.055806) | 3.797608 / 3.745712 (0.051895) | 1.876275 / 5.269862 (-3.393586) | 1.087458 / 4.565676 (-3.478218) | 0.066940 / 0.424275 (-0.357335) | 0.012432 / 0.007607 (0.004825) | 0.538346 / 0.226044 (0.312301) | 5.370968 / 2.268929 (3.102039) | 2.613718 / 55.444624 (-52.830906) | 2.246585 / 6.876477 (-4.629892) | 2.375695 / 2.142072 (0.233622) | 0.652227 / 4.805227 (-4.153001) | 0.143246 / 6.500664 (-6.357418) | 0.066163 / 0.075469 (-0.009306) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.291263 / 1.841788 (-0.550524) | 16.532281 / 8.074308 (8.457973) | 15.038471 / 10.191392 (4.847079) | 0.168139 / 0.680424 (-0.512285) | 0.017724 / 0.534201 (-0.516477) | 0.391636 / 0.579283 (-0.187648) | 0.429690 / 0.434364 (-0.004674) | 0.474941 / 0.540337 (-0.065396) | 0.579461 / 1.386936 (-0.807475) |

![](https://cml.dev/watermark.png#db690affa0373b08f7cef04e25fe2113ee831ef5 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006083 / 0.011353 (-0.005269) | 0.004085 / 0.011008 (-0.006923) | 0.098337 / 0.038508 (0.059829) | 0.027573 / 0.023109 (0.004464) | 0.305688 / 0.275898 (0.029790) | 0.341767 / 0.323480 (0.018287) | 0.005143 / 0.007986 (-0.002842) | 0.003396 / 0.004328 (-0.000932) | 0.076925 / 0.004250 (0.072674) | 0.041027 / 0.037052 (0.003975) | 0.307877 / 0.258489 (0.049388) | 0.346559 / 0.293841 (0.052718) | 0.025183 / 0.128546 (-0.103363) | 0.008575 / 0.075646 (-0.067071) | 0.319449 / 0.419271 (-0.099823) | 0.043378 / 0.043533 (-0.000154) | 0.304563 / 0.255139 (0.049424) | 0.332019 / 0.283200 (0.048819) | 0.087725 / 0.141683 (-0.053958) | 1.484904 / 1.452155 (0.032749) | 1.582780 / 1.492716 (0.090064) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.197503 / 0.018006 (0.179497) | 0.410370 / 0.000490 (0.409880) | 0.003840 / 0.000200 (0.003640) | 0.000067 / 0.000054 (0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024179 / 0.037411 (-0.013232) | 0.098876 / 0.014526 (0.084350) | 0.106189 / 0.176557 (-0.070367) | 0.168964 / 0.737135 (-0.568171) | 0.109723 / 0.296338 (-0.186616) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.429453 / 0.215209 (0.214244) | 4.295584 / 2.077655 (2.217929) | 2.014330 / 1.504120 (0.510210) | 1.841119 / 1.541195 (0.299924) | 1.928378 / 1.468490 (0.459888) | 0.554571 / 4.584777 (-4.030206) | 3.431769 / 3.745712 (-0.313943) | 1.716204 / 5.269862 (-3.553658) | 0.995054 / 4.565676 (-3.570622) | 0.067374 / 0.424275 (-0.356902) | 0.012557 / 0.007607 (0.004950) | 0.533785 / 0.226044 (0.307740) | 5.363360 / 2.268929 (3.094431) | 2.535190 / 55.444624 (-52.909434) | 2.191646 / 6.876477 (-4.684831) | 2.400799 / 2.142072 (0.258727) | 0.663961 / 4.805227 (-4.141266) | 0.135992 / 6.500664 (-6.364672) | 0.067378 / 0.075469 (-0.008092) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.235110 / 1.841788 (-0.606678) | 13.820695 / 8.074308 (5.746387) | 13.667202 / 10.191392 (3.475810) | 0.143025 / 0.680424 (-0.537399) | 0.016757 / 0.534201 (-0.517444) | 0.356262 / 0.579283 (-0.223021) | 0.401871 / 0.434364 (-0.032493) | 0.423928 / 0.540337 (-0.116410) | 0.514598 / 1.386936 (-0.872338) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006260 / 0.011353 (-0.005093) | 0.004159 / 0.011008 (-0.006850) | 0.076780 / 0.038508 (0.038272) | 0.027899 / 0.023109 (0.004789) | 0.412756 / 0.275898 (0.136858) | 0.455145 / 0.323480 (0.131665) | 0.005029 / 0.007986 (-0.002956) | 0.003482 / 0.004328 (-0.000847) | 0.076148 / 0.004250 (0.071898) | 0.038969 / 0.037052 (0.001917) | 0.429975 / 0.258489 (0.171486) | 0.465880 / 0.293841 (0.172039) | 0.025555 / 0.128546 (-0.102991) | 0.008612 / 0.075646 (-0.067034) | 0.082604 / 0.419271 (-0.336667) | 0.039690 / 0.043533 (-0.003842) | 0.403644 / 0.255139 (0.148505) | 0.440438 / 0.283200 (0.157238) | 0.090984 / 0.141683 (-0.050699) | 1.465915 / 1.452155 (0.013760) | 1.564227 / 1.492716 (0.071511) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.010502 / 0.018006 (-0.007504) | 0.410573 / 0.000490 (0.410083) | 0.000384 / 0.000200 (0.000184) | 0.000059 / 0.000054 (0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025726 / 0.037411 (-0.011686) | 0.101760 / 0.014526 (0.087235) | 0.110102 / 0.176557 (-0.066454) | 0.161321 / 0.737135 (-0.575815) | 0.112507 / 0.296338 (-0.183832) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.469925 / 0.215209 (0.254716) | 4.718740 / 2.077655 (2.641085) | 2.466272 / 1.504120 (0.962152) | 2.267357 / 1.541195 (0.726162) | 2.331343 / 1.468490 (0.862853) | 0.553448 / 4.584777 (-4.031329) | 3.464228 / 3.745712 (-0.281484) | 3.060957 / 5.269862 (-2.208905) | 1.387261 / 4.565676 (-3.178415) | 0.067989 / 0.424275 (-0.356286) | 0.012349 / 0.007607 (0.004741) | 0.575046 / 0.226044 (0.349001) | 5.740322 / 2.268929 (3.471394) | 2.925666 / 55.444624 (-52.518958) | 2.606535 / 6.876477 (-4.269942) | 2.658144 / 2.142072 (0.516072) | 0.655157 / 4.805227 (-4.150071) | 0.138520 / 6.500664 (-6.362144) | 0.069442 / 0.075469 (-0.006027) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.306523 / 1.841788 (-0.535265) | 14.400380 / 8.074308 (6.326072) | 14.231519 / 10.191392 (4.040127) | 0.146194 / 0.680424 (-0.534230) | 0.016632 / 0.534201 (-0.517569) | 0.361151 / 0.579283 (-0.218132) | 0.388838 / 0.434364 (-0.045526) | 0.419337 / 0.540337 (-0.121001) | 0.500483 / 1.386936 (-0.886453) |

![](https://cml.dev/watermark.png#c0429e9806bf7065d03dc5858c039a30c5af716c "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009430 / 0.011353 (-0.001923) | 0.006673 / 0.011008 (-0.004335) | 0.125151 / 0.038508 (0.086643) | 0.038258 / 0.023109 (0.015149) | 0.426383 / 0.275898 (0.150485) | 0.432327 / 0.323480 (0.108847) | 0.006964 / 0.007986 (-0.001022) | 0.005140 / 0.004328 (0.000811) | 0.100767 / 0.004250 (0.096517) | 0.058663 / 0.037052 (0.021610) | 0.424709 / 0.258489 (0.166220) | 0.453049 / 0.293841 (0.159208) | 0.051042 / 0.128546 (-0.077505) | 0.015291 / 0.075646 (-0.060355) | 0.456549 / 0.419271 (0.037278) | 0.067106 / 0.043533 (0.023573) | 0.408959 / 0.255139 (0.153820) | 0.445067 / 0.283200 (0.161867) | 0.115590 / 0.141683 (-0.026092) | 1.929439 / 1.452155 (0.477284) | 2.045709 / 1.492716 (0.552992) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.250726 / 0.018006 (0.232720) | 0.598976 / 0.000490 (0.598486) | 0.007542 / 0.000200 (0.007342) | 0.000101 / 0.000054 (0.000046) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030317 / 0.037411 (-0.007094) | 0.133177 / 0.014526 (0.118651) | 0.152761 / 0.176557 (-0.023795) | 0.233708 / 0.737135 (-0.503428) | 0.147303 / 0.296338 (-0.149036) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.633562 / 0.215209 (0.418353) | 6.235021 / 2.077655 (4.157366) | 2.652573 / 1.504120 (1.148454) | 2.223363 / 1.541195 (0.682168) | 2.231022 / 1.468490 (0.762531) | 0.942218 / 4.584777 (-3.642559) | 6.068661 / 3.745712 (2.322949) | 2.778604 / 5.269862 (-2.491257) | 1.787939 / 4.565676 (-2.777737) | 0.117749 / 0.424275 (-0.306526) | 0.015613 / 0.007607 (0.008006) | 0.810222 / 0.226044 (0.584177) | 7.931509 / 2.268929 (5.662581) | 3.260679 / 55.444624 (-52.183945) | 2.609085 / 6.876477 (-4.267391) | 2.867838 / 2.142072 (0.725766) | 1.144672 / 4.805227 (-3.660555) | 0.224379 / 6.500664 (-6.276285) | 0.084490 / 0.075469 (0.009021) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.650608 / 1.841788 (-0.191179) | 18.919748 / 8.074308 (10.845440) | 20.163162 / 10.191392 (9.971770) | 0.229427 / 0.680424 (-0.450997) | 0.033090 / 0.534201 (-0.501111) | 0.535549 / 0.579283 (-0.043734) | 0.658629 / 0.434364 (0.224265) | 0.631526 / 0.540337 (0.091189) | 0.748701 / 1.386936 (-0.638235) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009157 / 0.011353 (-0.002196) | 0.006153 / 0.011008 (-0.004856) | 0.106294 / 0.038508 (0.067786) | 0.040947 / 0.023109 (0.017837) | 0.493242 / 0.275898 (0.217344) | 0.563525 / 0.323480 (0.240045) | 0.007256 / 0.007986 (-0.000730) | 0.006757 / 0.004328 (0.002429) | 0.105151 / 0.004250 (0.100901) | 0.056262 / 0.037052 (0.019209) | 0.573341 / 0.258489 (0.314852) | 0.591125 / 0.293841 (0.297284) | 0.047935 / 0.128546 (-0.080611) | 0.015385 / 0.075646 (-0.060262) | 0.119457 / 0.419271 (-0.299814) | 0.066510 / 0.043533 (0.022977) | 0.485622 / 0.255139 (0.230483) | 0.540929 / 0.283200 (0.257730) | 0.132619 / 0.141683 (-0.009064) | 1.916905 / 1.452155 (0.464750) | 2.152722 / 1.492716 (0.660006) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.294823 / 0.018006 (0.276817) | 0.569371 / 0.000490 (0.568882) | 0.000642 / 0.000200 (0.000442) | 0.000091 / 0.000054 (0.000036) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034321 / 0.037411 (-0.003090) | 0.134165 / 0.014526 (0.119639) | 0.157871 / 0.176557 (-0.018685) | 0.210753 / 0.737135 (-0.526382) | 0.152961 / 0.296338 (-0.143377) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.686810 / 0.215209 (0.471601) | 6.890432 / 2.077655 (4.812778) | 3.182875 / 1.504120 (1.678755) | 2.770836 / 1.541195 (1.229641) | 2.790785 / 1.468490 (1.322295) | 0.938145 / 4.584777 (-3.646632) | 5.861093 / 3.745712 (2.115381) | 2.719862 / 5.269862 (-2.550000) | 1.760834 / 4.565676 (-2.804842) | 0.111317 / 0.424275 (-0.312958) | 0.015722 / 0.007607 (0.008115) | 0.863032 / 0.226044 (0.636988) | 8.482433 / 2.268929 (6.213504) | 3.892621 / 55.444624 (-51.552003) | 3.207370 / 6.876477 (-3.669106) | 3.344412 / 2.142072 (1.202339) | 1.133903 / 4.805227 (-3.671324) | 0.223456 / 6.500664 (-6.277209) | 0.084335 / 0.075469 (0.008866) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.794116 / 1.841788 (-0.047672) | 19.077447 / 8.074308 (11.003139) | 23.102309 / 10.191392 (12.910917) | 0.268806 / 0.680424 (-0.411617) | 0.027709 / 0.534201 (-0.506492) | 0.540488 / 0.579283 (-0.038796) | 0.658478 / 0.434364 (0.224114) | 0.604769 / 0.540337 (0.064431) | 0.722768 / 1.386936 (-0.664168) |

![](https://cml.dev/watermark.png#7e52021c66666e6953d5be0bd45a079e3ddb8c3f "CML watermark")

It was not reading the feature type from the parquet arrow schema


Optimize IterableDataset.from_file using ArrowExamplesIterable

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007439 / 0.011353 (-0.003914) | 0.004884 / 0.011008 (-0.006124) | 0.098750 / 0.038508 (0.060242) | 0.040723 / 0.023109 (0.017613) | 0.347242 / 0.275898 (0.071344) | 0.381202 / 0.323480 (0.057722) | 0.006814 / 0.007986 (-0.001171) | 0.004543 / 0.004328 (0.000215) | 0.075338 / 0.004250 (0.071088) | 0.058976 / 0.037052 (0.021924) | 0.344746 / 0.258489 (0.086257) | 0.406761 / 0.293841 (0.112920) | 0.028961 / 0.128546 (-0.099585) | 0.009531 / 0.075646 (-0.066115) | 0.337324 / 0.419271 (-0.081947) | 0.051071 / 0.043533 (0.007538) | 0.341251 / 0.255139 (0.086112) | 0.362773 / 0.283200 (0.079573) | 0.109423 / 0.141683 (-0.032260) | 1.457420 / 1.452155 (0.005266) | 1.588824 / 1.492716 (0.096108) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.288620 / 0.018006 (0.270614) | 0.568975 / 0.000490 (0.568485) | 0.003350 / 0.000200 (0.003150) | 0.000088 / 0.000054 (0.000034) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028732 / 0.037411 (-0.008680) | 0.117820 / 0.014526 (0.103294) | 0.120180 / 0.176557 (-0.056376) | 0.178736 / 0.737135 (-0.558399) | 0.126399 / 0.296338 (-0.169939) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.428357 / 0.215209 (0.213148) | 4.251989 / 2.077655 (2.174334) | 2.005239 / 1.504120 (0.501119) | 1.784009 / 1.541195 (0.242815) | 1.883763 / 1.468490 (0.415272) | 0.555429 / 4.584777 (-4.029348) | 3.868146 / 3.745712 (0.122434) | 2.081896 / 5.269862 (-3.187965) | 1.126047 / 4.565676 (-3.439629) | 0.069496 / 0.424275 (-0.354779) | 0.012926 / 0.007607 (0.005318) | 0.536989 / 0.226044 (0.310944) | 5.256052 / 2.268929 (2.987124) | 2.526802 / 55.444624 (-52.917822) | 2.233346 / 6.876477 (-4.643131) | 2.389063 / 2.142072 (0.246990) | 0.677107 / 4.805227 (-4.128120) | 0.147212 / 6.500664 (-6.353452) | 0.067061 / 0.075469 (-0.008408) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.210651 / 1.841788 (-0.631137) | 17.236898 / 8.074308 (9.162589) | 14.427301 / 10.191392 (4.235909) | 0.207194 / 0.680424 (-0.473229) | 0.018079 / 0.534201 (-0.516122) | 0.398355 / 0.579283 (-0.180929) | 0.462453 / 0.434364 (0.028089) | 0.484544 / 0.540337 (-0.055794) | 0.590119 / 1.386936 (-0.796817) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007392 / 0.011353 (-0.003961) | 0.005614 / 0.011008 (-0.005394) | 0.075587 / 0.038508 (0.037079) | 0.040429 / 0.023109 (0.017320) | 0.389901 / 0.275898 (0.114003) | 0.429466 / 0.323480 (0.105986) | 0.006790 / 0.007986 (-0.001196) | 0.006627 / 0.004328 (0.002299) | 0.075227 / 0.004250 (0.070976) | 0.060298 / 0.037052 (0.023246) | 0.391905 / 0.258489 (0.133416) | 0.449385 / 0.293841 (0.155544) | 0.028794 / 0.128546 (-0.099753) | 0.009461 / 0.075646 (-0.066185) | 0.083386 / 0.419271 (-0.335886) | 0.057968 / 0.043533 (0.014435) | 0.377327 / 0.255139 (0.122188) | 0.402825 / 0.283200 (0.119626) | 0.125477 / 0.141683 (-0.016206) | 1.462986 / 1.452155 (0.010832) | 1.595959 / 1.492716 (0.103243) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.304179 / 0.018006 (0.286173) | 0.543113 / 0.000490 (0.542623) | 0.004136 / 0.000200 (0.003936) | 0.000109 / 0.000054 (0.000054) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032617 / 0.037411 (-0.004794) | 0.123596 / 0.014526 (0.109070) | 0.128714 / 0.176557 (-0.047842) | 0.176344 / 0.737135 (-0.560792) | 0.132525 / 0.296338 (-0.163813) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.446041 / 0.215209 (0.230832) | 4.438799 / 2.077655 (2.361144) | 2.210815 / 1.504120 (0.706695) | 2.052025 / 1.541195 (0.510830) | 2.204687 / 1.468490 (0.736197) | 0.535219 / 4.584777 (-4.049558) | 3.858407 / 3.745712 (0.112695) | 3.826043 / 5.269862 (-1.443819) | 1.334149 / 4.565676 (-3.231527) | 0.067454 / 0.424275 (-0.356821) | 0.012566 / 0.007607 (0.004958) | 0.551597 / 0.226044 (0.325553) | 5.520054 / 2.268929 (3.251126) | 2.817976 / 55.444624 (-52.626649) | 2.528074 / 6.876477 (-4.348403) | 2.622391 / 2.142072 (0.480319) | 0.657632 / 4.805227 (-4.147595) | 0.147039 / 6.500664 (-6.353625) | 0.069603 / 0.075469 (-0.005866) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.300140 / 1.841788 (-0.541648) | 17.303907 / 8.074308 (9.229599) | 15.657887 / 10.191392 (5.466495) | 0.168991 / 0.680424 (-0.511433) | 0.021332 / 0.534201 (-0.512869) | 0.487261 / 0.579283 (-0.092022) | 0.450073 / 0.434364 (0.015709) | 0.465865 / 0.540337 (-0.074473) | 0.565501 / 1.386936 (-0.821435) |

![](https://cml.dev/watermark.png#f1723ab75a6b3a5e156ea0a41651e80e91fa9cc6 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006536 / 0.011353 (-0.004817) | 0.004254 / 0.011008 (-0.006755) | 0.095387 / 0.038508 (0.056878) | 0.032885 / 0.023109 (0.009776) | 0.298580 / 0.275898 (0.022682) | 0.319771 / 0.323480 (-0.003709) | 0.005510 / 0.007986 (-0.002476) | 0.003891 / 0.004328 (-0.000437) | 0.073763 / 0.004250 (0.069513) | 0.041625 / 0.037052 (0.004573) | 0.294896 / 0.258489 (0.036407) | 0.341308 / 0.293841 (0.047467) | 0.027898 / 0.128546 (-0.100648) | 0.008837 / 0.075646 (-0.066809) | 0.325055 / 0.419271 (-0.094216) | 0.050652 / 0.043533 (0.007119) | 0.298756 / 0.255139 (0.043617) | 0.318261 / 0.283200 (0.035061) | 0.098927 / 0.141683 (-0.042756) | 1.450356 / 1.452155 (-0.001798) | 1.508034 / 1.492716 (0.015318) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.209009 / 0.018006 (0.191003) | 0.439154 / 0.000490 (0.438665) | 0.004299 / 0.000200 (0.004099) | 0.000142 / 0.000054 (0.000087) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025938 / 0.037411 (-0.011473) | 0.105954 / 0.014526 (0.091429) | 0.113858 / 0.176557 (-0.062698) | 0.168887 / 0.737135 (-0.568249) | 0.121292 / 0.296338 (-0.175046) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.402050 / 0.215209 (0.186841) | 4.002310 / 2.077655 (1.924655) | 1.816190 / 1.504120 (0.312070) | 1.634404 / 1.541195 (0.093209) | 1.713632 / 1.468490 (0.245142) | 0.519633 / 4.584777 (-4.065144) | 3.740291 / 3.745712 (-0.005421) | 1.787602 / 5.269862 (-3.482260) | 1.038844 / 4.565676 (-3.526833) | 0.064973 / 0.424275 (-0.359302) | 0.012475 / 0.007607 (0.004868) | 0.498152 / 0.226044 (0.272108) | 4.970941 / 2.268929 (2.702013) | 2.287429 / 55.444624 (-53.157195) | 1.998050 / 6.876477 (-4.878427) | 2.091903 / 2.142072 (-0.050169) | 0.630363 / 4.805227 (-4.174864) | 0.138623 / 6.500664 (-6.362041) | 0.063293 / 0.075469 (-0.012176) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.201802 / 1.841788 (-0.639986) | 14.073836 / 8.074308 (5.999528) | 12.968665 / 10.191392 (2.777273) | 0.144653 / 0.680424 (-0.535771) | 0.017613 / 0.534201 (-0.516588) | 0.392067 / 0.579283 (-0.187216) | 0.416955 / 0.434364 (-0.017409) | 0.471492 / 0.540337 (-0.068845) | 0.554576 / 1.386936 (-0.832360) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006408 / 0.011353 (-0.004945) | 0.004452 / 0.011008 (-0.006556) | 0.073648 / 0.038508 (0.035140) | 0.032536 / 0.023109 (0.009427) | 0.358546 / 0.275898 (0.082648) | 0.387330 / 0.323480 (0.063850) | 0.005542 / 0.007986 (-0.002444) | 0.003882 / 0.004328 (-0.000447) | 0.073867 / 0.004250 (0.069617) | 0.044798 / 0.037052 (0.007746) | 0.362303 / 0.258489 (0.103814) | 0.400496 / 0.293841 (0.106655) | 0.028244 / 0.128546 (-0.100302) | 0.008931 / 0.075646 (-0.066715) | 0.080617 / 0.419271 (-0.338654) | 0.046575 / 0.043533 (0.003043) | 0.364283 / 0.255139 (0.109145) | 0.373215 / 0.283200 (0.090015) | 0.100080 / 0.141683 (-0.041603) | 1.430047 / 1.452155 (-0.022108) | 1.530957 / 1.492716 (0.038240) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.221061 / 0.018006 (0.203055) | 0.441753 / 0.000490 (0.441263) | 0.003626 / 0.000200 (0.003426) | 0.000088 / 0.000054 (0.000033) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029509 / 0.037411 (-0.007902) | 0.109578 / 0.014526 (0.095053) | 0.121009 / 0.176557 (-0.055548) | 0.168950 / 0.737135 (-0.568185) | 0.124475 / 0.296338 (-0.171864) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.431355 / 0.215209 (0.216146) | 4.295507 / 2.077655 (2.217852) | 2.167514 / 1.504120 (0.663394) | 2.013073 / 1.541195 (0.471879) | 1.973730 / 1.468490 (0.505240) | 0.529778 / 4.584777 (-4.054999) | 3.794702 / 3.745712 (0.048989) | 3.062940 / 5.269862 (-2.206922) | 1.503426 / 4.565676 (-3.062251) | 0.066692 / 0.424275 (-0.357583) | 0.011682 / 0.007607 (0.004075) | 0.539311 / 0.226044 (0.313266) | 5.406342 / 2.268929 (3.137414) | 2.652709 / 55.444624 (-52.791916) | 2.260066 / 6.876477 (-4.616410) | 2.295752 / 2.142072 (0.153680) | 0.647199 / 4.805227 (-4.158029) | 0.142981 / 6.500664 (-6.357683) | 0.065082 / 0.075469 (-0.010387) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.279788 / 1.841788 (-0.562000) | 14.982845 / 8.074308 (6.908536) | 14.277166 / 10.191392 (4.085774) | 0.145082 / 0.680424 (-0.535342) | 0.017885 / 0.534201 (-0.516316) | 0.392071 / 0.579283 (-0.187212) | 0.420425 / 0.434364 (-0.013939) | 0.461244 / 0.540337 (-0.079093) | 0.559956 / 1.386936 (-0.826980) |

![](https://cml.dev/watermark.png#651d96c1c4083a206c65f11602712d75f1f0453d "CML watermark")

following https://github.com/huggingface/datasets/pull/5893


add support for storage_options for load_dataset API

hi @lhoestq,
I saw some errors in my test and found all the failed reasons are `FileNotFoundError`  about `test_load_streaming_private_dataset_with_zipped_data` and `test_load_dataset_private_zipped_images` in `test_load.py `, I run pytest on my own Wins and Ubuntu system all the test in `test_load.py ` are succeed. could you help me to check the test environment of our server?

`2023-06-08T16:50:48.0828281Z FAILED tests/test_load.py::test_load_streaming_private_dataset_with_zipped_data - FileNotFoundError: Couldn't find a dataset script at D:\a\datasets\datasets\__DUMMY_TRANSFORMERS_USER__\repo_zipped_txt_data-16862429577813\repo_zipped_txt_data-16862429577813.py or any data file in the same directory. Couldn't find '__DUMMY_TRANSFORMERS_USER__/repo_zipped_txt_data-16862429577813' on the Hugging Face Hub either: FileNotFoundError: No (supported) data files or dataset script found in __DUMMY_TRANSFORMERS_USER__/repo_zipped_txt_data-16862429577813`
`2023-06-08T16:50:48.0830602Z FAILED tests/test_load.py::test_load_dataset_private_zipped_images[False-False] - FileNotFoundError: Couldn't find a dataset script at D:\a\datasets\datasets\__DUMMY_TRANSFORMERS_USER__\repo_zipped_img_data-16862429594168\repo_zipped_img_data-16862429594168.py or any data file in the same directory. Couldn't find '__DUMMY_TRANSFORMERS_USER__/repo_zipped_img_data-16862429594168' on the Hugging Face Hub either: FileNotFoundError: No (supported) data files or dataset script found in __DUMMY_TRANSFORMERS_USER__/repo_zipped_img_data-16862429594168`

> I just re-ran the CI, hopefully it's fixed

I just checked, still has the same error, maybe need someone to fix it

I think the issue comes from this PR somehow, since the CI fail is related to loading private repositories and this PR touches authentication related code. Let me check what's the issue, and I'll also review your PR later (sorry I don't have a ton of bandwidth atm)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5919). All of your documentation changes will be reflected on that endpoint.

@lhoestq Hi sorry to bother you, the CI check_code_quality failed and it said `would reformat /home/runner/work/datasets/datasets/src/datasets/download/streaming_download_manager.py` but I cant see any changes when I run `python3 -m black --check tests src benchmarks metrics` and `python3 -m ruff tests src benchmarks metrics` on my own computer, is there any version requirements on the tools? I didn't specific the version.

I just ran `make style` and pushed the changes.
You can install the right versions of black and ruff using `pip install -e .[quality]` ;)

I am working on this issue right now https://github.com/huggingface/datasets/issues/6017 which is strongly connected to your PR, and I might end up cherry-picking some of your commits (keeping attribution of course !). Would you be ok with that ?

it's totally ok for me, I just wish the S3 File system could support streaming too.

I already adjust the code and test on my local Mac, you can check it now, and you can make any changes to it.

Closing this PR in favor of https://github.com/huggingface/datasets/pull/6028 which includes your contribution :)

to solve the issue in #5880

1.  add s3 support in the link check step, previous we only check `http` and `https`,

2. change the parameter of `use_auth_token` to `download_config` to support both `storage_options` and `use_auth_token` parameter when trying to handle(list, open, read, etc,.) the remote files.

3.  integrate the check part's duplicate code to make adding or deleting other sources easier.


File not found for audio dataset

load_dataset () did not work for loading local files either

### Describe the bug

After loading an audio dataset, and looking at a sample entry, the `path` element, which is supposed to be the path to the audio file, doesn't actually exist.

### Steps to reproduce the bug

Run bug.py:

```py
import os.path

from datasets import load_dataset

def run() -> None:
    cv13 = load_dataset(
        "mozilla-foundation/common_voice_13_0",
        "hi",
        split="train",
    )

    print(cv13[0])
    audio_file = cv13[0]["path"]
    if not os.path.exists(audio_file):
        raise ValueError(f'File {audio_file} does not exist.')

if __name__ == "__main__":
    run()
```

The result (on my machine):

```json
{'client_id': '0f018a99663f33afbb7d38aee281fb1afcfd07f9e7acd00383f604e1e17c38d6ed8adf1bd2ccbf927a52c5adefb8ac4b158ce27a7c2ed9581e71202eb302dfb3', 'path': 'C:\\Users\\rober\\.cache\\huggingface\\datasets\\downloads\\extracted\\8d1479bc09b4609bc2675bd02d6869a4d5e09f7e6616f540bd55eacef46c6e2b\\common_voice_hi_26008353.mp3', 'audio': {'path': 'C:\\Users\\rober\\.cache\\huggingface\\datasets\\downloads\\extracted\\8d1479bc09b4609bc2675bd02d6869a4d5e09f7e6616f540bd55eacef46c6e2b\\common_voice_hi_26008353.mp3', 'array': array([ 6.46234854e-26, -1.35709319e-25, -8.07793567e-26, ...,
        1.06425944e-07,  4.46417090e-08,  2.61451660e-09]), 'sampling_rate': 48000}, 'sentence': 'हमने उसका जन्मदिन मनाया।', 'up_votes': 2, 'down_votes': 0, 'age': '', 'gender': '', 'accent': '', 'locale': 'hi', 'segment': '' ', 'variant': ''}
```

```txt
Traceback (most recent call last):
  File "F:\eo-reco\bug.py", line 18, in <module>
    run()
  File "F:\eo-reco\bug.py", line 15, in run
    raise ValueError(f'File {audio_file} does not exist.')
ValueError: File C:\Users\rober\.cache\huggingface\datasets\downloads\extracted\8d1479bc09b4609bc2675bd02d6869a4d5e09f7e6616f540bd55eacef46c6e2b\common_voice_hi_26008353.mp3 does not exist.
```

### Expected behavior

The `path` element points to the correct file, which happens to be:

```
C:\Users\rober\.cache\huggingface\datasets\downloads\extracted\8d1479bc09b4609bc2675bd02d6869a4d5e09f7e6616f540bd55eacef46c6e2b\hi_train_0\common_voice_hi_26008353.mp3
```

That is, there's an extra directory `hi_train_0` that is not in the `path` element.

### Environment info

- `datasets` version: 2.12.0
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.11.3
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1
-


Refactor extensions

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008358 / 0.011353 (-0.002995) | 0.005673 / 0.011008 (-0.005335) | 0.124034 / 0.038508 (0.085526) | 0.037550 / 0.023109 (0.014441) | 0.331301 / 0.275898 (0.055403) | 0.383542 / 0.323480 (0.060062) | 0.006940 / 0.007986 (-0.001046) | 0.005959 / 0.004328 (0.001631) | 0.084670 / 0.004250 (0.080419) | 0.054214 / 0.037052 (0.017162) | 0.359897 / 0.258489 (0.101408) | 0.383260 / 0.293841 (0.089419) | 0.047642 / 0.128546 (-0.080904) | 0.013902 / 0.075646 (-0.061744) | 0.380232 / 0.419271 (-0.039040) | 0.077790 / 0.043533 (0.034257) | 0.376648 / 0.255139 (0.121509) | 0.387536 / 0.283200 (0.104336) | 0.104644 / 0.141683 (-0.037038) | 1.618560 / 1.452155 (0.166406) | 1.742569 / 1.492716 (0.249853) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.257218 / 0.018006 (0.239212) | 0.636801 / 0.000490 (0.636311) | 0.000634 / 0.000200 (0.000434) | 0.000101 / 0.000054 (0.000047) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.037874 / 0.037411 (0.000462) | 0.107454 / 0.014526 (0.092928) | 0.117855 / 0.176557 (-0.058702) | 0.204067 / 0.737135 (-0.533068) | 0.134029 / 0.296338 (-0.162310) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.583657 / 0.215209 (0.368447) | 5.761289 / 2.077655 (3.683635) | 2.280201 / 1.504120 (0.776081) | 2.033442 / 1.541195 (0.492247) | 2.035343 / 1.468490 (0.566853) | 0.868122 / 4.584777 (-3.716655) | 5.352591 / 3.745712 (1.606879) | 2.432814 / 5.269862 (-2.837047) | 1.560765 / 4.565676 (-3.004911) | 0.098793 / 0.424275 (-0.325482) | 0.017327 / 0.007607 (0.009720) | 0.734676 / 0.226044 (0.508631) | 7.070318 / 2.268929 (4.801390) | 2.972701 / 55.444624 (-52.471924) | 2.442189 / 6.876477 (-4.434288) | 2.604379 / 2.142072 (0.462307) | 1.028853 / 4.805227 (-3.776374) | 0.210390 / 6.500664 (-6.290274) | 0.069329 / 0.075469 (-0.006140) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.469586 / 1.841788 (-0.372202) | 16.570305 / 8.074308 (8.495997) | 19.187845 / 10.191392 (8.996453) | 0.219162 / 0.680424 (-0.461262) | 0.026356 / 0.534201 (-0.507845) | 0.447370 / 0.579283 (-0.131913) | 0.555893 / 0.434364 (0.121529) | 0.574958 / 0.540337 (0.034621) | 0.639166 / 1.386936 (-0.747770) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008166 / 0.011353 (-0.003187) | 0.005577 / 0.011008 (-0.005431) | 0.103578 / 0.038508 (0.065070) | 0.040563 / 0.023109 (0.017454) | 0.441996 / 0.275898 (0.166098) | 0.483594 / 0.323480 (0.160114) | 0.007329 / 0.007986 (-0.000657) | 0.004546 / 0.004328 (0.000218) | 0.090471 / 0.004250 (0.086220) | 0.052740 / 0.037052 (0.015688) | 0.442197 / 0.258489 (0.183708) | 0.524310 / 0.293841 (0.230469) | 0.042487 / 0.128546 (-0.086060) | 0.012917 / 0.075646 (-0.062730) | 0.103992 / 0.419271 (-0.315280) | 0.060570 / 0.043533 (0.017037) | 0.441956 / 0.255139 (0.186817) | 0.477084 / 0.283200 (0.193885) | 0.103815 / 0.141683 (-0.037868) | 1.696963 / 1.452155 (0.244809) | 1.747849 / 1.492716 (0.255132) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.292465 / 0.018006 (0.274458) | 0.571518 / 0.000490 (0.571028) | 0.000476 / 0.000200 (0.000276) | 0.000077 / 0.000054 (0.000022) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028697 / 0.037411 (-0.008714) | 0.111671 / 0.014526 (0.097145) | 0.138826 / 0.176557 (-0.037731) | 0.189697 / 0.737135 (-0.547439) | 0.125454 / 0.296338 (-0.170884) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.619273 / 0.215209 (0.404064) | 6.138669 / 2.077655 (4.061015) | 2.558622 / 1.504120 (1.054502) | 2.201550 / 1.541195 (0.660356) | 2.279034 / 1.468490 (0.810544) | 0.850752 / 4.584777 (-3.734025) | 5.438185 / 3.745712 (1.692473) | 2.529343 / 5.269862 (-2.740518) | 1.572178 / 4.565676 (-2.993499) | 0.100768 / 0.424275 (-0.323507) | 0.013902 / 0.007607 (0.006295) | 0.726660 / 0.226044 (0.500616) | 7.794918 / 2.268929 (5.525990) | 3.311695 / 55.444624 (-52.132930) | 2.729167 / 6.876477 (-4.147310) | 2.630984 / 2.142072 (0.488911) | 1.018534 / 4.805227 (-3.786693) | 0.194602 / 6.500664 (-6.306062) | 0.070876 / 0.075469 (-0.004593) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.573005 / 1.841788 (-0.268783) | 17.042710 / 8.074308 (8.968401) | 19.615320 / 10.191392 (9.423928) | 0.229405 / 0.680424 (-0.451019) | 0.027560 / 0.534201 (-0.506641) | 0.447984 / 0.579283 (-0.131299) | 0.598392 / 0.434364 (0.164028) | 0.571769 / 0.540337 (0.031431) | 0.653025 / 1.386936 (-0.733911) |

![](https://cml.dev/watermark.png#9dca2ff89a8589595313e9535d16597ce10e3700 "CML watermark")

Related to:
- #5850