fixing the issue 6755(small typo)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6767). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005526 / 0.011353 (-0.005827) | 0.003839 / 0.011008 (-0.007169) | 0.064027 / 0.038508 (0.025519) | 0.032316 / 0.023109 (0.009206) | 0.250707 / 0.275898 (-0.025191) | 0.269222 / 0.323480 (-0.054258) | 0.004335 / 0.007986 (-0.003651) | 0.002703 / 0.004328 (-0.001626) | 0.049621 / 0.004250 (0.045370) | 0.047499 / 0.037052 (0.010446) | 0.262362 / 0.258489 (0.003873) | 0.292765 / 0.293841 (-0.001076) | 0.028661 / 0.128546 (-0.099885) | 0.010835 / 0.075646 (-0.064811) | 0.208910 / 0.419271 (-0.210362) | 0.036624 / 0.043533 (-0.006909) | 0.247448 / 0.255139 (-0.007691) | 0.270593 / 0.283200 (-0.012607) | 0.018988 / 0.141683 (-0.122695) | 1.141224 / 1.452155 (-0.310931) | 1.204944 / 1.492716 (-0.287772) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096324 / 0.018006 (0.078318) | 0.292495 / 0.000490 (0.292006) | 0.000232 / 0.000200 (0.000032) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018379 / 0.037411 (-0.019032) | 0.065216 / 0.014526 (0.050690) | 0.074071 / 0.176557 (-0.102486) | 0.120793 / 0.737135 (-0.616343) | 0.075882 / 0.296338 (-0.220456) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.286354 / 0.215209 (0.071145) | 2.800766 / 2.077655 (0.723111) | 1.474126 / 1.504120 (-0.029994) | 1.358232 / 1.541195 (-0.182963) | 1.400639 / 1.468490 (-0.067851) | 0.578354 / 4.584777 (-4.006423) | 2.454441 / 3.745712 (-1.291271) | 2.927003 / 5.269862 (-2.342859) | 1.826127 / 4.565676 (-2.739550) | 0.063049 / 0.424275 (-0.361226) | 0.005010 / 0.007607 (-0.002597) | 0.342174 / 0.226044 (0.116129) | 3.415900 / 2.268929 (1.146971) | 1.854096 / 55.444624 (-53.590528) | 1.568626 / 6.876477 (-5.307851) | 1.660138 / 2.142072 (-0.481934) | 0.664059 / 4.805227 (-4.141168) | 0.120496 / 6.500664 (-6.380168) | 0.044664 / 0.075469 (-0.030805) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.988434 / 1.841788 (-0.853353) | 12.525563 / 8.074308 (4.451255) | 10.016862 / 10.191392 (-0.174530) | 0.134043 / 0.680424 (-0.546381) | 0.014349 / 0.534201 (-0.519852) | 0.287173 / 0.579283 (-0.292110) | 0.266499 / 0.434364 (-0.167865) | 0.325425 / 0.540337 (-0.214912) | 0.418772 / 1.386936 (-0.968164) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005675 / 0.011353 (-0.005678) | 0.004238 / 0.011008 (-0.006770) | 0.051048 / 0.038508 (0.012540) | 0.033428 / 0.023109 (0.010319) | 0.283406 / 0.275898 (0.007508) | 0.309321 / 0.323480 (-0.014159) | 0.004354 / 0.007986 (-0.003631) | 0.003101 / 0.004328 (-0.001228) | 0.049369 / 0.004250 (0.045119) | 0.043252 / 0.037052 (0.006200) | 0.293097 / 0.258489 (0.034608) | 0.324392 / 0.293841 (0.030551) | 0.030524 / 0.128546 (-0.098022) | 0.010977 / 0.075646 (-0.064669) | 0.058546 / 0.419271 (-0.360726) | 0.033295 / 0.043533 (-0.010238) | 0.284929 / 0.255139 (0.029790) | 0.302925 / 0.283200 (0.019726) | 0.018586 / 0.141683 (-0.123097) | 1.156552 / 1.452155 (-0.295602) | 1.208856 / 1.492716 (-0.283860) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096938 / 0.018006 (0.078932) | 0.305375 / 0.000490 (0.304886) | 0.000227 / 0.000200 (0.000027) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022658 / 0.037411 (-0.014754) | 0.078125 / 0.014526 (0.063599) | 0.087892 / 0.176557 (-0.088665) | 0.127745 / 0.737135 (-0.609390) | 0.089806 / 0.296338 (-0.206533) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.292434 / 0.215209 (0.077225) | 2.862329 / 2.077655 (0.784674) | 1.607948 / 1.504120 (0.103828) | 1.487179 / 1.541195 (-0.054016) | 1.542234 / 1.468490 (0.073744) | 0.579446 / 4.584777 (-4.005331) | 2.478549 / 3.745712 (-1.267163) | 2.923493 / 5.269862 (-2.346369) | 1.833161 / 4.565676 (-2.732515) | 0.064289 / 0.424275 (-0.359986) | 0.005638 / 0.007607 (-0.001969) | 0.350111 / 0.226044 (0.124067) | 3.436035 / 2.268929 (1.167107) | 1.970592 / 55.444624 (-53.474032) | 1.717474 / 6.876477 (-5.159002) | 1.753150 / 2.142072 (-0.388922) | 0.660495 / 4.805227 (-4.144732) | 0.119302 / 6.500664 (-6.381362) | 0.042633 / 0.075469 (-0.032836) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.018761 / 1.841788 (-0.823027) | 12.859834 / 8.074308 (4.785525) | 10.547789 / 10.191392 (0.356397) | 0.131986 / 0.680424 (-0.548438) | 0.016469 / 0.534201 (-0.517732) | 0.288585 / 0.579283 (-0.290698) | 0.270499 / 0.434364 (-0.163865) | 0.325801 / 0.540337 (-0.214537) | 0.416551 / 1.386936 (-0.970385) |

![](https://cml.dev/watermark.png#7599f15537b094bfd18de5af7bb2a482c06d7a0e "CML watermark")

Fixed the issue #6755 on the typo mistake


Compatibility issue between s3fs, fsspec, and datasets

> Hi! Instead of running `pip install` separately for each package, you should pass all the packages to a single `pip install` call (in this case, `pip install datasets s3fs`) to let `pip` properly resolve their versions.

Thanks so much! My inexperience with pip is showing 😆  🙈

> Hi! Instead of running `pip install` separately for each package, you should pass all the packages to a single `pip install` call (in this case, `pip install datasets s3fs`) to let `pip` properly resolve their versions.

you are awesome bro

### Describe the bug

Here is the full error stack when installing:

```
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datasets 2.18.0 requires fsspec[http]<=2024.2.0,>=2023.1.0, but you have fsspec 2024.3.1 which is incompatible.
Successfully installed aiobotocore-2.12.1 aioitertools-0.11.0 botocore-1.34.51 fsspec-2024.3.1 jmespath-1.0.1 s3fs-2024.3.1 urllib3-2.0.7 wrapt-1.16.0
```
When I install with pip, pip allows this error to exist while still installing s3fs, but this error breaks poetry, since poetry will refuse to install s3fs because of the dependency conflict.

Maybe I'm missing something so maybe it's not a bug but some mistake on my end? Any input would be helpful. Thanks!

### Steps to reproduce the bug

1. conda create -n tmp python=3.10 -y
2. conda activate tmp
3. pip install datasets
4. pip install s3fs

### Expected behavior

I would expect there to be no error.

### Environment info

MacOS (ARM), Python3.10, conda 23.11.0.


load_dataset can't work with symbolic links

enhancement

New feature or request

### Feature request

Enable the `load_dataset` function to load local datasets with symbolic links.

E.g, this dataset can be loaded:
├── example_dataset/
│   ├── data/
│   │   ├── train/
│   │   │   ├── file0
│   │   │   ├── file1
│   │   ├── dev/
│   │   │   ├── file2
│   │   │   ├── file3
│   ├── metadata.csv

while this dataset can't:
├── example_dataset_symlink/
│   ├── data/
│   │   ├── train/
│   │   │   ├── sym0 -> file0
│   │   │   ├── sym1 -> file1
│   │   ├── dev/
│   │   │   ├── sym2 -> file2
│   │   │   ├── sym3 -> file3
│   ├── metadata.csv

I have created an example dataset in order to reproduce the problem:

1. Unzip `example_dataset.zip`.
2. Run `no_symlink.sh`. Training should start without issues.
3. Run `symlink.sh`. You will see that all four examples will be in train split, instead of having two examples in train and two examples in dev. The script won't load the correct audio files.

[example_dataset.zip](https://github.com/huggingface/datasets/files/14807053/example_dataset.zip)

### Motivation

I have a very large dataset locally. Instead of initiating training on the entire dataset, I need to start training on smaller subsets of the data. Due to the purpose of the experiments I am running, I will need to create many smaller datasets with overlapping data. Instead of copying the all the files for each subset, I would prefer copying symbolic links of the data. This way, the memory usage would not significantly increase beyond the initial dataset size.

Advantages of this approach:

- It would leave a smaller memory footprint on the hard drive
- Creating smaller datasets would be much faster

### Your contribution

I would gladly contribute, if this is something useful to the community. It seems like a simple change of code, something like `file_path = os.path.realpath(file_path)` should be added before loading the files. If anyone has insights on how to incorporate this functionality, I would greatly appreciate your knowledge and input.


Fix issue with case sensitivity when loading dataset from local cache

I also need this feature for ["Cnam-LMSSC/vibravox "](https://huggingface.co/datasets/Cnam-LMSSC/vibravox)

EDIT: Upgrading to `2.19.0` fixed my problem thanks to [this PR](https://github.com/huggingface/datasets/pull/6754)

When a dataset with upper-cases in its name is first loaded using `load_dataset()`, the local cache directory is created with all lowercase letters.

However, upon subsequent loads, the current version attempts to locate the cache directory using the dataset's original name, which includes uppercase letters. This discrepancy can lead to confusion and, particularly in offline mode, results in errors.

### Reproduce
```bash
~$ python
Python 3.9.19 (main, Mar 21 2024, 17:11:28)
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from datasets import load_dataset
>>> dataset = load_dataset("locuslab/TOFU", "full")
>>> quit()

~$ export HF_DATASETS_OFFLINE=1
~$ python
Python 3.9.19 (main, Mar 21 2024, 17:11:28)
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from datasets import load_dataset
>>> dataset = load_dataset("locuslab/TOFU", "full")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "xxxxxx/anaconda3/envs/llm/lib/python3.9/site-packages/datasets/load.py", line 2556, in load_dataset
    builder_instance = load_dataset_builder(
  File "xxxxxx/anaconda3/envs/llm/lib/python3.9/site-packages/datasets/load.py", line 2228, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "xxxxxx/anaconda3/envs/llm/lib/python3.9/site-packages/datasets/load.py", line 1871, in dataset_module_factory
    raise ConnectionError(f"Couldn't reach the Hugging Face Hub for dataset '{path}': {e1}") from None
ConnectionError: Couldn't reach the Hugging Face Hub for dataset 'locuslab/TOFU': Offline mode is enabled.
>>>
```

I fix this issue by lowering the dataset name (`.lower()`) when generating cache_dir.


Allow polars as valid output type

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6762). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I was trying out polars as an output for a map function and found that it wasn't a valid return type in `validate_function_output`. Thought that we should accommodate this by creating and adding it to the `allowed_processed_input_types` variable.


Remove deprecated code

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6761). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for cleaning this :) I'm also fine with renaming `hf_dataset_url` (and not `get_dataset_url` as you said in your OP)

(Yep, `hf_dataset_url` is fine, made a mistake writing the PR description)

@albertvillanova Sorry about that, tests are now fixed! :)

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005357 / 0.011353 (-0.005995) | 0.003788 / 0.011008 (-0.007220) | 0.063630 / 0.038508 (0.025122) | 0.031353 / 0.023109 (0.008244) | 0.247525 / 0.275898 (-0.028373) | 0.282052 / 0.323480 (-0.041428) | 0.004247 / 0.007986 (-0.003739) | 0.002750 / 0.004328 (-0.001579) | 0.049467 / 0.004250 (0.045217) | 0.046663 / 0.037052 (0.009610) | 0.266440 / 0.258489 (0.007951) | 0.295230 / 0.293841 (0.001389) | 0.028271 / 0.128546 (-0.100276) | 0.011116 / 0.075646 (-0.064530) | 0.222092 / 0.419271 (-0.197179) | 0.036627 / 0.043533 (-0.006906) | 0.252607 / 0.255139 (-0.002532) | 0.271231 / 0.283200 (-0.011969) | 0.019070 / 0.141683 (-0.122613) | 1.152645 / 1.452155 (-0.299509) | 1.211267 / 1.492716 (-0.281449) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095002 / 0.018006 (0.076996) | 0.304054 / 0.000490 (0.303564) | 0.000212 / 0.000200 (0.000012) | 0.000056 / 0.000054 (0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018251 / 0.037411 (-0.019161) | 0.061929 / 0.014526 (0.047403) | 0.074641 / 0.176557 (-0.101916) | 0.122643 / 0.737135 (-0.614492) | 0.076744 / 0.296338 (-0.219594) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.284605 / 0.215209 (0.069396) | 2.774638 / 2.077655 (0.696984) | 1.473907 / 1.504120 (-0.030213) | 1.351054 / 1.541195 (-0.190141) | 1.348840 / 1.468490 (-0.119650) | 0.576243 / 4.584777 (-4.008534) | 2.444110 / 3.745712 (-1.301602) | 2.814741 / 5.269862 (-2.455121) | 1.762666 / 4.565676 (-2.803010) | 0.063959 / 0.424275 (-0.360316) | 0.005011 / 0.007607 (-0.002596) | 0.338406 / 0.226044 (0.112361) | 3.361213 / 2.268929 (1.092284) | 1.832674 / 55.444624 (-53.611950) | 1.564229 / 6.876477 (-5.312248) | 1.570843 / 2.142072 (-0.571230) | 0.657134 / 4.805227 (-4.148093) | 0.120041 / 6.500664 (-6.380623) | 0.048594 / 0.075469 (-0.026875) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.965328 / 1.841788 (-0.876460) | 11.704441 / 8.074308 (3.630133) | 9.895462 / 10.191392 (-0.295930) | 0.131913 / 0.680424 (-0.548511) | 0.015175 / 0.534201 (-0.519026) | 0.292022 / 0.579283 (-0.287261) | 0.269752 / 0.434364 (-0.164612) | 0.330453 / 0.540337 (-0.209884) | 0.421659 / 1.386936 (-0.965277) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005472 / 0.011353 (-0.005881) | 0.003809 / 0.011008 (-0.007199) | 0.049594 / 0.038508 (0.011086) | 0.031858 / 0.023109 (0.008748) | 0.277622 / 0.275898 (0.001724) | 0.296092 / 0.323480 (-0.027388) | 0.004209 / 0.007986 (-0.003777) | 0.002726 / 0.004328 (-0.001603) | 0.048057 / 0.004250 (0.043806) | 0.043317 / 0.037052 (0.006265) | 0.288371 / 0.258489 (0.029882) | 0.312847 / 0.293841 (0.019007) | 0.029110 / 0.128546 (-0.099437) | 0.010792 / 0.075646 (-0.064854) | 0.058694 / 0.419271 (-0.360577) | 0.033315 / 0.043533 (-0.010218) | 0.281225 / 0.255139 (0.026086) | 0.297044 / 0.283200 (0.013844) | 0.018897 / 0.141683 (-0.122786) | 1.156417 / 1.452155 (-0.295738) | 1.221393 / 1.492716 (-0.271323) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095065 / 0.018006 (0.077059) | 0.304107 / 0.000490 (0.303618) | 0.000213 / 0.000200 (0.000014) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021658 / 0.037411 (-0.015753) | 0.075948 / 0.014526 (0.061423) | 0.087019 / 0.176557 (-0.089537) | 0.127309 / 0.737135 (-0.609827) | 0.092251 / 0.296338 (-0.204087) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.291906 / 0.215209 (0.076697) | 2.865007 / 2.077655 (0.787352) | 1.591647 / 1.504120 (0.087527) | 1.474499 / 1.541195 (-0.066696) | 1.496644 / 1.468490 (0.028154) | 0.575337 / 4.584777 (-4.009440) | 2.569426 / 3.745712 (-1.176287) | 2.872611 / 5.269862 (-2.397251) | 1.804278 / 4.565676 (-2.761399) | 0.064225 / 0.424275 (-0.360050) | 0.005574 / 0.007607 (-0.002033) | 0.347724 / 0.226044 (0.121680) | 3.426418 / 2.268929 (1.157490) | 1.966270 / 55.444624 (-53.478355) | 1.687790 / 6.876477 (-5.188686) | 1.728530 / 2.142072 (-0.413542) | 0.650251 / 4.805227 (-4.154977) | 0.118381 / 6.500664 (-6.382283) | 0.041693 / 0.075469 (-0.033776) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.014203 / 1.841788 (-0.827585) | 12.219496 / 8.074308 (4.145188) | 10.469677 / 10.191392 (0.278285) | 0.141840 / 0.680424 (-0.538584) | 0.015104 / 0.534201 (-0.519097) | 0.288453 / 0.579283 (-0.290830) | 0.287467 / 0.434364 (-0.146897) | 0.331046 / 0.540337 (-0.209292) | 0.423731 / 1.386936 (-0.963205) |

![](https://cml.dev/watermark.png#66d6242626eada79cfba4df39d99cd2bacb1cbea "CML watermark")

What does this PR do?
1. remove `list_files_info` in favor of `list_repo_tree`. As of `0.23`, `list_files_info` will be removed for good. `datasets` had a utility to support both pre-0.20 and post-0.20 versions. Since `hfh` version is already pinned to `>=0.21.2`, I removed the legacy part.
2. `preupload_lfs_files` had also a different behavior between `<0.20` and `>=0.20`. I remove it since huggingface_hub is now pinned to `>=0.21.2`
3. `hf_hub_url` is overwritten to default to the dataset repo_type. I do think it is misleading to keep the same method naming for it. I renamed it to `get_dataset_url` for clarity. Let me know if you prefer to see this change reverted.


Load codeparrot/apps raising UnicodeDecodeError in datasets-2.18.0

Unfortunately, I'm unable to reproduce this error locally or on Colab.

Here is the requirements.txt from a clean virtual environment (managed by conda) where I only install `datasets` by
`pip install datasets`.
The pip list:
```
aiohttp==3.9.3
aiosignal==1.3.1
attrs==23.2.0
certifi==2024.2.2
charset-normalizer==3.3.2
datasets==2.18.0
dill==0.3.8
filelock==3.13.3
frozenlist==1.4.1
fsspec==2024.2.0
huggingface-hub==0.22.2
idna==3.6
multidict==6.0.5
multiprocess==0.70.16
numpy==1.26.4
packaging==24.0
pandas==2.2.1
pyarrow==15.0.2
pyarrow-hotfix==0.6
python-dateutil==2.9.0.post0
pytz==2024.1
PyYAML==6.0.1
requests==2.31.0
six==1.16.0
tqdm==4.66.2
typing_extensions==4.11.0
tzdata==2024.1
urllib3==2.2.1
xxhash==3.4.1
yarl==1.9.4
```
And the error can be reproduced.

Downgrading to datasets==2.14.6 changes some packages' versions:

```
Successfully installed datasets-2.14.6 dill-0.3.7 fsspec-2023.10.0 multiprocess-0.70.15
```
and the dataset can be downloaded and loaded.

Then I upgrade the version to 2.18.0 again; now the dataset can be loaded with such a line:
```Using the latest cached version of the module from /home/xxx/.cache/huggingface/modules/datasets_modules/datasets/codeparrot--apps/04ac807715d07d6e5cc580f59cdc8213cd7dc4529d0bb819cca72c9f8e8c1aa5 (last modified on Sun Apr  7 09:06:43 2024) since it couldn't be found locally at codeparrot/apps, or remotely on the Hugging Face Hub. ```

So the latest version works wrong when requesting the dataset info.

**But if you cannot reproduce this, I may ignore some detailed information: I use `HF_ENDPOINT=https://hf-mirror.com` for some reason (if not use this I cannot connect to huggingface resources) and the error occurs when requesting the dataset's info card.**
Maybe the error is caused by this environment variable.
I'll open an issue in the author's repo now.

> Here is the requirements.txt from a clean virtual environment (managed by conda) where I only install `datasets` by `pip install datasets`. The pip list:
>
> ```
> aiohttp==3.9.3
> aiosignal==1.3.1
> attrs==23.2.0
> certifi==2024.2.2
> charset-normalizer==3.3.2
> datasets==2.18.0
> dill==0.3.8
> filelock==3.13.3
> frozenlist==1.4.1
> fsspec==2024.2.0
> huggingface-hub==0.22.2
> idna==3.6
> multidict==6.0.5
> multiprocess==0.70.16
> numpy==1.26.4
> packaging==24.0
> pandas==2.2.1
> pyarrow==15.0.2
> pyarrow-hotfix==0.6
> python-dateutil==2.9.0.post0
> pytz==2024.1
> PyYAML==6.0.1
> requests==2.31.0
> six==1.16.0
> tqdm==4.66.2
> typing_extensions==4.11.0
> tzdata==2024.1
> urllib3==2.2.1
> xxhash==3.4.1
> yarl==1.9.4
> ```
>
> And the error can be reproduced.
>
> Downgrading to datasets==2.14.6 changes some packages' versions:
>
> ```
> Successfully installed datasets-2.14.6 dill-0.3.7 fsspec-2023.10.0 multiprocess-0.70.15
> ```
>
> and the dataset can be downloaded and loaded.
>
> Then I upgrade the version to 2.18.0 again; now the dataset can be loaded with such a line: `Using the latest cached version of the module from /home/xxx/.cache/huggingface/modules/datasets_modules/datasets/codeparrot--apps/04ac807715d07d6e5cc580f59cdc8213cd7dc4529d0bb819cca72c9f8e8c1aa5 (last modified on Sun Apr 7 09:06:43 2024) since it couldn't be found locally at codeparrot/apps, or remotely on the Hugging Face Hub. `
>
> So the latest version works wrong when requesting the dataset info.
>
> **But if you cannot reproduce this, I may ignore some detailed information: I use `HF_ENDPOINT=https://hf-mirror.com` for some reason (if not use this I cannot connect to huggingface resources) and the error occurs when requesting the dataset's info card.** Maybe the error is caused by this environment variable. I'll open an issue in the author's repo now.

This is useful and my same error is settled!!!

### Describe the bug

This happens with datasets-2.18.0; I downgraded the version to 2.14.6 fixing this temporarily.
```
Traceback (most recent call last):
  File "/home/xxx/miniconda3/envs/py310/lib/python3.10/site-packages/datasets/load.py", line 2556, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/xxx/miniconda3/envs/py310/lib/python3.10/site-packages/datasets/load.py", line 2228, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/home/xxx/miniconda3/envs/py310/lib/python3.10/site-packages/datasets/load.py", line 1879, in dataset_module_factory
    raise e1 from None
  File "/home/xxx/miniconda3/envs/py310/lib/python3.10/site-packages/datasets/load.py", line 1831, in dataset_module_factory
    can_load_config_from_parquet_export = "DEFAULT_CONFIG_NAME" not in f.read()
  File "/home/xxx/miniconda3/envs/py310/lib/python3.10/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
```

### Steps to reproduce the bug

1. Using Python3.10/3.11
2. Install datasets-2.18.0
3. test with
```
from datasets import load_dataset
dataset = load_dataset("codeparrot/apps")
```

### Expected behavior

Normally it should manage to download and load the dataset without such error.

### Environment info

Ubuntu, Python3.10/3.11


Persistent multi-process Pool

New feature or request

### Feature request

Running .map and filter functions with `num_procs` consecutively instantiates several multiprocessing pools iteratively.

As instantiating a Pool is very resource intensive it can be a bottleneck to performing iteratively filtering.

My ideas:
1. There should be an option to declare `persistent_workers` similar to pytorch DataLoader. Downside would be that would be complex to determine the correct resource allocation and deallocation of the pool. i.e. the dataset can outlive the utility of the pool.
2. Provide a pool as an argument. Downside would be the expertise required by the user. Upside, is that there is better resource management.

### Motivation

Is really slow to iteratively perform map and filter operations on a dataset.

### Your contribution

If approved I could integrate it. I would need to know what method would be most suitable to implement from the two options above.


Passing `sample_by` to `load_dataset` when loading text data does not work

Thanks for reporting! We are working on a fix.

### Describe the bug

I have a dataset that consists of a bunch of text files, each representing an example. There is an undocumented `sample_by` argument for the `TextConfig` class that is used by `Text` to decide whether to split files into lines, paragraphs or take them whole. Passing `sample_by=“document”` to `load_dataset` results in files getting split into lines regardless. I have edited `src/datasets/packaged_modules/text/text.py` for myself to switch the default and it works fine.

As a side note, the `if-else` for `sample_by` will silently load an empty dataset if someone makes a typo in the argument, which is not ideal.

### Steps to reproduce the bug

1. Prepare data as a bunch of files in a directory.
2. Load that data via `load_dataset(“text”, data_files=<data_dir>/<files_glob>, …, sample_by=“document”)`.
3. Inspect the resultant dataset — every item should have the form of `{“text”: <a line from a file>}`.

### Expected behavior

`load_dataset(“text”, data_files=<data_dir>/<files_glob>, …, sample_by=“document”)` should result in a dataset with items of the form `{“text”: <one document>}`.

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.15.0-1046-nvidia-x86_64-with-glibc2.35
- Python version: 3.11.8
- `huggingface_hub` version: 0.21.4
- PyArrow version: 15.0.2
- Pandas version: 2.2.1
- `fsspec` version: 2024.2.0


Test disabling transformers containers in docs CI

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6757). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

On slack it was mentioned that it was actually slower for `datasets`, should we close this one or am I missing something ?

@lhoestq I converted to draft. Want to make some more tests and will let you know

Related to https://github.com/huggingface/doc-builder/pull/487 and [internal slack thread](https://huggingface.slack.com/archives/C04F8N7FQNL/p1711384899462349?thread_ts=1711041424.720769&cid=C04F8N7FQNL). There is now a `custom_container` option when building docs in CI. When set to `""` (instead of `"huggingface/transformers-doc-builder"` by default), we don't run the CI inside a container, therefore saving ~2min of download time. The plan is to test disabling the transformers container on a few "big" repo and if everything works correctly, we will stop making it the default container. More details on https://github.com/huggingface/doc-builder/pull/487.

cc @mishig25


Support SQLite files?

New feature or request

You can use `Dataset.from_sql(path_to_sql_file)` already. Though we haven't added the Sql dataset builder to the `_PACKAGED_DATASETS_MODULES` list or in `_EXTENSION_TO_MODULE` to map `.sqlite` to the Sql dataset builder

This would allow to load a dataset repository with a `.sqlite` file using `load_dataset` and enable the Dataset Viewer

Considering `Dataset.from_sql`'s (extremely) low usage, I don't think many users are interested in using this format for their datasets. Also, SQLite files are hard/impossible to stream efficiently and require custom logic to define splits/subsets, so IMO we shouldn't encourage people to use SQLite on the Hub.

@severo Do you have some real-world examples of datasets published in this format?

No. Indeed, it seems better to explicitly not support sqlite

### Feature request

Support loading a dataset from a SQLite file

https://huggingface.co/datasets/severo/test_iris_sqlite/tree/main

### Motivation

SQLite is a popular file format.

### Your contribution

See discussion on slack: https://huggingface.slack.com/archives/C04L6P8KNQ5/p1702481859117909 (internal)

In particular: a SQLite file can contain multiple tables, which might be matched to multiple configs. Maybe the detail of splits and configs should be defined in the README YAML, or use the same format as for ZIP files: `Iris.sqlite::Iris`.

See dataset here: https://huggingface.co/datasets/severo/test_iris_sqlite

Note: should we also support DuckDB files?


Small typo on the documentation

good first issue

Good for newcomers

Thanks for reporting @fostiropoulos! I've edited your comment to fix the link to the problematic line.

### Describe the bug

There is a small typo on https://github.com/huggingface/datasets/blob/d5468836fe94e8be1ae093397dd43d4a2503b926/src/datasets/dataset_dict.py#L938

It should be `caching is enabled`.

### Steps to reproduce the bug

Please visit

https://github.com/huggingface/datasets/blob/d5468836fe94e8be1ae093397dd43d4a2503b926/src/datasets/dataset_dict.py#L938

### Expected behavior

`caching is enabled`

### Environment info

- `datasets` version: 2.17.1
- Platform: Linux-5.15.0-101-generic-x86_64-with-glibc2.35
- Python version: 3.11.7
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.1
- `fsspec` version: 2023.10.0


Fix cache path to snakecase for `CachedDatasetModuleFactory` and `Cache`

@lhoestq hi 😃, is there something else I need to do to check this change?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6754). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @izhx! I have also faced this issue, happy to see it already addressed, looking forward for PR merge :)

@lhoestq What do you think of these tests? 😀

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005060 / 0.011353 (-0.006293) | 0.003251 / 0.011008 (-0.007757) | 0.063538 / 0.038508 (0.025030) | 0.031178 / 0.023109 (0.008069) | 0.249971 / 0.275898 (-0.025927) | 0.284828 / 0.323480 (-0.038652) | 0.004183 / 0.007986 (-0.003802) | 0.002656 / 0.004328 (-0.001673) | 0.049585 / 0.004250 (0.045335) | 0.042656 / 0.037052 (0.005604) | 0.270962 / 0.258489 (0.012473) | 0.296091 / 0.293841 (0.002250) | 0.028065 / 0.128546 (-0.100482) | 0.010545 / 0.075646 (-0.065102) | 0.207323 / 0.419271 (-0.211948) | 0.035977 / 0.043533 (-0.007556) | 0.257315 / 0.255139 (0.002176) | 0.272238 / 0.283200 (-0.010962) | 0.017984 / 0.141683 (-0.123699) | 1.131314 / 1.452155 (-0.320840) | 1.180259 / 1.492716 (-0.312457) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090977 / 0.018006 (0.072971) | 0.284021 / 0.000490 (0.283531) | 0.000264 / 0.000200 (0.000065) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.017852 / 0.037411 (-0.019559) | 0.061288 / 0.014526 (0.046762) | 0.073844 / 0.176557 (-0.102713) | 0.121371 / 0.737135 (-0.615764) | 0.075036 / 0.296338 (-0.221303) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.287599 / 0.215209 (0.072390) | 2.821172 / 2.077655 (0.743517) | 1.459904 / 1.504120 (-0.044216) | 1.340224 / 1.541195 (-0.200970) | 1.357350 / 1.468490 (-0.111140) | 0.557344 / 4.584777 (-4.027433) | 2.412177 / 3.745712 (-1.333535) | 2.745126 / 5.269862 (-2.524735) | 1.754600 / 4.565676 (-2.811077) | 0.062487 / 0.424275 (-0.361788) | 0.005306 / 0.007607 (-0.002301) | 0.338856 / 0.226044 (0.112811) | 3.354953 / 2.268929 (1.086024) | 1.803208 / 55.444624 (-53.641417) | 1.553051 / 6.876477 (-5.323426) | 1.554790 / 2.142072 (-0.587282) | 0.651380 / 4.805227 (-4.153847) | 0.117777 / 6.500664 (-6.382887) | 0.041992 / 0.075469 (-0.033477) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.977588 / 1.841788 (-0.864200) | 11.363058 / 8.074308 (3.288750) | 9.791770 / 10.191392 (-0.399622) | 0.130708 / 0.680424 (-0.549716) | 0.013798 / 0.534201 (-0.520403) | 0.288313 / 0.579283 (-0.290970) | 0.268170 / 0.434364 (-0.166194) | 0.324815 / 0.540337 (-0.215522) | 0.419260 / 1.386936 (-0.967676) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005187 / 0.011353 (-0.006166) | 0.003348 / 0.011008 (-0.007660) | 0.050309 / 0.038508 (0.011801) | 0.031334 / 0.023109 (0.008225) | 0.279542 / 0.275898 (0.003644) | 0.299608 / 0.323480 (-0.023872) | 0.004202 / 0.007986 (-0.003784) | 0.002735 / 0.004328 (-0.001593) | 0.050321 / 0.004250 (0.046070) | 0.039793 / 0.037052 (0.002740) | 0.289972 / 0.258489 (0.031483) | 0.313887 / 0.293841 (0.020046) | 0.028797 / 0.128546 (-0.099750) | 0.010166 / 0.075646 (-0.065480) | 0.059228 / 0.419271 (-0.360044) | 0.032667 / 0.043533 (-0.010866) | 0.278409 / 0.255139 (0.023270) | 0.292208 / 0.283200 (0.009008) | 0.017577 / 0.141683 (-0.124106) | 1.175046 / 1.452155 (-0.277109) | 1.200766 / 1.492716 (-0.291950) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092236 / 0.018006 (0.074230) | 0.298860 / 0.000490 (0.298370) | 0.000211 / 0.000200 (0.000011) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021475 / 0.037411 (-0.015936) | 0.074414 / 0.014526 (0.059888) | 0.087746 / 0.176557 (-0.088811) | 0.124757 / 0.737135 (-0.612378) | 0.088513 / 0.296338 (-0.207826) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.296583 / 0.215209 (0.081374) | 2.894978 / 2.077655 (0.817323) | 1.590806 / 1.504120 (0.086686) | 1.463251 / 1.541195 (-0.077944) | 1.478751 / 1.468490 (0.010261) | 0.571724 / 4.584777 (-4.013053) | 2.454356 / 3.745712 (-1.291356) | 2.789275 / 5.269862 (-2.480586) | 1.753866 / 4.565676 (-2.811811) | 0.064787 / 0.424275 (-0.359488) | 0.005321 / 0.007607 (-0.002287) | 0.348454 / 0.226044 (0.122410) | 3.453052 / 2.268929 (1.184124) | 1.972237 / 55.444624 (-53.472388) | 1.677822 / 6.876477 (-5.198655) | 1.674750 / 2.142072 (-0.467322) | 0.649353 / 4.805227 (-4.155874) | 0.117135 / 6.500664 (-6.383529) | 0.040018 / 0.075469 (-0.035451) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.029812 / 1.841788 (-0.811976) | 11.945063 / 8.074308 (3.870755) | 10.238380 / 10.191392 (0.046988) | 0.146225 / 0.680424 (-0.534199) | 0.015262 / 0.534201 (-0.518939) | 0.286632 / 0.579283 (-0.292651) | 0.272952 / 0.434364 (-0.161412) | 0.323098 / 0.540337 (-0.217239) | 0.423549 / 1.386936 (-0.963387) |

![](https://cml.dev/watermark.png#91b07b90915d7f7313d44ca3ff67673b9ad26bf4 "CML watermark")

Fix https://github.com/huggingface/datasets/issues/6750#issuecomment-2016678729

I didn't find a guideline on how to run the tests, so i just run the following steps to make sure that this bug is fixed.
1. `python test.py`,
2. then `HF_DATASETS_OFFLINE=1 python test.py`

The `test.py` is
```
  import datasets

  datasets.utils.logging.set_verbosity_info()

  ds = datasets.load_dataset('izhx/STS17-debug')
  print(ds)

  ds = datasets.load_dataset('C-MTEB/AFQMC', revision='b44c3b011063adb25877c13823db83bb193913c4')
  print(ds)
```


Type error when importing datasets on Kaggle

I have the same problem
It seems that it only appears when you are using GPU
It seems to work fine with the 2.17 version though

> I have the same problem
> It seems that it only appears when you are using GPU
> It seems to work fine with the 2.17 version though

I downgraded from 2.18 to 2.17, and it works with CPU/GPU .. except now pyarrow complains

```
...
File /opt/conda/lib/python3.10/site-packages/pyarrow/array.pxi:830, in pyarrow.lib._PandasConvertible.to_pandas()

File /opt/conda/lib/python3.10/site-packages/pyarrow/table.pxi:3989, in pyarrow.lib.Table._to_pandas()

ImportError: cannot import name table_to_blockmanager
```

see also https://www.kaggle.com/competitions/pii-detection-removal-from-educational-data/discussion/487474#2722594

Solved for me by downgrading `!pip install -U datasets==2.16.0` Works with gpu aswell

I think you should remain open this issue. It works at the previous version but not the latter versions. It is possible as a bug that the maintainer could take note for.

> Solved for me by downgrading `!pip install -U datasets==2.16.0` Works with gpu as well

Verified it's working w/ GPU if I make these 3 updates.

```
datasets==2.16.0
fsspec==2023.10.0
gcsfs==2023.10.0
```

but the issue shouldn't be closed, this is just a workaround until they get the issue with 2.18.0 resolved.

See also: https://www.kaggle.com/competitions/pii-detection-removal-from-educational-data/discussion/487474

> > Solved for me by downgrading `!pip install -U datasets==2.16.0` Works with gpu as well
>
> Verified it's working w/ GPU if I make these 3 updates.
>
> ```
> datasets==2.16.0
> fsspec==2023.10.0
> gcsfs==2023.10.0
> ```
>
> but the issue shouldn't be closed, this is just a workaround until they get the issue with 2.18.0 resolved.
>
> See also: https://www.kaggle.com/competitions/pii-detection-removal-from-educational-data/discussion/487474

This also works for me, thanks

### Describe the bug

When trying to run
```
import datasets
print(datasets.__version__)

```
It generates the following error

```
TypeError: expected string or bytes-like object
```

It looks like It cannot find the valid versions of `fsspec`

though fsspec version is fine when I checked Via command

```
import fsspec
print(fsspec.__version__)

# output: 2024.3.1
```

Detailed crash report
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[1], line 1
----> 1 import datasets
      2 print(datasets.__version__)

File /opt/conda/lib/python3.10/site-packages/datasets/__init__.py:18
      1 # ruff: noqa
      2 # Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors.
      3 #
   (...)
     13 # See the License for the specific language governing permissions and
     14 # limitations under the License.
     16 __version__ = "2.18.0"
---> 18 from .arrow_dataset import Dataset
     19 from .arrow_reader import ReadInstruction
     20 from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder

File /opt/conda/lib/python3.10/site-packages/datasets/arrow_dataset.py:66
     63 from multiprocess import Pool
     64 from tqdm.contrib.concurrent import thread_map
---> 66 from . import config
     67 from .arrow_reader import ArrowReader
     68 from .arrow_writer import ArrowWriter, OptimizedTypedSequence

File /opt/conda/lib/python3.10/site-packages/datasets/config.py:41
     39 # Imports
     40 DILL_VERSION = version.parse(importlib.metadata.version("dill"))
---> 41 FSSPEC_VERSION = version.parse(importlib.metadata.version("fsspec"))
     42 PANDAS_VERSION = version.parse(importlib.metadata.version("pandas"))
     43 PYARROW_VERSION = version.parse(importlib.metadata.version("pyarrow"))

File /opt/conda/lib/python3.10/site-packages/packaging/version.py:49, in parse(version)
     43 """
     44 Parse the given version string and return either a :class:`Version` object
     45 or a :class:`LegacyVersion` object depending on if the given version is
     46 a valid PEP 440 version or a legacy version.
     47 """
     48 try:
---> 49     return Version(version)
     50 except InvalidVersion:
     51     return LegacyVersion(version)

File /opt/conda/lib/python3.10/site-packages/packaging/version.py:264, in Version.__init__(self, version)
    261 def __init__(self, version: str) -> None:
    262
    263     # Validate the version and parse it into pieces
--> 264     match = self._regex.search(version)
    265     if not match:
    266         raise InvalidVersion(f"Invalid version: '{version}'")

TypeError: expected string or bytes-like object
```

### Steps to reproduce the bug

1. run `!pip install -U datasets` on kaggle
2. check datasets is installed via
```
import datasets
print(datasets.__version__)
```

### Expected behavior

Expected to print datasets version, like `2.18.0`

### Environment info

Running on Kaggle, latest enviornment , here is the notebook https://www.kaggle.com/code/jtv199/mistrial-7b-part2


Precision being changed from float16 to float32 unexpectedly

This is because of the formatter (`torch` in this case).
It defaults to `float32`.

You can load it in `float16` using `dataset.set_format("torch", dtype=torch.float16)`.

### Describe the bug

I'm loading a HuggingFace Dataset for images.

I'm running a preprocessing (map operation) step that runs a few operations, one of them being conversion to float16.  The Dataset features also say that the 'img' is of type float16.  Whenever I take an image from that HuggingFace Dataset instance, the type turns out to be float32.

### Steps to reproduce the bug

```python
import torchvision.transforms.v2 as transforms
from datasets import load_dataset

dataset = load_dataset('cifar10', split='test')
dataset = dataset.with_format("torch")

data_transform = transforms.Compose([transforms.Resize((32, 32)),
                                     transforms.ToDtype(torch.float16, scale=True),
                                     transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5]),
                                     ])
def _preprocess(examples):
    # Permutes from (BS x H x W x C) to (BS x C x H x W)
    images = torch.permute(examples['img'], (0, 3, 2, 1))
    examples['img'] = data_transform(images)
    return examples

dataset = dataset.map(_preprocess, batched=True, batch_size=8)
```

Now at this point the dataset.features are showing float16 which is great because that's what I want.

```python
print(data_loader.features['img'])

Sequence(feature=Sequence(feature=Sequence(feature=Value(dtype='float16', id=None), length=-1, id=None), length=-1, id=None), length=-1, id=None)
```

But when I try to sample an image from this dataloader; I'm getting a float32 image, when I'm expecting float16:
```python
print(next(iter(data_loader))['img'].dtype)

torch.float32
```

### Expected behavior

I'm expecting the images loaded after the transformation to stay in float16.

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.15.146.1-microsoft-standard-WSL2-x86_64-with-glibc2.31
- Python version: 3.10.9
- `huggingface_hub` version: 0.21.4
- PyArrow version: 14.0.2
- Pandas version: 2.0.3
- `fsspec` version: 2023.10.0


Use 'with' operator for some download functions

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6751). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I was mistaken on the intent of those functions, closing the PR.

Some functions in `streaming_download_manager.py` are not closing the file they open which lead to `Unclosed file` warnings in our code. This fixes a few of them.


`load_dataset` requires a network connection for local download?

> Are you using `HF_DATASETS_OFFLINE=1` ?

This doesn't work for me. `datasets=2.18.0`

`test.py`:
```
import datasets

datasets.utils.logging.set_verbosity_info()

ds = datasets.load_dataset('C-MTEB/AFQMC', revision='b44c3b011063adb25877c13823db83bb193913c4')

print(ds)
```

run `python test.py`
```
Generating dataset afqmc (/home/data/.cache/huggingface/datasets/C-MTEB___afqmc/default/0.0.0/b44c3b011063adb25877c13823db83bb193913c4)
Downloading and preparing dataset afqmc/default to /home/data/.cache/huggingface/datasets/C-MTEB___afqmc/default/0.0.0/b44c3b011063adb25877c13823db83bb193913c4...
Dataset not on Hf google storage. Downloading and preparing it from source
hf://datasets/C-MTEB/AFQMC@b44c3b011063adb25877c13823db83bb193913c4/data/validation-00000-of-00001-b8fc393b5ddedac7.parquet not found in cache or force_download set to True, downloading to /home/data/.cache/huggingface/datasets/downloads/78949f93104662359f4f3d5a2f7ec1ae37af5a5af44420a51212ea08c0be966b.incomplete
Downloading data: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 240k/240k [00:01<00:00, 178kB/s]
storing hf://datasets/C-MTEB/AFQMC@b44c3b011063adb25877c13823db83bb193913c4/data/validation-00000-of-00001-b8fc393b5ddedac7.parquet in cache at /home/data/.cache/huggingface/datasets/downloads/78949f93104662359f4f3d5a2f7ec1ae37af5a5af44420a51212ea08c0be966b
creating metadata file for /home/data/.cache/huggingface/datasets/downloads/78949f93104662359f4f3d5a2f7ec1ae37af5a5af44420a51212ea08c0be966b
Downloading took 0.0 min
Checksum Computation took 0.0 min
Generating test split
Generating test split: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3861/3861 [00:00<00:00, 3972.00 examples/s]
Generating train split
Generating train split: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 34334/34334 [00:00<00:00, 34355.50 examples/s]
Generating validation split
Generating validation split: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4316/4316 [00:00<00:00, 4477.00 examples/s]
All the splits matched successfully.
Dataset afqmc downloaded and prepared to /home/data/.cache/huggingface/datasets/C-MTEB___afqmc/default/0.0.0/b44c3b011063adb25877c13823db83bb193913c4. Subsequent calls will reuse this data.
DatasetDict({
    test: Dataset({
        features: ['sentence1', 'sentence2', 'score', 'idx'],
        num_rows: 3861
    })
    train: Dataset({
        features: ['sentence1', 'sentence2', 'score', 'idx'],
        num_rows: 34334
    })
    validation: Dataset({
        features: ['sentence1', 'sentence2', 'score', 'idx'],
        num_rows: 4316
    })
})
```

Then run `HF_DATASETS_OFFLINE=1 python test.py`
```
Traceback (most recent call last):
  File "test.py", line 9, in <module>
    ds = datasets.load_dataset('C-MTEB/AFQMC', revision='b44c3b011063adb25877c13823db83bb193913c4')
  File "/dev/shm/tmp_env/lib/python3.10/site-packages/datasets/load.py", line 2556, in load_dataset
    builder_instance = load_dataset_builder(
  File "/dev/shm/tmp_env/lib/python3.10/site-packages/datasets/load.py", line 2228, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/dev/shm/tmp_env/lib/python3.10/site-packages/datasets/load.py", line 1871, in dataset_module_factory
    raise ConnectionError(f"Couldn't reach the Hugging Face Hub for dataset '{path}': {e1}") from None
ConnectionError: Couldn't reach the Hugging Face Hub for dataset 'C-MTEB/AFQMC': Offline mode is enabled.
```

I was having similar inexplicable issues.

Doing this I *think* helped, but, `datasets` still *clearly* does not want to respect the cache:

```python
pip install --upgrade datasets  # now it is 2.18.0
HF_DATASETS_OFFLINE="1" python blah.py
```

Or similarly, I must spacify that env var to resuse the cache, IE, no arg to `load_dataset` helps it reuse the cache:

```python

import os
os.environ["HF_DATASETS_OFFLINE"] = "1"

import logging
logging.basicConfig(level=logging.DEBUG)

import datasets
# >>> datasets.__version__
# '2.18.0'

datasets.utils.logging.set_verbosity_info()
data = datasets.load_dataset("c-s-ale/dolly-15k-instruction-alpaca-format")
```

### Describe the bug

Hi all - I see that in the past a network dependency has been mistakenly introduced into `load_dataset` even for local loads. Is it possible this has happened again?

### Steps to reproduce the bug

```
>>> import datasets
>>> datasets.load_dataset("hh-rlhf")
Repo card metadata block was not found. Setting CardData to empty.
*hangs bc i'm firewalled*
````
stack trace from ctrl-c:
```
^CTraceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jobuser/.local/lib/python3.10/site-packages/datasets/load.py", line 2582, in load_dataset
    builder_instance.download_and_prepare(
    output_path = get_from_cache(                                                                                [0/122]
  File "/home/jobuser/.local/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 532, in get_from_cache
    response = http_head(
  File "/home/jobuser/.local/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 419, in http_head
    response = _request_with_retry(
  File "/home/jobuser/.local/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 304, in _request_with_retry
    response = requests.request(method=method.upper(), url=url, timeout=timeout, **params)
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/requests/adapters.py", line 487, in send
    resp = conn.urlopen(
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/urllib3/connection.py", line 363, in connect
    self.sock = conn = self._new_conn()
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
KeyboardInterrupt
```

### Expected behavior

loads the dataset

### Environment info

```
> pip show datasets
Name: datasets
Version: 2.18.0
```

Python 3.10.2


Fix fsspec tqdm callback

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6749). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005017 / 0.011353 (-0.006336) | 0.002958 / 0.011008 (-0.008050) | 0.063455 / 0.038508 (0.024946) | 0.028206 / 0.023109 (0.005096) | 0.230884 / 0.275898 (-0.045014) | 0.252688 / 0.323480 (-0.070792) | 0.002995 / 0.007986 (-0.004991) | 0.002613 / 0.004328 (-0.001716) | 0.046477 / 0.004250 (0.042226) | 0.040662 / 0.037052 (0.003609) | 0.241824 / 0.258489 (-0.016665) | 0.269063 / 0.293841 (-0.024778) | 0.027336 / 0.128546 (-0.101210) | 0.010614 / 0.075646 (-0.065032) | 0.216087 / 0.419271 (-0.203184) | 0.035667 / 0.043533 (-0.007866) | 0.238657 / 0.255139 (-0.016482) | 0.253433 / 0.283200 (-0.029767) | 0.017433 / 0.141683 (-0.124250) | 1.120856 / 1.452155 (-0.331299) | 1.157415 / 1.492716 (-0.335302) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.088028 / 0.018006 (0.070022) | 0.277368 / 0.000490 (0.276878) | 0.000204 / 0.000200 (0.000004) | 0.000049 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.017956 / 0.037411 (-0.019455) | 0.061061 / 0.014526 (0.046535) | 0.073323 / 0.176557 (-0.103234) | 0.119254 / 0.737135 (-0.617881) | 0.074308 / 0.296338 (-0.222031) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.285118 / 0.215209 (0.069908) | 2.785796 / 2.077655 (0.708142) | 1.476436 / 1.504120 (-0.027684) | 1.356505 / 1.541195 (-0.184690) | 1.362505 / 1.468490 (-0.105985) | 0.554064 / 4.584777 (-4.030713) | 2.395774 / 3.745712 (-1.349938) | 2.713703 / 5.269862 (-2.556159) | 1.701020 / 4.565676 (-2.864657) | 0.062370 / 0.424275 (-0.361905) | 0.004944 / 0.007607 (-0.002663) | 0.327948 / 0.226044 (0.101904) | 3.243739 / 2.268929 (0.974811) | 1.803881 / 55.444624 (-53.640743) | 1.551635 / 6.876477 (-5.324841) | 1.560627 / 2.142072 (-0.581446) | 0.628187 / 4.805227 (-4.177040) | 0.115824 / 6.500664 (-6.384840) | 0.041655 / 0.075469 (-0.033814) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.968797 / 1.841788 (-0.872991) | 11.220905 / 8.074308 (3.146597) | 9.322584 / 10.191392 (-0.868808) | 0.139629 / 0.680424 (-0.540795) | 0.013823 / 0.534201 (-0.520378) | 0.286700 / 0.579283 (-0.292583) | 0.263517 / 0.434364 (-0.170847) | 0.341264 / 0.540337 (-0.199074) | 0.418834 / 1.386936 (-0.968102) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005404 / 0.011353 (-0.005949) | 0.003630 / 0.011008 (-0.007378) | 0.048977 / 0.038508 (0.010469) | 0.029980 / 0.023109 (0.006871) | 0.274671 / 0.275898 (-0.001227) | 0.295671 / 0.323480 (-0.027808) | 0.004230 / 0.007986 (-0.003756) | 0.002656 / 0.004328 (-0.001672) | 0.048603 / 0.004250 (0.044353) | 0.044323 / 0.037052 (0.007271) | 0.286499 / 0.258489 (0.028010) | 0.313199 / 0.293841 (0.019358) | 0.030079 / 0.128546 (-0.098468) | 0.010480 / 0.075646 (-0.065166) | 0.058226 / 0.419271 (-0.361045) | 0.054920 / 0.043533 (0.011387) | 0.274921 / 0.255139 (0.019783) | 0.296559 / 0.283200 (0.013360) | 0.019164 / 0.141683 (-0.122519) | 1.154703 / 1.452155 (-0.297452) | 1.207015 / 1.492716 (-0.285701) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.089368 / 0.018006 (0.071362) | 0.301196 / 0.000490 (0.300706) | 0.000208 / 0.000200 (0.000008) | 0.000047 / 0.000054 (-0.000008) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021355 / 0.037411 (-0.016056) | 0.074688 / 0.014526 (0.060162) | 0.085840 / 0.176557 (-0.090716) | 0.125784 / 0.737135 (-0.611351) | 0.087103 / 0.296338 (-0.209235) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.296727 / 0.215209 (0.081518) | 2.884922 / 2.077655 (0.807267) | 1.586515 / 1.504120 (0.082395) | 1.474417 / 1.541195 (-0.066777) | 1.492105 / 1.468490 (0.023615) | 0.570016 / 4.584777 (-4.014761) | 2.435760 / 3.745712 (-1.309952) | 2.657999 / 5.269862 (-2.611863) | 1.740160 / 4.565676 (-2.825516) | 0.063743 / 0.424275 (-0.360532) | 0.005048 / 0.007607 (-0.002559) | 0.341279 / 0.226044 (0.115235) | 3.396185 / 2.268929 (1.127256) | 1.952825 / 55.444624 (-53.491800) | 1.676669 / 6.876477 (-5.199808) | 1.773158 / 2.142072 (-0.368915) | 0.650664 / 4.805227 (-4.154563) | 0.116815 / 6.500664 (-6.383849) | 0.040813 / 0.075469 (-0.034656) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.999836 / 1.841788 (-0.841952) | 11.854540 / 8.074308 (3.780232) | 10.245516 / 10.191392 (0.054124) | 0.141235 / 0.680424 (-0.539189) | 0.015562 / 0.534201 (-0.518639) | 0.287556 / 0.579283 (-0.291727) | 0.274946 / 0.434364 (-0.159418) | 0.324652 / 0.540337 (-0.215685) | 0.449204 / 1.386936 (-0.937733) |

![](https://cml.dev/watermark.png#ed2b406d045349dad16738985c947fe743260710 "CML watermark")

Following changes at https://github.com/fsspec/filesystem_spec/pull/1497 for `fsspec>=2024.2.0`


Strange slicing behavior

As explained in the [docs](https://huggingface.co/docs/datasets/v2.18.0/en/access#slicing), slicing a `Dataset` returns a dictionary that maps its column names to their values. So, `len(dataset[:300])=2` is expected, assuming your dataset has 2 columns (the returned dict has 2 keys, but each value in the dict has 300 items).
`

### Describe the bug

I have loaded a dataset, and then slice first 300 samples using `:` ops, however, the resulting dataset is not expected, as the output below:

```bash
len(dataset)=1050324
len(dataset[:300])=2
len(dataset[0:300])=2
len(dataset.select(range(300)))=300
```

### Steps to reproduce the bug

load a dataset then:
```bash
    dataset = load_from_disk(args.train_data_dir)

    print(f"{len(dataset)=}", flush=True)
    print(f"{len(dataset[:300])=}", flush=True)
    print(f"{len(dataset[0:300])=}", flush=True)
    print(f"{len(dataset.select(range(300)))=}", flush=True)
```

### Expected behavior

```bash
len(dataset)=1050324
len(dataset[:300])=300
len(dataset[0:300])=300
len(dataset.select(range(300)))=300
```

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.35
- Python version: 3.10.11
- `huggingface_hub` version: 0.20.2
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
- `fsspec` version: 2023.10.0


chore(deps): bump fsspec

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6747). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005129 / 0.011353 (-0.006224) | 0.003788 / 0.011008 (-0.007220) | 0.063456 / 0.038508 (0.024948) | 0.029079 / 0.023109 (0.005969) | 0.237228 / 0.275898 (-0.038670) | 0.260554 / 0.323480 (-0.062926) | 0.003090 / 0.007986 (-0.004895) | 0.002730 / 0.004328 (-0.001599) | 0.049040 / 0.004250 (0.044789) | 0.042432 / 0.037052 (0.005380) | 0.256954 / 0.258489 (-0.001535) | 0.285912 / 0.293841 (-0.007929) | 0.027568 / 0.128546 (-0.100978) | 0.010402 / 0.075646 (-0.065245) | 0.206773 / 0.419271 (-0.212499) | 0.035381 / 0.043533 (-0.008152) | 0.243147 / 0.255139 (-0.011992) | 0.259419 / 0.283200 (-0.023781) | 0.019503 / 0.141683 (-0.122180) | 1.145537 / 1.452155 (-0.306618) | 1.204070 / 1.492716 (-0.288646) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092298 / 0.018006 (0.074291) | 0.300042 / 0.000490 (0.299553) | 0.000236 / 0.000200 (0.000036) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018624 / 0.037411 (-0.018788) | 0.063832 / 0.014526 (0.049306) | 0.075849 / 0.176557 (-0.100707) | 0.120919 / 0.737135 (-0.616216) | 0.075878 / 0.296338 (-0.220461) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.275545 / 0.215209 (0.060336) | 2.706004 / 2.077655 (0.628349) | 1.406398 / 1.504120 (-0.097722) | 1.287154 / 1.541195 (-0.254041) | 1.298278 / 1.468490 (-0.170212) | 0.559763 / 4.584777 (-4.025014) | 2.434104 / 3.745712 (-1.311608) | 2.786338 / 5.269862 (-2.483523) | 1.720951 / 4.565676 (-2.844726) | 0.062082 / 0.424275 (-0.362193) | 0.004931 / 0.007607 (-0.002676) | 0.329998 / 0.226044 (0.103954) | 3.222105 / 2.268929 (0.953176) | 1.777539 / 55.444624 (-53.667085) | 1.533845 / 6.876477 (-5.342632) | 1.520357 / 2.142072 (-0.621715) | 0.638850 / 4.805227 (-4.166377) | 0.116718 / 6.500664 (-6.383946) | 0.042215 / 0.075469 (-0.033254) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.962791 / 1.841788 (-0.878997) | 11.509889 / 8.074308 (3.435581) | 9.507676 / 10.191392 (-0.683716) | 0.140780 / 0.680424 (-0.539644) | 0.014187 / 0.534201 (-0.520014) | 0.286363 / 0.579283 (-0.292920) | 0.263316 / 0.434364 (-0.171048) | 0.322099 / 0.540337 (-0.218239) | 0.415602 / 1.386936 (-0.971334) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005175 / 0.011353 (-0.006178) | 0.003631 / 0.011008 (-0.007377) | 0.050277 / 0.038508 (0.011769) | 0.031879 / 0.023109 (0.008770) | 0.269966 / 0.275898 (-0.005933) | 0.297229 / 0.323480 (-0.026251) | 0.004278 / 0.007986 (-0.003707) | 0.002936 / 0.004328 (-0.001393) | 0.048686 / 0.004250 (0.044436) | 0.044262 / 0.037052 (0.007209) | 0.284578 / 0.258489 (0.026089) | 0.313681 / 0.293841 (0.019840) | 0.029064 / 0.128546 (-0.099482) | 0.010700 / 0.075646 (-0.064946) | 0.058366 / 0.419271 (-0.360905) | 0.051341 / 0.043533 (0.007809) | 0.271262 / 0.255139 (0.016123) | 0.290791 / 0.283200 (0.007591) | 0.019044 / 0.141683 (-0.122639) | 1.149514 / 1.452155 (-0.302641) | 1.209277 / 1.492716 (-0.283439) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094879 / 0.018006 (0.076872) | 0.302196 / 0.000490 (0.301707) | 0.000217 / 0.000200 (0.000018) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021715 / 0.037411 (-0.015696) | 0.075122 / 0.014526 (0.060596) | 0.087393 / 0.176557 (-0.089164) | 0.125583 / 0.737135 (-0.611553) | 0.088722 / 0.296338 (-0.207617) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295158 / 0.215209 (0.079949) | 2.930208 / 2.077655 (0.852553) | 1.590197 / 1.504120 (0.086077) | 1.459038 / 1.541195 (-0.082156) | 1.471690 / 1.468490 (0.003200) | 0.570279 / 4.584777 (-4.014498) | 2.456971 / 3.745712 (-1.288741) | 2.675315 / 5.269862 (-2.594547) | 1.750122 / 4.565676 (-2.815554) | 0.062905 / 0.424275 (-0.361370) | 0.005118 / 0.007607 (-0.002489) | 0.344263 / 0.226044 (0.118219) | 3.472460 / 2.268929 (1.203532) | 1.931707 / 55.444624 (-53.512917) | 1.658537 / 6.876477 (-5.217939) | 1.785794 / 2.142072 (-0.356278) | 0.637149 / 4.805227 (-4.168078) | 0.115838 / 6.500664 (-6.384826) | 0.040771 / 0.075469 (-0.034698) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.002869 / 1.841788 (-0.838919) | 12.048825 / 8.074308 (3.974517) | 10.407979 / 10.191392 (0.216587) | 0.150300 / 0.680424 (-0.530124) | 0.015299 / 0.534201 (-0.518902) | 0.286277 / 0.579283 (-0.293006) | 0.312186 / 0.434364 (-0.122178) | 0.322633 / 0.540337 (-0.217704) | 0.438431 / 1.386936 (-0.948505) |

![](https://cml.dev/watermark.png#d5468836fe94e8be1ae093397dd43d4a2503b926 "CML watermark")

There were a few fixes released recently, some DVC ecosystem packages require newer version of `fsspec`.


ExpectedMoreSplits error when loading C4 dataset

Hi ! We updated the `allenai/c4` repository to allow people to specify which language to load easily (the the [c4 dataset page](https://huggingface.co/datasets/allenai/c4))

To fix this issue **you can update** `datasets` and remove the mention of the legacy configuration name "allenai--c4":

```python
traindata = load_dataset('allenai/c4', data_files={'train': 'en/c4-train.00000-of-01024.json.gz'}, split='train')
valdata = load_dataset('allenai/c4', data_files={'validation': 'en/c4-validation.00000-of-00008.json.gz'}, split='validation')
```

Did you solve this problem？I have the same bug.It is no use to delete "allenai--c4".

Did you solve it? I met this problem too.

But after I romove allenai--c4,it still fails

For me it works this way. I'm using datasets version 2.17.0

### Describe the bug

I encounter bug when running the example command line

```python
    python main.py \
    --model decapoda-research/llama-7b-hf \
    --prune_method wanda \
    --sparsity_ratio 0.5 \
    --sparsity_type unstructured \
    --save out/llama_7b/unstructured/wanda/
```

The bug occurred at these lines of code (when loading c4 dataset)

```python
traindata = load_dataset('allenai/c4', 'allenai--c4', data_files={'train': 'en/c4-train.00000-of-01024.json.gz'}, split='train')
valdata = load_dataset('allenai/c4', 'allenai--c4', data_files={'validation': 'en/c4-validation.00000-of-00008.json.gz'}, split='validation')
```

The error message states:
```
raise ExpectedMoreSplits(str(set(expected_splits) - set(recorded_splits)))
datasets.utils.info_utils.ExpectedMoreSplits: {'validation'}
```

### Steps to reproduce the bug

1. I encounter bug when running the example command line

### Expected behavior

The error message states:
```
raise ExpectedMoreSplits(str(set(expected_splits) - set(recorded_splits)))
datasets.utils.info_utils.ExpectedMoreSplits: {'validation'}
```

### Environment info

I'm using cuda 12.4, so I use ```pip install pytorch``` instead of conda provided in install.md

Also, I've tried another environment using the same commands in install.md, but the same bug occured


Scraping the whole of github including private repos is bad; kindly stop

enhancement

New feature or request

### Feature request

https://github.com/bigcode-project/opt-out-v2 - opt out is not consent. kindly quit this ridiculous nonsense.

### Motivation

[EDITED: insults not tolerated]

### Your contribution

[EDITED: insults not tolerated]


Option to disable file locking

New feature or request

### Feature request

Commands such as `load_dataset` creates file locks with `filelock.FileLock`. It would be good if there was a way to disable this.

### Motivation

File locking doesn't work on all file-systems (in my case NFS mounted Weka). If the `cache_dir` only had small files then it would be possible to point to local disk and the problem would be solved. However, as cache_dir is both where the small info files are written and the processed datasets are put this isn't a feasible solution.

Considering https://github.com/huggingface/datasets/issues/6395 I still do think this is something that belongs in HuggingFace. The possibility to control packages separately is valuable. It might be that a user has their dataset on a file-system that doesn't support file-locking while they are using file locking on local disk to control some other type of access.

### Your contribution

My suggested solution:
```
diff --git a/src/datasets/utils/_filelock.py b/src/datasets/utils/_filelock.py
index 19620e6e..58f41a02 100644
--- a/src/datasets/utils/_filelock.py
+++ b/src/datasets/utils/_filelock.py
@@ -18,11 +18,15 @@
 import os

 from filelock import FileLock as FileLock_
-from filelock import UnixFileLock
+from filelock import SoftFileLock, UnixFileLock
 from filelock import __version__ as _filelock_version
 from packaging import version

+if os.getenv('HF_USE_SOFTFILELOCK', 'false').lower() in ('true', '1'):
+    FileLock_ = SoftFileLock
+
+
 class FileLock(FileLock_):
     """
     A `filelock.FileLock` initializer that handles long paths.
```


Allow null values in dict columns

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6743). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005013 / 0.011353 (-0.006340) | 0.003228 / 0.011008 (-0.007780) | 0.062763 / 0.038508 (0.024255) | 0.028937 / 0.023109 (0.005828) | 0.240777 / 0.275898 (-0.035121) | 0.266972 / 0.323480 (-0.056508) | 0.003073 / 0.007986 (-0.004913) | 0.002769 / 0.004328 (-0.001560) | 0.049265 / 0.004250 (0.045015) | 0.042061 / 0.037052 (0.005009) | 0.261714 / 0.258489 (0.003225) | 0.284896 / 0.293841 (-0.008944) | 0.027717 / 0.128546 (-0.100829) | 0.010430 / 0.075646 (-0.065216) | 0.209022 / 0.419271 (-0.210249) | 0.035941 / 0.043533 (-0.007591) | 0.246849 / 0.255139 (-0.008290) | 0.263205 / 0.283200 (-0.019994) | 0.019489 / 0.141683 (-0.122193) | 1.102595 / 1.452155 (-0.349559) | 1.170493 / 1.492716 (-0.322223) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093611 / 0.018006 (0.075604) | 0.302041 / 0.000490 (0.301551) | 0.000223 / 0.000200 (0.000023) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018720 / 0.037411 (-0.018692) | 0.062199 / 0.014526 (0.047673) | 0.074888 / 0.176557 (-0.101669) | 0.120184 / 0.737135 (-0.616951) | 0.076756 / 0.296338 (-0.219583) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.287484 / 0.215209 (0.072275) | 2.787777 / 2.077655 (0.710123) | 1.488957 / 1.504120 (-0.015163) | 1.362678 / 1.541195 (-0.178517) | 1.364571 / 1.468490 (-0.103919) | 0.563139 / 4.584777 (-4.021638) | 2.422224 / 3.745712 (-1.323488) | 2.798011 / 5.269862 (-2.471850) | 1.751159 / 4.565676 (-2.814517) | 0.062740 / 0.424275 (-0.361536) | 0.004918 / 0.007607 (-0.002689) | 0.338285 / 0.226044 (0.112240) | 3.316012 / 2.268929 (1.047083) | 1.845975 / 55.444624 (-53.598650) | 1.553187 / 6.876477 (-5.323290) | 1.564582 / 2.142072 (-0.577490) | 0.645987 / 4.805227 (-4.159240) | 0.118216 / 6.500664 (-6.382448) | 0.041243 / 0.075469 (-0.034226) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.970265 / 1.841788 (-0.871522) | 11.783152 / 8.074308 (3.708844) | 9.516584 / 10.191392 (-0.674808) | 0.148086 / 0.680424 (-0.532338) | 0.013689 / 0.534201 (-0.520512) | 0.289657 / 0.579283 (-0.289626) | 0.265966 / 0.434364 (-0.168398) | 0.328483 / 0.540337 (-0.211854) | 0.433544 / 1.386936 (-0.953392) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005235 / 0.011353 (-0.006118) | 0.003515 / 0.011008 (-0.007493) | 0.049484 / 0.038508 (0.010976) | 0.029264 / 0.023109 (0.006154) | 0.278518 / 0.275898 (0.002620) | 0.298948 / 0.323480 (-0.024532) | 0.004308 / 0.007986 (-0.003678) | 0.002751 / 0.004328 (-0.001577) | 0.048952 / 0.004250 (0.044701) | 0.045379 / 0.037052 (0.008327) | 0.292633 / 0.258489 (0.034144) | 0.319405 / 0.293841 (0.025564) | 0.030201 / 0.128546 (-0.098345) | 0.010657 / 0.075646 (-0.064990) | 0.057842 / 0.419271 (-0.361430) | 0.053359 / 0.043533 (0.009826) | 0.281136 / 0.255139 (0.025997) | 0.295388 / 0.283200 (0.012188) | 0.018786 / 0.141683 (-0.122897) | 1.187181 / 1.452155 (-0.264974) | 1.198394 / 1.492716 (-0.294323) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093861 / 0.018006 (0.075855) | 0.304019 / 0.000490 (0.303529) | 0.000220 / 0.000200 (0.000020) | 0.000053 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021582 / 0.037411 (-0.015829) | 0.075381 / 0.014526 (0.060855) | 0.087886 / 0.176557 (-0.088671) | 0.125078 / 0.737135 (-0.612057) | 0.089339 / 0.296338 (-0.206999) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295797 / 0.215209 (0.080588) | 2.912021 / 2.077655 (0.834367) | 1.592191 / 1.504120 (0.088071) | 1.471270 / 1.541195 (-0.069925) | 1.475535 / 1.468490 (0.007045) | 0.564114 / 4.584777 (-4.020663) | 2.442882 / 3.745712 (-1.302830) | 2.679433 / 5.269862 (-2.590428) | 1.752097 / 4.565676 (-2.813579) | 0.062748 / 0.424275 (-0.361527) | 0.005068 / 0.007607 (-0.002539) | 0.345554 / 0.226044 (0.119509) | 3.456929 / 2.268929 (1.188000) | 1.962781 / 55.444624 (-53.481844) | 1.688313 / 6.876477 (-5.188164) | 1.817392 / 2.142072 (-0.324681) | 0.639588 / 4.805227 (-4.165639) | 0.116148 / 6.500664 (-6.384516) | 0.040851 / 0.075469 (-0.034618) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.009852 / 1.841788 (-0.831936) | 12.031749 / 8.074308 (3.957440) | 10.305107 / 10.191392 (0.113715) | 0.132960 / 0.680424 (-0.547464) | 0.014779 / 0.534201 (-0.519422) | 0.288903 / 0.579283 (-0.290381) | 0.275417 / 0.434364 (-0.158947) | 0.322628 / 0.540337 (-0.217709) | 0.445060 / 1.386936 (-0.941876) |

![](https://cml.dev/watermark.png#f234fce40d5ffc96fac5198d8cc89817970d87ee "CML watermark")

notify https://huggingface.co/datasets/chaoyi-wu/PMC-Inline/discussions/1 once it's merged in dataset-viewer

Fix #6738