fix: show correct package name to install biopython

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6662). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005135 / 0.011353 (-0.006218) | 0.003666 / 0.011008 (-0.007342) | 0.062660 / 0.038508 (0.024152) | 0.028656 / 0.023109 (0.005546) | 0.249601 / 0.275898 (-0.026297) | 0.265745 / 0.323480 (-0.057735) | 0.002935 / 0.007986 (-0.005051) | 0.002606 / 0.004328 (-0.001723) | 0.048774 / 0.004250 (0.044523) | 0.043643 / 0.037052 (0.006591) | 0.263114 / 0.258489 (0.004625) | 0.284596 / 0.293841 (-0.009245) | 0.027818 / 0.128546 (-0.100728) | 0.010726 / 0.075646 (-0.064921) | 0.205900 / 0.419271 (-0.213371) | 0.035646 / 0.043533 (-0.007887) | 0.245599 / 0.255139 (-0.009540) | 0.267706 / 0.283200 (-0.015493) | 0.018441 / 0.141683 (-0.123242) | 1.143365 / 1.452155 (-0.308790) | 1.191823 / 1.492716 (-0.300893) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.089703 / 0.018006 (0.071696) | 0.298073 / 0.000490 (0.297583) | 0.000209 / 0.000200 (0.000009) | 0.000042 / 0.000054 (-0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018068 / 0.037411 (-0.019343) | 0.061416 / 0.014526 (0.046890) | 0.075989 / 0.176557 (-0.100567) | 0.120765 / 0.737135 (-0.616370) | 0.075476 / 0.296338 (-0.220863) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.284043 / 0.215209 (0.068834) | 2.770282 / 2.077655 (0.692627) | 1.473040 / 1.504120 (-0.031080) | 1.349064 / 1.541195 (-0.192131) | 1.362783 / 1.468490 (-0.105708) | 0.560765 / 4.584777 (-4.024012) | 2.357731 / 3.745712 (-1.387981) | 2.745771 / 5.269862 (-2.524090) | 1.726764 / 4.565676 (-2.838913) | 0.061212 / 0.424275 (-0.363063) | 0.004902 / 0.007607 (-0.002705) | 0.336963 / 0.226044 (0.110919) | 3.324519 / 2.268929 (1.055591) | 1.825826 / 55.444624 (-53.618798) | 1.548811 / 6.876477 (-5.327666) | 1.570618 / 2.142072 (-0.571454) | 0.642411 / 4.805227 (-4.162816) | 0.116068 / 6.500664 (-6.384596) | 0.042433 / 0.075469 (-0.033036) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.988402 / 1.841788 (-0.853386) | 11.509601 / 8.074308 (3.435293) | 9.555338 / 10.191392 (-0.636054) | 0.138728 / 0.680424 (-0.541696) | 0.014107 / 0.534201 (-0.520094) | 0.285465 / 0.579283 (-0.293818) | 0.263086 / 0.434364 (-0.171278) | 0.327469 / 0.540337 (-0.212869) | 0.444799 / 1.386936 (-0.942137) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005359 / 0.011353 (-0.005993) | 0.003605 / 0.011008 (-0.007403) | 0.049734 / 0.038508 (0.011226) | 0.029792 / 0.023109 (0.006683) | 0.276384 / 0.275898 (0.000486) | 0.297915 / 0.323480 (-0.025564) | 0.004949 / 0.007986 (-0.003036) | 0.002713 / 0.004328 (-0.001616) | 0.049499 / 0.004250 (0.045249) | 0.044969 / 0.037052 (0.007917) | 0.284558 / 0.258489 (0.026069) | 0.315170 / 0.293841 (0.021329) | 0.029457 / 0.128546 (-0.099089) | 0.010573 / 0.075646 (-0.065073) | 0.058191 / 0.419271 (-0.361080) | 0.051461 / 0.043533 (0.007928) | 0.270744 / 0.255139 (0.015605) | 0.291664 / 0.283200 (0.008465) | 0.018607 / 0.141683 (-0.123076) | 1.158799 / 1.452155 (-0.293355) | 1.210509 / 1.492716 (-0.282208) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090277 / 0.018006 (0.072270) | 0.298748 / 0.000490 (0.298258) | 0.000228 / 0.000200 (0.000028) | 0.000060 / 0.000054 (0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021850 / 0.037411 (-0.015561) | 0.075433 / 0.014526 (0.060907) | 0.087171 / 0.176557 (-0.089386) | 0.125828 / 0.737135 (-0.611308) | 0.090343 / 0.296338 (-0.205996) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.297267 / 0.215209 (0.082058) | 2.865234 / 2.077655 (0.787579) | 1.595024 / 1.504120 (0.090904) | 1.476100 / 1.541195 (-0.065094) | 1.494896 / 1.468490 (0.026406) | 0.569086 / 4.584777 (-4.015691) | 2.401976 / 3.745712 (-1.343736) | 2.676091 / 5.269862 (-2.593771) | 1.742087 / 4.565676 (-2.823590) | 0.065161 / 0.424275 (-0.359114) | 0.005006 / 0.007607 (-0.002602) | 0.342302 / 0.226044 (0.116257) | 3.450571 / 2.268929 (1.181643) | 1.928754 / 55.444624 (-53.515871) | 1.672823 / 6.876477 (-5.203653) | 1.798830 / 2.142072 (-0.343243) | 0.648730 / 4.805227 (-4.156498) | 0.116433 / 6.500664 (-6.384231) | 0.040683 / 0.075469 (-0.034786) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.006158 / 1.841788 (-0.835630) | 12.200093 / 8.074308 (4.125785) | 10.180691 / 10.191392 (-0.010701) | 0.146620 / 0.680424 (-0.533804) | 0.015621 / 0.534201 (-0.518580) | 0.287956 / 0.579283 (-0.291327) | 0.277231 / 0.434364 (-0.157133) | 0.323815 / 0.540337 (-0.216522) | 0.429655 / 1.386936 (-0.957281) |

![](https://cml.dev/watermark.png#273e16f9a21d6eaba1fd40fbdf0c05e66642c5a7 "CML watermark")

When you try to download a dataset that uses [biopython](https://github.com/biopython/biopython), like `load_dataset("InstaDeepAI/multi_species_genomes")`, you get the error:

```
>>> from datasets import load_dataset
>>> dataset = load_dataset("InstaDeepAI/multi_species_genomes")
/home/j.vangoey/.pyenv/versions/multi_species_genomes/lib/python3.10/site-packages/datasets/load.py:1454: FutureWarning: The repository for InstaDeepAI/multi_species_genomes contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/InstaDeepAI/multi_species_genomes
You can avoid this message in future by passing the argument `trust_remote_code=True`.
Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.
  warnings.warn(
Downloading builder script: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7.51k/7.51k [00:00<00:00, 7.67MB/s]
Downloading readme: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 17.2k/17.2k [00:00<00:00, 11.0MB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/j.vangoey/.pyenv/versions/multi_species_genomes/lib/python3.10/site-packages/datasets/load.py", line 2548, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/j.vangoey/.pyenv/versions/multi_species_genomes/lib/python3.10/site-packages/datasets/load.py", line 2220, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/home/j.vangoey/.pyenv/versions/multi_species_genomes/lib/python3.10/site-packages/datasets/load.py", line 1871, in dataset_module_factory
    raise e1 from None
  File "/home/j.vangoey/.pyenv/versions/multi_species_genomes/lib/python3.10/site-packages/datasets/load.py", line 1844, in dataset_module_factory
    ).get_module()
  File "/home/j.vangoey/.pyenv/versions/multi_species_genomes/lib/python3.10/site-packages/datasets/load.py", line 1466, in get_module
    local_imports = _download_additional_modules(
  File "/home/j.vangoey/.pyenv/versions/multi_species_genomes/lib/python3.10/site-packages/datasets/load.py", line 346, in _download_additional_modules
    raise ImportError(
ImportError: To be able to use InstaDeepAI/multi_species_genomes, you need to install the following dependency: Bio.
Please install it using 'pip install Bio' for instance.
>>>
```

`Bio` comes from the `biopython` package that can be installed with `pip install biopython`, not with  `pip install Bio` as suggested.

This PR adds special logic to show the correct package name in the error message of ` _download_additional_modules`, similarly as is done for `sklearn` / `scikit-learn`  already.

There are more packages where importable module name differs from the PyPI package name, so this could be made more generic, like:

```
# Mapping of importable module names to their PyPI package names
package_map = {
    "sklearn": "scikit-learn",
    "Bio": "biopython",
    "PIL": "Pillow",
    "bs4": "beautifulsoup4"
}

for module_name, pypi_name in package_map.items():
    if module_name in needs_to_be_installed.keys():
        needs_to_be_installed[module_name] = pypi_name
```


Import error on Google Colab

Hi! This can happen if an incompatible `pyarrow` version (`pyarrow<12.0.0`) has been imported before the `datasets` installation and the Colab session hasn't been restarted afterward. To avoid the error, go to "Runtime -> Restart session" after `!pip install -U datasets` and before `import datasets`, or insert the `import os; os.kill(os.getpid(), 9)` cell between `!pip install -U datasets` and `import datasets` to do the same programmatically.

One possible cause might be the one pointed out by @mariosasko above, and you get the following warning on Colab:
```
WARNING: The following packages were previously imported in this runtime:
  [pyarrow]
You must restart the runtime in order to use newly installed versions.
```

On the other hand, if the old version of `pyarrow` is not previously imported (before the installation of `datasets`), the reported issue here is not reproducible: `datasets` can be installed, imported and used on Colab.

Google Colab now pre-installs PyArrow 14.0.2, making this issue unlikely to happen. So, I'm unpinning it.

### Describe the bug

Cannot be imported on Google Colab, the import throws the following error:
ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject

### Steps to reproduce the bug

1. `! pip install -U datasets`
2. `import datasets`

### Expected behavior

Should be possible to use the library

### Environment info

- `datasets` version: 2.17.0
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0


Automatic Conversion for uint16/uint32 to Compatible PyTorch Dtypes

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6660). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004995 / 0.011353 (-0.006357) | 0.003230 / 0.011008 (-0.007779) | 0.062836 / 0.038508 (0.024328) | 0.026684 / 0.023109 (0.003575) | 0.249286 / 0.275898 (-0.026612) | 0.272936 / 0.323480 (-0.050544) | 0.003952 / 0.007986 (-0.004033) | 0.002708 / 0.004328 (-0.001620) | 0.055346 / 0.004250 (0.051095) | 0.040023 / 0.037052 (0.002971) | 0.263350 / 0.258489 (0.004860) | 0.294727 / 0.293841 (0.000886) | 0.027280 / 0.128546 (-0.101266) | 0.010273 / 0.075646 (-0.065373) | 0.206035 / 0.419271 (-0.213236) | 0.035715 / 0.043533 (-0.007818) | 0.255474 / 0.255139 (0.000335) | 0.273960 / 0.283200 (-0.009240) | 0.018563 / 0.141683 (-0.123120) | 1.140013 / 1.452155 (-0.312142) | 1.188655 / 1.492716 (-0.304062) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091895 / 0.018006 (0.073888) | 0.284621 / 0.000490 (0.284131) | 0.000213 / 0.000200 (0.000013) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018610 / 0.037411 (-0.018801) | 0.061554 / 0.014526 (0.047028) | 0.072454 / 0.176557 (-0.104103) | 0.120283 / 0.737135 (-0.616853) | 0.073744 / 0.296338 (-0.222595) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.288850 / 0.215209 (0.073641) | 2.836761 / 2.077655 (0.759107) | 1.533407 / 1.504120 (0.029287) | 1.409394 / 1.541195 (-0.131801) | 1.421667 / 1.468490 (-0.046823) | 0.566183 / 4.584777 (-4.018594) | 2.390670 / 3.745712 (-1.355043) | 2.732031 / 5.269862 (-2.537831) | 1.730886 / 4.565676 (-2.834791) | 0.064280 / 0.424275 (-0.359995) | 0.004959 / 0.007607 (-0.002648) | 0.342664 / 0.226044 (0.116619) | 3.398969 / 2.268929 (1.130040) | 1.887354 / 55.444624 (-53.557270) | 1.572955 / 6.876477 (-5.303522) | 1.596179 / 2.142072 (-0.545894) | 0.645844 / 4.805227 (-4.159383) | 0.118050 / 6.500664 (-6.382614) | 0.042158 / 0.075469 (-0.033311) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.959170 / 1.841788 (-0.882617) | 11.276491 / 8.074308 (3.202183) | 9.471198 / 10.191392 (-0.720194) | 0.128346 / 0.680424 (-0.552078) | 0.013851 / 0.534201 (-0.520350) | 0.286125 / 0.579283 (-0.293158) | 0.266915 / 0.434364 (-0.167449) | 0.332811 / 0.540337 (-0.207526) | 0.444780 / 1.386936 (-0.942156) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005665 / 0.011353 (-0.005687) | 0.003267 / 0.011008 (-0.007741) | 0.050238 / 0.038508 (0.011730) | 0.032882 / 0.023109 (0.009773) | 0.269320 / 0.275898 (-0.006578) | 0.293140 / 0.323480 (-0.030340) | 0.004127 / 0.007986 (-0.003858) | 0.002728 / 0.004328 (-0.001601) | 0.049360 / 0.004250 (0.045109) | 0.043764 / 0.037052 (0.006712) | 0.291211 / 0.258489 (0.032722) | 0.319745 / 0.293841 (0.025904) | 0.029371 / 0.128546 (-0.099175) | 0.010212 / 0.075646 (-0.065434) | 0.059064 / 0.419271 (-0.360207) | 0.051148 / 0.043533 (0.007615) | 0.276698 / 0.255139 (0.021559) | 0.292329 / 0.283200 (0.009129) | 0.018349 / 0.141683 (-0.123334) | 1.150816 / 1.452155 (-0.301338) | 1.184292 / 1.492716 (-0.308425) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091646 / 0.018006 (0.073640) | 0.301737 / 0.000490 (0.301247) | 0.000214 / 0.000200 (0.000014) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021529 / 0.037411 (-0.015883) | 0.075596 / 0.014526 (0.061070) | 0.087912 / 0.176557 (-0.088645) | 0.125240 / 0.737135 (-0.611895) | 0.088035 / 0.296338 (-0.208303) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.305097 / 0.215209 (0.089888) | 2.979612 / 2.077655 (0.901957) | 1.647009 / 1.504120 (0.142889) | 1.520251 / 1.541195 (-0.020944) | 1.513361 / 1.468490 (0.044870) | 0.571733 / 4.584777 (-4.013044) | 2.415587 / 3.745712 (-1.330125) | 2.615983 / 5.269862 (-2.653879) | 1.732637 / 4.565676 (-2.833039) | 0.062830 / 0.424275 (-0.361445) | 0.004972 / 0.007607 (-0.002635) | 0.348559 / 0.226044 (0.122515) | 3.450567 / 2.268929 (1.181639) | 1.970743 / 55.444624 (-53.473882) | 1.702232 / 6.876477 (-5.174245) | 1.799592 / 2.142072 (-0.342480) | 0.649477 / 4.805227 (-4.155751) | 0.115940 / 6.500664 (-6.384724) | 0.040364 / 0.075469 (-0.035105) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.000014 / 1.841788 (-0.841773) | 11.937886 / 8.074308 (3.863578) | 10.169478 / 10.191392 (-0.021914) | 0.153359 / 0.680424 (-0.527064) | 0.015205 / 0.534201 (-0.518996) | 0.287812 / 0.579283 (-0.291471) | 0.278688 / 0.434364 (-0.155676) | 0.322831 / 0.540337 (-0.217507) | 0.425631 / 1.386936 (-0.961305) |

![](https://cml.dev/watermark.png#6e176efbed29374e7c2cd33da64aeeae3c11ca0f "CML watermark")

This PR addresses an issue encountered when utilizing uint16 or uint32 datatypes with datasets, followed by attempting to convert these datasets into PyTorch-compatible formats. Currently, doing so results in a TypeError due to incompatible datatype conversion, as illustrated by the following example:
```python
from datasets import Dataset, Sequence, Value, Features

def gen():
    for i in range(100):
        yield {'seq': list(range(i, i + 20))}

ds = Dataset.from_generator(gen, features=Features({'seq': Sequence(feature=Value(dtype='uint16'), length=-1)}))

ds.set_format('torch')

print(ds[0])
```
This code snippet triggers the following error due to the inability to convert numpy.uint16 arrays to a PyTorch-supported format:
```
TypeError: can't convert np.ndarray of type numpy.uint16. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.
```

This PR introduces an automatic mechanism to convert np.uint16 and np.uint32 datatypes to np.int64 for seamless compatibility with PyTorch formats, simplifying workflows and improving developer experience by eliminating the need for manual conversion handling.


Change default compression argument for JsonDatasetWriter

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6659). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005008 / 0.011353 (-0.006345) | 0.003267 / 0.011008 (-0.007741) | 0.064140 / 0.038508 (0.025632) | 0.027419 / 0.023109 (0.004309) | 0.246692 / 0.275898 (-0.029206) | 0.271303 / 0.323480 (-0.052177) | 0.004127 / 0.007986 (-0.003859) | 0.002698 / 0.004328 (-0.001631) | 0.050415 / 0.004250 (0.046165) | 0.040323 / 0.037052 (0.003271) | 0.265738 / 0.258489 (0.007249) | 0.291556 / 0.293841 (-0.002285) | 0.027924 / 0.128546 (-0.100622) | 0.010206 / 0.075646 (-0.065441) | 0.207106 / 0.419271 (-0.212165) | 0.036087 / 0.043533 (-0.007446) | 0.250412 / 0.255139 (-0.004727) | 0.269014 / 0.283200 (-0.014186) | 0.018102 / 0.141683 (-0.123581) | 1.135137 / 1.452155 (-0.317018) | 1.177718 / 1.492716 (-0.314998) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095557 / 0.018006 (0.077550) | 0.306235 / 0.000490 (0.305745) | 0.000214 / 0.000200 (0.000014) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018217 / 0.037411 (-0.019194) | 0.060993 / 0.014526 (0.046467) | 0.072748 / 0.176557 (-0.103808) | 0.119357 / 0.737135 (-0.617778) | 0.073719 / 0.296338 (-0.222619) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295924 / 0.215209 (0.080715) | 2.901071 / 2.077655 (0.823417) | 1.497316 / 1.504120 (-0.006804) | 1.371232 / 1.541195 (-0.169962) | 1.395643 / 1.468490 (-0.072847) | 0.577548 / 4.584777 (-4.007229) | 2.383813 / 3.745712 (-1.361899) | 2.764451 / 5.269862 (-2.505411) | 1.733074 / 4.565676 (-2.832602) | 0.063730 / 0.424275 (-0.360545) | 0.004933 / 0.007607 (-0.002674) | 0.347135 / 0.226044 (0.121090) | 3.390814 / 2.268929 (1.121885) | 1.849454 / 55.444624 (-53.595170) | 1.561801 / 6.876477 (-5.314675) | 1.587818 / 2.142072 (-0.554254) | 0.652061 / 4.805227 (-4.153166) | 0.117195 / 6.500664 (-6.383469) | 0.041922 / 0.075469 (-0.033548) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.949050 / 1.841788 (-0.892738) | 11.353664 / 8.074308 (3.279355) | 9.261581 / 10.191392 (-0.929811) | 0.140374 / 0.680424 (-0.540050) | 0.014254 / 0.534201 (-0.519946) | 0.288124 / 0.579283 (-0.291159) | 0.262888 / 0.434364 (-0.171476) | 0.330774 / 0.540337 (-0.209564) | 0.444777 / 1.386936 (-0.942159) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005162 / 0.011353 (-0.006191) | 0.003418 / 0.011008 (-0.007591) | 0.049764 / 0.038508 (0.011256) | 0.029336 / 0.023109 (0.006226) | 0.278570 / 0.275898 (0.002672) | 0.300676 / 0.323480 (-0.022804) | 0.004292 / 0.007986 (-0.003694) | 0.002745 / 0.004328 (-0.001584) | 0.049194 / 0.004250 (0.044943) | 0.044036 / 0.037052 (0.006984) | 0.299258 / 0.258489 (0.040769) | 0.324451 / 0.293841 (0.030610) | 0.029777 / 0.128546 (-0.098769) | 0.010426 / 0.075646 (-0.065221) | 0.057267 / 0.419271 (-0.362004) | 0.051276 / 0.043533 (0.007743) | 0.278012 / 0.255139 (0.022873) | 0.297099 / 0.283200 (0.013899) | 0.018340 / 0.141683 (-0.123343) | 1.179255 / 1.452155 (-0.272899) | 1.231536 / 1.492716 (-0.261180) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092546 / 0.018006 (0.074540) | 0.299959 / 0.000490 (0.299469) | 0.000220 / 0.000200 (0.000020) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021657 / 0.037411 (-0.015755) | 0.075440 / 0.014526 (0.060914) | 0.086246 / 0.176557 (-0.090310) | 0.126511 / 0.737135 (-0.610624) | 0.091303 / 0.296338 (-0.205036) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.294775 / 0.215209 (0.079566) | 2.868973 / 2.077655 (0.791319) | 1.666971 / 1.504120 (0.162851) | 1.545680 / 1.541195 (0.004486) | 1.559983 / 1.468490 (0.091493) | 0.572191 / 4.584777 (-4.012586) | 2.429317 / 3.745712 (-1.316395) | 2.673334 / 5.269862 (-2.596527) | 1.758114 / 4.565676 (-2.807563) | 0.063766 / 0.424275 (-0.360509) | 0.005070 / 0.007607 (-0.002537) | 0.345488 / 0.226044 (0.119443) | 3.464525 / 2.268929 (1.195596) | 1.975717 / 55.444624 (-53.468908) | 1.686671 / 6.876477 (-5.189806) | 1.825434 / 2.142072 (-0.316638) | 0.655853 / 4.805227 (-4.149374) | 0.116372 / 6.500664 (-6.384292) | 0.040647 / 0.075469 (-0.034822) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.014080 / 1.841788 (-0.827707) | 12.038496 / 8.074308 (3.964188) | 10.354536 / 10.191392 (0.163144) | 0.130285 / 0.680424 (-0.550139) | 0.015514 / 0.534201 (-0.518687) | 0.284743 / 0.579283 (-0.294540) | 0.280275 / 0.434364 (-0.154088) | 0.321175 / 0.540337 (-0.219162) | 0.425840 / 1.386936 (-0.961096) |

![](https://cml.dev/watermark.png#4bb6c6d46a171c4fa1b65167cb81998e2f863892 "CML watermark")

Change default compression type from `None` to "infer", to align with pandas' defaults.

Documentation asks the user to supply `to_json_kwargs` with arguments suitable for pandas' `to_json` method. At the same time, while pandas' by default uses ["infer"](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_json.html) for compression, datasets enforce `None` as default. This, likely, confuses user, as they expect the same behaviour, i.e they expect that if they name their output file as "dataset.jsonl.zst" then the compression would be inferred as "zstd" and file will be compressed before writing.

Moreover, while it is probably outside of the scope of this pull request, `compression` argument needs to be capable of taking `dict` as input (along with `str`), as it does in pandas, in order to allow user to specify compression parameters. Current implementation will likely fail with `NotImplementedError`, as it expects either `None` or `str` specifying compression algo.


[Resumable IterableDataset] Add IterableDataset state_dict

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6658). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

would be nice to have this feature in the new dataset release!

Before finalising this this I'd like to make sure this philosophy makes sense for other libs like `accelerate` for example.

cc @muellerzr I'd love your feedback on this one
cc @LysandreJik also if you think other people should take a look

> One design question though: what's the logic behind self._state_dict rather than having it all be state_dict?

The `_state_dict` is the internal object that is updated in-place while you iterate on the dataset.

We need to copy it every time the user accesses it.

Otherwise we would get
```python
state_dict = ds.state_dict()
for x in ds:
    assert ds.state_dict() == state_dict  # and actually `assert ds.state_dict() is state_dict`
```

The state is updated in-place since it's made of dictionaries that are shared with the steps in the IterableDataset pipeline.

What do you think of making it a full property with a docstring explicitly stating users shouldn’t call/modify it directly?

I can imagine some exploratory users getting curious

Just a small note mentioning returns a copy of the state dict should be enough imo

looking forward as well for this PR to be merge

> I don't think users read docstrings of properties that often. What about explaining the logic in the `.state_dict()` docstring ? This also feels aligned with the way `.state_dict()` and `.load_state_dict()` works in pytorch (you should use load_state_dict to load a modified copy of the state dict)

Hi, I'm experimenting with LLM pretraining using your code. I found that the time of resuming an iterable dataset can be reduced to 5% (my streaming process includes tokenization), but I'm not sure if I'm using it correctly. Could you help me check it? Thanks.

```
class CustomTrainer(Trainer):
    def _save_rng_state(self, output_dir):
        super()._save_rng_state(output_dir)
        if self.args.should_save:
            with open(os.path.join(output_dir, f'iterable_data_state_dict.json'), 'w', encoding='utf-8') as fo:
                json.dump(self.train_dataset.state_dict(), fo, ensure_ascii=False)
```

```
    dataset = <A IterableDataset constructed by (interleave, map(tokenization))>
    lask_ckpt_iterable_data_state_dict_file_path = os.path.join(training_args.resume_from_checkpoint, f'iterable_data_state_dict.json')
    if os.path.exists(lask_ckpt_iterable_data_state_dict_file_path) and finetuning_args.load_iteratable_state_dict:
        if not training_args.ignore_data_skip:
            raise ValueError(f'Found `iterable_data_state_dict_file_path`: `{lask_ckpt_iterable_data_state_dict_file_path}`. Please set `ignore_data_skip`=True to skip tokenization.')
        with open(lask_ckpt_iterable_data_state_dict_file_path) as f:
            lask_ckpt_iterable_data_state_dict = json.load(f)
            dataset.load_state_dict(lask_ckpt_iterable_data_state_dict)
            logger.info(f'Loading `iterable_data_state_dict` from {lask_ckpt_iterable_data_state_dict_file_path}')
```

@lhoestq Hi, if I set `prefetch`, does this dataset work well?

It does work well if you prefetch and then resume from a state, but you might lose the samples that were in the prefetch buffer of the DataLoader (which could be acceptable in some circumstances).

Fortunately we're about to ship an integration with the new StatefulDataLoader from torchdata which can help on this matter :)

yeah, what I meant is that prefetch might drop a few data entries. really looking forward to the new StatefulDataLoader. ：)

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005788 / 0.011353 (-0.005564) | 0.004036 / 0.011008 (-0.006972) | 0.064720 / 0.038508 (0.026212) | 0.034990 / 0.023109 (0.011881) | 0.245488 / 0.275898 (-0.030410) | 0.272596 / 0.323480 (-0.050884) | 0.003170 / 0.007986 (-0.004815) | 0.002867 / 0.004328 (-0.001461) | 0.049961 / 0.004250 (0.045711) | 0.050951 / 0.037052 (0.013899) | 0.257757 / 0.258489 (-0.000732) | 0.292957 / 0.293841 (-0.000884) | 0.027739 / 0.128546 (-0.100807) | 0.010942 / 0.075646 (-0.064705) | 0.205153 / 0.419271 (-0.214118) | 0.037892 / 0.043533 (-0.005641) | 0.247536 / 0.255139 (-0.007603) | 0.267239 / 0.283200 (-0.015960) | 0.021490 / 0.141683 (-0.120193) | 1.107306 / 1.452155 (-0.344848) | 1.144675 / 1.492716 (-0.348041) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.103212 / 0.018006 (0.085205) | 0.315174 / 0.000490 (0.314684) | 0.000229 / 0.000200 (0.000029) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019771 / 0.037411 (-0.017641) | 0.064033 / 0.014526 (0.049507) | 0.076751 / 0.176557 (-0.099805) | 0.122615 / 0.737135 (-0.614521) | 0.078490 / 0.296338 (-0.217848) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.286236 / 0.215209 (0.071027) | 2.841469 / 2.077655 (0.763814) | 1.514079 / 1.504120 (0.009959) | 1.393792 / 1.541195 (-0.147403) | 1.432741 / 1.468490 (-0.035749) | 0.571003 / 4.584777 (-4.013774) | 2.369031 / 3.745712 (-1.376681) | 2.825246 / 5.269862 (-2.444616) | 1.858524 / 4.565676 (-2.707153) | 0.065366 / 0.424275 (-0.358909) | 0.005107 / 0.007607 (-0.002500) | 0.341010 / 0.226044 (0.114965) | 3.443894 / 2.268929 (1.174966) | 1.879192 / 55.444624 (-53.565433) | 1.603046 / 6.876477 (-5.273431) | 1.807639 / 2.142072 (-0.334433) | 0.646726 / 4.805227 (-4.158502) | 0.119409 / 6.500664 (-6.381255) | 0.044564 / 0.075469 (-0.030905) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.971026 / 1.841788 (-0.870762) | 12.593884 / 8.074308 (4.519576) | 10.305243 / 10.191392 (0.113851) | 0.132018 / 0.680424 (-0.548406) | 0.014387 / 0.534201 (-0.519814) | 0.288597 / 0.579283 (-0.290686) | 0.267373 / 0.434364 (-0.166991) | 0.325626 / 0.540337 (-0.214711) | 0.488808 / 1.386936 (-0.898128) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005991 / 0.011353 (-0.005362) | 0.004028 / 0.011008 (-0.006980) | 0.051951 / 0.038508 (0.013443) | 0.036870 / 0.023109 (0.013761) | 0.263777 / 0.275898 (-0.012122) | 0.290914 / 0.323480 (-0.032566) | 0.004594 / 0.007986 (-0.003392) | 0.002971 / 0.004328 (-0.001357) | 0.049699 / 0.004250 (0.045449) | 0.044939 / 0.037052 (0.007887) | 0.275055 / 0.258489 (0.016566) | 0.316244 / 0.293841 (0.022403) | 0.030501 / 0.128546 (-0.098045) | 0.011197 / 0.075646 (-0.064449) | 0.058718 / 0.419271 (-0.360554) | 0.034926 / 0.043533 (-0.008607) | 0.259172 / 0.255139 (0.004033) | 0.280127 / 0.283200 (-0.003072) | 0.019775 / 0.141683 (-0.121908) | 1.169468 / 1.452155 (-0.282687) | 1.178098 / 1.492716 (-0.314619) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.101633 / 0.018006 (0.083626) | 0.314684 / 0.000490 (0.314194) | 0.000224 / 0.000200 (0.000024) | 0.000055 / 0.000054 (0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024071 / 0.037411 (-0.013341) | 0.079894 / 0.014526 (0.065368) | 0.090915 / 0.176557 (-0.085642) | 0.132397 / 0.737135 (-0.604738) | 0.091919 / 0.296338 (-0.204419) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.296237 / 0.215209 (0.081028) | 2.891752 / 2.077655 (0.814097) | 1.551937 / 1.504120 (0.047817) | 1.414179 / 1.541195 (-0.127016) | 1.450192 / 1.468490 (-0.018298) | 0.556272 / 4.584777 (-4.028504) | 0.952374 / 3.745712 (-2.793339) | 2.709450 / 5.269862 (-2.560411) | 1.771251 / 4.565676 (-2.794426) | 0.061873 / 0.424275 (-0.362402) | 0.005058 / 0.007607 (-0.002549) | 0.344790 / 0.226044 (0.118746) | 3.398982 / 2.268929 (1.130053) | 1.905832 / 55.444624 (-53.538792) | 1.632357 / 6.876477 (-5.244120) | 1.822913 / 2.142072 (-0.319160) | 0.643426 / 4.805227 (-4.161802) | 0.117321 / 6.500664 (-6.383343) | 0.042107 / 0.075469 (-0.033363) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.974921 / 1.841788 (-0.866867) | 12.497801 / 8.074308 (4.423493) | 11.216174 / 10.191392 (1.024782) | 0.135288 / 0.680424 (-0.545136) | 0.016731 / 0.534201 (-0.517470) | 0.287987 / 0.579283 (-0.291296) | 0.130246 / 0.434364 (-0.304117) | 0.323282 / 0.540337 (-0.217055) | 0.414595 / 1.386936 (-0.972341) |

![](https://cml.dev/watermark.png#43fd659eacab37b50abfe7f1b4efe2564055990c "CML watermark")

@lhoestq Hello, I'm wondering if there are any solutions to work with shuffle now. I've noticed the caveats in docs,
> examples from shuffle buffers are lost when resuming and the buffers are refilled with new data

Hi ! I haven't experimented with implementing state_dict for the shuffle buffer. Not sure if this is a good idea to add this, given a shuffle buffer can be quite big and poses serialization challenges.

It shouldn't be difficult to experiment with a simple implementation in `BufferShuffledExamplesIterable` though

@lhoestq thank you for your quick response! I'll try it :}

@lhoestq Hi, just revise the `BufferShuffledExamplesIterable` and it works
```py

class BufferShuffledExamplesIterable(datasets.iterable_dataset.BufferShuffledExamplesIterable):

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def _init_state_dict(self) -> dict:
        self._state_dict = self.ex_iterable._init_state_dict()
        self._state_dict['mem_buffer'] = ([],)
        self._state_dict['gloabl_example_index'] = 0
        return self._state_dict

    def __iter__(self):
        buffer_size = self.buffer_size
        rng = deepcopy(self.generator)
        indices_iterator = self._iter_random_indices(rng, buffer_size)
        # this is the shuffle buffer that we keep in memory
        mem_buffer = self._state_dict['mem_buffer'][0]
        gloabl_example_index_start = self._state_dict["gloabl_example_index"] if self._state_dict else 0
        # skip already consumed ones
        for i in range(gloabl_example_index_start):
            _ = next(indices_iterator)
        for x in self.ex_iterable:
            if len(mem_buffer) == buffer_size:  # if the buffer is full, pick and example from it
                i = next(indices_iterator)
                if self._state_dict:
                    self._state_dict['gloabl_example_index'] += 1
                yield mem_buffer[i]
                mem_buffer[i] = x  # replace the picked example by a new one
            else:  # otherwise, keep filling the buffer
                mem_buffer.append(x)
        # when we run out of examples, we shuffle the remaining examples in the buffer and yield them
        rng.shuffle(mem_buffer)
        yield from mem_buffer

    def shuffle_data_sources(self, generator: np.random.Generator) -> BufferShuffledExamplesIterable:
        """Shuffle the wrapped examples iterable as well as the shuffling buffer."""
        return BufferShuffledExamplesIterable(
            self.ex_iterable.shuffle_data_sources(generator), buffer_size=self.buffer_size, generator=generator
        )

    def shard_data_sources(self, worker_id: int, num_workers: int) -> BufferShuffledExamplesIterable:
        """Keep only the requested shard."""
        return BufferShuffledExamplesIterable(
            self.ex_iterable.shard_data_sources(worker_id, num_workers),
            buffer_size=self.buffer_size,
            generator=self.generator,
        )

    def load_state_dict(self, state_dict: dict) -> dict:
        def _inner_load_state_dict(state, new_state):
            if new_state is not None and isinstance(state, dict):
                for key in state:
                    state[key] = _inner_load_state_dict(state[key], new_state[key])
                return state
            elif new_state is not None and isinstance(state, list):
                for i in range(len(state)):
                    state[i] = _inner_load_state_dict(state[i], new_state[i])
                return state
            return new_state

        return _inner_load_state_dict(self._state_dict, state_dict)
```

I've noticed that it uses significantly more RAM than the original version and experiences a considerable decrease in GPU utilization. Could you offer some suggestions to address this issue?

or **is it prohibited** to maintain sth except for simple indices that small enough for each worker 😢

Some ExamplesIterable copy and store old versions of the state_dict of parent ExamplesIterable. It is the case for example for batched `map()` (state_dict of beginning of the batch) or `interleave_dataset()`  (state_dict of the previous step since it buffers one example to know if the iterable is exhausted).

Copying a shuffle buffer takes some RAM and some time, which can slow down the data loading pipeline.
Maybe the examples in the shuffle buffer shouldn't not be copied (only do a shallow copy of the list), this would surely help.

A simple implementation of a mechanism to resume an IterableDataset.
This is WIP and untested.

Example:

```python
from datasets import Dataset, concatenate_datasets

ds = Dataset.from_dict({"a": range(5)}).to_iterable_dataset(num_shards=3)
ds = concatenate_datasets([ds] * 2)

print(f"{ds.state_dict()=}")
for i, example in enumerate(ds):
    print(example)
    if i == 6:
        state_dict = ds.state_dict()
        print("checkpoint")
ds.load_state_dict(state_dict)
print(f"resuming from checkpoint {ds.state_dict()=}")
for example in ds:
    print(example)
```

returns

```
ds.state_dict()={'ex_iterable_idx': 0, 'ex_iterables': [{'shard_idx': 0, 'shard_example_idx': 0}, {'shard_idx': 0, 'shard_example_idx': 0}]}
{'a': 0}
{'a': 1}
{'a': 2}
{'a': 3}
{'a': 4}
{'a': 0}
{'a': 1}
checkpoint
{'a': 2}
{'a': 3}
{'a': 4}
resuming from checkpoint ds.state_dict()={'ex_iterable_idx': 1, 'ex_iterables': [{'shard_idx': 3, 'shard_example_idx': 0}, {'shard_idx': 0, 'shard_example_idx': 2}]}
{'a': 2}
{'a': 3}
{'a': 4}
```


Release not pushed to conda channel

Thanks for reporting, @atulsaurav.

We are investigating the issue.

I can't fix this issue because I do not appear as a team member of the huggingface datasets project: https://anaconda.org/huggingface/datasets

@lhoestq could you please add `datasets` team members to the corresponding Anaconda project?

Once this done, I could recreate and update the Anaconda token, as mentioned above it seems the current one has expired.

FYI it failed for 2.18.0 too: https://github.com/huggingface/datasets/actions/runs/8117132330/job/22188677936

We updated the token and I re-ran the conda releases :)

### Describe the bug

The github actions step to publish the release 2.17.0 to conda channel has failed due to expired token. Can some one please update the anaconda token rerun the failed action? @albertvillanova ?

![image](https://github.com/huggingface/datasets/assets/7138162/1b56ad3d-7643-4778-9cce-4bf531717700)

### Steps to reproduce the bug

Please see this actions [link](https://github.com/huggingface/datasets/actions/runs/7842473662)

### Expected behavior

The action runs successfully and the latest release is pushed to HuggingFace conda channel

### Environment info

Not applicable.


Error when loading a big local json file

I get similar when dealing with a large jsonl file (6k lines),

> TypeError: Couldn't cast array of type timestamp[us] to null

Yet when I split it into 1k lines, files, load_dataset works fine!

https://github.com/huggingface/course/issues/692

### Describe the bug

When trying to load big json files from a local directory, `load_dataset` throws the following error

```
Traceback (most recent call last):
  File "/miniconda3/envs/conda-env/lib/python3.10/site-packages/datasets/builder.py", line 1989, in _prepare_split_single
    writer.write_table(table)
  File "miniconda3/envs/conda-env/lib/python3.10/site-packages/datasets/arrow_writer.py", line 573, in write_table
    pa_table = pa_table.combine_chunks()
  File "pyarrow/table.pxi", line 3638, in pyarrow.lib.Table.combine_chunks
  File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays
```

### Steps to reproduce the bug

1. Download a big file, e.g. `https://dl.fbaipublicfiles.com/dpr/data/retriever/biencoder-nq-train.json.gz`
2. Load it like `data = load_dataset("json", data_files=["nq-train.json"], split="train")`

```python
from datasets import load_dataset

data = load_dataset("json", data_files=["nq-train.json"], split="train")
```

A similarly formatted but smaller file, e.g. e.g. `https://dl.fbaipublicfiles.com/dpr/data/retriever/biencoder-nq-dev.json.gz` is loaded without issues

```python
from datasets import load_dataset

data = load_dataset("json", data_files=["nq-dev.json"], split="train")
```

### Expected behavior

It should load normally

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-5.18.10-76051810-generic-x86_64-with-glibc2.31
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0


Cannot load the dataset go_emotions

Thanks for reporting, @arame.

I guess you have an old version of `transformers` (that submodule is present in `transformers` since version 3.0.1, since nearly 4 years ago). If you update it, the error should disappear:
```shell
pip install -U transformers
```

On the other hand, I am wondering: does it make sense to use `transformers` in this case, even if we don't need it to load the `go_emotions` dataset (already converted to Parquet files)?
- Maybe @mariosasko can give some insight, as he included these code lines:
  - #6454

https://github.com/huggingface/datasets/blob/9751fb14594d354e952f0ebdfaf31cb203b011e7/src/datasets/utils/_dill.py#L60-L63

The linked code lazily registers a custom reducer for `transformers.PreTrainedTokenizerBase` only if `transformers` have already been imported (imports are expensive, so we check `sys.modules`).

However, the logic does not account for `transformers<3`, so we should add a version check to fix that.

> The linked code lazily registers a custom reducer for `transformers.PreTrainedTokenizerBase` only if `transformers` have already been imported (imports are expensive, so we check `sys.modules`).
>
> However, the logic does not account for `transformers<3`, so we should add a version check to fix that.

Thank you for that Mario. Would this fix solve the problem and do you have any idea when it will be done?
I tried the pip install suggested by Albert and it made no difference.

I tried running the code today and the problem appears to be fixed.

### Describe the bug

When I run the following code I get an exception;
`go_emotions = load_dataset("go_emotions")`

> AttributeError                            Traceback (most recent call last)
Cell In[6], [line 1](vscode-notebook-cell:?execution_count=6&line=1)
----> [1](vscode-notebook-cell:?execution_count=6&line=1) go_emotions = load_dataset("go_emotions")
      [2](vscode-notebook-cell:?execution_count=6&line=2) data = go_emotions.data

File [c:\Users\hijik\anaconda3\Lib\site-packages\datasets\load.py:2523](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2523), in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, ignore_verifications, keep_in_memory, save_infos, revision, token, use_auth_token, task, streaming, num_proc, storage_options, trust_remote_code, **config_kwargs)
   [2518](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2518) verification_mode = VerificationMode(
   [2519](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2519)     (verification_mode or VerificationMode.BASIC_CHECKS) if not save_infos else VerificationMode.ALL_CHECKS
   [2520](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2520) )
   [2522](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2522) # Create a dataset builder
-> [2523](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2523) builder_instance = load_dataset_builder(
   [2524](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2524)     path=path,
   [2525](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2525)     name=name,
   [2526](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2526)     data_dir=data_dir,
   [2527](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2527)     data_files=data_files,
   [2528](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2528)     cache_dir=cache_dir,
   [2529](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2529)     features=features,
   [2530](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2530)     download_config=download_config,
   [2531](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2531)     download_mode=download_mode,
   [2532](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2532)     revision=revision,
   [2533](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2533)     token=token,
   [2534](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2534)     storage_options=storage_options,
   [2535](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2535)     trust_remote_code=trust_remote_code,
   [2536](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2536)     _require_default_config_name=name is None,
...
---> [63](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/utils/_dill.py:63)         if issubclass(obj_type, transformers.PreTrainedTokenizerBase):
     [64](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/utils/_dill.py:64)             pklregister(obj_type)(_save_transformersPreTrainedTokenizerBase)
     [66](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/utils/_dill.py:66) # Unwrap `torch.compile`-ed functions

AttributeError: module 'transformers' has no attribute 'PreTrainedTokenizerBase'
Output is truncated. View as a [scrollable element](command:cellOutput.enableScrolling?10bc0728-6947-456e-9a3e-f056872b04c6) or open in a [text editor](command:workbench.action.openLargeOutput?10bc0728-6947-456e-9a3e-f056872b04c6). Adjust cell output [settings](command:workbench.action.openSettings?%5B%22%40tag%3AnotebookOutputLayout%22%5D)...

### Steps to reproduce the bug

```
from datasets import load_dataset
go_emotions = load_dataset("go_emotions")
```

### Expected behavior

Should simply load the variable with the data from the file

### Environment info

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 2.16.1
- Platform: Windows-10-10.0.22631-SP0
- Python version: 3.11.4
- `huggingface_hub` version: 0.20.3
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.10.0


Batched dataset map throws exception that cannot cast fixed length array to Sequence

Hi ! This issue has been fixed by https://github.com/huggingface/datasets/pull/6283

Can you try again with the new release 2.17.0 ?

```
pip install -U datasets
```

### Describe the bug

I encountered a TypeError when batch processing a dataset with Sequence features in datasets package version 2.16.1. The error arises from a mismatch in handling fixed-size list arrays during the map function execution. Debugging pinpoints the issue to an if-statement in datasets/table.py, line 2093, failing to correctly process sequence lengths.

### Steps to reproduce the bug

Create virtual environment and activate
```
virtualenv venv
source venv/bin/activate
```

Then install the datasets package (I'm using the latest version)
```
pip install datasets==2.16.1
```

Then run
```python
# bug.py
from datasets import Dataset
from datasets.features import Features, Sequence, Value

data = {
    "num": [[1, 2], [3, 4]],
}
features = Features({'num': Sequence(feature=Value(dtype='int32'), length=2)})
dataset = Dataset.from_dict(data, features=features)
dataset.map(lambda x: x, batched=True, batch_size=1)
```

### Expected behavior

I get the following stack trace

```
Map:  50%|█████     | 1/2 [00:00<00:00, 423.92 examples/s]
Traceback (most recent call last):
  File "/PATH/TO/BUG_PORT/bug.py", line 9, in <module>
    dataset.map(lambda x: x, batched=True, batch_size=1)
  File "/PATH/TO/BUG_PORT/venv/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 592, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/PATH/TO/BUG_PORT/venv/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 557, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/PATH/TO/BUG_PORT/venv/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3093, in map
    for rank, done, content in Dataset._map_single(**dataset_kwargs):
  File "/PATH/TO/BUG_PORT/venv/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3489, in _map_single
    writer.write_batch(batch)
  File "/PATH/TO/BUG_PORT/venv/lib/python3.9/site-packages/datasets/arrow_writer.py", line 551, in write_batch
    array = cast_array_to_feature(col_values, col_type) if col_type is not None else col_values
  File "/PATH/TO/BUG_PORT/venv/lib/python3.9/site-packages/datasets/table.py", line 1797, in wrapper
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/PATH/TO/BUG_PORT/venv/lib/python3.9/site-packages/datasets/table.py", line 1797, in <listcomp>
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/PATH/TO/BUG_PORT/venv/lib/python3.9/site-packages/datasets/table.py", line 2111, in cast_array_to_feature
    raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{feature}")
TypeError: Couldn't cast array of type
fixed_size_list<item: int32>[2]
to
Sequence(feature=Value(dtype='int32', id=None), length=2, id=None)

```

After some debugging, I found that the if-statement that is actually failing is line 2093 in `datasets/table.py`
```python
# datasets/table.py
                ...
2093                if feature.length * len(array) == len(array_values):
2094                    return pa.FixedSizeListArray.from_arrays(_c(array_values, feature.feature), feature.length)
                ...
```

### Environment info

Platform: MacOS
Datasets version: datasets==2.16.1
Python version: 3.9.6


Set dev version

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6653). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005076 / 0.011353 (-0.006277) | 0.003424 / 0.011008 (-0.007584) | 0.064195 / 0.038508 (0.025687) | 0.031742 / 0.023109 (0.008633) | 0.244774 / 0.275898 (-0.031124) | 0.268529 / 0.323480 (-0.054951) | 0.003970 / 0.007986 (-0.004016) | 0.002657 / 0.004328 (-0.001672) | 0.048847 / 0.004250 (0.044597) | 0.042196 / 0.037052 (0.005144) | 0.266044 / 0.258489 (0.007555) | 0.282400 / 0.293841 (-0.011441) | 0.027617 / 0.128546 (-0.100929) | 0.010400 / 0.075646 (-0.065246) | 0.205910 / 0.419271 (-0.213362) | 0.035820 / 0.043533 (-0.007713) | 0.247750 / 0.255139 (-0.007389) | 0.267318 / 0.283200 (-0.015882) | 0.017980 / 0.141683 (-0.123703) | 1.107263 / 1.452155 (-0.344892) | 1.173208 / 1.492716 (-0.319509) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095830 / 0.018006 (0.077824) | 0.293891 / 0.000490 (0.293401) | 0.000257 / 0.000200 (0.000057) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018138 / 0.037411 (-0.019273) | 0.061631 / 0.014526 (0.047105) | 0.073038 / 0.176557 (-0.103519) | 0.118317 / 0.737135 (-0.618818) | 0.074190 / 0.296338 (-0.222148) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.287026 / 0.215209 (0.071817) | 2.786137 / 2.077655 (0.708482) | 1.472575 / 1.504120 (-0.031544) | 1.346919 / 1.541195 (-0.194276) | 1.388535 / 1.468490 (-0.079955) | 0.565731 / 4.584777 (-4.019046) | 2.382573 / 3.745712 (-1.363139) | 2.736926 / 5.269862 (-2.532935) | 1.716517 / 4.565676 (-2.849159) | 0.062168 / 0.424275 (-0.362108) | 0.004924 / 0.007607 (-0.002683) | 0.341897 / 0.226044 (0.115853) | 3.355715 / 2.268929 (1.086787) | 1.837014 / 55.444624 (-53.607611) | 1.532063 / 6.876477 (-5.344414) | 1.548193 / 2.142072 (-0.593880) | 0.634995 / 4.805227 (-4.170232) | 0.115622 / 6.500664 (-6.385042) | 0.042252 / 0.075469 (-0.033217) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.970713 / 1.841788 (-0.871075) | 11.727576 / 8.074308 (3.653268) | 9.806524 / 10.191392 (-0.384868) | 0.127622 / 0.680424 (-0.552802) | 0.014140 / 0.534201 (-0.520061) | 0.286832 / 0.579283 (-0.292451) | 0.266556 / 0.434364 (-0.167808) | 0.325940 / 0.540337 (-0.214398) | 0.421839 / 1.386936 (-0.965097) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005495 / 0.011353 (-0.005858) | 0.003676 / 0.011008 (-0.007332) | 0.054361 / 0.038508 (0.015853) | 0.030743 / 0.023109 (0.007633) | 0.277200 / 0.275898 (0.001302) | 0.313459 / 0.323480 (-0.010021) | 0.004316 / 0.007986 (-0.003670) | 0.002750 / 0.004328 (-0.001578) | 0.049491 / 0.004250 (0.045241) | 0.044268 / 0.037052 (0.007215) | 0.292529 / 0.258489 (0.034039) | 0.326524 / 0.293841 (0.032683) | 0.048040 / 0.128546 (-0.080507) | 0.010390 / 0.075646 (-0.065256) | 0.058459 / 0.419271 (-0.360813) | 0.033765 / 0.043533 (-0.009768) | 0.276003 / 0.255139 (0.020864) | 0.297299 / 0.283200 (0.014099) | 0.018532 / 0.141683 (-0.123151) | 1.157639 / 1.452155 (-0.294515) | 1.220492 / 1.492716 (-0.272225) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093903 / 0.018006 (0.075897) | 0.303005 / 0.000490 (0.302515) | 0.000224 / 0.000200 (0.000024) | 0.000051 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021580 / 0.037411 (-0.015831) | 0.076176 / 0.014526 (0.061650) | 0.086998 / 0.176557 (-0.089558) | 0.124148 / 0.737135 (-0.612987) | 0.088613 / 0.296338 (-0.207725) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.300623 / 0.215209 (0.085414) | 2.911876 / 2.077655 (0.834221) | 1.588398 / 1.504120 (0.084278) | 1.471251 / 1.541195 (-0.069944) | 1.505528 / 1.468490 (0.037038) | 0.570635 / 4.584777 (-4.014142) | 2.485769 / 3.745712 (-1.259943) | 2.785355 / 5.269862 (-2.484507) | 1.752944 / 4.565676 (-2.812732) | 0.063146 / 0.424275 (-0.361129) | 0.004980 / 0.007607 (-0.002627) | 0.354577 / 0.226044 (0.128532) | 3.477181 / 2.268929 (1.208253) | 1.951906 / 55.444624 (-53.492718) | 1.677169 / 6.876477 (-5.199307) | 1.686338 / 2.142072 (-0.455735) | 0.637156 / 4.805227 (-4.168071) | 0.117732 / 6.500664 (-6.382932) | 0.041091 / 0.075469 (-0.034378) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.010071 / 1.841788 (-0.831717) | 12.172242 / 8.074308 (4.097934) | 10.422811 / 10.191392 (0.231419) | 0.137185 / 0.680424 (-0.543239) | 0.014643 / 0.534201 (-0.519558) | 0.287248 / 0.579283 (-0.292035) | 0.272779 / 0.434364 (-0.161585) | 0.331761 / 0.540337 (-0.208576) | 0.417266 / 1.386936 (-0.969670) |

![](https://cml.dev/watermark.png#9751fb14594d354e952f0ebdfaf31cb203b011e7 "CML watermark")


Release: 2.17.0

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6652). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005207 / 0.011353 (-0.006145) | 0.003785 / 0.011008 (-0.007223) | 0.064221 / 0.038508 (0.025713) | 0.028981 / 0.023109 (0.005872) | 0.246215 / 0.275898 (-0.029683) | 0.268058 / 0.323480 (-0.055422) | 0.004028 / 0.007986 (-0.003958) | 0.002804 / 0.004328 (-0.001525) | 0.048878 / 0.004250 (0.044627) | 0.042641 / 0.037052 (0.005589) | 0.255590 / 0.258489 (-0.002899) | 0.287377 / 0.293841 (-0.006464) | 0.027772 / 0.128546 (-0.100774) | 0.010637 / 0.075646 (-0.065009) | 0.211526 / 0.419271 (-0.207746) | 0.035789 / 0.043533 (-0.007744) | 0.243042 / 0.255139 (-0.012097) | 0.268369 / 0.283200 (-0.014830) | 0.017907 / 0.141683 (-0.123776) | 1.138829 / 1.452155 (-0.313326) | 1.175732 / 1.492716 (-0.316984) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094205 / 0.018006 (0.076199) | 0.304317 / 0.000490 (0.303827) | 0.000206 / 0.000200 (0.000006) | 0.000049 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018424 / 0.037411 (-0.018987) | 0.061719 / 0.014526 (0.047193) | 0.073471 / 0.176557 (-0.103085) | 0.121577 / 0.737135 (-0.615558) | 0.075134 / 0.296338 (-0.221204) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.275178 / 0.215209 (0.059969) | 2.689222 / 2.077655 (0.611568) | 1.396680 / 1.504120 (-0.107439) | 1.278782 / 1.541195 (-0.262413) | 1.326632 / 1.468490 (-0.141858) | 0.566915 / 4.584777 (-4.017862) | 2.365928 / 3.745712 (-1.379784) | 2.785435 / 5.269862 (-2.484427) | 1.745131 / 4.565676 (-2.820546) | 0.062798 / 0.424275 (-0.361477) | 0.005107 / 0.007607 (-0.002500) | 0.330441 / 0.226044 (0.104396) | 3.266265 / 2.268929 (0.997337) | 1.792588 / 55.444624 (-53.652036) | 1.516021 / 6.876477 (-5.360455) | 1.562750 / 2.142072 (-0.579323) | 0.652964 / 4.805227 (-4.152264) | 0.117813 / 6.500664 (-6.382852) | 0.042372 / 0.075469 (-0.033097) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.010107 / 1.841788 (-0.831680) | 11.819910 / 8.074308 (3.745602) | 9.701673 / 10.191392 (-0.489719) | 0.178165 / 0.680424 (-0.502259) | 0.014438 / 0.534201 (-0.519763) | 0.297733 / 0.579283 (-0.281550) | 0.264914 / 0.434364 (-0.169450) | 0.324531 / 0.540337 (-0.215806) | 0.430207 / 1.386936 (-0.956729) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005848 / 0.011353 (-0.005505) | 0.003870 / 0.011008 (-0.007138) | 0.050379 / 0.038508 (0.011871) | 0.031238 / 0.023109 (0.008129) | 0.276839 / 0.275898 (0.000941) | 0.299488 / 0.323480 (-0.023992) | 0.005143 / 0.007986 (-0.002842) | 0.002725 / 0.004328 (-0.001604) | 0.048184 / 0.004250 (0.043934) | 0.046232 / 0.037052 (0.009180) | 0.287058 / 0.258489 (0.028569) | 0.322659 / 0.293841 (0.028818) | 0.047598 / 0.128546 (-0.080949) | 0.011116 / 0.075646 (-0.064530) | 0.058252 / 0.419271 (-0.361019) | 0.033404 / 0.043533 (-0.010128) | 0.277650 / 0.255139 (0.022511) | 0.295610 / 0.283200 (0.012410) | 0.018124 / 0.141683 (-0.123559) | 1.135052 / 1.452155 (-0.317103) | 1.194261 / 1.492716 (-0.298456) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095595 / 0.018006 (0.077588) | 0.306408 / 0.000490 (0.305918) | 0.000216 / 0.000200 (0.000016) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022027 / 0.037411 (-0.015385) | 0.076224 / 0.014526 (0.061698) | 0.087441 / 0.176557 (-0.089116) | 0.126636 / 0.737135 (-0.610499) | 0.089442 / 0.296338 (-0.206896) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.291315 / 0.215209 (0.076106) | 2.835304 / 2.077655 (0.757650) | 1.581102 / 1.504120 (0.076982) | 1.463046 / 1.541195 (-0.078149) | 1.481982 / 1.468490 (0.013492) | 0.559989 / 4.584777 (-4.024788) | 2.385262 / 3.745712 (-1.360450) | 2.773478 / 5.269862 (-2.496383) | 1.744427 / 4.565676 (-2.821249) | 0.062687 / 0.424275 (-0.361589) | 0.005149 / 0.007607 (-0.002458) | 0.374600 / 0.226044 (0.148555) | 3.376507 / 2.268929 (1.107579) | 1.935290 / 55.444624 (-53.509334) | 1.663227 / 6.876477 (-5.213250) | 1.678987 / 2.142072 (-0.463085) | 0.638970 / 4.805227 (-4.166258) | 0.120000 / 6.500664 (-6.380664) | 0.040862 / 0.075469 (-0.034608) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.008795 / 1.841788 (-0.832993) | 12.275084 / 8.074308 (4.200776) | 10.340088 / 10.191392 (0.148696) | 0.136454 / 0.680424 (-0.543970) | 0.014404 / 0.534201 (-0.519797) | 0.289478 / 0.579283 (-0.289805) | 0.279243 / 0.434364 (-0.155121) | 0.330992 / 0.540337 (-0.209346) | 0.422043 / 1.386936 (-0.964893) |

![](https://cml.dev/watermark.png#70633576ecf1f3f5e5cdfd8c9189246b3604f4b6 "CML watermark")


Slice splits support for datasets.load_from_disk

enhancement

New feature or request

### Feature request

Support for slice splits in `datasets.load_from_disk`, similar to how it's already supported for `datasets.load_dataset`.

### Motivation

Slice splits are convienient in a numer of cases - adding support to `datasets.load_from_disk` would make working with local datasets easier and homogenize the APIs of load_from_disk and load_dataset.

### Your contribution

Sure, if the devs think the feature request is sensible.


Hi! Does running the following code also return the same error on your machine?

```python
import copy
import pyarrow as pa
from datasets.table import InMemoryTable

copy.deepcopy(InMemoryTable(pa.table({"a": [1, 2, 3], "b": ["foo", "bar", "foobar"]})))
```

No, it doesn't, it runs fine.  But what's really strange is that the error just went away after I reran the data prep script for conversion from csv to a datasets object.  I realize that's not very helpful since the problem isn't reproducible.

### Describe the bug

```
Traceback (most recent call last):
  File "finetune.py", line 103, in <module>
    main(args)
  File "finetune.py", line 45, in main
    data_tokenized = data.map(partial(funcs.tokenize_function, tokenizer,
  File "/opt/conda/envs/ptca/lib/python3.8/site-packages/datasets/dataset_dict.py", line 868, in map
    {
  File "/opt/conda/envs/ptca/lib/python3.8/site-packages/datasets/dataset_dict.py", line 869, in <dictcomp>
    k: dataset.map(
  File "/opt/conda/envs/ptca/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 592, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/opt/conda/envs/ptca/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 557, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/opt/conda/envs/ptca/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 3093, in map
    for rank, done, content in Dataset._map_single(**dataset_kwargs):
  File "/opt/conda/envs/ptca/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 3432, in _map_single
    arrow_formatted_shard = shard.with_format("arrow")
  File "/opt/conda/envs/ptca/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2667, in with_format
    dataset = copy.deepcopy(self)
  File "/opt/conda/envs/ptca/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/opt/conda/envs/ptca/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/opt/conda/envs/ptca/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/opt/conda/envs/ptca/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/opt/conda/envs/ptca/lib/python3.8/copy.py", line 153, in deepcopy
    y = copier(memo)
  File "/opt/conda/envs/ptca/lib/python3.8/site-packages/datasets/table.py", line 176, in __deepcopy__
    memo[id(self._batches)] = list(self._batches)
AttributeError: 'InMemoryTable' object has no attribute '_batches'
```

### Steps to reproduce the bug

I'm running an MLOps flow using AzureML.
The error appears when I run the following function in my training script:

```python
data_tokenized = data.map(partial(funcs.tokenize_function, tokenizer,
                                      seq_length),
                              batched=True,
                              batch_size=batch_size,
                              remove_columns=['col1', 'col2'])
```

```python
def tokenize_function(tok, seq_length, example)
    # Pad so that each batch has the same sequence length
    inp = tok(example['col1'], padding=True, truncation=True)
    outp = tok(example['col2'], padding="max_length", max_length=seq_length)

    res = {
        'input_ids': inp['input_ids'],
        'attention_mask': inp['attention_mask'],
        'decoder_input_ids': outp['input_ids'],
        'labels': outp['input_ids'],
        'decoder_attention_mask': outp['attention_mask']
    }
    return res
```

### Expected behavior

Processing proceeds without errors.  I ran this same workflow 2 weeks ago without a problem.  I recreated the environment since then but it doesn't appear that datasets versions have changed since Dec. '23.

### Environment info

datasets 2.16.1
transformers 4.35.2
pyarrow 15.0.0
pyarrow-hotfix 0.6
torch 2.0.1

I'm not using the latest transformers version because there was an error due to a conflict with Azure mlflow when I tried the last time.