jsonlines files don't load with `load_dataset`

Hi @serenalotreck,

We use Apache Arrow `pyarrow` to read jsonlines and it throws an error when trying to load your data files:
```python
In [1]: import pyarrow as pa

In [2]: data = pa.json.read_json("train.jsonl")
---------------------------------------------------------------------------
ArrowInvalid                              Traceback (most recent call last)
<ipython-input-14-e9b104832528> in <module>
----> 1 data = pa.json.read_json("train.jsonl")

.../huggingface/datasets/venv/lib/python3.9/site-packages/pyarrow/_json.pyx in pyarrow._json.read_json()

.../huggingface/datasets/venv/lib/python3.9/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

.../huggingface/datasets/venv/lib/python3.9/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: JSON parse error: Column(/ner/[]/[]/[]) changed from number to string in row 0
```

I think it has to do with the data structure of the fields "ner" (and also "relations"):
```json
"ner": [
  [
    [0, 4, "Biochemical_process"],
    [15, 16, "Protein"]
  ],
```
Arrow interprets this data structure as an array, an arrays contain just a single data type:
- when reading sequentially, it finds first the `0` and infers that the data is of type `number`;
- when it finds the string `"Biochemical_process"`, it cannot cast it to number and throws the `ArrowInvalid` error

One solution could be to change the data structure of your data files. Any other ideas, @huggingface/datasets ?

Hi @albertvillanova,

Thanks for the explanation! To the best of my knowledge, arrays in a json [can contain multiple data types](https://docs.actian.com/ingres/11.2/index.html#page/SQLRef/Data_Types.htm), and I'm able to read these files with the `jsonlines` package. Is the requirement for arrays to only have one data type specific to PyArrow?

I'd prefer to keep the data structure as is, since it's a specific input requirement for the models this data was generated for. Any thoughts on how to enable the use of `load_dataset` with this dataset would be great!

Hi again @serenalotreck,

Yes, it is specific to PyArrow: as far as I know, Arrow does not support arrays with multiple data types.

As this is related specifically to your dataset structure (and not the `datasets` library), I have created a dedicated issue in your dataset page: https://huggingface.co/datasets/slotreck/pickle/discussions/1

Let's continue the discussion there! :hugs:

> Hi again @serenalotreck,
>
> Yes, it is specific to PyArrow: as far as I know, Arrow does not support arrays with multiple data types.
>
> As this is related specifically to your dataset structure (and not the `datasets` library), I have created a dedicated issue in your dataset page: https://huggingface.co/datasets/slotreck/pickle/discussions/1
>
> Let's continue the discussion there! 🤗

This is really terrible. My JSONL format data is very simple, but I still report this error
![image](https://github.com/huggingface/datasets/assets/58240629/e3fed922-ced4-406c-b5bc-90a4b891c4ee)
The error message is as follows：
  File "pyarrow/_json.pyx", line 290, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: JSON parse error: Column(/inputs) changed from string to number in row 208

### Describe the bug

While [the docs](https://huggingface.co/docs/datasets/upload_dataset#upload-dataset) seem to state that `.jsonl` is a supported extension for `datasets`, loading the dataset results in a `JSONDecodeError`.

### Steps to reproduce the bug

Code:
```
from datasets import load_dataset
dset = load_dataset('slotreck/pickle')
```

Traceback:
```
Downloading readme: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 925/925 [00:00<00:00, 3.11MB/s]
Downloading and preparing dataset json/slotreck--pickle to /mnt/home/lotrecks/.cache/huggingface/datasets/slotreck___json/slotreck--pickle-0c311f36ed032b04/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96...
Downloading data: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 589k/589k [00:00<00:00, 18.9MB/s]
Downloading data: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 104k/104k [00:00<00:00, 4.61MB/s]
Downloading data: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 170k/170k [00:00<00:00, 7.71MB/s]
Downloading data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00,  3.77it/s]
Extracting data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 523.92it/s]
Generating train split: 0 examples [00:00, ? examples/s]Failed to read file '/mnt/home/lotrecks/.cache/huggingface/datasets/downloads/6ec07bb2f279c9377036af6948532513fa8f48244c672d2644a2d7018ee5c9cb' with error <class 'pyarrow.lib.ArrowInvalid'>: JSON parse error: Column(/ner/[]/[]/[]) changed from number to string in row 0
Traceback (most recent call last):
  File "/mnt/home/lotrecks/anaconda3/envs/pickle/lib/python3.7/site-packages/datasets/packaged_modules/json/json.py", line 144, in _generate_tables
    dataset = json.load(f)
  File "/mnt/home/lotrecks/anaconda3/envs/pickle/lib/python3.7/json/__init__.py", line 296, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/mnt/home/lotrecks/anaconda3/envs/pickle/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/mnt/home/lotrecks/anaconda3/envs/pickle/lib/python3.7/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 3086)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/home/lotrecks/anaconda3/envs/pickle/lib/python3.7/site-packages/datasets/builder.py", line 1879, in _prepare_split_single
    for _, table in generator:
  File "/mnt/home/lotrecks/anaconda3/envs/pickle/lib/python3.7/site-packages/datasets/packaged_modules/json/json.py", line 147, in _generate_tables
    raise e
  File "/mnt/home/lotrecks/anaconda3/envs/pickle/lib/python3.7/site-packages/datasets/packaged_modules/json/json.py", line 122, in _generate_tables
    io.BytesIO(batch), read_options=paj.ReadOptions(block_size=block_size)
  File "pyarrow/_json.pyx", line 259, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: JSON parse error: Column(/ner/[]/[]/[]) changed from number to string in row 0

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/home/lotrecks/anaconda3/envs/pickle/lib/python3.7/site-packages/datasets/load.py", line 1815, in load_dataset
    storage_options=storage_options,
  File "/mnt/home/lotrecks/anaconda3/envs/pickle/lib/python3.7/site-packages/datasets/builder.py", line 913, in download_and_prepare
    **download_and_prepare_kwargs,
  File "/mnt/home/lotrecks/anaconda3/envs/pickle/lib/python3.7/site-packages/datasets/builder.py", line 1004, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/mnt/home/lotrecks/anaconda3/envs/pickle/lib/python3.7/site-packages/datasets/builder.py", line 1768, in _prepare_split
    gen_kwargs=gen_kwargs, job_id=job_id, **_prepare_split_args
  File "/mnt/home/lotrecks/anaconda3/envs/pickle/lib/python3.7/site-packages/datasets/builder.py", line 1912, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
```

### Expected behavior

For the dataset to be loaded without error.

### Environment info

- `datasets` version: 2.13.1
- Platform: Linux-3.10.0-1160.80.1.el7.x86_64-x86_64-with-centos-7.9.2009-Core
- Python version: 3.7.12
- Huggingface_hub version: 0.15.1
- PyArrow version: 8.0.0
- Pandas version: 1.3.5


Retrieve cached datasets that were pushed to hub when offline

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005292 / 0.011353 (-0.006061) | 0.003811 / 0.011008 (-0.007197) | 0.064912 / 0.038508 (0.026404) | 0.061199 / 0.023109 (0.038090) | 0.242953 / 0.275898 (-0.032945) | 0.271789 / 0.323480 (-0.051691) | 0.003994 / 0.007986 (-0.003991) | 0.002723 / 0.004328 (-0.001606) | 0.049952 / 0.004250 (0.045701) | 0.039489 / 0.037052 (0.002437) | 0.261143 / 0.258489 (0.002654) | 0.288800 / 0.293841 (-0.005041) | 0.028130 / 0.128546 (-0.100416) | 0.010724 / 0.075646 (-0.064922) | 0.208218 / 0.419271 (-0.211054) | 0.036224 / 0.043533 (-0.007309) | 0.247189 / 0.255139 (-0.007950) | 0.274702 / 0.283200 (-0.008498) | 0.019714 / 0.141683 (-0.121969) | 1.134853 / 1.452155 (-0.317301) | 1.192655 / 1.492716 (-0.300062) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096391 / 0.018006 (0.078385) | 0.303802 / 0.000490 (0.303312) | 0.000219 / 0.000200 (0.000019) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019530 / 0.037411 (-0.017881) | 0.061588 / 0.014526 (0.047062) | 0.075122 / 0.176557 (-0.101434) | 0.120980 / 0.737135 (-0.616155) | 0.075807 / 0.296338 (-0.220532) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.281672 / 0.215209 (0.066463) | 2.779884 / 2.077655 (0.702229) | 1.502026 / 1.504120 (-0.002094) | 1.369474 / 1.541195 (-0.171721) | 1.402694 / 1.468490 (-0.065796) | 0.559120 / 4.584777 (-4.025657) | 2.355320 / 3.745712 (-1.390393) | 2.823987 / 5.269862 (-2.445875) | 1.763888 / 4.565676 (-2.801788) | 0.061715 / 0.424275 (-0.362560) | 0.005015 / 0.007607 (-0.002592) | 0.342669 / 0.226044 (0.116625) | 3.360651 / 2.268929 (1.091722) | 1.887277 / 55.444624 (-53.557348) | 1.555613 / 6.876477 (-5.320864) | 1.614126 / 2.142072 (-0.527946) | 0.643797 / 4.805227 (-4.161430) | 0.118365 / 6.500664 (-6.382299) | 0.042596 / 0.075469 (-0.032873) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.951383 / 1.841788 (-0.890405) | 13.169812 / 8.074308 (5.095504) | 10.772460 / 10.191392 (0.581068) | 0.133248 / 0.680424 (-0.547176) | 0.014597 / 0.534201 (-0.519604) | 0.289758 / 0.579283 (-0.289525) | 0.266324 / 0.434364 (-0.168040) | 0.334811 / 0.540337 (-0.205526) | 0.445566 / 1.386936 (-0.941370) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005668 / 0.011353 (-0.005684) | 0.003583 / 0.011008 (-0.007425) | 0.050681 / 0.038508 (0.012173) | 0.063244 / 0.023109 (0.040135) | 0.279624 / 0.275898 (0.003726) | 0.308030 / 0.323480 (-0.015450) | 0.004160 / 0.007986 (-0.003826) | 0.002633 / 0.004328 (-0.001696) | 0.048475 / 0.004250 (0.044225) | 0.043106 / 0.037052 (0.006054) | 0.283678 / 0.258489 (0.025189) | 0.309730 / 0.293841 (0.015889) | 0.030290 / 0.128546 (-0.098256) | 0.011112 / 0.075646 (-0.064534) | 0.058234 / 0.419271 (-0.361038) | 0.033553 / 0.043533 (-0.009979) | 0.279902 / 0.255139 (0.024763) | 0.298041 / 0.283200 (0.014841) | 0.019367 / 0.141683 (-0.122316) | 1.142438 / 1.452155 (-0.309717) | 1.197305 / 1.492716 (-0.295411) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090875 / 0.018006 (0.072869) | 0.301174 / 0.000490 (0.300685) | 0.000216 / 0.000200 (0.000016) | 0.000053 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021544 / 0.037411 (-0.015867) | 0.071371 / 0.014526 (0.056846) | 0.080821 / 0.176557 (-0.095736) | 0.120054 / 0.737135 (-0.617082) | 0.082611 / 0.296338 (-0.213728) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.293787 / 0.215209 (0.078578) | 2.862610 / 2.077655 (0.784955) | 1.597282 / 1.504120 (0.093162) | 1.485094 / 1.541195 (-0.056101) | 1.507384 / 1.468490 (0.038893) | 0.558470 / 4.584777 (-4.026307) | 2.414137 / 3.745712 (-1.331575) | 2.863342 / 5.269862 (-2.406520) | 1.776973 / 4.565676 (-2.788704) | 0.062296 / 0.424275 (-0.361979) | 0.004954 / 0.007607 (-0.002653) | 0.346037 / 0.226044 (0.119993) | 3.441864 / 2.268929 (1.172935) | 1.969842 / 55.444624 (-53.474783) | 1.714878 / 6.876477 (-5.161599) | 1.738141 / 2.142072 (-0.403931) | 0.645929 / 4.805227 (-4.159298) | 0.117332 / 6.500664 (-6.383332) | 0.041963 / 0.075469 (-0.033507) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.983229 / 1.841788 (-0.858559) | 13.186932 / 8.074308 (5.112624) | 11.220549 / 10.191392 (1.029157) | 0.142105 / 0.680424 (-0.538319) | 0.015210 / 0.534201 (-0.518991) | 0.290055 / 0.579283 (-0.289228) | 0.274513 / 0.434364 (-0.159851) | 0.346834 / 0.540337 (-0.193504) | 0.575897 / 1.386936 (-0.811039) |

![](https://cml.dev/watermark.png#d3c0694d0c47a64a3cab5d468b4d9575ad7b1d96 "CML watermark")

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6459). All of your documentation changes will be reflected on that endpoint.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005308 / 0.011353 (-0.006045) | 0.003135 / 0.011008 (-0.007873) | 0.061820 / 0.038508 (0.023312) | 0.052005 / 0.023109 (0.028895) | 0.233507 / 0.275898 (-0.042391) | 0.257790 / 0.323480 (-0.065690) | 0.002848 / 0.007986 (-0.005138) | 0.002645 / 0.004328 (-0.001683) | 0.048379 / 0.004250 (0.044128) | 0.038320 / 0.037052 (0.001268) | 0.245470 / 0.258489 (-0.013019) | 0.274854 / 0.293841 (-0.018987) | 0.027335 / 0.128546 (-0.101211) | 0.010349 / 0.075646 (-0.065297) | 0.205872 / 0.419271 (-0.213400) | 0.035896 / 0.043533 (-0.007637) | 0.241645 / 0.255139 (-0.013494) | 0.260033 / 0.283200 (-0.023167) | 0.020325 / 0.141683 (-0.121358) | 1.116768 / 1.452155 (-0.335387) | 1.188067 / 1.492716 (-0.304649) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092622 / 0.018006 (0.074616) | 0.302663 / 0.000490 (0.302173) | 0.000227 / 0.000200 (0.000027) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018633 / 0.037411 (-0.018778) | 0.060117 / 0.014526 (0.045592) | 0.072713 / 0.176557 (-0.103844) | 0.119955 / 0.737135 (-0.617180) | 0.074698 / 0.296338 (-0.221640) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.277157 / 0.215209 (0.061948) | 2.699650 / 2.077655 (0.621995) | 1.413625 / 1.504120 (-0.090494) | 1.295900 / 1.541195 (-0.245295) | 1.306280 / 1.468490 (-0.162210) | 0.555354 / 4.584777 (-4.029423) | 2.386866 / 3.745712 (-1.358847) | 2.794069 / 5.269862 (-2.475793) | 1.736275 / 4.565676 (-2.829401) | 0.061812 / 0.424275 (-0.362464) | 0.004957 / 0.007607 (-0.002650) | 0.334533 / 0.226044 (0.108488) | 3.251096 / 2.268929 (0.982168) | 1.768193 / 55.444624 (-53.676431) | 1.473752 / 6.876477 (-5.402724) | 1.476320 / 2.142072 (-0.665753) | 0.642485 / 4.805227 (-4.162742) | 0.116986 / 6.500664 (-6.383678) | 0.042083 / 0.075469 (-0.033386) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.941364 / 1.841788 (-0.900424) | 11.587408 / 8.074308 (3.513100) | 10.500198 / 10.191392 (0.308806) | 0.129126 / 0.680424 (-0.551298) | 0.015206 / 0.534201 (-0.518995) | 0.286580 / 0.579283 (-0.292703) | 0.263566 / 0.434364 (-0.170798) | 0.331662 / 0.540337 (-0.208676) | 0.431423 / 1.386936 (-0.955513) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005151 / 0.011353 (-0.006202) | 0.003425 / 0.011008 (-0.007583) | 0.049301 / 0.038508 (0.010793) | 0.052005 / 0.023109 (0.028895) | 0.289594 / 0.275898 (0.013696) | 0.312630 / 0.323480 (-0.010849) | 0.003988 / 0.007986 (-0.003998) | 0.002705 / 0.004328 (-0.001624) | 0.048529 / 0.004250 (0.044279) | 0.039645 / 0.037052 (0.002592) | 0.293430 / 0.258489 (0.034941) | 0.311697 / 0.293841 (0.017856) | 0.029044 / 0.128546 (-0.099502) | 0.010282 / 0.075646 (-0.065364) | 0.057641 / 0.419271 (-0.361630) | 0.032733 / 0.043533 (-0.010800) | 0.293553 / 0.255139 (0.038414) | 0.308850 / 0.283200 (0.025651) | 0.018452 / 0.141683 (-0.123231) | 1.147931 / 1.452155 (-0.304224) | 1.173093 / 1.492716 (-0.319623) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.100862 / 0.018006 (0.082856) | 0.309286 / 0.000490 (0.308796) | 0.000223 / 0.000200 (0.000023) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021365 / 0.037411 (-0.016046) | 0.068987 / 0.014526 (0.054461) | 0.081092 / 0.176557 (-0.095465) | 0.119852 / 0.737135 (-0.617283) | 0.082850 / 0.296338 (-0.213489) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.288477 / 0.215209 (0.073268) | 2.833766 / 2.077655 (0.756111) | 1.576670 / 1.504120 (0.072550) | 1.431643 / 1.541195 (-0.109552) | 1.442132 / 1.468490 (-0.026358) | 0.556079 / 4.584777 (-4.028698) | 2.465042 / 3.745712 (-1.280670) | 2.786329 / 5.269862 (-2.483532) | 1.779428 / 4.565676 (-2.786249) | 0.062278 / 0.424275 (-0.361997) | 0.004867 / 0.007607 (-0.002740) | 0.348444 / 0.226044 (0.122399) | 3.389824 / 2.268929 (1.120896) | 1.919141 / 55.444624 (-53.525484) | 1.635411 / 6.876477 (-5.241066) | 1.654869 / 2.142072 (-0.487204) | 0.634467 / 4.805227 (-4.170761) | 0.114330 / 6.500664 (-6.386334) | 0.039900 / 0.075469 (-0.035569) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.970851 / 1.841788 (-0.870937) | 11.951660 / 8.074308 (3.877352) | 10.571115 / 10.191392 (0.379723) | 0.131040 / 0.680424 (-0.549384) | 0.015299 / 0.534201 (-0.518902) | 0.287851 / 0.579283 (-0.291432) | 0.278366 / 0.434364 (-0.155998) | 0.326468 / 0.540337 (-0.213870) | 0.552288 / 1.386936 (-0.834648) |

![](https://cml.dev/watermark.png#8214ff2a9f706427669a6c2a01ccabffa5bf0d2b "CML watermark")

I drafted the logic to retrieve a no-script dataset in the cache.
For example it can reload datasets that were pushed to hub if they exist in the cache.

example:

```python
>>> Dataset.from_dict({"a": [1, 2]}).push_to_hub("lhoestq/tmp")
>>> load_dataset("lhoestq/tmp")
DatasetDict({
    train: Dataset({
        features: ['a'],
        num_rows: 2
    })
})
```

and later, without connection:

```python
>>> load_dataset("lhoestq/tmp")
Using the latest cached version of the dataset from /Users/quentinlhoest/.cache/huggingface/datasets/lhoestq___tmp/*/*/0b3caccda1725efb(last modified on Wed Nov 29 16:50:27 2023) since it couldn't be found locally at lhoestq/tmp.
DatasetDict({
    train: Dataset({
        features: ['a'],
        num_rows: 2
    })
})
```

fix https://github.com/huggingface/datasets/issues/3547

## Implementation details (EDITED)

I continued in https://github.com/huggingface/datasets/pull/6493, see the changes there

TODO:
- [x] tests
- [ ] compatible with https://github.com/huggingface/datasets/pull/6458