Support the deserialization of json lines files comprised of lists

enhancement

New feature or request

Update: I ended up deciding to go back to use lines of dictionaries instead of arrays, not because of this issue as my users would be capable of downloading my corpus without `datasets`, but the speed and storage savings are not currently worth breaking my API and harming the backwards compatibility of each new revision.

With that said, for a static dataset that is not regularly updated like mine, and particularly for extremely large datasets with millions or billions of rows, using arrays could have a meaningful impact, and so there is probably still value in supporting this structure, provided the effort is not too much.

### Feature request

I manage a somewhat large and popular Hugging Face dataset known as the [Open Australian Legal Corpus](https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus). I recently updated my corpus to be stored in a json lines file where each line is an array and each element represents a value at a particular column. Previously, my corpus was stored as a json lines file where each line was a dictionary and the keys were the fields.

Essentially, a line in my json lines file used to look like this:
```json
{"version_id":"","type":"","jurisdiction":"","source":"","citation":"","url":"","when_scraped":"","text":""}
```

And now it looks like this:
```json
["","","","","","","",""]
```

This saves 65 bytes per document and allows me very quickly serialise and deserialise documents via `msgspec`.

After making this change, I found that `datasets` was incapable of deserialising my Corpus without a custom loading script, even if I ensured that the `dataset_info` field in my dataset card contained the desired names of my features.

I would like to request that functionality be added to support this format which is more memory-efficent and faster than using dictionaries.

### Motivation

The [documentation](https://huggingface.co/docs/datasets/en/dataset_script) for creating dataset loading scripts asserts that:
> In the next major release, the new safety features of 🤗 Datasets will disable running dataset loading scripts by default, and you will have to pass trust_remote_code=True to load datasets that require running a dataset script.

I would rather not require my users to pass `trust_remote_code=True` which means that I will need built-in support for this format.

### Your contribution

I would be happy to submit a PR for this if this is something you would incorporate into `datasets` and if I can be pointed to where the code would need to go.


irc_disentangle - Issue with splitting data

Thank you I will try this out!

On Tue, Jun 11, 2024 at 3:55 AM Vincent Lau ***@***.***>
wrote:

> I add a "streaming=True" after the name of the dataset, and it
> works.....hope it can help you
>
> And if you install the version datasets==2.15.0, this bug will not happen.
> I don't know why, but all of them works
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/6906#issuecomment-2160041812>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A3HXU7AMBT2MNO34SC3Z5G3ZG2UOXAVCNFSM6AAAAABH45CNPWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRQGA2DCOBRGI>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>

I still find out that there are some strange bug in v2.15.0 of datasets. it seems like that the *.arrow file cannot be established. it may be an index of the subsets. well I still try to debug it. but, one of the most efficient way may be using the google colab to build this index in the ~/huggingface/datasets, and than download them to replace the local file.....lol......it works!

Yeah I did try what you suggested and it didn’t work. I was able to get it
on a local from someone who access the dataset in the past. Let me know
when you end up fixing this bug.

On Tue, Jun 11, 2024 at 10:33 PM Vincent Lau ***@***.***>
wrote:

> I still find out that there are some strange bug in v2.15.0 of datasets.
> it seems like that the *.arrow file cannot be established. it may be an
> index of the subsets. well I still try to debug it. but, one of the most
> efficient way may be using the google colab to build this index in the
> ~/huggingface/datasets, and than download them to replace the local
> file.....lol......it works!
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/6906#issuecomment-2161988798>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A3HXU7BCJE2LOCWRVWPMNODZG6XPJAVCNFSM6AAAAABH45CNPWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRRHE4DQNZZHA>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>

### Describe the bug

I am trying to access your database through python using "datasets.load_dataset("irc_disentangle")" and I am getting this error message:

ValueError: Instruction "train" corresponds to no data!

### Steps to reproduce the bug

import datasets
ds = datasets.load_dataset('irc_disentangle')
ds

### Expected behavior

The data is supposed to load into ds and be accessable as such:
ds['train'][1050], ds['train'][1055]

### Environment info

I tired Python 3.12 and 3.10


Extraction protocol for arrow files is not defined

### Describe the bug

Passing files with `.arrow` extension into data_files argument, at least when `streaming=True` is very slow.

### Steps to reproduce the bug

 Basically it goes through the `_get_extraction_protocol` method located [here](https://github.com/huggingface/datasets/blob/main/src/datasets/utils/file_utils.py#L820)
The method then looks at some base known extensions where `arrow` is not defined so it proceeds to determine the compression with the magic number method which is slow when dealing with a lot of files which are stored in s3 and by looking at this predefined list, I don't see `arrow` in there either so in the end it return None:

```
MAGIC_NUMBER_TO_COMPRESSION_PROTOCOL = {
    bytes.fromhex("504B0304"): "zip",
    bytes.fromhex("504B0506"): "zip",  # empty archive
    bytes.fromhex("504B0708"): "zip",  # spanned archive
    bytes.fromhex("425A68"): "bz2",
    bytes.fromhex("1F8B"): "gzip",
    bytes.fromhex("FD377A585A00"): "xz",
    bytes.fromhex("04224D18"): "lz4",
    bytes.fromhex("28B52FFD"): "zstd",
}
```

### Expected behavior

My expectation is that `arrow` would be in the known lists so it would return None without going through the magic number method.

### Environment info

datasets 2.19.0


Fix decoding multi part extension

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6904). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

takign the liberty to merge this for the viewer and a new dataset being released

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005004 / 0.011353 (-0.006349) | 0.003352 / 0.011008 (-0.007657) | 0.063035 / 0.038508 (0.024527) | 0.032031 / 0.023109 (0.008922) | 0.244801 / 0.275898 (-0.031097) | 0.270622 / 0.323480 (-0.052857) | 0.003110 / 0.007986 (-0.004876) | 0.002629 / 0.004328 (-0.001700) | 0.048784 / 0.004250 (0.044534) | 0.045779 / 0.037052 (0.008726) | 0.258642 / 0.258489 (0.000153) | 0.291606 / 0.293841 (-0.002235) | 0.028237 / 0.128546 (-0.100310) | 0.010184 / 0.075646 (-0.065463) | 0.202455 / 0.419271 (-0.216816) | 0.036012 / 0.043533 (-0.007521) | 0.248209 / 0.255139 (-0.006930) | 0.267315 / 0.283200 (-0.015884) | 0.019249 / 0.141683 (-0.122434) | 1.120420 / 1.452155 (-0.331735) | 1.169515 / 1.492716 (-0.323201) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095193 / 0.018006 (0.077187) | 0.300544 / 0.000490 (0.300055) | 0.000214 / 0.000200 (0.000014) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019001 / 0.037411 (-0.018411) | 0.061857 / 0.014526 (0.047331) | 0.073379 / 0.176557 (-0.103178) | 0.121293 / 0.737135 (-0.615843) | 0.075665 / 0.296338 (-0.220673) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.285153 / 0.215209 (0.069944) | 2.875527 / 2.077655 (0.797873) | 1.479851 / 1.504120 (-0.024269) | 1.360691 / 1.541195 (-0.180504) | 1.385581 / 1.468490 (-0.082909) | 0.566312 / 4.584777 (-4.018465) | 2.400202 / 3.745712 (-1.345510) | 2.719241 / 5.269862 (-2.550620) | 1.706469 / 4.565676 (-2.859208) | 0.062129 / 0.424275 (-0.362146) | 0.005291 / 0.007607 (-0.002316) | 0.334585 / 0.226044 (0.108540) | 3.293347 / 2.268929 (1.024419) | 1.790490 / 55.444624 (-53.654134) | 1.505519 / 6.876477 (-5.370958) | 1.527730 / 2.142072 (-0.614343) | 0.644554 / 4.805227 (-4.160673) | 0.119775 / 6.500664 (-6.380889) | 0.056912 / 0.075469 (-0.018557) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.977512 / 1.841788 (-0.864275) | 11.293883 / 8.074308 (3.219575) | 9.669439 / 10.191392 (-0.521953) | 0.129910 / 0.680424 (-0.550514) | 0.014322 / 0.534201 (-0.519879) | 0.284967 / 0.579283 (-0.294316) | 0.265355 / 0.434364 (-0.169008) | 0.321965 / 0.540337 (-0.218372) | 0.415254 / 1.386936 (-0.971682) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005138 / 0.011353 (-0.006215) | 0.003321 / 0.011008 (-0.007687) | 0.049731 / 0.038508 (0.011223) | 0.032307 / 0.023109 (0.009198) | 0.266331 / 0.275898 (-0.009567) | 0.290863 / 0.323480 (-0.032617) | 0.004151 / 0.007986 (-0.003835) | 0.002684 / 0.004328 (-0.001644) | 0.048760 / 0.004250 (0.044510) | 0.042251 / 0.037052 (0.005199) | 0.280414 / 0.258489 (0.021925) | 0.305089 / 0.293841 (0.011248) | 0.029118 / 0.128546 (-0.099428) | 0.010276 / 0.075646 (-0.065370) | 0.057790 / 0.419271 (-0.361482) | 0.033290 / 0.043533 (-0.010243) | 0.267250 / 0.255139 (0.012111) | 0.285233 / 0.283200 (0.002034) | 0.018587 / 0.141683 (-0.123096) | 1.136198 / 1.452155 (-0.315957) | 1.185274 / 1.492716 (-0.307442) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096355 / 0.018006 (0.078349) | 0.301827 / 0.000490 (0.301337) | 0.000216 / 0.000200 (0.000016) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022607 / 0.037411 (-0.014805) | 0.075724 / 0.014526 (0.061198) | 0.088197 / 0.176557 (-0.088359) | 0.127864 / 0.737135 (-0.609271) | 0.089294 / 0.296338 (-0.207044) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.289321 / 0.215209 (0.074112) | 2.832456 / 2.077655 (0.754802) | 1.559208 / 1.504120 (0.055088) | 1.426229 / 1.541195 (-0.114966) | 1.424564 / 1.468490 (-0.043926) | 0.557754 / 4.584777 (-4.027023) | 0.940179 / 3.745712 (-2.805533) | 2.713640 / 5.269862 (-2.556222) | 1.697583 / 4.565676 (-2.868093) | 0.062024 / 0.424275 (-0.362251) | 0.005270 / 0.007607 (-0.002337) | 0.339450 / 0.226044 (0.113406) | 3.333024 / 2.268929 (1.064096) | 1.946087 / 55.444624 (-53.498537) | 1.601057 / 6.876477 (-5.275420) | 1.599862 / 2.142072 (-0.542210) | 0.642838 / 4.805227 (-4.162390) | 0.120470 / 6.500664 (-6.380194) | 0.040815 / 0.075469 (-0.034654) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.012904 / 1.841788 (-0.828884) | 11.917035 / 8.074308 (3.842727) | 9.717822 / 10.191392 (-0.473570) | 0.141730 / 0.680424 (-0.538694) | 0.015750 / 0.534201 (-0.518451) | 0.284470 / 0.579283 (-0.294813) | 0.125662 / 0.434364 (-0.308702) | 0.380740 / 0.540337 (-0.159598) | 0.418119 / 1.386936 (-0.968817) |

![](https://cml.dev/watermark.png#b3f772468b2bbf77a7510e265f9d41e9eb77d53f "CML watermark")

e.g. a field named `url.txt` should be a treated as text

I also included a small fix to support .npz correctly


Add the option of saving in parquet instead of arrow

New feature or request

I think [`Dataset.to_parquet`](https://huggingface.co/docs/datasets/v1.10.2/package_reference/main_classes.html#datasets.Dataset.to_parquet) is what you're looking for.

Let me know if I'm wrong

No, it does not save the metadata json.

We have to recode all meta json load/save
with another custome functions.

save_to_disk
and load should have option with
“Parquet” instead of “arrow”

since “arrow” is never user for production
(only parquet).

Thanks !

> On May 17, 2024, at 5:38, Frédéric Branchaud-Charron ***@***.***> wrote:
>
>
> I think Dataset.to_parquet is what you're looking for.
>
> Let me know if I'm wrong
>
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you authored the thread.

You can use `to_parquet` and `ds.info.write_to_directory()` to save the dataset info

Ok,

What about loading ?

Should we do in 2 steps ?

> On Jun 14, 2024, at 1:09, Quentin Lhoest ***@***.***> wrote:
>
>
> You can use to_parquet and ds.info.write_to_directory() to save the dataset info
>
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you authored the thread.

Isn’t easier to combine both
into load_dataset and save_dataset
with parquet options.

2) another question,
How can we download large dataset into disk directly without loading all in memory (!)

> On Jun 14, 2024, at 19:54, Quentin Lhoest ***@***.***> wrote:
>
>
> Yes, and there is DatasetInfo.from_directory(). to reload the info
>
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you authored the thread.

`load_dataset` doesn't load the dataset in memory, it progressively writes to disk in Arrow format and then memory maps the Arrow files. This allows to load datasets bigger than memory and without filling your RAM

Sure.
How memory map is managed ?
Managed by the OS ?

Why the need of save_dataset() ?

> On Jun 15, 2024, at 0:06, Quentin Lhoest ***@***.***> wrote:
>
>
> load_dataset doesn't load the dataset in memory, it progressively writes to disk in Arrow format and then memory maps the Arrow files. This allows to load datasets bigger than memory and without filling your RAM
>
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you authored the thread.

### Feature request

In dataset.save_to_disk('/path/to/save/dataset'),

add the option to save in parquet format

dataset.save_to_disk('/path/to/save/dataset', format="parquet"),

because arrow is not used for Production Big data.... (only parquet)

### Motivation

because arrow is not used for Production Big data.... (only parquet)

### Your contribution

I can do the testing !


Make CLI convert_to_parquet not raise error if no rights to create script branch

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6902). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005026 / 0.011353 (-0.006327) | 0.003672 / 0.011008 (-0.007336) | 0.062776 / 0.038508 (0.024268) | 0.032056 / 0.023109 (0.008947) | 0.245359 / 0.275898 (-0.030540) | 0.269371 / 0.323480 (-0.054109) | 0.004205 / 0.007986 (-0.003780) | 0.002774 / 0.004328 (-0.001555) | 0.048958 / 0.004250 (0.044708) | 0.046442 / 0.037052 (0.009390) | 0.263924 / 0.258489 (0.005434) | 0.291854 / 0.293841 (-0.001987) | 0.027299 / 0.128546 (-0.101248) | 0.010332 / 0.075646 (-0.065315) | 0.202677 / 0.419271 (-0.216595) | 0.037732 / 0.043533 (-0.005801) | 0.246028 / 0.255139 (-0.009111) | 0.272100 / 0.283200 (-0.011099) | 0.018497 / 0.141683 (-0.123186) | 1.101192 / 1.452155 (-0.350962) | 1.149683 / 1.492716 (-0.343033) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.097838 / 0.018006 (0.079832) | 0.305598 / 0.000490 (0.305108) | 0.000230 / 0.000200 (0.000030) | 0.000045 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019489 / 0.037411 (-0.017922) | 0.061902 / 0.014526 (0.047376) | 0.074825 / 0.176557 (-0.101732) | 0.121664 / 0.737135 (-0.615472) | 0.076440 / 0.296338 (-0.219898) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.279194 / 0.215209 (0.063985) | 2.756777 / 2.077655 (0.679123) | 1.429298 / 1.504120 (-0.074822) | 1.313423 / 1.541195 (-0.227771) | 1.340466 / 1.468490 (-0.128024) | 0.556349 / 4.584777 (-4.028428) | 2.355910 / 3.745712 (-1.389802) | 2.806733 / 5.269862 (-2.463128) | 1.741903 / 4.565676 (-2.823773) | 0.061556 / 0.424275 (-0.362719) | 0.005477 / 0.007607 (-0.002130) | 0.327856 / 0.226044 (0.101812) | 3.283092 / 2.268929 (1.014164) | 1.797776 / 55.444624 (-53.646848) | 1.498683 / 6.876477 (-5.377794) | 1.518501 / 2.142072 (-0.623572) | 0.632267 / 4.805227 (-4.172960) | 0.116505 / 6.500664 (-6.384159) | 0.042446 / 0.075469 (-0.033023) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.982841 / 1.841788 (-0.858947) | 11.709436 / 8.074308 (3.635128) | 9.570519 / 10.191392 (-0.620873) | 0.141968 / 0.680424 (-0.538456) | 0.014299 / 0.534201 (-0.519902) | 0.285101 / 0.579283 (-0.294182) | 0.267118 / 0.434364 (-0.167246) | 0.324720 / 0.540337 (-0.215617) | 0.423626 / 1.386936 (-0.963310) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005567 / 0.011353 (-0.005786) | 0.003703 / 0.011008 (-0.007306) | 0.050516 / 0.038508 (0.012008) | 0.032617 / 0.023109 (0.009508) | 0.276546 / 0.275898 (0.000648) | 0.299798 / 0.323480 (-0.023682) | 0.004282 / 0.007986 (-0.003704) | 0.002719 / 0.004328 (-0.001609) | 0.049424 / 0.004250 (0.045173) | 0.042924 / 0.037052 (0.005871) | 0.287785 / 0.258489 (0.029296) | 0.315490 / 0.293841 (0.021649) | 0.029533 / 0.128546 (-0.099013) | 0.010575 / 0.075646 (-0.065071) | 0.058210 / 0.419271 (-0.361061) | 0.033269 / 0.043533 (-0.010263) | 0.273325 / 0.255139 (0.018186) | 0.291762 / 0.283200 (0.008563) | 0.018922 / 0.141683 (-0.122761) | 1.118913 / 1.452155 (-0.333242) | 1.175554 / 1.492716 (-0.317162) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.099920 / 0.018006 (0.081914) | 0.317188 / 0.000490 (0.316698) | 0.000211 / 0.000200 (0.000011) | 0.000045 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022297 / 0.037411 (-0.015114) | 0.077775 / 0.014526 (0.063249) | 0.090239 / 0.176557 (-0.086317) | 0.130498 / 0.737135 (-0.606638) | 0.092010 / 0.296338 (-0.204328) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.293534 / 0.215209 (0.078325) | 2.866070 / 2.077655 (0.788415) | 1.547147 / 1.504120 (0.043027) | 1.419684 / 1.541195 (-0.121510) | 1.432128 / 1.468490 (-0.036362) | 0.571365 / 4.584777 (-4.013412) | 0.968879 / 3.745712 (-2.776833) | 2.797415 / 5.269862 (-2.472446) | 1.767821 / 4.565676 (-2.797856) | 0.063281 / 0.424275 (-0.360994) | 0.005072 / 0.007607 (-0.002535) | 0.344547 / 0.226044 (0.118502) | 3.383888 / 2.268929 (1.114959) | 1.879537 / 55.444624 (-53.565087) | 1.598392 / 6.876477 (-5.278085) | 1.627788 / 2.142072 (-0.514284) | 0.641199 / 4.805227 (-4.164028) | 0.116349 / 6.500664 (-6.384315) | 0.041940 / 0.075469 (-0.033529) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.002494 / 1.841788 (-0.839294) | 12.310056 / 8.074308 (4.235748) | 9.819718 / 10.191392 (-0.371674) | 0.134745 / 0.680424 (-0.545679) | 0.016223 / 0.534201 (-0.517978) | 0.284791 / 0.579283 (-0.294492) | 0.124665 / 0.434364 (-0.309699) | 0.381601 / 0.540337 (-0.158737) | 0.413007 / 1.386936 (-0.973929) |

![](https://cml.dev/watermark.png#6255b36be14ae22890c78749575f1f0793901f14 "CML watermark")

Make CLI convert_to_parquet not raise error if no rights to create "script" branch.

Not that before this PR, the error was not critical because it was raised at the end of the script, once all the rest of the steps were already performed.

Fix #6901.

Bug introduced in datasets-2.19.0 by:
- #6809


HTTPError 403 raised by CLI convert_to_parquet when creating script branch on 3rd party repos

bug

Something isn't working

CLI convert_to_parquet cannot create "script" branch on 3rd party repos.

It can only create it on repos where the user executing the script has write access.

Otherwise, a 403 Forbidden HTTPError is raised:
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_errors.py", line 304, in hf_raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://huggingface.co/api/datasets/ORG/DATASET/branch/script

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/datasets-cli", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/datasets/commands/datasets_cli.py", line 41, in main
    service.run()
  File "/usr/local/lib/python3.10/dist-packages/datasets/commands/convert_to_parquet.py", line 92, in run
    create_branch(dataset_id, branch="script", repo_type="dataset", token=token, exist_ok=True)
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/hf_api.py", line 5503, in create_branch
    hf_raise_for_status(response)
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_errors.py", line 367, in hf_raise_for_status
    raise HfHubHTTPError(message, response=response) from e
huggingface_hub.utils._errors.HfHubHTTPError:  (Request ID: Root=1-6645ee0d-4db1ed8a1fbe04956be15897;139a6e23-df7d-4f62-b5ba-adb6d8e6e696)

403 Forbidden: Forbidden: cannot write to script.
Cannot access content at: https://huggingface.co/api/datasets/ORG/DATASET/branch/script.
If you are trying to create or update content,make sure you have a token with the `write` role.
```


[WebDataset] KeyError with user-defined `Features` when a field is missing in an example

It shouldn't be difficult, I think it's just a matter of adding the missing fields from `self.config.features` in `example` here: before it iterates on image_field_names and audio_field_names. A missing field should have a value set to None

https://github.com/huggingface/datasets/blob/768cb35ede5a6c35fa7545aa3671f3e321c96440/src/datasets/packaged_modules/webdataset/webdataset.py#L113-L116

@lhoestq So like this then?

```
def _generate_examples(self, tar_paths, tar_iterators):
        image_field_names = [
            field_name for field_name, feature in self.info.features.items() if isinstance(feature, datasets.Image)
        ]
        audio_field_names = [
            field_name for field_name, feature in self.info.features.items() if isinstance(feature, datasets.Audio)
        ]

        all_field_names = list(self.config.features.keys())

        for tar_idx, (tar_path, tar_iterator) in enumerate(zip(tar_paths, tar_iterators)):
            for example_idx, example in enumerate(self._get_pipeline_from_tar(tar_path, tar_iterator)):
                for field_name in all_field_names:
                    if field_name not in example:
                        if field_name in self.config.features:
                            example[field_name] = self.config.features[field_name]
                        else:
                            example[field_name] = None

                # Process image and audio fields
                for field_name in image_field_names + audio_field_names:
                    if example[field_name] is not None:
                        example[field_name] = {"path": example["__key__"] + "." + field_name, "bytes": example[field_name]}

                yield f"{tar_idx}_{example_idx}", example
```

Or should we avoid trying add the missing values and just set them to None?

```
            for field_name in all_field_names:
                if field_name not in example:
                    example[field_name] = None
```

reported at https://huggingface.co/datasets/ProGamerGov/synthetic-dataset-1m-dalle3-high-quality-captions/discussions/1

```
File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/packaged_modules/webdataset/webdataset.py", line 109, in _generate_examples
    example[field_name] = {"path": example["__key__"] + "." + field_name, "bytes": example[field_name]}
```


List of dictionary features get standardized

### Describe the bug

Hi, i’m trying to create a HF dataset from a list using Dataset.from_list.

Each sample in the list is a dict with the same keys (which will be my features). The values for each feature are a list of dictionaries, and each such dictionary has a different set of keys. However, the datasets library standardizes all dictionaries under a feature and adds all possible keys (with None value) from all the dictionaries under that feature.

How can I keep the same set of keys as in the original list for each dictionary under a feature?

### Steps to reproduce the bug

```
from datasets import Dataset

# Define a function to generate a sample with "tools" feature
def generate_sample():
    # Generate random sample data
    sample_data = {
        "text": "Sample text",
        "feature_1": []
    }

    # Add feature_1 with random keys for this sample
    feature_1 = [{"key1": "value1"}, {"key2": "value2"}]  # Example feature_1 with random keys
    sample_data["feature_1"].extend(feature_1)

    return sample_data

# Generate multiple samples
num_samples = 10
samples = [generate_sample() for _ in range(num_samples)]

# Create a Hugging Face Dataset
dataset = Dataset.from_list(samples)
dataset[0]
```

```{'text': 'Sample text', 'feature_1': [{'key1': 'value1', 'key2': None}, {'key1': None, 'key2': 'value2'}]}```

### Expected behavior

```{'text': 'Sample text', 'feature_1': [{'key1': 'value1'}, {'key2': 'value2'}]}```

### Environment info

- `datasets` version: 2.19.1
- Platform: Linux-5.15.0-1040-nvidia-x86_64-with-glibc2.35
- Python version: 3.10.13
- `huggingface_hub` version: 0.23.0
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6898). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

After this PR, the README file looks like:

![Screenshot from 2024-05-14 14-19-29](https://github.com/huggingface/datasets/assets/8515462/1f665a06-98be-4dd7-ba7e-7cc025489503)

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004936 / 0.011353 (-0.006417) | 0.003591 / 0.011008 (-0.007418) | 0.062967 / 0.038508 (0.024459) | 0.031314 / 0.023109 (0.008205) | 0.248040 / 0.275898 (-0.027858) | 0.271630 / 0.323480 (-0.051850) | 0.003085 / 0.007986 (-0.004901) | 0.002605 / 0.004328 (-0.001724) | 0.049452 / 0.004250 (0.045202) | 0.044929 / 0.037052 (0.007876) | 0.264254 / 0.258489 (0.005765) | 0.287531 / 0.293841 (-0.006310) | 0.027197 / 0.128546 (-0.101349) | 0.009925 / 0.075646 (-0.065721) | 0.203165 / 0.419271 (-0.216107) | 0.035658 / 0.043533 (-0.007875) | 0.250207 / 0.255139 (-0.004932) | 0.269258 / 0.283200 (-0.013941) | 0.019975 / 0.141683 (-0.121708) | 1.093703 / 1.452155 (-0.358452) | 1.134031 / 1.492716 (-0.358685) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095089 / 0.018006 (0.077082) | 0.301410 / 0.000490 (0.300920) | 0.000251 / 0.000200 (0.000051) | 0.000047 / 0.000054 (-0.000007) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018453 / 0.037411 (-0.018958) | 0.061674 / 0.014526 (0.047148) | 0.073442 / 0.176557 (-0.103114) | 0.119743 / 0.737135 (-0.617392) | 0.074518 / 0.296338 (-0.221820) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.276351 / 0.215209 (0.061142) | 2.757670 / 2.077655 (0.680015) | 1.471199 / 1.504120 (-0.032921) | 1.363620 / 1.541195 (-0.177575) | 1.374175 / 1.468490 (-0.094315) | 0.556444 / 4.584777 (-4.028333) | 2.340637 / 3.745712 (-1.405075) | 2.728341 / 5.269862 (-2.541521) | 1.701214 / 4.565676 (-2.864463) | 0.061832 / 0.424275 (-0.362443) | 0.005287 / 0.007607 (-0.002320) | 0.331848 / 0.226044 (0.105804) | 3.334204 / 2.268929 (1.065276) | 1.791203 / 55.444624 (-53.653421) | 1.512246 / 6.876477 (-5.364231) | 1.529570 / 2.142072 (-0.612503) | 0.632193 / 4.805227 (-4.173034) | 0.116512 / 6.500664 (-6.384153) | 0.041271 / 0.075469 (-0.034198) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.981813 / 1.841788 (-0.859974) | 11.271398 / 8.074308 (3.197090) | 9.654613 / 10.191392 (-0.536780) | 0.140235 / 0.680424 (-0.540188) | 0.014336 / 0.534201 (-0.519865) | 0.284286 / 0.579283 (-0.294997) | 0.260265 / 0.434364 (-0.174099) | 0.321064 / 0.540337 (-0.219274) | 0.417554 / 1.386936 (-0.969382) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005265 / 0.011353 (-0.006088) | 0.003237 / 0.011008 (-0.007772) | 0.049723 / 0.038508 (0.011215) | 0.031705 / 0.023109 (0.008596) | 0.255548 / 0.275898 (-0.020350) | 0.281651 / 0.323480 (-0.041829) | 0.004099 / 0.007986 (-0.003886) | 0.002739 / 0.004328 (-0.001589) | 0.049713 / 0.004250 (0.045463) | 0.041563 / 0.037052 (0.004511) | 0.269500 / 0.258489 (0.011011) | 0.293948 / 0.293841 (0.000107) | 0.029259 / 0.128546 (-0.099287) | 0.010391 / 0.075646 (-0.065255) | 0.057772 / 0.419271 (-0.361500) | 0.033125 / 0.043533 (-0.010408) | 0.258838 / 0.255139 (0.003699) | 0.278616 / 0.283200 (-0.004584) | 0.017543 / 0.141683 (-0.124139) | 1.130319 / 1.452155 (-0.321835) | 1.185976 / 1.492716 (-0.306740) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094827 / 0.018006 (0.076821) | 0.296820 / 0.000490 (0.296331) | 0.000212 / 0.000200 (0.000012) | 0.000046 / 0.000054 (-0.000008) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022583 / 0.037411 (-0.014828) | 0.076318 / 0.014526 (0.061792) | 0.087435 / 0.176557 (-0.089121) | 0.127351 / 0.737135 (-0.609784) | 0.089051 / 0.296338 (-0.207287) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.289476 / 0.215209 (0.074267) | 2.842065 / 2.077655 (0.764410) | 1.536857 / 1.504120 (0.032737) | 1.393914 / 1.541195 (-0.147281) | 1.392636 / 1.468490 (-0.075854) | 0.570299 / 4.584777 (-4.014478) | 0.982246 / 3.745712 (-2.763466) | 2.758773 / 5.269862 (-2.511088) | 1.728615 / 4.565676 (-2.837062) | 0.063944 / 0.424275 (-0.360331) | 0.005014 / 0.007607 (-0.002593) | 0.347474 / 0.226044 (0.121430) | 3.398092 / 2.268929 (1.129164) | 1.855134 / 55.444624 (-53.589491) | 1.568705 / 6.876477 (-5.307772) | 1.574201 / 2.142072 (-0.567871) | 0.649466 / 4.805227 (-4.155761) | 0.116330 / 6.500664 (-6.384334) | 0.040730 / 0.075469 (-0.034739) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.000675 / 1.841788 (-0.841113) | 11.899660 / 8.074308 (3.825352) | 9.913335 / 10.191392 (-0.278058) | 0.132517 / 0.680424 (-0.547907) | 0.016467 / 0.534201 (-0.517734) | 0.282221 / 0.579283 (-0.297062) | 0.125205 / 0.434364 (-0.309159) | 0.374986 / 0.540337 (-0.165351) | 0.418666 / 1.386936 (-0.968270) |

![](https://cml.dev/watermark.png#e2f989d01b49e3d6f98b2014d9ece3307e885b7a "CML watermark")

Fix YAML error in README files appearing on GitHub.

See error message:
![Screenshot from 2024-05-14 06-58-02](https://github.com/huggingface/datasets/assets/8515462/7984cc4e-96ee-4e83-99a4-4c0c5791fa05)

Fix #6897.


datasets template guide :: issue in documentation YAML

Hello, @bghira.

Thanks for reporting. Please note that the text originating the error is not supposed to be valid YAML: it contains the instructions to generate the actual YAML content, that should replace the instructions comment.

On the other hand, I agree that it is not nice to have that YAML error message at the top of the page:
![Screenshot from 2024-05-14 06-58-02](https://github.com/huggingface/datasets/assets/8515462/28409eb4-99e7-4b24-8eaa-21a65a8f23b2)

I am proposing a change to make the YAML error disappear.

thanks albert! i looked at it for a while to figure it out. i think the `raw` view option is the correct way to look at it?

### Describe the bug

There is a YAML error at the top of the page, and I don't think it's supposed to be there

### Steps to reproduce the bug

1. Browse to [this tutorial document](https://github.com/huggingface/datasets/blob/main/templates/README_guide.md)
2. Observe a big red error at the top
3. The rest of the document remains functional

### Expected behavior

I think the YAML block should be displayed or ignored.

### Environment info

N/A


Regression bug: `NonMatchingSplitsSizesError` for (possibly) overwritten dataset

### Describe the bug

While trying to load the dataset `https://huggingface.co/datasets/pysentimiento/spanish-tweets-small`, I get this error:

```python
---------------------------------------------------------------------------
NonMatchingSplitsSizesError               Traceback (most recent call last)
[<ipython-input-1-d6a3c721d3b8>](https://localhost:8080/#) in <cell line: 3>()
      1 from datasets import load_dataset
      2
----> 3 ds = load_dataset("pysentimiento/spanish-tweets-small")

3 frames
[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://localhost:8080/#) in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, ignore_verifications, keep_in_memory, save_infos, revision, token, use_auth_token, task, streaming, num_proc, storage_options, **config_kwargs)
   2150
   2151     # Download and prepare data
-> 2152     builder_instance.download_and_prepare(
   2153         download_config=download_config,
   2154         download_mode=download_mode,

[/usr/local/lib/python3.10/dist-packages/datasets/builder.py](https://localhost:8080/#) in download_and_prepare(self, output_dir, download_config, download_mode, verification_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, file_format, max_shard_size, num_proc, storage_options, **download_and_prepare_kwargs)
    946                         if num_proc is not None:
    947                             prepare_split_kwargs["num_proc"] = num_proc
--> 948                         self._download_and_prepare(
    949                             dl_manager=dl_manager,
    950                             verification_mode=verification_mode,

[/usr/local/lib/python3.10/dist-packages/datasets/builder.py](https://localhost:8080/#) in _download_and_prepare(self, dl_manager, verification_mode, **prepare_split_kwargs)
   1059
   1060         if verification_mode == VerificationMode.BASIC_CHECKS or verification_mode == VerificationMode.ALL_CHECKS:
-> 1061             verify_splits(self.info.splits, split_dict)
   1062
   1063         # Update the info object with the splits.

[/usr/local/lib/python3.10/dist-packages/datasets/utils/info_utils.py](https://localhost:8080/#) in verify_splits(expected_splits, recorded_splits)
     98     ]
     99     if len(bad_splits) > 0:
--> 100         raise NonMatchingSplitsSizesError(str(bad_splits))
    101     logger.info("All the splits matched successfully.")
    102

NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=82649695458, num_examples=597433111, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='train', num_bytes=3358310095, num_examples=24898932, shard_lengths=[3626991, 3716991, 4036990, 3506990, 3676990, 3716990, 2616990], dataset_name='spanish-tweets-small')}]
```

I think I had this dataset updated, might be related to #6271

It is working fine as late in `2.10.0` , but not in `2.13.0` onwards.

### Steps to reproduce the bug

```python

from datasets import load_dataset

ds = load_dataset("pysentimiento/spanish-tweets-small")
```

You can run it in [this notebook](https://colab.research.google.com/drive/1FdhqLiVimHIlkn7B54DbhizeQ4U3vGVl#scrollTo=YgA50cBSibUg)

### Expected behavior

Load the dataset without any error

### Environment info

- `datasets` version: 2.13.0
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.20.3
- PyArrow version: 14.0.2
- Pandas version: 2.0.3


Document that to_json defaults to JSON Lines

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6895). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004914 / 0.011353 (-0.006439) | 0.003621 / 0.011008 (-0.007387) | 0.062841 / 0.038508 (0.024333) | 0.031630 / 0.023109 (0.008520) | 0.247666 / 0.275898 (-0.028232) | 0.288192 / 0.323480 (-0.035288) | 0.003145 / 0.007986 (-0.004841) | 0.002655 / 0.004328 (-0.001674) | 0.049484 / 0.004250 (0.045233) | 0.046593 / 0.037052 (0.009540) | 0.271550 / 0.258489 (0.013061) | 0.293228 / 0.293841 (-0.000613) | 0.026941 / 0.128546 (-0.101606) | 0.009936 / 0.075646 (-0.065710) | 0.201741 / 0.419271 (-0.217530) | 0.035435 / 0.043533 (-0.008098) | 0.251868 / 0.255139 (-0.003271) | 0.272082 / 0.283200 (-0.011118) | 0.019731 / 0.141683 (-0.121952) | 1.125752 / 1.452155 (-0.326403) | 1.152058 / 1.492716 (-0.340659) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.099695 / 0.018006 (0.081689) | 0.308306 / 0.000490 (0.307816) | 0.000223 / 0.000200 (0.000023) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018616 / 0.037411 (-0.018795) | 0.061886 / 0.014526 (0.047360) | 0.074059 / 0.176557 (-0.102498) | 0.124902 / 0.737135 (-0.612234) | 0.075108 / 0.296338 (-0.221230) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.336707 / 0.215209 (0.121498) | 2.805197 / 2.077655 (0.727542) | 1.565826 / 1.504120 (0.061706) | 1.443708 / 1.541195 (-0.097486) | 1.341167 / 1.468490 (-0.127323) | 0.566814 / 4.584777 (-4.017963) | 2.374536 / 3.745712 (-1.371176) | 2.804921 / 5.269862 (-2.464941) | 1.739848 / 4.565676 (-2.825829) | 0.062779 / 0.424275 (-0.361496) | 0.005341 / 0.007607 (-0.002266) | 0.326482 / 0.226044 (0.100438) | 3.273460 / 2.268929 (1.004531) | 1.803656 / 55.444624 (-53.640968) | 1.502518 / 6.876477 (-5.373958) | 1.523665 / 2.142072 (-0.618407) | 0.642443 / 4.805227 (-4.162784) | 0.117820 / 6.500664 (-6.382844) | 0.042540 / 0.075469 (-0.032929) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.963399 / 1.841788 (-0.878388) | 11.503648 / 8.074308 (3.429340) | 9.483957 / 10.191392 (-0.707435) | 0.129118 / 0.680424 (-0.551306) | 0.014136 / 0.534201 (-0.520065) | 0.286766 / 0.579283 (-0.292517) | 0.273328 / 0.434364 (-0.161036) | 0.324075 / 0.540337 (-0.216262) | 0.420408 / 1.386936 (-0.966528) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005099 / 0.011353 (-0.006254) | 0.003721 / 0.011008 (-0.007288) | 0.050614 / 0.038508 (0.012106) | 0.031882 / 0.023109 (0.008773) | 0.267619 / 0.275898 (-0.008279) | 0.291874 / 0.323480 (-0.031606) | 0.004254 / 0.007986 (-0.003731) | 0.002766 / 0.004328 (-0.001563) | 0.049291 / 0.004250 (0.045041) | 0.043302 / 0.037052 (0.006249) | 0.274891 / 0.258489 (0.016402) | 0.304977 / 0.293841 (0.011136) | 0.029088 / 0.128546 (-0.099459) | 0.010425 / 0.075646 (-0.065221) | 0.057781 / 0.419271 (-0.361491) | 0.033589 / 0.043533 (-0.009943) | 0.264293 / 0.255139 (0.009154) | 0.284861 / 0.283200 (0.001661) | 0.018025 / 0.141683 (-0.123658) | 1.124954 / 1.452155 (-0.327200) | 1.161957 / 1.492716 (-0.330759) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.103622 / 0.018006 (0.085615) | 0.310915 / 0.000490 (0.310425) | 0.000241 / 0.000200 (0.000041) | 0.000051 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022550 / 0.037411 (-0.014862) | 0.076466 / 0.014526 (0.061940) | 0.088297 / 0.176557 (-0.088260) | 0.128659 / 0.737135 (-0.608477) | 0.091823 / 0.296338 (-0.204516) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.293431 / 0.215209 (0.078222) | 2.888105 / 2.077655 (0.810450) | 1.559581 / 1.504120 (0.055461) | 1.421424 / 1.541195 (-0.119771) | 1.437941 / 1.468490 (-0.030549) | 0.577544 / 4.584777 (-4.007233) | 0.968840 / 3.745712 (-2.776872) | 2.799796 / 5.269862 (-2.470066) | 1.744791 / 4.565676 (-2.820885) | 0.064159 / 0.424275 (-0.360116) | 0.005043 / 0.007607 (-0.002564) | 0.341039 / 0.226044 (0.114995) | 3.354402 / 2.268929 (1.085474) | 1.904093 / 55.444624 (-53.540532) | 1.604046 / 6.876477 (-5.272431) | 1.610384 / 2.142072 (-0.531688) | 0.658129 / 4.805227 (-4.147098) | 0.119297 / 6.500664 (-6.381367) | 0.041396 / 0.075469 (-0.034073) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.001109 / 1.841788 (-0.840678) | 12.081856 / 8.074308 (4.007548) | 10.090943 / 10.191392 (-0.100449) | 0.150433 / 0.680424 (-0.529991) | 0.015850 / 0.534201 (-0.518351) | 0.286590 / 0.579283 (-0.292693) | 0.131137 / 0.434364 (-0.303227) | 0.389033 / 0.540337 (-0.151304) | 0.421382 / 1.386936 (-0.965554) |

![](https://cml.dev/watermark.png#22b7baed53f9f295a5dda2fe3eb0b7434bf57e89 "CML watermark")

Document that `Dataset.to_json` defaults to JSON Lines, by adding explanation in the corresponding docstring.

Fix #6894.


Better document defaults of to_json

documentation

Improvements or additions to documentation

Better document defaults of `to_json`: the default format is [JSON-Lines](https://jsonlines.org/).

Related to:
- #6891


Close gzipped files properly

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6893). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005388 / 0.011353 (-0.005965) | 0.003822 / 0.011008 (-0.007187) | 0.063285 / 0.038508 (0.024777) | 0.033780 / 0.023109 (0.010671) | 0.239580 / 0.275898 (-0.036318) | 0.264203 / 0.323480 (-0.059277) | 0.004207 / 0.007986 (-0.003778) | 0.002716 / 0.004328 (-0.001612) | 0.049569 / 0.004250 (0.045319) | 0.048591 / 0.037052 (0.011538) | 0.252606 / 0.258489 (-0.005884) | 0.285998 / 0.293841 (-0.007843) | 0.028650 / 0.128546 (-0.099896) | 0.010652 / 0.075646 (-0.064994) | 0.203962 / 0.419271 (-0.215310) | 0.036207 / 0.043533 (-0.007326) | 0.240374 / 0.255139 (-0.014765) | 0.263564 / 0.283200 (-0.019636) | 0.017722 / 0.141683 (-0.123961) | 1.143741 / 1.452155 (-0.308414) | 1.192452 / 1.492716 (-0.300264) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.141329 / 0.018006 (0.123323) | 0.320169 / 0.000490 (0.319679) | 0.000240 / 0.000200 (0.000041) | 0.000045 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019885 / 0.037411 (-0.017526) | 0.063322 / 0.014526 (0.048796) | 0.075446 / 0.176557 (-0.101110) | 0.122619 / 0.737135 (-0.614517) | 0.077175 / 0.296338 (-0.219163) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.281292 / 0.215209 (0.066083) | 2.796220 / 2.077655 (0.718565) | 1.456035 / 1.504120 (-0.048085) | 1.334445 / 1.541195 (-0.206750) | 1.380223 / 1.468490 (-0.088267) | 0.575895 / 4.584777 (-4.008882) | 2.375791 / 3.745712 (-1.369921) | 2.926273 / 5.269862 (-2.343589) | 1.832586 / 4.565676 (-2.733090) | 0.064323 / 0.424275 (-0.359952) | 0.005403 / 0.007607 (-0.002204) | 0.334088 / 0.226044 (0.108043) | 3.321174 / 2.268929 (1.052246) | 1.821432 / 55.444624 (-53.623193) | 1.520181 / 6.876477 (-5.356296) | 1.582487 / 2.142072 (-0.559585) | 0.645641 / 4.805227 (-4.159586) | 0.119596 / 6.500664 (-6.381068) | 0.043144 / 0.075469 (-0.032325) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.985104 / 1.841788 (-0.856684) | 12.518240 / 8.074308 (4.443932) | 10.017118 / 10.191392 (-0.174274) | 0.133900 / 0.680424 (-0.546524) | 0.014591 / 0.534201 (-0.519610) | 0.288326 / 0.579283 (-0.290957) | 0.262292 / 0.434364 (-0.172072) | 0.327601 / 0.540337 (-0.212736) | 0.421525 / 1.386936 (-0.965411) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005546 / 0.011353 (-0.005807) | 0.003961 / 0.011008 (-0.007047) | 0.051745 / 0.038508 (0.013237) | 0.032587 / 0.023109 (0.009478) | 0.266886 / 0.275898 (-0.009012) | 0.301327 / 0.323480 (-0.022153) | 0.004273 / 0.007986 (-0.003713) | 0.002851 / 0.004328 (-0.001477) | 0.049333 / 0.004250 (0.045082) | 0.044530 / 0.037052 (0.007478) | 0.286829 / 0.258489 (0.028340) | 0.310732 / 0.293841 (0.016892) | 0.029925 / 0.128546 (-0.098621) | 0.011270 / 0.075646 (-0.064377) | 0.059071 / 0.419271 (-0.360200) | 0.033899 / 0.043533 (-0.009633) | 0.270448 / 0.255139 (0.015309) | 0.286935 / 0.283200 (0.003735) | 0.019516 / 0.141683 (-0.122167) | 1.125815 / 1.452155 (-0.326339) | 1.179893 / 1.492716 (-0.312823) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096476 / 0.018006 (0.078470) | 0.305149 / 0.000490 (0.304660) | 0.000207 / 0.000200 (0.000008) | 0.000046 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023648 / 0.037411 (-0.013763) | 0.082847 / 0.014526 (0.068322) | 0.089210 / 0.176557 (-0.087347) | 0.130194 / 0.737135 (-0.606941) | 0.091700 / 0.296338 (-0.204639) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.290995 / 0.215209 (0.075786) | 2.870335 / 2.077655 (0.792680) | 1.595661 / 1.504120 (0.091541) | 1.452319 / 1.541195 (-0.088876) | 1.505647 / 1.468490 (0.037157) | 0.575856 / 4.584777 (-4.008921) | 1.005527 / 3.745712 (-2.740185) | 2.927824 / 5.269862 (-2.342038) | 1.791702 / 4.565676 (-2.773975) | 0.064804 / 0.424275 (-0.359471) | 0.005203 / 0.007607 (-0.002404) | 0.348615 / 0.226044 (0.122570) | 3.463989 / 2.268929 (1.195060) | 1.947758 / 55.444624 (-53.496866) | 1.669974 / 6.876477 (-5.206502) | 1.721663 / 2.142072 (-0.420410) | 0.650999 / 4.805227 (-4.154228) | 0.117769 / 6.500664 (-6.382895) | 0.041738 / 0.075469 (-0.033731) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.004140 / 1.841788 (-0.837648) | 13.035487 / 8.074308 (4.961179) | 10.318152 / 10.191392 (0.126760) | 0.143776 / 0.680424 (-0.536648) | 0.016272 / 0.534201 (-0.517929) | 0.286564 / 0.579283 (-0.292719) | 0.126579 / 0.434364 (-0.307785) | 0.397253 / 0.540337 (-0.143085) | 0.424968 / 1.386936 (-0.961968) |

![](https://cml.dev/watermark.png#ddb6a283d7dfccc81a9fb12e761b819fed86c7a0 "CML watermark")

close https://github.com/huggingface/datasets/issues/6877


Add support for categorical/dictionary types

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6892). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005388 / 0.011353 (-0.005965) | 0.004004 / 0.011008 (-0.007005) | 0.064037 / 0.038508 (0.025529) | 0.031666 / 0.023109 (0.008557) | 0.236493 / 0.275898 (-0.039405) | 0.269047 / 0.323480 (-0.054432) | 0.005008 / 0.007986 (-0.002977) | 0.002964 / 0.004328 (-0.001364) | 0.049926 / 0.004250 (0.045675) | 0.048092 / 0.037052 (0.011039) | 0.245563 / 0.258489 (-0.012926) | 0.282614 / 0.293841 (-0.011227) | 0.027488 / 0.128546 (-0.101058) | 0.010904 / 0.075646 (-0.064742) | 0.204892 / 0.419271 (-0.214379) | 0.037161 / 0.043533 (-0.006372) | 0.238488 / 0.255139 (-0.016651) | 0.258192 / 0.283200 (-0.025008) | 0.018819 / 0.141683 (-0.122864) | 1.131573 / 1.452155 (-0.320582) | 1.204084 / 1.492716 (-0.288632) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095852 / 0.018006 (0.077846) | 0.300225 / 0.000490 (0.299735) | 0.000217 / 0.000200 (0.000017) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018592 / 0.037411 (-0.018819) | 0.062297 / 0.014526 (0.047772) | 0.074344 / 0.176557 (-0.102212) | 0.120654 / 0.737135 (-0.616481) | 0.075567 / 0.296338 (-0.220772) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.287700 / 0.215209 (0.072491) | 2.829536 / 2.077655 (0.751882) | 1.446296 / 1.504120 (-0.057824) | 1.320912 / 1.541195 (-0.220283) | 1.362744 / 1.468490 (-0.105746) | 0.563732 / 4.584777 (-4.021045) | 2.399904 / 3.745712 (-1.345808) | 2.676706 / 5.269862 (-2.593156) | 1.744780 / 4.565676 (-2.820896) | 0.062884 / 0.424275 (-0.361391) | 0.004936 / 0.007607 (-0.002671) | 0.338084 / 0.226044 (0.112040) | 3.309532 / 2.268929 (1.040603) | 1.792791 / 55.444624 (-53.651833) | 1.502038 / 6.876477 (-5.374439) | 1.662417 / 2.142072 (-0.479655) | 0.642835 / 4.805227 (-4.162393) | 0.117002 / 6.500664 (-6.383662) | 0.041880 / 0.075469 (-0.033589) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.974814 / 1.841788 (-0.866974) | 11.430883 / 8.074308 (3.356575) | 10.314734 / 10.191392 (0.123342) | 0.139838 / 0.680424 (-0.540586) | 0.014939 / 0.534201 (-0.519262) | 0.288048 / 0.579283 (-0.291235) | 0.269146 / 0.434364 (-0.165218) | 0.324300 / 0.540337 (-0.216037) | 0.421612 / 1.386936 (-0.965324) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005660 / 0.011353 (-0.005692) | 0.003723 / 0.011008 (-0.007285) | 0.049909 / 0.038508 (0.011401) | 0.033079 / 0.023109 (0.009970) | 0.270940 / 0.275898 (-0.004958) | 0.291173 / 0.323480 (-0.032307) | 0.004336 / 0.007986 (-0.003650) | 0.002793 / 0.004328 (-0.001535) | 0.049619 / 0.004250 (0.045368) | 0.041062 / 0.037052 (0.004010) | 0.285026 / 0.258489 (0.026537) | 0.322119 / 0.293841 (0.028278) | 0.029653 / 0.128546 (-0.098894) | 0.010785 / 0.075646 (-0.064861) | 0.058680 / 0.419271 (-0.360591) | 0.033300 / 0.043533 (-0.010233) | 0.269452 / 0.255139 (0.014313) | 0.285426 / 0.283200 (0.002226) | 0.017655 / 0.141683 (-0.124028) | 1.144713 / 1.452155 (-0.307442) | 1.196828 / 1.492716 (-0.295888) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096719 / 0.018006 (0.078713) | 0.303532 / 0.000490 (0.303042) | 0.000223 / 0.000200 (0.000023) | 0.000049 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022620 / 0.037411 (-0.014791) | 0.077057 / 0.014526 (0.062532) | 0.088570 / 0.176557 (-0.087987) | 0.128715 / 0.737135 (-0.608421) | 0.090844 / 0.296338 (-0.205494) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.298101 / 0.215209 (0.082892) | 2.919861 / 2.077655 (0.842206) | 1.608945 / 1.504120 (0.104825) | 1.487756 / 1.541195 (-0.053439) | 1.520800 / 1.468490 (0.052310) | 0.576615 / 4.584777 (-4.008162) | 0.964250 / 3.745712 (-2.781462) | 2.852968 / 5.269862 (-2.416893) | 1.868768 / 4.565676 (-2.696908) | 0.063934 / 0.424275 (-0.360341) | 0.005093 / 0.007607 (-0.002514) | 0.352984 / 0.226044 (0.126939) | 3.507441 / 2.268929 (1.238513) | 1.944467 / 55.444624 (-53.500158) | 1.663985 / 6.876477 (-5.212492) | 1.847029 / 2.142072 (-0.295043) | 0.669228 / 4.805227 (-4.136000) | 0.118990 / 6.500664 (-6.381675) | 0.041788 / 0.075469 (-0.033681) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.004541 / 1.841788 (-0.837247) | 12.525181 / 8.074308 (4.450873) | 10.488167 / 10.191392 (0.296775) | 0.141182 / 0.680424 (-0.539242) | 0.016432 / 0.534201 (-0.517769) | 0.283682 / 0.579283 (-0.295601) | 0.128277 / 0.434364 (-0.306087) | 0.321933 / 0.540337 (-0.218404) | 0.416430 / 1.386936 (-0.970506) |

![](https://cml.dev/watermark.png#686f5df47442bf4b3a2a73ba255427ae8d659eea "CML watermark")

@lhoestq  Thanks a ton for helping this get merged!

Arrow has a very useful dictionary/categorical type (https://arrow.apache.org/docs/python/generated/pyarrow.dictionary.html). This data type has significant speed, memory and disk benefits over pa.string() when there are only a few unique text strings in a column.

Unfortunately, huggingface datasets currently does not support this type. So huggingface datasets cannot natively read many parquet files that use this datatype .This PR adds support for Huggingface Datasets to read categorical/dictionary data.

Note: This PR functions by simply converting those dictionary/categorical types to strings. This means that huggingface datasets cannot take advantage of the compute benefits of categoricals, but it significantly simplifies logic. At this time, I do not think it makes sense to optimize categorical support within huggingface datasets and that we should only try to optimize later, if necessary.

Closes #5706


Unable to load JSON saved using `to_json`

Hi @DarshanDeshpande,

Please note that the default format of the method `Dataset.to_json` is [JSON-Lines](https://jsonlines.org/): it passes `orient="records", lines=True` to `pandas.DataFrame.to_json`. This format is specially useful for large datasets, since unlike regular JSON files, it does not require loading all the data into memory at once, but can be done iteratively by batches.

In order to read this file using the `json` library, you should parse line by line:
```python
with open("full_dataset.json", "r") as f:
    data = [json.loads(line) for line in f]
len(data)
```
Maybe we should explain this better in our docs.

Now we explain this better in out docs:
- #6895

### Describe the bug

Datasets stored in the JSON format cannot be loaded using `json.load()`

### Steps to reproduce the bug

```
import json
from datasets import load_dataset

dataset = load_dataset("squad")
train_dataset, test_dataset = dataset["train"], dataset["validation"]
test_dataset.to_json("full_dataset.json")

# This works
loaded_test = load_dataset("json", data_files="full_dataset.json")

# This fails
loaded_test = json.load(open("full_dataset.json", "r"))
```

### Expected behavior

The JSON should be correctly formatted when writing so that it can be loaded using `json.load()`.

### Environment info

Colab: https://colab.research.google.com/drive/1st1iStFUVgu9ZPvnzSzL4vDeYWDwYpUm?usp=sharing


add `with_transform` and/or `set_transform` to IterableDataset

enhancement

New feature or request

### Feature request

when working with a really large dataset it would save us a lot of time (and compute resources) to use either with_transform or the set_transform from the Dataset class instead of waiting for the entire dataset to map

### Motivation

don't want to wait for a really long dataset to map, this would give IterableDataset an extra advantage over the Dataset class.
reducing time and resources

### Your contribution

I am a little busy with my job search lately, but would post about this feature in my social media.
Apologies again (dad going to kick me out soon), if I ever have some free time I will contribute to making this a reality, but that's going to be hard
      / (┬┬﹏┬┬)\


fix bug #6877

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6889). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Can you give more details on why this fix works ?

In order to locate this file handle problem, I defined a print_open_files_count() function using psutil library:
```python
def print_open_files_count(markstr):
    pid = os.getpid()
    p = psutil.Process(pid)
    open_files = p.open_files()
    print(f"{markstr}_Open files count: {len(open_files)}")

```

and added this function as below:
```python

with open(file, "rb") as f:
    print_open_files_count('Before')
...
...
      batch_idx += 1
print_open_files_count('After')
```
and the console output as below when loading the 'mteb/biblenlp-corpus-mmteb' dataset :
```shell
Before_Open files count: 1
After_Open files count: 1
Before_Open files count: 2
After_Open files count: 2
Before_Open files count: 3
After_Open files count: 3
...
```
which indicated there was a file handle leakage in the dataset loading process. So I tried to close the file handle manually using os library and found it works although  the core issue has not been found temporarily

I think it would be better to find the cause and have a cleaner fix, because while your suggested fix works for a simple case, it will lead to files that will stay open if there is an error during the dataset generation for example.

Btw I was not able to reproduce locally (macbook pro m2) or on colab, so it might be something related to your environment. Also `open()` should close the file at the end of the `with` block so I don't really get how you can get this issue :/

> Btw I was not able to reproduce locally (macbook pro m2) or on colab, so it might be something related to your environment. Also `open()` should close the file at the end of the `with` block so I don't really get how you can get this issue :/

how about setting the limitation of open files  to 1024?

I was able to reproduce on colab with

```
!ulimit -n 256 && python -c "from datasets import load_dataset; load_dataset('mteb/biblenlp-corpus-mmteb')"
```

(also needed to `!pip install -qq git+https://github.com/huggingface/huggingface_hub.git@less-paths-info-calls` to fix a rate limit for some reason)

which led to me find that the issue came from the `GzipFileSystem` that wasn't closing files.

to reproduce:

```python
import gzip
import os

import datasets
import fsspec

# os.mkdir("tmp")
# for i in range(300):
#     with gzip.open(f"tmp/{i}.txt.gz", "wt") as f:
#         f.write("yo")

for i in range(300):
    with fsspec.open(f"gzip://{i}.txt::tmp/{i}.txt.gz", "rb") as f:
        f.read()
```

I opened https://github.com/huggingface/datasets/pull/6893 to fix this, can you try if it works on your side ?

ok

---- Replied Message ----
| From | Quentin ***@***.***> |
| Date | 05/13/2024 20:28 |
| To | ***@***.***> |
| Cc | ***@***.***>***@***.***> |
| Subject | Re: [huggingface/datasets] fix bug #6877 (PR #6889) |

I was able to reproduce on colab with

!ulimit -n 256 && python -c "from datasets import load_dataset; load_dataset('mteb/biblenlp-corpus-mmteb')"

(also needed to !pip install -qq ***@***.*** to fix a rate limit for some reason)

which lead to me find that the issue came from the GzipFileSystem that wasn't closing files.

to reproduce:

importgzipimportosimportdatasetsimportfsspec# os.mkdir("tmp")# for i in range(300):#     with gzip.open(f"tmp/{i}.txt.gz", "wt") as f:#         f.write("yo")foriinrange(300):
    withfsspec.open(f"gzip://::tmp/{i}.txt.gz", "rb") asf:
        f.read()

I opened #6893 to fix this, can you try if it works on your side ?

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***>

fix bug #6877 due to maybe f becomes invaild after yield process
the results are below:

Resolving data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:01<00:00, 420.41it/s]
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 26148.48it/s]
Resolving data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 409731.44it/s]
Resolving data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 289720.84it/s]
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 26663.42it/s]
Resolving data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 434056.21it/s]
Downloading data: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 13222.33files/s]
Downloading data: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:04<00:00, 180.67files/s]
Downloading data: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [01:35<00:00,  8.70files/s]
Generating train split: 1571592 examples [00:08, 176736.09 examples/s]
Generating test split: 85533 examples [00:01, 48224.56 examples/s]
Generating validation split: 86246 examples [00:01, 50164.16 examples/s]

Fix https://github.com/huggingface/datasets/issues/6877.

CC: @natolambert


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6888). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I think webdataset splits the file name and extension using the first dot no ?

https://github.com/webdataset/webdataset/blob/945b251a872ec0d337be8f9ea17a9c5b0d017ff3/webdataset/tariterators.py#L226

links to this function that splits on first dot

```python

def base_plus_ext(path):
    """Split off all file extensions.

    Returns base, allext.

    Args:
        path: path with extensions

    Returns:
        path with all extensions removed
    """
    match = re.match(r"^((?:.*/|)[^.]+)[.]([^/]*)$", path)
    if not match:
        return None, None
    return match.group(1), match.group(2)
```

So maybe the original issue is actually due to one of the files containing a dot in its file name that is not for the extension

```python
>>> base_plus_ext("15_Cohen_1-s2.0-S0929664620300449-gr3_lrg-b.png")
('15_Cohen_1-s2', '0-S0929664620300449-gr3_lrg-b.png')
```

Thanks for your review, @lhoestq.

I was not aware that `webdataset` requires filenames without dots in their basenames.

I they can have dots for the extension (that becomes the column name) but not in the key used to group files into samples

Support WebDataset containing file basenames with dots.

Fix #6880.


FAISS load to None

Hello,

I'm not sure I understand.
The return value of `ds.load_faiss_index` is None as expected.

I see that loading an Index on a dataset that doesn't have an `embedding` column doesn't raise an Issue. Is that the issue?

So `ds` doesn't have an `embedding` column, but we load an index that looks for it. But this will raise an issue only when calling `ds.search`.

### Describe the bug

I've use FAISS with Datasets and save to FAISS.

Then load to save FAISS then no error, then ds to None

```python
ds.load_faiss_index('embeddings', 'my_index.faiss')
```

### Steps to reproduce the bug

# 1.
```python
ds_with_embeddings = ds.map(lambda example: {'embeddings': model(transforms(example['image']).unsqueeze(0)).squeeze()}, batch_size=64)

ds_with_embeddings.add_faiss_index(column='embeddings')

ds_with_embeddings.save_faiss_index('embeddings', 'index.faiss')
```

# 2.
```python
ds.load_faiss_index('embeddings', 'my_index.faiss')
```

### Expected behavior

Add column in Datasets.

### Environment info

Google Colab, SageMaker Notebook


load_dataset with data_dir and cache_dir set fail with not supported

### Describe the bug

with python 3.11 I execute:
```py
from transformers import Wav2Vec2Processor, Data2VecAudioModel
import torch
from torch import nn
from datasets import load_dataset, concatenate_datasets

# load demo audio and set processor
dataset_clean = load_dataset("librispeech_asr", "clean", split="validation", data_dir="data", cache_dir="cache")
```
This fails in the last line with
```log
Found cached dataset librispeech_asr (file:///Users/as/Documents/Project/git/audio2vec/cache/librispeech_asr/clean-data_dir=data/2.1.0/cff5df6e7955c80a67f80e27e7e655de71c689e2d2364bece785b972acb37fe7)
Traceback (most recent call last):
  File "/Users/as/Documents/Project/git/audio2vec/src/music2vec-v1.py", line 7, in <module>
    dataset_clean = load_dataset("librispeech_asr", "clean", split="validation", data_dir="data", cache_dir="cache")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/as/anaconda3/lib/python3.11/site-packages/datasets/load.py", line 1810, in load_dataset
    ds = builder_instance.as_dataset(split=split, verification_mode=verification_mode, in_memory=keep_in_memory)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/as/anaconda3/lib/python3.11/site-packages/datasets/builder.py", line 1113, in as_dataset
    raise NotImplementedError(f"Loading a dataset cached in a {type(self._fs).__name__} is not supported.")
NotImplementedError: Loading a dataset cached in a LocalFileSystem is not supported.
```

### Steps to reproduce the bug

I setup an venv with requirements.txt
```txt
transformers==4.40.2
torch==2.2.2
datasets==2.16.0
fsspec==2023.9.2
```

pip freeze is:
```
aiohttp==3.9.5
aiosignal==1.3.1
attrs==23.2.0
certifi==2024.2.2
charset-normalizer==3.3.2
datasets==2.16.0
dill==0.3.7
filelock==3.14.0
frozenlist==1.4.1
fsspec==2023.9.2
huggingface-hub==0.23.0
idna==3.7
Jinja2==3.1.4
MarkupSafe==2.1.5
mpmath==1.3.0
multidict==6.0.5
multiprocess==0.70.15
networkx==3.3
numpy==1.26.4
packaging==24.0
pandas==2.2.2
pyarrow==16.0.0
pyarrow-hotfix==0.6
python-dateutil==2.9.0.post0
pytz==2024.1
PyYAML==6.0.1
regex==2024.4.28
requests==2.31.0
safetensors==0.4.3
six==1.16.0
sympy==1.12
tokenizers==0.19.1
torch==2.2.2
tqdm==4.66.4
transformers==4.40.2
typing_extensions==4.11.0
tzdata==2024.1
urllib3==2.2.1
xxhash==3.4.1
yarl==1.9.4

```

I execute this on a M1 Mac.

### Expected behavior

I don't understand the error message. Why is "local" caching not supported. Would it possible to give some additional hint with the error message how to solve this issue?

### Environment info

source ....
python -u example.py