Make JSON builder support an array of strings

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6696). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005057 / 0.011353 (-0.006296) | 0.003665 / 0.011008 (-0.007343) | 0.063217 / 0.038508 (0.024709) | 0.028789 / 0.023109 (0.005679) | 0.233597 / 0.275898 (-0.042301) | 0.254792 / 0.323480 (-0.068687) | 0.003065 / 0.007986 (-0.004921) | 0.002686 / 0.004328 (-0.001642) | 0.050182 / 0.004250 (0.045932) | 0.042204 / 0.037052 (0.005151) | 0.254262 / 0.258489 (-0.004227) | 0.277099 / 0.293841 (-0.016742) | 0.027564 / 0.128546 (-0.100982) | 0.010768 / 0.075646 (-0.064878) | 0.207302 / 0.419271 (-0.211969) | 0.035737 / 0.043533 (-0.007796) | 0.242388 / 0.255139 (-0.012751) | 0.259833 / 0.283200 (-0.023367) | 0.019833 / 0.141683 (-0.121850) | 1.135928 / 1.452155 (-0.316227) | 1.162851 / 1.492716 (-0.329865) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.089209 / 0.018006 (0.071202) | 0.300493 / 0.000490 (0.300003) | 0.000216 / 0.000200 (0.000016) | 0.000049 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.017968 / 0.037411 (-0.019444) | 0.061773 / 0.014526 (0.047247) | 0.073835 / 0.176557 (-0.102722) | 0.118592 / 0.737135 (-0.618544) | 0.073606 / 0.296338 (-0.222732) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.287858 / 0.215209 (0.072649) | 2.822917 / 2.077655 (0.745262) | 1.485259 / 1.504120 (-0.018861) | 1.355922 / 1.541195 (-0.185273) | 1.364008 / 1.468490 (-0.104482) | 0.557713 / 4.584777 (-4.027064) | 2.378972 / 3.745712 (-1.366741) | 2.737218 / 5.269862 (-2.532643) | 1.718317 / 4.565676 (-2.847359) | 0.062362 / 0.424275 (-0.361913) | 0.004992 / 0.007607 (-0.002615) | 0.350765 / 0.226044 (0.124721) | 3.387579 / 2.268929 (1.118650) | 1.860408 / 55.444624 (-53.584216) | 1.569355 / 6.876477 (-5.307122) | 1.593013 / 2.142072 (-0.549059) | 0.639325 / 4.805227 (-4.165902) | 0.121769 / 6.500664 (-6.378895) | 0.042148 / 0.075469 (-0.033322) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.990594 / 1.841788 (-0.851194) | 11.460904 / 8.074308 (3.386596) | 9.438691 / 10.191392 (-0.752701) | 0.141884 / 0.680424 (-0.538540) | 0.013725 / 0.534201 (-0.520476) | 0.288847 / 0.579283 (-0.290436) | 0.278815 / 0.434364 (-0.155549) | 0.337108 / 0.540337 (-0.203229) | 0.441659 / 1.386936 (-0.945277) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005265 / 0.011353 (-0.006088) | 0.003734 / 0.011008 (-0.007274) | 0.049365 / 0.038508 (0.010857) | 0.030483 / 0.023109 (0.007373) | 0.275085 / 0.275898 (-0.000813) | 0.296004 / 0.323480 (-0.027475) | 0.004964 / 0.007986 (-0.003022) | 0.002542 / 0.004328 (-0.001787) | 0.048734 / 0.004250 (0.044483) | 0.044098 / 0.037052 (0.007046) | 0.292517 / 0.258489 (0.034028) | 0.319992 / 0.293841 (0.026151) | 0.029552 / 0.128546 (-0.098994) | 0.010669 / 0.075646 (-0.064977) | 0.058887 / 0.419271 (-0.360385) | 0.051163 / 0.043533 (0.007630) | 0.277266 / 0.255139 (0.022127) | 0.295347 / 0.283200 (0.012147) | 0.018403 / 0.141683 (-0.123280) | 1.151979 / 1.452155 (-0.300176) | 1.204583 / 1.492716 (-0.288134) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091157 / 0.018006 (0.073151) | 0.300109 / 0.000490 (0.299619) | 0.000211 / 0.000200 (0.000011) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021521 / 0.037411 (-0.015890) | 0.074954 / 0.014526 (0.060428) | 0.087010 / 0.176557 (-0.089546) | 0.125853 / 0.737135 (-0.611282) | 0.087877 / 0.296338 (-0.208461) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.297890 / 0.215209 (0.082681) | 2.912159 / 2.077655 (0.834504) | 1.619311 / 1.504120 (0.115192) | 1.501726 / 1.541195 (-0.039468) | 1.494143 / 1.468490 (0.025652) | 0.566744 / 4.584777 (-4.018033) | 2.497594 / 3.745712 (-1.248118) | 2.631403 / 5.269862 (-2.638459) | 1.727896 / 4.565676 (-2.837780) | 0.065937 / 0.424275 (-0.358339) | 0.005023 / 0.007607 (-0.002585) | 0.345747 / 0.226044 (0.119702) | 3.417615 / 2.268929 (1.148686) | 1.949970 / 55.444624 (-53.494654) | 1.680019 / 6.876477 (-5.196457) | 1.789879 / 2.142072 (-0.352193) | 0.648053 / 4.805227 (-4.157174) | 0.117408 / 6.500664 (-6.383256) | 0.040681 / 0.075469 (-0.034788) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.012535 / 1.841788 (-0.829252) | 11.935819 / 8.074308 (3.861511) | 10.241452 / 10.191392 (0.050060) | 0.130956 / 0.680424 (-0.549468) | 0.015396 / 0.534201 (-0.518805) | 0.289166 / 0.579283 (-0.290117) | 0.274149 / 0.434364 (-0.160215) | 0.325844 / 0.540337 (-0.214493) | 0.424919 / 1.386936 (-0.962017) |

![](https://cml.dev/watermark.png#cb834d9c63ab8cb14725ae8e4fc2da8672892a6d "CML watermark")

Support JSON file with an array of strings.

Fix #6695.


Support JSON file with an array of strings

enhancement

New feature or request

https://huggingface.co/datasets/CausalLM/Refined-Anime-Text/discussions/1 has been fixed, but how can we check if there are other datasets with the same error, in datasets-server's database? I don't know how to get the list of erroneous cache entries, since we only copied `Error code:   JobManagerCrashedError`, but not the traceback in `details`... Do you remember the error message, or the underlying exception, we had?

Support loading a dataset from a JSON file with an array of strings.

See: https://huggingface.co/datasets/CausalLM/Refined-Anime-Text/discussions/1


__add__ for Dataset, IterableDataset

Hi! You can find a reason why we are against this feature in https://github.com/huggingface/datasets/issues/3449.

> It's too cumbersome to write this command every time we perform a dataset merging operation

Explicit is better than implicit, so this isn't a good enough reason.

Thanks for the effort nonetheless :)!

It's too cumbersome to write this command every time we perform a dataset merging operation. ```pythonfrom datasets import concatenate_datasets``` We have added a simple `__add__` magic method to each class using `concatenate_datasets.`

```python
from datasets import load_dataset

bookcorpus = load_dataset("bookcorpus", split="train")
wiki = load_dataset("wikimedia/wikipedia", "20231101.ab", split="train")
wiki = wiki.remove_columns([col for col in wiki.column_names if col != "text"])  # only keep the 'text' column

bookcorpus + wiki

#Dataset({
#   features: ['text'],
#   num_rows: 74004228
#})
#Dataset({
#    features: ['text'],
#    num_rows: 6152
#})
#Dataset({
#   features: ['text'],
#    num_rows: 74010380
#})
```


Update the print message for chunked_dataset in process.mdx

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6693). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005069 / 0.011353 (-0.006284) | 0.003682 / 0.011008 (-0.007326) | 0.063733 / 0.038508 (0.025225) | 0.030377 / 0.023109 (0.007268) | 0.242962 / 0.275898 (-0.032936) | 0.262865 / 0.323480 (-0.060615) | 0.004760 / 0.007986 (-0.003225) | 0.002772 / 0.004328 (-0.001557) | 0.049094 / 0.004250 (0.044843) | 0.041093 / 0.037052 (0.004041) | 0.260423 / 0.258489 (0.001934) | 0.283908 / 0.293841 (-0.009933) | 0.027409 / 0.128546 (-0.101138) | 0.010548 / 0.075646 (-0.065098) | 0.208637 / 0.419271 (-0.210634) | 0.035386 / 0.043533 (-0.008147) | 0.242352 / 0.255139 (-0.012787) | 0.264201 / 0.283200 (-0.018999) | 0.017822 / 0.141683 (-0.123860) | 1.140792 / 1.452155 (-0.311363) | 1.166782 / 1.492716 (-0.325934) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094727 / 0.018006 (0.076720) | 0.308548 / 0.000490 (0.308059) | 0.000213 / 0.000200 (0.000013) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018106 / 0.037411 (-0.019305) | 0.062057 / 0.014526 (0.047531) | 0.073821 / 0.176557 (-0.102735) | 0.121269 / 0.737135 (-0.615867) | 0.074062 / 0.296338 (-0.222277) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282978 / 0.215209 (0.067768) | 2.788626 / 2.077655 (0.710971) | 1.479756 / 1.504120 (-0.024364) | 1.360620 / 1.541195 (-0.180575) | 1.363996 / 1.468490 (-0.104494) | 0.571646 / 4.584777 (-4.013131) | 2.430630 / 3.745712 (-1.315083) | 2.783909 / 5.269862 (-2.485953) | 1.744617 / 4.565676 (-2.821060) | 0.062771 / 0.424275 (-0.361504) | 0.004978 / 0.007607 (-0.002629) | 0.347929 / 0.226044 (0.121884) | 3.368837 / 2.268929 (1.099908) | 1.855635 / 55.444624 (-53.588990) | 1.581555 / 6.876477 (-5.294922) | 1.589888 / 2.142072 (-0.552184) | 0.655821 / 4.805227 (-4.149406) | 0.118990 / 6.500664 (-6.381674) | 0.042191 / 0.075469 (-0.033278) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.991099 / 1.841788 (-0.850688) | 11.627919 / 8.074308 (3.553611) | 9.554180 / 10.191392 (-0.637212) | 0.140541 / 0.680424 (-0.539882) | 0.014264 / 0.534201 (-0.519937) | 0.288465 / 0.579283 (-0.290818) | 0.266400 / 0.434364 (-0.167964) | 0.324400 / 0.540337 (-0.215938) | 0.423158 / 1.386936 (-0.963778) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005588 / 0.011353 (-0.005765) | 0.003784 / 0.011008 (-0.007224) | 0.049961 / 0.038508 (0.011453) | 0.031215 / 0.023109 (0.008105) | 0.280859 / 0.275898 (0.004961) | 0.306416 / 0.323480 (-0.017063) | 0.004310 / 0.007986 (-0.003676) | 0.002884 / 0.004328 (-0.001445) | 0.049662 / 0.004250 (0.045412) | 0.046611 / 0.037052 (0.009559) | 0.293353 / 0.258489 (0.034864) | 0.327839 / 0.293841 (0.033998) | 0.050784 / 0.128546 (-0.077763) | 0.010890 / 0.075646 (-0.064757) | 0.059612 / 0.419271 (-0.359659) | 0.033175 / 0.043533 (-0.010358) | 0.281085 / 0.255139 (0.025946) | 0.302746 / 0.283200 (0.019547) | 0.019201 / 0.141683 (-0.122481) | 1.126722 / 1.452155 (-0.325433) | 1.225678 / 1.492716 (-0.267038) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094335 / 0.018006 (0.076329) | 0.304774 / 0.000490 (0.304285) | 0.000207 / 0.000200 (0.000007) | 0.000050 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021648 / 0.037411 (-0.015763) | 0.077920 / 0.014526 (0.063394) | 0.087125 / 0.176557 (-0.089432) | 0.125481 / 0.737135 (-0.611654) | 0.089415 / 0.296338 (-0.206924) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.304955 / 0.215209 (0.089746) | 2.992587 / 2.077655 (0.914932) | 1.654609 / 1.504120 (0.150490) | 1.509114 / 1.541195 (-0.032081) | 1.530906 / 1.468490 (0.062416) | 0.572092 / 4.584777 (-4.012685) | 2.477902 / 3.745712 (-1.267810) | 2.731363 / 5.269862 (-2.538498) | 1.750000 / 4.565676 (-2.815677) | 0.063662 / 0.424275 (-0.360613) | 0.005008 / 0.007607 (-0.002600) | 0.353066 / 0.226044 (0.127022) | 3.528309 / 2.268929 (1.259380) | 2.009238 / 55.444624 (-53.435387) | 1.717792 / 6.876477 (-5.158685) | 1.861699 / 2.142072 (-0.280373) | 0.667392 / 4.805227 (-4.137835) | 0.119197 / 6.500664 (-6.381467) | 0.041131 / 0.075469 (-0.034338) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.032182 / 1.841788 (-0.809605) | 12.042613 / 8.074308 (3.968305) | 10.256293 / 10.191392 (0.064901) | 0.141180 / 0.680424 (-0.539244) | 0.015005 / 0.534201 (-0.519196) | 0.290081 / 0.579283 (-0.289202) | 0.281081 / 0.434364 (-0.153283) | 0.331425 / 0.540337 (-0.208912) | 0.418674 / 1.386936 (-0.968262) |

![](https://cml.dev/watermark.png#ad5b221c01a183a66cbf52a6d708f94e0cff0b53 "CML watermark")

Update documentation to align with `Dataset.__repr__` change after #423


Enhancement: Enable loading TSV files in load_dataset()

Hi @harsh1504660,

Thanks for your work, but this functionality already exists. See my comment in the corresponding issue: https://github.com/huggingface/datasets/issues/6691#issuecomment-1963449923

Next time you would like to contribute, I would suggest you take on an issue that is previously validated by one of the maintainers. Thanks anyway.

Fix #6691


load_dataset() does not support tsv

New feature or request

Hi @dipsivenkatesh,

Please note that this functionality is already implemented. Our CSV builder uses `pandas.read_csv` under the hood, and you can pass the parameter `delimiter="\t"` to read TSV files.

See the list of CSV config parameters in our docs: https://huggingface.co/docs/datasets/package_reference/loading_methods#datasets.packaged_modules.csv.CsvConfig

### Feature request

the load_dataset() for local functions support file types like csv, json etc but not of type tsv (tab separated values).

### Motivation

cant easily load files of type tsv, have to convert them to another type like csv then load

### Your contribution

Can try by raising a PR with a little help, currently went through the code but didn't fully understand


New feature or request

Add function to convert a script-dataset to Parquet and push it to the Hub, analogously to the Space: "Convert a Hugging Face dataset to Parquet"


.load_dataset() method defaults to zstandard

The dataset is made of JSON files compressed using zstandard, as you can see here: https://huggingface.co/datasets/cerebras/SlimPajama-627B/tree/main/test/chunk1

That's why it asks for zstandard to be installed.

Though I'm intrigued that you manage to load the dataset without zstandard installed. Maybe `pyarrow` that we use to load JSON data under the hood got support for zstandard at one point.

> The dataset is made of JSON files compressed using zstandard, as you can see here: https://huggingface.co/datasets/cerebras/SlimPajama-627B/tree/main/test/chunk1
>
> That's why it asks for zstandard to be installed.
>
> Though I'm intrigued that you manage to load the dataset without zstandard installed. Maybe `pyarrow` that we use to load JSON data under the hood got support for zstandard at one point.

Question, then.

When I loaded this dataset back in October, it downloaded all the files, and then loaded into memory just fine.

NOW, it has to sit there and unpack all these zstd files (3.6TB worth). Further, when they're in my harddrive, they're regular json files. It's only when looking at the LFS, or when the loading script runs, that I get asked to install zstd.

My question is, **is this normal?** As far as I can tell, there's no reason the dataset or the loading methods should have changed between then and now. Was my old behavior flawed, and the new behavior correct?

I mean, I got it working eventually, but it was pulling teeth, and it still doesn't load right, as I had to unpack each chunk separately, so there's no clean mapping between the chunks and the broader dataset.

The `ZstdExtractor` has been added 3 years ago and we haven't touched it since then. Same for the JSON loader.

`zstandard` is required as soon as you try to load a file with the `.zstd` extension or if a file starts with the Zstandard magic number `b"\x28\xb5\x2f\xfd"` (used to recognize Zstandard files).

Note that the extraction only has to happen once - if you reload the dataset it will be reloaded from your cache directly.

Not sure what happened between October and now unfortunately

Understood, thank you for clarifying that for me.

I'll look into how best to collate my stack of batches w/o creating duplicate arrow tables for each one.

### Describe the bug

Regardless of what method I use, datasets defaults to zstandard for unpacking my datasets.

This is poor behavior, because not only is zstandard not a dependency in the huggingface package (and therefore, your dataset loading will be interrupted while it asks you to install the package), but it happens on datasets that are uploaded in json format too, meaning the dataset loader will attempt to convert the data to a zstandard compatible format, and THEN try to unpackage it.

My 4tb drive runs out of room when using zstandard on slimpajama. It loads fine on 1.5tb when using json, however I lack the understanding of the "magic numbers" system used to select the unpackaging algorithm, so I can't push a change myself.

Commenting out this line, in "/datasets/utils/extract.py" fixes the issue, and causes SlimPajama to properly extract using rational amounts of storage, however it completely disables zstandard, which is probably undesirable behavior. Someone with an understanding of the "magic numbers" system should probably take a pass over this issue.

```
class Extractor:
    #  Put zip file to the last, b/c it is possible wrongly detected as zip (I guess it means: as tar or gzip)
    extractors: Dict[str, Type[BaseExtractor]] = {
        "tar": TarExtractor,
        "gzip": GzipExtractor,
        "zip": ZipExtractor,
        "xz": XzExtractor,
        #"zstd": ZstdExtractor,   # This line needs to go, in order for datasets to work w/o non-dependent packages
        "rar": RarExtractor,
        "bz2": Bzip2Extractor,
        "7z": SevenZipExtractor,  # <Added version="2.4.0"/>
        "lz4": Lz4Extractor,  # <Added version="2.4.0"/>
    }
```

### Steps to reproduce the bug

'''
from datasaets import load_dataset
load_dataset(path="/cerebras/SlimPajama-627B")
'''

This alone should trigger the error on any system that does not have zstandard pip installed.

### Expected behavior

This repository (which is encoded in json format, not zstandard) should check whether zstandard is installed before defaulting to it. Additionally, using zstandard should not use more than 3x the required space that other extraction mechanisms use.

### Environment info

- `datasets` version: 2.17.1
- Platform: Linux-6.5.0-18-generic-x86_64-with-glibc2.35
- Python version: 3.12.0
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0


Tensor type (e.g. from `return_tensors`) ignored in map

Hi, this is expected behavior since all the tensors are converted to Arrow data (the storage type behind a Dataset).

To get pytorch tensors back, you can set the dataset format to "torch":

```python
ds = ds.with_format("torch")
```

Thanks. Just one additional question. During the pipeline `<framework> -> arrow -> <framework>`, does `.with_format` zero-copies the tensors or is it a deep copy? And is this behavior framework-dependent?

Thanks again.

We do zero-copy Arrow <-> NumPy <-> PyTorch when the output dtype matches the original dtype, but for other frameworks it depends. For example JAX doesn't allow zero-copy NumPy -> JAX at all IIRC.

Currently tokenized data are formatted using a copy though, since tokens are stored as int32 and returned as int64 torch tensors.

### Describe the bug

I don't know if it is a bug or an expected behavior, but the tensor type seems to be ignored after applying map. For example, mapping over to tokenize text with a transformers' tokenizer always returns lists and it ignore the `return_tensors` argument.

If this is an expected behaviour (e.g., for caching/Arrow compatibility/etc.) it should be clearly documented. For example, current documentation (see [here](https://huggingface.co/docs/datasets/v2.17.1/en/nlp_process#map)) clearly state to "set  `return_tensors="np"` when you tokenize your text" to have Numpy arrays.

### Steps to reproduce the bug

```py
# %%%

import datasets
import numpy as np
import tensorflow as tf
import torch
from transformers import AutoTokenizer

# %%
ds = datasets.load_dataset("cnn_dailymail", "1.0.0", split="train[:1%]")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

#%%
for return_tensors in [None, "np", "pt", "tf", "jax"]:
  print(f"********** no map, return_tensors={return_tensors} **********")
  _ds = tokenizer(ds["article"], return_tensors=return_tensors, truncation=True, padding=True)
  print('Type <input_ids>:', type(_ds["input_ids"]))

# %%
for return_tensors in [None, "np", "pt", "tf", "jax"]:
  print(f"********** map, return_tensors={return_tensors} **********")
  _ds = ds.map(
    lambda examples: tokenizer(examples["article"], return_tensors=return_tensors, truncation=True, padding=True),
    batched=True,
    remove_columns=["article"],
  )

  print('Type <input_ids>:', type(_ds[0]["input_ids"]))
```

### Expected behavior

The output from the script above. I would expect the second half to be the same.

```
********** no map, return_tensors=None **********
Type <input_ids>: <class 'list'>
********** no map, return_tensors=np **********
Type <input_ids>: <class 'numpy.ndarray'>
********** no map, return_tensors=pt **********
Type <input_ids>: <class 'torch.Tensor'>
********** no map, return_tensors=tf **********
Type <input_ids>: <class 'tensorflow.python.framework.ops.EagerTensor'>
********** no map, return_tensors=jax **********
Type <input_ids>: <class 'jaxlib.xla_extension.ArrayImpl'>

********** map, return_tensors=None **********
Type <input_ids>: <class 'list'>
********** map, return_tensors=np **********
Type <input_ids>: <class 'list'>
********** map, return_tensors=pt **********
Type <input_ids>: <class 'list'>
********** map, return_tensors=tf **********
Type <input_ids>: <class 'list'>
********** map, return_tensors=jax **********
Type <input_ids>: <class 'list'>
```

### Environment info

- `datasets` version: 2.17.1
- Platform: Redacted (linux)
- Python version: 3.10.12
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.1.3
- `fsspec` version: 2023.10.0


fsspec: support fsspec>=2023.12.0 glob changes

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6687). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Looking into the CI failure, this PR is incompatible with `huggingface-hub>=0.20.0`. It looks like there were several changes made to HfFileSystem in 0.20.0, @lhoestq any ideas on what the issue might be in particular?

a bisect indicates that it's related to https://github.com/huggingface/huggingface_hub/pull/1815

It looks like huggingface-hub's `HfFileSystem.glob` is broken for exact string matches (that don't contain glob wildcards) when combining `huggingface-hub>=0.20.0` and `fsspec>=2023.12.0`.

I did a quick test with huggingface-hub `main`, and adding this test case to `tests/test_hf_filesystem::HfFileSystemTests::test_glob` (https://github.com/huggingface/huggingface_hub/blob/main/tests/test_hf_file_system.py) passes with `fsspec==2023.10.0` and fails with `fsspec==2023.12.0`
```python
        self.assertEqual(
            sorted(self.hffs.glob(self.hf_path + "/.gitattributes")),
            sorted([self.hf_path + "/.gitattributes"]),
        )

```

the `hffs.glob()` call with a pattern that does not contain any wildcards returns an empty list:
```
E       AssertionError: Lists differ: [] != ['datasets/__DUMMY_TRANSFORMERS_USER__/rep[35 chars]tes']
E
E       Second list contains 1 additional elements.
E       First extra element 0:
E       'datasets/__DUMMY_TRANSFORMERS_USER__/repo-7d0ae9-17091013467064/.gitattributes'
E
E       - []
E       + ['datasets/__DUMMY_TRANSFORMERS_USER__/repo-7d0ae9-17091013467064/.gitattributes']
```
(and with the compatible/passing older fsspec versions the glob call returns the single exact file match as expected)

So it looks like the CI failure here isn't directly related to this PR. The failing patterns that don't contain any `*` wildcards are generated by `datasets` with or without this PR, but now that this PR installs the incompatible fsspec version, the underlying `HfFileSystem.glob()` call ends up failing.

I just opened https://github.com/huggingface/huggingface_hub/pull/2056 to fix this.

Do you mind if I continue this PR to run the CI against `huggingface_hub@main` until the fix is released ?

EDIT: the fix has been released in `huggingface_hub` 0.21.2 - I removed my commits that were using `huggingface_hub@main`

I just added two additional patterns to cover cases like `test-data/xxx.csv` and `data-test/xxx.csv`

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005461 / 0.011353 (-0.005892) | 0.003861 / 0.011008 (-0.007148) | 0.063252 / 0.038508 (0.024744) | 0.031474 / 0.023109 (0.008364) | 0.250321 / 0.275898 (-0.025577) | 0.275198 / 0.323480 (-0.048282) | 0.003275 / 0.007986 (-0.004710) | 0.002874 / 0.004328 (-0.001454) | 0.049499 / 0.004250 (0.045248) | 0.045334 / 0.037052 (0.008282) | 0.266347 / 0.258489 (0.007858) | 0.308974 / 0.293841 (0.015133) | 0.027742 / 0.128546 (-0.100804) | 0.010274 / 0.075646 (-0.065373) | 0.207516 / 0.419271 (-0.211755) | 0.036538 / 0.043533 (-0.006995) | 0.247949 / 0.255139 (-0.007190) | 0.268986 / 0.283200 (-0.014214) | 0.019842 / 0.141683 (-0.121841) | 1.117547 / 1.452155 (-0.334607) | 1.175813 / 1.492716 (-0.316903) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.103661 / 0.018006 (0.085655) | 0.331023 / 0.000490 (0.330534) | 0.000240 / 0.000200 (0.000040) | 0.000041 / 0.000054 (-0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019767 / 0.037411 (-0.017645) | 0.061500 / 0.014526 (0.046974) | 0.075899 / 0.176557 (-0.100658) | 0.122240 / 0.737135 (-0.614895) | 0.074621 / 0.296338 (-0.221717) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.287501 / 0.215209 (0.072292) | 2.794737 / 2.077655 (0.717082) | 1.505362 / 1.504120 (0.001242) | 1.379481 / 1.541195 (-0.161713) | 1.394836 / 1.468490 (-0.073654) | 0.545803 / 4.584777 (-4.038974) | 2.364167 / 3.745712 (-1.381545) | 2.800923 / 5.269862 (-2.468939) | 1.723910 / 4.565676 (-2.841766) | 0.061270 / 0.424275 (-0.363005) | 0.005006 / 0.007607 (-0.002601) | 0.334952 / 0.226044 (0.108908) | 3.367122 / 2.268929 (1.098194) | 1.839822 / 55.444624 (-53.604803) | 1.553774 / 6.876477 (-5.322703) | 1.583585 / 2.142072 (-0.558488) | 0.624680 / 4.805227 (-4.180547) | 0.116364 / 6.500664 (-6.384300) | 0.042412 / 0.075469 (-0.033057) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.975207 / 1.841788 (-0.866580) | 11.843126 / 8.074308 (3.768818) | 9.418537 / 10.191392 (-0.772855) | 0.130648 / 0.680424 (-0.549775) | 0.013747 / 0.534201 (-0.520454) | 0.288195 / 0.579283 (-0.291088) | 0.269861 / 0.434364 (-0.164503) | 0.326732 / 0.540337 (-0.213606) | 0.441256 / 1.386936 (-0.945680) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005185 / 0.011353 (-0.006168) | 0.003836 / 0.011008 (-0.007172) | 0.050057 / 0.038508 (0.011549) | 0.030929 / 0.023109 (0.007820) | 0.263558 / 0.275898 (-0.012340) | 0.284553 / 0.323480 (-0.038927) | 0.004331 / 0.007986 (-0.003655) | 0.002815 / 0.004328 (-0.001513) | 0.050187 / 0.004250 (0.045936) | 0.048431 / 0.037052 (0.011379) | 0.271005 / 0.258489 (0.012515) | 0.304749 / 0.293841 (0.010908) | 0.029286 / 0.128546 (-0.099260) | 0.010598 / 0.075646 (-0.065048) | 0.058111 / 0.419271 (-0.361160) | 0.053665 / 0.043533 (0.010132) | 0.257574 / 0.255139 (0.002436) | 0.285802 / 0.283200 (0.002602) | 0.018917 / 0.141683 (-0.122766) | 1.206517 / 1.452155 (-0.245638) | 1.220572 / 1.492716 (-0.272144) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.122466 / 0.018006 (0.104460) | 0.567887 / 0.000490 (0.567397) | 0.000321 / 0.000200 (0.000121) | 0.000049 / 0.000054 (-0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022120 / 0.037411 (-0.015292) | 0.075456 / 0.014526 (0.060931) | 0.086385 / 0.176557 (-0.090171) | 0.126045 / 0.737135 (-0.611091) | 0.087502 / 0.296338 (-0.208837) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.304847 / 0.215209 (0.089638) | 3.008095 / 2.077655 (0.930441) | 1.726178 / 1.504120 (0.222058) | 1.592332 / 1.541195 (0.051138) | 1.603714 / 1.468490 (0.135224) | 0.576875 / 4.584777 (-4.007902) | 2.450884 / 3.745712 (-1.294828) | 2.719073 / 5.269862 (-2.550789) | 1.775261 / 4.565676 (-2.790415) | 0.063144 / 0.424275 (-0.361131) | 0.005122 / 0.007607 (-0.002485) | 0.350004 / 0.226044 (0.123960) | 3.467146 / 2.268929 (1.198218) | 2.062907 / 55.444624 (-53.381717) | 1.798793 / 6.876477 (-5.077684) | 1.921204 / 2.142072 (-0.220868) | 0.651832 / 4.805227 (-4.153396) | 0.122326 / 6.500664 (-6.378338) | 0.041396 / 0.075469 (-0.034073) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.024859 / 1.841788 (-0.816928) | 12.569744 / 8.074308 (4.495436) | 10.448487 / 10.191392 (0.257095) | 0.131529 / 0.680424 (-0.548895) | 0.014853 / 0.534201 (-0.519348) | 0.287683 / 0.579283 (-0.291600) | 0.289814 / 0.434364 (-0.144550) | 0.323935 / 0.540337 (-0.216403) | 0.425208 / 1.386936 (-0.961728) |

![](https://cml.dev/watermark.png#ba71e92c59c9bd9d1ee6168691977f0c4728ed6e "CML watermark")

> EDIT: the fix has been released in `huggingface_hub` 0.21.2 - I removed my commits that were using `huggingface_hub@main`

Please note that people using `huggingface_hub` < 0.21.2 and latest `fsspec` will have issues when using `datasets`:
- https://github.com/huggingface/lighteval/actions/runs/8139147047/job/22241658122?pr=86
- https://github.com/huggingface/lighteval/pull/84

CC: @clefourrier

- adds support for the `fs.glob` changes introduced in `fsspec==2023.12.0` and unpins the current upper bound

Should close #6644
Should close #6645

The `test_data_files` glob/pattern tests pass for me in:
- `fsspec==2023.10.0` (the pinned max version in datasets `main`)
- `fsspec==2023.12.0` (#6644)
- `fsspec==2024.2.0` (#6645)


Question: Is there any way for uploading a large image dataset?

```
import pandas as pd
from datasets import Dataset, Image

# Read the CSV file
data = pd.read_csv("XXXX.csv")

# Create a Hugging Face Dataset
dataset = Dataset.from_pandas(data)
dataset = dataset.cast_column("file_name", Image())

# Upload to Hugging Face Hub (make sure authentication is set up)
dataset.push_to_hub("XXXXX"")
```

stuck in "Casting the dataset
![截屏2024-05-02 11 44 50](https://github.com/huggingface/datasets/assets/48406770/dc012dc5-16f6-4fd5-9e02-1b705c552c5b)
"

I am uploading an image dataset like this:
```
dataset = load_dataset(
    "json",
    data_files={"train": "data/custom_dataset/train.json", "validation": "data/custom_dataset/val.json"},
)
dataset = dataset.cast_column("images", Sequence(Image()))
dataset.push_to_hub("StanfordAIMI/custom_dataset", max_shard_size="1GB")
```
where it takes a long time in the `Map` process. Do you think I can use multi-processing to map all the image data to the memory first? For the `Map()` function, I can set `num_proc`. But for `push_to_hub` and `cast_column`, I can not find it.

Thanks in advance!

Best,


Updated Quickstart Notebook link

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6685). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005386 / 0.011353 (-0.005967) | 0.003707 / 0.011008 (-0.007301) | 0.062661 / 0.038508 (0.024153) | 0.029058 / 0.023109 (0.005949) | 0.249669 / 0.275898 (-0.026230) | 0.280996 / 0.323480 (-0.042484) | 0.004041 / 0.007986 (-0.003945) | 0.002713 / 0.004328 (-0.001616) | 0.047914 / 0.004250 (0.043664) | 0.042014 / 0.037052 (0.004961) | 0.265209 / 0.258489 (0.006720) | 0.297320 / 0.293841 (0.003479) | 0.028323 / 0.128546 (-0.100223) | 0.010844 / 0.075646 (-0.064802) | 0.205895 / 0.419271 (-0.213377) | 0.035997 / 0.043533 (-0.007536) | 0.245069 / 0.255139 (-0.010070) | 0.266159 / 0.283200 (-0.017040) | 0.017590 / 0.141683 (-0.124093) | 1.132046 / 1.452155 (-0.320109) | 1.177496 / 1.492716 (-0.315220) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.105441 / 0.018006 (0.087435) | 0.301321 / 0.000490 (0.300831) | 0.000211 / 0.000200 (0.000011) | 0.000064 / 0.000054 (0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018687 / 0.037411 (-0.018724) | 0.061221 / 0.014526 (0.046695) | 0.072556 / 0.176557 (-0.104001) | 0.119641 / 0.737135 (-0.617495) | 0.073781 / 0.296338 (-0.222557) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.284564 / 0.215209 (0.069354) | 2.795786 / 2.077655 (0.718131) | 1.437059 / 1.504120 (-0.067061) | 1.309319 / 1.541195 (-0.231876) | 1.315849 / 1.468490 (-0.152641) | 0.578571 / 4.584777 (-4.006206) | 2.350754 / 3.745712 (-1.394958) | 2.758499 / 5.269862 (-2.511362) | 1.705545 / 4.565676 (-2.860131) | 0.063660 / 0.424275 (-0.360615) | 0.005506 / 0.007607 (-0.002101) | 0.334915 / 0.226044 (0.108871) | 3.295922 / 2.268929 (1.026994) | 1.796513 / 55.444624 (-53.648111) | 1.488113 / 6.876477 (-5.388364) | 1.523042 / 2.142072 (-0.619031) | 0.648169 / 4.805227 (-4.157058) | 0.119321 / 6.500664 (-6.381343) | 0.041932 / 0.075469 (-0.033537) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.982432 / 1.841788 (-0.859356) | 11.344780 / 8.074308 (3.270472) | 9.627219 / 10.191392 (-0.564173) | 0.142590 / 0.680424 (-0.537834) | 0.013899 / 0.534201 (-0.520302) | 0.286335 / 0.579283 (-0.292948) | 0.266552 / 0.434364 (-0.167812) | 0.320361 / 0.540337 (-0.219977) | 0.420303 / 1.386936 (-0.966633) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005251 / 0.011353 (-0.006102) | 0.003515 / 0.011008 (-0.007494) | 0.049344 / 0.038508 (0.010836) | 0.032055 / 0.023109 (0.008945) | 0.280653 / 0.275898 (0.004755) | 0.303989 / 0.323480 (-0.019491) | 0.004402 / 0.007986 (-0.003584) | 0.002758 / 0.004328 (-0.001570) | 0.050947 / 0.004250 (0.046697) | 0.044405 / 0.037052 (0.007353) | 0.292856 / 0.258489 (0.034367) | 0.325307 / 0.293841 (0.031466) | 0.047720 / 0.128546 (-0.080827) | 0.010589 / 0.075646 (-0.065057) | 0.057728 / 0.419271 (-0.361543) | 0.033842 / 0.043533 (-0.009691) | 0.285443 / 0.255139 (0.030304) | 0.300013 / 0.283200 (0.016814) | 0.017444 / 0.141683 (-0.124238) | 1.152880 / 1.452155 (-0.299275) | 1.200670 / 1.492716 (-0.292046) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092355 / 0.018006 (0.074349) | 0.307907 / 0.000490 (0.307418) | 0.000226 / 0.000200 (0.000026) | 0.000053 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021624 / 0.037411 (-0.015787) | 0.075855 / 0.014526 (0.061329) | 0.087109 / 0.176557 (-0.089447) | 0.124859 / 0.737135 (-0.612276) | 0.088933 / 0.296338 (-0.207406) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.294213 / 0.215209 (0.079004) | 2.893146 / 2.077655 (0.815491) | 1.595061 / 1.504120 (0.090942) | 1.480959 / 1.541195 (-0.060236) | 1.528277 / 1.468490 (0.059787) | 0.570273 / 4.584777 (-4.014504) | 2.412948 / 3.745712 (-1.332764) | 2.675009 / 5.269862 (-2.594852) | 1.724005 / 4.565676 (-2.841671) | 0.063359 / 0.424275 (-0.360916) | 0.005008 / 0.007607 (-0.002599) | 0.346570 / 0.226044 (0.120526) | 3.456566 / 2.268929 (1.187637) | 1.973109 / 55.444624 (-53.471515) | 1.657562 / 6.876477 (-5.218915) | 1.790086 / 2.142072 (-0.351986) | 0.655277 / 4.805227 (-4.149950) | 0.117985 / 6.500664 (-6.382679) | 0.041128 / 0.075469 (-0.034342) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.001428 / 1.841788 (-0.840360) | 11.953458 / 8.074308 (3.879150) | 10.188439 / 10.191392 (-0.002953) | 0.140863 / 0.680424 (-0.539561) | 0.015278 / 0.534201 (-0.518923) | 0.288193 / 0.579283 (-0.291090) | 0.281732 / 0.434364 (-0.152632) | 0.328034 / 0.540337 (-0.212304) | 0.414571 / 1.386936 (-0.972365) |

![](https://cml.dev/watermark.png#531f35e688f81ec6b4c9044856a89a6b48142bd8 "CML watermark")

Fixed Quickstart Notebook Link in the [Overview notebook](https://github.com/huggingface/datasets/blob/main/notebooks/Overview.ipynb)


Improve error message for gated datasets on load

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6684). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Thank you ! Should we also add the link to the dataset page ?

Good idea! Done in https://github.com/huggingface/datasets/pull/6684/commits/4ab55210dca1815b6c2f23901598bfb29fc92a47

Looks like a test is failing: `test_load_dataset_cached_local_script `.

Apparently your new message is also shown for datasets that don't exist, which is maybe not ideal

> Looks like a test is failing: `test_load_dataset_cached_local_script `.
>
> Apparently your new message is also shown for datasets that don't exist, which is maybe not ideal

Fixed by reverting the error message root + added a small clarifying part

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005634 / 0.011353 (-0.005719) | 0.003786 / 0.011008 (-0.007222) | 0.064245 / 0.038508 (0.025737) | 0.031228 / 0.023109 (0.008119) | 0.248162 / 0.275898 (-0.027736) | 0.273454 / 0.323480 (-0.050026) | 0.003176 / 0.007986 (-0.004809) | 0.002814 / 0.004328 (-0.001515) | 0.049234 / 0.004250 (0.044984) | 0.046075 / 0.037052 (0.009023) | 0.262410 / 0.258489 (0.003921) | 0.290597 / 0.293841 (-0.003244) | 0.028545 / 0.128546 (-0.100001) | 0.010881 / 0.075646 (-0.064766) | 0.212098 / 0.419271 (-0.207173) | 0.036406 / 0.043533 (-0.007127) | 0.244571 / 0.255139 (-0.010568) | 0.269537 / 0.283200 (-0.013663) | 0.019574 / 0.141683 (-0.122109) | 1.120369 / 1.452155 (-0.331785) | 1.170188 / 1.492716 (-0.322529) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.108088 / 0.018006 (0.090082) | 0.299836 / 0.000490 (0.299346) | 0.000204 / 0.000200 (0.000004) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.020881 / 0.037411 (-0.016531) | 0.065290 / 0.014526 (0.050764) | 0.074283 / 0.176557 (-0.102274) | 0.122189 / 0.737135 (-0.614947) | 0.077772 / 0.296338 (-0.218566) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.278329 / 0.215209 (0.063120) | 2.709885 / 2.077655 (0.632230) | 1.428824 / 1.504120 (-0.075296) | 1.314338 / 1.541195 (-0.226857) | 1.349445 / 1.468490 (-0.119045) | 0.571863 / 4.584777 (-4.012914) | 2.358306 / 3.745712 (-1.387407) | 2.873498 / 5.269862 (-2.396364) | 1.779897 / 4.565676 (-2.785779) | 0.062828 / 0.424275 (-0.361447) | 0.005416 / 0.007607 (-0.002191) | 0.337645 / 0.226044 (0.111601) | 3.328868 / 2.268929 (1.059940) | 1.793387 / 55.444624 (-53.651238) | 1.539201 / 6.876477 (-5.337276) | 1.589552 / 2.142072 (-0.552520) | 0.645454 / 4.805227 (-4.159773) | 0.116966 / 6.500664 (-6.383698) | 0.043339 / 0.075469 (-0.032130) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.995743 / 1.841788 (-0.846045) | 12.096551 / 8.074308 (4.022243) | 10.214299 / 10.191392 (0.022907) | 0.133025 / 0.680424 (-0.547399) | 0.014393 / 0.534201 (-0.519808) | 0.289018 / 0.579283 (-0.290266) | 0.267879 / 0.434364 (-0.166485) | 0.324362 / 0.540337 (-0.215976) | 0.425596 / 1.386936 (-0.961340) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005739 / 0.011353 (-0.005614) | 0.003992 / 0.011008 (-0.007017) | 0.051362 / 0.038508 (0.012854) | 0.031707 / 0.023109 (0.008598) | 0.274807 / 0.275898 (-0.001091) | 0.298897 / 0.323480 (-0.024583) | 0.004363 / 0.007986 (-0.003622) | 0.002862 / 0.004328 (-0.001466) | 0.050462 / 0.004250 (0.046212) | 0.048158 / 0.037052 (0.011106) | 0.282759 / 0.258489 (0.024270) | 0.317766 / 0.293841 (0.023926) | 0.060245 / 0.128546 (-0.068301) | 0.011279 / 0.075646 (-0.064367) | 0.061175 / 0.419271 (-0.358097) | 0.035876 / 0.043533 (-0.007656) | 0.273963 / 0.255139 (0.018824) | 0.288788 / 0.283200 (0.005589) | 0.019690 / 0.141683 (-0.121992) | 1.167074 / 1.452155 (-0.285080) | 1.206344 / 1.492716 (-0.286372) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091211 / 0.018006 (0.073205) | 0.299295 / 0.000490 (0.298805) | 0.000216 / 0.000200 (0.000016) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022718 / 0.037411 (-0.014693) | 0.079483 / 0.014526 (0.064957) | 0.087437 / 0.176557 (-0.089120) | 0.126977 / 0.737135 (-0.610159) | 0.089678 / 0.296338 (-0.206660) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.294719 / 0.215209 (0.079510) | 2.864505 / 2.077655 (0.786851) | 1.583993 / 1.504120 (0.079873) | 1.455079 / 1.541195 (-0.086115) | 1.504080 / 1.468490 (0.035590) | 0.569040 / 4.584777 (-4.015737) | 2.423472 / 3.745712 (-1.322240) | 2.742848 / 5.269862 (-2.527014) | 1.785244 / 4.565676 (-2.780432) | 0.062655 / 0.424275 (-0.361620) | 0.005027 / 0.007607 (-0.002580) | 0.343863 / 0.226044 (0.117818) | 3.376286 / 2.268929 (1.107358) | 1.933846 / 55.444624 (-53.510779) | 1.667316 / 6.876477 (-5.209161) | 1.815621 / 2.142072 (-0.326451) | 0.639378 / 4.805227 (-4.165850) | 0.116514 / 6.500664 (-6.384150) | 0.042191 / 0.075469 (-0.033279) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.007103 / 1.841788 (-0.834685) | 12.791193 / 8.074308 (4.716885) | 10.870575 / 10.191392 (0.679183) | 0.131040 / 0.680424 (-0.549384) | 0.016510 / 0.534201 (-0.517691) | 0.288372 / 0.579283 (-0.290911) | 0.275574 / 0.434364 (-0.158790) | 0.327801 / 0.540337 (-0.212536) | 0.415942 / 1.386936 (-0.970994) |

![](https://cml.dev/watermark.png#b775390900132834e5edf487f5cbbf1299af1d88 "CML watermark")

Internal Slack discussion: https://huggingface.slack.com/archives/C02V51Q3800/p1708424971135029


Fix imagefolder dataset url

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6683). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005501 / 0.011353 (-0.005851) | 0.003907 / 0.011008 (-0.007101) | 0.063524 / 0.038508 (0.025016) | 0.031773 / 0.023109 (0.008664) | 0.244672 / 0.275898 (-0.031226) | 0.293342 / 0.323480 (-0.030138) | 0.004091 / 0.007986 (-0.003895) | 0.002837 / 0.004328 (-0.001491) | 0.049181 / 0.004250 (0.044930) | 0.044515 / 0.037052 (0.007462) | 0.263932 / 0.258489 (0.005443) | 0.288412 / 0.293841 (-0.005429) | 0.028338 / 0.128546 (-0.100208) | 0.010865 / 0.075646 (-0.064781) | 0.207979 / 0.419271 (-0.211293) | 0.036149 / 0.043533 (-0.007384) | 0.250674 / 0.255139 (-0.004465) | 0.263232 / 0.283200 (-0.019968) | 0.017919 / 0.141683 (-0.123763) | 1.127794 / 1.452155 (-0.324360) | 1.172071 / 1.492716 (-0.320645) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090435 / 0.018006 (0.072429) | 0.300041 / 0.000490 (0.299552) | 0.000217 / 0.000200 (0.000018) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018986 / 0.037411 (-0.018426) | 0.064872 / 0.014526 (0.050346) | 0.074738 / 0.176557 (-0.101818) | 0.121577 / 0.737135 (-0.615558) | 0.076416 / 0.296338 (-0.219923) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.279471 / 0.215209 (0.064262) | 2.743066 / 2.077655 (0.665411) | 1.429511 / 1.504120 (-0.074609) | 1.315391 / 1.541195 (-0.225804) | 1.371255 / 1.468490 (-0.097235) | 0.570708 / 4.584777 (-4.014069) | 2.373047 / 3.745712 (-1.372666) | 2.813198 / 5.269862 (-2.456663) | 1.768928 / 4.565676 (-2.796749) | 0.066031 / 0.424275 (-0.358244) | 0.005074 / 0.007607 (-0.002533) | 0.333484 / 0.226044 (0.107440) | 3.295002 / 2.268929 (1.026074) | 1.796089 / 55.444624 (-53.648535) | 1.521849 / 6.876477 (-5.354627) | 1.604417 / 2.142072 (-0.537655) | 0.645235 / 4.805227 (-4.159992) | 0.119226 / 6.500664 (-6.381439) | 0.043275 / 0.075469 (-0.032194) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.986350 / 1.841788 (-0.855438) | 11.921886 / 8.074308 (3.847578) | 9.878841 / 10.191392 (-0.312551) | 0.141072 / 0.680424 (-0.539352) | 0.014514 / 0.534201 (-0.519687) | 0.304060 / 0.579283 (-0.275223) | 0.267844 / 0.434364 (-0.166520) | 0.324881 / 0.540337 (-0.215457) | 0.421426 / 1.386936 (-0.965510) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005322 / 0.011353 (-0.006030) | 0.003942 / 0.011008 (-0.007066) | 0.050629 / 0.038508 (0.012121) | 0.031176 / 0.023109 (0.008066) | 0.279627 / 0.275898 (0.003729) | 0.302667 / 0.323480 (-0.020813) | 0.004281 / 0.007986 (-0.003705) | 0.002900 / 0.004328 (-0.001428) | 0.048168 / 0.004250 (0.043918) | 0.046094 / 0.037052 (0.009042) | 0.290714 / 0.258489 (0.032224) | 0.321336 / 0.293841 (0.027496) | 0.047934 / 0.128546 (-0.080612) | 0.010773 / 0.075646 (-0.064873) | 0.059439 / 0.419271 (-0.359832) | 0.033644 / 0.043533 (-0.009889) | 0.273710 / 0.255139 (0.018571) | 0.295144 / 0.283200 (0.011944) | 0.018115 / 0.141683 (-0.123568) | 1.150302 / 1.452155 (-0.301853) | 1.197304 / 1.492716 (-0.295412) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090262 / 0.018006 (0.072255) | 0.300727 / 0.000490 (0.300238) | 0.000228 / 0.000200 (0.000028) | 0.000055 / 0.000054 (0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022706 / 0.037411 (-0.014706) | 0.077420 / 0.014526 (0.062894) | 0.089119 / 0.176557 (-0.087437) | 0.126760 / 0.737135 (-0.610375) | 0.090702 / 0.296338 (-0.205637) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.296558 / 0.215209 (0.081349) | 2.865311 / 2.077655 (0.787656) | 1.587355 / 1.504120 (0.083235) | 1.491660 / 1.541195 (-0.049534) | 1.513604 / 1.468490 (0.045114) | 0.565209 / 4.584777 (-4.019568) | 2.450648 / 3.745712 (-1.295064) | 2.709941 / 5.269862 (-2.559921) | 1.775032 / 4.565676 (-2.790645) | 0.063767 / 0.424275 (-0.360508) | 0.005047 / 0.007607 (-0.002560) | 0.347406 / 0.226044 (0.121361) | 3.416671 / 2.268929 (1.147743) | 1.949653 / 55.444624 (-53.494971) | 1.669885 / 6.876477 (-5.206592) | 1.848125 / 2.142072 (-0.293947) | 0.648179 / 4.805227 (-4.157048) | 0.116374 / 6.500664 (-6.384290) | 0.041816 / 0.075469 (-0.033653) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.007009 / 1.841788 (-0.834779) | 12.749964 / 8.074308 (4.675656) | 10.765890 / 10.191392 (0.574498) | 0.141743 / 0.680424 (-0.538681) | 0.016077 / 0.534201 (-0.518124) | 0.293275 / 0.579283 (-0.286008) | 0.277064 / 0.434364 (-0.157300) | 0.327039 / 0.540337 (-0.213299) | 0.421784 / 1.386936 (-0.965152) |

![](https://cml.dev/watermark.png#f807cd4c733a3616011a3f7f53a9fa56f7d5f685 "CML watermark")


Update GitHub Actions to Node 20

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6682). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005292 / 0.011353 (-0.006060) | 0.003354 / 0.011008 (-0.007654) | 0.063150 / 0.038508 (0.024642) | 0.028616 / 0.023109 (0.005507) | 0.242267 / 0.275898 (-0.033631) | 0.267305 / 0.323480 (-0.056175) | 0.003041 / 0.007986 (-0.004944) | 0.003346 / 0.004328 (-0.000982) | 0.048268 / 0.004250 (0.044018) | 0.042070 / 0.037052 (0.005018) | 0.256526 / 0.258489 (-0.001963) | 0.279744 / 0.293841 (-0.014097) | 0.027862 / 0.128546 (-0.100684) | 0.010786 / 0.075646 (-0.064861) | 0.206998 / 0.419271 (-0.212273) | 0.035503 / 0.043533 (-0.008030) | 0.248454 / 0.255139 (-0.006685) | 0.265639 / 0.283200 (-0.017561) | 0.019590 / 0.141683 (-0.122093) | 1.134445 / 1.452155 (-0.317709) | 1.194956 / 1.492716 (-0.297761) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090987 / 0.018006 (0.072981) | 0.301907 / 0.000490 (0.301418) | 0.000213 / 0.000200 (0.000013) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018324 / 0.037411 (-0.019088) | 0.061492 / 0.014526 (0.046966) | 0.074166 / 0.176557 (-0.102391) | 0.119990 / 0.737135 (-0.617145) | 0.074554 / 0.296338 (-0.221785) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.279646 / 0.215209 (0.064437) | 2.773819 / 2.077655 (0.696164) | 1.436460 / 1.504120 (-0.067660) | 1.310303 / 1.541195 (-0.230892) | 1.315328 / 1.468490 (-0.153162) | 0.558328 / 4.584777 (-4.026449) | 2.383819 / 3.745712 (-1.361893) | 2.735034 / 5.269862 (-2.534827) | 1.724413 / 4.565676 (-2.841263) | 0.061476 / 0.424275 (-0.362799) | 0.004899 / 0.007607 (-0.002708) | 0.333195 / 0.226044 (0.107151) | 3.228900 / 2.268929 (0.959971) | 1.787315 / 55.444624 (-53.657309) | 1.526949 / 6.876477 (-5.349527) | 1.539816 / 2.142072 (-0.602257) | 0.636926 / 4.805227 (-4.168302) | 0.117533 / 6.500664 (-6.383131) | 0.041859 / 0.075469 (-0.033610) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.964637 / 1.841788 (-0.877151) | 11.296021 / 8.074308 (3.221713) | 9.375436 / 10.191392 (-0.815956) | 0.140330 / 0.680424 (-0.540094) | 0.013638 / 0.534201 (-0.520563) | 0.287046 / 0.579283 (-0.292237) | 0.265054 / 0.434364 (-0.169310) | 0.331548 / 0.540337 (-0.208790) | 0.438418 / 1.386936 (-0.948518) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005284 / 0.011353 (-0.006069) | 0.003853 / 0.011008 (-0.007155) | 0.049301 / 0.038508 (0.010793) | 0.030477 / 0.023109 (0.007368) | 0.278507 / 0.275898 (0.002609) | 0.298245 / 0.323480 (-0.025235) | 0.004225 / 0.007986 (-0.003761) | 0.002736 / 0.004328 (-0.001593) | 0.049345 / 0.004250 (0.045094) | 0.045141 / 0.037052 (0.008088) | 0.290992 / 0.258489 (0.032503) | 0.317430 / 0.293841 (0.023589) | 0.029623 / 0.128546 (-0.098924) | 0.010351 / 0.075646 (-0.065295) | 0.058027 / 0.419271 (-0.361244) | 0.051306 / 0.043533 (0.007773) | 0.279947 / 0.255139 (0.024808) | 0.296916 / 0.283200 (0.013717) | 0.018859 / 0.141683 (-0.122823) | 1.153484 / 1.452155 (-0.298670) | 1.189141 / 1.492716 (-0.303575) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091030 / 0.018006 (0.073024) | 0.301305 / 0.000490 (0.300815) | 0.000230 / 0.000200 (0.000030) | 0.000049 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021801 / 0.037411 (-0.015611) | 0.075162 / 0.014526 (0.060636) | 0.086455 / 0.176557 (-0.090102) | 0.125431 / 0.737135 (-0.611705) | 0.087797 / 0.296338 (-0.208542) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295950 / 0.215209 (0.080741) | 2.895839 / 2.077655 (0.818184) | 1.603121 / 1.504120 (0.099001) | 1.482162 / 1.541195 (-0.059033) | 1.474231 / 1.468490 (0.005741) | 0.571370 / 4.584777 (-4.013407) | 2.466864 / 3.745712 (-1.278848) | 2.607279 / 5.269862 (-2.662582) | 1.723106 / 4.565676 (-2.842571) | 0.062068 / 0.424275 (-0.362208) | 0.004958 / 0.007607 (-0.002649) | 0.345213 / 0.226044 (0.119168) | 3.403916 / 2.268929 (1.134987) | 1.935538 / 55.444624 (-53.509086) | 1.658930 / 6.876477 (-5.217547) | 1.767611 / 2.142072 (-0.374461) | 0.645780 / 4.805227 (-4.159447) | 0.116077 / 6.500664 (-6.384587) | 0.040774 / 0.075469 (-0.034695) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.025952 / 1.841788 (-0.815836) | 11.935970 / 8.074308 (3.861662) | 9.935799 / 10.191392 (-0.255593) | 0.131081 / 0.680424 (-0.549343) | 0.016010 / 0.534201 (-0.518191) | 0.285476 / 0.579283 (-0.293807) | 0.274928 / 0.434364 (-0.159435) | 0.325788 / 0.540337 (-0.214550) | 0.421666 / 1.386936 (-0.965270) |

![](https://cml.dev/watermark.png#7a79064b7a2255c0d6950dc998509ecefb893689 "CML watermark")

Update GitHub Actions to Node 20.

Fix #6679.


Update release instructions

maintenance

Maintenance tasks

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6681). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005410 / 0.011353 (-0.005943) | 0.003862 / 0.011008 (-0.007146) | 0.063457 / 0.038508 (0.024949) | 0.030081 / 0.023109 (0.006972) | 0.250657 / 0.275898 (-0.025241) | 0.275483 / 0.323480 (-0.047997) | 0.004048 / 0.007986 (-0.003938) | 0.002818 / 0.004328 (-0.001511) | 0.048940 / 0.004250 (0.044689) | 0.043397 / 0.037052 (0.006345) | 0.262160 / 0.258489 (0.003671) | 0.294154 / 0.293841 (0.000313) | 0.030028 / 0.128546 (-0.098519) | 0.010789 / 0.075646 (-0.064857) | 0.209665 / 0.419271 (-0.209606) | 0.035297 / 0.043533 (-0.008236) | 0.253169 / 0.255139 (-0.001970) | 0.271775 / 0.283200 (-0.011424) | 0.018332 / 0.141683 (-0.123351) | 1.152420 / 1.452155 (-0.299735) | 1.262767 / 1.492716 (-0.229949) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.089990 / 0.018006 (0.071984) | 0.298552 / 0.000490 (0.298062) | 0.000217 / 0.000200 (0.000017) | 0.000050 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018414 / 0.037411 (-0.018997) | 0.061566 / 0.014526 (0.047040) | 0.075360 / 0.176557 (-0.101196) | 0.123470 / 0.737135 (-0.613665) | 0.075215 / 0.296338 (-0.221124) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.279563 / 0.215209 (0.064354) | 2.725212 / 2.077655 (0.647557) | 1.446413 / 1.504120 (-0.057707) | 1.321665 / 1.541195 (-0.219529) | 1.352475 / 1.468490 (-0.116015) | 0.568440 / 4.584777 (-4.016337) | 2.393217 / 3.745712 (-1.352495) | 2.793150 / 5.269862 (-2.476711) | 1.764316 / 4.565676 (-2.801360) | 0.063157 / 0.424275 (-0.361118) | 0.005117 / 0.007607 (-0.002491) | 0.333310 / 0.226044 (0.107265) | 3.291000 / 2.268929 (1.022071) | 1.824654 / 55.444624 (-53.619971) | 1.558681 / 6.876477 (-5.317795) | 1.580558 / 2.142072 (-0.561514) | 0.649831 / 4.805227 (-4.155396) | 0.118674 / 6.500664 (-6.381990) | 0.042247 / 0.075469 (-0.033222) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.976552 / 1.841788 (-0.865236) | 11.847361 / 8.074308 (3.773053) | 9.490786 / 10.191392 (-0.700606) | 0.141643 / 0.680424 (-0.538781) | 0.013653 / 0.534201 (-0.520548) | 0.284345 / 0.579283 (-0.294938) | 0.268314 / 0.434364 (-0.166050) | 0.339586 / 0.540337 (-0.200751) | 0.445239 / 1.386936 (-0.941697) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005754 / 0.011353 (-0.005599) | 0.004038 / 0.011008 (-0.006970) | 0.050027 / 0.038508 (0.011519) | 0.033598 / 0.023109 (0.010488) | 0.286514 / 0.275898 (0.010616) | 0.302493 / 0.323480 (-0.020986) | 0.004254 / 0.007986 (-0.003731) | 0.002827 / 0.004328 (-0.001502) | 0.050433 / 0.004250 (0.046182) | 0.046106 / 0.037052 (0.009054) | 0.301522 / 0.258489 (0.043033) | 0.325784 / 0.293841 (0.031943) | 0.030014 / 0.128546 (-0.098532) | 0.010891 / 0.075646 (-0.064756) | 0.059899 / 0.419271 (-0.359373) | 0.057252 / 0.043533 (0.013719) | 0.280276 / 0.255139 (0.025137) | 0.295632 / 0.283200 (0.012433) | 0.019060 / 0.141683 (-0.122622) | 1.141423 / 1.452155 (-0.310731) | 1.226960 / 1.492716 (-0.265757) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091919 / 0.018006 (0.073913) | 0.300769 / 0.000490 (0.300279) | 0.000220 / 0.000200 (0.000020) | 0.000053 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022467 / 0.037411 (-0.014945) | 0.075342 / 0.014526 (0.060816) | 0.087988 / 0.176557 (-0.088569) | 0.128304 / 0.737135 (-0.608831) | 0.089058 / 0.296338 (-0.207280) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.294662 / 0.215209 (0.079453) | 2.887743 / 2.077655 (0.810088) | 1.591756 / 1.504120 (0.087636) | 1.469249 / 1.541195 (-0.071945) | 1.495639 / 1.468490 (0.027149) | 0.575507 / 4.584777 (-4.009270) | 2.449674 / 3.745712 (-1.296038) | 2.737217 / 5.269862 (-2.532645) | 1.783066 / 4.565676 (-2.782610) | 0.063388 / 0.424275 (-0.360887) | 0.005044 / 0.007607 (-0.002563) | 0.344807 / 0.226044 (0.118763) | 3.410845 / 2.268929 (1.141916) | 1.967452 / 55.444624 (-53.477173) | 1.699884 / 6.876477 (-5.176593) | 1.862466 / 2.142072 (-0.279607) | 0.663714 / 4.805227 (-4.141513) | 0.118356 / 6.500664 (-6.382308) | 0.041176 / 0.075469 (-0.034293) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.013523 / 1.841788 (-0.828264) | 12.498866 / 8.074308 (4.424558) | 10.382595 / 10.191392 (0.191203) | 0.141757 / 0.680424 (-0.538667) | 0.015992 / 0.534201 (-0.518209) | 0.295639 / 0.579283 (-0.283644) | 0.278382 / 0.434364 (-0.155982) | 0.330351 / 0.540337 (-0.209986) | 0.431293 / 1.386936 (-0.955643) |

![](https://cml.dev/watermark.png#cf1aaa32eddd73076cf6600125661df4a32cb20a "CML watermark")

Update release instructions.