Minor multi gpu doc improvement

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6649). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005197 / 0.011353 (-0.006156) | 0.003469 / 0.011008 (-0.007539) | 0.062306 / 0.038508 (0.023798) | 0.028417 / 0.023109 (0.005308) | 0.241147 / 0.275898 (-0.034751) | 0.270910 / 0.323480 (-0.052569) | 0.003053 / 0.007986 (-0.004933) | 0.003343 / 0.004328 (-0.000985) | 0.048044 / 0.004250 (0.043794) | 0.043738 / 0.037052 (0.006686) | 0.259274 / 0.258489 (0.000785) | 0.282522 / 0.293841 (-0.011319) | 0.027807 / 0.128546 (-0.100739) | 0.010413 / 0.075646 (-0.065234) | 0.206322 / 0.419271 (-0.212950) | 0.035770 / 0.043533 (-0.007763) | 0.243465 / 0.255139 (-0.011674) | 0.261596 / 0.283200 (-0.021604) | 0.018613 / 0.141683 (-0.123070) | 1.115509 / 1.452155 (-0.336645) | 1.189403 / 1.492716 (-0.303314) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.086075 / 0.018006 (0.068069) | 0.296140 / 0.000490 (0.295650) | 0.000198 / 0.000200 (-0.000002) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018238 / 0.037411 (-0.019173) | 0.061783 / 0.014526 (0.047257) | 0.072014 / 0.176557 (-0.104543) | 0.118746 / 0.737135 (-0.618389) | 0.073279 / 0.296338 (-0.223060) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.278281 / 0.215209 (0.063072) | 2.772209 / 2.077655 (0.694555) | 1.404503 / 1.504120 (-0.099617) | 1.274753 / 1.541195 (-0.266441) | 1.304394 / 1.468490 (-0.164096) | 0.556903 / 4.584777 (-4.027874) | 2.335428 / 3.745712 (-1.410284) | 2.712255 / 5.269862 (-2.557606) | 1.722252 / 4.565676 (-2.843425) | 0.061268 / 0.424275 (-0.363007) | 0.005029 / 0.007607 (-0.002578) | 0.326112 / 0.226044 (0.100067) | 3.207917 / 2.268929 (0.938988) | 1.743513 / 55.444624 (-53.701111) | 1.476418 / 6.876477 (-5.400059) | 1.489776 / 2.142072 (-0.652297) | 0.628181 / 4.805227 (-4.177046) | 0.115959 / 6.500664 (-6.384706) | 0.041854 / 0.075469 (-0.033615) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.969039 / 1.841788 (-0.872749) | 11.178646 / 8.074308 (3.104338) | 9.639716 / 10.191392 (-0.551676) | 0.139750 / 0.680424 (-0.540674) | 0.014230 / 0.534201 (-0.519971) | 0.285318 / 0.579283 (-0.293965) | 0.260788 / 0.434364 (-0.173576) | 0.324183 / 0.540337 (-0.216154) | 0.416326 / 1.386936 (-0.970610) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005149 / 0.011353 (-0.006204) | 0.003469 / 0.011008 (-0.007539) | 0.049761 / 0.038508 (0.011253) | 0.030723 / 0.023109 (0.007614) | 0.271562 / 0.275898 (-0.004336) | 0.297843 / 0.323480 (-0.025637) | 0.004296 / 0.007986 (-0.003690) | 0.002704 / 0.004328 (-0.001624) | 0.048890 / 0.004250 (0.044640) | 0.044776 / 0.037052 (0.007723) | 0.285490 / 0.258489 (0.027001) | 0.312888 / 0.293841 (0.019047) | 0.046239 / 0.128546 (-0.082307) | 0.010238 / 0.075646 (-0.065408) | 0.057968 / 0.419271 (-0.361304) | 0.033295 / 0.043533 (-0.010238) | 0.274320 / 0.255139 (0.019181) | 0.296199 / 0.283200 (0.012999) | 0.017856 / 0.141683 (-0.123827) | 1.147532 / 1.452155 (-0.304622) | 1.211647 / 1.492716 (-0.281070) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.089655 / 0.018006 (0.071649) | 0.297275 / 0.000490 (0.296785) | 0.000207 / 0.000200 (0.000007) | 0.000043 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021739 / 0.037411 (-0.015672) | 0.075041 / 0.014526 (0.060515) | 0.085754 / 0.176557 (-0.090802) | 0.124512 / 0.737135 (-0.612623) | 0.086926 / 0.296338 (-0.209412) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.290306 / 0.215209 (0.075097) | 2.847404 / 2.077655 (0.769749) | 1.606175 / 1.504120 (0.102055) | 1.483220 / 1.541195 (-0.057974) | 1.514551 / 1.468490 (0.046061) | 0.559332 / 4.584777 (-4.025445) | 2.403089 / 3.745712 (-1.342624) | 2.715179 / 5.269862 (-2.554683) | 1.688340 / 4.565676 (-2.877337) | 0.062057 / 0.424275 (-0.362218) | 0.004955 / 0.007607 (-0.002652) | 0.338909 / 0.226044 (0.112865) | 3.356882 / 2.268929 (1.087954) | 1.942259 / 55.444624 (-53.502366) | 1.675195 / 6.876477 (-5.201282) | 1.688158 / 2.142072 (-0.453914) | 0.637270 / 4.805227 (-4.167957) | 0.114314 / 6.500664 (-6.386350) | 0.040677 / 0.075469 (-0.034792) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.022126 / 1.841788 (-0.819661) | 11.783359 / 8.074308 (3.709051) | 10.247652 / 10.191392 (0.056260) | 0.138188 / 0.680424 (-0.542236) | 0.014850 / 0.534201 (-0.519351) | 0.287414 / 0.579283 (-0.291869) | 0.274393 / 0.434364 (-0.159971) | 0.327255 / 0.540337 (-0.213082) | 0.416355 / 1.386936 (-0.970581) |

![](https://cml.dev/watermark.png#727a952367966a98b759d54f333b1e2c28cfd4d4 "CML watermark")

just added torch.no_grad and eval()


Document usage of hfh cli instead of git

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6648). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004951 / 0.011353 (-0.006402) | 0.003187 / 0.011008 (-0.007821) | 0.062959 / 0.038508 (0.024451) | 0.028037 / 0.023109 (0.004928) | 0.241374 / 0.275898 (-0.034524) | 0.262792 / 0.323480 (-0.060688) | 0.004132 / 0.007986 (-0.003854) | 0.002766 / 0.004328 (-0.001563) | 0.051416 / 0.004250 (0.047165) | 0.040957 / 0.037052 (0.003904) | 0.260760 / 0.258489 (0.002271) | 0.282018 / 0.293841 (-0.011823) | 0.027689 / 0.128546 (-0.100857) | 0.010433 / 0.075646 (-0.065214) | 0.211598 / 0.419271 (-0.207674) | 0.035447 / 0.043533 (-0.008086) | 0.244333 / 0.255139 (-0.010806) | 0.263192 / 0.283200 (-0.020008) | 0.016816 / 0.141683 (-0.124867) | 1.103188 / 1.452155 (-0.348967) | 1.179093 / 1.492716 (-0.313623) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092412 / 0.018006 (0.074406) | 0.301226 / 0.000490 (0.300736) | 0.000208 / 0.000200 (0.000008) | 0.000051 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018146 / 0.037411 (-0.019265) | 0.061447 / 0.014526 (0.046921) | 0.072162 / 0.176557 (-0.104394) | 0.118965 / 0.737135 (-0.618170) | 0.073756 / 0.296338 (-0.222583) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.285361 / 0.215209 (0.070152) | 2.776928 / 2.077655 (0.699273) | 1.506859 / 1.504120 (0.002739) | 1.379119 / 1.541195 (-0.162075) | 1.401798 / 1.468490 (-0.066692) | 0.572512 / 4.584777 (-4.012265) | 2.403793 / 3.745712 (-1.341919) | 2.740496 / 5.269862 (-2.529366) | 1.714611 / 4.565676 (-2.851065) | 0.063496 / 0.424275 (-0.360780) | 0.005009 / 0.007607 (-0.002598) | 0.342438 / 0.226044 (0.116393) | 3.368129 / 2.268929 (1.099200) | 1.831200 / 55.444624 (-53.613424) | 1.553611 / 6.876477 (-5.322866) | 1.578116 / 2.142072 (-0.563956) | 0.653034 / 4.805227 (-4.152193) | 0.117724 / 6.500664 (-6.382940) | 0.041188 / 0.075469 (-0.034282) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.972520 / 1.841788 (-0.869268) | 11.186297 / 8.074308 (3.111989) | 9.485829 / 10.191392 (-0.705563) | 0.139715 / 0.680424 (-0.540708) | 0.013705 / 0.534201 (-0.520496) | 0.287384 / 0.579283 (-0.291899) | 0.266784 / 0.434364 (-0.167580) | 0.320789 / 0.540337 (-0.219548) | 0.417484 / 1.386936 (-0.969452) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005570 / 0.011353 (-0.005783) | 0.003416 / 0.011008 (-0.007592) | 0.051160 / 0.038508 (0.012652) | 0.031082 / 0.023109 (0.007973) | 0.279336 / 0.275898 (0.003438) | 0.300529 / 0.323480 (-0.022951) | 0.004320 / 0.007986 (-0.003666) | 0.002781 / 0.004328 (-0.001548) | 0.049642 / 0.004250 (0.045391) | 0.044379 / 0.037052 (0.007327) | 0.293797 / 0.258489 (0.035308) | 0.317844 / 0.293841 (0.024003) | 0.049697 / 0.128546 (-0.078849) | 0.010624 / 0.075646 (-0.065023) | 0.058834 / 0.419271 (-0.360437) | 0.033869 / 0.043533 (-0.009664) | 0.280547 / 0.255139 (0.025408) | 0.300685 / 0.283200 (0.017486) | 0.017010 / 0.141683 (-0.124673) | 1.172277 / 1.452155 (-0.279878) | 1.205359 / 1.492716 (-0.287358) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092914 / 0.018006 (0.074907) | 0.303561 / 0.000490 (0.303071) | 0.000219 / 0.000200 (0.000019) | 0.000054 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022379 / 0.037411 (-0.015032) | 0.075460 / 0.014526 (0.060934) | 0.085795 / 0.176557 (-0.090762) | 0.124776 / 0.737135 (-0.612360) | 0.088260 / 0.296338 (-0.208079) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.302873 / 0.215209 (0.087664) | 2.936173 / 2.077655 (0.858519) | 1.589251 / 1.504120 (0.085131) | 1.477552 / 1.541195 (-0.063643) | 1.479322 / 1.468490 (0.010832) | 0.570481 / 4.584777 (-4.014296) | 2.434137 / 3.745712 (-1.311575) | 2.774012 / 5.269862 (-2.495849) | 1.718103 / 4.565676 (-2.847574) | 0.061951 / 0.424275 (-0.362324) | 0.004992 / 0.007607 (-0.002615) | 0.352250 / 0.226044 (0.126205) | 3.457417 / 2.268929 (1.188488) | 1.934587 / 55.444624 (-53.510037) | 1.646904 / 6.876477 (-5.229573) | 1.669429 / 2.142072 (-0.472643) | 0.649665 / 4.805227 (-4.155562) | 0.116630 / 6.500664 (-6.384034) | 0.040669 / 0.075469 (-0.034800) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.011488 / 1.841788 (-0.830300) | 11.866394 / 8.074308 (3.792086) | 10.144588 / 10.191392 (-0.046804) | 0.129931 / 0.680424 (-0.550493) | 0.014885 / 0.534201 (-0.519316) | 0.287463 / 0.579283 (-0.291821) | 0.280754 / 0.434364 (-0.153610) | 0.330139 / 0.540337 (-0.210199) | 0.414653 / 1.386936 (-0.972283) |

![](https://cml.dev/watermark.png#585275b8deaebd1bdcbd3725fa63172395791c73 "CML watermark")

(basically the same content as the hfh upload docs, but adapted for datasets)


Update loading.mdx to include "jsonl" file loading.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6647). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Thanks for adding the explicit loading command.
>
> However, I would move it just below, where we present the JSON-Lines example.
>
> * Maybe adding that this format is called JSON-Lines
> * Add the example after the JSON-Lines data example
>
> https://github.com/huggingface/datasets/blob/14d9afbb7ae1b787c450261ca0ff374551993031/docs/source/loading.mdx#L135-L138

Thank you @albertvillanova for the feedback! I moved the jsonl file loading example to a more appropriate location.

* A small update to the documentation, noting the ability to load jsonl files.


Better multi-gpu example

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6646). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005598 / 0.011353 (-0.005755) | 0.003640 / 0.011008 (-0.007369) | 0.064557 / 0.038508 (0.026049) | 0.029645 / 0.023109 (0.006536) | 0.243695 / 0.275898 (-0.032203) | 0.261252 / 0.323480 (-0.062228) | 0.004067 / 0.007986 (-0.003919) | 0.002883 / 0.004328 (-0.001446) | 0.049192 / 0.004250 (0.044942) | 0.045299 / 0.037052 (0.008246) | 0.273207 / 0.258489 (0.014718) | 0.288668 / 0.293841 (-0.005173) | 0.028114 / 0.128546 (-0.100432) | 0.010597 / 0.075646 (-0.065049) | 0.215345 / 0.419271 (-0.203927) | 0.036119 / 0.043533 (-0.007414) | 0.243718 / 0.255139 (-0.011421) | 0.266657 / 0.283200 (-0.016543) | 0.018176 / 0.141683 (-0.123507) | 1.127926 / 1.452155 (-0.324229) | 1.168066 / 1.492716 (-0.324650) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096001 / 0.018006 (0.077994) | 0.304317 / 0.000490 (0.303828) | 0.000209 / 0.000200 (0.000009) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018241 / 0.037411 (-0.019170) | 0.061505 / 0.014526 (0.046979) | 0.072456 / 0.176557 (-0.104101) | 0.118315 / 0.737135 (-0.618821) | 0.075154 / 0.296338 (-0.221184) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.278748 / 0.215209 (0.063538) | 2.729923 / 2.077655 (0.652268) | 1.416835 / 1.504120 (-0.087285) | 1.294016 / 1.541195 (-0.247179) | 1.323249 / 1.468490 (-0.145241) | 0.575389 / 4.584777 (-4.009388) | 2.404923 / 3.745712 (-1.340789) | 2.769233 / 5.269862 (-2.500629) | 1.742340 / 4.565676 (-2.823336) | 0.062664 / 0.424275 (-0.361611) | 0.004951 / 0.007607 (-0.002656) | 0.335024 / 0.226044 (0.108979) | 3.291446 / 2.268929 (1.022518) | 1.797095 / 55.444624 (-53.647530) | 1.532963 / 6.876477 (-5.343513) | 1.529315 / 2.142072 (-0.612758) | 0.654922 / 4.805227 (-4.150305) | 0.118772 / 6.500664 (-6.381892) | 0.042034 / 0.075469 (-0.033435) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.983646 / 1.841788 (-0.858141) | 11.518625 / 8.074308 (3.444317) | 9.538781 / 10.191392 (-0.652611) | 0.140300 / 0.680424 (-0.540124) | 0.013966 / 0.534201 (-0.520235) | 0.287071 / 0.579283 (-0.292212) | 0.270201 / 0.434364 (-0.164163) | 0.323294 / 0.540337 (-0.217044) | 0.418130 / 1.386936 (-0.968806) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005508 / 0.011353 (-0.005844) | 0.003714 / 0.011008 (-0.007294) | 0.050031 / 0.038508 (0.011523) | 0.031866 / 0.023109 (0.008756) | 0.272248 / 0.275898 (-0.003650) | 0.295105 / 0.323480 (-0.028375) | 0.005179 / 0.007986 (-0.002807) | 0.002820 / 0.004328 (-0.001508) | 0.048896 / 0.004250 (0.044646) | 0.045975 / 0.037052 (0.008922) | 0.287662 / 0.258489 (0.029173) | 0.321139 / 0.293841 (0.027298) | 0.049242 / 0.128546 (-0.079304) | 0.010732 / 0.075646 (-0.064914) | 0.057943 / 0.419271 (-0.361328) | 0.033527 / 0.043533 (-0.010006) | 0.271746 / 0.255139 (0.016607) | 0.291404 / 0.283200 (0.008204) | 0.019351 / 0.141683 (-0.122332) | 1.157221 / 1.452155 (-0.294934) | 1.215757 / 1.492716 (-0.276959) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096950 / 0.018006 (0.078944) | 0.312002 / 0.000490 (0.311512) | 0.000223 / 0.000200 (0.000023) | 0.000055 / 0.000054 (0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022288 / 0.037411 (-0.015123) | 0.075282 / 0.014526 (0.060756) | 0.087445 / 0.176557 (-0.089112) | 0.125617 / 0.737135 (-0.611519) | 0.088878 / 0.296338 (-0.207460) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.291961 / 0.215209 (0.076752) | 2.881445 / 2.077655 (0.803790) | 1.586128 / 1.504120 (0.082008) | 1.458636 / 1.541195 (-0.082558) | 1.487001 / 1.468490 (0.018511) | 0.575466 / 4.584777 (-4.009311) | 2.454941 / 3.745712 (-1.290771) | 2.878077 / 5.269862 (-2.391785) | 1.787215 / 4.565676 (-2.778462) | 0.064010 / 0.424275 (-0.360265) | 0.005092 / 0.007607 (-0.002516) | 0.360500 / 0.226044 (0.134455) | 3.465574 / 2.268929 (1.196646) | 1.957516 / 55.444624 (-53.487108) | 1.666282 / 6.876477 (-5.210195) | 1.690070 / 2.142072 (-0.452002) | 0.661323 / 4.805227 (-4.143905) | 0.117824 / 6.500664 (-6.382840) | 0.042286 / 0.075469 (-0.033183) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.026517 / 1.841788 (-0.815270) | 12.083347 / 8.074308 (4.009039) | 10.269319 / 10.191392 (0.077927) | 0.139253 / 0.680424 (-0.541171) | 0.016258 / 0.534201 (-0.517943) | 0.290583 / 0.579283 (-0.288700) | 0.284338 / 0.434364 (-0.150026) | 0.335865 / 0.540337 (-0.204473) | 0.416600 / 1.386936 (-0.970336) |

![](https://cml.dev/watermark.png#ba3cfad91e9366cda0ba203700fc745d8bcd1f17 "CML watermark")

Thanks, I was needing this example today <3

Use Qwen1.5-0.5B-Chat as an easy example for multi-GPU

the previous example was using a model for translation and the way it was setup was not really the right way to use the model.


Support fsspec 2024.2

enhancement

New feature or request

I'd be very grateful. This upper bound banished me straight into dependency hell today. :(

Support fsspec 2024.2.

First, we should address:
- #6644


Support fsspec 2023.12

New feature or request

The pinned fsspec version range dependency conflict has been affecting several of our users in https://github.com/iterative/dvc. I've opened an initial PR that I think should resolve the glob behavior changes with using datasets + the latest fsspec release.

Please let us know if there's any other fsspec related behavior in datasets that needs to be updated to get 2024.2 supported, we'd like to get this conflict resolved as quickly as possible and we're willing to contribute any additional work that's required here.

cc @dberenbaum

Support fsspec 2023.12 by handling previous and new glob behavior.


Faiss GPU index cannot be serialised when passed to trainer

Hi ! make sure your query embeddings are numpy arrays, not torch tensors ;)

Hi Quentin, not sure how that solves the problem number 1. I am trying to pass on a dataset with a faiss gpu for training to the standard trainer but getting this serialisation error. What is a workaround this? I do not want to remove the faiss index, as I would want to use it to create batches of retrieved samples from the dataset.
Thanks in advance for your help!

Issue number one seems to be an issue with FAISS indexes not being compatible with copy.deepcopy.

Maybe you try to not remove the columns, e.g. by passing `remove_unused_columns=False`

### Describe the bug

I am working on a retrieval project and encountering I have encountered two issues in the hugging face faiss integration:

1. I am trying to pass in a dataset with a faiss index to the Huggingface trainer. The code works for a cpu faiss index, but doesn't for a gpu one, getting error:

```
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/site-packages/transformers/trainer.py", line 1543, in train
    return inner_training_loop(
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/site-packages/transformers/trainer.py", line 1555, in _inner_training_loop
    train_dataloader = self.get_train_dataloader()
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/site-packages/transformers/trainer.py", line 831, in get_train_dataloader
    train_dataset = self._remove_unused_columns(train_dataset, description="training")
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/site-packages/transformers/trainer.py", line 725, in _remove_unused_columns
    return dataset.remove_columns(ignored_columns)
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 592, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 557, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/site-packages/datasets/fingerprint.py", line 481, in wrapper
    out = func(dataset, *args, **kwargs)
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2146, in remove_columns
    dataset = copy.deepcopy(self)
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/copy.py", line 161, in deepcopy
    rv = reductor(4)
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/site-packages/faiss/__init__.py", line 556, in index_getstate
    return {"this": serialize_index(self).tobytes()}
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/site-packages/faiss/__init__.py", line 1607, in serialize_index
    write_index(index, writer)
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/site-packages/faiss/swigfaiss.py", line 9843, in write_index
    return _swigfaiss.write_index(*args)
RuntimeError: Error in void faiss::write_index(const faiss::Index*, faiss::IOWriter*) at /project/faiss/faiss/impl/index_write.cpp:590: don't know how to serialize this type of index
```
The index was created with the add_faiss_index method
```
train_dataset.add_faiss_index(
                column='embeddings',
                index_name='embeddings',
                string_factory=faiss_index_string,
                train_size=config.faiss_train_size,
                device=0,  # Use -1 for CPU, or specify GPU device ID
                faiss_verbose=True
            )

```
2. Athough faiss is written to be compatible on the gpu for searching [https://github.com/facebookresearch/faiss/wiki/Faiss-on-the-GPU](https://github.com/facebookresearch/faiss/wiki/Faiss-on-the-GPU) I am getting error when trying to use the hugggingface code to do the search on gpu. This seems to be caused by this line https://github.com/huggingface/datasets/blob/f9975f636542df7f95c27065ea93147440d690b7/src/datasets/search.py#L376 producing error

```
    total_scores, total_examples = self.dataset.get_nearest_examples_batch('embeddings', embeddings, k=self.k)
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/site-packages/datasets/search.py", line 773, in get_nearest_examples_batch
    total_scores, total_indices = self.search_batch(index_name, queries, k, **kwargs)
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/site-packages/datasets/search.py", line 727, in search_batch
    return self._indexes[index_name].search_batch(queries, k, **kwargs)
  File "/users/rubman/.conda/envs/protein_npt_env/lib/python3.10/site-packages/datasets/search.py", line 376, in search_batch
    if not queries.flags.c_contiguous:
AttributeError: 'Tensor' object has no attribute 'flags'
```

### Steps to reproduce the bug

```
train_dataset.add_faiss_index(
                column='embeddings',
                index_name='embeddings',
                string_factory=faiss_index_string,
                train_size=config.faiss_train_size,
                device=0,  # Use -1 for CPU, or specify GPU device ID
                faiss_verbose=True
            )
Trainer(
            model=model,
            args=args,
            train_dataset=train_dataset,
            eval_dataset=eval_dataset,
            data_collator=data_collator,
            tokenizer=tokenizer
        )
train_dataset.get_nearest_examples_batch('embeddings', embeddings, k=self.k)
```

### Expected behavior

I would expect the faiss database code to be gpu compatible

### Environment info

huggingface Version: 2.16.1


Differently dataset object saved than it is loaded.

I see now, that I have to use `load_from_disk`, in order to load dataset properly, not `load_dataset`. Why is this behavior split? Why do we need both, `load_dataset` and `load_from_disk`?

Unless answered, I believe this might be helpful for other hf datasets newbies.

Anyway, made a `load_dataset` compatible dataset in a following way. I created a directory, and just copied jsonl there as `train.jsonl/test.jsonl`.
```python
output_folder = os.path.join(args.output_folder, f"{task_meta_type}_{task_type}")
os.makedirs(output_folder, exist_ok=True)
file = f"{task_meta_type}_{task_type}_train.jsonl"
shutil.copy(os.path.join(input_folder, file),
            os.path.join(output_folder, "train.jsonl"))
# now test
file = f"{task_meta_type}_{task_type}_test.jsonl"
shutil.copy(os.path.join(input_folder, file),
            os.path.join(output_folder, "test.jsonl"))
```

Hi @MFajcik,

You can find information about save_to_disk/load_from_disk in our docs:
- https://huggingface.co/docs/datasets/v2.16.1/en/process#save
- https://huggingface.co/docs/datasets/v2.16.1/en/package_reference/main_classes#datasets.Dataset.save_to_disk
- https://huggingface.co/docs/datasets/v2.16.1/en/package_reference/main_classes#datasets.Dataset.load_from_disk

### Describe the bug

Differently sized object is saved than it is loaded.

### Steps to reproduce the bug

Hi, I save dataset in a following way:
```
        dataset = load_dataset("json",
                               data_files={
                                   "train": os.path.join(input_folder, f"{task_meta_type}_{task_type}_train.jsonl"),
                                   "test": os.path.join(input_folder, f"{task_meta_type}_{task_type}_test.jsonl")})
        print(os.path.join(output_folder, f"{task_meta_type}_{task_type}"))
        print(f"Length of train dataset: {len(dataset['train'])}")
        print(f"Length of test dataset: {len(dataset['test'])}")
        dataset.save_to_disk(os.path.join(output_folder, f"{task_meta_type}_{task_type}"))
```

this yields output
```
.data/hf_dataset/propaganda_zanr
Length of train dataset: 7642
Length of test dataset: 1000
```

Everything looks fine.

Then I load the dataset
```python
from datasets import load_dataset

dataset_path = ".data/hf_dataset/propaganda_zanr"
dataset = load_dataset(dataset_path)

print(f"Length of train dataset: {len(dataset['train'])}")
print(f"Length of test dataset: {len(dataset['test'])}")

```

this prints
```
Generating train split: 1 examples [00:00, 72.10 examples/s]
Generating test split: 1 examples [00:00, 100.69 examples/s]
Length of train dataset: 1
Length of test dataset: 1
```

I dont' understand :(

### Expected behavior

same object is loaded

### Environment info

datasets==2.16.1


Hi @Hughhuh.

I have formatted the issue because it was not easily readable. Additionally, the environment info is incomplete: it seems you did not run the proposed CLI command `datasets-cli env` and essential information is missing: version of `datasets`, version of `pyarrow`,...

With the information you provided, it seems an issue with the specific "samsum" dataset. I'm transferring the issue to the corresponding dataset page: https://huggingface.co/datasets/samsum/discussions/5

### Describe the bug

unicodedecodeerror: 'utf-8' codec can't decode byte 0xac in position 25: invalid start byte

### Steps to reproduce the bug
```
import sys
sys.getdefaultencoding()
'utf-8'

from datasets import load_dataset

print(f"Train dataset size: {len(dataset['train'])}")
print(f"Test dataset size: {len(dataset['test'])}")

Resolving data files: 100%
159/159 [00:00<00:00, 9909.28it/s]
Using custom data configuration samsum-0b1209637541c9e6
Downloading and preparing dataset json/samsum to C:/Users/Administrator/.cache/huggingface/datasets/json/samsum-0b1209637541c9e6/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51...
Downloading data files: 100%
3/3 [00:00<00:00, 119.99it/s]
Extracting data files: 100%
3/3 [00:00<00:00, 9.54it/s]
Generating train split:
88392/0 [00:15<00:00, 86848.17 examples/s]
Generating test split:
0/0 [00:00<?, ? examples/s]
---------------------------------------------------------------------------
ArrowInvalid                              Traceback (most recent call last)
File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\datasets\packaged_modules\json\json.py:132, in Json._generate_tables(self, files)
    131 try:
--> 132     pa_table = paj.read_json(
    133         io.BytesIO(batch), read_options=paj.ReadOptions(block_size=block_size)
    134     )
    135     break

File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\pyarrow\_json.pyx:290, in pyarrow._json.read_json()

File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\pyarrow\error.pxi:144, in pyarrow.lib.pyarrow_internal_check_status()

File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\pyarrow\error.pxi:100, in pyarrow.lib.check_status()

ArrowInvalid: JSON parse error: Invalid value. in row 0

During handling of the above exception, another exception occurred:

UnicodeDecodeError                        Traceback (most recent call last)
File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\datasets\builder.py:1819, in ArrowBasedBuilder._prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
   1818 _time = time.time()
-> 1819 for _, table in generator:
   1820     if max_shard_size is not None and writer._num_bytes > max_shard_size:

File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\datasets\packaged_modules\json\json.py:153, in Json._generate_tables(self, files)
    152     with open(file, encoding="utf-8") as f:
--> 153         dataset = json.load(f)
    154 except json.JSONDecodeError:

File ~\AppData\Local\Programs\Python\Python310\lib\json\__init__.py:293, in load(fp, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    276 """Deserialize ``fp`` (a ``.read()``-supporting file-like object containing
    277 a JSON document) to a Python object.
    278
   (...)
    291 kwarg; otherwise ``JSONDecoder`` is used.
    292 """
--> 293 return loads(fp.read(),
    294     cls=cls, object_hook=object_hook,
    295     parse_float=parse_float, parse_int=parse_int,
    296     parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)

File ~\AppData\Local\Programs\Python\Python310\lib\codecs.py:322, in BufferedIncrementalDecoder.decode(self, input, final)
    321 data = self.buffer + input
--> 322 (result, consumed) = self._buffer_decode(data, self.errors, final)
    323 # keep undecoded input until the next call

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xac in position 25: invalid start byte

The above exception was the direct cause of the following exception:

DatasetGenerationError                    Traceback (most recent call last)
Cell In[81], line 5
      1 from datasets import load_dataset
      3 # Load dataset from the hub
      4 #dataset = load_dataset("json",data_files="C:/Users/Administrator/Desktop/samsum/samsum/data/corpus/train.json",field="data")
----> 5 dataset = load_dataset('json',"samsum")
      6 #dataset = load_dataset("samsum")
      7 print(f"Train dataset size: {len(dataset['train'])}")

File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\datasets\load.py:1758, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, num_proc, **config_kwargs)
   1755 try_from_hf_gcs = path not in _PACKAGED_DATASETS_MODULES
   1757 # Download and prepare data
-> 1758 builder_instance.download_and_prepare(
   1759     download_config=download_config,
   1760     download_mode=download_mode,
   1761     ignore_verifications=ignore_verifications,
   1762     try_from_hf_gcs=try_from_hf_gcs,
   1763     num_proc=num_proc,
   1764 )
   1766 # Build dataset for splits
   1767 keep_in_memory = (
   1768     keep_in_memory if keep_in_memory is not None else is_small_dataset(builder_instance.info.dataset_size)
   1769 )

File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\datasets\builder.py:860, in DatasetBuilder.download_and_prepare(self, output_dir, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, file_format, max_shard_size, num_proc, storage_options, **download_and_prepare_kwargs)
    858     if num_proc is not None:
    859         prepare_split_kwargs["num_proc"] = num_proc
--> 860     self._download_and_prepare(
    861         dl_manager=dl_manager,
    862         verify_infos=verify_infos,
    863         **prepare_split_kwargs,
    864         **download_and_prepare_kwargs,
    865     )
    866 # Sync info
    867 self.info.dataset_size = sum(split.num_bytes for split in self.info.splits.values())

File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\datasets\builder.py:953, in DatasetBuilder._download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    949 split_dict.add(split_generator.split_info)
    951 try:
    952     # Prepare split will record examples associated to the split
--> 953     self._prepare_split(split_generator, **prepare_split_kwargs)
    954 except OSError as e:
    955     raise OSError(
    956         "Cannot find data file. "
    957         + (self.manual_download_instructions or "")
    958         + "\nOriginal error:\n"
    959         + str(e)
    960     ) from None

File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\datasets\builder.py:1708, in ArrowBasedBuilder._prepare_split(self, split_generator, file_format, num_proc, max_shard_size)
   1706 gen_kwargs = split_generator.gen_kwargs
   1707 job_id = 0
-> 1708 for job_id, done, content in self._prepare_split_single(
   1709     gen_kwargs=gen_kwargs, job_id=job_id, **_prepare_split_args
   1710 ):
   1711     if done:
   1712         result = content

File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\datasets\builder.py:1851, in ArrowBasedBuilder._prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
   1849     if isinstance(e, SchemaInferenceError) and e.__context__ is not None:
   1850         e = e.__context__
-> 1851     raise DatasetGenerationError("An error occurred while generating the dataset") from e
   1853 yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_shards, shard_lengths)

DatasetGenerationError: An error occurred while generating the dataset
```

### Expected behavior

can't load dataset

### Environment info

dataset:samsum
system :win10
gpu:m40 24G


Sign Language Support

New feature or request

### Feature request

Currently, there are only several Sign Language labels, I would like to propose adding all the Signed Languages as new labels which are described in this ISO standard: https://www.evertype.com/standards/iso639/sign-language.html

### Motivation

Datasets currently only have labels for several signed languages. There are more signed languages in the world. Furthermore, some signed languages that have a lot of online data cannot be found because of this reason (for instance, German Sign Language, and there is no German Sign Language label on huggingface datasets even though there are a lot of readily available sign language datasets exist for German Sign Language, which are used very frequently in Sign Language Processing papers, and models.)

### Your contribution

I can submit a PR for this as well, adding the ISO codes and languages to the labels in datasets.


Run download_and_prepare if missing splits

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6639). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

A first step towards https://github.com/huggingface/datasets/issues/6529


Cannot download wmt16 dataset

Looks like it works with latest datasets repository
```
- `datasets` version: 2.16.2.dev0
- Platform: Linux-5.15.0-92-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.0.1
- `fsspec` version: 2023.10.0
```

Could you explain which is the minimum version that fixes this?
Edit: Looks like that's 2.16.0, will close out issue

### Describe the bug

As of this morning (PST) 2/1/2024, seeing the wmt16 dataset is missing from opus , could you suggest an alternative?

```
Downloading data files:   0%|                                                                                | 0/4 [00:00<?, ?it/s]Traceback (most recent call last):
  File "test.py", line 2, in <module>
    raw_datasets = load_dataset("wmt16","ro-en",split="train")
  File "/usr/local/lib/python3.8/dist-packages/datasets/load.py", line 2153, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.8/dist-packages/datasets/builder.py", line 954, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.8/dist-packages/datasets/builder.py", line 1717, in _download_and_prepare
    super()._download_and_prepare(
  File "/usr/local/lib/python3.8/dist-packages/datasets/builder.py", line 1027, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/root/.cache/huggingface/modules/datasets_modules/datasets/wmt16/746749a11d25c02058042da7502d973ff410e73457f3d305fc1177dc0e8c4227/wmt_utils.py", line 754, in _split_generators
    downloaded_files = dl_manager.download_and_extract(urls_to_download)
  File "/usr/local/lib/python3.8/dist-packages/datasets/download/download_manager.py", line 565, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/usr/local/lib/python3.8/dist-packages/datasets/download/download_manager.py", line 428, in download
    downloaded_path_or_paths = map_nested(
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/py_utils.py", line 464, in map_nested
    mapped = [
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/py_utils.py", line 465, in <listcomp>
    _single_map_nested((function, obj, types, None, True, None))
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/py_utils.py", line 384, in _single_map_nested
    mapped = [_single_map_nested((function, v, types, None, True, None)) for v in pbar]
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/py_utils.py", line 384, in <listcomp>
    mapped = [_single_map_nested((function, v, types, None, True, None)) for v in pbar]
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/py_utils.py", line 367, in _single_map_nested
    return function(data_struct)
  File "/usr/local/lib/python3.8/dist-packages/datasets/download/download_manager.py", line 454, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/file_utils.py", line 182, in cached_path
    output_path = get_from_cache(
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/file_utils.py", line 596, in get_from_cache
    raise FileNotFoundError(f"Couldn't find file at {url}")
FileNotFoundError: Couldn't find file at https://opus.nlpl.eu/download.php?f=SETIMES/v2/tmx/en-ro.tmx.gz
```

### Steps to reproduce the bug

```
from datasets import load_dataset
raw_datasets = load_dataset("wmt16","ro-en",split="train")
```

### Expected behavior

Expect the dataset to be downloaded/ at least a clean exit with error explaining dataset is missing and a suggestion for next steps

### Environment info

- `datasets` version: 2.14.7
- Platform: Linux-5.15.0-92-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.17.3
- PyArrow version: 15.0.0
- Pandas version: 2.0.1


'with_format' is extremely slow when used together with 'interleave_datasets' or 'shuffle' on IterableDatasets

The "torch" formatting is usually fast because we do zero-copy conversion from the Arrow data on your disk to Torch tensors. However IterableDataset shuffling seems to do data copies that slow down the pipeline, and it shuffles python objects instead of Arrow data.

To fix this we need to implement `BufferShuffledExamplesIterable.iter_arrow()` (same as regular `BufferShuffledExamplesIterable.__iter__()` but yields Arrow tables)

https://github.com/huggingface/datasets/blob/b7d854b7fd3e9a330e21b76ee8421d4a7ebb4a7a/src/datasets/iterable_dataset.py#L968-L974

### Describe the bug

If you:

1. Interleave two iterable datasets together with the interleave_datasets function, or shuffle an iterable dataset
2. Set the output format to torch tensors with .with_format('torch')

Then iterating through the dataset becomes over 100x slower than it is if you don't apply the torch formatting.

### Steps to reproduce the bug

```python
import datasets
import torch
from tqdm import tqdm

rand_a = torch.randn(3,224,224)
rand_b = torch.randn(3,224,224)

a = torch.stack([rand_a] * 1000)
b = torch.stack([rand_b] * 1000)

features = datasets.Features({"tensor": datasets.Array3D(shape=(3,224,224), dtype="float32")})
ds_a = datasets.Dataset.from_dict({"tensor": a}, features=features).to_iterable_dataset()
ds_b = datasets.Dataset.from_dict({"tensor": b}, features=features).to_iterable_dataset()

# Iterating through either dataset with torch formatting is really fast (2000it/s on my machine)
for example in tqdm(ds_a.with_format('torch')):
    pass

# Iterating through either dataset shuffled is also pretty fast (100it/s on my machine)
for example in tqdm(ds_a.shuffle()):
    pass

# Iterating through this interleaved dataset is pretty fast (200it/s on my machine)
ds_fast = datasets.interleave_datasets([ds_a, ds_b])
for example in tqdm(ds_fast):
    pass

# Iterating through either dataset with torch formatting *after shuffling* is really slow... (<2it/s on my machine)
for example in tqdm(ds_a.shuffle().with_format('torch')):
    pass

# Iterating through this torch formatted interleaved dataset is also really slow (<2it/s on my machine)...
ds_slow = datasets.interleave_datasets([ds_a, ds_b]).with_format('torch')
for example in tqdm(ds_slow):
    pass

# Even doing this is way faster!! (70it/s on my machine)
for example in tqdm(ds_fast):
    test = torch.tensor(example['tensor'])

```

### Expected behavior

Applying torch formatting to the interleaved dataset shouldn't increase the time taken to iterate through the dataset by very much, since even explicitly converting every example is over 70x faster than calling .with_format('torch').

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-6.5.0-15-generic-x86_64-with-glibc2.38
- Python version: 3.11.6
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0


Faster column validation and reordering

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6636). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks @mariosasko, I made the changes. However, I did some tests with `map` and I still saw that it took ~3.5 minutes per batch on 6000 features when using `dataset.map(lambda x: x, batched=True)`. From the profile, the culprits were mainly with `ArrowWriter.write_batch` and `ArrowWriter._build_writer`. The slow down from `_build_writer` is due to updating existing features with the inferred ones. I don't think this can be optimized any further, but fortunately, I can avoid this by setting the `features` in `map`. On the other hand, `write_batch` selects cols based on intersection and difference between schema names and example keys using two for loops. The same exists in `ArrowWriter.write_examples_on_file`. Optimizing the column selection using set operations effectively brings it from 3.5 minutes per batch down to 6 seconds per batch. Can we add these changes along with this PR?

Edit: Ah just realized you can avoid the issue with inferring features altogether when you set the format to arrow (or pandas).

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004990 / 0.011353 (-0.006363) | 0.003138 / 0.011008 (-0.007870) | 0.062368 / 0.038508 (0.023860) | 0.028634 / 0.023109 (0.005524) | 0.241297 / 0.275898 (-0.034601) | 0.264433 / 0.323480 (-0.059047) | 0.003133 / 0.007986 (-0.004852) | 0.003444 / 0.004328 (-0.000885) | 0.048522 / 0.004250 (0.044271) | 0.043700 / 0.037052 (0.006648) | 0.257054 / 0.258489 (-0.001435) | 0.277551 / 0.293841 (-0.016290) | 0.027132 / 0.128546 (-0.101414) | 0.010395 / 0.075646 (-0.065251) | 0.208003 / 0.419271 (-0.211269) | 0.035814 / 0.043533 (-0.007719) | 0.250098 / 0.255139 (-0.005041) | 0.266726 / 0.283200 (-0.016474) | 0.018424 / 0.141683 (-0.123259) | 1.129242 / 1.452155 (-0.322912) | 1.167674 / 1.492716 (-0.325042) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091808 / 0.018006 (0.073802) | 0.298726 / 0.000490 (0.298236) | 0.000219 / 0.000200 (0.000019) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019119 / 0.037411 (-0.018292) | 0.061969 / 0.014526 (0.047443) | 0.073392 / 0.176557 (-0.103165) | 0.119460 / 0.737135 (-0.617675) | 0.074072 / 0.296338 (-0.222266) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.281435 / 0.215209 (0.066226) | 2.702094 / 2.077655 (0.624439) | 1.411541 / 1.504120 (-0.092579) | 1.284084 / 1.541195 (-0.257111) | 1.302638 / 1.468490 (-0.165852) | 0.562420 / 4.584777 (-4.022357) | 2.364890 / 3.745712 (-1.380822) | 2.744033 / 5.269862 (-2.525828) | 1.699000 / 4.565676 (-2.866677) | 0.062315 / 0.424275 (-0.361961) | 0.004982 / 0.007607 (-0.002625) | 0.334385 / 0.226044 (0.108341) | 3.203268 / 2.268929 (0.934339) | 1.766998 / 55.444624 (-53.677627) | 1.497164 / 6.876477 (-5.379313) | 1.509996 / 2.142072 (-0.632077) | 0.633014 / 4.805227 (-4.172213) | 0.115317 / 6.500664 (-6.385347) | 0.041120 / 0.075469 (-0.034349) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.965877 / 1.841788 (-0.875911) | 11.219909 / 8.074308 (3.145601) | 9.333822 / 10.191392 (-0.857570) | 0.136482 / 0.680424 (-0.543941) | 0.013632 / 0.534201 (-0.520569) | 0.287251 / 0.579283 (-0.292032) | 0.262786 / 0.434364 (-0.171578) | 0.322893 / 0.540337 (-0.217444) | 0.418180 / 1.386936 (-0.968756) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005444 / 0.011353 (-0.005909) | 0.003147 / 0.011008 (-0.007862) | 0.049242 / 0.038508 (0.010734) | 0.030944 / 0.023109 (0.007834) | 0.281901 / 0.275898 (0.006003) | 0.303820 / 0.323480 (-0.019660) | 0.004326 / 0.007986 (-0.003659) | 0.002696 / 0.004328 (-0.001632) | 0.048306 / 0.004250 (0.044055) | 0.044145 / 0.037052 (0.007093) | 0.297253 / 0.258489 (0.038764) | 0.324062 / 0.293841 (0.030221) | 0.046724 / 0.128546 (-0.081823) | 0.010079 / 0.075646 (-0.065567) | 0.057635 / 0.419271 (-0.361636) | 0.033621 / 0.043533 (-0.009912) | 0.282303 / 0.255139 (0.027164) | 0.300761 / 0.283200 (0.017561) | 0.017116 / 0.141683 (-0.124567) | 1.156519 / 1.452155 (-0.295636) | 1.216087 / 1.492716 (-0.276630) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093011 / 0.018006 (0.075005) | 0.301310 / 0.000490 (0.300820) | 0.000223 / 0.000200 (0.000023) | 0.000053 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023112 / 0.037411 (-0.014299) | 0.075192 / 0.014526 (0.060666) | 0.086213 / 0.176557 (-0.090343) | 0.125853 / 0.737135 (-0.611282) | 0.087754 / 0.296338 (-0.208585) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.301095 / 0.215209 (0.085886) | 2.911769 / 2.077655 (0.834114) | 1.614708 / 1.504120 (0.110588) | 1.494497 / 1.541195 (-0.046698) | 1.506978 / 1.468490 (0.038488) | 0.572743 / 4.584777 (-4.012034) | 2.417142 / 3.745712 (-1.328570) | 2.755338 / 5.269862 (-2.514523) | 1.711026 / 4.565676 (-2.854650) | 0.062732 / 0.424275 (-0.361543) | 0.005031 / 0.007607 (-0.002576) | 0.352343 / 0.226044 (0.126298) | 3.465183 / 2.268929 (1.196255) | 1.958795 / 55.444624 (-53.485829) | 1.682239 / 6.876477 (-5.194238) | 1.688897 / 2.142072 (-0.453176) | 0.643311 / 4.805227 (-4.161916) | 0.115426 / 6.500664 (-6.385238) | 0.040338 / 0.075469 (-0.035131) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.005322 / 1.841788 (-0.836466) | 11.779380 / 8.074308 (3.705072) | 10.041574 / 10.191392 (-0.149818) | 0.127617 / 0.680424 (-0.552807) | 0.015840 / 0.534201 (-0.518361) | 0.286905 / 0.579283 (-0.292378) | 0.275180 / 0.434364 (-0.159183) | 0.332498 / 0.540337 (-0.207840) | 0.410719 / 1.386936 (-0.976217) |

![](https://cml.dev/watermark.png#32b206d47f582380f9c64578dcfa6c48252db3b8 "CML watermark")

I work with bioinformatics data and often these tables have thousands and even tens of thousands of features. These tables are also accompanied by metadata that I do not want to pass in the model. When I perform `set_format('pt', columns=large_column_list)` , it can take several minutes before it finishes. The culprit is when the following check is performed: `any(col not in self._data.column_names for col in columns)`. Replacing this by  `set(columns) - (self._data.column_names)` is more efficient.


Fix missing info when loading some datasets from Parquet export

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6635). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005577 / 0.011353 (-0.005776) | 0.004452 / 0.011008 (-0.006556) | 0.067849 / 0.038508 (0.029341) | 0.032328 / 0.023109 (0.009219) | 0.256924 / 0.275898 (-0.018974) | 0.273410 / 0.323480 (-0.050070) | 0.004359 / 0.007986 (-0.003626) | 0.003484 / 0.004328 (-0.000845) | 0.053880 / 0.004250 (0.049630) | 0.058142 / 0.037052 (0.021089) | 0.268863 / 0.258489 (0.010374) | 0.307977 / 0.293841 (0.014136) | 0.028840 / 0.128546 (-0.099707) | 0.011808 / 0.075646 (-0.063839) | 0.216277 / 0.419271 (-0.202995) | 0.039245 / 0.043533 (-0.004288) | 0.250420 / 0.255139 (-0.004719) | 0.273642 / 0.283200 (-0.009557) | 0.019340 / 0.141683 (-0.122342) | 1.176734 / 1.452155 (-0.275421) | 1.250643 / 1.492716 (-0.242074) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.181210 / 0.018006 (0.163204) | 1.070750 / 0.000490 (1.070261) | 0.000315 / 0.000200 (0.000115) | 0.000045 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022905 / 0.037411 (-0.014507) | 0.064549 / 0.014526 (0.050023) | 0.077113 / 0.176557 (-0.099443) | 0.131976 / 0.737135 (-0.605159) | 0.081266 / 0.296338 (-0.215072) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.291363 / 0.215209 (0.076154) | 2.851691 / 2.077655 (0.774036) | 1.592815 / 1.504120 (0.088695) | 1.494550 / 1.541195 (-0.046645) | 1.516464 / 1.468490 (0.047974) | 0.583244 / 4.584777 (-4.001532) | 2.504907 / 3.745712 (-1.240805) | 3.183490 / 5.269862 (-2.086371) | 1.932854 / 4.565676 (-2.632823) | 0.067564 / 0.424275 (-0.356711) | 0.006587 / 0.007607 (-0.001020) | 0.346368 / 0.226044 (0.120324) | 3.428256 / 2.268929 (1.159327) | 1.994176 / 55.444624 (-53.450448) | 1.688116 / 6.876477 (-5.188360) | 1.767653 / 2.142072 (-0.374420) | 0.673867 / 4.805227 (-4.131360) | 0.125582 / 6.500664 (-6.375082) | 0.047198 / 0.075469 (-0.028271) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.002895 / 1.841788 (-0.838893) | 16.332893 / 8.074308 (8.258585) | 10.781993 / 10.191392 (0.590601) | 0.153919 / 0.680424 (-0.526505) | 0.015528 / 0.534201 (-0.518673) | 0.306182 / 0.579283 (-0.273101) | 0.296380 / 0.434364 (-0.137984) | 0.341432 / 0.540337 (-0.198905) | 0.455900 / 1.386936 (-0.931036) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006442 / 0.011353 (-0.004911) | 0.004433 / 0.011008 (-0.006576) | 0.053327 / 0.038508 (0.014819) | 0.035966 / 0.023109 (0.012856) | 0.280913 / 0.275898 (0.005015) | 0.308419 / 0.323480 (-0.015061) | 0.005842 / 0.007986 (-0.002144) | 0.003789 / 0.004328 (-0.000539) | 0.053983 / 0.004250 (0.049732) | 0.069052 / 0.037052 (0.032000) | 0.299225 / 0.258489 (0.040736) | 0.336470 / 0.293841 (0.042629) | 0.068170 / 0.128546 (-0.060377) | 0.012259 / 0.075646 (-0.063388) | 0.064166 / 0.419271 (-0.355106) | 0.037291 / 0.043533 (-0.006241) | 0.281318 / 0.255139 (0.026179) | 0.297093 / 0.283200 (0.013893) | 0.021358 / 0.141683 (-0.120324) | 1.189584 / 1.452155 (-0.262571) | 1.256985 / 1.492716 (-0.235731) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.216726 / 0.018006 (0.198720) | 2.496957 / 0.000490 (2.496467) | 0.000336 / 0.000200 (0.000136) | 0.000070 / 0.000054 (0.000016) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026604 / 0.037411 (-0.010807) | 0.080398 / 0.014526 (0.065873) | 0.094475 / 0.176557 (-0.082082) | 0.136263 / 0.737135 (-0.600873) | 0.097898 / 0.296338 (-0.198440) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295171 / 0.215209 (0.079962) | 2.947530 / 2.077655 (0.869875) | 1.607531 / 1.504120 (0.103411) | 1.485045 / 1.541195 (-0.056150) | 1.524899 / 1.468490 (0.056409) | 0.572934 / 4.584777 (-4.011843) | 2.544320 / 3.745712 (-1.201393) | 3.292630 / 5.269862 (-1.977232) | 1.927138 / 4.565676 (-2.638539) | 0.068560 / 0.424275 (-0.355715) | 0.005982 / 0.007607 (-0.001625) | 0.345833 / 0.226044 (0.119789) | 3.424253 / 2.268929 (1.155324) | 2.195017 / 55.444624 (-53.249608) | 1.712037 / 6.876477 (-5.164440) | 1.763899 / 2.142072 (-0.378174) | 0.653776 / 4.805227 (-4.151451) | 0.123056 / 6.500664 (-6.377609) | 0.044572 / 0.075469 (-0.030897) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.033400 / 1.841788 (-0.808388) | 15.409887 / 8.074308 (7.335579) | 11.220990 / 10.191392 (1.029597) | 0.153603 / 0.680424 (-0.526821) | 0.016866 / 0.534201 (-0.517335) | 0.311945 / 0.579283 (-0.267338) | 0.307048 / 0.434364 (-0.127316) | 0.350422 / 0.540337 (-0.189915) | 0.447308 / 1.386936 (-0.939628) |

![](https://cml.dev/watermark.png#14d9afbb7ae1b787c450261ca0ff374551993031 "CML watermark")

Fix getting the info for script-based datasets with Parquet export with a single config not named "default".

E.g.

```python
from datasets import load_dataset_builder

b = load_dataset_builder("bookcorpus")
print(b.info.features)
# should print {'text': Value(dtype='string', id=None)}
```

I fixed this by setting the default config name when there is only one config.


Support data_dir parameter in push_to_hub

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6634). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@huggingface/datasets, feel free to review this PR so that it can be included in the next release.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005125 / 0.011353 (-0.006228) | 0.003772 / 0.011008 (-0.007236) | 0.063258 / 0.038508 (0.024750) | 0.029479 / 0.023109 (0.006370) | 0.245554 / 0.275898 (-0.030344) | 0.266395 / 0.323480 (-0.057085) | 0.003063 / 0.007986 (-0.004922) | 0.003298 / 0.004328 (-0.001031) | 0.049242 / 0.004250 (0.044991) | 0.042390 / 0.037052 (0.005338) | 0.258176 / 0.258489 (-0.000313) | 0.279935 / 0.293841 (-0.013906) | 0.027910 / 0.128546 (-0.100637) | 0.011033 / 0.075646 (-0.064613) | 0.207763 / 0.419271 (-0.211509) | 0.036127 / 0.043533 (-0.007405) | 0.247363 / 0.255139 (-0.007776) | 0.261309 / 0.283200 (-0.021890) | 0.020259 / 0.141683 (-0.121424) | 1.152760 / 1.452155 (-0.299395) | 1.194853 / 1.492716 (-0.297863) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.088936 / 0.018006 (0.070930) | 0.298396 / 0.000490 (0.297906) | 0.000211 / 0.000200 (0.000011) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018434 / 0.037411 (-0.018977) | 0.061991 / 0.014526 (0.047466) | 0.072786 / 0.176557 (-0.103771) | 0.120437 / 0.737135 (-0.616698) | 0.078375 / 0.296338 (-0.217964) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.275821 / 0.215209 (0.060612) | 2.703358 / 2.077655 (0.625703) | 1.446783 / 1.504120 (-0.057337) | 1.333556 / 1.541195 (-0.207639) | 1.325753 / 1.468490 (-0.142737) | 0.565196 / 4.584777 (-4.019581) | 2.411193 / 3.745712 (-1.334520) | 2.702764 / 5.269862 (-2.567098) | 1.727425 / 4.565676 (-2.838252) | 0.062966 / 0.424275 (-0.361309) | 0.004985 / 0.007607 (-0.002622) | 0.333473 / 0.226044 (0.107428) | 3.270615 / 2.268929 (1.001687) | 1.822213 / 55.444624 (-53.622411) | 1.546572 / 6.876477 (-5.329905) | 1.568767 / 2.142072 (-0.573305) | 0.655907 / 4.805227 (-4.149321) | 0.117173 / 6.500664 (-6.383491) | 0.042415 / 0.075469 (-0.033054) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.987966 / 1.841788 (-0.853822) | 11.851206 / 8.074308 (3.776898) | 10.327751 / 10.191392 (0.136359) | 0.127929 / 0.680424 (-0.552494) | 0.013781 / 0.534201 (-0.520420) | 0.286910 / 0.579283 (-0.292373) | 0.273615 / 0.434364 (-0.160749) | 0.323373 / 0.540337 (-0.216965) | 0.426407 / 1.386936 (-0.960529) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005412 / 0.011353 (-0.005941) | 0.003619 / 0.011008 (-0.007389) | 0.049603 / 0.038508 (0.011095) | 0.031246 / 0.023109 (0.008136) | 0.279723 / 0.275898 (0.003825) | 0.298557 / 0.323480 (-0.024923) | 0.004253 / 0.007986 (-0.003733) | 0.002758 / 0.004328 (-0.001570) | 0.048931 / 0.004250 (0.044680) | 0.044245 / 0.037052 (0.007193) | 0.295876 / 0.258489 (0.037387) | 0.322720 / 0.293841 (0.028879) | 0.046746 / 0.128546 (-0.081800) | 0.010841 / 0.075646 (-0.064805) | 0.058528 / 0.419271 (-0.360744) | 0.034224 / 0.043533 (-0.009308) | 0.279192 / 0.255139 (0.024053) | 0.299775 / 0.283200 (0.016576) | 0.017862 / 0.141683 (-0.123820) | 1.154478 / 1.452155 (-0.297677) | 1.190483 / 1.492716 (-0.302234) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.088717 / 0.018006 (0.070710) | 0.297905 / 0.000490 (0.297415) | 0.000209 / 0.000200 (0.000009) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021458 / 0.037411 (-0.015953) | 0.075616 / 0.014526 (0.061090) | 0.087080 / 0.176557 (-0.089476) | 0.125315 / 0.737135 (-0.611821) | 0.088958 / 0.296338 (-0.207381) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.287085 / 0.215209 (0.071876) | 2.807798 / 2.077655 (0.730143) | 1.552201 / 1.504120 (0.048081) | 1.422374 / 1.541195 (-0.118820) | 1.437908 / 1.468490 (-0.030582) | 0.569738 / 4.584777 (-4.015039) | 2.493921 / 3.745712 (-1.251791) | 2.648376 / 5.269862 (-2.621486) | 1.741721 / 4.565676 (-2.823955) | 0.063023 / 0.424275 (-0.361253) | 0.005166 / 0.007607 (-0.002441) | 0.336927 / 0.226044 (0.110882) | 3.384517 / 2.268929 (1.115588) | 1.909888 / 55.444624 (-53.534736) | 1.641879 / 6.876477 (-5.234597) | 1.727734 / 2.142072 (-0.414338) | 0.647127 / 4.805227 (-4.158100) | 0.115831 / 6.500664 (-6.384833) | 0.041161 / 0.075469 (-0.034309) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.016310 / 1.841788 (-0.825477) | 12.088500 / 8.074308 (4.014192) | 10.799730 / 10.191392 (0.608338) | 0.129049 / 0.680424 (-0.551375) | 0.015379 / 0.534201 (-0.518822) | 0.291352 / 0.579283 (-0.287931) | 0.284579 / 0.434364 (-0.149785) | 0.331214 / 0.540337 (-0.209124) | 0.422902 / 1.386936 (-0.964034) |

![](https://cml.dev/watermark.png#991169ed4901d129d0e0ab8d7fccd6a0728da4b8 "CML watermark")

Support `data_dir` parameter in `push_to_hub`.

This allows users to organize the data files according to their specific needs. For example, "wikimedia/wikipedia" files could be organized by year and/or date, e.g. "2024/20240101/20240101.en".


dataset viewer requires no-script

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6633). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005172 / 0.011353 (-0.006181) | 0.003694 / 0.011008 (-0.007314) | 0.063098 / 0.038508 (0.024590) | 0.028161 / 0.023109 (0.005052) | 0.262288 / 0.275898 (-0.013610) | 0.281867 / 0.323480 (-0.041613) | 0.004088 / 0.007986 (-0.003898) | 0.002745 / 0.004328 (-0.001583) | 0.049071 / 0.004250 (0.044820) | 0.040629 / 0.037052 (0.003577) | 0.282766 / 0.258489 (0.024277) | 0.297998 / 0.293841 (0.004157) | 0.028057 / 0.128546 (-0.100489) | 0.010878 / 0.075646 (-0.064768) | 0.207410 / 0.419271 (-0.211861) | 0.035600 / 0.043533 (-0.007933) | 0.260157 / 0.255139 (0.005018) | 0.273252 / 0.283200 (-0.009948) | 0.017403 / 0.141683 (-0.124280) | 1.150798 / 1.452155 (-0.301356) | 1.200485 / 1.492716 (-0.292231) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093783 / 0.018006 (0.075777) | 0.302112 / 0.000490 (0.301622) | 0.000225 / 0.000200 (0.000025) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018254 / 0.037411 (-0.019158) | 0.061083 / 0.014526 (0.046557) | 0.074899 / 0.176557 (-0.101657) | 0.119616 / 0.737135 (-0.617520) | 0.075269 / 0.296338 (-0.221069) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.275878 / 0.215209 (0.060669) | 2.694778 / 2.077655 (0.617123) | 1.423810 / 1.504120 (-0.080310) | 1.309444 / 1.541195 (-0.231750) | 1.327898 / 1.468490 (-0.140592) | 0.568621 / 4.584777 (-4.016155) | 2.345849 / 3.745712 (-1.399863) | 2.901281 / 5.269862 (-2.368580) | 1.777959 / 4.565676 (-2.787717) | 0.063539 / 0.424275 (-0.360736) | 0.005011 / 0.007607 (-0.002596) | 0.331212 / 0.226044 (0.105168) | 3.200379 / 2.268929 (0.931451) | 1.780766 / 55.444624 (-53.663859) | 1.517178 / 6.876477 (-5.359299) | 1.587307 / 2.142072 (-0.554765) | 0.651939 / 4.805227 (-4.153288) | 0.116646 / 6.500664 (-6.384018) | 0.043325 / 0.075469 (-0.032144) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.996894 / 1.841788 (-0.844894) | 11.495397 / 8.074308 (3.421089) | 10.255784 / 10.191392 (0.064392) | 0.129006 / 0.680424 (-0.551418) | 0.013967 / 0.534201 (-0.520234) | 0.284847 / 0.579283 (-0.294436) | 0.265610 / 0.434364 (-0.168754) | 0.320176 / 0.540337 (-0.220162) | 0.429526 / 1.386936 (-0.957410) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005582 / 0.011353 (-0.005771) | 0.003867 / 0.011008 (-0.007142) | 0.050416 / 0.038508 (0.011908) | 0.030996 / 0.023109 (0.007887) | 0.275987 / 0.275898 (0.000089) | 0.289487 / 0.323480 (-0.033993) | 0.005149 / 0.007986 (-0.002837) | 0.002806 / 0.004328 (-0.001522) | 0.049617 / 0.004250 (0.045366) | 0.046949 / 0.037052 (0.009897) | 0.281596 / 0.258489 (0.023107) | 0.330948 / 0.293841 (0.037108) | 0.049645 / 0.128546 (-0.078901) | 0.010953 / 0.075646 (-0.064693) | 0.058546 / 0.419271 (-0.360725) | 0.034010 / 0.043533 (-0.009523) | 0.270525 / 0.255139 (0.015386) | 0.289749 / 0.283200 (0.006550) | 0.018755 / 0.141683 (-0.122927) | 1.163072 / 1.452155 (-0.289082) | 1.213400 / 1.492716 (-0.279316) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092397 / 0.018006 (0.074390) | 0.299376 / 0.000490 (0.298886) | 0.000211 / 0.000200 (0.000011) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022496 / 0.037411 (-0.014916) | 0.076886 / 0.014526 (0.062361) | 0.087186 / 0.176557 (-0.089371) | 0.126092 / 0.737135 (-0.611044) | 0.088832 / 0.296338 (-0.207507) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.288885 / 0.215209 (0.073676) | 2.839851 / 2.077655 (0.762196) | 1.587556 / 1.504120 (0.083436) | 1.470249 / 1.541195 (-0.070945) | 1.518080 / 1.468490 (0.049590) | 0.569646 / 4.584777 (-4.015131) | 2.417574 / 3.745712 (-1.328138) | 2.737368 / 5.269862 (-2.532494) | 1.784419 / 4.565676 (-2.781257) | 0.064104 / 0.424275 (-0.360171) | 0.005138 / 0.007607 (-0.002469) | 0.346214 / 0.226044 (0.120169) | 3.439541 / 2.268929 (1.170612) | 1.944792 / 55.444624 (-53.499832) | 1.675762 / 6.876477 (-5.200714) | 1.851871 / 2.142072 (-0.290201) | 0.652932 / 4.805227 (-4.152295) | 0.118953 / 6.500664 (-6.381711) | 0.041011 / 0.075469 (-0.034459) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.017690 / 1.841788 (-0.824098) | 12.610531 / 8.074308 (4.536223) | 11.223165 / 10.191392 (1.031773) | 0.131637 / 0.680424 (-0.548786) | 0.016733 / 0.534201 (-0.517468) | 0.288491 / 0.579283 (-0.290792) | 0.275899 / 0.434364 (-0.158465) | 0.331837 / 0.540337 (-0.208500) | 0.421695 / 1.386936 (-0.965241) |

![](https://cml.dev/watermark.png#5d9dfa9a8c077c783729a279623926faa9e2f3f1 "CML watermark")