Fix wrong type hints in data_files

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6910). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005135 / 0.011353 (-0.006218) | 0.003757 / 0.011008 (-0.007251) | 0.063122 / 0.038508 (0.024614) | 0.029837 / 0.023109 (0.006727) | 0.246120 / 0.275898 (-0.029778) | 0.268529 / 0.323480 (-0.054951) | 0.004136 / 0.007986 (-0.003849) | 0.002650 / 0.004328 (-0.001678) | 0.048749 / 0.004250 (0.044499) | 0.045279 / 0.037052 (0.008226) | 0.257970 / 0.258489 (-0.000519) | 0.285993 / 0.293841 (-0.007848) | 0.027612 / 0.128546 (-0.100935) | 0.010175 / 0.075646 (-0.065471) | 0.207373 / 0.419271 (-0.211899) | 0.037672 / 0.043533 (-0.005861) | 0.249603 / 0.255139 (-0.005536) | 0.271081 / 0.283200 (-0.012119) | 0.018174 / 0.141683 (-0.123509) | 1.116703 / 1.452155 (-0.335452) | 1.169261 / 1.492716 (-0.323455) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095161 / 0.018006 (0.077155) | 0.301112 / 0.000490 (0.300623) | 0.000221 / 0.000200 (0.000021) | 0.000042 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023218 / 0.037411 (-0.014193) | 0.063125 / 0.014526 (0.048599) | 0.075857 / 0.176557 (-0.100699) | 0.137922 / 0.737135 (-0.599213) | 0.076989 / 0.296338 (-0.219349) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.279272 / 0.215209 (0.064063) | 2.776463 / 2.077655 (0.698809) | 1.472220 / 1.504120 (-0.031900) | 1.347105 / 1.541195 (-0.194090) | 1.361014 / 1.468490 (-0.107476) | 0.589233 / 4.584777 (-3.995544) | 2.395212 / 3.745712 (-1.350500) | 2.794855 / 5.269862 (-2.475007) | 1.698350 / 4.565676 (-2.867327) | 0.063328 / 0.424275 (-0.360947) | 0.005020 / 0.007607 (-0.002588) | 0.335872 / 0.226044 (0.109828) | 3.293486 / 2.268929 (1.024558) | 1.837270 / 55.444624 (-53.607354) | 1.535694 / 6.876477 (-5.340782) | 1.559696 / 2.142072 (-0.582376) | 0.639302 / 4.805227 (-4.165925) | 0.116554 / 6.500664 (-6.384110) | 0.042305 / 0.075469 (-0.033164) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.971562 / 1.841788 (-0.870226) | 11.710500 / 8.074308 (3.636192) | 9.505935 / 10.191392 (-0.685457) | 0.139161 / 0.680424 (-0.541263) | 0.014351 / 0.534201 (-0.519850) | 0.285790 / 0.579283 (-0.293493) | 0.265718 / 0.434364 (-0.168646) | 0.323558 / 0.540337 (-0.216780) | 0.412635 / 1.386936 (-0.974301) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005987 / 0.011353 (-0.005366) | 0.003787 / 0.011008 (-0.007221) | 0.049839 / 0.038508 (0.011331) | 0.032817 / 0.023109 (0.009708) | 0.268304 / 0.275898 (-0.007594) | 0.303409 / 0.323480 (-0.020071) | 0.004924 / 0.007986 (-0.003061) | 0.002740 / 0.004328 (-0.001589) | 0.048906 / 0.004250 (0.044655) | 0.044266 / 0.037052 (0.007213) | 0.290506 / 0.258489 (0.032017) | 0.314124 / 0.293841 (0.020283) | 0.030242 / 0.128546 (-0.098304) | 0.010555 / 0.075646 (-0.065091) | 0.058849 / 0.419271 (-0.360423) | 0.033540 / 0.043533 (-0.009993) | 0.267833 / 0.255139 (0.012694) | 0.291056 / 0.283200 (0.007857) | 0.018611 / 0.141683 (-0.123072) | 1.137620 / 1.452155 (-0.314534) | 1.199554 / 1.492716 (-0.293162) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096716 / 0.018006 (0.078709) | 0.302033 / 0.000490 (0.301543) | 0.000217 / 0.000200 (0.000017) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023208 / 0.037411 (-0.014203) | 0.076231 / 0.014526 (0.061705) | 0.088672 / 0.176557 (-0.087884) | 0.129033 / 0.737135 (-0.608103) | 0.090709 / 0.296338 (-0.205630) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.297033 / 0.215209 (0.081824) | 2.951181 / 2.077655 (0.873526) | 1.567690 / 1.504120 (0.063570) | 1.436809 / 1.541195 (-0.104385) | 1.469696 / 1.468490 (0.001206) | 0.567963 / 4.584777 (-4.016813) | 0.954168 / 3.745712 (-2.791544) | 2.700473 / 5.269862 (-2.569389) | 1.742144 / 4.565676 (-2.823532) | 0.065027 / 0.424275 (-0.359248) | 0.005319 / 0.007607 (-0.002288) | 0.346459 / 0.226044 (0.120415) | 3.446117 / 2.268929 (1.177189) | 1.953142 / 55.444624 (-53.491483) | 1.639131 / 6.876477 (-5.237346) | 1.830664 / 2.142072 (-0.311409) | 0.657807 / 4.805227 (-4.147420) | 0.117987 / 6.500664 (-6.382678) | 0.040726 / 0.075469 (-0.034744) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.992666 / 1.841788 (-0.849122) | 12.305377 / 8.074308 (4.231069) | 10.274829 / 10.191392 (0.083437) | 0.141731 / 0.680424 (-0.538692) | 0.015100 / 0.534201 (-0.519101) | 0.282298 / 0.579283 (-0.296985) | 0.124301 / 0.434364 (-0.310063) | 0.320914 / 0.540337 (-0.219424) | 0.445855 / 1.386936 (-0.941081) |

![](https://cml.dev/watermark.png#3b66daa02b3307079a90fbfd13856e9bec0fc1ab "CML watermark")

Fix wrong type hints in data_files introduced in:
- #6493


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6909). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005375 / 0.011353 (-0.005978) | 0.004005 / 0.011008 (-0.007003) | 0.062407 / 0.038508 (0.023899) | 0.032241 / 0.023109 (0.009131) | 0.256092 / 0.275898 (-0.019806) | 0.285740 / 0.323480 (-0.037740) | 0.004146 / 0.007986 (-0.003839) | 0.002831 / 0.004328 (-0.001497) | 0.049179 / 0.004250 (0.044928) | 0.048303 / 0.037052 (0.011251) | 0.270841 / 0.258489 (0.012352) | 0.303209 / 0.293841 (0.009368) | 0.027642 / 0.128546 (-0.100905) | 0.010661 / 0.075646 (-0.064985) | 0.201999 / 0.419271 (-0.217272) | 0.036532 / 0.043533 (-0.007001) | 0.262441 / 0.255139 (0.007302) | 0.280944 / 0.283200 (-0.002256) | 0.018369 / 0.141683 (-0.123314) | 1.122249 / 1.452155 (-0.329906) | 1.171352 / 1.492716 (-0.321364) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096433 / 0.018006 (0.078427) | 0.297272 / 0.000490 (0.296782) | 0.000222 / 0.000200 (0.000023) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019645 / 0.037411 (-0.017766) | 0.062744 / 0.014526 (0.048219) | 0.076096 / 0.176557 (-0.100460) | 0.121882 / 0.737135 (-0.615253) | 0.076267 / 0.296338 (-0.220072) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.274159 / 0.215209 (0.058950) | 2.729371 / 2.077655 (0.651716) | 1.454328 / 1.504120 (-0.049792) | 1.330517 / 1.541195 (-0.210678) | 1.338832 / 1.468490 (-0.129658) | 0.600252 / 4.584777 (-3.984525) | 2.388658 / 3.745712 (-1.357054) | 2.837717 / 5.269862 (-2.432145) | 1.747329 / 4.565676 (-2.818347) | 0.064620 / 0.424275 (-0.359655) | 0.004955 / 0.007607 (-0.002653) | 0.340253 / 0.226044 (0.114209) | 3.351559 / 2.268929 (1.082630) | 1.822718 / 55.444624 (-53.621907) | 1.518663 / 6.876477 (-5.357814) | 1.548066 / 2.142072 (-0.594006) | 0.663525 / 4.805227 (-4.141702) | 0.118334 / 6.500664 (-6.382331) | 0.042060 / 0.075469 (-0.033410) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.976509 / 1.841788 (-0.865278) | 11.703321 / 8.074308 (3.629013) | 9.305605 / 10.191392 (-0.885787) | 0.131016 / 0.680424 (-0.549408) | 0.014299 / 0.534201 (-0.519902) | 0.293963 / 0.579283 (-0.285320) | 0.264018 / 0.434364 (-0.170345) | 0.330265 / 0.540337 (-0.210073) | 0.427239 / 1.386936 (-0.959697) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005437 / 0.011353 (-0.005916) | 0.003774 / 0.011008 (-0.007234) | 0.049927 / 0.038508 (0.011419) | 0.032246 / 0.023109 (0.009137) | 0.271808 / 0.275898 (-0.004090) | 0.295652 / 0.323480 (-0.027828) | 0.004220 / 0.007986 (-0.003766) | 0.002803 / 0.004328 (-0.001525) | 0.049656 / 0.004250 (0.045406) | 0.041938 / 0.037052 (0.004885) | 0.282199 / 0.258489 (0.023710) | 0.310206 / 0.293841 (0.016365) | 0.030389 / 0.128546 (-0.098157) | 0.010593 / 0.075646 (-0.065054) | 0.057862 / 0.419271 (-0.361409) | 0.033937 / 0.043533 (-0.009596) | 0.268920 / 0.255139 (0.013781) | 0.286000 / 0.283200 (0.002800) | 0.018766 / 0.141683 (-0.122917) | 1.118556 / 1.452155 (-0.333599) | 1.175083 / 1.492716 (-0.317633) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095135 / 0.018006 (0.077129) | 0.304735 / 0.000490 (0.304245) | 0.000210 / 0.000200 (0.000010) | 0.000054 / 0.000054 (-0.000000) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022971 / 0.037411 (-0.014441) | 0.076204 / 0.014526 (0.061678) | 0.090801 / 0.176557 (-0.085756) | 0.130149 / 0.737135 (-0.606987) | 0.090986 / 0.296338 (-0.205352) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.298535 / 0.215209 (0.083326) | 2.882959 / 2.077655 (0.805304) | 1.574018 / 1.504120 (0.069899) | 1.445251 / 1.541195 (-0.095944) | 1.483651 / 1.468490 (0.015160) | 0.572012 / 4.584777 (-4.012765) | 0.972223 / 3.745712 (-2.773489) | 2.745776 / 5.269862 (-2.524085) | 1.783980 / 4.565676 (-2.781697) | 0.063910 / 0.424275 (-0.360365) | 0.005397 / 0.007607 (-0.002210) | 0.349104 / 0.226044 (0.123059) | 3.433303 / 2.268929 (1.164374) | 1.961506 / 55.444624 (-53.483119) | 1.665905 / 6.876477 (-5.210571) | 1.800977 / 2.142072 (-0.341095) | 0.655843 / 4.805227 (-4.149384) | 0.118320 / 6.500664 (-6.382345) | 0.041748 / 0.075469 (-0.033722) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.006835 / 1.841788 (-0.834952) | 12.506123 / 8.074308 (4.431815) | 10.564310 / 10.191392 (0.372918) | 0.143121 / 0.680424 (-0.537303) | 0.016340 / 0.534201 (-0.517861) | 0.284181 / 0.579283 (-0.295102) | 0.125975 / 0.434364 (-0.308389) | 0.324369 / 0.540337 (-0.215969) | 0.443713 / 1.386936 (-0.943223) |

![](https://cml.dev/watermark.png#60d21efbc01e15d0b596ac1072750cbecd91548a "CML watermark")

Update requests >=2.32.1 to fix vulnerability.


Fail to load "stas/c4-en-10k" dataset since 2.16 version

I am not able to reproduce the error with datasets 2.19.1:
```python
In [1]: from datasets import load_dataset; ds = load_dataset("stas/c4-en-10k", streaming=True); item = next(iter(ds["train"])); item
Out[1]: {'text': 'Beginners BBQ Class Taking Place in Missoula!\nDo you want to get better at making delicious BBQ? You will have the opportunity, put this on your calendar now. Thursday, September 22nd join World Class BBQ Champion, Tony Balay from Lonestar Smoke Rangers. He will be teaching a beginner level class for everyone who wants to get better with their culinary skills.\nHe will teach you everything you need to know to compete in a KCBS BBQ competition, including techniques, recipes, timelines, meat selection and trimming, plus smoker and fire information.\nThe cost to be in the class is $35 per person, and for spectators it is free. Included in the cost will be either a t-shirt or apron and you will be tasting samples of each meat that is prepared.'}

In [2]: from datasets import load_dataset; ds = load_dataset("stas/c4-en-10k", download_mode="force_redownload"); ds
Downloading data: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 13.3M/13.3M [00:00<00:00, 18.7MB/s]
Generating train split: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10000/10000 [00:00<00:00, 78548.55 examples/s]
Out[2]:
DatasetDict({
    train: Dataset({
        features: ['text'],
        num_rows: 10000
    })
})
```

Looking at your error traceback, I notice that the code line numbers do not correspond to the ones of datasets 2.19.1.

Additionally, I can't reproduce the issue with `HfFileSystem`:
```python
In [1]: from huggingface_hub import HfFileSystem

In [2]: fs = HfFileSystem()

In [3]: with fs.open("datasets/stas/c4-en-10k/c4-en-10k.py", "rb") as f:
   ...:     data = f.read()
   ...:

In [4]: data[:20]
Out[4]: b'# coding=utf-8\n# Cop'
```

Could you please verify the `datasets` and `huggingface_hub` versions you are indeed using?
```python
import datasets; print(datasets.__version__)

import huggingface_hub; print(huggingface_hub.__version__)
```

Thanks for your reply! After I update the datasets version from 2.15.0 back to 2.19.1 again, it seems everything work well. Sorry for bordering you!

### Describe the bug

When update datasets library to version 2.16+ ( I test it on 2.16, 2.19.0 and 2.19.1), using the following code to load stas/c4-en-10k dataset

```python
from datasets import load_dataset, Dataset
dataset = load_dataset('stas/c4-en-10k')
```

and then it raise UnicodeDecodeError like

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/*/conda3/envs/watermark/lib/python3.10/site-packages/datasets/load.py", line 2523, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/*/conda3/envs/watermark/lib/python3.10/site-packages/datasets/load.py", line 2195, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/home/*/conda3/envs/watermark/lib/python3.10/site-packages/datasets/load.py", line 1846, in dataset_module_factory
    raise e1 from None
  File "/home/*/conda3/envs/watermark/lib/python3.10/site-packages/datasets/load.py", line 1798, in dataset_module_factory
    can_load_config_from_parquet_export = "DEFAULT_CONFIG_NAME" not in f.read()
  File "/home/*/conda3/envs/watermark/lib/python3.10/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
```

I found that fs.open loads a gzip file and parses it like plain text using utf-8 encoder.

```python
fs = HfFileSystem('https://huggingface.co')
fs.open("datasets/stas/c4-en-10k/c4-en-10k.py", "rb")
data = fs.read()    # data is gzip bytes begin with b'\x1f\x8b\x08\x00\x00\tn\x88\x00...'
data2 = unzip_gzip_bytes(data)    #  data2 is what we want: '# coding=utf-8\n# Copyright 2020 The HuggingFace Datasets...'
```

### Steps to reproduce the bug

1. Install datasets between version 2.16 and 2.19
2. Use `datasets.load_dataset` method to load `stas/c4-en-10k` dataset.

### Expected behavior

Load dataset normally.

### Environment info

Platform = Linux-5.4.0-159-generic-x86_64-with-glibc2.35
Python = 3.10.14
Datasets = 2.19


Support the deserialization of json lines files comprised of lists

enhancement

New feature or request

Update: I ended up deciding to go back to use lines of dictionaries instead of arrays, not because of this issue as my users would be capable of downloading my corpus without `datasets`, but the speed and storage savings are not currently worth breaking my API and harming the backwards compatibility of each new revision.

With that said, for a static dataset that is not regularly updated like mine, and particularly for extremely large datasets with millions or billions of rows, using arrays could have a meaningful impact, and so there is probably still value in supporting this structure, provided the effort is not too much.

### Feature request

I manage a somewhat large and popular Hugging Face dataset known as the [Open Australian Legal Corpus](https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus). I recently updated my corpus to be stored in a json lines file where each line is an array and each element represents a value at a particular column. Previously, my corpus was stored as a json lines file where each line was a dictionary and the keys were the fields.

Essentially, a line in my json lines file used to look like this:
```json
{"version_id":"","type":"","jurisdiction":"","source":"","citation":"","url":"","when_scraped":"","text":""}
```

And now it looks like this:
```json
["","","","","","","",""]
```

This saves 65 bytes per document and allows me very quickly serialise and deserialise documents via `msgspec`.

After making this change, I found that `datasets` was incapable of deserialising my Corpus without a custom loading script, even if I ensured that the `dataset_info` field in my dataset card contained the desired names of my features.

I would like to request that functionality be added to support this format which is more memory-efficent and faster than using dictionaries.

### Motivation

The [documentation](https://huggingface.co/docs/datasets/en/dataset_script) for creating dataset loading scripts asserts that:
> In the next major release, the new safety features of 🤗 Datasets will disable running dataset loading scripts by default, and you will have to pass trust_remote_code=True to load datasets that require running a dataset script.

I would rather not require my users to pass `trust_remote_code=True` which means that I will need built-in support for this format.

### Your contribution

I would be happy to submit a PR for this if this is something you would incorporate into `datasets` and if I can be pointed to where the code would need to go.


irc_disentangle - Issue with splitting data

Thank you I will try this out!

On Tue, Jun 11, 2024 at 3:55 AM Vincent Lau ***@***.***>
wrote:

> I add a "streaming=True" after the name of the dataset, and it
> works.....hope it can help you
>
> And if you install the version datasets==2.15.0, this bug will not happen.
> I don't know why, but all of them works
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/6906#issuecomment-2160041812>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A3HXU7AMBT2MNO34SC3Z5G3ZG2UOXAVCNFSM6AAAAABH45CNPWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRQGA2DCOBRGI>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>

I still find out that there are some strange bug in v2.15.0 of datasets. it seems like that the *.arrow file cannot be established. it may be an index of the subsets. well I still try to debug it. but, one of the most efficient way may be using the google colab to build this index in the ~/huggingface/datasets, and than download them to replace the local file.....lol......it works!

Yeah I did try what you suggested and it didn’t work. I was able to get it
on a local from someone who access the dataset in the past. Let me know
when you end up fixing this bug.

On Tue, Jun 11, 2024 at 10:33 PM Vincent Lau ***@***.***>
wrote:

> I still find out that there are some strange bug in v2.15.0 of datasets.
> it seems like that the *.arrow file cannot be established. it may be an
> index of the subsets. well I still try to debug it. but, one of the most
> efficient way may be using the google colab to build this index in the
> ~/huggingface/datasets, and than download them to replace the local
> file.....lol......it works!
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/6906#issuecomment-2161988798>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A3HXU7BCJE2LOCWRVWPMNODZG6XPJAVCNFSM6AAAAABH45CNPWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRRHE4DQNZZHA>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>

### Describe the bug

I am trying to access your database through python using "datasets.load_dataset("irc_disentangle")" and I am getting this error message:

ValueError: Instruction "train" corresponds to no data!

### Steps to reproduce the bug

import datasets
ds = datasets.load_dataset('irc_disentangle')
ds

### Expected behavior

The data is supposed to load into ds and be accessable as such:
ds['train'][1050], ds['train'][1055]

### Environment info

I tired Python 3.12 and 3.10


Extraction protocol for arrow files is not defined

### Describe the bug

Passing files with `.arrow` extension into data_files argument, at least when `streaming=True` is very slow.

### Steps to reproduce the bug

 Basically it goes through the `_get_extraction_protocol` method located [here](https://github.com/huggingface/datasets/blob/main/src/datasets/utils/file_utils.py#L820)
The method then looks at some base known extensions where `arrow` is not defined so it proceeds to determine the compression with the magic number method which is slow when dealing with a lot of files which are stored in s3 and by looking at this predefined list, I don't see `arrow` in there either so in the end it return None:

```
MAGIC_NUMBER_TO_COMPRESSION_PROTOCOL = {
    bytes.fromhex("504B0304"): "zip",
    bytes.fromhex("504B0506"): "zip",  # empty archive
    bytes.fromhex("504B0708"): "zip",  # spanned archive
    bytes.fromhex("425A68"): "bz2",
    bytes.fromhex("1F8B"): "gzip",
    bytes.fromhex("FD377A585A00"): "xz",
    bytes.fromhex("04224D18"): "lz4",
    bytes.fromhex("28B52FFD"): "zstd",
}
```

### Expected behavior

My expectation is that `arrow` would be in the known lists so it would return None without going through the magic number method.

### Environment info

datasets 2.19.0


Fix decoding multi part extension

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6904). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

takign the liberty to merge this for the viewer and a new dataset being released

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005004 / 0.011353 (-0.006349) | 0.003352 / 0.011008 (-0.007657) | 0.063035 / 0.038508 (0.024527) | 0.032031 / 0.023109 (0.008922) | 0.244801 / 0.275898 (-0.031097) | 0.270622 / 0.323480 (-0.052857) | 0.003110 / 0.007986 (-0.004876) | 0.002629 / 0.004328 (-0.001700) | 0.048784 / 0.004250 (0.044534) | 0.045779 / 0.037052 (0.008726) | 0.258642 / 0.258489 (0.000153) | 0.291606 / 0.293841 (-0.002235) | 0.028237 / 0.128546 (-0.100310) | 0.010184 / 0.075646 (-0.065463) | 0.202455 / 0.419271 (-0.216816) | 0.036012 / 0.043533 (-0.007521) | 0.248209 / 0.255139 (-0.006930) | 0.267315 / 0.283200 (-0.015884) | 0.019249 / 0.141683 (-0.122434) | 1.120420 / 1.452155 (-0.331735) | 1.169515 / 1.492716 (-0.323201) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095193 / 0.018006 (0.077187) | 0.300544 / 0.000490 (0.300055) | 0.000214 / 0.000200 (0.000014) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019001 / 0.037411 (-0.018411) | 0.061857 / 0.014526 (0.047331) | 0.073379 / 0.176557 (-0.103178) | 0.121293 / 0.737135 (-0.615843) | 0.075665 / 0.296338 (-0.220673) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.285153 / 0.215209 (0.069944) | 2.875527 / 2.077655 (0.797873) | 1.479851 / 1.504120 (-0.024269) | 1.360691 / 1.541195 (-0.180504) | 1.385581 / 1.468490 (-0.082909) | 0.566312 / 4.584777 (-4.018465) | 2.400202 / 3.745712 (-1.345510) | 2.719241 / 5.269862 (-2.550620) | 1.706469 / 4.565676 (-2.859208) | 0.062129 / 0.424275 (-0.362146) | 0.005291 / 0.007607 (-0.002316) | 0.334585 / 0.226044 (0.108540) | 3.293347 / 2.268929 (1.024419) | 1.790490 / 55.444624 (-53.654134) | 1.505519 / 6.876477 (-5.370958) | 1.527730 / 2.142072 (-0.614343) | 0.644554 / 4.805227 (-4.160673) | 0.119775 / 6.500664 (-6.380889) | 0.056912 / 0.075469 (-0.018557) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.977512 / 1.841788 (-0.864275) | 11.293883 / 8.074308 (3.219575) | 9.669439 / 10.191392 (-0.521953) | 0.129910 / 0.680424 (-0.550514) | 0.014322 / 0.534201 (-0.519879) | 0.284967 / 0.579283 (-0.294316) | 0.265355 / 0.434364 (-0.169008) | 0.321965 / 0.540337 (-0.218372) | 0.415254 / 1.386936 (-0.971682) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005138 / 0.011353 (-0.006215) | 0.003321 / 0.011008 (-0.007687) | 0.049731 / 0.038508 (0.011223) | 0.032307 / 0.023109 (0.009198) | 0.266331 / 0.275898 (-0.009567) | 0.290863 / 0.323480 (-0.032617) | 0.004151 / 0.007986 (-0.003835) | 0.002684 / 0.004328 (-0.001644) | 0.048760 / 0.004250 (0.044510) | 0.042251 / 0.037052 (0.005199) | 0.280414 / 0.258489 (0.021925) | 0.305089 / 0.293841 (0.011248) | 0.029118 / 0.128546 (-0.099428) | 0.010276 / 0.075646 (-0.065370) | 0.057790 / 0.419271 (-0.361482) | 0.033290 / 0.043533 (-0.010243) | 0.267250 / 0.255139 (0.012111) | 0.285233 / 0.283200 (0.002034) | 0.018587 / 0.141683 (-0.123096) | 1.136198 / 1.452155 (-0.315957) | 1.185274 / 1.492716 (-0.307442) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096355 / 0.018006 (0.078349) | 0.301827 / 0.000490 (0.301337) | 0.000216 / 0.000200 (0.000016) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022607 / 0.037411 (-0.014805) | 0.075724 / 0.014526 (0.061198) | 0.088197 / 0.176557 (-0.088359) | 0.127864 / 0.737135 (-0.609271) | 0.089294 / 0.296338 (-0.207044) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.289321 / 0.215209 (0.074112) | 2.832456 / 2.077655 (0.754802) | 1.559208 / 1.504120 (0.055088) | 1.426229 / 1.541195 (-0.114966) | 1.424564 / 1.468490 (-0.043926) | 0.557754 / 4.584777 (-4.027023) | 0.940179 / 3.745712 (-2.805533) | 2.713640 / 5.269862 (-2.556222) | 1.697583 / 4.565676 (-2.868093) | 0.062024 / 0.424275 (-0.362251) | 0.005270 / 0.007607 (-0.002337) | 0.339450 / 0.226044 (0.113406) | 3.333024 / 2.268929 (1.064096) | 1.946087 / 55.444624 (-53.498537) | 1.601057 / 6.876477 (-5.275420) | 1.599862 / 2.142072 (-0.542210) | 0.642838 / 4.805227 (-4.162390) | 0.120470 / 6.500664 (-6.380194) | 0.040815 / 0.075469 (-0.034654) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.012904 / 1.841788 (-0.828884) | 11.917035 / 8.074308 (3.842727) | 9.717822 / 10.191392 (-0.473570) | 0.141730 / 0.680424 (-0.538694) | 0.015750 / 0.534201 (-0.518451) | 0.284470 / 0.579283 (-0.294813) | 0.125662 / 0.434364 (-0.308702) | 0.380740 / 0.540337 (-0.159598) | 0.418119 / 1.386936 (-0.968817) |

![](https://cml.dev/watermark.png#b3f772468b2bbf77a7510e265f9d41e9eb77d53f "CML watermark")

e.g. a field named `url.txt` should be a treated as text

I also included a small fix to support .npz correctly


Add the option of saving in parquet instead of arrow

New feature or request

I think [`Dataset.to_parquet`](https://huggingface.co/docs/datasets/v1.10.2/package_reference/main_classes.html#datasets.Dataset.to_parquet) is what you're looking for.

Let me know if I'm wrong

No, it does not save the metadata json.

We have to recode all meta json load/save
with another custome functions.

save_to_disk
and load should have option with
“Parquet” instead of “arrow”

since “arrow” is never user for production
(only parquet).

Thanks !

> On May 17, 2024, at 5:38, Frédéric Branchaud-Charron ***@***.***> wrote:
>
>
> I think Dataset.to_parquet is what you're looking for.
>
> Let me know if I'm wrong
>
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you authored the thread.

You can use `to_parquet` and `ds.info.write_to_directory()` to save the dataset info

Ok,

What about loading ?

Should we do in 2 steps ?

> On Jun 14, 2024, at 1:09, Quentin Lhoest ***@***.***> wrote:
>
>
> You can use to_parquet and ds.info.write_to_directory() to save the dataset info
>
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you authored the thread.

Isn’t easier to combine both
into load_dataset and save_dataset
with parquet options.

2) another question,
How can we download large dataset into disk directly without loading all in memory (!)

> On Jun 14, 2024, at 19:54, Quentin Lhoest ***@***.***> wrote:
>
>
> Yes, and there is DatasetInfo.from_directory(). to reload the info
>
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you authored the thread.

`load_dataset` doesn't load the dataset in memory, it progressively writes to disk in Arrow format and then memory maps the Arrow files. This allows to load datasets bigger than memory and without filling your RAM

Sure.
How memory map is managed ?
Managed by the OS ?

Why the need of save_dataset() ?

> On Jun 15, 2024, at 0:06, Quentin Lhoest ***@***.***> wrote:
>
>
> load_dataset doesn't load the dataset in memory, it progressively writes to disk in Arrow format and then memory maps the Arrow files. This allows to load datasets bigger than memory and without filling your RAM
>
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you authored the thread.

### Feature request

In dataset.save_to_disk('/path/to/save/dataset'),

add the option to save in parquet format

dataset.save_to_disk('/path/to/save/dataset', format="parquet"),

because arrow is not used for Production Big data.... (only parquet)

### Motivation

because arrow is not used for Production Big data.... (only parquet)

### Your contribution

I can do the testing !


Make CLI convert_to_parquet not raise error if no rights to create script branch

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6902). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005026 / 0.011353 (-0.006327) | 0.003672 / 0.011008 (-0.007336) | 0.062776 / 0.038508 (0.024268) | 0.032056 / 0.023109 (0.008947) | 0.245359 / 0.275898 (-0.030540) | 0.269371 / 0.323480 (-0.054109) | 0.004205 / 0.007986 (-0.003780) | 0.002774 / 0.004328 (-0.001555) | 0.048958 / 0.004250 (0.044708) | 0.046442 / 0.037052 (0.009390) | 0.263924 / 0.258489 (0.005434) | 0.291854 / 0.293841 (-0.001987) | 0.027299 / 0.128546 (-0.101248) | 0.010332 / 0.075646 (-0.065315) | 0.202677 / 0.419271 (-0.216595) | 0.037732 / 0.043533 (-0.005801) | 0.246028 / 0.255139 (-0.009111) | 0.272100 / 0.283200 (-0.011099) | 0.018497 / 0.141683 (-0.123186) | 1.101192 / 1.452155 (-0.350962) | 1.149683 / 1.492716 (-0.343033) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.097838 / 0.018006 (0.079832) | 0.305598 / 0.000490 (0.305108) | 0.000230 / 0.000200 (0.000030) | 0.000045 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019489 / 0.037411 (-0.017922) | 0.061902 / 0.014526 (0.047376) | 0.074825 / 0.176557 (-0.101732) | 0.121664 / 0.737135 (-0.615472) | 0.076440 / 0.296338 (-0.219898) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.279194 / 0.215209 (0.063985) | 2.756777 / 2.077655 (0.679123) | 1.429298 / 1.504120 (-0.074822) | 1.313423 / 1.541195 (-0.227771) | 1.340466 / 1.468490 (-0.128024) | 0.556349 / 4.584777 (-4.028428) | 2.355910 / 3.745712 (-1.389802) | 2.806733 / 5.269862 (-2.463128) | 1.741903 / 4.565676 (-2.823773) | 0.061556 / 0.424275 (-0.362719) | 0.005477 / 0.007607 (-0.002130) | 0.327856 / 0.226044 (0.101812) | 3.283092 / 2.268929 (1.014164) | 1.797776 / 55.444624 (-53.646848) | 1.498683 / 6.876477 (-5.377794) | 1.518501 / 2.142072 (-0.623572) | 0.632267 / 4.805227 (-4.172960) | 0.116505 / 6.500664 (-6.384159) | 0.042446 / 0.075469 (-0.033023) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.982841 / 1.841788 (-0.858947) | 11.709436 / 8.074308 (3.635128) | 9.570519 / 10.191392 (-0.620873) | 0.141968 / 0.680424 (-0.538456) | 0.014299 / 0.534201 (-0.519902) | 0.285101 / 0.579283 (-0.294182) | 0.267118 / 0.434364 (-0.167246) | 0.324720 / 0.540337 (-0.215617) | 0.423626 / 1.386936 (-0.963310) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005567 / 0.011353 (-0.005786) | 0.003703 / 0.011008 (-0.007306) | 0.050516 / 0.038508 (0.012008) | 0.032617 / 0.023109 (0.009508) | 0.276546 / 0.275898 (0.000648) | 0.299798 / 0.323480 (-0.023682) | 0.004282 / 0.007986 (-0.003704) | 0.002719 / 0.004328 (-0.001609) | 0.049424 / 0.004250 (0.045173) | 0.042924 / 0.037052 (0.005871) | 0.287785 / 0.258489 (0.029296) | 0.315490 / 0.293841 (0.021649) | 0.029533 / 0.128546 (-0.099013) | 0.010575 / 0.075646 (-0.065071) | 0.058210 / 0.419271 (-0.361061) | 0.033269 / 0.043533 (-0.010263) | 0.273325 / 0.255139 (0.018186) | 0.291762 / 0.283200 (0.008563) | 0.018922 / 0.141683 (-0.122761) | 1.118913 / 1.452155 (-0.333242) | 1.175554 / 1.492716 (-0.317162) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.099920 / 0.018006 (0.081914) | 0.317188 / 0.000490 (0.316698) | 0.000211 / 0.000200 (0.000011) | 0.000045 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022297 / 0.037411 (-0.015114) | 0.077775 / 0.014526 (0.063249) | 0.090239 / 0.176557 (-0.086317) | 0.130498 / 0.737135 (-0.606638) | 0.092010 / 0.296338 (-0.204328) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.293534 / 0.215209 (0.078325) | 2.866070 / 2.077655 (0.788415) | 1.547147 / 1.504120 (0.043027) | 1.419684 / 1.541195 (-0.121510) | 1.432128 / 1.468490 (-0.036362) | 0.571365 / 4.584777 (-4.013412) | 0.968879 / 3.745712 (-2.776833) | 2.797415 / 5.269862 (-2.472446) | 1.767821 / 4.565676 (-2.797856) | 0.063281 / 0.424275 (-0.360994) | 0.005072 / 0.007607 (-0.002535) | 0.344547 / 0.226044 (0.118502) | 3.383888 / 2.268929 (1.114959) | 1.879537 / 55.444624 (-53.565087) | 1.598392 / 6.876477 (-5.278085) | 1.627788 / 2.142072 (-0.514284) | 0.641199 / 4.805227 (-4.164028) | 0.116349 / 6.500664 (-6.384315) | 0.041940 / 0.075469 (-0.033529) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.002494 / 1.841788 (-0.839294) | 12.310056 / 8.074308 (4.235748) | 9.819718 / 10.191392 (-0.371674) | 0.134745 / 0.680424 (-0.545679) | 0.016223 / 0.534201 (-0.517978) | 0.284791 / 0.579283 (-0.294492) | 0.124665 / 0.434364 (-0.309699) | 0.381601 / 0.540337 (-0.158737) | 0.413007 / 1.386936 (-0.973929) |

![](https://cml.dev/watermark.png#6255b36be14ae22890c78749575f1f0793901f14 "CML watermark")

Make CLI convert_to_parquet not raise error if no rights to create "script" branch.

Not that before this PR, the error was not critical because it was raised at the end of the script, once all the rest of the steps were already performed.

Fix #6901.

Bug introduced in datasets-2.19.0 by:
- #6809


HTTPError 403 raised by CLI convert_to_parquet when creating script branch on 3rd party repos

bug

Something isn't working

CLI convert_to_parquet cannot create "script" branch on 3rd party repos.

It can only create it on repos where the user executing the script has write access.

Otherwise, a 403 Forbidden HTTPError is raised:
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_errors.py", line 304, in hf_raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://huggingface.co/api/datasets/ORG/DATASET/branch/script

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/datasets-cli", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/datasets/commands/datasets_cli.py", line 41, in main
    service.run()
  File "/usr/local/lib/python3.10/dist-packages/datasets/commands/convert_to_parquet.py", line 92, in run
    create_branch(dataset_id, branch="script", repo_type="dataset", token=token, exist_ok=True)
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/hf_api.py", line 5503, in create_branch
    hf_raise_for_status(response)
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_errors.py", line 367, in hf_raise_for_status
    raise HfHubHTTPError(message, response=response) from e
huggingface_hub.utils._errors.HfHubHTTPError:  (Request ID: Root=1-6645ee0d-4db1ed8a1fbe04956be15897;139a6e23-df7d-4f62-b5ba-adb6d8e6e696)

403 Forbidden: Forbidden: cannot write to script.
Cannot access content at: https://huggingface.co/api/datasets/ORG/DATASET/branch/script.
If you are trying to create or update content,make sure you have a token with the `write` role.
```


[WebDataset] KeyError with user-defined `Features` when a field is missing in an example

It shouldn't be difficult, I think it's just a matter of adding the missing fields from `self.config.features` in `example` here: before it iterates on image_field_names and audio_field_names. A missing field should have a value set to None

https://github.com/huggingface/datasets/blob/768cb35ede5a6c35fa7545aa3671f3e321c96440/src/datasets/packaged_modules/webdataset/webdataset.py#L113-L116

@lhoestq So like this then?

```
def _generate_examples(self, tar_paths, tar_iterators):
        image_field_names = [
            field_name for field_name, feature in self.info.features.items() if isinstance(feature, datasets.Image)
        ]
        audio_field_names = [
            field_name for field_name, feature in self.info.features.items() if isinstance(feature, datasets.Audio)
        ]

        all_field_names = list(self.config.features.keys())

        for tar_idx, (tar_path, tar_iterator) in enumerate(zip(tar_paths, tar_iterators)):
            for example_idx, example in enumerate(self._get_pipeline_from_tar(tar_path, tar_iterator)):
                for field_name in all_field_names:
                    if field_name not in example:
                        if field_name in self.config.features:
                            example[field_name] = self.config.features[field_name]
                        else:
                            example[field_name] = None

                # Process image and audio fields
                for field_name in image_field_names + audio_field_names:
                    if example[field_name] is not None:
                        example[field_name] = {"path": example["__key__"] + "." + field_name, "bytes": example[field_name]}

                yield f"{tar_idx}_{example_idx}", example
```

Or should we avoid trying add the missing values and just set them to None?

```
            for field_name in all_field_names:
                if field_name not in example:
                    example[field_name] = None
```

reported at https://huggingface.co/datasets/ProGamerGov/synthetic-dataset-1m-dalle3-high-quality-captions/discussions/1

```
File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/packaged_modules/webdataset/webdataset.py", line 109, in _generate_examples
    example[field_name] = {"path": example["__key__"] + "." + field_name, "bytes": example[field_name]}
```


List of dictionary features get standardized

### Describe the bug

Hi, i’m trying to create a HF dataset from a list using Dataset.from_list.

Each sample in the list is a dict with the same keys (which will be my features). The values for each feature are a list of dictionaries, and each such dictionary has a different set of keys. However, the datasets library standardizes all dictionaries under a feature and adds all possible keys (with None value) from all the dictionaries under that feature.

How can I keep the same set of keys as in the original list for each dictionary under a feature?

### Steps to reproduce the bug

```
from datasets import Dataset

# Define a function to generate a sample with "tools" feature
def generate_sample():
    # Generate random sample data
    sample_data = {
        "text": "Sample text",
        "feature_1": []
    }

    # Add feature_1 with random keys for this sample
    feature_1 = [{"key1": "value1"}, {"key2": "value2"}]  # Example feature_1 with random keys
    sample_data["feature_1"].extend(feature_1)

    return sample_data

# Generate multiple samples
num_samples = 10
samples = [generate_sample() for _ in range(num_samples)]

# Create a Hugging Face Dataset
dataset = Dataset.from_list(samples)
dataset[0]
```

```{'text': 'Sample text', 'feature_1': [{'key1': 'value1', 'key2': None}, {'key1': None, 'key2': 'value2'}]}```

### Expected behavior

```{'text': 'Sample text', 'feature_1': [{'key1': 'value1'}, {'key2': 'value2'}]}```

### Environment info

- `datasets` version: 2.19.1
- Platform: Linux-5.15.0-1040-nvidia-x86_64-with-glibc2.35
- Python version: 3.10.13
- `huggingface_hub` version: 0.23.0
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0


Remove tasks

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6999). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Remove tasks, as part of the 3.0 release.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6998). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005396 / 0.011353 (-0.005957) | 0.003974 / 0.011008 (-0.007034) | 0.063490 / 0.038508 (0.024982) | 0.030299 / 0.023109 (0.007189) | 0.244489 / 0.275898 (-0.031409) | 0.274116 / 0.323480 (-0.049364) | 0.003187 / 0.007986 (-0.004798) | 0.003433 / 0.004328 (-0.000896) | 0.049313 / 0.004250 (0.045062) | 0.043677 / 0.037052 (0.006624) | 0.260198 / 0.258489 (0.001709) | 0.283558 / 0.293841 (-0.010283) | 0.029728 / 0.128546 (-0.098819) | 0.011950 / 0.075646 (-0.063696) | 0.204371 / 0.419271 (-0.214901) | 0.035712 / 0.043533 (-0.007821) | 0.252715 / 0.255139 (-0.002424) | 0.268906 / 0.283200 (-0.014293) | 0.021153 / 0.141683 (-0.120529) | 1.125599 / 1.452155 (-0.326556) | 1.163122 / 1.492716 (-0.329594) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095089 / 0.018006 (0.077083) | 0.298576 / 0.000490 (0.298086) | 0.000214 / 0.000200 (0.000014) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018567 / 0.037411 (-0.018844) | 0.062337 / 0.014526 (0.047811) | 0.074231 / 0.176557 (-0.102326) | 0.120960 / 0.737135 (-0.616175) | 0.076124 / 0.296338 (-0.220215) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.286936 / 0.215209 (0.071727) | 2.816656 / 2.077655 (0.739001) | 1.486772 / 1.504120 (-0.017348) | 1.373289 / 1.541195 (-0.167905) | 1.392739 / 1.468490 (-0.075752) | 0.708091 / 4.584777 (-3.876686) | 2.410034 / 3.745712 (-1.335678) | 2.912701 / 5.269862 (-2.357161) | 1.850924 / 4.565676 (-2.714752) | 0.078896 / 0.424275 (-0.345380) | 0.005116 / 0.007607 (-0.002491) | 0.332275 / 0.226044 (0.106231) | 3.306562 / 2.268929 (1.037633) | 1.853051 / 55.444624 (-53.591573) | 1.556210 / 6.876477 (-5.320267) | 1.558892 / 2.142072 (-0.583181) | 0.789917 / 4.805227 (-4.015310) | 0.133683 / 6.500664 (-6.366981) | 0.042566 / 0.075469 (-0.032904) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.957050 / 1.841788 (-0.884738) | 11.401462 / 8.074308 (3.327154) | 9.782988 / 10.191392 (-0.408404) | 0.142127 / 0.680424 (-0.538296) | 0.014730 / 0.534201 (-0.519471) | 0.302647 / 0.579283 (-0.276636) | 0.264654 / 0.434364 (-0.169710) | 0.341340 / 0.540337 (-0.198998) | 0.425808 / 1.386936 (-0.961128) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005679 / 0.011353 (-0.005674) | 0.003513 / 0.011008 (-0.007495) | 0.050135 / 0.038508 (0.011627) | 0.031614 / 0.023109 (0.008505) | 0.260064 / 0.275898 (-0.015834) | 0.285816 / 0.323480 (-0.037664) | 0.004428 / 0.007986 (-0.003558) | 0.002816 / 0.004328 (-0.001512) | 0.048441 / 0.004250 (0.044191) | 0.039622 / 0.037052 (0.002570) | 0.274940 / 0.258489 (0.016451) | 0.311837 / 0.293841 (0.017996) | 0.031439 / 0.128546 (-0.097107) | 0.012056 / 0.075646 (-0.063590) | 0.060109 / 0.419271 (-0.359163) | 0.033123 / 0.043533 (-0.010409) | 0.261563 / 0.255139 (0.006424) | 0.282640 / 0.283200 (-0.000560) | 0.017168 / 0.141683 (-0.124515) | 1.127859 / 1.452155 (-0.324295) | 1.182414 / 1.492716 (-0.310303) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095517 / 0.018006 (0.077510) | 0.300578 / 0.000490 (0.300088) | 0.000212 / 0.000200 (0.000012) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022192 / 0.037411 (-0.015220) | 0.076617 / 0.014526 (0.062091) | 0.087405 / 0.176557 (-0.089151) | 0.127011 / 0.737135 (-0.610124) | 0.088706 / 0.296338 (-0.207632) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.294260 / 0.215209 (0.079051) | 2.872879 / 2.077655 (0.795224) | 1.531374 / 1.504120 (0.027254) | 1.399232 / 1.541195 (-0.141962) | 1.400708 / 1.468490 (-0.067782) | 0.714003 / 4.584777 (-3.870773) | 0.943144 / 3.745712 (-2.802568) | 2.833396 / 5.269862 (-2.436466) | 1.890570 / 4.565676 (-2.675106) | 0.077664 / 0.424275 (-0.346611) | 0.005651 / 0.007607 (-0.001956) | 0.349476 / 0.226044 (0.123431) | 3.405768 / 2.268929 (1.136840) | 1.869739 / 55.444624 (-53.574885) | 1.575293 / 6.876477 (-5.301184) | 1.692981 / 2.142072 (-0.449092) | 0.795363 / 4.805227 (-4.009865) | 0.131532 / 6.500664 (-6.369132) | 0.041183 / 0.075469 (-0.034286) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.000821 / 1.841788 (-0.840967) | 11.987795 / 8.074308 (3.913487) | 10.147652 / 10.191392 (-0.043740) | 0.141314 / 0.680424 (-0.539110) | 0.015506 / 0.534201 (-0.518695) | 0.305090 / 0.579283 (-0.274193) | 0.123403 / 0.434364 (-0.310960) | 0.346507 / 0.540337 (-0.193831) | 0.471318 / 1.386936 (-0.915618) |

![](https://cml.dev/watermark.png#186b560eb2393c7d1913f4b3e76e9e04a081e09b "CML watermark")

Fix tests using hf-internal-testing/librispeech_asr_dummy once that dataset has been converted to Parquet.

Fix #6997.


CI is broken for tests using hf-internal-testing/librispeech_asr_dummy

maintenance

Maintenance tasks

CI is broken: https://github.com/huggingface/datasets/actions/runs/9657882317/job/26637998686?pr=6996
```
FAILED tests/test_inspect.py::test_get_dataset_config_names[hf-internal-testing/librispeech_asr_dummy-expected4] - AssertionError: assert ['clean'] == ['clean', 'other']

  Right contains one more item: 'other'

  Full diff:
    [
        'clean',
  -     'other',
    ]
FAILED tests/test_inspect.py::test_get_dataset_default_config_name[hf-internal-testing/librispeech_asr_dummy-None] - AssertionError: assert 'clean' is None
```

Note that repository was recently converted to Parquet: https://huggingface.co/datasets/hf-internal-testing/librispeech_asr_dummy/commit/5be91486e11a2d616f4ec5db8d3fd248585ac07a


Remove deprecated code

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6996). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Remove deprecated code, as part of the 3.0 release.


ImportError when importing datasets.load_dataset

What is the version of your installed `huggingface-hub`:
```python
import huggingface_hub
print(huggingface_hub.__version__)
```

It seems you have a very old version of `huggingface-hub`, where `CommitInfo` was not still implemented. You need to update it:
```
pip install -U huggingface-hub
```

Note that `CommitInfo` was implemented in huggingface-hub 0.10.0 and datasets requires "huggingface-hub>=0.21.2"

The version of my huggingface-hub is 0.23.4.

The error message says there is no CommitInfo in your installed huggingface-hub library:
```
ImportError: cannot import name 'CommitInfo' from 'huggingface_hub' (D:\Anaconda3\envs\CS224S\Lib\site-packages\huggingface_hub_init_.py)
```

And this is implemented since version 0.10.0:
- https://github.com/huggingface/huggingface_hub/pull/1066

### Describe the bug

I encountered an ImportError while trying to import `load_dataset` from the `datasets` module in Hugging Face. The error message indicates a problem with importing 'CommitInfo' from 'huggingface_hub'.

### Steps to reproduce the bug

1. pip install git+https://github.com/huggingface/datasets
2. from datasets import load_dataset

### Expected behavior

ImportError                               Traceback (most recent call last)
Cell In[7], [line 1](vscode-notebook-cell:?execution_count=7&line=1)
----> [1](vscode-notebook-cell:?execution_count=7&line=1) from datasets import load_dataset
      [3](vscode-notebook-cell:?execution_count=7&line=3) train_set = load_dataset("mispeech/speechocean762", split="train")
      [4](vscode-notebook-cell:?execution_count=7&line=4) test_set = load_dataset("mispeech/speechocean762", split="test")

File d:\Anaconda3\envs\CS224S\Lib\site-packages\datasets\__init__.py:[1](file:///D:/Anaconda3/envs/CS224S/Lib/site-packages/datasets/__init__.py:1)7
      1 # Copyright 2020 The HuggingFace Datasets Authors and the TensorFlow Datasets Authors.
      [2](file:///D:/Anaconda3/envs/CS224S/Lib/site-packages/datasets/__init__.py:2) #
      [3](file:///D:/Anaconda3/envs/CS224S/Lib/site-packages/datasets/__init__.py:3) # Licensed under the Apache License, Version 2.0 (the "License");
   (...)
     [12](file:///D:/Anaconda3/envs/CS224S/Lib/site-packages/datasets/__init__.py:12) # See the License for the specific language governing permissions and
     [13](file:///D:/Anaconda3/envs/CS224S/Lib/site-packages/datasets/__init__.py:13) # limitations under the License.
     [15](file:///D:/Anaconda3/envs/CS224S/Lib/site-packages/datasets/__init__.py:15) __version__ = "2.20.1.dev0"
---> [17](file:///D:/Anaconda3/envs/CS224S/Lib/site-packages/datasets/__init__.py:17) from .arrow_dataset import Dataset
     [18](file:///D:/Anaconda3/envs/CS224S/Lib/site-packages/datasets/__init__.py:18) from .arrow_reader import ReadInstruction
     [19](file:///D:/Anaconda3/envs/CS224S/Lib/site-packages/datasets/__init__.py:19) from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder

File d:\Anaconda3\envs\CS224S\Lib\site-packages\datasets\arrow_dataset.py:63
     [61](file:///D:/Anaconda3/envs/CS224S/Lib/site-packages/datasets/arrow_dataset.py:61) import pyarrow.compute as pc
     [62](file:///D:/Anaconda3/envs/CS224S/Lib/site-packages/datasets/arrow_dataset.py:62) from fsspec.core import url_to_fs
---> [63](file:///D:/Anaconda3/envs/CS224S/Lib/site-packages/datasets/arrow_dataset.py:63) from huggingface_hub import (
     [64](file:///D:/Anaconda3/envs/CS224S/Lib/site-packages/datasets/arrow_dataset.py:64)     CommitInfo,
     [65](file:///D:/Anaconda3/envs/CS224S/Lib/site-packages/datasets/arrow_dataset.py:65)     CommitOperationAdd,
...
     [70](file:///D:/Anaconda3/envs/CS224S/Lib/site-packages/datasets/arrow_dataset.py:70) )
     [71](file:///D:/Anaconda3/envs/CS224S/Lib/site-packages/datasets/arrow_dataset.py:71) from huggingface_hub.hf_api import RepoFile
     [72](file:///D:/Anaconda3/envs/CS224S/Lib/site-packages/datasets/arrow_dataset.py:72) from multiprocess import Pool

ImportError: cannot import name 'CommitInfo' from 'huggingface_hub' (d:\Anaconda3\envs\CS224S\Lib\site-packages\huggingface_hub\__init__.py)
Output is truncated. View as a [scrollable element](command:cellOutput.enableScrolling?580889ab-0f61-4f37-9214-eaa2b3807f85) or open in a [text editor](command:workbench.action.openLargeOutput?580889ab-0f61-4f37-9214-eaa2b3807f85). Adjust cell output [settings](command:workbench.action.openSettings?%5B%22%40tag%3AnotebookOutputLayout%22%5D)...

### Environment info

Leo@DESKTOP-9NHUAMI MSYS /d/Anaconda3/envs/CS224S/Lib/site-packages/huggingface_hub
$ datasets-cli env
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "D:\Anaconda3\envs\CS224S\Scripts\datasets-cli.exe\__main__.py", line 4, in <module>
  File "D:\Anaconda3\envs\CS224S\Lib\site-packages\datasets\__init__.py", line 17, in <module>
    from .arrow_dataset import Dataset
  File "D:\Anaconda3\envs\CS224S\Lib\site-packages\datasets\arrow_dataset.py", line 63, in <module>
    from huggingface_hub import (
ImportError: cannot import name 'CommitInfo' from 'huggingface_hub' (D:\Anaconda3\envs\CS224S\Lib\site-packages\huggingface_hub\__init__.py)
(CS224S)


Fix incorrect rank value in data splitting (#6990)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6994). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005538 / 0.011353 (-0.005815) | 0.003997 / 0.011008 (-0.007011) | 0.063444 / 0.038508 (0.024935) | 0.032552 / 0.023109 (0.009442) | 0.266574 / 0.275898 (-0.009324) | 0.282841 / 0.323480 (-0.040639) | 0.004279 / 0.007986 (-0.003706) | 0.002788 / 0.004328 (-0.001540) | 0.049226 / 0.004250 (0.044976) | 0.044688 / 0.037052 (0.007636) | 0.275464 / 0.258489 (0.016975) | 0.305278 / 0.293841 (0.011437) | 0.030097 / 0.128546 (-0.098450) | 0.012237 / 0.075646 (-0.063410) | 0.205526 / 0.419271 (-0.213745) | 0.036145 / 0.043533 (-0.007388) | 0.267395 / 0.255139 (0.012256) | 0.289149 / 0.283200 (0.005949) | 0.019044 / 0.141683 (-0.122639) | 1.162294 / 1.452155 (-0.289861) | 1.183642 / 1.492716 (-0.309074) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.139125 / 0.018006 (0.121119) | 0.301743 / 0.000490 (0.301253) | 0.000260 / 0.000200 (0.000061) | 0.000053 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019494 / 0.037411 (-0.017917) | 0.063078 / 0.014526 (0.048552) | 0.076989 / 0.176557 (-0.099567) | 0.121363 / 0.737135 (-0.615773) | 0.080040 / 0.296338 (-0.216298) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.284401 / 0.215209 (0.069192) | 2.805397 / 2.077655 (0.727742) | 1.555609 / 1.504120 (0.051489) | 1.405662 / 1.541195 (-0.135533) | 1.459492 / 1.468490 (-0.008999) | 0.718376 / 4.584777 (-3.866401) | 2.395918 / 3.745712 (-1.349794) | 2.976753 / 5.269862 (-2.293108) | 1.883938 / 4.565676 (-2.681738) | 0.078867 / 0.424275 (-0.345408) | 0.005207 / 0.007607 (-0.002400) | 0.335178 / 0.226044 (0.109133) | 3.313414 / 2.268929 (1.044485) | 1.856929 / 55.444624 (-53.587696) | 1.565319 / 6.876477 (-5.311158) | 1.592723 / 2.142072 (-0.549350) | 0.793621 / 4.805227 (-4.011606) | 0.134208 / 6.500664 (-6.366456) | 0.042853 / 0.075469 (-0.032616) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.981553 / 1.841788 (-0.860235) | 11.810438 / 8.074308 (3.736130) | 9.529874 / 10.191392 (-0.661518) | 0.142216 / 0.680424 (-0.538207) | 0.014303 / 0.534201 (-0.519898) | 0.304600 / 0.579283 (-0.274684) | 0.261869 / 0.434364 (-0.172495) | 0.347301 / 0.540337 (-0.193036) | 0.437395 / 1.386936 (-0.949541) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005881 / 0.011353 (-0.005472) | 0.004039 / 0.011008 (-0.006969) | 0.050241 / 0.038508 (0.011733) | 0.032670 / 0.023109 (0.009561) | 0.264940 / 0.275898 (-0.010959) | 0.287105 / 0.323480 (-0.036374) | 0.004844 / 0.007986 (-0.003142) | 0.002867 / 0.004328 (-0.001462) | 0.048083 / 0.004250 (0.043833) | 0.040965 / 0.037052 (0.003913) | 0.274390 / 0.258489 (0.015901) | 0.312107 / 0.293841 (0.018266) | 0.031714 / 0.128546 (-0.096832) | 0.012603 / 0.075646 (-0.063043) | 0.060698 / 0.419271 (-0.358573) | 0.033130 / 0.043533 (-0.010402) | 0.264444 / 0.255139 (0.009305) | 0.282797 / 0.283200 (-0.000403) | 0.027872 / 0.141683 (-0.113811) | 1.139026 / 1.452155 (-0.313129) | 1.181431 / 1.492716 (-0.311285) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.097314 / 0.018006 (0.079308) | 0.301326 / 0.000490 (0.300836) | 0.000215 / 0.000200 (0.000015) | 0.000049 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023394 / 0.037411 (-0.014018) | 0.076270 / 0.014526 (0.061744) | 0.089065 / 0.176557 (-0.087491) | 0.129996 / 0.737135 (-0.607139) | 0.089642 / 0.296338 (-0.206697) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295390 / 0.215209 (0.080181) | 2.877849 / 2.077655 (0.800194) | 1.537129 / 1.504120 (0.033009) | 1.409441 / 1.541195 (-0.131754) | 1.432468 / 1.468490 (-0.036023) | 0.718054 / 4.584777 (-3.866722) | 0.930872 / 3.745712 (-2.814841) | 2.841028 / 5.269862 (-2.428834) | 1.921990 / 4.565676 (-2.643686) | 0.077638 / 0.424275 (-0.346637) | 0.005494 / 0.007607 (-0.002113) | 0.336331 / 0.226044 (0.110287) | 3.330490 / 2.268929 (1.061561) | 1.887994 / 55.444624 (-53.556630) | 1.593332 / 6.876477 (-5.283144) | 1.726956 / 2.142072 (-0.415116) | 0.783612 / 4.805227 (-4.021615) | 0.129926 / 6.500664 (-6.370738) | 0.040792 / 0.075469 (-0.034677) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.980274 / 1.841788 (-0.861514) | 12.193871 / 8.074308 (4.119563) | 10.348934 / 10.191392 (0.157542) | 0.141584 / 0.680424 (-0.538840) | 0.015737 / 0.534201 (-0.518464) | 0.300725 / 0.579283 (-0.278558) | 0.127190 / 0.434364 (-0.307174) | 0.341142 / 0.540337 (-0.199196) | 0.459523 / 1.386936 (-0.927413) |

![](https://cml.dev/watermark.png#637246baf96f07b19b193ed101f34b65cb35cffb "CML watermark")


less script docs

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6993). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

+ mark as legacy in some parts of the docs since we'll not build new features for script datasets