Fix resuming arrow format

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6964). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005735 / 0.011353 (-0.005618) | 0.003746 / 0.011008 (-0.007263) | 0.063115 / 0.038508 (0.024606) | 0.033557 / 0.023109 (0.010447) | 0.247599 / 0.275898 (-0.028299) | 0.275310 / 0.323480 (-0.048170) | 0.004203 / 0.007986 (-0.003783) | 0.002770 / 0.004328 (-0.001558) | 0.050951 / 0.004250 (0.046700) | 0.046609 / 0.037052 (0.009557) | 0.256237 / 0.258489 (-0.002252) | 0.292050 / 0.293841 (-0.001791) | 0.027991 / 0.128546 (-0.100556) | 0.010367 / 0.075646 (-0.065279) | 0.202295 / 0.419271 (-0.216977) | 0.037287 / 0.043533 (-0.006246) | 0.250330 / 0.255139 (-0.004809) | 0.281250 / 0.283200 (-0.001950) | 0.018832 / 0.141683 (-0.122851) | 1.117303 / 1.452155 (-0.334852) | 1.141593 / 1.492716 (-0.351123) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.097318 / 0.018006 (0.079312) | 0.304853 / 0.000490 (0.304364) | 0.000220 / 0.000200 (0.000020) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.020353 / 0.037411 (-0.017058) | 0.065497 / 0.014526 (0.050971) | 0.076205 / 0.176557 (-0.100351) | 0.122471 / 0.737135 (-0.614665) | 0.079522 / 0.296338 (-0.216816) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282604 / 0.215209 (0.067395) | 2.743198 / 2.077655 (0.665543) | 1.480436 / 1.504120 (-0.023684) | 1.373935 / 1.541195 (-0.167260) | 1.388901 / 1.468490 (-0.079589) | 0.571961 / 4.584777 (-4.012816) | 2.431790 / 3.745712 (-1.313922) | 2.942126 / 5.269862 (-2.327736) | 1.857361 / 4.565676 (-2.708316) | 0.063535 / 0.424275 (-0.360740) | 0.005039 / 0.007607 (-0.002568) | 0.331726 / 0.226044 (0.105682) | 3.282504 / 2.268929 (1.013576) | 1.852303 / 55.444624 (-53.592321) | 1.506665 / 6.876477 (-5.369812) | 1.577524 / 2.142072 (-0.564548) | 0.646267 / 4.805227 (-4.158960) | 0.118706 / 6.500664 (-6.381958) | 0.043437 / 0.075469 (-0.032033) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.978073 / 1.841788 (-0.863714) | 12.028575 / 8.074308 (3.954267) | 10.066303 / 10.191392 (-0.125090) | 0.131763 / 0.680424 (-0.548661) | 0.016479 / 0.534201 (-0.517722) | 0.286012 / 0.579283 (-0.293271) | 0.266824 / 0.434364 (-0.167540) | 0.328452 / 0.540337 (-0.211885) | 0.414562 / 1.386936 (-0.972374) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005943 / 0.011353 (-0.005409) | 0.003992 / 0.011008 (-0.007016) | 0.051159 / 0.038508 (0.012651) | 0.033805 / 0.023109 (0.010695) | 0.268425 / 0.275898 (-0.007474) | 0.295662 / 0.323480 (-0.027818) | 0.004473 / 0.007986 (-0.003512) | 0.002910 / 0.004328 (-0.001418) | 0.048595 / 0.004250 (0.044345) | 0.043724 / 0.037052 (0.006671) | 0.280552 / 0.258489 (0.022063) | 0.319052 / 0.293841 (0.025211) | 0.031269 / 0.128546 (-0.097278) | 0.010976 / 0.075646 (-0.064671) | 0.060128 / 0.419271 (-0.359144) | 0.034198 / 0.043533 (-0.009335) | 0.269664 / 0.255139 (0.014525) | 0.292249 / 0.283200 (0.009049) | 0.019950 / 0.141683 (-0.121733) | 1.143073 / 1.452155 (-0.309082) | 1.188553 / 1.492716 (-0.304164) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095188 / 0.018006 (0.077182) | 0.300207 / 0.000490 (0.299717) | 0.000205 / 0.000200 (0.000005) | 0.000051 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023610 / 0.037411 (-0.013802) | 0.082868 / 0.014526 (0.068342) | 0.089059 / 0.176557 (-0.087498) | 0.131735 / 0.737135 (-0.605401) | 0.091467 / 0.296338 (-0.204872) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.302497 / 0.215209 (0.087287) | 2.985794 / 2.077655 (0.908140) | 1.590783 / 1.504120 (0.086663) | 1.468819 / 1.541195 (-0.072375) | 1.503115 / 1.468490 (0.034625) | 0.575109 / 4.584777 (-4.009668) | 0.972370 / 3.745712 (-2.773342) | 2.727976 / 5.269862 (-2.541886) | 1.793438 / 4.565676 (-2.772238) | 0.068840 / 0.424275 (-0.355435) | 0.005440 / 0.007607 (-0.002167) | 0.351843 / 0.226044 (0.125799) | 3.523108 / 2.268929 (1.254180) | 1.928576 / 55.444624 (-53.516049) | 1.627939 / 6.876477 (-5.248538) | 1.837618 / 2.142072 (-0.304454) | 0.669351 / 4.805227 (-4.135876) | 0.121822 / 6.500664 (-6.378842) | 0.042056 / 0.075469 (-0.033413) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.020081 / 1.841788 (-0.821707) | 13.417448 / 8.074308 (5.343140) | 10.974516 / 10.191392 (0.783124) | 0.135240 / 0.680424 (-0.545184) | 0.017581 / 0.534201 (-0.516620) | 0.289080 / 0.579283 (-0.290203) | 0.127679 / 0.434364 (-0.306685) | 0.331818 / 0.540337 (-0.208520) | 0.453143 / 1.386936 (-0.933793) |

![](https://cml.dev/watermark.png#ef2fb358433678b322d275c0bdee3239fa6485b2 "CML watermark")

following https://github.com/huggingface/datasets/pull/6658


[Streaming] retry on requests errors

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6963). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

reported in https://discuss.huggingface.co/t/speeding-up-streaming-of-large-datasets-fineweb/90714/6  when training using a streaming a dataloader

cc @Wauplin it looks like the retries from `hfh` are not always enough. In this PR I let `datasets` do additional retries (that users can configure in `datasets.config`) since I couldn't find an easy way to increase the max_retries for `hfh` users in general.


fix(ci): remove unnecessary permissions

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6962). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005520 / 0.011353 (-0.005833) | 0.003989 / 0.011008 (-0.007019) | 0.064786 / 0.038508 (0.026278) | 0.031075 / 0.023109 (0.007966) | 0.241619 / 0.275898 (-0.034279) | 0.275341 / 0.323480 (-0.048139) | 0.003139 / 0.007986 (-0.004847) | 0.002820 / 0.004328 (-0.001508) | 0.049766 / 0.004250 (0.045515) | 0.045047 / 0.037052 (0.007995) | 0.251906 / 0.258489 (-0.006583) | 0.285889 / 0.293841 (-0.007952) | 0.028297 / 0.128546 (-0.100249) | 0.010683 / 0.075646 (-0.064963) | 0.206467 / 0.419271 (-0.212805) | 0.036267 / 0.043533 (-0.007266) | 0.250720 / 0.255139 (-0.004419) | 0.268565 / 0.283200 (-0.014635) | 0.020394 / 0.141683 (-0.121289) | 1.114283 / 1.452155 (-0.337872) | 1.163884 / 1.492716 (-0.328833) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.112698 / 0.018006 (0.094692) | 0.302740 / 0.000490 (0.302251) | 0.000209 / 0.000200 (0.000009) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019337 / 0.037411 (-0.018075) | 0.062854 / 0.014526 (0.048328) | 0.077088 / 0.176557 (-0.099468) | 0.120926 / 0.737135 (-0.616209) | 0.075594 / 0.296338 (-0.220744) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.290787 / 0.215209 (0.075578) | 2.867894 / 2.077655 (0.790239) | 1.490043 / 1.504120 (-0.014076) | 1.356383 / 1.541195 (-0.184812) | 1.400229 / 1.468490 (-0.068261) | 0.582076 / 4.584777 (-4.002701) | 2.398270 / 3.745712 (-1.347442) | 2.856459 / 5.269862 (-2.413403) | 1.815545 / 4.565676 (-2.750131) | 0.063259 / 0.424275 (-0.361016) | 0.005056 / 0.007607 (-0.002551) | 0.347699 / 0.226044 (0.121655) | 3.466511 / 2.268929 (1.197582) | 1.862096 / 55.444624 (-53.582528) | 1.532324 / 6.876477 (-5.344152) | 1.599411 / 2.142072 (-0.542661) | 0.657350 / 4.805227 (-4.147878) | 0.118981 / 6.500664 (-6.381683) | 0.042224 / 0.075469 (-0.033245) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.965649 / 1.841788 (-0.876139) | 11.896501 / 8.074308 (3.822193) | 9.873923 / 10.191392 (-0.317469) | 0.141165 / 0.680424 (-0.539258) | 0.013885 / 0.534201 (-0.520316) | 0.291464 / 0.579283 (-0.287819) | 0.273153 / 0.434364 (-0.161211) | 0.324395 / 0.540337 (-0.215942) | 0.422040 / 1.386936 (-0.964897) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005640 / 0.011353 (-0.005713) | 0.004035 / 0.011008 (-0.006973) | 0.050831 / 0.038508 (0.012323) | 0.032841 / 0.023109 (0.009732) | 0.272226 / 0.275898 (-0.003672) | 0.297880 / 0.323480 (-0.025599) | 0.004397 / 0.007986 (-0.003588) | 0.002762 / 0.004328 (-0.001566) | 0.049887 / 0.004250 (0.045637) | 0.040372 / 0.037052 (0.003320) | 0.286337 / 0.258489 (0.027848) | 0.320015 / 0.293841 (0.026174) | 0.029992 / 0.128546 (-0.098554) | 0.010781 / 0.075646 (-0.064865) | 0.059391 / 0.419271 (-0.359880) | 0.034410 / 0.043533 (-0.009123) | 0.273024 / 0.255139 (0.017885) | 0.288953 / 0.283200 (0.005754) | 0.018072 / 0.141683 (-0.123611) | 1.125742 / 1.452155 (-0.326413) | 1.175233 / 1.492716 (-0.317483) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093470 / 0.018006 (0.075463) | 0.313248 / 0.000490 (0.312758) | 0.000324 / 0.000200 (0.000124) | 0.000081 / 0.000054 (0.000026) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023529 / 0.037411 (-0.013882) | 0.077305 / 0.014526 (0.062779) | 0.088916 / 0.176557 (-0.087640) | 0.128792 / 0.737135 (-0.608344) | 0.090141 / 0.296338 (-0.206197) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.291110 / 0.215209 (0.075901) | 2.848118 / 2.077655 (0.770464) | 1.581664 / 1.504120 (0.077544) | 1.446390 / 1.541195 (-0.094804) | 1.452594 / 1.468490 (-0.015896) | 0.571213 / 4.584777 (-4.013564) | 0.976382 / 3.745712 (-2.769330) | 2.756192 / 5.269862 (-2.513670) | 1.770274 / 4.565676 (-2.795403) | 0.064513 / 0.424275 (-0.359763) | 0.005334 / 0.007607 (-0.002273) | 0.347380 / 0.226044 (0.121335) | 3.424800 / 2.268929 (1.155871) | 1.942374 / 55.444624 (-53.502250) | 1.636069 / 6.876477 (-5.240407) | 1.795327 / 2.142072 (-0.346745) | 0.658942 / 4.805227 (-4.146285) | 0.119542 / 6.500664 (-6.381123) | 0.041826 / 0.075469 (-0.033643) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.007230 / 1.841788 (-0.834558) | 12.293084 / 8.074308 (4.218776) | 10.618104 / 10.191392 (0.426712) | 0.133691 / 0.680424 (-0.546733) | 0.015725 / 0.534201 (-0.518476) | 0.288860 / 0.579283 (-0.290423) | 0.130546 / 0.434364 (-0.303818) | 0.327279 / 0.540337 (-0.213059) | 0.428768 / 1.386936 (-0.958168) |

![](https://cml.dev/watermark.png#af3acfdfcf76bb980dbac871540e30c2cade0cf9 "CML watermark")

### What does this PR do?

Remove unnecessary permissions granted to the actions workflow.

Sorry for the mishap.


Manual downloads should count as downloads

enhancement

New feature or request

We're unlikely to add more features/support for datasets with python loading scripts, which include datasets with manual download. Sorry for the inconvenience

### Feature request

I would like to request that manual downloads of data files from Hugging Face dataset repositories count as downloads of a dataset. According to the documentation for the Hugging Face Hub, that is currently not the case: https://huggingface.co/docs/hub/en/datasets-download-stats

### Motivation

This would ensure that downloads are accurately reported to end users.

### Your contribution

N/A


feat(ci): add trufflehog secrets detection

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6960). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005007 / 0.011353 (-0.006346) | 0.003603 / 0.011008 (-0.007405) | 0.062719 / 0.038508 (0.024211) | 0.029327 / 0.023109 (0.006217) | 0.250360 / 0.275898 (-0.025538) | 0.265095 / 0.323480 (-0.058385) | 0.004205 / 0.007986 (-0.003781) | 0.002713 / 0.004328 (-0.001616) | 0.049209 / 0.004250 (0.044958) | 0.045162 / 0.037052 (0.008110) | 0.260439 / 0.258489 (0.001950) | 0.287778 / 0.293841 (-0.006063) | 0.027458 / 0.128546 (-0.101088) | 0.010169 / 0.075646 (-0.065477) | 0.199487 / 0.419271 (-0.219784) | 0.036584 / 0.043533 (-0.006949) | 0.254523 / 0.255139 (-0.000616) | 0.269902 / 0.283200 (-0.013298) | 0.017138 / 0.141683 (-0.124545) | 1.099285 / 1.452155 (-0.352869) | 1.150878 / 1.492716 (-0.341839) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092868 / 0.018006 (0.074862) | 0.300421 / 0.000490 (0.299932) | 0.000213 / 0.000200 (0.000013) | 0.000053 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018810 / 0.037411 (-0.018601) | 0.062341 / 0.014526 (0.047815) | 0.074779 / 0.176557 (-0.101777) | 0.120641 / 0.737135 (-0.616494) | 0.075020 / 0.296338 (-0.221318) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.277782 / 0.215209 (0.062573) | 2.716427 / 2.077655 (0.638772) | 1.434204 / 1.504120 (-0.069916) | 1.335990 / 1.541195 (-0.205205) | 1.336636 / 1.468490 (-0.131854) | 0.557562 / 4.584777 (-4.027215) | 2.323517 / 3.745712 (-1.422196) | 2.647937 / 5.269862 (-2.621925) | 1.728735 / 4.565676 (-2.836941) | 0.061888 / 0.424275 (-0.362387) | 0.004981 / 0.007607 (-0.002627) | 0.329429 / 0.226044 (0.103385) | 3.324708 / 2.268929 (1.055779) | 1.832641 / 55.444624 (-53.611983) | 1.514386 / 6.876477 (-5.362091) | 1.656912 / 2.142072 (-0.485160) | 0.630706 / 4.805227 (-4.174521) | 0.116250 / 6.500664 (-6.384414) | 0.042598 / 0.075469 (-0.032871) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.969217 / 1.841788 (-0.872570) | 11.232580 / 8.074308 (3.158272) | 9.541306 / 10.191392 (-0.650086) | 0.139544 / 0.680424 (-0.540880) | 0.014441 / 0.534201 (-0.519760) | 0.285834 / 0.579283 (-0.293449) | 0.261950 / 0.434364 (-0.172414) | 0.325449 / 0.540337 (-0.214889) | 0.415501 / 1.386936 (-0.971435) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005422 / 0.011353 (-0.005931) | 0.003528 / 0.011008 (-0.007480) | 0.049582 / 0.038508 (0.011074) | 0.032683 / 0.023109 (0.009574) | 0.277309 / 0.275898 (0.001411) | 0.298598 / 0.323480 (-0.024882) | 0.004325 / 0.007986 (-0.003661) | 0.002741 / 0.004328 (-0.001588) | 0.047933 / 0.004250 (0.043683) | 0.040778 / 0.037052 (0.003726) | 0.287492 / 0.258489 (0.029003) | 0.311408 / 0.293841 (0.017567) | 0.029482 / 0.128546 (-0.099064) | 0.010630 / 0.075646 (-0.065016) | 0.057745 / 0.419271 (-0.361526) | 0.033501 / 0.043533 (-0.010031) | 0.279880 / 0.255139 (0.024741) | 0.297421 / 0.283200 (0.014221) | 0.017907 / 0.141683 (-0.123776) | 1.152221 / 1.452155 (-0.299934) | 1.189332 / 1.492716 (-0.303385) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094464 / 0.018006 (0.076457) | 0.300769 / 0.000490 (0.300279) | 0.000196 / 0.000200 (-0.000004) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022232 / 0.037411 (-0.015179) | 0.076626 / 0.014526 (0.062100) | 0.087807 / 0.176557 (-0.088750) | 0.128847 / 0.737135 (-0.608288) | 0.092135 / 0.296338 (-0.204203) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.299013 / 0.215209 (0.083804) | 2.929788 / 2.077655 (0.852133) | 1.614185 / 1.504120 (0.110065) | 1.486720 / 1.541195 (-0.054475) | 1.492473 / 1.468490 (0.023983) | 0.563699 / 4.584777 (-4.021078) | 0.928820 / 3.745712 (-2.816892) | 2.597271 / 5.269862 (-2.672590) | 1.716534 / 4.565676 (-2.849142) | 0.062568 / 0.424275 (-0.361707) | 0.005168 / 0.007607 (-0.002439) | 0.353781 / 0.226044 (0.127737) | 3.493732 / 2.268929 (1.224803) | 2.018343 / 55.444624 (-53.426282) | 1.694516 / 6.876477 (-5.181961) | 1.796950 / 2.142072 (-0.345123) | 0.634846 / 4.805227 (-4.170382) | 0.115230 / 6.500664 (-6.385434) | 0.040816 / 0.075469 (-0.034654) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.986212 / 1.841788 (-0.855575) | 11.954392 / 8.074308 (3.880084) | 10.299670 / 10.191392 (0.108278) | 0.128358 / 0.680424 (-0.552066) | 0.016313 / 0.534201 (-0.517888) | 0.289621 / 0.579283 (-0.289662) | 0.124708 / 0.434364 (-0.309656) | 0.325269 / 0.540337 (-0.215068) | 0.415133 / 1.386936 (-0.971803) |

![](https://cml.dev/watermark.png#97513be330114a8aa07e5199ec252ac662aeb76d "CML watermark")

### What does this PR do?

Adding a GH action to scan for leaked secrets on each commit.


Better error handling in `dataset_module_factory`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6959). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Test should be fixed by https://github.com/huggingface/datasets/pull/6959/commits/ef8f7cee79ffb070d9b5190f21128fc523b3d3ee (tested locally). Let's see what CI says :crossed_fingers:

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005678 / 0.011353 (-0.005675) | 0.004119 / 0.011008 (-0.006889) | 0.063901 / 0.038508 (0.025393) | 0.032071 / 0.023109 (0.008961) | 0.243182 / 0.275898 (-0.032716) | 0.280709 / 0.323480 (-0.042770) | 0.004195 / 0.007986 (-0.003791) | 0.002810 / 0.004328 (-0.001518) | 0.048722 / 0.004250 (0.044472) | 0.049381 / 0.037052 (0.012328) | 0.257816 / 0.258489 (-0.000673) | 0.288460 / 0.293841 (-0.005381) | 0.028518 / 0.128546 (-0.100029) | 0.010775 / 0.075646 (-0.064871) | 0.203149 / 0.419271 (-0.216122) | 0.038792 / 0.043533 (-0.004741) | 0.248502 / 0.255139 (-0.006637) | 0.268251 / 0.283200 (-0.014949) | 0.019536 / 0.141683 (-0.122147) | 1.133935 / 1.452155 (-0.318220) | 1.182855 / 1.492716 (-0.309862) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.097531 / 0.018006 (0.079525) | 0.303612 / 0.000490 (0.303122) | 0.000222 / 0.000200 (0.000022) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019670 / 0.037411 (-0.017741) | 0.063439 / 0.014526 (0.048913) | 0.075119 / 0.176557 (-0.101438) | 0.122419 / 0.737135 (-0.614717) | 0.076965 / 0.296338 (-0.219374) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.286780 / 0.215209 (0.071571) | 2.811860 / 2.077655 (0.734206) | 1.485165 / 1.504120 (-0.018954) | 1.373296 / 1.541195 (-0.167898) | 1.412700 / 1.468490 (-0.055790) | 0.566442 / 4.584777 (-4.018335) | 2.382616 / 3.745712 (-1.363096) | 2.677214 / 5.269862 (-2.592647) | 1.760073 / 4.565676 (-2.805603) | 0.062673 / 0.424275 (-0.361602) | 0.005050 / 0.007607 (-0.002557) | 0.341701 / 0.226044 (0.115657) | 3.321182 / 2.268929 (1.052253) | 1.811715 / 55.444624 (-53.632909) | 1.554986 / 6.876477 (-5.321491) | 1.727448 / 2.142072 (-0.414624) | 0.642193 / 4.805227 (-4.163034) | 0.117878 / 6.500664 (-6.382786) | 0.042814 / 0.075469 (-0.032655) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.985894 / 1.841788 (-0.855894) | 12.195975 / 8.074308 (4.121667) | 9.890180 / 10.191392 (-0.301212) | 0.142638 / 0.680424 (-0.537786) | 0.015207 / 0.534201 (-0.518994) | 0.283140 / 0.579283 (-0.296143) | 0.266016 / 0.434364 (-0.168348) | 0.325518 / 0.540337 (-0.214820) | 0.418994 / 1.386936 (-0.967942) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005978 / 0.011353 (-0.005374) | 0.003915 / 0.011008 (-0.007093) | 0.051592 / 0.038508 (0.013084) | 0.033338 / 0.023109 (0.010229) | 0.267925 / 0.275898 (-0.007973) | 0.296011 / 0.323480 (-0.027469) | 0.004503 / 0.007986 (-0.003483) | 0.002854 / 0.004328 (-0.001475) | 0.049958 / 0.004250 (0.045707) | 0.041708 / 0.037052 (0.004656) | 0.287185 / 0.258489 (0.028696) | 0.322715 / 0.293841 (0.028874) | 0.030088 / 0.128546 (-0.098458) | 0.010709 / 0.075646 (-0.064938) | 0.059736 / 0.419271 (-0.359536) | 0.034294 / 0.043533 (-0.009239) | 0.264316 / 0.255139 (0.009177) | 0.285471 / 0.283200 (0.002272) | 0.019197 / 0.141683 (-0.122486) | 1.135571 / 1.452155 (-0.316583) | 1.190019 / 1.492716 (-0.302698) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.099251 / 0.018006 (0.081245) | 0.305357 / 0.000490 (0.304867) | 0.000215 / 0.000200 (0.000015) | 0.000045 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023206 / 0.037411 (-0.014205) | 0.077835 / 0.014526 (0.063310) | 0.090242 / 0.176557 (-0.086315) | 0.131208 / 0.737135 (-0.605928) | 0.091726 / 0.296338 (-0.204612) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.292487 / 0.215209 (0.077278) | 2.837044 / 2.077655 (0.759389) | 1.553155 / 1.504120 (0.049035) | 1.433645 / 1.541195 (-0.107550) | 1.476702 / 1.468490 (0.008212) | 0.561926 / 4.584777 (-4.022851) | 0.954630 / 3.745712 (-2.791082) | 2.752286 / 5.269862 (-2.517575) | 1.782746 / 4.565676 (-2.782931) | 0.062984 / 0.424275 (-0.361291) | 0.005056 / 0.007607 (-0.002551) | 0.341700 / 0.226044 (0.115656) | 3.343726 / 2.268929 (1.074798) | 1.953390 / 55.444624 (-53.491234) | 1.616989 / 6.876477 (-5.259488) | 1.785104 / 2.142072 (-0.356969) | 0.643465 / 4.805227 (-4.161763) | 0.115905 / 6.500664 (-6.384759) | 0.041678 / 0.075469 (-0.033791) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.000237 / 1.841788 (-0.841550) | 12.633517 / 8.074308 (4.559208) | 10.553485 / 10.191392 (0.362092) | 0.143188 / 0.680424 (-0.537236) | 0.016020 / 0.534201 (-0.518181) | 0.286739 / 0.579283 (-0.292544) | 0.128488 / 0.434364 (-0.305876) | 0.321932 / 0.540337 (-0.218405) | 0.418635 / 1.386936 (-0.968301) |

![](https://cml.dev/watermark.png#9510252f03fded02b8cc87ca6dfa3195d17594ba "CML watermark")

cc @cakiki who reported it on [slack](https://huggingface.slack.com/archives/C039P47V1L5/p1717754405578539) (private link)

This PR updates how errors are handled in `dataset_module_factory` when the  `dataset_info` cannot be accessed:
1. Use multiple `except ... as e` instead of using `isinstance(e, ...)`
2. Always raise `DatasetNotFoundError` with `from e` so that the initial error is explicitly logged in the stacktrace.
3. Differentiate `RepoNotFoundError` / `GatedRepoError` / `RevisionNotFoundError` cases


My Private Dataset doesn't exist on the Hub or cannot be accessed

I can load public dataset, but for my private dataset it fails

I have checked the API HTTP link. Repository Not Found for url: https://huggingface.co/api/datasets/xxx/xxx.

![image](https://github.com/huggingface/datasets/assets/39621324/4aceef59-0c65-4161-9665-676d25d73225)

It just works fine.

It seems that everything is in a mass huh....

![image](https://github.com/huggingface/datasets/assets/39621324/fb2fe12c-4f0a-4bf6-9656-63ba50347b10)

https://github.com/huggingface/datasets/blob/main/templates/new_dataset_script.py#L81 can not use this, too complex. I just need a def to load my file to a dict

I am facing the same issue. Did you find a fix?

### Describe the bug

```
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 1852, in dataset_module_factory
    raise DatasetNotFoundError(msg + f" at revision '{revision}'" if revision else msg)
datasets.exceptions.DatasetNotFoundError: Dataset 'xxx' doesn't exist on the Hub or cannot be accessed
>>> dataset = load_dataset("xxxx", token=True)

404 error 404 Client Error. (Request ID: Root=xxxx)

Repository Not Found for url: https://huggingface.co/api/datasets/xxx/xxx.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 2593, in load_dataset
    builder_instance = load_dataset_builder(
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 2265, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 1910, in dataset_module_factory
    raise e1 from None
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 1852, in dataset_module_factory
    raise DatasetNotFoundError(msg + f" at revision '{revision}'" if revision else msg)
datasets.exceptions.DatasetNotFoundError: Dataset 'xxx' doesn't exist on the Hub or cannot be accessed
```

### Steps to reproduce the bug

123

### Expected behavior

123

### Environment info

123


Fix typos in docs

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6957). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005371 / 0.011353 (-0.005982) | 0.003834 / 0.011008 (-0.007174) | 0.063032 / 0.038508 (0.024524) | 0.031623 / 0.023109 (0.008514) | 0.250008 / 0.275898 (-0.025890) | 0.273998 / 0.323480 (-0.049482) | 0.004114 / 0.007986 (-0.003871) | 0.002821 / 0.004328 (-0.001508) | 0.049470 / 0.004250 (0.045220) | 0.046586 / 0.037052 (0.009534) | 0.276807 / 0.258489 (0.018318) | 0.288607 / 0.293841 (-0.005234) | 0.027427 / 0.128546 (-0.101119) | 0.010634 / 0.075646 (-0.065012) | 0.202451 / 0.419271 (-0.216821) | 0.036346 / 0.043533 (-0.007187) | 0.250426 / 0.255139 (-0.004713) | 0.274104 / 0.283200 (-0.009096) | 0.018461 / 0.141683 (-0.123222) | 1.120326 / 1.452155 (-0.331829) | 1.157635 / 1.492716 (-0.335081) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.102287 / 0.018006 (0.084281) | 0.313145 / 0.000490 (0.312655) | 0.000255 / 0.000200 (0.000055) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019494 / 0.037411 (-0.017917) | 0.063252 / 0.014526 (0.048727) | 0.075318 / 0.176557 (-0.101239) | 0.122194 / 0.737135 (-0.614942) | 0.076837 / 0.296338 (-0.219501) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.284098 / 0.215209 (0.068889) | 2.822301 / 2.077655 (0.744647) | 1.490185 / 1.504120 (-0.013935) | 1.366723 / 1.541195 (-0.174472) | 1.398832 / 1.468490 (-0.069658) | 0.563661 / 4.584777 (-4.021116) | 2.385129 / 3.745712 (-1.360583) | 2.689823 / 5.269862 (-2.580039) | 1.731271 / 4.565676 (-2.834405) | 0.063351 / 0.424275 (-0.360924) | 0.004974 / 0.007607 (-0.002633) | 0.332163 / 0.226044 (0.106119) | 3.314906 / 2.268929 (1.045977) | 1.811331 / 55.444624 (-53.633294) | 1.513357 / 6.876477 (-5.363120) | 1.718454 / 2.142072 (-0.423618) | 0.639663 / 4.805227 (-4.165564) | 0.120377 / 6.500664 (-6.380287) | 0.043254 / 0.075469 (-0.032215) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.978534 / 1.841788 (-0.863253) | 11.622313 / 8.074308 (3.548005) | 9.608732 / 10.191392 (-0.582660) | 0.131339 / 0.680424 (-0.549085) | 0.015226 / 0.534201 (-0.518975) | 0.287317 / 0.579283 (-0.291966) | 0.266647 / 0.434364 (-0.167717) | 0.324243 / 0.540337 (-0.216094) | 0.442025 / 1.386936 (-0.944911) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005673 / 0.011353 (-0.005680) | 0.003722 / 0.011008 (-0.007286) | 0.049483 / 0.038508 (0.010975) | 0.033308 / 0.023109 (0.010199) | 0.261912 / 0.275898 (-0.013986) | 0.291151 / 0.323480 (-0.032329) | 0.004389 / 0.007986 (-0.003596) | 0.002762 / 0.004328 (-0.001567) | 0.048970 / 0.004250 (0.044719) | 0.041509 / 0.037052 (0.004457) | 0.273288 / 0.258489 (0.014798) | 0.308351 / 0.293841 (0.014510) | 0.029958 / 0.128546 (-0.098589) | 0.010500 / 0.075646 (-0.065146) | 0.058253 / 0.419271 (-0.361019) | 0.033820 / 0.043533 (-0.009713) | 0.261089 / 0.255139 (0.005950) | 0.282179 / 0.283200 (-0.001021) | 0.018543 / 0.141683 (-0.123140) | 1.121303 / 1.452155 (-0.330852) | 1.166141 / 1.492716 (-0.326575) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.099209 / 0.018006 (0.081203) | 0.316920 / 0.000490 (0.316430) | 0.000216 / 0.000200 (0.000016) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023339 / 0.037411 (-0.014072) | 0.077127 / 0.014526 (0.062602) | 0.088160 / 0.176557 (-0.088396) | 0.129449 / 0.737135 (-0.607686) | 0.093159 / 0.296338 (-0.203180) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.281262 / 0.215209 (0.066053) | 2.797504 / 2.077655 (0.719850) | 1.513354 / 1.504120 (0.009234) | 1.383034 / 1.541195 (-0.158161) | 1.395202 / 1.468490 (-0.073288) | 0.563180 / 4.584777 (-4.021597) | 0.979330 / 3.745712 (-2.766383) | 2.674008 / 5.269862 (-2.595853) | 1.762174 / 4.565676 (-2.803502) | 0.062333 / 0.424275 (-0.361942) | 0.004991 / 0.007607 (-0.002616) | 0.336043 / 0.226044 (0.109999) | 3.313500 / 2.268929 (1.044571) | 1.848083 / 55.444624 (-53.596541) | 1.554723 / 6.876477 (-5.321754) | 1.743485 / 2.142072 (-0.398587) | 0.657117 / 4.805227 (-4.148111) | 0.115736 / 6.500664 (-6.384928) | 0.040527 / 0.075469 (-0.034942) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.005876 / 1.841788 (-0.835911) | 12.525895 / 8.074308 (4.451587) | 10.492961 / 10.191392 (0.301569) | 0.143443 / 0.680424 (-0.536981) | 0.016652 / 0.534201 (-0.517548) | 0.288236 / 0.579283 (-0.291047) | 0.131401 / 0.434364 (-0.302963) | 0.322885 / 0.540337 (-0.217452) | 0.416048 / 1.386936 (-0.970888) |

![](https://cml.dev/watermark.png#6548e0e282aeeda7bfb18beafbc65ebecd780c63 "CML watermark")

Fix typos in docs introduced by:
- #6956

Typos:
- `comparisions` => `comparisons`
- two consecutive sentences both ending in colon
- split one sentence into two

Sorry, I did not have time to review that PR.

CC: @lhoestq


update docs on N-dim arrays

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6956). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005348 / 0.011353 (-0.006005) | 0.003785 / 0.011008 (-0.007223) | 0.061674 / 0.038508 (0.023166) | 0.032127 / 0.023109 (0.009017) | 0.247095 / 0.275898 (-0.028803) | 0.276466 / 0.323480 (-0.047014) | 0.004197 / 0.007986 (-0.003789) | 0.002734 / 0.004328 (-0.001594) | 0.049604 / 0.004250 (0.045354) | 0.048553 / 0.037052 (0.011500) | 0.253230 / 0.258489 (-0.005259) | 0.286954 / 0.293841 (-0.006887) | 0.028181 / 0.128546 (-0.100365) | 0.010602 / 0.075646 (-0.065044) | 0.200719 / 0.419271 (-0.218552) | 0.037278 / 0.043533 (-0.006254) | 0.251565 / 0.255139 (-0.003574) | 0.269026 / 0.283200 (-0.014174) | 0.017632 / 0.141683 (-0.124050) | 1.136216 / 1.452155 (-0.315939) | 1.181158 / 1.492716 (-0.311559) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.004892 / 0.018006 (-0.013114) | 0.312921 / 0.000490 (0.312431) | 0.000247 / 0.000200 (0.000047) | 0.000054 / 0.000054 (-0.000000) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019303 / 0.037411 (-0.018108) | 0.062699 / 0.014526 (0.048174) | 0.075227 / 0.176557 (-0.101329) | 0.122919 / 0.737135 (-0.614217) | 0.076506 / 0.296338 (-0.219833) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.277299 / 0.215209 (0.062090) | 2.754771 / 2.077655 (0.677116) | 1.457164 / 1.504120 (-0.046956) | 1.318878 / 1.541195 (-0.222317) | 1.374245 / 1.468490 (-0.094245) | 0.566253 / 4.584777 (-4.018524) | 2.352589 / 3.745712 (-1.393123) | 2.764263 / 5.269862 (-2.505599) | 1.843141 / 4.565676 (-2.722535) | 0.063996 / 0.424275 (-0.360279) | 0.005045 / 0.007607 (-0.002562) | 0.336703 / 0.226044 (0.110658) | 3.342538 / 2.268929 (1.073609) | 1.836664 / 55.444624 (-53.607960) | 1.528901 / 6.876477 (-5.347576) | 1.769562 / 2.142072 (-0.372511) | 0.674192 / 4.805227 (-4.131035) | 0.122421 / 6.500664 (-6.378243) | 0.043714 / 0.075469 (-0.031756) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.989432 / 1.841788 (-0.852356) | 12.178341 / 8.074308 (4.104033) | 9.730838 / 10.191392 (-0.460554) | 0.146751 / 0.680424 (-0.533673) | 0.014720 / 0.534201 (-0.519481) | 0.285821 / 0.579283 (-0.293462) | 0.266474 / 0.434364 (-0.167889) | 0.327886 / 0.540337 (-0.212451) | 0.455672 / 1.386936 (-0.931264) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005691 / 0.011353 (-0.005662) | 0.004089 / 0.011008 (-0.006919) | 0.049878 / 0.038508 (0.011370) | 0.033578 / 0.023109 (0.010469) | 0.268295 / 0.275898 (-0.007603) | 0.288918 / 0.323480 (-0.034561) | 0.005092 / 0.007986 (-0.002894) | 0.002916 / 0.004328 (-0.001412) | 0.049489 / 0.004250 (0.045239) | 0.042495 / 0.037052 (0.005442) | 0.276253 / 0.258489 (0.017764) | 0.313321 / 0.293841 (0.019480) | 0.029386 / 0.128546 (-0.099160) | 0.010926 / 0.075646 (-0.064720) | 0.071747 / 0.419271 (-0.347525) | 0.033642 / 0.043533 (-0.009891) | 0.264950 / 0.255139 (0.009811) | 0.282962 / 0.283200 (-0.000238) | 0.018878 / 0.141683 (-0.122805) | 1.170685 / 1.452155 (-0.281470) | 1.198321 / 1.492716 (-0.294396) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.100422 / 0.018006 (0.082415) | 0.311750 / 0.000490 (0.311260) | 0.000235 / 0.000200 (0.000035) | 0.000063 / 0.000054 (0.000008) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023093 / 0.037411 (-0.014318) | 0.076934 / 0.014526 (0.062408) | 0.088959 / 0.176557 (-0.087598) | 0.129511 / 0.737135 (-0.607624) | 0.090151 / 0.296338 (-0.206187) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.301646 / 0.215209 (0.086437) | 2.961780 / 2.077655 (0.884126) | 1.656051 / 1.504120 (0.151931) | 1.533154 / 1.541195 (-0.008041) | 1.585152 / 1.468490 (0.116662) | 0.582157 / 4.584777 (-4.002620) | 0.954881 / 3.745712 (-2.790831) | 2.813174 / 5.269862 (-2.456688) | 1.842840 / 4.565676 (-2.722837) | 0.065598 / 0.424275 (-0.358677) | 0.005306 / 0.007607 (-0.002301) | 0.359610 / 0.226044 (0.133565) | 3.575320 / 2.268929 (1.306391) | 2.015327 / 55.444624 (-53.429297) | 1.734086 / 6.876477 (-5.142391) | 1.919081 / 2.142072 (-0.222991) | 0.671178 / 4.805227 (-4.134049) | 0.120109 / 6.500664 (-6.380555) | 0.042353 / 0.075469 (-0.033116) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.011726 / 1.841788 (-0.830062) | 13.007806 / 8.074308 (4.933498) | 10.632486 / 10.191392 (0.441094) | 0.148535 / 0.680424 (-0.531889) | 0.015988 / 0.534201 (-0.518213) | 0.290023 / 0.579283 (-0.289260) | 0.130685 / 0.434364 (-0.303679) | 0.322912 / 0.540337 (-0.217425) | 0.420596 / 1.386936 (-0.966340) |

![](https://cml.dev/watermark.png#336512dcba4fdb4c349d5ecb632b6ced80e038d5 "CML watermark")


Fix small typo

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005507 / 0.011353 (-0.005845) | 0.003757 / 0.011008 (-0.007251) | 0.063274 / 0.038508 (0.024766) | 0.029720 / 0.023109 (0.006610) | 0.247974 / 0.275898 (-0.027924) | 0.272283 / 0.323480 (-0.051197) | 0.004186 / 0.007986 (-0.003799) | 0.002820 / 0.004328 (-0.001508) | 0.049070 / 0.004250 (0.044820) | 0.050026 / 0.037052 (0.012973) | 0.256501 / 0.258489 (-0.001988) | 0.297082 / 0.293841 (0.003241) | 0.028549 / 0.128546 (-0.099997) | 0.010361 / 0.075646 (-0.065285) | 0.213202 / 0.419271 (-0.206070) | 0.038117 / 0.043533 (-0.005416) | 0.258878 / 0.255139 (0.003739) | 0.282980 / 0.283200 (-0.000220) | 0.018911 / 0.141683 (-0.122772) | 1.118857 / 1.452155 (-0.333298) | 1.157763 / 1.492716 (-0.334953) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.004499 / 0.018006 (-0.013507) | 0.310445 / 0.000490 (0.309956) | 0.000218 / 0.000200 (0.000018) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019275 / 0.037411 (-0.018137) | 0.063257 / 0.014526 (0.048731) | 0.075833 / 0.176557 (-0.100724) | 0.122323 / 0.737135 (-0.614812) | 0.079046 / 0.296338 (-0.217292) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.292811 / 0.215209 (0.077602) | 2.903501 / 2.077655 (0.825846) | 1.592434 / 1.504120 (0.088314) | 1.450833 / 1.541195 (-0.090362) | 1.481285 / 1.468490 (0.012795) | 0.570150 / 4.584777 (-4.014627) | 2.388618 / 3.745712 (-1.357094) | 2.699322 / 5.269862 (-2.570540) | 1.781405 / 4.565676 (-2.784272) | 0.063451 / 0.424275 (-0.360824) | 0.004979 / 0.007607 (-0.002628) | 0.353346 / 0.226044 (0.127302) | 3.541217 / 2.268929 (1.272289) | 1.972335 / 55.444624 (-53.472289) | 1.634780 / 6.876477 (-5.241697) | 1.815944 / 2.142072 (-0.326128) | 0.651559 / 4.805227 (-4.153669) | 0.118398 / 6.500664 (-6.382266) | 0.041962 / 0.075469 (-0.033507) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.971435 / 1.841788 (-0.870352) | 11.843740 / 8.074308 (3.769431) | 9.716333 / 10.191392 (-0.475059) | 0.145923 / 0.680424 (-0.534501) | 0.015073 / 0.534201 (-0.519128) | 0.293307 / 0.579283 (-0.285976) | 0.265505 / 0.434364 (-0.168859) | 0.327578 / 0.540337 (-0.212760) | 0.436409 / 1.386936 (-0.950527) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005647 / 0.011353 (-0.005706) | 0.003669 / 0.011008 (-0.007339) | 0.050234 / 0.038508 (0.011726) | 0.033033 / 0.023109 (0.009924) | 0.269303 / 0.275898 (-0.006595) | 0.282472 / 0.323480 (-0.041008) | 0.004283 / 0.007986 (-0.003703) | 0.002821 / 0.004328 (-0.001507) | 0.050887 / 0.004250 (0.046637) | 0.041618 / 0.037052 (0.004565) | 0.277628 / 0.258489 (0.019139) | 0.310539 / 0.293841 (0.016698) | 0.030036 / 0.128546 (-0.098511) | 0.010401 / 0.075646 (-0.065245) | 0.058845 / 0.419271 (-0.360427) | 0.033676 / 0.043533 (-0.009857) | 0.261148 / 0.255139 (0.006009) | 0.295232 / 0.283200 (0.012032) | 0.018603 / 0.141683 (-0.123080) | 1.132182 / 1.452155 (-0.319972) | 1.173763 / 1.492716 (-0.318953) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.100594 / 0.018006 (0.082588) | 0.308101 / 0.000490 (0.307611) | 0.000217 / 0.000200 (0.000017) | 0.000055 / 0.000054 (0.000000) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023040 / 0.037411 (-0.014371) | 0.080676 / 0.014526 (0.066150) | 0.094687 / 0.176557 (-0.081870) | 0.129780 / 0.737135 (-0.607356) | 0.092241 / 0.296338 (-0.204097) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.294799 / 0.215209 (0.079590) | 2.957570 / 2.077655 (0.879915) | 1.576795 / 1.504120 (0.072675) | 1.446869 / 1.541195 (-0.094326) | 1.463133 / 1.468490 (-0.005357) | 0.568511 / 4.584777 (-4.016266) | 1.011502 / 3.745712 (-2.734211) | 2.759571 / 5.269862 (-2.510291) | 1.771738 / 4.565676 (-2.793939) | 0.064104 / 0.424275 (-0.360171) | 0.005160 / 0.007607 (-0.002448) | 0.347554 / 0.226044 (0.121510) | 3.463905 / 2.268929 (1.194976) | 1.931843 / 55.444624 (-53.512781) | 1.622765 / 6.876477 (-5.253712) | 1.809146 / 2.142072 (-0.332926) | 0.653388 / 4.805227 (-4.151839) | 0.122703 / 6.500664 (-6.377961) | 0.041680 / 0.075469 (-0.033790) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.000428 / 1.841788 (-0.841359) | 12.503003 / 8.074308 (4.428695) | 10.434802 / 10.191392 (0.243410) | 0.144684 / 0.680424 (-0.535740) | 0.015988 / 0.534201 (-0.518213) | 0.287179 / 0.579283 (-0.292104) | 0.124811 / 0.434364 (-0.309553) | 0.327855 / 0.540337 (-0.212482) | 0.425144 / 1.386936 (-0.961792) |

![](https://cml.dev/watermark.png#f7170067f819222153fcd45682db61279bdfe673 "CML watermark")


Remove default `trust_remote_code=True`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6954). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004881 / 0.011353 (-0.006472) | 0.003246 / 0.011008 (-0.007762) | 0.062496 / 0.038508 (0.023988) | 0.030760 / 0.023109 (0.007651) | 0.241500 / 0.275898 (-0.034398) | 0.272073 / 0.323480 (-0.051407) | 0.004123 / 0.007986 (-0.003863) | 0.002796 / 0.004328 (-0.001533) | 0.049015 / 0.004250 (0.044764) | 0.047095 / 0.037052 (0.010043) | 0.257002 / 0.258489 (-0.001487) | 0.287602 / 0.293841 (-0.006239) | 0.027281 / 0.128546 (-0.101265) | 0.010132 / 0.075646 (-0.065514) | 0.203699 / 0.419271 (-0.215572) | 0.036553 / 0.043533 (-0.006980) | 0.246221 / 0.255139 (-0.008918) | 0.268137 / 0.283200 (-0.015062) | 0.017260 / 0.141683 (-0.124423) | 1.100677 / 1.452155 (-0.351478) | 1.148367 / 1.492716 (-0.344349) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.102519 / 0.018006 (0.084513) | 0.301929 / 0.000490 (0.301439) | 0.000223 / 0.000200 (0.000023) | 0.000046 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018590 / 0.037411 (-0.018821) | 0.061615 / 0.014526 (0.047089) | 0.074579 / 0.176557 (-0.101978) | 0.121415 / 0.737135 (-0.615720) | 0.075696 / 0.296338 (-0.220642) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.283842 / 0.215209 (0.068633) | 2.788321 / 2.077655 (0.710666) | 1.481376 / 1.504120 (-0.022743) | 1.356064 / 1.541195 (-0.185131) | 1.380592 / 1.468490 (-0.087898) | 0.575577 / 4.584777 (-4.009199) | 2.471858 / 3.745712 (-1.273854) | 2.760769 / 5.269862 (-2.509093) | 1.808638 / 4.565676 (-2.757038) | 0.064930 / 0.424275 (-0.359345) | 0.005056 / 0.007607 (-0.002551) | 0.337794 / 0.226044 (0.111750) | 3.359444 / 2.268929 (1.090515) | 1.829540 / 55.444624 (-53.615084) | 1.518660 / 6.876477 (-5.357817) | 1.671612 / 2.142072 (-0.470460) | 0.664286 / 4.805227 (-4.140941) | 0.119593 / 6.500664 (-6.381071) | 0.042519 / 0.075469 (-0.032950) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.993152 / 1.841788 (-0.848636) | 11.733054 / 8.074308 (3.658746) | 9.746734 / 10.191392 (-0.444658) | 0.143026 / 0.680424 (-0.537398) | 0.014900 / 0.534201 (-0.519301) | 0.292243 / 0.579283 (-0.287040) | 0.261301 / 0.434364 (-0.173063) | 0.330838 / 0.540337 (-0.209500) | 0.523719 / 1.386936 (-0.863217) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005707 / 0.011353 (-0.005646) | 0.003523 / 0.011008 (-0.007485) | 0.052265 / 0.038508 (0.013757) | 0.034296 / 0.023109 (0.011187) | 0.266589 / 0.275898 (-0.009309) | 0.288441 / 0.323480 (-0.035039) | 0.004507 / 0.007986 (-0.003478) | 0.002745 / 0.004328 (-0.001583) | 0.049417 / 0.004250 (0.045167) | 0.042679 / 0.037052 (0.005627) | 0.278518 / 0.258489 (0.020029) | 0.328751 / 0.293841 (0.034911) | 0.029530 / 0.128546 (-0.099016) | 0.010373 / 0.075646 (-0.065274) | 0.058207 / 0.419271 (-0.361064) | 0.033434 / 0.043533 (-0.010099) | 0.267902 / 0.255139 (0.012763) | 0.288192 / 0.283200 (0.004993) | 0.018866 / 0.141683 (-0.122817) | 1.132734 / 1.452155 (-0.319421) | 1.172879 / 1.492716 (-0.319837) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.097787 / 0.018006 (0.079780) | 0.305509 / 0.000490 (0.305019) | 0.000268 / 0.000200 (0.000068) | 0.000060 / 0.000054 (0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023230 / 0.037411 (-0.014181) | 0.076637 / 0.014526 (0.062111) | 0.088386 / 0.176557 (-0.088171) | 0.131079 / 0.737135 (-0.606057) | 0.091142 / 0.296338 (-0.205197) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295586 / 0.215209 (0.080377) | 2.872090 / 2.077655 (0.794435) | 1.538152 / 1.504120 (0.034032) | 1.405695 / 1.541195 (-0.135500) | 1.421058 / 1.468490 (-0.047432) | 0.561179 / 4.584777 (-4.023598) | 0.943954 / 3.745712 (-2.801758) | 2.684381 / 5.269862 (-2.585481) | 1.757457 / 4.565676 (-2.808220) | 0.062903 / 0.424275 (-0.361372) | 0.004998 / 0.007607 (-0.002610) | 0.370290 / 0.226044 (0.144245) | 3.374988 / 2.268929 (1.106059) | 1.899282 / 55.444624 (-53.545342) | 1.598787 / 6.876477 (-5.277690) | 1.735371 / 2.142072 (-0.406702) | 0.647367 / 4.805227 (-4.157860) | 0.116975 / 6.500664 (-6.383689) | 0.040811 / 0.075469 (-0.034658) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.996380 / 1.841788 (-0.845408) | 12.225657 / 8.074308 (4.151349) | 10.291221 / 10.191392 (0.099829) | 0.142791 / 0.680424 (-0.537633) | 0.016087 / 0.534201 (-0.518114) | 0.299978 / 0.579283 (-0.279305) | 0.149444 / 0.434364 (-0.284920) | 0.321354 / 0.540337 (-0.218984) | 0.414492 / 1.386936 (-0.972444) |

![](https://cml.dev/watermark.png#a2dc287cbef5311cf1a32ad4e3685f4052db227c "CML watermark")

@lhoestq Thanks for the PR, Is there a way to detect if `trust_remote_code=True` will be required for loading the dataset, without loading it? It would be great if you could please point me to the relevant documentation.

You can check the presence of a python loading script in the repository.

If there is a .py file named after the repository name, then it requires trust_remote_code.

TODO:
- [x] fix tests


Remove canonical datasets from docs

documentation

Improvements or additions to documentation

Remove canonical datasets from docs, now that we no longer have canonical datasets.


Move info_utils errors to exceptions module

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6952). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005232 / 0.011353 (-0.006121) | 0.003744 / 0.011008 (-0.007264) | 0.064089 / 0.038508 (0.025581) | 0.032409 / 0.023109 (0.009300) | 0.255886 / 0.275898 (-0.020013) | 0.276033 / 0.323480 (-0.047447) | 0.004165 / 0.007986 (-0.003821) | 0.002741 / 0.004328 (-0.001588) | 0.052145 / 0.004250 (0.047894) | 0.043863 / 0.037052 (0.006811) | 0.258844 / 0.258489 (0.000355) | 0.290108 / 0.293841 (-0.003733) | 0.027390 / 0.128546 (-0.101156) | 0.010543 / 0.075646 (-0.065103) | 0.206936 / 0.419271 (-0.212335) | 0.036778 / 0.043533 (-0.006755) | 0.254331 / 0.255139 (-0.000808) | 0.279037 / 0.283200 (-0.004163) | 0.018564 / 0.141683 (-0.123119) | 1.112765 / 1.452155 (-0.339390) | 1.160099 / 1.492716 (-0.332617) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092148 / 0.018006 (0.074142) | 0.297156 / 0.000490 (0.296667) | 0.000211 / 0.000200 (0.000011) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018797 / 0.037411 (-0.018615) | 0.062992 / 0.014526 (0.048466) | 0.076361 / 0.176557 (-0.100195) | 0.121168 / 0.737135 (-0.615968) | 0.075845 / 0.296338 (-0.220494) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.293842 / 0.215209 (0.078633) | 2.880720 / 2.077655 (0.803065) | 1.477779 / 1.504120 (-0.026341) | 1.345136 / 1.541195 (-0.196059) | 1.352153 / 1.468490 (-0.116337) | 0.574722 / 4.584777 (-4.010055) | 2.373925 / 3.745712 (-1.371787) | 2.750704 / 5.269862 (-2.519157) | 1.725979 / 4.565676 (-2.839697) | 0.063006 / 0.424275 (-0.361269) | 0.005019 / 0.007607 (-0.002588) | 0.341228 / 0.226044 (0.115184) | 3.352576 / 2.268929 (1.083647) | 1.821363 / 55.444624 (-53.623261) | 1.529441 / 6.876477 (-5.347036) | 1.543401 / 2.142072 (-0.598671) | 0.634282 / 4.805227 (-4.170945) | 0.115565 / 6.500664 (-6.385099) | 0.042514 / 0.075469 (-0.032956) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.987532 / 1.841788 (-0.854255) | 11.483853 / 8.074308 (3.409545) | 9.565657 / 10.191392 (-0.625735) | 0.141247 / 0.680424 (-0.539176) | 0.015026 / 0.534201 (-0.519175) | 0.299905 / 0.579283 (-0.279378) | 0.267667 / 0.434364 (-0.166697) | 0.320661 / 0.540337 (-0.219676) | 0.427368 / 1.386936 (-0.959568) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005448 / 0.011353 (-0.005905) | 0.003726 / 0.011008 (-0.007283) | 0.049776 / 0.038508 (0.011268) | 0.032733 / 0.023109 (0.009624) | 0.261387 / 0.275898 (-0.014511) | 0.280087 / 0.323480 (-0.043393) | 0.004351 / 0.007986 (-0.003634) | 0.002842 / 0.004328 (-0.001487) | 0.049440 / 0.004250 (0.045190) | 0.039585 / 0.037052 (0.002533) | 0.266331 / 0.258489 (0.007842) | 0.299643 / 0.293841 (0.005802) | 0.029649 / 0.128546 (-0.098897) | 0.010381 / 0.075646 (-0.065265) | 0.058596 / 0.419271 (-0.360676) | 0.033271 / 0.043533 (-0.010262) | 0.251070 / 0.255139 (-0.004069) | 0.272850 / 0.283200 (-0.010349) | 0.016728 / 0.141683 (-0.124955) | 1.146952 / 1.452155 (-0.305202) | 1.182602 / 1.492716 (-0.310114) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091673 / 0.018006 (0.073667) | 0.297228 / 0.000490 (0.296738) | 0.000197 / 0.000200 (-0.000003) | 0.000051 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023174 / 0.037411 (-0.014237) | 0.078866 / 0.014526 (0.064341) | 0.088436 / 0.176557 (-0.088121) | 0.129650 / 0.737135 (-0.607485) | 0.091100 / 0.296338 (-0.205238) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.293882 / 0.215209 (0.078673) | 2.882667 / 2.077655 (0.805012) | 1.562949 / 1.504120 (0.058829) | 1.435104 / 1.541195 (-0.106090) | 1.450815 / 1.468490 (-0.017675) | 0.584090 / 4.584777 (-4.000687) | 0.984176 / 3.745712 (-2.761536) | 2.668740 / 5.269862 (-2.601121) | 1.766993 / 4.565676 (-2.798683) | 0.064710 / 0.424275 (-0.359565) | 0.005329 / 0.007607 (-0.002278) | 0.346008 / 0.226044 (0.119964) | 3.414576 / 2.268929 (1.145647) | 1.911388 / 55.444624 (-53.533236) | 1.660357 / 6.876477 (-5.216120) | 1.818628 / 2.142072 (-0.323444) | 0.659585 / 4.805227 (-4.145643) | 0.116980 / 6.500664 (-6.383684) | 0.041364 / 0.075469 (-0.034105) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.005659 / 1.841788 (-0.836129) | 12.023761 / 8.074308 (3.949453) | 10.351086 / 10.191392 (0.159694) | 0.143261 / 0.680424 (-0.537162) | 0.016143 / 0.534201 (-0.518058) | 0.287793 / 0.579283 (-0.291490) | 0.123698 / 0.434364 (-0.310666) | 0.325241 / 0.540337 (-0.215097) | 0.418772 / 1.386936 (-0.968164) |

![](https://cml.dev/watermark.png#37a603679f451826cfafd8aae00738b01dcb9d58 "CML watermark")

Move `info_utils` errors to `exceptions` module.

Additionally rename some of them, deprecate the former ones, and make the deprecation backward compatible (by making the new errors inherit from the former ones).


load_dataset() should load all subsets, if no specific subset is specified

enhancement

New feature or request

Feel free to open a PR in `m-a-p/COIG-CQIA` to define a default subset. Currently there is no default.

You can find some documentation at https://huggingface.co/docs/hub/datasets-manual-configuration#multiple-configurations

@lhoestq

Whilst having a default subset readily available (e.g. `all`) by the dataset author is an ideal solution, it is not always the reality.

Without the ability to fork the dataset, this can be problematic.

As far as I know, it is not possible at all to specify multiple subsets in a generalized programmatic way without hard coding subset names for a specific dataset.

Even the ability to fetch subset names and loop over them would be sufficient.

Please note that each subset can have different feature columns, thus making it impossible to load them all into a unique Dataset instance.

That is why subsets were created: to support different but related datasets to coexist in a single dataset repository.

If you would like to programmatically get the list of subset names, you can use `datasets.get_dataset_config_names`: https://huggingface.co/docs/datasets/v2.20.0/en/load_hub#configurations

### Feature request

Currently load_dataset() is forcing users to specify a subset. Example

`from datasets import load_dataset
dataset = load_dataset("m-a-p/COIG-CQIA")`

```---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-10-c0cb49385da6>](https://localhost:8080/#) in <cell line: 2>()
      1 from datasets import load_dataset
----> 2 dataset = load_dataset("m-a-p/COIG-CQIA")

3 frames
[/usr/local/lib/python3.10/dist-packages/datasets/builder.py](https://localhost:8080/#) in _create_builder_config(self, config_name, custom_features, **config_kwargs)
    582                     if not config_kwargs:
    583                         example_of_usage = f"load_dataset('{self.dataset_name}', '{self.BUILDER_CONFIGS[0].name}')"
--> 584                         raise ValueError(
    585                             "Config name is missing."
    586                             f"\nPlease pick one among the available configs: {list(self.builder_configs.keys())}"

ValueError: Config name is missing.
Please pick one among the available configs: ['chinese_traditional', 'coig_pc', 'exam', 'finance', 'douban', 'human_value', 'logi_qa', 'ruozhiba', 'segmentfault', 'wiki', 'wikihow', 'xhs', 'zhihu']
Example of usage:
	`load_dataset('coig-cqia', 'chinese_traditional')`
```
This means a dataset cannot contain all the subsets at the same time. I guess one workaround is to manually specify the subset files like in [here](https://huggingface.co/datasets/m-a-p/COIG-CQIA/discussions/1#658698b44bb41498f75c5622), which is clumsy.

### Motivation

Ideally, if not subset is specified, the API should just try to load all subsets. This makes it much easier to handle datasets w/ subsets.

### Your contribution

Not sure since I'm not familiar w/ the lib src.


`Dataset.with_format` behaves inconsistently with documentation

Hi ! It seems the documentation was outdated in this paragraph

I fixed it here: https://github.com/huggingface/datasets/pull/6956

### Describe the bug

The actual behavior of the interface `Dataset.with_format` is inconsistent with the documentation.
https://huggingface.co/docs/datasets/use_with_pytorch#n-dimensional-arrays
https://huggingface.co/docs/datasets/v2.19.0/en/use_with_tensorflow#n-dimensional-arrays

> If your dataset consists of N-dimensional arrays, you will see that by default they are considered as nested lists.
> In particular, a PyTorch formatted dataset outputs nested lists instead of a single tensor.
> A TensorFlow formatted dataset outputs a RaggedTensor instead of a single tensor.

But I get a single tensor by default, which is  inconsistent with the description.

Actually the current behavior seems more reasonable to me. Therefore, the document needs to be modified.

### Steps to reproduce the bug

```python
>>> from datasets import Dataset
>>> data = [[[1, 2],[3, 4]],[[5, 6],[7, 8]]]
>>> ds = Dataset.from_dict({"data": data})
>>> ds = ds.with_format("torch")
>>> ds[0]
{'data': tensor([[1, 2],
        [3, 4]])}
>>> ds = ds.with_format("tf")
>>> ds[0]
{'data': <tf.Tensor: shape=(2, 2), dtype=int64, numpy=
array([[1, 2],
       [3, 4]])>}
```

### Expected behavior

```python
>>> from datasets import Dataset
>>> data = [[[1, 2],[3, 4]],[[5, 6],[7, 8]]]
>>> ds = Dataset.from_dict({"data": data})
>>> ds = ds.with_format("torch")
>>> ds[0]
{'data': [tensor([1, 2]), tensor([3, 4])]}
>>> ds = ds.with_format("tf")
>>> ds[0]
{'data': <tf.RaggedTensor [[1, 2], [3, 4]]>}
```

### Environment info

datasets==2.19.1
torch==2.1.0
tensorflow==2.13.1


load_dataset error

Hi, @lion-ops.

In our Continuous Integration we have many tests on loading JSON files and all of them work properly.

Could you please share your "train.json" file, so that we can try to reproduce the issue you have?

> Hi, @lion-ops.
>
> In our Continuous Integration we have many tests on loading JSON files and all of them work properly.
>
> Could you please share your "train.json" file, so that we can try to reproduce the issue you have?

Thank you for your reply. I can load it normally in another server. Is it possible that the disk of my server is a network disk in the LAN, so it will be downloaded from the LAN and get stuck?

### Describe the bug

Why does the program get stuck when I use load_dataset method, and it still gets stuck after loading for several hours? In fact, my json file is only 21m, and I can load it in one go using open('', 'r').

### Steps to reproduce the bug

1. pip install datasets==2.19.2
2. from datasets import Dataset, DatasetDict, NamedSplit, Split, load_dataset
3. data = load_dataset('json', data_files='train.json')

### Expected behavior

It is able to load my json correctly

### Environment info

datasets==2.19.2


to_tf_dataset: Visible devices cannot be modified after being initialized

### Describe the bug

When trying to use to_tf_dataset with a custom data_loader collate_fn when I use parallelism I am met with the following error as many times as number of workers there were in ``num_workers``.

  File "/opt/miniconda/envs/env/lib/python3.11/site-packages/multiprocess/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/miniconda/envs/env/lib/python3.11/site-packages/multiprocess/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/miniconda/envs/env/lib/python3.11/site-packages/datasets/utils/tf_utils.py", line 438, in worker_loop
    tf.config.set_visible_devices([], "GPU")  # Make sure workers don't try to allocate GPU memory
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda/envs/env/lib/python3.11/site-packages/tensorflow/python/framework/config.py", line 566, in set_visible_devices
    context.context().set_visible_devices(devices, device_type)
  File "/opt/miniconda/envs/env/lib/python3.11/site-packages/tensorflow/python/eager/context.py", line 1737, in set_visible_devices
    raise RuntimeError(
RuntimeError: Visible devices cannot be modified after being initialized

### Steps to reproduce the bug

1. Download a dataset using HuggingFace load_dataset
2. Define a function that transforms the data in some way to be used in the collate_fn argument
3. Provide a ``batch_size`` and ``num_workers`` value in the ``to_tf_dataset`` function
4. Either retrieve directly or use tfds benchmark to test the dataset

``` python
from datasets import load_datasets
import tensorflow_datasets as tfds
from keras_cv.layers import Resizing

def data_loader(examples):
      x = Resizing(examples[0]['image'], 256, 256, crop_to_aspect_ratio=True)
      return {X[0]: x}

ds = load_datasets("logasja/FDF", split="test")

ds = ds.to_tf_dataset(collate_fn=data_loader, batch_size=16, num_workers=2)

tfds.benchmark(ds)
```

### Expected behavior

Use multiple processes to apply transformations from the collate_fn to the tf dataset on the CPU.

### Environment info

- `datasets` version: 2.19.1
- Platform: Linux-6.5.0-1023-oracle-x86_64-with-glibc2.35
- Python version: 3.11.8
- `huggingface_hub` version: 0.22.2
- PyArrow version: 15.0.2
- Pandas version: 2.2.1
- `fsspec` version: 2024.2.0