Fix archive fs test

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008664 / 0.011353 (-0.002689) | 0.004622 / 0.011008 (-0.006387) | 0.101716 / 0.038508 (0.063208) | 0.030044 / 0.023109 (0.006935) | 0.298476 / 0.275898 (0.022578) | 0.360873 / 0.323480 (0.037393) | 0.007012 / 0.007986 (-0.000974) | 0.003409 / 0.004328 (-0.000919) | 0.077731 / 0.004250 (0.073480) | 0.035493 / 0.037052 (-0.001560) | 0.311474 / 0.258489 (0.052985) | 0.357276 / 0.293841 (0.063435) | 0.033909 / 0.128546 (-0.094638) | 0.011315 / 0.075646 (-0.064332) | 0.323149 / 0.419271 (-0.096122) | 0.040678 / 0.043533 (-0.002855) | 0.298487 / 0.255139 (0.043348) | 0.323107 / 0.283200 (0.039907) | 0.086641 / 0.141683 (-0.055042) | 1.452905 / 1.452155 (0.000750) | 1.510953 / 1.492716 (0.018237) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.190607 / 0.018006 (0.172601) | 0.409786 / 0.000490 (0.409297) | 0.000818 / 0.000200 (0.000618) | 0.000075 / 0.000054 (0.000020) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023267 / 0.037411 (-0.014144) | 0.095390 / 0.014526 (0.080864) | 0.104381 / 0.176557 (-0.072175) | 0.150735 / 0.737135 (-0.586401) | 0.106876 / 0.296338 (-0.189462) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.434259 / 0.215209 (0.219050) | 4.326978 / 2.077655 (2.249323) | 2.036690 / 1.504120 (0.532570) | 1.836459 / 1.541195 (0.295264) | 1.904003 / 1.468490 (0.435513) | 0.697265 / 4.584777 (-3.887512) | 3.435911 / 3.745712 (-0.309802) | 3.240918 / 5.269862 (-2.028944) | 1.629220 / 4.565676 (-2.936456) | 0.083158 / 0.424275 (-0.341117) | 0.012604 / 0.007607 (0.004997) | 0.539818 / 0.226044 (0.313773) | 5.397860 / 2.268929 (3.128932) | 2.483890 / 55.444624 (-52.960735) | 2.132404 / 6.876477 (-4.744072) | 2.162583 / 2.142072 (0.020510) | 0.817773 / 4.805227 (-3.987454) | 0.151677 / 6.500664 (-6.348987) | 0.066569 / 0.075469 (-0.008900) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.243449 / 1.841788 (-0.598339) | 13.699854 / 8.074308 (5.625546) | 13.930979 / 10.191392 (3.739587) | 0.165344 / 0.680424 (-0.515079) | 0.028910 / 0.534201 (-0.505291) | 0.396201 / 0.579283 (-0.183082) | 0.404448 / 0.434364 (-0.029916) | 0.482031 / 0.540337 (-0.058306) | 0.570023 / 1.386936 (-0.816913) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006785 / 0.011353 (-0.004568) | 0.004643 / 0.011008 (-0.006365) | 0.076755 / 0.038508 (0.038247) | 0.027893 / 0.023109 (0.004783) | 0.342539 / 0.275898 (0.066641) | 0.379103 / 0.323480 (0.055623) | 0.005107 / 0.007986 (-0.002879) | 0.003413 / 0.004328 (-0.000915) | 0.075779 / 0.004250 (0.071528) | 0.039251 / 0.037052 (0.002199) | 0.343425 / 0.258489 (0.084935) | 0.385292 / 0.293841 (0.091451) | 0.032229 / 0.128546 (-0.096317) | 0.011666 / 0.075646 (-0.063980) | 0.086452 / 0.419271 (-0.332819) | 0.042918 / 0.043533 (-0.000615) | 0.343145 / 0.255139 (0.088006) | 0.367916 / 0.283200 (0.084717) | 0.090810 / 0.141683 (-0.050873) | 1.471679 / 1.452155 (0.019524) | 1.566683 / 1.492716 (0.073966) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.220343 / 0.018006 (0.202336) | 0.396155 / 0.000490 (0.395665) | 0.003831 / 0.000200 (0.003631) | 0.000080 / 0.000054 (0.000025) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024990 / 0.037411 (-0.012421) | 0.101270 / 0.014526 (0.086744) | 0.110115 / 0.176557 (-0.066442) | 0.161770 / 0.737135 (-0.575365) | 0.112187 / 0.296338 (-0.184151) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.436199 / 0.215209 (0.220989) | 4.329084 / 2.077655 (2.251429) | 2.043335 / 1.504120 (0.539215) | 1.836799 / 1.541195 (0.295604) | 1.908362 / 1.468490 (0.439872) | 0.700518 / 4.584777 (-3.884259) | 3.418003 / 3.745712 (-0.327710) | 1.860621 / 5.269862 (-3.409241) | 1.171343 / 4.565676 (-3.394334) | 0.083150 / 0.424275 (-0.341125) | 0.012543 / 0.007607 (0.004936) | 0.533528 / 0.226044 (0.307483) | 5.339660 / 2.268929 (3.070732) | 2.499494 / 55.444624 (-52.945131) | 2.154773 / 6.876477 (-4.721704) | 2.198734 / 2.142072 (0.056661) | 0.803383 / 4.805227 (-4.001844) | 0.150980 / 6.500664 (-6.349684) | 0.068050 / 0.075469 (-0.007419) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.309487 / 1.841788 (-0.532301) | 14.177068 / 8.074308 (6.102760) | 13.218912 / 10.191392 (3.027520) | 0.156857 / 0.680424 (-0.523567) | 0.016534 / 0.534201 (-0.517667) | 0.383986 / 0.579283 (-0.195297) | 0.395264 / 0.434364 (-0.039100) | 0.442310 / 0.540337 (-0.098027) | 0.535535 / 1.386936 (-0.851401) |

![](https://cml.dev/watermark.png#64e24bca88be711f4fdcb9c18edaddc1db0bbe2e "CML watermark")

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009446 / 0.011353 (-0.001907) | 0.005061 / 0.011008 (-0.005948) | 0.099783 / 0.038508 (0.061275) | 0.036379 / 0.023109 (0.013270) | 0.296769 / 0.275898 (0.020871) | 0.368990 / 0.323480 (0.045510) | 0.007891 / 0.007986 (-0.000094) | 0.003940 / 0.004328 (-0.000389) | 0.076284 / 0.004250 (0.072034) | 0.044390 / 0.037052 (0.007337) | 0.313373 / 0.258489 (0.054884) | 0.361118 / 0.293841 (0.067277) | 0.039058 / 0.128546 (-0.089488) | 0.012016 / 0.075646 (-0.063631) | 0.334239 / 0.419271 (-0.085033) | 0.047028 / 0.043533 (0.003495) | 0.297766 / 0.255139 (0.042627) | 0.312853 / 0.283200 (0.029653) | 0.099117 / 0.141683 (-0.042566) | 1.475487 / 1.452155 (0.023332) | 1.557487 / 1.492716 (0.064771) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.206243 / 0.018006 (0.188237) | 0.443920 / 0.000490 (0.443430) | 0.001404 / 0.000200 (0.001205) | 0.000078 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026347 / 0.037411 (-0.011065) | 0.105880 / 0.014526 (0.091354) | 0.116227 / 0.176557 (-0.060330) | 0.157404 / 0.737135 (-0.579732) | 0.121668 / 0.296338 (-0.174671) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.398614 / 0.215209 (0.183405) | 3.970657 / 2.077655 (1.893002) | 1.778899 / 1.504120 (0.274779) | 1.591806 / 1.541195 (0.050611) | 1.687717 / 1.468490 (0.219227) | 0.695399 / 4.584777 (-3.889378) | 3.829281 / 3.745712 (0.083569) | 2.140856 / 5.269862 (-3.129006) | 1.355027 / 4.565676 (-3.210650) | 0.085714 / 0.424275 (-0.338561) | 0.012130 / 0.007607 (0.004523) | 0.505807 / 0.226044 (0.279762) | 5.053098 / 2.268929 (2.784170) | 2.321694 / 55.444624 (-53.122931) | 2.015909 / 6.876477 (-4.860568) | 2.100862 / 2.142072 (-0.041210) | 0.855689 / 4.805227 (-3.949539) | 0.167192 / 6.500664 (-6.333472) | 0.062376 / 0.075469 (-0.013093) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.196647 / 1.841788 (-0.645141) | 14.971356 / 8.074308 (6.897048) | 13.897184 / 10.191392 (3.705792) | 0.193267 / 0.680424 (-0.487157) | 0.029252 / 0.534201 (-0.504949) | 0.444885 / 0.579283 (-0.134398) | 0.452792 / 0.434364 (0.018429) | 0.550157 / 0.540337 (0.009819) | 0.658524 / 1.386936 (-0.728412) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007774 / 0.011353 (-0.003579) | 0.005304 / 0.011008 (-0.005704) | 0.075530 / 0.038508 (0.037022) | 0.034930 / 0.023109 (0.011821) | 0.343879 / 0.275898 (0.067981) | 0.386487 / 0.323480 (0.063008) | 0.005998 / 0.007986 (-0.001987) | 0.005619 / 0.004328 (0.001291) | 0.075865 / 0.004250 (0.071614) | 0.050499 / 0.037052 (0.013446) | 0.345503 / 0.258489 (0.087014) | 0.392081 / 0.293841 (0.098240) | 0.037118 / 0.128546 (-0.091429) | 0.012540 / 0.075646 (-0.063107) | 0.086202 / 0.419271 (-0.333069) | 0.050672 / 0.043533 (0.007139) | 0.343622 / 0.255139 (0.088483) | 0.353853 / 0.283200 (0.070653) | 0.105408 / 0.141683 (-0.036274) | 1.460695 / 1.452155 (0.008540) | 1.524270 / 1.492716 (0.031554) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.219356 / 0.018006 (0.201350) | 0.440740 / 0.000490 (0.440251) | 0.014313 / 0.000200 (0.014114) | 0.000103 / 0.000054 (0.000048) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030297 / 0.037411 (-0.007115) | 0.108723 / 0.014526 (0.094197) | 0.125085 / 0.176557 (-0.051471) | 0.176664 / 0.737135 (-0.560471) | 0.126659 / 0.296338 (-0.169680) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.445790 / 0.215209 (0.230581) | 4.241046 / 2.077655 (2.163391) | 2.027381 / 1.504120 (0.523261) | 1.821070 / 1.541195 (0.279876) | 1.934417 / 1.468490 (0.465927) | 0.710897 / 4.584777 (-3.873880) | 3.840397 / 3.745712 (0.094685) | 3.959196 / 5.269862 (-1.310666) | 1.646069 / 4.565676 (-2.919608) | 0.088615 / 0.424275 (-0.335660) | 0.012321 / 0.007607 (0.004714) | 0.523463 / 0.226044 (0.297418) | 5.240147 / 2.268929 (2.971218) | 2.521639 / 55.444624 (-52.922986) | 2.246535 / 6.876477 (-4.629942) | 2.365913 / 2.142072 (0.223841) | 0.851288 / 4.805227 (-3.953939) | 0.170179 / 6.500664 (-6.330485) | 0.064732 / 0.075469 (-0.010737) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.255505 / 1.841788 (-0.586283) | 15.305457 / 8.074308 (7.231148) | 13.214186 / 10.191392 (3.022794) | 0.188971 / 0.680424 (-0.491453) | 0.018972 / 0.534201 (-0.515229) | 0.429621 / 0.579283 (-0.149662) | 0.428738 / 0.434364 (-0.005626) | 0.536241 / 0.540337 (-0.004096) | 0.632998 / 1.386936 (-0.753938) |

![](https://cml.dev/watermark.png#b64fae9509f6e9da9cabf0ce677966598fc61e38 "CML watermark")

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008435 / 0.011353 (-0.002918) | 0.004454 / 0.011008 (-0.006554) | 0.099091 / 0.038508 (0.060583) | 0.028890 / 0.023109 (0.005781) | 0.297450 / 0.275898 (0.021551) | 0.329025 / 0.323480 (0.005545) | 0.006584 / 0.007986 (-0.001401) | 0.004669 / 0.004328 (0.000340) | 0.077387 / 0.004250 (0.073137) | 0.033701 / 0.037052 (-0.003352) | 0.301272 / 0.258489 (0.042783) | 0.345401 / 0.293841 (0.051560) | 0.033473 / 0.128546 (-0.095073) | 0.011244 / 0.075646 (-0.064402) | 0.321941 / 0.419271 (-0.097330) | 0.040646 / 0.043533 (-0.002887) | 0.306686 / 0.255139 (0.051547) | 0.321868 / 0.283200 (0.038668) | 0.084281 / 0.141683 (-0.057401) | 1.491414 / 1.452155 (0.039259) | 1.542799 / 1.492716 (0.050083) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.188368 / 0.018006 (0.170362) | 0.398595 / 0.000490 (0.398105) | 0.000805 / 0.000200 (0.000605) | 0.000075 / 0.000054 (0.000021) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022690 / 0.037411 (-0.014721) | 0.096795 / 0.014526 (0.082269) | 0.104037 / 0.176557 (-0.072520) | 0.149409 / 0.737135 (-0.587727) | 0.108022 / 0.296338 (-0.188317) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.419316 / 0.215209 (0.204107) | 4.186850 / 2.077655 (2.109196) | 1.920182 / 1.504120 (0.416062) | 1.715493 / 1.541195 (0.174298) | 1.757767 / 1.468490 (0.289277) | 0.692296 / 4.584777 (-3.892480) | 3.342330 / 3.745712 (-0.403382) | 1.842063 / 5.269862 (-3.427798) | 1.150190 / 4.565676 (-3.415487) | 0.082792 / 0.424275 (-0.341483) | 0.012540 / 0.007607 (0.004933) | 0.528867 / 0.226044 (0.302822) | 5.297818 / 2.268929 (3.028890) | 2.313173 / 55.444624 (-53.131451) | 1.941723 / 6.876477 (-4.934754) | 1.982948 / 2.142072 (-0.159125) | 0.808951 / 4.805227 (-3.996276) | 0.149338 / 6.500664 (-6.351326) | 0.064838 / 0.075469 (-0.010631) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.187865 / 1.841788 (-0.653923) | 13.381918 / 8.074308 (5.307610) | 13.730627 / 10.191392 (3.539234) | 0.149976 / 0.680424 (-0.530447) | 0.028249 / 0.534201 (-0.505952) | 0.392591 / 0.579283 (-0.186692) | 0.403451 / 0.434364 (-0.030912) | 0.467484 / 0.540337 (-0.072853) | 0.560296 / 1.386936 (-0.826640) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006440 / 0.011353 (-0.004913) | 0.004488 / 0.011008 (-0.006521) | 0.077875 / 0.038508 (0.039367) | 0.027284 / 0.023109 (0.004174) | 0.341625 / 0.275898 (0.065727) | 0.374960 / 0.323480 (0.051480) | 0.005581 / 0.007986 (-0.002405) | 0.003326 / 0.004328 (-0.001003) | 0.076928 / 0.004250 (0.072677) | 0.038205 / 0.037052 (0.001153) | 0.345933 / 0.258489 (0.087444) | 0.383675 / 0.293841 (0.089834) | 0.031908 / 0.128546 (-0.096638) | 0.011724 / 0.075646 (-0.063922) | 0.086974 / 0.419271 (-0.332298) | 0.043084 / 0.043533 (-0.000449) | 0.339663 / 0.255139 (0.084524) | 0.363782 / 0.283200 (0.080582) | 0.090934 / 0.141683 (-0.050749) | 1.459718 / 1.452155 (0.007563) | 1.541104 / 1.492716 (0.048388) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.224005 / 0.018006 (0.205998) | 0.400727 / 0.000490 (0.400238) | 0.000427 / 0.000200 (0.000227) | 0.000061 / 0.000054 (0.000007) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024604 / 0.037411 (-0.012807) | 0.099813 / 0.014526 (0.085287) | 0.104034 / 0.176557 (-0.072523) | 0.156245 / 0.737135 (-0.580890) | 0.108739 / 0.296338 (-0.187600) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.440500 / 0.215209 (0.225291) | 4.379934 / 2.077655 (2.302279) | 2.075826 / 1.504120 (0.571706) | 1.867635 / 1.541195 (0.326441) | 1.919035 / 1.468490 (0.450545) | 0.696613 / 4.584777 (-3.888164) | 3.334993 / 3.745712 (-0.410720) | 1.857139 / 5.269862 (-3.412723) | 1.160598 / 4.565676 (-3.405079) | 0.083120 / 0.424275 (-0.341155) | 0.012475 / 0.007607 (0.004868) | 0.544607 / 0.226044 (0.318563) | 5.436808 / 2.268929 (3.167879) | 2.518562 / 55.444624 (-52.926063) | 2.158434 / 6.876477 (-4.718042) | 2.170691 / 2.142072 (0.028618) | 0.811297 / 4.805227 (-3.993930) | 0.150675 / 6.500664 (-6.349990) | 0.065655 / 0.075469 (-0.009814) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.277627 / 1.841788 (-0.564160) | 13.833501 / 8.074308 (5.759193) | 13.038718 / 10.191392 (2.847325) | 0.148837 / 0.680424 (-0.531587) | 0.016440 / 0.534201 (-0.517761) | 0.379147 / 0.579283 (-0.200136) | 0.379753 / 0.434364 (-0.054611) | 0.460197 / 0.540337 (-0.080141) | 0.544152 / 1.386936 (-0.842784) |

![](https://cml.dev/watermark.png#6e2a235cbab1c91dc5eca0cb123f9c9d9f743461 "CML watermark")


Version mismatch with multiprocess and dill on Python 3.10

Sorry, I just found https://github.com/apache/beam/issues/24458. It seems this issue is being worked on.

Reopening, since I think the docs should inform the user of this problem. For example, [this page](https://huggingface.co/docs/datasets/installation) says
> Datasets is tested on Python 3.7+.

but it should probably say that Beam Datasets do not work with Python 3.10 (or link to a known issues page).

Same problem on Colab using a vanilla setup running :
Python 3.10.11
apache-beam 2.47.0
datasets 2.12.0

Same problem,
py 3.10.11
apache-beam==2.47.0
datasets==2.12.0

I have made a workaround by forcing an install of the version of `multiprocess` version `0.70.15` (after installing `datasets` and `apache-beam`). I can confirm that (on Python 3.10 in [this colab notebook](https://colab.research.google.com/drive/1PTeGlshamFcJZix_GiS3vMXX_YzAhGv0?usp=sharing)) `datasets` can download pre-processed Wikipedia dumps and can download non-pre-processed dumps using `beam_runner="DirectRunner"`. I don't know if/how other `beam_runner`s can be made compatible.

Same problem.

```
python = "^3.10"
apache-beam = { extras = ["gcp"], version = "2.54.0" }
datasets = "^2.18.0"
```

### Describe the bug

Grabbing the latest version of `datasets` and `apache-beam` with `poetry` using Python 3.10 gives a crash at runtime. The crash is
```
File "/Users/adpauls/sc/git/DSI-transformers/data/NQ/create_NQ_train_vali.py", line 1, in <module>
    import datasets
  File "/Users/adpauls/Library/Caches/pypoetry/virtualenvs/yyy-oPbZ7mKM-py3.10/lib/python3.10/site-packages/datasets/__init__.py", line 43, in <module>
    from .arrow_dataset import Dataset
  File "/Users/adpauls/Library/Caches/pypoetry/virtualenvs/yyy-oPbZ7mKM-py3.10/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 65, in <module>
    from .arrow_reader import ArrowReader
  File "/Users/adpauls/Library/Caches/pypoetry/virtualenvs/yyy-oPbZ7mKM-py3.10/lib/python3.10/site-packages/datasets/arrow_reader.py", line 30, in <module>
    from .download.download_config import DownloadConfig
  File "/Users/adpauls/Library/Caches/pypoetry/virtualenvs/yyy-oPbZ7mKM-py3.10/lib/python3.10/site-packages/datasets/download/__init__.py", line 9, in <module>
    from .download_manager import DownloadManager, DownloadMode
  File "/Users/adpauls/Library/Caches/pypoetry/virtualenvs/yyy-oPbZ7mKM-py3.10/lib/python3.10/site-packages/datasets/download/download_manager.py", line 35, in <module>
    from ..utils.py_utils import NestedDataStructure, map_nested, size_str
  File "/Users/adpauls/Library/Caches/pypoetry/virtualenvs/yyy-oPbZ7mKM-py3.10/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 40, in <module>
    import multiprocess.pool
  File "/Users/adpauls/Library/Caches/pypoetry/virtualenvs/yyy-oPbZ7mKM-py3.10/lib/python3.10/site-packages/multiprocess/pool.py", line 609, in <module>
    class ThreadPool(Pool):
  File "/Users/adpauls/Library/Caches/pypoetry/virtualenvs/yyy-oPbZ7mKM-py3.10/lib/python3.10/site-packages/multiprocess/pool.py", line 611, in ThreadPool
    from .dummy import Process
  File "/Users/adpauls/Library/Caches/pypoetry/virtualenvs/yyy-oPbZ7mKM-py3.10/lib/python3.10/site-packages/multiprocess/dummy/__init__.py", line 87, in <module>
    class Condition(threading._Condition):
AttributeError: module 'threading' has no attribute '_Condition'. Did you mean: 'Condition'?
```

I think this is a bad interaction of versions from `dill`, `multiprocess`, `apache-beam`, and `threading` from the Python (3.10) standard lib. Upgrading `multiprocess` to a version that does not crash like this is not possible because `apache-beam` pins `dill` to and old version:
```
Because multiprocess (0.70.10) depends on dill (>=0.3.2)
 and apache-beam (2.45.0) depends on dill (>=0.3.1.1,<0.3.2), multiprocess (0.70.10) is incompatible with apache-beam (2.45.0).
And because no versions of apache-beam match >2.45.0,<3.0.0, multiprocess (0.70.10) is incompatible with apache-beam (>=2.45.0,<3.0.0).
So, because yyy depends on both apache-beam (^2.45.0) and multiprocess (0.70.10), version solving failed.
```

Perhaps it is not right to file a bug here, but I'm not totally sure whose fault it is. And in any case, this is an immediate blocker to using `datasets` out of the box.

Possibly related to https://github.com/huggingface/datasets/issues/5232.

### Steps to reproduce the bug

Steps to reproduce:
1. Make a poetry project with this configuration
   ```
   [tool.poetry]
   name = "yyy"
   version = "0.1.0"
   description = ""
   authors = ["Adam Pauls <adpauls@gmail.com>"]
   readme = "README.md"
   packages = [{ include = "xxx" }]

   [tool.poetry.dependencies]
   python = ">=3.10,<3.11"
   datasets = "^2.10.1"
   apache-beam = "^2.45.0"

   [build-system]
   requires = ["poetry-core"]
   build-backend = "poetry.core.masonry.api"
   ```
   2. `poetry install`.
   3. `poetry run python -c "import datasets"`.

### Expected behavior

Script runs.

### Environment info

Python 3.10. Here are the versions installed by `poetry`:

```
•• Installing frozenlist (1.3.3)
  • Installing idna (3.4)
  • Installing multidict (6.0.4)
  • Installing aiosignal (1.3.1)
  • Installing async-timeout (4.0.2)
  • Installing attrs (22.2.0)
  • Installing certifi (2022.12.7)
  • Installing charset-normalizer (3.1.0)
  • Installing six (1.16.0)
  • Installing urllib3 (1.26.14)
  • Installing yarl (1.8.2)
  • Installing aiohttp (3.8.4)
  • Installing dill (0.3.1.1)
  • Installing docopt (0.6.2)
  • Installing filelock (3.9.0)
  • Installing numpy (1.22.4)
  • Installing pyparsing (3.0.9)
  • Installing protobuf (3.19.4)
  • Installing packaging (23.0)
  • Installing python-dateutil (2.8.2)
  • Installing pytz (2022.7.1)
  • Installing pyyaml (6.0)
  • Installing requests (2.28.2)
  • Installing tqdm (4.65.0)
  • Installing typing-extensions (4.5.0)
  • Installing cloudpickle (2.2.1)
  • Installing crcmod (1.7)
  • Installing fastavro (1.7.2)
  • Installing fasteners (0.18)
  • Installing fsspec (2023.3.0)
  • Installing grpcio (1.51.3)
  • Installing hdfs (2.7.0)
  • Installing httplib2 (0.20.4)
  • Installing huggingface-hub (0.12.1)
  • Installing multiprocess (0.70.9)
  • Installing objsize (0.6.1)
  • Installing orjson (3.8.7)
  • Installing pandas (1.5.3)
  • Installing proto-plus (1.22.2)
  • Installing pyarrow (9.0.0)
  • Installing pydot (1.4.2)
  • Installing pymongo (3.13.0)
  • Installing regex (2022.10.31)
  • Installing responses (0.18.0)
  • Installing xxhash (3.2.0)
  • Installing zstandard (0.20.0)
  • Installing apache-beam (2.45.0)
  • Installing datasets (2.10.1)
```


Arrow map type in parquet files unsupported

I'm attaching a minimal reproducible example:
```python
from datasets import load_dataset
import pyarrow as pa
import pyarrow.parquet as pq

table_with_map = pa.Table.from_pydict(
    {"a": [1, 2], "b": [[("a", 2)], [("b", 4)]]},
    schema=pa.schema({"a": pa.int32(), "b": pa.map_(pa.string(), pa.int32())})
)
pq.write_table(table_with_map, "parquet_with_map.parquet")
dset = load_dataset("parquet", data_files="parquet_with_map.parquet", split="train") # error unless streaming=True
```

For a dataset generated with the packaged loaders (CSV, JSON, Parquet), `streaming=True` sets the dataset's features to `None` (unless explicitly provided in `load_dataset`), hence no error will be thrown as long as the features stay "unresolved" (resolving the features with `_resolve_features` will lead to an error).

I've also been wondering about datasets support for Arrow Map datatypes. I had a situation where I had a pandas series of dict[str, float] with hundreds of different possible key values (ie. not bounded), and this got converted to a sequence of structs where every single struct had the entire set of keys.

I worked around it, by explicitly creating a sequence of [str, float], but given that pyarrow has an explicit Map datatype, it would be good to be able to explicitly cast/force this data type combination.

(feel free to ignore) polars will not support this type: https://github.com/pola-rs/polars/issues/3942#issuecomment-1202331210

> Polars will not add the map dtype. It's benefit do not outweigh the extra complexity. Maybe we can investigate conversion of maps to struct. But I will have to explore that.

Looks like they chose to convert every instance with https://github.com/pola-rs/polars/pull/4226

### Describe the bug

When I try to load parquet files that were processed with Spark, I get the following issue:

`ValueError: Arrow type map<string, string ('warc_headers')> does not have a datasets dtype equivalent.`

Strangely, loading the dataset with `streaming=True` solves the issue.

### Steps to reproduce the bug

The dataset is private, but this can be reproduced with any dataset that has Arrow maps.

### Expected behavior

Loading the dataset no matter whether streaming is True or not.

### Environment info

- `datasets` version: 2.10.1
- Platform: Linux-5.15.0-1029-gcp-x86_64-with-glibc2.31
- Python version: 3.10.7
- PyArrow version: 8.0.0
- Pandas version: 1.4.2


add Dataset.to_list

_The documentation is not available anymore as the PR was closed or merged._

Hi, thanks for working on this! `Table.to_pylist` requires PyArrow 7.0+, and our minimal version requirement is 6.0, so we need to bump the version requirement to avoid CI failure. I'll do this in a separate PR.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006857 / 0.011353 (-0.004496) | 0.004711 / 0.011008 (-0.006297) | 0.098332 / 0.038508 (0.059824) | 0.028547 / 0.023109 (0.005438) | 0.307647 / 0.275898 (0.031749) | 0.334891 / 0.323480 (0.011411) | 0.005252 / 0.007986 (-0.002734) | 0.003495 / 0.004328 (-0.000833) | 0.075529 / 0.004250 (0.071279) | 0.042167 / 0.037052 (0.005114) | 0.308509 / 0.258489 (0.050020) | 0.348294 / 0.293841 (0.054453) | 0.032042 / 0.128546 (-0.096504) | 0.011684 / 0.075646 (-0.063962) | 0.321740 / 0.419271 (-0.097531) | 0.057725 / 0.043533 (0.014193) | 0.309431 / 0.255139 (0.054292) | 0.326818 / 0.283200 (0.043618) | 0.093261 / 0.141683 (-0.048422) | 1.475344 / 1.452155 (0.023190) | 1.563952 / 1.492716 (0.071236) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.205056 / 0.018006 (0.187050) | 0.421656 / 0.000490 (0.421166) | 0.004167 / 0.000200 (0.003967) | 0.000075 / 0.000054 (0.000020) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023935 / 0.037411 (-0.013476) | 0.097220 / 0.014526 (0.082695) | 0.104942 / 0.176557 (-0.071615) | 0.170339 / 0.737135 (-0.566796) | 0.107556 / 0.296338 (-0.188782) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.424509 / 0.215209 (0.209300) | 4.223637 / 2.077655 (2.145982) | 2.090700 / 1.504120 (0.586580) | 1.902537 / 1.541195 (0.361343) | 1.981192 / 1.468490 (0.512701) | 0.695272 / 4.584777 (-3.889505) | 3.570169 / 3.745712 (-0.175544) | 1.885007 / 5.269862 (-3.384854) | 1.162828 / 4.565676 (-3.402848) | 0.084956 / 0.424275 (-0.339319) | 0.012818 / 0.007607 (0.005210) | 0.534395 / 0.226044 (0.308351) | 5.354318 / 2.268929 (3.085389) | 2.436875 / 55.444624 (-53.007749) | 2.111365 / 6.876477 (-4.765112) | 2.232874 / 2.142072 (0.090802) | 0.804703 / 4.805227 (-4.000524) | 0.152406 / 6.500664 (-6.348258) | 0.066926 / 0.075469 (-0.008543) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.198621 / 1.841788 (-0.643166) | 13.907491 / 8.074308 (5.833183) | 14.356286 / 10.191392 (4.164894) | 0.140714 / 0.680424 (-0.539710) | 0.016440 / 0.534201 (-0.517761) | 0.380868 / 0.579283 (-0.198415) | 0.396004 / 0.434364 (-0.038360) | 0.448275 / 0.540337 (-0.092062) | 0.537818 / 1.386936 (-0.849118) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006789 / 0.011353 (-0.004564) | 0.004652 / 0.011008 (-0.006356) | 0.076449 / 0.038508 (0.037941) | 0.028389 / 0.023109 (0.005280) | 0.378644 / 0.275898 (0.102746) | 0.423870 / 0.323480 (0.100391) | 0.005824 / 0.007986 (-0.002162) | 0.003398 / 0.004328 (-0.000931) | 0.075575 / 0.004250 (0.071324) | 0.039656 / 0.037052 (0.002604) | 0.370072 / 0.258489 (0.111583) | 0.441812 / 0.293841 (0.147971) | 0.031817 / 0.128546 (-0.096729) | 0.011701 / 0.075646 (-0.063946) | 0.085759 / 0.419271 (-0.333513) | 0.042328 / 0.043533 (-0.001205) | 0.364103 / 0.255139 (0.108964) | 0.413910 / 0.283200 (0.130711) | 0.090871 / 0.141683 (-0.050812) | 1.505749 / 1.452155 (0.053594) | 1.608555 / 1.492716 (0.115839) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.212533 / 0.018006 (0.194527) | 0.404519 / 0.000490 (0.404030) | 0.000373 / 0.000200 (0.000174) | 0.000059 / 0.000054 (0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024849 / 0.037411 (-0.012562) | 0.100769 / 0.014526 (0.086243) | 0.110450 / 0.176557 (-0.066107) | 0.161715 / 0.737135 (-0.575420) | 0.113599 / 0.296338 (-0.182739) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.436780 / 0.215209 (0.221571) | 4.387103 / 2.077655 (2.309448) | 2.081942 / 1.504120 (0.577822) | 1.873661 / 1.541195 (0.332466) | 1.947718 / 1.468490 (0.479228) | 0.696434 / 4.584777 (-3.888343) | 3.405300 / 3.745712 (-0.340412) | 1.897388 / 5.269862 (-3.372474) | 1.169969 / 4.565676 (-3.395707) | 0.083085 / 0.424275 (-0.341190) | 0.012480 / 0.007607 (0.004873) | 0.535635 / 0.226044 (0.309591) | 5.364462 / 2.268929 (3.095533) | 2.531168 / 55.444624 (-52.913457) | 2.184324 / 6.876477 (-4.692153) | 2.228613 / 2.142072 (0.086541) | 0.807127 / 4.805227 (-3.998100) | 0.151971 / 6.500664 (-6.348693) | 0.068430 / 0.075469 (-0.007039) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.306401 / 1.841788 (-0.535387) | 14.479552 / 8.074308 (6.405244) | 14.428398 / 10.191392 (4.237006) | 0.159505 / 0.680424 (-0.520919) | 0.016856 / 0.534201 (-0.517344) | 0.375197 / 0.579283 (-0.204086) | 0.384328 / 0.434364 (-0.050036) | 0.440688 / 0.540337 (-0.099650) | 0.524998 / 1.386936 (-0.861938) |

![](https://cml.dev/watermark.png#50b887b840cf3cab86b0394b41050b579c4b79ba "CML watermark")

close https://github.com/huggingface/datasets/issues/5606

This PR is for adding the `Dataset.to_list` method.
Thank you in advance.


Same problem,
transformers 4.28.1
datasets 2.12.0

leak around 100Mb per 10 seconds when use dataloader_num_werker > 0 in training argumennts for transformer train, possile bug in transformers repo, but still not found solution :(

found an article described a problem, may be helpful for somebody:
https://ppwwyyxx.com/blog/2022/Demystify-RAM-Usage-in-Multiprocess-DataLoader/
I confirm, it`s not memory leak, after some time memory growing has stopped

"After some time" - from your description, it sounds like memory growth can happen for 12 hours+, even days, before it stops? That seems very scary.

### Describe the bug

use datasets streaming mode in trainer ddp mode cause memory leak

### Steps to reproduce the bug

import os
import time
import datetime
import sys
import numpy as np
import random

import torch
from torch.utils.data import Dataset, DataLoader, random_split, RandomSampler, SequentialSampler,DistributedSampler,BatchSampler
torch.manual_seed(42)

from transformers import GPT2LMHeadModel, GPT2Tokenizer, GPT2Config, GPT2Model,DataCollatorForLanguageModeling,AutoModelForCausalLM
from transformers import AdamW, get_linear_schedule_with_warmup

hf_model_path ='./Wenzhong-GPT2-110M'
tokenizer = GPT2Tokenizer.from_pretrained(hf_model_path)
tokenizer.add_special_tokens({'pad_token': '<|pad|>'})

from datasets import load_dataset

gpus=8
max_len = 576
batch_size_node = 17
save_step = 5000
gradient_accumulation = 2
dataloader_num = 4
max_step = 351000*1000//batch_size_node//gradient_accumulation//gpus
#max_step = -1
print("total_step:%d"%(max_step))
import datasets

datasets.version

dataset = load_dataset("text", data_files="./gpt_data_v1/*",split='train',cache_dir='./dataset_cache',streaming=True)

print('load over')
shuffled_dataset = dataset.shuffle(seed=42)
print('shuffle over')

def dataset_tokener(example,max_lenth=max_len):
example['text'] = list(map(lambda x : x.strip()+'<|endoftext|>',example['text'] ))
return tokenizer(example['text'], truncation=True, max_length=max_lenth, padding="longest")

new_new_dataset = shuffled_dataset.map(dataset_tokener, batched=True, remove_columns=["text"])
print('map over')

configuration = GPT2Config.from_pretrained(hf_model_path, output_hidden_states=False)

model = AutoModelForCausalLM.from_pretrained(hf_model_path)

model.resize_token_embeddings(len(tokenizer))

seed_val = 42

random.seed(seed_val)
np.random.seed(seed_val)
torch.manual_seed(seed_val)
torch.cuda.manual_seed_all(seed_val)

from transformers import Trainer,TrainingArguments
import os
print("strat train")
training_args = TrainingArguments(output_dir="./test_trainer",
num_train_epochs=1.0,
report_to="none",
do_train=True,
dataloader_num_workers=dataloader_num,
local_rank=int(os.environ.get('LOCAL_RANK', -1)),
overwrite_output_dir=True,
logging_strategy='steps',
logging_first_step=True,
logging_dir="./logs",
log_on_each_node=False,
per_device_train_batch_size=batch_size_node,
warmup_ratio=0.03,
save_steps=save_step,
save_total_limit=5,
gradient_accumulation_steps=gradient_accumulation,
max_steps=max_step,
disable_tqdm=False,
data_seed=42
)

trainer = Trainer(
model=model,
args=training_args,
train_dataset=new_new_dataset,
eval_dataset=None,
tokenizer=tokenizer,
data_collator=DataCollatorForLanguageModeling(tokenizer,mlm=False),
#compute_metrics=compute_metrics if training_args.do_eval and not is_torch_tpu_available() else None,
#preprocess_logits_for_metrics=preprocess_logits_for_metrics
#if training_args.do_eval and not is_torch_tpu_available()
#else None,
)
trainer.train(resume_from_checkpoint=True)

### Expected behavior

use the train code uppper
my dataset ./gpt_data_v1 have 1000 files, each file size is 120mb
start cmd is : python -m torch.distributed.launch --nproc_per_node=8 my_train.py
here is result:

![image](https://user-images.githubusercontent.com/15223544/223026042-1a81489f-897a-43e4-8339-65a202fd5dc7.png)

here is memory usage monitor in 12 hours

![image](https://user-images.githubusercontent.com/15223544/223027076-14e32e8b-9608-4282-9a80-f15d0277026d.png)

every dataloader work allocate over 24gb cpu memory
according to memory usage monitor in 12 hours,sometime small memory releases, but total memory usage is increase.
i think datasets streaming mode should not used so much memery,so maybe somewhere has memory leak.

### Environment info

pytorch 1.11.0
py 3.8
cuda 11.3
transformers 4.26.1
datasets 2.9.0


`load_from_disk` vs `load_dataset` performance.

Hi! We've recently made some improvements to `save_to_disk`/`list_to_disk` (100x faster in some scenarios), so it would help if you could install `datasets` directly from `main` (`pip install git+https://github.com/huggingface/datasets.git`) and re-run the "benchmark".

Great to hear! I'll give it a try when I've got a moment.

@mariosasko is that fix released to pip in the meantime? Asking cause im facing still the same issue (regarding loading images from local paths):
```
dataset = load_dataset("csv", cache_dir="cache", data_files=["/STORAGE/DATA/mijam/vit/code/list_filtered.csv"], num_proc=16, split="train").cast_column("image", Image())
dataset = dataset.class_encode_column("label")
```
quite fast.

Then I do `save_to_disk()` and some time later:
```
dataset = load_from_disk('/STORAGE/DATA/mijam/accel/saved_arrow_big')
```
really slow. In theory it should be quicked since it only loads arrow files, no conversions and so on.

@mjamroz I assume your CSV file stores image file paths. This means `save_to_disk` needs to embed the image bytes resulting in a much bigger Arrow file (than the initial one). Maybe specifying `num_shards` to make the Arrow files smaller can help (large Arrow files on some systems take a long time to load).

### Describe the bug

I have downloaded `openwebtext` (~12GB) and filtered out a small amount of junk (it's still huge). Now, I would like to use this filtered version for future work. It seems I have two choices:
1. Use `load_dataset` each time, relying on the cache mechanism, and re-run my filtering.
2. `save_to_disk` and then use `load_from_disk` to load the filtered version.

The performance of these two approaches is wildly different:
* Using `load_dataset` takes about 20 seconds to load the dataset, and a few seconds to re-filter (thanks to the brilliant filter/map caching)
* Using `load_from_disk` takes 14 minutes! And the second time I tried, the session just crashed (on a machine with 32GB of RAM)

I don't know if you'd call this a bug, but it seems like there shouldn't need to be two methods to load from disk, or that they should  not take such wildly different amounts of time, or that one should not crash. Or maybe that the docs could offer some guidance about when to pick which method and why two methods exist, or just how do most people do it?

Something I couldn't work out from reading the docs was this: can I modify a dataset from the hub, save it (locally) and use `load_dataset` to load it? This [post seemed to suggest that the answer is no](https://discuss.huggingface.co/t/save-and-load-datasets/9260).

### Steps to reproduce the bug

See above

### Expected behavior

Load times should be about the same.

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.31
- Python version: 3.10.8
- PyArrow version: 11.0.0
- Pandas version: 1.5.3


audiofolder only creates dataset of 13 rows (files) when the data folder it's reading from has 20,000 mp3 files.

Hi!

> naming convention of mp3 files

Yes, this could be the problem. MP3 files should end with  `.mp3`/`.MP3` to be recognized as audio files.

If the file names are not the culprit, can you paste the audio folder's directory structure to help us reproduce the error (e.g., by running the `tree "x"` command)?

Hi! I'm sorry, I don't want to reveal my entire dataset, but here's a snippet (all of the mp3 files below are some of the ones not being recognized by audiofolder. Also, for another dataset, audiofolder loaded zero mp3 files because "train" was in the name of one of the mp3 files.
my_dataset
├── data
│   ├── VHA_Innovation_Stories_-_Day_2-123.mp3
│   ├── VHA_Innovation_Stories_-_Day_2-124.mp3
│   ├── ASSOCIATION_OF_GENERAL_PRACTITIONERS_OF_JAMAICA_NEPHROLOGY_CONFERENCE_-_JULY_3,_2022-93.mp3
│   ├── ASSOCIATION_OF_GENERAL_PRACTITIONERS_OF_JAMAICA_NEPHROLOGY_CONFERENCE_-_JULY_3,_2022-94.mp3
│   ├── ASSOCIATION_OF_GENERAL_PRACTITIONERS_OF_JAMAICA_NEPHROLOGY_CONFERENCE_-_JULY_3,_2022-95.mp3
│   ├── Your_Impact\357\274\232_Neurosurgery_equipment-5.mp3
│   └── Your_Impact\357\274\232_Neurosurgery_equipment-6.mp3
└── metadata.csv

Here's a few of the 13 files recognized by the dataset:
British_Heart_Foundation_-_Your_guide_to_a_Coronary_Angiogram,_a_test_for_heart_disease-1.mp3
British_Heart_Foundation_-_Your_guide_to_a_Coronary_Angiogram,_a_test_for_heart_disease-2.mp3
British_Heart_Foundation_-_Your_guide_to_a_Coronary_Angiogram,_a_test_for_heart_disease-3.mp3
IVP_⧸_IVU_test_Procedure_for_Kidneys_intravenous_pyelogram_-_medical_radiology_X-ray_ivp-1.mp3
IVP_⧸_IVU_test_Procedure_for_Kidneys_intravenous_pyelogram_-_medical_radiology_X-ray_ivp-2.mp3

### Describe the bug

x = load_dataset("audiofolder", data_dir="x")
When running this, x is a dataset of 13 rows (files) when it should be 20,000 rows (files) as the data_dir "x" has 20,000 mp3 files. Does anyone know what could possibly cause this (naming convention of mp3 files, etc.)

### Steps to reproduce the bug

x = load_dataset("audiofolder", data_dir="x")

### Expected behavior

x = load_dataset("audiofolder", data_dir="x") should create a dataset of 20,000 rows (files).

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-3.10.0-1160.80.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.16
- PyArrow version: 11.0.0
- Pandas version: 1.5.3


Fix outdated `verification_mode` values

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006142 / 0.011353 (-0.005211) | 0.004506 / 0.011008 (-0.006502) | 0.100224 / 0.038508 (0.061715) | 0.026988 / 0.023109 (0.003879) | 0.301625 / 0.275898 (0.025727) | 0.346337 / 0.323480 (0.022857) | 0.004642 / 0.007986 (-0.003343) | 0.003481 / 0.004328 (-0.000847) | 0.075847 / 0.004250 (0.071597) | 0.036959 / 0.037052 (-0.000094) | 0.302697 / 0.258489 (0.044208) | 0.351917 / 0.293841 (0.058076) | 0.030719 / 0.128546 (-0.097828) | 0.011591 / 0.075646 (-0.064056) | 0.319709 / 0.419271 (-0.099563) | 0.042000 / 0.043533 (-0.001532) | 0.306854 / 0.255139 (0.051715) | 0.326903 / 0.283200 (0.043703) | 0.082711 / 0.141683 (-0.058972) | 1.486616 / 1.452155 (0.034461) | 1.603229 / 1.492716 (0.110513) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.198990 / 0.018006 (0.180983) | 0.427733 / 0.000490 (0.427243) | 0.003612 / 0.000200 (0.003412) | 0.000071 / 0.000054 (0.000016) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022932 / 0.037411 (-0.014480) | 0.096969 / 0.014526 (0.082443) | 0.105749 / 0.176557 (-0.070807) | 0.166101 / 0.737135 (-0.571034) | 0.108646 / 0.296338 (-0.187692) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.428174 / 0.215209 (0.212965) | 4.271452 / 2.077655 (2.193797) | 1.907588 / 1.504120 (0.403468) | 1.680870 / 1.541195 (0.139675) | 1.761336 / 1.468490 (0.292846) | 0.700380 / 4.584777 (-3.884396) | 3.415168 / 3.745712 (-0.330544) | 1.886122 / 5.269862 (-3.383740) | 1.276814 / 4.565676 (-3.288863) | 0.083429 / 0.424275 (-0.340846) | 0.012988 / 0.007607 (0.005381) | 0.518821 / 0.226044 (0.292776) | 5.188284 / 2.268929 (2.919356) | 2.433084 / 55.444624 (-53.011540) | 1.988034 / 6.876477 (-4.888443) | 2.100275 / 2.142072 (-0.041797) | 0.808252 / 4.805227 (-3.996976) | 0.158102 / 6.500664 (-6.342562) | 0.067686 / 0.075469 (-0.007783) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.204171 / 1.841788 (-0.637616) | 13.548756 / 8.074308 (5.474448) | 14.339805 / 10.191392 (4.148413) | 0.142853 / 0.680424 (-0.537571) | 0.016529 / 0.534201 (-0.517672) | 0.383800 / 0.579283 (-0.195483) | 0.380362 / 0.434364 (-0.054002) | 0.437716 / 0.540337 (-0.102621) | 0.524306 / 1.386936 (-0.862630) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006730 / 0.011353 (-0.004623) | 0.004652 / 0.011008 (-0.006356) | 0.077476 / 0.038508 (0.038968) | 0.027584 / 0.023109 (0.004475) | 0.340907 / 0.275898 (0.065009) | 0.377950 / 0.323480 (0.054470) | 0.005946 / 0.007986 (-0.002040) | 0.003548 / 0.004328 (-0.000780) | 0.076270 / 0.004250 (0.072019) | 0.037483 / 0.037052 (0.000431) | 0.346390 / 0.258489 (0.087901) | 0.384739 / 0.293841 (0.090898) | 0.031744 / 0.128546 (-0.096802) | 0.011598 / 0.075646 (-0.064049) | 0.085651 / 0.419271 (-0.333620) | 0.047308 / 0.043533 (0.003775) | 0.344704 / 0.255139 (0.089565) | 0.363410 / 0.283200 (0.080211) | 0.095009 / 0.141683 (-0.046674) | 1.478307 / 1.452155 (0.026152) | 1.576808 / 1.492716 (0.084092) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.197545 / 0.018006 (0.179539) | 0.431984 / 0.000490 (0.431494) | 0.001529 / 0.000200 (0.001329) | 0.000079 / 0.000054 (0.000025) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025452 / 0.037411 (-0.011959) | 0.100176 / 0.014526 (0.085651) | 0.108222 / 0.176557 (-0.068335) | 0.160556 / 0.737135 (-0.576580) | 0.112748 / 0.296338 (-0.183591) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.436326 / 0.215209 (0.221117) | 4.378443 / 2.077655 (2.300788) | 2.056001 / 1.504120 (0.551881) | 1.853406 / 1.541195 (0.312211) | 1.931645 / 1.468490 (0.463155) | 0.698340 / 4.584777 (-3.886437) | 3.368961 / 3.745712 (-0.376751) | 2.583622 / 5.269862 (-2.686239) | 1.501274 / 4.565676 (-3.064402) | 0.083034 / 0.424275 (-0.341241) | 0.012725 / 0.007607 (0.005117) | 0.539991 / 0.226044 (0.313947) | 5.418413 / 2.268929 (3.149485) | 2.517205 / 55.444624 (-52.927420) | 2.179332 / 6.876477 (-4.697144) | 2.215376 / 2.142072 (0.073304) | 0.806133 / 4.805227 (-3.999094) | 0.151499 / 6.500664 (-6.349165) | 0.067270 / 0.075469 (-0.008199) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.308324 / 1.841788 (-0.533464) | 14.357361 / 8.074308 (6.283053) | 14.684768 / 10.191392 (4.493376) | 0.139575 / 0.680424 (-0.540849) | 0.016409 / 0.534201 (-0.517792) | 0.374087 / 0.579283 (-0.205196) | 0.390628 / 0.434364 (-0.043735) | 0.443102 / 0.540337 (-0.097235) | 0.536089 / 1.386936 (-0.850847) |

![](https://cml.dev/watermark.png#778d4e1c13ece980e706f8c7cb06e8473fd61315 "CML watermark")

~I think it makes sense not to save `dataset_info.json` file to a dataset cache directory when loading dataset with `verification_mode="no_checks"` because otherwise when next time the dataset is loaded **without** `verification_mode="no_checks"`, it will be loaded successfully, despite some values in info might not correspond to the ones in the repo which was the reason for using `verification_mode="no_checks"` first.~

Updated values of `verification_mode` to the current ones in some places ("none" -> "no_checks", "all" -> "all_checks")


Add `Dataset.to_list` to the API

enhancement

New feature or request

good first issue

Good for newcomers

Hello, I have an interest in this issue.
Is the `Dataset.to_dict` you are describing correct in the code here?

https://github.com/huggingface/datasets/blob/35b789e8f6826b6b5a6b48fcc2416c890a1f326a/src/datasets/arrow_dataset.py#L4633-L4667

Yes, this is where `Dataset.to_dict` is defined.

Since there is `Dataset.from_list` in the API, we should also add `Dataset.to_list` to be consistent.

Regarding the implementation, we can re-use `Dataset.to_dict`'s code and replace the `to_pydict` calls with `to_pylist`.


Update README logo

Are you sure it's safe to remove? https://github.com/huggingface/datasets/pull/3866

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009520 / 0.011353 (-0.001833) | 0.005319 / 0.011008 (-0.005690) | 0.099372 / 0.038508 (0.060863) | 0.036173 / 0.023109 (0.013064) | 0.295752 / 0.275898 (0.019853) | 0.362882 / 0.323480 (0.039402) | 0.008442 / 0.007986 (0.000456) | 0.004225 / 0.004328 (-0.000103) | 0.076645 / 0.004250 (0.072394) | 0.044198 / 0.037052 (0.007146) | 0.311948 / 0.258489 (0.053459) | 0.342963 / 0.293841 (0.049122) | 0.038613 / 0.128546 (-0.089933) | 0.012127 / 0.075646 (-0.063519) | 0.334427 / 0.419271 (-0.084844) | 0.048309 / 0.043533 (0.004776) | 0.297046 / 0.255139 (0.041907) | 0.314562 / 0.283200 (0.031363) | 0.105797 / 0.141683 (-0.035886) | 1.460967 / 1.452155 (0.008812) | 1.500907 / 1.492716 (0.008190) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.216185 / 0.018006 (0.198179) | 0.438924 / 0.000490 (0.438435) | 0.001210 / 0.000200 (0.001011) | 0.000081 / 0.000054 (0.000027) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026193 / 0.037411 (-0.011219) | 0.105888 / 0.014526 (0.091363) | 0.115812 / 0.176557 (-0.060744) | 0.158748 / 0.737135 (-0.578387) | 0.121514 / 0.296338 (-0.174824) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.399837 / 0.215209 (0.184628) | 3.996992 / 2.077655 (1.919338) | 1.784964 / 1.504120 (0.280844) | 1.591078 / 1.541195 (0.049883) | 1.666424 / 1.468490 (0.197934) | 0.711450 / 4.584777 (-3.873327) | 3.787814 / 3.745712 (0.042102) | 2.056776 / 5.269862 (-3.213085) | 1.332163 / 4.565676 (-3.233514) | 0.085755 / 0.424275 (-0.338520) | 0.012033 / 0.007607 (0.004426) | 0.511500 / 0.226044 (0.285455) | 5.098999 / 2.268929 (2.830071) | 2.288261 / 55.444624 (-53.156364) | 1.947483 / 6.876477 (-4.928994) | 1.987838 / 2.142072 (-0.154234) | 0.852241 / 4.805227 (-3.952986) | 0.164781 / 6.500664 (-6.335883) | 0.061825 / 0.075469 (-0.013644) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.202253 / 1.841788 (-0.639534) | 14.632608 / 8.074308 (6.558300) | 13.331320 / 10.191392 (3.139928) | 0.157944 / 0.680424 (-0.522480) | 0.029284 / 0.534201 (-0.504917) | 0.446636 / 0.579283 (-0.132647) | 0.437009 / 0.434364 (0.002645) | 0.521883 / 0.540337 (-0.018455) | 0.606687 / 1.386936 (-0.780249) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007528 / 0.011353 (-0.003825) | 0.005274 / 0.011008 (-0.005734) | 0.073524 / 0.038508 (0.035016) | 0.033893 / 0.023109 (0.010784) | 0.335432 / 0.275898 (0.059534) | 0.379981 / 0.323480 (0.056501) | 0.005954 / 0.007986 (-0.002031) | 0.004126 / 0.004328 (-0.000203) | 0.072891 / 0.004250 (0.068641) | 0.046517 / 0.037052 (0.009465) | 0.337241 / 0.258489 (0.078752) | 0.385562 / 0.293841 (0.091721) | 0.036410 / 0.128546 (-0.092136) | 0.012246 / 0.075646 (-0.063401) | 0.085974 / 0.419271 (-0.333298) | 0.049665 / 0.043533 (0.006133) | 0.330919 / 0.255139 (0.075780) | 0.352041 / 0.283200 (0.068841) | 0.103751 / 0.141683 (-0.037931) | 1.468851 / 1.452155 (0.016696) | 1.565380 / 1.492716 (0.072663) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.260431 / 0.018006 (0.242425) | 0.444554 / 0.000490 (0.444064) | 0.016055 / 0.000200 (0.015855) | 0.000283 / 0.000054 (0.000228) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029130 / 0.037411 (-0.008281) | 0.112002 / 0.014526 (0.097476) | 0.120769 / 0.176557 (-0.055788) | 0.169345 / 0.737135 (-0.567790) | 0.129609 / 0.296338 (-0.166730) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.432211 / 0.215209 (0.217002) | 4.293008 / 2.077655 (2.215353) | 2.071291 / 1.504120 (0.567171) | 1.859322 / 1.541195 (0.318127) | 1.971434 / 1.468490 (0.502943) | 0.704042 / 4.584777 (-3.880735) | 3.791696 / 3.745712 (0.045983) | 3.142632 / 5.269862 (-2.127230) | 1.735151 / 4.565676 (-2.830525) | 0.086203 / 0.424275 (-0.338072) | 0.012542 / 0.007607 (0.004935) | 0.534870 / 0.226044 (0.308826) | 5.326042 / 2.268929 (3.057113) | 2.547960 / 55.444624 (-52.896664) | 2.212730 / 6.876477 (-4.663747) | 2.296177 / 2.142072 (0.154105) | 0.840311 / 4.805227 (-3.964917) | 0.168353 / 6.500664 (-6.332311) | 0.065949 / 0.075469 (-0.009520) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.255589 / 1.841788 (-0.586199) | 14.947344 / 8.074308 (6.873036) | 13.253721 / 10.191392 (3.062329) | 0.162349 / 0.680424 (-0.518075) | 0.017579 / 0.534201 (-0.516622) | 0.420758 / 0.579283 (-0.158525) | 0.430030 / 0.434364 (-0.004334) | 0.524669 / 0.540337 (-0.015669) | 0.623920 / 1.386936 (-0.763016) |

![](https://cml.dev/watermark.png#35b789e8f6826b6b5a6b48fcc2416c890a1f326a "CML watermark")


Problems with downloading The Pile

Hi!

You can specify `download_config=DownloadConfig(resume_download=True))` in `load_dataset` to resume the download when re-running the code after the timeout error:
```python
from datasets import load_dataset, DownloadConfig
dataset = load_dataset('the_pile', split='train', cache_dir='F:\datasets', download_config=DownloadConfig(resume_download=True))
```

@mariosasko , I used your suggestion but its not saving anything , just stops and runs from the same point .
below is the script to download and save on disk .

```
from datasets import load_dataset, DownloadConfig

#load the Pile dataset from Hugging Face Datasets
#dataset = load_dataset('the_pile')
dataset = load_dataset('the_pile', split='train', cache_dir='datasets', download_config=DownloadConfig(resume_download=True))

# save each file in the dataset to disk
for i, example in enumerate(dataset['train']):
    filename = f'pile_file_{i}.json'
    with open(filename, 'w') as f:
        f.write(str(example))

print("Finished saving Pile dataset files to disk.")
```

@mariosasko , it shows nothing in dataset folder

```
 du -sh /mnt/nlp/hugging_face/*
20K     /mnt/nlp/hugging_face/datasets
4.0K    /mnt/nlp/hugging_face/download_pile.py
```

@mariosasko

```
root@d20f0ab8f4f8:/mnt/hugging_face# python3 download_pile.py
No config specified, defaulting to: the_pile/all
Downloading and preparing dataset the_pile/all to /mnt/hugging_face/datasets/the_pile/all/0.0.0/6fadc480ecb32470826cbf5900a9558b791ce55d5e9a0fdc8ad653e7b64bb349...
Downloading data files:   0%|                                                                                | 0/3 [00:00<?, ?it/s]

Downloading data:  70%|████████████████████████████████████████████████████████████████████▊                             | 10.7G/15.2G [12:09<11:53, 6.36MB/s]
Downloading data: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 15.2G/15.2G [22:15<00:00, 7.25MB/s]
Downloading data: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 15.2G/15.2G [46:17<00:00, 5.48MB/s]
Downloading data:  40%|██████████████████████████████████████▏                                                         | 6.07G/15.3G [50:49<1:17:02, 1.99MB/s]
Traceback (most recent call last):██████████████████████████▊                                                         | 6.07G/15.3G [50:49<25:35:23, 99.9kB/s]
  File "/usr/local/lib/python3.8/dist-packages/urllib3/response.py", line 444, in _error_catcher
    yield
  File "/usr/local/lib/python3.8/dist-packages/urllib3/response.py", line 567, in read
    data = self._fp_read(amt) if not fp_closed else b""
  File "/usr/local/lib/python3.8/dist-packages/urllib3/response.py", line 525, in _fp_read
    data = self._fp.read(chunk_amt)
  File "/usr/lib/python3.8/http/client.py", line 459, in read
    n = self.readinto(b)
  File "/usr/lib/python3.8/http/client.py", line 503, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 816, in generate
    yield from self.raw.stream(chunk_size, decode_content=True)
  File "/usr/local/lib/python3.8/dist-packages/urllib3/response.py", line 628, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/usr/local/lib/python3.8/dist-packages/urllib3/response.py", line 593, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.8/dist-packages/urllib3/response.py", line 461, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "download_pile.py", line 6, in <module>
    dataset = load_dataset('the_pile', split='train', cache_dir='datasets', download_config=DownloadConfig(resume_download=True))
  File "/usr/local/lib/python3.8/dist-packages/datasets/load.py", line 1782, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.8/dist-packages/datasets/builder.py", line 872, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.8/dist-packages/datasets/builder.py", line 1649, in _download_and_prepare
    super()._download_and_prepare(
  File "/usr/local/lib/python3.8/dist-packages/datasets/builder.py", line 945, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/root/.cache/huggingface/modules/datasets_modules/datasets/the_pile/6fadc480ecb32470826cbf5900a9558b791ce55d5e9a0fdc8ad653e7b64bb349/the_pile.py", line 192, in _split_generators
    data_dir = dl_manager.download(_DATA_URLS[self.config.name])
  File "/usr/local/lib/python3.8/dist-packages/datasets/download/download_manager.py", line 427, in download
    downloaded_path_or_paths = map_nested(
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/py_utils.py", line 443, in map_nested
    mapped = [
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/py_utils.py", line 444, in <listcomp>
    _single_map_nested((function, obj, types, None, True, None))
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/py_utils.py", line 363, in _single_map_nested
    mapped = [_single_map_nested((function, v, types, None, True, None)) for v in pbar]
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/py_utils.py", line 363, in <listcomp>
    mapped = [_single_map_nested((function, v, types, None, True, None)) for v in pbar]
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/py_utils.py", line 346, in _single_map_nested
    return function(data_struct)
  File "/usr/local/lib/python3.8/dist-packages/datasets/download/download_manager.py", line 453, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/file_utils.py", line 182, in cached_path
    output_path = get_from_cache(
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/file_utils.py", line 575, in get_from_cache
    http_get(
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/file_utils.py", line 379, in http_get
    for chunk in response.iter_content(chunk_size=1024):
  File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 818, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
```

Users with slow internet speed are doomed (4MB/s). The dataset downloads fine at minimum speed 10MB/s.

Also, when the train splits were generated and then I removed the downloads folder to save up disk space, it started redownloading the whole dataset. Is there any way to use the already generated splits instead?

@sentialx @mariosasko , anytime on my above script , am I downloading and saving dataset correctly . Please suggest :)

@sentialx probably worth noting that `resume_download=True` doesn't directly save the dataset to disk, but instead just helps in resuming the dataset resume on interruption as @mariosasko mentions. resolving resumptions after a crash is [an open issue](https://github.com/huggingface/datasets/issues/5380) at the moment.

### Describe the bug

The downloads in the screenshot seem to be interrupted after some time and the last download throws a "Read timed out" error.

![image](https://user-images.githubusercontent.com/11065386/222687870-ec5fcb65-84e8-467d-9593-4ad7bdac4d50.png)

Here are the downloaded files:
![image](https://user-images.githubusercontent.com/11065386/222688200-454c2288-49e5-4682-96e6-1eb69aca0852.png)

They should be all 14GB like here (https://the-eye.eu/public/AI/pile/train/).

Alternatively, can I somehow download the files by myself and use the datasets preparing script?

### Steps to reproduce the bug

dataset = load_dataset('the_pile', split='train', cache_dir='F:\datasets')

### Expected behavior

The files should be downloaded correctly.

### Environment info

- `datasets` version: 2.10.1
- Platform: Windows-10-10.0.22623-SP0
- Python version: 3.10.5
- PyArrow version: 9.0.0
- Pandas version: 1.4.2