Temporarily pin fsspec < 2023.9.0

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006494 / 0.011353 (-0.004859) | 0.003896 / 0.011008 (-0.007112) | 0.083940 / 0.038508 (0.045432) | 0.068335 / 0.023109 (0.045225) | 0.365770 / 0.275898 (0.089872) | 0.403702 / 0.323480 (0.080222) | 0.004005 / 0.007986 (-0.003981) | 0.003276 / 0.004328 (-0.001052) | 0.064877 / 0.004250 (0.060626) | 0.053524 / 0.037052 (0.016472) | 0.372951 / 0.258489 (0.114462) | 0.420935 / 0.293841 (0.127094) | 0.030656 / 0.128546 (-0.097890) | 0.009048 / 0.075646 (-0.066599) | 0.287607 / 0.419271 (-0.131665) | 0.052042 / 0.043533 (0.008509) | 0.371446 / 0.255139 (0.116307) | 0.408781 / 0.283200 (0.125581) | 0.024228 / 0.141683 (-0.117455) | 1.483325 / 1.452155 (0.031170) | 1.544321 / 1.492716 (0.051605) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.212355 / 0.018006 (0.194349) | 0.463298 / 0.000490 (0.462808) | 0.005170 / 0.000200 (0.004970) | 0.000087 / 0.000054 (0.000032) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027824 / 0.037411 (-0.009587) | 0.081880 / 0.014526 (0.067354) | 0.094886 / 0.176557 (-0.081670) | 0.150024 / 0.737135 (-0.587111) | 0.096643 / 0.296338 (-0.199696) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.388521 / 0.215209 (0.173312) | 3.877251 / 2.077655 (1.799596) | 1.931085 / 1.504120 (0.426965) | 1.766525 / 1.541195 (0.225330) | 1.814802 / 1.468490 (0.346312) | 0.489478 / 4.584777 (-4.095299) | 3.570973 / 3.745712 (-0.174739) | 3.190211 / 5.269862 (-2.079651) | 2.015670 / 4.565676 (-2.550006) | 0.057773 / 0.424275 (-0.366503) | 0.007611 / 0.007607 (0.000004) | 0.462162 / 0.226044 (0.236117) | 4.616173 / 2.268929 (2.347244) | 2.360531 / 55.444624 (-53.084094) | 2.053680 / 6.876477 (-4.822797) | 2.228057 / 2.142072 (0.085985) | 0.584921 / 4.805227 (-4.220306) | 0.132470 / 6.500664 (-6.368194) | 0.060482 / 0.075469 (-0.014987) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.263393 / 1.841788 (-0.578394) | 19.416841 / 8.074308 (11.342532) | 14.049032 / 10.191392 (3.857640) | 0.162822 / 0.680424 (-0.517602) | 0.018189 / 0.534201 (-0.516012) | 0.391142 / 0.579283 (-0.188141) | 0.409367 / 0.434364 (-0.024997) | 0.454589 / 0.540337 (-0.085748) | 0.632946 / 1.386936 (-0.753990) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006568 / 0.011353 (-0.004785) | 0.004026 / 0.011008 (-0.006982) | 0.064522 / 0.038508 (0.026014) | 0.071738 / 0.023109 (0.048629) | 0.395771 / 0.275898 (0.119873) | 0.421553 / 0.323480 (0.098073) | 0.005291 / 0.007986 (-0.002694) | 0.003266 / 0.004328 (-0.001063) | 0.064464 / 0.004250 (0.060214) | 0.054622 / 0.037052 (0.017569) | 0.395010 / 0.258489 (0.136521) | 0.433895 / 0.293841 (0.140054) | 0.031670 / 0.128546 (-0.096876) | 0.008536 / 0.075646 (-0.067111) | 0.071059 / 0.419271 (-0.348212) | 0.047117 / 0.043533 (0.003584) | 0.391210 / 0.255139 (0.136071) | 0.411685 / 0.283200 (0.128486) | 0.022779 / 0.141683 (-0.118904) | 1.479900 / 1.452155 (0.027746) | 1.551853 / 1.492716 (0.059137) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.332814 / 0.018006 (0.314807) | 0.460654 / 0.000490 (0.460164) | 0.062257 / 0.000200 (0.062057) | 0.000374 / 0.000054 (0.000319) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031801 / 0.037411 (-0.005610) | 0.090730 / 0.014526 (0.076204) | 0.102955 / 0.176557 (-0.073602) | 0.155928 / 0.737135 (-0.581207) | 0.103028 / 0.296338 (-0.193310) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.434460 / 0.215209 (0.219251) | 4.331550 / 2.077655 (2.253895) | 2.335990 / 1.504120 (0.831870) | 2.183985 / 1.541195 (0.642790) | 2.233086 / 1.468490 (0.764595) | 0.488484 / 4.584777 (-4.096293) | 3.603856 / 3.745712 (-0.141856) | 3.229833 / 5.269862 (-2.040029) | 2.007366 / 4.565676 (-2.558311) | 0.057658 / 0.424275 (-0.366617) | 0.007339 / 0.007607 (-0.000268) | 0.512812 / 0.226044 (0.286768) | 5.141497 / 2.268929 (2.872569) | 2.847383 / 55.444624 (-52.597241) | 2.467010 / 6.876477 (-4.409467) | 2.644995 / 2.142072 (0.502923) | 0.581385 / 4.805227 (-4.223842) | 0.130755 / 6.500664 (-6.369909) | 0.058834 / 0.075469 (-0.016635) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.350162 / 1.841788 (-0.491626) | 19.768412 / 8.074308 (11.694104) | 15.079196 / 10.191392 (4.887804) | 0.167083 / 0.680424 (-0.513341) | 0.020372 / 0.534201 (-0.513829) | 0.402685 / 0.579283 (-0.176598) | 0.408338 / 0.434364 (-0.026026) | 0.476788 / 0.540337 (-0.063550) | 0.654765 / 1.386936 (-0.732171) |

![](https://cml.dev/watermark.png#ff803c7e9f256c5a137c25c090e18d844f9fc6e4 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008047 / 0.011353 (-0.003305) | 0.004662 / 0.011008 (-0.006346) | 0.102487 / 0.038508 (0.063978) | 0.096832 / 0.023109 (0.073723) | 0.375298 / 0.275898 (0.099400) | 0.420604 / 0.323480 (0.097124) | 0.004655 / 0.007986 (-0.003330) | 0.005699 / 0.004328 (0.001370) | 0.077681 / 0.004250 (0.073430) | 0.065987 / 0.037052 (0.028935) | 0.393146 / 0.258489 (0.134657) | 0.436324 / 0.293841 (0.142483) | 0.036168 / 0.128546 (-0.092378) | 0.010398 / 0.075646 (-0.065248) | 0.347579 / 0.419271 (-0.071693) | 0.061723 / 0.043533 (0.018190) | 0.377439 / 0.255139 (0.122300) | 0.416666 / 0.283200 (0.133467) | 0.031874 / 0.141683 (-0.109809) | 1.818885 / 1.452155 (0.366730) | 1.904749 / 1.492716 (0.412032) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.240497 / 0.018006 (0.222491) | 0.507907 / 0.000490 (0.507417) | 0.004574 / 0.000200 (0.004374) | 0.000098 / 0.000054 (0.000044) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033504 / 0.037411 (-0.003907) | 0.102919 / 0.014526 (0.088393) | 0.113014 / 0.176557 (-0.063543) | 0.181111 / 0.737135 (-0.556024) | 0.115047 / 0.296338 (-0.181291) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.453640 / 0.215209 (0.238431) | 4.514604 / 2.077655 (2.436949) | 2.219758 / 1.504120 (0.715638) | 2.004735 / 1.541195 (0.463541) | 2.112817 / 1.468490 (0.644327) | 0.579534 / 4.584777 (-4.005243) | 4.095994 / 3.745712 (0.350282) | 3.887204 / 5.269862 (-1.382658) | 2.461755 / 4.565676 (-2.103921) | 0.068930 / 0.424275 (-0.355345) | 0.009102 / 0.007607 (0.001495) | 0.540031 / 0.226044 (0.313987) | 5.394324 / 2.268929 (3.125396) | 2.738906 / 55.444624 (-52.705719) | 2.332041 / 6.876477 (-4.544436) | 2.600764 / 2.142072 (0.458692) | 0.697859 / 4.805227 (-4.107368) | 0.159247 / 6.500664 (-6.341417) | 0.073339 / 0.075469 (-0.002130) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.561082 / 1.841788 (-0.280706) | 23.581031 / 8.074308 (15.506723) | 17.011085 / 10.191392 (6.819693) | 0.196115 / 0.680424 (-0.484308) | 0.022050 / 0.534201 (-0.512151) | 0.470865 / 0.579283 (-0.108418) | 0.480539 / 0.434364 (0.046175) | 0.546458 / 0.540337 (0.006120) | 0.744353 / 1.386936 (-0.642583) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007884 / 0.011353 (-0.003468) | 0.004723 / 0.011008 (-0.006286) | 0.076431 / 0.038508 (0.037923) | 0.087016 / 0.023109 (0.063907) | 0.501880 / 0.275898 (0.225982) | 0.546286 / 0.323480 (0.222806) | 0.006224 / 0.007986 (-0.001762) | 0.003858 / 0.004328 (-0.000471) | 0.076485 / 0.004250 (0.072234) | 0.066758 / 0.037052 (0.029706) | 0.510090 / 0.258489 (0.251601) | 0.553935 / 0.293841 (0.260094) | 0.037785 / 0.128546 (-0.090761) | 0.009946 / 0.075646 (-0.065700) | 0.084001 / 0.419271 (-0.335270) | 0.056732 / 0.043533 (0.013199) | 0.490724 / 0.255139 (0.235585) | 0.528367 / 0.283200 (0.245168) | 0.026082 / 0.141683 (-0.115601) | 1.769200 / 1.452155 (0.317045) | 1.847559 / 1.492716 (0.354843) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.306752 / 0.018006 (0.288745) | 0.481215 / 0.000490 (0.480725) | 0.048231 / 0.000200 (0.048031) | 0.000249 / 0.000054 (0.000194) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.039517 / 0.037411 (0.002106) | 0.112884 / 0.014526 (0.098359) | 0.123858 / 0.176557 (-0.052698) | 0.188260 / 0.737135 (-0.548875) | 0.125819 / 0.296338 (-0.170520) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.515260 / 0.215209 (0.300051) | 5.125038 / 2.077655 (3.047383) | 2.785122 / 1.504120 (1.281003) | 2.590753 / 1.541195 (1.049558) | 2.682084 / 1.468490 (1.213594) | 0.581162 / 4.584777 (-4.003615) | 4.241776 / 3.745712 (0.496063) | 3.860979 / 5.269862 (-1.408883) | 2.434203 / 4.565676 (-2.131473) | 0.068580 / 0.424275 (-0.355695) | 0.008700 / 0.007607 (0.001093) | 0.604712 / 0.226044 (0.378667) | 6.044240 / 2.268929 (3.775311) | 3.379734 / 55.444624 (-52.064890) | 2.968906 / 6.876477 (-3.907571) | 3.195775 / 2.142072 (1.053703) | 0.702431 / 4.805227 (-4.102796) | 0.158752 / 6.500664 (-6.341912) | 0.072795 / 0.075469 (-0.002674) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.616354 / 1.841788 (-0.225434) | 24.258731 / 8.074308 (16.184423) | 17.505483 / 10.191392 (7.314091) | 0.173445 / 0.680424 (-0.506979) | 0.023215 / 0.534201 (-0.510986) | 0.472975 / 0.579283 (-0.106308) | 0.478425 / 0.434364 (0.044061) | 0.566950 / 0.540337 (0.026612) | 0.767648 / 1.386936 (-0.619288) |

![](https://cml.dev/watermark.png#a1d520a5226268f2c6f0303de3e8bfd72198b082 "CML watermark")

Temporarily pin fsspec < 2023.9.0 until permanent solution is found.

Hot fix #6209.


CI is broken with AssertionError: 3 failed, 12 errors

bug

Something isn't working

Our CI is broken: 3 failed, 12 errors

See: https://github.com/huggingface/datasets/actions/runs/6069947111/job/16465138041
```
=========================== short test summary info ============================
FAILED tests/test_load.py::ModuleFactoryTest::test_LocalDatasetModuleFactoryWithoutScript_with_data_dir - AssertionError: assert ({NamedSplit('train'): ['/tmp/pytest-of-runner/pytest-0/popen-gw1/test_LocalDatasetModuleFactory2/data_dir2/subdir1/train.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_LocalDatasetModuleFactory2/data_dir2/subdir1/train.txt'], NamedSplit('test'): ['/tmp/pytest-of-runner/pytest-0/popen-gw1/test_LocalDatasetModuleFactory2/data_dir2/subdir1/test.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_LocalDatasetModuleFactory2/data_dir2/subdir1/test.txt']} is not None and 2 == 1)
 +  where 2 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw1/test_LocalDatasetModuleFactory2/data_dir2/subdir1/train.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_LocalDatasetModuleFactory2/data_dir2/subdir1/train.txt'])
FAILED tests/test_load.py::test_load_dataset_arrow[False] - AssertionError: assert 20 == 10
 +  where 20 = Dataset({\n    features: ['col_1'],\n    num_rows: 20\n}).num_rows
FAILED tests/test_load.py::test_load_dataset_arrow[True] - assert 20 == 10
ERROR tests/packaged_modules/test_audiofolder.py::test_data_files_with_metadata_and_multiple_splits[jsonl-False] - AssertionError: assert 6 == 3
 +  where 6 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_2/audiofolder_data_dir_with_metadata/train/audio_file.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_2/audiofolder_data_dir_with_metadata/train/audio_file2.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_2/audiofolder_data_dir_with_metadata/train/metadata.jsonl', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_2/audiofolder_data_dir_with_metadata/train/audio_file.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_2/audiofolder_data_dir_with_metadata/train/audio_file2.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_2/audiofolder_data_dir_with_metadata/train/metadata.jsonl'])
ERROR tests/packaged_modules/test_audiofolder.py::test_data_files_with_metadata_and_multiple_splits[jsonl-True] - AssertionError: assert 6 == 3
 +  where 6 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_3/audiofolder_data_dir_with_metadata/train/audio_file.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_3/audiofolder_data_dir_with_metadata/train/audio_file2.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_3/audiofolder_data_dir_with_metadata/train/metadata.jsonl', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_3/audiofolder_data_dir_with_metadata/train/audio_file.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_3/audiofolder_data_dir_with_metadata/train/audio_file2.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_3/audiofolder_data_dir_with_metadata/train/metadata.jsonl'])
ERROR tests/packaged_modules/test_audiofolder.py::test_data_files_with_metadata_and_multiple_splits[csv-False] - AssertionError: assert 6 == 3
 +  where 6 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_4/audiofolder_data_dir_with_metadata/train/audio_file.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_4/audiofolder_data_dir_with_metadata/train/audio_file2.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_4/audiofolder_data_dir_with_metadata/train/metadata.csv', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_4/audiofolder_data_dir_with_metadata/train/audio_file.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_4/audiofolder_data_dir_with_metadata/train/audio_file2.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_4/audiofolder_data_dir_with_metadata/train/metadata.csv'])
ERROR tests/packaged_modules/test_audiofolder.py::test_data_files_with_metadata_and_multiple_splits[csv-True] - AssertionError: assert 6 == 3
 +  where 6 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_5/audiofolder_data_dir_with_metadata/train/audio_file.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_5/audiofolder_data_dir_with_metadata/train/audio_file2.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_5/audiofolder_data_dir_with_metadata/train/metadata.csv', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_5/audiofolder_data_dir_with_metadata/train/audio_file.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_5/audiofolder_data_dir_with_metadata/train/audio_file2.wav', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_5/audiofolder_data_dir_with_metadata/train/metadata.csv'])
ERROR tests/packaged_modules/test_folder_based_builder.py::test_data_files_with_metadata_and_splits[1-False] - AssertionError: assert 6 == 3
 +  where 6 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_3/autofolder_data_dir_with_metadata_two_splits/train/file.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_3/autofolder_data_dir_with_metadata_two_splits/train/file2.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_3/autofolder_data_dir_with_metadata_two_splits/train/metadata.jsonl', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_3/autofolder_data_dir_with_metadata_two_splits/train/file.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_3/autofolder_data_dir_with_metadata_two_splits/train/file2.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_3/autofolder_data_dir_with_metadata_two_splits/train/metadata.jsonl'])
ERROR tests/packaged_modules/test_folder_based_builder.py::test_data_files_with_metadata_and_splits[1-True] - AssertionError: assert 6 == 3
 +  where 6 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_4/autofolder_data_dir_with_metadata_two_splits/train/file.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_4/autofolder_data_dir_with_metadata_two_splits/train/file2.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_4/autofolder_data_dir_with_metadata_two_splits/train/metadata.jsonl', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_4/autofolder_data_dir_with_metadata_two_splits/train/file.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_4/autofolder_data_dir_with_metadata_two_splits/train/file2.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_4/autofolder_data_dir_with_metadata_two_splits/train/metadata.jsonl'])
ERROR tests/packaged_modules/test_folder_based_builder.py::test_data_files_with_metadata_and_splits[2-False] - AssertionError: assert 6 == 3
 +  where 6 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_5/autofolder_data_dir_with_metadata_two_splits/train/file.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_5/autofolder_data_dir_with_metadata_two_splits/train/file2.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_5/autofolder_data_dir_with_metadata_two_splits/train/metadata.jsonl', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_5/autofolder_data_dir_with_metadata_two_splits/train/file.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_5/autofolder_data_dir_with_metadata_two_splits/train/file2.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_5/autofolder_data_dir_with_metadata_two_splits/train/metadata.jsonl'])
ERROR tests/packaged_modules/test_imagefolder.py::test_data_files_with_metadata_and_multiple_splits[jsonl-False] - AssertionError: assert 6 == 3
 +  where 6 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_12/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_12/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb2.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_12/imagefolder_data_dir_with_metadata_two_splits/train/metadata.jsonl', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_12/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_12/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb2.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_12/imagefolder_data_dir_with_metadata_two_splits/train/metadata.jsonl'])
ERROR tests/packaged_modules/test_imagefolder.py::test_data_files_with_metadata_and_multiple_splits[jsonl-True] - AssertionError: assert 6 == 3
 +  where 6 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_13/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_13/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb2.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_13/imagefolder_data_dir_with_metadata_two_splits/train/metadata.jsonl', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_13/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_13/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb2.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_13/imagefolder_data_dir_with_metadata_two_splits/train/metadata.jsonl'])
ERROR tests/packaged_modules/test_folder_based_builder.py::test_data_files_with_metadata_and_splits[2-True] - AssertionError: assert 6 == 3
 +  where 6 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_6/autofolder_data_dir_with_metadata_two_splits/train/file.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_6/autofolder_data_dir_with_metadata_two_splits/train/file2.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_6/autofolder_data_dir_with_metadata_two_splits/train/metadata.jsonl', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_6/autofolder_data_dir_with_metadata_two_splits/train/file.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_6/autofolder_data_dir_with_metadata_two_splits/train/file2.txt', '/tmp/pytest-of-runner/pytest-0/popen-gw0/test_data_files_with_metadata_6/autofolder_data_dir_with_metadata_two_splits/train/metadata.jsonl'])
ERROR tests/packaged_modules/test_imagefolder.py::test_data_files_with_metadata_and_multiple_splits[csv-False] - AssertionError: assert 6 == 3
 +  where 6 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_14/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_14/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb2.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_14/imagefolder_data_dir_with_metadata_two_splits/train/metadata.csv', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_14/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_14/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb2.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_14/imagefolder_data_dir_with_metadata_two_splits/train/metadata.csv'])
ERROR tests/packaged_modules/test_imagefolder.py::test_data_files_with_metadata_and_multiple_splits[csv-True] - AssertionError: assert 6 == 3
 +  where 6 = len(['/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_15/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_15/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb2.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_15/imagefolder_data_dir_with_metadata_two_splits/train/metadata.csv', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_15/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_15/imagefolder_data_dir_with_metadata_two_splits/train/image_rgb2.jpg', '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_data_files_with_metadata_15/imagefolder_data_dir_with_metadata_two_splits/train/metadata.csv'])
= 3 failed, 2383 passed, 26 skipped, 9 warnings, 12 errors in 280.79s (0:04:40) =
```


Do not filter out .zip extensions from no-script datasets

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006797 / 0.011353 (-0.004556) | 0.003966 / 0.011008 (-0.007042) | 0.085296 / 0.038508 (0.046788) | 0.076873 / 0.023109 (0.053764) | 0.355795 / 0.275898 (0.079897) | 0.397132 / 0.323480 (0.073652) | 0.005325 / 0.007986 (-0.002660) | 0.003343 / 0.004328 (-0.000986) | 0.064966 / 0.004250 (0.060716) | 0.054519 / 0.037052 (0.017467) | 0.357864 / 0.258489 (0.099374) | 0.409238 / 0.293841 (0.115397) | 0.031620 / 0.128546 (-0.096926) | 0.008529 / 0.075646 (-0.067117) | 0.288502 / 0.419271 (-0.130769) | 0.053260 / 0.043533 (0.009728) | 0.355245 / 0.255139 (0.100106) | 0.384139 / 0.283200 (0.100939) | 0.024507 / 0.141683 (-0.117176) | 1.494696 / 1.452155 (0.042541) | 1.579847 / 1.492716 (0.087130) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.204011 / 0.018006 (0.186005) | 0.451729 / 0.000490 (0.451239) | 0.004628 / 0.000200 (0.004428) | 0.000081 / 0.000054 (0.000026) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028342 / 0.037411 (-0.009069) | 0.084647 / 0.014526 (0.070121) | 0.096174 / 0.176557 (-0.080383) | 0.151753 / 0.737135 (-0.585382) | 0.096347 / 0.296338 (-0.199991) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.387179 / 0.215209 (0.171970) | 3.861552 / 2.077655 (1.783898) | 1.844033 / 1.504120 (0.339913) | 1.678811 / 1.541195 (0.137616) | 1.793207 / 1.468490 (0.324717) | 0.485836 / 4.584777 (-4.098941) | 3.566274 / 3.745712 (-0.179438) | 3.269888 / 5.269862 (-1.999974) | 2.042850 / 4.565676 (-2.522827) | 0.057088 / 0.424275 (-0.367187) | 0.007627 / 0.007607 (0.000019) | 0.460510 / 0.226044 (0.234465) | 4.602019 / 2.268929 (2.333090) | 2.390984 / 55.444624 (-53.053641) | 1.976150 / 6.876477 (-4.900327) | 2.193394 / 2.142072 (0.051322) | 0.582775 / 4.805227 (-4.222453) | 0.133408 / 6.500664 (-6.367256) | 0.060577 / 0.075469 (-0.014893) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.248505 / 1.841788 (-0.593283) | 19.771301 / 8.074308 (11.696993) | 14.327871 / 10.191392 (4.136479) | 0.155288 / 0.680424 (-0.525136) | 0.018310 / 0.534201 (-0.515891) | 0.393664 / 0.579283 (-0.185619) | 0.410578 / 0.434364 (-0.023786) | 0.459301 / 0.540337 (-0.081037) | 0.631921 / 1.386936 (-0.755015) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006827 / 0.011353 (-0.004526) | 0.004094 / 0.011008 (-0.006915) | 0.065299 / 0.038508 (0.026791) | 0.079496 / 0.023109 (0.056387) | 0.403661 / 0.275898 (0.127763) | 0.434449 / 0.323480 (0.110969) | 0.005398 / 0.007986 (-0.002588) | 0.003410 / 0.004328 (-0.000919) | 0.064832 / 0.004250 (0.060582) | 0.056303 / 0.037052 (0.019250) | 0.397848 / 0.258489 (0.139359) | 0.438244 / 0.293841 (0.144403) | 0.032637 / 0.128546 (-0.095909) | 0.008584 / 0.075646 (-0.067063) | 0.071406 / 0.419271 (-0.347866) | 0.048265 / 0.043533 (0.004732) | 0.397814 / 0.255139 (0.142675) | 0.421601 / 0.283200 (0.138402) | 0.023815 / 0.141683 (-0.117868) | 1.504814 / 1.452155 (0.052659) | 1.577185 / 1.492716 (0.084469) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.231775 / 0.018006 (0.213769) | 0.445437 / 0.000490 (0.444948) | 0.005252 / 0.000200 (0.005052) | 0.000093 / 0.000054 (0.000039) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032777 / 0.037411 (-0.004634) | 0.095054 / 0.014526 (0.080528) | 0.106429 / 0.176557 (-0.070127) | 0.160111 / 0.737135 (-0.577024) | 0.108075 / 0.296338 (-0.188263) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.426034 / 0.215209 (0.210825) | 4.244668 / 2.077655 (2.167013) | 2.257938 / 1.504120 (0.753818) | 2.087993 / 1.541195 (0.546798) | 2.170878 / 1.468490 (0.702387) | 0.485228 / 4.584777 (-4.099549) | 3.725912 / 3.745712 (-0.019800) | 3.286925 / 5.269862 (-1.982937) | 2.059929 / 4.565676 (-2.505748) | 0.057813 / 0.424275 (-0.366462) | 0.007518 / 0.007607 (-0.000089) | 0.506632 / 0.226044 (0.280588) | 5.048340 / 2.268929 (2.779411) | 2.744756 / 55.444624 (-52.699869) | 2.406636 / 6.876477 (-4.469841) | 2.617552 / 2.142072 (0.475480) | 0.588476 / 4.805227 (-4.216751) | 0.133518 / 6.500664 (-6.367146) | 0.060778 / 0.075469 (-0.014691) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.356416 / 1.841788 (-0.485372) | 20.467516 / 8.074308 (12.393208) | 15.265443 / 10.191392 (5.074051) | 0.169201 / 0.680424 (-0.511223) | 0.020087 / 0.534201 (-0.514114) | 0.402332 / 0.579283 (-0.176951) | 0.414848 / 0.434364 (-0.019516) | 0.470422 / 0.540337 (-0.069916) | 0.647266 / 1.386936 (-0.739670) |

![](https://cml.dev/watermark.png#eb001b4cee7f1d71e393c3ad489a8a5cd8119df5 "CML watermark")

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005804 / 0.011353 (-0.005549) | 0.003519 / 0.011008 (-0.007489) | 0.080003 / 0.038508 (0.041495) | 0.055419 / 0.023109 (0.032309) | 0.395254 / 0.275898 (0.119356) | 0.432714 / 0.323480 (0.109234) | 0.004438 / 0.007986 (-0.003548) | 0.002832 / 0.004328 (-0.001496) | 0.062026 / 0.004250 (0.057775) | 0.044334 / 0.037052 (0.007282) | 0.401278 / 0.258489 (0.142789) | 0.451516 / 0.293841 (0.157675) | 0.026791 / 0.128546 (-0.101755) | 0.007946 / 0.075646 (-0.067700) | 0.265166 / 0.419271 (-0.154106) | 0.044119 / 0.043533 (0.000586) | 0.399621 / 0.255139 (0.144482) | 0.422808 / 0.283200 (0.139609) | 0.019998 / 0.141683 (-0.121685) | 1.433559 / 1.452155 (-0.018596) | 1.596902 / 1.492716 (0.104186) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.195662 / 0.018006 (0.177656) | 0.423167 / 0.000490 (0.422677) | 0.003426 / 0.000200 (0.003227) | 0.000066 / 0.000054 (0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023318 / 0.037411 (-0.014094) | 0.072532 / 0.014526 (0.058006) | 0.082181 / 0.176557 (-0.094375) | 0.142214 / 0.737135 (-0.594921) | 0.083423 / 0.296338 (-0.212915) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.402270 / 0.215209 (0.187061) | 4.027607 / 2.077655 (1.949953) | 2.059803 / 1.504120 (0.555684) | 1.865115 / 1.541195 (0.323920) | 1.934976 / 1.468490 (0.466485) | 0.502145 / 4.584777 (-4.082632) | 2.970865 / 3.745712 (-0.774847) | 2.784155 / 5.269862 (-2.485707) | 1.822003 / 4.565676 (-2.743673) | 0.057699 / 0.424275 (-0.366576) | 0.006668 / 0.007607 (-0.000939) | 0.471164 / 0.226044 (0.245120) | 4.733079 / 2.268929 (2.464150) | 2.445119 / 55.444624 (-52.999505) | 2.132956 / 6.876477 (-4.743521) | 2.335998 / 2.142072 (0.193926) | 0.594881 / 4.805227 (-4.210347) | 0.125801 / 6.500664 (-6.374863) | 0.060780 / 0.075469 (-0.014689) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.233170 / 1.841788 (-0.608618) | 17.942205 / 8.074308 (9.867897) | 13.587020 / 10.191392 (3.395628) | 0.142110 / 0.680424 (-0.538314) | 0.016600 / 0.534201 (-0.517601) | 0.328659 / 0.579283 (-0.250624) | 0.347759 / 0.434364 (-0.086605) | 0.378651 / 0.540337 (-0.161687) | 0.523474 / 1.386936 (-0.863462) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006028 / 0.011353 (-0.005325) | 0.003552 / 0.011008 (-0.007456) | 0.062175 / 0.038508 (0.023667) | 0.057602 / 0.023109 (0.034493) | 0.444585 / 0.275898 (0.168687) | 0.471238 / 0.323480 (0.147758) | 0.004562 / 0.007986 (-0.003423) | 0.002871 / 0.004328 (-0.001457) | 0.063101 / 0.004250 (0.058851) | 0.046072 / 0.037052 (0.009020) | 0.448253 / 0.258489 (0.189764) | 0.478734 / 0.293841 (0.184893) | 0.028463 / 0.128546 (-0.100084) | 0.008090 / 0.075646 (-0.067557) | 0.068142 / 0.419271 (-0.351130) | 0.040517 / 0.043533 (-0.003016) | 0.447145 / 0.255139 (0.192006) | 0.469472 / 0.283200 (0.186273) | 0.019391 / 0.141683 (-0.122291) | 1.471195 / 1.452155 (0.019040) | 1.532966 / 1.492716 (0.040249) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.259894 / 0.018006 (0.241888) | 0.412987 / 0.000490 (0.412497) | 0.020780 / 0.000200 (0.020580) | 0.000084 / 0.000054 (0.000030) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026352 / 0.037411 (-0.011060) | 0.080024 / 0.014526 (0.065498) | 0.088041 / 0.176557 (-0.088516) | 0.142987 / 0.737135 (-0.594148) | 0.090108 / 0.296338 (-0.206231) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.458874 / 0.215209 (0.243665) | 4.573005 / 2.077655 (2.495351) | 2.507885 / 1.504120 (1.003765) | 2.335432 / 1.541195 (0.794238) | 2.379617 / 1.468490 (0.911126) | 0.503331 / 4.584777 (-4.081446) | 3.078284 / 3.745712 (-0.667428) | 2.750580 / 5.269862 (-2.519282) | 1.828100 / 4.565676 (-2.737577) | 0.057572 / 0.424275 (-0.366703) | 0.006553 / 0.007607 (-0.001054) | 0.532283 / 0.226044 (0.306239) | 5.310584 / 2.268929 (3.041656) | 2.943559 / 55.444624 (-52.501065) | 2.587544 / 6.876477 (-4.288932) | 2.718261 / 2.142072 (0.576188) | 0.590267 / 4.805227 (-4.214961) | 0.123229 / 6.500664 (-6.377435) | 0.060219 / 0.075469 (-0.015250) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.340773 / 1.841788 (-0.501014) | 18.420766 / 8.074308 (10.346458) | 14.630550 / 10.191392 (4.439158) | 0.146666 / 0.680424 (-0.533758) | 0.017905 / 0.534201 (-0.516296) | 0.332483 / 0.579283 (-0.246801) | 0.355490 / 0.434364 (-0.078874) | 0.382618 / 0.540337 (-0.157720) | 0.531336 / 1.386936 (-0.855600) |

![](https://cml.dev/watermark.png#d438617fc577bc0222527714edafea0c52ebf239 "CML watermark")

There were CI errors unrelated to this PR.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008702 / 0.011353 (-0.002651) | 0.005060 / 0.011008 (-0.005948) | 0.097017 / 0.038508 (0.058509) | 0.073740 / 0.023109 (0.050631) | 0.435138 / 0.275898 (0.159240) | 0.512776 / 0.323480 (0.189296) | 0.006186 / 0.007986 (-0.001800) | 0.003970 / 0.004328 (-0.000358) | 0.089523 / 0.004250 (0.085273) | 0.054441 / 0.037052 (0.017389) | 0.447415 / 0.258489 (0.188926) | 0.464851 / 0.293841 (0.171010) | 0.050264 / 0.128546 (-0.078283) | 0.016643 / 0.075646 (-0.059004) | 0.350565 / 0.419271 (-0.068707) | 0.071220 / 0.043533 (0.027687) | 0.432531 / 0.255139 (0.177392) | 0.472994 / 0.283200 (0.189795) | 0.040229 / 0.141683 (-0.101454) | 1.743431 / 1.452155 (0.291276) | 1.778653 / 1.492716 (0.285936) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.261992 / 0.018006 (0.243986) | 0.571979 / 0.000490 (0.571489) | 0.006270 / 0.000200 (0.006071) | 0.000109 / 0.000054 (0.000054) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027821 / 0.037411 (-0.009590) | 0.081874 / 0.014526 (0.067348) | 0.103725 / 0.176557 (-0.072831) | 0.170593 / 0.737135 (-0.566542) | 0.108749 / 0.296338 (-0.187590) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.690774 / 0.215209 (0.475565) | 6.770902 / 2.077655 (4.693247) | 2.887218 / 1.504120 (1.383098) | 2.456226 / 1.541195 (0.915032) | 2.509422 / 1.468490 (1.040932) | 0.768451 / 4.584777 (-3.816326) | 4.988933 / 3.745712 (1.243221) | 4.151460 / 5.269862 (-1.118402) | 2.640472 / 4.565676 (-1.925205) | 0.093522 / 0.424275 (-0.330753) | 0.008614 / 0.007607 (0.001007) | 0.696281 / 0.226044 (0.470237) | 6.721077 / 2.268929 (4.452149) | 3.229760 / 55.444624 (-52.214864) | 2.668521 / 6.876477 (-4.207956) | 2.866420 / 2.142072 (0.724347) | 0.945328 / 4.805227 (-3.859899) | 0.197645 / 6.500664 (-6.303019) | 0.074442 / 0.075469 (-0.001027) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.630468 / 1.841788 (-0.211320) | 22.991661 / 8.074308 (14.917353) | 19.816919 / 10.191392 (9.625527) | 0.257410 / 0.680424 (-0.423014) | 0.027228 / 0.534201 (-0.506973) | 0.444515 / 0.579283 (-0.134768) | 0.597067 / 0.434364 (0.162703) | 0.528151 / 0.540337 (-0.012186) | 0.771276 / 1.386936 (-0.615660) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009154 / 0.011353 (-0.002199) | 0.004648 / 0.011008 (-0.006360) | 0.073054 / 0.038508 (0.034546) | 0.077146 / 0.023109 (0.054037) | 0.481659 / 0.275898 (0.205761) | 0.516985 / 0.323480 (0.193505) | 0.007447 / 0.007986 (-0.000538) | 0.003890 / 0.004328 (-0.000438) | 0.078701 / 0.004250 (0.074450) | 0.059183 / 0.037052 (0.022131) | 0.475350 / 0.258489 (0.216861) | 0.547834 / 0.293841 (0.253993) | 0.058440 / 0.128546 (-0.070106) | 0.013563 / 0.075646 (-0.062083) | 0.084320 / 0.419271 (-0.334951) | 0.065965 / 0.043533 (0.022433) | 0.483541 / 0.255139 (0.228402) | 0.513940 / 0.283200 (0.230740) | 0.042889 / 0.141683 (-0.098794) | 1.676050 / 1.452155 (0.223895) | 1.759206 / 1.492716 (0.266489) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.274848 / 0.018006 (0.256841) | 0.588965 / 0.000490 (0.588475) | 0.006312 / 0.000200 (0.006112) | 0.000120 / 0.000054 (0.000065) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033871 / 0.037411 (-0.003540) | 0.104013 / 0.014526 (0.089487) | 0.118457 / 0.176557 (-0.058099) | 0.178268 / 0.737135 (-0.558868) | 0.116972 / 0.296338 (-0.179366) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.609952 / 0.215209 (0.394743) | 5.788754 / 2.077655 (3.711100) | 2.812166 / 1.504120 (1.308046) | 2.362861 / 1.541195 (0.821666) | 2.641295 / 1.468490 (1.172804) | 0.767601 / 4.584777 (-3.817176) | 5.027439 / 3.745712 (1.281727) | 4.612511 / 5.269862 (-0.657351) | 2.654364 / 4.565676 (-1.911312) | 0.103100 / 0.424275 (-0.321175) | 0.012233 / 0.007607 (0.004626) | 0.749283 / 0.226044 (0.523238) | 7.511093 / 2.268929 (5.242165) | 3.585867 / 55.444624 (-51.858757) | 3.255110 / 6.876477 (-3.621366) | 3.260174 / 2.142072 (1.118102) | 0.958422 / 4.805227 (-3.846806) | 0.209096 / 6.500664 (-6.291568) | 0.075014 / 0.075469 (-0.000455) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.728283 / 1.841788 (-0.113504) | 25.411147 / 8.074308 (17.336839) | 21.335202 / 10.191392 (11.143810) | 0.199090 / 0.680424 (-0.481334) | 0.031288 / 0.534201 (-0.502913) | 0.449226 / 0.579283 (-0.130057) | 0.555570 / 0.434364 (0.121206) | 0.570297 / 0.540337 (0.029960) | 0.758673 / 1.386936 (-0.628263) |

![](https://cml.dev/watermark.png#fa696b4b4f0d11c5b8592eb31cb1d54a707e3d33 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006862 / 0.011353 (-0.004491) | 0.003959 / 0.011008 (-0.007049) | 0.087219 / 0.038508 (0.048711) | 0.078335 / 0.023109 (0.055226) | 0.319019 / 0.275898 (0.043121) | 0.342871 / 0.323480 (0.019391) | 0.004065 / 0.007986 (-0.003921) | 0.004346 / 0.004328 (0.000017) | 0.065243 / 0.004250 (0.060993) | 0.056698 / 0.037052 (0.019646) | 0.326906 / 0.258489 (0.068417) | 0.354323 / 0.293841 (0.060482) | 0.031252 / 0.128546 (-0.097295) | 0.008587 / 0.075646 (-0.067060) | 0.300323 / 0.419271 (-0.118948) | 0.052810 / 0.043533 (0.009277) | 0.323866 / 0.255139 (0.068727) | 0.346011 / 0.283200 (0.062811) | 0.025584 / 0.141683 (-0.116099) | 1.464475 / 1.452155 (0.012320) | 1.530868 / 1.492716 (0.038152) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.208927 / 0.018006 (0.190921) | 0.454147 / 0.000490 (0.453657) | 0.003945 / 0.000200 (0.003746) | 0.000081 / 0.000054 (0.000026) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029901 / 0.037411 (-0.007511) | 0.088889 / 0.014526 (0.074363) | 0.098181 / 0.176557 (-0.078375) | 0.156787 / 0.737135 (-0.580349) | 0.099015 / 0.296338 (-0.197324) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.384981 / 0.215209 (0.169772) | 3.831040 / 2.077655 (1.753386) | 1.858312 / 1.504120 (0.354192) | 1.686846 / 1.541195 (0.145651) | 1.771509 / 1.468490 (0.303019) | 0.485618 / 4.584777 (-4.099159) | 3.430961 / 3.745712 (-0.314751) | 3.264489 / 5.269862 (-2.005372) | 2.040125 / 4.565676 (-2.525551) | 0.057218 / 0.424275 (-0.367057) | 0.007640 / 0.007607 (0.000033) | 0.468072 / 0.226044 (0.242027) | 4.677214 / 2.268929 (2.408286) | 2.348425 / 55.444624 (-53.096199) | 1.994352 / 6.876477 (-4.882125) | 2.217020 / 2.142072 (0.074948) | 0.587467 / 4.805227 (-4.217760) | 0.133550 / 6.500664 (-6.367114) | 0.060571 / 0.075469 (-0.014898) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.271003 / 1.841788 (-0.570785) | 19.986365 / 8.074308 (11.912057) | 14.574046 / 10.191392 (4.382654) | 0.146212 / 0.680424 (-0.534212) | 0.018320 / 0.534201 (-0.515881) | 0.394524 / 0.579283 (-0.184759) | 0.399707 / 0.434364 (-0.034657) | 0.458965 / 0.540337 (-0.081372) | 0.619940 / 1.386936 (-0.766996) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006982 / 0.011353 (-0.004371) | 0.004061 / 0.011008 (-0.006947) | 0.064520 / 0.038508 (0.026012) | 0.076828 / 0.023109 (0.053719) | 0.402989 / 0.275898 (0.127090) | 0.439697 / 0.323480 (0.116217) | 0.005511 / 0.007986 (-0.002475) | 0.003378 / 0.004328 (-0.000950) | 0.064727 / 0.004250 (0.060477) | 0.058114 / 0.037052 (0.021062) | 0.402054 / 0.258489 (0.143565) | 0.442377 / 0.293841 (0.148536) | 0.032808 / 0.128546 (-0.095738) | 0.008604 / 0.075646 (-0.067043) | 0.070994 / 0.419271 (-0.348278) | 0.048738 / 0.043533 (0.005205) | 0.399786 / 0.255139 (0.144647) | 0.423537 / 0.283200 (0.140338) | 0.022397 / 0.141683 (-0.119286) | 1.504613 / 1.452155 (0.052458) | 1.571064 / 1.492716 (0.078348) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.226876 / 0.018006 (0.208870) | 0.451477 / 0.000490 (0.450987) | 0.004511 / 0.000200 (0.004311) | 0.000095 / 0.000054 (0.000041) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032998 / 0.037411 (-0.004413) | 0.095843 / 0.014526 (0.081317) | 0.105684 / 0.176557 (-0.070873) | 0.158175 / 0.737135 (-0.578960) | 0.107297 / 0.296338 (-0.189041) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.434912 / 0.215209 (0.219703) | 4.326394 / 2.077655 (2.248740) | 2.287310 / 1.504120 (0.783190) | 2.127987 / 1.541195 (0.586793) | 2.202485 / 1.468490 (0.733995) | 0.494305 / 4.584777 (-4.090472) | 3.575176 / 3.745712 (-0.170536) | 3.354358 / 5.269862 (-1.915504) | 2.074293 / 4.565676 (-2.491383) | 0.058967 / 0.424275 (-0.365308) | 0.007712 / 0.007607 (0.000105) | 0.513734 / 0.226044 (0.287690) | 5.107538 / 2.268929 (2.838610) | 2.776190 / 55.444624 (-52.668434) | 2.425051 / 6.876477 (-4.451426) | 2.666715 / 2.142072 (0.524643) | 0.598844 / 4.805227 (-4.206383) | 0.134186 / 6.500664 (-6.366478) | 0.062403 / 0.075469 (-0.013066) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.346730 / 1.841788 (-0.495058) | 20.533190 / 8.074308 (12.458882) | 15.174443 / 10.191392 (4.983051) | 0.167204 / 0.680424 (-0.513219) | 0.020619 / 0.534201 (-0.513582) | 0.399033 / 0.579283 (-0.180250) | 0.394428 / 0.434364 (-0.039936) | 0.468792 / 0.540337 (-0.071545) | 0.640122 / 1.386936 (-0.746814) |

![](https://cml.dev/watermark.png#2c4c2b529e2a262a5006e4caa55fbc003378006a "CML watermark")

This PR is a hotfix of:
- #6207

That PR introduced the filtering out of `.zip` extensions. This PR reverts that.

Hot fix #6207.

Maybe we should do patch releases: the bug was introduced in 2.13.1.

CC: @lhoestq


No-script datasets with ZIP files do not load

Something isn't working

While investigating an issue on a Hub dataset, I have discovered the no-script datasets containing ZIP files do not load.

For example, that no-script dataset containing ZIP files, raises NonMatchingSplitsSizesError:
```python
In [2]: ds = load_dataset("sidovic/LearningQ-qg")

NonMatchingSplitsSizesError: [
  {
    'expected': SplitInfo(name='train', num_bytes=0, num_examples=188660, shard_lengths=None, dataset_name=None),
    'recorded': SplitInfo(name='train', num_bytes=0, num_examples=0, shard_lengths=None, dataset_name='learning_q-qg')
  }, {
    'expected': SplitInfo(name='validation', num_bytes=0, num_examples=20630, shard_lengths=None, dataset_name=None),
    'recorded': SplitInfo(name='validation', num_bytes=0, num_examples=0, shard_lengths=None, dataset_name='learning_q-qg')
  }, {
    'expected': SplitInfo(name='test', num_bytes=0, num_examples=18227, shard_lengths=None, dataset_name=None),
    'recorded': SplitInfo(name='test', num_bytes=0, num_examples=0, shard_lengths=None, dataset_name='learning_q-qg')
  }
]
```

As another example, a no-script dataset containing just a (CSV)-ZIP file, raises a DatasetGenerationError:
```
>               num_examples, num_bytes = writer.finalize()

src/datasets/builder.py:1949:

>           raise SchemaInferenceError("Please pass `features` or at least one example when writing data")
E           datasets.arrow_writer.SchemaInferenceError: Please pass `features` or at least one example when writing data

src/datasets/arrow_writer.py:598: SchemaInferenceError

The above exception was the direct cause of the following exception:

src/datasets/load.py:2143: in load_dataset
    builder_instance.download_and_prepare(
src/datasets/builder.py:954: in download_and_prepare
    self._download_and_prepare(
src/datasets/builder.py:1049: in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
src/datasets/builder.py:1813: in _prepare_split
    for job_id, done, content in self._prepare_split_single(

>           raise DatasetGenerationError("An error occurred while generating the dataset") from e
E           datasets.builder.DatasetGenerationError: An error occurred while generating the dataset

src/datasets/builder.py:1958: DatasetGenerationError
```

After investigating, I think this bug was introduced in this PR:
- #5972

Related to:
- https://huggingface.co/datasets/sidovic/LearningQ-qg/discussions/1

CC: @lhoestq


When calling load_dataset, raise error: pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays

I solved the problem by modifying the "self DEFAULT_WRITER_BATCH_SIZE" in "class MyDataset (datasets. GeneratorBasedBuilder) : __init__"

same problem, and this solution worked me also - you can set this var by setting the keyword argument `writer_batch_size=...` in `load_dataset(...,writer_batch_size=...)`

### Describe the bug

When calling load_dataset, raise error
```
Traceback (most recent call last):
  File "/home/aihao/miniconda3/envs/torch/lib/python3.11/site-packages/datasets/builder.py", line 1694, in _pre
pare_split_single
    writer.write(example, key)
  File "/home/aihao/miniconda3/envs/torch/lib/python3.11/site-packages/datasets/arrow_writer.py", line 490, in
write
    self.write_examples_on_file()
  File "/home/aihao/miniconda3/envs/torch/lib/python3.11/site-packages/datasets/arrow_writer.py", line 448, in
write_examples_on_file
    self.write_batch(batch_examples=batch_examples)
  File "/home/aihao/miniconda3/envs/torch/lib/python3.11/site-packages/datasets/arrow_writer.py", line 559, in
write_batch
    self.write_table(pa_table, writer_batch_size)
  File "/home/aihao/miniconda3/envs/torch/lib/python3.11/site-packages/datasets/arrow_writer.py", line 571, in
write_table
    pa_table = pa_table.combine_chunks()
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/table.pxi", line 3439, in pyarrow.lib.Table.combine_chunks
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
dataset = load_dataset(
              ^^^^^^^^^^^^^
  File "/home/aihao/miniconda3/envs/torch/lib/python3.11/site-packages/datasets/load.py", line 2133, in load_da
taset
    builder_instance.download_and_prepare(
  File "/home/aihao/miniconda3/envs/torch/lib/python3.11/site-packages/datasets/builder.py", line 954, in downl
oad_and_prepare
    self._download_and_prepare(
  File "/home/aihao/miniconda3/envs/torch/lib/python3.11/site-packages/datasets/builder.py", line 1717, in _dow
nload_and_prepare
    super()._download_and_prepare(
  File "/home/aihao/miniconda3/envs/torch/lib/python3.11/site-packages/datasets/builder.py", line 1049, in _dow
nload_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/aihao/miniconda3/envs/torch/lib/python3.11/site-packages/datasets/builder.py", line 1555, in _pre
pare_split
    for job_id, done, content in self._prepare_split_single(
  File "/home/aihao/miniconda3/envs/torch/lib/python3.11/site-packages/datasets/builder.py", line 1712, in _pre
pare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
Setting num_proc from 8 back to 1 for the train split to disable multiprocessing as it only contains one shard.
09/04/2023 12:02:04 - WARNING - datasets.builder - Setting num_proc from 8 back to 1 for the train split to dis
able multiprocessing as it only contains one shard.
```

### Steps to reproduce the bug

Call load_dataset with the large image as feature

### Expected behavior

no error

### Environment info

- `datasets` version: 2.14.3
- Platform: Linux-6.2.0-31-generic-x86_64-with-glibc2.35
- Python version: 3.11.4
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3


Support loading from a DVC remote repository

enhancement

New feature or request

(cross-posting from the linked DVC issue)

I think this should already work out of the box with the current `datasets` and `dvc.api` releases by passing the correct `storage_options` into the datasets calls. `storage_options` is essentially just the kwargs dict that gets passed to the fsspec fs constructor.

The main thing to note here is that the fsspec DVCFileSystem URL should be `dvc://folder/file.json` (i.e. this should be the DVCFileSystem path that is relative to the DVC repo root). You cannot use a URL like `https://gitlab.com/user/repo/folder/file.json`.

I think something like this should work for you (in a venv where both DVC and datasets are installed):
```python
import datasets

# load a dataset from Git/DVC repository where Git repo is located at https://gitlab.com/user/repo.git
# and path to dataset (relative to git/dvc repo root) is 'folder/file.json'
datasets.load_from_disk(
    "dvc://folder/file.json",
    storage_options={"url": "https://gitlab.com/user/repo.git"},
)
```

basically the `dvc://` is what tells fsspec to create a `DVCFileSystem` and it will construct it like
```python
fs = DVCFileSystem(**storage_options)
```

Then the subsequent calls use the rest of the `dvc://...` URL like
```python
fs.exists("folder/file.json")
```

Hi @pmrowla Thank you for your help, that's very helpful, I was indeed using `fsspec` incorrectly here. There is still an issue with `datasets`:

```python
import datasets
dataset = datasets.load_dataset("json", data_files="dvc://folder/file.jsonl", storage_options={"url": "https://gitlab.com/repo/folder/"})
```

results in the following exception:

```
Traceback (most recent call last):
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/scmrepo/fs.py", line 217, in info
    ret = self.trie.info(key)
          ^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/scmrepo/git/objects.py", line 141, in info
    obj = self.trie[key]
          ~~~~~~~~~^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/pygtrie.py", line 937, in __getitem__
    node, _ = self._get_node(key_or_slice)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/pygtrie.py", line 630, in _get_node
    raise KeyError(key)
KeyError: ('dvc:', 'datasets', 'spider', 'train.jsonl')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/datasets/load.py", line 2129, in load_dataset
    builder_instance = load_dataset_builder(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/datasets/load.py", line 1815, in load_dataset_builder
    dataset_module = dataset_module_factory(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/datasets/load.py", line 1430, in dataset_module_factory
    ).get_module()
      ^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/datasets/load.py", line 958, in get_module
    data_files = DataFilesDict.from_patterns(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/datasets/data_files.py", line 674, in from_patterns
    DataFilesList.from_patterns(
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/datasets/data_files.py", line 589, in from_patterns
    origin_metadata = _get_origin_metadata(data_files, download_config=download_config)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/datasets/data_files.py", line 504, in _get_origin_metadata
    return thread_map(
           ^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/tqdm/contrib/concurrent.py", line 69, in thread_map
    return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/tqdm/contrib/concurrent.py", line 51, in _executor_map
    return list(tqdm_class(ex.map(fn, *iterables, chunksize=chunksize), **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/.pyenv/versions/3.11.4/lib/python3.11/concurrent/futures/_base.py", line 619, in result_iterator
    yield _result_or_cancel(fs.pop())
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/.pyenv/versions/3.11.4/lib/python3.11/concurrent/futures/_base.py", line 317, in _result_or_cancel
    return fut.result(timeout)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/.pyenv/versions/3.11.4/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/.pyenv/versions/3.11.4/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/Users/bilelomrani/.pyenv/versions/3.11.4/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/datasets/data_files.py", line 491, in _get_single_origin_metadata
    info = fs.info(data_file)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/dvc/fs/dvc.py", line 357, in info
    return self._info(key, path, ignore_subrepos=ignore_subrepos)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/dvc/fs/dvc.py", line 377, in _info
    fs_info = fs.info(fs_path)
              ^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/dvc_objects/fs/base.py", line 501, in info
    return self.fs.info(path, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bilelomrani/Documents/ILLUIN.nosync/instructions-finetuning/.venv/lib/python3.11/site-packages/scmrepo/fs.py", line 221, in info
    raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), path)
FileNotFoundError: [Errno 2] No such file or directory: '/dvc:/folder/file.jsonl'
```

Somehow the URL gets turned into `/dvc:/folder/file.jsonl` inside `datasets`. Otherwise I can confirm that using `fsspec` properly with DVC works as expected.

For the record, there was a `dvc.api.DVCFileSystem` bug which is fixed in DVC `main` and will be available in the next DVC release.

To use DVC with `datasets` you just need to pass the Git/DVC repo `url` in `storage_options` as discussed above.

(note that this requires having both `datasets` and `dvc` installed in your python environment)
```python
>>> from datasets import load_dataset
>>> load_dataset(
...   "json",
...   data_files="dvc://eval/metrics.json",
...   storage_options={"url": "https://github.com/iterative/example-get-started.git"},
... )
DatasetDict({
    train: Dataset({
        features: ['avg_prec', 'roc_auc'],
        num_rows: 1
    })
})
```

Any additional `DVCFileSystem` args can be passed in the same way, so to get a specific branch/tag/commit from the DVC repo you just need to specify the `rev` in `storage_options` like
```
storage_options={"url": "https://github.com/iterative/example-get-started.git", "rev": "main"}
```

I think this issue can probably be closed now.

### Feature request

Adding support for loading a file from a DVC repository, tracked remotely on a SCM.

### Motivation

DVC is a popular version control system to version and manage datasets. The files are stored on a remote object storage platform, but they are tracked using Git. Integration with DVC is possible through the `DVCFileSystem`.

I have a Gitlab repository where multiple files are tracked using DVC and stored in a GCP bucket. I would like to be able to load these files using `datasets` directly using an URL. My goal is to write a generic code that abstracts the storage layer, such that my users will only have to pass in an `fsspec`-compliant URL and the corresponding files will be loaded.

### Your contribution

I managed to instantiate a `DVCFileSystem` pointing to a Gitlab repo from a `fsspec` chained URL in [this pull request](https://github.com/iterative/dvc/pull/9903) to DVC.

```python
from fsspec.core import url_to_fs

fs, _ = url_to_fs("dvc::https://gitlab.com/repository/group/my-repo")
```

From now I'm not sure how to continue, it seems that `datasets` expects the URL to be fully qualified like so: `dvc::https://gitlab.com/repository/group/my-repo/my-folder/my-file.json` but this fails because `DVCFileSystem` expects the URL to point to the root of an SCM repo. Is there a way to make this work with `datasets`?


avoid downgrading jax version

New feature or request

https://github.com/huggingface/datasets/blob/main/setup.py#L236
Currently has the highest version at 0.3.25; Not sure if there is any reason for this, other than that was the tested version?

### Feature request

Whenever I `pip install datasets[jax]` it downgrades jax to version 0.3.25. I seem to be able to install this library first then upgrade jax back to version 0.4.13.

### Motivation

It would be nice to not overwrite currently installed version of jax if possible.

### Your contribution

I would be willing to beta test. Or maybe write some code if I could get pointed in the right direction, I'm not super familiar with this codebase.