Replace deprecated license_file in setup.cfg

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006884 / 0.011353 (-0.004469) | 0.004132 / 0.011008 (-0.006877) | 0.085993 / 0.038508 (0.047485) | 0.084049 / 0.023109 (0.060940) | 0.346194 / 0.275898 (0.070296) | 0.386999 / 0.323480 (0.063519) | 0.004185 / 0.007986 (-0.003801) | 0.004354 / 0.004328 (0.000026) | 0.065137 / 0.004250 (0.060886) | 0.057629 / 0.037052 (0.020577) | 0.353639 / 0.258489 (0.095150) | 0.400815 / 0.293841 (0.106974) | 0.031370 / 0.128546 (-0.097176) | 0.008719 / 0.075646 (-0.066927) | 0.289579 / 0.419271 (-0.129693) | 0.052826 / 0.043533 (0.009293) | 0.351110 / 0.255139 (0.095971) | 0.375663 / 0.283200 (0.092464) | 0.025892 / 0.141683 (-0.115791) | 1.481943 / 1.452155 (0.029789) | 1.541494 / 1.492716 (0.048778) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.240007 / 0.018006 (0.222000) | 0.456216 / 0.000490 (0.455726) | 0.009348 / 0.000200 (0.009148) | 0.000370 / 0.000054 (0.000315) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029541 / 0.037411 (-0.007870) | 0.088394 / 0.014526 (0.073868) | 0.098460 / 0.176557 (-0.078096) | 0.154053 / 0.737135 (-0.583083) | 0.098821 / 0.296338 (-0.197518) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.386751 / 0.215209 (0.171542) | 3.809818 / 2.077655 (1.732164) | 1.833439 / 1.504120 (0.329319) | 1.686924 / 1.541195 (0.145729) | 1.796882 / 1.468490 (0.328392) | 0.488853 / 4.584777 (-4.095924) | 3.606369 / 3.745712 (-0.139343) | 3.460003 / 5.269862 (-1.809858) | 2.087493 / 4.565676 (-2.478184) | 0.056838 / 0.424275 (-0.367437) | 0.007679 / 0.007607 (0.000072) | 0.455080 / 0.226044 (0.229036) | 4.539227 / 2.268929 (2.270299) | 2.337245 / 55.444624 (-53.107379) | 1.988195 / 6.876477 (-4.888281) | 2.067473 / 2.142072 (-0.074600) | 0.576640 / 4.805227 (-4.228587) | 0.132140 / 6.500664 (-6.368525) | 0.060737 / 0.075469 (-0.014732) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.268866 / 1.841788 (-0.572922) | 19.695296 / 8.074308 (11.620988) | 14.431254 / 10.191392 (4.239862) | 0.166779 / 0.680424 (-0.513645) | 0.018262 / 0.534201 (-0.515939) | 0.390406 / 0.579283 (-0.188877) | 0.411284 / 0.434364 (-0.023080) | 0.456696 / 0.540337 (-0.083642) | 0.629660 / 1.386936 (-0.757276) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007210 / 0.011353 (-0.004143) | 0.004124 / 0.011008 (-0.006884) | 0.065877 / 0.038508 (0.027368) | 0.086242 / 0.023109 (0.063133) | 0.420087 / 0.275898 (0.144189) | 0.454327 / 0.323480 (0.130847) | 0.005586 / 0.007986 (-0.002399) | 0.003465 / 0.004328 (-0.000863) | 0.065153 / 0.004250 (0.060902) | 0.059337 / 0.037052 (0.022285) | 0.420913 / 0.258489 (0.162424) | 0.458552 / 0.293841 (0.164711) | 0.032335 / 0.128546 (-0.096211) | 0.008672 / 0.075646 (-0.066974) | 0.072029 / 0.419271 (-0.347242) | 0.048148 / 0.043533 (0.004615) | 0.423334 / 0.255139 (0.168196) | 0.440616 / 0.283200 (0.157416) | 0.023761 / 0.141683 (-0.117922) | 1.487022 / 1.452155 (0.034868) | 1.554028 / 1.492716 (0.061312) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.216693 / 0.018006 (0.198687) | 0.446359 / 0.000490 (0.445869) | 0.005294 / 0.000200 (0.005094) | 0.000100 / 0.000054 (0.000045) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034655 / 0.037411 (-0.002756) | 0.099479 / 0.014526 (0.084953) | 0.111822 / 0.176557 (-0.064735) | 0.160675 / 0.737135 (-0.576461) | 0.108718 / 0.296338 (-0.187621) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.440270 / 0.215209 (0.225061) | 4.389013 / 2.077655 (2.311358) | 2.408007 / 1.504120 (0.903887) | 2.237233 / 1.541195 (0.696038) | 2.344131 / 1.468490 (0.875641) | 0.493143 / 4.584777 (-4.091634) | 3.620024 / 3.745712 (-0.125688) | 3.335810 / 5.269862 (-1.934052) | 2.079256 / 4.565676 (-2.486420) | 0.058324 / 0.424275 (-0.365951) | 0.007410 / 0.007607 (-0.000197) | 0.512057 / 0.226044 (0.286013) | 5.120629 / 2.268929 (2.851701) | 2.913268 / 55.444624 (-52.531356) | 2.558214 / 6.876477 (-4.318262) | 2.784146 / 2.142072 (0.642074) | 0.593308 / 4.805227 (-4.211920) | 0.134941 / 6.500664 (-6.365723) | 0.062292 / 0.075469 (-0.013177) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.351795 / 1.841788 (-0.489993) | 20.489559 / 8.074308 (12.415251) | 15.046116 / 10.191392 (4.854724) | 0.166339 / 0.680424 (-0.514085) | 0.020449 / 0.534201 (-0.513752) | 0.406570 / 0.579283 (-0.172713) | 0.423405 / 0.434364 (-0.010959) | 0.474541 / 0.540337 (-0.065796) | 0.653280 / 1.386936 (-0.733656) |

![](https://cml.dev/watermark.png#3bde0f0f0e556e55b95c72b0f83bdcf7145c813c "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006362 / 0.011353 (-0.004991) | 0.003990 / 0.011008 (-0.007018) | 0.084020 / 0.038508 (0.045512) | 0.072198 / 0.023109 (0.049089) | 0.335992 / 0.275898 (0.060094) | 0.362056 / 0.323480 (0.038576) | 0.005298 / 0.007986 (-0.002688) | 0.003421 / 0.004328 (-0.000908) | 0.065343 / 0.004250 (0.061092) | 0.053310 / 0.037052 (0.016258) | 0.344855 / 0.258489 (0.086366) | 0.385524 / 0.293841 (0.091683) | 0.030209 / 0.128546 (-0.098337) | 0.008465 / 0.075646 (-0.067181) | 0.287359 / 0.419271 (-0.131912) | 0.051371 / 0.043533 (0.007838) | 0.338716 / 0.255139 (0.083577) | 0.351730 / 0.283200 (0.068530) | 0.023581 / 0.141683 (-0.118102) | 1.473772 / 1.452155 (0.021617) | 1.560594 / 1.492716 (0.067878) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.309019 / 0.018006 (0.291013) | 0.561428 / 0.000490 (0.560939) | 0.007237 / 0.000200 (0.007038) | 0.000266 / 0.000054 (0.000212) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028172 / 0.037411 (-0.009239) | 0.081050 / 0.014526 (0.066524) | 0.095952 / 0.176557 (-0.080604) | 0.151796 / 0.737135 (-0.585340) | 0.096132 / 0.296338 (-0.200206) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.384287 / 0.215209 (0.169078) | 3.840797 / 2.077655 (1.763142) | 1.891120 / 1.504120 (0.387000) | 1.743498 / 1.541195 (0.202303) | 1.821037 / 1.468490 (0.352547) | 0.484946 / 4.584777 (-4.099831) | 3.586053 / 3.745712 (-0.159659) | 3.446215 / 5.269862 (-1.823647) | 2.054352 / 4.565676 (-2.511325) | 0.057315 / 0.424275 (-0.366960) | 0.007541 / 0.007607 (-0.000066) | 0.464088 / 0.226044 (0.238044) | 4.634005 / 2.268929 (2.365076) | 2.355818 / 55.444624 (-53.088806) | 2.045584 / 6.876477 (-4.830893) | 2.039455 / 2.142072 (-0.102617) | 0.576137 / 4.805227 (-4.229090) | 0.132071 / 6.500664 (-6.368593) | 0.059611 / 0.075469 (-0.015858) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.280078 / 1.841788 (-0.561710) | 19.054079 / 8.074308 (10.979771) | 14.291090 / 10.191392 (4.099698) | 0.170607 / 0.680424 (-0.509817) | 0.018489 / 0.534201 (-0.515712) | 0.391802 / 0.579283 (-0.187481) | 0.418945 / 0.434364 (-0.015419) | 0.464084 / 0.540337 (-0.076254) | 0.638099 / 1.386936 (-0.748837) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006735 / 0.011353 (-0.004618) | 0.004133 / 0.011008 (-0.006876) | 0.064620 / 0.038508 (0.026112) | 0.076395 / 0.023109 (0.053286) | 0.399659 / 0.275898 (0.123761) | 0.426821 / 0.323480 (0.103341) | 0.006407 / 0.007986 (-0.001578) | 0.003472 / 0.004328 (-0.000857) | 0.064922 / 0.004250 (0.060671) | 0.058312 / 0.037052 (0.021260) | 0.403286 / 0.258489 (0.144797) | 0.437772 / 0.293841 (0.143931) | 0.032323 / 0.128546 (-0.096223) | 0.008727 / 0.075646 (-0.066919) | 0.071344 / 0.419271 (-0.347927) | 0.048673 / 0.043533 (0.005141) | 0.400693 / 0.255139 (0.145554) | 0.418668 / 0.283200 (0.135468) | 0.022871 / 0.141683 (-0.118812) | 1.517691 / 1.452155 (0.065536) | 1.552021 / 1.492716 (0.059305) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.305279 / 0.018006 (0.287272) | 0.520054 / 0.000490 (0.519564) | 0.007247 / 0.000200 (0.007047) | 0.000098 / 0.000054 (0.000044) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032001 / 0.037411 (-0.005410) | 0.091273 / 0.014526 (0.076747) | 0.106480 / 0.176557 (-0.070077) | 0.163122 / 0.737135 (-0.574014) | 0.105244 / 0.296338 (-0.191094) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.432207 / 0.215209 (0.216998) | 4.304856 / 2.077655 (2.227202) | 2.326790 / 1.504120 (0.822670) | 2.150081 / 1.541195 (0.608886) | 2.150558 / 1.468490 (0.682068) | 0.488808 / 4.584777 (-4.095969) | 3.690435 / 3.745712 (-0.055277) | 3.302625 / 5.269862 (-1.967236) | 2.044193 / 4.565676 (-2.521483) | 0.057520 / 0.424275 (-0.366755) | 0.007281 / 0.007607 (-0.000326) | 0.521078 / 0.226044 (0.295034) | 5.162620 / 2.268929 (2.893691) | 2.744041 / 55.444624 (-52.700583) | 2.407211 / 6.876477 (-4.469266) | 2.606290 / 2.142072 (0.464217) | 0.586412 / 4.805227 (-4.218815) | 0.132152 / 6.500664 (-6.368512) | 0.059424 / 0.075469 (-0.016045) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.351879 / 1.841788 (-0.489908) | 19.460608 / 8.074308 (11.386299) | 14.643413 / 10.191392 (4.452021) | 0.168062 / 0.680424 (-0.512362) | 0.020396 / 0.534201 (-0.513805) | 0.395885 / 0.579283 (-0.183398) | 0.439551 / 0.434364 (0.005187) | 0.473051 / 0.540337 (-0.067286) | 0.644614 / 1.386936 (-0.742322) |

![](https://cml.dev/watermark.png#732b2ed47728fffc8d74f92691c21de8ac7423fe "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.014708 / 0.011353 (0.003355) | 0.008309 / 0.011008 (-0.002699) | 0.138986 / 0.038508 (0.100478) | 0.121781 / 0.023109 (0.098671) | 0.495536 / 0.275898 (0.219637) | 0.565195 / 0.323480 (0.241715) | 0.008018 / 0.007986 (0.000032) | 0.004904 / 0.004328 (0.000575) | 0.080622 / 0.004250 (0.076371) | 0.078917 / 0.037052 (0.041865) | 0.489424 / 0.258489 (0.230935) | 0.540496 / 0.293841 (0.246656) | 0.061110 / 0.128546 (-0.067437) | 0.021443 / 0.075646 (-0.054203) | 0.395789 / 0.419271 (-0.023482) | 0.076727 / 0.043533 (0.033194) | 0.427808 / 0.255139 (0.172669) | 0.519672 / 0.283200 (0.236473) | 0.041607 / 0.141683 (-0.100076) | 2.098675 / 1.452155 (0.646520) | 2.175123 / 1.492716 (0.682407) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.275784 / 0.018006 (0.257777) | 0.707103 / 0.000490 (0.706613) | 0.011524 / 0.000200 (0.011324) | 0.000390 / 0.000054 (0.000336) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032897 / 0.037411 (-0.004514) | 0.123239 / 0.014526 (0.108713) | 0.151815 / 0.176557 (-0.024741) | 0.214790 / 0.737135 (-0.522345) | 0.139166 / 0.296338 (-0.157173) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.740662 / 0.215209 (0.525453) | 7.540376 / 2.077655 (5.462721) | 3.168207 / 1.504120 (1.664087) | 2.745663 / 1.541195 (1.204468) | 2.714020 / 1.468490 (1.245530) | 1.182632 / 4.584777 (-3.402145) | 6.365807 / 3.745712 (2.620095) | 6.317228 / 5.269862 (1.047366) | 4.061107 / 4.565676 (-0.504569) | 0.146939 / 0.424275 (-0.277336) | 0.011765 / 0.007607 (0.004158) | 0.910564 / 0.226044 (0.684519) | 9.020618 / 2.268929 (6.751689) | 4.180748 / 55.444624 (-51.263876) | 3.290257 / 6.876477 (-3.586220) | 3.363172 / 2.142072 (1.221099) | 1.239142 / 4.805227 (-3.566086) | 0.294965 / 6.500664 (-6.205699) | 0.088520 / 0.075469 (0.013051) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.867528 / 1.841788 (0.025741) | 29.494058 / 8.074308 (21.419750) | 31.386703 / 10.191392 (21.195311) | 0.302488 / 0.680424 (-0.377936) | 0.036116 / 0.534201 (-0.498085) | 0.622112 / 0.579283 (0.042829) | 0.775658 / 0.434364 (0.341294) | 0.632452 / 0.540337 (0.092115) | 0.909424 / 1.386936 (-0.477512) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.016002 / 0.011353 (0.004649) | 0.007007 / 0.011008 (-0.004002) | 0.100463 / 0.038508 (0.061955) | 0.124423 / 0.023109 (0.101314) | 0.556014 / 0.275898 (0.280116) | 0.600909 / 0.323480 (0.277429) | 0.007272 / 0.007986 (-0.000714) | 0.006743 / 0.004328 (0.002415) | 0.088575 / 0.004250 (0.084324) | 0.066003 / 0.037052 (0.028951) | 0.580080 / 0.258489 (0.321591) | 0.655567 / 0.293841 (0.361726) | 0.065295 / 0.128546 (-0.063252) | 0.021105 / 0.075646 (-0.054541) | 0.120044 / 0.419271 (-0.299227) | 0.081133 / 0.043533 (0.037600) | 0.570322 / 0.255139 (0.315183) | 0.581134 / 0.283200 (0.297934) | 0.046298 / 0.141683 (-0.095385) | 2.113200 / 1.452155 (0.661045) | 2.344187 / 1.492716 (0.851471) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.284517 / 0.018006 (0.266511) | 0.611834 / 0.000490 (0.611345) | 0.005581 / 0.000200 (0.005381) | 0.000153 / 0.000054 (0.000098) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.042162 / 0.037411 (0.004750) | 0.114496 / 0.014526 (0.099970) | 0.134034 / 0.176557 (-0.042523) | 0.201649 / 0.737135 (-0.535486) | 0.143235 / 0.296338 (-0.153103) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.764863 / 0.215209 (0.549654) | 7.603076 / 2.077655 (5.525421) | 3.318911 / 1.504120 (1.814791) | 2.939815 / 1.541195 (1.398620) | 2.870911 / 1.468490 (1.402421) | 1.171978 / 4.584777 (-3.412799) | 6.479933 / 3.745712 (2.734221) | 5.944387 / 5.269862 (0.674526) | 4.282625 / 4.565676 (-0.283051) | 0.123672 / 0.424275 (-0.300603) | 0.009666 / 0.007607 (0.002059) | 0.870683 / 0.226044 (0.644638) | 9.187788 / 2.268929 (6.918859) | 4.431818 / 55.444624 (-51.012807) | 3.460457 / 6.876477 (-3.416020) | 3.708198 / 2.142072 (1.566126) | 1.353673 / 4.805227 (-3.451554) | 0.264274 / 6.500664 (-6.236390) | 0.074943 / 0.075469 (-0.000526) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 2.073810 / 1.841788 (0.232023) | 29.182464 / 8.074308 (21.108156) | 30.527040 / 10.191392 (20.335648) | 0.307561 / 0.680424 (-0.372863) | 0.047384 / 0.534201 (-0.486817) | 0.662760 / 0.579283 (0.083477) | 0.768321 / 0.434364 (0.333957) | 0.692296 / 0.540337 (0.151959) | 0.955197 / 1.386936 (-0.431739) |

![](https://cml.dev/watermark.png#1e82d6f017c7fc0ab6b65847c1e34772c880d3b7 "CML watermark")

Replace deprecated license_file in `setup.cfg`.

See: https://github.com/huggingface/datasets/actions/runs/6610930650/job/17953825724?pr=6331
```
      /tmp/pip-build-env-a51hls20/overlay/lib/python3.8/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
      !!

              ********************************************************************************
              The license_file parameter is deprecated, use license_files instead.

              By 2023-Oct-30, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.

              See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
              ********************************************************************************

      !!
```


Temporarily pin fsspec < 2023.10.0

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009605 / 0.011353 (-0.001747) | 0.004864 / 0.011008 (-0.006144) | 0.114605 / 0.038508 (0.076097) | 0.090874 / 0.023109 (0.067765) | 0.429203 / 0.275898 (0.153305) | 0.489888 / 0.323480 (0.166408) | 0.006542 / 0.007986 (-0.001443) | 0.004585 / 0.004328 (0.000257) | 0.090251 / 0.004250 (0.086001) | 0.066612 / 0.037052 (0.029560) | 0.437491 / 0.258489 (0.179002) | 0.515196 / 0.293841 (0.221355) | 0.047756 / 0.128546 (-0.080791) | 0.013587 / 0.075646 (-0.062059) | 0.376960 / 0.419271 (-0.042311) | 0.069701 / 0.043533 (0.026168) | 0.430850 / 0.255139 (0.175711) | 0.475061 / 0.283200 (0.191861) | 0.034800 / 0.141683 (-0.106883) | 1.799947 / 1.452155 (0.347793) | 1.941863 / 1.492716 (0.449147) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.316685 / 0.018006 (0.298679) | 0.595098 / 0.000490 (0.594608) | 0.015447 / 0.000200 (0.015247) | 0.000463 / 0.000054 (0.000409) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.039364 / 0.037411 (0.001953) | 0.091295 / 0.014526 (0.076769) | 0.109380 / 0.176557 (-0.067177) | 0.185454 / 0.737135 (-0.551681) | 0.104476 / 0.296338 (-0.191862) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.626291 / 0.215209 (0.411082) | 5.869948 / 2.077655 (3.792293) | 2.466267 / 1.504120 (0.962147) | 2.183572 / 1.541195 (0.642377) | 2.208286 / 1.468490 (0.739796) | 0.817175 / 4.584777 (-3.767602) | 5.255141 / 3.745712 (1.509429) | 4.878668 / 5.269862 (-0.391193) | 2.917020 / 4.565676 (-1.648657) | 0.104995 / 0.424275 (-0.319280) | 0.008687 / 0.007607 (0.001080) | 0.678993 / 0.226044 (0.452948) | 7.004983 / 2.268929 (4.736054) | 3.444040 / 55.444624 (-52.000584) | 2.745075 / 6.876477 (-4.131402) | 2.720151 / 2.142072 (0.578078) | 0.995803 / 4.805227 (-3.809424) | 0.205928 / 6.500664 (-6.294736) | 0.077053 / 0.075469 (0.001584) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.587354 / 1.841788 (-0.254434) | 23.843227 / 8.074308 (15.768919) | 21.355771 / 10.191392 (11.164379) | 0.225593 / 0.680424 (-0.454831) | 0.029054 / 0.534201 (-0.505147) | 0.469676 / 0.579283 (-0.109607) | 0.582619 / 0.434364 (0.148255) | 0.576932 / 0.540337 (0.036594) | 0.946182 / 1.386936 (-0.440754) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009819 / 0.011353 (-0.001534) | 0.005562 / 0.011008 (-0.005446) | 0.075512 / 0.038508 (0.037004) | 0.084294 / 0.023109 (0.061185) | 0.549516 / 0.275898 (0.273618) | 0.550364 / 0.323480 (0.226884) | 0.006603 / 0.007986 (-0.001383) | 0.004587 / 0.004328 (0.000259) | 0.084040 / 0.004250 (0.079789) | 0.066815 / 0.037052 (0.029762) | 0.549224 / 0.258489 (0.290735) | 0.556213 / 0.293841 (0.262372) | 0.048538 / 0.128546 (-0.080008) | 0.014050 / 0.075646 (-0.061596) | 0.088955 / 0.419271 (-0.330317) | 0.062393 / 0.043533 (0.018860) | 0.528770 / 0.255139 (0.273631) | 0.564854 / 0.283200 (0.281655) | 0.033976 / 0.141683 (-0.107707) | 1.858558 / 1.452155 (0.406403) | 1.894616 / 1.492716 (0.401899) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.378597 / 0.018006 (0.360591) | 0.650586 / 0.000490 (0.650097) | 0.033179 / 0.000200 (0.032979) | 0.000477 / 0.000054 (0.000423) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031779 / 0.037411 (-0.005632) | 0.103393 / 0.014526 (0.088867) | 0.119810 / 0.176557 (-0.056747) | 0.192188 / 0.737135 (-0.544948) | 0.114545 / 0.296338 (-0.181794) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.623571 / 0.215209 (0.408362) | 6.350249 / 2.077655 (4.272594) | 3.207773 / 1.504120 (1.703653) | 2.861118 / 1.541195 (1.319923) | 2.864445 / 1.468490 (1.395955) | 0.827451 / 4.584777 (-3.757326) | 5.323860 / 3.745712 (1.578148) | 4.569197 / 5.269862 (-0.700665) | 2.967595 / 4.565676 (-1.598081) | 0.090926 / 0.424275 (-0.333349) | 0.007820 / 0.007607 (0.000213) | 0.731610 / 0.226044 (0.505565) | 7.342651 / 2.268929 (5.073723) | 3.781727 / 55.444624 (-51.662897) | 3.222100 / 6.876477 (-3.654377) | 3.546145 / 2.142072 (1.404073) | 1.030500 / 4.805227 (-3.774728) | 0.226563 / 6.500664 (-6.274101) | 0.078633 / 0.075469 (0.003164) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.733677 / 1.841788 (-0.108111) | 24.650616 / 8.074308 (16.576308) | 22.033745 / 10.191392 (11.842353) | 0.211055 / 0.680424 (-0.469369) | 0.031658 / 0.534201 (-0.502543) | 0.467190 / 0.579283 (-0.112094) | 0.598303 / 0.434364 (0.163939) | 0.569318 / 0.540337 (0.028981) | 0.825984 / 1.386936 (-0.560952) |

![](https://cml.dev/watermark.png#7eef7749031232d0b29f7ca10e3fa9f997b19ef7 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006873 / 0.011353 (-0.004479) | 0.004174 / 0.011008 (-0.006835) | 0.085874 / 0.038508 (0.047366) | 0.074207 / 0.023109 (0.051098) | 0.307342 / 0.275898 (0.031444) | 0.339972 / 0.323480 (0.016493) | 0.005522 / 0.007986 (-0.002463) | 0.003576 / 0.004328 (-0.000753) | 0.065680 / 0.004250 (0.061430) | 0.056274 / 0.037052 (0.019222) | 0.313121 / 0.258489 (0.054632) | 0.364699 / 0.293841 (0.070858) | 0.031297 / 0.128546 (-0.097249) | 0.008652 / 0.075646 (-0.066994) | 0.288431 / 0.419271 (-0.130840) | 0.053081 / 0.043533 (0.009548) | 0.309076 / 0.255139 (0.053937) | 0.329251 / 0.283200 (0.046052) | 0.024840 / 0.141683 (-0.116843) | 1.484155 / 1.452155 (0.032001) | 1.598665 / 1.492716 (0.105949) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.270933 / 0.018006 (0.252927) | 0.565867 / 0.000490 (0.565377) | 0.006964 / 0.000200 (0.006764) | 0.000298 / 0.000054 (0.000244) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028393 / 0.037411 (-0.009018) | 0.081756 / 0.014526 (0.067230) | 0.095733 / 0.176557 (-0.080823) | 0.152426 / 0.737135 (-0.584710) | 0.096655 / 0.296338 (-0.199683) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.403992 / 0.215209 (0.188783) | 4.027230 / 2.077655 (1.949576) | 2.031102 / 1.504120 (0.526982) | 1.843727 / 1.541195 (0.302532) | 1.898342 / 1.468490 (0.429852) | 0.479186 / 4.584777 (-4.105591) | 3.488153 / 3.745712 (-0.257559) | 3.523953 / 5.269862 (-1.745909) | 2.078392 / 4.565676 (-2.487284) | 0.056104 / 0.424275 (-0.368171) | 0.007368 / 0.007607 (-0.000239) | 0.479630 / 0.226044 (0.253585) | 4.787400 / 2.268929 (2.518471) | 2.488268 / 55.444624 (-52.956356) | 2.229955 / 6.876477 (-4.646522) | 2.260468 / 2.142072 (0.118396) | 0.587934 / 4.805227 (-4.217294) | 0.147124 / 6.500664 (-6.353540) | 0.059954 / 0.075469 (-0.015515) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.283155 / 1.841788 (-0.558632) | 19.013574 / 8.074308 (10.939266) | 13.915188 / 10.191392 (3.723796) | 0.174101 / 0.680424 (-0.506323) | 0.018172 / 0.534201 (-0.516029) | 0.390322 / 0.579283 (-0.188961) | 0.405493 / 0.434364 (-0.028871) | 0.456914 / 0.540337 (-0.083424) | 0.635213 / 1.386936 (-0.751723) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006622 / 0.011353 (-0.004731) | 0.003997 / 0.011008 (-0.007011) | 0.064542 / 0.038508 (0.026034) | 0.074165 / 0.023109 (0.051056) | 0.392285 / 0.275898 (0.116387) | 0.423522 / 0.323480 (0.100042) | 0.006361 / 0.007986 (-0.001625) | 0.003463 / 0.004328 (-0.000866) | 0.064891 / 0.004250 (0.060641) | 0.058485 / 0.037052 (0.021433) | 0.425217 / 0.258489 (0.166728) | 0.435907 / 0.293841 (0.142066) | 0.031501 / 0.128546 (-0.097045) | 0.008575 / 0.075646 (-0.067071) | 0.072094 / 0.419271 (-0.347178) | 0.047904 / 0.043533 (0.004371) | 0.397174 / 0.255139 (0.142035) | 0.417940 / 0.283200 (0.134741) | 0.023324 / 0.141683 (-0.118358) | 1.517245 / 1.452155 (0.065090) | 1.586497 / 1.492716 (0.093781) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.268311 / 0.018006 (0.250305) | 0.561118 / 0.000490 (0.560628) | 0.004352 / 0.000200 (0.004152) | 0.000095 / 0.000054 (0.000041) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033039 / 0.037411 (-0.004373) | 0.091596 / 0.014526 (0.077071) | 0.111520 / 0.176557 (-0.065036) | 0.161755 / 0.737135 (-0.575381) | 0.107681 / 0.296338 (-0.188657) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.427170 / 0.215209 (0.211961) | 4.252648 / 2.077655 (2.174994) | 2.257623 / 1.504120 (0.753503) | 2.098446 / 1.541195 (0.557251) | 2.128544 / 1.468490 (0.660054) | 0.496639 / 4.584777 (-4.088138) | 3.593385 / 3.745712 (-0.152328) | 3.396367 / 5.269862 (-1.873494) | 2.073369 / 4.565676 (-2.492308) | 0.058386 / 0.424275 (-0.365889) | 0.007515 / 0.007607 (-0.000093) | 0.502358 / 0.226044 (0.276313) | 5.015224 / 2.268929 (2.746296) | 2.735740 / 55.444624 (-52.708885) | 2.388368 / 6.876477 (-4.488109) | 2.682857 / 2.142072 (0.540785) | 0.595003 / 4.805227 (-4.210225) | 0.135419 / 6.500664 (-6.365245) | 0.062824 / 0.075469 (-0.012645) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.367507 / 1.841788 (-0.474281) | 19.569288 / 8.074308 (11.494979) | 14.748693 / 10.191392 (4.557301) | 0.198659 / 0.680424 (-0.481765) | 0.020954 / 0.534201 (-0.513247) | 0.414858 / 0.579283 (-0.164426) | 0.421226 / 0.434364 (-0.013138) | 0.477774 / 0.540337 (-0.062563) | 0.676173 / 1.386936 (-0.710763) |

![](https://cml.dev/watermark.png#87ed467af77d87ad7e38279cf9a24c341545cbac "CML watermark")

Temporarily pin fsspec < 2023.10.0 until permanent solution is found.

Hot fix #6330.

See: https://github.com/huggingface/datasets/actions/runs/6610904287/job/17953774987
```
...
ERROR tests/test_iterable_dataset.py::test_iterable_dataset_from_file - NotImplementedError: Loading a dataset cached in a LocalFileSystem is not supported.
= 373 failed, 2055 passed, 17 skipped, 8 warnings, 6 errors in 228.14s (0:03:48) =
```


Latest fsspec==2023.10.0 issue with streaming datasets

I also encountered a similar error below.
Appreciate the team could shed some light on this issue.

```
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
[/home/ubuntu/work/EveryDream2trainer/prepare_dataset.ipynb](https://vscode-remote+ssh-002dremote-002braspberry-002dg5-002e4x.vscode-resource.vscode-cdn.net/home/ubuntu/work/EveryDream2trainer/prepare_dataset.ipynb) Cell 1 line 4
      [1](vscode-notebook-cell://ssh-remote%2Braspberry-g5.4x/home/ubuntu/work/EveryDream2trainer/prepare_dataset.ipynb#W0sdnNjb2RlLXJlbW90ZQ%3D%3D?line=0) from datasets import load_dataset, load_dataset
      [3](vscode-notebook-cell://ssh-remote%2Braspberry-g5.4x/home/ubuntu/work/EveryDream2trainer/prepare_dataset.ipynb#W0sdnNjb2RlLXJlbW90ZQ%3D%3D?line=2) # ds = load_dataset("parquet", data_dir="/home/ubuntu/work/EveryDream2trainer/datasets/monse_v1/data")
----> [4](vscode-notebook-cell://ssh-remote%2Braspberry-g5.4x/home/ubuntu/work/EveryDream2trainer/prepare_dataset.ipynb#W0sdnNjb2RlLXJlbW90ZQ%3D%3D?line=3) ds = load_dataset("Raspberry-ai/monse-v1")

File [/opt/conda/envs/everydream/lib/python3.10/site-packages/datasets/load.py:1804](https://vscode-remote+ssh-002dremote-002braspberry-002dg5-002e4x.vscode-resource.vscode-cdn.net/opt/conda/envs/everydream/lib/python3.10/site-packages/datasets/load.py:1804), in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, num_proc, storage_options, **config_kwargs)
   1800 # Build dataset for splits
   1801 keep_in_memory = (
   1802     keep_in_memory if keep_in_memory is not None else is_small_dataset(builder_instance.info.dataset_size)
   1803 )
-> 1804 ds = builder_instance.as_dataset(split=split, verification_mode=verification_mode, in_memory=keep_in_memory)
   1805 # Rename and cast features to match task schema
   1806 if task is not None:

File [/opt/conda/envs/everydream/lib/python3.10/site-packages/datasets/builder.py:1108](https://vscode-remote+ssh-002dremote-002braspberry-002dg5-002e4x.vscode-resource.vscode-cdn.net/opt/conda/envs/everydream/lib/python3.10/site-packages/datasets/builder.py:1108), in DatasetBuilder.as_dataset(self, split, run_post_process, verification_mode, ignore_verifications, in_memory)
   1106 is_local = not is_remote_filesystem(self._fs)
   1107 if not is_local:
-> 1108     raise NotImplementedError(f"Loading a dataset cached in a {type(self._fs).__name__} is not supported.")
   1109 if not os.path.exists(self._output_dir):
   1110     raise FileNotFoundError(
   1111         f"Dataset {self.name}: could not find data in {self._output_dir}. Please make sure to call "
   1112         "builder.download_and_prepare(), or use "
   1113         "datasets.load_dataset() before trying to access the Dataset object."
   1114     )

NotImplementedError: Loading a dataset cached in a LocalFileSystem is not supported.
```

Code to reproduce the issue:

```
from datasets import load_dataset

ds = load_dataset("Raspberry-ai/monse-v1")
```

Dependencies:
```
Package                   Version
------------------------- ------------
absl-py                   2.0.0
accelerate                0.23.0
aiohttp                   3.8.4
aiosignal                 1.3.1
antlr4-python3-runtime    4.9.3
anyio                     4.0.0
appdirs                   1.4.4
argon2-cffi               23.1.0
argon2-cffi-bindings      21.2.0
arrow                     1.3.0
asttokens                 2.4.0
async-lru                 2.0.4
async-timeout             4.0.3
attrs                     23.1.0
Babel                     2.13.0
backcall                  0.2.0
beautifulsoup4            4.12.2
bitsandbytes              0.41.1
bleach                    6.1.0
braceexpand               0.1.7
cachetools                5.3.1
certifi                   2023.7.22
cffi                      1.16.0
charset-normalizer        3.3.1
click                     8.1.7
cmake                     3.27.7
colorama                  0.4.6
comm                      0.1.4
compel                    1.1.6
datasets                  2.11.0
debugpy                   1.8.0
decorator                 5.1.1
defusedxml                0.7.1
diffusers                 0.18.0
dill                      0.3.6
docker-pycreds            0.4.0
dowg                      0.3.1
einops                    0.7.0
einops-exts               0.0.4
exceptiongroup            1.1.3
executing                 2.0.0
fastjsonschema            2.18.1
filelock                  3.12.4
fqdn                      1.5.1
frozenlist                1.4.0
fsspec                    2023.10.0
ftfy                      6.1.1
gitdb                     4.0.11
GitPython                 3.1.40
google-auth               2.23.3
google-auth-oauthlib      1.1.0
grpcio                    1.59.0
huggingface-hub           0.18.0
idna                      3.4
importlib-metadata        6.8.0
inflection                0.5.1
ipykernel                 6.25.2
ipython                   8.16.1
isoduration               20.11.0
jedi                      0.19.1
Jinja2                    3.1.2
joblib                    1.3.2
json5                     0.9.14
jsonpointer               2.4
jsonschema                4.19.1
jsonschema-specifications 2023.7.1
jupyter_client            8.4.0
jupyter_core              5.4.0
jupyter-events            0.8.0
jupyter-lsp               2.2.0
jupyter_server            2.8.0
jupyter_server_terminals  0.4.4
jupyterlab                4.0.7
jupyterlab-pygments       0.2.2
jupyterlab_server         2.25.0
lightning-utilities       0.9.0
lion-pytorch              0.1.2
lit                       17.0.3
Markdown                  3.5
MarkupSafe                2.1.3
matplotlib-inline         0.1.6
mistune                   3.0.2
more-itertools            10.1.0
mpmath                    1.3.0
multidict                 6.0.4
multiprocess              0.70.14
mypy-extensions           1.0.0
nbclient                  0.8.0
nbconvert                 7.9.2
nbformat                  5.9.2
nest-asyncio              1.5.8
networkx                  3.2
nltk                      3.8.1
notebook_shim             0.2.3
numpy                     1.23.5
oauthlib                  3.2.2
omegaconf                 2.2.3
open-clip-torch           2.22.0
open-flamingo             2.0.0
overrides                 7.4.0
packaging                 23.2
pandas                    2.1.1
pandocfilters             1.5.0
parso                     0.8.3
pathtools                 0.1.2
pexpect                   4.8.0
pickleshare               0.7.5
Pillow                    10.1.0
pip                       23.3.1
platformdirs              3.11.0
prometheus-client         0.17.1
prompt-toolkit            3.0.39
protobuf                  3.20.1
psutil                    5.9.6
ptyprocess                0.7.0
pure-eval                 0.2.2
pyarrow                   13.0.0
pyasn1                    0.5.0
pyasn1-modules            0.3.0
pycparser                 2.21
pyDeprecate               0.3.2
Pygments                  2.16.1
pynvml                    11.4.1
pyparsing                 3.1.1
pyre-extensions           0.0.29
python-dateutil           2.8.2
python-json-logger        2.0.7
pytorch-lightning         1.6.5
pytz                      2023.3.post1
PyYAML                    6.0.1
pyzmq                     25.1.1
referencing               0.30.2
regex                     2023.10.3
requests                  2.31.0
requests-oauthlib         1.3.1
responses                 0.18.0
rfc3339-validator         0.1.4
rfc3986-validator         0.1.1
rpds-py                   0.10.6
rsa                       4.9
safetensors               0.4.0
scipy                     1.11.3
Send2Trash                1.8.2
sentencepiece             0.1.98
sentry-sdk                1.32.0
setproctitle              1.3.3
setuptools                68.2.2
six                       1.16.0
smmap                     5.0.1
sniffio                   1.3.0
soupsieve                 2.5
stack-data                0.6.3
sympy                     1.12
tensorboard               2.15.0
tensorboard-data-server   0.7.1
terminado                 0.17.1
timm                      0.9.8
tinycss2                  1.2.1
tokenizers                0.13.3
tomli                     2.0.1
torch                     2.0.1+cu118
torchmetrics              1.2.0
torchvision               0.15.2+cu118
tornado                   6.3.3
tqdm                      4.66.1
traitlets                 5.11.2
transformers              4.29.2
triton                    2.0.0
types-python-dateutil     2.8.19.14
typing_extensions         4.8.0
typing-inspect            0.9.0
tzdata                    2023.3
uri-template              1.3.0
urllib3                   2.0.7
wandb                     0.15.12
wcwidth                   0.2.8
webcolors                 1.13
webdataset                0.2.62
webencodings              0.5.1
websocket-client          1.6.4
Werkzeug                  3.0.0
wheel                     0.41.2
xformers                  0.0.20
xxhash                    3.4.1
yarl                      1.9.2
zipp                      3.17.0
```

@humpydonkey FWIW setting fsspec down to 2023.9.2 fixed the issue

`pip install fsspec==2023.9.2`

Thanks for reporting and for the investigation, @ZachNagengast! :hugs:

We are investigating the root cause of the issue. In the meantime, we are going to pin fsspec < 2023.10.0.

You can also update `datasets`:

```
pip install -U datasets
```

It will also update `fsspec` to use the right version

This seems to work fine in 2.19.0. Hopefully it will not break again

### Describe the bug

Loading a streaming dataset with this version of fsspec fails with the following error:

`NotImplementedError: Loading a streaming dataset cached in a LocalFileSystem is not supported yet.`

I suspect the issue is with this PR

https://github.com/fsspec/filesystem_spec/pull/1381

### Steps to reproduce the bug

1. Upgrade fsspec to version `2023.10.0`
2. Attempt to load a streaming dataset e.g. `load_dataset("laion/gpt4v-emotion-dataset", split="train", streaming=True)`
3. Observe the following exception:

```
  File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/datasets/load.py", line 2146, in load_dataset
    return builder_instance.as_streaming_dataset(split=split)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/datasets/builder.py", line 1318, in as_streaming_dataset
    raise NotImplementedError(
NotImplementedError: Loading a streaming dataset cached in a LocalFileSystem is not supported yet.
```

### Expected behavior

Should stream the dataset as normal.

### Environment info

datasets@main
fsspec==2023.10.0


شبکه های متن به گفتار ابتدا متن داده شده را به بازنمایی میانی


FileNotFoundError when trying to load the downloaded dataset with `load_dataset(..., streaming=True)`

You can clone the `togethercomputer/RedPajama-Data-1T-Sample` repo and load the dataset with `load_dataset("path/to/cloned_repo")` to use it offline.

@mariosasko Thank you for your kind reply! I'll try it as a workaround.
Does that mean that currently it's not supported to simply load with a short name?

It is, but manually downloading repo files to the cache can easily lead to failure (the HF cache is not meant to be modified by a user besides deleting the files 🙂), as in your case. Hence, the clone + `load_dataset("path/to/cloned_repo")` workflow should be used instead.

### Describe the bug

Hi, I'm trying to load the dataset `togethercomputer/RedPajama-Data-1T-Sample` with `load_dataset` in streaming mode, i.e., `streaming=True`, but `FileNotFoundError` occurs.

### Steps to reproduce the bug

I've downloaded the dataset and save it to the cache dir in advance. My hope is loading the files in offline environment and without taking too much hours to prepross the entire data before running into the training process.
So I try the following code to load the files streamingly
```py
dataset = load_dataset('togethercomputer/RedPajama-Data-1T-Sample', streaming=True)
print(next(iter(dataset['train'])))
```
Sadly, it raises the following:
```
FileNotFoundError: [Errno 2] No such file or directory: 'CURRENT_CODE_PATH/arxiv_sample.jsonl'
```
I've noticed that the dataset can be properly found in the begining
```
Using the latest cached version of the module from /root/.cache/huggingface/modules/datasets_modules/datasets/togethercomputer--RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 (last modified on Sat Oct 21 20:12:57 2023) since it couldn't be found locally at togethercomputer/RedPajama-Data-1T-Sample., or remotely on the Hugging Face Hub.
```
But it seems that the paths couldn't be properly parsed when loading iteratively.

How should I fix this error. I've tried specifying `data_files` or `data_dir` as `.../arxiv_sample.jsonl` but none of them works.

Thanks.

### Expected behavior

Properly load the dataset.

### Environment info

`datasets==2.14.5`


Create battery_analysis.py


Conversion to Arrow fails due to wrong type heuristic

Unlike Pandas, Arrow is strict with types, so converting the problematic strings to ints (or ints to strings) to ensure all the values have the same type is the only fix.

JSON support has been requested in Arrow [here](https://github.com/apache/arrow/issues/32538), but I don't expect this to be implemented soon.

Also, this type could be represented with the Arrow Union type. However, due to low usage, the Union type has limited support in the Arrow ecosystem (e.g., IIRC Parquet still does not support it). So, we should probably wait a bit more before adding support for it in `datasets`

> Unlike Pandas, Arrow is strict with types, so converting the problematic strings to ints (or ints to strings) to ensure all the values have the same type is the only fix.
>
> JSON support has been requested in Arrow [here](https://github.com/apache/arrow/issues/32538), but I don't expect this to be implemented soon.
>
> Also, this type could be represented with the Arrow Union type. However, due to low usage, the Union type has limited support in the Arrow ecosystem (e.g., IIRC Parquet still does not support it). So, we should probably wait a bit more before adding support for it in `datasets`

Ok many thanks, I was able to mitigate the problem by manually checking and converting all problematic fields now.

### Describe the bug

I have a list of dictionaries with valid/JSON-serializable values.

One key is the denominator for a paragraph. In 99.9% of cases its a number, but there are some occurences of '1a', '2b' and so on.

If trying to convert this list to a dataset with `Dataset.from_list()`, I always get
`ArrowInvalid: Could not convert '1' with type str: tried to convert to int64`, presumably because pyarrow tries to convert the keys to integers.

Is there any way to circumvent this and fix dtypes? I didn't find anything in the documentation.

### Steps to reproduce the bug

* create a list of dicts with one key being a string of an integer for the first few thousand occurences and try to convert to dataset.

### Expected behavior

There shouldn't be an error (e.g. some flag to turn off automatic str to numeric conversion).

### Environment info

- `datasets` version: 2.14.5
- Platform: Linux-5.15.0-84-generic-x86_64-with-glibc2.35
- Python version: 3.9.18
- Huggingface_hub version: 0.17.3
- PyArrow version: 13.0.0
- Pandas version: 2.1.1


Loading dataset from large GCS bucket very slow since 2.14

### Describe the bug

Since updating to >2.14 we have very slow access to our parquet files on GCS when loading a dataset (>30 min vs 3s). Our GCS bucket has many objects and resolving globs is very slow. I could track down the problem to this change:
https://github.com/huggingface/datasets/blame/bade7af74437347a760830466eb74f7a8ce0d799/src/datasets/data_files.py#L348
The underlying implementation with gcsfs is really slow. Could you go back to the old way if we are simply giving the parquet files and no glob pattern?

Thank you.

### Steps to reproduce the bug

Load a dataset from a GCS bucket that has many files.

### Expected behavior

Used to be fast (3s) in 2.13

### Environment info

datasets==2.14.5


Fix regex `get_data_files` formatting for base paths

> The reason why I used the the glob_pattern_to_regex in the entire pattern is because otherwise I got an error for Windows local paths: a base_path like 'C:\\Users\\runneradmin... made the function string_to_dict raise re.error: incomplete escape \U at position 2

What is the expected inputs and outputs for the windows `base_path`

> That issue was fixed once we pass the base_path as POSIX.

I'm not sure what you meant by that, are there still changes needed?

We took the liberty of continuing this PR to include it in today's patch release :)
I hope you don't mind

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007109 / 0.011353 (-0.004244) | 0.004209 / 0.011008 (-0.006799) | 0.097401 / 0.038508 (0.058892) | 0.079532 / 0.023109 (0.056423) | 0.341300 / 0.275898 (0.065402) | 0.402165 / 0.323480 (0.078685) | 0.005838 / 0.007986 (-0.002148) | 0.003310 / 0.004328 (-0.001018) | 0.072804 / 0.004250 (0.068553) | 0.059418 / 0.037052 (0.022366) | 0.339277 / 0.258489 (0.080788) | 0.418495 / 0.293841 (0.124654) | 0.035975 / 0.128546 (-0.092571) | 0.008101 / 0.075646 (-0.067546) | 0.339236 / 0.419271 (-0.080035) | 0.059326 / 0.043533 (0.015794) | 0.326880 / 0.255139 (0.071741) | 0.393614 / 0.283200 (0.110414) | 0.025830 / 0.141683 (-0.115852) | 1.657726 / 1.452155 (0.205571) | 1.817250 / 1.492716 (0.324534) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.256015 / 0.018006 (0.238008) | 0.482447 / 0.000490 (0.481957) | 0.012166 / 0.000200 (0.011966) | 0.000343 / 0.000054 (0.000288) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029898 / 0.037411 (-0.007514) | 0.088218 / 0.014526 (0.073692) | 0.102353 / 0.176557 (-0.074203) | 0.165863 / 0.737135 (-0.571272) | 0.100342 / 0.296338 (-0.195996) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.429362 / 0.215209 (0.214153) | 4.147327 / 2.077655 (2.069672) | 2.014653 / 1.504120 (0.510533) | 1.824394 / 1.541195 (0.283199) | 1.936408 / 1.468490 (0.467917) | 0.542960 / 4.584777 (-4.041817) | 3.917215 / 3.745712 (0.171503) | 3.714825 / 5.269862 (-1.555036) | 2.180279 / 4.565676 (-2.385398) | 0.057808 / 0.424275 (-0.366467) | 0.008426 / 0.007607 (0.000819) | 0.472372 / 0.226044 (0.246327) | 4.879656 / 2.268929 (2.610728) | 2.602729 / 55.444624 (-52.841896) | 2.142593 / 6.876477 (-4.733884) | 2.206070 / 2.142072 (0.063997) | 0.635591 / 4.805227 (-4.169636) | 0.140928 / 6.500664 (-6.359736) | 0.065119 / 0.075469 (-0.010350) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.455909 / 1.841788 (-0.385879) | 20.803592 / 8.074308 (12.729284) | 14.788713 / 10.191392 (4.597321) | 0.170546 / 0.680424 (-0.509878) | 0.021189 / 0.534201 (-0.513012) | 0.432368 / 0.579283 (-0.146915) | 0.444664 / 0.434364 (0.010300) | 0.517744 / 0.540337 (-0.022593) | 0.699265 / 1.386936 (-0.687671) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007592 / 0.011353 (-0.003760) | 0.004045 / 0.011008 (-0.006964) | 0.073434 / 0.038508 (0.034926) | 0.076962 / 0.023109 (0.053853) | 0.468873 / 0.275898 (0.192975) | 0.479968 / 0.323480 (0.156488) | 0.006270 / 0.007986 (-0.001716) | 0.003652 / 0.004328 (-0.000677) | 0.069893 / 0.004250 (0.065643) | 0.061902 / 0.037052 (0.024850) | 0.443379 / 0.258489 (0.184890) | 0.492627 / 0.293841 (0.198786) | 0.035967 / 0.128546 (-0.092579) | 0.009276 / 0.075646 (-0.066370) | 0.083060 / 0.419271 (-0.336212) | 0.050870 / 0.043533 (0.007337) | 0.438246 / 0.255139 (0.183107) | 0.472074 / 0.283200 (0.188874) | 0.023724 / 0.141683 (-0.117959) | 1.677178 / 1.452155 (0.225023) | 1.732273 / 1.492716 (0.239557) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.244693 / 0.018006 (0.226687) | 0.470067 / 0.000490 (0.469577) | 0.005574 / 0.000200 (0.005374) | 0.000105 / 0.000054 (0.000051) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036242 / 0.037411 (-0.001169) | 0.099166 / 0.014526 (0.084641) | 0.116785 / 0.176557 (-0.059772) | 0.174986 / 0.737135 (-0.562149) | 0.118130 / 0.296338 (-0.178209) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.475907 / 0.215209 (0.260698) | 4.708125 / 2.077655 (2.630470) | 2.600855 / 1.504120 (1.096735) | 2.446498 / 1.541195 (0.905303) | 2.538786 / 1.468490 (1.070296) | 0.566787 / 4.584777 (-4.017990) | 4.066187 / 3.745712 (0.320475) | 3.743632 / 5.269862 (-1.526229) | 2.337737 / 4.565676 (-2.227939) | 0.068402 / 0.424275 (-0.355873) | 0.008674 / 0.007607 (0.001067) | 0.593428 / 0.226044 (0.367384) | 5.840687 / 2.268929 (3.571759) | 3.194937 / 55.444624 (-52.249688) | 2.899033 / 6.876477 (-3.977444) | 2.977870 / 2.142072 (0.835797) | 0.683673 / 4.805227 (-4.121554) | 0.154933 / 6.500664 (-6.345731) | 0.071619 / 0.075469 (-0.003850) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.501895 / 1.841788 (-0.339893) | 21.709792 / 8.074308 (13.635484) | 15.679556 / 10.191392 (5.488164) | 0.188028 / 0.680424 (-0.492396) | 0.022555 / 0.534201 (-0.511646) | 0.439840 / 0.579283 (-0.139443) | 0.452140 / 0.434364 (0.017776) | 0.526421 / 0.540337 (-0.013916) | 0.731692 / 1.386936 (-0.655244) |

![](https://cml.dev/watermark.png#02ecc84a2e2ed664f574ccbcab0e525a7377a01d "CML watermark")

With this pr https://github.com/huggingface/datasets/pull/6309, it is formatting the entire base path into regex, which results in the undesired formatting error `doesn't match the pattern` because of the line in `glob_pattern_to_regex`: `.replace("//", "/")`:
- Input: `hf://datasets/...`
- Output: `hf:/datasets/...`

This fix will only convert the `split_pattern` to regex and keep the `base_path` unchanged.

cc @albertvillanova hopefully this still works with your implementation


Fix typos

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007809 / 0.011353 (-0.003544) | 0.004573 / 0.011008 (-0.006435) | 0.101201 / 0.038508 (0.062693) | 0.089703 / 0.023109 (0.066594) | 0.416502 / 0.275898 (0.140604) | 0.463352 / 0.323480 (0.139872) | 0.006101 / 0.007986 (-0.001885) | 0.003783 / 0.004328 (-0.000545) | 0.076531 / 0.004250 (0.072281) | 0.064017 / 0.037052 (0.026964) | 0.422453 / 0.258489 (0.163964) | 0.485926 / 0.293841 (0.192085) | 0.036797 / 0.128546 (-0.091749) | 0.010172 / 0.075646 (-0.065474) | 0.344442 / 0.419271 (-0.074829) | 0.062240 / 0.043533 (0.018707) | 0.422685 / 0.255139 (0.167546) | 0.451457 / 0.283200 (0.168257) | 0.027831 / 0.141683 (-0.113852) | 1.737187 / 1.452155 (0.285033) | 1.847631 / 1.492716 (0.354915) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.270336 / 0.018006 (0.252330) | 0.500540 / 0.000490 (0.500050) | 0.017042 / 0.000200 (0.016842) | 0.000704 / 0.000054 (0.000650) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033450 / 0.037411 (-0.003962) | 0.100314 / 0.014526 (0.085788) | 0.117216 / 0.176557 (-0.059340) | 0.182352 / 0.737135 (-0.554784) | 0.114903 / 0.296338 (-0.181436) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.458562 / 0.215209 (0.243353) | 4.570492 / 2.077655 (2.492837) | 2.230286 / 1.504120 (0.726167) | 2.032229 / 1.541195 (0.491034) | 2.130431 / 1.468490 (0.661941) | 0.563254 / 4.584777 (-4.021523) | 4.108455 / 3.745712 (0.362743) | 3.994059 / 5.269862 (-1.275802) | 2.424589 / 4.565676 (-2.141087) | 0.067534 / 0.424275 (-0.356741) | 0.008774 / 0.007607 (0.001167) | 0.546356 / 0.226044 (0.320312) | 5.527772 / 2.268929 (3.258843) | 2.934410 / 55.444624 (-52.510215) | 2.536871 / 6.876477 (-4.339605) | 2.598704 / 2.142072 (0.456632) | 0.676721 / 4.805227 (-4.128506) | 0.155904 / 6.500664 (-6.344760) | 0.073274 / 0.075469 (-0.002195) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.559170 / 1.841788 (-0.282618) | 23.228524 / 8.074308 (15.154216) | 16.743246 / 10.191392 (6.551854) | 0.184113 / 0.680424 (-0.496310) | 0.021804 / 0.534201 (-0.512397) | 0.466158 / 0.579283 (-0.113125) | 0.539911 / 0.434364 (0.105547) | 0.544377 / 0.540337 (0.004040) | 0.765779 / 1.386936 (-0.621157) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008249 / 0.011353 (-0.003104) | 0.004734 / 0.011008 (-0.006275) | 0.077083 / 0.038508 (0.038575) | 0.096959 / 0.023109 (0.073850) | 0.497501 / 0.275898 (0.221603) | 0.530687 / 0.323480 (0.207207) | 0.006379 / 0.007986 (-0.001607) | 0.003899 / 0.004328 (-0.000430) | 0.076165 / 0.004250 (0.071915) | 0.069406 / 0.037052 (0.032354) | 0.515847 / 0.258489 (0.257358) | 0.540639 / 0.293841 (0.246798) | 0.038334 / 0.128546 (-0.090213) | 0.010112 / 0.075646 (-0.065534) | 0.084918 / 0.419271 (-0.334353) | 0.056866 / 0.043533 (0.013333) | 0.495555 / 0.255139 (0.240416) | 0.518988 / 0.283200 (0.235789) | 0.028556 / 0.141683 (-0.113127) | 1.799320 / 1.452155 (0.347165) | 1.874647 / 1.492716 (0.381931) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.264283 / 0.018006 (0.246277) | 0.510278 / 0.000490 (0.509788) | 0.015219 / 0.000200 (0.015019) | 0.000160 / 0.000054 (0.000105) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.038462 / 0.037411 (0.001051) | 0.115420 / 0.014526 (0.100894) | 0.124250 / 0.176557 (-0.052306) | 0.187724 / 0.737135 (-0.549411) | 0.126674 / 0.296338 (-0.169664) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.499345 / 0.215209 (0.284136) | 4.983924 / 2.077655 (2.906269) | 2.705099 / 1.504120 (1.200980) | 2.516344 / 1.541195 (0.975149) | 2.621103 / 1.468490 (1.152613) | 0.583254 / 4.584777 (-4.001523) | 4.231215 / 3.745712 (0.485503) | 4.028326 / 5.269862 (-1.241536) | 2.459171 / 4.565676 (-2.106505) | 0.069194 / 0.424275 (-0.355081) | 0.008850 / 0.007607 (0.001243) | 0.593878 / 0.226044 (0.367834) | 5.926478 / 2.268929 (3.657549) | 3.287435 / 55.444624 (-52.157189) | 2.902104 / 6.876477 (-3.974372) | 3.151307 / 2.142072 (1.009234) | 0.696922 / 4.805227 (-4.108306) | 0.161140 / 6.500664 (-6.339524) | 0.073728 / 0.075469 (-0.001741) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.636456 / 1.841788 (-0.205331) | 23.884606 / 8.074308 (15.810298) | 17.180875 / 10.191392 (6.989483) | 0.176782 / 0.680424 (-0.503642) | 0.023731 / 0.534201 (-0.510470) | 0.475191 / 0.579283 (-0.104092) | 0.506603 / 0.434364 (0.072239) | 0.571976 / 0.540337 (0.031638) | 0.826935 / 1.386936 (-0.560002) |

![](https://cml.dev/watermark.png#2b19f6b30f49e09b0d1f0c4a38b10d76f35ac483 "CML watermark")


Dataset slice splits can't load training and validation at the same time

The expression "train+test" concatenates the splits.

The individual splits as separate datasets can be obtained as follows:
```python
train_ds, test_ds = load_dataset("<dataset_name>", split=["train", "test"])
train_10pct_ds, test_10pct_ds = load_dataset("<dataset_name>", split=["train[:10%]", "test[:%10]"])
```

### Describe the bug

According to the [documentation](https://huggingface.co/docs/datasets/v2.14.5/loading#slice-splits) is should be possible to run the following command:

`train_test_ds = datasets.load_dataset("bookcorpus", split="train+test")`

to load the train and test sets from the dataset.

However executing the equivalent code:

`speech_commands_v1 = load_dataset("superb", "ks", split="train+test")`

only yields the following output:

> Dataset({
>     features: ['file', 'audio', 'label'],
>     num_rows: 54175
> })

Where loading the dataset without the split argument yields:

> DatasetDict({
>     train: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 51094
>     })
>     validation: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 6798
>     })
>     test: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 3081
>     })
> })

Thus, the API seems to be broken in this regard.

This is a bit annoying since I want to be able to use the split argument with `split="train[:10%]+test[:10%]"` to have smaller dataset to work with when validating my model is working correctly.

### Steps to reproduce the bug

`speech_commands_v1 = load_dataset("superb", "ks", split="train+test")`

### Expected behavior

> DatasetDict({
>     train: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 51094
>     })
>     test: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 3081
>     })
> })

### Environment info

```
import datasets
print(datasets.__version__)
```
> 2.14.5

```
import sys
print(sys.version)
```
> 3.9.17 (main, Jul  5 2023, 20:41:20)
> [GCC 11.2.0]