Ignore CI lint rule violation in Pickler.memoize

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006536 / 0.011353 (-0.004817) | 0.003890 / 0.011008 (-0.007118) | 0.084044 / 0.038508 (0.045536) | 0.071893 / 0.023109 (0.048784) | 0.346926 / 0.275898 (0.071028) | 0.397487 / 0.323480 (0.074007) | 0.004065 / 0.007986 (-0.003921) | 0.003218 / 0.004328 (-0.001111) | 0.064670 / 0.004250 (0.060420) | 0.052414 / 0.037052 (0.015362) | 0.355413 / 0.258489 (0.096924) | 0.398894 / 0.293841 (0.105053) | 0.030763 / 0.128546 (-0.097783) | 0.008590 / 0.075646 (-0.067056) | 0.286857 / 0.419271 (-0.132415) | 0.051126 / 0.043533 (0.007593) | 0.346125 / 0.255139 (0.090986) | 0.395673 / 0.283200 (0.112474) | 0.025766 / 0.141683 (-0.115917) | 1.466238 / 1.452155 (0.014084) | 1.543117 / 1.492716 (0.050400) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.213210 / 0.018006 (0.195204) | 0.451981 / 0.000490 (0.451491) | 0.003784 / 0.000200 (0.003585) | 0.000096 / 0.000054 (0.000041) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027756 / 0.037411 (-0.009655) | 0.082446 / 0.014526 (0.067920) | 0.095414 / 0.176557 (-0.081142) | 0.151812 / 0.737135 (-0.585323) | 0.096296 / 0.296338 (-0.200042) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.383729 / 0.215209 (0.168520) | 3.835126 / 2.077655 (1.757471) | 1.891972 / 1.504120 (0.387852) | 1.719934 / 1.541195 (0.178739) | 1.899980 / 1.468490 (0.431490) | 0.488741 / 4.584777 (-4.096036) | 3.634120 / 3.745712 (-0.111592) | 3.243314 / 5.269862 (-2.026547) | 2.028382 / 4.565676 (-2.537294) | 0.057355 / 0.424275 (-0.366920) | 0.007717 / 0.007607 (0.000110) | 0.459835 / 0.226044 (0.233790) | 4.591793 / 2.268929 (2.322864) | 2.346861 / 55.444624 (-53.097764) | 2.067357 / 6.876477 (-4.809120) | 2.254954 / 2.142072 (0.112882) | 0.587016 / 4.805227 (-4.218211) | 0.133918 / 6.500664 (-6.366746) | 0.060311 / 0.075469 (-0.015158) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.250016 / 1.841788 (-0.591772) | 19.674333 / 8.074308 (11.600025) | 14.522764 / 10.191392 (4.331372) | 0.145741 / 0.680424 (-0.534683) | 0.018593 / 0.534201 (-0.515608) | 0.392833 / 0.579283 (-0.186450) | 0.408194 / 0.434364 (-0.026170) | 0.455164 / 0.540337 (-0.085174) | 0.622722 / 1.386936 (-0.764214) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006583 / 0.011353 (-0.004770) | 0.004008 / 0.011008 (-0.007000) | 0.064688 / 0.038508 (0.026180) | 0.074969 / 0.023109 (0.051860) | 0.360504 / 0.275898 (0.084606) | 0.396926 / 0.323480 (0.073446) | 0.005190 / 0.007986 (-0.002796) | 0.003363 / 0.004328 (-0.000966) | 0.064372 / 0.004250 (0.060122) | 0.054428 / 0.037052 (0.017376) | 0.361204 / 0.258489 (0.102715) | 0.400917 / 0.293841 (0.107077) | 0.031117 / 0.128546 (-0.097429) | 0.008406 / 0.075646 (-0.067241) | 0.069655 / 0.419271 (-0.349617) | 0.048582 / 0.043533 (0.005049) | 0.365396 / 0.255139 (0.110257) | 0.381344 / 0.283200 (0.098145) | 0.023809 / 0.141683 (-0.117874) | 1.472926 / 1.452155 (0.020772) | 1.547298 / 1.492716 (0.054582) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.276912 / 0.018006 (0.258906) | 0.449096 / 0.000490 (0.448607) | 0.018921 / 0.000200 (0.018721) | 0.000111 / 0.000054 (0.000056) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030237 / 0.037411 (-0.007174) | 0.088610 / 0.014526 (0.074084) | 0.101529 / 0.176557 (-0.075027) | 0.154070 / 0.737135 (-0.583065) | 0.103471 / 0.296338 (-0.192867) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.416047 / 0.215209 (0.200838) | 4.152374 / 2.077655 (2.074719) | 2.111181 / 1.504120 (0.607061) | 1.943582 / 1.541195 (0.402387) | 2.031729 / 1.468490 (0.563239) | 0.486740 / 4.584777 (-4.098037) | 3.631547 / 3.745712 (-0.114165) | 3.251202 / 5.269862 (-2.018660) | 2.041272 / 4.565676 (-2.524405) | 0.057287 / 0.424275 (-0.366988) | 0.007303 / 0.007607 (-0.000304) | 0.491027 / 0.226044 (0.264982) | 4.906757 / 2.268929 (2.637829) | 2.581694 / 55.444624 (-52.862931) | 2.250996 / 6.876477 (-4.625481) | 2.441771 / 2.142072 (0.299698) | 0.600714 / 4.805227 (-4.204514) | 0.133233 / 6.500664 (-6.367431) | 0.060856 / 0.075469 (-0.014613) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.340062 / 1.841788 (-0.501725) | 19.973899 / 8.074308 (11.899591) | 14.347381 / 10.191392 (4.155989) | 0.166651 / 0.680424 (-0.513773) | 0.018691 / 0.534201 (-0.515510) | 0.393580 / 0.579283 (-0.185703) | 0.409425 / 0.434364 (-0.024939) | 0.474409 / 0.540337 (-0.065929) | 0.649423 / 1.386936 (-0.737514) |

![](https://cml.dev/watermark.png#c5da68102297c3639207a7901952d2765a4cdb8b "CML watermark")

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006593 / 0.011353 (-0.004760) | 0.004123 / 0.011008 (-0.006885) | 0.084424 / 0.038508 (0.045916) | 0.076867 / 0.023109 (0.053758) | 0.309149 / 0.275898 (0.033251) | 0.348572 / 0.323480 (0.025092) | 0.005463 / 0.007986 (-0.002523) | 0.003440 / 0.004328 (-0.000889) | 0.064604 / 0.004250 (0.060353) | 0.053920 / 0.037052 (0.016868) | 0.345221 / 0.258489 (0.086732) | 0.363209 / 0.293841 (0.069368) | 0.031209 / 0.128546 (-0.097337) | 0.008690 / 0.075646 (-0.066956) | 0.288851 / 0.419271 (-0.130421) | 0.052239 / 0.043533 (0.008707) | 0.308643 / 0.255139 (0.053504) | 0.346407 / 0.283200 (0.063207) | 0.023935 / 0.141683 (-0.117748) | 1.469207 / 1.452155 (0.017052) | 1.532855 / 1.492716 (0.040138) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.290885 / 0.018006 (0.272879) | 0.580561 / 0.000490 (0.580071) | 0.004698 / 0.000200 (0.004498) | 0.000286 / 0.000054 (0.000231) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028015 / 0.037411 (-0.009396) | 0.081172 / 0.014526 (0.066646) | 0.096822 / 0.176557 (-0.079735) | 0.151355 / 0.737135 (-0.585781) | 0.098017 / 0.296338 (-0.198321) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.384069 / 0.215209 (0.168859) | 3.828635 / 2.077655 (1.750980) | 1.829311 / 1.504120 (0.325192) | 1.672520 / 1.541195 (0.131325) | 1.743944 / 1.468490 (0.275453) | 0.481594 / 4.584777 (-4.103183) | 3.556204 / 3.745712 (-0.189509) | 3.279499 / 5.269862 (-1.990363) | 2.033243 / 4.565676 (-2.532434) | 0.056525 / 0.424275 (-0.367750) | 0.007717 / 0.007607 (0.000109) | 0.466815 / 0.226044 (0.240771) | 4.657022 / 2.268929 (2.388094) | 2.438600 / 55.444624 (-53.006024) | 2.097999 / 6.876477 (-4.778478) | 2.263122 / 2.142072 (0.121049) | 0.636001 / 4.805227 (-4.169226) | 0.147727 / 6.500664 (-6.352937) | 0.059293 / 0.075469 (-0.016176) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.243111 / 1.841788 (-0.598677) | 19.558379 / 8.074308 (11.484071) | 14.141017 / 10.191392 (3.949625) | 0.169840 / 0.680424 (-0.510583) | 0.017912 / 0.534201 (-0.516289) | 0.391325 / 0.579283 (-0.187958) | 0.417169 / 0.434364 (-0.017195) | 0.457129 / 0.540337 (-0.083209) | 0.629907 / 1.386936 (-0.757029) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006687 / 0.011353 (-0.004666) | 0.004165 / 0.011008 (-0.006844) | 0.064738 / 0.038508 (0.026230) | 0.077286 / 0.023109 (0.054177) | 0.364236 / 0.275898 (0.088338) | 0.393228 / 0.323480 (0.069748) | 0.005451 / 0.007986 (-0.002535) | 0.003547 / 0.004328 (-0.000781) | 0.065761 / 0.004250 (0.061510) | 0.056526 / 0.037052 (0.019474) | 0.365523 / 0.258489 (0.107034) | 0.403331 / 0.293841 (0.109490) | 0.030900 / 0.128546 (-0.097646) | 0.008757 / 0.075646 (-0.066889) | 0.070961 / 0.419271 (-0.348311) | 0.048394 / 0.043533 (0.004861) | 0.365908 / 0.255139 (0.110769) | 0.381197 / 0.283200 (0.097998) | 0.022940 / 0.141683 (-0.118743) | 1.487909 / 1.452155 (0.035754) | 1.532931 / 1.492716 (0.040215) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.317506 / 0.018006 (0.299500) | 0.513391 / 0.000490 (0.512902) | 0.005464 / 0.000200 (0.005264) | 0.000214 / 0.000054 (0.000159) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032289 / 0.037411 (-0.005122) | 0.090157 / 0.014526 (0.075631) | 0.103514 / 0.176557 (-0.073043) | 0.158236 / 0.737135 (-0.578899) | 0.106554 / 0.296338 (-0.189784) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.406455 / 0.215209 (0.191246) | 4.061563 / 2.077655 (1.983908) | 2.082201 / 1.504120 (0.578081) | 1.914433 / 1.541195 (0.373238) | 2.039342 / 1.468490 (0.570852) | 0.478444 / 4.584777 (-4.106333) | 3.599755 / 3.745712 (-0.145957) | 3.294453 / 5.269862 (-1.975409) | 2.028519 / 4.565676 (-2.537158) | 0.056118 / 0.424275 (-0.368157) | 0.007325 / 0.007607 (-0.000282) | 0.493177 / 0.226044 (0.267132) | 4.926218 / 2.268929 (2.657289) | 2.605033 / 55.444624 (-52.839591) | 2.239933 / 6.876477 (-4.636544) | 2.454210 / 2.142072 (0.312137) | 0.571905 / 4.805227 (-4.233322) | 0.133251 / 6.500664 (-6.367413) | 0.062422 / 0.075469 (-0.013047) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.352752 / 1.841788 (-0.489036) | 20.265109 / 8.074308 (12.190801) | 14.293064 / 10.191392 (4.101672) | 0.169267 / 0.680424 (-0.511157) | 0.018607 / 0.534201 (-0.515594) | 0.393655 / 0.579283 (-0.185628) | 0.402132 / 0.434364 (-0.032232) | 0.477566 / 0.540337 (-0.062772) | 0.651773 / 1.386936 (-0.735163) |

![](https://cml.dev/watermark.png#80023f36b2b6678347979421ef973d8969d31306 "CML watermark")

This PR ignores the violation of the lint rule E721 in `Pickler.memoize`.

The lint rule violation was introduced in this PR:
- #3182

@lhoestq is there a reason you did not use `isinstance` instead?

As a hotfix, we just ignore the violation of the lint rule.

Fix #6136.


(`from_spark()`) Unable to connect HDFS in pyspark YARN setting

### Describe the bug

related issue: https://github.com/apache/arrow/issues/37057#issue-1841013613

---

Hello. I'm trying to interact with HDFS storage from a driver and workers of pyspark YARN cluster. Precisely I'm using **huggingface's `datasets`** ([link](https://github.com/huggingface/datasets)) library that relies on pyarrow to communicate with HDFS. The `from_spark()` ([link](https://huggingface.co/docs/datasets/use_with_spark#load-from-spark)) is what I'm invoking in my script.

Below is the error I'm encountering. Note that I've masked sensitive paths. My code is sent to worker containers (docker) from driver container then executed. I confirmed that in both driver and worker images I can connect to HDFS using pyarrow since the envs and required jars are properly set, but strangely that becomes impossible when the same image runs as remote worker process.

These are some peculiarities in my environment that might caused this issue.
* **Cluster requires kerberos authentication**
  * But I think the error message implies that's not the problem in this case
* **The user that runs the worker process is different from that built the docker image**
  * To avoid permission-related issues I made all directories that are accessed from the script accessible to everyone
* **Pyspark-part of my code has no problem interacting with HDFS.**
  * Even pyarrow doesn't experience problem when I run the code in interactive session of the same docker images (driver, worker)
  * The problem occurs only when it runs as cluster's worker runtime

Hope I could get some help. Thanks.

```bash
2023-08-08 18:51:19,638 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2023-08-08 18:51:20,280 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
23/08/08 18:51:22 WARN TaskSetManager: Lost task 0.0 in stage 142.0 (TID 9732) (ac3bax2062.bdp.bdata.ai executor 1): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "<MASKED>/application_1682476586273_25865777/container_e143_1682476586273_25865777_01_000003/pyspark.zip/pyspark/worker.py", line 830, in main
    process()
  File "<MASKED>/application_1682476586273_25865777/container_e143_1682476586273_25865777_01_000003/pyspark.zip/pyspark/worker.py", line 820, in process
    out_iter = func(split_index, iterator)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/spark/python/pyspark/rdd.py", line 5405, in pipeline_func
  File "/root/spark/python/pyspark/rdd.py", line 828, in func
  File "/opt/conda/lib/python3.11/site-packages/datasets/packaged_modules/spark/spark.py", line 130, in create_cache_and_write_probe
    open(probe_file, "a")
  File "/opt/conda/lib/python3.11/site-packages/datasets/streaming.py", line 74, in wrapper
    return function(*args, download_config=download_config, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/datasets/download/streaming_download_manager.py", line 496, in xopen
    file_obj = fsspec.open(file, mode=mode, *args, **kwargs).open()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/fsspec/core.py", line 439, in open
    out = open_files(
          ^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/fsspec/core.py", line 282, in open_files
    fs, fs_token, paths = get_fs_token_paths(
                          ^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/fsspec/core.py", line 609, in get_fs_token_paths
    fs = filesystem(protocol, **inkwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/fsspec/registry.py", line 267, in filesystem
    return cls(**storage_options)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/fsspec/spec.py", line 79, in __call__
    obj = super().__call__(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/fsspec/implementations/arrow.py", line 278, in __init__
    fs = HadoopFileSystem(
         ^^^^^^^^^^^^^^^^^
  File "pyarrow/_hdfs.pyx", line 96, in pyarrow._hdfs.HadoopFileSystem.__init__
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: HDFS connection failed

	at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:561)
	at org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:767)
	at org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:749)
	at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:514)
	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
	at scala.collection.Iterator.foreach(Iterator.scala:943)
	at scala.collection.Iterator.foreach$(Iterator.scala:943)
	at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
	at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
	at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
	at scala.collection.TraversableOnce.to(TraversableOnce.scala:366)
	at scala.collection.TraversableOnce.to$(TraversableOnce.scala:364)
	at org.apache.spark.InterruptibleIterator.to(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358)
	at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358)
	at org.apache.spark.InterruptibleIterator.toBuffer(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345)
	at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339)
	at org.apache.spark.InterruptibleIterator.toArray(InterruptibleIterator.scala:28)
	at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1019)
	at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2303)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)
	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
	at org.apache.spark.scheduler.Task.run(Task.scala:139)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:554)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1529)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:557)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

23/08/08 18:51:24 WARN TaskSetManager: Lost task 0.1 in stage 142.0 (TID 9733) (ac3iax2079.bdp.bdata.ai executor 2): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "<MASKED>/application_1682476586273_25865777/container_e143_1682476586273_25865777_01_000005/pyspark.zip/pyspark/worker.py", line 830, in main
    process()
  File "<MASKED>/application_1682476586273_25865777/container_e143_1682476586273_25865777_01_000005/pyspark.zip/pyspark/worker.py", line 820, in process
    out_iter = func(split_index, iterator)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/spark/python/pyspark/rdd.py", line 5405, in pipeline_func
  File "/root/spark/python/pyspark/rdd.py", line 828, in func
  File "/opt/conda/lib/python3.11/site-packages/datasets/packaged_modules/spark/spark.py", line 130, in create_cache_and_write_probe
    open(probe_file, "a")
  File "/opt/conda/lib/python3.11/site-packages/datasets/streaming.py", line 74, in wrapper
    return function(*args, download_config=download_config, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/datasets/download/streaming_download_manager.py", line 496, in xopen
    file_obj = fsspec.open(file, mode=mode, *args, **kwargs).open()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/fsspec/core.py", line 439, in open
    out = open_files(
          ^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/fsspec/core.py", line 282, in open_files
    fs, fs_token, paths = get_fs_token_paths(
                          ^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/fsspec/core.py", line 609, in get_fs_token_paths
    fs = filesystem(protocol, **inkwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/fsspec/registry.py", line 267, in filesystem
    return cls(**storage_options)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/fsspec/spec.py", line 79, in __call__
    obj = super().__call__(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/fsspec/implementations/arrow.py", line 278, in __init__
    fs = HadoopFileSystem(
         ^^^^^^^^^^^^^^^^^
  File "pyarrow/_hdfs.pyx", line 96, in pyarrow._hdfs.HadoopFileSystem.__init__
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: HDFS connection failed

	at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:561)
	at org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:767)
	at org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:749)
	at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:514)
	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
	at scala.collection.Iterator.foreach(Iterator.scala:943)
	at scala.collection.Iterator.foreach$(Iterator.scala:943)
	at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
	at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
	at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
	at scala.collection.TraversableOnce.to(TraversableOnce.scala:366)
	at scala.collection.TraversableOnce.to$(TraversableOnce.scala:364)
	at org.apache.spark.InterruptibleIterator.to(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358)
	at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358)
	at org.apache.spark.InterruptibleIterator.toBuffer(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345)
	at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339)
	at org.apache.spark.InterruptibleIterator.toArray(InterruptibleIterator.scala:28)
	at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1019)
	at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2303)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)
	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
	at org.apache.spark.scheduler.Task.run(Task.scala:139)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:554)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1529)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:557)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

23/08/08 18:51:38 WARN TaskSetManager: Lost task 0.2 in stage 142.0 (TID 9734) (<MASKED> executor 4): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "<MASKED>/application_1682476586273_25865777/container_e143_1682476586273_25865777_01_000008/pyspark.zip/pyspark/worker.py", line 830, in main
    process()
  File "<MASKED>/application_1682476586273_25865777/container_e143_1682476586273_25865777_01_000008/pyspark.zip/pyspark/worker.py", line 820, in process
    out_iter = func(split_index, iterator)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/spark/python/pyspark/rdd.py", line 5405, in pipeline_func
  File "/root/spark/python/pyspark/rdd.py", line 828, in func
  File "/opt/conda/lib/python3.11/site-packages/datasets/packaged_modules/spark/spark.py", line 130, in create_cache_and_write_probe
    open(probe_file, "a")
  File "/opt/conda/lib/python3.11/site-packages/datasets/streaming.py", line 74, in wrapper
    return function(*args, download_config=download_config, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/datasets/download/streaming_download_manager.py", line 496, in xopen
    file_obj = fsspec.open(file, mode=mode, *args, **kwargs).open()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/fsspec/core.py", line 439, in open
    out = open_files(
          ^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/fsspec/core.py", line 282, in open_files
    fs, fs_token, paths = get_fs_token_paths(
                          ^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/fsspec/core.py", line 609, in get_fs_token_paths
    fs = filesystem(protocol, **inkwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/fsspec/registry.py", line 267, in filesystem
    return cls(**storage_options)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/fsspec/spec.py", line 79, in __call__
    obj = super().__call__(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/fsspec/implementations/arrow.py", line 278, in __init__
    fs = HadoopFileSystem(
         ^^^^^^^^^^^^^^^^^
  File "pyarrow/_hdfs.pyx", line 96, in pyarrow._hdfs.HadoopFileSystem.__init__
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: HDFS connection failed

	at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:561)
	at org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:767)
	at org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:749)
	at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:514)
	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
	at scala.collection.Iterator.foreach(Iterator.scala:943)
	at scala.collection.Iterator.foreach$(Iterator.scala:943)
	at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
	at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
	at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
	at scala.collection.TraversableOnce.to(TraversableOnce.scala:366)
	at scala.collection.TraversableOnce.to$(TraversableOnce.scala:364)
	at org.apache.spark.InterruptibleIterator.to(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358)
	at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358)
	at org.apache.spark.InterruptibleIterator.toBuffer(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345)
	at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339)
	at org.apache.spark.InterruptibleIterator.toArray(InterruptibleIterator.scala:28)
	at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1019)
	at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2303)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)
	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
	at org.apache.spark.scheduler.Task.run(Task.scala:139)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:554)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1529)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:557)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
```

### Steps to reproduce the bug

Use `from_spark()` function in pyspark YARN setting. I set `cache_dir` to HDFS path.

### Expected behavior

Work as described in document

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-4.18.0-425.19.2.el8_7.x86_64-x86_64-with-glibc2.17
- Python version: 3.11.4
- Huggingface_hub version: 0.16.4
- PyArrow version: 10.0.1
- Pandas version: 1.5.3


CI check_code_quality error: E721 Do not compare types, use `isinstance()`

maintenance

Maintenance tasks

After latest release of `ruff` (https://pypi.org/project/ruff/0.0.284/), we get the following CI error:
```
src/datasets/utils/py_utils.py:689:12: E721 Do not compare types, use `isinstance()`
```


Remove unused allowed_extensions param

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009055 / 0.011353 (-0.002298) | 0.008835 / 0.011008 (-0.002173) | 0.117048 / 0.038508 (0.078540) | 0.096268 / 0.023109 (0.073159) | 0.474678 / 0.275898 (0.198780) | 0.550509 / 0.323480 (0.227029) | 0.005552 / 0.007986 (-0.002434) | 0.004315 / 0.004328 (-0.000013) | 0.094336 / 0.004250 (0.090086) | 0.061945 / 0.037052 (0.024892) | 0.461422 / 0.258489 (0.202933) | 0.521271 / 0.293841 (0.227430) | 0.049116 / 0.128546 (-0.079430) | 0.015007 / 0.075646 (-0.060639) | 0.414351 / 0.419271 (-0.004920) | 0.137520 / 0.043533 (0.093987) | 0.465627 / 0.255139 (0.210488) | 0.537244 / 0.283200 (0.254044) | 0.068577 / 0.141683 (-0.073106) | 1.921373 / 1.452155 (0.469219) | 2.506653 / 1.492716 (1.013937) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.273970 / 0.018006 (0.255963) | 0.750295 / 0.000490 (0.749805) | 0.004241 / 0.000200 (0.004041) | 0.000128 / 0.000054 (0.000073) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033793 / 0.037411 (-0.003618) | 0.105562 / 0.014526 (0.091037) | 0.131771 / 0.176557 (-0.044786) | 0.196890 / 0.737135 (-0.540245) | 0.119842 / 0.296338 (-0.176496) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.634881 / 0.215209 (0.419672) | 6.069221 / 2.077655 (3.991566) | 2.678765 / 1.504120 (1.174646) | 2.460309 / 1.541195 (0.919114) | 2.517579 / 1.468490 (1.049089) | 0.869558 / 4.584777 (-3.715219) | 5.407686 / 3.745712 (1.661974) | 4.920687 / 5.269862 (-0.349175) | 3.130066 / 4.565676 (-1.435611) | 0.100337 / 0.424275 (-0.323938) | 0.009615 / 0.007607 (0.002008) | 0.745275 / 0.226044 (0.519231) | 7.577890 / 2.268929 (5.308962) | 3.607887 / 55.444624 (-51.836738) | 2.922211 / 6.876477 (-3.954266) | 3.205592 / 2.142072 (1.063519) | 1.052298 / 4.805227 (-3.752929) | 0.218798 / 6.500664 (-6.281866) | 0.082137 / 0.075469 (0.006667) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.696551 / 1.841788 (-0.145237) | 24.946074 / 8.074308 (16.871766) | 23.114202 / 10.191392 (12.922810) | 0.220498 / 0.680424 (-0.459925) | 0.029388 / 0.534201 (-0.504813) | 0.494721 / 0.579283 (-0.084562) | 0.603085 / 0.434364 (0.168722) | 0.573093 / 0.540337 (0.032756) | 0.784937 / 1.386936 (-0.601999) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009642 / 0.011353 (-0.001711) | 0.007551 / 0.011008 (-0.003457) | 0.085224 / 0.038508 (0.046716) | 0.099493 / 0.023109 (0.076384) | 0.503824 / 0.275898 (0.227926) | 0.546583 / 0.323480 (0.223103) | 0.006385 / 0.007986 (-0.001601) | 0.004751 / 0.004328 (0.000423) | 0.084699 / 0.004250 (0.080449) | 0.067875 / 0.037052 (0.030823) | 0.485313 / 0.258489 (0.226824) | 0.535808 / 0.293841 (0.241967) | 0.049935 / 0.128546 (-0.078611) | 0.014427 / 0.075646 (-0.061219) | 0.095531 / 0.419271 (-0.323741) | 0.068487 / 0.043533 (0.024954) | 0.502204 / 0.255139 (0.247065) | 0.514393 / 0.283200 (0.231193) | 0.037350 / 0.141683 (-0.104333) | 1.849380 / 1.452155 (0.397226) | 1.920151 / 1.492716 (0.427434) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.298363 / 0.018006 (0.280357) | 0.651555 / 0.000490 (0.651065) | 0.005910 / 0.000200 (0.005710) | 0.000103 / 0.000054 (0.000048) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.039170 / 0.037411 (0.001758) | 0.106436 / 0.014526 (0.091910) | 0.129880 / 0.176557 (-0.046677) | 0.185401 / 0.737135 (-0.551734) | 0.125732 / 0.296338 (-0.170607) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.643248 / 0.215209 (0.428039) | 6.374807 / 2.077655 (4.297152) | 3.057296 / 1.504120 (1.553176) | 2.779534 / 1.541195 (1.238340) | 2.790165 / 1.468490 (1.321675) | 0.841580 / 4.584777 (-3.743197) | 5.371478 / 3.745712 (1.625766) | 4.973251 / 5.269862 (-0.296610) | 3.235817 / 4.565676 (-1.329860) | 0.097276 / 0.424275 (-0.326999) | 0.008840 / 0.007607 (0.001233) | 0.728678 / 0.226044 (0.502634) | 7.526382 / 2.268929 (5.257454) | 3.792550 / 55.444624 (-51.652074) | 3.439134 / 6.876477 (-3.437342) | 3.466626 / 2.142072 (1.324553) | 1.035894 / 4.805227 (-3.769333) | 0.211670 / 6.500664 (-6.288994) | 0.087596 / 0.075469 (0.012127) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.782755 / 1.841788 (-0.059033) | 25.704407 / 8.074308 (17.630099) | 23.799672 / 10.191392 (13.608280) | 0.233952 / 0.680424 (-0.446472) | 0.030810 / 0.534201 (-0.503391) | 0.505857 / 0.579283 (-0.073426) | 0.629331 / 0.434364 (0.194967) | 0.608530 / 0.540337 (0.068192) | 0.813688 / 1.386936 (-0.573248) |

![](https://cml.dev/watermark.png#ed4d6bb5f1331576c41b04acd9872a5349a0915c "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006401 / 0.011353 (-0.004952) | 0.003916 / 0.011008 (-0.007092) | 0.083976 / 0.038508 (0.045468) | 0.072583 / 0.023109 (0.049474) | 0.322747 / 0.275898 (0.046849) | 0.345159 / 0.323480 (0.021679) | 0.005366 / 0.007986 (-0.002620) | 0.003399 / 0.004328 (-0.000930) | 0.064232 / 0.004250 (0.059982) | 0.053313 / 0.037052 (0.016261) | 0.353127 / 0.258489 (0.094638) | 0.361398 / 0.293841 (0.067557) | 0.030604 / 0.128546 (-0.097942) | 0.008615 / 0.075646 (-0.067031) | 0.285806 / 0.419271 (-0.133466) | 0.050887 / 0.043533 (0.007354) | 0.312293 / 0.255139 (0.057154) | 0.349716 / 0.283200 (0.066516) | 0.024546 / 0.141683 (-0.117137) | 1.472318 / 1.452155 (0.020163) | 1.536063 / 1.492716 (0.043347) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.280012 / 0.018006 (0.262006) | 0.593574 / 0.000490 (0.593085) | 0.004083 / 0.000200 (0.003883) | 0.000195 / 0.000054 (0.000141) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027715 / 0.037411 (-0.009696) | 0.081392 / 0.014526 (0.066866) | 0.096445 / 0.176557 (-0.080112) | 0.152131 / 0.737135 (-0.585004) | 0.094825 / 0.296338 (-0.201514) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.380749 / 0.215209 (0.165540) | 3.806994 / 2.077655 (1.729339) | 1.842544 / 1.504120 (0.338424) | 1.682829 / 1.541195 (0.141635) | 1.701679 / 1.468490 (0.233189) | 0.484830 / 4.584777 (-4.099947) | 3.517359 / 3.745712 (-0.228353) | 3.231211 / 5.269862 (-2.038651) | 2.029371 / 4.565676 (-2.536306) | 0.057199 / 0.424275 (-0.367077) | 0.007653 / 0.007607 (0.000046) | 0.458572 / 0.226044 (0.232528) | 4.579835 / 2.268929 (2.310907) | 2.326467 / 55.444624 (-53.118157) | 1.939646 / 6.876477 (-4.936831) | 2.133150 / 2.142072 (-0.008922) | 0.596251 / 4.805227 (-4.208976) | 0.131979 / 6.500664 (-6.368686) | 0.059226 / 0.075469 (-0.016243) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.234833 / 1.841788 (-0.606955) | 19.475522 / 8.074308 (11.401214) | 14.102760 / 10.191392 (3.911368) | 0.159657 / 0.680424 (-0.520767) | 0.018292 / 0.534201 (-0.515909) | 0.391079 / 0.579283 (-0.188204) | 0.406736 / 0.434364 (-0.027628) | 0.459159 / 0.540337 (-0.081178) | 0.618159 / 1.386936 (-0.768777) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006592 / 0.011353 (-0.004761) | 0.004052 / 0.011008 (-0.006957) | 0.064536 / 0.038508 (0.026028) | 0.075051 / 0.023109 (0.051942) | 0.379596 / 0.275898 (0.103698) | 0.412413 / 0.323480 (0.088933) | 0.005377 / 0.007986 (-0.002608) | 0.003466 / 0.004328 (-0.000863) | 0.064958 / 0.004250 (0.060708) | 0.055265 / 0.037052 (0.018213) | 0.391505 / 0.258489 (0.133016) | 0.425345 / 0.293841 (0.131504) | 0.030750 / 0.128546 (-0.097796) | 0.008652 / 0.075646 (-0.066994) | 0.072107 / 0.419271 (-0.347165) | 0.048340 / 0.043533 (0.004807) | 0.387714 / 0.255139 (0.132575) | 0.402602 / 0.283200 (0.119402) | 0.023492 / 0.141683 (-0.118191) | 1.528377 / 1.452155 (0.076222) | 1.574827 / 1.492716 (0.082110) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.316999 / 0.018006 (0.298993) | 0.528391 / 0.000490 (0.527901) | 0.005183 / 0.000200 (0.004983) | 0.000085 / 0.000054 (0.000031) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029670 / 0.037411 (-0.007741) | 0.087130 / 0.014526 (0.072604) | 0.099897 / 0.176557 (-0.076660) | 0.154074 / 0.737135 (-0.583062) | 0.104309 / 0.296338 (-0.192030) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.408804 / 0.215209 (0.193595) | 4.072248 / 2.077655 (1.994593) | 2.103333 / 1.504120 (0.599213) | 1.931972 / 1.541195 (0.390777) | 1.980132 / 1.468490 (0.511642) | 0.482623 / 4.584777 (-4.102154) | 3.532789 / 3.745712 (-0.212923) | 3.304962 / 5.269862 (-1.964899) | 2.036672 / 4.565676 (-2.529004) | 0.056944 / 0.424275 (-0.367331) | 0.007190 / 0.007607 (-0.000417) | 0.490650 / 0.226044 (0.264606) | 4.903604 / 2.268929 (2.634675) | 2.586247 / 55.444624 (-52.858377) | 2.227631 / 6.876477 (-4.648846) | 2.397286 / 2.142072 (0.255214) | 0.579167 / 4.805227 (-4.226060) | 0.132037 / 6.500664 (-6.368627) | 0.059971 / 0.075469 (-0.015498) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.336430 / 1.841788 (-0.505358) | 19.915846 / 8.074308 (11.841538) | 14.102781 / 10.191392 (3.911389) | 0.147956 / 0.680424 (-0.532468) | 0.018192 / 0.534201 (-0.516009) | 0.397949 / 0.579283 (-0.181334) | 0.408529 / 0.434364 (-0.025835) | 0.479382 / 0.540337 (-0.060955) | 0.659735 / 1.386936 (-0.727201) |

![](https://cml.dev/watermark.png#98074122449bc031f7269f298f1c55f20e39b975 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005880 / 0.011353 (-0.005473) | 0.003677 / 0.011008 (-0.007332) | 0.080022 / 0.038508 (0.041514) | 0.055554 / 0.023109 (0.032445) | 0.397449 / 0.275898 (0.121551) | 0.428346 / 0.323480 (0.104867) | 0.004613 / 0.007986 (-0.003373) | 0.002873 / 0.004328 (-0.001455) | 0.062226 / 0.004250 (0.057976) | 0.044721 / 0.037052 (0.007669) | 0.404792 / 0.258489 (0.146303) | 0.437467 / 0.293841 (0.143626) | 0.027166 / 0.128546 (-0.101381) | 0.008077 / 0.075646 (-0.067569) | 0.260469 / 0.419271 (-0.158803) | 0.043551 / 0.043533 (0.000018) | 0.401712 / 0.255139 (0.146573) | 0.427294 / 0.283200 (0.144094) | 0.021243 / 0.141683 (-0.120440) | 1.464553 / 1.452155 (0.012398) | 1.507112 / 1.492716 (0.014396) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.198415 / 0.018006 (0.180408) | 0.427940 / 0.000490 (0.427450) | 0.004236 / 0.000200 (0.004036) | 0.000067 / 0.000054 (0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023759 / 0.037411 (-0.013652) | 0.073262 / 0.014526 (0.058736) | 0.677113 / 0.176557 (0.500557) | 0.194964 / 0.737135 (-0.542172) | 0.086121 / 0.296338 (-0.210217) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.401176 / 0.215209 (0.185967) | 4.028688 / 2.077655 (1.951034) | 2.026804 / 1.504120 (0.522685) | 1.887964 / 1.541195 (0.346770) | 2.008991 / 1.468490 (0.540501) | 0.498847 / 4.584777 (-4.085930) | 3.015920 / 3.745712 (-0.729792) | 2.837019 / 5.269862 (-2.432843) | 1.849976 / 4.565676 (-2.715701) | 0.057545 / 0.424275 (-0.366730) | 0.006645 / 0.007607 (-0.000962) | 0.470225 / 0.226044 (0.244180) | 4.720910 / 2.268929 (2.451982) | 2.473693 / 55.444624 (-52.970931) | 2.177525 / 6.876477 (-4.698952) | 2.374702 / 2.142072 (0.232630) | 0.588253 / 4.805227 (-4.216974) | 0.125512 / 6.500664 (-6.375152) | 0.061247 / 0.075469 (-0.014222) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.255829 / 1.841788 (-0.585959) | 18.251689 / 8.074308 (10.177381) | 13.690373 / 10.191392 (3.498981) | 0.146928 / 0.680424 (-0.533496) | 0.016534 / 0.534201 (-0.517667) | 0.335249 / 0.579283 (-0.244034) | 0.338940 / 0.434364 (-0.095424) | 0.382170 / 0.540337 (-0.158168) | 0.529570 / 1.386936 (-0.857366) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005920 / 0.011353 (-0.005433) | 0.003557 / 0.011008 (-0.007451) | 0.062776 / 0.038508 (0.024267) | 0.058473 / 0.023109 (0.035364) | 0.358780 / 0.275898 (0.082882) | 0.394161 / 0.323480 (0.070682) | 0.004636 / 0.007986 (-0.003349) | 0.002865 / 0.004328 (-0.001463) | 0.062033 / 0.004250 (0.057782) | 0.047154 / 0.037052 (0.010101) | 0.367718 / 0.258489 (0.109229) | 0.400814 / 0.293841 (0.106973) | 0.026919 / 0.128546 (-0.101628) | 0.008071 / 0.075646 (-0.067575) | 0.067802 / 0.419271 (-0.351469) | 0.040894 / 0.043533 (-0.002638) | 0.358757 / 0.255139 (0.103618) | 0.384971 / 0.283200 (0.101771) | 0.020019 / 0.141683 (-0.121664) | 1.458578 / 1.452155 (0.006423) | 1.525059 / 1.492716 (0.032342) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.207795 / 0.018006 (0.189789) | 0.413201 / 0.000490 (0.412712) | 0.005199 / 0.000200 (0.004999) | 0.000085 / 0.000054 (0.000031) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025716 / 0.037411 (-0.011696) | 0.078434 / 0.014526 (0.063908) | 0.086920 / 0.176557 (-0.089637) | 0.138327 / 0.737135 (-0.598808) | 0.088120 / 0.296338 (-0.208219) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.434344 / 0.215209 (0.219135) | 4.343114 / 2.077655 (2.265459) | 2.384439 / 1.504120 (0.880319) | 2.253929 / 1.541195 (0.712735) | 2.306811 / 1.468490 (0.838321) | 0.497572 / 4.584777 (-4.087205) | 3.028794 / 3.745712 (-0.716919) | 2.833484 / 5.269862 (-2.436377) | 1.878918 / 4.565676 (-2.686759) | 0.057133 / 0.424275 (-0.367143) | 0.006357 / 0.007607 (-0.001251) | 0.508019 / 0.226044 (0.281975) | 5.076935 / 2.268929 (2.808007) | 2.745784 / 55.444624 (-52.698841) | 2.476291 / 6.876477 (-4.400186) | 2.677264 / 2.142072 (0.535191) | 0.587173 / 4.805227 (-4.218054) | 0.126373 / 6.500664 (-6.374291) | 0.062815 / 0.075469 (-0.012654) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.355482 / 1.841788 (-0.486305) | 18.818227 / 8.074308 (10.743919) | 13.954289 / 10.191392 (3.762896) | 0.143413 / 0.680424 (-0.537011) | 0.016844 / 0.534201 (-0.517357) | 0.338334 / 0.579283 (-0.240949) | 0.344559 / 0.434364 (-0.089805) | 0.400669 / 0.540337 (-0.139669) | 0.563835 / 1.386936 (-0.823101) |

![](https://cml.dev/watermark.png#c02a44715c036b5261686669727394b1308a3a4b "CML watermark")

This PR removes unused `allowed_extensions` parameter from `create_builder_configs_from_metadata_configs`.


`datasets` cannot be installed alongside `apache-beam`

I noticed that this is actually covered by issue #5613, which for some reason I didn't see when I searched the issues in this repo the first time.

### Describe the bug

If one installs `apache-beam` alongside `datasets` (which is required for the [wikipedia](https://huggingface.co/datasets/wikipedia#dataset-summary) dataset) in certain environments (such as a Google Colab notebook), they appear to install successfully, however, actually trying to do something such as importing the `load_dataset` method from `datasets` results in a crashing error.

I think the problem is that `apache-beam` version 2.49.0 requires `dill>=0.3.1.1,<0.3.2`, but the latest version of `multiprocess` (0.70.15) (on which `datasets` depends) requires  `dill>=0.3.7,`, so this is causing the dependency resolver to use an older version of `multiprocess` which leads to the `datasets` crashing since it doesn't actually appear to be compatible with older versions.

### Steps to reproduce the bug

See this [Google Colab notebook](https://colab.research.google.com/drive/1PTeGlshamFcJZix_GiS3vMXX_YzAhGv0?usp=sharing) to easily reproduce the bug.

In some environments, I have been able to reproduce the bug by running the following in Bash:

```bash
$ pip install datasets apache-beam
```

then the following in a Python shell:

```python
from datasets import load_dataset
```

Here is my stacktrace from running on Google Colab:

<details>
<summary>stacktrace</summary>

```
[/usr/local/lib/python3.10/dist-packages/datasets/__init__.py](https://localhost:8080/#) in <module>
     20 __version__ = "2.14.4"
     21
---> 22 from .arrow_dataset import Dataset
     23 from .arrow_reader import ReadInstruction
     24 from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder

[/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in <module>
     64
     65 from . import config
---> 66 from .arrow_reader import ArrowReader
     67 from .arrow_writer import ArrowWriter, OptimizedTypedSequence
     68 from .data_files import sanitize_patterns

[/usr/local/lib/python3.10/dist-packages/datasets/arrow_reader.py](https://localhost:8080/#) in <module>
     28 import pyarrow.parquet as pq
     29
---> 30 from .download.download_config import DownloadConfig
     31 from .naming import _split_re, filenames_for_dataset_split
     32 from .table import InMemoryTable, MemoryMappedTable, Table, concat_tables

[/usr/local/lib/python3.10/dist-packages/datasets/download/__init__.py](https://localhost:8080/#) in <module>
      7
      8 from .download_config import DownloadConfig
----> 9 from .download_manager import DownloadManager, DownloadMode
     10 from .streaming_download_manager import StreamingDownloadManager

[/usr/local/lib/python3.10/dist-packages/datasets/download/download_manager.py](https://localhost:8080/#) in <module>
     33 from ..utils.info_utils import get_size_checksum_dict
     34 from ..utils.logging import get_logger, is_progress_bar_enabled, tqdm
---> 35 from ..utils.py_utils import NestedDataStructure, map_nested, size_str
     36 from .download_config import DownloadConfig
     37

[/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py](https://localhost:8080/#) in <module>
     38 import dill
     39 import multiprocess
---> 40 import multiprocess.pool
     41 import numpy as np
     42 from packaging import version

[/usr/local/lib/python3.10/dist-packages/multiprocess/pool.py](https://localhost:8080/#) in <module>
    607 #
    608
--> 609 class ThreadPool(Pool):
    610
    611     from .dummy import Process

[/usr/local/lib/python3.10/dist-packages/multiprocess/pool.py](https://localhost:8080/#) in ThreadPool()
    609 class ThreadPool(Pool):
    610
--> 611     from .dummy import Process
    612
    613     def __init__(self, processes=None, initializer=None, initargs=()):

[/usr/local/lib/python3.10/dist-packages/multiprocess/dummy/__init__.py](https://localhost:8080/#) in <module>
     85 #
     86
---> 87 class Condition(threading._Condition):
     88     # XXX
     89     if sys.version_info < (3, 0):

AttributeError: module 'threading' has no attribute '_Condition'
```

</details>

I've also found that attempting to install these `datasets` and `apache-beam` in certain environments (e.g. via pip inside a conda env) simply causes pip to hang indefinitely.

### Expected behavior

I would expect to be able to import methods from `datasets` without crashing. I have tested that this is possible as long as I do not attempt to install `apache-beam`.

### Environment info

Google Colab


Dataset is slower after calling `to_iterable_dataset`

It's roughly the same code between the two so we can expected roughly the same speed, could you share a benchmark ?

### Describe the bug

Can anyone explain why looping over a dataset becomes slower after calling `to_iterable_dataset` to convert to `IterableDataset`

### Steps to reproduce the bug

Any dataset after converting to `IterableDataset`

### Expected behavior

Maybe it should be faster on big dataset? I only test on small dataset

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.17
- Python version: 3.8.15
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.3


### Describe the bug

to_iterable_dataset is missing in document

### Steps to reproduce the bug

to_iterable_dataset is missing in document

### Expected behavior

document enhancement

### Environment info

unrelated


default config name doesn't work when config kwargs are specified.

What should be the behavior in this case ? Should it override the default config with the added parameter ?

I know why it should be treated as a new config if overriding parameters are passed. But in some case, I just pass in some common fields like `data_dir`.

For example, I want to extend the FolderBasedBuilder as a multi-config version, the `data_dir` or `data_files` are always passed by user and should not be considered as overriding the default config. In current state, I cannot leverage the feature of default config since passing `data_dir` will disable the default config.

Thinking more about it I think the current behavior is the right one.

Provided parameters should be passed to instantiate a new BuilderConfig.

What's the error you're getting ?

For example, this works to use default config with name '_all_':
```python
datasets.load_dataset("indonesian-nlp/librivox-indonesia", split="train")
```
while this failed to use default config
```python
datasets.load_dataset("indonesian-nlp/librivox-indonesia", split="train", data_dir='.')
```
After manually specifying it, it works again.
```python
datasets.load_dataset("indonesian-nlp/librivox-indonesia", "_all_", split="train", data_dir='.')
```

It should work if you explicitly ask for the config you want to override

```python
load_dataset('/dataset/with/multiple/config', 'name_of_the_default_config', some_field_in_config='some')
```

Alternatively you can have a BuilderConfig class that when instantiated returns a config with the right default values. In this case this code would instantiate this config with the default values except for the parameter to override:

```python
load_dataset('/dataset/with/multiple/config', some_field_in_config='some')
```

@lhoestq Yes. But it doesn't work for me.

Here's my dataset for example.
```
lass MyDatasetConfig(datasets.BuilderConfig):
    def __init__(self, name: str, version: str, **kwargs):
        self.option1 = kwargs.pop("option1", False)
        self.option2 = kwargs.pop("option2", 5)

        super().__init__(
            name=name,
            version=datasets.Version(version),
            **kwargs)

class MyDataset(datasets.GeneratorBasedBuilder):
    DEFAULT_CONFIG_NAME = "v1"

    BUILDER_CONFIGS = [
        UnifiedTtsDatasetConfig(
            name="v1",
            version="1.0.0",
            description="Initial version of the dataset"
        ),
    ]

    def _info(self) -> DatasetInfo:
        _ = self.option1
        ....
```

Here it's okay to use `load_dataset('my_dataset.py')` for loading the default config `v1`.

But if I want to override the default values in config with `load_dataset('my_dataset.py', option2=3)`, it failed to find my default config `v1.

Unless I use `load_dataset('my_dataset.py', 'v1', option2=3)`

So according to your advice, how can I modify my dataset to be able to override default config without manually specifying it.

What's the error ? It should try to instantiate `MyDatasetConfig` with `option2=3`

@lhoestq The error is
```
def _info(self) -> DatasetInfo:
        _ = self.option1  <-
        ....
AttributeError: 'BuilderConfig' object has no attribute 'option1'
```
which seems to find another unknown config.

You can try this line `datasets.load_dataset("indonesian-nlp/librivox-indonesia", split="train", data_dir='.')`, it's a multi-config dataset on HF hub and the error is the same.

My insights:
https://github.com/huggingface/datasets/blob/12cfc1196e62847e2e8239fbd727a02cbc86ddec/src/datasets/builder.py#L518
if `config_kwargs` is provided here, the if branch is skipped.

I see, you just have to set this class attribute to your builder class :)

```python
BUILDER_CONFIG_CLASS = MyDatasetConfig
```

So what does this attribute do? In most cases it's not used and the [documents for multi-config dataset](https://huggingface.co/docs/datasets/main/en/image_dataset#multiple-configurations) never mentioned that.

It tells which builder config class to instantiate if additional config parameters are passed to load_dataset

@lhoestq maybe we can enhance the document to say something about the common attributes of `DatasetBuilder`

Ah indeed it's missing in the docs, thanks for reporting. I'm opening a PR

### Describe the bug

https://github.com/huggingface/datasets/blob/12cfc1196e62847e2e8239fbd727a02cbc86ddec/src/datasets/builder.py#L518-L522

If `config_name` is `None`, `DEFAULT_CONFIG_NAME` should be select. But once users pass `config_kwargs` to their customized `BuilderConfig`, the logic is ignored, and dataset cannot select the default config from multiple configs.

### Steps to reproduce the bug

```python
import datasets

datasets.load_dataset('/dataset/with/multiple/config'')  # Ok
datasets.load_dataset('/dataset/with/multiple/config', some_field_in_config='some')  # Err
```

### Expected behavior

Default config behavior should be consistent.

### Environment info

- `datasets` version: 2.14.3
- Platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.17
- Python version: 3.8.15
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.3