Fix CI 404 errors

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008220 / 0.011353 (-0.003133) | 0.005560 / 0.011008 (-0.005448) | 0.100147 / 0.038508 (0.061639) | 0.070106 / 0.023109 (0.046996) | 0.411906 / 0.275898 (0.136008) | 0.432825 / 0.323480 (0.109345) | 0.004795 / 0.007986 (-0.003190) | 0.004094 / 0.004328 (-0.000235) | 0.075719 / 0.004250 (0.071468) | 0.067426 / 0.037052 (0.030374) | 0.428531 / 0.258489 (0.170042) | 0.437114 / 0.293841 (0.143273) | 0.045603 / 0.128546 (-0.082943) | 0.013333 / 0.075646 (-0.062313) | 0.353137 / 0.419271 (-0.066134) | 0.067902 / 0.043533 (0.024369) | 0.396633 / 0.255139 (0.141494) | 0.399185 / 0.283200 (0.115985) | 0.036377 / 0.141683 (-0.105306) | 1.624249 / 1.452155 (0.172094) | 1.792575 / 1.492716 (0.299859) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.315847 / 0.018006 (0.297840) | 0.595009 / 0.000490 (0.594519) | 0.018876 / 0.000200 (0.018676) | 0.000613 / 0.000054 (0.000558) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029886 / 0.037411 (-0.007526) | 0.085765 / 0.014526 (0.071239) | 0.108680 / 0.176557 (-0.067877) | 0.174588 / 0.737135 (-0.562548) | 0.104494 / 0.296338 (-0.191844) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.594429 / 0.215209 (0.379220) | 5.912352 / 2.077655 (3.834698) | 2.408501 / 1.504120 (0.904381) | 2.050914 / 1.541195 (0.509720) | 2.199349 / 1.468490 (0.730859) | 0.813797 / 4.584777 (-3.770980) | 5.169577 / 3.745712 (1.423864) | 4.653951 / 5.269862 (-0.615911) | 2.805423 / 4.565676 (-1.760253) | 0.092278 / 0.424275 (-0.331997) | 0.007394 / 0.007607 (-0.000213) | 0.684029 / 0.226044 (0.457985) | 6.964260 / 2.268929 (4.695331) | 3.108408 / 55.444624 (-52.336217) | 2.470907 / 6.876477 (-4.405569) | 2.460153 / 2.142072 (0.318081) | 0.986445 / 4.805227 (-3.818782) | 0.213069 / 6.500664 (-6.287596) | 0.074061 / 0.075469 (-0.001408) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.590732 / 1.841788 (-0.251056) | 23.736918 / 8.074308 (15.662609) | 21.223910 / 10.191392 (11.032518) | 0.236173 / 0.680424 (-0.444251) | 0.030056 / 0.534201 (-0.504145) | 0.489461 / 0.579283 (-0.089822) | 0.607582 / 0.434364 (0.173218) | 0.539889 / 0.540337 (-0.000449) | 0.817942 / 1.386936 (-0.568994) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008042 / 0.011353 (-0.003311) | 0.004836 / 0.011008 (-0.006173) | 0.075434 / 0.038508 (0.036926) | 0.080818 / 0.023109 (0.057709) | 0.474797 / 0.275898 (0.198899) | 0.526168 / 0.323480 (0.202689) | 0.006463 / 0.007986 (-0.001522) | 0.004031 / 0.004328 (-0.000297) | 0.074141 / 0.004250 (0.069891) | 0.068265 / 0.037052 (0.031212) | 0.562550 / 0.258489 (0.304061) | 0.544820 / 0.293841 (0.250979) | 0.047263 / 0.128546 (-0.081283) | 0.014113 / 0.075646 (-0.061534) | 0.086061 / 0.419271 (-0.333210) | 0.062475 / 0.043533 (0.018942) | 0.479912 / 0.255139 (0.224773) | 0.494784 / 0.283200 (0.211584) | 0.035847 / 0.141683 (-0.105836) | 1.726452 / 1.452155 (0.274297) | 1.770113 / 1.492716 (0.277396) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.286713 / 0.018006 (0.268707) | 0.609704 / 0.000490 (0.609214) | 0.009342 / 0.000200 (0.009143) | 0.000134 / 0.000054 (0.000080) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035137 / 0.037411 (-0.002275) | 0.099331 / 0.014526 (0.084805) | 0.108971 / 0.176557 (-0.067586) | 0.170952 / 0.737135 (-0.566183) | 0.111736 / 0.296338 (-0.184603) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.617434 / 0.215209 (0.402225) | 6.204351 / 2.077655 (4.126697) | 2.854347 / 1.504120 (1.350227) | 2.557424 / 1.541195 (1.016229) | 2.638173 / 1.468490 (1.169683) | 0.854234 / 4.584777 (-3.730543) | 5.383288 / 3.745712 (1.637576) | 4.698098 / 5.269862 (-0.571763) | 2.903860 / 4.565676 (-1.661817) | 0.094689 / 0.424275 (-0.329586) | 0.007892 / 0.007607 (0.000285) | 0.729420 / 0.226044 (0.503376) | 7.356691 / 2.268929 (5.087763) | 3.708039 / 55.444624 (-51.736585) | 2.979734 / 6.876477 (-3.896743) | 2.978983 / 2.142072 (0.836911) | 1.040554 / 4.805227 (-3.764673) | 0.211246 / 6.500664 (-6.289418) | 0.079880 / 0.075469 (0.004411) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.676057 / 1.841788 (-0.165731) | 23.428443 / 8.074308 (15.354135) | 21.016293 / 10.191392 (10.824901) | 0.260927 / 0.680424 (-0.419497) | 0.030689 / 0.534201 (-0.503512) | 0.495652 / 0.579283 (-0.083632) | 0.622976 / 0.434364 (0.188612) | 0.561175 / 0.540337 (0.020837) | 0.786733 / 1.386936 (-0.600203) |

![](https://cml.dev/watermark.png#fb621b9630a69643255d25f192fdb011935122b1 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005942 / 0.011353 (-0.005410) | 0.003706 / 0.011008 (-0.007302) | 0.081002 / 0.038508 (0.042493) | 0.056854 / 0.023109 (0.033745) | 0.358668 / 0.275898 (0.082770) | 0.369718 / 0.323480 (0.046238) | 0.005202 / 0.007986 (-0.002784) | 0.002841 / 0.004328 (-0.001487) | 0.062976 / 0.004250 (0.058726) | 0.051308 / 0.037052 (0.014255) | 0.373636 / 0.258489 (0.115147) | 0.390480 / 0.293841 (0.096639) | 0.027480 / 0.128546 (-0.101067) | 0.007960 / 0.075646 (-0.067686) | 0.262719 / 0.419271 (-0.156552) | 0.046488 / 0.043533 (0.002955) | 0.347299 / 0.255139 (0.092160) | 0.393448 / 0.283200 (0.110249) | 0.019445 / 0.141683 (-0.122238) | 1.431314 / 1.452155 (-0.020841) | 1.495578 / 1.492716 (0.002862) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.223724 / 0.018006 (0.205718) | 0.416929 / 0.000490 (0.416440) | 0.005253 / 0.000200 (0.005053) | 0.000217 / 0.000054 (0.000163) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023571 / 0.037411 (-0.013841) | 0.073503 / 0.014526 (0.058978) | 0.081366 / 0.176557 (-0.095190) | 0.142716 / 0.737135 (-0.594420) | 0.082612 / 0.296338 (-0.213727) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.407319 / 0.215209 (0.192109) | 4.141404 / 2.077655 (2.063749) | 1.910842 / 1.504120 (0.406722) | 1.731694 / 1.541195 (0.190499) | 1.805228 / 1.468490 (0.336738) | 0.497109 / 4.584777 (-4.087668) | 3.107624 / 3.745712 (-0.638088) | 2.890687 / 5.269862 (-2.379174) | 1.795913 / 4.565676 (-2.769763) | 0.057099 / 0.424275 (-0.367176) | 0.006414 / 0.007607 (-0.001194) | 0.482127 / 0.226044 (0.256083) | 4.835158 / 2.268929 (2.566229) | 2.368909 / 55.444624 (-53.075715) | 2.001608 / 6.876477 (-4.874868) | 2.004492 / 2.142072 (-0.137580) | 0.579910 / 4.805227 (-4.225317) | 0.123541 / 6.500664 (-6.377123) | 0.059651 / 0.075469 (-0.015818) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.242364 / 1.841788 (-0.599424) | 16.982676 / 8.074308 (8.908368) | 13.718885 / 10.191392 (3.527493) | 0.132759 / 0.680424 (-0.547665) | 0.017012 / 0.534201 (-0.517189) | 0.333447 / 0.579283 (-0.245836) | 0.360149 / 0.434364 (-0.074215) | 0.385526 / 0.540337 (-0.154811) | 0.536915 / 1.386936 (-0.850021) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005946 / 0.011353 (-0.005407) | 0.003442 / 0.011008 (-0.007566) | 0.062595 / 0.038508 (0.024087) | 0.058699 / 0.023109 (0.035590) | 0.442626 / 0.275898 (0.166728) | 0.473773 / 0.323480 (0.150293) | 0.004622 / 0.007986 (-0.003364) | 0.002812 / 0.004328 (-0.001516) | 0.064099 / 0.004250 (0.059849) | 0.046784 / 0.037052 (0.009731) | 0.466049 / 0.258489 (0.207560) | 0.487912 / 0.293841 (0.194071) | 0.028372 / 0.128546 (-0.100174) | 0.007992 / 0.075646 (-0.067654) | 0.068151 / 0.419271 (-0.351120) | 0.041010 / 0.043533 (-0.002523) | 0.442331 / 0.255139 (0.187192) | 0.469686 / 0.283200 (0.186487) | 0.019694 / 0.141683 (-0.121989) | 1.467928 / 1.452155 (0.015774) | 1.525635 / 1.492716 (0.032918) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.204459 / 0.018006 (0.186453) | 0.407766 / 0.000490 (0.407276) | 0.003898 / 0.000200 (0.003698) | 0.000077 / 0.000054 (0.000023) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025909 / 0.037411 (-0.011503) | 0.080341 / 0.014526 (0.065816) | 0.088231 / 0.176557 (-0.088325) | 0.144056 / 0.737135 (-0.593079) | 0.089769 / 0.296338 (-0.206569) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.462876 / 0.215209 (0.247667) | 4.625983 / 2.077655 (2.548329) | 2.580079 / 1.504120 (1.075959) | 2.402792 / 1.541195 (0.861597) | 2.424982 / 1.468490 (0.956491) | 0.503654 / 4.584777 (-4.081123) | 3.178995 / 3.745712 (-0.566717) | 2.956126 / 5.269862 (-2.313735) | 1.847837 / 4.565676 (-2.717840) | 0.057964 / 0.424275 (-0.366311) | 0.006405 / 0.007607 (-0.001202) | 0.536036 / 0.226044 (0.309992) | 5.374416 / 2.268929 (3.105487) | 3.036440 / 55.444624 (-52.408184) | 2.682054 / 6.876477 (-4.194422) | 2.683462 / 2.142072 (0.541390) | 0.592751 / 4.805227 (-4.212477) | 0.124313 / 6.500664 (-6.376351) | 0.061127 / 0.075469 (-0.014342) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.383539 / 1.841788 (-0.458249) | 17.766221 / 8.074308 (9.691913) | 15.306600 / 10.191392 (5.115208) | 0.145035 / 0.680424 (-0.535389) | 0.018078 / 0.534201 (-0.516123) | 0.330102 / 0.579283 (-0.249181) | 0.375380 / 0.434364 (-0.058984) | 0.388531 / 0.540337 (-0.151807) | 0.548720 / 1.386936 (-0.838216) |

![](https://cml.dev/watermark.png#0082342ac792a05f4a615e4985d1c791e155115a "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006757 / 0.011353 (-0.004596) | 0.004110 / 0.011008 (-0.006898) | 0.084727 / 0.038508 (0.046219) | 0.074328 / 0.023109 (0.051219) | 0.310467 / 0.275898 (0.034569) | 0.343209 / 0.323480 (0.019729) | 0.004228 / 0.007986 (-0.003757) | 0.003400 / 0.004328 (-0.000929) | 0.065546 / 0.004250 (0.061296) | 0.063057 / 0.037052 (0.026005) | 0.315023 / 0.258489 (0.056534) | 0.356395 / 0.293841 (0.062554) | 0.031959 / 0.128546 (-0.096588) | 0.008577 / 0.075646 (-0.067069) | 0.289075 / 0.419271 (-0.130196) | 0.055011 / 0.043533 (0.011478) | 0.308861 / 0.255139 (0.053722) | 0.328691 / 0.283200 (0.045491) | 0.027037 / 0.141683 (-0.114646) | 1.464314 / 1.452155 (0.012159) | 1.549644 / 1.492716 (0.056927) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.238330 / 0.018006 (0.220324) | 0.451570 / 0.000490 (0.451080) | 0.010873 / 0.000200 (0.010673) | 0.000341 / 0.000054 (0.000286) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029909 / 0.037411 (-0.007503) | 0.085222 / 0.014526 (0.070696) | 0.100180 / 0.176557 (-0.076377) | 0.154842 / 0.737135 (-0.582293) | 0.099253 / 0.296338 (-0.197086) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.401603 / 0.215209 (0.186394) | 4.009781 / 2.077655 (1.932126) | 2.021807 / 1.504120 (0.517687) | 1.861017 / 1.541195 (0.319822) | 2.009072 / 1.468490 (0.540582) | 0.483798 / 4.584777 (-4.100979) | 3.580394 / 3.745712 (-0.165318) | 3.464587 / 5.269862 (-1.805275) | 2.018400 / 4.565676 (-2.547276) | 0.057134 / 0.424275 (-0.367141) | 0.007303 / 0.007607 (-0.000304) | 0.473627 / 0.226044 (0.247582) | 4.722634 / 2.268929 (2.453706) | 2.490884 / 55.444624 (-52.953741) | 2.121568 / 6.876477 (-4.754909) | 2.200699 / 2.142072 (0.058626) | 0.576728 / 4.805227 (-4.228499) | 0.135633 / 6.500664 (-6.365032) | 0.061625 / 0.075469 (-0.013844) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.250545 / 1.841788 (-0.591243) | 19.167642 / 8.074308 (11.093334) | 14.189891 / 10.191392 (3.998499) | 0.164552 / 0.680424 (-0.515872) | 0.018215 / 0.534201 (-0.515986) | 0.389962 / 0.579283 (-0.189321) | 0.413972 / 0.434364 (-0.020392) | 0.460253 / 0.540337 (-0.080085) | 0.647897 / 1.386936 (-0.739039) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006714 / 0.011353 (-0.004639) | 0.004081 / 0.011008 (-0.006927) | 0.065627 / 0.038508 (0.027119) | 0.077644 / 0.023109 (0.054535) | 0.409950 / 0.275898 (0.134052) | 0.442940 / 0.323480 (0.119460) | 0.005523 / 0.007986 (-0.002463) | 0.003366 / 0.004328 (-0.000962) | 0.065425 / 0.004250 (0.061174) | 0.056222 / 0.037052 (0.019169) | 0.429928 / 0.258489 (0.171439) | 0.457136 / 0.293841 (0.163296) | 0.032356 / 0.128546 (-0.096190) | 0.008676 / 0.075646 (-0.066970) | 0.071785 / 0.419271 (-0.347486) | 0.048458 / 0.043533 (0.004925) | 0.408003 / 0.255139 (0.152864) | 0.433529 / 0.283200 (0.150330) | 0.023232 / 0.141683 (-0.118450) | 1.483640 / 1.452155 (0.031485) | 1.552425 / 1.492716 (0.059709) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.282347 / 0.018006 (0.264341) | 0.448742 / 0.000490 (0.448253) | 0.039590 / 0.000200 (0.039390) | 0.000407 / 0.000054 (0.000353) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032516 / 0.037411 (-0.004896) | 0.095269 / 0.014526 (0.080744) | 0.106363 / 0.176557 (-0.070193) | 0.157945 / 0.737135 (-0.579191) | 0.106783 / 0.296338 (-0.189556) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.436334 / 0.215209 (0.221125) | 4.348147 / 2.077655 (2.270492) | 2.326830 / 1.504120 (0.822710) | 2.162586 / 1.541195 (0.621391) | 2.257769 / 1.468490 (0.789279) | 0.491677 / 4.584777 (-4.093099) | 3.707385 / 3.745712 (-0.038328) | 3.567147 / 5.269862 (-1.702715) | 2.099451 / 4.565676 (-2.466226) | 0.058486 / 0.424275 (-0.365789) | 0.007324 / 0.007607 (-0.000283) | 0.510962 / 0.226044 (0.284917) | 5.106550 / 2.268929 (2.837622) | 2.785723 / 55.444624 (-52.658901) | 2.452928 / 6.876477 (-4.423548) | 2.545034 / 2.142072 (0.402961) | 0.611124 / 4.805227 (-4.194103) | 0.133503 / 6.500664 (-6.367161) | 0.061118 / 0.075469 (-0.014351) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.386640 / 1.841788 (-0.455148) | 20.485670 / 8.074308 (12.411362) | 15.332223 / 10.191392 (5.140831) | 0.164070 / 0.680424 (-0.516354) | 0.019962 / 0.534201 (-0.514239) | 0.394217 / 0.579283 (-0.185066) | 0.428442 / 0.434364 (-0.005922) | 0.473784 / 0.540337 (-0.066553) | 0.665141 / 1.386936 (-0.721795) |

![](https://cml.dev/watermark.png#c722eb75a6cc56eac530c44a17ff679ca805aa89 "CML watermark")

The CI errors seem unrelated to this PR but I think they need further investigation in another PR.
```
FAILED tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_multiple_files - KeyError: 'url'
```

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008766 / 0.011353 (-0.002587) | 0.005289 / 0.011008 (-0.005720) | 0.097220 / 0.038508 (0.058712) | 0.072246 / 0.023109 (0.049137) | 0.369359 / 0.275898 (0.093461) | 0.422571 / 0.323480 (0.099091) | 0.004941 / 0.007986 (-0.003044) | 0.006103 / 0.004328 (0.001774) | 0.075828 / 0.004250 (0.071578) | 0.065795 / 0.037052 (0.028743) | 0.412835 / 0.258489 (0.154346) | 0.430062 / 0.293841 (0.136221) | 0.045806 / 0.128546 (-0.082741) | 0.013760 / 0.075646 (-0.061887) | 0.351542 / 0.419271 (-0.067729) | 0.064836 / 0.043533 (0.021304) | 0.370162 / 0.255139 (0.115023) | 0.434949 / 0.283200 (0.151749) | 0.039198 / 0.141683 (-0.102485) | 1.670940 / 1.452155 (0.218785) | 1.809677 / 1.492716 (0.316961) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.295104 / 0.018006 (0.277097) | 0.594584 / 0.000490 (0.594095) | 0.010923 / 0.000200 (0.010723) | 0.000479 / 0.000054 (0.000425) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029174 / 0.037411 (-0.008237) | 0.094637 / 0.014526 (0.080111) | 0.102948 / 0.176557 (-0.073608) | 0.171048 / 0.737135 (-0.566087) | 0.111465 / 0.296338 (-0.184873) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.582017 / 0.215209 (0.366808) | 5.727008 / 2.077655 (3.649354) | 2.563211 / 1.504120 (1.059091) | 2.308912 / 1.541195 (0.767717) | 2.301258 / 1.468490 (0.832768) | 0.819594 / 4.584777 (-3.765183) | 5.177536 / 3.745712 (1.431824) | 4.473602 / 5.269862 (-0.796260) | 2.743819 / 4.565676 (-1.821857) | 0.090052 / 0.424275 (-0.334223) | 0.007903 / 0.007607 (0.000295) | 0.679142 / 0.226044 (0.453097) | 6.887891 / 2.268929 (4.618962) | 3.337926 / 55.444624 (-52.106699) | 2.659228 / 6.876477 (-4.217249) | 2.641289 / 2.142072 (0.499216) | 0.974829 / 4.805227 (-3.830398) | 0.205775 / 6.500664 (-6.294890) | 0.075268 / 0.075469 (-0.000201) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.500562 / 1.841788 (-0.341226) | 22.688483 / 8.074308 (14.614175) | 19.634878 / 10.191392 (9.443486) | 0.227409 / 0.680424 (-0.453015) | 0.029794 / 0.534201 (-0.504407) | 0.475204 / 0.579283 (-0.104079) | 0.579379 / 0.434364 (0.145016) | 0.541244 / 0.540337 (0.000907) | 0.739187 / 1.386936 (-0.647749) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008641 / 0.011353 (-0.002712) | 0.006139 / 0.011008 (-0.004870) | 0.075048 / 0.038508 (0.036540) | 0.074070 / 0.023109 (0.050961) | 0.508288 / 0.275898 (0.232390) | 0.539770 / 0.323480 (0.216290) | 0.006092 / 0.007986 (-0.001894) | 0.003748 / 0.004328 (-0.000581) | 0.077945 / 0.004250 (0.073695) | 0.056989 / 0.037052 (0.019936) | 0.526889 / 0.258489 (0.268400) | 0.560862 / 0.293841 (0.267021) | 0.046507 / 0.128546 (-0.082040) | 0.013249 / 0.075646 (-0.062397) | 0.088363 / 0.419271 (-0.330908) | 0.058776 / 0.043533 (0.015243) | 0.495869 / 0.255139 (0.240730) | 0.538615 / 0.283200 (0.255415) | 0.034055 / 0.141683 (-0.107628) | 1.658713 / 1.452155 (0.206558) | 1.736599 / 1.492716 (0.243883) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.288355 / 0.018006 (0.270349) | 0.571481 / 0.000490 (0.570991) | 0.006765 / 0.000200 (0.006565) | 0.000101 / 0.000054 (0.000047) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031836 / 0.037411 (-0.005575) | 0.101312 / 0.014526 (0.086786) | 0.111433 / 0.176557 (-0.065124) | 0.169599 / 0.737135 (-0.567536) | 0.114595 / 0.296338 (-0.181743) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.645258 / 0.215209 (0.430049) | 6.446653 / 2.077655 (4.368998) | 2.983498 / 1.504120 (1.479379) | 2.573820 / 1.541195 (1.032625) | 2.624286 / 1.468490 (1.155796) | 0.815997 / 4.584777 (-3.768780) | 5.140248 / 3.745712 (1.394536) | 4.636915 / 5.269862 (-0.632947) | 2.866313 / 4.565676 (-1.699364) | 0.096643 / 0.424275 (-0.327633) | 0.008452 / 0.007607 (0.000845) | 0.765837 / 0.226044 (0.539793) | 7.622897 / 2.268929 (5.353968) | 3.796247 / 55.444624 (-51.648378) | 3.019349 / 6.876477 (-3.857128) | 3.034187 / 2.142072 (0.892115) | 1.001682 / 4.805227 (-3.803546) | 0.211841 / 6.500664 (-6.288823) | 0.073351 / 0.075469 (-0.002119) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.740254 / 1.841788 (-0.101534) | 23.465619 / 8.074308 (15.391311) | 21.651670 / 10.191392 (11.460278) | 0.226129 / 0.680424 (-0.454294) | 0.029611 / 0.534201 (-0.504590) | 0.441140 / 0.579283 (-0.138143) | 0.605591 / 0.434364 (0.171227) | 0.552427 / 0.540337 (0.012090) | 0.771975 / 1.386936 (-0.614961) |

![](https://cml.dev/watermark.png#ef5751522c424c758df0647ff9a449b8b0404b6a "CML watermark")

> The CI errors seem unrelated to this PR but I think they need further investigation in another PR.
>
> ```
> FAILED tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_multiple_files - KeyError: 'url'
> ```

We need to wait for `huggingface_hub`'s next release to fix this (see https://github.com/huggingface/huggingface_hub/pull/1675; 409 error is currently ignored, hence the `KeyError`)

Also, we should be able to fix `test_push_dataset_dict_to_hub_overwrite_files` by inserting `gc.collect()` (to drop the  "reference" to an Arrow file) between the `load_dataset` calls to avoid the `PermissionError` (also reported in https://github.com/huggingface/datasets/issues/3139)

(Indeed, this can be addressed in subsequent PRs.)

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008988 / 0.011353 (-0.002365) | 0.005270 / 0.011008 (-0.005738) | 0.114577 / 0.038508 (0.076068) | 0.091630 / 0.023109 (0.068521) | 0.409217 / 0.275898 (0.133319) | 0.440903 / 0.323480 (0.117424) | 0.005226 / 0.007986 (-0.002760) | 0.004289 / 0.004328 (-0.000040) | 0.082246 / 0.004250 (0.077995) | 0.084926 / 0.037052 (0.047873) | 0.407822 / 0.258489 (0.149333) | 0.440891 / 0.293841 (0.147051) | 0.052225 / 0.128546 (-0.076321) | 0.014218 / 0.075646 (-0.061429) | 0.436994 / 0.419271 (0.017722) | 0.066433 / 0.043533 (0.022901) | 0.413909 / 0.255139 (0.158770) | 0.425729 / 0.283200 (0.142530) | 0.039576 / 0.141683 (-0.102107) | 1.905604 / 1.452155 (0.453449) | 1.907032 / 1.492716 (0.414315) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.313662 / 0.018006 (0.295655) | 0.614541 / 0.000490 (0.614051) | 0.015631 / 0.000200 (0.015431) | 0.000507 / 0.000054 (0.000453) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029049 / 0.037411 (-0.008362) | 0.094626 / 0.014526 (0.080100) | 0.104718 / 0.176557 (-0.071838) | 0.187346 / 0.737135 (-0.549790) | 0.108001 / 0.296338 (-0.188337) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.578997 / 0.215209 (0.363788) | 5.815546 / 2.077655 (3.737892) | 2.411301 / 1.504120 (0.907181) | 2.110088 / 1.541195 (0.568893) | 2.147839 / 1.468490 (0.679349) | 0.861285 / 4.584777 (-3.723492) | 5.264245 / 3.745712 (1.518533) | 4.695786 / 5.269862 (-0.574076) | 2.867522 / 4.565676 (-1.698154) | 0.096523 / 0.424275 (-0.327752) | 0.008777 / 0.007607 (0.001170) | 0.716316 / 0.226044 (0.490272) | 7.257574 / 2.268929 (4.988645) | 3.141502 / 55.444624 (-52.303123) | 2.480604 / 6.876477 (-4.395872) | 2.530031 / 2.142072 (0.387958) | 1.054274 / 4.805227 (-3.750953) | 0.210781 / 6.500664 (-6.289883) | 0.073837 / 0.075469 (-0.001632) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.607689 / 1.841788 (-0.234099) | 23.856780 / 8.074308 (15.782472) | 19.507196 / 10.191392 (9.315804) | 0.232712 / 0.680424 (-0.447712) | 0.027037 / 0.534201 (-0.507164) | 0.466613 / 0.579283 (-0.112670) | 0.571139 / 0.434364 (0.136775) | 0.543109 / 0.540337 (0.002771) | 0.785558 / 1.386936 (-0.601378) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008104 / 0.011353 (-0.003249) | 0.004923 / 0.011008 (-0.006086) | 0.075093 / 0.038508 (0.036585) | 0.075218 / 0.023109 (0.052109) | 0.476615 / 0.275898 (0.200717) | 0.506984 / 0.323480 (0.183504) | 0.006371 / 0.007986 (-0.001614) | 0.004818 / 0.004328 (0.000489) | 0.075634 / 0.004250 (0.071383) | 0.059513 / 0.037052 (0.022461) | 0.523763 / 0.258489 (0.265274) | 0.531858 / 0.293841 (0.238017) | 0.048168 / 0.128546 (-0.080379) | 0.014110 / 0.075646 (-0.061537) | 0.086052 / 0.419271 (-0.333219) | 0.058369 / 0.043533 (0.014836) | 0.475537 / 0.255139 (0.220398) | 0.509429 / 0.283200 (0.226229) | 0.033924 / 0.141683 (-0.107758) | 1.657490 / 1.452155 (0.205336) | 1.762544 / 1.492716 (0.269828) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.263863 / 0.018006 (0.245857) | 0.584468 / 0.000490 (0.583978) | 0.007063 / 0.000200 (0.006863) | 0.000181 / 0.000054 (0.000126) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032229 / 0.037411 (-0.005183) | 0.096750 / 0.014526 (0.082224) | 0.117798 / 0.176557 (-0.058758) | 0.173376 / 0.737135 (-0.563760) | 0.117241 / 0.296338 (-0.179098) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.701935 / 0.215209 (0.486726) | 6.544655 / 2.077655 (4.467001) | 3.055531 / 1.504120 (1.551411) | 2.896339 / 1.541195 (1.355144) | 3.013157 / 1.468490 (1.544667) | 0.852989 / 4.584777 (-3.731788) | 5.399355 / 3.745712 (1.653643) | 5.119811 / 5.269862 (-0.150051) | 3.167269 / 4.565676 (-1.398407) | 0.096962 / 0.424275 (-0.327313) | 0.008843 / 0.007607 (0.001236) | 0.776170 / 0.226044 (0.550125) | 7.735093 / 2.268929 (5.466164) | 3.792629 / 55.444624 (-51.651996) | 3.249911 / 6.876477 (-3.626565) | 3.235590 / 2.142072 (1.093517) | 1.046426 / 4.805227 (-3.758801) | 0.239854 / 6.500664 (-6.260810) | 0.100648 / 0.075469 (0.025179) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.774488 / 1.841788 (-0.067300) | 25.646958 / 8.074308 (17.572650) | 23.181577 / 10.191392 (12.990185) | 0.231948 / 0.680424 (-0.448476) | 0.030147 / 0.534201 (-0.504054) | 0.464161 / 0.579283 (-0.115122) | 0.598980 / 0.434364 (0.164616) | 0.571156 / 0.540337 (0.030819) | 0.833221 / 1.386936 (-0.553715) |

![](https://cml.dev/watermark.png#ad876e8908188dcd56759a35c4da182bf121535a "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006010 / 0.011353 (-0.005343) | 0.003662 / 0.011008 (-0.007346) | 0.079971 / 0.038508 (0.041463) | 0.066790 / 0.023109 (0.043681) | 0.311387 / 0.275898 (0.035489) | 0.346781 / 0.323480 (0.023301) | 0.003500 / 0.007986 (-0.004485) | 0.002831 / 0.004328 (-0.001498) | 0.063238 / 0.004250 (0.058988) | 0.056163 / 0.037052 (0.019110) | 0.317456 / 0.258489 (0.058967) | 0.356106 / 0.293841 (0.062265) | 0.027358 / 0.128546 (-0.101188) | 0.007906 / 0.075646 (-0.067741) | 0.261779 / 0.419271 (-0.157492) | 0.046385 / 0.043533 (0.002852) | 0.312587 / 0.255139 (0.057448) | 0.339513 / 0.283200 (0.056314) | 0.021474 / 0.141683 (-0.120209) | 1.418637 / 1.452155 (-0.033518) | 1.510257 / 1.492716 (0.017540) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.211761 / 0.018006 (0.193755) | 0.424387 / 0.000490 (0.423898) | 0.002579 / 0.000200 (0.002379) | 0.000065 / 0.000054 (0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024038 / 0.037411 (-0.013374) | 0.072524 / 0.014526 (0.057998) | 0.083443 / 0.176557 (-0.093113) | 0.144835 / 0.737135 (-0.592300) | 0.084754 / 0.296338 (-0.211585) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.392423 / 0.215209 (0.177214) | 3.927220 / 2.077655 (1.849565) | 1.877853 / 1.504120 (0.373733) | 1.699275 / 1.541195 (0.158081) | 1.793144 / 1.468490 (0.324654) | 0.503809 / 4.584777 (-4.080968) | 3.052569 / 3.745712 (-0.693143) | 2.907432 / 5.269862 (-2.362429) | 1.811220 / 4.565676 (-2.754457) | 0.057249 / 0.424275 (-0.367026) | 0.006433 / 0.007607 (-0.001174) | 0.463257 / 0.226044 (0.237213) | 4.631038 / 2.268929 (2.362109) | 2.315870 / 55.444624 (-53.128754) | 2.000476 / 6.876477 (-4.876001) | 2.043581 / 2.142072 (-0.098492) | 0.588911 / 4.805227 (-4.216317) | 0.125370 / 6.500664 (-6.375295) | 0.061721 / 0.075469 (-0.013748) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.244486 / 1.841788 (-0.597301) | 17.862422 / 8.074308 (9.788114) | 13.890205 / 10.191392 (3.698813) | 0.145467 / 0.680424 (-0.534957) | 0.016856 / 0.534201 (-0.517345) | 0.329357 / 0.579283 (-0.249926) | 0.367550 / 0.434364 (-0.066814) | 0.377541 / 0.540337 (-0.162796) | 0.534087 / 1.386936 (-0.852849) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006030 / 0.011353 (-0.005323) | 0.003650 / 0.011008 (-0.007359) | 0.063300 / 0.038508 (0.024792) | 0.058877 / 0.023109 (0.035767) | 0.454662 / 0.275898 (0.178764) | 0.489362 / 0.323480 (0.165882) | 0.004856 / 0.007986 (-0.003130) | 0.002909 / 0.004328 (-0.001420) | 0.063356 / 0.004250 (0.059105) | 0.047867 / 0.037052 (0.010814) | 0.465461 / 0.258489 (0.206972) | 0.506684 / 0.293841 (0.212843) | 0.028599 / 0.128546 (-0.099947) | 0.008076 / 0.075646 (-0.067570) | 0.068695 / 0.419271 (-0.350576) | 0.041487 / 0.043533 (-0.002045) | 0.448676 / 0.255139 (0.193537) | 0.471206 / 0.283200 (0.188007) | 0.020401 / 0.141683 (-0.121282) | 1.461181 / 1.452155 (0.009026) | 1.517079 / 1.492716 (0.024363) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.222827 / 0.018006 (0.204821) | 0.425074 / 0.000490 (0.424585) | 0.004153 / 0.000200 (0.003953) | 0.000081 / 0.000054 (0.000026) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026980 / 0.037411 (-0.010431) | 0.080786 / 0.014526 (0.066260) | 0.092040 / 0.176557 (-0.084517) | 0.146082 / 0.737135 (-0.591053) | 0.092739 / 0.296338 (-0.203600) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.461663 / 0.215209 (0.246454) | 4.604828 / 2.077655 (2.527173) | 2.566926 / 1.504120 (1.062806) | 2.394419 / 1.541195 (0.853224) | 2.458375 / 1.468490 (0.989885) | 0.505140 / 4.584777 (-4.079637) | 3.155916 / 3.745712 (-0.589796) | 3.014474 / 5.269862 (-2.255388) | 1.900296 / 4.565676 (-2.665380) | 0.058063 / 0.424275 (-0.366212) | 0.006409 / 0.007607 (-0.001198) | 0.541165 / 0.226044 (0.315120) | 5.410700 / 2.268929 (3.141772) | 3.010239 / 55.444624 (-52.434386) | 2.668103 / 6.876477 (-4.208373) | 2.730418 / 2.142072 (0.588346) | 0.603471 / 4.805227 (-4.201756) | 0.129852 / 6.500664 (-6.370812) | 0.061507 / 0.075469 (-0.013962) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.355272 / 1.841788 (-0.486516) | 18.170088 / 8.074308 (10.095780) | 15.583855 / 10.191392 (5.392463) | 0.146246 / 0.680424 (-0.534178) | 0.018093 / 0.534201 (-0.516108) | 0.331695 / 0.579283 (-0.247588) | 0.380845 / 0.434364 (-0.053519) | 0.388564 / 0.540337 (-0.151774) | 0.551465 / 1.386936 (-0.835471) |

![](https://cml.dev/watermark.png#afc3c2b034481a3502f5476186a110cf8613a248 "CML watermark")

Currently our CI usually raises 404 errors when trying to delete temporary repositories. See, e.g.: https://github.com/huggingface/datasets/actions/runs/6314980985/job/17146507884
```
FAILED tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_multiple_files_with_max_shard_size - huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6512fb99-4a52c561752ece3d77eb6d57;2b61cae4-613d-4a73-bbb1-2faf9e32b02d)
Repository Not Found for url: https://hub-ci.huggingface.co/api/repos/delete.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.

FAILED tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_to_hub_custom_features_audio - huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6512fbb2-0333dd666d42f0e173c2bb68;dfdc4271-b49b-4008-8c49-f05cf7c1d53d)
Repository Not Found for url: https://hub-ci.huggingface.co/api/repos/delete.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.

FAILED tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_custom_splits - huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6512fbca-167690694f39770a5b3a444e;baeaa905-0a57-4585-ac97-9aaae12dd47d)
Repository Not Found for url: https://hub-ci.huggingface.co/api/repos/delete.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.
```

I think this can be caused by collisions in temporary repository IDs because we create them in multiprocessing:
```python
with temporary_repo(f"{CI_HUB_USER}/test-{int(time.time() * 10e3)}") as ds_name:
```
This can also be caused when there is another issue that does not allow the creation of the repository, thus making it impossible to delete it.

This PR tries to fix this issue by increasing the precision of the number on the repository ID: `10e6` instead of `10e3`.
Additionally, this PR catches RepositoryNotFoundError.


Can't load a dataset

`JourneyDB/JourneyDB` is a gated dataset, so this error means you are not authenticated to access it, either by using an invalid token or by not agreeing to the terms in the dialog on the dataset page.

> I believe is due to the fact that doesn't work with .tgz files.

Indeed, the dataset's data files structure is not supported natively by `datasets`. To load it, one option is to clone the repo (or download it with `huggingface_hub.snapshot_download`) and use `Dataset.from_generator` to process the files.

> JourneyDB/JourneyDB is a gated dataset, so this error means you are not authenticated to access it, either by using an invalid token or by not agreeing to the terms in the dialog on the dataset page.´

I did authentication with:

```
from huggingface_hub import notebook_login
notebook_login()
```

Isn't that the correct way to do it?

> Indeed, the dataset's data files structure is not supported natively by datasets. To load it, one option is to clone the repo (or download it with huggingface_hub.snapshot_download) and use Dataset.from_generator to process the files.

Great suggestion I will give it a try.

Have you accepted the terms in the dialog [here](https://huggingface.co/datasets/JourneyDB/JourneyDB)?

IIRC Kaggle preinstalls an outdated `datasets` version, so it's also a good idea to update it before importing `datasets` (and do the same for `huggingface_hub`)

Sorry for the late reply.  Yes, I did. Thanks for the tip!

### Describe the bug

Can't seem to load the JourneyDB dataset.

It throws the following error:

 ```
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[15], line 2
      1 # If the dataset is gated/private, make sure you have run huggingface-cli login
----> 2 dataset = load_dataset("JourneyDB/JourneyDB", data_files="data", use_auth_token=True)

File /opt/conda/lib/python3.10/site-packages/datasets/load.py:1664, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1661 ignore_verifications = ignore_verifications or save_infos
   1663 # Create a dataset builder
-> 1664 builder_instance = load_dataset_builder(
   1665     path=path,
   1666     name=name,
   1667     data_dir=data_dir,
   1668     data_files=data_files,
   1669     cache_dir=cache_dir,
   1670     features=features,
   1671     download_config=download_config,
   1672     download_mode=download_mode,
   1673     revision=revision,
   1674     use_auth_token=use_auth_token,
   1675     **config_kwargs,
   1676 )
   1678 # Return iterable dataset in case of streaming
   1679 if streaming:

File /opt/conda/lib/python3.10/site-packages/datasets/load.py:1490, in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, use_auth_token, **config_kwargs)
   1488     download_config = download_config.copy() if download_config else DownloadConfig()
   1489     download_config.use_auth_token = use_auth_token
-> 1490 dataset_module = dataset_module_factory(
   1491     path,
   1492     revision=revision,
   1493     download_config=download_config,
   1494     download_mode=download_mode,
   1495     data_dir=data_dir,
   1496     data_files=data_files,
   1497 )
   1499 # Get dataset builder class from the processing script
   1500 builder_cls = import_main_class(dataset_module.module_path)

File /opt/conda/lib/python3.10/site-packages/datasets/load.py:1238, in dataset_module_factory(path, revision, download_config, download_mode, force_local_path, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1236                 raise ConnectionError(f"Couln't reach the Hugging Face Hub for dataset '{path}': {e1}") from None
   1237             if isinstance(e1, FileNotFoundError):
-> 1238                 raise FileNotFoundError(
   1239                     f"Couldn't find a dataset script at {relative_to_absolute_path(combined_path)} or any data file in the same directory. "
   1240                     f"Couldn't find '{path}' on the Hugging Face Hub either: {type(e1).__name__}: {e1}"
   1241                 ) from None
   1242             raise e1 from None
   1243 else:

FileNotFoundError: Couldn't find a dataset script at /kaggle/working/JourneyDB/JourneyDB/JourneyDB.py or any data file in the same directory. Couldn't find 'JourneyDB/JourneyDB' on the Hugging Face Hub either: FileNotFoundError: Unable to find data in dataset repository JourneyDB/JourneyDB with any supported extension ['csv', 'tsv', 'json', 'jsonl', 'parquet', 'txt', 'blp', 'bmp', 'dib', 'bufr', 'cur', 'pcx', 'dcx', 'dds', 'ps', 'eps', 'fit', 'fits', 'fli', 'flc', 'ftc', 'ftu', 'gbr', 'gif', 'grib', 'h5', 'hdf', 'png', 'apng', 'jp2', 'j2k', 'jpc', 'jpf', 'jpx', 'j2c', 'icns', 'ico', 'im', 'iim', 'tif', 'tiff', 'jfif', 'jpe', 'jpg', 'jpeg', 'mpg', 'mpeg', 'msp', 'pcd', 'pxr', 'pbm', 'pgm', 'ppm', 'pnm', 'psd', 'bw', 'rgb', 'rgba', 'sgi', 'ras', 'tga', 'icb', 'vda', 'vst', 'webp', 'wmf', 'emf', 'xbm', 'xpm', 'zip']

```

### Steps to reproduce the bug

1)
```
from huggingface_hub import notebook_login
notebook_login()
```

2)
```
!pip install -q datasets
from datasets import load_dataset
```

3)
`dataset = load_dataset("JourneyDB/JourneyDB", data_files="data", use_auth_token=True)`

### Expected behavior

Load the dataset

### Environment info

Notebook


REUSE_DATASET_IF_EXISTS don't work

Hi! Unfortunately, the current behavior is to delete the downloaded data when this error happens. So, I've opened a PR that removes the problematic import to avoid losing data due to `apache_beam` not being installed (we host the preprocessed version of `natual_questions` on the HF GCS, so requiring `apache_beam` in that case doesn't make sense)

Thanks for your reply. I met another question that I set `export HF_DATASETS_CACHE=/data/lxy/.cache` , but each time I run load_datasets, the datasets module still looking for NQ in the wrong default cache dir '/home/lxy/.cache' 。How to avoid this incorrect behavior. I am sure HF_DATASETS_CACHE was set correctly since I use echo & to check it.
![image](https://github.com/huggingface/datasets/assets/88258534/e7029f27-b9f9-496c-8948-6234ef695646)
by the way I delete the file in '/home/lxy/.cache' since I found there has some kb size file seems useless.

You need to set this variable before the `datasets` import. Then, you can use `import datasets; datasets.config.HF_DATASETS_CACHE` to verify the cache location.

### Describe the bug

I use the following code  to download natural_question dataset. Even though I have completely download it, the next time I run this code, the new download procedure will start and cover the original /data/lxy/NQ

config=datasets.DownloadConfig(resume_download=True,max_retries=100,cache_dir=r'/data/lxy/NQ',download_desc='NQ')

data=datasets.load_dataset('natural_questions',cache_dir=r'/data/lxy/NQ',download_config=config,download_mode=DownloadMode.REUSE_DATASET_IF_EXISTS)

---
Since I don't have apache_beam installed, it throw a exception. After I pip install apache_beam ,the download restart..

![image](https://github.com/huggingface/datasets/assets/88258534/f28ce7fe-29ea-4348-b87f-e69182a8bd41)

### Steps to reproduce the bug

run this two line code

config=datasets.DownloadConfig(resume_download=True,max_retries=100,cache_dir=r'/data/lxy/NQ',download_desc='NQ')

data=datasets.load_dataset('natural_questions',cache_dir=r'/data/lxy/NQ',download_config=config,download_mode=DownloadMode.REUSE_DATASET_IF_EXISTS)

### Expected behavior

Download behavior can be correctly follow DownloadMode

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-3.10.0-1160.88.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.17
- Huggingface_hub version: 0.16.4
- PyArrow version: 11.0.0
- Pandas version: 2.0.3


Duplicated Rows When Loading Parquet Files from Root Directory with Subdirectories

Thanks for reporting this issue! We should be able to avoid this by making our `glob` patterns more precise. In the meantime, you can load the dataset by directly assigning splits to the data files:
```python
from datasets import load_dataset
ds = load_dataset("parquet", data_files={"train": "testing123/train/output_train.parquet", "validation": "testing123/val/output_val.parquet"})
```

### Describe the bug

When parquet files are saved in "train" and "val" subdirectories under a root directory, and datasets are then loaded using `load_dataset("parquet", data_dir="root_directory")`, the resulting dataset has duplicated rows for both the training and validation sets.

### Steps to reproduce the bug

1. Create a root directory, e.g., "testing123".
2. Under "testing123", create two subdirectories: "train" and "val".
3. Create and save a parquet file with 3 unique rows in the "train" subdirectory.
4. Create and save a parquet file with 4 unique rows in the "val" subdirectory.
5. Load the datasets from the root directory using `load_dataset("parquet", data_dir="testing123")`
6. Iterate through the datasets and print the rows

Here's a collab reproducing these steps:

https://colab.research.google.com/drive/11NEdImnQ3OqJlwKSHRMhr7jCBesNdLY4?usp=sharing

### Expected behavior

- Training set should contain 3 unique rows.
- Validation set should contain 4 unique rows.

### Environment info

- `datasets` version: 2.14.5
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.17.2
- PyArrow version: 9.0.0
- Pandas version: 1.5.3


[DOCS] Fix typo: Elasticsearch

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006131 / 0.011353 (-0.005222) | 0.003682 / 0.011008 (-0.007327) | 0.081108 / 0.038508 (0.042600) | 0.061580 / 0.023109 (0.038471) | 0.395880 / 0.275898 (0.119982) | 0.427429 / 0.323480 (0.103949) | 0.003570 / 0.007986 (-0.004416) | 0.003874 / 0.004328 (-0.000455) | 0.063322 / 0.004250 (0.059072) | 0.049742 / 0.037052 (0.012690) | 0.396547 / 0.258489 (0.138058) | 0.434759 / 0.293841 (0.140918) | 0.028137 / 0.128546 (-0.100409) | 0.008103 / 0.075646 (-0.067544) | 0.262504 / 0.419271 (-0.156767) | 0.045944 / 0.043533 (0.002411) | 0.397659 / 0.255139 (0.142520) | 0.416479 / 0.283200 (0.133280) | 0.022870 / 0.141683 (-0.118813) | 1.478280 / 1.452155 (0.026126) | 1.543748 / 1.492716 (0.051031) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.228851 / 0.018006 (0.210845) | 0.432845 / 0.000490 (0.432355) | 0.005922 / 0.000200 (0.005722) | 0.000227 / 0.000054 (0.000172) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025545 / 0.037411 (-0.011867) | 0.073506 / 0.014526 (0.058980) | 0.087622 / 0.176557 (-0.088935) | 0.145455 / 0.737135 (-0.591680) | 0.085236 / 0.296338 (-0.211102) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.433083 / 0.215209 (0.217874) | 4.323121 / 2.077655 (2.245466) | 2.297947 / 1.504120 (0.793827) | 2.126405 / 1.541195 (0.585211) | 2.201635 / 1.468490 (0.733145) | 0.509902 / 4.584777 (-4.074875) | 3.116877 / 3.745712 (-0.628835) | 2.892949 / 5.269862 (-2.376912) | 1.866833 / 4.565676 (-2.698844) | 0.058087 / 0.424275 (-0.366189) | 0.006464 / 0.007607 (-0.001143) | 0.503594 / 0.226044 (0.277550) | 5.027634 / 2.268929 (2.758705) | 2.718030 / 55.444624 (-52.726595) | 2.373876 / 6.876477 (-4.502600) | 2.515496 / 2.142072 (0.373423) | 0.602648 / 4.805227 (-4.202579) | 0.126119 / 6.500664 (-6.374545) | 0.060623 / 0.075469 (-0.014846) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.236429 / 1.841788 (-0.605359) | 17.760532 / 8.074308 (9.686224) | 13.970093 / 10.191392 (3.778701) | 0.145455 / 0.680424 (-0.534969) | 0.017110 / 0.534201 (-0.517091) | 0.329649 / 0.579283 (-0.249634) | 0.366942 / 0.434364 (-0.067421) | 0.384418 / 0.540337 (-0.155920) | 0.552330 / 1.386936 (-0.834606) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006302 / 0.011353 (-0.005051) | 0.003677 / 0.011008 (-0.007331) | 0.062836 / 0.038508 (0.024328) | 0.063317 / 0.023109 (0.040207) | 0.449970 / 0.275898 (0.174072) | 0.480903 / 0.323480 (0.157423) | 0.005013 / 0.007986 (-0.002972) | 0.002934 / 0.004328 (-0.001394) | 0.062975 / 0.004250 (0.058724) | 0.051285 / 0.037052 (0.014233) | 0.448417 / 0.258489 (0.189928) | 0.486022 / 0.293841 (0.192181) | 0.029215 / 0.128546 (-0.099332) | 0.008189 / 0.075646 (-0.067457) | 0.068203 / 0.419271 (-0.351068) | 0.041942 / 0.043533 (-0.001591) | 0.445749 / 0.255139 (0.190610) | 0.465442 / 0.283200 (0.182243) | 0.020681 / 0.141683 (-0.121002) | 1.500704 / 1.452155 (0.048549) | 1.550511 / 1.492716 (0.057795) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.224922 / 0.018006 (0.206915) | 0.419714 / 0.000490 (0.419224) | 0.003804 / 0.000200 (0.003604) | 0.000082 / 0.000054 (0.000028) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026924 / 0.037411 (-0.010487) | 0.082400 / 0.014526 (0.067874) | 0.092193 / 0.176557 (-0.084363) | 0.147045 / 0.737135 (-0.590090) | 0.093173 / 0.296338 (-0.203166) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.462510 / 0.215209 (0.247300) | 4.635249 / 2.077655 (2.557594) | 2.627127 / 1.504120 (1.123007) | 2.442879 / 1.541195 (0.901684) | 2.502456 / 1.468490 (1.033966) | 0.506607 / 4.584777 (-4.078170) | 3.127348 / 3.745712 (-0.618364) | 2.901818 / 5.269862 (-2.368044) | 1.906876 / 4.565676 (-2.658801) | 0.058025 / 0.424275 (-0.366250) | 0.006442 / 0.007607 (-0.001165) | 0.534438 / 0.226044 (0.308394) | 5.352481 / 2.268929 (3.083553) | 3.058068 / 55.444624 (-52.386556) | 2.697310 / 6.876477 (-4.179167) | 2.873141 / 2.142072 (0.731069) | 0.594517 / 4.805227 (-4.210710) | 0.125369 / 6.500664 (-6.375295) | 0.061411 / 0.075469 (-0.014058) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.369549 / 1.841788 (-0.472238) | 17.933507 / 8.074308 (9.859199) | 14.890107 / 10.191392 (4.698715) | 0.154398 / 0.680424 (-0.526026) | 0.018021 / 0.534201 (-0.516180) | 0.335163 / 0.579283 (-0.244120) | 0.350396 / 0.434364 (-0.083968) | 0.397694 / 0.540337 (-0.142643) | 0.554853 / 1.386936 (-0.832083) |

![](https://cml.dev/watermark.png#f56fd9d6c877ffa6fb44fb832c13b61227c9cc5b "CML watermark")

Not ElasticSearch :)


HfHubHTTPError - exceeded our hourly quotas for action: commit

how is your dataset structured? (file types, how many commits and files are you trying to push, etc)

I succeeded in uploading it after several attempts with an hour gap between each attempt (inconvenient but worked). The final dataset is [here](https://huggingface.co/datasets/yuvalkirstain/pickapic_v2), code and context to the dataset can be found [here](https://github.com/yuvalkirstain/PickScore/).
I can close the issue if this behavior is intended, as most users probably do not need to upload large-scale datasets.

We could fix this by creating a single commit for all the (Parquet) shards in `push_to_hub` instead of one commit per shard, as we currently do.

@Wauplin Any updates on the 2-step commit process suggested by you that we need to implement this?

> Any updates on the 2-step commit process suggested by you that we need to implement this?

Re-prioritizing this, sorry. Will let you know but probably can be done this week.

### Describe the bug

I try to upload a very large dataset of images, and get the following error:
```
File /fsx-multigen/yuvalkirstain/miniconda/envs/pickapic/lib/python3.10/site-packages/huggingface_hub/hf_api.py:2712, in HfApi.create_commit(self, repo_id, operations, commit_message, commit_description, token, repo_type, revision, create_pr, num_threads, parent_commit, run_as_future)
   2710 try:
   2711     commit_resp = get_session().post(url=commit_url, headers=headers, data=data, params=params)
-> 2712     hf_raise_for_status(commit_resp, endpoint_name="commit")
   2713 except RepositoryNotFoundError as e:
   2714     e.append_to_message(_CREATE_COMMIT_NO_REPO_ERROR_MESSAGE)

File /fsx-multigen/yuvalkirstain/miniconda/envs/pickapic/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py:301, in hf_raise_for_status(response, endpoint_name)
    297     raise BadRequestError(message, response=response) from e
    299 # Convert `HTTPError` into a `HfHubHTTPError` to display request information
    300 # as well (request id and/or server error message)
--> 301 raise HfHubHTTPError(str(e), response=response) from e

HfHubHTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/api/datasets/yuvalkirstain/pickapic_v2/commit/main (Request ID: Root=1-65112399-12d63f7d7f28bfa40a36a0fd)

You have exceeded our hourly quotas for action: commit. We invite you to retry later.
```

this makes it much less convenient to host large datasets on HF hub.

### Steps to reproduce the bug

Upload a very large dataset of images

### Expected behavior

the upload to work well

### Environment info

- `datasets` version: 2.13.1
- Platform: Linux-5.15.0-1033-aws-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 1.5.3


load_dataset() function's cache_dir does not seems to work

Can you share the error message?

Also, it would help if you could check whether `huggingface_hub`'s download behaves the same:
```python
from huggingface_hub import snapshot_download
snapshot_download("trec", repo_type="dataset", cache_dir='/path/to/my/dir)
```

In the next major release, we aim to switch to `huggingface_hub` for file download/caching, but we could align the `cache_dir`'s `umask` behavior earlier than this if their solution works for your use case.

### Describe the bug

datasets version: 2.14.5

when trying to run the following command
trec = load_dataset('trec', split='train[:1000]', cache_dir='/path/to/my/dir')

I keep getting error saying the command does not have permission to the default cache directory on my macbook pro machine.

It seems the cache_dir parameter cannot change the dataset saving directory from the default

what ever explained in the https://huggingface.co/docs/datasets/cache does not seem to work

### Steps to reproduce the bug

datasets version: 2.14.5

when trying to run the following command
trec = load_dataset('trec', split='train[:1000]', cache_dir='/path/to/my/dir')

I keep getting error saying the command does not have permission to the default cache directory on my macbook pro machine.

It seems the cache_dir parameter cannot change the dataset saving directory from the default

what ever explained in the https://huggingface.co/docs/datasets/cache does not seem to work

### Expected behavior

the dataset should be saved to the cache_dir points to

### Environment info

datasets version: 2.14.5
macos X: Ventura 13.4.1 (c)