Fix ArrayXD cast

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006920 / 0.011353 (-0.004433) | 0.004306 / 0.011008 (-0.006703) | 0.085961 / 0.038508 (0.047453) | 0.087008 / 0.023109 (0.063899) | 0.308953 / 0.275898 (0.033055) | 0.349919 / 0.323480 (0.026440) | 0.005705 / 0.007986 (-0.002281) | 0.003565 / 0.004328 (-0.000763) | 0.066272 / 0.004250 (0.062022) | 0.056438 / 0.037052 (0.019385) | 0.312927 / 0.258489 (0.054437) | 0.363081 / 0.293841 (0.069240) | 0.031947 / 0.128546 (-0.096600) | 0.008801 / 0.075646 (-0.066845) | 0.288657 / 0.419271 (-0.130615) | 0.053746 / 0.043533 (0.010213) | 0.305815 / 0.255139 (0.050676) | 0.327174 / 0.283200 (0.043975) | 0.024863 / 0.141683 (-0.116820) | 1.489718 / 1.452155 (0.037563) | 1.566726 / 1.492716 (0.074009) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.289273 / 0.018006 (0.271266) | 0.555519 / 0.000490 (0.555029) | 0.006522 / 0.000200 (0.006322) | 0.000105 / 0.000054 (0.000051) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031968 / 0.037411 (-0.005443) | 0.085113 / 0.014526 (0.070587) | 0.103931 / 0.176557 (-0.072625) | 0.158471 / 0.737135 (-0.578665) | 0.102633 / 0.296338 (-0.193705) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.399592 / 0.215209 (0.184383) | 4.004453 / 2.077655 (1.926798) | 2.047224 / 1.504120 (0.543104) | 1.896203 / 1.541195 (0.355008) | 1.974056 / 1.468490 (0.505566) | 0.485964 / 4.584777 (-4.098813) | 3.650648 / 3.745712 (-0.095064) | 3.475953 / 5.269862 (-1.793908) | 2.168105 / 4.565676 (-2.397571) | 0.058167 / 0.424275 (-0.366108) | 0.007517 / 0.007607 (-0.000090) | 0.475386 / 0.226044 (0.249342) | 4.758300 / 2.268929 (2.489372) | 2.527540 / 55.444624 (-52.917085) | 2.180544 / 6.876477 (-4.695933) | 2.460148 / 2.142072 (0.318076) | 0.589944 / 4.805227 (-4.215284) | 0.136474 / 6.500664 (-6.364190) | 0.061462 / 0.075469 (-0.014007) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.245816 / 1.841788 (-0.595972) | 20.376958 / 8.074308 (12.302650) | 14.764579 / 10.191392 (4.573187) | 0.152436 / 0.680424 (-0.527988) | 0.018580 / 0.534201 (-0.515621) | 0.394680 / 0.579283 (-0.184603) | 0.424162 / 0.434364 (-0.010202) | 0.465604 / 0.540337 (-0.074733) | 0.658531 / 1.386936 (-0.728405) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007105 / 0.011353 (-0.004248) | 0.004441 / 0.011008 (-0.006567) | 0.068792 / 0.038508 (0.030284) | 0.080371 / 0.023109 (0.057262) | 0.430263 / 0.275898 (0.154365) | 0.451743 / 0.323480 (0.128263) | 0.005987 / 0.007986 (-0.001999) | 0.003639 / 0.004328 (-0.000690) | 0.065462 / 0.004250 (0.061212) | 0.059852 / 0.037052 (0.022800) | 0.438390 / 0.258489 (0.179901) | 0.458679 / 0.293841 (0.164838) | 0.033044 / 0.128546 (-0.095502) | 0.008845 / 0.075646 (-0.066802) | 0.071772 / 0.419271 (-0.347500) | 0.048840 / 0.043533 (0.005307) | 0.415707 / 0.255139 (0.160568) | 0.431216 / 0.283200 (0.148017) | 0.024422 / 0.141683 (-0.117260) | 1.502249 / 1.452155 (0.050094) | 1.566767 / 1.492716 (0.074050) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.311352 / 0.018006 (0.293346) | 0.550395 / 0.000490 (0.549906) | 0.005190 / 0.000200 (0.004990) | 0.000116 / 0.000054 (0.000062) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034264 / 0.037411 (-0.003147) | 0.098712 / 0.014526 (0.084186) | 0.110906 / 0.176557 (-0.065651) | 0.161670 / 0.737135 (-0.575465) | 0.111023 / 0.296338 (-0.185316) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.435296 / 0.215209 (0.220087) | 4.331231 / 2.077655 (2.253576) | 2.305009 / 1.504120 (0.800889) | 2.154492 / 1.541195 (0.613297) | 2.344017 / 1.468490 (0.875527) | 0.496924 / 4.584777 (-4.087853) | 3.750782 / 3.745712 (0.005070) | 3.380193 / 5.269862 (-1.889669) | 2.161239 / 4.565676 (-2.404438) | 0.058456 / 0.424275 (-0.365819) | 0.007395 / 0.007607 (-0.000212) | 0.507824 / 0.226044 (0.281780) | 5.081564 / 2.268929 (2.812635) | 2.824080 / 55.444624 (-52.620544) | 2.458835 / 6.876477 (-4.417642) | 2.747897 / 2.142072 (0.605824) | 0.600727 / 4.805227 (-4.204500) | 0.135085 / 6.500664 (-6.365579) | 0.060506 / 0.075469 (-0.014963) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.376873 / 1.841788 (-0.464915) | 21.211922 / 8.074308 (13.137614) | 15.022845 / 10.191392 (4.831453) | 0.195388 / 0.680424 (-0.485036) | 0.020268 / 0.534201 (-0.513933) | 0.398971 / 0.579283 (-0.180312) | 0.427588 / 0.434364 (-0.006776) | 0.478044 / 0.540337 (-0.062293) | 0.687904 / 1.386936 (-0.699033) |

![](https://cml.dev/watermark.png#7fb5fae8f79b3db4a94013aa2af7c63796ef2d64 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006134 / 0.011353 (-0.005219) | 0.003655 / 0.011008 (-0.007354) | 0.081295 / 0.038508 (0.042787) | 0.060202 / 0.023109 (0.037093) | 0.330005 / 0.275898 (0.054107) | 0.361219 / 0.323480 (0.037739) | 0.004766 / 0.007986 (-0.003220) | 0.002942 / 0.004328 (-0.001386) | 0.063322 / 0.004250 (0.059072) | 0.047844 / 0.037052 (0.010791) | 0.340375 / 0.258489 (0.081886) | 0.406301 / 0.293841 (0.112460) | 0.027474 / 0.128546 (-0.101072) | 0.007991 / 0.075646 (-0.067655) | 0.262746 / 0.419271 (-0.156526) | 0.045575 / 0.043533 (0.002042) | 0.324123 / 0.255139 (0.068984) | 0.344399 / 0.283200 (0.061199) | 0.021806 / 0.141683 (-0.119877) | 1.425390 / 1.452155 (-0.026765) | 1.487920 / 1.492716 (-0.004796) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.217504 / 0.018006 (0.199498) | 0.420878 / 0.000490 (0.420388) | 0.007312 / 0.000200 (0.007112) | 0.000218 / 0.000054 (0.000163) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023507 / 0.037411 (-0.013905) | 0.073493 / 0.014526 (0.058967) | 0.084857 / 0.176557 (-0.091700) | 0.145130 / 0.737135 (-0.592005) | 0.085204 / 0.296338 (-0.211135) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.388767 / 0.215209 (0.173557) | 3.877998 / 2.077655 (1.800344) | 1.881447 / 1.504120 (0.377327) | 1.714555 / 1.541195 (0.173360) | 1.772551 / 1.468490 (0.304061) | 0.505146 / 4.584777 (-4.079631) | 3.045471 / 3.745712 (-0.700241) | 2.834436 / 5.269862 (-2.435426) | 1.859896 / 4.565676 (-2.705780) | 0.057806 / 0.424275 (-0.366469) | 0.006378 / 0.007607 (-0.001229) | 0.458339 / 0.226044 (0.232294) | 4.588125 / 2.268929 (2.319196) | 2.302215 / 55.444624 (-53.142409) | 1.981297 / 6.876477 (-4.895180) | 2.152967 / 2.142072 (0.010895) | 0.590166 / 4.805227 (-4.215061) | 0.125753 / 6.500664 (-6.374911) | 0.061583 / 0.075469 (-0.013887) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.232195 / 1.841788 (-0.609593) | 17.761159 / 8.074308 (9.686851) | 13.829498 / 10.191392 (3.638106) | 0.131936 / 0.680424 (-0.548488) | 0.016909 / 0.534201 (-0.517292) | 0.332615 / 0.579283 (-0.246668) | 0.358149 / 0.434364 (-0.076215) | 0.384251 / 0.540337 (-0.156087) | 0.536453 / 1.386936 (-0.850483) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006253 / 0.011353 (-0.005100) | 0.003639 / 0.011008 (-0.007370) | 0.062810 / 0.038508 (0.024302) | 0.063761 / 0.023109 (0.040652) | 0.450538 / 0.275898 (0.174640) | 0.483793 / 0.323480 (0.160313) | 0.004973 / 0.007986 (-0.003013) | 0.002918 / 0.004328 (-0.001411) | 0.062140 / 0.004250 (0.057889) | 0.050328 / 0.037052 (0.013275) | 0.455860 / 0.258489 (0.197371) | 0.492399 / 0.293841 (0.198558) | 0.028928 / 0.128546 (-0.099618) | 0.008166 / 0.075646 (-0.067481) | 0.067860 / 0.419271 (-0.351411) | 0.040990 / 0.043533 (-0.002542) | 0.451343 / 0.255139 (0.196204) | 0.473769 / 0.283200 (0.190569) | 0.021585 / 0.141683 (-0.120097) | 1.451040 / 1.452155 (-0.001115) | 1.516065 / 1.492716 (0.023349) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.230994 / 0.018006 (0.212988) | 0.428404 / 0.000490 (0.427915) | 0.003777 / 0.000200 (0.003577) | 0.000074 / 0.000054 (0.000020) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027394 / 0.037411 (-0.010018) | 0.081692 / 0.014526 (0.067166) | 0.091568 / 0.176557 (-0.084988) | 0.146149 / 0.737135 (-0.590987) | 0.092200 / 0.296338 (-0.204139) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.467086 / 0.215209 (0.251877) | 4.664862 / 2.077655 (2.587207) | 2.575703 / 1.504120 (1.071583) | 2.396587 / 1.541195 (0.855392) | 2.506064 / 1.468490 (1.037574) | 0.511942 / 4.584777 (-4.072834) | 3.196320 / 3.745712 (-0.549392) | 2.916627 / 5.269862 (-2.353235) | 1.919372 / 4.565676 (-2.646305) | 0.058769 / 0.424275 (-0.365506) | 0.006487 / 0.007607 (-0.001120) | 0.539095 / 0.226044 (0.313051) | 5.404675 / 2.268929 (3.135746) | 2.988962 / 55.444624 (-52.455662) | 2.670134 / 6.876477 (-4.206343) | 2.837414 / 2.142072 (0.695342) | 0.614776 / 4.805227 (-4.190451) | 0.125806 / 6.500664 (-6.374858) | 0.061593 / 0.075469 (-0.013876) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.346171 / 1.841788 (-0.495617) | 18.374626 / 8.074308 (10.300318) | 14.508723 / 10.191392 (4.317331) | 0.146771 / 0.680424 (-0.533652) | 0.018438 / 0.534201 (-0.515763) | 0.336944 / 0.579283 (-0.242339) | 0.385631 / 0.434364 (-0.048733) | 0.391922 / 0.540337 (-0.148416) | 0.568904 / 1.386936 (-0.818032) |

![](https://cml.dev/watermark.png#3e8d420808718c9a1453a2e7ee3484ca12c9c70d "CML watermark")

Fix #6291


Move `exceptions.py` to `utils/exceptions.py`

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006695 / 0.011353 (-0.004658) | 0.004321 / 0.011008 (-0.006687) | 0.084558 / 0.038508 (0.046050) | 0.076290 / 0.023109 (0.053181) | 0.312331 / 0.275898 (0.036433) | 0.349854 / 0.323480 (0.026374) | 0.004267 / 0.007986 (-0.003719) | 0.003595 / 0.004328 (-0.000733) | 0.065077 / 0.004250 (0.060826) | 0.057461 / 0.037052 (0.020409) | 0.314989 / 0.258489 (0.056500) | 0.364767 / 0.293841 (0.070926) | 0.031726 / 0.128546 (-0.096820) | 0.008674 / 0.075646 (-0.066972) | 0.288282 / 0.419271 (-0.130990) | 0.052845 / 0.043533 (0.009312) | 0.317501 / 0.255139 (0.062362) | 0.333241 / 0.283200 (0.050041) | 0.026412 / 0.141683 (-0.115271) | 1.475648 / 1.452155 (0.023493) | 1.551656 / 1.492716 (0.058939) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.276512 / 0.018006 (0.258506) | 0.576350 / 0.000490 (0.575861) | 0.009518 / 0.000200 (0.009318) | 0.000280 / 0.000054 (0.000226) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029332 / 0.037411 (-0.008079) | 0.082904 / 0.014526 (0.068379) | 0.102516 / 0.176557 (-0.074041) | 0.159355 / 0.737135 (-0.577780) | 0.104112 / 0.296338 (-0.192226) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.379144 / 0.215209 (0.163935) | 3.785283 / 2.077655 (1.707629) | 1.833753 / 1.504120 (0.329633) | 1.667906 / 1.541195 (0.126711) | 1.751551 / 1.468490 (0.283061) | 0.480998 / 4.584777 (-4.103779) | 3.533433 / 3.745712 (-0.212279) | 3.343363 / 5.269862 (-1.926498) | 2.094169 / 4.565676 (-2.471508) | 0.056613 / 0.424275 (-0.367662) | 0.007410 / 0.007607 (-0.000197) | 0.455077 / 0.226044 (0.229033) | 4.541380 / 2.268929 (2.272452) | 2.269151 / 55.444624 (-53.175473) | 1.955663 / 6.876477 (-4.920814) | 2.227663 / 2.142072 (0.085591) | 0.580597 / 4.805227 (-4.224630) | 0.135034 / 6.500664 (-6.365630) | 0.062091 / 0.075469 (-0.013378) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.276295 / 1.841788 (-0.565492) | 20.072827 / 8.074308 (11.998519) | 14.296462 / 10.191392 (4.105070) | 0.164936 / 0.680424 (-0.515488) | 0.018415 / 0.534201 (-0.515786) | 0.390894 / 0.579283 (-0.188389) | 0.415515 / 0.434364 (-0.018849) | 0.462798 / 0.540337 (-0.077540) | 0.650099 / 1.386936 (-0.736837) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007218 / 0.011353 (-0.004135) | 0.004246 / 0.011008 (-0.006763) | 0.065818 / 0.038508 (0.027310) | 0.087315 / 0.023109 (0.064206) | 0.406449 / 0.275898 (0.130551) | 0.442008 / 0.323480 (0.118528) | 0.005752 / 0.007986 (-0.002233) | 0.003624 / 0.004328 (-0.000704) | 0.065349 / 0.004250 (0.061099) | 0.062423 / 0.037052 (0.025371) | 0.410099 / 0.258489 (0.151610) | 0.448929 / 0.293841 (0.155088) | 0.032498 / 0.128546 (-0.096048) | 0.008877 / 0.075646 (-0.066770) | 0.071611 / 0.419271 (-0.347661) | 0.048038 / 0.043533 (0.004506) | 0.407957 / 0.255139 (0.152818) | 0.424045 / 0.283200 (0.140846) | 0.025222 / 0.141683 (-0.116461) | 1.496191 / 1.452155 (0.044037) | 1.580765 / 1.492716 (0.088048) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.274798 / 0.018006 (0.256792) | 0.581410 / 0.000490 (0.580920) | 0.007302 / 0.000200 (0.007102) | 0.000160 / 0.000054 (0.000106) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034068 / 0.037411 (-0.003343) | 0.096116 / 0.014526 (0.081590) | 0.110234 / 0.176557 (-0.066323) | 0.163246 / 0.737135 (-0.573889) | 0.110250 / 0.296338 (-0.186089) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.442381 / 0.215209 (0.227172) | 4.427061 / 2.077655 (2.349406) | 2.361013 / 1.504120 (0.856893) | 2.185048 / 1.541195 (0.643853) | 2.312544 / 1.468490 (0.844054) | 0.498347 / 4.584777 (-4.086430) | 3.640839 / 3.745712 (-0.104873) | 3.353405 / 5.269862 (-1.916457) | 2.082038 / 4.565676 (-2.483638) | 0.058786 / 0.424275 (-0.365489) | 0.007403 / 0.007607 (-0.000205) | 0.517894 / 0.226044 (0.291850) | 5.184257 / 2.268929 (2.915329) | 2.838467 / 55.444624 (-52.606157) | 2.511116 / 6.876477 (-4.365361) | 2.757816 / 2.142072 (0.615743) | 0.644050 / 4.805227 (-4.161177) | 0.136446 / 6.500664 (-6.364218) | 0.062219 / 0.075469 (-0.013250) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.350916 / 1.841788 (-0.490872) | 20.549280 / 8.074308 (12.474972) | 14.697569 / 10.191392 (4.506177) | 0.149818 / 0.680424 (-0.530606) | 0.020187 / 0.534201 (-0.514014) | 0.396008 / 0.579283 (-0.183275) | 0.427535 / 0.434364 (-0.006829) | 0.484544 / 0.540337 (-0.055794) | 0.687076 / 1.386936 (-0.699860) |

![](https://cml.dev/watermark.png#02a0d7cc9bdbc745c355c0bf8a210d8bf0b90327 "CML watermark")

I'd rather be consistent with `huggingface_hub` and have this module in `utils/` with the exceptions exposed in `utils/__init__.py` ...

Ok, I'll close this PR.

> Maybe we could ask huggingface_hub to align with the rest of open-source libraries and expose the errors/exceptions at the root of the library...

cc @Wauplin

It would be nice to have an HF style guide to ensure consistency across our libraries 🙂.

I can expose exceptions at root level yes.

About having guidelines and consistency, let's try to do our best but it's not really in the essence of HF to formalize stuff in libraries :unamused:

I didn't notice the path while reviewing the PR yesterday :(


Fix parquet columns argument in streaming mode

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008112 / 0.011353 (-0.003241) | 0.004762 / 0.011008 (-0.006247) | 0.101349 / 0.038508 (0.062841) | 0.092361 / 0.023109 (0.069252) | 0.418429 / 0.275898 (0.142531) | 0.427332 / 0.323480 (0.103852) | 0.006112 / 0.007986 (-0.001874) | 0.003920 / 0.004328 (-0.000408) | 0.076813 / 0.004250 (0.072563) | 0.064361 / 0.037052 (0.027309) | 0.420526 / 0.258489 (0.162037) | 0.441576 / 0.293841 (0.147735) | 0.044760 / 0.128546 (-0.083787) | 0.010054 / 0.075646 (-0.065592) | 0.346063 / 0.419271 (-0.073209) | 0.077453 / 0.043533 (0.033920) | 0.412871 / 0.255139 (0.157732) | 0.408307 / 0.283200 (0.125107) | 0.033398 / 0.141683 (-0.108285) | 1.755825 / 1.452155 (0.303671) | 1.852347 / 1.492716 (0.359630) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.274201 / 0.018006 (0.256194) | 0.536375 / 0.000490 (0.535885) | 0.008076 / 0.000200 (0.007876) | 0.000159 / 0.000054 (0.000105) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033567 / 0.037411 (-0.003845) | 0.102378 / 0.014526 (0.087852) | 0.114176 / 0.176557 (-0.062381) | 0.180576 / 0.737135 (-0.556560) | 0.114801 / 0.296338 (-0.181538) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.450300 / 0.215209 (0.235091) | 4.490940 / 2.077655 (2.413285) | 2.172412 / 1.504120 (0.668292) | 1.978746 / 1.541195 (0.437551) | 2.065602 / 1.468490 (0.597112) | 0.571260 / 4.584777 (-4.013517) | 4.185485 / 3.745712 (0.439773) | 3.885594 / 5.269862 (-1.384268) | 2.532942 / 4.565676 (-2.032735) | 0.067612 / 0.424275 (-0.356663) | 0.008694 / 0.007607 (0.001087) | 0.533375 / 0.226044 (0.307331) | 5.321261 / 2.268929 (3.052333) | 2.697788 / 55.444624 (-52.746836) | 2.331328 / 6.876477 (-4.545149) | 2.585168 / 2.142072 (0.443096) | 0.681760 / 4.805227 (-4.123467) | 0.157687 / 6.500664 (-6.342977) | 0.071014 / 0.075469 (-0.004455) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.525689 / 1.841788 (-0.316098) | 23.162280 / 8.074308 (15.087972) | 16.644941 / 10.191392 (6.453548) | 0.182588 / 0.680424 (-0.497836) | 0.021653 / 0.534201 (-0.512548) | 0.466556 / 0.579283 (-0.112727) | 0.511902 / 0.434364 (0.077538) | 0.553707 / 0.540337 (0.013370) | 0.777830 / 1.386936 (-0.609106) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007954 / 0.011353 (-0.003399) | 0.004645 / 0.011008 (-0.006363) | 0.079096 / 0.038508 (0.040587) | 0.088200 / 0.023109 (0.065090) | 0.508882 / 0.275898 (0.232984) | 0.545986 / 0.323480 (0.222506) | 0.006233 / 0.007986 (-0.001752) | 0.004016 / 0.004328 (-0.000312) | 0.078103 / 0.004250 (0.073853) | 0.066354 / 0.037052 (0.029302) | 0.504132 / 0.258489 (0.245643) | 0.543714 / 0.293841 (0.249873) | 0.038140 / 0.128546 (-0.090407) | 0.011201 / 0.075646 (-0.064446) | 0.085713 / 0.419271 (-0.333559) | 0.057169 / 0.043533 (0.013637) | 0.488161 / 0.255139 (0.233022) | 0.516231 / 0.283200 (0.233031) | 0.027868 / 0.141683 (-0.113814) | 1.794084 / 1.452155 (0.341930) | 1.884993 / 1.492716 (0.392276) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.263108 / 0.018006 (0.245102) | 0.495761 / 0.000490 (0.495272) | 0.007056 / 0.000200 (0.006856) | 0.000117 / 0.000054 (0.000062) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.039089 / 0.037411 (0.001678) | 0.113332 / 0.014526 (0.098806) | 0.130137 / 0.176557 (-0.046419) | 0.189330 / 0.737135 (-0.547805) | 0.125860 / 0.296338 (-0.170479) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.530496 / 0.215209 (0.315287) | 5.349235 / 2.077655 (3.271581) | 2.975886 / 1.504120 (1.471766) | 2.786368 / 1.541195 (1.245173) | 2.920448 / 1.468490 (1.451958) | 0.575677 / 4.584777 (-4.009100) | 4.215535 / 3.745712 (0.469823) | 3.879984 / 5.269862 (-1.389878) | 2.420193 / 4.565676 (-2.145484) | 0.068506 / 0.424275 (-0.355769) | 0.008785 / 0.007607 (0.001178) | 0.611471 / 0.226044 (0.385427) | 6.118399 / 2.268929 (3.849471) | 3.509376 / 55.444624 (-51.935248) | 3.149219 / 6.876477 (-3.727257) | 3.413861 / 2.142072 (1.271788) | 0.697586 / 4.805227 (-4.107641) | 0.157767 / 6.500664 (-6.342897) | 0.071539 / 0.075469 (-0.003930) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.625196 / 1.841788 (-0.216591) | 24.347319 / 8.074308 (16.273011) | 17.365789 / 10.191392 (7.174397) | 0.217590 / 0.680424 (-0.462834) | 0.023885 / 0.534201 (-0.510316) | 0.477226 / 0.579283 (-0.102057) | 0.529319 / 0.434364 (0.094955) | 0.622299 / 0.540337 (0.081962) | 0.835295 / 1.386936 (-0.551641) |

![](https://cml.dev/watermark.png#3de42c8fae86c602fc71ac6d166e5c77f4149446 "CML watermark")

CI errors are unrelated or due to flaky tests

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006288 / 0.011353 (-0.005065) | 0.003836 / 0.011008 (-0.007172) | 0.080958 / 0.038508 (0.042450) | 0.065934 / 0.023109 (0.042825) | 0.312597 / 0.275898 (0.036699) | 0.351216 / 0.323480 (0.027736) | 0.004864 / 0.007986 (-0.003121) | 0.002961 / 0.004328 (-0.001368) | 0.063142 / 0.004250 (0.058892) | 0.049822 / 0.037052 (0.012770) | 0.320305 / 0.258489 (0.061816) | 0.363151 / 0.293841 (0.069310) | 0.027561 / 0.128546 (-0.100985) | 0.008176 / 0.075646 (-0.067470) | 0.261290 / 0.419271 (-0.157982) | 0.045517 / 0.043533 (0.001984) | 0.309218 / 0.255139 (0.054079) | 0.340140 / 0.283200 (0.056940) | 0.021000 / 0.141683 (-0.120683) | 1.448699 / 1.452155 (-0.003456) | 1.523904 / 1.492716 (0.031188) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.224294 / 0.018006 (0.206288) | 0.434928 / 0.000490 (0.434439) | 0.007541 / 0.000200 (0.007341) | 0.000286 / 0.000054 (0.000232) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025257 / 0.037411 (-0.012154) | 0.077364 / 0.014526 (0.062838) | 0.085825 / 0.176557 (-0.090732) | 0.148121 / 0.737135 (-0.589014) | 0.086838 / 0.296338 (-0.209500) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.396900 / 0.215209 (0.181691) | 3.953381 / 2.077655 (1.875727) | 1.933561 / 1.504120 (0.429441) | 1.760549 / 1.541195 (0.219354) | 1.824014 / 1.468490 (0.355523) | 0.495385 / 4.584777 (-4.089392) | 3.005558 / 3.745712 (-0.740154) | 2.931022 / 5.269862 (-2.338840) | 1.905113 / 4.565676 (-2.660563) | 0.057232 / 0.424275 (-0.367043) | 0.006472 / 0.007607 (-0.001135) | 0.464261 / 0.226044 (0.238216) | 4.629388 / 2.268929 (2.360459) | 2.342004 / 55.444624 (-53.102620) | 1.977295 / 6.876477 (-4.899181) | 2.167151 / 2.142072 (0.025079) | 0.582483 / 4.805227 (-4.222744) | 0.129444 / 6.500664 (-6.371220) | 0.061057 / 0.075469 (-0.014412) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.259444 / 1.841788 (-0.582344) | 18.189338 / 8.074308 (10.115030) | 14.313174 / 10.191392 (4.121782) | 0.146209 / 0.680424 (-0.534215) | 0.017115 / 0.534201 (-0.517086) | 0.336643 / 0.579283 (-0.242640) | 0.370824 / 0.434364 (-0.063540) | 0.387032 / 0.540337 (-0.153306) | 0.546688 / 1.386936 (-0.840248) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006371 / 0.011353 (-0.004982) | 0.003693 / 0.011008 (-0.007315) | 0.062499 / 0.038508 (0.023991) | 0.066367 / 0.023109 (0.043257) | 0.451481 / 0.275898 (0.175583) | 0.482495 / 0.323480 (0.159015) | 0.005676 / 0.007986 (-0.002310) | 0.002940 / 0.004328 (-0.001389) | 0.063011 / 0.004250 (0.058760) | 0.051500 / 0.037052 (0.014447) | 0.455482 / 0.258489 (0.196993) | 0.488888 / 0.293841 (0.195047) | 0.028714 / 0.128546 (-0.099832) | 0.008178 / 0.075646 (-0.067468) | 0.067218 / 0.419271 (-0.352053) | 0.041323 / 0.043533 (-0.002210) | 0.454007 / 0.255139 (0.198868) | 0.476241 / 0.283200 (0.193041) | 0.021530 / 0.141683 (-0.120153) | 1.457859 / 1.452155 (0.005705) | 1.506437 / 1.492716 (0.013721) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.228280 / 0.018006 (0.210274) | 0.427574 / 0.000490 (0.427084) | 0.003793 / 0.000200 (0.003593) | 0.000076 / 0.000054 (0.000022) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028420 / 0.037411 (-0.008992) | 0.087935 / 0.014526 (0.073409) | 0.092761 / 0.176557 (-0.083796) | 0.148084 / 0.737135 (-0.589051) | 0.095301 / 0.296338 (-0.201037) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.462457 / 0.215209 (0.247248) | 4.618016 / 2.077655 (2.540361) | 2.540531 / 1.504120 (1.036412) | 2.384696 / 1.541195 (0.843501) | 2.493108 / 1.468490 (1.024618) | 0.511689 / 4.584777 (-4.073088) | 3.173701 / 3.745712 (-0.572011) | 2.917046 / 5.269862 (-2.352816) | 1.916294 / 4.565676 (-2.649382) | 0.058969 / 0.424275 (-0.365306) | 0.006461 / 0.007607 (-0.001147) | 0.540997 / 0.226044 (0.314952) | 5.406596 / 2.268929 (3.137667) | 3.071189 / 55.444624 (-52.373435) | 2.701982 / 6.876477 (-4.174494) | 2.860194 / 2.142072 (0.718121) | 0.602684 / 4.805227 (-4.202543) | 0.127384 / 6.500664 (-6.373280) | 0.061718 / 0.075469 (-0.013751) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.340587 / 1.841788 (-0.501201) | 18.543831 / 8.074308 (10.469523) | 14.847319 / 10.191392 (4.655927) | 0.146523 / 0.680424 (-0.533901) | 0.018172 / 0.534201 (-0.516029) | 0.333276 / 0.579283 (-0.246007) | 0.375874 / 0.434364 (-0.058490) | 0.396766 / 0.540337 (-0.143572) | 0.572562 / 1.386936 (-0.814374) |

![](https://cml.dev/watermark.png#f2d9fcc0840f9d94f63635e9b40a1a7f11b34ea2 "CML watermark")

It was failing when there's a DatasetInfo with non-None info.features from the YAML (therefore containing columns that should be ignored)

Fix https://github.com/huggingface/datasets/issues/6293


IndexError: Invalid key is out of bounds for size 0 despite having a populated dataset

It looks to be the same issue as the one reported in https://discuss.huggingface.co/t/indexerror-invalid-key-16-is-out-of-bounds-for-size-0.

Can you check the length of `train_dataset` before the `train_sampler = self._get_train_sampler()` (and after `_remove_unused_columns`) line?

### Describe the bug

I am encountering an `IndexError` when trying to access data from a DataLoader which wraps around a dataset I've loaded using the `datasets` library. The error suggests that the dataset size is `0`, but when I check the length and print the dataset, it's clear that it has `1166` entries.

### Steps to reproduce the bug

1. Load a dataset with `1166` entries.
2. Create a DataLoader using this dataset.
3. Try iterating over the DataLoader.

code:
```python
def get_train_dataloader(self) -> DataLoader:
        if self.train_dataset is None:
            raise ValueError("Trainer: training requires a train_dataset.")

        train_dataset = self.train_dataset
        data_collator = self.data_collator
        print(len(train_dataset))
        print(train_dataset)
        if is_datasets_available() and isinstance(train_dataset, datasets.Dataset):
            train_dataset = self._remove_unused_columns(train_dataset, description="training")
        else:
            data_collator = self._get_collator_with_removed_columns(data_collator, description="training")
        train_sampler = self._get_train_sampler()
        dl = DataLoader(
            train_dataset,
            batch_size=self._train_batch_size,
            sampler=train_sampler,
            collate_fn=data_collator,
            drop_last=self.args.dataloader_drop_last,
            num_workers=self.args.dataloader_num_workers,
            pin_memory=self.args.dataloader_pin_memory,
            worker_init_fn=seed_worker,
        )
        print(dl)
        print(len(dl))
        for i in dl:
            print(i)
            break
        return dl
```

output :
```
1166
Dataset({
    features: ['input_ids', 'special_tokens_mask'],
    num_rows: 1166
})
<torch.utils.data.dataloader.DataLoader object ...>
146
```
Error:
```
Traceback (most recent call last):
  File "/home/dl/zym/llamaJP/TestUseContinuePretrainLlama.py", line 266, in <module>
    train()
  File "/home/dl/zym/llamaJP/TestUseContinuePretrainLlama.py", line 260, in train
    trainer.train()
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/transformers/trainer.py", line 1506, in train
    return inner_training_loop(
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/transformers/trainer.py", line 1520, in _inner_training_loop
    train_dataloader = self.get_train_dataloader()
  File "/home/dl/zym/llamaJP/TestUseContinuePretrainLlama.py", line 80, in get_train_dataloader
    for i in dl:
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
    data = self._next_data()
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 674, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = self.dataset.__getitems__(possibly_batched_index)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2807, in __getitems__
    batch = self.__getitem__(keys)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2803, in __getitem__
    return self._getitem(key)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2787, in _getitem
    pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 583, in query_table
    _check_valid_index_key(key, size)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 536, in _check_valid_index_key
    _check_valid_index_key(int(max(key)), size=size)
  File "/root/miniconda3/envs/LLM/lib/python3.10/site-packages/datasets/formatting/formatting.py", line 526, in _check_valid_index_key
    raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
IndexError: Invalid key: 1116 is out of bounds for size 0
```

### Expected behavior

I expect to be able to iterate over the DataLoader without encountering an IndexError since the dataset is populated.

### Environment info

- `datasets` library version: [2.14.5]
- Platform: [Linux]
- Python version: 3.10
- Other libraries involved: HuggingFace Transformers


Choose columns to stream parquet data in streaming mode

bug

Something isn't working

Currently passing columns= to load_dataset in streaming mode fails

```
Tried to load parquet data with columns '['link']' with mismatching features '{'caption': Value(dtype='string', id=None), 'image': {'bytes': Value(dtype='binary', id=None), 'path': Value(dtype='null', id=None)}, 'link': Value(dtype='string', id=None), 'message_id': Value(dtype='string', id=None), 'timestamp': Value(dtype='string', id=None)}'
```

similar to https://github.com/huggingface/datasets/issues/6039

reported at https://huggingface.co/datasets/laion/dalle-3-dataset/discussions/3#65259a09617407d4520f4ad9


how to load the image of dtype float32 or float64

Hi! Can you provide a code that reproduces the issue?

Also, which version of `datasets` are you using? You can check this by running `python -c "import datasets; print(datasets.__version__)"` inside the env. We added support for "float images" in `datasets 2.9`.

_FEATURES = datasets.Features(
    {
        "image": datasets.Image(),
        "text": datasets.Value("string"),
    },
)
The datasets builder seems only support the unit8 data. How to load the float dtype data?


Casting type from Array2D int to Array2D float crashes

Thanks for reporting! I've opened a PR with a fix

### Describe the bug

I am on a school project and the initial type for feature annotations are `Array2D(shape=(None, 4))`. I am trying to cast this type to a `float64` and pyarrow gives me this error :

```
Traceback (most recent call last):
  File "/home/alan/dev/ClassezDesImagesAvecDesAlgorithmesDeDeeplearning/src/sdd/data/dataset.py", line 141, in <module>
    dataset = StanfordDogsDataset(size, 5).original(True).demo()
  File "<attrs generated init __main__.StanfordDogsDataset>", line 4, in __init__
  File "/home/alan/dev/ClassezDesImagesAvecDesAlgorithmesDeDeeplearning/src/sdd/data/dataset.py", line 33, in __attrs_post_init__
    self.dataset = self.dataset.cast_column(
  File "/home/alan/.cache/pypoetry/virtualenvs/sdd-2XWLAjSi-py3.10/lib/python3.10/site-packages/datasets/fingerprint.py", line 511, in wrapper
    out = func(dataset, *args, **kwargs)
  File "/home/alan/.cache/pypoetry/virtualenvs/sdd-2XWLAjSi-py3.10/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2110, in cast_column
    return self.cast(features)
  File "/home/alan/.cache/pypoetry/virtualenvs/sdd-2XWLAjSi-py3.10/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 2055, in cast
    dataset = dataset.map(
  File "/home/alan/.cache/pypoetry/virtualenvs/sdd-2XWLAjSi-py3.10/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 592, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/alan/.cache/pypoetry/virtualenvs/sdd-2XWLAjSi-py3.10/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 557, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/alan/.cache/pypoetry/virtualenvs/sdd-2XWLAjSi-py3.10/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3097, in map
    for rank, done, content in Dataset._map_single(**dataset_kwargs):
  File "/home/alan/.cache/pypoetry/virtualenvs/sdd-2XWLAjSi-py3.10/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3474, in _map_single
    batch = apply_function_on_filtered_inputs(
  File "/home/alan/.cache/pypoetry/virtualenvs/sdd-2XWLAjSi-py3.10/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3353, in apply_function_on_filtered_inputs
    processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
  File "/home/alan/.cache/pypoetry/virtualenvs/sdd-2XWLAjSi-py3.10/lib/python3.10/site-packages/datasets/table.py", line 2328, in table_cast
    return cast_table_to_schema(table, schema)
  File "/home/alan/.cache/pypoetry/virtualenvs/sdd-2XWLAjSi-py3.10/lib/python3.10/site-packages/datasets/table.py", line 2287, in cast_table_to_schema
    arrays = [cast_array_to_feature(table[name], feature) for name, feature in features.items()]
  File "/home/alan/.cache/pypoetry/virtualenvs/sdd-2XWLAjSi-py3.10/lib/python3.10/site-packages/datasets/table.py", line 2287, in <listcomp>
    arrays = [cast_array_to_feature(table[name], feature) for name, feature in features.items()]
  File "/home/alan/.cache/pypoetry/virtualenvs/sdd-2XWLAjSi-py3.10/lib/python3.10/site-packages/datasets/table.py", line 1831, in wrapper
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/home/alan/.cache/pypoetry/virtualenvs/sdd-2XWLAjSi-py3.10/lib/python3.10/site-packages/datasets/table.py", line 1831, in <listcomp>
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/home/alan/.cache/pypoetry/virtualenvs/sdd-2XWLAjSi-py3.10/lib/python3.10/site-packages/datasets/table.py", line 2143, in cast_array_to_feature
    return array_cast(array, feature(), allow_number_to_str=allow_number_to_str)
  File "/home/alan/.cache/pypoetry/virtualenvs/sdd-2XWLAjSi-py3.10/lib/python3.10/site-packages/datasets/table.py", line 1833, in wrapper
    return func(array, *args, **kwargs)
  File "/home/alan/.cache/pypoetry/virtualenvs/sdd-2XWLAjSi-py3.10/lib/python3.10/site-packages/datasets/table.py", line 1967, in array_cast
    return pa_type.wrap_array(array)
  File "pyarrow/types.pxi", line 1369, in pyarrow.lib.BaseExtensionType.wrap_array
TypeError: Incompatible storage type for extension<arrow.py_extension_type<Array2DExtensionType>>: expected list<item: list<item: double>>, got list<item: list<item: int32>>
```

### Steps to reproduce the bug

```python
dataset = datasets.load_dataset("Alanox/stanford-dogs", split="full")
dataset = dataset.cast_column("annotations", Array2D((None, 4), "float64"))
```

### Expected behavior

It should simply cast the column feature type to a `float64` without error

### Environment info

datasets == 2.14.5


Incremental dataset (e.g. `.push_to_hub(..., append=True)`)

enhancement

New feature or request

Yea I think waiting for #6269 would be best, or branching from it. For reference, this [PR](https://github.com/LAION-AI/Discord-Scrapers/pull/2) is progressing pretty well which will do similar using the hf hub for our LAION dataset bot https://github.com/LAION-AI/Discord-Scrapers/pull/2.

### Feature request

Have the possibility to do `ds.push_to_hub(..., append=True)`.

### Motivation

Requested in this [comment](https://huggingface.co/datasets/laion/dalle-3-dataset/discussions/3#65252597c4edc168202a5eaa) and
this [comment](https://huggingface.co/datasets/laion/dalle-3-dataset/discussions/4#6524f675c9607bdffb208d8f). Discussed internally on [slack](https://huggingface.slack.com/archives/C02EMARJ65P/p1696950642610639?thread_ts=1690554266.830949&cid=C02EMARJ65P).

### Your contribution

What I suggest to do for parquet datasets is to use `CommitOperationCopy` + `CommitOperationDelete` from `huggingface_hub`:
1. list files
2. copy files from parquet-0001-of-0004 to parquet-0001-of-0005
3. delete files like parquet-0001-of-0004
4. generate + add last parquet file parquet-0005-of-0005

=> make a single commit with all commit operations at once

I think it should be quite straightforward to implement. Happy to review a PR (maybe conflicting with the ongoing "1 commit push_to_hub" PR https://github.com/huggingface/datasets/pull/6269)


testing doc-builder

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006424 / 0.011353 (-0.004929) | 0.003960 / 0.011008 (-0.007048) | 0.084022 / 0.038508 (0.045514) | 0.070770 / 0.023109 (0.047661) | 0.320525 / 0.275898 (0.044627) | 0.354507 / 0.323480 (0.031027) | 0.003939 / 0.007986 (-0.004047) | 0.004161 / 0.004328 (-0.000168) | 0.064754 / 0.004250 (0.060503) | 0.053630 / 0.037052 (0.016578) | 0.323948 / 0.258489 (0.065459) | 0.376908 / 0.293841 (0.083067) | 0.031063 / 0.128546 (-0.097483) | 0.008470 / 0.075646 (-0.067177) | 0.288110 / 0.419271 (-0.131161) | 0.053062 / 0.043533 (0.009529) | 0.328176 / 0.255139 (0.073037) | 0.345203 / 0.283200 (0.062003) | 0.024579 / 0.141683 (-0.117104) | 1.471649 / 1.452155 (0.019495) | 1.561458 / 1.492716 (0.068742) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.223591 / 0.018006 (0.205585) | 0.450758 / 0.000490 (0.450269) | 0.003751 / 0.000200 (0.003552) | 0.000079 / 0.000054 (0.000025) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027859 / 0.037411 (-0.009552) | 0.080607 / 0.014526 (0.066081) | 0.093835 / 0.176557 (-0.082722) | 0.150466 / 0.737135 (-0.586669) | 0.094381 / 0.296338 (-0.201957) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.394011 / 0.215209 (0.178802) | 3.918318 / 2.077655 (1.840664) | 1.928684 / 1.504120 (0.424564) | 1.765944 / 1.541195 (0.224749) | 1.784716 / 1.468490 (0.316226) | 0.487189 / 4.584777 (-4.097588) | 3.537705 / 3.745712 (-0.208008) | 3.312162 / 5.269862 (-1.957699) | 2.024520 / 4.565676 (-2.541156) | 0.057571 / 0.424275 (-0.366704) | 0.007203 / 0.007607 (-0.000404) | 0.467253 / 0.226044 (0.241208) | 4.659934 / 2.268929 (2.391005) | 2.377764 / 55.444624 (-53.066860) | 2.021984 / 6.876477 (-4.854492) | 2.197468 / 2.142072 (0.055395) | 0.586415 / 4.805227 (-4.218812) | 0.136636 / 6.500664 (-6.364028) | 0.060885 / 0.075469 (-0.014584) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.241879 / 1.841788 (-0.599908) | 18.719327 / 8.074308 (10.645019) | 14.408689 / 10.191392 (4.217297) | 0.155778 / 0.680424 (-0.524646) | 0.018475 / 0.534201 (-0.515726) | 0.392316 / 0.579283 (-0.186967) | 0.409803 / 0.434364 (-0.024561) | 0.458701 / 0.540337 (-0.081637) | 0.630561 / 1.386936 (-0.756375) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006541 / 0.011353 (-0.004812) | 0.003915 / 0.011008 (-0.007094) | 0.064292 / 0.038508 (0.025784) | 0.069174 / 0.023109 (0.046065) | 0.402048 / 0.275898 (0.126150) | 0.423960 / 0.323480 (0.100480) | 0.005355 / 0.007986 (-0.002631) | 0.003295 / 0.004328 (-0.001033) | 0.065212 / 0.004250 (0.060962) | 0.054292 / 0.037052 (0.017240) | 0.402930 / 0.258489 (0.144441) | 0.441840 / 0.293841 (0.147999) | 0.032732 / 0.128546 (-0.095814) | 0.008565 / 0.075646 (-0.067081) | 0.070705 / 0.419271 (-0.348567) | 0.047908 / 0.043533 (0.004375) | 0.401400 / 0.255139 (0.146261) | 0.422682 / 0.283200 (0.139483) | 0.022244 / 0.141683 (-0.119439) | 1.532018 / 1.452155 (0.079864) | 1.597955 / 1.492716 (0.105239) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.226277 / 0.018006 (0.208271) | 0.475578 / 0.000490 (0.475088) | 0.005456 / 0.000200 (0.005256) | 0.000140 / 0.000054 (0.000085) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033111 / 0.037411 (-0.004300) | 0.093138 / 0.014526 (0.078613) | 0.104619 / 0.176557 (-0.071937) | 0.157972 / 0.737135 (-0.579164) | 0.105017 / 0.296338 (-0.191321) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.441771 / 0.215209 (0.226562) | 4.396981 / 2.077655 (2.319326) | 2.410745 / 1.504120 (0.906625) | 2.258359 / 1.541195 (0.717164) | 2.372628 / 1.468490 (0.904138) | 0.491411 / 4.584777 (-4.093366) | 3.650084 / 3.745712 (-0.095628) | 3.279557 / 5.269862 (-1.990304) | 2.011377 / 4.565676 (-2.554300) | 0.058283 / 0.424275 (-0.365992) | 0.007435 / 0.007607 (-0.000172) | 0.507212 / 0.226044 (0.281167) | 5.080104 / 2.268929 (2.811176) | 2.822680 / 55.444624 (-52.621945) | 2.507608 / 6.876477 (-4.368869) | 2.719349 / 2.142072 (0.577277) | 0.586157 / 4.805227 (-4.219071) | 0.132851 / 6.500664 (-6.367813) | 0.059944 / 0.075469 (-0.015525) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.374801 / 1.841788 (-0.466987) | 19.089359 / 8.074308 (11.015051) | 14.525861 / 10.191392 (4.334469) | 0.184758 / 0.680424 (-0.495666) | 0.020206 / 0.534201 (-0.513995) | 0.397309 / 0.579283 (-0.181975) | 0.418120 / 0.434364 (-0.016244) | 0.471817 / 0.540337 (-0.068520) | 0.681691 / 1.386936 (-0.705245) |

![](https://cml.dev/watermark.png#2076cb857e90cf7a6050bba230f586993c5e034a "CML watermark")

_The documentation is not available anymore as the PR was closed or merged._

testing https://github.com/huggingface/doc-builder/pull/426


Dataset.from_pandas with a DataFrame of PIL.Images

New feature or request

A duplicate of https://github.com/huggingface/datasets/issues/4796.

We could get this for free by implementing the `Image` feature as an extension type, as shown in [this](https://colab.research.google.com/drive/1Uzm_tXVpGTwbzleDConWcNjacwO1yxE4?usp=sharing) Colab (example with UUIDs).

+1 to this
Calling this line with a df that contains a PIL image (as they are returned from load_dataset)
`ds = Dataset.from_pandas(df)`
Results in this error:
`ArrowInvalid: ('Could not convert <PIL.PngImagePlugin.PngImageFile image mode=RGB size=1024x1024 at 0x2B41F2D70> with type PngImageFile: did not recognize Python value type when inferring an Arrow data type', 'Conversion failed for column image with type object')`

Currently type inference doesn't know what to do with a Pandas Series of PIL.Image objects, though it would be nice to get a Dataset with the Image type this way


map() not recognizing "text"

There is no "text" column in the `amazon_reviews_multi`, hence the `KeyError`. You can get the column names by running `dataset.column_names`.

### Describe the bug

The [map() documentation](https://huggingface.co/docs/datasets/v2.14.5/en/package_reference/main_classes#datasets.Dataset.map) reads:
`
ds = ds.map(lambda x: tokenizer(x['text'], truncation=True, padding=True), batched=True)`

I have been trying to reproduce it in my code as:

`tokenizedDataset = dataset.map(lambda x: tokenizer(x['text']), batched=True)`

But it doesn't work as it throws the error:

> KeyError: 'text'

Can you please guide me on how to fix it?

### Steps to reproduce the bug

1. `from datasets import load_dataset

dataset = load_dataset("amazon_reviews_multi")`

2. Then this code: `from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")`
3. The line I quoted above (which I have been trying)

### Expected behavior

As mentioned in the documentation, it should run without any error and map the tokenization on the whole dataset.

### Environment info

Python 3.10.2


Create DefunctDatasetError

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009157 / 0.011353 (-0.002195) | 0.004275 / 0.011008 (-0.006734) | 0.099341 / 0.038508 (0.060833) | 0.080634 / 0.023109 (0.057525) | 0.373598 / 0.275898 (0.097700) | 0.445048 / 0.323480 (0.121568) | 0.006541 / 0.007986 (-0.001444) | 0.003550 / 0.004328 (-0.000779) | 0.071034 / 0.004250 (0.066784) | 0.062637 / 0.037052 (0.025585) | 0.379110 / 0.258489 (0.120621) | 0.447896 / 0.293841 (0.154055) | 0.047739 / 0.128546 (-0.080807) | 0.012575 / 0.075646 (-0.063071) | 0.332314 / 0.419271 (-0.086957) | 0.065500 / 0.043533 (0.021967) | 0.365919 / 0.255139 (0.110780) | 0.438611 / 0.283200 (0.155412) | 0.034243 / 0.141683 (-0.107440) | 1.628034 / 1.452155 (0.175880) | 1.802970 / 1.492716 (0.310253) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.224528 / 0.018006 (0.206522) | 0.482094 / 0.000490 (0.481604) | 0.012752 / 0.000200 (0.012552) | 0.000570 / 0.000054 (0.000515) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025456 / 0.037411 (-0.011956) | 0.082281 / 0.014526 (0.067756) | 0.100050 / 0.176557 (-0.076506) | 0.156931 / 0.737135 (-0.580204) | 0.108229 / 0.296338 (-0.188110) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.560688 / 0.215209 (0.345479) | 5.171711 / 2.077655 (3.094056) | 2.273178 / 1.504120 (0.769058) | 1.948158 / 1.541195 (0.406963) | 1.879744 / 1.468490 (0.411254) | 0.789216 / 4.584777 (-3.795561) | 4.529370 / 3.745712 (0.783658) | 4.008743 / 5.269862 (-1.261118) | 2.633555 / 4.565676 (-1.932121) | 0.085411 / 0.424275 (-0.338864) | 0.007256 / 0.007607 (-0.000351) | 0.623254 / 0.226044 (0.397209) | 6.327256 / 2.268929 (4.058327) | 2.911787 / 55.444624 (-52.532837) | 2.240610 / 6.876477 (-4.635867) | 2.352811 / 2.142072 (0.210738) | 0.930114 / 4.805227 (-3.875114) | 0.185028 / 6.500664 (-6.315636) | 0.062115 / 0.075469 (-0.013354) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.394261 / 1.841788 (-0.447527) | 19.689376 / 8.074308 (11.615067) | 17.242289 / 10.191392 (7.050897) | 0.209122 / 0.680424 (-0.471302) | 0.027205 / 0.534201 (-0.506996) | 0.408613 / 0.579283 (-0.170670) | 0.503836 / 0.434364 (0.069472) | 0.485179 / 0.540337 (-0.055158) | 0.674333 / 1.386936 (-0.712603) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007506 / 0.011353 (-0.003847) | 0.004683 / 0.011008 (-0.006325) | 0.067584 / 0.038508 (0.029076) | 0.065635 / 0.023109 (0.042525) | 0.458814 / 0.275898 (0.182916) | 0.477549 / 0.323480 (0.154069) | 0.005212 / 0.007986 (-0.002774) | 0.003393 / 0.004328 (-0.000936) | 0.075307 / 0.004250 (0.071057) | 0.051989 / 0.037052 (0.014937) | 0.484229 / 0.258489 (0.225740) | 0.470889 / 0.293841 (0.177048) | 0.043528 / 0.128546 (-0.085018) | 0.014685 / 0.075646 (-0.060962) | 0.084199 / 0.419271 (-0.335073) | 0.053970 / 0.043533 (0.010437) | 0.432362 / 0.255139 (0.177223) | 0.467472 / 0.283200 (0.184272) | 0.031109 / 0.141683 (-0.110574) | 1.525938 / 1.452155 (0.073784) | 1.631993 / 1.492716 (0.139276) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.200196 / 0.018006 (0.182190) | 0.479316 / 0.000490 (0.478827) | 0.010146 / 0.000200 (0.009947) | 0.000118 / 0.000054 (0.000063) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027911 / 0.037411 (-0.009500) | 0.089720 / 0.014526 (0.075194) | 0.097000 / 0.176557 (-0.079557) | 0.157549 / 0.737135 (-0.579587) | 0.098247 / 0.296338 (-0.198092) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.581401 / 0.215209 (0.366192) | 5.703829 / 2.077655 (3.626174) | 2.688272 / 1.504120 (1.184152) | 2.321691 / 1.541195 (0.780496) | 2.355987 / 1.468490 (0.887497) | 0.759109 / 4.584777 (-3.825668) | 4.711288 / 3.745712 (0.965576) | 4.093019 / 5.269862 (-1.176843) | 2.648240 / 4.565676 (-1.917437) | 0.087839 / 0.424275 (-0.336436) | 0.007060 / 0.007607 (-0.000547) | 0.702783 / 0.226044 (0.476739) | 6.986924 / 2.268929 (4.717996) | 3.365970 / 55.444624 (-52.078654) | 2.670876 / 6.876477 (-4.205600) | 2.776431 / 2.142072 (0.634358) | 0.920005 / 4.805227 (-3.885222) | 0.197521 / 6.500664 (-6.303143) | 0.069974 / 0.075469 (-0.005495) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.596947 / 1.841788 (-0.244841) | 20.606007 / 8.074308 (12.531699) | 18.437425 / 10.191392 (8.246033) | 0.222445 / 0.680424 (-0.457978) | 0.028610 / 0.534201 (-0.505591) | 0.419748 / 0.579283 (-0.159535) | 0.513409 / 0.434364 (0.079045) | 0.487517 / 0.540337 (-0.052820) | 0.706637 / 1.386936 (-0.680299) |

![](https://cml.dev/watermark.png#d664439eb82d62889c21c5236a5869dae75ae779 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007744 / 0.011353 (-0.003609) | 0.004678 / 0.011008 (-0.006330) | 0.101243 / 0.038508 (0.062735) | 0.085653 / 0.023109 (0.062543) | 0.383772 / 0.275898 (0.107874) | 0.422151 / 0.323480 (0.098671) | 0.004566 / 0.007986 (-0.003419) | 0.003900 / 0.004328 (-0.000429) | 0.077778 / 0.004250 (0.073528) | 0.063761 / 0.037052 (0.026709) | 0.385505 / 0.258489 (0.127016) | 0.436186 / 0.293841 (0.142345) | 0.036172 / 0.128546 (-0.092374) | 0.009935 / 0.075646 (-0.065711) | 0.341434 / 0.419271 (-0.077837) | 0.061866 / 0.043533 (0.018333) | 0.385020 / 0.255139 (0.129881) | 0.399455 / 0.283200 (0.116256) | 0.029324 / 0.141683 (-0.112358) | 1.784749 / 1.452155 (0.332594) | 1.845926 / 1.492716 (0.353209) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.266322 / 0.018006 (0.248316) | 0.508708 / 0.000490 (0.508218) | 0.013680 / 0.000200 (0.013480) | 0.000868 / 0.000054 (0.000814) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033887 / 0.037411 (-0.003525) | 0.096709 / 0.014526 (0.082183) | 0.109472 / 0.176557 (-0.067084) | 0.174422 / 0.737135 (-0.562713) | 0.110830 / 0.296338 (-0.185509) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.457533 / 0.215209 (0.242324) | 4.615229 / 2.077655 (2.537575) | 2.418820 / 1.504120 (0.914700) | 2.181079 / 1.541195 (0.639884) | 2.229164 / 1.468490 (0.760674) | 0.554861 / 4.584777 (-4.029916) | 4.323787 / 3.745712 (0.578075) | 3.769396 / 5.269862 (-1.500466) | 2.376850 / 4.565676 (-2.188826) | 0.065030 / 0.424275 (-0.359245) | 0.008397 / 0.007607 (0.000790) | 0.541109 / 0.226044 (0.315065) | 5.477540 / 2.268929 (3.208612) | 2.957049 / 55.444624 (-52.487576) | 2.511732 / 6.876477 (-4.364744) | 2.703953 / 2.142072 (0.561881) | 0.660822 / 4.805227 (-4.144405) | 0.147035 / 6.500664 (-6.353630) | 0.066045 / 0.075469 (-0.009424) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.526481 / 1.841788 (-0.315307) | 22.020256 / 8.074308 (13.945948) | 16.854566 / 10.191392 (6.663174) | 0.192958 / 0.680424 (-0.487466) | 0.021505 / 0.534201 (-0.512696) | 0.462867 / 0.579283 (-0.116416) | 0.514813 / 0.434364 (0.080449) | 0.546147 / 0.540337 (0.005809) | 0.767853 / 1.386936 (-0.619083) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007770 / 0.011353 (-0.003583) | 0.004671 / 0.011008 (-0.006337) | 0.080862 / 0.038508 (0.042354) | 0.087049 / 0.023109 (0.063940) | 0.479497 / 0.275898 (0.203599) | 0.559787 / 0.323480 (0.236307) | 0.007168 / 0.007986 (-0.000818) | 0.003829 / 0.004328 (-0.000500) | 0.079018 / 0.004250 (0.074768) | 0.067359 / 0.037052 (0.030307) | 0.516140 / 0.258489 (0.257651) | 0.547000 / 0.293841 (0.253159) | 0.037955 / 0.128546 (-0.090591) | 0.010007 / 0.075646 (-0.065639) | 0.087673 / 0.419271 (-0.331598) | 0.059309 / 0.043533 (0.015777) | 0.473920 / 0.255139 (0.218781) | 0.529216 / 0.283200 (0.246017) | 0.028236 / 0.141683 (-0.113447) | 1.771127 / 1.452155 (0.318972) | 1.918878 / 1.492716 (0.426162) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.242010 / 0.018006 (0.224004) | 0.494944 / 0.000490 (0.494454) | 0.006319 / 0.000200 (0.006119) | 0.000111 / 0.000054 (0.000056) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.039220 / 0.037411 (0.001809) | 0.113805 / 0.014526 (0.099279) | 0.125704 / 0.176557 (-0.050853) | 0.189198 / 0.737135 (-0.547937) | 0.126334 / 0.296338 (-0.170004) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.502226 / 0.215209 (0.287017) | 5.039133 / 2.077655 (2.961478) | 2.782352 / 1.504120 (1.278232) | 2.587654 / 1.541195 (1.046460) | 2.692588 / 1.468490 (1.224098) | 0.585672 / 4.584777 (-3.999105) | 4.553078 / 3.745712 (0.807366) | 3.864739 / 5.269862 (-1.405123) | 2.536109 / 4.565676 (-2.029567) | 0.069567 / 0.424275 (-0.354708) | 0.008749 / 0.007607 (0.001142) | 0.620645 / 0.226044 (0.394601) | 6.247286 / 2.268929 (3.978357) | 3.345293 / 55.444624 (-52.099332) | 2.873970 / 6.876477 (-4.002507) | 3.123190 / 2.142072 (0.981118) | 0.687391 / 4.805227 (-4.117837) | 0.159046 / 6.500664 (-6.341618) | 0.071019 / 0.075469 (-0.004450) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.728724 / 1.841788 (-0.113064) | 22.828390 / 8.074308 (14.754082) | 17.305225 / 10.191392 (7.113833) | 0.176571 / 0.680424 (-0.503853) | 0.023837 / 0.534201 (-0.510364) | 0.467935 / 0.579283 (-0.111348) | 0.503701 / 0.434364 (0.069337) | 0.558140 / 0.540337 (0.017803) | 0.789326 / 1.386936 (-0.597610) |

![](https://cml.dev/watermark.png#7d357eb4b499cd530c3f4e626f2825a50ee6c8aa "CML watermark")

Create `DefunctDatasetError` as a specific error to be raised when a dataset is defunct and no longer accessible.

See Hub discussion: https://huggingface.co/datasets/the_pile_books3/discussions/7#6523c13a94f3a1a2092d251b


You should be able to load the images by modifying the `load_dataset` call like this:
```python
dataset = load_dataset("imagefolder", data_dir="/content/datasets/PotholeDetectionYOLOv8-1")
```

The `imagefolder` builder expects the image files to be in `path/label/image_file` (e.g. .`.../train/dog/image_1.jpg`), so the solution for the labels in your case is to create metadata files (one for each split; as explained [here](https://huggingface.co/docs/datasets/image_dataset#imagefolder)) that map the images to their labels.

> You should be able to load the images by modifying the `load_dataset` call like this:
>
> ```python
> dataset = load_dataset("imagefolder", data_dir="/content/datasets/PotholeDetectionYOLOv8-1")
> ```
>
> The `imagefolder` builder expects the image files to be in `path/label/image_file` (e.g. .`.../train/dog/image_1.jpg`), so the solution for the labels in your case is to create metadata files (one for each split; as explained [here](https://huggingface.co/docs/datasets/image_dataset#imagefolder)) that map the images to their labels.

I tried like this but only uploads images and not labels, Andyrasika/potholes-dataset

As explained in my previous comment, you need to define metadata files to load the labels or update the paths to be in the format `train/label/image` (`train- image /n -labels` is not supported by the loader).

I downloaded my file after annotating using roboflow . It gives train-
images, labels ,  test- images, labels , valid- images, labels . I hope it
gives you an idea of the dataset . Please advise on this dataset

On Tue, Oct 10, 2023 at 18:12 Mario Šaško ***@***.***> wrote:

> As explained in my previous comment, you need to define metadata files to
> load the labels or update the paths to be in the format train/label/image
> (train- image /n -labels is not supported by the loader).
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/6285#issuecomment-1755335215>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AE4LJNN56FWWTSBYTSTUWHLX6U7CVAVCNFSM6AAAAAA5YHCSTGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJVGMZTKMRRGU>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>

### Describe the bug

my dataset is in form : train- image /n -labels

and tried the code:
```
from datasets import load_dataset

data_files = {
    "train": "/content/datasets/PotholeDetectionYOLOv8-1/train/",
      "validation": "/content/datasets/PotholeDetectionYOLOv8-1/valid/",
      "test": "/content/datasets/PotholeDetectionYOLOv8-1/test/"
}
dataset = load_dataset("imagefolder", data_dir=data_files)
dataset
```
got error:
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-29-2ef1926f73d9>](https://localhost:8080/#) in <cell line: 8>()
      6       "test": "/content/datasets/PotholeDetectionYOLOv8-1/test/"
      7 }
----> 8 dataset = load_dataset("imagefolder", data_dir=data_files)
      9 dataset

6 frames
[/usr/lib/python3.10/pathlib.py](https://localhost:8080/#) in _parse_args(cls, args)
    576                 parts += a._parts
    577             else:
--> 578                 a = os.fspath(a)
    579                 if isinstance(a, str):
    580                     # Force-cast str subclasses to str (issue #21127)

TypeError: expected str, bytes or os.PathLike object, not dict
```

### Steps to reproduce the bug

as share above

### Expected behavior

load images and labels , but my dataset only uploads images
- https://huggingface.co/datasets/Andyrasika/potholes-dataset

### Environment info

colab pro


Add Belebele multiple-choice machine reading comprehension (MRC) dataset

New feature or request

This dataset is already available on the Hub: https://huggingface.co/datasets/facebook/belebele.

### Feature request

Belebele is a multiple-choice machine reading comprehension (MRC) dataset spanning 122 language variants. This dataset enables the evaluation of mono- and multi-lingual models in high-, medium-, and low-resource languages. Each question has four multiple-choice answers and is linked to a short passage from the [FLORES-200](https://github.com/facebookresearch/flores/tree/main/flores200) dataset. The human annotation procedure was carefully curated to create questions that discriminate between different levels of generalizable language comprehension and is reinforced by extensive quality checks. While all questions directly relate to the passage, the English dataset on its own proves difficult enough to challenge state-of-the-art language models. Being fully parallel, this dataset enables direct comparison of model performance across all languages. Belebele opens up new avenues for evaluating and analyzing the multilingual abilities of language models and NLP systems.

Please refer to paper for more details, [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://arxiv.org/abs/2308.16884).

## Composition

- 900 questions per language variant
- 488 distinct passages, there are 1-2 associated questions for each.
- For each question, there is 4 multiple-choice answers, exactly 1 of which is correct.
- 122 language/language variants (including English).
- 900 x 122 = 109,800 total questions.

### Motivation

official repo https://github.com/facebookresearch/belebele

### Your contribution

-