Fix array cast/embed with null values

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006278 / 0.011353 (-0.005075) | 0.003692 / 0.011008 (-0.007316) | 0.080464 / 0.038508 (0.041956) | 0.064751 / 0.023109 (0.041642) | 0.318586 / 0.275898 (0.042688) | 0.351435 / 0.323480 (0.027955) | 0.005044 / 0.007986 (-0.002942) | 0.003034 / 0.004328 (-0.001295) | 0.063710 / 0.004250 (0.059460) | 0.050607 / 0.037052 (0.013555) | 0.318491 / 0.258489 (0.060001) | 0.365688 / 0.293841 (0.071847) | 0.027818 / 0.128546 (-0.100729) | 0.008119 / 0.075646 (-0.067527) | 0.262141 / 0.419271 (-0.157131) | 0.044710 / 0.043533 (0.001177) | 0.318875 / 0.255139 (0.063736) | 0.344559 / 0.283200 (0.061360) | 0.022861 / 0.141683 (-0.118822) | 1.452402 / 1.452155 (0.000247) | 1.502340 / 1.492716 (0.009624) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.219355 / 0.018006 (0.201349) | 0.433311 / 0.000490 (0.432822) | 0.006545 / 0.000200 (0.006345) | 0.000078 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024538 / 0.037411 (-0.012874) | 0.073346 / 0.014526 (0.058821) | 0.083824 / 0.176557 (-0.092733) | 0.145176 / 0.737135 (-0.591959) | 0.085941 / 0.296338 (-0.210397) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.395153 / 0.215209 (0.179944) | 3.944734 / 2.077655 (1.867080) | 1.883910 / 1.504120 (0.379790) | 1.690560 / 1.541195 (0.149365) | 1.775180 / 1.468490 (0.306690) | 0.506873 / 4.584777 (-4.077904) | 3.111095 / 3.745712 (-0.634617) | 2.915358 / 5.269862 (-2.354504) | 1.892886 / 4.565676 (-2.672791) | 0.058690 / 0.424275 (-0.365585) | 0.006550 / 0.007607 (-0.001057) | 0.463372 / 0.226044 (0.237328) | 4.640511 / 2.268929 (2.371583) | 2.321051 / 55.444624 (-53.123573) | 1.986330 / 6.876477 (-4.890147) | 2.160046 / 2.142072 (0.017973) | 0.597833 / 4.805227 (-4.207394) | 0.127946 / 6.500664 (-6.372718) | 0.059709 / 0.075469 (-0.015760) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.278966 / 1.841788 (-0.562822) | 17.863102 / 8.074308 (9.788794) | 13.896057 / 10.191392 (3.704665) | 0.147512 / 0.680424 (-0.532912) | 0.016771 / 0.534201 (-0.517430) | 0.335260 / 0.579283 (-0.244024) | 0.383019 / 0.434364 (-0.051345) | 0.384821 / 0.540337 (-0.155516) | 0.550143 / 1.386936 (-0.836793) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006234 / 0.011353 (-0.005118) | 0.003695 / 0.011008 (-0.007313) | 0.062654 / 0.038508 (0.024146) | 0.059397 / 0.023109 (0.036287) | 0.458375 / 0.275898 (0.182477) | 0.488951 / 0.323480 (0.165471) | 0.004971 / 0.007986 (-0.003014) | 0.002914 / 0.004328 (-0.001415) | 0.061184 / 0.004250 (0.056934) | 0.051246 / 0.037052 (0.014194) | 0.458035 / 0.258489 (0.199546) | 0.490838 / 0.293841 (0.196997) | 0.028746 / 0.128546 (-0.099800) | 0.008167 / 0.075646 (-0.067480) | 0.068006 / 0.419271 (-0.351265) | 0.041809 / 0.043533 (-0.001724) | 0.453896 / 0.255139 (0.198757) | 0.477583 / 0.283200 (0.194383) | 0.020906 / 0.141683 (-0.120777) | 1.443275 / 1.452155 (-0.008879) | 1.493431 / 1.492716 (0.000714) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.219903 / 0.018006 (0.201896) | 0.410275 / 0.000490 (0.409785) | 0.003919 / 0.000200 (0.003719) | 0.000078 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027850 / 0.037411 (-0.009561) | 0.080444 / 0.014526 (0.065918) | 0.089943 / 0.176557 (-0.086614) | 0.145810 / 0.737135 (-0.591326) | 0.090908 / 0.296338 (-0.205430) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.464386 / 0.215209 (0.249177) | 4.633787 / 2.077655 (2.556133) | 2.581658 / 1.504120 (1.077538) | 2.408486 / 1.541195 (0.867291) | 2.460491 / 1.468490 (0.992001) | 0.507512 / 4.584777 (-4.077265) | 3.190363 / 3.745712 (-0.555349) | 2.895581 / 5.269862 (-2.374280) | 1.871506 / 4.565676 (-2.694171) | 0.058469 / 0.424275 (-0.365806) | 0.006526 / 0.007607 (-0.001082) | 0.537641 / 0.226044 (0.311596) | 5.396660 / 2.268929 (3.127731) | 3.027028 / 55.444624 (-52.417596) | 2.703771 / 6.876477 (-4.172705) | 2.865576 / 2.142072 (0.723503) | 0.600103 / 4.805227 (-4.205124) | 0.127109 / 6.500664 (-6.373555) | 0.060985 / 0.075469 (-0.014484) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.365030 / 1.841788 (-0.476758) | 17.988218 / 8.074308 (9.913909) | 14.900796 / 10.191392 (4.709404) | 0.158211 / 0.680424 (-0.522213) | 0.018291 / 0.534201 (-0.515910) | 0.337437 / 0.579283 (-0.241846) | 0.383710 / 0.434364 (-0.050654) | 0.392341 / 0.540337 (-0.147997) | 0.561584 / 1.386936 (-0.825352) |

![](https://cml.dev/watermark.png#b7571ab4b0d9b67b767c55db400b4ffac0f752f1 "CML watermark")

I also plan to address https://github.com/huggingface/datasets/issues/6280#issuecomment-1749310065 in this PR :).

Oh ok, ping me again whenever you want another review :)

Have you had a chance to continue this ? I can also take a look if you want

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6283). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq Feel free to review this again. I've bumped PyArrow to 12.0.0 to simplify the implementation (no need for custom `array_concat` and less `pa.Array.from_buffers`). However, this makes `apache-beam` complain as it only supports `<12.0.0`. The next `apache-beam` release will set this boundary to `<15.0.0.`, so I think the only solution is to wait for it to be published.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005188 / 0.011353 (-0.006165) | 0.003997 / 0.011008 (-0.007011) | 0.062642 / 0.038508 (0.024134) | 0.028913 / 0.023109 (0.005804) | 0.248289 / 0.275898 (-0.027609) | 0.268084 / 0.323480 (-0.055396) | 0.004093 / 0.007986 (-0.003893) | 0.002822 / 0.004328 (-0.001506) | 0.048263 / 0.004250 (0.044012) | 0.041520 / 0.037052 (0.004468) | 0.263277 / 0.258489 (0.004788) | 0.289835 / 0.293841 (-0.004006) | 0.027621 / 0.128546 (-0.100925) | 0.010793 / 0.075646 (-0.064853) | 0.207624 / 0.419271 (-0.211648) | 0.035597 / 0.043533 (-0.007936) | 0.245706 / 0.255139 (-0.009433) | 0.268157 / 0.283200 (-0.015043) | 0.017310 / 0.141683 (-0.124373) | 1.130656 / 1.452155 (-0.321499) | 1.162134 / 1.492716 (-0.330583) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094081 / 0.018006 (0.076075) | 0.302298 / 0.000490 (0.301809) | 0.000220 / 0.000200 (0.000020) | 0.000048 / 0.000054 (-0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019072 / 0.037411 (-0.018339) | 0.061162 / 0.014526 (0.046636) | 0.072820 / 0.176557 (-0.103737) | 0.122628 / 0.737135 (-0.614507) | 0.074962 / 0.296338 (-0.221377) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.277858 / 0.215209 (0.062649) | 2.688478 / 2.077655 (0.610823) | 1.397366 / 1.504120 (-0.106754) | 1.285078 / 1.541195 (-0.256117) | 1.291559 / 1.468490 (-0.176931) | 0.553646 / 4.584777 (-4.031131) | 2.355737 / 3.745712 (-1.389975) | 2.773025 / 5.269862 (-2.496836) | 1.731195 / 4.565676 (-2.834481) | 0.061372 / 0.424275 (-0.362903) | 0.004928 / 0.007607 (-0.002679) | 0.321703 / 0.226044 (0.095659) | 3.212927 / 2.268929 (0.943999) | 1.727104 / 55.444624 (-53.717521) | 1.479430 / 6.876477 (-5.397047) | 1.513436 / 2.142072 (-0.628637) | 0.629913 / 4.805227 (-4.175315) | 0.114607 / 6.500664 (-6.386057) | 0.041707 / 0.075469 (-0.033762) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.976060 / 1.841788 (-0.865727) | 11.575163 / 8.074308 (3.500855) | 9.521390 / 10.191392 (-0.670003) | 0.138725 / 0.680424 (-0.541699) | 0.013752 / 0.534201 (-0.520449) | 0.286252 / 0.579283 (-0.293031) | 0.263420 / 0.434364 (-0.170944) | 0.325531 / 0.540337 (-0.214806) | 0.419466 / 1.386936 (-0.967470) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005615 / 0.011353 (-0.005738) | 0.003884 / 0.011008 (-0.007124) | 0.049563 / 0.038508 (0.011055) | 0.032573 / 0.023109 (0.009464) | 0.276917 / 0.275898 (0.001019) | 0.298403 / 0.323480 (-0.025077) | 0.004367 / 0.007986 (-0.003618) | 0.002794 / 0.004328 (-0.001534) | 0.049105 / 0.004250 (0.044855) | 0.045597 / 0.037052 (0.008545) | 0.289762 / 0.258489 (0.031273) | 0.318440 / 0.293841 (0.024599) | 0.051883 / 0.128546 (-0.076664) | 0.010644 / 0.075646 (-0.065003) | 0.057455 / 0.419271 (-0.361816) | 0.033667 / 0.043533 (-0.009866) | 0.274424 / 0.255139 (0.019285) | 0.295890 / 0.283200 (0.012690) | 0.017029 / 0.141683 (-0.124654) | 1.130123 / 1.452155 (-0.322031) | 1.214827 / 1.492716 (-0.277889) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094882 / 0.018006 (0.076876) | 0.302505 / 0.000490 (0.302015) | 0.000228 / 0.000200 (0.000028) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021695 / 0.037411 (-0.015716) | 0.075196 / 0.014526 (0.060670) | 0.086641 / 0.176557 (-0.089915) | 0.124893 / 0.737135 (-0.612243) | 0.088765 / 0.296338 (-0.207574) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.303388 / 0.215209 (0.088179) | 2.934506 / 2.077655 (0.856852) | 1.608607 / 1.504120 (0.104487) | 1.494632 / 1.541195 (-0.046563) | 1.512801 / 1.468490 (0.044310) | 0.558563 / 4.584777 (-4.026214) | 2.383212 / 3.745712 (-1.362500) | 2.634629 / 5.269862 (-2.635233) | 1.729319 / 4.565676 (-2.836357) | 0.062345 / 0.424275 (-0.361930) | 0.004981 / 0.007607 (-0.002626) | 0.358333 / 0.226044 (0.132289) | 3.484229 / 2.268929 (1.215301) | 2.010043 / 55.444624 (-53.434581) | 1.693733 / 6.876477 (-5.182744) | 1.824150 / 2.142072 (-0.317922) | 0.650835 / 4.805227 (-4.154392) | 0.115933 / 6.500664 (-6.384732) | 0.041270 / 0.075469 (-0.034199) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.007949 / 1.841788 (-0.833838) | 12.000085 / 8.074308 (3.925776) | 10.453119 / 10.191392 (0.261727) | 0.143583 / 0.680424 (-0.536840) | 0.015937 / 0.534201 (-0.518264) | 0.286653 / 0.579283 (-0.292631) | 0.272359 / 0.434364 (-0.162005) | 0.330520 / 0.540337 (-0.209818) | 0.417015 / 1.386936 (-0.969921) |

![](https://cml.dev/watermark.png#ac05bac20fcc8e0e22a852707162e15a7e2ae357 "CML watermark")

Fixes issues with casting/embedding PyArrow list arrays with null values. It also bumps the required PyArrow version to 12.0.0 (over 9 months old) to simplify the implementation.

Fix #6280, fix #6311, fix #6360

(Also fixes https://github.com/huggingface/datasets/issues/5430 to make Beam compatible with PyArrow>=12.0.0)


Drop data_files duplicates

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006934 / 0.011353 (-0.004419) | 0.004097 / 0.011008 (-0.006911) | 0.084662 / 0.038508 (0.046154) | 0.077106 / 0.023109 (0.053996) | 0.355035 / 0.275898 (0.079137) | 0.381466 / 0.323480 (0.057986) | 0.004182 / 0.007986 (-0.003803) | 0.003411 / 0.004328 (-0.000917) | 0.065279 / 0.004250 (0.061029) | 0.058192 / 0.037052 (0.021140) | 0.372363 / 0.258489 (0.113874) | 0.401621 / 0.293841 (0.107780) | 0.031719 / 0.128546 (-0.096827) | 0.008753 / 0.075646 (-0.066893) | 0.287125 / 0.419271 (-0.132146) | 0.052943 / 0.043533 (0.009410) | 0.349680 / 0.255139 (0.094541) | 0.364004 / 0.283200 (0.080805) | 0.026705 / 0.141683 (-0.114977) | 1.472708 / 1.452155 (0.020553) | 1.556559 / 1.492716 (0.063842) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.224868 / 0.018006 (0.206862) | 0.458793 / 0.000490 (0.458304) | 0.009434 / 0.000200 (0.009234) | 0.000356 / 0.000054 (0.000301) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029670 / 0.037411 (-0.007741) | 0.086517 / 0.014526 (0.071991) | 0.097342 / 0.176557 (-0.079215) | 0.153722 / 0.737135 (-0.583413) | 0.098465 / 0.296338 (-0.197874) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.400739 / 0.215209 (0.185530) | 3.998087 / 2.077655 (1.920432) | 2.025772 / 1.504120 (0.521652) | 1.858679 / 1.541195 (0.317485) | 1.951573 / 1.468490 (0.483083) | 0.483028 / 4.584777 (-4.101749) | 3.554085 / 3.745712 (-0.191627) | 3.306983 / 5.269862 (-1.962879) | 2.087043 / 4.565676 (-2.478633) | 0.057127 / 0.424275 (-0.367148) | 0.007252 / 0.007607 (-0.000355) | 0.480180 / 0.226044 (0.254136) | 4.787183 / 2.268929 (2.518255) | 2.489667 / 55.444624 (-52.954957) | 2.150774 / 6.876477 (-4.725703) | 2.403197 / 2.142072 (0.261124) | 0.581843 / 4.805227 (-4.223384) | 0.134915 / 6.500664 (-6.365749) | 0.061283 / 0.075469 (-0.014186) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.285700 / 1.841788 (-0.556088) | 19.474093 / 8.074308 (11.399785) | 14.336349 / 10.191392 (4.144957) | 0.170932 / 0.680424 (-0.509492) | 0.018348 / 0.534201 (-0.515853) | 0.391909 / 0.579283 (-0.187374) | 0.414706 / 0.434364 (-0.019658) | 0.458156 / 0.540337 (-0.082182) | 0.656303 / 1.386936 (-0.730633) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006738 / 0.011353 (-0.004615) | 0.004029 / 0.011008 (-0.006979) | 0.064411 / 0.038508 (0.025903) | 0.078225 / 0.023109 (0.055116) | 0.408468 / 0.275898 (0.132569) | 0.445585 / 0.323480 (0.122105) | 0.005490 / 0.007986 (-0.002495) | 0.003419 / 0.004328 (-0.000910) | 0.063966 / 0.004250 (0.059715) | 0.056779 / 0.037052 (0.019727) | 0.415258 / 0.258489 (0.156769) | 0.461258 / 0.293841 (0.167418) | 0.032051 / 0.128546 (-0.096495) | 0.008471 / 0.075646 (-0.067176) | 0.071004 / 0.419271 (-0.348267) | 0.049068 / 0.043533 (0.005536) | 0.409575 / 0.255139 (0.154436) | 0.430748 / 0.283200 (0.147548) | 0.023784 / 0.141683 (-0.117899) | 1.507894 / 1.452155 (0.055739) | 1.586575 / 1.492716 (0.093859) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.228574 / 0.018006 (0.210568) | 0.451389 / 0.000490 (0.450900) | 0.006312 / 0.000200 (0.006112) | 0.000100 / 0.000054 (0.000045) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033391 / 0.037411 (-0.004020) | 0.096816 / 0.014526 (0.082290) | 0.107269 / 0.176557 (-0.069288) | 0.159749 / 0.737135 (-0.577387) | 0.108240 / 0.296338 (-0.188098) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.437643 / 0.215209 (0.222434) | 4.378173 / 2.077655 (2.300518) | 2.367218 / 1.504120 (0.863098) | 2.229493 / 1.541195 (0.688298) | 2.329849 / 1.468490 (0.861359) | 0.494985 / 4.584777 (-4.089792) | 3.578540 / 3.745712 (-0.167172) | 3.338220 / 5.269862 (-1.931642) | 2.092482 / 4.565676 (-2.473194) | 0.058495 / 0.424275 (-0.365780) | 0.007396 / 0.007607 (-0.000211) | 0.511001 / 0.226044 (0.284957) | 5.113497 / 2.268929 (2.844568) | 2.806215 / 55.444624 (-52.638409) | 2.485428 / 6.876477 (-4.391048) | 2.764907 / 2.142072 (0.622835) | 0.598824 / 4.805227 (-4.206404) | 0.134988 / 6.500664 (-6.365676) | 0.061752 / 0.075469 (-0.013717) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.365583 / 1.841788 (-0.476205) | 20.270297 / 8.074308 (12.195989) | 15.331673 / 10.191392 (5.140281) | 0.166152 / 0.680424 (-0.514272) | 0.020678 / 0.534201 (-0.513523) | 0.394821 / 0.579283 (-0.184462) | 0.420493 / 0.434364 (-0.013871) | 0.468551 / 0.540337 (-0.071787) | 0.654903 / 1.386936 (-0.732033) |

![](https://cml.dev/watermark.png#5f268dd4ad4fb6dada15937d57fb367cb2810162 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007803 / 0.011353 (-0.003550) | 0.004664 / 0.011008 (-0.006344) | 0.099908 / 0.038508 (0.061400) | 0.090674 / 0.023109 (0.067565) | 0.406009 / 0.275898 (0.130111) | 0.465098 / 0.323480 (0.141618) | 0.004667 / 0.007986 (-0.003319) | 0.003880 / 0.004328 (-0.000449) | 0.076552 / 0.004250 (0.072301) | 0.066345 / 0.037052 (0.029292) | 0.419195 / 0.258489 (0.160706) | 0.478581 / 0.293841 (0.184741) | 0.036967 / 0.128546 (-0.091579) | 0.010000 / 0.075646 (-0.065647) | 0.347126 / 0.419271 (-0.072145) | 0.062265 / 0.043533 (0.018733) | 0.406653 / 0.255139 (0.151514) | 0.439044 / 0.283200 (0.155845) | 0.031289 / 0.141683 (-0.110394) | 1.797674 / 1.452155 (0.345520) | 1.835183 / 1.492716 (0.342467) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.268194 / 0.018006 (0.250187) | 0.493614 / 0.000490 (0.493124) | 0.015636 / 0.000200 (0.015436) | 0.000417 / 0.000054 (0.000362) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034188 / 0.037411 (-0.003223) | 0.099127 / 0.014526 (0.084601) | 0.113949 / 0.176557 (-0.062607) | 0.181209 / 0.737135 (-0.555926) | 0.114943 / 0.296338 (-0.181395) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.455767 / 0.215209 (0.240558) | 4.542947 / 2.077655 (2.465293) | 2.214605 / 1.504120 (0.710485) | 2.015163 / 1.541195 (0.473969) | 2.084945 / 1.468490 (0.616455) | 0.583827 / 4.584777 (-4.000950) | 4.187009 / 3.745712 (0.441297) | 3.920841 / 5.269862 (-1.349020) | 2.447260 / 4.565676 (-2.118417) | 0.069139 / 0.424275 (-0.355137) | 0.008734 / 0.007607 (0.001127) | 0.544673 / 0.226044 (0.318629) | 5.445094 / 2.268929 (3.176165) | 2.788284 / 55.444624 (-52.656340) | 2.395863 / 6.876477 (-4.480614) | 2.622632 / 2.142072 (0.480560) | 0.703931 / 4.805227 (-4.101297) | 0.160502 / 6.500664 (-6.340162) | 0.073734 / 0.075469 (-0.001735) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.498992 / 1.841788 (-0.342795) | 22.761476 / 8.074308 (14.687168) | 17.123919 / 10.191392 (6.932527) | 0.170272 / 0.680424 (-0.510151) | 0.021307 / 0.534201 (-0.512894) | 0.467548 / 0.579283 (-0.111735) | 0.480777 / 0.434364 (0.046413) | 0.542168 / 0.540337 (0.001830) | 0.771092 / 1.386936 (-0.615844) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007923 / 0.011353 (-0.003430) | 0.004664 / 0.011008 (-0.006344) | 0.077795 / 0.038508 (0.039286) | 0.090293 / 0.023109 (0.067184) | 0.494682 / 0.275898 (0.218784) | 0.539973 / 0.323480 (0.216494) | 0.006302 / 0.007986 (-0.001684) | 0.003794 / 0.004328 (-0.000535) | 0.076567 / 0.004250 (0.072317) | 0.067141 / 0.037052 (0.030089) | 0.501279 / 0.258489 (0.242790) | 0.555670 / 0.293841 (0.261829) | 0.037773 / 0.128546 (-0.090773) | 0.009930 / 0.075646 (-0.065716) | 0.084839 / 0.419271 (-0.334433) | 0.056876 / 0.043533 (0.013344) | 0.499329 / 0.255139 (0.244190) | 0.518449 / 0.283200 (0.235249) | 0.026041 / 0.141683 (-0.115642) | 1.787259 / 1.452155 (0.335105) | 1.853505 / 1.492716 (0.360788) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.238413 / 0.018006 (0.220407) | 0.488889 / 0.000490 (0.488399) | 0.007476 / 0.000200 (0.007277) | 0.000141 / 0.000054 (0.000087) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.038701 / 0.037411 (0.001290) | 0.115391 / 0.014526 (0.100865) | 0.125553 / 0.176557 (-0.051004) | 0.190267 / 0.737135 (-0.546868) | 0.126401 / 0.296338 (-0.169937) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.509270 / 0.215209 (0.294061) | 5.087631 / 2.077655 (3.009976) | 2.745863 / 1.504120 (1.241743) | 2.560259 / 1.541195 (1.019064) | 2.653124 / 1.468490 (1.184634) | 0.582118 / 4.584777 (-4.002659) | 4.181144 / 3.745712 (0.435431) | 3.871179 / 5.269862 (-1.398683) | 2.459849 / 4.565676 (-2.105827) | 0.068844 / 0.424275 (-0.355431) | 0.008672 / 0.007607 (0.001065) | 0.604898 / 0.226044 (0.378854) | 6.073263 / 2.268929 (3.804334) | 3.366638 / 55.444624 (-52.077986) | 2.937261 / 6.876477 (-3.939215) | 3.181173 / 2.142072 (1.039100) | 0.700478 / 4.805227 (-4.104750) | 0.158361 / 6.500664 (-6.342303) | 0.072860 / 0.075469 (-0.002609) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.621363 / 1.841788 (-0.220425) | 23.614315 / 8.074308 (15.540007) | 17.607213 / 10.191392 (7.415821) | 0.198031 / 0.680424 (-0.482393) | 0.023859 / 0.534201 (-0.510342) | 0.474674 / 0.579283 (-0.104609) | 0.491173 / 0.434364 (0.056809) | 0.581995 / 0.540337 (0.041658) | 0.792168 / 1.386936 (-0.594768) |

![](https://cml.dev/watermark.png#56fa9645fd24e083adee3cfd0f7d972fce391f0e "CML watermark")

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6282). All of your documentation changes will be reflected on that endpoint.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004779 / 0.011353 (-0.006574) | 0.002916 / 0.011008 (-0.008092) | 0.061962 / 0.038508 (0.023454) | 0.029537 / 0.023109 (0.006428) | 0.242574 / 0.275898 (-0.033324) | 0.268585 / 0.323480 (-0.054894) | 0.004006 / 0.007986 (-0.003979) | 0.002434 / 0.004328 (-0.001895) | 0.048289 / 0.004250 (0.044039) | 0.045534 / 0.037052 (0.008481) | 0.248251 / 0.258489 (-0.010239) | 0.277037 / 0.293841 (-0.016804) | 0.023728 / 0.128546 (-0.104818) | 0.007295 / 0.075646 (-0.068351) | 0.205813 / 0.419271 (-0.213459) | 0.059093 / 0.043533 (0.015560) | 0.244336 / 0.255139 (-0.010803) | 0.262865 / 0.283200 (-0.020335) | 0.017232 / 0.141683 (-0.124451) | 1.126729 / 1.452155 (-0.325426) | 1.198987 / 1.492716 (-0.293729) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091246 / 0.018006 (0.073240) | 0.300747 / 0.000490 (0.300258) | 0.000202 / 0.000200 (0.000003) | 0.000041 / 0.000054 (-0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018681 / 0.037411 (-0.018731) | 0.063567 / 0.014526 (0.049041) | 0.074019 / 0.176557 (-0.102538) | 0.120856 / 0.737135 (-0.616279) | 0.076525 / 0.296338 (-0.219814) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282230 / 0.215209 (0.067021) | 2.731502 / 2.077655 (0.653847) | 1.473901 / 1.504120 (-0.030219) | 1.351165 / 1.541195 (-0.190030) | 1.390582 / 1.468490 (-0.077908) | 0.398443 / 4.584777 (-4.186334) | 2.360497 / 3.745712 (-1.385215) | 2.548158 / 5.269862 (-2.721703) | 1.552416 / 4.565676 (-3.013260) | 0.045659 / 0.424275 (-0.378616) | 0.004778 / 0.007607 (-0.002829) | 0.330191 / 0.226044 (0.104146) | 3.262510 / 2.268929 (0.993582) | 1.823076 / 55.444624 (-53.621549) | 1.541206 / 6.876477 (-5.335271) | 1.589069 / 2.142072 (-0.553004) | 0.472265 / 4.805227 (-4.332963) | 0.099712 / 6.500664 (-6.400952) | 0.042803 / 0.075469 (-0.032666) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.963022 / 1.841788 (-0.878766) | 11.998807 / 8.074308 (3.924499) | 10.526006 / 10.191392 (0.334614) | 0.140965 / 0.680424 (-0.539459) | 0.014197 / 0.534201 (-0.520004) | 0.271668 / 0.579283 (-0.307615) | 0.263993 / 0.434364 (-0.170371) | 0.307213 / 0.540337 (-0.233124) | 0.427411 / 1.386936 (-0.959525) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004761 / 0.011353 (-0.006592) | 0.002652 / 0.011008 (-0.008357) | 0.047949 / 0.038508 (0.009441) | 0.049714 / 0.023109 (0.026604) | 0.274021 / 0.275898 (-0.001877) | 0.292413 / 0.323480 (-0.031067) | 0.003912 / 0.007986 (-0.004074) | 0.002290 / 0.004328 (-0.002038) | 0.047320 / 0.004250 (0.043069) | 0.038061 / 0.037052 (0.001009) | 0.279318 / 0.258489 (0.020829) | 0.305167 / 0.293841 (0.011326) | 0.024595 / 0.128546 (-0.103952) | 0.006976 / 0.075646 (-0.068671) | 0.052987 / 0.419271 (-0.366285) | 0.032454 / 0.043533 (-0.011079) | 0.273986 / 0.255139 (0.018847) | 0.297641 / 0.283200 (0.014442) | 0.017680 / 0.141683 (-0.124003) | 1.141218 / 1.452155 (-0.310937) | 1.222543 / 1.492716 (-0.270173) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092880 / 0.018006 (0.074873) | 0.305080 / 0.000490 (0.304590) | 0.000215 / 0.000200 (0.000016) | 0.000051 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021050 / 0.037411 (-0.016362) | 0.069676 / 0.014526 (0.055150) | 0.081082 / 0.176557 (-0.095475) | 0.119234 / 0.737135 (-0.617902) | 0.081242 / 0.296338 (-0.215096) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295916 / 0.215209 (0.080707) | 2.909769 / 2.077655 (0.832115) | 1.623118 / 1.504120 (0.118998) | 1.502297 / 1.541195 (-0.038898) | 1.540290 / 1.468490 (0.071800) | 0.401176 / 4.584777 (-4.183601) | 2.427764 / 3.745712 (-1.317948) | 2.568610 / 5.269862 (-2.701252) | 1.550486 / 4.565676 (-3.015190) | 0.046895 / 0.424275 (-0.377380) | 0.004800 / 0.007607 (-0.002807) | 0.344524 / 0.226044 (0.118479) | 3.429189 / 2.268929 (1.160261) | 1.949738 / 55.444624 (-53.494887) | 1.681440 / 6.876477 (-5.195037) | 1.675304 / 2.142072 (-0.466769) | 0.469663 / 4.805227 (-4.335564) | 0.097470 / 6.500664 (-6.403194) | 0.040121 / 0.075469 (-0.035348) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.957947 / 1.841788 (-0.883841) | 11.968455 / 8.074308 (3.894147) | 10.809763 / 10.191392 (0.618371) | 0.140603 / 0.680424 (-0.539820) | 0.015562 / 0.534201 (-0.518638) | 0.276406 / 0.579283 (-0.302877) | 0.295267 / 0.434364 (-0.139097) | 0.315744 / 0.540337 (-0.224593) | 0.417985 / 1.386936 (-0.968951) |

![](https://cml.dev/watermark.png#12e01642a6978cbb9d5778c8b7f1c6b20a9887d5 "CML watermark")

I've opened #6704 with a cleaner fix for the issue :)

I just added drop_duplicates=True to `.from_patterns`. I used a dict to deduplicate and preserve the order

close https://github.com/huggingface/datasets/issues/6259
close https://github.com/huggingface/datasets/issues/6272


Improve documentation of dataset.from_generator

I have looked at the doc failures, and I do not think that my change caused the doc build failure, but I'm not 100% sure about that.
I have high confidence that the integration test failures are not something I introduced:-)

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008557 / 0.011353 (-0.002796) | 0.005224 / 0.011008 (-0.005784) | 0.109402 / 0.038508 (0.070893) | 0.075008 / 0.023109 (0.051899) | 0.388910 / 0.275898 (0.113012) | 0.425481 / 0.323480 (0.102002) | 0.005046 / 0.007986 (-0.002939) | 0.004166 / 0.004328 (-0.000162) | 0.079890 / 0.004250 (0.075639) | 0.061992 / 0.037052 (0.024940) | 0.409933 / 0.258489 (0.151444) | 0.444096 / 0.293841 (0.150255) | 0.043958 / 0.128546 (-0.084588) | 0.013655 / 0.075646 (-0.061991) | 0.402620 / 0.419271 (-0.016651) | 0.062784 / 0.043533 (0.019251) | 0.399653 / 0.255139 (0.144514) | 0.432926 / 0.283200 (0.149727) | 0.034631 / 0.141683 (-0.107052) | 1.801450 / 1.452155 (0.349296) | 1.965007 / 1.492716 (0.472290) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.305744 / 0.018006 (0.287738) | 0.590825 / 0.000490 (0.590335) | 0.014561 / 0.000200 (0.014361) | 0.000430 / 0.000054 (0.000375) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030449 / 0.037411 (-0.006962) | 0.091753 / 0.014526 (0.077227) | 0.106259 / 0.176557 (-0.070298) | 0.174599 / 0.737135 (-0.562537) | 0.107069 / 0.296338 (-0.189269) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.607544 / 0.215209 (0.392335) | 6.182592 / 2.077655 (4.104937) | 2.699782 / 1.504120 (1.195663) | 2.386915 / 1.541195 (0.845720) | 2.441763 / 1.468490 (0.973273) | 0.811360 / 4.584777 (-3.773417) | 5.253799 / 3.745712 (1.508087) | 4.762054 / 5.269862 (-0.507807) | 3.045161 / 4.565676 (-1.520515) | 0.095983 / 0.424275 (-0.328292) | 0.008653 / 0.007607 (0.001046) | 0.714218 / 0.226044 (0.488174) | 7.279325 / 2.268929 (5.010397) | 3.356107 / 55.444624 (-52.088517) | 2.765867 / 6.876477 (-4.110610) | 2.997756 / 2.142072 (0.855684) | 1.008740 / 4.805227 (-3.796487) | 0.201462 / 6.500664 (-6.299202) | 0.075780 / 0.075469 (0.000311) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.677034 / 1.841788 (-0.164754) | 23.546919 / 8.074308 (15.472610) | 21.576985 / 10.191392 (11.385593) | 0.239253 / 0.680424 (-0.441171) | 0.028740 / 0.534201 (-0.505460) | 0.468519 / 0.579283 (-0.110765) | 0.593935 / 0.434364 (0.159571) | 0.536830 / 0.540337 (-0.003507) | 0.779925 / 1.386936 (-0.607011) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009582 / 0.011353 (-0.001771) | 0.004971 / 0.011008 (-0.006037) | 0.081304 / 0.038508 (0.042796) | 0.077588 / 0.023109 (0.054478) | 0.486610 / 0.275898 (0.210712) | 0.580228 / 0.323480 (0.256748) | 0.006707 / 0.007986 (-0.001279) | 0.004325 / 0.004328 (-0.000004) | 0.086170 / 0.004250 (0.081920) | 0.060591 / 0.037052 (0.023539) | 0.501723 / 0.258489 (0.243234) | 0.548633 / 0.293841 (0.254793) | 0.050306 / 0.128546 (-0.078240) | 0.017458 / 0.075646 (-0.058188) | 0.093295 / 0.419271 (-0.325977) | 0.064588 / 0.043533 (0.021056) | 0.519395 / 0.255139 (0.264256) | 0.526021 / 0.283200 (0.242821) | 0.035795 / 0.141683 (-0.105888) | 1.792927 / 1.452155 (0.340772) | 1.956499 / 1.492716 (0.463783) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.296249 / 0.018006 (0.278243) | 0.594482 / 0.000490 (0.593992) | 0.007318 / 0.000200 (0.007118) | 0.000182 / 0.000054 (0.000128) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036110 / 0.037411 (-0.001301) | 0.107924 / 0.014526 (0.093399) | 0.119975 / 0.176557 (-0.056582) | 0.177499 / 0.737135 (-0.559636) | 0.123299 / 0.296338 (-0.173039) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.632994 / 0.215209 (0.417785) | 6.481663 / 2.077655 (4.404008) | 3.231259 / 1.504120 (1.727139) | 2.768298 / 1.541195 (1.227103) | 2.694543 / 1.468490 (1.226053) | 0.837384 / 4.584777 (-3.747393) | 5.405278 / 3.745712 (1.659566) | 4.639424 / 5.269862 (-0.630437) | 2.944251 / 4.565676 (-1.621426) | 0.094978 / 0.424275 (-0.329297) | 0.008716 / 0.007607 (0.001108) | 0.795820 / 0.226044 (0.569776) | 8.514233 / 2.268929 (6.245304) | 3.800463 / 55.444624 (-51.644161) | 3.000005 / 6.876477 (-3.876472) | 3.298853 / 2.142072 (1.156781) | 0.994112 / 4.805227 (-3.811115) | 0.209435 / 6.500664 (-6.291229) | 0.075610 / 0.075469 (0.000141) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.681127 / 1.841788 (-0.160661) | 23.874465 / 8.074308 (15.800156) | 21.638567 / 10.191392 (11.447175) | 0.233303 / 0.680424 (-0.447121) | 0.032504 / 0.534201 (-0.501697) | 0.460462 / 0.579283 (-0.118821) | 0.560043 / 0.434364 (0.125679) | 0.555059 / 0.540337 (0.014721) | 0.831444 / 1.386936 (-0.555492) |

![](https://cml.dev/watermark.png#faada1742e1f25fce9cc5691ec11d3f91d4aa120 "CML watermark")

Improve documentation to clarify sharding behavior (#6270)


Couldn't cast array of type fixed_size_list to Sequence(Value(float64))

Thanks for reporting! I've opened a PR with a fix.

Thanks for the quick response @mariosasko!  I just installed your branch via `poetry add 'git+https://github.com/huggingface/datasets#fix-array_values'` and I can confirm it works on the example provided.

Follow up question for you, should `None`s be supported in these types of features as they are in others?

For example, the following script:

```
from datasets import Features, Value, Sequence, ClassLabel, Dataset

dataset_features = Features({
    'text': Value('string'),
    'embedding': Sequence(Value('double'), length=2),
    'categories': Sequence(ClassLabel(names=sorted([
        'one',
        'two',
        'three'
    ]))),
})

dataset = Dataset.from_dict(
    {
        'text': ['A'] * 10000,
        "embedding": [None] * 10000, # THIS LINE CHANGED
        'categories': [[0]] * 10000,
    },
    features=dataset_features
)

def test_mapper(r):
    r['text'] = list(map(lambda t: t + ' b', r['text']))
    return r

dataset = dataset.map(test_mapper, batched=True, batch_size=10, features=dataset_features, num_proc=2)
```

fails with

```
Traceback (most recent call last):
  File "/home/jmif/.virtualenvs/llm-training/lib/python3.10/site-packages/multiprocess/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/jmif/.virtualenvs/llm-training/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 1354, in _write_generator_to_queue
    for i, result in enumerate(func(**kwargs)):
  File "/home/jmif/.virtualenvs/llm-training/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3493, in _map_single
    writer.write_batch(batch)
  File "/home/jmif/.virtualenvs/llm-training/lib/python3.10/site-packages/datasets/arrow_writer.py", line 549, in write_batch
    array = cast_array_to_feature(col_values, col_type) if col_type is not None else col_values
  File "/home/jmif/.virtualenvs/llm-training/lib/python3.10/site-packages/datasets/table.py", line 1831, in wrapper
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/home/jmif/.virtualenvs/llm-training/lib/python3.10/site-packages/datasets/table.py", line 1831, in <listcomp>
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/home/jmif/.virtualenvs/llm-training/lib/python3.10/site-packages/datasets/table.py", line 2160, in cast_array_to_feature
    raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{feature}")
TypeError: Couldn't cast array of type
fixed_size_list<item: double>[2]
to
Sequence(feature=Value(dtype='float64', id=None), length=2, id=None)
```

Ideally we can have empty embedding columns as well!

This part of PyArrow is buggy and inconsistent regarding features implemented across the types, so the only option is to operate on the Arrow buffer level to fix issues such as the above one.

Ok - can you take the POC I did [here](https://github.com/huggingface/datasets/commit/15443098e9ce053943172f7ec6fce3769d7dff6e)?  Happy to turn this into an actual PR but would appreciate feedback on the implementation before I take another pass!

### Describe the bug

I have a dataset with an embedding column, when I try to map that dataset I get the following exception:

```
Traceback (most recent call last):
  File "/Users/jmif/.virtualenvs/llm-training/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3189, in map
    for rank, done, content in iflatmap_unordered(
  File "/Users/jmif/.virtualenvs/llm-training/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 1387, in iflatmap_unordered
    [async_result.get(timeout=0.05) for async_result in async_results]
  File "/Users/jmif/.virtualenvs/llm-training/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 1387, in <listcomp>
    [async_result.get(timeout=0.05) for async_result in async_results]
  File "/Users/jmif/.virtualenvs/llm-training/lib/python3.10/site-packages/multiprocess/pool.py", line 774, in get
    raise self._value
TypeError: Couldn't cast array of type
fixed_size_list<item: float>[2]
to
Sequence(feature=Value(dtype='float32', id=None), length=2, id=None)
```

### Steps to reproduce the bug

Here's a simple repro script:

```
from datasets import Features, Value, Sequence, ClassLabel, Dataset

dataset_features = Features({
    'text': Value('string'),
    'embedding': Sequence(Value('double'), length=2),
    'categories': Sequence(ClassLabel(names=sorted([
        'one',
        'two',
        'three'
    ]))),
})

dataset = Dataset.from_dict(
    {
        'text': ['A'] * 10000,
        'embedding': [[0.0, 0.1]] * 10000,
        'categories': [[0]] * 10000,
    },
    features=dataset_features
)

def test_mapper(r):
    r['text'] = list(map(lambda t: t + ' b', r['text']))
    return r

dataset = dataset.map(test_mapper, batched=True, batch_size=10, features=dataset_features, num_proc=2)
```

Removing the embedding column fixes the issue!

### Expected behavior

The mapping completes successfully.

### Environment info

- `datasets` version: 2.14.4
- Platform: macOS-14.0-arm64-arm-64bit
- Python version: 3.10.12
- Huggingface_hub version: 0.17.1
- PyArrow version: 13.0.0
- Pandas version: 2.0.3


Batched IterableDataset

enhancement

New feature or request

This is exactly what I was looking for. It would also be very useful for me :-)

This issue is really smashing the selling point of HF datasets... The only workaround I've found so far is to create a customized IterableDataloader which improves the loading speed to some extent.

For example I've a HF dataset `dt_train` with `len(dt_train) == 1M`. Using plain DataLoader is extremely slow:
```
%%time
dl_train = DataLoader(dt_train, batch_size=128, shuffle = True)
for batch in dl_train:
    pass
```

```
CPU times: user 24min 35s, sys: 704 ms, total: 24min 36s
Wall time: 24min 37s
```
And DataLoader works even worse with HF's iterable_dataset:
```
%%time
dt_train_ = dt_train.with_format(None).to_iterable_dataset(num_shards=64).shuffle(buffer_size=10_000)
dl_train = DataLoader(dt_train_, batch_size=128)
for batch in dl_train:
    pass
```
```
CPU times: user 1h 6min 2s, sys: 4.28 s, total: 1h 6min 6s
Wall time: 1h 7min 53s
```
Workaround by running a customized wrapper:
```
%%time
from torch.utils.data import DataLoader, IterableDataset

class Dataset2Iterable(IterableDataset):
    """
    Wrapper to use a HF dataset as pytorch IterableDataset to speed up data loading.
    """
    def __init__(self, dataset, batch_size=1, shuffle=True):
        super(Dataset2Iterable).__init__()
        self.dataset = dataset
        self.batch_size = batch_size
        self.shuffle = shuffle

    def __iter__(self):
        if self.shuffle: self.dataset.shuffle()
        return self.dataset.iter(batch_size=self.batch_size)

dl_train = DataLoader(Dataset2Iterable(dt_train, batch_size = 128), batch_size=1, num_workers=0)
for n in range(2):
    for batch in dl_train:
        pass
```
The speed still is slower than using tensorflow's loader but improved a lot than previous code:
```
CPU times: user 4min 18s, sys: 0 ns, total: 4min 18s
Wall time: 4min 20s
```
Note that the way I implemented `Dataset2Iterable` will only work with `num_workers=0`.

### Feature request

Hi,

could you add an implementation of a batched `IterableDataset`. It already support an option to do batch iteration via `.iter(batch_size=...)` but this cannot be used in combination with a torch `DataLoader` since it just returns an iterator.

### Motivation

The current implementation loads each element of a batch individually which can be very slow in cases of a big batch_size. I did some experiments [here](https://discuss.huggingface.co/t/slow-dataloader-with-big-batch-size/57224) and using a batched iteration would speed up data loading significantly.

### Your contribution

N/A


No data files duplicates

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009624 / 0.011353 (-0.001729) | 0.005121 / 0.011008 (-0.005887) | 0.105560 / 0.038508 (0.067052) | 0.090749 / 0.023109 (0.067640) | 0.430274 / 0.275898 (0.154376) | 0.443399 / 0.323480 (0.119919) | 0.006575 / 0.007986 (-0.001411) | 0.004396 / 0.004328 (0.000068) | 0.080900 / 0.004250 (0.076649) | 0.064921 / 0.037052 (0.027868) | 0.410092 / 0.258489 (0.151603) | 0.470058 / 0.293841 (0.176217) | 0.054160 / 0.128546 (-0.074386) | 0.014367 / 0.075646 (-0.061279) | 0.384844 / 0.419271 (-0.034428) | 0.072818 / 0.043533 (0.029285) | 0.429341 / 0.255139 (0.174202) | 0.430968 / 0.283200 (0.147769) | 0.038437 / 0.141683 (-0.103246) | 1.814456 / 1.452155 (0.362301) | 1.832122 / 1.492716 (0.339406) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.329266 / 0.018006 (0.311260) | 0.596848 / 0.000490 (0.596358) | 0.018291 / 0.000200 (0.018091) | 0.000113 / 0.000054 (0.000058) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030505 / 0.037411 (-0.006907) | 0.097394 / 0.014526 (0.082869) | 0.127144 / 0.176557 (-0.049412) | 0.190251 / 0.737135 (-0.546884) | 0.116543 / 0.296338 (-0.179795) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.592124 / 0.215209 (0.376915) | 5.979801 / 2.077655 (3.902146) | 2.837753 / 1.504120 (1.333633) | 2.492942 / 1.541195 (0.951747) | 2.548083 / 1.468490 (1.079593) | 0.870446 / 4.584777 (-3.714330) | 5.493718 / 3.745712 (1.748006) | 4.945135 / 5.269862 (-0.324727) | 3.133994 / 4.565676 (-1.431683) | 0.097742 / 0.424275 (-0.326533) | 0.008750 / 0.007607 (0.001143) | 0.723304 / 0.226044 (0.497260) | 7.353766 / 2.268929 (5.084838) | 3.504808 / 55.444624 (-51.939816) | 2.872490 / 6.876477 (-4.003987) | 3.186628 / 2.142072 (1.044556) | 1.035470 / 4.805227 (-3.769758) | 0.211980 / 6.500664 (-6.288684) | 0.080356 / 0.075469 (0.004887) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.623389 / 1.841788 (-0.218399) | 23.492350 / 8.074308 (15.418042) | 21.053525 / 10.191392 (10.862133) | 0.225668 / 0.680424 (-0.454756) | 0.028311 / 0.534201 (-0.505890) | 0.472672 / 0.579283 (-0.106611) | 0.581536 / 0.434364 (0.147172) | 0.525180 / 0.540337 (-0.015158) | 0.790420 / 1.386936 (-0.596516) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009091 / 0.011353 (-0.002262) | 0.004978 / 0.011008 (-0.006030) | 0.077633 / 0.038508 (0.039125) | 0.103189 / 0.023109 (0.080080) | 0.500194 / 0.275898 (0.224296) | 0.524310 / 0.323480 (0.200831) | 0.006656 / 0.007986 (-0.001329) | 0.004586 / 0.004328 (0.000257) | 0.075535 / 0.004250 (0.071284) | 0.065100 / 0.037052 (0.028048) | 0.513776 / 0.258489 (0.255287) | 0.528483 / 0.293841 (0.234642) | 0.049877 / 0.128546 (-0.078669) | 0.012494 / 0.075646 (-0.063152) | 0.090225 / 0.419271 (-0.329046) | 0.054648 / 0.043533 (0.011116) | 0.510369 / 0.255139 (0.255230) | 0.540042 / 0.283200 (0.256842) | 0.035966 / 0.141683 (-0.105717) | 1.825965 / 1.452155 (0.373810) | 1.965647 / 1.492716 (0.472931) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.295921 / 0.018006 (0.277914) | 0.605751 / 0.000490 (0.605262) | 0.007243 / 0.000200 (0.007043) | 0.000134 / 0.000054 (0.000079) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032954 / 0.037411 (-0.004457) | 0.093613 / 0.014526 (0.079087) | 0.120010 / 0.176557 (-0.056546) | 0.176168 / 0.737135 (-0.560967) | 0.113978 / 0.296338 (-0.182360) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.682904 / 0.215209 (0.467695) | 6.674640 / 2.077655 (4.596986) | 3.360660 / 1.504120 (1.856540) | 3.227246 / 1.541195 (1.686051) | 3.188852 / 1.468490 (1.720362) | 0.862293 / 4.584777 (-3.722484) | 5.518455 / 3.745712 (1.772743) | 4.881904 / 5.269862 (-0.387957) | 3.066964 / 4.565676 (-1.498712) | 0.099284 / 0.424275 (-0.324991) | 0.008644 / 0.007607 (0.001037) | 0.789231 / 0.226044 (0.563186) | 7.872017 / 2.268929 (5.603089) | 4.037105 / 55.444624 (-51.407519) | 3.318921 / 6.876477 (-3.557555) | 3.621953 / 2.142072 (1.479881) | 1.012049 / 4.805227 (-3.793178) | 0.204541 / 6.500664 (-6.296123) | 0.074509 / 0.075469 (-0.000960) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.748215 / 1.841788 (-0.093573) | 24.274974 / 8.074308 (16.200665) | 20.582389 / 10.191392 (10.390997) | 0.251001 / 0.680424 (-0.429423) | 0.032390 / 0.534201 (-0.501811) | 0.479211 / 0.579283 (-0.100072) | 0.607482 / 0.434364 (0.173118) | 0.587867 / 0.540337 (0.047530) | 0.822399 / 1.386936 (-0.564537) |

![](https://cml.dev/watermark.png#f2b6b2fd90ba47f19e9ab125f6f7656903dd065f "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009715 / 0.011353 (-0.001638) | 0.005449 / 0.011008 (-0.005559) | 0.108556 / 0.038508 (0.070048) | 0.080512 / 0.023109 (0.057403) | 0.450736 / 0.275898 (0.174838) | 0.487771 / 0.323480 (0.164291) | 0.005155 / 0.007986 (-0.002830) | 0.004213 / 0.004328 (-0.000115) | 0.087247 / 0.004250 (0.082997) | 0.063962 / 0.037052 (0.026909) | 0.454153 / 0.258489 (0.195664) | 0.499917 / 0.293841 (0.206076) | 0.052605 / 0.128546 (-0.075942) | 0.013019 / 0.075646 (-0.062627) | 0.379716 / 0.419271 (-0.039555) | 0.073241 / 0.043533 (0.029708) | 0.473488 / 0.255139 (0.218349) | 0.482944 / 0.283200 (0.199745) | 0.041541 / 0.141683 (-0.100142) | 1.829415 / 1.452155 (0.377261) | 1.953280 / 1.492716 (0.460564) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.313725 / 0.018006 (0.295719) | 0.591336 / 0.000490 (0.590847) | 0.021224 / 0.000200 (0.021025) | 0.000969 / 0.000054 (0.000914) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031874 / 0.037411 (-0.005537) | 0.099786 / 0.014526 (0.085260) | 0.116987 / 0.176557 (-0.059569) | 0.205538 / 0.737135 (-0.531597) | 0.118716 / 0.296338 (-0.177622) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.617145 / 0.215209 (0.401936) | 6.079144 / 2.077655 (4.001489) | 2.567233 / 1.504120 (1.063113) | 2.265301 / 1.541195 (0.724107) | 2.314001 / 1.468490 (0.845511) | 0.871561 / 4.584777 (-3.713216) | 5.477049 / 3.745712 (1.731337) | 4.720552 / 5.269862 (-0.549309) | 3.107515 / 4.565676 (-1.458162) | 0.100438 / 0.424275 (-0.323838) | 0.008586 / 0.007607 (0.000979) | 0.716913 / 0.226044 (0.490869) | 7.108417 / 2.268929 (4.839489) | 3.391336 / 55.444624 (-52.053288) | 2.734052 / 6.876477 (-4.142425) | 2.857226 / 2.142072 (0.715153) | 1.024121 / 4.805227 (-3.781106) | 0.216735 / 6.500664 (-6.283929) | 0.081605 / 0.075469 (0.006136) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.678176 / 1.841788 (-0.163611) | 23.606037 / 8.074308 (15.531729) | 21.485331 / 10.191392 (11.293939) | 0.218312 / 0.680424 (-0.462112) | 0.027061 / 0.534201 (-0.507140) | 0.481188 / 0.579283 (-0.098096) | 0.620592 / 0.434364 (0.186228) | 0.574778 / 0.540337 (0.034441) | 0.831529 / 1.386936 (-0.555407) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.011666 / 0.011353 (0.000313) | 0.005187 / 0.011008 (-0.005821) | 0.080692 / 0.038508 (0.042184) | 0.079159 / 0.023109 (0.056049) | 0.530823 / 0.275898 (0.254925) | 0.577807 / 0.323480 (0.254327) | 0.006246 / 0.007986 (-0.001740) | 0.004355 / 0.004328 (0.000026) | 0.080702 / 0.004250 (0.076452) | 0.062279 / 0.037052 (0.025226) | 0.553712 / 0.258489 (0.295223) | 0.579112 / 0.293841 (0.285271) | 0.056374 / 0.128546 (-0.072172) | 0.014681 / 0.075646 (-0.060966) | 0.097110 / 0.419271 (-0.322161) | 0.061040 / 0.043533 (0.017507) | 0.524718 / 0.255139 (0.269579) | 0.568586 / 0.283200 (0.285386) | 0.035774 / 0.141683 (-0.105909) | 1.864590 / 1.452155 (0.412435) | 1.953715 / 1.492716 (0.460998) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.271315 / 0.018006 (0.253309) | 0.571343 / 0.000490 (0.570854) | 0.015812 / 0.000200 (0.015612) | 0.000115 / 0.000054 (0.000060) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.038582 / 0.037411 (0.001170) | 0.117523 / 0.014526 (0.102997) | 0.128864 / 0.176557 (-0.047693) | 0.191164 / 0.737135 (-0.545971) | 0.133161 / 0.296338 (-0.163178) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.679305 / 0.215209 (0.464096) | 6.814451 / 2.077655 (4.736796) | 3.377431 / 1.504120 (1.873311) | 3.011008 / 1.541195 (1.469813) | 3.093200 / 1.468490 (1.624710) | 0.905827 / 4.584777 (-3.678950) | 5.456094 / 3.745712 (1.710382) | 4.848511 / 5.269862 (-0.421351) | 3.064230 / 4.565676 (-1.501447) | 0.107478 / 0.424275 (-0.316798) | 0.009234 / 0.007607 (0.001627) | 0.833944 / 0.226044 (0.607899) | 8.286100 / 2.268929 (6.017171) | 4.241455 / 55.444624 (-51.203169) | 3.405460 / 6.876477 (-3.471017) | 3.660618 / 2.142072 (1.518546) | 1.046310 / 4.805227 (-3.758917) | 0.210891 / 6.500664 (-6.289773) | 0.079413 / 0.075469 (0.003944) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.825448 / 1.841788 (-0.016340) | 24.639059 / 8.074308 (16.564750) | 21.970417 / 10.191392 (11.779025) | 0.247708 / 0.680424 (-0.432715) | 0.033810 / 0.534201 (-0.500391) | 0.495517 / 0.579283 (-0.083766) | 0.601820 / 0.434364 (0.167456) | 0.585618 / 0.540337 (0.045280) | 0.858722 / 1.386936 (-0.528214) |

![](https://cml.dev/watermark.png#0477e20dccb77b68f0add77fd5c9b4cb05473235 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006137 / 0.011353 (-0.005216) | 0.003685 / 0.011008 (-0.007324) | 0.079985 / 0.038508 (0.041476) | 0.060937 / 0.023109 (0.037828) | 0.390583 / 0.275898 (0.114685) | 0.425307 / 0.323480 (0.101827) | 0.003433 / 0.007986 (-0.004552) | 0.002868 / 0.004328 (-0.001461) | 0.062572 / 0.004250 (0.058322) | 0.048642 / 0.037052 (0.011590) | 0.401096 / 0.258489 (0.142607) | 0.436988 / 0.293841 (0.143147) | 0.027645 / 0.128546 (-0.100901) | 0.007973 / 0.075646 (-0.067673) | 0.261997 / 0.419271 (-0.157275) | 0.045393 / 0.043533 (0.001860) | 0.394266 / 0.255139 (0.139127) | 0.414448 / 0.283200 (0.131248) | 0.022551 / 0.141683 (-0.119131) | 1.438458 / 1.452155 (-0.013697) | 1.501568 / 1.492716 (0.008852) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.224335 / 0.018006 (0.206329) | 0.421918 / 0.000490 (0.421428) | 0.006883 / 0.000200 (0.006683) | 0.000210 / 0.000054 (0.000155) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023505 / 0.037411 (-0.013906) | 0.072438 / 0.014526 (0.057912) | 0.083576 / 0.176557 (-0.092981) | 0.142906 / 0.737135 (-0.594229) | 0.083910 / 0.296338 (-0.212428) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.396004 / 0.215209 (0.180795) | 3.969852 / 2.077655 (1.892197) | 1.966000 / 1.504120 (0.461880) | 1.786453 / 1.541195 (0.245258) | 1.866082 / 1.468490 (0.397592) | 0.502633 / 4.584777 (-4.082144) | 3.114331 / 3.745712 (-0.631382) | 2.940003 / 5.269862 (-2.329859) | 1.901844 / 4.565676 (-2.663832) | 0.058109 / 0.424275 (-0.366166) | 0.006502 / 0.007607 (-0.001105) | 0.463465 / 0.226044 (0.237420) | 4.641531 / 2.268929 (2.372603) | 2.315759 / 55.444624 (-53.128865) | 2.253088 / 6.876477 (-4.623389) | 2.151399 / 2.142072 (0.009326) | 0.592225 / 4.805227 (-4.213002) | 0.125072 / 6.500664 (-6.375592) | 0.059966 / 0.075469 (-0.015503) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.231392 / 1.841788 (-0.610396) | 17.533893 / 8.074308 (9.459585) | 13.710478 / 10.191392 (3.519086) | 0.147389 / 0.680424 (-0.533035) | 0.017932 / 0.534201 (-0.516269) | 0.334144 / 0.579283 (-0.245139) | 0.368817 / 0.434364 (-0.065547) | 0.383790 / 0.540337 (-0.156547) | 0.540262 / 1.386936 (-0.846674) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006066 / 0.011353 (-0.005287) | 0.003804 / 0.011008 (-0.007205) | 0.062474 / 0.038508 (0.023966) | 0.060547 / 0.023109 (0.037437) | 0.448643 / 0.275898 (0.172745) | 0.487005 / 0.323480 (0.163525) | 0.004884 / 0.007986 (-0.003102) | 0.002911 / 0.004328 (-0.001418) | 0.062950 / 0.004250 (0.058700) | 0.049672 / 0.037052 (0.012620) | 0.477491 / 0.258489 (0.219002) | 0.488234 / 0.293841 (0.194393) | 0.028711 / 0.128546 (-0.099835) | 0.008101 / 0.075646 (-0.067545) | 0.068333 / 0.419271 (-0.350939) | 0.040959 / 0.043533 (-0.002574) | 0.450716 / 0.255139 (0.195577) | 0.471089 / 0.283200 (0.187890) | 0.020710 / 0.141683 (-0.120973) | 1.474850 / 1.452155 (0.022695) | 1.540115 / 1.492716 (0.047399) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.229811 / 0.018006 (0.211805) | 0.419526 / 0.000490 (0.419036) | 0.003818 / 0.000200 (0.003618) | 0.000084 / 0.000054 (0.000030) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026045 / 0.037411 (-0.011366) | 0.080325 / 0.014526 (0.065799) | 0.091549 / 0.176557 (-0.085007) | 0.145253 / 0.737135 (-0.591882) | 0.091849 / 0.296338 (-0.204489) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.463047 / 0.215209 (0.247838) | 4.598727 / 2.077655 (2.521072) | 2.558996 / 1.504120 (1.054877) | 2.405896 / 1.541195 (0.864701) | 2.447291 / 1.468490 (0.978801) | 0.510393 / 4.584777 (-4.074384) | 3.173344 / 3.745712 (-0.572368) | 2.901201 / 5.269862 (-2.368661) | 1.896440 / 4.565676 (-2.669236) | 0.058374 / 0.424275 (-0.365901) | 0.006449 / 0.007607 (-0.001158) | 0.539653 / 0.226044 (0.313608) | 5.408217 / 2.268929 (3.139289) | 3.042453 / 55.444624 (-52.402172) | 2.656724 / 6.876477 (-4.219753) | 2.838165 / 2.142072 (0.696092) | 0.598663 / 4.805227 (-4.206565) | 0.126211 / 6.500664 (-6.374453) | 0.062830 / 0.075469 (-0.012639) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.392412 / 1.841788 (-0.449376) | 18.195170 / 8.074308 (10.120862) | 14.788251 / 10.191392 (4.596859) | 0.132579 / 0.680424 (-0.547845) | 0.017867 / 0.534201 (-0.516334) | 0.340020 / 0.579283 (-0.239263) | 0.386719 / 0.434364 (-0.047645) | 0.398863 / 0.540337 (-0.141475) | 0.579320 / 1.386936 (-0.807617) |

![](https://cml.dev/watermark.png#a2569fdfcf387f8885974a35fafa409fbc6dd059 "CML watermark")

closing in favor of https://github.com/huggingface/datasets/pull/6282

I added a new DataFilesSet class to disallow duplicate data files.
I also deprecated DataFilesList.

EDIT: actually I might just add drop_duplicates=True to `.from_patterns`

close https://github.com/huggingface/datasets/issues/6259
close https://github.com/huggingface/datasets/issues/6272

TODO:
- [ ] tests
- [ ] preserve data files order


`evaluate.load("paws-x", "es")` throws the error because there is no such metric in the `evaluate` lib.

So, this is unrelated to our lib.

### Describe the bug

I'm encountering a "FileNotFoundError" while attempting to use the "paws-x" dataset to retrain the DistilRoBERTa-base model. The error message is as follows:

FileNotFoundError: Couldn't find a module script at /content/paws-x/paws-x.py. Module 'paws-x' doesn't exist on the Hugging Face Hub either.

### Steps to reproduce the bug

https://colab.research.google.com/drive/11xUUFxloClpmqLvDy_Xxfmo3oUzjY5nx#scrollTo=kUn74FigzhHm

### Expected behavior

The the trained model

### Environment info

colab, "paws-x" dataset , DistilRoBERTa-base model