Add repo_id to DatasetInfo

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6268). All of your documentation changes will be reflected on that endpoint.

In https://github.com/huggingface/datasets/issues/4129 we want to track the origin of a dataset, e.g. if it comes from multiple datasets.

I think it's out of scope of DatasetInfo alone, which has info for one dataset only.
Therefore it makes sense to add repo_id, which is for one dataset only.

IMO if we want to track multiple origins we will need a new DatasetInfo that would have fields relevant to a mix of datasets (out of scope of this PR)

cc @mariosasko I'd like your opinion on this

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009009 / 0.011353 (-0.002344) | 0.004169 / 0.011008 (-0.006840) | 0.098634 / 0.038508 (0.060126) | 0.069526 / 0.023109 (0.046417) | 0.337963 / 0.275898 (0.062065) | 0.379737 / 0.323480 (0.056257) | 0.004318 / 0.007986 (-0.003668) | 0.005347 / 0.004328 (0.001019) | 0.069875 / 0.004250 (0.065624) | 0.055964 / 0.037052 (0.018912) | 0.340305 / 0.258489 (0.081816) | 0.429718 / 0.293841 (0.135877) | 0.045101 / 0.128546 (-0.083445) | 0.012610 / 0.075646 (-0.063036) | 0.312366 / 0.419271 (-0.106905) | 0.064711 / 0.043533 (0.021178) | 0.345216 / 0.255139 (0.090077) | 0.367245 / 0.283200 (0.084046) | 0.034638 / 0.141683 (-0.107045) | 1.541947 / 1.452155 (0.089793) | 1.645268 / 1.492716 (0.152551) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.233501 / 0.018006 (0.215495) | 0.514207 / 0.000490 (0.513717) | 0.014271 / 0.000200 (0.014072) | 0.000366 / 0.000054 (0.000311) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026288 / 0.037411 (-0.011124) | 0.083206 / 0.014526 (0.068680) | 0.098172 / 0.176557 (-0.078385) | 0.158529 / 0.737135 (-0.578606) | 0.095183 / 0.296338 (-0.201155) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.538300 / 0.215209 (0.323091) | 5.486939 / 2.077655 (3.409285) | 2.321812 / 1.504120 (0.817692) | 2.002124 / 1.541195 (0.460929) | 2.045043 / 1.468490 (0.576553) | 0.852772 / 4.584777 (-3.732005) | 5.014897 / 3.745712 (1.269185) | 4.428115 / 5.269862 (-0.841746) | 2.750126 / 4.565676 (-1.815550) | 0.099028 / 0.424275 (-0.325247) | 0.007678 / 0.007607 (0.000070) | 0.664463 / 0.226044 (0.438418) | 6.617811 / 2.268929 (4.348883) | 2.888382 / 55.444624 (-52.556242) | 2.190753 / 6.876477 (-4.685724) | 2.414586 / 2.142072 (0.272513) | 1.010302 / 4.805227 (-3.794925) | 0.194925 / 6.500664 (-6.305739) | 0.063490 / 0.075469 (-0.011979) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.543464 / 1.841788 (-0.298323) | 20.566666 / 8.074308 (12.492358) | 19.410745 / 10.191392 (9.219353) | 0.207077 / 0.680424 (-0.473347) | 0.028895 / 0.534201 (-0.505306) | 0.427525 / 0.579283 (-0.151758) | 0.535450 / 0.434364 (0.101086) | 0.494632 / 0.540337 (-0.045705) | 0.723705 / 1.386936 (-0.663231) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008209 / 0.011353 (-0.003144) | 0.004184 / 0.011008 (-0.006824) | 0.072420 / 0.038508 (0.033912) | 0.066851 / 0.023109 (0.043742) | 0.424137 / 0.275898 (0.148239) | 0.473156 / 0.323480 (0.149676) | 0.005394 / 0.007986 (-0.002591) | 0.003898 / 0.004328 (-0.000430) | 0.069996 / 0.004250 (0.065746) | 0.053113 / 0.037052 (0.016061) | 0.453214 / 0.258489 (0.194725) | 0.495921 / 0.293841 (0.202080) | 0.043028 / 0.128546 (-0.085519) | 0.012320 / 0.075646 (-0.063326) | 0.080270 / 0.419271 (-0.339002) | 0.053337 / 0.043533 (0.009804) | 0.436604 / 0.255139 (0.181465) | 0.463422 / 0.283200 (0.180223) | 0.030277 / 0.141683 (-0.111406) | 1.560261 / 1.452155 (0.108106) | 1.647209 / 1.492716 (0.154493) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.232556 / 0.018006 (0.214550) | 0.502387 / 0.000490 (0.501897) | 0.006688 / 0.000200 (0.006488) | 0.000118 / 0.000054 (0.000064) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030204 / 0.037411 (-0.007207) | 0.089438 / 0.014526 (0.074912) | 0.118939 / 0.176557 (-0.057617) | 0.160537 / 0.737135 (-0.576598) | 0.113432 / 0.296338 (-0.182906) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.586469 / 0.215209 (0.371260) | 5.916156 / 2.077655 (3.838502) | 2.904960 / 1.504120 (1.400840) | 2.346838 / 1.541195 (0.805644) | 2.373688 / 1.468490 (0.905198) | 0.829917 / 4.584777 (-3.754860) | 4.851283 / 3.745712 (1.105571) | 4.220103 / 5.269862 (-1.049758) | 2.706139 / 4.565676 (-1.859538) | 0.094095 / 0.424275 (-0.330180) | 0.008201 / 0.007607 (0.000594) | 0.699099 / 0.226044 (0.473054) | 7.046940 / 2.268929 (4.778011) | 3.374837 / 55.444624 (-52.069788) | 2.690839 / 6.876477 (-4.185638) | 2.845717 / 2.142072 (0.703645) | 0.989698 / 4.805227 (-3.815529) | 0.190413 / 6.500664 (-6.310251) | 0.066233 / 0.075469 (-0.009236) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.513607 / 1.841788 (-0.328180) | 21.544200 / 8.074308 (13.469892) | 20.297337 / 10.191392 (10.105945) | 0.216390 / 0.680424 (-0.464034) | 0.029962 / 0.534201 (-0.504239) | 0.451531 / 0.579283 (-0.127752) | 0.530147 / 0.434364 (0.095783) | 0.520739 / 0.540337 (-0.019598) | 0.716431 / 1.386936 (-0.670505) |

![](https://cml.dev/watermark.png#fcaa9f218ad1505bb5474060889b4b9578e24423 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006509 / 0.011353 (-0.004844) | 0.003987 / 0.011008 (-0.007022) | 0.085233 / 0.038508 (0.046725) | 0.077765 / 0.023109 (0.054656) | 0.310467 / 0.275898 (0.034569) | 0.343363 / 0.323480 (0.019883) | 0.005557 / 0.007986 (-0.002429) | 0.003430 / 0.004328 (-0.000898) | 0.064948 / 0.004250 (0.060697) | 0.056864 / 0.037052 (0.019812) | 0.314005 / 0.258489 (0.055516) | 0.360638 / 0.293841 (0.066798) | 0.031134 / 0.128546 (-0.097412) | 0.008869 / 0.075646 (-0.066777) | 0.286409 / 0.419271 (-0.132862) | 0.051338 / 0.043533 (0.007805) | 0.311329 / 0.255139 (0.056190) | 0.334373 / 0.283200 (0.051174) | 0.024816 / 0.141683 (-0.116867) | 1.502872 / 1.452155 (0.050718) | 1.569941 / 1.492716 (0.077224) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.269639 / 0.018006 (0.251633) | 0.558510 / 0.000490 (0.558020) | 0.011748 / 0.000200 (0.011548) | 0.000234 / 0.000054 (0.000180) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029139 / 0.037411 (-0.008272) | 0.083586 / 0.014526 (0.069060) | 0.102426 / 0.176557 (-0.074131) | 0.162398 / 0.737135 (-0.574737) | 0.101364 / 0.296338 (-0.194975) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.382281 / 0.215209 (0.167072) | 3.826412 / 2.077655 (1.748758) | 1.815911 / 1.504120 (0.311791) | 1.644539 / 1.541195 (0.103344) | 1.688487 / 1.468490 (0.219996) | 0.482115 / 4.584777 (-4.102662) | 3.574773 / 3.745712 (-0.170939) | 3.262733 / 5.269862 (-2.007129) | 2.058115 / 4.565676 (-2.507562) | 0.056367 / 0.424275 (-0.367908) | 0.007233 / 0.007607 (-0.000374) | 0.456859 / 0.226044 (0.230815) | 4.565935 / 2.268929 (2.297006) | 2.311802 / 55.444624 (-53.132823) | 1.943936 / 6.876477 (-4.932541) | 2.129811 / 2.142072 (-0.012261) | 0.575098 / 4.805227 (-4.230129) | 0.130495 / 6.500664 (-6.370169) | 0.059757 / 0.075469 (-0.015712) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.238495 / 1.841788 (-0.603293) | 18.940000 / 8.074308 (10.865692) | 14.034240 / 10.191392 (3.842848) | 0.166418 / 0.680424 (-0.514006) | 0.018420 / 0.534201 (-0.515781) | 0.395330 / 0.579283 (-0.183953) | 0.413518 / 0.434364 (-0.020846) | 0.461499 / 0.540337 (-0.078838) | 0.661371 / 1.386936 (-0.725565) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006673 / 0.011353 (-0.004680) | 0.004335 / 0.011008 (-0.006673) | 0.064568 / 0.038508 (0.026060) | 0.072763 / 0.023109 (0.049653) | 0.429488 / 0.275898 (0.153590) | 0.456900 / 0.323480 (0.133420) | 0.005481 / 0.007986 (-0.002505) | 0.003649 / 0.004328 (-0.000680) | 0.064975 / 0.004250 (0.060724) | 0.056839 / 0.037052 (0.019786) | 0.439451 / 0.258489 (0.180962) | 0.461691 / 0.293841 (0.167850) | 0.031455 / 0.128546 (-0.097092) | 0.008848 / 0.075646 (-0.066798) | 0.071719 / 0.419271 (-0.347553) | 0.047116 / 0.043533 (0.003583) | 0.429055 / 0.255139 (0.173916) | 0.434204 / 0.283200 (0.151004) | 0.022594 / 0.141683 (-0.119089) | 1.539231 / 1.452155 (0.087077) | 1.568111 / 1.492716 (0.075394) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.267374 / 0.018006 (0.249368) | 0.553202 / 0.000490 (0.552712) | 0.005410 / 0.000200 (0.005210) | 0.000101 / 0.000054 (0.000046) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031478 / 0.037411 (-0.005933) | 0.092438 / 0.014526 (0.077912) | 0.103874 / 0.176557 (-0.072682) | 0.158428 / 0.737135 (-0.578708) | 0.111617 / 0.296338 (-0.184721) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.434783 / 0.215209 (0.219574) | 4.332536 / 2.077655 (2.254881) | 2.354522 / 1.504120 (0.850402) | 2.220271 / 1.541195 (0.679076) | 2.338524 / 1.468490 (0.870034) | 0.494508 / 4.584777 (-4.090269) | 3.619592 / 3.745712 (-0.126120) | 3.320897 / 5.269862 (-1.948964) | 2.107475 / 4.565676 (-2.458202) | 0.058479 / 0.424275 (-0.365796) | 0.007427 / 0.007607 (-0.000180) | 0.509298 / 0.226044 (0.283254) | 5.067940 / 2.268929 (2.799012) | 2.815336 / 55.444624 (-52.629288) | 2.470958 / 6.876477 (-4.405519) | 2.672801 / 2.142072 (0.530728) | 0.588199 / 4.805227 (-4.217028) | 0.134062 / 6.500664 (-6.366602) | 0.060951 / 0.075469 (-0.014518) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.353955 / 1.841788 (-0.487832) | 20.386012 / 8.074308 (12.311704) | 15.032463 / 10.191392 (4.841071) | 0.167243 / 0.680424 (-0.513181) | 0.020426 / 0.534201 (-0.513775) | 0.396815 / 0.579283 (-0.182469) | 0.421806 / 0.434364 (-0.012558) | 0.471866 / 0.540337 (-0.068471) | 0.667206 / 1.386936 (-0.719730) |

![](https://cml.dev/watermark.png#aade5a0c79398c84632a3ff253111e694c7b598b "CML watermark")

Really happy to see this! It could also be helpful to track some other metadata about how the dataset was built in the future. i.e. for the Stack loaded like this:

```
ds = load_dataset("bigcode/the-stack", data_dir="data/dockerfile", split="train")
```
It could be helpful to have easy access to the `data_dir` argument used during loading since that changes the training data quite a bit vs. loading the full dataset. You can also recover this from `download_checksums`, which seems a bit hacky. That is not necessary for this PR, though.

Perhaps it is also interesting to track the revision? I suppose the version also kind of covers that.

That said, this is looking great already! I'm quite excited about this. Losing the `repo_id` after merging (different) datasets also makes sense to me, well done.

One other thought. Is it worth tracking if a `token` was passed during loading?

The Hub ID for private datasets could in some cases contain information someone wouldn't want to make public i.e. `davanstrien/super_secret_dataset_using_GPT_created_data`.

Adding a bool like `is_private` could then be used by another library to determine if the dataset ID should be shared or not (or default to not sharing the ID for private datasets). i.e. in SpanMarker @tomaarsen might do a check like

```python
if ds.is_private and not push_hub_id_for_private_ds:
	ds_name  = None
```
Potentially this is overkill but could be useful for downstream libraries who might use this information for creating automatic model cards.

We should probably find a way to remove `DatasetInfo`, as (most of) its attributes are outdated (homepage, description, etc.), not introduce new ones :). But I guess storing `repo_id` there is the simplest solution for now, so I'm OK with it.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007757 / 0.011353 (-0.003595) | 0.004543 / 0.011008 (-0.006465) | 0.100193 / 0.038508 (0.061685) | 0.082333 / 0.023109 (0.059224) | 0.374586 / 0.275898 (0.098688) | 0.412617 / 0.323480 (0.089137) | 0.006148 / 0.007986 (-0.001838) | 0.003826 / 0.004328 (-0.000503) | 0.077077 / 0.004250 (0.072827) | 0.064057 / 0.037052 (0.027005) | 0.391435 / 0.258489 (0.132946) | 0.436439 / 0.293841 (0.142599) | 0.036534 / 0.128546 (-0.092012) | 0.009986 / 0.075646 (-0.065660) | 0.344243 / 0.419271 (-0.075028) | 0.062013 / 0.043533 (0.018480) | 0.378113 / 0.255139 (0.122974) | 0.398476 / 0.283200 (0.115276) | 0.026552 / 0.141683 (-0.115131) | 1.740505 / 1.452155 (0.288350) | 1.835684 / 1.492716 (0.342968) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.267917 / 0.018006 (0.249911) | 0.510676 / 0.000490 (0.510186) | 0.010810 / 0.000200 (0.010610) | 0.000383 / 0.000054 (0.000328) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032113 / 0.037411 (-0.005299) | 0.097679 / 0.014526 (0.083153) | 0.113213 / 0.176557 (-0.063344) | 0.177897 / 0.737135 (-0.559238) | 0.111761 / 0.296338 (-0.184577) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.450544 / 0.215209 (0.235335) | 4.476746 / 2.077655 (2.399091) | 2.205391 / 1.504120 (0.701271) | 2.006457 / 1.541195 (0.465262) | 2.058859 / 1.468490 (0.590369) | 0.571549 / 4.584777 (-4.013228) | 4.175039 / 3.745712 (0.429327) | 3.815445 / 5.269862 (-1.454416) | 2.376673 / 4.565676 (-2.189004) | 0.067048 / 0.424275 (-0.357227) | 0.008544 / 0.007607 (0.000937) | 0.536384 / 0.226044 (0.310340) | 5.386232 / 2.268929 (3.117304) | 2.825620 / 55.444624 (-52.619004) | 2.339821 / 6.876477 (-4.536656) | 2.535736 / 2.142072 (0.393663) | 0.679572 / 4.805227 (-4.125655) | 0.156799 / 6.500664 (-6.343865) | 0.071667 / 0.075469 (-0.003802) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.512198 / 1.841788 (-0.329590) | 21.786760 / 8.074308 (13.712452) | 16.386274 / 10.191392 (6.194882) | 0.169108 / 0.680424 (-0.511316) | 0.021312 / 0.534201 (-0.512889) | 0.466153 / 0.579283 (-0.113130) | 0.496192 / 0.434364 (0.061829) | 0.549420 / 0.540337 (0.009082) | 0.780974 / 1.386936 (-0.605962) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007644 / 0.011353 (-0.003709) | 0.004654 / 0.011008 (-0.006354) | 0.075280 / 0.038508 (0.036772) | 0.083044 / 0.023109 (0.059935) | 0.481704 / 0.275898 (0.205805) | 0.514828 / 0.323480 (0.191348) | 0.006245 / 0.007986 (-0.001740) | 0.003715 / 0.004328 (-0.000614) | 0.074498 / 0.004250 (0.070248) | 0.064406 / 0.037052 (0.027353) | 0.481874 / 0.258489 (0.223385) | 0.518527 / 0.293841 (0.224686) | 0.037549 / 0.128546 (-0.090997) | 0.010106 / 0.075646 (-0.065541) | 0.084266 / 0.419271 (-0.335006) | 0.056659 / 0.043533 (0.013126) | 0.497707 / 0.255139 (0.242568) | 0.503201 / 0.283200 (0.220001) | 0.027086 / 0.141683 (-0.114597) | 1.834715 / 1.452155 (0.382560) | 1.919927 / 1.492716 (0.427210) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.249288 / 0.018006 (0.231282) | 0.500950 / 0.000490 (0.500460) | 0.005856 / 0.000200 (0.005656) | 0.000120 / 0.000054 (0.000065) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.037674 / 0.037411 (0.000263) | 0.111141 / 0.014526 (0.096615) | 0.123408 / 0.176557 (-0.053149) | 0.186604 / 0.737135 (-0.550531) | 0.125360 / 0.296338 (-0.170979) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.520480 / 0.215209 (0.305271) | 5.171108 / 2.077655 (3.093453) | 2.812746 / 1.504120 (1.308626) | 2.602941 / 1.541195 (1.061746) | 2.666196 / 1.468490 (1.197706) | 0.578684 / 4.584777 (-4.006092) | 4.238722 / 3.745712 (0.493010) | 3.844361 / 5.269862 (-1.425501) | 2.369214 / 4.565676 (-2.196462) | 0.068543 / 0.424275 (-0.355732) | 0.008695 / 0.007607 (0.001088) | 0.621869 / 0.226044 (0.395825) | 6.200566 / 2.268929 (3.931637) | 3.340846 / 55.444624 (-52.103779) | 2.920691 / 6.876477 (-3.955786) | 3.132438 / 2.142072 (0.990366) | 0.697394 / 4.805227 (-4.107834) | 0.158385 / 6.500664 (-6.342280) | 0.072566 / 0.075469 (-0.002903) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.599070 / 1.841788 (-0.242717) | 22.767139 / 8.074308 (14.692831) | 17.053988 / 10.191392 (6.862596) | 0.188414 / 0.680424 (-0.492009) | 0.023409 / 0.534201 (-0.510792) | 0.472092 / 0.579283 (-0.107191) | 0.486107 / 0.434364 (0.051743) | 0.562190 / 0.540337 (0.021852) | 0.791606 / 1.386936 (-0.595330) |

![](https://cml.dev/watermark.png#aacbaf45c93f88e8c95924f6224153fb37c3064a "CML watermark")

```python
from datasets import load_dataset

ds = load_dataset("lhoestq/demo1", split="train")
ds = ds.map(lambda x: {}, num_proc=2).filter(lambda x: True).remove_columns(["id"])
print(ds.repo_id)
# lhoestq/demo1
```

- repo_id is None when the dataset doesn't come from the Hub, e.g. from Dataset.from_dict
- repo_id is set to None when concatenating datasets with different repo ids

related to https://github.com/huggingface/datasets/issues/4129

TODO:
- [ ] discuss if it's ok for now
- [ ] tests


Multi label class encoding

enhancement

New feature or request

You can use a `Sequence(ClassLabel(...))` feature type to represent a list of labels, and `cast_column`/`cast` to perform the "string to label" conversion  (`class_encode_column` does support nested fields), e.g., in your case:
```python
from datasets import Dataset, Sequence, ClassLabel
data = {
    'text': ['one', 'two', 'three', 'four'],
    'labels': [['a', 'b'], ['b'], ['b', 'c'], ['a', 'd']]
}

dataset = Dataset.from_dict(data)
dataset = dataset.cast_column('labels', Sequence(ClassLabel(names=["a", "b", "c", "d"])))
```

Great!  Can you elaborate on "class_encode_column does support nested fields"?  Do you mean that there is a way to `class_encode_column` on a Sequence?

Yes, exactly! This would be a nice contribution, though.

Sorry, I'm still not following.   Are you saying that there currently exists a way to call `class_encode_column` on a `Sequence(ClassLabel)` type?  Or that the underlying data structures support it and a contribution of a method to do that would be welcome?

`class_encode_column ` currently does not support `Sequence(ClassLabel)`. Implementing support for this would be a nice contribution.

In the meantime, this limitation can be circumvented by fetching (unique) labels and calling `.cast_column(col, Sequence(ClassLabel(names=labels)))`.

Ok makes sense, can you take a look at the POC implementation I did [here](https://github.com/huggingface/datasets/commit/15443098e9ce053943172f7ec6fce3769d7dff6e)?  Happy to take another pass / submit as a PR but would be helpful if I got a thumbs up that this was directionally correct with respect to implementation / architecture.

There is no need to introduce a new type (`MultiLabel`) for this feature. Also, I think we can keep the logic inside a single method instead of separating the two cases.

Maybe https://github.com/huggingface/datasets/pull/4277 can help with the implementation. We extended `align_labels_with_mapping` to support `Sequence(ClassLabel(...))` in that PR (initially, it only worked with `ClassLabel(...)`)

### Feature request

I have a multi label dataset and I'd like to be able to class encode the column and store the mapping directly in the features just as I can with a single label column.  `class_encode_column` currently does not support multi labels.

Here's an example of what I'd like to encode:

```
data = {
    'text': ['one', 'two', 'three', 'four'],
    'labels': [['a', 'b'], ['b'], ['b', 'c'], ['a', 'd']]
}

dataset = Dataset.from_dict(data)
dataset = dataset.class_encode_column('labels')
```

I did some digging into the code base to evaluate the feasibility of this (note I'm very new to this code base) and from what I noticed the `ClassLabel` feature is still stored as an underlying raw data type of int so I thought a `MultiLabel` feature could similarly be stored as a Sequence of ints, thus not requiring significant serialization / conversion work to / from arrow.

I did a POC of this [here](https://github.com/huggingface/datasets/commit/15443098e9ce053943172f7ec6fce3769d7dff6e) and included a simple test case (please excuse all the commented out tests, going for speed of POC here and didn't want to fight IDE to debug a single test).   In the test I just assert that `num_classes` is the same to show that things are properly serializing, but if you break after loading from disk you'll see the dataset correct and the dataset feature is as expected.

After digging more I did notice a few issues
- After loading from disk I noticed type of the `labels` class is `Sequence` not `MultiLabel` (though the added `feature` attribute came through).  This doesn't happen for `ClassLabel` but I couldn't find the encode / decode code paths that handle this.
- I subclass `Sequence` in `MultiLabel` to leverage existing serialization, but this does miss the custom encode logic that `ClassLabel` has.  I'm not sure of the best way to approach this as I haven't fully understood the encode / decode flow for datasets.  I suspect my simple implementation will need some improvement as it'll require a significant amount of repeated logic to mimic `ClassLabel` behavior.

### Motivation

See above - would like to support multi label class encodings.

### Your contribution

This would be a big help for us and we're open to contributing but I'll likely need some guidance on how to implement to fit the encode / decode flow.  Some suggestions on tests / would be great too, I'm guessing in addition to the class encode tests (that I'll need to expand) we'll need encode / decode tests.


Use LibYAML with PyYAML if available

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6266). All of your documentation changes will be reflected on that endpoint.

On Ubuntu, if `libyaml-dev` is installed, you can install PyYAML 6.0.1 with LibYAML with the following command (as it's automatically detected):

```bash
pip install git+https://github.com/yaml/pyyaml.git@6.0.1
```

We use `huggingface_hub`'s RepoCard API instead of these modules to parse the YAML block (notice the deprecations), so the `huggingface_hub` repo is the right place to suggest these changes.

Personally, I'm not a fan of these changes, as a single non-standard usage of the `ClassLabel` type is not a sufficient reason to merge them. Also, the dataset in question stores data in a single Parquet file, with the features info embedded in its (schema) metadata, which means the YAML parsing can be skipped while preserving the features by directly loading the Parquet file:
```python
from datasets import load_dataset
ds = load_dataset("parquet", data_files="https://huggingface.co/datasets/HuggingFaceM4/SugarCrepe_swap_obj/resolve/main/data/test-00000-of-00001-ca2ae6017a2336d7.parquet")
```

PS: Yes, these tests are flaky. We are working on fixing them.

Oh, I didn't realize they were deprecated. Thanks for the tip on how to work around this issue!

For future reference, the places to change the code in `huggingface_hub` would be:

https://github.com/huggingface/huggingface_hub/blob/89cc69105074f1d071e0471144605f3cdfe1dab3/src/huggingface_hub/repocard.py#L506

https://github.com/huggingface/huggingface_hub/blob/89cc69105074f1d071e0471144605f3cdfe1dab3/src/huggingface_hub/utils/_fixes.py#L34

PyYAML, the YAML framework used in this library, allows the use of LibYAML to accelerate the methods `load` and `dump`. To use it, a user would need to first install a PyYAML version that uses LibYAML (not available in PyPI; needs to be manually installed). Then, to actually use them, PyYAML suggests importing the LibYAML version of the `Loader` and `Dumper` and falling back to the default ones. This PR implements this change. See [PyYAML docs](https://pyyaml.org/wiki/PyYAMLDocumentation) for more info.

This change was motivated after trying to use any of [the SugarCREPE datasets in the Hub](https://huggingface.co/datasets?search=sugarcrepe) provided by [the org HuggingFaceM4](https://huggingface.co/datasets/HuggingFaceM4). Such datasets save a lot of information (~1MB) in the YAML metadata from the `README.md` file and I noticed this slowed down the data loading process. BTW, I also noticed cache files for it is also slow because it tries to hash an instance of `DatasetInfo`, which in turn has all this metadata.

Also, I changed two list comprehensions into generator expressions to avoid allocating extra memory unnecessarily.

And BTW, there's [an issue in PyYAML suggesting to make this automatic](https://github.com/yaml/pyyaml/issues/437).


Remove `apache_beam` import in `BeamBasedBuilder._save_info`

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005896 / 0.011353 (-0.005457) | 0.003642 / 0.011008 (-0.007366) | 0.081917 / 0.038508 (0.043409) | 0.059513 / 0.023109 (0.036404) | 0.341422 / 0.275898 (0.065524) | 0.359278 / 0.323480 (0.035798) | 0.004707 / 0.007986 (-0.003279) | 0.002938 / 0.004328 (-0.001390) | 0.063095 / 0.004250 (0.058845) | 0.051777 / 0.037052 (0.014725) | 0.321114 / 0.258489 (0.062625) | 0.363823 / 0.293841 (0.069982) | 0.027590 / 0.128546 (-0.100957) | 0.007846 / 0.075646 (-0.067800) | 0.261197 / 0.419271 (-0.158074) | 0.045812 / 0.043533 (0.002279) | 0.319787 / 0.255139 (0.064648) | 0.341839 / 0.283200 (0.058640) | 0.021913 / 0.141683 (-0.119770) | 1.397525 / 1.452155 (-0.054630) | 1.495902 / 1.492716 (0.003186) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.224815 / 0.018006 (0.206809) | 0.425780 / 0.000490 (0.425290) | 0.006934 / 0.000200 (0.006734) | 0.000225 / 0.000054 (0.000171) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024342 / 0.037411 (-0.013070) | 0.073923 / 0.014526 (0.059398) | 0.082108 / 0.176557 (-0.094448) | 0.143017 / 0.737135 (-0.594119) | 0.083163 / 0.296338 (-0.213175) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.398244 / 0.215209 (0.183035) | 3.957688 / 2.077655 (1.880033) | 1.904615 / 1.504120 (0.400495) | 1.710353 / 1.541195 (0.169158) | 1.798980 / 1.468490 (0.330490) | 0.499307 / 4.584777 (-4.085470) | 3.026734 / 3.745712 (-0.718978) | 2.923940 / 5.269862 (-2.345922) | 1.831870 / 4.565676 (-2.733807) | 0.058551 / 0.424275 (-0.365724) | 0.006403 / 0.007607 (-0.001204) | 0.464164 / 0.226044 (0.238119) | 4.644556 / 2.268929 (2.375628) | 2.341455 / 55.444624 (-53.103169) | 2.004385 / 6.876477 (-4.872092) | 2.051819 / 2.142072 (-0.090253) | 0.585610 / 4.805227 (-4.219617) | 0.124735 / 6.500664 (-6.375929) | 0.061150 / 0.075469 (-0.014319) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.224665 / 1.841788 (-0.617122) | 17.476227 / 8.074308 (9.401919) | 13.867617 / 10.191392 (3.676225) | 0.144177 / 0.680424 (-0.536247) | 0.017045 / 0.534201 (-0.517156) | 0.337468 / 0.579283 (-0.241815) | 0.374476 / 0.434364 (-0.059888) | 0.393428 / 0.540337 (-0.146910) | 0.535335 / 1.386936 (-0.851601) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006208 / 0.011353 (-0.005145) | 0.003650 / 0.011008 (-0.007359) | 0.062843 / 0.038508 (0.024335) | 0.062272 / 0.023109 (0.039162) | 0.446336 / 0.275898 (0.170438) | 0.477476 / 0.323480 (0.153996) | 0.004862 / 0.007986 (-0.003124) | 0.002822 / 0.004328 (-0.001506) | 0.063427 / 0.004250 (0.059177) | 0.049023 / 0.037052 (0.011971) | 0.453633 / 0.258489 (0.195144) | 0.486494 / 0.293841 (0.192653) | 0.028634 / 0.128546 (-0.099912) | 0.008187 / 0.075646 (-0.067460) | 0.068846 / 0.419271 (-0.350425) | 0.041104 / 0.043533 (-0.002429) | 0.446646 / 0.255139 (0.191507) | 0.468860 / 0.283200 (0.185660) | 0.020980 / 0.141683 (-0.120703) | 1.455565 / 1.452155 (0.003410) | 1.511142 / 1.492716 (0.018426) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.224242 / 0.018006 (0.206236) | 0.408483 / 0.000490 (0.407993) | 0.003495 / 0.000200 (0.003296) | 0.000076 / 0.000054 (0.000022) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027286 / 0.037411 (-0.010125) | 0.081151 / 0.014526 (0.066625) | 0.096598 / 0.176557 (-0.079959) | 0.146193 / 0.737135 (-0.590942) | 0.092213 / 0.296338 (-0.204125) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.463837 / 0.215209 (0.248628) | 4.636820 / 2.077655 (2.559165) | 2.576100 / 1.504120 (1.071980) | 2.396974 / 1.541195 (0.855779) | 2.461526 / 1.468490 (0.993036) | 0.502360 / 4.584777 (-4.082417) | 3.099973 / 3.745712 (-0.645739) | 2.937260 / 5.269862 (-2.332602) | 1.871274 / 4.565676 (-2.694402) | 0.057913 / 0.424275 (-0.366362) | 0.006511 / 0.007607 (-0.001096) | 0.536917 / 0.226044 (0.310873) | 5.396966 / 2.268929 (3.128038) | 3.015646 / 55.444624 (-52.428978) | 2.673793 / 6.876477 (-4.202684) | 2.712376 / 2.142072 (0.570304) | 0.591632 / 4.805227 (-4.213595) | 0.124872 / 6.500664 (-6.375792) | 0.061820 / 0.075469 (-0.013649) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.356828 / 1.841788 (-0.484960) | 18.076995 / 8.074308 (10.002687) | 15.116482 / 10.191392 (4.925090) | 0.151375 / 0.680424 (-0.529049) | 0.017867 / 0.534201 (-0.516334) | 0.335012 / 0.579283 (-0.244271) | 0.384137 / 0.434364 (-0.050226) | 0.397792 / 0.540337 (-0.142546) | 0.551521 / 1.386936 (-0.835415) |

![](https://cml.dev/watermark.png#46a0506765d0f92916ed5c37eb19e5fa1a77736a "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009418 / 0.011353 (-0.001935) | 0.005186 / 0.011008 (-0.005822) | 0.112270 / 0.038508 (0.073761) | 0.114856 / 0.023109 (0.091747) | 0.402267 / 0.275898 (0.126369) | 0.445213 / 0.323480 (0.121733) | 0.005588 / 0.007986 (-0.002398) | 0.004315 / 0.004328 (-0.000013) | 0.083561 / 0.004250 (0.079311) | 0.087319 / 0.037052 (0.050267) | 0.400989 / 0.258489 (0.142500) | 0.455636 / 0.293841 (0.161795) | 0.045168 / 0.128546 (-0.083378) | 0.010939 / 0.075646 (-0.064707) | 0.400120 / 0.419271 (-0.019151) | 0.071599 / 0.043533 (0.028066) | 0.418112 / 0.255139 (0.162973) | 0.443889 / 0.283200 (0.160690) | 0.032433 / 0.141683 (-0.109250) | 1.886313 / 1.452155 (0.434159) | 2.012909 / 1.492716 (0.520193) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.306991 / 0.018006 (0.288985) | 0.590426 / 0.000490 (0.589937) | 0.011811 / 0.000200 (0.011611) | 0.000596 / 0.000054 (0.000542) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.042520 / 0.037411 (0.005108) | 0.129808 / 0.014526 (0.115283) | 0.125481 / 0.176557 (-0.051075) | 0.199181 / 0.737135 (-0.537954) | 0.130426 / 0.296338 (-0.165913) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.526455 / 0.215209 (0.311246) | 5.213304 / 2.077655 (3.135649) | 2.643406 / 1.504120 (1.139286) | 2.611214 / 1.541195 (1.070019) | 2.586730 / 1.468490 (1.118240) | 0.639103 / 4.584777 (-3.945674) | 5.197421 / 3.745712 (1.451709) | 4.634642 / 5.269862 (-0.635220) | 2.741079 / 4.565676 (-1.824598) | 0.073064 / 0.424275 (-0.351211) | 0.009441 / 0.007607 (0.001834) | 0.635984 / 0.226044 (0.409940) | 6.283268 / 2.268929 (4.014339) | 3.337205 / 55.444624 (-52.107419) | 3.192362 / 6.876477 (-3.684114) | 2.910367 / 2.142072 (0.768294) | 0.767937 / 4.805227 (-4.037290) | 0.177467 / 6.500664 (-6.323198) | 0.081162 / 0.075469 (0.005693) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.803717 / 1.841788 (-0.038071) | 26.823235 / 8.074308 (18.748927) | 19.714471 / 10.191392 (9.523079) | 0.204048 / 0.680424 (-0.476376) | 0.025992 / 0.534201 (-0.508209) | 0.521438 / 0.579283 (-0.057845) | 0.596524 / 0.434364 (0.162160) | 0.600763 / 0.540337 (0.060425) | 0.945971 / 1.386936 (-0.440965) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009126 / 0.011353 (-0.002226) | 0.005109 / 0.011008 (-0.005899) | 0.083046 / 0.038508 (0.044538) | 0.115930 / 0.023109 (0.092821) | 0.534311 / 0.275898 (0.258413) | 0.552846 / 0.323480 (0.229366) | 0.007240 / 0.007986 (-0.000746) | 0.004617 / 0.004328 (0.000289) | 0.083927 / 0.004250 (0.079676) | 0.075926 / 0.037052 (0.038873) | 0.534750 / 0.258489 (0.276261) | 0.575122 / 0.293841 (0.281281) | 0.041001 / 0.128546 (-0.087545) | 0.010851 / 0.075646 (-0.064795) | 0.096574 / 0.419271 (-0.322697) | 0.063533 / 0.043533 (0.020001) | 0.546850 / 0.255139 (0.291711) | 0.547122 / 0.283200 (0.263922) | 0.032437 / 0.141683 (-0.109245) | 1.926191 / 1.452155 (0.474036) | 2.029841 / 1.492716 (0.537125) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.275582 / 0.018006 (0.257576) | 0.574212 / 0.000490 (0.573722) | 0.006863 / 0.000200 (0.006663) | 0.000236 / 0.000054 (0.000181) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.045340 / 0.037411 (0.007928) | 0.129196 / 0.014526 (0.114670) | 0.136637 / 0.176557 (-0.039920) | 0.200040 / 0.737135 (-0.537096) | 0.136328 / 0.296338 (-0.160011) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.612379 / 0.215209 (0.397170) | 5.874664 / 2.077655 (3.797010) | 3.070626 / 1.504120 (1.566506) | 2.999319 / 1.541195 (1.458124) | 3.000571 / 1.468490 (1.532081) | 0.732119 / 4.584777 (-3.852658) | 5.193226 / 3.745712 (1.447514) | 4.714571 / 5.269862 (-0.555291) | 2.870438 / 4.565676 (-1.695239) | 0.075793 / 0.424275 (-0.348482) | 0.009238 / 0.007607 (0.001631) | 0.695192 / 0.226044 (0.469148) | 6.897996 / 2.268929 (4.629067) | 3.923474 / 55.444624 (-51.521150) | 3.458326 / 6.876477 (-3.418151) | 3.331652 / 2.142072 (1.189579) | 0.821132 / 4.805227 (-3.984095) | 0.182252 / 6.500664 (-6.318412) | 0.084730 / 0.075469 (0.009260) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.919861 / 1.841788 (0.078073) | 27.437228 / 8.074308 (19.362920) | 21.109899 / 10.191392 (10.918507) | 0.245998 / 0.680424 (-0.434426) | 0.025817 / 0.534201 (-0.508384) | 0.517757 / 0.579283 (-0.061526) | 0.576375 / 0.434364 (0.142011) | 0.625283 / 0.540337 (0.084945) | 0.956877 / 1.386936 (-0.430059) |

![](https://cml.dev/watermark.png#8ddee15a8650a0ea52073477036d8c973da50f11 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008099 / 0.011353 (-0.003254) | 0.004815 / 0.011008 (-0.006194) | 0.099657 / 0.038508 (0.061149) | 0.064737 / 0.023109 (0.041628) | 0.461773 / 0.275898 (0.185875) | 0.444810 / 0.323480 (0.121330) | 0.004247 / 0.007986 (-0.003739) | 0.004956 / 0.004328 (0.000628) | 0.068664 / 0.004250 (0.064414) | 0.052039 / 0.037052 (0.014986) | 0.406750 / 0.258489 (0.148261) | 0.452832 / 0.293841 (0.158991) | 0.044518 / 0.128546 (-0.084028) | 0.013220 / 0.075646 (-0.062426) | 0.317713 / 0.419271 (-0.101558) | 0.061897 / 0.043533 (0.018364) | 0.398664 / 0.255139 (0.143525) | 0.531494 / 0.283200 (0.248294) | 0.064033 / 0.141683 (-0.077650) | 1.590385 / 1.452155 (0.138231) | 1.769918 / 1.492716 (0.277202) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.230795 / 0.018006 (0.212789) | 0.568797 / 0.000490 (0.568308) | 0.013498 / 0.000200 (0.013298) | 0.000448 / 0.000054 (0.000393) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028394 / 0.037411 (-0.009017) | 0.081973 / 0.014526 (0.067447) | 0.097623 / 0.176557 (-0.078934) | 0.158691 / 0.737135 (-0.578445) | 0.101548 / 0.296338 (-0.194791) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.574459 / 0.215209 (0.359249) | 5.709871 / 2.077655 (3.632217) | 2.521460 / 1.504120 (1.017340) | 2.239463 / 1.541195 (0.698268) | 2.195067 / 1.468490 (0.726577) | 0.792390 / 4.584777 (-3.792387) | 4.841665 / 3.745712 (1.095952) | 4.201620 / 5.269862 (-1.068241) | 2.664081 / 4.565676 (-1.901595) | 0.097661 / 0.424275 (-0.326614) | 0.008428 / 0.007607 (0.000821) | 0.698729 / 0.226044 (0.472684) | 6.908867 / 2.268929 (4.639939) | 3.247480 / 55.444624 (-52.197145) | 2.563921 / 6.876477 (-4.312556) | 2.738249 / 2.142072 (0.596177) | 0.972066 / 4.805227 (-3.833161) | 0.191196 / 6.500664 (-6.309468) | 0.064732 / 0.075469 (-0.010737) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.421910 / 1.841788 (-0.419877) | 20.633538 / 8.074308 (12.559230) | 18.054562 / 10.191392 (7.863170) | 0.194125 / 0.680424 (-0.486299) | 0.028097 / 0.534201 (-0.506104) | 0.417857 / 0.579283 (-0.161426) | 0.518758 / 0.434364 (0.084394) | 0.500199 / 0.540337 (-0.040138) | 0.754662 / 1.386936 (-0.632274) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008452 / 0.011353 (-0.002901) | 0.004646 / 0.011008 (-0.006362) | 0.077286 / 0.038508 (0.038778) | 0.072507 / 0.023109 (0.049398) | 0.439580 / 0.275898 (0.163682) | 0.506166 / 0.323480 (0.182686) | 0.006035 / 0.007986 (-0.001950) | 0.003886 / 0.004328 (-0.000442) | 0.075091 / 0.004250 (0.070841) | 0.063163 / 0.037052 (0.026110) | 0.468550 / 0.258489 (0.210061) | 0.523273 / 0.293841 (0.229432) | 0.048728 / 0.128546 (-0.079818) | 0.012991 / 0.075646 (-0.062655) | 0.087964 / 0.419271 (-0.331308) | 0.058920 / 0.043533 (0.015387) | 0.451247 / 0.255139 (0.196108) | 0.489827 / 0.283200 (0.206628) | 0.031164 / 0.141683 (-0.110519) | 1.675504 / 1.452155 (0.223349) | 1.806098 / 1.492716 (0.313382) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.253567 / 0.018006 (0.235561) | 0.508971 / 0.000490 (0.508481) | 0.010882 / 0.000200 (0.010682) | 0.000111 / 0.000054 (0.000057) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029490 / 0.037411 (-0.007921) | 0.090255 / 0.014526 (0.075729) | 0.110075 / 0.176557 (-0.066482) | 0.159375 / 0.737135 (-0.577760) | 0.109313 / 0.296338 (-0.187025) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.580252 / 0.215209 (0.365043) | 5.911741 / 2.077655 (3.834086) | 2.659405 / 1.504120 (1.155285) | 2.344943 / 1.541195 (0.803749) | 2.390748 / 1.468490 (0.922258) | 0.827823 / 4.584777 (-3.756954) | 4.973544 / 3.745712 (1.227832) | 4.300220 / 5.269862 (-0.969642) | 2.826181 / 4.565676 (-1.739495) | 0.101013 / 0.424275 (-0.323263) | 0.008025 / 0.007607 (0.000418) | 0.728414 / 0.226044 (0.502369) | 7.508045 / 2.268929 (5.239117) | 3.687627 / 55.444624 (-51.756997) | 2.902953 / 6.876477 (-3.973524) | 3.094624 / 2.142072 (0.952551) | 1.054696 / 4.805227 (-3.750531) | 0.212297 / 6.500664 (-6.288367) | 0.070211 / 0.075469 (-0.005258) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.567117 / 1.841788 (-0.274670) | 21.420746 / 8.074308 (13.346438) | 19.857467 / 10.191392 (9.666075) | 0.228554 / 0.680424 (-0.451870) | 0.032278 / 0.534201 (-0.501923) | 0.459966 / 0.579283 (-0.119317) | 0.541219 / 0.434364 (0.106855) | 0.549599 / 0.540337 (0.009261) | 0.731476 / 1.386936 (-0.655460) |

![](https://cml.dev/watermark.png#0cc77d7f45c73698c31eab4f8cfff901044d0020 "CML watermark")

... to avoid an `ImportError` raised in `BeamBasedBuilder._save_info` when `apache_beam` is not installed (e.g., when downloading the processed version of a dataset from the HF GCS)

Fix https://github.com/huggingface/datasets/issues/6260


Temporarily pin tensorflow < 2.14.0

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008356 / 0.011353 (-0.002997) | 0.004553 / 0.011008 (-0.006455) | 0.101025 / 0.038508 (0.062517) | 0.090194 / 0.023109 (0.067085) | 0.427127 / 0.275898 (0.151229) | 0.469116 / 0.323480 (0.145636) | 0.007593 / 0.007986 (-0.000393) | 0.003751 / 0.004328 (-0.000578) | 0.077432 / 0.004250 (0.073182) | 0.082744 / 0.037052 (0.045692) | 0.433638 / 0.258489 (0.175149) | 0.482387 / 0.293841 (0.188546) | 0.040658 / 0.128546 (-0.087888) | 0.009799 / 0.075646 (-0.065848) | 0.345274 / 0.419271 (-0.073998) | 0.076642 / 0.043533 (0.033109) | 0.424417 / 0.255139 (0.169278) | 0.457045 / 0.283200 (0.173846) | 0.033642 / 0.141683 (-0.108041) | 1.765446 / 1.452155 (0.313291) | 1.859279 / 1.492716 (0.366562) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.273629 / 0.018006 (0.255623) | 0.505743 / 0.000490 (0.505253) | 0.009300 / 0.000200 (0.009100) | 0.000359 / 0.000054 (0.000305) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032510 / 0.037411 (-0.004901) | 0.099628 / 0.014526 (0.085103) | 0.112904 / 0.176557 (-0.063652) | 0.179118 / 0.737135 (-0.558018) | 0.115946 / 0.296338 (-0.180393) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.456431 / 0.215209 (0.241222) | 4.556559 / 2.077655 (2.478904) | 2.207893 / 1.504120 (0.703773) | 2.024706 / 1.541195 (0.483512) | 2.165424 / 1.468490 (0.696934) | 0.571745 / 4.584777 (-4.013031) | 4.341017 / 3.745712 (0.595305) | 3.980520 / 5.269862 (-1.289342) | 2.333077 / 4.565676 (-2.232599) | 0.067200 / 0.424275 (-0.357075) | 0.008563 / 0.007607 (0.000956) | 0.545294 / 0.226044 (0.319250) | 5.445152 / 2.268929 (3.176224) | 2.740657 / 55.444624 (-52.703968) | 2.370635 / 6.876477 (-4.505842) | 2.451642 / 2.142072 (0.309570) | 0.679385 / 4.805227 (-4.125842) | 0.155967 / 6.500664 (-6.344697) | 0.072812 / 0.075469 (-0.002657) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.494483 / 1.841788 (-0.347305) | 23.673700 / 8.074308 (15.599392) | 16.608529 / 10.191392 (6.417137) | 0.170220 / 0.680424 (-0.510204) | 0.021630 / 0.534201 (-0.512571) | 0.470771 / 0.579283 (-0.108512) | 0.535874 / 0.434364 (0.101510) | 0.550376 / 0.540337 (0.010039) | 0.776633 / 1.386936 (-0.610303) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007899 / 0.011353 (-0.003454) | 0.004581 / 0.011008 (-0.006427) | 0.076520 / 0.038508 (0.038012) | 0.090374 / 0.023109 (0.067265) | 0.495016 / 0.275898 (0.219118) | 0.532384 / 0.323480 (0.208904) | 0.006160 / 0.007986 (-0.001825) | 0.003780 / 0.004328 (-0.000548) | 0.077164 / 0.004250 (0.072914) | 0.064444 / 0.037052 (0.027391) | 0.501642 / 0.258489 (0.243153) | 0.549170 / 0.293841 (0.255329) | 0.038051 / 0.128546 (-0.090495) | 0.010081 / 0.075646 (-0.065565) | 0.083752 / 0.419271 (-0.335520) | 0.061334 / 0.043533 (0.017801) | 0.493502 / 0.255139 (0.238363) | 0.518018 / 0.283200 (0.234818) | 0.029534 / 0.141683 (-0.112149) | 1.929432 / 1.452155 (0.477277) | 1.889985 / 1.492716 (0.397268) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.254802 / 0.018006 (0.236795) | 0.494463 / 0.000490 (0.493974) | 0.005040 / 0.000200 (0.004840) | 0.000120 / 0.000054 (0.000065) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.038372 / 0.037411 (0.000960) | 0.112247 / 0.014526 (0.097721) | 0.124365 / 0.176557 (-0.052191) | 0.187142 / 0.737135 (-0.549993) | 0.126070 / 0.296338 (-0.170269) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.513418 / 0.215209 (0.298209) | 5.132267 / 2.077655 (3.054613) | 2.773676 / 1.504120 (1.269556) | 2.576840 / 1.541195 (1.035645) | 2.681729 / 1.468490 (1.213238) | 0.581809 / 4.584777 (-4.002968) | 4.327075 / 3.745712 (0.581363) | 4.040264 / 5.269862 (-1.229598) | 2.436192 / 4.565676 (-2.129484) | 0.067819 / 0.424275 (-0.356456) | 0.008760 / 0.007607 (0.001153) | 0.610765 / 0.226044 (0.384720) | 6.105679 / 2.268929 (3.836750) | 3.341341 / 55.444624 (-52.103284) | 2.926695 / 6.876477 (-3.949781) | 3.017269 / 2.142072 (0.875196) | 0.707289 / 4.805227 (-4.097938) | 0.157379 / 6.500664 (-6.343285) | 0.072549 / 0.075469 (-0.002920) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.666738 / 1.841788 (-0.175050) | 23.698567 / 8.074308 (15.624259) | 17.806437 / 10.191392 (7.615045) | 0.172103 / 0.680424 (-0.508321) | 0.023508 / 0.534201 (-0.510693) | 0.473171 / 0.579283 (-0.106112) | 0.524834 / 0.434364 (0.090470) | 0.562562 / 0.540337 (0.022224) | 0.788667 / 1.386936 (-0.598269) |

![](https://cml.dev/watermark.png#1e7338259b26b32a095d251d5cdbc779c3573307 "CML watermark")

CI 404 errors are unrelated. See:
- #6262

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006657 / 0.011353 (-0.004696) | 0.003975 / 0.011008 (-0.007033) | 0.084614 / 0.038508 (0.046106) | 0.074557 / 0.023109 (0.051448) | 0.309213 / 0.275898 (0.033315) | 0.338245 / 0.323480 (0.014765) | 0.005375 / 0.007986 (-0.002610) | 0.003355 / 0.004328 (-0.000973) | 0.064406 / 0.004250 (0.060156) | 0.061763 / 0.037052 (0.024711) | 0.313405 / 0.258489 (0.054916) | 0.352149 / 0.293841 (0.058308) | 0.031597 / 0.128546 (-0.096949) | 0.008499 / 0.075646 (-0.067147) | 0.289098 / 0.419271 (-0.130174) | 0.054415 / 0.043533 (0.010882) | 0.313210 / 0.255139 (0.058071) | 0.326728 / 0.283200 (0.043528) | 0.024597 / 0.141683 (-0.117086) | 1.449916 / 1.452155 (-0.002239) | 1.526314 / 1.492716 (0.033598) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.231435 / 0.018006 (0.213429) | 0.537224 / 0.000490 (0.536734) | 0.007287 / 0.000200 (0.007088) | 0.000227 / 0.000054 (0.000172) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028340 / 0.037411 (-0.009071) | 0.084085 / 0.014526 (0.069560) | 0.428211 / 0.176557 (0.251655) | 0.157360 / 0.737135 (-0.579775) | 0.139470 / 0.296338 (-0.156868) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.389311 / 0.215209 (0.174102) | 3.871329 / 2.077655 (1.793674) | 1.861533 / 1.504120 (0.357413) | 1.688082 / 1.541195 (0.146887) | 1.804036 / 1.468490 (0.335546) | 0.489154 / 4.584777 (-4.095623) | 3.603843 / 3.745712 (-0.141869) | 3.424868 / 5.269862 (-1.844994) | 2.013525 / 4.565676 (-2.552152) | 0.057387 / 0.424275 (-0.366888) | 0.007274 / 0.007607 (-0.000333) | 0.462340 / 0.226044 (0.236295) | 4.620095 / 2.268929 (2.351167) | 2.326641 / 55.444624 (-53.117984) | 1.990082 / 6.876477 (-4.886395) | 2.037841 / 2.142072 (-0.104232) | 0.581973 / 4.805227 (-4.223254) | 0.135932 / 6.500664 (-6.364732) | 0.061092 / 0.075469 (-0.014377) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.249586 / 1.841788 (-0.592202) | 19.036233 / 8.074308 (10.961925) | 14.083365 / 10.191392 (3.891973) | 0.169802 / 0.680424 (-0.510622) | 0.018547 / 0.534201 (-0.515654) | 0.392926 / 0.579283 (-0.186357) | 0.409993 / 0.434364 (-0.024371) | 0.460081 / 0.540337 (-0.080257) | 0.643836 / 1.386936 (-0.743100) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006889 / 0.011353 (-0.004464) | 0.004060 / 0.011008 (-0.006948) | 0.064332 / 0.038508 (0.025824) | 0.077067 / 0.023109 (0.053958) | 0.401235 / 0.275898 (0.125337) | 0.437139 / 0.323480 (0.113659) | 0.005510 / 0.007986 (-0.002476) | 0.003338 / 0.004328 (-0.000991) | 0.064446 / 0.004250 (0.060195) | 0.055537 / 0.037052 (0.018485) | 0.432689 / 0.258489 (0.174200) | 0.460005 / 0.293841 (0.166164) | 0.033122 / 0.128546 (-0.095424) | 0.008637 / 0.075646 (-0.067010) | 0.071088 / 0.419271 (-0.348183) | 0.049024 / 0.043533 (0.005491) | 0.400258 / 0.255139 (0.145119) | 0.419324 / 0.283200 (0.136124) | 0.022050 / 0.141683 (-0.119632) | 1.475744 / 1.452155 (0.023589) | 1.546565 / 1.492716 (0.053848) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.226241 / 0.018006 (0.208235) | 0.448574 / 0.000490 (0.448085) | 0.004732 / 0.000200 (0.004533) | 0.000097 / 0.000054 (0.000042) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033260 / 0.037411 (-0.004151) | 0.092622 / 0.014526 (0.078096) | 0.105501 / 0.176557 (-0.071056) | 0.157981 / 0.737135 (-0.579155) | 0.105993 / 0.296338 (-0.190345) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.445716 / 0.215209 (0.230507) | 4.451848 / 2.077655 (2.374194) | 2.404769 / 1.504120 (0.900649) | 2.232594 / 1.541195 (0.691399) | 2.312735 / 1.468490 (0.844245) | 0.491208 / 4.584777 (-4.093569) | 3.561629 / 3.745712 (-0.184083) | 3.444269 / 5.269862 (-1.825592) | 2.060365 / 4.565676 (-2.505311) | 0.057723 / 0.424275 (-0.366552) | 0.007392 / 0.007607 (-0.000215) | 0.526447 / 0.226044 (0.300403) | 5.264307 / 2.268929 (2.995379) | 2.951481 / 55.444624 (-52.493143) | 2.593178 / 6.876477 (-4.283299) | 2.689780 / 2.142072 (0.547707) | 0.588649 / 4.805227 (-4.216579) | 0.133566 / 6.500664 (-6.367098) | 0.060462 / 0.075469 (-0.015008) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.381008 / 1.841788 (-0.460780) | 19.452394 / 8.074308 (11.378086) | 15.255912 / 10.191392 (5.064520) | 0.171043 / 0.680424 (-0.509381) | 0.020395 / 0.534201 (-0.513806) | 0.396429 / 0.579283 (-0.182854) | 0.422820 / 0.434364 (-0.011544) | 0.477305 / 0.540337 (-0.063032) | 0.658274 / 1.386936 (-0.728663) |

![](https://cml.dev/watermark.png#faedc670ca896584d0f8edcb1fd9c13d1d6cc903 "CML watermark")

Temporarily pin tensorflow < 2.14.0 until permanent solution is found.

Hot fix #6263.


CI is broken: ImportError: cannot import name 'context' from 'tensorflow.python'

bug

Something isn't working

Python 3.10 CI is broken for `test_py310`.

See: https://github.com/huggingface/datasets/actions/runs/6322990957/job/17169678812?pr=6262

```
FAILED tests/test_py_utils.py::TempSeedTest::test_tensorflow - ImportError: cannot import name 'context' from 'tensorflow.python' (/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/tensorflow/python/__init__.py)
```

```
_________________________ TempSeedTest.test_tensorflow _________________________
[gw1] linux -- Python 3.10.13 /opt/hostedtoolcache/Python/3.10.13/x64/bin/python

self = <tests.test_py_utils.TempSeedTest testMethod=test_tensorflow>

    @require_tf
    def test_tensorflow(self):
        import tensorflow as tf
        from tensorflow.keras import layers

        model = layers.Dense(2)

        def gen_random_output():
            x = tf.random.uniform((1, 3))
            return model(x).numpy()

>       with temp_seed(42, set_tensorflow=True):

tests/test_py_utils.py:155:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/contextlib.py:135: in __enter__
    return next(self.gen)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

seed = 42, set_pytorch = False, set_tensorflow = True

    @contextmanager
    def temp_seed(seed: int, set_pytorch=False, set_tensorflow=False):
        """Temporarily set the random seed. This works for python numpy, pytorch and tensorflow."""
        np_state = np.random.get_state()
        np.random.seed(seed)

        if set_pytorch and config.TORCH_AVAILABLE:
            import torch

            torch_state = torch.random.get_rng_state()
            torch.random.manual_seed(seed)

            if torch.cuda.is_available():
                torch_cuda_states = torch.cuda.get_rng_state_all()
                torch.cuda.manual_seed_all(seed)

        if set_tensorflow and config.TF_AVAILABLE:
            import tensorflow as tf
>           from tensorflow.python import context as tfpycontext
E           ImportError: cannot import name 'context' from 'tensorflow.python' (/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/tensorflow/python/__init__.py)

/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/datasets/utils/py_utils.py:257: ImportError
```