Skip dataset verifications by default

100% agree that the checksum verification is overkill and not super useful. But I think this PR would also disable the check on num_examples no ?

As a user I would like to know if the dataset I'm loading changed significantly.
And I also think it can be useful to make sure the metadata are up to date.

What do you think ?

We could have a default `ignore_verifications="ignore_checksums"`

> We could have a default `ignore_verifications="ignore_checksums"`

Accepting multiple types (booleans and strings) at the same time is not the best design. Maybe we could define an enum for this parameter?

so we can have three verification levels, - smth like "ignore_all" (to skip both checksums and all other info like num_examples verification), "ignore_checksums" (to skip only checksums verification), and "verify_all" (to perform all verification)?
and deprecate `ignore_verifications` param.

@mariosasko if you're not going to work on this PR in the coming days, I can take over it if you want (this PR will help me with [this issue](https://github.com/huggingface/datasets/issues/5315), not super urgent though).

Okay, I propose deprecating `ignore_verifications` in favor of `verification_mode` (`load_dataset` already has `download_mode`; some other projects use this name for verification control). `verification_mode` would accept the following enum (or strings in the same manner as `download_mode` does):

```python
class VerificationMode(enum.Enum):
    FULL = "full"           # runs all verification checks
    BASIC = "basic"     # default, runs only the cheap ones (skips the checksum check)
    NONE = "none"      # skips all the checks
```

WDTY?

(copy paste from my message on slack)

What do you think of a config variable in config.py to switch from one verification mode to another ? This way we don’t deprecate anything

Many users are familiar with ignore_verifications=True, it might be overkill to deprecate it

@lhoestq So we have "basic" verification mode in `config.py` and continue to have `False` as a default
value for `ignore_verifications`? That way running all verifications including checksums would not be possible without switching the config var, right?

I like having a `VerificationMode` enum because it's aligned with `DownloadMode` and sounds more natural to me (`ignore_verifications` feels a bit semantically reverted but this is probably just my feeling) and it's flexible (no need to worry about `config.py`, I'm not sure that users even know it exists, wdyt?).

The usage point seems also valid to me, but cases when users are stuck with NonMatchingX errors also happen from time to time and to figure out what's wrong is non-trivial here.

As a note aside - I suggest to add instructions to the NonMatchingX error message (how to use `ignore_verifications` / `verification_mode`), this would save users who don't know about this param a lot of time.

Ok I see. I'm fine with the new parameter then (even though I had a small pref for the config variable) :)

I like the idea of an enum and the `verification_mode` parameter.

In relation with the config parameter, we could additionally add a `DEFAULT_VERIFICATION_MODE`, maybe only if users require it. Note that until now there wasn't any config parameter for a default `ignore_verifications` value: I guess people are explicitly passing `ignore_verifications=True`...

As a note aside, I like the suggestion by @polinaeterna: we could give actionable messages when verifying checksums. This could be done in other PR.

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.012891 / 0.011353 (0.001538) | 0.006474 / 0.011008 (-0.004535) | 0.144038 / 0.038508 (0.105530) | 0.036151 / 0.023109 (0.013042) | 0.404366 / 0.275898 (0.128468) | 0.479988 / 0.323480 (0.156508) | 0.010219 / 0.007986 (0.002233) | 0.005319 / 0.004328 (0.000990) | 0.099705 / 0.004250 (0.095455) | 0.046639 / 0.037052 (0.009586) | 0.398997 / 0.258489 (0.140508) | 0.478431 / 0.293841 (0.184590) | 0.069125 / 0.128546 (-0.059421) | 0.019603 / 0.075646 (-0.056043) | 0.400829 / 0.419271 (-0.018443) | 0.066549 / 0.043533 (0.023016) | 0.398343 / 0.255139 (0.143204) | 0.417928 / 0.283200 (0.134728) | 0.121124 / 0.141683 (-0.020559) | 1.751513 / 1.452155 (0.299358) | 1.821239 / 1.492716 (0.328523) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.251603 / 0.018006 (0.233597) | 0.579916 / 0.000490 (0.579427) | 0.003257 / 0.000200 (0.003058) | 0.000109 / 0.000054 (0.000054) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031502 / 0.037411 (-0.005909) | 0.134688 / 0.014526 (0.120162) | 0.152306 / 0.176557 (-0.024251) | 0.198943 / 0.737135 (-0.538192) | 0.142551 / 0.296338 (-0.153788) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.634672 / 0.215209 (0.419463) | 6.370215 / 2.077655 (4.292561) | 2.548123 / 1.504120 (1.044003) | 2.184263 / 1.541195 (0.643069) | 2.239026 / 1.468490 (0.770536) | 1.233340 / 4.584777 (-3.351437) | 5.791824 / 3.745712 (2.046112) | 5.093032 / 5.269862 (-0.176830) | 2.849833 / 4.565676 (-1.715844) | 0.143787 / 0.424275 (-0.280488) | 0.015279 / 0.007607 (0.007672) | 0.757984 / 0.226044 (0.531939) | 7.883604 / 2.268929 (5.614675) | 3.321591 / 55.444624 (-52.123033) | 2.671777 / 6.876477 (-4.204700) | 2.685215 / 2.142072 (0.543142) | 1.546709 / 4.805227 (-3.258519) | 0.247186 / 6.500664 (-6.253478) | 0.085117 / 0.075469 (0.009648) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.679809 / 1.841788 (-0.161979) | 18.528893 / 8.074308 (10.454585) | 23.168590 / 10.191392 (12.977198) | 0.277618 / 0.680424 (-0.402806) | 0.045109 / 0.534201 (-0.489092) | 0.568873 / 0.579283 (-0.010410) | 0.695017 / 0.434364 (0.260653) | 0.671024 / 0.540337 (0.130687) | 0.823817 / 1.386936 (-0.563119) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009809 / 0.011353 (-0.001544) | 0.006890 / 0.011008 (-0.004118) | 0.099211 / 0.038508 (0.060703) | 0.035387 / 0.023109 (0.012278) | 0.507603 / 0.275898 (0.231705) | 0.535553 / 0.323480 (0.212073) | 0.007346 / 0.007986 (-0.000640) | 0.007559 / 0.004328 (0.003231) | 0.099132 / 0.004250 (0.094882) | 0.048048 / 0.037052 (0.010996) | 0.518096 / 0.258489 (0.259607) | 0.561134 / 0.293841 (0.267294) | 0.057580 / 0.128546 (-0.070966) | 0.023665 / 0.075646 (-0.051982) | 0.138409 / 0.419271 (-0.280862) | 0.061989 / 0.043533 (0.018456) | 0.510568 / 0.255139 (0.255429) | 0.552722 / 0.283200 (0.269522) | 0.115990 / 0.141683 (-0.025693) | 1.884900 / 1.452155 (0.432745) | 1.990604 / 1.492716 (0.497888) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.280638 / 0.018006 (0.262632) | 0.592837 / 0.000490 (0.592347) | 0.000465 / 0.000200 (0.000265) | 0.000078 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030253 / 0.037411 (-0.007158) | 0.141580 / 0.014526 (0.127054) | 0.135114 / 0.176557 (-0.041443) | 0.190003 / 0.737135 (-0.547133) | 0.160230 / 0.296338 (-0.136109) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.699762 / 0.215209 (0.484553) | 6.632344 / 2.077655 (4.554689) | 2.718803 / 1.504120 (1.214683) | 2.485294 / 1.541195 (0.944099) | 2.579889 / 1.468490 (1.111399) | 1.268795 / 4.584777 (-3.315982) | 5.777745 / 3.745712 (2.032033) | 3.232551 / 5.269862 (-2.037311) | 2.127699 / 4.565676 (-2.437977) | 0.146570 / 0.424275 (-0.277705) | 0.015971 / 0.007607 (0.008364) | 0.803181 / 0.226044 (0.577137) | 8.377192 / 2.268929 (6.108264) | 3.551242 / 55.444624 (-51.893382) | 2.865228 / 6.876477 (-4.011249) | 2.774869 / 2.142072 (0.632797) | 1.553856 / 4.805227 (-3.251371) | 0.264510 / 6.500664 (-6.236154) | 0.087918 / 0.075469 (0.012449) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.653396 / 1.841788 (-0.188391) | 18.703863 / 8.074308 (10.629555) | 22.067331 / 10.191392 (11.875939) | 0.257424 / 0.680424 (-0.422999) | 0.026448 / 0.534201 (-0.507753) | 0.550100 / 0.579283 (-0.029183) | 0.647296 / 0.434364 (0.212932) | 0.657476 / 0.540337 (0.117138) | 0.781119 / 1.386936 (-0.605817) |

![](https://cml.dev/watermark.png#8c4a9cb95f8742a2850f11d59abbef71d6c1f60c "CML watermark")

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008889 / 0.011353 (-0.002464) | 0.004563 / 0.011008 (-0.006445) | 0.101627 / 0.038508 (0.063118) | 0.030526 / 0.023109 (0.007417) | 0.297175 / 0.275898 (0.021277) | 0.368454 / 0.323480 (0.044974) | 0.007246 / 0.007986 (-0.000740) | 0.003565 / 0.004328 (-0.000763) | 0.078644 / 0.004250 (0.074394) | 0.038616 / 0.037052 (0.001564) | 0.310521 / 0.258489 (0.052032) | 0.348014 / 0.293841 (0.054173) | 0.033463 / 0.128546 (-0.095083) | 0.011544 / 0.075646 (-0.064102) | 0.323281 / 0.419271 (-0.095990) | 0.040187 / 0.043533 (-0.003346) | 0.298015 / 0.255139 (0.042876) | 0.326392 / 0.283200 (0.043193) | 0.088730 / 0.141683 (-0.052952) | 1.503387 / 1.452155 (0.051233) | 1.548704 / 1.492716 (0.055988) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.185983 / 0.018006 (0.167977) | 0.451889 / 0.000490 (0.451400) | 0.001433 / 0.000200 (0.001233) | 0.000080 / 0.000054 (0.000026) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023396 / 0.037411 (-0.014015) | 0.118236 / 0.014526 (0.103710) | 0.124594 / 0.176557 (-0.051962) | 0.159089 / 0.737135 (-0.578047) | 0.129369 / 0.296338 (-0.166969) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.423161 / 0.215209 (0.207952) | 4.228211 / 2.077655 (2.150556) | 1.853862 / 1.504120 (0.349742) | 1.649471 / 1.541195 (0.108276) | 1.708631 / 1.468490 (0.240141) | 0.697456 / 4.584777 (-3.887321) | 3.473244 / 3.745712 (-0.272468) | 1.942586 / 5.269862 (-3.327275) | 1.291592 / 4.565676 (-3.274084) | 0.082758 / 0.424275 (-0.341517) | 0.012256 / 0.007607 (0.004649) | 0.528355 / 0.226044 (0.302311) | 5.277620 / 2.268929 (3.008691) | 2.299604 / 55.444624 (-53.145020) | 1.954940 / 6.876477 (-4.921537) | 2.055543 / 2.142072 (-0.086529) | 0.814723 / 4.805227 (-3.990505) | 0.149937 / 6.500664 (-6.350727) | 0.064529 / 0.075469 (-0.010941) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.266240 / 1.841788 (-0.575547) | 14.144016 / 8.074308 (6.069708) | 14.331733 / 10.191392 (4.140340) | 0.138963 / 0.680424 (-0.541461) | 0.029034 / 0.534201 (-0.505167) | 0.397325 / 0.579283 (-0.181958) | 0.405293 / 0.434364 (-0.029071) | 0.480745 / 0.540337 (-0.059592) | 0.573386 / 1.386936 (-0.813550) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007214 / 0.011353 (-0.004139) | 0.004569 / 0.011008 (-0.006439) | 0.078718 / 0.038508 (0.040209) | 0.031104 / 0.023109 (0.007995) | 0.342562 / 0.275898 (0.066664) | 0.387802 / 0.323480 (0.064322) | 0.005378 / 0.007986 (-0.002608) | 0.003414 / 0.004328 (-0.000915) | 0.077249 / 0.004250 (0.072999) | 0.044337 / 0.037052 (0.007285) | 0.341397 / 0.258489 (0.082907) | 0.385536 / 0.293841 (0.091695) | 0.033257 / 0.128546 (-0.095289) | 0.011825 / 0.075646 (-0.063821) | 0.086723 / 0.419271 (-0.332549) | 0.045951 / 0.043533 (0.002418) | 0.340914 / 0.255139 (0.085775) | 0.367126 / 0.283200 (0.083926) | 0.096326 / 0.141683 (-0.045357) | 1.608612 / 1.452155 (0.156458) | 1.687251 / 1.492716 (0.194534) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.227595 / 0.018006 (0.209589) | 0.418502 / 0.000490 (0.418013) | 0.000392 / 0.000200 (0.000192) | 0.000059 / 0.000054 (0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026232 / 0.037411 (-0.011179) | 0.101020 / 0.014526 (0.086494) | 0.110017 / 0.176557 (-0.066539) | 0.153497 / 0.737135 (-0.583639) | 0.110602 / 0.296338 (-0.185737) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.433789 / 0.215209 (0.218579) | 4.329350 / 2.077655 (2.251696) | 2.052136 / 1.504120 (0.548016) | 1.848457 / 1.541195 (0.307262) | 1.936791 / 1.468490 (0.468301) | 0.700609 / 4.584777 (-3.884168) | 3.391983 / 3.745712 (-0.353729) | 1.903220 / 5.269862 (-3.366642) | 1.179463 / 4.565676 (-3.386213) | 0.084025 / 0.424275 (-0.340250) | 0.012743 / 0.007607 (0.005136) | 0.536816 / 0.226044 (0.310772) | 5.420230 / 2.268929 (3.151302) | 2.507438 / 55.444624 (-52.937187) | 2.178907 / 6.876477 (-4.697570) | 2.228586 / 2.142072 (0.086514) | 0.812527 / 4.805227 (-3.992701) | 0.153382 / 6.500664 (-6.347282) | 0.069932 / 0.075469 (-0.005537) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.256861 / 1.841788 (-0.584927) | 14.309236 / 8.074308 (6.234928) | 13.740323 / 10.191392 (3.548931) | 0.142698 / 0.680424 (-0.537726) | 0.016998 / 0.534201 (-0.517203) | 0.385489 / 0.579283 (-0.193794) | 0.391515 / 0.434364 (-0.042849) | 0.472704 / 0.540337 (-0.067633) | 0.565042 / 1.386936 (-0.821894) |

![](https://cml.dev/watermark.png#4b0713ddf2e2e7129d9ccda791d265684c96675c "CML watermark")

This is ready for review.

If `verification_mode` is None, it defaults to `VerificationMode.BASIC` instead of `VerificationMode.NONE`, so maybe we should find a better name for the latter to avoid confusion.

PS: `ignore_verifications` is still present in the `test`/`run_beam` commands for simplicity. Let me know if you think these commands should support all three modes.

> I would also prefer to change the name for the NONE verification mode, but don't have really good ideas in mind. maybe smth like SKIP_ALL ?

I decided to go with the following names:
* `no_checks` (previously `none`)
* `basic_checks` (previously `basic`)
* `all_checks` (previously `full`)

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008900 / 0.011353 (-0.002453) | 0.004492 / 0.011008 (-0.006516) | 0.100957 / 0.038508 (0.062449) | 0.030145 / 0.023109 (0.007036) | 0.302531 / 0.275898 (0.026633) | 0.344072 / 0.323480 (0.020592) | 0.007032 / 0.007986 (-0.000953) | 0.004150 / 0.004328 (-0.000178) | 0.078272 / 0.004250 (0.074021) | 0.034142 / 0.037052 (-0.002910) | 0.310798 / 0.258489 (0.052308) | 0.350077 / 0.293841 (0.056236) | 0.034497 / 0.128546 (-0.094050) | 0.011417 / 0.075646 (-0.064230) | 0.323427 / 0.419271 (-0.095844) | 0.045664 / 0.043533 (0.002132) | 0.304688 / 0.255139 (0.049549) | 0.336591 / 0.283200 (0.053391) | 0.086116 / 0.141683 (-0.055567) | 1.519278 / 1.452155 (0.067123) | 1.576728 / 1.492716 (0.084011) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.242482 / 0.018006 (0.224476) | 0.403548 / 0.000490 (0.403058) | 0.001217 / 0.000200 (0.001017) | 0.000073 / 0.000054 (0.000018) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023466 / 0.037411 (-0.013945) | 0.095220 / 0.014526 (0.080694) | 0.104119 / 0.176557 (-0.072438) | 0.141107 / 0.737135 (-0.596029) | 0.107236 / 0.296338 (-0.189102) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.416290 / 0.215209 (0.201081) | 4.159068 / 2.077655 (2.081413) | 1.846014 / 1.504120 (0.341894) | 1.634789 / 1.541195 (0.093594) | 1.724687 / 1.468490 (0.256196) | 0.696887 / 4.584777 (-3.887890) | 3.313861 / 3.745712 (-0.431851) | 1.907239 / 5.269862 (-3.362622) | 1.266815 / 4.565676 (-3.298861) | 0.081660 / 0.424275 (-0.342615) | 0.012290 / 0.007607 (0.004683) | 0.522866 / 0.226044 (0.296822) | 5.237356 / 2.268929 (2.968428) | 2.294645 / 55.444624 (-53.149979) | 1.946407 / 6.876477 (-4.930069) | 1.995441 / 2.142072 (-0.146632) | 0.808340 / 4.805227 (-3.996887) | 0.149670 / 6.500664 (-6.350994) | 0.065162 / 0.075469 (-0.010307) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.219476 / 1.841788 (-0.622312) | 13.868709 / 8.074308 (5.794401) | 14.115783 / 10.191392 (3.924391) | 0.149403 / 0.680424 (-0.531021) | 0.028514 / 0.534201 (-0.505686) | 0.398194 / 0.579283 (-0.181089) | 0.410898 / 0.434364 (-0.023466) | 0.485763 / 0.540337 (-0.054574) | 0.574924 / 1.386936 (-0.812012) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006906 / 0.011353 (-0.004447) | 0.004446 / 0.011008 (-0.006562) | 0.075936 / 0.038508 (0.037428) | 0.027693 / 0.023109 (0.004584) | 0.339505 / 0.275898 (0.063607) | 0.383315 / 0.323480 (0.059835) | 0.005138 / 0.007986 (-0.002847) | 0.004636 / 0.004328 (0.000308) | 0.074829 / 0.004250 (0.070578) | 0.040327 / 0.037052 (0.003274) | 0.340516 / 0.258489 (0.082027) | 0.388569 / 0.293841 (0.094729) | 0.031562 / 0.128546 (-0.096984) | 0.011585 / 0.075646 (-0.064061) | 0.084753 / 0.419271 (-0.334518) | 0.041310 / 0.043533 (-0.002223) | 0.338272 / 0.255139 (0.083133) | 0.367243 / 0.283200 (0.084043) | 0.092653 / 0.141683 (-0.049029) | 1.515973 / 1.452155 (0.063818) | 1.582869 / 1.492716 (0.090152) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.229366 / 0.018006 (0.211360) | 0.414404 / 0.000490 (0.413914) | 0.002922 / 0.000200 (0.002723) | 0.000075 / 0.000054 (0.000020) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026391 / 0.037411 (-0.011020) | 0.106754 / 0.014526 (0.092228) | 0.110718 / 0.176557 (-0.065839) | 0.145786 / 0.737135 (-0.591350) | 0.113180 / 0.296338 (-0.183159) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.446340 / 0.215209 (0.231131) | 4.499756 / 2.077655 (2.422101) | 2.071485 / 1.504120 (0.567365) | 1.873223 / 1.541195 (0.332029) | 1.931562 / 1.468490 (0.463071) | 0.699270 / 4.584777 (-3.885507) | 3.452383 / 3.745712 (-0.293329) | 2.970630 / 5.269862 (-2.299232) | 1.300859 / 4.565676 (-3.264817) | 0.083971 / 0.424275 (-0.340304) | 0.012489 / 0.007607 (0.004882) | 0.544190 / 0.226044 (0.318146) | 5.460097 / 2.268929 (3.191169) | 2.700244 / 55.444624 (-52.744380) | 2.396694 / 6.876477 (-4.479783) | 2.376334 / 2.142072 (0.234262) | 0.812845 / 4.805227 (-3.992382) | 0.154441 / 6.500664 (-6.346223) | 0.069510 / 0.075469 (-0.005959) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.278836 / 1.841788 (-0.562952) | 14.153158 / 8.074308 (6.078850) | 13.821290 / 10.191392 (3.629898) | 0.160464 / 0.680424 (-0.519960) | 0.016742 / 0.534201 (-0.517459) | 0.379840 / 0.579283 (-0.199443) | 0.391903 / 0.434364 (-0.042461) | 0.461646 / 0.540337 (-0.078691) | 0.550691 / 1.386936 (-0.836245) |

![](https://cml.dev/watermark.png#aeb637daab938d51b8b15ad4d175d06817e99512 "CML watermark")

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009858 / 0.011353 (-0.001495) | 0.005383 / 0.011008 (-0.005625) | 0.100527 / 0.038508 (0.062019) | 0.037176 / 0.023109 (0.014067) | 0.295204 / 0.275898 (0.019306) | 0.364511 / 0.323480 (0.041031) | 0.008486 / 0.007986 (0.000500) | 0.004273 / 0.004328 (-0.000055) | 0.076538 / 0.004250 (0.072288) | 0.046250 / 0.037052 (0.009197) | 0.307102 / 0.258489 (0.048613) | 0.339313 / 0.293841 (0.045472) | 0.040783 / 0.128546 (-0.087763) | 0.012323 / 0.075646 (-0.063323) | 0.336216 / 0.419271 (-0.083055) | 0.050480 / 0.043533 (0.006947) | 0.293689 / 0.255139 (0.038550) | 0.315034 / 0.283200 (0.031834) | 0.113775 / 0.141683 (-0.027908) | 1.438738 / 1.452155 (-0.013416) | 1.499874 / 1.492716 (0.007157) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.202392 / 0.018006 (0.184386) | 0.442784 / 0.000490 (0.442295) | 0.003004 / 0.000200 (0.002804) | 0.000087 / 0.000054 (0.000033) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027792 / 0.037411 (-0.009620) | 0.110886 / 0.014526 (0.096360) | 0.121041 / 0.176557 (-0.055515) | 0.166803 / 0.737135 (-0.570333) | 0.127617 / 0.296338 (-0.168722) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.409762 / 0.215209 (0.194553) | 4.073297 / 2.077655 (1.995643) | 1.836375 / 1.504120 (0.332255) | 1.651507 / 1.541195 (0.110312) | 1.734134 / 1.468490 (0.265644) | 0.690900 / 4.584777 (-3.893877) | 3.812045 / 3.745712 (0.066333) | 2.101378 / 5.269862 (-3.168483) | 1.438242 / 4.565676 (-3.127434) | 0.083256 / 0.424275 (-0.341020) | 0.012436 / 0.007607 (0.004829) | 0.501702 / 0.226044 (0.275658) | 5.007679 / 2.268929 (2.738751) | 2.315158 / 55.444624 (-53.129466) | 2.003934 / 6.876477 (-4.872543) | 2.154658 / 2.142072 (0.012586) | 0.831749 / 4.805227 (-3.973478) | 0.165058 / 6.500664 (-6.335606) | 0.062166 / 0.075469 (-0.013303) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.212435 / 1.841788 (-0.629353) | 15.022673 / 8.074308 (6.948365) | 14.649631 / 10.191392 (4.458239) | 0.172121 / 0.680424 (-0.508303) | 0.028791 / 0.534201 (-0.505410) | 0.440290 / 0.579283 (-0.138993) | 0.437359 / 0.434364 (0.002995) | 0.543603 / 0.540337 (0.003265) | 0.643241 / 1.386936 (-0.743695) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007572 / 0.011353 (-0.003781) | 0.005207 / 0.011008 (-0.005801) | 0.074427 / 0.038508 (0.035919) | 0.033384 / 0.023109 (0.010275) | 0.334538 / 0.275898 (0.058640) | 0.371556 / 0.323480 (0.048076) | 0.006453 / 0.007986 (-0.001532) | 0.004010 / 0.004328 (-0.000319) | 0.073488 / 0.004250 (0.069238) | 0.048082 / 0.037052 (0.011030) | 0.337325 / 0.258489 (0.078836) | 0.395143 / 0.293841 (0.101302) | 0.036714 / 0.128546 (-0.091832) | 0.012089 / 0.075646 (-0.063557) | 0.086008 / 0.419271 (-0.333263) | 0.049277 / 0.043533 (0.005744) | 0.333848 / 0.255139 (0.078709) | 0.354003 / 0.283200 (0.070803) | 0.105012 / 0.141683 (-0.036671) | 1.450769 / 1.452155 (-0.001386) | 1.554538 / 1.492716 (0.061821) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.208407 / 0.018006 (0.190400) | 0.438778 / 0.000490 (0.438288) | 0.000399 / 0.000200 (0.000199) | 0.000059 / 0.000054 (0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030180 / 0.037411 (-0.007232) | 0.115432 / 0.014526 (0.100906) | 0.126106 / 0.176557 (-0.050451) | 0.167508 / 0.737135 (-0.569627) | 0.130566 / 0.296338 (-0.165772) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.421408 / 0.215209 (0.206198) | 4.208492 / 2.077655 (2.130838) | 2.024177 / 1.504120 (0.520057) | 1.834356 / 1.541195 (0.293161) | 1.923234 / 1.468490 (0.454744) | 0.699548 / 4.584777 (-3.885229) | 3.933775 / 3.745712 (0.188063) | 2.124526 / 5.269862 (-3.145336) | 1.360934 / 4.565676 (-3.204742) | 0.086568 / 0.424275 (-0.337707) | 0.012351 / 0.007607 (0.004744) | 0.517431 / 0.226044 (0.291387) | 5.175428 / 2.268929 (2.906499) | 2.471031 / 55.444624 (-52.973593) | 2.131529 / 6.876477 (-4.744948) | 2.202512 / 2.142072 (0.060440) | 0.849364 / 4.805227 (-3.955863) | 0.171505 / 6.500664 (-6.329159) | 0.065864 / 0.075469 (-0.009605) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.270054 / 1.841788 (-0.571734) | 15.254502 / 8.074308 (7.180194) | 13.874969 / 10.191392 (3.683577) | 0.144131 / 0.680424 (-0.536293) | 0.017743 / 0.534201 (-0.516458) | 0.421990 / 0.579283 (-0.157293) | 0.423924 / 0.434364 (-0.010439) | 0.522560 / 0.540337 (-0.017778) | 0.626159 / 1.386936 (-0.760777) |

![](https://cml.dev/watermark.png#05bd726a575a3c1c337022424fa7d226f1a2ebee "CML watermark")

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008643 / 0.011353 (-0.002710) | 0.004479 / 0.011008 (-0.006529) | 0.102372 / 0.038508 (0.063864) | 0.029703 / 0.023109 (0.006594) | 0.301479 / 0.275898 (0.025581) | 0.370970 / 0.323480 (0.047490) | 0.007044 / 0.007986 (-0.000942) | 0.004868 / 0.004328 (0.000540) | 0.079568 / 0.004250 (0.075318) | 0.035344 / 0.037052 (-0.001708) | 0.308091 / 0.258489 (0.049602) | 0.353812 / 0.293841 (0.059971) | 0.033406 / 0.128546 (-0.095140) | 0.011476 / 0.075646 (-0.064170) | 0.324343 / 0.419271 (-0.094929) | 0.040293 / 0.043533 (-0.003240) | 0.300007 / 0.255139 (0.044868) | 0.334410 / 0.283200 (0.051210) | 0.086553 / 0.141683 (-0.055130) | 1.463814 / 1.452155 (0.011659) | 1.501580 / 1.492716 (0.008864) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.198032 / 0.018006 (0.180025) | 0.409970 / 0.000490 (0.409480) | 0.001075 / 0.000200 (0.000875) | 0.000076 / 0.000054 (0.000022) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022941 / 0.037411 (-0.014471) | 0.097320 / 0.014526 (0.082794) | 0.106445 / 0.176557 (-0.070111) | 0.139073 / 0.737135 (-0.598063) | 0.108408 / 0.296338 (-0.187930) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.419315 / 0.215209 (0.204106) | 4.199273 / 2.077655 (2.121618) | 1.877689 / 1.504120 (0.373569) | 1.670442 / 1.541195 (0.129247) | 1.735034 / 1.468490 (0.266544) | 0.694691 / 4.584777 (-3.890086) | 3.323644 / 3.745712 (-0.422069) | 2.884349 / 5.269862 (-2.385513) | 1.518882 / 4.565676 (-3.046794) | 0.082390 / 0.424275 (-0.341886) | 0.012884 / 0.007607 (0.005277) | 0.525103 / 0.226044 (0.299058) | 5.277297 / 2.268929 (3.008369) | 2.328639 / 55.444624 (-53.115985) | 1.983210 / 6.876477 (-4.893267) | 2.037985 / 2.142072 (-0.104088) | 0.809520 / 4.805227 (-3.995707) | 0.150150 / 6.500664 (-6.350514) | 0.065578 / 0.075469 (-0.009891) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.221971 / 1.841788 (-0.619817) | 13.692361 / 8.074308 (5.618052) | 13.874582 / 10.191392 (3.683190) | 0.138182 / 0.680424 (-0.542242) | 0.028618 / 0.534201 (-0.505583) | 0.395104 / 0.579283 (-0.184179) | 0.397169 / 0.434364 (-0.037195) | 0.457509 / 0.540337 (-0.082829) | 0.537275 / 1.386936 (-0.849661) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006835 / 0.011353 (-0.004518) | 0.004585 / 0.011008 (-0.006423) | 0.076877 / 0.038508 (0.038369) | 0.027305 / 0.023109 (0.004196) | 0.349085 / 0.275898 (0.073187) | 0.401416 / 0.323480 (0.077936) | 0.004912 / 0.007986 (-0.003074) | 0.003315 / 0.004328 (-0.001014) | 0.075676 / 0.004250 (0.071425) | 0.038960 / 0.037052 (0.001907) | 0.346196 / 0.258489 (0.087707) | 0.403185 / 0.293841 (0.109344) | 0.032054 / 0.128546 (-0.096493) | 0.011742 / 0.075646 (-0.063905) | 0.086631 / 0.419271 (-0.332640) | 0.041633 / 0.043533 (-0.001900) | 0.343519 / 0.255139 (0.088380) | 0.385413 / 0.283200 (0.102213) | 0.091430 / 0.141683 (-0.050253) | 1.478886 / 1.452155 (0.026731) | 1.546873 / 1.492716 (0.054156) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.167882 / 0.018006 (0.149876) | 0.396464 / 0.000490 (0.395974) | 0.003629 / 0.000200 (0.003429) | 0.000085 / 0.000054 (0.000030) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024829 / 0.037411 (-0.012583) | 0.099607 / 0.014526 (0.085081) | 0.106187 / 0.176557 (-0.070370) | 0.142379 / 0.737135 (-0.594756) | 0.109307 / 0.296338 (-0.187032) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.442276 / 0.215209 (0.227067) | 4.427099 / 2.077655 (2.349444) | 2.093407 / 1.504120 (0.589287) | 1.880973 / 1.541195 (0.339778) | 1.915592 / 1.468490 (0.447102) | 0.708196 / 4.584777 (-3.876581) | 3.417649 / 3.745712 (-0.328063) | 2.859953 / 5.269862 (-2.409909) | 1.528380 / 4.565676 (-3.037297) | 0.084054 / 0.424275 (-0.340221) | 0.012585 / 0.007607 (0.004978) | 0.537614 / 0.226044 (0.311569) | 5.409915 / 2.268929 (3.140987) | 2.555853 / 55.444624 (-52.888771) | 2.195075 / 6.876477 (-4.681402) | 2.232775 / 2.142072 (0.090703) | 0.814994 / 4.805227 (-3.990233) | 0.152882 / 6.500664 (-6.347782) | 0.067467 / 0.075469 (-0.008002) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.306007 / 1.841788 (-0.535780) | 13.923981 / 8.074308 (5.849673) | 13.385881 / 10.191392 (3.194489) | 0.150712 / 0.680424 (-0.529712) | 0.016731 / 0.534201 (-0.517470) | 0.376557 / 0.579283 (-0.202726) | 0.379396 / 0.434364 (-0.054968) | 0.456251 / 0.540337 (-0.084087) | 0.545731 / 1.386936 (-0.841205) |

![](https://cml.dev/watermark.png#cc637d107ef3e3b9948691379312a8099b6476aa "CML watermark")

Skip the dataset verifications (split and checksum verifications, duplicate keys check) by default unless a dataset is being tested (`datasets-cli test/run_beam`). The main goal is to avoid running the checksum check in the default case due to how expensive it can be for large datasets.

PS: Maybe we should deprecate `ignore_verifications`, which is `True` now by default, and give it a different name?


Improve `use_auth_token` docstring and deprecate `use_auth_token` in `download_and_prepare`

_The documentation is not available anymore as the PR was closed or merged._

Clarify in the docstrings what happens when `use_auth_token` is `None` and deprecate the `use_auth_token` param in `download_and_prepare`.


Return a split Dataset in load_dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5301). All of your documentation changes will be reflected on that endpoint.

Just noticed that now we have to deal with indexed & split datasets. The remaining tests are failing because one should be able to get an indexed dataset when accessing the split of a dataset made of indexed splits (right now the index is just trashed)

...instead of a DatasetDict.

```python
# now supported
ds = load_dataset("squad")
ds[0]
for example in ds:
    pass

# still works
ds["train"]
ds["validation"]

# new
ds.splits  # Dict[str, Dataset] | None

# soon to be supported (not in this PR)
ds = load_dataset("dataset_with_no_splits")
ds[0]
for example in ds:
    pass
```

I implemented `Dataset.__getitem__` and `IterableDataset.__getitem__` to be able to get a split from a dataset.
The splits are defined by the `ds.info.splits` dictionary.

Therefore a dataset is a table that optionally has some splits defined in the dataset info. And a split dataset is the concatenation of all its splits.

I made as little breaking changes as possible. Notable breaking changes:
- `load_dataset("potato").keys() / .items() / .values() /` don't work anymore, since we don't return a dict
- same for `for split_name in load_dataset("potato")`, since we now iterate on the examples
- ..

TODO:
- [x] Update push_to_hub
- [x] Update save_to_disk/load_from_disk
- [ ] check for other breaking changes
- [ ] fix existing tests
- [ ] add new tests
- [ ] docs

This is related to https://github.com/huggingface/datasets/issues/5189, to extend `load_dataset` to return datasets without splits


Use same `num_proc` for dataset download and generation

I noticed this bug the other day and was going to look into it! "Where are these processes coming from?" ;-)

Use the same `num_proc` value for data download and generation. Additionally, do not set `num_proc` to 16 in `DownloadManager` by default (`num_proc` now has to be specified explicitly).


Fix xopen for Windows pathnames

This PR fixes a bug in `xopen` function for Windows pathnames.

Fix #5298.


Bug in xopen with Windows pathnames

bug

Something isn't working

Currently, `xopen` function has a bug with local Windows pathnames:

From its implementation:
```python
def xopen(file: str, mode="r", *args, **kwargs):
    file = _as_posix(PurePath(file))
    main_hop, *rest_hops = file.split("::")
    if is_local_path(main_hop):
        return open(file, mode, *args, **kwargs)
```

On a Windows machine, if we pass the argument:
```python
xopen("C:\\Users\\USERNAME\\filename.txt")
```
it returns
```python
open("C:/Users/USERNAME/filename.txt")
```


Fix xjoin for Windows pathnames

This PR fixes a bug in `xjoin` function with Windows pathnames.

Fix #5296.


Bug in xjoin with Windows pathnames

Something isn't working

Currently, `xjoin` function has a bug with local Windows pathnames: instead of returning the OS-dependent join pathname, it always returns it in POSIX format.

```python
from datasets.download.streaming_download_manager import xjoin

path = xjoin("C:\\Users\\USERNAME", "filename.txt")
```

Join path should be:
```python
"C:\\Users\\USERNAME\\filename.txt"
```
However it is:
```python
"C:/Users/USERNAME/filename.txt"
```


Extractions failed when .zip file located on read-only path (e.g., SageMaker FastFile mode)

Hi ! Thanks for reporting. Indeed the lock file should be placed in a directory with write permission (e.g. in the directory where the archive is extracted).

I opened https://github.com/huggingface/datasets/pull/5320 to fix this - it places the lock file in the cache directory instead of trying to put in next to the ZIP where it's read-only

### Describe the bug

Hi,

`load_dataset()` does not work .zip files located on a read-only directory. Looks like it's because Dataset creates a lock file in the [same directory](https://github.com/huggingface/datasets/blob/df4bdd365f2abb695f113cbf8856a925bc70901b/src/datasets/utils/extract.py)  as the .zip file.

Encountered this when attempting `load_dataset()` on a datadir with SageMaker FastFile mode.

### Steps to reproduce the bug

```python
# Showing relevant lines only.

hyperparameters = {
    "dataset_name": "ydshieh/coco_dataset_script",
    "dataset_config_name": 2017,
    "data_dir": "/opt/ml/input/data/coco",
    "cache_dir": "/tmp/huggingface-cache",  # Fix dataset complains out-of-space.
    ...
}

estimator = PyTorch(
    base_job_name="clip",
    source_dir="../src/sm-entrypoint",
    entry_point="run_clip.py",  # Transformers/src/examples/pytorch/contrastive-image-text/run_clip.py
    framework_version="1.12",
    py_version="py38",
    hyperparameters=hyperparameters,
    instance_count=1,
    instance_type="ml.p3.16xlarge",
    volume_size=100,
    distribution={"smdistributed": {"dataparallel": {"enabled": True}}},
)

fast_file = lambda x: TrainingInput(x, input_mode='FastFile')
estimator.fit(
    {
        "pre-trained": fast_file("s3://vm-sagemakerr-us-east-1/clip/pre-trained-checkpoint/"),
        "coco": fast_file("s3://vm-sagemakerr-us-east-1/clip/coco-zip-files/"),
    }
)
```

Error message:

```text
ErrorMessage "OSError: [Errno 30] Read-only file system: '/opt/ml/input/data/coco/image_info_test2017.zip.lock'
 """

 The above exception was the direct cause of the following exception
 Traceback (most recent call last)
 File "/opt/conda/lib/python3.8/runpy.py", line 194, in _run_module_as_main
 return _run_code(code, main_globals, None,
 File "/opt/conda/lib/python3.8/runpy.py", line 87, in _run_code
 exec(code, run_globals)
 File "/opt/conda/lib/python3.8/site-packages/mpi4py/__main__.py", line 7, in <module>
 main()
 File "/opt/conda/lib/python3.8/site-packages/mpi4py/run.py", line 198, in main
 run_command_line(args)
 File "/opt/conda/lib/python3.8/site-packages/mpi4py/run.py", line 47, in run_command_line
 run_path(sys.argv[0], run_name='__main__')
 File "/opt/conda/lib/python3.8/runpy.py", line 265, in run_path
 return _run_module_code(code, init_globals, run_name,
 File "/opt/conda/lib/python3.8/runpy.py", line 97, in _run_module_code
 _run_code(code, mod_globals, init_globals,
 File "run_clip_smddp.py", line 594, in <module>
 File "run_clip_smddp.py", line 327, in main
 dataset = load_dataset(
 File "/opt/conda/lib/python3.8/site-packages/datasets/load.py", line 1741, in load_dataset
 builder_instance.download_and_prepare(
 File "/opt/conda/lib/python3.8/site-packages/datasets/builder.py", line 822, in download_and_prepare
 self._download_and_prepare(
 File "/opt/conda/lib/python3.8/site-packages/datasets/builder.py", line 1555, in _download_and_prepare
 super()._download_and_prepare(
 File "/opt/conda/lib/python3.8/site-packages/datasets/builder.py", line 891, in _download_and_prepare
 split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
 File "/root/.cache/huggingface/modules/datasets_modules/datasets/ydshieh--coco_dataset_script/e033205c0266a54c10be132f9264f2a39dcf893e798f6756d224b1ff5078998f/coco_dataset_script.py", line 123, in _split_generators
 archive_path = dl_manager.download_and_extract(_DL_URLS)
 File "/opt/conda/lib/python3.8/site-packages/datasets/download/download_manager.py", line 447, in download_and_extract
 return self.extract(self.download(url_or_urls))
 File "/opt/conda/lib/python3.8/site-packages/datasets/download/download_manager.py", line 419, in extract
 extracted_paths = map_nested(
 File "/opt/conda/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 472, in map_nested
 mapped = pool.map(_single_map_nested, split_kwds)
 File "/opt/conda/lib/python3.8/multiprocessing/pool.py", line 364, in map
 return self._map_async(func, iterable, mapstar, chunksize).get()
 File "/opt/conda/lib/python3.8/multiprocessing/pool.py", line 771, in get
 raise self._value
 OSError: [Errno 30] Read-only file system: '/opt/ml/input/data/coco/image_info_test2017.zip.lock'"

```

### Expected behavior

`load_dataset()` to succeed, just like when .zip file is passed in SageMaker File mode.

### Environment info

* datasets-2.7.1
* transformers-4.24.0
* python-3.8
* torch-1.12
* SageMaker PyTorch DLC


Support streaming datasets with pathlib.Path.with_suffix

This PR extends the support in streaming mode for datasets that use `pathlib.Path.with_suffix`.

Fix #5293.


enhancement

New feature or request

Extend support for streaming datasets that use `pathlib.Path.with_suffix`.

This feature will be useful e.g. for datasets containing text files and annotated files with the same name but different extension.


Missing documentation build for versions 2.7.1 and 2.6.2

maintenance

Maintenance tasks

- Build docs for 2.6.2:
  - Commit: a6a5a1cf4cdf1e0be65168aed5a327f543001fe8
  - Build docs GH Action: https://github.com/huggingface/datasets/actions/runs/3539470622/jobs/5941404044
- Build docs for 2.7.1:
  - Commit: 5ef1ab1cc06c2b7a574bf2df454cd9fcb071ccb2
  - Build docs GH Action: https://github.com/huggingface/datasets/actions/runs/3539574442/jobs/5941636792

After the patch releases [2.7.1](https://github.com/huggingface/datasets/releases/tag/2.7.1) and [2.6.2](https://github.com/huggingface/datasets/releases/tag/2.6.2), the online docs were not properly built (the build_documentation workflow was not triggered).

There was a fix by:
- #5291

However, both documentations were built from main branch, instead of their corresponding version branch.

We are rebuilding them.


[build doc] for v2.7.1 & v2.6.2

doc versions are built https://huggingface.co/docs/datasets/index

Do NOT merge. Using this PR to build docs for [v2.7.1](https://github.com/huggingface/datasets/pull/5291/commits/f4914af20700f611b9331a9e3ba34743bbeff934) & [v2.6.2](https://github.com/huggingface/datasets/pull/5291/commits/025f85300a0874eeb90a20393c62f25ac0accaa0)


fix error where reading breaks when batch missing an assigned column feature

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5290). All of your documentation changes will be reflected on that endpoint.


Added support for JXL images.

I'm fine with the addition of jxl in the list of known image extensions, this way users that have the plugin can work with their JXL datasets. WDYT @mariosasko ?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5289). All of your documentation changes will be reflected on that endpoint.

I think we should wait for official support from Pillow. Plus, the linked plugin doesn't support `Image.save`, which is one of the requirements for a format to be included in `IMAGE_EXTENSIONS`.

@alexjc In the meantime, one option is to add these lines to the card:
```python
import importlib
import datasets

if ".jxl" not in datasets.packaged_modules.imagefolder.IMAGE_EXTENSIONS:
  datasets.packaged_modules.imagefolder.IMAGE_EXTENSIONS.append(".jxl")

if "jxl" not in datasets.packaged_modules._EXTENSION_TO_MODULE:
  datasets.packaged_modules._EXTENSION_TO_MODULE["jxl"] = ("imagefolder", {})

importlib.reload(datasets.load)
ds = datasets.load_dataset("texturedesign/td01_natural-ground-textures")
```
And you can add a note to the card that this dataset requires the "jxlpy" package to work.

In this case, you can also disable the viewer to avoid the discrepancy between the data displayed in the preview and the loaded data.

Another option is to define the loading script and add `jxlpy` to the list of dependencies [here](https://github.com/huggingface/datasets-server/blob/3012da62054a025467616abc14b0b46e1f11ea13/workers/first_rows/pyproject.toml#L8) to enable the viewer. This option requires more work, so let us know if you need help.

Thank you both for your thoughtful replies!

One questions and and update:
* The jxlpy plugin does support saving, in the `_save` function of the JXLImagePlugin file.  Did it not work?  I'm working on the upgrade to the latest JXL, so it'd be good to know if it failed so I can fix it.
* I wrote to the Pillow maintainer and the preferred solution would be to keep JXL as a separate plugin because they're a small team don't have the resources to maintain more code.

With that in mind, let me share the minimal set of features I'd need for this to work within the `datasets` library:
1. Using `load_dataset()` with the HuggingFace dataset name correctly downloads the JXL files so they are available locally. Even if the `file_name` field is left intact and not loaded as a PIL image, this is the first step.
2. With minimal monkey-patching, having the `load_dataset` correctly expand `file_name` into PIL `image` fields if JXL support is available.

If both of these work, then I can use HuggingFace's hub and the `datasets` library for an MVP even if not all features are there.  I don't need automatic thumbnails or previews of the dataset on the server.

Given the reply from the Pillow maintainer, what solution can we come up with that works in a more permanent way than waiting for Pillow integration (which may not happen) — assuming users install the `jxlpy` plugin separately?

Link to my upgrade for the latest `libjxl`, pending review and merge. I tested load/save via Pillow extensively for this: https://github.com/olokelo/jxlpy/pull/13

After more research, here's my latest suggestion:
* Depending on the build of pillow, the source (pip or conda), the platform even, certain formats may or may not be available — despite them being in the list.  For example, webp support is not consistently available.
* I'd suggest adding JXL to the list and simply catching the `PIL.UnidentifiedImageError` — printing a useful error message that sends them to a Wiki page to find out what to do.
* On that page would be included instructions how to install support for the format and what to do for the dataset to load correctly on any platform, both with or without conda, etc.

What do you think?

> The jxlpy plugin does support saving, in the _save function of the JXLImagePlugin file. Did it not work? I'm working on the upgrade to the latest JXL, so it'd be good to know if it failed so I can fix it.

My bad, I was referring to [this](https://github.com/google/brunsli/blob/2dd949e53ed05796eb44a31cc759fbf9e6c53e2f/contrib/py/jxl_library_patches/jxl_pillow.py) version of the plugin.

I still think this involves too much work:
* would require a new doc page
* unofficial plugins have to be imported explicitly, leading to messier code on our side
* etc.

For now, it seems more reasonable to create a loading script (faster than ImageFolder, as ImageFolder has to resolve the image files first) for this particular case and add `jxlpy` to the list of the `datasets-server`'s dependencies. Also, one additional advantage of this approach is that it reports if any of the modules imported in a script is missing, which is handy in your case for the plugin lib. WDYT?

OK, let me try it it and I'll report back.

Will the JXL files (even if unknown format) be automatically downloaded if they are linked from the `.jsonl` file?

(I had trouble getting that working before this patch.)

> Will the JXL files (even if unknown format) be automatically downloaded if they are linked from the .jsonl file?

No, they need to be downloaded explicitly.

Feel free to use 🤗 Hub discussions in your dataset repo to ping us for help (our usernames are the same there)

Is it possible to add support for JXL files being downloaded without needing to add server-side rendering support?

In the loading script, data files are downloaded with `DownloadManager` (`dl_manager` in `_split_generators`), which doesn't have any requirements regarding the actual type of the downloaded files.

PS: Let's use the forum or Hub discussions for further questions to avoid pinging other participants

JPEG-XL is the most advanced of the next-generation of image codecs, supporting both lossless and lossy files — with better compression and quality than PNG and JPG respectively.  It has reduced the disk sizes and bandwidth required for many of the datasets I use.

Pillow does not yet support JXL, but there's a plugin as a separate Python library that does (`pip install jxlpy`), and I've tested that this change works as expected when the plugin is imported.

Dataset used for testing, you must `git pull` as loading it from Python won't work until `datasets-server` is also changed to support JXL files:
https://huggingface.co/datasets/texturedesign/td01_natural-ground-textures

The case where the plugin is not imported first raises an error:
```
PIL.UnidentifiedImageError: cannot identify image file 'td01/train/set01/01_145523.jxl'
```

In order to enable support for JXL even before pillow supports this, should this exception be handled with a better error message?  I'd expect/hope JXL support to follow in one of the pillow quarterly releases in the next 6-9 months.


Lossy json serialization - deserialization of dataset info

Hi ! JSON is a lossy format indeed. If you want to keep the feature types or other metadata I'd encourage you to store them as well. For example you can use `dataset.info.write_to_directory` and `DatasetInfo.from_directory` to store the feature types, split info, description, license etc.

### Describe the bug

Saving a dataset to disk as json (using `to_json`) and then loading it again (using `load_dataset`) results in features whose labels are not type-cast correctly. In the code snippet below, `features.label` should have a label of type `ClassLabel` but has type `Value` instead.

### Steps to reproduce the bug

```
from datasets import load_dataset

def test_serdes_from_json(d):
    dataset = load_dataset(d, split="train")
    dataset.to_json('_test')
    dataset_loaded = load_dataset("json", data_files='_test', split='train')
    try:
        assert dataset_loaded.info.features == dataset.info.features, "features unequal!"
    except Exception as ex:
        print(f'{ex}')
        print(f'expected {dataset.info.features}, \nactual { dataset_loaded.info.features }')

test_serdes_from_json('rotten_tomatoes')
```

Output
```
features unequal!
expected {'text': Value(dtype='string', id=None), 'label': ClassLabel(names=['neg', 'pos'], id=None)},
actual {'text': Value(dtype='string', id=None), 'label': Value(dtype='int64', id=None)}
```

### Expected behavior

The deserialized `features.label` should have type `ClassLabel`.

### Environment info

- `datasets` version: 2.6.1
- Platform: Linux-5.10.144-127.601.amzn2.x86_64-x86_64-with-glibc2.17
- Python version: 3.7.13
- PyArrow version: 7.0.0
- Pandas version: 1.2.3


Fix methods using `IterableDataset.map` that lead to `features=None`

_The documentation is not available anymore as the PR was closed or merged._

Maybe other options are:
* Keep the `info.features` to `None` if those were initially `None`
* Infer the features with pre-fetching just if the `info.features` is `None`
* If the `info.features` are there, make sure that after `map` features is not `None`

Hi @lhoestq something that's still not clear to me is: should we infer the features always when applying a `map` if those are initially `None`, or just assume that if the features are initially `None` those should be left that way unless the user specifically sets those (or during iter)?

In this PR I'm using `from datasets.iterable_dataset import _infer_features_from_batch` to infer the features when those are `None` using pre-fetch of `self._head()`, but I'm not sure if that's the expected behavior.

Thanks in advance for your help!

Also, the PR still has some more work to do, but probably the most relevant thing to fix right now is that the `features` are being set to `None` in the functions `IterableDataset.rename_column`, `IterableDataset.rename_columns`, and `IterableDataset.remove_columns` when the `features` originally had a value. So once that's fixed maybe we can focus on improving the current `map`'s behavior, so as to avoid this from happening also when the user uses `map` directly and not through the functions mentioned above.

> Cool thank you ! Resolving the features can be expensive sometimes, so maybe we don't resolve the features and we can just rename/remove columns if the features are known (i.e. if they're not None). What do you think ?

Thanks for the feedback! Makes sense to me 👍🏻 I'll commit the comments now!

Already done @lhoestq, feel free to merge whenever you want! Also before merging, can you please link the following issues https://github.com/huggingface/datasets/issues/3888, https://github.com/huggingface/datasets/issues/5245, and https://github.com/huggingface/datasets/issues/5284, so that those are closed upon merge? Thanks!

As currently `IterableDataset.map` is setting the `info.features` to `None` every time as we don't know the output of the dataset in advance, `IterableDataset` methods such as `rename_column`, `rename_columns`, and `remove_columns`. that internally use `map` lead to the features being `None`.

This PR is related to #3888, #5245, and #5284

## ✅ Current solution

The code in this PR is basically making sure that if the features were there since the beginning and a `rename_column`/`rename_columns` happens, those are kept and the rename is applied to the `Features` too. Also, if the features were not there before applying `rename_column`, `rename_columns` or `remove_columns`, a batch is prefetched and the features are being inferred (that could potentially be part of `IterableDataset.__init__` in case the `info.features` value is `None`).

## 💡 Ideas

Some ideas were proposed in https://github.com/huggingface/datasets/issues/3888, but probably the most consistent solution even though it may take some time is to actually do the type inferencing during the `IterableDataset.__init__` in case the provided `info.features` is `None`, otherwise, we can just use the provided features.

Additionally, as mentioned at https://github.com/huggingface/datasets/issues/3888, we could also include a `features` parameter to the `map` function, but that's probably more tedious.

Also thanks to @lhoestq for sharing some ideas in both https://github.com/huggingface/datasets/issues/3888 and https://github.com/huggingface/datasets/issues/5245 :hugs:


I found a solution

If you specifically install datasets==1.18 and then run

import datasets
wiki = datasets.load_dataset('wikipedia', '20200501.en')
then this should work (it worked for me.)

I have the same problem here but installing datasets==1.18 wont work for me

### Describe the bug

I follow the steps provided on the website [https://huggingface.co/datasets/wikipedia](https://huggingface.co/datasets/wikipedia)

$ pip install apache_beam mwparserfromhell
>>> from datasets import load_dataset
>>> load_dataset("wikipedia", "20220301.en")

however this results in the following error:

 raise MissingBeamOptions(
datasets.builder.MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory).
Example of usage:
	`load_dataset('wikipedia', '20220301.en', beam_runner='DirectRunner')`

If I then prompt the system with:

>>> load_dataset('wikipedia', '20220301.en', beam_runner='DirectRunner')

the following error occurs:

raise FileNotFoundError(f"Couldn't find file at {url}")
FileNotFoundError: Couldn't find file at https://dumps.wikimedia.org/enwiki/20220301/dumpstatus.json

Here is the exact code:

Python 3.10.6 (main, Nov  2 2022, 18:53:38) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from datasets import load_dataset
>>> load_dataset('wikipedia', '20220301.en')
Downloading and preparing dataset wikipedia/20220301.en to /home/[EDITED]/.cache/huggingface/datasets/wikipedia/20220301.en/2.0.0/aa542ed919df55cc5d3347f42dd4521d05ca68751f50dbc32bae2a7f1e167559...
Downloading: 100%|████████████████████████████████████████████████████████████████████████████| 15.3k/15.3k [00:00<00:00, 22.2MB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 1741, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 822, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1879, in _download_and_prepare
    raise MissingBeamOptions(
datasets.builder.MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory).
Example of usage:
	`load_dataset('wikipedia', '20220301.en', beam_runner='DirectRunner')`

>>> load_dataset('wikipedia', '20220301.en', beam_runner='DirectRunner')
Downloading and preparing dataset wikipedia/20220301.en to /home/[EDITED]/.cache/huggingface/datasets/wikipedia/20220301.en/2.0.0/aa542ed919df55cc5d3347f42dd4521d05ca68751f50dbc32bae2a7f1e167559...
Downloading: 100%|████████████████████████████████████████████████████████████████████████████| 15.3k/15.3k [00:00<00:00, 18.8MB/s]
Downloading data files:   0%|                                                                                | 0/1 [00:00<?, ?it/s]Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 1741, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 822, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1909, in _download_and_prepare
    super()._download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 891, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/rorytol/.cache/huggingface/modules/datasets_modules/datasets/wikipedia/aa542ed919df55cc5d3347f42dd4521d05ca68751f50dbc32bae2a7f1e167559/wikipedia.py", line 945, in _split_generators
    downloaded_files = dl_manager.download_and_extract({"info": info_url})
  File "/usr/local/lib/python3.10/dist-packages/datasets/download/download_manager.py", line 447, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/usr/local/lib/python3.10/dist-packages/datasets/download/download_manager.py", line 311, in download
    downloaded_path_or_paths = map_nested(
  File "/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py", line 444, in map_nested
    mapped = [
  File "/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py", line 445, in <listcomp>
    _single_map_nested((function, obj, types, None, True, None))
  File "/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py", line 346, in _single_map_nested
    return function(data_struct)
  File "/usr/local/lib/python3.10/dist-packages/datasets/download/download_manager.py", line 338, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/usr/local/lib/python3.10/dist-packages/datasets/utils/file_utils.py", line 183, in cached_path
    output_path = get_from_cache(
  File "/usr/local/lib/python3.10/dist-packages/datasets/utils/file_utils.py", line 530, in get_from_cache
    raise FileNotFoundError(f"Couldn't find file at {url}")
FileNotFoundError: Couldn't find file at https://dumps.wikimedia.org/enwiki/20220301/dumpstatus.json

### Steps to reproduce the bug

$ pip install apache_beam mwparserfromhell
>>> from datasets import load_dataset
>>> load_dataset("wikipedia", "20220301.en")
>>> load_dataset('wikipedia', '20220301.en', beam_runner='DirectRunner')

### Expected behavior

Download the dataset

### Environment info

Running linux on a remote workstation operated through a macbook terminal

Python 3.10.6


Save file name in embed_storage

I updated the tests, met le know if it sounds good to you now :)

Having the file name is useful in case we need to check the extension of the file (e.g. mp3), or in general in case it includes some metadata information (track id, image id etc.)

Related to https://github.com/huggingface/datasets/issues/5276


Features of IterableDataset set to None by remove column

bug

Something isn't working

Thanks @lhoestq and @alvarobartt!

This would be extremely helpful to have working for the Whisper fine-tuning event - we're **only** training using streaming mode, so it'll be quite important to have this feature working to make training as easy as possible!

_c.f._ https://twitter.com/sanchitgandhi99/status/1592188332171493377

> Thanks @lhoestq and @alvarobartt!
>
>
>
> This would be extremely helpful to have working for the Whisper fine-tuning event - we're **only** training using streaming mode, so it'll be quite important to have this feature working to make training as easy as possible!
>
>
>
> _c.f._ https://twitter.com/sanchitgandhi99/status/1592188332171493377

I'm almost done with at least a temporary fix to `rename_column`, `rename_columns`, and `remove_columns`, just trying to figure out how to extend it to the `map` function itself!

I'll probably open the PR for review either tomorrow or Sunday hopefully! Glad I can help you and HuggingFace 🤗

Awesome - thank you so much for this PR @alvarobartt! Is much appreciated!

@sanchit-gandhi PR is ready and open for review at #5287, but there's still one issue I may need @lhoestq's input :hugs:

Let us know @sanchit-gandhi if you need a new release of `datasets` soon with this fix included :)

Thanks for the fix guys! We can direct people to install `datasets` from main if that's easier!

Hey guys, any update around this? I'm facing the same issue with a streamable dataset.

Hi @asennoussi so this was already fixed and released as part of https://github.com/huggingface/datasets/releases/tag/2.8.0, so you should be able to install it as `pip install datasets==2.8.0` or just to use `pip install datasets --upgrade` to get the latest version, as of now, the https://github.com/huggingface/datasets/releases/tag/2.9.0 released last week! 🤗

Still facing the same issue though:
```
from datasets import IterableDatasetDict, load_dataset

raw_datasets = vectorized_datasets = IterableDatasetDict()

raw_datasets["train"] = load_dataset("asennoussi/private", split="train", use_auth_token=True, streaming=True)
raw_datasets["test"] = load_dataset("asennoussi/private", split="test", use_auth_token=True, streaming=True)

print("Original features: ", raw_datasets['train'].features.keys())

...

def prepare_dataset(batch):

    # load and (possibly) resample audio datato 16kHz
    audio = batch["audio"]

    # compute log-Mel input features from input audio array
    batch["input_features"] = processor.feature_extractor(audio["array"], sampling_rate=audio["sampling_rate"]).input_features[0]
    # compute input length of audio sample in seconds
    batch["input_length"] = len(audio["array"]) / audio["sampling_rate"]

    # optional pre-processing steps
    transcription = batch["sentence"]

    # encode target text to label ids
    batch["labels"] = processor.tokenizer(transcription).input_ids
    batch["labels_length"] = len(batch["labels"])
    return batch
...
vectorized_datasets = vectorized_datasets.remove_columns(['input_length', 'labels_length']+list(next(iter(raw_datasets.values())).features))
print("Processed features: ", vectorized_datasets['train'].features)
print("First sample:", next(iter(vectorized_datasets['train'])))

```

Output:
```
Original features:  dict_keys(['path', 'audio', 'sentence'])
Processed features:  None
```

Hmm weird, could you try to print

```python
print("Processed features: ", vectorized_datasets['train'].features)
```

again after iterating over the `vectorized_datasets`? In the code above, should be last line :)

Didn't seem to fix it:
```
Original features:  dict_keys(['path', 'audio', 'sentence'])
Processed features:  None
Processed features:  None
```

Actually the culprit looks to be this one:
`vectorized_datasets = raw_datasets.map(prepare_dataset).with_format("torch")`
When I remove this line: `vectorized_datasets = vectorized_datasets.remove_columns(['input_length', 'labels_length']+list(next(iter(raw_datasets.values())).features))`

I still get
```
Processed features:  None
```

> The culprit is definitely `.map` Just validated it. Any idea please?

Yes, indeed `.map` losses the features, because AFAIK pre-fetching the data to infer the features is expensive and not ideal, that's part of this issue https://github.com/huggingface/datasets/issues/3888

Anyway, now you can pass the `features` as a param to `.map` as follows:

```python
from datasets import Features
vectorized_datasets = raw_datasets.map(
    prepare_dataset,
    features=Features(
        {"path": raw_datasets["train"].info.features["path"], "audio": raw_datasets["train"].info.features["audio"], "sentence": raw_datasets["train"].info.features["sentence"]}
    ),
).with_format("torch")
```

Also, to let you know, when calling `.remove_columns` over an `IterableDataset`, the `features` are not lost, as well as `.rename_column` and `rename_columns` :)

More information about the latter at https://github.com/huggingface/datasets/pull/5287

@asennoussi alternatively you can just call `._resolve_features()` from your `IterableDataset` and it will pre-fetch the data to resolve the features, but note that feature-inference is not as accurate as if you manually specify which features and feature-types the `IterableDataset` has, as mentioned in the comment above, the alternative is to provide `features` param to `.map` :hugs:

### Describe the bug

The `remove_column` method of the IterableDataset sets the dataset features to None.

### Steps to reproduce the bug

```python
from datasets import Audio, load_dataset

# load LS in streaming mode
dataset = load_dataset("librispeech_asr", "clean", split="validation", streaming=True)

# check original features
print("Original features: ", dataset.features.keys())

# define features to remove: we KEEP audio and text
COLUMNS_TO_REMOVE = ['chapter_id', 'speaker_id', 'file', 'id']

dataset = dataset.remove_columns(COLUMNS_TO_REMOVE)

# check processed features, uh-oh!
print("Processed features: ", dataset.features)

# streaming the first audio sample still works
print("First sample:", next(iter(ds)))
```

**Print Output:**

```
Original features:  dict_keys(['file', 'audio', 'text', 'speaker_id', 'chapter_id', 'id'])
Processed features:  None
First sample: {'audio': {'path': '2277-149896-0000.flac', 'array': array([ 0.00186157,  0.0005188 ,  0.00024414, ..., -0.00097656,
       -0.00109863, -0.00146484]), 'sampling_rate': 16000}, 'text': "HE WAS IN A FEVERED STATE OF MIND OWING TO THE BLIGHT HIS WIFE'S ACTION THREATENED TO CAST UPON HIS ENTIRE FUTURE"}
```

### Expected behavior

The features should be those **not** removed by the `remove_column` method, i.e. audio and text.

### Environment info

- `datasets` version: 2.7.1
- Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.15
- PyArrow version: 9.0.0
- Pandas version: 1.3.5

(Running on Google Colab for a blog post: https://colab.research.google.com/drive/1ySCQREPZEl4msLfxb79pYYOWjUZhkr9y#scrollTo=8pRDGiVmH2ml)

cc @polinaeterna @lhoestq


Release: 2.6.2

_The documentation is not available anymore as the PR was closed or merged._


Release: 2.7.1