Remove py.typed

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008283 / 0.011353 (-0.003070) | 0.004450 / 0.011008 (-0.006558) | 0.099773 / 0.038508 (0.061265) | 0.029068 / 0.023109 (0.005959) | 0.296799 / 0.275898 (0.020901) | 0.350946 / 0.323480 (0.027466) | 0.007331 / 0.007986 (-0.000655) | 0.004550 / 0.004328 (0.000222) | 0.077603 / 0.004250 (0.073352) | 0.034307 / 0.037052 (-0.002746) | 0.313174 / 0.258489 (0.054685) | 0.342270 / 0.293841 (0.048429) | 0.033463 / 0.128546 (-0.095083) | 0.011421 / 0.075646 (-0.064225) | 0.317188 / 0.419271 (-0.102083) | 0.040985 / 0.043533 (-0.002548) | 0.300800 / 0.255139 (0.045661) | 0.360171 / 0.283200 (0.076972) | 0.086702 / 0.141683 (-0.054981) | 1.474679 / 1.452155 (0.022525) | 1.518319 / 1.492716 (0.025603) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.198059 / 0.018006 (0.180052) | 0.403502 / 0.000490 (0.403012) | 0.002663 / 0.000200 (0.002463) | 0.000218 / 0.000054 (0.000164) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022946 / 0.037411 (-0.014465) | 0.096466 / 0.014526 (0.081940) | 0.104092 / 0.176557 (-0.072465) | 0.138499 / 0.737135 (-0.598636) | 0.106941 / 0.296338 (-0.189397) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.416000 / 0.215209 (0.200791) | 4.153120 / 2.077655 (2.075465) | 1.843957 / 1.504120 (0.339837) | 1.650391 / 1.541195 (0.109197) | 1.684765 / 1.468490 (0.216275) | 0.688917 / 4.584777 (-3.895860) | 3.442797 / 3.745712 (-0.302916) | 1.834685 / 5.269862 (-3.435176) | 1.148046 / 4.565676 (-3.417631) | 0.082299 / 0.424275 (-0.341976) | 0.012399 / 0.007607 (0.004792) | 0.521099 / 0.226044 (0.295054) | 5.223695 / 2.268929 (2.954767) | 2.270970 / 55.444624 (-53.173654) | 1.921321 / 6.876477 (-4.955156) | 1.954675 / 2.142072 (-0.187398) | 0.809383 / 4.805227 (-3.995845) | 0.148562 / 6.500664 (-6.352102) | 0.064764 / 0.075469 (-0.010705) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.212687 / 1.841788 (-0.629101) | 13.491641 / 8.074308 (5.417333) | 12.972926 / 10.191392 (2.781534) | 0.137036 / 0.680424 (-0.543388) | 0.028591 / 0.534201 (-0.505610) | 0.391980 / 0.579283 (-0.187303) | 0.394474 / 0.434364 (-0.039889) | 0.456582 / 0.540337 (-0.083755) | 0.535984 / 1.386936 (-0.850952) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006419 / 0.011353 (-0.004934) | 0.004295 / 0.011008 (-0.006713) | 0.077702 / 0.038508 (0.039194) | 0.027368 / 0.023109 (0.004259) | 0.336713 / 0.275898 (0.060815) | 0.370074 / 0.323480 (0.046594) | 0.004657 / 0.007986 (-0.003328) | 0.003308 / 0.004328 (-0.001021) | 0.075747 / 0.004250 (0.071496) | 0.037323 / 0.037052 (0.000271) | 0.342382 / 0.258489 (0.083893) | 0.381109 / 0.293841 (0.087269) | 0.031804 / 0.128546 (-0.096742) | 0.011761 / 0.075646 (-0.063885) | 0.086818 / 0.419271 (-0.332454) | 0.042058 / 0.043533 (-0.001475) | 0.346295 / 0.255139 (0.091156) | 0.366857 / 0.283200 (0.083658) | 0.088666 / 0.141683 (-0.053016) | 1.533711 / 1.452155 (0.081556) | 1.537422 / 1.492716 (0.044705) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.220416 / 0.018006 (0.202410) | 0.387393 / 0.000490 (0.386903) | 0.003739 / 0.000200 (0.003539) | 0.000076 / 0.000054 (0.000021) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024083 / 0.037411 (-0.013329) | 0.098036 / 0.014526 (0.083510) | 0.102908 / 0.176557 (-0.073648) | 0.139512 / 0.737135 (-0.597623) | 0.107703 / 0.296338 (-0.188635) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.437615 / 0.215209 (0.222406) | 4.373140 / 2.077655 (2.295486) | 2.065063 / 1.504120 (0.560943) | 1.863938 / 1.541195 (0.322743) | 1.907955 / 1.468490 (0.439465) | 0.695830 / 4.584777 (-3.888947) | 3.394248 / 3.745712 (-0.351464) | 1.842794 / 5.269862 (-3.427068) | 1.156928 / 4.565676 (-3.408748) | 0.082505 / 0.424275 (-0.341771) | 0.012405 / 0.007607 (0.004798) | 0.538041 / 0.226044 (0.311997) | 5.363508 / 2.268929 (3.094579) | 2.509383 / 55.444624 (-52.935241) | 2.160416 / 6.876477 (-4.716061) | 2.162054 / 2.142072 (0.019982) | 0.802419 / 4.805227 (-4.002809) | 0.150529 / 6.500664 (-6.350135) | 0.066418 / 0.075469 (-0.009051) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.257221 / 1.841788 (-0.584567) | 13.748839 / 8.074308 (5.674531) | 13.310555 / 10.191392 (3.119163) | 0.152997 / 0.680424 (-0.527427) | 0.016618 / 0.534201 (-0.517583) | 0.375443 / 0.579283 (-0.203840) | 0.374942 / 0.434364 (-0.059422) | 0.466704 / 0.540337 (-0.073633) | 0.553563 / 1.386936 (-0.833373) |

![](https://cml.dev/watermark.png#1ac8343af4e2dc6fe0771d0be70eaf8a6e5a8fbc "CML watermark")

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009260 / 0.011353 (-0.002092) | 0.005213 / 0.011008 (-0.005795) | 0.102151 / 0.038508 (0.063643) | 0.035619 / 0.023109 (0.012510) | 0.296266 / 0.275898 (0.020368) | 0.359884 / 0.323480 (0.036404) | 0.008176 / 0.007986 (0.000190) | 0.005031 / 0.004328 (0.000703) | 0.077178 / 0.004250 (0.072927) | 0.041898 / 0.037052 (0.004846) | 0.305640 / 0.258489 (0.047151) | 0.346275 / 0.293841 (0.052434) | 0.037684 / 0.128546 (-0.090863) | 0.011816 / 0.075646 (-0.063831) | 0.334853 / 0.419271 (-0.084419) | 0.046535 / 0.043533 (0.003002) | 0.291544 / 0.255139 (0.036405) | 0.317194 / 0.283200 (0.033994) | 0.103212 / 0.141683 (-0.038471) | 1.424994 / 1.452155 (-0.027161) | 1.486216 / 1.492716 (-0.006501) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.011816 / 0.018006 (-0.006190) | 0.442092 / 0.000490 (0.441602) | 0.001297 / 0.000200 (0.001097) | 0.000078 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028277 / 0.037411 (-0.009134) | 0.110431 / 0.014526 (0.095905) | 0.118456 / 0.176557 (-0.058100) | 0.156778 / 0.737135 (-0.580357) | 0.123036 / 0.296338 (-0.173302) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.399006 / 0.215209 (0.183797) | 3.990367 / 2.077655 (1.912712) | 1.798739 / 1.504120 (0.294620) | 1.607133 / 1.541195 (0.065938) | 1.748897 / 1.468490 (0.280407) | 0.690666 / 4.584777 (-3.894111) | 3.795892 / 3.745712 (0.050180) | 3.479317 / 5.269862 (-1.790545) | 1.861268 / 4.565676 (-2.704409) | 0.085235 / 0.424275 (-0.339040) | 0.012997 / 0.007607 (0.005390) | 0.512489 / 0.226044 (0.286445) | 5.039515 / 2.268929 (2.770587) | 2.258079 / 55.444624 (-53.186545) | 1.907178 / 6.876477 (-4.969299) | 1.985953 / 2.142072 (-0.156119) | 0.843595 / 4.805227 (-3.961633) | 0.165286 / 6.500664 (-6.335378) | 0.063026 / 0.075469 (-0.012443) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.186680 / 1.841788 (-0.655108) | 14.976016 / 8.074308 (6.901708) | 14.436941 / 10.191392 (4.245549) | 0.172620 / 0.680424 (-0.507804) | 0.028760 / 0.534201 (-0.505441) | 0.443505 / 0.579283 (-0.135778) | 0.435665 / 0.434364 (0.001301) | 0.520164 / 0.540337 (-0.020174) | 0.608348 / 1.386936 (-0.778588) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007510 / 0.011353 (-0.003842) | 0.005012 / 0.011008 (-0.005996) | 0.077865 / 0.038508 (0.039357) | 0.033610 / 0.023109 (0.010500) | 0.365996 / 0.275898 (0.090098) | 0.416393 / 0.323480 (0.092913) | 0.005672 / 0.007986 (-0.002314) | 0.005334 / 0.004328 (0.001006) | 0.074948 / 0.004250 (0.070698) | 0.045962 / 0.037052 (0.008909) | 0.362209 / 0.258489 (0.103719) | 0.410522 / 0.293841 (0.116681) | 0.036247 / 0.128546 (-0.092299) | 0.012432 / 0.075646 (-0.063214) | 0.088754 / 0.419271 (-0.330517) | 0.048848 / 0.043533 (0.005315) | 0.370994 / 0.255139 (0.115855) | 0.382476 / 0.283200 (0.099277) | 0.103443 / 0.141683 (-0.038240) | 1.483127 / 1.452155 (0.030972) | 1.573366 / 1.492716 (0.080650) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.224163 / 0.018006 (0.206157) | 0.475136 / 0.000490 (0.474646) | 0.000394 / 0.000200 (0.000194) | 0.000057 / 0.000054 (0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030612 / 0.037411 (-0.006799) | 0.113983 / 0.014526 (0.099457) | 0.121835 / 0.176557 (-0.054722) | 0.160092 / 0.737135 (-0.577043) | 0.127431 / 0.296338 (-0.168908) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.421389 / 0.215209 (0.206179) | 4.207638 / 2.077655 (2.129984) | 2.040265 / 1.504120 (0.536145) | 1.868617 / 1.541195 (0.327422) | 1.979016 / 1.468490 (0.510526) | 0.712499 / 4.584777 (-3.872278) | 3.783091 / 3.745712 (0.037379) | 2.124293 / 5.269862 (-3.145569) | 1.382028 / 4.565676 (-3.183649) | 0.087133 / 0.424275 (-0.337142) | 0.012634 / 0.007607 (0.005027) | 0.518965 / 0.226044 (0.292920) | 5.188330 / 2.268929 (2.919401) | 2.556593 / 55.444624 (-52.888031) | 2.243081 / 6.876477 (-4.633396) | 2.340420 / 2.142072 (0.198347) | 0.858010 / 4.805227 (-3.947218) | 0.169165 / 6.500664 (-6.331499) | 0.065177 / 0.075469 (-0.010292) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.297350 / 1.841788 (-0.544438) | 15.404241 / 8.074308 (7.329933) | 13.806039 / 10.191392 (3.614647) | 0.182055 / 0.680424 (-0.498369) | 0.017789 / 0.534201 (-0.516412) | 0.422828 / 0.579283 (-0.156455) | 0.418269 / 0.434364 (-0.016095) | 0.521561 / 0.540337 (-0.018777) | 0.642526 / 1.386936 (-0.744410) |

![](https://cml.dev/watermark.png#0009eea6819c32a888f65b0fdb5889b6d311c436 "CML watermark")

Fix https://github.com/huggingface/datasets/issues/3841


`with_format("numpy")` silently downcasts float64 to float32 features

Next major release

3.0

Hi! This behavior stems from these lines:

https://github.com/huggingface/datasets/blob/b065547654efa0ec633cf373ac1512884c68b2e1/src/datasets/formatting/np_formatter.py#L45-L46

I agree we should preserve the original type whenever possible and downcast explicitly with a warning.

@lhoestq Do you remember why we need this "default dtype" logic in our formatters?

I was also wondering why the default type logic is needed. Me just deleting it is probably too naive of a solution.

Hmm I think the idea was to end up with the usual default precision for deep learning models - no matter how the data was stored or where it comes from.

For example in NLP we store tokens using an optimized low precision to save disk space, but when we set the format to `torch` we actually need to get `int64`. Although the need for a default for integers also comes from numpy not returning the same integer precision depending on your machine. Finally I guess we added a default for floats as well for consistency.

I'm a bit embarrassed by this though, as a user I'd have expected to get the same precision indeed as well and get a zero copy view.

Will you fix this or should I open a PR?

Unfortunately removing it for integers is a breaking change for most `transformers` + `datasets` users for NLP (which is a common case). Removing it for floats is a breaking change for `transformers` + `datasets` for ASR as well. And it also is a breaking change for the other users relying on this behavior.

Therefore I think that the only short term solution is for the user to provide `dtype=` manually and document better this behavior. We could also extend `dtype` to accept a value that means "return the same dtype as the underlying storage" and make it easier to do zero copy.

@lhoestq It should be fine to remove this conversion in Datasets 3.0, no? For now, we can warn the user (with a log message) about the future change when the default type is changed.

Let's see with the transformers team if it sounds reasonable ? We'd have to fix multiple example scripts though.

If it's not ok we can also explore keeping this behavior only for tokens and audio data.

IMO being coupled with Transformers can lead to unexpected behavior when one tries to use our lib without pairing it with Transformers, so I think it's still important to "fix" this, even if it means we will need to update Transformers' example scripts afterward.

Ideally let's update the `transformers` example scripts before the change :P

For others that run into the same issue: A temporary workaround for me is this:
```python
def numpy_transform(batch):
    return {key: np.asarray(val) for key, val in batch.items()}

dataset = dataset.with_transform(numpy_transform)
```

This behavior (silent upcast from `int32` to `int64`) is also unexpected for the user in https://discuss.huggingface.co/t/standard-getitem-returns-wrong-data-type-for-arrays/62470/2

Hi, I stumbled on a variation that upcasts uint8 to int64. I would expect the dtype to be the same as it was when I generated the dataset.

```
import numpy as np
import datasets as ds

foo = np.random.randint(0, 256, size=(5, 10, 10), dtype=np.uint8)

features = ds.Features({"foo": ds.Array2D((10, 10), "uint8")})
dataset = ds.Dataset.from_dict({"foo": foo}, features=features)
dataset.set_format("torch")
print("feature dtype:", dataset.features["foo"].dtype)
print("array dtype:", dataset["foo"].dtype)

# feature dtype: uint8
# array dtype: torch.int64
```

workaround to remove torch upcasting

```
import datasets as ds
import torch

class FixedTorchFormatter(ds.formatting.TorchFormatter):
    def _tensorize(self, value):
        return torch.from_numpy(value)

ds.formatting._register_formatter(FixedTorchFormatter, "torch")
```

### Describe the bug

When I create a dataset with a `float64` feature, then apply numpy formatting the returned numpy arrays are silently downcasted to `float32`.

### Steps to reproduce the bug

```python
import datasets
dataset = datasets.Dataset.from_dict({'a': [1.0, 2.0, 3.0]}).with_format("numpy")
print("feature dtype:", dataset.features['a'].dtype)
print("array dtype:", dataset['a'].dtype)
```
output:
```
feature dtype: float64
array dtype: float32
```

### Expected behavior

```
feature dtype: float64
array dtype: float64
```

### Environment info

- `datasets` version: 2.8.0
- Platform: Linux-5.4.0-135-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 10.0.1
- Pandas version: 1.4.4

### Suggested Fix

Changing [the `_tensorize` function of the numpy formatter](https://github.com/huggingface/datasets/blob/b065547654efa0ec633cf373ac1512884c68b2e1/src/datasets/formatting/np_formatter.py#L32) to
```python
    def _tensorize(self, value):

        if isinstance(value, (str, bytes, type(None))):
            return value
        elif isinstance(value, (np.character, np.ndarray)) and np.issubdtype(value.dtype, np.character):
            return value
        elif isinstance(value, np.number):
            return value

        return np.asarray(value, **self.np_array_kwargs)
```
fixes this particular issue for me. Not sure if this would break other tests. This should also avoid unnecessary copying of the array.


Reload features from Parquet metadata

Thanks a lot for your help @lhoestq. I've simplified what turned out to be a simple fix and added the unit test.

Does this look ready to be merged or is there anything I'm still missing?

Cool ! I think you just need to remove the unused import in `io/parquet.py`
```
src/datasets/io/parquet.py:4:1: F401 'pyarrow as pa' imported but unused
```
and we're good to merge :)

_The documentation is not available anymore as the PR was closed or merged._

> Cool ! I think you just need to remove the unused import in `io/parquet.py`
>
> ```
> src/datasets/io/parquet.py:4:1: F401 'pyarrow as pa' imported but unused
> ```
>
> and we're good to merge :)

Done! Thanks a lot, this was fun :)

Resolves #5482.

Attaches feature metadata to parquet files serialised using `Dataset.to_parquet`.

This allows retrieving data with "rich" feature types (e.g., `datasets.features.image.Image` or `datasets.features.audio.Audio`) from parquet files without cumbersome casting (for an example, see #5482).

@lhoestq It seems that it is sufficient to attach metadata to the schema prior to serialising and features are loaded back with correct types afterwards automatically.

I used the following script to test the implementation:

```python
from pathlib import Path

import datasets

dataset_name = "Maysee/tiny-imagenet"
ds = datasets.load_dataset(dataset_name, split=datasets.Split.TRAIN)
output_directory_path = Path(__file__).parent.joinpath("example_test_outputs", dataset_name.replace("/", "_"))
output_directory_path.mkdir(exist_ok=True, parents=True)
output_filepath = output_directory_path.joinpath("ds.parquet")
ds.to_parquet(str(output_filepath))

reloaded_ds = datasets.load_dataset(str(output_directory_path), split=datasets.Split.TRAIN)

assert ds.features == reloaded_ds.features
```

Prior to the change in this PR this script raises an `AssertionError` and the `Image` features lose their type after serialisation. After the change in this PR, the assertion does not raise an error and manual inspection of the features shows type `Image` for the respective columns of `reloaded_ds `.

Some open questions:
* How/where can I best add new unit tests for this implementation?
* What dataset would I best use in the tests? I chose `Maysee/tiny-imagenet` mainly because it is small and contains an ?Image` feature that can be used to test, but I'd be happy for suggestions on a suitable data source to use.
* Currently I'm calling `datasets.arrow_writer.ArrowWriter._build_metadata` as I need the same logic. However, I'm not happy with the coupling between `datasets.io.parquet` and `datasets.arrow_writer` it leaves me with. Suggest to factor this common logic out into a helper function and reuse it from both of these. Do you agree and if yes, could you please guide me where I would best place this function?

Many thanks in advance and kind regards,
MFreidank


Unify `load_from_cache_file` type and logic

The commit also includes the changes to the `DatasetDict` methods or am I missing something?

Oh, indeed. Feel free to mark the PR as "Ready for review" then.

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010149 / 0.011353 (-0.001204) | 0.005606 / 0.011008 (-0.005402) | 0.103455 / 0.038508 (0.064947) | 0.042934 / 0.023109 (0.019825) | 0.308365 / 0.275898 (0.032467) | 0.394188 / 0.323480 (0.070708) | 0.008760 / 0.007986 (0.000774) | 0.004567 / 0.004328 (0.000239) | 0.077959 / 0.004250 (0.073708) | 0.050115 / 0.037052 (0.013063) | 0.318009 / 0.258489 (0.059520) | 0.358578 / 0.293841 (0.064737) | 0.039231 / 0.128546 (-0.089315) | 0.012381 / 0.075646 (-0.063265) | 0.340046 / 0.419271 (-0.079226) | 0.048366 / 0.043533 (0.004834) | 0.307643 / 0.255139 (0.052504) | 0.342886 / 0.283200 (0.059687) | 0.109628 / 0.141683 (-0.032055) | 1.457297 / 1.452155 (0.005142) | 1.518067 / 1.492716 (0.025351) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.295590 / 0.018006 (0.277584) | 0.531515 / 0.000490 (0.531026) | 0.005677 / 0.000200 (0.005477) | 0.000095 / 0.000054 (0.000041) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030901 / 0.037411 (-0.006511) | 0.118312 / 0.014526 (0.103786) | 0.123146 / 0.176557 (-0.053410) | 0.163608 / 0.737135 (-0.573527) | 0.128604 / 0.296338 (-0.167734) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.404143 / 0.215209 (0.188934) | 4.000118 / 2.077655 (1.922464) | 1.804502 / 1.504120 (0.300382) | 1.597287 / 1.541195 (0.056093) | 1.738512 / 1.468490 (0.270022) | 0.704658 / 4.584777 (-3.880119) | 3.830101 / 3.745712 (0.084389) | 2.186598 / 5.269862 (-3.083263) | 1.367873 / 4.565676 (-3.197804) | 0.085550 / 0.424275 (-0.338725) | 0.012226 / 0.007607 (0.004619) | 0.505760 / 0.226044 (0.279716) | 5.054583 / 2.268929 (2.785655) | 2.284942 / 55.444624 (-53.159682) | 1.961413 / 6.876477 (-4.915064) | 2.059449 / 2.142072 (-0.082623) | 0.845009 / 4.805227 (-3.960218) | 0.167204 / 6.500664 (-6.333460) | 0.065998 / 0.075469 (-0.009471) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.221861 / 1.841788 (-0.619927) | 15.925213 / 8.074308 (7.850905) | 15.359308 / 10.191392 (5.167916) | 0.171776 / 0.680424 (-0.508648) | 0.029234 / 0.534201 (-0.504967) | 0.446349 / 0.579283 (-0.132934) | 0.447873 / 0.434364 (0.013509) | 0.527400 / 0.540337 (-0.012937) | 0.610208 / 1.386936 (-0.776728) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008030 / 0.011353 (-0.003323) | 0.005686 / 0.011008 (-0.005322) | 0.076204 / 0.038508 (0.037696) | 0.037131 / 0.023109 (0.014022) | 0.341461 / 0.275898 (0.065563) | 0.378734 / 0.323480 (0.055255) | 0.006580 / 0.007986 (-0.001406) | 0.004379 / 0.004328 (0.000050) | 0.073983 / 0.004250 (0.069732) | 0.055895 / 0.037052 (0.018842) | 0.342667 / 0.258489 (0.084178) | 0.401464 / 0.293841 (0.107623) | 0.037710 / 0.128546 (-0.090837) | 0.012604 / 0.075646 (-0.063042) | 0.087563 / 0.419271 (-0.331709) | 0.050887 / 0.043533 (0.007354) | 0.333491 / 0.255139 (0.078352) | 0.357437 / 0.283200 (0.074237) | 0.109566 / 0.141683 (-0.032117) | 1.423372 / 1.452155 (-0.028783) | 1.569423 / 1.492716 (0.076706) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.340986 / 0.018006 (0.322980) | 0.530885 / 0.000490 (0.530395) | 0.004172 / 0.000200 (0.003972) | 0.000115 / 0.000054 (0.000060) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030424 / 0.037411 (-0.006987) | 0.121191 / 0.014526 (0.106666) | 0.129066 / 0.176557 (-0.047491) | 0.166938 / 0.737135 (-0.570198) | 0.132000 / 0.296338 (-0.164338) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.418718 / 0.215209 (0.203509) | 4.163973 / 2.077655 (2.086318) | 1.982665 / 1.504120 (0.478545) | 1.798866 / 1.541195 (0.257671) | 1.918867 / 1.468490 (0.450377) | 0.724634 / 4.584777 (-3.860143) | 3.864549 / 3.745712 (0.118837) | 3.697768 / 5.269862 (-1.572093) | 1.983942 / 4.565676 (-2.581735) | 0.086818 / 0.424275 (-0.337457) | 0.012336 / 0.007607 (0.004728) | 0.522314 / 0.226044 (0.296269) | 5.216813 / 2.268929 (2.947884) | 2.516187 / 55.444624 (-52.928437) | 2.172057 / 6.876477 (-4.704420) | 2.342773 / 2.142072 (0.200701) | 0.851805 / 4.805227 (-3.953422) | 0.170139 / 6.500664 (-6.330525) | 0.068494 / 0.075469 (-0.006975) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.307370 / 1.841788 (-0.534418) | 16.737937 / 8.074308 (8.663629) | 14.483384 / 10.191392 (4.291992) | 0.172418 / 0.680424 (-0.508006) | 0.018241 / 0.534201 (-0.515960) | 0.432049 / 0.579283 (-0.147234) | 0.447590 / 0.434364 (0.013227) | 0.550332 / 0.540337 (0.009994) | 0.646756 / 1.386936 (-0.740180) |

![](https://cml.dev/watermark.png#819bc6e9f88459f363e6fb6948e9cbe5c231500d "CML watermark")

* Updating type annotations for  #`load_from_cache_file`
* Added logic for cache checking if needed
* Updated documentation following the wording of `Dataset.map`


Improve inconsistency of `Dataset.map` interface for `load_from_cache_file`

enhancement

New feature or request

Hi, thanks for noticing this! We can't just remove the cache control as this allows us to control where the arrow files generated by the ops are written (cached on disk if enabled or a temporary directory if disabled). The right way to address this inconsistency would be by having `load_from_cache_file=None` by default everywhere.

Hi! Yes, this seems more plausible. I can implement that. One last thing is the type annotation `load_from_cache_file: bool = None`. Which I then would change to `load_from_cache_file: Optional[bool] = None`.

Yes, `Optional[bool]` is the correct type annotation and thanks for the PR.

### Feature request

1. Replace the `load_from_cache_file` default value to `True`.
2. Remove or alter checks from `is_caching_enabled` logic.

### Motivation

I stumbled over an inconsistency in the `Dataset.map` interface. The documentation (and source) states for the parameter `load_from_cache_file`:
```
load_from_cache_file (`bool`, defaults to `True` if caching is enabled):
    If a cache file storing the current computation from `function`
    can be identified, use it instead of recomputing.
```

1. `load_from_cache_file` default value is `None`, while being annotated as `bool`
2. It is inconsistent with other method signatures like `filter`, that have the default value `True`
3. The logic is inconsistent, as the `map` method checks if caching is enabled through `is_caching_enabled`. This logic is not used for other similar methods.

### Your contribution

I am not fully aware of the logic behind caching checks. If this is just a inconsistency that historically grew, I would suggest to remove the `is_caching_enabled` logic as the "default" logic. Maybe someone can give insights, if environment variables have a higher priority than local variables or vice versa.

If this is clarified, I could adjust the source  according to the "Feature request" section of this issue.


Some functions use a param named `type` shouldn't that be avoided since it's a Python reserved name?

Hi! Let's not do this - renaming it would be a breaking change, and going through the deprecation cycle is only worth it if it improves user experience.

Hi @mariosasko, ok it makes sense. Anyway, don't you think it's worth it at some point to start a deprecation cycle e.g. `fs` in `load_from_disk`? It doesn't affect user experience but it's for sure a bad practice IMO, but's up to you 😄 Feel free to close this issue otherwise!

I don't think deprecating a param name in this particular instance is worth the hassle, so I'm closing the issue 🙂.

Hi @mariosasko, @lhoestq, or whoever reads this! :)

After going through `ArrowDataset.set_format` I found out that the `type` param is actually named `type` which is a Python reserved name as you may already know, shouldn't that be renamed to `format_type` before the 3.0.0 is released?

Just wanted to get your input, and if applicable, tackle this issue myself! Thanks 🤗