[docs] Split pattern search order

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007841 / 0.011353 (-0.003512) | 0.005640 / 0.011008 (-0.005368) | 0.096465 / 0.038508 (0.057957) | 0.036476 / 0.023109 (0.013367) | 0.306431 / 0.275898 (0.030533) | 0.339545 / 0.323480 (0.016065) | 0.006064 / 0.007986 (-0.001922) | 0.004404 / 0.004328 (0.000076) | 0.073130 / 0.004250 (0.068879) | 0.052765 / 0.037052 (0.015713) | 0.309895 / 0.258489 (0.051406) | 0.354037 / 0.293841 (0.060196) | 0.037127 / 0.128546 (-0.091420) | 0.012387 / 0.075646 (-0.063260) | 0.333503 / 0.419271 (-0.085769) | 0.059799 / 0.043533 (0.016266) | 0.305496 / 0.255139 (0.050358) | 0.324122 / 0.283200 (0.040922) | 0.107007 / 0.141683 (-0.034676) | 1.416743 / 1.452155 (-0.035411) | 1.520772 / 1.492716 (0.028055) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.261233 / 0.018006 (0.243227) | 0.573806 / 0.000490 (0.573316) | 0.000390 / 0.000200 (0.000190) | 0.000058 / 0.000054 (0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027672 / 0.037411 (-0.009740) | 0.112803 / 0.014526 (0.098278) | 0.121085 / 0.176557 (-0.055471) | 0.176056 / 0.737135 (-0.561080) | 0.127171 / 0.296338 (-0.169167) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.414756 / 0.215209 (0.199547) | 4.148743 / 2.077655 (2.071088) | 1.883940 / 1.504120 (0.379820) | 1.698771 / 1.541195 (0.157576) | 1.811926 / 1.468490 (0.343436) | 0.708293 / 4.584777 (-3.876484) | 3.780456 / 3.745712 (0.034744) | 2.098556 / 5.269862 (-3.171306) | 1.323512 / 4.565676 (-3.242164) | 0.086253 / 0.424275 (-0.338022) | 0.012587 / 0.007607 (0.004980) | 0.514824 / 0.226044 (0.288779) | 5.157415 / 2.268929 (2.888487) | 2.382519 / 55.444624 (-53.062105) | 2.014539 / 6.876477 (-4.861938) | 2.215239 / 2.142072 (0.073166) | 0.847178 / 4.805227 (-3.958049) | 0.170053 / 6.500664 (-6.330611) | 0.066461 / 0.075469 (-0.009008) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.199056 / 1.841788 (-0.642732) | 15.244999 / 8.074308 (7.170691) | 14.661593 / 10.191392 (4.470201) | 0.168855 / 0.680424 (-0.511569) | 0.017889 / 0.534201 (-0.516312) | 0.424961 / 0.579283 (-0.154322) | 0.428632 / 0.434364 (-0.005732) | 0.502680 / 0.540337 (-0.037658) | 0.597827 / 1.386936 (-0.789109) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007749 / 0.011353 (-0.003604) | 0.005527 / 0.011008 (-0.005482) | 0.074774 / 0.038508 (0.036266) | 0.035367 / 0.023109 (0.012258) | 0.340594 / 0.275898 (0.064696) | 0.373970 / 0.323480 (0.050490) | 0.006094 / 0.007986 (-0.001892) | 0.004428 / 0.004328 (0.000100) | 0.074120 / 0.004250 (0.069869) | 0.054852 / 0.037052 (0.017800) | 0.357173 / 0.258489 (0.098684) | 0.388877 / 0.293841 (0.095036) | 0.037002 / 0.128546 (-0.091545) | 0.012337 / 0.075646 (-0.063309) | 0.086962 / 0.419271 (-0.332310) | 0.050370 / 0.043533 (0.006837) | 0.342989 / 0.255139 (0.087850) | 0.358065 / 0.283200 (0.074865) | 0.111063 / 0.141683 (-0.030620) | 1.516704 / 1.452155 (0.064549) | 1.634359 / 1.492716 (0.141643) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.261493 / 0.018006 (0.243487) | 0.566288 / 0.000490 (0.565799) | 0.000439 / 0.000200 (0.000239) | 0.000056 / 0.000054 (0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030426 / 0.037411 (-0.006985) | 0.114606 / 0.014526 (0.100080) | 0.126134 / 0.176557 (-0.050423) | 0.175324 / 0.737135 (-0.561812) | 0.132766 / 0.296338 (-0.163573) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.426785 / 0.215209 (0.211576) | 4.243555 / 2.077655 (2.165900) | 2.089631 / 1.504120 (0.585511) | 1.994562 / 1.541195 (0.453367) | 2.140284 / 1.468490 (0.671794) | 0.698645 / 4.584777 (-3.886132) | 3.807471 / 3.745712 (0.061759) | 3.275343 / 5.269862 (-1.994519) | 1.796756 / 4.565676 (-2.768921) | 0.085986 / 0.424275 (-0.338289) | 0.012213 / 0.007607 (0.004606) | 0.536815 / 0.226044 (0.310771) | 5.344611 / 2.268929 (3.075683) | 2.498578 / 55.444624 (-52.946047) | 2.153260 / 6.876477 (-4.723217) | 2.251310 / 2.142072 (0.109237) | 0.839104 / 4.805227 (-3.966123) | 0.169639 / 6.500664 (-6.331025) | 0.065880 / 0.075469 (-0.009589) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.268610 / 1.841788 (-0.573178) | 15.624915 / 8.074308 (7.550606) | 15.163684 / 10.191392 (4.972292) | 0.172992 / 0.680424 (-0.507432) | 0.018154 / 0.534201 (-0.516047) | 0.440485 / 0.579283 (-0.138798) | 0.431949 / 0.434364 (-0.002415) | 0.547935 / 0.540337 (0.007597) | 0.662442 / 1.386936 (-0.724494) |

![](https://cml.dev/watermark.png#5c8a6ba43c4aaa0ca0665d8dadd87ef33e28e8e4 "CML watermark")

This PR addresses #5681 about the order of split patterns 🤗 Datasets searches for when generating dataset splits.


pyarrow.lib.ArrowInvalid: Unable to merge: Field <field> has incompatible types

> Hi! The link pointing to the code that generated the dataset is broken. Can you please fix it to make debugging easier?

Sorry about that, it's fixed now.

@cyanic-selkie could you explain how you fixed it? I met the same error in loading other datasets, is it due to the version of the library enviroment?

@MingsYang I never fixed it. If you're referring to my comment above, I only meant I fixed the link to my code.

Anyway, I managed to work around the issue by using `streaming` when loading the dataset.

@cyanic-selkie Emm, I get it. I just tried to use a new version python enviroment, and it show no errors anymore.

Upgrade pyarrow to the latest version solves this problem in my case.

### Describe the bug

When loading the dataset [wikianc-en](https://huggingface.co/datasets/cyanic-selkie/wikianc-en) which I created using [this](https://github.com/cyanic-selkie/wikianc) code, I get the following error:

```
Traceback (most recent call last):
  File "/home/sven/code/rector/answer-detection/train.py", line 106, in <module>
    (dataset, weights) = get_dataset(args.dataset, tokenizer, labels, args.padding)
  File "/home/sven/code/rector/answer-detection/dataset.py", line 106, in get_dataset
    dataset = load_dataset("cyanic-selkie/wikianc-en")
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/load.py", line 1794, in load_dataset
    ds = builder_instance.as_dataset(split=split, verification_mode=verification_mode, in_memory=keep_in_memory)
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/builder.py", line 1106, in as_dataset
    datasets = map_nested(
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 443, in map_nested
    mapped = [
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 444, in <listcomp>
    _single_map_nested((function, obj, types, None, True, None))
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 346, in _single_map_nested
    return function(data_struct)
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/builder.py", line 1136, in _build_single_dataset
    ds = self._as_dataset(
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/builder.py", line 1207, in _as_dataset
    dataset_kwargs = ArrowReader(cache_dir, self.info).read(
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/arrow_reader.py", line 239, in read
    return self.read_files(files=files, original_instructions=instructions, in_memory=in_memory)
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/arrow_reader.py", line 260, in read_files
    pa_table = self._read_files(files, in_memory=in_memory)
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/arrow_reader.py", line 203, in _read_files
    pa_table = concat_tables(pa_tables) if len(pa_tables) != 1 else pa_tables[0]
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/table.py", line 1808, in concat_tables
    return ConcatenationTable.from_tables(tables, axis=axis)
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/table.py", line 1514, in from_tables
    return cls.from_blocks(blocks)
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/table.py", line 1427, in from_blocks
    table = cls._concat_blocks(blocks, axis=0)
  File "/home/sven/.cache/pypoetry/virtualenvs/rector-Z2mdKRnn-py3.10/lib/python3.10/site-packages/datasets/table.py", line 1373, in _concat_blocks
    return pa.concat_tables(pa_tables, promote=True)
  File "pyarrow/table.pxi", line 5224, in pyarrow.lib.concat_tables
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Unable to merge: Field paragraph_anchors has incompatible types: list<: struct<start: uint32 not null, end: uint32 not null, qid: uint32, pageid: uint32, title: string not null> not null> vs list<item: struct<start: uint32, end: uint32, qid: uint32, pageid: uint32, title: string>>
```

This only happens when I load the `train` split, indicating that the size of the dataset is the deciding factor.

### Steps to reproduce the bug

```python
from datasets import load_dataset

dataset = load_dataset("cyanic-selkie/wikianc-en", split="train")
```

### Expected behavior

The dataset should load normally without any errors.

### Environment info

- `datasets` version: 2.10.1
- Platform: Linux-6.2.8-arch1-1-x86_64-with-glibc2.37
- Python version: 3.10.10
- PyArrow version: 11.0.0
- Pandas version: 1.5.3


[docs] Compress data files

_The documentation is not available anymore as the PR was closed or merged._

[Confirmed](https://huggingface.slack.com/archives/C02EMARJ65P/p1680541667004199) with the Hub team the file size limit for the Hugging Face Hub is 10MB :)

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006789 / 0.011353 (-0.004564) | 0.004935 / 0.011008 (-0.006073) | 0.096796 / 0.038508 (0.058288) | 0.032485 / 0.023109 (0.009376) | 0.335342 / 0.275898 (0.059444) | 0.354999 / 0.323480 (0.031519) | 0.005467 / 0.007986 (-0.002519) | 0.005267 / 0.004328 (0.000939) | 0.073988 / 0.004250 (0.069737) | 0.044402 / 0.037052 (0.007350) | 0.331156 / 0.258489 (0.072666) | 0.363595 / 0.293841 (0.069754) | 0.035301 / 0.128546 (-0.093245) | 0.012141 / 0.075646 (-0.063505) | 0.333164 / 0.419271 (-0.086107) | 0.048818 / 0.043533 (0.005286) | 0.331458 / 0.255139 (0.076319) | 0.343567 / 0.283200 (0.060367) | 0.094963 / 0.141683 (-0.046720) | 1.444383 / 1.452155 (-0.007772) | 1.520093 / 1.492716 (0.027377) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.212311 / 0.018006 (0.194305) | 0.436413 / 0.000490 (0.435923) | 0.000333 / 0.000200 (0.000133) | 0.000057 / 0.000054 (0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026670 / 0.037411 (-0.010742) | 0.105774 / 0.014526 (0.091248) | 0.115796 / 0.176557 (-0.060760) | 0.176504 / 0.737135 (-0.560631) | 0.121883 / 0.296338 (-0.174456) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.400783 / 0.215209 (0.185574) | 4.006608 / 2.077655 (1.928953) | 1.817659 / 1.504120 (0.313539) | 1.619777 / 1.541195 (0.078582) | 1.684247 / 1.468490 (0.215757) | 0.701116 / 4.584777 (-3.883661) | 3.684056 / 3.745712 (-0.061656) | 2.065258 / 5.269862 (-3.204603) | 1.425460 / 4.565676 (-3.140217) | 0.084519 / 0.424275 (-0.339757) | 0.011949 / 0.007607 (0.004342) | 0.496793 / 0.226044 (0.270749) | 4.978864 / 2.268929 (2.709935) | 2.303388 / 55.444624 (-53.141237) | 1.978341 / 6.876477 (-4.898135) | 2.055744 / 2.142072 (-0.086329) | 0.832022 / 4.805227 (-3.973206) | 0.164715 / 6.500664 (-6.335949) | 0.062701 / 0.075469 (-0.012768) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.178723 / 1.841788 (-0.663065) | 14.583986 / 8.074308 (6.509678) | 14.189402 / 10.191392 (3.998010) | 0.183867 / 0.680424 (-0.496557) | 0.017565 / 0.534201 (-0.516636) | 0.421345 / 0.579283 (-0.157938) | 0.420235 / 0.434364 (-0.014129) | 0.496758 / 0.540337 (-0.043580) | 0.591558 / 1.386936 (-0.795378) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007019 / 0.011353 (-0.004334) | 0.004996 / 0.011008 (-0.006012) | 0.073345 / 0.038508 (0.034836) | 0.033077 / 0.023109 (0.009968) | 0.335954 / 0.275898 (0.060056) | 0.372616 / 0.323480 (0.049136) | 0.005678 / 0.007986 (-0.002308) | 0.003906 / 0.004328 (-0.000423) | 0.072841 / 0.004250 (0.068591) | 0.046829 / 0.037052 (0.009777) | 0.335177 / 0.258489 (0.076688) | 0.382862 / 0.293841 (0.089021) | 0.038406 / 0.128546 (-0.090141) | 0.012110 / 0.075646 (-0.063536) | 0.085796 / 0.419271 (-0.333476) | 0.049896 / 0.043533 (0.006363) | 0.338232 / 0.255139 (0.083093) | 0.361054 / 0.283200 (0.077855) | 0.103171 / 0.141683 (-0.038512) | 1.556692 / 1.452155 (0.104538) | 1.540023 / 1.492716 (0.047306) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.223705 / 0.018006 (0.205699) | 0.438771 / 0.000490 (0.438282) | 0.002838 / 0.000200 (0.002639) | 0.000081 / 0.000054 (0.000026) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028423 / 0.037411 (-0.008988) | 0.110560 / 0.014526 (0.096035) | 0.121629 / 0.176557 (-0.054928) | 0.173638 / 0.737135 (-0.563498) | 0.127062 / 0.296338 (-0.169277) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.425806 / 0.215209 (0.210597) | 4.251051 / 2.077655 (2.173397) | 2.059735 / 1.504120 (0.555615) | 1.864886 / 1.541195 (0.323692) | 1.941553 / 1.468490 (0.473063) | 0.700084 / 4.584777 (-3.884693) | 3.753150 / 3.745712 (0.007438) | 3.218606 / 5.269862 (-2.051256) | 1.439648 / 4.565676 (-3.126028) | 0.085239 / 0.424275 (-0.339037) | 0.012026 / 0.007607 (0.004419) | 0.521564 / 0.226044 (0.295520) | 5.217902 / 2.268929 (2.948973) | 2.557831 / 55.444624 (-52.886793) | 2.240223 / 6.876477 (-4.636254) | 2.364664 / 2.142072 (0.222591) | 0.825884 / 4.805227 (-3.979343) | 0.167800 / 6.500664 (-6.332864) | 0.063552 / 0.075469 (-0.011917) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.255532 / 1.841788 (-0.586256) | 14.747783 / 8.074308 (6.673475) | 14.352263 / 10.191392 (4.160871) | 0.143659 / 0.680424 (-0.536765) | 0.017517 / 0.534201 (-0.516684) | 0.419863 / 0.579283 (-0.159421) | 0.416674 / 0.434364 (-0.017690) | 0.485694 / 0.540337 (-0.054643) | 0.584810 / 1.386936 (-0.802126) |

![](https://cml.dev/watermark.png#61db0e9c936bc67c18b37b0960e2f0bb1f8ffdcd "CML watermark")

This PR addresses the comments in #5687 about compressing text file extensions before uploading to the Hub. Also clarified what "too large" means based on the GitLFS [docs](https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-git-large-file-storage).


Support streaming Beam datasets from HF GCS preprocessed data

```python
In [1]: from datasets import load_dataset

In [2]: ds = load_dataset("wikipedia", "20220301.en", split="train", streaming=True); item = next(iter(ds)); item
Out[2]:
{'id': '12',
 'url': 'https://en.wikipedia.org/wiki/Anarchism',
 'title': 'Anarchism',
 'text': 'Anarchism is a political philosophy and movement that is sceptical of authority and rejects all involuntary, coercive forms of hierarchy. Anarchism calls for the abolition of the state, which it holds to be unnecessary, undesirable, and harmful. As a historically left-wing movement, placed on the farthest left of the political spectrum, it is usually described alongside communalism and libertarian Marxism as the libertarian wing (libertarian socialism) of the socialist movement,...}
```

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007859 / 0.011353 (-0.003493) | 0.005129 / 0.011008 (-0.005879) | 0.098070 / 0.038508 (0.059562) | 0.036500 / 0.023109 (0.013391) | 0.311575 / 0.275898 (0.035677) | 0.338351 / 0.323480 (0.014872) | 0.005962 / 0.007986 (-0.002024) | 0.004060 / 0.004328 (-0.000268) | 0.072970 / 0.004250 (0.068719) | 0.049289 / 0.037052 (0.012237) | 0.310303 / 0.258489 (0.051814) | 0.347449 / 0.293841 (0.053608) | 0.046912 / 0.128546 (-0.081634) | 0.011952 / 0.075646 (-0.063694) | 0.333600 / 0.419271 (-0.085671) | 0.052700 / 0.043533 (0.009167) | 0.325486 / 0.255139 (0.070347) | 0.326920 / 0.283200 (0.043720) | 0.107683 / 0.141683 (-0.034000) | 1.416679 / 1.452155 (-0.035476) | 1.502418 / 1.492716 (0.009702) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.216520 / 0.018006 (0.198514) | 0.448450 / 0.000490 (0.447960) | 0.004213 / 0.000200 (0.004013) | 0.000082 / 0.000054 (0.000028) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027081 / 0.037411 (-0.010331) | 0.110989 / 0.014526 (0.096463) | 0.116087 / 0.176557 (-0.060470) | 0.173771 / 0.737135 (-0.563364) | 0.121240 / 0.296338 (-0.175099) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.399938 / 0.215209 (0.184729) | 4.017665 / 2.077655 (1.940010) | 1.782327 / 1.504120 (0.278207) | 1.612955 / 1.541195 (0.071761) | 1.698839 / 1.468490 (0.230349) | 0.706702 / 4.584777 (-3.878075) | 4.533425 / 3.745712 (0.787713) | 2.102611 / 5.269862 (-3.167250) | 1.461429 / 4.565676 (-3.104248) | 0.085719 / 0.424275 (-0.338556) | 0.012104 / 0.007607 (0.004497) | 0.507397 / 0.226044 (0.281352) | 5.061572 / 2.268929 (2.792643) | 2.272106 / 55.444624 (-53.172518) | 1.935575 / 6.876477 (-4.940901) | 2.102541 / 2.142072 (-0.039532) | 0.838395 / 4.805227 (-3.966832) | 0.168573 / 6.500664 (-6.332091) | 0.064234 / 0.075469 (-0.011235) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.190077 / 1.841788 (-0.651710) | 15.765587 / 8.074308 (7.691279) | 14.694626 / 10.191392 (4.503234) | 0.142912 / 0.680424 (-0.537512) | 0.017669 / 0.534201 (-0.516532) | 0.421502 / 0.579283 (-0.157781) | 0.452732 / 0.434364 (0.018368) | 0.497480 / 0.540337 (-0.042857) | 0.586310 / 1.386936 (-0.800626) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007629 / 0.011353 (-0.003724) | 0.005330 / 0.011008 (-0.005679) | 0.076366 / 0.038508 (0.037858) | 0.034703 / 0.023109 (0.011593) | 0.356300 / 0.275898 (0.080402) | 0.392909 / 0.323480 (0.069429) | 0.005959 / 0.007986 (-0.002026) | 0.004140 / 0.004328 (-0.000188) | 0.075289 / 0.004250 (0.071039) | 0.047880 / 0.037052 (0.010828) | 0.357289 / 0.258489 (0.098800) | 0.404554 / 0.293841 (0.110714) | 0.037182 / 0.128546 (-0.091365) | 0.012266 / 0.075646 (-0.063380) | 0.088554 / 0.419271 (-0.330718) | 0.049698 / 0.043533 (0.006165) | 0.353453 / 0.255139 (0.098314) | 0.373252 / 0.283200 (0.090052) | 0.101892 / 0.141683 (-0.039791) | 1.481534 / 1.452155 (0.029380) | 1.553818 / 1.492716 (0.061102) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.229891 / 0.018006 (0.211884) | 0.452444 / 0.000490 (0.451954) | 0.000434 / 0.000200 (0.000234) | 0.000058 / 0.000054 (0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030170 / 0.037411 (-0.007241) | 0.115097 / 0.014526 (0.100571) | 0.122094 / 0.176557 (-0.054463) | 0.171352 / 0.737135 (-0.565784) | 0.128441 / 0.296338 (-0.167898) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.428347 / 0.215209 (0.213138) | 4.266243 / 2.077655 (2.188588) | 2.148327 / 1.504120 (0.644207) | 1.874141 / 1.541195 (0.332946) | 1.968737 / 1.468490 (0.500246) | 0.715320 / 4.584777 (-3.869457) | 4.166097 / 3.745712 (0.420384) | 2.169550 / 5.269862 (-3.100312) | 1.377441 / 4.565676 (-3.188236) | 0.086376 / 0.424275 (-0.337899) | 0.012018 / 0.007607 (0.004411) | 0.517433 / 0.226044 (0.291388) | 5.167327 / 2.268929 (2.898398) | 2.545822 / 55.444624 (-52.898803) | 2.241726 / 6.876477 (-4.634751) | 2.327220 / 2.142072 (0.185147) | 0.841618 / 4.805227 (-3.963609) | 0.169473 / 6.500664 (-6.331191) | 0.065505 / 0.075469 (-0.009964) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.270476 / 1.841788 (-0.571312) | 17.049885 / 8.074308 (8.975577) | 14.847615 / 10.191392 (4.656223) | 0.168671 / 0.680424 (-0.511753) | 0.017564 / 0.534201 (-0.516637) | 0.424780 / 0.579283 (-0.154503) | 0.517392 / 0.434364 (0.083028) | 0.561197 / 0.540337 (0.020859) | 0.697792 / 1.386936 (-0.689144) |

![](https://cml.dev/watermark.png#ce06edf0afb70027ffbd3c2ddec5d28037e9bd31 "CML watermark")

This PR implements streaming Apache Beam datasets that are already preprocessed by us and stored in the HF Google Cloud Storage:
- natural_questions
- wiki40b
- wikipedia

This is done by streaming from the prepared Arrow files in HF Google Cloud Storage.

This will fix their corresponding dataset viewers. Related to:
- https://github.com/huggingface/datasets-server/pull/988#discussion_r1150767138

Related to:
- https://huggingface.co/datasets/natural_questions/discussions/4
- https://huggingface.co/datasets/wiki40b/discussions/2
- https://huggingface.co/datasets/wikipedia/discussions/9

CC: @severo


bug

Something isn't working

Hi @wccccp, thanks for reporting.
That's weird since `huggingface_hub` _has_ a module called `hf_api` and you are using a recent version of it.

Which version of `datasets` are you using? And is it a bug that you experienced only recently? (cc @lhoestq can it be somehow related to the recent release of `datasets`?)

~@wccccp what I can suggest you is to uninstall and reinstall completely huggingface_hub and datasets? My first guess is that there is a discrepancy somewhere in your setup 😕~

@wccccp Actually I have also been able to reproduce the error so it's not an issue with your setup.

@huggingface/datasets I found this issue quite weird. Is this a module that is not used very often?
The problematic line is [this one](https://github.com/huggingface/datasets/blame/c33e8ce68b5000988bf6b2e4bca27ffaa469acea/src/datasets/data_files.py#L476) where `huggingface_hub.hf_api.DatasetInfo` is used. `huggingface_hub` is imported [here](https://github.com/huggingface/datasets/blame/c33e8ce68b5000988bf6b2e4bca27ffaa469acea/src/datasets/data_files.py#L6) as `import huggingface_hub`. However since modules are lazy-loaded in `hfh` you need to explicitly import them (i.e. `import huggingface_hub.hf_api`).

What's weird is that nothing has changed for months. Datasets code seems that it didn't change for 2 years when I git-blame this part. And lazy-loading was introduced 1 year ago in `huggingface_hub`. Could it be that `data_files.py` is a file almost never used?

For context, I tried to run `import huggingface_hub; huggingface_hub.hf_api.DatasetInfo` in the terminal with different versions of `hfh` and I need to go back to `huggingface_hub==0.7.0` to make it work (latest is 0.13.3).

Before the error happens at line 120 in `data_files.py`, `datasets.filesystems.hffilesystem` is imported at the top of `data_files.py` and this file does `from huggingface_hub.hf_api import DatasetInfo` - so `huggingface_hub.hf_api` is imported. Not sure how the error could happen, what version of `datasets` are you using @wccccp ?

### Describe the bug

rta.sh
Traceback (most recent call last):
  File "run.py", line 7, in <module>
    import datasets
  File "/home/appuser/miniconda3/envs/pt2/lib/python3.8/site-packages/datasets/__init__.py", line 37, in <module>
    from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
  File "/home/appuser/miniconda3/envs/pt2/lib/python3.8/site-packages/datasets/builder.py", line 44, in <module>
    from .data_files import DataFilesDict, _sanitize_patterns
  File "/home/appuser/miniconda3/envs/pt2/lib/python3.8/site-packages/datasets/data_files.py", line 120, in <module>
    dataset_info: huggingface_hub.hf_api.DatasetInfo,
  File "/home/appuser/miniconda3/envs/pt2/lib/python3.8/site-packages/huggingface_hub/__init__.py", line 290, in __getattr__
    raise AttributeError(f"No {package_name} attribute {name}")
AttributeError: No huggingface_hub attribute hf_api

### Reproduction

_No response_

### Logs

```shell
Traceback (most recent call last):
  File "run.py", line 7, in <module>
    import datasets
  File "/home/appuser/miniconda3/envs/pt2/lib/python3.8/site-packages/datasets/__init__.py", line 37, in <module>
    from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
  File "/home/appuser/miniconda3/envs/pt2/lib/python3.8/site-packages/datasets/builder.py", line 44, in <module>
    from .data_files import DataFilesDict, _sanitize_patterns
  File "/home/appuser/miniconda3/envs/pt2/lib/python3.8/site-packages/datasets/data_files.py", line 120, in <module>
    dataset_info: huggingface_hub.hf_api.DatasetInfo,
  File "/home/appuser/miniconda3/envs/pt2/lib/python3.8/site-packages/huggingface_hub/__init__.py", line 290, in __getattr__
    raise AttributeError(f"No {package_name} attribute {name}")
AttributeError: No huggingface_hub attribute hf_api
```

### System info

```shell
- huggingface_hub version: 0.13.2
- Platform: Linux-5.4.0-144-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Token path ?: /home/appuser/.cache/huggingface/token
- Has saved token ?: False
- Configured git credential helpers:
- FastAI: N/A
- Tensorflow: N/A
- Torch: 1.7.1
- Jinja2: N/A
- Graphviz: N/A
- Pydot: N/A
- Pillow: 9.3.0
- hf_transfer: N/A
- ENDPOINT: https://huggingface.co
- HUGGINGFACE_HUB_CACHE: /home/appuser/.cache/huggingface/hub
- HUGGINGFACE_ASSETS_CACHE: /home/appuser/.cache/huggingface/assets
- HF_TOKEN_PATH: /home/appuser/.cache/huggingface/token
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
```


Wikipedia download_and_prepare for GCS

Hi @adrianfagerland, thanks for reporting.

Please note that "wikipedia" is a special dataset, with an Apache Beam builder: https://beam.apache.org/
You can find more info about Beam datasets in our docs: https://huggingface.co/docs/datasets/beam

It was implemented to be run in parallel processing, using one of the distributed back-ends supported by Apache Beam: https://beam.apache.org/get-started/beam-overview/#apache-beam-pipeline-runners

That is, you are trying to process the source wikipedia data on your machine (not distributed) when passing `beam_runner="DirectRunner"`.

As documented in the wikipedia dataset page (https://huggingface.co/datasets/wikipedia):

    Some subsets of Wikipedia have already been processed by HuggingFace, and you can load them just with:

        from datasets import load_dataset

        load_dataset("wikipedia", "20220301.en")

    The list of pre-processed subsets is:
    - "20220301.de"
    - "20220301.en"
    - "20220301.fr"
    - "20220301.frr"
    - "20220301.it"
    - "20220301.simple"

To download the available processed data (in Arrow format):
```python
builder = datasets.load_dataset_builder("wikipedia", "20220301.en")
builder.download_and_prepare(your_path)
```

When running this using :
```
import datasets
from apache_beam.options.pipeline_options import PipelineOptions
from gcsfs import GCSFileSystem

storage_options = {"project":"tdt4310", "token":"cloud"}
fs = GCSFileSystem(**storage_options)

output_dir = "gcs://quiz_transformer/"
beam_options = PipelineOptions(
        region="europe-west4",
        project="tdt4310",
        temp_location=output_dir+"tmp/")

builder = datasets.load_dataset_builder("wikipedia", "20220301.en", beam_runner="dataflow", beam_options=beam_options)
builder.download_and_prepare(
        output_dir, storage_options=storage_options, file_format="parquet")
```
I now get this error:
```
raise FileNotFoundError(f"Couldn't find file at {url}")
FileNotFoundError: Couldn't find file at https://dumps.wikimedia.org/enwiki/20220301/dumpstatus.json
Downloading data files:   0%|                                                            | 0/1 [00:00<?, ?it/s]
```

I get the same error for this:
```
import datasets
from gcsfs import GCSFileSystem

storage_options = {"project":"tdt4310", "token":"cloud"}
fs = GCSFileSystem(**storage_options)

output_dir = "gcs://quiz_transformer/"
builder = datasets.load_dataset_builder("wikipedia", "20220301.en")
builder.download_and_prepare(
        output_dir, storage_options=storage_options, file_format="parquet")
```

`wikipedia` is no longer a Beam dataset, so the above code should work now.

PS: You can use [these files](https://huggingface.co/datasets/wikipedia/tree/main/data/20220301.en) (or a newer dump at https://huggingface.co/datasets/wikimedia/wikipedia/tree/main/20231101.en) instead of generating the Parquet version yourself

### Describe the bug

I am unable to download the wikipedia dataset onto GCS.

When I run the script provided the memory firstly gets eaten up, then it crashes.
I tried running this on a VM with 128GB RAM and all I got was a two empty files: _data_builder.lock_, _data.incomplete/beam-temp-wikipedia-train-1ab2039acf3611ed87a9893475de0093_

I have troubleshot this for two straight days now, but I am just unable to get the dataset into storage.

### Steps to reproduce the bug

Run this and insert a path:
```
import datasets

builder = datasets.load_dataset_builder(
    "wikipedia", language="en", date="20230320", beam_runner="DirectRunner")

builder.download_and_prepare({path}, file_format="parquet")
```
This is where the problem of it eating RAM occurs.

I have also tried several versions of this, based on the docs:
```
import gcsfs
import datasets

storage_options = {"project": "tdt4310", "token": "cloud"}
fs = gcsfs.GCSFileSystem(**storage_options)

output_dir = "gcs://wikipediadata/"
builder = datasets.load_dataset_builder(
    "wikipedia", date="20230320", language="en",  beam_runner="DirectRunner")
builder.download_and_prepare(
    output_dir, storage_options=storage_options, file_format="parquet")
```
The error message that is received here is:

> ValueError: Unable to get filesystem from specified path, please use the correct path or ensure the required dependency is installed, e.g., pip install apache-beam[gcp]. Path specified: gcs://wikipediadata/wikipedia-train [while running 'train/Save to parquet/Write/WriteImpl/InitializeWrite']

I have ran `pip install apache-beam[gcp]`

### Expected behavior

The wikipedia data loaded into GCS

Everything worked when testing with a smaller demo dataset found somewhere in the docs

### Environment info

Newest published version of datasets. Python 3.9. Also tested with Python 3.7. 128GB RAM Google Cloud VM instance.


Document to compress data files before uploading

documentation

Improvements or additions to documentation

Great idea!

Should we also take this opportunity to include some audio/image file formats? Currently, it still reads very text heavy. Something like:

> We support many text, audio, and image data extensions such as `.zip`, `.rar`, `.mp3`, and `.jpg` among many others. For data extensions like `.csv`, `.json`, `.jsonl`, and `txt`, we recommend compressing them before uploading to the Hub. These file extensions are not tracked by Git LFS by default, and if they're too large, they will not be committed and uploaded. Take a look at the `.gitattributes` file in your repository for a complete list of supported file extensions.

Hi @stevhliu, thanks for your suggestion.

I agree it is a good opportunity to mention that audio/image file formats are also supported.

Nit:
I would not mention .zip, .rar after "text, audio, and image data extensions". Those are "compression" extensions and not "text, audio, and image data extensions".

What about something similar to:
> We support many text, audio, and image data extensions such as `.csv`, `.mp3`, and `.jpg` among many others. For text data extensions like `.csv`, `.json`, `.jsonl`, and `.txt`, we recommend compressing them before uploading to the Hub (to `.zip` or `.gz` file extension for example).
>
> Note that text file extensions are not tracked by Git LFS by default, and if they're too large, they will not be committed and uploaded. Take a look at the `.gitattributes` file in your repository for a complete list of tracked file extensions by default.

Note that for compressions I have mentioned:
- gz, to compress individual files
- zip, to compress and archive multiple files; zip is preferred rather than tar because it supports streaming out of the box

Perfect, thanks for making the distinction between compression and data extensions!

In our docs to [Share a dataset to the Hub](https://huggingface.co/docs/datasets/upload_dataset), we tell users to upload directly their data files, like CSV, JSON, JSON-Lines, text,... However, these extensions are not tracked by Git LFS by default, as they are not in the `.giattributes` file. Therefore, if they are too large, Git will fail to commit/upload them.

I think for those file extensions (.csv, .json, .jsonl, .txt), we should better recommend to **compress** their data files (using ZIP for example) before uploading them to the Hub.
- Compressed files are tracked by Git LFS in our default `.gitattributes` file

What do you think?
CC: @stevhliu

See related issue:
- https://huggingface.co/datasets/tcor0005/langchain-docs-400-chunksize/discussions/1


set dev version

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5686). All of your documentation changes will be reflected on that endpoint.

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008460 / 0.011353 (-0.002893) | 0.006114 / 0.011008 (-0.004894) | 0.121496 / 0.038508 (0.082987) | 0.035030 / 0.023109 (0.011920) | 0.397778 / 0.275898 (0.121880) | 0.429020 / 0.323480 (0.105540) | 0.007811 / 0.007986 (-0.000174) | 0.006269 / 0.004328 (0.001940) | 0.098895 / 0.004250 (0.094645) | 0.045407 / 0.037052 (0.008355) | 0.413679 / 0.258489 (0.155189) | 0.437491 / 0.293841 (0.143650) | 0.053207 / 0.128546 (-0.075339) | 0.018471 / 0.075646 (-0.057175) | 0.414800 / 0.419271 (-0.004472) | 0.060864 / 0.043533 (0.017332) | 0.398501 / 0.255139 (0.143362) | 0.421142 / 0.283200 (0.137942) | 0.114908 / 0.141683 (-0.026775) | 1.678630 / 1.452155 (0.226475) | 1.782313 / 1.492716 (0.289596) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.280783 / 0.018006 (0.262777) | 0.591573 / 0.000490 (0.591083) | 0.005797 / 0.000200 (0.005597) | 0.000115 / 0.000054 (0.000060) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030431 / 0.037411 (-0.006981) | 0.117342 / 0.014526 (0.102816) | 0.128456 / 0.176557 (-0.048101) | 0.198782 / 0.737135 (-0.538354) | 0.128501 / 0.296338 (-0.167838) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.603073 / 0.215209 (0.387864) | 6.101354 / 2.077655 (4.023699) | 2.527812 / 1.504120 (1.023692) | 2.101468 / 1.541195 (0.560273) | 2.092813 / 1.468490 (0.624323) | 1.182150 / 4.584777 (-3.402627) | 5.389278 / 3.745712 (1.643566) | 5.041001 / 5.269862 (-0.228860) | 2.650581 / 4.565676 (-1.915095) | 0.138761 / 0.424275 (-0.285514) | 0.014209 / 0.007607 (0.006602) | 0.748596 / 0.226044 (0.522552) | 7.373937 / 2.268929 (5.105008) | 3.245882 / 55.444624 (-52.198742) | 2.523569 / 6.876477 (-4.352908) | 2.581343 / 2.142072 (0.439270) | 1.340436 / 4.805227 (-3.464791) | 0.241388 / 6.500664 (-6.259276) | 0.076634 / 0.075469 (0.001164) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.480237 / 1.841788 (-0.361551) | 16.781338 / 8.074308 (8.707030) | 19.735028 / 10.191392 (9.543636) | 0.256872 / 0.680424 (-0.423551) | 0.029211 / 0.534201 (-0.504990) | 0.503292 / 0.579283 (-0.075991) | 0.584510 / 0.434364 (0.150146) | 0.580293 / 0.540337 (0.039955) | 0.678863 / 1.386936 (-0.708073) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009972 / 0.011353 (-0.001381) | 0.006107 / 0.011008 (-0.004902) | 0.096188 / 0.038508 (0.057680) | 0.033320 / 0.023109 (0.010210) | 0.420789 / 0.275898 (0.144891) | 0.460488 / 0.323480 (0.137008) | 0.006492 / 0.007986 (-0.001493) | 0.005325 / 0.004328 (0.000997) | 0.094974 / 0.004250 (0.090723) | 0.047708 / 0.037052 (0.010655) | 0.426689 / 0.258489 (0.168200) | 0.476440 / 0.293841 (0.182599) | 0.052776 / 0.128546 (-0.075770) | 0.018779 / 0.075646 (-0.056868) | 0.119598 / 0.419271 (-0.299673) | 0.061800 / 0.043533 (0.018267) | 0.421305 / 0.255139 (0.166166) | 0.441125 / 0.283200 (0.157925) | 0.114221 / 0.141683 (-0.027462) | 1.712681 / 1.452155 (0.260526) | 1.852316 / 1.492716 (0.359600) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.272412 / 0.018006 (0.254405) | 0.583996 / 0.000490 (0.583506) | 0.000505 / 0.000200 (0.000305) | 0.000077 / 0.000054 (0.000022) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029553 / 0.037411 (-0.007858) | 0.124921 / 0.014526 (0.110395) | 0.133338 / 0.176557 (-0.043218) | 0.193811 / 0.737135 (-0.543325) | 0.147973 / 0.296338 (-0.148365) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.595241 / 0.215209 (0.380032) | 6.012015 / 2.077655 (3.934360) | 2.611295 / 1.504120 (1.107175) | 2.290127 / 1.541195 (0.748932) | 2.300366 / 1.468490 (0.831876) | 1.197602 / 4.584777 (-3.387175) | 5.439064 / 3.745712 (1.693352) | 2.906088 / 5.269862 (-2.363773) | 1.919183 / 4.565676 (-2.646493) | 0.132166 / 0.424275 (-0.292109) | 0.014544 / 0.007607 (0.006937) | 0.726377 / 0.226044 (0.500333) | 7.361023 / 2.268929 (5.092094) | 3.289266 / 55.444624 (-52.155358) | 2.635570 / 6.876477 (-4.240907) | 2.595691 / 2.142072 (0.453619) | 1.329458 / 4.805227 (-3.475769) | 0.239419 / 6.500664 (-6.261245) | 0.076316 / 0.075469 (0.000847) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.547616 / 1.841788 (-0.294172) | 17.374315 / 8.074308 (9.300007) | 20.216275 / 10.191392 (10.024883) | 0.252102 / 0.680424 (-0.428322) | 0.027535 / 0.534201 (-0.506665) | 0.524618 / 0.579283 (-0.054666) | 0.596803 / 0.434364 (0.162439) | 0.652632 / 0.540337 (0.112294) | 0.762272 / 1.386936 (-0.624664) |

![](https://cml.dev/watermark.png#8c7d4b2f981f8cf639dcbd80f40a41aa5b1693c6 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008236 / 0.011353 (-0.003117) | 0.006186 / 0.011008 (-0.004822) | 0.117852 / 0.038508 (0.079344) | 0.034711 / 0.023109 (0.011602) | 0.447564 / 0.275898 (0.171666) | 0.438727 / 0.323480 (0.115247) | 0.006576 / 0.007986 (-0.001410) | 0.005903 / 0.004328 (0.001574) | 0.094309 / 0.004250 (0.090059) | 0.042760 / 0.037052 (0.005708) | 0.393269 / 0.258489 (0.134780) | 0.438061 / 0.293841 (0.144220) | 0.059029 / 0.128546 (-0.069517) | 0.020296 / 0.075646 (-0.055350) | 0.412057 / 0.419271 (-0.007215) | 0.059808 / 0.043533 (0.016275) | 0.407243 / 0.255139 (0.152104) | 0.414290 / 0.283200 (0.131090) | 0.107701 / 0.141683 (-0.033981) | 1.671522 / 1.452155 (0.219367) | 1.775055 / 1.492716 (0.282338) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.275242 / 0.018006 (0.257236) | 0.599698 / 0.000490 (0.599208) | 0.001289 / 0.000200 (0.001089) | 0.000101 / 0.000054 (0.000046) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029579 / 0.037411 (-0.007832) | 0.127249 / 0.014526 (0.112723) | 0.137431 / 0.176557 (-0.039126) | 0.220330 / 0.737135 (-0.516805) | 0.133540 / 0.296338 (-0.162798) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.571989 / 0.215209 (0.356780) | 5.931503 / 2.077655 (3.853848) | 2.526646 / 1.504120 (1.022527) | 2.189476 / 1.541195 (0.648281) | 2.151935 / 1.468490 (0.683444) | 1.242440 / 4.584777 (-3.342337) | 5.599675 / 3.745712 (1.853963) | 3.242035 / 5.269862 (-2.027826) | 2.368361 / 4.565676 (-2.197315) | 0.145659 / 0.424275 (-0.278616) | 0.013813 / 0.007607 (0.006206) | 0.782495 / 0.226044 (0.556451) | 7.861619 / 2.268929 (5.592690) | 3.241001 / 55.444624 (-52.203623) | 2.611025 / 6.876477 (-4.265452) | 2.667263 / 2.142072 (0.525191) | 1.429992 / 4.805227 (-3.375235) | 0.243008 / 6.500664 (-6.257656) | 0.083686 / 0.075469 (0.008217) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.565526 / 1.841788 (-0.276262) | 18.260815 / 8.074308 (10.186507) | 22.586133 / 10.191392 (12.394741) | 0.231864 / 0.680424 (-0.448559) | 0.030877 / 0.534201 (-0.503324) | 0.569726 / 0.579283 (-0.009557) | 0.678638 / 0.434364 (0.244274) | 0.611810 / 0.540337 (0.071472) | 0.718771 / 1.386936 (-0.668165) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009398 / 0.011353 (-0.001955) | 0.006452 / 0.011008 (-0.004556) | 0.103352 / 0.038508 (0.064844) | 0.034773 / 0.023109 (0.011664) | 0.523782 / 0.275898 (0.247884) | 0.523554 / 0.323480 (0.200074) | 0.006990 / 0.007986 (-0.000996) | 0.004994 / 0.004328 (0.000666) | 0.102199 / 0.004250 (0.097949) | 0.050087 / 0.037052 (0.013035) | 0.496662 / 0.258489 (0.238173) | 0.563130 / 0.293841 (0.269289) | 0.052851 / 0.128546 (-0.075695) | 0.019824 / 0.075646 (-0.055822) | 0.122657 / 0.419271 (-0.296614) | 0.057714 / 0.043533 (0.014181) | 0.470502 / 0.255139 (0.215363) | 0.518908 / 0.283200 (0.235708) | 0.114374 / 0.141683 (-0.027309) | 1.795918 / 1.452155 (0.343763) | 1.957461 / 1.492716 (0.464744) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.303921 / 0.018006 (0.285915) | 0.584406 / 0.000490 (0.583916) | 0.000444 / 0.000200 (0.000244) | 0.000096 / 0.000054 (0.000042) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032254 / 0.037411 (-0.005158) | 0.129966 / 0.014526 (0.115440) | 0.151000 / 0.176557 (-0.025557) | 0.234060 / 0.737135 (-0.503076) | 0.149444 / 0.296338 (-0.146895) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.666627 / 0.215209 (0.451418) | 7.054701 / 2.077655 (4.977046) | 2.836895 / 1.504120 (1.332775) | 2.561994 / 1.541195 (1.020799) | 2.672460 / 1.468490 (1.203970) | 1.411929 / 4.584777 (-3.172848) | 6.026918 / 3.745712 (2.281206) | 3.341745 / 5.269862 (-1.928116) | 2.280317 / 4.565676 (-2.285359) | 0.156635 / 0.424275 (-0.267641) | 0.014256 / 0.007607 (0.006649) | 0.804830 / 0.226044 (0.578786) | 8.106960 / 2.268929 (5.838031) | 3.597452 / 55.444624 (-51.847172) | 3.002847 / 6.876477 (-3.873630) | 2.931160 / 2.142072 (0.789088) | 1.484172 / 4.805227 (-3.321056) | 0.254166 / 6.500664 (-6.246498) | 0.080554 / 0.075469 (0.005085) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.809909 / 1.841788 (-0.031879) | 18.988994 / 8.074308 (10.914686) | 23.153442 / 10.191392 (12.962050) | 0.250554 / 0.680424 (-0.429870) | 0.048677 / 0.534201 (-0.485524) | 0.574109 / 0.579283 (-0.005174) | 0.640917 / 0.434364 (0.206553) | 0.725215 / 0.540337 (0.184878) | 0.878234 / 1.386936 (-0.508702) |

![](https://cml.dev/watermark.png#e3667d6e17d68503469c8e88ec344b7cccfa2346 "CML watermark")


Broken Image render on the hub website

Hi!

You can fix the viewer by adding the `dataset_info` YAML field deleted in https://huggingface.co/datasets/Francesco/cell-towers/commit/b95b59ddd91ebe9c12920f0efe0ed415cd0d4298 back to the metadata section of the card.

To avoid this issue in the feature, you can use `huggingface_hub`'s [RepoCard](https://huggingface.co/docs/huggingface_hub/package_reference/cards) API  to update the dataset card instead of `upload_file`:
```python
from huggingface_hub import DatasetCard
# Load card
card = DatasetCard.load("<namespace>/<repo_id>")
# Modify card content
card.content = ...
# Push card to the Hub
card.push_to_hub("<namespace>/<repo_id>")
```

However, the best solution would be to use the features info stored in the header of the Parquet shards generated with `push_to_hub` on the viewer side to avoid unexpected issues such as this one. This shouldn't be too hard to address.

Thanks for reporting @FrancescoSaverioZuppichini.

For future issues with your specific dataset, you can use its "Community" tab to start a conversation: https://huggingface.co/datasets/Francesco/cell-towers/discussions/new

Thanks @albertvillanova , @mariosasko I was not aware of this requirement from the doc (must have skipped :sweat_smile: )

Confirmed, adding back `dataset_info` fixed the issu

### Describe the bug

Hi :wave:

Not sure if this is the right place to ask, but I am trying to load a huge amount of datasets on the hub (:partying_face: ) but I am facing a little issue with the `image` type

![image](https://user-images.githubusercontent.com/15908060/228587875-427a37f1-3a31-4e17-8bbe-0f759003910d.png)

See this [dataset](https://huggingface.co/datasets/Francesco/cell-towers), basically for some reason the first image has numerical bytes inside, not sure if that is okay, but the image render feature **doesn't work**

So the dataset is stored in the following way

```python
    builder.download_and_prepare(output_dir=str(output_dir))

    ds = builder.as_dataset(split="train")
    # [NOTE] no idea how to push it from the builder folder
    ds.push_to_hub(repo_id=repo_id)
    builder.as_dataset(split="validation").push_to_hub(repo_id=repo_id)
    ds = builder.as_dataset(split="test")
    ds.push_to_hub(repo_id=repo_id)
   ```

   The build is this class

   ```python
       class COCOLikeDatasetBuilder(datasets.GeneratorBasedBuilder):

        VERSION = datasets.Version("1.0.0")

        def _info(self):
            features = datasets.Features(
                {
                    "image_id": datasets.Value("int64"),
                    "image": datasets.Image(),
                    "width": datasets.Value("int32"),
                    "height": datasets.Value("int32"),
                    "objects": datasets.Sequence(
                        {
                            "id": datasets.Value("int64"),
                            "area": datasets.Value("int64"),
                            "bbox": datasets.Sequence(
                                datasets.Value("float32"), length=4
                            ),
                            "category": datasets.ClassLabel(names=categories),
                        }
                    ),
                }
            )
            return datasets.DatasetInfo(
                description=description,
                features=features,
                homepage=homepage,
                license=license,
                citation=citation,
            )

        def _split_generators(self, dl_manager):
            archive = dl_manager.download(url)

            return [
                datasets.SplitGenerator(
                    name=datasets.Split.TRAIN,
                    gen_kwargs={
                        "annotation_file_path": "train/_annotations.coco.json",
                        "files": dl_manager.iter_archive(archive),
                    },
                ),
                datasets.SplitGenerator(
                    name=datasets.Split.VALIDATION,
                    gen_kwargs={
                        "annotation_file_path": "test/_annotations.coco.json",
                        "files": dl_manager.iter_archive(archive),
                    },
                ),
                datasets.SplitGenerator(
                    name=datasets.Split.TEST,
                    gen_kwargs={
                        "annotation_file_path": "valid/_annotations.coco.json",
                        "files": dl_manager.iter_archive(archive),
                    },
                ),
            ]

        def _generate_examples(self, annotation_file_path, files):
            def process_annot(annot, category_id_to_category):
                return {
                    "id": annot["id"],
                    "area": annot["area"],
                    "bbox": annot["bbox"],
                    "category": category_id_to_category[annot["category_id"]],
                }

            image_id_to_image = {}
            idx = 0

            # This loop relies on the ordering of the files in the archive:
            # Annotation files come first, then the images.
            for path, f in files:
                file_name = os.path.basename(path)
                if annotation_file_path in path:
                    annotations = json.load(f)
                    category_id_to_category = {
                        category["id"]: category["name"]
                        for category in annotations["categories"]
                    }
                    print(category_id_to_category)
                    image_id_to_annotations = collections.defaultdict(list)
                    for annot in annotations["annotations"]:
                        image_id_to_annotations[annot["image_id"]].append(annot)
                    image_id_to_image = {
                        annot["file_name"]: annot for annot in annotations["images"]
                    }
                elif file_name in image_id_to_image:
                    image = image_id_to_image[file_name]
                    objects = [
                        process_annot(annot, category_id_to_category)
                        for annot in image_id_to_annotations[image["id"]]
                    ]
                    print(file_name)
                    yield idx, {
                        "image_id": image["id"],
                        "image": {"path": path, "bytes": f.read()},
                        "width": image["width"],
                        "height": image["height"],
                        "objects": objects,
                    }
                    idx += 1
```

Basically, I want to add to the hub every dataset I come across on coco format

Thanks

Fra

### Steps to reproduce the bug

In this case, you can just navigate on the [dataset](https://huggingface.co/datasets/Francesco/cell-towers)

### Expected behavior

I was expecting the image rendering feature to work

### Environment info

Not a lot to share, I am using `datasets` from a fresh venv