Tutorial for creating a dataset

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.012018 / 0.011353 (0.000665) | 0.006204 / 0.011008 (-0.004804) | 0.134119 / 0.038508 (0.095611) | 0.038436 / 0.023109 (0.015327) | 0.381397 / 0.275898 (0.105499) | 0.456362 / 0.323480 (0.132882) | 0.009826 / 0.007986 (0.001840) | 0.004746 / 0.004328 (0.000417) | 0.103755 / 0.004250 (0.099505) | 0.043867 / 0.037052 (0.006815) | 0.395322 / 0.258489 (0.136833) | 0.475812 / 0.293841 (0.181971) | 0.057865 / 0.128546 (-0.070682) | 0.019919 / 0.075646 (-0.055727) | 0.465343 / 0.419271 (0.046072) | 0.061574 / 0.043533 (0.018041) | 0.371668 / 0.255139 (0.116529) | 0.400375 / 0.283200 (0.117176) | 0.106539 / 0.141683 (-0.035144) | 1.822931 / 1.452155 (0.370776) | 1.875535 / 1.492716 (0.382819) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.013583 / 0.018006 (-0.004423) | 0.535515 / 0.000490 (0.535025) | 0.007920 / 0.000200 (0.007720) | 0.000305 / 0.000054 (0.000250) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030204 / 0.037411 (-0.007207) | 0.131671 / 0.014526 (0.117145) | 0.143977 / 0.176557 (-0.032579) | 0.175498 / 0.737135 (-0.561637) | 0.166134 / 0.296338 (-0.130204) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.630995 / 0.215209 (0.415786) | 6.152275 / 2.077655 (4.074620) | 2.519887 / 1.504120 (1.015767) | 2.110926 / 1.541195 (0.569732) | 2.207555 / 1.468490 (0.739064) | 1.296197 / 4.584777 (-3.288580) | 5.510619 / 3.745712 (1.764906) | 3.167468 / 5.269862 (-2.102394) | 2.043924 / 4.565676 (-2.521753) | 0.144772 / 0.424275 (-0.279503) | 0.014456 / 0.007607 (0.006848) | 0.783629 / 0.226044 (0.557585) | 7.836962 / 2.268929 (5.568033) | 3.248593 / 55.444624 (-52.196032) | 2.577092 / 6.876477 (-4.299385) | 2.671918 / 2.142072 (0.529846) | 1.471586 / 4.805227 (-3.333641) | 0.251391 / 6.500664 (-6.249273) | 0.091947 / 0.075469 (0.016478) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.594839 / 1.841788 (-0.246949) | 18.250630 / 8.074308 (10.176322) | 23.948781 / 10.191392 (13.757389) | 0.275505 / 0.680424 (-0.404919) | 0.045202 / 0.534201 (-0.488999) | 0.545552 / 0.579283 (-0.033731) | 0.639352 / 0.434364 (0.204989) | 0.666345 / 0.540337 (0.126008) | 0.795614 / 1.386936 (-0.591322) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.011234 / 0.011353 (-0.000119) | 0.005983 / 0.011008 (-0.005025) | 0.109144 / 0.038508 (0.070636) | 0.036070 / 0.023109 (0.012961) | 0.429313 / 0.275898 (0.153415) | 0.490615 / 0.323480 (0.167135) | 0.007448 / 0.007986 (-0.000538) | 0.004424 / 0.004328 (0.000095) | 0.097100 / 0.004250 (0.092850) | 0.049719 / 0.037052 (0.012667) | 0.412719 / 0.258489 (0.154230) | 0.485717 / 0.293841 (0.191876) | 0.061168 / 0.128546 (-0.067378) | 0.021510 / 0.075646 (-0.054136) | 0.116598 / 0.419271 (-0.302673) | 0.066116 / 0.043533 (0.022583) | 0.426212 / 0.255139 (0.171073) | 0.448368 / 0.283200 (0.165168) | 0.116003 / 0.141683 (-0.025680) | 1.799329 / 1.452155 (0.347175) | 1.967256 / 1.492716 (0.474540) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.214893 / 0.018006 (0.196887) | 0.497843 / 0.000490 (0.497354) | 0.000464 / 0.000200 (0.000264) | 0.000094 / 0.000054 (0.000039) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031758 / 0.037411 (-0.005653) | 0.131182 / 0.014526 (0.116656) | 0.141251 / 0.176557 (-0.035305) | 0.186526 / 0.737135 (-0.550609) | 0.142975 / 0.296338 (-0.153363) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.662094 / 0.215209 (0.446885) | 6.664841 / 2.077655 (4.587186) | 2.690613 / 1.504120 (1.186493) | 2.305399 / 1.541195 (0.764205) | 2.383697 / 1.468490 (0.915207) | 1.280692 / 4.584777 (-3.304085) | 5.629215 / 3.745712 (1.883503) | 5.007083 / 5.269862 (-0.262778) | 2.482163 / 4.565676 (-2.083513) | 0.147662 / 0.424275 (-0.276613) | 0.017770 / 0.007607 (0.010163) | 0.818380 / 0.226044 (0.592335) | 8.006521 / 2.268929 (5.737592) | 3.472262 / 55.444624 (-51.972363) | 2.709550 / 6.876477 (-4.166926) | 2.775138 / 2.142072 (0.633066) | 1.570545 / 4.805227 (-3.234683) | 0.266323 / 6.500664 (-6.234341) | 0.090591 / 0.075469 (0.015122) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.657927 / 1.841788 (-0.183861) | 18.448981 / 8.074308 (10.374673) | 20.336909 / 10.191392 (10.145517) | 0.230322 / 0.680424 (-0.450102) | 0.025972 / 0.534201 (-0.508229) | 0.561361 / 0.579283 (-0.017922) | 0.623758 / 0.434364 (0.189394) | 0.664120 / 0.540337 (0.123783) | 0.763144 / 1.386936 (-0.623792) |

![](https://cml.dev/watermark.png#29de6179766418c937fb33b0cc8803ec24a39e9e "CML watermark")

A tutorial for creating datasets based on the folder-based builders and `from_dict` and `from_generator` methods. I've also mentioned loading scripts as a next step, but I think we should keep the tutorial focused on the low-code methods. Let me know what you think! 🙂


good first issue

Good for newcomers

Hi! The `set_transform` does not apply a custom formatting transform on a single example but the entire batch, so the fixed version of your transform would look as follows:
```python
from datasets import load_dataset
import torch

dataset = load_dataset("lambdalabs/pokemon-blip-captions", split='train')
def t(batch):
    return {"test": torch.tensor([1] * len(batch[next(iter(batch))]))}

dataset.set_transform(t)
d_0 = dataset[0]
```

Still, the formatter's error message should mention that a dict of **sequences** is expected as the returned value (not just a dict) to make debugging easier.

> Hi! The `set_transform` does not apply a custom formatting transform on a single example but the entire batch, so the fixed version of your transform would look as follows:
>
> ```python
> from datasets import load_dataset
> import torch
>
> dataset = load_dataset("lambdalabs/pokemon-blip-captions", split='train')
> def t(batch):
>     return {"test": torch.tensor([1] * len(batch[next(iter(batch))]))}
>
> dataset.set_transform(t)
> d_0 = dataset[0]
> ```
>
> Still, the formatter's error message should mention that a dict of **sequences** is expected as the returned value (not just a dict) to make debugging easier.

ok, will change it according to suggestion. Thanks for the reply!

### Describe the bug

When dataset contains a 0-dim tensor, formatting.py raises a following error and fails.
```bash
Traceback (most recent call last):
  File "<path>/lib/python3.8/site-packages/datasets/formatting/formatting.py", line 501, in format_row
    return _unnest(formatted_batch)
  File "<path>/lib/python3.8/site-packages/datasets/formatting/formatting.py", line 137, in _unnest
    return {key: array[0] for key, array in py_dict.items()}
  File "<path>/lib/python3.8/site-packages/datasets/formatting/formatting.py", line 137, in <dictcomp>
    return {key: array[0] for key, array in py_dict.items()}
IndexError: invalid index of a 0-dim tensor. Use `tensor.item()` in Python or `tensor.item<T>()` in C++ to convert a 0-dim tensor to a number
```

### Steps to reproduce the bug

Load whichever dataset and add transform method to add 0-dim tensor. Or create/find a dataset containing 0-dim tensor. E.g.

```python
from datasets import load_dataset
import torch

dataset = load_dataset("lambdalabs/pokemon-blip-captions", split='train')
def t(batch):
    return {"test": torch.tensor(1)}

dataset.set_transform(t)
d_0 = dataset[0]
```

### Expected behavior

Extractor will correctly get a row from the dataset, even if it contains 0-dim tensor.

### Environment info

`datasets==2.8.0`, but it looks like it is also applicable to main branch version (as of 16th February)


load_dataset in seaborn is not working for me. getting this error.

Hi! `seaborn`'s `load_dataset` pulls datasets from [here](https://github.com/mwaskom/seaborn-data) and not from our Hub, so this issue is not related to our library in any way and should be reported in their repo instead.

TimeoutError                              Traceback (most recent call last)
~\anaconda3\lib\urllib\request.py in do_open(self, http_class, req, **http_conn_args)
   1345             try:
-> 1346                 h.request(req.get_method(), req.selector, req.data, headers,
   1347                           encode_chunked=req.has_header('Transfer-encoding'))

~\anaconda3\lib\http\client.py in request(self, method, url, body, headers, encode_chunked)
   1278         """Send a complete request to the server."""
-> 1279         self._send_request(method, url, body, headers, encode_chunked)
   1280

~\anaconda3\lib\http\client.py in _send_request(self, method, url, body, headers, encode_chunked)
   1324             body = _encode(body, 'body')
-> 1325         self.endheaders(body, encode_chunked=encode_chunked)
   1326

~\anaconda3\lib\http\client.py in endheaders(self, message_body, encode_chunked)
   1273             raise CannotSendHeader()
-> 1274         self._send_output(message_body, encode_chunked=encode_chunked)
   1275

~\anaconda3\lib\http\client.py in _send_output(self, message_body, encode_chunked)
   1033         del self._buffer[:]
-> 1034         self.send(msg)
   1035

~\anaconda3\lib\http\client.py in send(self, data)
    973             if self.auto_open:
--> 974                 self.connect()
    975             else:

~\anaconda3\lib\http\client.py in connect(self)
   1440
-> 1441             super().connect()
   1442

~\anaconda3\lib\http\client.py in connect(self)
    944         """Connect to the host and port specified in __init__."""
--> 945         self.sock = self._create_connection(
    946             (self.host,self.port), self.timeout, self.source_address)

~\anaconda3\lib\socket.py in create_connection(address, timeout, source_address)
    843         try:
--> 844             raise err
    845         finally:

~\anaconda3\lib\socket.py in create_connection(address, timeout, source_address)
    831                 sock.bind(source_address)
--> 832             sock.connect(sa)
    833             # Break explicitly a reference cycle

TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

During handling of the above exception, another exception occurred:

URLError                                  Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_12220/2927704185.py in <module>
      1 import seaborn as sn
----> 2 iris = sn.load_dataset('iris')

~\anaconda3\lib\site-packages\seaborn\utils.py in load_dataset(name, cache, data_home, **kws)
    594             if name not in get_dataset_names():
    595                 raise ValueError(f"'{name}' is not one of the example datasets.")
--> 596             urlretrieve(url, cache_path)
    597         full_path = cache_path
    598     else:

~\anaconda3\lib\urllib\request.py in urlretrieve(url, filename, reporthook, data)
    237     url_type, path = _splittype(url)
    238
--> 239     with contextlib.closing(urlopen(url, data)) as fp:
    240         headers = fp.info()
    241

~\anaconda3\lib\urllib\request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
    212     else:
    213         opener = _opener
--> 214     return opener.open(url, data, timeout)
    215
    216 def install_opener(opener):

~\anaconda3\lib\urllib\request.py in open(self, fullurl, data, timeout)
    515
    516         sys.audit('urllib.Request', req.full_url, req.data, req.headers, req.get_method())
--> 517         response = self._open(req, data)
    518
    519         # post-process response

~\anaconda3\lib\urllib\request.py in _open(self, req, data)
    532
    533         protocol = req.type
--> 534         result = self._call_chain(self.handle_open, protocol, protocol +
    535                                   '_open', req)
    536         if result:

~\anaconda3\lib\urllib\request.py in _call_chain(self, chain, kind, meth_name, *args)
    492         for handler in handlers:
    493             func = getattr(handler, meth_name)
--> 494             result = func(*args)
    495             if result is not None:
    496                 return result

~\anaconda3\lib\urllib\request.py in https_open(self, req)
   1387
   1388         def https_open(self, req):
-> 1389             return self.do_open(http.client.HTTPSConnection, req,
   1390                 context=self._context, check_hostname=self._check_hostname)
   1391

~\anaconda3\lib\urllib\request.py in do_open(self, http_class, req, **http_conn_args)
   1347                           encode_chunked=req.has_header('Transfer-encoding'))
   1348             except OSError as err: # timeout error
-> 1349                 raise URLError(err)
   1350             r = h.getresponse()
   1351         except:

URLError: <urlopen error [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>


Increase speed of data files resolution

enhancement

New feature or request

good second issue

Issues a bit more difficult than "Good First" issues

You were right, if `self.dir_cache` is not None in glob, it is exactly the same as what is returned by find, at least for all the tests we have, and some extended evaluation I did across a random sample of about 1000 datasets.

Thanks for the nice hints, and let me know if this is not exactly what we want here!

see PR: https://github.com/huggingface/datasets/pull/5704

I think we can make the data files resolution (significantly) faster in 2 steps:

1. `glob` calls `find` (which in turn calls `ls`), so we need `find` to be fast, and this can be achieved by fetching all the entries in a single API call and avoiding calls to `ls`. Implementing this for `HfFileSystem.find` (the one in `huggingface_hub`) is on my TO-DO list.
2. caching the repeated `find` calls in `_get_data_files_patterns` when the `data_files` patterns are not provided in `load_dataset`. To address this, we can introduce a `_resolve_single_pattern` function that would accept a filesystem object and a list of regex patterns to resolve. Then we can wrap this filesystem object in `_get_data_files_patterns` with an object that would cache the find calls before resolving the patterns with `_resolve_single_pattern`. (Feel free to suggest a cleaner implementation)

WDYT?

Good idea :)

For 2:

That would work ! It's also possible to have a FileSystem with a cache on `.find` and use it inside the resolver passed to `_get_data_files_patterns`. Right now they're pretty simple:

```python
# for remote repositories
resolver = partial(_resolve_single_pattern_in_dataset_repository, dataset_info, base_path=base_path)
# for local
resolver = partial(_resolve_single_pattern_locally, base_path)
```

something like this maybe (with Quentin's reimplementation of `HfFilesystem.find`)?

 ```
 @lru_cache(max_size=None)
 def _find(self, path, maxdepth=None, withdirs=False, detail=False, **kwargs):
```

In any case please let me know if I can help in any way!

Certain datasets like `bigcode/the-stack-dedup` have so many files that loading them takes forever right from the data files resolution step.

`datasets` uses file patterns to check the structure of the repository but it takes too much time to iterate over and over again on all the data files.

This comes from `resolve_patterns_in_dataset_repository` which calls `_resolve_single_pattern_in_dataset_repository`, which iterates on all the files at
```python
glob_iter = [PurePath(filepath) for filepath in fs.glob(PurePath(pattern).as_posix()) if fs.isfile(filepath)]
```
but calling `glob` on such a dataset is too expensive. Indeed it calls `ls()` in `hffilesystem.py` too many times.

Maybe `glob` can be more optimized in `hffilesystem.py`, or the data files resolution can directly be implemented in the filesystem by checking its `dir_cache` ?


Failure to hash function when using .map()

Hi ! `enc` is not hashable:
```python
import tiktoken
from datasets.fingerprint import Hasher

enc = tiktoken.get_encoding("gpt2")
Hasher.hash(enc)
# raises TypeError: cannot pickle 'builtins.CoreBPE' object
```
It happens because it's not picklable, and because of that it's not possible to cache the result of `map`, hence the warning message.

You can find more details about caching here: https://huggingface.co/docs/datasets/about_cache

You can also provide your own unique hash in `map` if you want, with the `new_fingerprint` argument.
Or disable caching using
```python
import datasets
datasets.disable_caching()
```

@lhoestq Thank you for the explanation and advice. Will relay all of this to the repo where this (non)issue arose.

Great job with huggingface!

We made tiktoken tokenizers hashable in #5552, which is included in today's release `datasets==2.10.0`

Just a heads up that when I'm trying to use TikToken along with the a given Dataset `.map()` method, I am still met with the following error :

```
  File "/opt/conda/lib/python3.8/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/opt/conda/lib/python3.8/pickle.py", line 578, in save
    rv = reduce(self.proto)
TypeError: cannot pickle 'builtins.CoreBPE' object
```

My current environment is running datasets v2.10.0.

@lhoestq @edhenry I am also seeing this, do you have any suggested solution?

With which `datasets` version ? Can you try to udpate ?

@lhoestq @edhenry I am on datasets version `'2.12.0'. I see the same `TypeError: cannot pickle 'builtins.CoreBPE' object` that others are seeing.

I am able to reproduce this on datasets 2.14.2. The `datasets.disable_caching()` doesn't work around it.

@lhoestq - you might want to reopen this issue. Because of this issue folks won't be able run Karpathy's NanoGPT :(.

update： temporarily solved the problem by setting
```
--preprocess_num_workers  1
```

-------------
I  have met the same problem, here is my env:
```
datasets                      2.14.4
transformers                  4.31.0
tiktoken                      0.4.0
torch                         1.13.1
```

@mengban I cannot reproduce the issue even with these versions installed. It would help if you could provide info about your system and the `pip list` output.

@mariosasko Please take a look at this
```python
from typing import Any
from datasets import Dataset
import tiktoken

dataset = Dataset.from_list([{"n": str(i)} for i in range(20)])
enc = tiktoken.get_encoding("gpt2")

class A:
    tokenizer = enc #tiktoken.get_encoding("gpt2")

    def __call__(self, example) -> Any:
        ids = self.tokenizer.encode(example["n"])
        example["len"] = len(ids)
        return example

a = A()

def process(example):
    ids = a.tokenizer.encode(example["n"])
    example["len"] = len(ids)
    return example

# success
tokenized = dataset.map(process, desc="tiktoken", num_proc=2)

# raise TypeError: cannot pickle 'builtins.CoreBPE' object
tokenized = dataset.map(a, desc="tiktoken", num_proc=2)
```

pip list
```
datasets                      2.14.4
tiktoken                      0.4.0
```

Thanks @maxwellzh! Our `Hasher` works with this snippet, but the problem is running multiprocessing with a non-serializable `tiktoken.Encoding` object.

Inserting the following code before the `map` should fix this:
```python
import copyreg

def pickle_Encoding(enc):
  return (functools.partial(tiktoken.core.Encoding, enc.name, pat_str=enc._pat_str, mergeable_ranks=enc._mergeable_ranks, special_tokens=enc._special_tokens), ())

copyreg.pickle(tiktoken.core.Encoding, pickle_Encoding)
```

But the best fix would be implementing `__reduce__` for `tiktoken.Encoding` or `tiktoken.CoreBPE`. If I find time, I'll try to fix this in the `tiktoken` repo.

I think the right way to fix this would be to have new tokenizer instance for each process. This applies to many other tokenizers that don't support multi-process or have bugs. To do this, first define tokenizer factory class like this:

```
    class TikTokenFactory:
        def __init__(self):
            self._enc = None
            self.eot_token = None

        def encode_ordinary(self, text):
            if self._enc is None:
                self._enc = tiktoken.get_encoding("gpt2")
                self.eot_token = self._enc.eot_token
            return self._enc.encode_ordinary(text)
```

Now use this in `.map()` like this:

```
    # tokenize the dataset
    tokenized = dataset.map(
        partial(process, TikTokenFactory()),
        remove_columns=['text'],
        desc="tokenizing the splits",
        num_proc=max(1, cpu_count()//2),
    )
```

A full working example is here: https://github.com/sytelus/nanoGPT/blob/refactor/nanogpt_common/hf_data_prepare.py

### Describe the bug

_Parameter 'function'=<function process at 0x7f1ec4388af0> of the transform datasets.arrow_dataset.Dataset.\_map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed._

This issue with `.map()` happens for me consistently, as also described in closed issue #4506

Dataset indices can be individually serialized using dill and pickle without any errors.  I'm using tiktoken to encode in the function passed to map(). Similarly, indices can be individually encoded without error.

### Steps to reproduce the bug

```py
from datasets import load_dataset
import tiktoken

dataset = load_dataset("stas/openwebtext-10k")

enc = tiktoken.get_encoding("gpt2")

tokenized = dataset.map(
    process,
    remove_columns=['text'],
    desc="tokenizing the OWT splits",
)

def process(example):
        ids = enc.encode(example['text'])
        ids.append(enc.eot_token)
        out = {'ids': ids, 'len': len(ids)}
        return out
```

### Expected behavior

Should encode simple text objects.

### Environment info

Python versions tried: both 3.8 and 3.10.10
`PYTHONUTF8=1` as env variable

Datasets tried:
- stas/openwebtext-10k
- rotten_tomatoes
- local text file

OS: Ubuntu Linux 20.04

Package versions:
- torch 1.13.1
- dill 0.3.4 (if using 0.3.6 - same issue)
- datasets 2.9.0
- tiktoken 0.2.0


Add JAX-formatting documentation

_The documentation is not available anymore as the PR was closed or merged._

> Awesome thank you !
>
> Could you also explain how to use certain types like ClassLabel, Image or Audio with jax ? You can get a lot of inspiration from the "Other feature types" section in the [PyTorch page](https://huggingface.co/docs/datasets/use_with_pytorch)
>
> I also think it's be nice if this page had the same structure as the pytorch or tf ones, with sections named
>
>     * Dataset format
>
>     * N-dimensional arrays
>
>     * Other feature types
>
>     * Data loading

Sure @lhoestq I'll do that later this afternoon whenever I'm done working! Thanks for the feedback as always 🤗

Also, @lhoestq do you want me to elaborate more on the `## Data loading` section on how to use `datasets` to train a JAX model offering alternatives e.g. `Flax`, or do I keep it pure JAX? Thanks!

If you have a good example with `flax` it can also be helpful for users

For now, I think that probably it's not worth adding a `Flax` example, as train loops need to be done manually as in pure JAX, so probably the JAX example is enough. Anyway, let me know if you see something missing/incomplete/misleading/etc. and I'll update that ASAP 👍🏻

P.S. I see that the `benchmark` action is being triggered on every PR, is it worth it? e.g. now I'm just editing the docs, so does it make any sense to trigger still the whole CI pipeline (including `benchmark`)? Just asking because in this PR for example it could be skipped.

> P.S. I see that the benchmark action is being triggered on every PR, is it worth it? e.g. now I'm just editing the docs, so does it make any sense to trigger still the whole CI pipeline (including benchmark)? Just asking because in this PR for example it could be skipped.

We could restrict it to PRs modifying files in src/ indeed ^^'

> LGTM :)

Cool thanks! My bad I didn't update those code blocks 🙃 Thanks for doing so before merge!

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009336 / 0.011353 (-0.002017) | 0.005037 / 0.011008 (-0.005971) | 0.102168 / 0.038508 (0.063659) | 0.035351 / 0.023109 (0.012242) | 0.299616 / 0.275898 (0.023718) | 0.333269 / 0.323480 (0.009789) | 0.008215 / 0.007986 (0.000229) | 0.005047 / 0.004328 (0.000718) | 0.074257 / 0.004250 (0.070007) | 0.045080 / 0.037052 (0.008028) | 0.300657 / 0.258489 (0.042168) | 0.357569 / 0.293841 (0.063728) | 0.038614 / 0.128546 (-0.089932) | 0.011995 / 0.075646 (-0.063651) | 0.369141 / 0.419271 (-0.050130) | 0.047603 / 0.043533 (0.004070) | 0.297694 / 0.255139 (0.042555) | 0.315380 / 0.283200 (0.032180) | 0.105009 / 0.141683 (-0.036674) | 1.421077 / 1.452155 (-0.031078) | 1.550024 / 1.492716 (0.057308) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.239026 / 0.018006 (0.221020) | 0.550010 / 0.000490 (0.549520) | 0.003294 / 0.000200 (0.003094) | 0.000093 / 0.000054 (0.000038) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027180 / 0.037411 (-0.010231) | 0.107942 / 0.014526 (0.093416) | 0.121092 / 0.176557 (-0.055464) | 0.161028 / 0.737135 (-0.576108) | 0.124615 / 0.296338 (-0.171723) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.399492 / 0.215209 (0.184283) | 3.984685 / 2.077655 (1.907030) | 1.794784 / 1.504120 (0.290664) | 1.604849 / 1.541195 (0.063654) | 1.682994 / 1.468490 (0.214504) | 0.691197 / 4.584777 (-3.893580) | 3.741816 / 3.745712 (-0.003897) | 2.092151 / 5.269862 (-3.177711) | 1.319106 / 4.565676 (-3.246570) | 0.083875 / 0.424275 (-0.340400) | 0.012473 / 0.007607 (0.004866) | 0.514057 / 0.226044 (0.288012) | 5.110217 / 2.268929 (2.841288) | 2.259105 / 55.444624 (-53.185519) | 1.914021 / 6.876477 (-4.962455) | 1.958371 / 2.142072 (-0.183701) | 0.819800 / 4.805227 (-3.985428) | 0.161153 / 6.500664 (-6.339511) | 0.061967 / 0.075469 (-0.013502) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.198553 / 1.841788 (-0.643234) | 14.793201 / 8.074308 (6.718893) | 14.646807 / 10.191392 (4.455415) | 0.152805 / 0.680424 (-0.527619) | 0.029206 / 0.534201 (-0.504995) | 0.440875 / 0.579283 (-0.138408) | 0.434925 / 0.434364 (0.000561) | 0.533495 / 0.540337 (-0.006842) | 0.624479 / 1.386936 (-0.762457) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007346 / 0.011353 (-0.004007) | 0.005422 / 0.011008 (-0.005586) | 0.073930 / 0.038508 (0.035422) | 0.032978 / 0.023109 (0.009869) | 0.335182 / 0.275898 (0.059284) | 0.371916 / 0.323480 (0.048436) | 0.005851 / 0.007986 (-0.002135) | 0.005582 / 0.004328 (0.001254) | 0.073090 / 0.004250 (0.068839) | 0.048395 / 0.037052 (0.011342) | 0.353921 / 0.258489 (0.095432) | 0.380678 / 0.293841 (0.086837) | 0.036628 / 0.128546 (-0.091919) | 0.012392 / 0.075646 (-0.063254) | 0.086265 / 0.419271 (-0.333006) | 0.049262 / 0.043533 (0.005729) | 0.334790 / 0.255139 (0.079651) | 0.355278 / 0.283200 (0.072078) | 0.102714 / 0.141683 (-0.038969) | 1.536366 / 1.452155 (0.084211) | 1.565984 / 1.492716 (0.073268) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.216050 / 0.018006 (0.198043) | 0.554972 / 0.000490 (0.554482) | 0.002432 / 0.000200 (0.002232) | 0.000110 / 0.000054 (0.000055) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028602 / 0.037411 (-0.008809) | 0.123681 / 0.014526 (0.109155) | 0.136763 / 0.176557 (-0.039793) | 0.170083 / 0.737135 (-0.567052) | 0.138771 / 0.296338 (-0.157567) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.420036 / 0.215209 (0.204827) | 4.188734 / 2.077655 (2.111079) | 2.014758 / 1.504120 (0.510638) | 1.818423 / 1.541195 (0.277228) | 1.940790 / 1.468490 (0.472300) | 0.691420 / 4.584777 (-3.893357) | 3.782996 / 3.745712 (0.037284) | 2.131278 / 5.269862 (-3.138583) | 1.363043 / 4.565676 (-3.202633) | 0.087182 / 0.424275 (-0.337093) | 0.012448 / 0.007607 (0.004841) | 0.519296 / 0.226044 (0.293252) | 5.220397 / 2.268929 (2.951469) | 2.474243 / 55.444624 (-52.970381) | 2.139726 / 6.876477 (-4.736751) | 2.200700 / 2.142072 (0.058627) | 0.841171 / 4.805227 (-3.964056) | 0.169234 / 6.500664 (-6.331430) | 0.063879 / 0.075469 (-0.011590) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.260262 / 1.841788 (-0.581526) | 14.853209 / 8.074308 (6.778901) | 13.944085 / 10.191392 (3.752693) | 0.192014 / 0.680424 (-0.488410) | 0.017811 / 0.534201 (-0.516390) | 0.427166 / 0.579283 (-0.152117) | 0.438263 / 0.434364 (0.003899) | 0.538815 / 0.540337 (-0.001523) | 0.641398 / 1.386936 (-0.745538) |

![](https://cml.dev/watermark.png#139e9ae67a88cd79274bbf8315d861ee8bc7175f "CML watermark")

## What's in this PR?

As a follow-up of #5522, I've created this entry in the documentation to explain how to use `.with_format("jax")` and why is it useful.

@lhoestq Feel free to drop any feedback and/or suggestion, as probably more useful features can be included there!


map() breaks at certain dataset size when using Array3D

Hi! This code works for me locally or in Colab. What's the output of `python -c "import pyarrow as pa; print(pa.__version__)"` when you run it inside your environment?

Thanks for looking into this!
The output of  `python -c "import pyarrow as pa; print(pa.__version__)"` is:
```
11.0.0
```

I did the following to setup the environment:
```
conda create -n datasets_debug python=3.9
conda activate datasets_debug
pip install datasets==2.9.0
```

I just tested this on another machine (Ubuntu 18.04.6 LTS) with the same result as mentioned in the issue description.

### Describe the bug

`map()` magically breaks when using a `Array3D` feature and mapping it. I created a very simple dummy dataset (see below). When filtering it down to 95 elements I can apply map, but it breaks when filtering it down to just 96 entries with the following exception:
```
Traceback (most recent call last):
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3255, in _map_single
    writer.finalize()  # close_stream=bool(buf_writer is None))  # We only close if we are writing in a file
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/arrow_writer.py", line 581, in finalize
    self.write_examples_on_file()
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/arrow_writer.py", line 440, in write_examples_on_file
    batch_examples[col] = array_concat(arrays)
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/table.py", line 1931, in array_concat
    return _concat_arrays(arrays)
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/table.py", line 1901, in _concat_arrays
    return array_type.wrap_array(_concat_arrays([array.storage for array in arrays]))
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/table.py", line 1922, in _concat_arrays
    _concat_arrays([array.values for array in arrays]),
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/table.py", line 1922, in _concat_arrays
    _concat_arrays([array.values for array in arrays]),
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/table.py", line 1920, in _concat_arrays
    return pa.ListArray.from_arrays(
  File "pyarrow/array.pxi", line 1997, in pyarrow.lib.ListArray.from_arrays
  File "pyarrow/array.pxi", line 1527, in pyarrow.lib.Array.validate
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Negative offsets in list array

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 2815, in map
    return self._map_single(
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 546, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 513, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/fingerprint.py", line 480, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3259, in _map_single
    writer.finalize()
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/arrow_writer.py", line 581, in finalize
    self.write_examples_on_file()
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/arrow_writer.py", line 440, in write_examples_on_file
    batch_examples[col] = array_concat(arrays)
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/table.py", line 1931, in array_concat
    return _concat_arrays(arrays)
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/table.py", line 1901, in _concat_arrays
    return array_type.wrap_array(_concat_arrays([array.storage for array in arrays]))
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/table.py", line 1922, in _concat_arrays
    _concat_arrays([array.values for array in arrays]),
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/table.py", line 1922, in _concat_arrays
    _concat_arrays([array.values for array in arrays]),
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/table.py", line 1920, in _concat_arrays
    return pa.ListArray.from_arrays(
  File "pyarrow/array.pxi", line 1997, in pyarrow.lib.ListArray.from_arrays
  File "pyarrow/array.pxi", line 1527, in pyarrow.lib.Array.validate
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Negative offsets in list array

```

### Steps to reproduce the bug

1. put following dataset loading script into: debug/debug.py
```python
import datasets
import numpy as np

class DEBUG(datasets.GeneratorBasedBuilder):
    """DEBUG dataset."""

    def _info(self):
        return datasets.DatasetInfo(
            features=datasets.Features(
                {
                    "id": datasets.Value("uint8"),
                    "img_data": datasets.Array3D(shape=(3, 224, 224), dtype="uint8"),
                },
            ),
            supervised_keys=None,
        )

    def _split_generators(self, dl_manager):
        return [datasets.SplitGenerator(name=datasets.Split.TRAIN)]

    def _generate_examples(self):
        for i in range(149):
            image_np = np.zeros(shape=(3, 224, 224), dtype=np.int8).tolist()
            yield f"id_{i}", {"id": i, "img_data": image_np}

```

2. try the following code:

```python
import datasets

def add_dummy_col(ex):
    ex["dummy"] = "test"
    return ex

ds = datasets.load_dataset(path="debug", split="train")

# works
ds_filtered_works = ds.filter(lambda example: example["id"] < 95)
print(f"filtered result size: {len(ds_filtered_works)}")
# output:
# filtered result size: 95
ds_mapped_works = ds_filtered_works.map(add_dummy_col)

# fails
ds_filtered_error = ds.filter(lambda example: example["id"] < 96)
print(f"filtered result size: {len(ds_filtered_error)}")
# output:
# filtered result size: 96
ds_mapped_error = ds_filtered_error.map(add_dummy_col)
```

### Expected behavior

The example code does not fail.

### Environment info

Python 3.9.16 (main, Jan 11 2023, 16:05:54); [GCC 11.2.0] :: Anaconda, Inc. on linux
datasets 2.9.0


Add reduce function

I agree that it would be a good idea to introduce a `combiner` argument in another PR.

I did take quite a lot of inspiration from the implementation of  `map`, but it did not seem obvious how to resuse `map` for the implementation. Do you have any suggestions, i could give a try?

Those were exactly my thoughts, regarding the non-obvious initializer for batched and formatted datasets, so i agree! I'll  introduce a `initializer` argument, and have it mandatory when `batched=True`.

I added `initializer`. It is optional for `batched=False` and mandatory for `batched=True`. It has to be of the same length as `input_columns`, if `input_columns=None` it has to have the same length as `_data.column_names`.

If the initializer is not set for `batched=False` the first example is set as the `initializer`.

The initializer is used to initiliaze for each shard, so that means if that:
```python
dset = Dataset.from_dict({"x": [1, 2, 3]})
sum_reduce = lambda x, y: x + y
reduction = dset.reduce(sum_reduce, batched=True, initializer=1, input_columns='x', num_proc=2)
# reduction is 8, i.e. reduction + num_proc * initializer
```

> I added initializer. It is optional for batched=False and mandatory for batched=True. It has to be of the same length as input_columns, if input_columns=None it has to have the same length as _data.column_names.
>
> If the initializer is not set for batched=False the first example is set as the initializer.

Sounds good to me !

> The initializer is used to initiliaze for each shard, so that means if that:
>
> ```python
> dset = Dataset.from_dict({"x": [1, 2, 3]})
> sum_reduce = lambda x, y: x + y
> reduction = dset.reduce(sum_reduce, batched=True, initializer=1, input_columns='x', num_proc=2)
> # reduction is 8, i.e. reduction + num_proc * initializer
> ```

Hmm this can be confusing for some users. Maybe we should consider making `combiner` mandatory for multiprocessing.

If we agree on this, maybe for this PR you can either:
- remove multiprocessing (and we add combiner + multiprocessing in a subsequent PR)
- OR add `combiner` directly

Maybe we can get more feedback from @huggingface/datasets as well

> > I added initializer. It is optional for batched=False and mandatory for batched=True. It has to be of the same length as input_columns, if input_columns=None it has to have the same length as _data.column_names.
> > If the initializer is not set for batched=False the first example is set as the initializer.
>
> Sounds good to me !
>
> > The initializer is used to initiliaze for each shard, so that means if that:
> > ```python
> > dset = Dataset.from_dict({"x": [1, 2, 3]})
> > sum_reduce = lambda x, y: x + y
> > reduction = dset.reduce(sum_reduce, batched=True, initializer=1, input_columns='x', num_proc=2)
> > # reduction is 8, i.e. reduction + num_proc * initializer
> > ```
>
> Hmm this can be confusing for some users. Maybe we should consider making `combiner` mandatory for multiprocessing.
>
> If we agree on this, maybe for this PR you can either:
>
> * remove multiprocessing (and we add combiner + multiprocessing in a subsequent PR)
> * OR add `combiner` directly
>
> Maybe we can get more feedback from @huggingface/datasets as well

I think i prefer adding `combiner` in this PR. I think ill make `combiner` mandatory for `batched=True`, instead of assuming that `combiner=function`. Ill look at this one of the coming days. Also at some point i have to define `reduce` for `DatasetDict`, and not just `Dataset`.

I added the `combiner` parameter as described. I added some examples in the docstring, as i felt it might still be a bit confusing what happens during multiprocessing / batching.

Still need to look at `DatasetDict`.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5533). All of your documentation changes will be reflected on that endpoint.

Feel free to merge `main` into your branch - we fixed some CI failures today

The proposed API doesn't seem intuitive to me - one can already use `functools.reduce` or `Dataset.map` for this purpose ([Colab](https://colab.research.google.com/drive/1jCLv31Y4cDfqD0lhO0AnqEv3Or-LLvWe?usp=sharing) with examples), so perhaps we could have a section in the docs that uses these methods to perform reductions rather than introducing a new method (which needs to be maintained later)

Thanks for sharing this google colab, it has nice examples !

Though I still think `functools.reduce` with multiprocessing can be a pain - we offer something easier here:
- no need to use a pool yourself
- no need to use `map` just to iterate on the dataset (not its main purpose)
- native support for lambdas (using dill)
- the combiner is **mandatory** for multiprocessing to avoid ending up with an incorrect result as in your example

However I agree that maintaining this can be challenging, especially if you think about how `map` already is, and if we also have to deal with dataset formatting.

> native support for lambdas (using dill)

Replacing `multiprocessing` with `multiprocess` in the example would allow that.

> no need to use map just to iterate on the dataset (not its main purpose)

Not the main purpose, but this was mentioned as a "feature" in the previous docs if I remember.

And all this is related to the multi-processing case, which we can document.

Besides the linked issue, I can't find requests for `Dataset.reduce`, which makes me think `functools.reduce` does the job for most users.

> Besides the linked issue, I can't find requests for Dataset.reduce, which makes me think functools.reduce does the job for most users.

I think @srush was looking for a way to do a word count but ended up using a single processed `map`. I also saw some users on the forum wanting to compute `max`

> Not the main purpose, but this was mentioned as a "feature" in the previous docs if I remember.
>
> And all this is related to the multi-processing case, which we can document.

Yup indeed

While counting is one example, I often find I want to compute different statistics over a dataset. This seems like a natural way to do it in a stateless manner.

I guess you could use functools reduce, but that wouldn't allow batching, right?

I've updated the [Colab](https://colab.research.google.com/drive/1jCLv31Y4cDfqD0lhO0AnqEv3Or-LLvWe?usp=sharing) with an example that reduces batches with `map` and then computes the final result. It would be nice to have a similar example (explained in detail) in the docs to show the full power of `map`.

Plus, for simple reductions such as `max`, one can do `pc.max(ds.with_format("arrow")["col"])` to directly get the result (without loading the entire column in RAM).

@srush

> I guess you could use functools reduce, but that wouldn't allow batching, right?

You can use `.iter(batch_size)` to get batches

That `functools` tools example is clean. I didn't know about `iter`. That would handle my use case.

The stateful `map` with a global variable is pretty hairy. I don't think we should recommend people do that.

Whenever I in the past wanted to calculate statistics for datasets I used `functools` similarly to how it's described in the colab, but I always felt it was a bit of a hassle to use it together with multiprocessing, which is why I picked up the issue, to do it "once and for all".

Should i close this and open another PR, with descriptions of how to use `map` for reduction, or?

Yes I think good documentation is the way to go here. @mariosasko 's examples are clear and efficient.

Maybe we could have an `Aggregations` section in the `Process` page with some guides on how to:
- use `.map()` to compute aggregates
- use `.with_format("arrow")` for max, min, etc. to save RAM and get max speed
- use a multiprocessed `.map()` to get partial results in parallel and combine them (max text length example)
- (advanced) use multiprocessing with an arbitrary accumulator (word count example)

And also a new conceptual guide on `Multiprocessed mapping` to say that it helps speed up CPU intensive processing but why it may lead to incorrect results when computing aggregates.

cc @stevhliu for visibility and if you have some comments

I would create a `Reduce` - to be more exact - subsection under `Map` to demonstrate these examples since we're showing how they can be done with the `Dataset.map` function. It'd also be good to add a link to the new concept guide from this section to solidify user understanding :)

Coolio. Ill close this PR and get going on another one adding what we've discussed during the next couple of days!

This PR closes #5496 .

I tried to imitate the `reduce`-method from `functools`, i.e. the function input must be a binary operation. I assume that the input type has an empty element, i.e. `input_type()` is defined, as the acumulant is instantiated as this object - im not sure that is this a reasonable assumption?

If `batched= True`  the reduction of each shard is _not_ returned, but the reduction of the entire dataset. I was unsure wether this was an intuitive API, or it would make more sense to return the reduction of each shard?


train_test_split in arrow_dataset does not ensure to keep single classes in test set

Hi! You can get this behavior by specifying `stratify_by_column="label"` in `train_test_split`.

This is the full example:
```python
import numpy as np
from datasets import Dataset, ClassLabel

data = [
    {'label': 0, 'text': "example1"},
    {'label': 1, 'text': "example2"},
    {'label': 1, 'text': "example3"},
    {'label': 1, 'text': "example4"},
    {'label': 0, 'text': "example5"},
    {'label': 1, 'text': "example6"},
    {'label': 2, 'text': "example7"},
    {'label': 2, 'text': "example8"}
]

for _ in range(10):
    data_set = Dataset.from_list(data)
    data_set = data_set.cast_column("label", ClassLabel(num_classes=3))
    data_set = data_set.train_test_split(test_size=0.5, stratify_by_column="label")
    unique_labels_train = np.unique(data_set["train"][:]["label"])
    unique_labels_test = np.unique(data_set["test"][:]["label"])
    assert len(unique_labels_train) >= len(unique_labels_test)
```

### Describe the bug

When I have a dataset with very few (e.g. 1) examples per class and I call the train_test_split function on it, sometimes the single class will be in the test set. thus will never be considered for training.

### Steps to reproduce the bug

```
import numpy as np
from datasets import Dataset

data = [
    {'label': 0, 'text': "example1"},
    {'label': 1, 'text': "example2"},
    {'label': 1, 'text': "example3"},
    {'label': 1, 'text': "example4"},
    {'label': 0, 'text': "example5"},
    {'label': 1, 'text': "example6"},
    {'label': 2, 'text': "example7"},
    {'label': 2, 'text': "example8"}
]

for _ in range(10):
    data_set = Dataset.from_list(data)
    data_set = data_set.train_test_split(test_size=0.5)
    data_set["train"]

    unique_labels_train = np.unique(data_set["train"][:]["label"])
    unique_labels_test = np.unique(data_set["test"][:]["label"])
    assert len(unique_labels_train) >= len(unique_labels_test)
```

### Expected behavior

I expect to have every available class at least once in my training set.

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-5.15.65+-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- PyArrow version: 11.0.0
- Pandas version: 1.3.5


Invalid Arrow data from JSONL

bug

Something isn't working

This code fails:
```python
from datasets import Dataset

ds = Dataset.from_json(path_to_file)
ds.data.validate()
```
raises

```python
ArrowInvalid: Column 2: In chunk 1: Invalid: Struct child array #3 invalid: Invalid: Length spanned by list offsets (4064) larger than values array (length 4063)
```

This causes many issues for @TevenLeScao:

- `map` fails because it fails to rewrite invalid arrow arrays

    ```python
    ~/Desktop/hf/datasets/src/datasets/arrow_writer.py in write_examples_on_file(self)
        438             if all(isinstance(row[0][col], (pa.Array, pa.ChunkedArray)) for row in self.current_examples):
        439                 arrays = [row[0][col] for row in self.current_examples]
    --> 440                 batch_examples[col] = array_concat(arrays)
        441             else:
        442                 batch_examples[col] = [

    ~/Desktop/hf/datasets/src/datasets/table.py in array_concat(arrays)
    1885
    1886     if not _is_extension_type(array_type):
    -> 1887         return pa.concat_arrays(arrays)
    1888
    1889     def _offsets_concat(offsets):

    ~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.concat_arrays()

    ~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

    ~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

    ArrowIndexError: array slice would exceed array length
    ```

- `to_dict()` **segfaults** ⚠

    ```python
    /Users/runner/work/crossbow/crossbow/arrow/cpp/src/arrow/array/data.cc:99:  Check failed: (off) <= (length) Slice offset greater
    than array length
    ```

To reproduce: unzip the archive and run the above code using `sanity_oscar_en.jsonl`
[sanity_oscar_en.jsonl.zip](https://github.com/huggingface/datasets/files/10734124/sanity_oscar_en.jsonl.zip)

PS: reading using pandas and converting to Arrow works though (note that the dataset lives in RAM in this case):
```python
ds = Dataset.from_pandas(pd.read_json(path_to_file, lines=True))
ds.data.validate()
```


Add missing license in `NumpyFormatter`

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008837 / 0.011353 (-0.002516) | 0.004608 / 0.011008 (-0.006400) | 0.101821 / 0.038508 (0.063312) | 0.030300 / 0.023109 (0.007191) | 0.301275 / 0.275898 (0.025377) | 0.365027 / 0.323480 (0.041547) | 0.007043 / 0.007986 (-0.000943) | 0.003493 / 0.004328 (-0.000835) | 0.078444 / 0.004250 (0.074194) | 0.036963 / 0.037052 (-0.000089) | 0.310510 / 0.258489 (0.052020) | 0.343769 / 0.293841 (0.049928) | 0.033560 / 0.128546 (-0.094986) | 0.011427 / 0.075646 (-0.064220) | 0.323542 / 0.419271 (-0.095730) | 0.043063 / 0.043533 (-0.000470) | 0.308869 / 0.255139 (0.053730) | 0.326436 / 0.283200 (0.043236) | 0.091775 / 0.141683 (-0.049908) | 1.471020 / 1.452155 (0.018865) | 1.494328 / 1.492716 (0.001612) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.009299 / 0.018006 (-0.008707) | 0.415705 / 0.000490 (0.415215) | 0.002406 / 0.000200 (0.002206) | 0.000066 / 0.000054 (0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022959 / 0.037411 (-0.014452) | 0.097111 / 0.014526 (0.082585) | 0.103399 / 0.176557 (-0.073157) | 0.144385 / 0.737135 (-0.592750) | 0.109069 / 0.296338 (-0.187269) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.417796 / 0.215209 (0.202587) | 4.158198 / 2.077655 (2.080543) | 1.862036 / 1.504120 (0.357916) | 1.650130 / 1.541195 (0.108936) | 1.717150 / 1.468490 (0.248660) | 0.691704 / 4.584777 (-3.893073) | 3.328254 / 3.745712 (-0.417458) | 1.850070 / 5.269862 (-3.419792) | 1.154331 / 4.565676 (-3.411346) | 0.082199 / 0.424275 (-0.342076) | 0.012226 / 0.007607 (0.004619) | 0.522491 / 0.226044 (0.296446) | 5.244181 / 2.268929 (2.975253) | 2.286651 / 55.444624 (-53.157973) | 1.954439 / 6.876477 (-4.922038) | 1.992052 / 2.142072 (-0.150020) | 0.804779 / 4.805227 (-4.000449) | 0.147341 / 6.500664 (-6.353323) | 0.063863 / 0.075469 (-0.011606) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.270778 / 1.841788 (-0.571010) | 13.676378 / 8.074308 (5.602070) | 14.253498 / 10.191392 (4.062106) | 0.170748 / 0.680424 (-0.509676) | 0.028451 / 0.534201 (-0.505750) | 0.395034 / 0.579283 (-0.184249) | 0.407512 / 0.434364 (-0.026852) | 0.466740 / 0.540337 (-0.073598) | 0.564338 / 1.386936 (-0.822598) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006733 / 0.011353 (-0.004620) | 0.004635 / 0.011008 (-0.006373) | 0.075464 / 0.038508 (0.036956) | 0.027732 / 0.023109 (0.004623) | 0.343622 / 0.275898 (0.067724) | 0.380388 / 0.323480 (0.056908) | 0.005177 / 0.007986 (-0.002808) | 0.003435 / 0.004328 (-0.000893) | 0.074546 / 0.004250 (0.070296) | 0.039115 / 0.037052 (0.002063) | 0.342207 / 0.258489 (0.083718) | 0.390324 / 0.293841 (0.096483) | 0.031665 / 0.128546 (-0.096882) | 0.011695 / 0.075646 (-0.063951) | 0.085788 / 0.419271 (-0.333484) | 0.042423 / 0.043533 (-0.001110) | 0.340748 / 0.255139 (0.085609) | 0.372813 / 0.283200 (0.089614) | 0.092395 / 0.141683 (-0.049288) | 1.502158 / 1.452155 (0.050004) | 1.618233 / 1.492716 (0.125516) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.224451 / 0.018006 (0.206444) | 0.398712 / 0.000490 (0.398222) | 0.002739 / 0.000200 (0.002539) | 0.000074 / 0.000054 (0.000020) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025393 / 0.037411 (-0.012018) | 0.100480 / 0.014526 (0.085954) | 0.106913 / 0.176557 (-0.069644) | 0.148639 / 0.737135 (-0.588496) | 0.110098 / 0.296338 (-0.186240) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.439359 / 0.215209 (0.224150) | 4.396801 / 2.077655 (2.319146) | 2.069809 / 1.504120 (0.565689) | 1.851014 / 1.541195 (0.309820) | 1.885003 / 1.468490 (0.416513) | 0.701387 / 4.584777 (-3.883390) | 3.404943 / 3.745712 (-0.340769) | 1.874506 / 5.269862 (-3.395355) | 1.174925 / 4.565676 (-3.390752) | 0.083282 / 0.424275 (-0.340993) | 0.012352 / 0.007607 (0.004745) | 0.543058 / 0.226044 (0.317013) | 5.458186 / 2.268929 (3.189258) | 2.562159 / 55.444624 (-52.882466) | 2.198810 / 6.876477 (-4.677667) | 2.238976 / 2.142072 (0.096903) | 0.810958 / 4.805227 (-3.994269) | 0.153341 / 6.500664 (-6.347323) | 0.067773 / 0.075469 (-0.007696) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.303938 / 1.841788 (-0.537850) | 14.170363 / 8.074308 (6.096055) | 13.727012 / 10.191392 (3.535620) | 0.129118 / 0.680424 (-0.551306) | 0.016746 / 0.534201 (-0.517455) | 0.382759 / 0.579283 (-0.196524) | 0.391070 / 0.434364 (-0.043294) | 0.461197 / 0.540337 (-0.079141) | 0.557641 / 1.386936 (-0.829295) |

![](https://cml.dev/watermark.png#004bba88db03fb87d57252e38a4d7abdb0a5f0a9 "CML watermark")

## What's in this PR?

As discussed with @lhoestq in https://github.com/huggingface/datasets/pull/5522, the license for `NumpyFormatter` at `datasets/formatting/np_formatter.py` was missing, but present on the rest of the `formatting/*.py` files. So this PR is basically to include it there.


Fix `datasets.load_from_disk`, `DatasetDict.load_from_disk` and `Dataset.load_from_disk`

Hmm, should this also be updated in `Dataset.load_from_disk` and `DatasetDict.load_from_disk`? https://github.com/huggingface/datasets/pull/5466 As there the paths are joined using `Path(..., ...)` and it won't work on Windows OS according to that PR, right?

Hi, @lhoestq could you review this PR? Thank you in advance and sorry for the ping 🤗

Besides that, I was also thinking of adding a `skip_validation` boolean arg in both `Dataset.load_from_disk` and `DatasetDict.load_from_disk` to avoid duplicating those calls too when those functions are called from `datasets.load_from_disk`.

So that `skip_validation` is set to `False` by default, but passed as `True` if called from `datasets.load_from_disk`, and that just affects the file checking part of the code on both functions, do you agree @lhoestq?

> I think we should always verify

But with the current way we're also verifying twice right? First on `datasets.load_from_disk` then on `Dataset.load_from_disk`, right?

Maybe a warning before calling either `Dataset.load_from_disk` or `DatasetDict.load_from_disk` is enough?

e.g. **"Consider using `Dataset.load_from_disk` instead to avoid `fsspec` from verifying the presence of `dataset_info.json` and `state.json` in the remote filesystem twice."** to be showed just when `fs` is remote.

> I don't think it's worth adding a new argument just for that. Usually we keep the set of arguments to the strict minimum

What about the warning?

Anyway, if you don't think that's worth it feel free to merge 👍🏻

> What about the warning?

We may show warnings for suggestions, but only if the user does a very unoptimized thing. Here we're not at that level ^^'

Thanks for the explanation and feedback @lhoestq 🤗

> Thank you :) Added my last suggestions:

Thanks for the feedback, I agree with everything besides one nit! 👍🏻

Show benchmarks

PyArrow==6.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.011556 / 0.011353 (0.000203) | 0.006213 / 0.011008 (-0.004796) | 0.132390 / 0.038508 (0.093882) | 0.034609 / 0.023109 (0.011500) | 0.361156 / 0.275898 (0.085258) | 0.402524 / 0.323480 (0.079044) | 0.009138 / 0.007986 (0.001152) | 0.005728 / 0.004328 (0.001399) | 0.115406 / 0.004250 (0.111156) | 0.041440 / 0.037052 (0.004388) | 0.370232 / 0.258489 (0.111742) | 0.409944 / 0.293841 (0.116103) | 0.053803 / 0.128546 (-0.074744) | 0.022029 / 0.075646 (-0.053617) | 0.400325 / 0.419271 (-0.018946) | 0.055324 / 0.043533 (0.011791) | 0.368699 / 0.255139 (0.113560) | 0.391836 / 0.283200 (0.108636) | 0.099356 / 0.141683 (-0.042327) | 1.687881 / 1.452155 (0.235726) | 1.752202 / 1.492716 (0.259485) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.012992 / 0.018006 (-0.005014) | 0.518756 / 0.000490 (0.518267) | 0.004702 / 0.000200 (0.004502) | 0.000105 / 0.000054 (0.000050) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028371 / 0.037411 (-0.009041) | 0.127058 / 0.014526 (0.112532) | 0.136908 / 0.176557 (-0.039649) | 0.210168 / 0.737135 (-0.526968) | 0.139600 / 0.296338 (-0.156738) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.570901 / 0.215209 (0.355692) | 5.967213 / 2.077655 (3.889558) | 2.286745 / 1.504120 (0.782626) | 1.950682 / 1.541195 (0.409487) | 2.062536 / 1.468490 (0.594046) | 1.255671 / 4.584777 (-3.329106) | 5.454951 / 3.745712 (1.709238) | 3.076429 / 5.269862 (-2.193433) | 2.082871 / 4.565676 (-2.482806) | 0.150069 / 0.424275 (-0.274206) | 0.014864 / 0.007607 (0.007257) | 0.774672 / 0.226044 (0.548627) | 7.873992 / 2.268929 (5.605064) | 3.196165 / 55.444624 (-52.248459) | 2.366854 / 6.876477 (-4.509623) | 2.407381 / 2.142072 (0.265309) | 1.419130 / 4.805227 (-3.386097) | 0.249210 / 6.500664 (-6.251454) | 0.088648 / 0.075469 (0.013179) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.528368 / 1.841788 (-0.313420) | 17.554000 / 8.074308 (9.479692) | 20.773300 / 10.191392 (10.581908) | 0.216903 / 0.680424 (-0.463521) | 0.046544 / 0.534201 (-0.487657) | 0.538238 / 0.579283 (-0.041045) | 0.673926 / 0.434364 (0.239562) | 0.656108 / 0.540337 (0.115770) | 0.774026 / 1.386936 (-0.612910) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010177 / 0.011353 (-0.001176) | 0.006334 / 0.011008 (-0.004675) | 0.100097 / 0.038508 (0.061589) | 0.039996 / 0.023109 (0.016887) | 0.420225 / 0.275898 (0.144327) | 0.437694 / 0.323480 (0.114214) | 0.007987 / 0.007986 (0.000002) | 0.005782 / 0.004328 (0.001454) | 0.106421 / 0.004250 (0.102171) | 0.046993 / 0.037052 (0.009941) | 0.397304 / 0.258489 (0.138815) | 0.441780 / 0.293841 (0.147939) | 0.064594 / 0.128546 (-0.063952) | 0.020823 / 0.075646 (-0.054823) | 0.108854 / 0.419271 (-0.310417) | 0.076457 / 0.043533 (0.032924) | 0.401712 / 0.255139 (0.146573) | 0.459292 / 0.283200 (0.176093) | 0.125044 / 0.141683 (-0.016639) | 1.765531 / 1.452155 (0.313377) | 1.845429 / 1.492716 (0.352713) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.225549 / 0.018006 (0.207543) | 0.524402 / 0.000490 (0.523913) | 0.006994 / 0.000200 (0.006794) | 0.000120 / 0.000054 (0.000065) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033787 / 0.037411 (-0.003624) | 0.144895 / 0.014526 (0.130369) | 0.147185 / 0.176557 (-0.029371) | 0.228227 / 0.737135 (-0.508908) | 0.164967 / 0.296338 (-0.131371) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.628242 / 0.215209 (0.413033) | 6.348176 / 2.077655 (4.270522) | 2.615832 / 1.504120 (1.111712) | 2.217481 / 1.541195 (0.676286) | 2.287058 / 1.468490 (0.818568) | 1.322854 / 4.584777 (-3.261923) | 5.547831 / 3.745712 (1.802119) | 3.199467 / 5.269862 (-2.070395) | 2.135297 / 4.565676 (-2.430380) | 0.165134 / 0.424275 (-0.259141) | 0.014753 / 0.007607 (0.007146) | 0.778579 / 0.226044 (0.552535) | 7.982329 / 2.268929 (5.713401) | 3.331712 / 55.444624 (-52.112913) | 2.642606 / 6.876477 (-4.233871) | 2.699362 / 2.142072 (0.557290) | 1.572268 / 4.805227 (-3.232959) | 0.273348 / 6.500664 (-6.227316) | 0.082975 / 0.075469 (0.007506) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.730421 / 1.841788 (-0.111367) | 18.154495 / 8.074308 (10.080187) | 20.969885 / 10.191392 (10.778493) | 0.233652 / 0.680424 (-0.446772) | 0.026609 / 0.534201 (-0.507592) | 0.546874 / 0.579283 (-0.032410) | 0.602891 / 0.434364 (0.168527) | 0.641073 / 0.540337 (0.100736) | 0.772138 / 1.386936 (-0.614798) |

![](https://cml.dev/watermark.png#20703458e3c42ee7bfc1a26e47805c0db4dda2d6 "CML watermark")

## What's in this PR?

After playing around a little bit with 🤗`datasets` in Google Cloud Storage (GCS), I found out some things that should be fixed IMO in the code:

* `datasets.load_from_disk` is not checking whether `state.json` is there too when trying to load a `Dataset`, just `dataset_info.json` is checked
* `DatasetDict.load_from_disk` is not checking whether `state.json` is there too when redirecting the user to load it as `datasets.load_from_disk`, just `dataset_info.json` is checked, which is misleading, as it won't be loadable that way either
* `Dataset.load_from_disk` is missing the `extract_path_from_uri` call before checking in the `fs` whether `dataset_info.json` and `dataset_dict.json` exist, which when using `gcsfs` leads to 400 error code (not blocking) due to `gcsfs.retry.HttpError: Invalid bucket name: 'gs:', 400`
* And, finally, the exception messages are a little bit misleading / incomplete IMO so I've tried to include all the relevant information in the messages to avoid issues when interpreting the exceptions


Push to hub in a pull request

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5528). All of your documentation changes will be reflected on that endpoint.

It seems that the parameter `create_pr` is available for [`0.8.0`](https://huggingface.co/docs/huggingface_hub/v0.8.1/en/package_reference/hf_api#huggingface_hub.HfApi.upload_file) (its not here: [`0.7.0`](https://huggingface.co/docs/huggingface_hub/v0.7.0.rc0/en/package_reference/hf_api#huggingface_hub.HfApi.upload_file)) and onwards. I included a warning, informing the user that no PR was created.

@nateraw you are completely right! Actually, the dataset shards is never added to the created pr, only the metadata, as the code is now. Ill look into you suggestion asap. Thank!

@nateraw Nothing more to add, that's a perfect usage of `huggingface_hub` as far as I can tell ! :fire:

A very nit improvement would be to use the [for .. else ... python statement](https://book.pythontips.com/en/latest/for_-_else.html).
i.e:

```py
if create_pr is True and revision is not None:
    for discussion in get_repo_discussions(repo_id, repo_type='dataset'):
        if discussion.is_pull_request and discussion.git_reference == revision:
            create_pr = False
            break
    else:
        raise ValueError("Provided revision not found")
```
No need for the `revision_found` temporary flag when do so. Yeah ok, it's niche :wink:

I added the suggestions from @nateraw and @Wauplin .

> Thanks. Some comments/suggestions below...
>
> Why have you removed the test for create_pr? You could add it again and just add a pytest skipif when version of huggingface_hub is lower than 0.8.1.

I have added the test again. I removed it because i kept getting errors when calling `create_pull_request` with `repo_id=ds_name` where `temporary_repo = ds_name`, and thought i might look more thoroughly at it later. I have added a test called `test_test` showing this, it gives:
```
tests/test_upstream_hub.py:360:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py:124: in _inner_fn
    return fn(*args, **kwargs)
.venv/lib/python3.10/site-packages/huggingface_hub/hf_api.py:3451: in create_pull_request
    return self.create_discussion(
.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py:124: in _inner_fn
    return fn(*args, **kwargs)
.venv/lib/python3.10/site-packages/huggingface_hub/hf_api.py:3393: in create_discussion
    hf_raise_for_status(resp)
(...)
E               huggingface_hub.utils._errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-63ecd2cb-2cf2557a332c86ad27f687b3)
E
E               Repository Not Found for url: https://huggingface.co/api/models/__DUMMY_TRANSFORMERS_USER__/test-16764648321590/discussions.
E               Please make sure you specified the correct `repo_id` and `repo_type`.
E               If you are trying to access a private or gated repo, make sure you are authenticated.
E               Invalid username or password.
```

> > Thanks. Some comments/suggestions below...
> > Why have you removed the test for create_pr? You could add it again and just add a pytest skipif when version of huggingface_hub is lower than 0.8.1.
>
> I have added the test again. I removed it because i kept getting errors when calling `create_pull_request` with `repo_id=ds_name` where `temporary_repo = ds_name`, and thought i might look more thoroughly at it later. I have added a test called `test_test` showing this, it gives:
>
> ```
> tests/test_upstream_hub.py:360:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> .venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py:124: in _inner_fn
>     return fn(*args, **kwargs)
> .venv/lib/python3.10/site-packages/huggingface_hub/hf_api.py:3451: in create_pull_request
>     return self.create_discussion(
> .venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py:124: in _inner_fn
>     return fn(*args, **kwargs)
> .venv/lib/python3.10/site-packages/huggingface_hub/hf_api.py:3393: in create_discussion
>     hf_raise_for_status(resp)
> (...)
> E               huggingface_hub.utils._errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-63ecd2cb-2cf2557a332c86ad27f687b3)
> E
> E               Repository Not Found for url: https://huggingface.co/api/models/__DUMMY_TRANSFORMERS_USER__/test-16764648321590/discussions.
> E               Please make sure you specified the correct `repo_id` and `repo_type`.
> E               If you are trying to access a private or gated repo, make sure you are authenticated.
> E               Invalid username or password.
> ```

@albertvillanova, @lhoestq , FYI I have looked at this again, and i haven't figured it out, so the test`test_push_dataset_to_hub_with_pull_request` and the minimal example `test_test` are still failing locally, while the other tests succeed. Do you have any advice?

I tried to move all of the "create pr safely"-logic to a seperate function in `_hf_hub_fixes`. I looked at how the exceptions were raised before `huggingface_hub.utils.RepositoryNotFoundError`existed, and make changes accordingly.

`create_pr` was set during `push_to_hub`, even though it was `None` from the outset, hence causing tests to fail for older versions of `huggingface_hub`. This is now fixed.

With the implementation of `_hf_hub_fixes.upload_file` the function call expected `commit_message`, `commit_description`. If these are not set we call the function without them, even though we are on a version of `huggingface_hub` where they are not available in `upload_file`.

When `huggingface_hub < 0.5.0` we assume `repo_id` of them form `organisation/name`, so now that we are calling `create_repo` in the tests with `repo_id` not of this form, we need to handle this case, this is now done.

Many tests failed for `dataset_dict` for the above reasons, so the fixes from `arrow_dataset.py` were also added to `dataset_dict.py`.

**All tests are now passing locally for `huggingface_hub==0.2.0` and `huggingface_hub==0.12.1`…** Im sorry I should have downgraded and went through this a long time ago, but I didn’t realise the extend of these version fixes until recently…

Hi ! FYI bumped the `huggingface-hub` dependency to 0.11 and removed the `_hf_hub_fixes.py` - which should make this PR much easier

Just now finding this - seems like a cool issue to contribute to. If any more help is needed please ping me! @AJDERS

Fixes #5492.

Introduce new kwarg `create_pr` in `push_to_hub`, which is passed to `HFapi.upload_file`.