Datasets.map is severely broken

Hi! Instead of processing a single example at a time, you should use the batched `map` for the best performance (with `num_proc=1`) - the fast tokenizers can process a batch's samples in parallel in that scenario.

E.g., the following code in Colab takes an hour to complete:
```python
# !pip install datasets transformers
from datasets import load_dataset
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
dataset = dataset.map(lambda ex: tokenizer(ex["text"]), batched=True, remove_columns=["text", "meta"])
```

Batched is far worse. A single batch of 1000 took hours and that was only 1%

On Thu, Oct 19, 2023, 2:26 PM Mario Šaško ***@***.***> wrote:

> Hi! You should use the batched map for the best performance (with
> num_proc=1) - the fast tokenizers can process a batch's samples in
> parallel.
>
> E.g., the following code in Colab takes an hour to complete:
>
> # !pip install datasets transformersfrom datasets import load_datasetfrom transformers import AutoTokenizertokenizer = AutoTokenizer.from_pretrained("bert-base-cased")dataset = dataset.map(lambda ex: tokenizer(ex["text"]), batched=True, remove_columns=["text", "meta"])
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/6319#issuecomment-1771503757>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABDD3ZJHPSRVDEXFNMXR2N3YAFWFZAVCNFSM6AAAAAA6HDKPSCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZRGUYDGNZVG4>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>

Which specific version of datasets are you using?

What is the architecture of your colab setup? Ram? Cores? OS?

On Thu, Oct 19, 2023, 2:27 PM pensive introvert ***@***.***>
wrote:

> Batched is far worse. A single batch of 1000 took hours and that was only
> 1%
>
>
> On Thu, Oct 19, 2023, 2:26 PM Mario Šaško ***@***.***>
> wrote:
>
>> Hi! You should use the batched map for the best performance (with
>> num_proc=1) - the fast tokenizers can process a batch's samples in
>> parallel.
>>
>> E.g., the following code in Colab takes an hour to complete:
>>
>> # !pip install datasets transformersfrom datasets import load_datasetfrom transformers import AutoTokenizertokenizer = AutoTokenizer.from_pretrained("bert-base-cased")dataset = dataset.map(lambda ex: tokenizer(ex["text"]), batched=True, remove_columns=["text", "meta"])
>>
>> —
>> Reply to this email directly, view it on GitHub
>> <https://github.com/huggingface/datasets/issues/6319#issuecomment-1771503757>,
>> or unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/ABDD3ZJHPSRVDEXFNMXR2N3YAFWFZAVCNFSM6AAAAAA6HDKPSCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZRGUYDGNZVG4>
>> .
>> You are receiving this because you authored the thread.Message ID:
>> ***@***.***>
>>
>

from functools import partial
import transformers
from datasets import load_dataset, concatenate_datasets, load_from_disk

model_name_or_path="/opt/data/data/daryl149/llama-2-7b-chat-hf"
output_dir="/opt/data/data/LongLoRA/checkpoints"
cache_dir="/opt/data/data/LongLoRA/cache"
model_max_length=16384

IGNORE_INDEX = -100
DEFAULT_PAD_TOKEN = "[PAD]"
DEFAULT_EOS_TOKEN = "</s>"
DEFAULT_BOS_TOKEN = "<s>"
DEFAULT_UNK_TOKEN = "<unk>"

tokenizer = transformers.LlamaTokenizerFast.from_pretrained(
        model_name_or_path,
        cache_dir=cache_dir,
        model_max_length=model_max_length,
        padding_side="right",
        use_fast=True,
        #use_fast=False
)

special_tokens_dict = dict()
if tokenizer.pad_token is None:
    special_tokens_dict["pad_token"] = DEFAULT_PAD_TOKEN
if tokenizer.eos_token is None:
    special_tokens_dict["eos_token"] = DEFAULT_EOS_TOKEN
if tokenizer.bos_token is None:
    special_tokens_dict["bos_token"] = DEFAULT_BOS_TOKEN
if tokenizer.unk_token is None:
    special_tokens_dict["unk_token"] = DEFAULT_UNK_TOKEN

tokenizer.add_special_tokens(special_tokens_dict)

def tokenize_fn(tokenizer, example):
    context_length = tokenizer.model_max_length
    outputs = tokenizer(
        tokenizer.eos_token.join(example["text"]),
        #truncation=False,
        truncation=True,
        return_tensors="pt",
        #return_tensors="np",
        pad_to_multiple_of=context_length,
        padding=True,
    )
    return {"input_ids": outputs["input_ids"].view(-1, context_length)}

for idx in range(100):
    dataset = load_dataset("togethercomputer/RedPajama-Data-1T-Sample",
cache_dir=cache_dir, split=f'train[{idx}%:{idx+1}%]')
    dataset = dataset.map(partial(tokenize_fn, tokenizer), batched=False,
num_proc=16, remove_columns=["text", "meta"])
    dataset.save_to_disk(training_args.cache_dir + f"/training_data_{idx}")

On Thu, Oct 19, 2023 at 2:30 PM Mario Šaško ***@***.***>
wrote:

> Can you please provide a self-contained reproducer?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/6319#issuecomment-1771509229>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABDD3ZNBZ3BE7Q4EQZZK6MLYAFWURAVCNFSM6AAAAAA6HDKPSCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZRGUYDSMRSHE>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>

I changed the tokenizer to one without "Fast suffix, and something changed.
The fraction, although still slowed a lot at 80% was able to get over the
finish line of 100%

I have to do more testng, see if the whole set can be processed

On Thu, Oct 19, 2023 at 3:03 PM pensive introvert <
***@***.***> wrote:

> from functools import partial
> import transformers
> from datasets import load_dataset, concatenate_datasets, load_from_disk
>
> model_name_or_path="/opt/data/data/daryl149/llama-2-7b-chat-hf"
> output_dir="/opt/data/data/LongLoRA/checkpoints"
> cache_dir="/opt/data/data/LongLoRA/cache"
> model_max_length=16384
>
> IGNORE_INDEX = -100
> DEFAULT_PAD_TOKEN = "[PAD]"
> DEFAULT_EOS_TOKEN = "</s>"
> DEFAULT_BOS_TOKEN = "<s>"
> DEFAULT_UNK_TOKEN = "<unk>"
>
>
> tokenizer = transformers.LlamaTokenizerFast.from_pretrained(
>         model_name_or_path,
>         cache_dir=cache_dir,
>         model_max_length=model_max_length,
>         padding_side="right",
>         use_fast=True,
>         #use_fast=False
> )
>
> special_tokens_dict = dict()
> if tokenizer.pad_token is None:
>     special_tokens_dict["pad_token"] = DEFAULT_PAD_TOKEN
> if tokenizer.eos_token is None:
>     special_tokens_dict["eos_token"] = DEFAULT_EOS_TOKEN
> if tokenizer.bos_token is None:
>     special_tokens_dict["bos_token"] = DEFAULT_BOS_TOKEN
> if tokenizer.unk_token is None:
>     special_tokens_dict["unk_token"] = DEFAULT_UNK_TOKEN
>
> tokenizer.add_special_tokens(special_tokens_dict)
>
> def tokenize_fn(tokenizer, example):
>     context_length = tokenizer.model_max_length
>     outputs = tokenizer(
>         tokenizer.eos_token.join(example["text"]),
>         #truncation=False,
>         truncation=True,
>         return_tensors="pt",
>         #return_tensors="np",
>         pad_to_multiple_of=context_length,
>         padding=True,
>     )
>     return {"input_ids": outputs["input_ids"].view(-1, context_length)}
>
> for idx in range(100):
>     dataset = load_dataset("togethercomputer/RedPajama-Data-1T-Sample",
> cache_dir=cache_dir, split=f'train[{idx}%:{idx+1}%]')
>     dataset = dataset.map(partial(tokenize_fn, tokenizer), batched=False,
> num_proc=16, remove_columns=["text", "meta"])
>     dataset.save_to_disk(training_args.cache_dir + f"/training_data_{idx}")
>
>
> On Thu, Oct 19, 2023 at 2:30 PM Mario Šaško ***@***.***>
> wrote:
>
>> Can you please provide a self-contained reproducer?
>>
>> —
>> Reply to this email directly, view it on GitHub
>> <https://github.com/huggingface/datasets/issues/6319#issuecomment-1771509229>,
>> or unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/ABDD3ZNBZ3BE7Q4EQZZK6MLYAFWURAVCNFSM6AAAAAA6HDKPSCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZRGUYDSMRSHE>
>> .
>> You are receiving this because you authored the thread.Message ID:
>> ***@***.***>
>>
>

So, using LlamaTokenizerFast was the problem. Changing it to LlamaTokenizer
fixed things,

On Thu, Oct 19, 2023 at 4:04 PM pensive introvert <
***@***.***> wrote:

> I changed the tokenizer to one without "Fast suffix, and something
> changed. The fraction, although still slowed a lot at 80% was able to get
> over the finish line of 100%
>
> I have to do more testng, see if the whole set can be processed
>
>
>
> On Thu, Oct 19, 2023 at 3:03 PM pensive introvert <
> ***@***.***> wrote:
>
>> from functools import partial
>> import transformers
>> from datasets import load_dataset, concatenate_datasets, load_from_disk
>>
>> model_name_or_path="/opt/data/data/daryl149/llama-2-7b-chat-hf"
>> output_dir="/opt/data/data/LongLoRA/checkpoints"
>> cache_dir="/opt/data/data/LongLoRA/cache"
>> model_max_length=16384
>>
>> IGNORE_INDEX = -100
>> DEFAULT_PAD_TOKEN = "[PAD]"
>> DEFAULT_EOS_TOKEN = "</s>"
>> DEFAULT_BOS_TOKEN = "<s>"
>> DEFAULT_UNK_TOKEN = "<unk>"
>>
>>
>> tokenizer = transformers.LlamaTokenizerFast.from_pretrained(
>>         model_name_or_path,
>>         cache_dir=cache_dir,
>>         model_max_length=model_max_length,
>>         padding_side="right",
>>         use_fast=True,
>>         #use_fast=False
>> )
>>
>> special_tokens_dict = dict()
>> if tokenizer.pad_token is None:
>>     special_tokens_dict["pad_token"] = DEFAULT_PAD_TOKEN
>> if tokenizer.eos_token is None:
>>     special_tokens_dict["eos_token"] = DEFAULT_EOS_TOKEN
>> if tokenizer.bos_token is None:
>>     special_tokens_dict["bos_token"] = DEFAULT_BOS_TOKEN
>> if tokenizer.unk_token is None:
>>     special_tokens_dict["unk_token"] = DEFAULT_UNK_TOKEN
>>
>> tokenizer.add_special_tokens(special_tokens_dict)
>>
>> def tokenize_fn(tokenizer, example):
>>     context_length = tokenizer.model_max_length
>>     outputs = tokenizer(
>>         tokenizer.eos_token.join(example["text"]),
>>         #truncation=False,
>>         truncation=True,
>>         return_tensors="pt",
>>         #return_tensors="np",
>>         pad_to_multiple_of=context_length,
>>         padding=True,
>>     )
>>     return {"input_ids": outputs["input_ids"].view(-1, context_length)}
>>
>> for idx in range(100):
>>     dataset = load_dataset("togethercomputer/RedPajama-Data-1T-Sample",
>> cache_dir=cache_dir, split=f'train[{idx}%:{idx+1}%]')
>>     dataset = dataset.map(partial(tokenize_fn, tokenizer), batched=False,
>> num_proc=16, remove_columns=["text", "meta"])
>>     dataset.save_to_disk(training_args.cache_dir +
>> f"/training_data_{idx}")
>>
>>
>> On Thu, Oct 19, 2023 at 2:30 PM Mario Šaško ***@***.***>
>> wrote:
>>
>>> Can you please provide a self-contained reproducer?
>>>
>>> —
>>> Reply to this email directly, view it on GitHub
>>> <https://github.com/huggingface/datasets/issues/6319#issuecomment-1771509229>,
>>> or unsubscribe
>>> <https://github.com/notifications/unsubscribe-auth/ABDD3ZNBZ3BE7Q4EQZZK6MLYAFWURAVCNFSM6AAAAAA6HDKPSCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZRGUYDSMRSHE>
>>> .
>>> You are receiving this because you authored the thread.Message ID:
>>> ***@***.***>
>>>
>>

Indeed, the tokenizer is super slow. Perhaps @ArthurZucker knows the reason why.

([This](https://colab.research.google.com/drive/1VgeurX-4Fl2X6aBQTwh_X4kuQKZ6K9L1?usp=sharing) simplified Colab can be used to reproduce the behavior)

same issue here
sample to reproduce: https://github.com/philschmid/document-ai-transformers/blob/main/training/donut_sroie.ipynb
with following map line
https://github.com/philschmid/document-ai-transformers/blob/main/training/donut_sroie.ipynb

If I directly iterate over the dataset and call the mapping method, it is very fast
```py
for sample in dataset:
    def preprocess_documents_for_donut(sample):
```

if i removed `.convert('RGB')` It can run to completion without getting stuck. I suspect it has something to do with the Image.

If I use batch, it's even slower.

@ewfian

> If I directly iterate over the dataset and call the mapping method, it is very fast

`Dataset.map` must also convert the images into bytes to write them to an Arrow file (the write itself takes some time, too).

You can make the `map` faster by manually converting the images into an "arrow-compatible" representation. Otherwise,  the Pillow defaults are used when saving an image, which seems particularly slow for the notebook's case.

```python
def preprocess_documents_for_donut(sample):
    text = json.loads(sample["text"])
    d_doc = task_start_token + json2token(text) + eos_token
    image = sample["image"].convert('RGB')
    # convert image to bytes
    buffer = io.BytesIO()
    image.save(buffer, format="PNG", compress_level=1)
    return {"image": {"bytes": buffer.getvalue()}, "text": d_doc}

proc_dataset = dataset.map(preprocess_documents_for_donut, writer_batch_size=50)
```

The problem I had was to do with map using fork and copying locks from the
parent process in acquired state. I ended up changing the context to use
forkserver instead.

On Wed, Nov 29, 2023, 10:04 PM Mario Šaško ***@***.***> wrote:

> @ewfian <https://github.com/ewfian>
>
> If I directly iterate over the dataset and call the mapping method, it is
> very fast
>
> Dataset.map must also convert the images into bytes to write them to an
> Arrow file (the write itself takes some time, too).
>
> You can make the map faster by manually converting the images into an
> "arrow-compatible" representation. Otherwise, the Pillow defaults are used
> when saving an image, which seems particularly slow for the notebook's case.
>
> def preprocess_documents_for_donut(sample):
>     text = json.loads(sample["text"])
>     d_doc = task_start_token + json2token(text) + eos_token
>     image = sample["image"].convert('RGB')
>     # convert image to bytes
>     buffer = io.BytesIO()
>     image.save(buffer, format="PNG", compress_level=1)
>     return {"image": {"bytes": buffer.getvalue()}, "text": d_doc}
> proc_dataset = dataset.map(preprocess_documents_for_donut, writer_batch_size=50)
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/6319#issuecomment-1833033973>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABDD3ZKKEKJVWBFH7QHLRJ3YG7ZUJAVCNFSM6AAAAAA6HDKPSCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZTGAZTGOJXGM>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>

### Describe the bug

Regardless of how many cores I used, I have 16 or 32 threads, map slows down to a crawl at around 80% done, lingers maybe until 97% extremely slowly and NEVER finishes the job. It just hangs.

After watching this for 27 hours I control-C out of it. Until the end one process appears to be doing something, but it never ends.

I saw some comments about fast tokenizers using Rust and all and tried different variations. NOTHING works.

### Steps to reproduce the bug

Running it without breaking the dataset into parts results in the same behavior. The loop was an attempt to see if this was a RAM issue.

for idx in range(100):
    dataset = load_dataset("togethercomputer/RedPajama-Data-1T-Sample", cache_dir=cache_dir, split=f'train[{idx}%:{idx+1}%]')
    dataset = dataset.map(partial(tokenize_fn, tokenizer), batched=False, num_proc=1, remove_columns=["text", "meta"])
    dataset.save_to_disk(training_args.cache_dir + f"/training_data_{idx}")

### Expected behavior

I expect map to run at more or less the same speed it starts with and FINISH its processing.

### Environment info

Python 3.8, same with 3.10 makes no difference.
Ubuntu 20.04,


Deterministic set hash

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006827 / 0.011353 (-0.004526) | 0.004468 / 0.011008 (-0.006540) | 0.088687 / 0.038508 (0.050179) | 0.072560 / 0.023109 (0.049451) | 0.333421 / 0.275898 (0.057523) | 0.374977 / 0.323480 (0.051497) | 0.005829 / 0.007986 (-0.002156) | 0.003284 / 0.004328 (-0.001045) | 0.068929 / 0.004250 (0.064678) | 0.057212 / 0.037052 (0.020160) | 0.328911 / 0.258489 (0.070422) | 0.389107 / 0.293841 (0.095266) | 0.033518 / 0.128546 (-0.095029) | 0.009919 / 0.075646 (-0.065728) | 0.308100 / 0.419271 (-0.111171) | 0.059380 / 0.043533 (0.015847) | 0.345587 / 0.255139 (0.090448) | 0.353703 / 0.283200 (0.070503) | 0.026454 / 0.141683 (-0.115229) | 1.573309 / 1.452155 (0.121155) | 1.663812 / 1.492716 (0.171095) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.255081 / 0.018006 (0.237075) | 0.472613 / 0.000490 (0.472123) | 0.016120 / 0.000200 (0.015920) | 0.000383 / 0.000054 (0.000328) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028219 / 0.037411 (-0.009192) | 0.086600 / 0.014526 (0.072074) | 0.099484 / 0.176557 (-0.077073) | 0.154604 / 0.737135 (-0.582531) | 0.099168 / 0.296338 (-0.197171) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.421703 / 0.215209 (0.206494) | 4.188600 / 2.077655 (2.110945) | 2.037575 / 1.504120 (0.533456) | 1.843389 / 1.541195 (0.302194) | 1.912554 / 1.468490 (0.444064) | 0.517452 / 4.584777 (-4.067325) | 3.838002 / 3.745712 (0.092290) | 3.698899 / 5.269862 (-1.570963) | 2.175393 / 4.565676 (-2.390283) | 0.066059 / 0.424275 (-0.358216) | 0.008455 / 0.007607 (0.000848) | 0.506813 / 0.226044 (0.280768) | 4.826994 / 2.268929 (2.558066) | 2.544437 / 55.444624 (-52.900187) | 2.164938 / 6.876477 (-4.711539) | 2.171725 / 2.142072 (0.029652) | 0.603757 / 4.805227 (-4.201470) | 0.149113 / 6.500664 (-6.351551) | 0.065093 / 0.075469 (-0.010376) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.366887 / 1.841788 (-0.474901) | 20.508089 / 8.074308 (12.433780) | 14.836531 / 10.191392 (4.645139) | 0.167418 / 0.680424 (-0.513006) | 0.019707 / 0.534201 (-0.514494) | 0.409897 / 0.579283 (-0.169387) | 0.439412 / 0.434364 (0.005048) | 0.495784 / 0.540337 (-0.044553) | 0.685367 / 1.386936 (-0.701569) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007604 / 0.011353 (-0.003749) | 0.004368 / 0.011008 (-0.006640) | 0.072628 / 0.038508 (0.034120) | 0.084187 / 0.023109 (0.061077) | 0.461396 / 0.275898 (0.185498) | 0.481429 / 0.323480 (0.157949) | 0.005894 / 0.007986 (-0.002092) | 0.003472 / 0.004328 (-0.000857) | 0.068717 / 0.004250 (0.064466) | 0.061066 / 0.037052 (0.024014) | 0.464217 / 0.258489 (0.205728) | 0.498061 / 0.293841 (0.204220) | 0.035458 / 0.128546 (-0.093089) | 0.009474 / 0.075646 (-0.066173) | 0.079633 / 0.419271 (-0.339639) | 0.053966 / 0.043533 (0.010433) | 0.454911 / 0.255139 (0.199772) | 0.470837 / 0.283200 (0.187637) | 0.026358 / 0.141683 (-0.115325) | 1.665131 / 1.452155 (0.212976) | 1.730365 / 1.492716 (0.237648) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.234810 / 0.018006 (0.216804) | 0.453672 / 0.000490 (0.453183) | 0.004620 / 0.000200 (0.004420) | 0.000119 / 0.000054 (0.000064) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035310 / 0.037411 (-0.002101) | 0.100379 / 0.014526 (0.085853) | 0.118802 / 0.176557 (-0.057754) | 0.173853 / 0.737135 (-0.563282) | 0.115714 / 0.296338 (-0.180624) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.466797 / 0.215209 (0.251588) | 4.698324 / 2.077655 (2.620670) | 2.446897 / 1.504120 (0.942777) | 2.277346 / 1.541195 (0.736151) | 2.347211 / 1.468490 (0.878721) | 0.514377 / 4.584777 (-4.070400) | 3.931269 / 3.745712 (0.185557) | 3.573575 / 5.269862 (-1.696286) | 2.208122 / 4.565676 (-2.357554) | 0.061081 / 0.424275 (-0.363194) | 0.007803 / 0.007607 (0.000196) | 0.544376 / 0.226044 (0.318332) | 5.440003 / 2.268929 (3.171074) | 3.012559 / 55.444624 (-52.432065) | 2.617286 / 6.876477 (-4.259191) | 2.863978 / 2.142072 (0.721906) | 0.610024 / 4.805227 (-4.195203) | 0.133643 / 6.500664 (-6.367021) | 0.064766 / 0.075469 (-0.010703) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.465225 / 1.841788 (-0.376563) | 21.308351 / 8.074308 (13.234043) | 15.176634 / 10.191392 (4.985242) | 0.172701 / 0.680424 (-0.507723) | 0.020345 / 0.534201 (-0.513855) | 0.433923 / 0.579283 (-0.145360) | 0.450183 / 0.434364 (0.015819) | 0.514048 / 0.540337 (-0.026289) | 0.736302 / 1.386936 (-0.650634) |

![](https://cml.dev/watermark.png#7f1a7d621fff3b08ace02643466097654a5e010f "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008305 / 0.011353 (-0.003048) | 0.006007 / 0.011008 (-0.005001) | 0.103521 / 0.038508 (0.065013) | 0.075776 / 0.023109 (0.052666) | 0.378888 / 0.275898 (0.102990) | 0.405245 / 0.323480 (0.081765) | 0.004596 / 0.007986 (-0.003390) | 0.003687 / 0.004328 (-0.000641) | 0.079043 / 0.004250 (0.074792) | 0.055895 / 0.037052 (0.018843) | 0.406565 / 0.258489 (0.148076) | 0.433869 / 0.293841 (0.140028) | 0.045321 / 0.128546 (-0.083226) | 0.014317 / 0.075646 (-0.061329) | 0.345312 / 0.419271 (-0.073960) | 0.064485 / 0.043533 (0.020953) | 0.381744 / 0.255139 (0.126605) | 0.401162 / 0.283200 (0.117962) | 0.035973 / 0.141683 (-0.105709) | 1.829616 / 1.452155 (0.377461) | 1.868487 / 1.492716 (0.375771) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.245432 / 0.018006 (0.227426) | 0.494249 / 0.000490 (0.493759) | 0.010878 / 0.000200 (0.010678) | 0.000492 / 0.000054 (0.000437) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032778 / 0.037411 (-0.004633) | 0.103418 / 0.014526 (0.088892) | 0.108010 / 0.176557 (-0.068547) | 0.176477 / 0.737135 (-0.560658) | 0.107732 / 0.296338 (-0.188606) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.572471 / 0.215209 (0.357262) | 5.647039 / 2.077655 (3.569384) | 2.385069 / 1.504120 (0.880949) | 2.048928 / 1.541195 (0.507733) | 2.108538 / 1.468490 (0.640048) | 0.861436 / 4.584777 (-3.723341) | 4.933452 / 3.745712 (1.187739) | 4.735219 / 5.269862 (-0.534642) | 2.926971 / 4.565676 (-1.638705) | 0.097687 / 0.424275 (-0.326588) | 0.008346 / 0.007607 (0.000739) | 0.677754 / 0.226044 (0.451709) | 6.798433 / 2.268929 (4.529504) | 3.129862 / 55.444624 (-52.314762) | 2.454033 / 6.876477 (-4.422444) | 2.464590 / 2.142072 (0.322517) | 1.034497 / 4.805227 (-3.770730) | 0.205753 / 6.500664 (-6.294911) | 0.076618 / 0.075469 (0.001149) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.617569 / 1.841788 (-0.224219) | 22.091489 / 8.074308 (14.017181) | 20.406312 / 10.191392 (10.214920) | 0.222012 / 0.680424 (-0.458411) | 0.027787 / 0.534201 (-0.506414) | 0.441669 / 0.579283 (-0.137615) | 0.564773 / 0.434364 (0.130409) | 0.510389 / 0.540337 (-0.029948) | 0.753672 / 1.386936 (-0.633264) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.011107 / 0.011353 (-0.000246) | 0.004973 / 0.011008 (-0.006035) | 0.078331 / 0.038508 (0.039823) | 0.083964 / 0.023109 (0.060855) | 0.518980 / 0.275898 (0.243082) | 0.528264 / 0.323480 (0.204784) | 0.007452 / 0.007986 (-0.000534) | 0.003931 / 0.004328 (-0.000397) | 0.079724 / 0.004250 (0.075474) | 0.061739 / 0.037052 (0.024686) | 0.517804 / 0.258489 (0.259315) | 0.582764 / 0.293841 (0.288923) | 0.049674 / 0.128546 (-0.078873) | 0.014540 / 0.075646 (-0.061106) | 0.093130 / 0.419271 (-0.326141) | 0.060647 / 0.043533 (0.017114) | 0.492628 / 0.255139 (0.237489) | 0.549761 / 0.283200 (0.266562) | 0.034313 / 0.141683 (-0.107369) | 1.824574 / 1.452155 (0.372419) | 2.013664 / 1.492716 (0.520947) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.231335 / 0.018006 (0.213329) | 0.521477 / 0.000490 (0.520987) | 0.011314 / 0.000200 (0.011114) | 0.000397 / 0.000054 (0.000343) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033303 / 0.037411 (-0.004108) | 0.098238 / 0.014526 (0.083712) | 0.119527 / 0.176557 (-0.057030) | 0.169163 / 0.737135 (-0.567972) | 0.114536 / 0.296338 (-0.181803) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.578401 / 0.215209 (0.363191) | 5.966438 / 2.077655 (3.888783) | 2.646370 / 1.504120 (1.142250) | 2.361833 / 1.541195 (0.820638) | 2.476573 / 1.468490 (1.008083) | 0.777411 / 4.584777 (-3.807366) | 4.811070 / 3.745712 (1.065357) | 4.314221 / 5.269862 (-0.955641) | 2.743317 / 4.565676 (-1.822359) | 0.110394 / 0.424275 (-0.313881) | 0.008333 / 0.007607 (0.000726) | 0.729588 / 0.226044 (0.503543) | 7.743226 / 2.268929 (5.474298) | 3.606294 / 55.444624 (-51.838330) | 2.838069 / 6.876477 (-4.038408) | 3.087494 / 2.142072 (0.945421) | 1.053341 / 4.805227 (-3.751886) | 0.205105 / 6.500664 (-6.295559) | 0.075204 / 0.075469 (-0.000265) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.561959 / 1.841788 (-0.279829) | 21.407849 / 8.074308 (13.333541) | 19.084263 / 10.191392 (8.892871) | 0.226129 / 0.680424 (-0.454295) | 0.029695 / 0.534201 (-0.504506) | 0.427035 / 0.579283 (-0.152248) | 0.565353 / 0.434364 (0.130989) | 0.526789 / 0.540337 (-0.013548) | 0.734820 / 1.386936 (-0.652116) |

![](https://cml.dev/watermark.png#5b52536f4e39df3b98f7e0b03ee71b24c4fff49a "CML watermark")

Sort the items in a set according to their `datasets.fingerprint.Hasher.hash` hash to get a deterministic hash of sets.

This is useful to get deterministic hashes of tokenizers that use a trie based on python sets.

reported in https://github.com/huggingface/datasets/issues/3847


sentiment140 dataset unavailable

Thanks for reporting. We are investigating the issue.

We have opened an issue in the corresponding Hub dataset: https://huggingface.co/datasets/sentiment140/discussions/3

Let's continue the discussion there.

### Describe the bug

loading the dataset using load_dataset("sentiment140") returns the following error
ConnectionError: Couldn't reach http://cs.stanford.edu/people/alecmgo/trainingandtestdata.zip (error 403)

### Steps to reproduce the bug

Run the following code (version should not matter).

```
from datasets import load_dataset
data = load_dataset("sentiment140")
```

### Expected behavior

The dataset should be loaded just like any other.
The main issue is that it is no longer hosted by stanford. It is still available from a [Google Drive Link](https://docs.google.com/file/d/0B04GJPshIjmPRnZManQwWEdTZjg/edit).

### Environment info

- `datasets` version: 2.14.5
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.10.8
- Huggingface_hub version: 0.17.3
- PyArrow version: 13.0.0
- Pandas version: 2.1.1


Fix loading Hub datasets with CSV metadata file

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008896 / 0.011353 (-0.002456) | 0.005811 / 0.011008 (-0.005197) | 0.108582 / 0.038508 (0.070074) | 0.096509 / 0.023109 (0.073399) | 0.481725 / 0.275898 (0.205827) | 0.534743 / 0.323480 (0.211263) | 0.005517 / 0.007986 (-0.002468) | 0.006479 / 0.004328 (0.002151) | 0.081313 / 0.004250 (0.077062) | 0.063578 / 0.037052 (0.026525) | 0.493977 / 0.258489 (0.235488) | 0.551897 / 0.293841 (0.258056) | 0.051835 / 0.128546 (-0.076711) | 0.014105 / 0.075646 (-0.061541) | 0.385866 / 0.419271 (-0.033405) | 0.069131 / 0.043533 (0.025598) | 0.484780 / 0.255139 (0.229641) | 0.493221 / 0.283200 (0.210021) | 0.039560 / 0.141683 (-0.102123) | 1.782331 / 1.452155 (0.330176) | 1.899193 / 1.492716 (0.406477) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.329978 / 0.018006 (0.311972) | 0.600839 / 0.000490 (0.600349) | 0.013187 / 0.000200 (0.012987) | 0.000499 / 0.000054 (0.000444) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031835 / 0.037411 (-0.005576) | 0.103740 / 0.014526 (0.089214) | 0.115875 / 0.176557 (-0.060681) | 0.189880 / 0.737135 (-0.547255) | 0.132614 / 0.296338 (-0.163725) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.596255 / 0.215209 (0.381046) | 5.967993 / 2.077655 (3.890339) | 2.612675 / 1.504120 (1.108555) | 2.251461 / 1.541195 (0.710266) | 2.308585 / 1.468490 (0.840095) | 0.816516 / 4.584777 (-3.768261) | 5.241791 / 3.745712 (1.496079) | 4.680745 / 5.269862 (-0.589117) | 2.997370 / 4.565676 (-1.568307) | 0.098632 / 0.424275 (-0.325643) | 0.010912 / 0.007607 (0.003305) | 0.659092 / 0.226044 (0.433047) | 6.825562 / 2.268929 (4.556634) | 3.323844 / 55.444624 (-52.120780) | 2.796203 / 6.876477 (-4.080274) | 2.946994 / 2.142072 (0.804922) | 1.002814 / 4.805227 (-3.802413) | 0.202613 / 6.500664 (-6.298051) | 0.072011 / 0.075469 (-0.003459) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.613873 / 1.841788 (-0.227914) | 24.500990 / 8.074308 (16.426682) | 21.941599 / 10.191392 (11.750207) | 0.214450 / 0.680424 (-0.465974) | 0.031227 / 0.534201 (-0.502974) | 0.498297 / 0.579283 (-0.080986) | 0.597460 / 0.434364 (0.163096) | 0.558152 / 0.540337 (0.017815) | 0.789693 / 1.386936 (-0.597243) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.011299 / 0.011353 (-0.000053) | 0.005103 / 0.011008 (-0.005905) | 0.083161 / 0.038508 (0.044653) | 0.094201 / 0.023109 (0.071092) | 0.560457 / 0.275898 (0.284559) | 0.590459 / 0.323480 (0.266980) | 0.007059 / 0.007986 (-0.000926) | 0.004418 / 0.004328 (0.000090) | 0.081343 / 0.004250 (0.077093) | 0.067069 / 0.037052 (0.030016) | 0.538137 / 0.258489 (0.279648) | 0.600416 / 0.293841 (0.306575) | 0.049046 / 0.128546 (-0.079500) | 0.014299 / 0.075646 (-0.061347) | 0.093631 / 0.419271 (-0.325641) | 0.062536 / 0.043533 (0.019003) | 0.557238 / 0.255139 (0.302099) | 0.571050 / 0.283200 (0.287850) | 0.035881 / 0.141683 (-0.105802) | 1.918487 / 1.452155 (0.466332) | 2.013979 / 1.492716 (0.521263) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.400995 / 0.018006 (0.382989) | 0.634898 / 0.000490 (0.634408) | 0.041809 / 0.000200 (0.041609) | 0.000279 / 0.000054 (0.000224) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034160 / 0.037411 (-0.003251) | 0.109996 / 0.014526 (0.095470) | 0.124335 / 0.176557 (-0.052222) | 0.188100 / 0.737135 (-0.549035) | 0.135897 / 0.296338 (-0.160442) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.639751 / 0.215209 (0.424542) | 6.403312 / 2.077655 (4.325657) | 3.146453 / 1.504120 (1.642333) | 2.840358 / 1.541195 (1.299164) | 2.908667 / 1.468490 (1.440177) | 0.818767 / 4.584777 (-3.766010) | 5.416939 / 3.745712 (1.671227) | 4.853498 / 5.269862 (-0.416364) | 3.023526 / 4.565676 (-1.542150) | 0.110850 / 0.424275 (-0.313425) | 0.013103 / 0.007607 (0.005496) | 0.799720 / 0.226044 (0.573676) | 7.837704 / 2.268929 (5.568775) | 4.016526 / 55.444624 (-51.428099) | 3.338965 / 6.876477 (-3.537512) | 3.715721 / 2.142072 (1.573648) | 1.088340 / 4.805227 (-3.716887) | 0.213610 / 6.500664 (-6.287054) | 0.079244 / 0.075469 (0.003775) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.833175 / 1.841788 (-0.008612) | 25.307218 / 8.074308 (17.232910) | 23.716075 / 10.191392 (13.524683) | 0.259114 / 0.680424 (-0.421310) | 0.035171 / 0.534201 (-0.499029) | 0.530128 / 0.579283 (-0.049155) | 0.651484 / 0.434364 (0.217120) | 0.589414 / 0.540337 (0.049077) | 0.862691 / 1.386936 (-0.524245) |

![](https://cml.dev/watermark.png#1bdfba93b8a739b9d885b8fb1909d47ff689bbc2 "CML watermark")

Me too, I thought the same... quite surprised... :open_mouth:

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006929 / 0.011353 (-0.004423) | 0.004345 / 0.011008 (-0.006663) | 0.085522 / 0.038508 (0.047014) | 0.083380 / 0.023109 (0.060271) | 0.310332 / 0.275898 (0.034434) | 0.350525 / 0.323480 (0.027045) | 0.004367 / 0.007986 (-0.003618) | 0.005503 / 0.004328 (0.001175) | 0.066311 / 0.004250 (0.062061) | 0.059545 / 0.037052 (0.022492) | 0.314090 / 0.258489 (0.055601) | 0.366661 / 0.293841 (0.072821) | 0.031581 / 0.128546 (-0.096965) | 0.008852 / 0.075646 (-0.066794) | 0.289312 / 0.419271 (-0.129960) | 0.052960 / 0.043533 (0.009427) | 0.308134 / 0.255139 (0.052995) | 0.330342 / 0.283200 (0.047142) | 0.026157 / 0.141683 (-0.115526) | 1.488463 / 1.452155 (0.036308) | 1.561441 / 1.492716 (0.068725) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.327735 / 0.018006 (0.309729) | 0.568162 / 0.000490 (0.567672) | 0.012097 / 0.000200 (0.011897) | 0.000438 / 0.000054 (0.000383) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029503 / 0.037411 (-0.007909) | 0.084327 / 0.014526 (0.069801) | 0.102065 / 0.176557 (-0.074492) | 0.157392 / 0.737135 (-0.579744) | 0.101428 / 0.296338 (-0.194910) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.386767 / 0.215209 (0.171558) | 3.870757 / 2.077655 (1.793102) | 1.870048 / 1.504120 (0.365928) | 1.678221 / 1.541195 (0.137026) | 1.799423 / 1.468490 (0.330933) | 0.477718 / 4.584777 (-4.107059) | 3.618351 / 3.745712 (-0.127361) | 3.577921 / 5.269862 (-1.691941) | 2.146217 / 4.565676 (-2.419459) | 0.056290 / 0.424275 (-0.367985) | 0.007378 / 0.007607 (-0.000229) | 0.460678 / 0.226044 (0.234633) | 4.606243 / 2.268929 (2.337314) | 2.303460 / 55.444624 (-53.141164) | 1.982662 / 6.876477 (-4.893814) | 2.103891 / 2.142072 (-0.038182) | 0.570700 / 4.805227 (-4.234527) | 0.131747 / 6.500664 (-6.368918) | 0.060915 / 0.075469 (-0.014554) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.286364 / 1.841788 (-0.555424) | 20.106330 / 8.074308 (12.032022) | 14.780833 / 10.191392 (4.589441) | 0.164301 / 0.680424 (-0.516123) | 0.018730 / 0.534201 (-0.515471) | 0.398530 / 0.579283 (-0.180754) | 0.418084 / 0.434364 (-0.016280) | 0.468735 / 0.540337 (-0.071602) | 0.690122 / 1.386936 (-0.696814) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007262 / 0.011353 (-0.004091) | 0.004228 / 0.011008 (-0.006780) | 0.065866 / 0.038508 (0.027358) | 0.096151 / 0.023109 (0.073042) | 0.409352 / 0.275898 (0.133454) | 0.441234 / 0.323480 (0.117754) | 0.005946 / 0.007986 (-0.002039) | 0.003630 / 0.004328 (-0.000698) | 0.066271 / 0.004250 (0.062020) | 0.061567 / 0.037052 (0.024515) | 0.409097 / 0.258489 (0.150608) | 0.447675 / 0.293841 (0.153834) | 0.032804 / 0.128546 (-0.095743) | 0.008793 / 0.075646 (-0.066853) | 0.070790 / 0.419271 (-0.348482) | 0.048650 / 0.043533 (0.005117) | 0.411021 / 0.255139 (0.155882) | 0.421398 / 0.283200 (0.138198) | 0.025305 / 0.141683 (-0.116378) | 1.494826 / 1.452155 (0.042671) | 1.580441 / 1.492716 (0.087724) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.321871 / 0.018006 (0.303865) | 0.526471 / 0.000490 (0.525982) | 0.006913 / 0.000200 (0.006713) | 0.000108 / 0.000054 (0.000054) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034889 / 0.037411 (-0.002522) | 0.096096 / 0.014526 (0.081570) | 0.111920 / 0.176557 (-0.064636) | 0.166103 / 0.737135 (-0.571032) | 0.111162 / 0.296338 (-0.185176) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.428037 / 0.215209 (0.212828) | 4.294150 / 2.077655 (2.216495) | 2.270331 / 1.504120 (0.766211) | 2.108235 / 1.541195 (0.567041) | 2.242560 / 1.468490 (0.774070) | 0.489941 / 4.584777 (-4.094836) | 3.688111 / 3.745712 (-0.057601) | 3.450180 / 5.269862 (-1.819681) | 2.175106 / 4.565676 (-2.390570) | 0.057657 / 0.424275 (-0.366619) | 0.007478 / 0.007607 (-0.000130) | 0.505242 / 0.226044 (0.279198) | 5.047817 / 2.268929 (2.778888) | 2.724125 / 55.444624 (-52.720500) | 2.419765 / 6.876477 (-4.456711) | 2.723231 / 2.142072 (0.581159) | 0.602382 / 4.805227 (-4.202846) | 0.132362 / 6.500664 (-6.368302) | 0.060600 / 0.075469 (-0.014869) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.363356 / 1.841788 (-0.478431) | 21.446474 / 8.074308 (13.372165) | 15.074732 / 10.191392 (4.883340) | 0.191837 / 0.680424 (-0.488587) | 0.020565 / 0.534201 (-0.513636) | 0.396692 / 0.579283 (-0.182591) | 0.432390 / 0.434364 (-0.001974) | 0.491747 / 0.540337 (-0.048591) | 0.699203 / 1.386936 (-0.687733) |

![](https://cml.dev/watermark.png#c3a8a87c841426495d3a7ed1863c26660a6a551f "CML watermark")

Currently, the reading of the metadata file infers the file extension (.jsonl or .csv) from the passed filename. However, downloaded files from the Hub don't have file extension. For example:
- the original file: `hf://datasets/__DUMMY_TRANSFORMERS_USER__/test-dataset-5916a4-16977085077831/metadata.jsonl`
- corresponds to the downloaded path: `/tmp/pytest-of-username/pytest-46/cache/datasets/downloads/9f5374dbb470f711f6b89d66a5eec1f19cc96324b26bcbebe29138bda6cb20e6`, which does not have extension

In the case where the metadata file does not have an extension, the reader assumes it is a JSONL file, thus the reported error when trying to read a CSV file as a JSONL one: `ArrowInvalid: JSON parse error: Invalid value. in row 0`

This behavior was introduced by:
- #4837

This PR extracts the metadata file extension from the original filename (instead of the downloaded one) and passes it as a parameter to the read_metadata function.

Fix #6315.


Hub datasets with CSV metadata raise ArrowInvalid: JSON parse error: Invalid value. in row 0

bug

Something isn't working

When trying to load a Hub dataset that contains a CSV metadata file, it raises an `ArrowInvalid` error:
```
E   pyarrow.lib.ArrowInvalid: JSON parse error: Invalid value. in row 0

pyarrow/error.pxi:100: ArrowInvalid
```

See: https://huggingface.co/datasets/lukarape/public_small_papers/discussions/1


Support creating new branch in push_to_hub

This adds support for creating a new branch when pushing a dataset to the hub.  Tested both methods locally and branches are created.


Fix commit message formatting in multi-commit uploads

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006760 / 0.011353 (-0.004593) | 0.003918 / 0.011008 (-0.007091) | 0.084016 / 0.038508 (0.045508) | 0.069927 / 0.023109 (0.046818) | 0.307898 / 0.275898 (0.032000) | 0.337453 / 0.323480 (0.013973) | 0.004132 / 0.007986 (-0.003854) | 0.003248 / 0.004328 (-0.001081) | 0.064526 / 0.004250 (0.060275) | 0.056424 / 0.037052 (0.019371) | 0.316313 / 0.258489 (0.057824) | 0.356302 / 0.293841 (0.062461) | 0.030634 / 0.128546 (-0.097912) | 0.008467 / 0.075646 (-0.067180) | 0.286676 / 0.419271 (-0.132595) | 0.051813 / 0.043533 (0.008280) | 0.309874 / 0.255139 (0.054735) | 0.332513 / 0.283200 (0.049313) | 0.023919 / 0.141683 (-0.117764) | 1.509033 / 1.452155 (0.056878) | 1.549636 / 1.492716 (0.056920) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.221464 / 0.018006 (0.203458) | 0.447873 / 0.000490 (0.447384) | 0.002408 / 0.000200 (0.002208) | 0.000090 / 0.000054 (0.000035) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027634 / 0.037411 (-0.009777) | 0.081802 / 0.014526 (0.067276) | 0.781489 / 0.176557 (0.604933) | 0.165184 / 0.737135 (-0.571951) | 0.121526 / 0.296338 (-0.174813) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.408215 / 0.215209 (0.193006) | 4.091192 / 2.077655 (2.013538) | 2.062608 / 1.504120 (0.558488) | 1.895747 / 1.541195 (0.354552) | 1.873682 / 1.468490 (0.405192) | 0.484184 / 4.584777 (-4.100593) | 3.469096 / 3.745712 (-0.276616) | 3.365325 / 5.269862 (-1.904537) | 2.000333 / 4.565676 (-2.565343) | 0.056661 / 0.424275 (-0.367614) | 0.007100 / 0.007607 (-0.000507) | 0.478587 / 0.226044 (0.252542) | 4.768703 / 2.268929 (2.499774) | 2.472432 / 55.444624 (-52.972192) | 2.133611 / 6.876477 (-4.742865) | 2.154296 / 2.142072 (0.012223) | 0.582293 / 4.805227 (-4.222934) | 0.131932 / 6.500664 (-6.368732) | 0.060259 / 0.075469 (-0.015211) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.259167 / 1.841788 (-0.582620) | 18.465604 / 8.074308 (10.391296) | 14.024528 / 10.191392 (3.833136) | 0.162320 / 0.680424 (-0.518104) | 0.018144 / 0.534201 (-0.516057) | 0.389931 / 0.579283 (-0.189352) | 0.396456 / 0.434364 (-0.037908) | 0.454734 / 0.540337 (-0.085603) | 0.636406 / 1.386936 (-0.750530) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006565 / 0.011353 (-0.004788) | 0.004008 / 0.011008 (-0.007000) | 0.064526 / 0.038508 (0.026018) | 0.071963 / 0.023109 (0.048854) | 0.415456 / 0.275898 (0.139557) | 0.441199 / 0.323480 (0.117719) | 0.005619 / 0.007986 (-0.002366) | 0.003261 / 0.004328 (-0.001067) | 0.064817 / 0.004250 (0.060567) | 0.055349 / 0.037052 (0.018296) | 0.425172 / 0.258489 (0.166683) | 0.452629 / 0.293841 (0.158788) | 0.031676 / 0.128546 (-0.096870) | 0.008432 / 0.075646 (-0.067214) | 0.071752 / 0.419271 (-0.347519) | 0.047176 / 0.043533 (0.003643) | 0.408641 / 0.255139 (0.153502) | 0.428579 / 0.283200 (0.145380) | 0.021548 / 0.141683 (-0.120135) | 1.495153 / 1.452155 (0.042999) | 1.557933 / 1.492716 (0.065217) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.212749 / 0.018006 (0.194743) | 0.441263 / 0.000490 (0.440773) | 0.005831 / 0.000200 (0.005631) | 0.000092 / 0.000054 (0.000037) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031844 / 0.037411 (-0.005567) | 0.091590 / 0.014526 (0.077064) | 0.102859 / 0.176557 (-0.073697) | 0.155859 / 0.737135 (-0.581276) | 0.104717 / 0.296338 (-0.191622) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.425924 / 0.215209 (0.210715) | 4.292829 / 2.077655 (2.215174) | 2.314350 / 1.504120 (0.810230) | 2.163087 / 1.541195 (0.621892) | 2.217310 / 1.468490 (0.748820) | 0.490889 / 4.584777 (-4.093887) | 3.498287 / 3.745712 (-0.247425) | 3.224980 / 5.269862 (-2.044881) | 1.987739 / 4.565676 (-2.577938) | 0.057486 / 0.424275 (-0.366790) | 0.007199 / 0.007607 (-0.000408) | 0.501194 / 0.226044 (0.275149) | 5.015202 / 2.268929 (2.746273) | 2.816307 / 55.444624 (-52.628318) | 2.474593 / 6.876477 (-4.401884) | 2.649510 / 2.142072 (0.507437) | 0.597167 / 4.805227 (-4.208060) | 0.131199 / 6.500664 (-6.369465) | 0.059532 / 0.075469 (-0.015938) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.384053 / 1.841788 (-0.457734) | 18.964201 / 8.074308 (10.889893) | 14.336209 / 10.191392 (4.144817) | 0.187522 / 0.680424 (-0.492902) | 0.020201 / 0.534201 (-0.514000) | 0.394778 / 0.579283 (-0.184505) | 0.408393 / 0.434364 (-0.025971) | 0.470965 / 0.540337 (-0.069373) | 0.667974 / 1.386936 (-0.718962) |

![](https://cml.dev/watermark.png#3b3333d790800ddaa3bf386ee71dc800258c921c "CML watermark")

Currently, the commit message keeps on adding:

- `Upload dataset (part 00000-of-00002)`
- `Upload dataset (part 00000-of-00002) (part 00001-of-00002)`

Introduced in https://github.com/huggingface/datasets/pull/6269

This PR fixes this issue to have

- `Upload dataset (part 00000-of-00002)`
- `Upload dataset (part 00001-of-00002)`


docs: resolving namespace conflict, refactored variable

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006209 / 0.011353 (-0.005144) | 0.003708 / 0.011008 (-0.007300) | 0.080435 / 0.038508 (0.041926) | 0.060105 / 0.023109 (0.036995) | 0.392962 / 0.275898 (0.117064) | 0.429381 / 0.323480 (0.105902) | 0.003596 / 0.007986 (-0.004390) | 0.003849 / 0.004328 (-0.000480) | 0.062377 / 0.004250 (0.058127) | 0.048718 / 0.037052 (0.011666) | 0.400906 / 0.258489 (0.142417) | 0.440335 / 0.293841 (0.146494) | 0.027807 / 0.128546 (-0.100739) | 0.008066 / 0.075646 (-0.067580) | 0.262542 / 0.419271 (-0.156730) | 0.045513 / 0.043533 (0.001980) | 0.399608 / 0.255139 (0.144469) | 0.418007 / 0.283200 (0.134807) | 0.023475 / 0.141683 (-0.118208) | 1.476563 / 1.452155 (0.024409) | 1.528898 / 1.492716 (0.036182) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.223798 / 0.018006 (0.205792) | 0.430526 / 0.000490 (0.430036) | 0.009232 / 0.000200 (0.009032) | 0.000082 / 0.000054 (0.000028) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024921 / 0.037411 (-0.012490) | 0.077692 / 0.014526 (0.063166) | 0.085382 / 0.176557 (-0.091174) | 0.146220 / 0.737135 (-0.590915) | 0.086396 / 0.296338 (-0.209943) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.439986 / 0.215209 (0.224777) | 4.384552 / 2.077655 (2.306897) | 2.373697 / 1.504120 (0.869577) | 2.176138 / 1.541195 (0.634943) | 2.225914 / 1.468490 (0.757424) | 0.505776 / 4.584777 (-4.079001) | 3.053744 / 3.745712 (-0.691968) | 3.080443 / 5.269862 (-2.189419) | 1.904392 / 4.565676 (-2.661285) | 0.058112 / 0.424275 (-0.366163) | 0.006631 / 0.007607 (-0.000976) | 0.503409 / 0.226044 (0.277365) | 5.053375 / 2.268929 (2.784447) | 2.789963 / 55.444624 (-52.654661) | 2.452659 / 6.876477 (-4.423818) | 2.512353 / 2.142072 (0.370280) | 0.590095 / 4.805227 (-4.215132) | 0.126267 / 6.500664 (-6.374397) | 0.061246 / 0.075469 (-0.014223) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.249884 / 1.841788 (-0.591903) | 17.684730 / 8.074308 (9.610422) | 13.967467 / 10.191392 (3.776075) | 0.144202 / 0.680424 (-0.536222) | 0.017004 / 0.534201 (-0.517197) | 0.333634 / 0.579283 (-0.245649) | 0.387251 / 0.434364 (-0.047113) | 0.390189 / 0.540337 (-0.150148) | 0.535662 / 1.386936 (-0.851274) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006379 / 0.011353 (-0.004974) | 0.003681 / 0.011008 (-0.007327) | 0.063005 / 0.038508 (0.024497) | 0.064221 / 0.023109 (0.041112) | 0.446074 / 0.275898 (0.170176) | 0.471997 / 0.323480 (0.148517) | 0.005074 / 0.007986 (-0.002911) | 0.002945 / 0.004328 (-0.001383) | 0.063305 / 0.004250 (0.059054) | 0.050608 / 0.037052 (0.013556) | 0.443260 / 0.258489 (0.184771) | 0.478497 / 0.293841 (0.184656) | 0.028980 / 0.128546 (-0.099566) | 0.008145 / 0.075646 (-0.067502) | 0.068412 / 0.419271 (-0.350859) | 0.041552 / 0.043533 (-0.001980) | 0.436649 / 0.255139 (0.181510) | 0.462397 / 0.283200 (0.179198) | 0.019929 / 0.141683 (-0.121753) | 1.530248 / 1.452155 (0.078093) | 1.611117 / 1.492716 (0.118401) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.232894 / 0.018006 (0.214888) | 0.421451 / 0.000490 (0.420961) | 0.003984 / 0.000200 (0.003784) | 0.000084 / 0.000054 (0.000030) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027776 / 0.037411 (-0.009635) | 0.081632 / 0.014526 (0.067106) | 0.094031 / 0.176557 (-0.082526) | 0.147930 / 0.737135 (-0.589206) | 0.094226 / 0.296338 (-0.202112) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.471722 / 0.215209 (0.256513) | 4.713241 / 2.077655 (2.635587) | 2.662660 / 1.504120 (1.158540) | 2.490778 / 1.541195 (0.949583) | 2.555786 / 1.468490 (1.087296) | 0.512209 / 4.584777 (-4.072568) | 3.210612 / 3.745712 (-0.535100) | 2.863346 / 5.269862 (-2.406516) | 1.884664 / 4.565676 (-2.681012) | 0.058514 / 0.424275 (-0.365761) | 0.006473 / 0.007607 (-0.001134) | 0.543279 / 0.226044 (0.317235) | 5.441485 / 2.268929 (3.172556) | 3.145398 / 55.444624 (-52.299226) | 2.749603 / 6.876477 (-4.126874) | 2.925738 / 2.142072 (0.783666) | 0.598725 / 4.805227 (-4.206502) | 0.125616 / 6.500664 (-6.375048) | 0.061314 / 0.075469 (-0.014155) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.384270 / 1.841788 (-0.457518) | 18.307618 / 8.074308 (10.233310) | 14.635768 / 10.191392 (4.444376) | 0.148787 / 0.680424 (-0.531637) | 0.018191 / 0.534201 (-0.516010) | 0.333166 / 0.579283 (-0.246117) | 0.405116 / 0.434364 (-0.029247) | 0.392798 / 0.540337 (-0.147540) | 0.582299 / 1.386936 (-0.804637) |

![](https://cml.dev/watermark.png#7004f0f2ec59832fe53af033efdca10d00377760 "CML watermark")

In docs of about_arrow.md, in the below example code
![image](https://github.com/huggingface/datasets/assets/74114936/fc70e152-e15f-422e-949a-1c4c4c9aa116)
The variable name 'time' was being used in a way that could potentially lead to a namespace conflict with Python's built-in 'time' module. It is not a good convention and can lead to unintended variable shadowing for any user re-using the example code.
To ensure code clarity, and prevent potential naming conflicts renamed the variable 'time' to 'elapsed_time' in the example  code.


cast_column to Sequence with length=4 occur exception raise in datasets/table.py:2146

Thanks for reporting! We've spotted the bugs with the `array.values` handling and are fixing them in https://github.com/huggingface/datasets/pull/6283 (should be part of the next release).

> Thanks for reporting! We've spotted the bugs with the `array.values` handling and are fixing them in #6283 (should be part of the next release).

i encounter another exception while cast_column to type `Sequence(feature={"points": Array2D(shape=(-1, 2), dtype="int64"), "label": ClassLabel(num_classes=num_classes, names=names)})`

while my data like this: '{"points": [[0.6,0.6], [0.7,0.7], [0.8,0.8]], "label": "A1"}'

here is the backtrace info:

```
    out = func(dataset, *args, **kwargs)
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 2110, in cast_column
    return self.cast(features)
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 2055, in cast
    dataset = dataset.map(
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 592, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 557, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3097, in map
    for rank, done, content in Dataset._map_single(**dataset_kwargs):
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3474, in _map_single
    batch = apply_function_on_filtered_inputs(
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3353, in apply_function_on_filtered_inputs
    processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/table.py", line 2329, in table_cast
    return cast_table_to_schema(table, schema)
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/table.py", line 2288, in cast_table_to_schema
    arrays = [cast_array_to_feature(table[name], feature) for name, feature in features.items()]
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/table.py", line 2288, in <listcomp>
    arrays = [cast_array_to_feature(table[name], feature) for name, feature in features.items()]
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/table.py", line 1831, in wrapper
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/table.py", line 1831, in <listcomp>
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/table.py", line 2073, in cast_array_to_feature
    arrays = [_c(array.field(name), subfeature) for name, subfeature in feature.items()]
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/table.py", line 2073, in <listcomp>
    arrays = [_c(array.field(name), subfeature) for name, subfeature in feature.items()]
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/table.py", line 1833, in wrapper
    return func(array, *args, **kwargs)
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/table.py", line 2095, in cast_array_to_feature
    casted_values = _c(array.values, feature.feature)
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/table.py", line 1833, in wrapper
    return func(array, *args, **kwargs)
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/table.py", line 2144, in cast_array_to_feature
    return array_cast(array, feature(), allow_number_to_str=allow_number_to_str)
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/table.py", line 1833, in wrapper
    return func(array, *args, **kwargs)
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/table.py", line 1967, in array_cast
    return pa_type.wrap_array(array)
  File "pyarrow/types.pxi", line 1369, in pyarrow.lib.BaseExtensionType.wrap_array
TypeError: Incompatible storage type for extension<arrow.py_extension_type<Array2DExtensionType>>: expected list<item: list<item: double>>, got list<item: double>
```
and i print(array) in datasets/table.py:1967 indeed get 2D list. is that same issue in #6283 ?

besides this, hugging face datasets seems don't  naturally support multi-labels which means `Sequence(ClassLabel)` illegal if data is ["label1", "label2"]. so i have to define a class derived from `ClassLabel`, like this:

```
class AisClassLabels(ClassLabel):
    def encode_example(self, example_data):
        if self.num_classes is None:
            raise ValueError(
                "Trying to use ClassLabel feature with undefined number of class. "
                "Please set ClassLabel.names or num_classes."
            )
        if not isinstance(example_data, list):
            example_data = [example_data]

        for i in range(len(example_data)):
            if isinstance(example_data[i], str):
                example_data[i] = self.str2int(example_data[i])
                if not -1 <= example_data[i] < self.num_classes:
                    raise ValueError(f"Class label {example_data:d} greater than configured num_classes {self.num_classes}")
        return example_data
```
and it works well in my case. but is there any recommend way to implement multi-labels?

`Incompatible storage type for extension<arrow.py_extension_type<Array2DExtensionType>>: expected list<item: list<item: double>>, got list<item: double>`
if i change `Array2D(shape=(-1, 2), dtype="int64")` to `Sequence(Value("int64"))` , every thing goes well. but my data is 2D int list

i test Sequence(ClassLabel) is ok if one column is label list. but it is not ok in nested column such as `Sequence(feature= {"points": Sequence(Value("int32")), "label": Sequence(ClassLabel(num_classes....)))`. in this case i need override ClassLabels. encode_example as i given above.

### Describe the bug

i load a dataset from local csv file which has 187383612 examples, then use `map` to generate new columns for test.
here is my code :

```
import os

from datasets import load_dataset
from datasets.features import Sequence, Value

def add_new_path(example):

    example["ais_bbox"] = [100,100,200,200]
    example["ais_image_path"] = os.path.join("images", example["image_path"]) if example["image_path"] else ""
    return example

ais_dataset = load_dataset("/data/ryan.gao/ais_dataset_cache/raw/1749/")
hf_ds = ais_dataset.map(add_new_path, batched=False, num_proc=32)
ds = hf_ds.cast_column("ais_bbox", Sequence(Value("int32"), length=4))
```

and the `cast_column` raise an exception
```
Casting the dataset:   3%|███▉
...
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 2110, in cast_column
    return self.cast(features)
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 2055, in cast
    dataset = dataset.map(
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 592, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 557, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3097, in map
    for rank, done, content in Dataset._map_single(**dataset_kwargs):
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3474, in _map_single
    batch = apply_function_on_filtered_inputs(
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3353, in apply_function_on_filtered_inputs
    processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/table.py", line 2329, in table_cast
    return cast_table_to_schema(table, schema)
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/table.py", line 2288, in cast_table_to_schema
    arrays = [cast_array_to_feature(table[name], feature) for name, feature in features.items()]
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/table.py", line 2288, in <listcomp>
    arrays = [cast_array_to_feature(table[name], feature) for name, feature in features.items()]
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/table.py", line 1831, in wrapper
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/table.py", line 1831, in <listcomp>
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/table.py", line 2145, in cast_array_to_feature
    raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{feature}")
TypeError: Couldn't cast array of type
list<item: int64>
to
Sequence(feature=Value(dtype='int32', id=None), length=4, id=None)
```
i check the source code and make debug info:
in datasets/table.py:2092
```
2091             if feature.length > -1:
2092                 if feature.length * len(array) == len(array.values):
2093                     return pa.FixedSizeListArray.from_arrays(_c(array.values, feature.feature), feature.length)
2094                 print(len(array))
2095                 print(len(array.values))
```
my feature.length is 4.  but feature.length * len(array) == len(array.values) is false.
print(len(array)) is 262
print(len(array.values)) is 4000

then I use "for item in array" to print each item then get 262 * [100,100,200,200]
and use "for item in array.values" to print each item and get 4000 int32 which are 1000 * [100,100,200,200]

i'm wondering the `chunk` in each `array.chunks`, the "chunk.values" may get all the chunks's value rather than single chunk? but i check the pyarrow's doc seems chunk.values is chunk's value not all.

### Steps to reproduce the bug

code provided above.

### Expected behavior

feature.length * len(array) == len(array.values) should be true. and there should not has Exception.

### Environment info

python3.9
x86_64
datasets: 2.14.4
pyarrow: 13.0.0 or 10.0.0


Add return_file_name in load_dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6310). All of your documentation changes will be reflected on that endpoint.

> Thanks for the change !
>
> Since `return` in python often refers to what is actually returned by the function (here `load_dataset`), I think we can use another word for the parameter. Maybe name it `with_file_names`?
>
> cc @mariosasko in case you have an opinion

I changed the argument name to your suggestion, I agree that it should be less confusing :)

> Thanks! I've left some comments.
>
> @lhoestq WDYT about returning a data file's name (the last part) instead of the full path? This way we could have the same values in the streaming and the non-streaming mode. (In the non-streaming mode, we would also have to iterate over remote files to not output the files' hash (from the HF cache))

Concerning the last part of the file name, do you have suggestions on how to do that? Because it can happen that the files are located in different folders with the same name so I am wondering what would be the way to go.

Proposition to fix #5806.

Added an optional parameter `return_file_name` in the dataset builder config. When set to `True`, the function will include the file name corresponding to the sample in the returned output.

There is a difference between arrow-based and folder-based datasets to return the file name:
- for arrow-based: a column is concatenated after the table is cast.
- for folder-based: `dataset.info.features` has the entry `file_name` and the original file name is passed to the `sample_metadata` dictionary.

The difference in behavior might be a concern, also I do not know whether the `file_name` should return the original file path or the downloaded one for folder-based datasets.

I added some tests for the datasets that already had a test file.