[FR] `map` should reuse unchanged columns from the previous dataset to avoid disk usage

enhancement

New feature or request

good second issue

Issues a bit more difficult than "Good First" issues

You can use the `remove_columns` parameter in `map` to avoid duplicating the columns (and save disk space) and then concatenate the original dataset with the map result:
```python
from datasets import concatenate_datasets
# dummy example
ds_new = ds.map(lambda x: {"new_col": x["col"] + 2}, remove_columns=ds.column_names)
ds_combined = concatenate_datasets([ds, ds_new], axis=1)
```

Doing this automatically is hard to implement efficiently unless we know ahead of time which existing columns will be modified by a `map` transform. We have this info when `input_columns` are specified, so I think this is the only case we can optimize.

### Feature request

Currently adding a new column with `map` will cause all the data in the dataset to be duplicated and stored/cached on the disk again. It should reuse unchanged columns.

### Motivation

This allows having datasets with different columns but sharing some basic columns. Currently, these datasets would become too expensive to store and one would need some kind of on-the-fly join; which also doesn't seem implemented.

### Your contribution

_


[FR] Transform Chaining, Lazy Mapping

enhancement

New feature or request

You can use `with_transform` to get a new dataset object.

Support for lazy `map` has already been discussed [here](https://github.com/huggingface/datasets/issues/3385) a little bit. Personally, I'm not a fan, as this would make `map` even more complex.

> You can use `with_transform` to get a new dataset object.
>
> Support for lazy `map` has already been discussed [here](https://github.com/huggingface/datasets/issues/3385) a little bit. Personally, I'm not a fan, as this would make `map` even more complex.

I read about IterableDataset, and it seems to have lazy mapping. But I can't figure out how to convert an IterableDataset into a normal one when needed.

`with_transform` still does not chain AFAIU.

> I read about IterableDataset, and it seems to have lazy mapping. But I can't figure out how to convert an IterableDataset into a normal one when needed.

You must cache an `IterableDataset` to disk to load it as a `Dataset`. One way to do this is with `Dataset.from_generator`:
```python
from functools import partial
from datasets import Dataset

def gen_from_iterable_dataset(iterable_ds)
    yield from iterable_ds

ds = Dataset.from_generator(partial(gen_from_iterable_dataset, iterable_ds), features=iterable_ds.features})
```

> with_transform still does not chain AFAIU.

Yes, not supported yet - the solution is to combine the transforms into a single one.

I wonder if it would be beneficial to have a dedicated method to do that ? Maybe a `.save_to_disk()` so that the user can reload the resulting dataset later ?

> ```python
> from functools import partial
> from datasets import Dataset
>
> def gen_from_iterable_dataset(iterable_ds)
>     yield from iterable_ds
>
> ds = Dataset.from_generator(partial(gen_from_iterable_dataset, iterable_ds), features=iterable_ds.features})
> ```

@mariosasko With these complex mapping functions, what hash will be used to cache this dataset?

The params passed to `Dataset.from_generator` will be used to compute the hash (`partial` encapsulates the `iterable_ds` value, so changing it will also change the hash)

Hi, I think this feature would be very useful. I want to concatenate large datasets with heterogeneous columns. I dislike `map` since I don't want multiple copy of that datasets locally. I tried to use "set_transform" on each dataset to convert it to a standard features format, but `datasets.concatenate_datasets` ignores the updated format of the datasets.  A work around is to use `torch.utils.data.ConcatDataset`. Is there a neat way to do it using HF datasets?

### Feature request

Currently using a `map` call processes and duplicates the whole dataset, which takes both time and disk space.

The solution is to allow lazy mapping, which is essentially a saved chain of transforms that are applied on the fly whenever a slice of the dataset is requested.

The API should look like `map`, as `set_transform` changes the current dataset while `map` returns another dataset.

### Motivation

Lazy processing allows lower disk usage and faster experimentation.

### Your contribution

_


Documentation: wiki_dpr Dataset has no metric_type for Faiss Index

Hi! You can do `ds.get_index("embeddings").faiss_index.metric_type` to get the metric type and then match the result with the FAISS metric [enum](https://github.com/facebookresearch/faiss/blob/43d86e30736ede853c384b24667fc3ab897d6ba9/faiss/MetricType.h#L22-L36) (should be L2).

Ah! Thank you for pointing this out. FYI: the enum indicates it's using the inner product. Using `torch.inner` or `torch.dot` still produces a discrepancy compared to the built-in score. I think this is because of the compression/quantization that occurs with the FAISS index.

### Describe the bug

After loading `wiki_dpr` using:
```py
ds = load_dataset(path='wiki_dpr', name='psgs_w100.multiset.compressed', split='train')
print(ds.get_index("embeddings").metric_type) # prints nothing because the value is None
```
the index does not have a defined `metric_type`. This is an issue because I do not know how the `scores` are being computed for `get_nearest_examples()`.

### Steps to reproduce the bug

System: Python 3.9.16, Transformers 4.30.2, WSL

After loading `wiki_dpr` using:
```py
ds = load_dataset(path='wiki_dpr', name='psgs_w100.multiset.compressed', split='train')
print(ds.get_index("embeddings").metric_type) # prints nothing because the value is None
```
the index does not have a defined `metric_type`. This is an issue because I do not know how the `scores` are being computed for `get_nearest_examples()`.

```py
from transformers import DPRQuestionEncoder, DPRContextEncoder, DPRQuestionEncoderTokenizer, DPRContextEncoderTokenizer

tokenizer = DPRQuestionEncoderTokenizer.from_pretrained("facebook/dpr-question_encoder-multiset-base")
encoder = DPRQuestionEncoder.from_pretrained("facebook/dpr-question_encoder-multiset-base")

def encode_question(query, tokenizer=tokenizer, encoder=encoder):
    inputs = tokenizer(query, return_tensors='pt')
    question_embedding = encoder(**inputs)[0].detach().numpy()
    return question_embedding

def get_knn(query, k=5, tokenizer=tokenizer, encoder=encoder, verbose=False):
    enc_question = encode_question(query, tokenizer, encoder)
    topk_results = ds.get_nearest_examples(index_name='embeddings',
                                           query=enc_question,
                                          k=k)

    a = torch.tensor(enc_question[0]).reshape(768)
    b = torch.tensor(topk_results.examples['embeddings'][0])
    print(a.shape, b.shape)
    print(torch.dot(a, b))
    print((a-b).pow(2).sum())

    return topk_results
```

The [FAISS documentation](https://github.com/facebookresearch/faiss/wiki/MetricType-and-distances) suggests the metric is usually L2 distance (without the square root) or the inner product. I compute both for the sample query:
```py
query = """ it catapulted into popular culture along with a line of action figures and other toys by Bandai.[2] By 2001, the media franchise had generated over $6 billion in toy sales.
Despite initial criticism that its action violence targeted child audiences, the franchise has been commercially successful."""
get_knn(query,k=5)
```

Here, I get dot product of 80.6020 and L2 distance of 77.6616 and
```py
NearestExamplesResults(scores=array([76.20431 , 75.312416, 74.945404, 74.866394, 74.68506 ],
      dtype=float32), examples={'id': ['3081096', '2004811', '8908258', '9594124', '286575'], 'text': ['actors, resulting in the "Power Rangers" franchise which has continued since then into sequel TV series (with "Power Rangers Beast Morphers" set to premiere in 2019), comic books, video games, and three feature films, with a further cinematic universe planned. Following from the success of "Power Rangers", Saban acquired the rights to more of Toei\'s library, creating "VR Troopers" and "Big Bad Beetleborgs" from several Metal Hero Series shows and "Masked Rider" from Kamen Rider Series footage. DIC Entertainment joined this boom by acquiring the rights to "Gridman the Hyper Agent" and turning it into "Superhuman Samurai Syber-Squad". In 2002,',
```

Doing `k=1` indicates the higher the outputted number, the better the match, so the metric should not be L2 distance. However, my manually computed inner product (80.6) has a discrepancy with the reported (76.2). Perhaps, this has to do with me using the `compressed` embeddings?

### Expected behavior

```py
ds = load_dataset(path='wiki_dpr', name='psgs_w100.multiset.compressed', split='train')
print(ds.get_index("embeddings").metric_type) # METRIC_INNER_PRODUCT
```

### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-4.18.0-477.13.1.el8_8.x86_64-x86_64-with-glibc2.28
- Python version: 3.9.16
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1


Improve `Dataset`'s string representation

New feature or request

I want to take a shot at this if possible

Yes, feel free to work on this.

You can check the PyArrow Table `__repr__`  and Polars DataFrame `__repr__`/`_repr_html_` implementations for some pointers/ideas.

@mariosasko are there any other similar issues that I could work on? I see this has been already solved.

Currently, `Dataset.__repr__` outputs a dataset's column names and the number of rows. We could improve it by printing its features and the first few rows.

We should also implement `_repr_html_` to have a rich HTML representation in notebooks/Streamlit.


Fix cast for dictionaries with no keys

_The documentation is not available anymore as the PR was closed or merged._

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006961 / 0.011353 (-0.004392) | 0.004390 / 0.011008 (-0.006618) | 0.103249 / 0.038508 (0.064741) | 0.048084 / 0.023109 (0.024975) | 0.351213 / 0.275898 (0.075315) | 0.416918 / 0.323480 (0.093439) | 0.005539 / 0.007986 (-0.002446) | 0.003555 / 0.004328 (-0.000774) | 0.079306 / 0.004250 (0.075055) | 0.066937 / 0.037052 (0.029884) | 0.382601 / 0.258489 (0.124112) | 0.406125 / 0.293841 (0.112284) | 0.032269 / 0.128546 (-0.096277) | 0.009133 / 0.075646 (-0.066514) | 0.354449 / 0.419271 (-0.064822) | 0.068978 / 0.043533 (0.025445) | 0.352314 / 0.255139 (0.097175) | 0.390398 / 0.283200 (0.107199) | 0.025640 / 0.141683 (-0.116043) | 1.553865 / 1.452155 (0.101710) | 1.601292 / 1.492716 (0.108576) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.208310 / 0.018006 (0.190303) | 0.440076 / 0.000490 (0.439586) | 0.000363 / 0.000200 (0.000163) | 0.000059 / 0.000054 (0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029173 / 0.037411 (-0.008238) | 0.111323 / 0.014526 (0.096797) | 0.123001 / 0.176557 (-0.053556) | 0.180180 / 0.737135 (-0.556955) | 0.125804 / 0.296338 (-0.170534) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.419919 / 0.215209 (0.204710) | 4.194515 / 2.077655 (2.116860) | 1.881234 / 1.504120 (0.377114) | 1.672914 / 1.541195 (0.131720) | 1.723102 / 1.468490 (0.254612) | 0.543584 / 4.584777 (-4.041193) | 3.822477 / 3.745712 (0.076765) | 1.837946 / 5.269862 (-3.431915) | 1.094975 / 4.565676 (-3.470701) | 0.066788 / 0.424275 (-0.357487) | 0.011689 / 0.007607 (0.004082) | 0.520983 / 0.226044 (0.294938) | 5.209245 / 2.268929 (2.940316) | 2.392916 / 55.444624 (-53.051708) | 2.060042 / 6.876477 (-4.816434) | 2.162291 / 2.142072 (0.020219) | 0.668472 / 4.805227 (-4.136755) | 0.144373 / 6.500664 (-6.356291) | 0.066152 / 0.075469 (-0.009318) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.251256 / 1.841788 (-0.590532) | 15.161338 / 8.074308 (7.087030) | 14.416133 / 10.191392 (4.224741) | 0.166145 / 0.680424 (-0.514279) | 0.018168 / 0.534201 (-0.516033) | 0.433364 / 0.579283 (-0.145919) | 0.417484 / 0.434364 (-0.016880) | 0.502543 / 0.540337 (-0.037794) | 0.602904 / 1.386936 (-0.784032) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006946 / 0.011353 (-0.004407) | 0.004248 / 0.011008 (-0.006761) | 0.079707 / 0.038508 (0.041199) | 0.046226 / 0.023109 (0.023117) | 0.375864 / 0.275898 (0.099966) | 0.430740 / 0.323480 (0.107260) | 0.006222 / 0.007986 (-0.001764) | 0.003474 / 0.004328 (-0.000854) | 0.079622 / 0.004250 (0.075372) | 0.066666 / 0.037052 (0.029613) | 0.379487 / 0.258489 (0.120998) | 0.423002 / 0.293841 (0.129161) | 0.032836 / 0.128546 (-0.095710) | 0.008976 / 0.075646 (-0.066670) | 0.086578 / 0.419271 (-0.332693) | 0.055651 / 0.043533 (0.012118) | 0.360787 / 0.255139 (0.105648) | 0.384265 / 0.283200 (0.101065) | 0.025350 / 0.141683 (-0.116333) | 1.547880 / 1.452155 (0.095725) | 1.605850 / 1.492716 (0.113134) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.184227 / 0.018006 (0.166220) | 0.442071 / 0.000490 (0.441582) | 0.002887 / 0.000200 (0.002687) | 0.000088 / 0.000054 (0.000034) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031923 / 0.037411 (-0.005488) | 0.119093 / 0.014526 (0.104568) | 0.128704 / 0.176557 (-0.047853) | 0.187065 / 0.737135 (-0.550070) | 0.134135 / 0.296338 (-0.162204) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.455731 / 0.215209 (0.240522) | 4.562911 / 2.077655 (2.485256) | 2.247431 / 1.504120 (0.743311) | 2.053346 / 1.541195 (0.512151) | 2.049611 / 1.468490 (0.581121) | 0.546069 / 4.584777 (-4.038708) | 3.821852 / 3.745712 (0.076140) | 3.358497 / 5.269862 (-1.911364) | 1.667697 / 4.565676 (-2.897979) | 0.067968 / 0.424275 (-0.356307) | 0.012344 / 0.007607 (0.004737) | 0.550864 / 0.226044 (0.324820) | 5.496867 / 2.268929 (3.227939) | 2.680031 / 55.444624 (-52.764594) | 2.328673 / 6.876477 (-4.547804) | 2.436754 / 2.142072 (0.294682) | 0.681195 / 4.805227 (-4.124033) | 0.148761 / 6.500664 (-6.351904) | 0.067716 / 0.075469 (-0.007753) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.353798 / 1.841788 (-0.487990) | 15.992965 / 8.074308 (7.918657) | 14.051539 / 10.191392 (3.860147) | 0.181087 / 0.680424 (-0.499337) | 0.018653 / 0.534201 (-0.515548) | 0.433499 / 0.579283 (-0.145784) | 0.428845 / 0.434364 (-0.005519) | 0.501100 / 0.540337 (-0.039238) | 0.603666 / 1.386936 (-0.783270) |

![](https://cml.dev/watermark.png#10cfa871a2f387fe9c6360e1873ea74c6d69ff67 "CML watermark")

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010983 / 0.011353 (-0.000370) | 0.005630 / 0.011008 (-0.005378) | 0.109967 / 0.038508 (0.071458) | 0.101580 / 0.023109 (0.078471) | 0.490205 / 0.275898 (0.214307) | 0.534653 / 0.323480 (0.211173) | 0.008365 / 0.007986 (0.000379) | 0.004317 / 0.004328 (-0.000012) | 0.082429 / 0.004250 (0.078179) | 0.080556 / 0.037052 (0.043504) | 0.494627 / 0.258489 (0.236138) | 0.544189 / 0.293841 (0.250348) | 0.049419 / 0.128546 (-0.079127) | 0.014033 / 0.075646 (-0.061613) | 0.370406 / 0.419271 (-0.048866) | 0.083468 / 0.043533 (0.039935) | 0.463829 / 0.255139 (0.208690) | 0.507516 / 0.283200 (0.224316) | 0.053266 / 0.141683 (-0.088417) | 1.778680 / 1.452155 (0.326525) | 1.916616 / 1.492716 (0.423900) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.267646 / 0.018006 (0.249640) | 0.617824 / 0.000490 (0.617334) | 0.007720 / 0.000200 (0.007520) | 0.000139 / 0.000054 (0.000085) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034464 / 0.037411 (-0.002948) | 0.113626 / 0.014526 (0.099100) | 0.118911 / 0.176557 (-0.057646) | 0.194701 / 0.737135 (-0.542434) | 0.123431 / 0.296338 (-0.172907) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.606073 / 0.215209 (0.390863) | 6.086393 / 2.077655 (4.008738) | 2.568712 / 1.504120 (1.064593) | 2.260801 / 1.541195 (0.719606) | 2.411798 / 1.468490 (0.943307) | 0.876433 / 4.584777 (-3.708344) | 5.521280 / 3.745712 (1.775568) | 5.969722 / 5.269862 (0.699861) | 3.671028 / 4.565676 (-0.894649) | 0.097082 / 0.424275 (-0.327193) | 0.011354 / 0.007607 (0.003747) | 0.713842 / 0.226044 (0.487798) | 7.291172 / 2.268929 (5.022244) | 3.315272 / 55.444624 (-52.129352) | 2.777487 / 6.876477 (-4.098990) | 3.025449 / 2.142072 (0.883377) | 1.014115 / 4.805227 (-3.791112) | 0.217928 / 6.500664 (-6.282736) | 0.083097 / 0.075469 (0.007627) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.640060 / 1.841788 (-0.201728) | 25.342172 / 8.074308 (17.267864) | 22.776510 / 10.191392 (12.585118) | 0.227300 / 0.680424 (-0.453124) | 0.032233 / 0.534201 (-0.501968) | 0.507547 / 0.579283 (-0.071736) | 0.647044 / 0.434364 (0.212680) | 0.607019 / 0.540337 (0.066682) | 0.823548 / 1.386936 (-0.563388) |

PyArrow==latest

Show updated benchmarks!

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009576 / 0.011353 (-0.001777) | 0.009322 / 0.011008 (-0.001687) | 0.087184 / 0.038508 (0.048676) | 0.100795 / 0.023109 (0.077685) | 0.492138 / 0.275898 (0.216240) | 0.528386 / 0.323480 (0.204906) | 0.006689 / 0.007986 (-0.001296) | 0.004735 / 0.004328 (0.000406) | 0.085519 / 0.004250 (0.081269) | 0.072648 / 0.037052 (0.035595) | 0.496068 / 0.258489 (0.237579) | 0.549634 / 0.293841 (0.255793) | 0.049709 / 0.128546 (-0.078837) | 0.015077 / 0.075646 (-0.060569) | 0.099445 / 0.419271 (-0.319826) | 0.068080 / 0.043533 (0.024547) | 0.500426 / 0.255139 (0.245287) | 0.531437 / 0.283200 (0.248238) | 0.053176 / 0.141683 (-0.088507) | 1.827942 / 1.452155 (0.375787) | 1.914286 / 1.492716 (0.421570) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.247658 / 0.018006 (0.229652) | 0.590805 / 0.000490 (0.590315) | 0.005319 / 0.000200 (0.005119) | 0.000165 / 0.000054 (0.000110) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036993 / 0.037411 (-0.000418) | 0.112944 / 0.014526 (0.098419) | 0.118964 / 0.176557 (-0.057593) | 0.194867 / 0.737135 (-0.542269) | 0.120816 / 0.296338 (-0.175523) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.638062 / 0.215209 (0.422853) | 6.246785 / 2.077655 (4.169130) | 2.957779 / 1.504120 (1.453659) | 2.739118 / 1.541195 (1.197924) | 2.795362 / 1.468490 (1.326872) | 0.890532 / 4.584777 (-3.694245) | 5.508198 / 3.745712 (1.762486) | 5.222315 / 5.269862 (-0.047547) | 3.152731 / 4.565676 (-1.412946) | 0.098344 / 0.424275 (-0.325931) | 0.008800 / 0.007607 (0.001193) | 0.757889 / 0.226044 (0.531845) | 7.545715 / 2.268929 (5.276787) | 3.694536 / 55.444624 (-51.750088) | 3.112872 / 6.876477 (-3.763605) | 3.182358 / 2.142072 (1.040285) | 1.028171 / 4.805227 (-3.777056) | 0.215223 / 6.500664 (-6.285441) | 0.085856 / 0.075469 (0.010387) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.853138 / 1.841788 (0.011350) | 25.939672 / 8.074308 (17.865364) | 23.118029 / 10.191392 (12.926637) | 0.250599 / 0.680424 (-0.429825) | 0.029942 / 0.534201 (-0.504259) | 0.508748 / 0.579283 (-0.070535) | 0.593966 / 0.434364 (0.159602) | 0.605499 / 0.540337 (0.065162) | 0.863827 / 1.386936 (-0.523109) |

![](https://cml.dev/watermark.png#5d15950d99677e9473cdcd31cfd83aa17e313e28 "CML watermark")

Fix #5677


Dataset.from_generator consistently freezes at ~1000 rows

By default, we write data to disk (so it can be memory-mapped) every 1000 rows/samples. You can control this with the `writer_batch_size` parameter. Also, when working with fixed-size arrays, the `ArrayXD` feature types yield better performance (e.g., in your case, `features=datasets.Features({"i": datasets.Array3D(shape=(512,512,3), dtype="float32")})` should be faster).

Our support for multi-dim arrays could be better, and we plan to improve it as part of https://github.com/huggingface/datasets/issues/5272.

> By default, we write data to disk (so it can be memory-mapped) every 1000 rows/samples. You can control this with the `writer_batch_size` parameter. Also, when working with fixed-size arrays, the `ArrayXD` feature types yield better performance (e.g., in your case, `features=datasets.Features({"i": datasets.Array3D(shape=(512,512,3), dtype="float32")})` should be faster).
>
> Our support for multi-dim arrays could be better, and we plan to improve it as part of #5272.

Thanks for the explanation! The Image array was just for demonstration, I use PIL Images in practice. Does that make a difference? What's the best approach for a dataset with PIL Images as rows?

It's best to use the `datasets.Image()` feature type for PIL images (to save space) :)

### Describe the bug

Whenever I try to create a dataset which contains images using `Dataset.from_generator`, it freezes around 996 rows. I suppose it has something to do with memory consumption, but there's more memory available. I

Somehow it worked a few times but mostly this makes the datasets library much more cumbersome to work with because generators are the easiest way to turn an existing dataset into a Hugging Face dataset.

I've let it run in the frozen state for way longer than it can possibly take to load the actual dataset.

Let me know if you have ideas how to resolve it!

### Steps to reproduce the bug

```python
from datasets import Dataset
import numpy as np

def gen():
    for row in range(10000):
        yield {"i": np.random.rand(512, 512, 3)}

Dataset.from_generator(gen)
# -> 90% of the time gets stuck around 1000 rows
```

### Expected behavior

Should continue and go through all the examples yielded by the generator, or at least throw an error or somehow communicate what's going on.

### Environment info

- `datasets` version: 2.8.0
- Platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 12.0.1
- Pandas version: 1.5.1


Get an error "OverflowError: Python int too large to convert to C long" when loading a large dataset

arrow

Related to Apache Arrow

This error means that one of the int32 (`Value("int32")`) columns in the dataset has a value that is out of the valid (int32) range.

I'll open a PR to print the name of a problematic column to make debugging such errors easier.

I am afraid int32 is not the reason for this error.

I have submitted a commit to use int64 for all ints in the dataset:
https://huggingface.co/datasets/liwu/MNBVC/commit/857ac00d9eab96a6708ad6a82bd9001686042a9e

and I have updated my env to the latest datasets release:
Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 2.13.1
- Platform: macOS-13.2.1-arm64-arm-64bit
- Python version: 3.11.2
- Huggingface_hub version: 0.13.4
- PyArrow version: 11.0.0
- Pandas version: 1.5.3

But the error still exist

```
Downloading and preparing dataset mnbvc/news_peoples_daily to /Users/silver/.cache/huggingface/datasets/liwu___mnbvc/news_peoples_daily/0.0.1/ee380f6309fe9b8b0d1fb14d77118f132444f22c8c4b28bf5c1645312688e051...
Downloading data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 9070.40it/s]
Extracting data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 2697.16it/s]
---------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
File ~/git/venv/lib/python3.11/site-packages/datasets/builder.py:1647, in GeneratorBasedBuilder._prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, split_info, check_duplicate_keys, job_id)
   1646 example = self.info.features.encode_example(record) if self.info.features is not None else record
-> 1647 writer.write(example, key)
   1648 num_examples_progress_update += 1

File ~/git/venv/lib/python3.11/site-packages/datasets/arrow_writer.py:490, in ArrowWriter.write(self, example, key, writer_batch_size)
    488     self.hkey_record = []
--> 490 self.write_examples_on_file()

File ~/git/venv/lib/python3.11/site-packages/datasets/arrow_writer.py:448, in ArrowWriter.write_examples_on_file(self)
    444         batch_examples[col] = [
    445             row[0][col].to_pylist()[0] if isinstance(row[0][col], (pa.Array, pa.ChunkedArray)) else row[0][col]
    446             for row in self.current_examples
    447         ]
--> 448 self.write_batch(batch_examples=batch_examples)
    449 self.current_examples = []

File ~/git/venv/lib/python3.11/site-packages/datasets/arrow_writer.py:553, in ArrowWriter.write_batch(self, batch_examples, writer_batch_size)
    552 typed_sequence = OptimizedTypedSequence(col_values, type=col_type, try_type=col_try_type, col=col)
--> 553 arrays.append(pa.array(typed_sequence))
    554 inferred_features[col] = typed_sequence.get_inferred_type()

File ~/git/venv/lib/python3.11/site-packages/pyarrow/array.pxi:236, in pyarrow.lib.array()

File ~/git/venv/lib/python3.11/site-packages/pyarrow/array.pxi:110, in pyarrow.lib._handle_arrow_array_protocol()

File ~/git/venv/lib/python3.11/site-packages/datasets/arrow_writer.py:189, in TypedSequence.__arrow_array__(self, type)
    188     trying_cast_to_python_objects = True
--> 189     out = pa.array(cast_to_python_objects(data, only_1d_for_numpy=True))
    190 # use smaller integer precisions if possible

File ~/git/venv/lib/python3.11/site-packages/pyarrow/array.pxi:320, in pyarrow.lib.array()

File ~/git/venv/lib/python3.11/site-packages/pyarrow/array.pxi:39, in pyarrow.lib._sequence_to_array()

File ~/git/venv/lib/python3.11/site-packages/pyarrow/error.pxi:144, in pyarrow.lib.pyarrow_internal_check_status()

OverflowError: Python int too large to convert to C long

During handling of the above exception, another exception occurred:

OverflowError                             Traceback (most recent call last)
File ~/git/venv/lib/python3.11/site-packages/datasets/builder.py:1656, in GeneratorBasedBuilder._prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, split_info, check_duplicate_keys, job_id)
   1655 num_shards = shard_id + 1
-> 1656 num_examples, num_bytes = writer.finalize()
   1657 writer.close()

File ~/git/venv/lib/python3.11/site-packages/datasets/arrow_writer.py:584, in ArrowWriter.finalize(self, close_stream)
    583     self.hkey_record = []
--> 584 self.write_examples_on_file()
    585 # If schema is known, infer features even if no examples were written

File ~/git/venv/lib/python3.11/site-packages/datasets/arrow_writer.py:448, in ArrowWriter.write_examples_on_file(self)
    444         batch_examples[col] = [
    445             row[0][col].to_pylist()[0] if isinstance(row[0][col], (pa.Array, pa.ChunkedArray)) else row[0][col]
    446             for row in self.current_examples
    447         ]
--> 448 self.write_batch(batch_examples=batch_examples)
    449 self.current_examples = []

File ~/git/venv/lib/python3.11/site-packages/datasets/arrow_writer.py:553, in ArrowWriter.write_batch(self, batch_examples, writer_batch_size)
    552 typed_sequence = OptimizedTypedSequence(col_values, type=col_type, try_type=col_try_type, col=col)
--> 553 arrays.append(pa.array(typed_sequence))
    554 inferred_features[col] = typed_sequence.get_inferred_type()

File ~/git/venv/lib/python3.11/site-packages/pyarrow/array.pxi:236, in pyarrow.lib.array()

File ~/git/venv/lib/python3.11/site-packages/pyarrow/array.pxi:110, in pyarrow.lib._handle_arrow_array_protocol()

File ~/git/venv/lib/python3.11/site-packages/datasets/arrow_writer.py:189, in TypedSequence.__arrow_array__(self, type)
    188     trying_cast_to_python_objects = True
--> 189     out = pa.array(cast_to_python_objects(data, only_1d_for_numpy=True))
    190 # use smaller integer precisions if possible

File ~/git/venv/lib/python3.11/site-packages/pyarrow/array.pxi:320, in pyarrow.lib.array()

File ~/git/venv/lib/python3.11/site-packages/pyarrow/array.pxi:39, in pyarrow.lib._sequence_to_array()

File ~/git/venv/lib/python3.11/site-packages/pyarrow/error.pxi:144, in pyarrow.lib.pyarrow_internal_check_status()

OverflowError: Python int too large to convert to C long

The above exception was the direct cause of the following exception:

DatasetGenerationError                    Traceback (most recent call last)
Cell In[2], line 1
----> 1 dataset = load_dataset("liwu/MNBVC", 'news_peoples_daily', split='train')

File ~/git/venv/lib/python3.11/site-packages/datasets/load.py:1809, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, num_proc, storage_options, **config_kwargs)
   1806 try_from_hf_gcs = path not in _PACKAGED_DATASETS_MODULES
   1808 # Download and prepare data
-> 1809 builder_instance.download_and_prepare(
   1810     download_config=download_config,
   1811     download_mode=download_mode,
   1812     verification_mode=verification_mode,
   1813     try_from_hf_gcs=try_from_hf_gcs,
   1814     num_proc=num_proc,
   1815     storage_options=storage_options,
   1816 )
   1818 # Build dataset for splits
   1819 keep_in_memory = (
   1820     keep_in_memory if keep_in_memory is not None else is_small_dataset(builder_instance.info.dataset_size)
   1821 )

File ~/git/venv/lib/python3.11/site-packages/datasets/builder.py:909, in DatasetBuilder.download_and_prepare(self, output_dir, download_config, download_mode, verification_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, file_format, max_shard_size, num_proc, storage_options, **download_and_prepare_kwargs)
    907     if num_proc is not None:
    908         prepare_split_kwargs["num_proc"] = num_proc
--> 909     self._download_and_prepare(
    910         dl_manager=dl_manager,
    911         verification_mode=verification_mode,
    912         **prepare_split_kwargs,
    913         **download_and_prepare_kwargs,
    914     )
    915 # Sync info
    916 self.info.dataset_size = sum(split.num_bytes for split in self.info.splits.values())

File ~/git/venv/lib/python3.11/site-packages/datasets/builder.py:1670, in GeneratorBasedBuilder._download_and_prepare(self, dl_manager, verification_mode, **prepare_splits_kwargs)
   1669 def _download_and_prepare(self, dl_manager, verification_mode, **prepare_splits_kwargs):
-> 1670     super()._download_and_prepare(
   1671         dl_manager,
   1672         verification_mode,
   1673         check_duplicate_keys=verification_mode == VerificationMode.BASIC_CHECKS
   1674         or verification_mode == VerificationMode.ALL_CHECKS,
   1675         **prepare_splits_kwargs,
   1676     )

File ~/git/venv/lib/python3.11/site-packages/datasets/builder.py:1004, in DatasetBuilder._download_and_prepare(self, dl_manager, verification_mode, **prepare_split_kwargs)
   1000 split_dict.add(split_generator.split_info)
   1002 try:
   1003     # Prepare split will record examples associated to the split
-> 1004     self._prepare_split(split_generator, **prepare_split_kwargs)
   1005 except OSError as e:
   1006     raise OSError(
   1007         "Cannot find data file. "
   1008         + (self.manual_download_instructions or "")
   1009         + "\nOriginal error:\n"
   1010         + str(e)
   1011     ) from None

File ~/git/venv/lib/python3.11/site-packages/datasets/builder.py:1508, in GeneratorBasedBuilder._prepare_split(self, split_generator, check_duplicate_keys, file_format, num_proc, max_shard_size)
   1506 job_id = 0
   1507 with pbar:
-> 1508     for job_id, done, content in self._prepare_split_single(
   1509         gen_kwargs=gen_kwargs, job_id=job_id, **_prepare_split_args
   1510     ):
   1511         if done:
   1512             result = content

File ~/git/venv/lib/python3.11/site-packages/datasets/builder.py:1665, in GeneratorBasedBuilder._prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, split_info, check_duplicate_keys, job_id)
   1663     if isinstance(e, SchemaInferenceError) and e.__context__ is not None:
   1664         e = e.__context__
-> 1665     raise DatasetGenerationError("An error occurred while generating the dataset") from e
   1667 yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_shards, shard_lengths)

DatasetGenerationError: An error occurred while generating the dataset
```

Besides, it works fine when I am using streamed dataset.

`simhash` is the problematic column - it has values such as `18329103420363166823` that are out of the int64 range. You can fix this by setting the feature type to `Value("string")` (it's advised to use this type for hash values in general)

> Besides, it works fine when I am using streamed dataset.

Streaming yields Python dictionaries from the script without converting them to the Arrow representation, as this conversion step is not that cheap performance-wise.

i am using uint64 for simhash

uint64 ranges up to about 3.69E19.

18329103420363166823 is less than this value.

moreover, our simhash algorithm use 64 bits. it should fit in uint64.

You are right. I overlooked the feature type.

This is a reproducer:
```python
import pyarrow as pa
from datasets.arrow_writer import TypedSequence

pa.array(TypedSequence([18329103420363166823], type=Value("uint64")))
```

`pa.array([18329103420363166823])` also fails with the same error, so it seems PyArrow does not always infer the correct type as NumPy does (`uint64` in this case).

I'll report this issue in the Arrow repo.

`pa.array([18329103420363166823], pa.uint64)` works, so maybe we can implement a temporary fix (supporting complex input such as `[{"image": pil_image, "num": uint64_value}]` would be hard though).

In the meantime, you should be able to bypass this error by returning the `simhash` values as NumPy scalars in the script:
```python
def _generate_examples(self, ...):
    ...
    yield {...,  "simhash": np.uint64(simhash), ...}
```

Thank you for checking this issue in detail.

However, it seems that using `np.uint64(simhash)` does not work. The same issue still exists.

https://huggingface.co/datasets/liwu/MNBVC/commit/1e44f1e400b7e61052647d44c99cdae3bae9c830

Anyway, we decide to use string type for these simhash values. Hope pyarrow can fix their bug soon.

May be something read your training data line by line.
Then your  training data just only one line.
It is so large.
I guess.

### Describe the bug

When load a large dataset with the following code

```python
from datasets import load_dataset
dataset = load_dataset("liwu/MNBVC", 'news_peoples_daily', split='train')
```

We encountered the error: "OverflowError: Python int too large to convert to C long"
The error look something like:

```
OverflowError: Python int too large to convert to C long

During handling of the above exception, another exception occurred:

OverflowError                             Traceback (most recent call last)
<ipython-input-7-0ed8700e662d> in <module>
----> 1 dataset = load_dataset("liwu/MNBVC", 'news_peoples_daily', split='train', cache_dir='/sfs/MNBVC/.cache/')

/sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1749         ignore_verifications=ignore_verifications,
   1750         try_from_hf_gcs=try_from_hf_gcs,
-> 1751         use_auth_token=use_auth_token,
   1752     )
   1753

/sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    703                     if not downloaded_from_gcs:
    704                         self._download_and_prepare(
--> 705                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    706                         )
    707                     # Sync info

/sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos)
   1225
   1226     def _download_and_prepare(self, dl_manager, verify_infos):
-> 1227         super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
   1228
   1229     def _get_examples_iterable_for_split(self, split_generator: SplitGenerator) -> ExamplesIterable:

/sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    791             try:
    792                 # Prepare split will record examples associated to the split
--> 793                 self._prepare_split(split_generator, **prepare_split_kwargs)
    794             except OSError as e:
    795                 raise OSError(

/sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/builder.py in _prepare_split(self, split_generator, check_duplicate_keys)
   1219                     writer.write(example, key)
   1220             finally:
-> 1221                 num_examples, num_bytes = writer.finalize()
   1222
   1223         split_generator.split_info.num_examples = num_examples

/sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/arrow_writer.py in finalize(self, close_stream)
    536             # Re-intializing to empty list for next batch
    537             self.hkey_record = []
--> 538         self.write_examples_on_file()
    539         if self.pa_writer is None:
    540             if self.schema:

/sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/arrow_writer.py in write_examples_on_file(self)
    407             # Since current_examples contains (example, key) tuples
    408             batch_examples[col] = [row[0][col] for row in self.current_examples]
--> 409         self.write_batch(batch_examples=batch_examples)
    410         self.current_examples = []
    411

/sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/arrow_writer.py in write_batch(self, batch_examples, writer_batch_size)
    506             col_try_type = try_features[col] if try_features is not None and col in try_features else None
    507             typed_sequence = OptimizedTypedSequence(batch_examples[col], type=col_type, try_type=col_try_type, col=col)
--> 508             arrays.append(pa.array(typed_sequence))
    509             inferred_features[col] = typed_sequence.get_inferred_type()
    510         schema = inferred_features.arrow_schema if self.pa_writer is None else self.schema

/sfs/MNBVC/venv/lib64/python3.6/site-packages/pyarrow/array.pxi in pyarrow.lib.array()

/sfs/MNBVC/venv/lib64/python3.6/site-packages/pyarrow/array.pxi in pyarrow.lib._handle_arrow_array_protocol()

/sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/arrow_writer.py in __arrow_array__(self, type)
    180             else:
    181                 trying_cast_to_python_objects = True
--> 182                 out = pa.array(cast_to_python_objects(data, only_1d_for_numpy=True))
    183             # use smaller integer precisions if possible
    184             if self.trying_int_optimization:

/sfs/MNBVC/venv/lib64/python3.6/site-packages/pyarrow/array.pxi in pyarrow.lib.array()

/sfs/MNBVC/venv/lib64/python3.6/site-packages/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()

/sfs/MNBVC/venv/lib64/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

OverflowError: Python int too large to convert to C long
```

However, that dataset can be loaded in a streaming manner:

```python
from datasets import load_dataset
dataset = load_dataset("liwu/MNBVC", 'news_peoples_daily', split='train', streaming=True)

for i in dataset:
    pass  # it work well
```

Another issue is reported in our dataset hub:
https://huggingface.co/datasets/liwu/MNBVC/discussions/2

### Steps to reproduce the bug

from datasets import load_dataset
dataset = load_dataset("liwu/MNBVC", 'news_peoples_daily', split='train')

### Expected behavior

the dataset can be safely loaded

### Environment info

- `datasets` version: 2.4.0
- Platform: Linux-3.10.0-1160.an7.x86_64-x86_64-with-centos-7.9
- Python version: 3.6.8
- PyArrow version: 6.0.1
- Pandas version: 1.1.5


NotADirectoryError when loading gigawords

issue due to corrupted download files. resolved after cleaning download cache. sorry for any inconvinence.

### Describe the bug

got `NotADirectoryError` whtn loading gigawords dataset

### Steps to reproduce the bug

When running
```
import datasets
datasets.load_dataset('gigaword')
```

Got the following exception:
```bash
Traceback (most recent call last):                                                                                [0/1862]
  File "/home/x/.conda/envs/dataproc/lib/python3.8/site-packages/datasets/builder.py", line 1629, in _prepare_split_single
    for key, record in generator:
  File "/home/x/.cache/huggingface/modules/datasets_modules/datasets/gigaword/ea83a8b819190acac5f2dae011fad51dccf269a0604ec5dd24795b
64efb424b6/gigaword.py", line 115, in _generate_examples
    with open(src_path, encoding="utf-8") as f_d, open(tgt_path, encoding="utf-8") as f_s:
  File "/home/x/.conda/envs/dataproc/lib/python3.8/site-packages/datasets/streaming.py", line 71, in wrapper
    return function(*args, use_auth_token=use_auth_token, **kwargs)
  File "/home/x/.conda/envs/dataproc/lib/python3.8/site-packages/datasets/download/streaming_download_manager.py", line 493, in xope
n
    return open(main_hop, mode, *args, **kwargs)
NotADirectoryError: [Errno 20] Not a directory: '/home/x/.cache/huggingface/datasets/downloads/6da52431bb5124d90cf51a0187d2dbee9046e
89780c4be7599794a4f559048ec/org_data/train.src.txt'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "gigaword.py", line 38, in <module>
    main()
  File "gigaword.py", line 35, in main
    train, dev, test = dataset.generate_k_shot_data(k=32, seed=seed, path="../data/")
  File "/home/x/MICL/preprocess/fewshot_gym_dataset.py", line 199, in generate_k_shot_data
    dataset = self.load_dataset()
  File "gigaword.py", line 29, in load_dataset
    return datasets.load_dataset('gigaword')
  File "/home/x/.conda/envs/dataproc/lib/python3.8/site-packages/datasets/load.py", line 1809, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/x/.conda/envs/dataproc/lib/python3.8/site-packages/datasets/builder.py", line 909, in download_and_prepare
    self._download_and_prepare(
  File "/home/x/.conda/envs/dataproc/lib/python3.8/site-packages/datasets/builder.py", line 1670, in _download_and_prepare
    super()._download_and_prepare(
  File "/home/x/.conda/envs/dataproc/lib/python3.8/site-packages/datasets/builder.py", line 1004, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/x/.conda/envs/dataproc/lib/python3.8/site-packages/datasets/builder.py", line 1508, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/home/x/.conda/envs/dataproc/lib/python3.8/site-packages/datasets/builder.py", line 1665, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
```

### Expected behavior

Download and process the dataset successfully

### Environment info

- `datasets` version: 2.13.1
- Platform: Linux-5.0.0-1032-azure-x86_64-with-glibc2.10
- Python version: 3.8.0
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 2.0.3