`map()` fails midway due to format incompatibility

bug

Something isn't working

You can end up with a list instead of a tensor if all the tensors inside the list can't be stacked together - can you make sure all your inputs are tensors with the same shape ?

You can make sure your `tokenize` function always return tensors of the same shape

I modified my `tokenize()` function to be like so:

```py
def tokenize(batch):
    return tokenizer(batch["text"], padding="longest")
```

so that the padding always happens w.r.t to the length of the longest sequence in a batch. The issue still persists. Is there any other way?

tbh I though your first implementation was fine
```python
def tokenize(batch):
    return tokenizer(batch["text"], padding=True, truncation=True)
```

Maybe you can try to see what the erroring data looks like by adding a try/except in `get_test_accuracy` ?

If you use the same batch_size for your two maps, you should get the exact same batches - therefore all containing the same shapes

### Describe the bug

I am using the `emotion` dataset from Hub for sequence classification. After training the model, I am using it to generate predictions for all the entries present in the `validation` split of the dataset.

```py
def get_test_accuracy(model):
    def fn(batch):
        inputs = {k:v.to(device) for k,v in batch.items()
                  if k in tokenizer.model_input_names}
        with torch.no_grad():
            output = model(**inputs)
            pred_label = torch.argmax(output.logits, axis=-1)
        return {"predicted_label": pred_label.cpu().numpy()}
    return fn
```

This is how the `get_test_accuracy()` is being used:

```py

emotions = load_dataset("emotion")

def tokenize(batch):
    return tokenizer(batch["text"], padding=True, truncation=True)

emotions_encoded = emotions.map(tokenize, batched=True)
emotions_encoded.set_format("torch",
                            columns=["input_ids", "attention_mask", "label"])

new_dataset = emotions_encoded["validation"].map(
    accuracy_fn, batched=True, batch_size=128
)
```

Complete code is available in the Colab Notebook provided below.

The `map()` process fails midway giving:

```shell
AttributeError                            Traceback (most recent call last)
<ipython-input-8-ad24ac288eb4> in <module>
      2
      3 new_dataset = emotions_encoded["validation"].map(
----> 4     accuracy_fn, batched=True, batch_size=128
      5 )

7 frames
/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py in map(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   2588                 new_fingerprint=new_fingerprint,
   2589                 disable_tqdm=disable_tqdm,
-> 2590                 desc=desc,
   2591             )
   2592         else:

/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    582             self: "Dataset" = kwargs.pop("self")
    583         # apply actual function
--> 584         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    585         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    586         for dataset in datasets:

/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    549         }
    550         # apply actual function
--> 551         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    552         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    553         # re-apply format to the output

/usr/local/lib/python3.7/dist-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    478             # Call actual function
    479
--> 480             out = func(self, *args, **kwargs)
    481
    482             # Update fingerprint of in-place transforms + update in-place history of transforms

/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py in _map_single(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, disable_tqdm, desc, cache_only)
   2970                                 indices,
   2971                                 check_same_num_examples=len(input_dataset.list_indexes()) > 0,
-> 2972                                 offset=offset,
   2973                             )
   2974                         except NumExamplesMismatchError:

/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py in apply_function_on_filtered_inputs(inputs, indices, check_same_num_examples, offset)
   2850             if with_rank:
   2851                 additional_args += (rank,)
-> 2852             processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
   2853             if update_data is None:
   2854                 # Check if the function returns updated examples

<ipython-input-6-4e0d280426f6> in fn(batch)
      1 def get_test_accuracy(model):
      2     def fn(batch):
----> 3         inputs = {k:v.to(device) for k,v in batch.items()
      4                   if k in tokenizer.model_input_names}
      5         with torch.no_grad():

<ipython-input-6-4e0d280426f6> in <dictcomp>(.0)
      2     def fn(batch):
      3         inputs = {k:v.to(device) for k,v in batch.items()
----> 4                   if k in tokenizer.model_input_names}
      5         with torch.no_grad():
      6             output = model(**inputs)

AttributeError: 'list' object has no attribute 'to'
```

As you'd notice in the notebook, the process fails _midway_ and not at the beginning.

Is this expected?

### Steps to reproduce the bug

Colab Notebook:

https://colab.research.google.com/gist/sayakpaul/d1570d537faf39040d02d77b1ed7de07/scratchpad.ipynb

### Expected behavior

The mapping process should complete as is. If you switch the `split` to `test` it works as expected.

### Environment info

Colab


Unable to download the Chinese `wikipedia`, the dumpstatus.json not found!

Hi, @beyondguo, thanks for reporting.

You have all the information in the dataset card: https://huggingface.co/datasets/wikipedia

> Then, you can load any subset of Wikipedia per language and per date this way:
> ```python
> from datasets import load_dataset
>
> load_dataset("wikipedia", language="sw", date="20220120", beam_runner=...)
> ```
> where you can pass as beam_runner any Apache Beam supported runner for (distributed) data processing (see [here](https://beam.apache.org/documentation/runners/capability-matrix/)). Pass "DirectRunner" to run it on your machine.
>
> You can find the full list of languages and dates [here](https://dumps.wikimedia.org/backup-index.html).

Note that you have to pass the language and date as keyword arguments, and the available dates depend on the language and can be found on Wikimedia website.

Also:
> Some subsets of Wikipedia have already been processed by HuggingFace, and you can load them just with:
> ```python
> load_dataset("wikipedia", "20220301.en")
> ```
> The list of pre-processed subsets is:
> - "20220301.de"
> - "20220301.en"
> - "20220301.fr"
> - "20220301.frr"
> - "20220301.it"
> - "20220301.simple"

### Describe the bug

I tried:
`data = load_dataset('wikipedia', '20220301.zh', beam_runner='DirectRunner')`
and
`data = load_dataset("wikipedia", language="zh", date="20220301", beam_runner='DirectRunner')`
but both got:
`FileNotFoundError: Couldn't find file at https://dumps.wikimedia.org/zhwiki/20220301/dumpstatus.json`

the full report is:
```
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-13-d07c5021090c> in <module>
      1 from datasets import load_dataset
      2
----> 3 data = load_dataset("wikipedia", language="zh", date="20220301", beam_runner='DirectRunner')<?, ?it/s]

/opt/conda/lib/python3.8/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1740
   1741     # Download and prepare data
-> 1742     builder_instance.download_and_prepare(
   1743         download_config=download_config,
   1744         download_mode=download_mode,

/opt/conda/lib/python3.8/site-packages/datasets/builder.py in download_and_prepare(self, output_dir, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, file_format, max_shard_size, storage_options, **download_and_prepare_kwargs)
    812                             **download_and_prepare_kwargs,
    813                         }
--> 814                         self._download_and_prepare(
    815                             dl_manager=dl_manager,
    816                             verify_infos=verify_infos,

/opt/conda/lib/python3.8/site-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_splits_kwargs)
   1645             options=beam_options,
   1646         )
-> 1647         super()._download_and_prepare(
   1648             dl_manager, verify_infos=False, pipeline=pipeline, **prepare_splits_kwargs
   1649         )  # TODO handle verify_infos in beam datasets

/opt/conda/lib/python3.8/site-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    881         split_dict = SplitDict(dataset_name=self.name)
    882         split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 883         split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    884
    885         # Checksums verification

~/.cache/huggingface/modules/datasets_modules/datasets/wikipedia/aa542ed919df55cc5d3347f42dd4521d05ca68751f50dbc32bae2a7f1e167559/wikipedia.py in _split_generators(self, dl_manager, pipeline)
    943         info_url = _base_url(lang) + _INFO_FILE
    944         # Use dictionary since testing mock always returns the same result.
--> 945         downloaded_files = dl_manager.download_and_extract({"info": info_url})
    946
    947         xml_urls = []

/opt/conda/lib/python3.8/site-packages/datasets/download/download_manager.py in download_and_extract(self, url_or_urls)
    431             extracted_path(s): `str`, extracted paths of given URL(s).
    432         """
--> 433         return self.extract(self.download(url_or_urls))
    434
    435     def get_recorded_sizes_checksums(self):

/opt/conda/lib/python3.8/site-packages/datasets/download/download_manager.py in download(self, url_or_urls)
    308
    309         start_time = datetime.now()
--> 310         downloaded_path_or_paths = map_nested(
    311             download_func,
    312             url_or_urls,

/opt/conda/lib/python3.8/site-packages/datasets/utils/py_utils.py in map_nested(function, data_struct, dict_only, map_list, map_tuple, map_numpy, num_proc, parallel_min_length, types, disable_tqdm, desc)
    427         num_proc = 1
    428     if num_proc <= 1 or len(iterable) < parallel_min_length:
--> 429         mapped = [
    430             _single_map_nested((function, obj, types, None, True, None))
    431             for obj in logging.tqdm(iterable, disable=disable_tqdm, desc=desc)

/opt/conda/lib/python3.8/site-packages/datasets/utils/py_utils.py in <listcomp>(.0)
    428     if num_proc <= 1 or len(iterable) < parallel_min_length:
    429         mapped = [
--> 430             _single_map_nested((function, obj, types, None, True, None))
    431             for obj in logging.tqdm(iterable, disable=disable_tqdm, desc=desc)
    432         ]

/opt/conda/lib/python3.8/site-packages/datasets/utils/py_utils.py in _single_map_nested(args)
    329     # Singleton first to spare some computation
    330     if not isinstance(data_struct, dict) and not isinstance(data_struct, types):
--> 331         return function(data_struct)
    332
    333     # Reduce logging to keep things readable in multiprocessing with tqdm

/opt/conda/lib/python3.8/site-packages/datasets/download/download_manager.py in _download(self, url_or_filename, download_config)
    335             # append the relative path to the base_path
    336             url_or_filename = url_or_path_join(self._base_path, url_or_filename)
--> 337         return cached_path(url_or_filename, download_config=download_config)
    338
    339     def iter_archive(self, path_or_buf: Union[str, io.BufferedReader]):

/opt/conda/lib/python3.8/site-packages/datasets/utils/file_utils.py in cached_path(url_or_filename, download_config, **download_kwargs)
    186     if is_remote_url(url_or_filename):
    187         # URL, so get it from the cache (downloading if necessary)
--> 188         output_path = get_from_cache(
    189             url_or_filename,
    190             cache_dir=cache_dir,

/opt/conda/lib/python3.8/site-packages/datasets/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag, max_retries, use_auth_token, ignore_url_params, download_desc)
    533             )
    534         elif response is not None and response.status_code == 404:
--> 535             raise FileNotFoundError(f"Couldn't find file at {url}")
    536         _raise_if_offline_mode_is_enabled(f"Tried to reach {url}")
    537         if head_error is not None:

FileNotFoundError: Couldn't find file at https://dumps.wikimedia.org/zhwiki/20220301/dumpstatus.json
```

### Steps to reproduce the bug

`data = load_dataset('wikipedia', '20220301.zh', beam_runner='DirectRunner')`

### Expected behavior

download the data

### Environment info

python3.6
latest datasets/transformers version


Update create image dataset docs

documentation

Improvements or additions to documentation

_The documentation is not available anymore as the PR was closed or merged._

Based on @osanseviero and community feedback, it wasn't super clear how to upload a dataset to the Hub after creating something like an image captioning dataset. This PR adds a brief section on how to upload the dataset with `push_to_hub`.


prepare dataset for cloud storage doesn't work

It looks like an issue with `gcsfs`, are you able to instantiate a `GCSFileSystem` manually ?

closing since it was probably due to gcsfs

### Describe the bug

Following the [documentation](https://huggingface.co/docs/datasets/filesystems#load-and-save-your-datasets-using-your-cloud-storage-filesystem) and [this PR](https://github.com/huggingface/datasets/pull/4724), I was downloading and storing huggingface dataset to cloud storage.
```
from datasets import load_dataset, load_dataset_builder
dataset = load_dataset_builder("wikipedia", "20220301.en", cache_dir='LOCAL_PATH')
dataset.download_and_prepare("gs://Bucket_NAME", file_format="parquet")
```
The above code successfully downloaded dataset, however, it returns error from `download_and_prepare`.

> Traceback (most recent call last):
>   File "/shared/zhuiai/research/wiki/wiki/gcsfs.py", line 12, in <module>
>     dataset.download_and_prepare("gs://upgen/dataset/wiki", file_format="parquet")
>   File "/shared/zhuiai/.conda/envs/wiki/lib/python3.9/site-packages/datasets/builder.py", line 671, in download_and_prepare
>     fs_token_paths = fsspec.get_fs_token_paths(output_dir, storage_options=storage_options)
>   File "/shared/zhuiai/.conda/envs/wiki/lib/python3.9/site-packages/fsspec/core.py", line 635, in get_fs_token_paths
>     cls = get_filesystem_class(protocol)
>   File "/shared/zhuiai/.conda/envs/wiki/lib/python3.9/site-packages/fsspec/registry.py", line 234, in get_filesystem_class
>     register_implementation(protocol, _import_class(bit["class"]))
>   File "/shared/zhuiai/.conda/envs/wiki/lib/python3.9/site-packages/fsspec/registry.py", line 257, in _import_class
>     mod = importlib.import_module(mod)
>   File "/shared/zhuiai/.conda/envs/wiki/lib/python3.9/importlib/__init__.py", line 127, in import_module
>     return _bootstrap._gcd_import(name[level:], package, level)
>   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
>   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
>   File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
>   File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
>   File "<frozen importlib._bootstrap_external>", line 850, in exec_module
>   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
>   File "/shared/zhuiai/research/wiki/wiki/gcsfs.py", line 12, in <module>
>     dataset.download_and_prepare("gs://upgen/dataset/wiki", file_format="parquet")
>   File "/shared/zhuiai/.conda/envs/wiki/lib/python3.9/site-packages/datasets/builder.py", line 671, in download_and_prepare
>     fs_token_paths = fsspec.get_fs_token_paths(output_dir, storage_options=storage_options)
>   File "/shared/zhuiai/.conda/envs/wiki/lib/python3.9/site-packages/fsspec/core.py", line 635, in get_fs_token_paths
>     cls = get_filesystem_class(protocol)
>   File "/shared/zhuiai/.conda/envs/wiki/lib/python3.9/site-packages/fsspec/registry.py", line 234, in get_filesystem_class
>     register_implementation(protocol, _import_class(bit["class"]))
>   File "/shared/zhuiai/.conda/envs/wiki/lib/python3.9/site-packages/fsspec/registry.py", line 258, in _import_class
>     return getattr(mod, name)
> AttributeError: partially initialized module 'gcsfs' has no attribute 'GCSFileSystem' (most likely due to a circular import)

### Steps to reproduce the bug

1. pip install datasets==2.6.1 gcsfs==2022.8.2
2. Run the following code will reproduce the issue (change `LOCAL_PATH` and `Bucket_NAME` accordingly)
```
from datasets import load_dataset, load_dataset_builder
dataset = load_dataset_builder("wikipedia", "20220301.en", cache_dir='LOCAL_PATH')
dataset.download_and_prepare("gs://Bucket_NAME", file_format="parquet")
```

### Expected behavior

Expecting successful downloading dataset and uploading it to cloud storage.

### Environment info

- `datasets` version: 2.6.1
- Platform: Linux-5.15.0-25-generic-x86_64-with-glibc2.35
- Python version: 3.9.12
- PyArrow version: 7.0.0
- Pandas version: 1.5.1


Loading an external NER dataset

I need to use huggingface datasets to load a custom dataset similar to conll2003 but with more entities and each the files contain only two columns: word and ner tag.

I tried this code snnipet that I found here as an answer to a similar issue:

from datasets import Dataset

INPUT_COLUMNS = "ID Text NER".split()

def read_conll(file):
    example = {col: [] for col in INPUT_COLUMNS}
    idx = 0
    with open(file) as f:
        for line in f:
            if line.startswith("-DOCSTART-") or line == "\n" or not line:
                if example[next(iter(example))]:
                    yield idx, example
                idx += 1
                example = {col: [] for col in INPUT_COLUMNS}
            else:
                row_cols = line.split()
                for i, col in enumerate(example):
                    example[col] = row_cols[i].rstrip()

train = Dataset.from_generator(read_conll, gen_kwargs={"file": "some_path"})

But the following error happened:

ValueError: Please pass `features` or at least one example when writing data


Preserve None in list type cast in PyArrow 10

The `ListArray` type in PyArrow 10.0.0 supports the `mask` parameter, which allows us to preserve Nones in nested lists in `cast` instead of replacing them with empty lists.

Fix https://github.com/huggingface/datasets/issues/3676


Raise ffmpeg warnings only once

Our warnings looks nice now.

`librosa` warning that was raised at each decoding:
```
/usr/local/lib/python3.7/dist-packages/librosa/core/audio.py:165: UserWarning: PySoundFile failed. Trying audioread instead.
  warnings.warn("PySoundFile failed. Trying audioread instead.")
```
is suppressed with `filterwarnings("ignore")` in a context manager. That means the first warning is also ignored (setting `filterwarnings("once")` didn't work!), so I added info that audioread is used for decoding to our message. Hope it's enough.

Tests failed at first because they used to check if the warning was raised at (each) decoding in `librosa` case but now we throw only one warning (at first decoding). I removed this check for warnings, do you think it's fine?


Inconsistency behavior between handling local file protocol and other FS protocols

### Describe the bug

These lines us used during load_from_disk:
```
if is_remote_filesystem(fs):
    dest_dataset_dict_path = extract_path_from_uri(dataset_dict_path)
else:
    fs = fsspec.filesystem("file")
    dest_dataset_dict_path = dataset_dict_path
```
If a local FS is given, then it will the URL as the path name. If a remote Fs is given, then it will use the path of the URL. This is an inconsistent behavior when handling a file: when using remote FS, you must write a URL, but for local FS, even if you passed LocalFileSystem as `fs` you still can't use a `file://` URL. It will be recognized as a directory named `file:`.

### Steps to reproduce the bug

```
import fsspec.core
url = "hdfs:///somewhere/MNIST"
# url = "file:///somewhere/MNIST"
fs, path = fsspec.core.url_to_fs(url)

fs.ls(path) # this will always work
load_from_disk(path, fs) # only works for local FS
load_from_disk(url, fs) # only works for remote FS

```

### Expected behavior

one of `url` or `path` should always work

I think we extract path from given URL by using `fsspec.core.url_to_fs` instead of using `is_remote_filesystem` and `extract_path_from_uri` will fix this, since:
```
fsspec.core.url_to_fs("/somewhere/MNIST") ->  LocalFs, '/somewhere/MNIST'
fsspec.core.url_to_fs("file:///somewhere/MNIST") ->  LocalFs, '/somewhere/MNIST'
fsspec.core.url_to_fs("hdfs:///somewhere/MNIST") ->  HDFS, '/somewhere/MNIST'
```
and
```
fsspec.core.url_to_fs("file:///somewhere/MNIST") == fsspec.core.url_to_fs("/somewhere/MNIST")
```
In theory, this wouldn't break anything, since giving local path and remote uri still works. It will only affect local URI (make it works too)

### Environment info

- `datasets` version: 2.5.1
- Platform: Linux-5.4.205.1**HIDDEN**
- Python version: 3.7.10
- PyArrow version: 8.0.0
- Pandas version: 1.2.4


Add PB and TB in convert_file_size_to_int


[Caching] Deterministic hashing of torch tensors

enhancement

New feature or request

Currently this fails
```python
import torch
from datasets.fingerprint import Hasher

t = torch.tensor([1.])

def func(x):
    return t + x

hash1 = Hasher.hash(func)
t = torch.tensor([1.])
hash2 = Hasher.hash(func)
assert hash1 == hash2
```

Also as noticed in https://discuss.huggingface.co/t/dataset-cant-cache-models-outputs/24945, using a model in a `map` function doesn't work well with caching. Indeed the `bert-base-uncased` model has a different hash every time you reload it. Supporting torch tensors may also help in this case.

This can be fixed by registering a custom pickling functions for torch tensors - as we did for other objects such as CodeType, FunctionType and Regex in `py_utils.py`


Add "ipykernel" to list of `co_filename`s to remove

I don't know how I could add some tests for this, although jupyter is not among the dependencies so at least that would need to be added. If someone can tell a recommended way I will try to do it!

So testing by myself and looking around the jupyter codebase it looks like the `co_filename` of objects created within jupyter is of the form `f"{tempfile.tempdir}/ipykernel_{id1}/{id2}.py"` however I can't find the exact command setting it so I [asked in discourse](https://discourse.jupyter.org/t/co-filename-within-notebooks/16538). For now adapted the `co_filename` filter and added tests according to this I hope to get an answer and possibly fix based on that.

Ok ! I think it's fine to just check if the parent folder is named like `ipykernel_*` then

see the source code of how it's created:

https://github.com/ipython/ipykernel/blob/7f73ff705510b35d1e2faad7f5a676c620ce08d4/ipykernel/compiler.py#L72-L75

Should look better now didn't notice the duplicated tests

I did the changes you suggested and tried to rebase, the first part went fine, the second less so :(

If you have time to spare, can you tell me what should I do now to fix this? thanks

Instead of rebasing you can just merge `main` into your branch, otherwise the GitHub preview of your PR shows changes of from `main`.

Feel free to close this PR and create a new one. Or alternatively your can force push to this PR with a new clean git history.

I have force-pushed and merged main, only shows the right changes, if you can run CI one more time it should be ok now

Hi, sorry I have been busy, the thing is I can't really understand why the test fail, besides the ugly thing I had done in the last commit to check if within CI smth stange happened with `os`, locally tests pass

The CI wasn't passing when using the latest version `dill==0.3.6`. We have a separate function to dump CodeType objects for 0.3.6

I applied the same changes you did to this other function as well - it should be all good now

> The CI wasn't passing when using the latest version `dill==0.3.6`. We have a separate function to dump CodeType objects for 0.3.6
>
> I applied the same changes you did to this other function as well - it should be all good now

Thanks, it would have taken a long time to figure out :)

Should resolve #5157


Fix CI require beam

I'm merging this PR because it is quite a trivial fix and this is required by:
- #5166

This PR:
- Fixes the CI `require_beam`: before it was requiring PyTorch instead
  ```python
  def require_beam(test_case):
      if not config.TORCH_AVAILABLE:
          test_case = unittest.skip("test requires PyTorch")(test_case)
      return test_case
  ```
- Fixes a missing `require_beam` in `test_beam_based_builder_download_and_prepare_as_parquet`
- Refactors `require_beam` to use `pytest` (`skipif`) instead


Add ffmpeg4 installation instructions in warnings

To make it warn only once, feel free to use a global counter in python - and if the warning has already been done, you don't do it again

> Added the same formatting for the error message :)

nice!! thank you!

> Oh and regarding the warning counter, you can do it in another PR maybe ?

Yes, more warnings is better then no warnings.... I'll merge when the CI passes

Adds instructions on how to install `ffmpeg=4` on Linux (relevant for Colab users).

Looks pretty ugly because I didn't find a way to check `ffmpeg` version from python (without `subprocess.call()`; `ctypes.util.find_library` doesn't work`), so the warning is raised on each decoding. Any suggestions on how to make it look nice are welcome!

This is how it looks on Colab:
![image](https://user-images.githubusercontent.com/16348744/198052412-d48018d1-4416-4aa5-9114-f7f9b4af031f.png)


Support dill 0.3.6

I think it hasn't been merged ? https://github.com/uqfoundation/dill/pull/501

Though I can see that the CI is green because it uses dill 0.3.1.1 - we should probably fix the dill version in both CIs:
- use 0.3.1.1 for the CI with the minimum requirements
- use latest for the CI with the latest requirements

I have noticed our CI uses `dill-0.3.1.1`, so not really testing dill 0.3.6...

The dill version in our CI is due to `apache-beam`...

I've tested locally: we need a specific fix for 0.3.6 (different from the previous ones)...

I think we can force the version of dill to be whatever we want in the CI - no matter what beam says. The alternative would be to run beam tests separately but it's more work

@lhoestq I tried the easiest solution: force dill==0.3.6 ignoring the requirement of apache-beam. But it doesn't work:
- For example, for `tests/test_builder.py::test_beam_based_builder_download_and_prepare_as_parquet`:
```
    @dill.dill.register(dill.dill.ModuleType)
    def save_module(pickler, obj):
      if dill.dill.is_dill(pickler) and obj is pickler._main:
        return old_save_module(pickler, obj)
      else:
>       dill.dill.log.info('M2: %s' % obj)
E       AttributeError: module 'dill._dill' has no attribute 'log'

venv/lib/python3.9/site-packages/apache_beam/internal/dill_pickler.py:170: AttributeError
```
  - Apache Beam registers some dill functions (`save_module`) which are incompatible with dill 0.3.6 (in 0.3.6 'dill._dill' has no attribute 'log' but 'logger')
  - This has an impact in CI tests using either Apache Beam or `multiprocess` (even without using Apache Beam!):
```
FAILED tests/test_beam.py::BeamBuilderTest::test_download_and_prepare - AttributeError: module 'dill._dill' has no attribute 'log'
FAILED tests/test_beam.py::BeamBuilderTest::test_nested_features - AttributeError: module 'dill._dill' has no attribute 'log'
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_filter_multiprocessing_in_memory - AttributeError: module 'dill._dill' has no attribute 'log'
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_filter_multiprocessing_on_disk - AttributeError: module 'dill._dill' has no attribute 'log'
FAILED tests/test_builder.py::test_beam_based_download_and_prepare - AttributeError: module 'dill._dill' has no attribute 'log'
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_map_caching_in_memory - AttributeError: module 'dill._dill' has no attribute 'log'
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_map_caching_on_disk - AttributeError: module 'dill._dill' has no attribute 'log'
FAILED tests/test_builder.py::test_beam_based_as_dataset - AttributeError: module 'dill._dill' has no attribute 'log'
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_map_multiprocessing_in_memory - AttributeError: module 'dill._dill' has no attribute 'log'
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_map_multiprocessing_on_disk - AttributeError: module 'dill._dill' has no attribute 'log'
FAILED tests/test_builder.py::test_beam_based_builder_download_and_prepare_as_parquet - AttributeError: module 'dill._dill' has no attribute 'log'
```

I guess we should implement the other option: run beam tests separately.

I'm opening another PR for the CI refactoring.

Ah crap >< maybe only install apache_beam for the "minimum requirements" CI

@lhoestq if we install apache-beam only in the "minimum requirements" CI, then this other PR should be merged first:
- #5168

Otherwise, our CI for "latest" will fail because it will try to run the beam tests (because PyTorch is installed but indeed apache-beam is not installed).

One of the test is failing because we set
```python
# google colab doesn't allow to pickle loggers
# so we want to make sure each tests passes without pickling the logger
def reduce_ex(self):
    raise pickle.PicklingError()

datasets.arrow_dataset.logger.__reduce_ex__ = reduce_ex
```
in `test_arrow_dataset.py` to avoid pickling the logger because it used to fail on google colab.

Now pickling the logger seems to be working on google colab again - so you can remove it, and it should fix some tests

For the other 2 errors:
- FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_map_caching_in_memory - _pickle.PicklingError: Can't pickle <class 'unittest.mock.MagicMock'>: it's not the same object as unittest.mock.MagicMock
- FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_map_caching_on_disk - _pickle.PicklingError: Can't pickle <class 'unittest.mock.MagicMock'>: it's not the same object as unittest.mock.MagicMock

I have implemented a pickable MagicMock.

This PR:
- ~~Unpins dill to allow installing dill>=0.3.6~~
- ~~Removes the fix on dill for >=0.3.6 because they implemented a deterministic mode (to be confirmed by @anivegesana)~~
- Pins dill<0.3.7 to allow latest dill 0.3.6
- Implements a fix for dill `save_function` for dill 0.3.6
- Additionally had to implement a fix for dill `save_code` and `_save_regex` for dill 0.3.6
- Fixes the CI so that the latest dill version is tested (besides the minimum 0.3.1.1 required by apache-beam 2.42.0)

Fix #5162.


Memory explosion when trying to access 4d tensors in datasets cast to torch or np

### Describe the bug

When trying to access an item by index, in a datasets.Dataset cast to torch/np using `set_format` or `with_format`, we get a memory explosion if the item contains 4d (or above) tensors.

### Steps to reproduce the bug

MWE:

```python
from datasets import load_dataset
import numpy as np

def create_4d_tensor(item):
    i = item["num_nodes"]
    item["x_big"] = np.random.rand(i, 2*i, int(i/2), 1) + 1 # we create a big 4d tensor
    return item

if __name__ == "__main__":
    dataset = load_dataset(path=f"graphs-datasets/PROTEINS")

    # This works
    print(dataset["train"].format)
    print(dataset["train"][0].keys())

    dataset = dataset.map(
        create_4d_tensor,
        batched=False,
        writer_batch_size=100,
    )

    # This works
    print(dataset["train"].format)
    print(dataset["train"][0].keys())

    dataset.set_format("torch")

    print(dataset["train"].format)
    # This gets killed :(
    print(dataset["train"][0].keys())
```

The problem likely comes from `format_table` [here](https://cs.github.com/huggingface/datasets/blob/f09f781be3278156ce3aa6ec90c1926b1846a78f/src/datasets/arrow_dataset.py#L2328)

### Expected behavior

No memory explosion when trying to access dataset items after cast.

### Environment info

- `datasets` version: 2.3.2
- Platform: Linux-5.14.0-1054-oem-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


WIP: drop labels in Image and Audio folders by default

_The documentation is not available anymore as the PR was closed or merged._

close in favor of https://github.com/huggingface/datasets/pull/5192

will fix https://github.com/huggingface/datasets/issues/5153 and redundant labels displaying for most of the images datasets on the Hub (which are used just to store files)

TODO: discuss adding `drop_labels` (and `drop_metadata`) params to yaml


Reduce default max `writer_batch_size`

Reduce the default writer_batch_size from 10k to 1k examples. Additionally, align the default values of `batch_size` and `writer_batch_size` in `Dataset.cast` with the values from the corresponding docstring.


Pip-compile: Could not find a version that matches dill<0.3.6,>=0.3.6

Thanks for reporting, @Rijgersberg.

We were waiting for the release of `dill` 0.3.6, that happened 2 days ago (24 Oct 2022): https://github.com/uqfoundation/dill/releases/tag/dill-0.3.6
- See comment: https://github.com/huggingface/datasets/pull/4397#discussion_r880629543

Also `multiprocess` 0.70.14 was released 2 days ago: https://github.com/uqfoundation/multiprocess/releases/tag/multiprocess-0.70.14

We are addressing this issue to align dependencies.

In your specific setup, I guess the compatible configuration is with `multiprocess` 0.70.13 (instead of 0.70.14).

> @Rijgersberg this issue is fixed. It will be available in our next `datasets` release.

Any chance you have a eta?

@StefanSamba we are disussing about making a release early this week.

@Rijgersberg, please also that you can make `pip-compile` work by using the backtracking resolver (instead of the legacy one): https://pip-tools.readthedocs.io/en/latest/#a-note-on-resolvers
```
pip-compile --resolver=backtracking requirements.in
```
This resolver will automatically use `multiprocess` 0.70.13 version.

### Describe the bug

When using `pip-compile` (part of `pip-tools`) to generate a pinned requirements file that includes `datasets`, a version conflict of `dill` appears.

It is caused by a transitive dependency conflict between `datasets` and `multiprocess`.

### Steps to reproduce the bug

```bash
$ echo "datasets" > requirements.in
$ pip install pip-tools
$ pip-compile requirements.in
Could not find a version that matches dill<0.3.6,>=0.3.6 (from datasets==2.6.1->-r requirements.in (line 1))
Tried: 0.2, 0.2, 0.2.1, 0.2.1, 0.2.2, 0.2.2, 0.2.3, 0.2.3, 0.2.4, 0.2.4, 0.2.5, 0.2.5, 0.2.6, 0.2.7, 0.2.7.1, 0.2.8, 0.2.8.1, 0.2.8.2, 0.2.9, 0.3.0, 0.3.1, 0.3.1.1, 0.3.2, 0.3.3, 0.3.3, 0.3.4, 0.3.4, 0.3.5, 0.3.5, 0.3.5.1, 0.3.5.1, 0.3.6, 0.3.6
Skipped pre-versions: 0.1a1, 0.2a1, 0.2a1, 0.2b1, 0.2b1
There are incompatible versions in the resolved dependencies:
  dill<0.3.6 (from datasets==2.6.1->-r requirements.in (line 1))
  dill>=0.3.6 (from multiprocess==0.70.14->datasets==2.6.1->-r requirements.in (line 1))
```

### Expected behavior

A correctly generated file `requirements.txt` with pinned dependencies

### Environment info

Tested with versions `2.6.1, 2.6.0, 2.5.2` on Python 3.8 and 3.10 on Ubuntu 20.04LTS and Python 3.10 on MacOS 12.6 (M1).


Dataset can’t cache model’s outputs

Addressed in https://github.com/huggingface/datasets/pull/5191 (torch.Tensor objects now produce deterministic hashes)

### Describe the bug

Hi,

I try to cache some outputs of teacher model( Knowledge Distillation ) by using map function of Dataset library, while every time I run my code, I still recompute all the sequences. I tested Bert Model like this, I got different hash every single run, so any idea to deal with this?

### Steps to reproduce the bug

1. run below code
2. get different hash
```
from transformers import BertModel
from transformers import AutoTokenizer
import torch
token = ['hello']
model = BertModel.from_pretrained("bert-base-uncased").eval()
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
def abcd():
    with torch.no_grad():

        out = model(**tok(token,return_tensors='pt'))[0]
        # out = tok(token)
    return out
from datasets.fingerprint import Hasher
my_func = abcd
print(Hasher.hash(my_func))

print(abcd())
```

### Expected behavior

I wanna cache all the model output

### Environment info

datasets:2.5.0


Automatically add filename for image/audio folder

New feature or request

BTW the exact same holds true for the audio folder

I'm fine with adding a new column with the file name personally. Not sure how breaking this is though

@patrickvonplaten do you mean just filename or full relative path inside the repo?
I think it shouldn't be breaking, at least I cannot come up with any case where it is. Maybe @mariosasko can?

also I think that the problem here and in general is that Image/AudioFolder has default configuration which implies automatic label creation if there is not metadata file. It can be changed when you load the dataset with `load_dataset` but not on it's Hub page.

> also I think that the problem here and in general Image/AudioFolder has default configuration which implies automatic label creation if there is not metadata file

Yea I agree it's often the wrong default. We can also imagine adding the builder's parameters as YAML in the repo.

@lhoestq yes I also got the idea of some YAML config! not sure of what priority it is though.

but it would actually also solve this issue: https://github.com/huggingface/datasets/issues/5153

I meant just the file name (no path) that would already be super helpful IMO :-) (maybe dir+filename if there are dirs in the folder)

@patrickvonplaten one more time, to be sure I understand you.
For example, we have data structure like this:
```
├─ data/
│  └─ subdir/
│     └── cats/
│          ├── 0.jpg
│          ├── 1.jpg
│          └── 2.jpg
│     └── dogs/
│          ├── 0.jpg
│          ├── 1.jpg
│          └── 2.jpg
└── another_subdir/
      ├── 10.jpg
      ├── 11.jpg
      └── 12.jpg
```
Is it okay to provide `"data/subdir/cats/0.jpg"`, `"data/subdir/dogs/0.jpg"`, `"data/another_subdir/10.jpg"`?
I think providing just filenames might be confusing if they are not unique, as in this example.

Yes I think the relative path as you proposed makes a lot of sense :-)

### Feature request

When creating a custom audio of image dataset, it would be great to automatically have access to the filename. It should be both:
a) Automatically displayed in the viewer
b) Automatically added as a column to the dataset when doing `load_dataset`

In `diffusers` our test rely quite heavily on images and audio files now and it's a bit tedious at the moment to download specific images from a datasets repo.

E.g. we have a dataset of images for tests in `diffusers`: https://huggingface.co/datasets/hf-internal-testing/diffusers-images
where it would be extremely nice to have direct access to the filename both visually on the datasets page (@severo ) as well as via the `load_datasets` function. We currently have some akward functionality to download images by path name: https://github.com/huggingface/diffusers/blob/2fb8fafa4b761f6fc144cf75a6f6f0ea6af3a1c1/src/diffusers/utils/testing_utils.py#L131
It would be much nicer to just go over `load_dataset(...)`

### Motivation

Intuitively the filename is something people understand directly. E.g if you upload a folder of images online, it's nice if you recognize the image as well as the filename next to it directly and that you're able to use it right away.

The label on the other hand is less intuitive to understand as you haven't added it yourself.

### Your contribution

Not sure if I have the time to add it myself anytime soon, but it would help us a lot for `diffusers`.


fsspec lock reset in multiprocessing

_The documentation is not available anymore as the PR was closed or merged._

`fsspec` added a clean way of resetting its lock - instead of doing it manually


Fix language and license tag names in all Hub datasets

dataset contribution

Contribution to a dataset script

There are currently 402 datasets with deprecated "languages" or "licenses".

hey @albertvillanova ,i would love to work on this issue if you like.

Hi @ayushthe1, thanks for your offer.

But as you can see, I self-assigned this issue.

I have already fixed 200 out of the 402 datasets. My script is still running and fixing the rest.

For example: https://huggingface.co/datasets/fhamborg/news_sentiment_newsmtsc/discussions/2/files

Thanks for your time. Will try next time. 😇

@ayushthe1 feel free to take one of the non-assigned open issues: https://github.com/huggingface/datasets/issues

While working on this:
- #5137

we realized there are still many datasets with deprecated "languages" and "licenses" tag names (instead of "language" and "license").

This is a blocking issue: no subsequent PR can be opened to modify their metadata: a ValueError will be thrown.

We should fix the "language" and "license" tag names in all Hub datasets.

TODO:
- [x] Fix language and license tag names in 402 Hub datasets

CC: @julien-c


Consistent caching between python and jupyter

enhancement

New feature or request

Hi ! Maybe it's possible to have a consistent hash for a function defined in `__main__` and a function define in a notebook.

However for functions imported from another location, pickle uses the location to identify the code, so in that case we can't do much I believe.

Would it be ok for you if we only try to do this for functions in `__main__` / jupyter ?

If you'd like to contribute, you can read this part of the code and let me know if you have questions:

https://github.com/huggingface/datasets/blob/7feeb5648a63b6135a8259dedc3b1e19185ee4c7/src/datasets/utils/py_utils.py#L617-L643

I think the key here would be to also ignore the "co_filename" of functions defined in `__main__`

Seems like a good solution, I will start a PR and see if I understood the changes needed. Thanks!

### Feature request

I hope this is not my mistake, currently if I use `load_dataset` from a python session on a custom dataset to do the preprocessing, it will be saved in the cache and in other python sessions it will be loaded from the cache, however calling the same from a jupyter notebook does not work, meaning the preprocessing starts from scratch.

If adjusting the hashes is impossible, is there a way to manually set dataset fingerprint to "force" this behaviour?

### Motivation

If this is not already the case and I am doing something wrong, it would be useful to have the two fingerprints consistent so one can create the dataset once and then try small things on jupyter without preprocessing everything again.

### Your contribution

I am happy to try a PR if you give me some pointers where the changes should happen


Unable to download dataset using Azure Data Lake Gen 2

Hi ! From the `adlfs` docs, there are two filesystems you can use:
> To use the Gen1 filesystem:
> - known_implementations[‘adl’] = {‘class’: ‘adlfs.AzureDatalakeFileSystem’}
>
> To use the Gen2 filesystem:
>  - known_implementations[‘abfs’] = {‘class’: ‘adlfs.AzureBlobFileSystem’}

If I'm not mistaken you're using the second one - so you should use `abfs://` instead of `adl://`, and also run this at the beginning of your script:
```python
from fsspec.registry import known_implementations
known_implementations['abfs'] = {'class': 'adlfs.AzureDatalakeFileSystem'}
```

Thank you @lhoestq . Great call.
Using the default class from `known_implementations` dict solved my problem
```
known_implementations[‘abfs’] = {‘class’: ‘adlfs.AzureBlobFileSystem’}
```
I'm closing this issue.

> Thank you @lhoestq . Great call. Using the default class from `known_implementations` dict solved my problem
>
> ```
> known_implementations[‘abfs’] = {‘class’: ‘adlfs.AzureBlobFileSystem’}
> ```
>
> I'm closing this issue.

Hi so here  `Saving serialized datasets

After you have processed your dataset, you can save it to your cloud storage with [Dataset.save_to_disk()](https://huggingface.co/docs/datasets/v2.17.0/en/package_reference/main_classes#datasets.Dataset.save_to_disk):` what is the encoded dataset I have failed to save it

Uploading failed ? Did you get an error message ?

### Describe the bug

When using the DatasetBuilder method with the credentials for the cloud storage Azure Data Lake (adl) Gen2, the following error is showed:
```
Traceback (most recent call last):
  File "download_hf_dataset.py", line 143, in <module>
    main()
  File "download_hf_dataset.py", line 102, in main
    builder.download_and_prepare(save_dir, storage_options=storage_options, max_shard_size="250MB", file_format="parquet")
  File "/home/clarisses/miniconda3/envs/hf_datasets_env/lib/python3.8/site-packages/datasets/builder.py", line 671, in download_and_prepare
    fs_token_paths = fsspec.get_fs_token_paths(output_dir, storage_options=storage_options)
  File "/home/clarisses/miniconda3/envs/hf_datasets_env/lib/python3.8/site-packages/fsspec/core.py", line 639, in get_fs_token_paths
    fs = cls(**options)
  File "/home/clarisses/miniconda3/envs/hf_datasets_env/lib/python3.8/site-packages/fsspec/spec.py", line 76, in __call__
    obj = super().__call__(*args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'account_name'
```

If I don't pass the storage_options argument (leave it as None), it requires the credentials used in ADL Gen 1:
`TypeError: __init__() missing 3 required positional arguments: 'tenant_id', 'client_id', and 'client_secret'`

Thus, it is not possible to download a dataset from the cloud using Azure Data Lake (adl) Gen2.

### Steps to reproduce the bug

Assuming that you have an account on Azure and at Storage Account that can be used for reproduce:

1. Create a dict with the format to connect to Azure Data Lake Gen 2
```
storage_options = {"account_name": ACCOUNT_NAME, "account_key": ACCOUNT_KEY)  # gen 2 filesystem
```
2. Create a dataset builder for any HF hosted dataset
```
builder = load_dataset_builder(dataset_name)
```
3. Try to download the dataset passing the storage_options as an argument
```
save_dir = 'adl://my_save_dir'
builder.download_and_prepare(save_dir, storage_options=storage_options, max_shard_size="250MB", file_format="parquet")
```

### Expected behavior

Not seeing the error mentioned above and being able to download the dataset to the provided path on ADL

### Environment info

- `datasets` version: 2.6.1
- Platform: Linux-5.15.0-46-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- PyArrow version: 9.0.0
- Pandas version: 1.5.1


TextConfig: added "errors"

[**@lhoestq**](https://github.com/lhoestq) commented on [Oct 27, 2022, 4:08 PM GMT+3:30](https://github.com/huggingface/datasets/pull/5155#issuecomment-1293464680 "2022-10-27T12:38:04Z - Replied by Github Reply Comments"):
> Thanks for adding this ! You can fix the CI by formatting your code using the `make  style` command :)

I ran this and force pushed the changes.

This patch adds the ability to set the `errors` option of `open` for loading text datasets. I needed it because some data I had scraped had bad bytes in it, so I needed `errors='ignore'`.


Test latest fsspec in CI

actually the latest fsspec is already installed

Following the discussion in https://discuss.huggingface.co/t/attributeerror-module-fsspec-has-no-attribute-asyn/19255 I think we need to test the latest fsspec in the CI


default Image/AudioFolder infers labels when there is no metadata files even if there is only one dir

bug

Something isn't working

Makes sense! For the last structure, we could count the path segments (delimited by "/" for URLs and `os.sep`  for local paths) to ensure all inferred labels are on the same level. Otherwise, I think it's safe to assume they are meaningless and ignore them.

### Describe the bug

By default FolderBasedBuilder infers labels if there is not metadata files, even if it's meaningless (for example, they are in a single directory or in the root folder, see this repo as an example: https://huggingface.co/datasets/patrickvonplaten/audios

As this is a corner case for quick exploration of images or audios on the Hub.

### Steps to reproduce the bug

If you have directory like this:

```
repo
   image1.jpg
   image2.jpg
   image3.jpg
```
or

```
repo
   data
      image1.jpg
      image2.jpg
      image3.jpg
```
doing `ds = load_dataset(repo)` would create `label` feature:
```python
print(ds["train"][0])
>> {'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x375 at 0x7FB5326468E0>, 'label': 0}
```

Also, if you have the following structure:

```
repo
   data
      image1.jpg
      image2.jpg
      image3.jpg
   image4.jpg
   image5.jpg
   image6.jpg
```
it will infer two labels:

```python
print(ds["train"][0])
print(ds["train"][-1])
>> {'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x375 at 0x7FB5326468E0>, 'label': 1}
>> {'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x415 at 0x7FB5326555B0>, 'label': 0}
```

### Expected behavior

We should have only one base feature (Image/Audio) in such cases.

### Environment info

all versions of `datasets`


refactor FolderBasedBuilder and Image/AudioFolder tests

refactoring

Restructuring existing code without changing its external behavior

Tests for FolderBasedBuilder, ImageFolder and AudioFolder are mostly duplicating each other. They need to be refactored and Audio/ImageFolder should have only tests specific to the loader.


Add support to create different configs with `push_to_hub` (+ inferring configs from directories with package managers?)

enhancement

New feature or request

also asked in https://discuss.huggingface.co/t/create-multiple-dataset-configs-with-push-to-hub-method/25480

Now one can push only different splits within one default config of a dataset.

Would be nice to allow something like:
```
ds.push_to_hub(repo_name, config=config_name)
```

I'm not sure, but this will probably require changes in `data_files.py` patterns. If so, it would also allow to create different configs for packaged modules datasets.


Problems after upgrading to 2.6.1

Hi! I can't reproduce the error following these steps. Can you please provide a reproducible example?

I faced the same issue:

### Repro
```
!pip install datasets==2.6.1
import datasets as Dataset
dataset = Dataset.from_pandas(dataframe)
dataset.save_to_disk(local)

!pip install datasets==2.5.2
import datasets as Dataset
dataset = Dataset.load_from_disk(local)
```

@Lokiiiiii And what are the contents of the "dataframe" in your example?

I bumped into the issue too. @Lokiiiiii thanks for steps. I "solved" if for now by `pip install datasets>=2.6.1` everywhere.

Hi all,
I experienced the same issue.
Please note that the pull request is related to the IMDB example provided in the doc, and is a fix for that, in that context, to make sure that people can follow the doc example and have a working system.
It does not provide a fix for Datasets itself.

im getting the same error.
- using the base AWS HF container that uses a datasets <2.
- updating the AWS HF container to use dataset 2.4

Same here, running on our SageMaker pipelines. It's only happening for some but not all of our saved Datasets.

I am also receiving this error on Sagemaker but not locally, I have noticed that this occurs when the `.dataset/` folder does not contain a single file like:

`dataset.arrow`

but instead contains multiple files like:

`data-00000-of-00002.arrow`
`data-00001-of-00002.arrow`

I think that it may have something to do with this recent PR that updated the behaviour of `dataset.save_to_disk` by introducing sharding: https://github.com/huggingface/datasets/pull/5268

For now I can get around this by forcing datasets==2.8.0 on machine that creates dataset and in the huggingface instance for training (by running this at the start of training script `os.system("pip install datasets==2.8.0")`)

To ensure the dataset is a single shard when saving the dataset locally:

```python3
dataset.flatten_indices().save_to_disk('path/to/dataset', num_shards=1)
```

 and then manually changing the name afterwards from `path/to/dataset/data-00000-of-00001.arrow` to `path/to/dataset/dataset.arrow` and updating the `path/to/dataset/state.json` to reflect this name change. i.e. by changing `state.json` to this:

```javascript
{
  "_data_files": [
    {
      "filename": "dataset.arrow"
    }
  ],
  "_fingerprint": "420086f0636f8727",
  "_format_columns": null,
  "_format_kwargs": {},
  "_format_type": null,
  "_output_all_columns": false,
  "_split": null
}
```

Does anyone know if this has been resolved?

I have the same issue in datasets version 2.3.2

### Describe the bug

Loading a dataset_dict from disk with `load_from_disk` is now creating a `KeyError "length"` that was not occurring in v2.5.2.

Context:
- Each individual dataset in the dict is created with `Dataset.from_pandas`
- The dataset_dict is create from a dict of `Dataset`s, e.g., `DatasetDict({"train": train_ds, "validation": val_ds})
- The pandas dataframe, besides text columns, has a column with a dictionary inside and potentially different keys in each row. Correctly the `Dataset.from_pandas` function adds `key: None` to all dictionaries in each row so that the schema can be correctly inferred.

### Steps to reproduce the bug

Steps to reproduce:

- Upgrade to datasets==2.6.1
- Create a dataset from pandas dataframe with `Dataset.from_pandas`
- Create a dataset_dict from a dict of `Dataset`s, e.g., `DatasetDict({"train": train_ds, "validation": val_ds})
- Save to disk with the `save` function

### Expected behavior

Same as in v2.5.2, that is load from disk without errors

### Environment info

- `datasets` version: 2.6.1
- Platform: Linux-5.4.209-129.367.amzn2int.x86_64-x86_64-with-glibc2.26
- Python version: 3.9.13
- PyArrow version: 9.0.0
- Pandas version: 1.5.1


Make iter_files deterministic

_The documentation is not available anymore as the PR was closed or merged._

Fix #5145.


Cannot find the rvl_cdip dataset

Hi, @santule.

We have transferred all dataset scripts from GitHub to the Hugging Face Hub: https://huggingface.co/datasets
- Concretely, you have "rvl_cdip" here: https://huggingface.co/datasets/rvl_cdip

To be able to load them, you should update your `datasets` library:
```
pip install -U datasets
```

Hi,
I am trying to use load_dataset to load the official "rvl_cdip" dataset but getting an error.

dataset = load_dataset("rvl_cdip")

Couldn't find 'rvl_cdip' on the Hugging Face Hub either: FileNotFoundError: Couldn't find the file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/rvl_cdip/rvl_cdip.py

Regards,


Allow ignoring kwargs inside fn_kwargs during dataset.map's fingerprinting

New feature or request

Hi ! In the `transformers` issue the object to not hash is a `Pool` - I think you can instantiate it inside your function instead of passing it as a parameter. It's good practice that your function and all its fn_kwargs are picklable, in case you want to parallelize `map` using `num_proc>1`

For the other case `def fn(example, verbose=False):` however, I agree it would be nice to let the user specify that "verbose" needs to be ignored.

Do you think providing a decorator could help ? Maybe
```python
@datasets.hashing.register(ignore_kwargs=["verbose"])
def func(example, verbose=False):
    ...
```

Hi @lhoestq! Thanks for your response.

A `Pool` shouldn't be instantiated within the function, because there's a huge overhead in doing so. The main idea is that the same `Pool` should be used across all function calls. Parallel `map` is not helpful/desired in that specific scenario, because the heavy parallel computation is done by another lib (`pyctcdecode`, called within `transformer`'s model inference code).

But yes, it makes sense to be able to leverage parallel processing by just doing `num_proc>1` when possible.

Your decorator suggestions seems like a pretty clean API to me. I didn't find a `datasets.hashing` module though. Would it be created for this specific purpose? Any downsides in just using `datasets.fingerprint`?

And would `datasets.hashing.register` just add some metadata to `func` in your approach (so it could be inspected from `fingerprint_transform`)?

And looking to the `datasets.Dataset` API, `.filter` would also benefited from this.

> Would it be created for this specific purpose? Any downsides in just using datasets.fingerprint?

This can also go in datasets.fingerprint indeed - but maybe datasets.hashing tells more about what the register function does (i.e. register this function to have a custom hashing) ?

> And would datasets.hashing.register just add some metadata to func in your approach (so it could be inspected from fingerprint_transform)?

Yup that's the idea :)

> And looking to the datasets.Dataset API, .filter would also benefited from this.

Indeed !

-----

If you would like to contribute this you can assign yourself to this issue by posting #self-assign
And of course if you have questions or if I can help, feel free to ping me !

> This can also go in datasets.fingerprint indeed - but maybe datasets.hashing tells more about what the register function does (i.e. register this function to have a custom hashing) ?

Sure, it makes sense.

---

I don't plan to work on it right now, so I'll let it unassigned in case somebody wants to join. I'll get back at it as soon as possible though.

### Feature request

`dataset.map` accepts a `fn_kwargs` that is passed to `fn`. Currently, the whole `fn_kwargs` is used by `fingerprint_transform` to calculate the new fingerprint.

I'd like to be able to inform `fingerprint_transform` which `fn_kwargs` shoud/shouldn't be taken into account during hashing.

Of course, users should be aware to properly use this new feature, just like the internal usages of `fingerprint_transform` [does](https://github.com/huggingface/datasets/blob/2699593b33ee63d17aad2a2bfddedd38a8df57b8/src/datasets/arrow_dataset.py#L2700).

### Motivation

This is originally motivated by https://github.com/huggingface/transformers/pull/18351#issuecomment-1263588680.

Nonetheless, consider a more general processing function that accepts a kwarg that does not influence it's output:
```python
def fn(example, verbose=False):
    ...
```

Then `dataset.map(fn, verbose=True)` would not benefit from dataset caching.

I'm not sure if other methods in the `Dataset` API could benefit from this feature.

### Your contribution

Based on `fingerprint_transform `'s `wrapper` function [here](https://github.com/huggingface/datasets/blob/c59cc34fcd2a369d27b77cc678017f5976a926a9/src/datasets/fingerprint.py#L443), it seems to me that it should be possible to make `.map`/`._map_single` accept something like `fn_use_fingerprint_kwargs`/`fn_ignore_fingerprint_kwargs` (probably another arg name). This would then be used by `fingerprint_transform.wrapper` to better/more flexibly hash the transformation.

I could contribute with a PR if this feature and approach look good to you.


Delete duplicate issue template file

A conflict between two PRs:
- #5116
- #5136

was not properly resolved, resulting in a duplicate issue template.

This PR removes the duplicate template.


Dataset order is not deterministic with ZIP archives and `iter_files`

Thanks for reporting ! The issue doesn't come from shuffling, but from `beans` row order not being deterministic:

https://huggingface.co/datasets/beans/blob/main/beans.py uses `dl_manager.iter_files` on ZIP archives and the file order doesn't seen to be deterministic and changes across machines

This is still a bug, so I'd keep this one open if you don't mind ;)

Besides the linked PR, to make the loading process fully deterministic, I believe we should also sort the data files [here](https://github.com/huggingface/datasets/blob/df4bdd365f2abb695f113cbf8856a925bc70901b/src/datasets/data_files.py#L276) and [here](https://github.com/huggingface/datasets/blob/df4bdd365f2abb695f113cbf8856a925bc70901b/src/datasets/data_files.py#L485) (e.g. fsspec's `LocalFileSystem.glob` relies on `os.scandir`, which yields the contents in arbitrary order). My concern is the overhead of these sorts... Maybe we could introduce a new flag to `load_dataset` similar to TFDS' [`shuffle_files`](https://www.tensorflow.org/datasets/determinism#determinism_when_reading) or sort only if the number of data files is small?

We already return the result sorted at the end of `_resolve_single_pattern_locally` and `_resolve_single_pattern_in_dataset_repository` if I'm not mistaken

@lhoestq Oh, you are right. Feel free to ignore my comment.

I think the corresponding PR is ready to be merged :hugs:

### Describe the bug

For the `beans` dataset (did not try on other), the order of samples is not the same on different machines. Tested on my local laptop, github actions machine, and ec2 instance. The three yield a different order.

### Steps to reproduce the bug

In a clean docker container or conda environment with datasets==2.6.1, run

```python
from datasets import load_dataset
from pprint import pprint

data = load_dataset("beans", split="validation")

pprint(data["image_file_path"])
```

### Expected behavior

The order of the images is the same on all machines.

### Environment info

On the EC2 instance:
```
- `datasets` version: 2.6.1
- Platform: Linux-4.14.291-218.527.amzn2.x86_64-x86_64-with-glibc2.2.5
- Python version: 3.7.10
- PyArrow version: 9.0.0
- Pandas version: 1.3.5
- Numpy version: not checked
```

On my local laptop:
```
- `datasets` version: 2.6.1
- Platform: Linux-5.15.0-50-generic-x86_64-with-glibc2.35
- Python version: 3.9.12
- PyArrow version: 7.0.0
- Pandas version: 1.3.5
- Numpy version: 1.23.1
```

On github actions:
```
- `datasets` version: 2.6.1
- Platform: Linux-5.15.0-1022-azure-x86_64-with-glibc2.2.5
- Python version: 3.8.14
- PyArrow version: 9.0.0
- Pandas version: 1.5.1
- Numpy version: 1.23.4
```


Inconsistent documentation on map remove_columns

duplicate

This issue or pull request already exists

good first issue

Good for newcomers

Thanks for reporting, @zhaowei-wang-nlp.

You are right, the documentation is confusing on the behavior of `remove_columns`. We should better explain it.

This is a duplicate of https://github.com/huggingface/datasets/issues/2343.

I'm closing this issue because as @mariosasko pointed out, it is a duplicate of:
- #2343

### Describe the bug

The page [process](https://huggingface.co/docs/datasets/process) says this about the parameter `remove_columns` of the function `map`:
When you remove a column, it is only removed after the example has been provided to the mapped function.

So it seems that the `remove_columns` parameter removes after the mapped functions.

However, another page, [the documentation of the function map](https://huggingface.co/docs/datasets/v2.6.1/en/package_reference/main_classes#datasets.Dataset.map.remove_columns) says:
Columns will be removed before updating the examples with the output of `function`, i.e. if `function` is adding columns with names in remove_columns, these columns will be kept.

So one page says "after the mapped function" and another says "before the mapped function."
Is there something wrong?

### Steps to reproduce the bug

Not about code.

### Expected behavior

consistent about the descriptions of the behavior of the parameter `remove_columns` in the function `map`.

### Environment info

datasets V2.6.0


DownloadManager Git LFS support

enhancement

New feature or request

Hey ! Actually it works, just pass the right URL ;)
The URL must be the one with “/resolve/”

e.g. https://huggingface.co/datasets/imagenet-1k/resolve/main/data/test_images.tar.gz

You can even pass a relative path to the dl_manager instead, like `dl_manager.download("data/test_images.tar.gz")`

### Feature request

Maybe I'm mistaken but the `DownloadManager` does not support extracting git lfs files out of the box right?
Using `dl_manager.download()` or `dl_manager.download_and_extract()` still returns lfs files afaict.

Is there a good way to write a dataset loading script for a repo with lfs files?

### Motivation

/

### Your contribution

/


Deprecate num_proc parameter in  DownloadManager.extract

Hey @mariosasko . Can you please help me with why the tests keep failing.  I have reviewed the code changes multiple times but can't spot any mistakes.

You can fix this failure by formatting your code with the `make style` command (run it from the root of the cloned repo).

hey @mariosasko ,i cant understand how to use the `make style` command .I searched for it on the internet but cant find any results.
So i formatted the code using vs-code document formatter. Hope this helps.

`make style` runs the "style" target defined here: https://github.com/huggingface/datasets/blob/f09f781be3278156ce3aa6ec90c1926b1846a78f/Makefile#L12

This seems to be a good tutorial on Makefiles: https://opensource.com/article/18/8/what-how-makefile

> `make style` runs the "style" target defined here:
>
> https://github.com/huggingface/datasets/blob/f09f781be3278156ce3aa6ec90c1926b1846a78f/Makefile#L12
>
> This seems to be a good tutorial on Makefiles: https://opensource.com/article/18/8/what-how-makefile

Thanks! I will look into this :relaxed:

fixes #5132 : Deprecated the `num_proc` parameter in `DownloadManager.extract` by passing `num_proc` parameter to `map_nested` .


Raise ImportError instead of OSError

_The documentation is not available anymore as the PR was closed or merged._

Thanks @mariosasko ,i commited the changes as you said.

fixes #5134  : Replaced OSError with ImportError if required extraction library is not installed.


Make the KeyHasher FIPS compliant

MD5 is not FIPS compliant thus I am proposing this minimal change to make datasets package FIPS compliant


Align task tags in dataset metadata

dataset contribution

Contribution to a dataset script

I removed all the invalid task_ids in datasts without namespace, based on the <s>(internal)</s> types.ts

I have opened PRs to fix the task_ids in all datasets within a namespace as well.

Working on task_categories...

For future reference: this fix had some complications

When trying to open a PR to fix the task tags, an exception was thrown if:
- the metadata contained "languages" or "licenses" (instead of "language" or "license")
- the metadata contained a non-valid language: `en-US` (instead of `en`), `no` (instead of `'no'`),...
- the metadata contained a non-valid license
- either `task_categories` or `task_ids` was not an array (a dict for each config)
- the metadata contained non-valid tag names

Errors:
```
ValueError: - Error: "languages" is deprecated. Use "language" instead.
```
```
ValueError: - Error: "licenses" is deprecated. Use "license" instead.
```
```
ValueError: - Error: "language[17]" must only contain lowercase characters
```
```
ValueError: - Error: "language[0]" with value "cz, de, it" is not valid. It must be an ISO 639-1, 639-2 or 639-3 code (two/three letters), or a special value like "code", "multilingual". If you want to use BCP-47 identifiers, you can specify them in language_bcp47.
```
```
ValueError: - Error: "task_ids" must be an array
```

great job! did you have feedback from Hub users/i.E. repo authors?

Yes, @julien-c. These are some of the feedbacks:
- Most people just thank for the fix: [cahya/librivox-indonesia](https://huggingface.co/datasets/cahya/librivox-indonesia/discussions/1#6357cd8a292a050ebd705f84), [TurkuNLP/xlsum-fi](https://huggingface.co/datasets/TurkuNLP/xlsum-fi/discussions/1#6357828aa1f8ad1c31bcbe46), [coastalcph/fairlex](https://huggingface.co/datasets/coastalcph/fairlex/discussions/4#6351a527a8e595171ab1aef2)
- Why are we changing their task names? [joelito/lextreme](https://huggingface.co/datasets/joelito/lextreme/discussions/1#6351b576fe367c0d9b12041b)
  - I take note of this for the next bulk operation; besides the PR title, we should also add a description to explain the reason for the change and also maybe putting a link to some pertinent GH Issue page
- Some of them ask where to find the list of the supported task values is: [dennlinger/klexikon](https://huggingface.co/datasets/dennlinger/klexikon/discussions/3#6356b3ea80f8cb3ab777ac5c), [lmqg/qg_jaquad](https://huggingface.co/datasets/lmqg/qg_jaquad/discussions/1#635262467e4cc3135fd09f58)
  - Currently, the list is here: https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Types.ts#L85
  - Maybe we could made them more easily accessible
- Some people do not agree about current "hierarchy":
  - text-scoring: [emrecan/nli_tr_for_simcse](https://huggingface.co/datasets/emrecan/nli_tr_for_simcse/discussions/1#6357c1b128792d8cdd51e9f9) (but referring to [emrecan/nli_tr_for_simcse](https://huggingface.co/datasets/emrecan/nli_tr_for_simcse/discussions/2/files))
    - Before "text-scoring" was a task_category, with task_ids ["semantic-similarity-scoring", "sentiment-scoring"]
    - Now all three are task_ids ["text-scoring", "semantic-similarity-scoring", "sentiment-scoring"] under the task_category "text-classification"
    - People complain that their scoring tasks are not classification task
  - binary-classification: why don't we have binary-classification? We have multi-class-classification, multi-label-classification and sentiment-classification, but not binary-classification
  - symbolic-regression: [yoshitomo-matsubara/srsd-feynman_hard](https://huggingface.co/datasets/yoshitomo-matsubara/srsd-feynman_hard/discussions/2#63614194c12a09b8a31457cc), [yoshitomo-matsubara/srsd-feynman_medium](https://huggingface.co/datasets/yoshitomo-matsubara/srsd-feynman_medium/discussions/2#6361418aeee0d27f04379e43), [yoshitomo-matsubara/srsd-feynman_easy](https://huggingface.co/datasets/yoshitomo-matsubara/srsd-feynman_easy/discussions/2#6361416e00905b1ffb8d0112)
    - Why don't we have symbolic-regression task?

NOTE: I'm editing this comment to add more feedback

As someone with feedback on the updates (which I highly appreciate seeing included here :D), a few comments from a "user perspective":

* I think the general confusion for me was also surrounding the hierarchy; it doesn't really become super clear (even when using the tagger space) that one is a subset of the other, especially since it seems to be still possible to include fine-grained tasks without the "parent category"?
* The datasets explorer still shows tags that are no longer valid (e.g., super specific ones such as `summarization-other-paper-abstract-generation`, but also ones that should be `task_categories`, such as `summarization`). I'm assuming this will be fixed soon, but until then it can confuse people who don't understand why they suddenly can't use seemingly still valid tags anymore.
* As I mentioned to @albertvillanova, having a dedicated page in the docs with explanations (especially wrt the difference between `task_categories` and `task_ids`) would be super helpful. However, I think it would have been sufficient to just include some description in the dataset PRs where you can link to the Github/other discussion on the topic :) That way, I can check myself what changes are expected to happen.

Thanks again for the streamlining process, I personally learned a fair bit about the tagging structure in the meantime!
Best,
Dennis

Thanks to you both for your feedback! super useful! cc'ing @osanseviero too 🙂

> The datasets explorer still shows tags that are no longer valid

wait which explorer is that? is it https://huggingface.co/datasets/viewer/ ?

Sorry, this one: https://huggingface.co/datasets
And then selecting the "Fine-Grained Tasks".

- Some people do not agree about current "hierarchy":
  - symbolic-regression: [yoshitomo-matsubara/srsd-feynman_hard](https://huggingface.co/datasets/yoshitomo-matsubara/srsd-feynman_hard/discussions/2#63614194c12a09b8a31457cc), [yoshitomo-matsubara/srsd-feynman_medium](https://huggingface.co/datasets/yoshitomo-matsubara/srsd-feynman_medium/discussions/2#6361418aeee0d27f04379e43), [yoshitomo-matsubara/srsd-feynman_easy](https://huggingface.co/datasets/yoshitomo-matsubara/srsd-feynman_easy/discussions/2#6361416e00905b1ffb8d0112)
    - Why don't we have symbolic-regression task?

@albertvillanova
Thank you for sharing our voice here!

Yes, we want `symbolic-regression` to be listed as a task. This task has been attracting attention from the machine learning/deep learning community, and unfortunately existing symbolic regression datasets are de-centralized in the community (hosted at individual platforms like author website, github, etc).
It would be great for the community if Hugging Face can support the task.

## Describe
Once we have agreed on a common naming for task tags for all open source projects, we should align on them.

## Steps
- [x] Align task tags in canonical datasets
  - [x] task_categories: 4 datasets
  - [x] task_ids (by @lhoestq)
- [x] Open PRs in community datasets
  - [x] task_categories: 451 datasets
  - [x] task_ids: 556 datasets


Update docs once dataset scripts transferred to the Hub

Todo:
- [x] Update docs:
  - [x] Datasets on GitHub (legacy)
  - [x] Load: offline
  - [x] About dataset load:
    - [x] Maintaining integrity
    - [x] Security
- [x] Update docstrings:
  - [x] Inspect:
    - [x] get_dataset_config_info
    - [x] get_dataset_split_names
  - [x] Load:
    - [x] dataset_module_factory
    - [x] load_dataset_builder
    - [x] load_dataset
- [x] Remove `ADD_NEW_DATASET.md`
- [x] Update `.github/ISSUE_TEMPLATE/config.yml`

Fix #5135.


## Describe the bug
As discussed in:
- https://github.com/huggingface/hub-docs/pull/423#pullrequestreview-1146083701

we should update our docs once dataset scripts have been transferred to the Hub (and removed from GitHub):
- #4974

Concretely:
- [x] Datasets on GitHub (legacy): https://huggingface.co/docs/datasets/main/en/share#datasets-on-github-legacy
- [x] ADD_NEW_DATASET: https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md
- ...

This PR complements the work of:
- #5067

This PR is a follow-up of PRs:
- #3777

CC: @julien-c


Raise ImportError instead of OSError if required extraction library is not installed

enhancement

New feature or request

good first issue

Good for newcomers

hey ,i would like to work on this issue . Please assign it to me.

hey @mariosasko , i made a pr for this issue. Could you please review it.
Also i found multiple `OSError` in `extract.py` file which i thought could be replaced too but wasn't sure about them.
Please do tell if that also needs to be done.

According to the official Python docs, `OSError` should be thrown in the following situations:
> This exception is raised when a system function returns a system-related error, including I/O failures such as “file not found” or “disk full” (not for illegal argument types or other incidental errors).

Hence, it makes more sense to raise `ImportError` instead of `OSError` when the required extraction/decompression library is not installed.


Tensor operation not functioning in dataset mapping

bug

Something isn't working

> Hi! The Torch ops in your snippet are not equivalent to the NumPy ones, hence the difference. You can get the same behavior by replacing the line `feature = torch.mean(feature, dim=1)` with `feature = feature.squeeze().mean(1)` .

Thank you.

## Describe the bug
I'm doing a torch.mean() operation in data preprocessing, and it's not working.

## Steps to reproduce the bug
```
from transformers import pipeline
import torch
import numpy as np
from datasets import load_dataset
device = 'cuda:0'
raw_dataset = load_dataset("glue", "sst2")
feature_extraction = pipeline('feature-extraction', 'bert-base-uncased', device=device)
def extracted_data(examples):
    # feature = torch.tensor(feature_extraction(examples['sentence'], batch_size=16), device=device)
    # feature = torch.mean(feature, dim=1)
    feature = np.asarray(feature_extraction(examples['sentence'], batch_size=16)).squeeze().mean(1)
    print(feature.shape)
    return {'feature': feature}

extracted_dataset = raw_dataset.map(extracted_data, batched=True, batch_size=16)
```

## Results
When running with torch.mean(), the shape printed out is [16, seq_len, 768], which is exactly the same before the operation. While numpy works just fine, which gives [16, 768].

## Environment info
- `datasets` version: 2.6.1
- Platform: Linux-4.4.0-142-generic-x86_64-with-glibc2.31
- Python version: 3.10.6
- PyArrow version: 9.0.0
- Pandas version: 1.5.0


Depracate `num_proc` parameter in `DownloadManager.extract`

enhancement

New feature or request

good first issue

Good for newcomers

@lazarust i'm already working on this issue :smile:

hey @mariosasko , i made a pr for this issue. Could you please review it.

The `num_proc` parameter is only present in `DownloadManager.extract` but not in `StreamingDownloadManager.extract`, making it impossible to support streaming in the dataset scripts that use it (`openwebtext` and `the_pile_stack_exchange`). We can avoid this situation by deprecating this parameter and passing `DownloadConfig`'s `num_proc` to `map_nested` instead, as it's done in `DownloadManager.download`.


WikiText 103 tokenizer hangs

bug

Something isn't working

any updates on this? It happens to me on [OpenWikiText-20%](https://huggingface.co/datasets/Bingsu/openwebtext_20p) dataset, but not on [OpenWebText-10k](https://huggingface.co/datasets/stas/openwebtext-10k). This is really strange because I don't change anything else in my running script.

transformers version 4.18.0.dev0
datasets version 1.18.0

See issue here: https://github.com/huggingface/transformers/issues/19702


Avoid extra cast in `class_encode_column`

_The documentation is not available anymore as the PR was closed or merged._

Pass the updated features to `map` to avoid the `cast` in `class_encode_column`.


unexpected `cast` or `class_encode_column` result after `rename_column`

Something isn't working

Hi, 方子东. I tried running the code with exact the same configuration (both datasets 2.5.2 and 2.6.1, python, pyarrow, pandas), but on Linux. The results seem to be the expected `{<pyarrow.Int64Scalar: 4>, <pyarrow.Int64Scalar: 2>, <pyarrow.Int64Scalar: 3>, <pyarrow.Int64Scalar: 0>, <pyarrow.Int64Scalar: 1>}`.
I don't have a Mac device. I can't verify whether this is a M1 chip-specific problem.

I've just tested the code on my M1 Mac, and it behaves as expected.

> Hi! Unfortunately, I can't reproduce this issue locally (in Python 3.7/3.10) or in Colab. I would assume this is due to a bug we fixed in the latest release, but your version is up-to-date, so I'm not sure if there is something we can do to help...

Thank you for your attention and feel sorry to take your time. Since this is a bug of old version, I think mybe my problem is because `cast` operation directaly used cached data generated by older verion of `datasets`. I tried to deleted the cached data and I got expected result.

## Describe the bug
When invoke `cast` or `class_encode_column` to a colunm renamed by `rename_column` , it will convert all the variables in this column into one variable. I also run this script in version 2.5.2, this bug does not appear. So I switched to the older version.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("amazon_reviews_multi", "en")
data = dataset['train']

data = data.remove_columns(
                [
                    "review_id",
                    "product_id",
                    "reviewer_id",
                    "review_title",
                    "language",
                    "product_category",
                ]
            )
data = data.rename_column("review_body", "text")
data1 = data.class_encode_column("stars")
print(set(data1.data.columns[0]))
# output: {<pyarrow.Int64Scalar: 4>, <pyarrow.Int64Scalar: 2>, <pyarrow.Int64Scalar: 3>, <pyarrow.Int64Scalar: 0>, <pyarrow.Int64Scalar: 1>}

data = data.rename_column("stars", "label")
print(set(data.data.columns[0]))
# output: {<pyarrow.Int32Scalar: 5>, <pyarrow.Int32Scalar: 4>, <pyarrow.Int32Scalar: 1>, <pyarrow.Int32Scalar: 3>, <pyarrow.Int32Scalar: 2>}
data2 = data.class_encode_column("label")
print(set(data2.data.columns[0]))
# output: {<pyarrow.Int64Scalar: 0>}
```

## Expected results
the last print should be:
{<pyarrow.Int64Scalar: 4>, <pyarrow.Int64Scalar: 2>, <pyarrow.Int64Scalar: 3>, <pyarrow.Int64Scalar: 0>, <pyarrow.Int64Scalar: 1>}

## Actual results
but it output:
{<pyarrow.Int64Scalar: 0>}

## Environment info
- `datasets` version: 2.6.1
- Platform: macOS-12.5.1-arm64-arm-64bit
- Python version: 3.10.6
- PyArrow version: 9.0.0
- Pandas version: 1.5.0


Make filename matching more robust

> I think we should also modify one of the metadata files in the `folder_based_builder` tests to make sure "./" is ignored now in the `file_name`

@mariosasko what do you mean here? I'm not sure which metadata file I should modify here

You can modify this line for instance: https://github.com/huggingface/datasets/blob/2699593b33ee63d17aad2a2bfddedd38a8df57b8/tests/packaged_modules/test_folder_based_builder.py#L135

Fix  #5046


[WIP] WebDataset export

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5127). All of your documentation changes will be reflected on that endpoint.

I added a first draft of the `IterableDataset.to_wds` method.

You can use it to savea dataset loaded in streamign mode as a webdataset locally.
The API can be further improved to allow to export to a cloud storage like the HF Hub.

I also included sharding with a default max shard size of 500MB (uncompressed), and it is single-processed fo rnow.

Choosing the number of shards is not implemented yet - though if we know the size of the `IterableDataset` this is probably doable`.

For example
```python
>>> from datasets import load_dataset
>>> ds = load_dataset("rotten_tomatoes", split="train", streaming=True)
>>> ds.to_wds("output_dir", compress=True)
>>> import webdataset as wds
>>> ds = wds.WebDataset("output_dir/rotten_tomatoes-train-000000.tar.gz").decode()
>>> next(iter(ds))
{'__key__': '0',
'__url__': 'output_dir/rotten_tomatoes-train-000000.tar.gz',
'label.cls': 1,
'text.txt': 'the rock is destined to be the 21st century\'s new ..., jean-claud van damme or steven segal .'}
```

### Implementation details

The WebDataset format is made of TAR archives containing a series of files per example. For example one pair of `image.jpg` and `label.cls` for image classification.

WebDataset automatically decodes serialized data based on the extension of the files, and output a dictionary. For example `{"image.png": np.array(...), "label.cls": 0}` if you choose the numpy decoding.

To use the automatic decoding, I store each field of each example as a file with its corresponding extension (jpg, json, cls, etc.)

While this is useful to end up with a dictionary with one key per column and appropriate decoding, it can create huge TAR archives if the dataset is made of small samples of text - probably because of useless TAR metadata for each file. This also makes loading super slow: iterating on SQuAD takes 50sec vs 7sec using `datasets` in streaming mode.

I haven't taken a look at alternatives for text datasets made out of small samples, but for image datasets this can already be used to run some benchmarks.


Fix class name of symbolic link

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5126). All of your documentation changes will be reflected on that endpoint.

I have removed the reference to the Issue in the PR title, so that we avoid to have both references (to the issue and to the PR) in the merge commit to the main branch.

Instead, it should be commented in the PR description, so that the PR is appropriately linked by GitHub to its corresponding Issue:

> Fix #5098.

@albertvillanova What should I test in your opinion? Also, where should I save the test file and how should I name it? Thanks for your support

The regression test to be implemented should test what your PR fixes: that is, that `_resolve_single_pattern_locally` function does not resolve any symbolic link when passed a directory that does contain any.

As you are testing a function in `data_files.py`, the corresponding test should be in `tests/test_data_files.py`.

You could name the test something lilke: `test_resolve_single_pattern_locally_does_not_resolve_symbolic_links`

You could take inspiration from other tests there in that file.

Fix #5098


Add `pyproject.toml` for `black`

Add `pyproject.toml` as a config file for the `black` tool to support VS Code's auto-formatting on save (and to be more consistent with the other HF projects).


Install tensorflow-macos dependency conditionally

Fix #5118.