# Filesystem API

The `HfFileSystem` class provides a pythonic file interface to the Hugging Face Hub based on [`fsspec`](https://filesystem-spec.readthedocs.io/en/latest/).

## HfFileSystem[[huggingface_hub.HfFileSystem]]

`HfFileSystem` is based on [fsspec](https://filesystem-spec.readthedocs.io/en/latest/), so it is compatible with most of the APIs that it offers. For more details, check out [our guide](../guides/hf_file_system) and fsspec's [API Reference](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem).

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class huggingface_hub.HfFileSystem</name><anchor>huggingface_hub.HfFileSystem</anchor><source>https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/hf_file_system.py#L59</source><parameters>[{"name": "*args", "val": ""}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **token** (`str` or `bool`, *optional*) --
  A valid user access token (string). Defaults to the locally saved
  token, which is the recommended method for authentication (see
  https://huggingface.co/docs/huggingface_hub/quick-start#authentication).
  To disable authentication, pass `False`.
- **endpoint** (`str`, *optional*) --
  Endpoint of the Hub. Defaults to <https://huggingface.co>.</paramsdesc><paramgroups>0</paramgroups></docstring>

Access a remote Hugging Face Hub repository as if were a local file system.

> [!WARNING]
> [HfFileSystem](/docs/huggingface_hub/main/en/package_reference/hf_file_system#huggingface_hub.HfFileSystem) provides fsspec compatibility, which is useful for libraries that require it (e.g., reading
>     Hugging Face datasets directly with `pandas`). However, it introduces additional overhead due to this compatibility
>     layer. For better performance and reliability, it's recommended to use `HfApi` methods when possible.



<ExampleCodeBlock anchor="huggingface_hub.HfFileSystem.example">

Usage:

```python
>>> from huggingface_hub import HfFileSystem

>>> fs = HfFileSystem()

>>> # List files
>>> fs.glob("my-username/my-model/*.bin")
['my-username/my-model/pytorch_model.bin']
>>> fs.ls("datasets/my-username/my-dataset", detail=False)
['datasets/my-username/my-dataset/.gitattributes', 'datasets/my-username/my-dataset/README.md', 'datasets/my-username/my-dataset/data.json']

>>> # Read/write files
>>> with fs.open("my-username/my-model/pytorch_model.bin") as f:
...     data = f.read()
>>> with fs.open("my-username/my-model/pytorch_model.bin", "wb") as f:
...     f.write(data)
```

</ExampleCodeBlock>



<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>__init__</name><anchor>huggingface_hub.HfFileSystem.__init__</anchor><source>https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/hf_file_system.py#L100</source><parameters>[{"name": "*args", "val": ""}, {"name": "endpoint", "val": ": typing.Optional[str] = None"}, {"name": "token", "val": ": typing.Union[bool, str, NoneType] = None"}, {"name": "block_size", "val": ": typing.Optional[int] = None"}, {"name": "**storage_options", "val": ""}]</parameters></docstring>


</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>cp_file</name><anchor>huggingface_hub.HfFileSystem.cp_file</anchor><source>https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/hf_file_system.py#L594</source><parameters>[{"name": "path1", "val": ": str"}, {"name": "path2", "val": ": str"}, {"name": "revision", "val": ": typing.Optional[str] = None"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **path1** (`str`) --
  Source path to copy from.
- **path2** (`str`) --
  Destination path to copy to.
- **revision** (`str`, *optional*) --
  The git revision to copy from.</paramsdesc><paramgroups>0</paramgroups></docstring>

Copy a file within or between repositories.

> [!WARNING]
> Note: When possible, use `HfApi.upload_file()` for better performance.




</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>exists</name><anchor>huggingface_hub.HfFileSystem.exists</anchor><source>https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/hf_file_system.py#L764</source><parameters>[{"name": "path", "val": ""}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **path** (`str`) --
  Path to check.</paramsdesc><paramgroups>0</paramgroups><rettype>`bool`</rettype><retdesc>True if file exists, False otherwise.</retdesc></docstring>

Check if a file exists.

For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.exists).

> [!WARNING]
> Note: When possible, use `HfApi.file_exists()` for better performance.








</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>find</name><anchor>huggingface_hub.HfFileSystem.find</anchor><source>https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/hf_file_system.py#L531</source><parameters>[{"name": "path", "val": ": str"}, {"name": "maxdepth", "val": ": typing.Optional[int] = None"}, {"name": "withdirs", "val": ": bool = False"}, {"name": "detail", "val": ": bool = False"}, {"name": "refresh", "val": ": bool = False"}, {"name": "revision", "val": ": typing.Optional[str] = None"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **path** (`str`) --
  Root path to list files from.
- **maxdepth** (`int`, *optional*) --
  Maximum depth to descend into subdirectories.
- **withdirs** (`bool`, *optional*) --
  Include directory paths in the output. Defaults to False.
- **detail** (`bool`, *optional*) --
  If True, returns a dict mapping paths to file information. Defaults to False.
- **refresh** (`bool`, *optional*) --
  If True, bypass the cache and fetch the latest data. Defaults to False.
- **revision** (`str`, *optional*) --
  The git revision to list from.</paramsdesc><paramgroups>0</paramgroups><rettype>`Union[list[str], dict[str, dict[str, Any]]]`</rettype><retdesc>List of paths or dict of file information.</retdesc></docstring>

List all files below path.

For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.find).








</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>get_file</name><anchor>huggingface_hub.HfFileSystem.get_file</anchor><source>https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/hf_file_system.py#L848</source><parameters>[{"name": "rpath", "val": ""}, {"name": "lpath", "val": ""}, {"name": "callback", "val": " = <fsspec.callbacks.NoOpCallback object at 0x7fac60c9c760>"}, {"name": "outfile", "val": " = None"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **rpath** (`str`) --
  Remote path to download from.
- **lpath** (`str`) --
  Local path to download to.
- **callback** (`Callback`, *optional*) --
  Optional callback to track download progress. Defaults to no callback.
- **outfile** (`IO`, *optional*) --
  Optional file-like object to write to. If provided, `lpath` is ignored.</paramsdesc><paramgroups>0</paramgroups></docstring>

Copy single remote file to local.

> [!WARNING]
> Note: When possible, use `HfApi.hf_hub_download()` for better performance.




</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>glob</name><anchor>huggingface_hub.HfFileSystem.glob</anchor><source>https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/hf_file_system.py#L515</source><parameters>[{"name": "path", "val": ": str"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **path** (`str`) --
  Path pattern to match.</paramsdesc><paramgroups>0</paramgroups><rettype>`list[str]`</rettype><retdesc>List of paths matching the pattern.</retdesc></docstring>

Find files by glob-matching.

For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.glob).








</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>info</name><anchor>huggingface_hub.HfFileSystem.info</anchor><source>https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/hf_file_system.py#L666</source><parameters>[{"name": "path", "val": ": str"}, {"name": "refresh", "val": ": bool = False"}, {"name": "revision", "val": ": typing.Optional[str] = None"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **path** (`str`) --
  Path to get info for.
- **refresh** (`bool`, *optional*) --
  If True, bypass the cache and fetch the latest data. Defaults to False.
- **revision** (`str`, *optional*) --
  The git revision to get info from.</paramsdesc><paramgroups>0</paramgroups><rettype>`dict[str, Any]`</rettype><retdesc>Dictionary containing file information (type, size, commit info, etc.).</retdesc></docstring>

Get information about a file or directory.

For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.info).

> [!WARNING]
> Note: When possible, use `HfApi.get_paths_info()` or `HfApi.repo_info()`  for better performance.








</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>invalidate_cache</name><anchor>huggingface_hub.HfFileSystem.invalidate_cache</anchor><source>https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/hf_file_system.py#L236</source><parameters>[{"name": "path", "val": ": typing.Optional[str] = None"}]</parameters><paramsdesc>- **path** (`str`, *optional*) --
  Path to clear from cache. If not provided, clear the entire cache.</paramsdesc><paramgroups>0</paramgroups></docstring>

Clear the cache for a given path.

For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.invalidate_cache).




</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>isdir</name><anchor>huggingface_hub.HfFileSystem.isdir</anchor><source>https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/hf_file_system.py#L789</source><parameters>[{"name": "path", "val": ""}]</parameters><paramsdesc>- **path** (`str`) --
  Path to check.</paramsdesc><paramgroups>0</paramgroups><rettype>`bool`</rettype><retdesc>True if path is a directory, False otherwise.</retdesc></docstring>

Check if a path is a directory.

For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.isdir).








</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>isfile</name><anchor>huggingface_hub.HfFileSystem.isfile</anchor><source>https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/hf_file_system.py#L807</source><parameters>[{"name": "path", "val": ""}]</parameters><paramsdesc>- **path** (`str`) --
  Path to check.</paramsdesc><paramgroups>0</paramgroups><rettype>`bool`</rettype><retdesc>True if path is a file, False otherwise.</retdesc></docstring>

Check if a path is a file.

For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.isfile).








</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>ls</name><anchor>huggingface_hub.HfFileSystem.ls</anchor><source>https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/hf_file_system.py#L341</source><parameters>[{"name": "path", "val": ": str"}, {"name": "detail", "val": ": bool = True"}, {"name": "refresh", "val": ": bool = False"}, {"name": "revision", "val": ": typing.Optional[str] = None"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **path** (`str`) --
  Path to the directory.
- **detail** (`bool`, *optional*) --
  If True, returns a list of dictionaries containing file information. If False,
  returns a list of file paths. Defaults to True.
- **refresh** (`bool`, *optional*) --
  If True, bypass the cache and fetch the latest data. Defaults to False.
- **revision** (`str`, *optional*) --
  The git revision to list from.</paramsdesc><paramgroups>0</paramgroups><rettype>`list[Union[str, dict[str, Any]]]`</rettype><retdesc>List of file paths (if detail=False) or list of file information
dictionaries (if detail=True).</retdesc></docstring>

List the contents of a directory.

For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.ls).

> [!WARNING]
> Note: When possible, use `HfApi.list_repo_tree()` for better performance.








</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>modified</name><anchor>huggingface_hub.HfFileSystem.modified</anchor><source>https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/hf_file_system.py#L650</source><parameters>[{"name": "path", "val": ": str"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **path** (`str`) --
  Path to the file.</paramsdesc><paramgroups>0</paramgroups><rettype>`datetime`</rettype><retdesc>Last commit date of the file.</retdesc></docstring>

Get the last modified time of a file.

For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.modified).








</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>resolve_path</name><anchor>huggingface_hub.HfFileSystem.resolve_path</anchor><source>https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/hf_file_system.py#L141</source><parameters>[{"name": "path", "val": ": str"}, {"name": "revision", "val": ": typing.Optional[str] = None"}]</parameters><paramsdesc>- **path** (`str`) --
  Path to resolve.
- **revision** (`str`, *optional*) --
  The revision of the repo to resolve. Defaults to the revision specified in the path.</paramsdesc><paramgroups>0</paramgroups><rettype>`HfFileSystemResolvedPath`</rettype><retdesc>Resolved path information containing `repo_type`, `repo_id`, `revision` and `path_in_repo`.</retdesc><raises>- ``ValueError`` -- 
  If path contains conflicting revision information.
- ``NotImplementedError`` -- 
  If trying to list repositories.</raises><raisederrors>``ValueError`` or ``NotImplementedError``</raisederrors></docstring>

Resolve a Hugging Face file system path into its components.












</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>rm</name><anchor>huggingface_hub.HfFileSystem.rm</anchor><source>https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/hf_file_system.py#L295</source><parameters>[{"name": "path", "val": ": str"}, {"name": "recursive", "val": ": bool = False"}, {"name": "maxdepth", "val": ": typing.Optional[int] = None"}, {"name": "revision", "val": ": typing.Optional[str] = None"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **path** (`str`) --
  Path to delete.
- **recursive** (`bool`, *optional*) --
  If True, delete directory and all its contents. Defaults to False.
- **maxdepth** (`int`, *optional*) --
  Maximum number of subdirectories to visit when deleting recursively.
- **revision** (`str`, *optional*) --
  The git revision to delete from.</paramsdesc><paramgroups>0</paramgroups></docstring>

Delete files from a repository.

For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.rm).

> [!WARNING]
> Note: When possible, use `HfApi.delete_file()` for better performance.




</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>url</name><anchor>huggingface_hub.HfFileSystem.url</anchor><source>https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/hf_file_system.py#L825</source><parameters>[{"name": "path", "val": ": str"}]</parameters><paramsdesc>- **path** (`str`) --
  Path to get URL for.</paramsdesc><paramgroups>0</paramgroups><rettype>`str`</rettype><retdesc>HTTP URL to access the file or directory on the Hub.</retdesc></docstring>

Get the HTTP URL of the given path.








</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>walk</name><anchor>huggingface_hub.HfFileSystem.walk</anchor><source>https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/hf_file_system.py#L499</source><parameters>[{"name": "path", "val": ": str"}, {"name": "*args", "val": ""}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **path** (`str`) --
  Root path to list files from.</paramsdesc><paramgroups>0</paramgroups><rettype>`Iterator[tuple[str, list[str], list[str]]]`</rettype><retdesc>An iterator of (path, list of directory names, list of file names) tuples.</retdesc></docstring>

Return all files below the given path.

For more details, refer to [fsspec documentation](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.walk).








</div></div>

<EditOnGithub source="https://github.com/huggingface/huggingface_hub/blob/main/docs/source/en/package_reference/hf_file_system.md" />