# Kwargs handlers

The following objects can be passed to the main [Accelerator](/docs/accelerate/v1.11.0/en/package_reference/accelerator#accelerate.Accelerator) to customize how some PyTorch objects
related to distributed training or mixed precision are created.

## AutocastKwargs[[accelerate.AutocastKwargs]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class accelerate.AutocastKwargs</name><anchor>accelerate.AutocastKwargs</anchor><source>https://github.com/huggingface/accelerate/blob/v1.11.0/src/accelerate/utils/dataclasses.py#L112</source><parameters>[{"name": "enabled", "val": ": bool = True"}, {"name": "cache_enabled", "val": ": typing.Optional[bool] = None"}]</parameters></docstring>

Use this object in your [Accelerator](/docs/accelerate/v1.11.0/en/package_reference/accelerator#accelerate.Accelerator) to customize how `torch.autocast` behaves. Please refer to the
documentation of this [context manager](https://pytorch.org/docs/stable/amp.html#torch.autocast) for more
information on each argument.

<ExampleCodeBlock anchor="accelerate.AutocastKwargs.example">

Example:

```python
from accelerate import Accelerator
from accelerate.utils import AutocastKwargs

kwargs = AutocastKwargs(cache_enabled=True)
accelerator = Accelerator(kwargs_handlers=[kwargs])
```

</ExampleCodeBlock>


</div>

## DistributedDataParallelKwargs[[accelerate.DistributedDataParallelKwargs]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class accelerate.DistributedDataParallelKwargs</name><anchor>accelerate.DistributedDataParallelKwargs</anchor><source>https://github.com/huggingface/accelerate/blob/v1.11.0/src/accelerate/utils/dataclasses.py#L154</source><parameters>[{"name": "dim", "val": ": int = 0"}, {"name": "broadcast_buffers", "val": ": bool = True"}, {"name": "bucket_cap_mb", "val": ": int = 25"}, {"name": "find_unused_parameters", "val": ": bool = False"}, {"name": "check_reduction", "val": ": bool = False"}, {"name": "gradient_as_bucket_view", "val": ": bool = False"}, {"name": "static_graph", "val": ": bool = False"}, {"name": "comm_hook", "val": ": DDPCommunicationHookType = <DDPCommunicationHookType.NO: 'no'>"}, {"name": "comm_wrapper", "val": ": typing.Literal[<DDPCommunicationHookType.NO: 'no'>, <DDPCommunicationHookType.FP16: 'fp16'>, <DDPCommunicationHookType.BF16: 'bf16'>] = <DDPCommunicationHookType.NO: 'no'>"}, {"name": "comm_state_option", "val": ": dict = <factory>"}]</parameters></docstring>

Use this object in your [Accelerator](/docs/accelerate/v1.11.0/en/package_reference/accelerator#accelerate.Accelerator) to customize how your model is wrapped in a
`torch.nn.parallel.DistributedDataParallel`. Please refer to the documentation of this
[wrapper](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html) for more
information on each argument.

<Tip warning={true}>

`gradient_as_bucket_view` is only available in PyTorch 1.7.0 and later versions.

`static_graph` is only available in PyTorch 1.11.0 and later versions.

</Tip>

<ExampleCodeBlock anchor="accelerate.DistributedDataParallelKwargs.example">

Example:

```python
from accelerate import Accelerator
from accelerate.utils import DistributedDataParallelKwargs

kwargs = DistributedDataParallelKwargs(find_unused_parameters=True)
accelerator = Accelerator(kwargs_handlers=[kwargs])
```

</ExampleCodeBlock>


</div>

## FP8RecipeKwargs[[accelerate.utils.FP8RecipeKwargs]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class accelerate.utils.FP8RecipeKwargs</name><anchor>accelerate.utils.FP8RecipeKwargs</anchor><source>https://github.com/huggingface/accelerate/blob/v1.11.0/src/accelerate/utils/dataclasses.py#L425</source><parameters>[{"name": "opt_level", "val": ": typing.Literal['O1', 'O2'] = None"}, {"name": "use_autocast_during_eval", "val": ": typing.Optional[bool] = None"}, {"name": "margin", "val": ": typing.Optional[int] = None"}, {"name": "interval", "val": ": typing.Optional[int] = None"}, {"name": "fp8_format", "val": ": typing.Literal['HYBRID', 'E4M3', 'E5M2'] = None"}, {"name": "amax_history_len", "val": ": typing.Optional[int] = None"}, {"name": "amax_compute_algo", "val": ": typing.Literal['max', 'most_recent'] = None"}, {"name": "override_linear_precision", "val": ": tuple = None"}, {"name": "use_mxfp8_block_scaling", "val": ": typing.Optional[bool] = None"}, {"name": "backend", "val": ": typing.Literal['MSAMP', 'TE'] = None"}]</parameters></docstring>

Deprecated. Please use one of the proper FP8 recipe kwargs classes such as `TERecipeKwargs` or `MSAMPRecipeKwargs`
instead.


</div>

## ProfileKwargs[[accelerate.ProfileKwargs]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class accelerate.ProfileKwargs</name><anchor>accelerate.ProfileKwargs</anchor><source>https://github.com/huggingface/accelerate/blob/v1.11.0/src/accelerate/utils/dataclasses.py#L454</source><parameters>[{"name": "activities", "val": ": typing.Optional[list[typing.Literal['cpu', 'xpu', 'mtia', 'cuda', 'hpu']]] = None"}, {"name": "schedule_option", "val": ": typing.Optional[dict[str, int]] = None"}, {"name": "on_trace_ready", "val": ": typing.Optional[typing.Callable] = None"}, {"name": "record_shapes", "val": ": bool = False"}, {"name": "profile_memory", "val": ": bool = False"}, {"name": "with_stack", "val": ": bool = False"}, {"name": "with_flops", "val": ": bool = False"}, {"name": "with_modules", "val": ": bool = False"}, {"name": "output_trace_dir", "val": ": typing.Optional[str] = None"}]</parameters><paramsdesc>- **activities** (`List[str]`, *optional*, default to `None`) --
  The list of activity groups to use in profiling. Must be one of `"cpu"`, `"xpu"`, `"mtia"`, "hpu" or
  `"cuda"`.
- **schedule_option** (`Dict[str, int]`, *optional*, default to `None`) --
  The schedule option to use for the profiler. Available keys are `wait`, `warmup`, `active`, `repeat` and
  `skip_first`. The profiler will skip the first `skip_first` steps, then wait for `wait` steps, then do the
  warmup for the next `warmup` steps, then do the active recording for the next `active` steps and then
  repeat the cycle starting with `wait` steps. The optional number of cycles is specified with the `repeat`
  parameter, the zero value means that the cycles will continue until the profiling is finished.
- **on_trace_ready** (`Callable`, *optional*, default to `None`) --
  Callable that is called at each step when schedule returns `ProfilerAction.RECORD_AND_SAVE` during the
  profiling.
- **record_shapes** (`bool`, *optional*, default to `False`) --
  Save information about operator’s input shapes.
- **profile_memory** (`bool`, *optional*, default to `False`) --
  Track tensor memory allocation/deallocation
- **with_stack** (`bool`, *optional*, default to `False`) --
  Record source information (file and line number) for the ops.
- **with_flops** (`bool`, *optional*, default to `False`) --
  Use formula to estimate the FLOPS of specific operators
- **with_modules** (`bool`, *optional*, default to `False`) --
  Record module hierarchy (including function names) corresponding to the callstack of the op.
- **output_trace_dir** (`str`, *optional*, default to `None`) --
  Exports the collected trace in Chrome JSON format. Chrome use 'chrome://tracing' view json file. Defaults
  to None, which means profiling does not store json files.</paramsdesc><paramgroups>0</paramgroups></docstring>

Use this object in your [Accelerator](/docs/accelerate/v1.11.0/en/package_reference/accelerator#accelerate.Accelerator) to customize the initialization of the profiler. Please refer to the
documentation of this [context manager](https://pytorch.org/docs/stable/profiler.html#torch.profiler.profile) for
more information on each argument.

<Tip warning={true}>

`torch.profiler` is only available in PyTorch 1.8.1 and later versions.

</Tip>

<ExampleCodeBlock anchor="accelerate.ProfileKwargs.example">

Example:

```python
from accelerate import Accelerator
from accelerate.utils import ProfileKwargs

kwargs = ProfileKwargs(activities=["cpu", "cuda"])
accelerator = Accelerator(kwargs_handlers=[kwargs])
```

</ExampleCodeBlock>





<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>build</name><anchor>accelerate.ProfileKwargs.build</anchor><source>https://github.com/huggingface/accelerate/blob/v1.11.0/src/accelerate/utils/dataclasses.py#L544</source><parameters>[]</parameters><rettype>torch.profiler.profile</rettype><retdesc>The profiler object.</retdesc></docstring>

Build a profiler object with the current configuration.






</div></div>

## GradScalerKwargs[[accelerate.GradScalerKwargs]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class accelerate.GradScalerKwargs</name><anchor>accelerate.GradScalerKwargs</anchor><source>https://github.com/huggingface/accelerate/blob/v1.11.0/src/accelerate/utils/dataclasses.py#L240</source><parameters>[{"name": "init_scale", "val": ": float = 65536.0"}, {"name": "growth_factor", "val": ": float = 2.0"}, {"name": "backoff_factor", "val": ": float = 0.5"}, {"name": "growth_interval", "val": ": int = 2000"}, {"name": "enabled", "val": ": bool = True"}]</parameters></docstring>

Use this object in your [Accelerator](/docs/accelerate/v1.11.0/en/package_reference/accelerator#accelerate.Accelerator) to customize the behavior of mixed precision, specifically how the
`torch.amp.GradScaler` or `torch.cuda.amp.GradScaler` used is created. Please refer to the documentation of this
[scaler](https://pytorch.org/docs/stable/amp.html?highlight=gradscaler) for more information on each argument.

<Tip warning={true}>

`torch.cuda.amp.GradScaler` is only available in PyTorch 1.5.0 and later versions, and `torch.amp.GradScaler` is
only available in PyTorch 2.4.0 and later versions.

</Tip>

<ExampleCodeBlock anchor="accelerate.GradScalerKwargs.example">

Example:

```python
from accelerate import Accelerator
from accelerate.utils import GradScalerKwargs

kwargs = GradScalerKwargs(backoff_factor=0.25)
accelerator = Accelerator(kwargs_handlers=[kwargs])
```

</ExampleCodeBlock>


</div>

## InitProcessGroupKwargs[[accelerate.InitProcessGroupKwargs]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class accelerate.InitProcessGroupKwargs</name><anchor>accelerate.InitProcessGroupKwargs</anchor><source>https://github.com/huggingface/accelerate/blob/v1.11.0/src/accelerate/utils/dataclasses.py#L272</source><parameters>[{"name": "backend", "val": ": typing.Optional[str] = 'nccl'"}, {"name": "init_method", "val": ": typing.Optional[str] = None"}, {"name": "timeout", "val": ": typing.Optional[datetime.timedelta] = None"}]</parameters></docstring>

Use this object in your [Accelerator](/docs/accelerate/v1.11.0/en/package_reference/accelerator#accelerate.Accelerator) to customize the initialization of the distributed processes. Please refer
to the documentation of this
[method](https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group) for more
information on each argument.

Note: If `timeout` is set to `None`, the default will be based upon how `backend` is set.

<ExampleCodeBlock anchor="accelerate.InitProcessGroupKwargs.example">

```python
from datetime import timedelta
from accelerate import Accelerator
from accelerate.utils import InitProcessGroupKwargs

kwargs = InitProcessGroupKwargs(timeout=timedelta(seconds=800))
accelerator = Accelerator(kwargs_handlers=[kwargs])
```

</ExampleCodeBlock>


</div>

## KwargsHandler[[accelerate.utils.KwargsHandler]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class accelerate.utils.KwargsHandler</name><anchor>accelerate.utils.KwargsHandler</anchor><source>https://github.com/huggingface/accelerate/blob/v1.11.0/src/accelerate/utils/dataclasses.py#L67</source><parameters>[]</parameters></docstring>

Internal mixin that implements a `to_kwargs()` method for a dataclass.



<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>to_kwargs</name><anchor>accelerate.utils.KwargsHandler.to_kwargs</anchor><source>https://github.com/huggingface/accelerate/blob/v1.11.0/src/accelerate/utils/dataclasses.py#L75</source><parameters>[]</parameters></docstring>

Returns a dictionary containing the attributes with values different from the default of this class.


</div></div>

<EditOnGithub source="https://github.com/huggingface/accelerate/blob/main/docs/source/package_reference/kwargs.md" />