# Optimization

This page contains the API reference documentation for learning rate optimizers included in `timm`.

## Optimizers

### Factory functions[[timm.optim.create_optimizer_v2]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>timm.optim.create_optimizer_v2</name><anchor>timm.optim.create_optimizer_v2</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/_optim_factory.py#L1140</source><parameters>[{"name": "model_or_params", "val": ": typing.Union[torch.nn.modules.module.Module, collections.abc.Iterable[torch.Tensor], collections.abc.Iterable[dict[str, typing.Any]], collections.abc.Iterable[tuple[str, torch.Tensor]]]"}, {"name": "opt", "val": ": str = 'sgd'"}, {"name": "lr", "val": ": typing.Optional[float] = None"}, {"name": "weight_decay", "val": ": float = 0.0"}, {"name": "momentum", "val": ": float = 0.9"}, {"name": "foreach", "val": ": typing.Optional[bool] = None"}, {"name": "filter_bias_and_bn", "val": ": bool = True"}, {"name": "layer_decay", "val": ": typing.Optional[float] = None"}, {"name": "layer_decay_min_scale", "val": ": float = 0.0"}, {"name": "layer_decay_no_opt_scale", "val": ": typing.Optional[float] = None"}, {"name": "param_group_fn", "val": ": typing.Optional[typing.Callable[[torch.nn.modules.module.Module], typing.Union[collections.abc.Iterable[torch.Tensor], collections.abc.Iterable[dict[str, typing.Any]], collections.abc.Iterable[tuple[str, torch.Tensor]]]]] = None"}, {"name": "**kwargs", "val": ": typing.Any"}]</parameters><paramsdesc>- **model_or_params** -- A PyTorch model or an iterable of parameters/parameter groups.
  If a model is provided, parameters will be automatically extracted and grouped
  based on the other arguments.
- **opt** -- Name of the optimizer to create (e.g., 'adam', 'adamw', 'sgd').
  Use list_optimizers() to see available options.
- **lr** -- Learning rate. If None, will use the optimizer's default.
- **weight_decay** -- Weight decay factor. Will be used to create param groups if model_or_params is a model.
- **momentum** -- Momentum factor for optimizers that support it. Only used if the
  chosen optimizer accepts a momentum parameter.
- **foreach** -- Enable/disable foreach (multi-tensor) implementation if available.
  If None, will use optimizer-specific defaults.
- **filter_bias_and_bn** -- If True, bias, norm layer parameters (all 1d params) will not have
  weight decay applied. Only used when model_or_params is a model and
  weight_decay > 0.
- **layer_decay** -- Optional layer-wise learning rate decay factor. If provided,
  learning rates will be scaled by layer_decay^(max_depth - layer_depth).
  Only used when model_or_params is a model.
- **param_group_fn** -- Optional function to create custom parameter groups.
  If provided, other parameter grouping options will be ignored.
- ****kwargs** -- Additional optimizer-specific arguments (e.g., betas for Adam).</paramsdesc><paramgroups>0</paramgroups><retdesc>Configured optimizer instance.</retdesc></docstring>
Create an optimizer instance via timm registry.

Creates and configures an optimizer with appropriate parameter groups and settings.
Supports automatic parameter group creation for weight decay and layer-wise learning
rates, as well as custom parameter grouping.





Examples:
>>> # Basic usage with a model
>>> optimizer = create_optimizer_v2(model, 'adamw', lr=1e-3)

>>> # SGD with momentum and weight decay
>>> optimizer = create_optimizer_v2(
...     model, 'sgd', lr=0.1, momentum=0.9, weight_decay=1e-4
... )

>>> # Adam with layer-wise learning rate decay
>>> optimizer = create_optimizer_v2(
...     model, 'adam', lr=1e-3, layer_decay=0.7
... )

>>> # Custom parameter groups
>>> def group_fn(model):
...     return [
...         {'params': model.backbone.parameters(), 'lr': 1e-4},
...         {'params': model.head.parameters(), 'lr': 1e-3}
...     ]
>>> optimizer = create_optimizer_v2(
...     model, 'sgd', param_group_fn=group_fn
... )

Note:
Parameter group handling precedence:
1. If param_group_fn is provided, it will be used exclusively
2. If layer_decay is provided, layer-wise groups will be created
3. If weight_decay > 0 and filter_bias_and_bn is True, weight decay groups will be created
4. Otherwise, all parameters will be in a single group


</div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>timm.optim.list_optimizers</name><anchor>timm.optim.list_optimizers</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/_optim_factory.py#L1043</source><parameters>[{"name": "filter", "val": ": typing.Union[str, typing.List[str]] = ''"}, {"name": "exclude_filters", "val": ": typing.Optional[typing.List[str]] = None"}, {"name": "with_description", "val": ": bool = False"}]</parameters><paramsdesc>- **filter** -- Wildcard style filter string or list of filter strings
  (e.g., 'adam*' for all Adam variants, or ['adam*', '*8bit'] for
  Adam variants and 8-bit optimizers). Empty string means no filtering.
- **exclude_filters** -- Optional list of wildcard patterns to exclude. For example,
  ['*8bit', 'fused*'] would exclude 8-bit and fused implementations.
- **with_description** -- If True, returns tuples of (name, description) instead of
  just names. Descriptions provide brief explanations of optimizer characteristics.</paramsdesc><paramgroups>0</paramgroups><rettype>If with_description is False</rettype><retdesc>List of optimizer names as strings (e.g., ['adam', 'adamw', ...])
If with_description is True:
List of tuples of (name, description) (e.g., [('adam', 'Adaptive Moment...'), ...])</retdesc></docstring>
List available optimizer names, optionally filtered.

List all registered optimizers, with optional filtering using wildcard patterns.
Optimizers can be filtered using include and exclude patterns, and can optionally
return descriptions with each optimizer name.







Examples:
>>> list_optimizers()
['adam', 'adamw', 'sgd', ...]

>>> list_optimizers(['la*', 'nla*'])  # List lamb & lars
['lamb', 'lambc', 'larc', 'lars', 'nlarc', 'nlars']

>>> list_optimizers('*adam*', exclude_filters=['bnb*', 'fused*'])  # Exclude bnb & apex adam optimizers
['adam', 'adamax', 'adamp', 'adamw', 'nadam', 'nadamw', 'radam']

>>> list_optimizers(with_description=True)  # Get descriptions
[('adabelief', 'Adapts learning rate based on gradient prediction error'),
('adadelta', 'torch.optim Adadelta, Adapts learning rates based on running windows of gradients'),
('adafactor', 'Memory-efficient implementation of Adam with factored gradients'),
...]


</div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>timm.optim.get_optimizer_class</name><anchor>timm.optim.get_optimizer_class</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/_optim_factory.py#L1103</source><parameters>[{"name": "name", "val": ": str"}, {"name": "bind_defaults", "val": ": bool = True"}]</parameters><paramsdesc>- **name** -- Name of the optimizer to retrieve (e.g., 'adam', 'sgd')
- **bind_defaults** -- If True, returns a partial function with default arguments from OptimInfo bound.
  If False, returns the raw optimizer class.</paramsdesc><paramgroups>0</paramgroups><rettype>If bind_defaults is False</rettype><retdesc>The optimizer class (e.g., torch.optim.Adam)
If bind_defaults is True:
A partial function with default arguments bound</retdesc><raises>- ``ValueError`` -- If optimizer name is not found in registry</raises><raisederrors>``ValueError``</raisederrors></docstring>
Get optimizer class by name with option to bind default arguments.

Retrieves the optimizer class or a partial function with default arguments bound.
This allows direct instantiation of optimizers with their default configurations
without going through the full factory.











Examples:
>>> # Get SGD with nesterov momentum default
>>> SGD = get_optimizer_class('sgd')  # nesterov=True bound
>>> opt = SGD(model.parameters(), lr=0.1, momentum=0.9)

>>> # Get raw optimizer class
>>> SGD = get_optimizer_class('sgd')
>>> opt = SGD(model.parameters(), lr=1e-3, momentum=0.9)



</div>

### Optimizer Classes[[timm.optim.AdaBelief]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.optim.AdaBelief</name><anchor>timm.optim.AdaBelief</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/adabelief.py#L6</source><parameters>[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.001"}, {"name": "betas", "val": " = (0.9, 0.999)"}, {"name": "eps", "val": " = 1e-16"}, {"name": "weight_decay", "val": " = 0"}, {"name": "amsgrad", "val": " = False"}, {"name": "decoupled_decay", "val": " = True"}, {"name": "fixed_decay", "val": " = False"}, {"name": "rectify", "val": " = True"}, {"name": "degenerated_to_sgd", "val": " = True"}]</parameters><paramsdesc>- **params** (iterable) -- iterable of parameters to optimize or dicts defining
  parameter groups
- **lr** (float, optional) -- learning rate (default: 1e-3)
- **betas** (Tuple[float, float], optional) -- coefficients used for computing
  running averages of gradient and its square (default: (0.9, 0.999))
- **eps** (float, optional) -- term added to the denominator to improve
  numerical stability (default: 1e-16)
- **weight_decay** (float, optional) -- weight decay (L2 penalty) (default: 0)
- **amsgrad** (boolean, optional) -- whether to use the AMSGrad variant of this
  algorithm from the paper `On the Convergence of Adam and Beyond`_
  (default: False)
- **decoupled_decay** (boolean, optional) -- (default: True) If set as True, then
  the optimizer uses decoupled weight decay as in AdamW
- **fixed_decay** (boolean, optional) -- (default: False) This is used when weight_decouple
  is set as True.
  When fixed_decay == True, the weight decay is performed as
  $W_{new} = W_{old} - W_{old} \times decay$.
  When fixed_decay == False, the weight decay is performed as
  $W_{new} = W_{old} - W_{old} \times decay \times lr$. Note that in this case, the
  weight decay ratio decreases with learning rate (lr).
- **rectify** (boolean, optional) -- (default: True) If set as True, then perform the rectified
  update similar to RAdam
- **degenerated_to_sgd** (boolean, optional) (default --True) If set as True, then perform SGD update
  when variance of gradient is high</paramsdesc><paramgroups>0</paramgroups></docstring>
Implements AdaBelief algorithm. Modified from Adam in PyTorch



reference: AdaBelief Optimizer, adapting stepsizes by the belief in observed gradients, NeurIPS 2020

For a complete table of recommended hyperparameters, see https://github.com/juntang-zhuang/Adabelief-Optimizer'
For example train/args for EfficientNet see these gists
- link to train_script: https://gist.github.com/juntang-zhuang/0a501dd51c02278d952cf159bc233037
- link to args.yaml: https://gist.github.com/juntang-zhuang/517ce3c27022b908bb93f78e4f786dc3



<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>step</name><anchor>timm.optim.AdaBelief.step</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/adabelief.py#L106</source><parameters>[{"name": "closure", "val": " = None"}]</parameters><paramsdesc>- **closure** (callable, optional) -- A closure that reevaluates the model
  and returns the loss.</paramsdesc><paramgroups>0</paramgroups></docstring>
Performs a single optimization step.



</div></div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.optim.Adafactor</name><anchor>timm.optim.Adafactor</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/adafactor.py#L21</source><parameters>[{"name": "params", "val": ": typing.Union[collections.abc.Iterable[torch.Tensor], collections.abc.Iterable[dict[str, typing.Any]], collections.abc.Iterable[tuple[str, torch.Tensor]]]"}, {"name": "lr", "val": ": typing.Optional[float] = None"}, {"name": "eps", "val": ": float = 1e-30"}, {"name": "eps_scale", "val": ": float = 0.001"}, {"name": "clip_threshold", "val": ": float = 1.0"}, {"name": "decay_rate", "val": ": float = -0.8"}, {"name": "betas", "val": ": typing.Optional[typing.Tuple[float, float]] = None"}, {"name": "weight_decay", "val": ": float = 0.0"}, {"name": "scale_parameter", "val": ": bool = True"}, {"name": "warmup_init", "val": ": bool = False"}, {"name": "min_dim_size_to_factor", "val": ": int = 16"}, {"name": "caution", "val": ": bool = False"}]</parameters></docstring>
Implements Adafactor algorithm.

This implementation is based on: `Adafactor: Adaptive Learning Rates with Sublinear Memory Cost`
(see https://arxiv.org/abs/1804.04235)

Note that this optimizer internally adjusts the learning rate depending on the
*scale_parameter*, *relative_step* and *warmup_init* options.

To use a manual (external) learning rate schedule you should set `scale_parameter=False` and
`relative_step=False`.

Ags:
params: iterable of parameters to optimize or dicts defining parameter groups
lr: external learning rate
eps: regularization constants for square gradient and parameter scale respectively
eps_scale: regularization constants for parameter scale respectively
clip_threshold: threshold of root-mean-square of final gradient update
decay_rate: coefficient used to compute running averages of square gradient
beta1: coefficient used for computing running averages of gradient
weight_decay: weight decay
scale_parameter: if True, learning rate is scaled by root-mean-square of parameter
warmup_init: time-dependent learning rate computation depends on whether warm-up initialization is being used



<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>step</name><anchor>timm.optim.Adafactor.step</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/adafactor.py#L127</source><parameters>[{"name": "closure", "val": " = None"}]</parameters><paramsdesc>- **closure** (callable, optional) -- A closure that reevaluates the model and returns the loss.</paramsdesc><paramgroups>0</paramgroups></docstring>
Performs a single optimization step.



</div></div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.optim.AdafactorBigVision</name><anchor>timm.optim.AdafactorBigVision</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/adafactor_bv.py#L53</source><parameters>[{"name": "params", "val": ": typing.Union[collections.abc.Iterable[torch.Tensor], collections.abc.Iterable[dict[str, typing.Any]], collections.abc.Iterable[tuple[str, torch.Tensor]]]"}, {"name": "lr", "val": ": float = 1.0"}, {"name": "min_dim_size_to_factor", "val": ": int = 16"}, {"name": "decay_rate", "val": ": float = 0.8"}, {"name": "decay_offset", "val": ": int = 0"}, {"name": "beta2_cap", "val": ": float = 0.999"}, {"name": "momentum", "val": ": typing.Optional[float] = 0.9"}, {"name": "momentum_dtype", "val": ": typing.Union[str, torch.dtype] = torch.bfloat16"}, {"name": "eps", "val": ": typing.Optional[float] = None"}, {"name": "weight_decay", "val": ": float = 0.0"}, {"name": "clipping_threshold", "val": ": typing.Optional[float] = None"}, {"name": "unscaled_wd", "val": ": bool = False"}, {"name": "caution", "val": ": bool = False"}, {"name": "corrected_weight_decay", "val": ": bool = False"}, {"name": "foreach", "val": ": typing.Optional[bool] = False"}]</parameters></docstring>

PyTorch implementation of BigVision's Adafactor variant with both single and multi tensor implementations.

Adapted from https://github.com/google-research/big_vision by Ross Wightman


</div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.optim.Adahessian</name><anchor>timm.optim.Adahessian</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/adahessian.py#L9</source><parameters>[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.1"}, {"name": "betas", "val": " = (0.9, 0.999)"}, {"name": "eps", "val": " = 1e-08"}, {"name": "weight_decay", "val": " = 0.0"}, {"name": "hessian_power", "val": " = 1.0"}, {"name": "update_each", "val": " = 1"}, {"name": "n_samples", "val": " = 1"}, {"name": "avg_conv_kernel", "val": " = False"}]</parameters><paramsdesc>- **params** (iterable) -- iterable of parameters to optimize or dicts defining parameter groups
- **lr** (float, optional) -- learning rate (default: 0.1)
- **betas** ((float, float), optional) -- coefficients used for computing running averages of gradient and the
  squared hessian trace (default: (0.9, 0.999))
- **eps** (float, optional) -- term added to the denominator to improve numerical stability (default: 1e-8)
- **weight_decay** (float, optional) -- weight decay (L2 penalty) (default: 0.0)
- **hessian_power** (float, optional) -- exponent of the hessian trace (default: 1.0)
- **update_each** (int, optional) -- compute the hessian trace approximation only after *this* number of steps
  (to save time) (default: 1)
- **n_samples** (int, optional) -- how many times to sample `z` for the approximation of the hessian trace (default: 1)</paramsdesc><paramgroups>0</paramgroups></docstring>

Implements the AdaHessian algorithm from "ADAHESSIAN: An Adaptive Second OrderOptimizer for Machine Learning"





<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>get_params</name><anchor>timm.optim.Adahessian.get_params</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/adahessian.py#L74</source><parameters>[]</parameters></docstring>

Gets all parameters in all param_groups with gradients


</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>set_hessian</name><anchor>timm.optim.Adahessian.set_hessian</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/adahessian.py#L90</source><parameters>[]</parameters></docstring>

Computes the Hutchinson approximation of the hessian trace and accumulates it for each trainable parameter.


</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>step</name><anchor>timm.optim.Adahessian.step</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/adahessian.py#L118</source><parameters>[{"name": "closure", "val": " = None"}]</parameters><paramsdesc>- **closure** (callable, optional) -- a closure that reevaluates the model and returns the loss (default -- None)</paramsdesc><paramgroups>0</paramgroups></docstring>

Performs a single optimization step.



</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>zero_hessian</name><anchor>timm.optim.Adahessian.zero_hessian</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/adahessian.py#L81</source><parameters>[]</parameters></docstring>

Zeros out the accumulated hessian traces.


</div></div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.optim.AdamP</name><anchor>timm.optim.AdamP</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/adamp.py#L43</source><parameters>[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.001"}, {"name": "betas", "val": " = (0.9, 0.999)"}, {"name": "eps", "val": " = 1e-08"}, {"name": "weight_decay", "val": " = 0"}, {"name": "delta", "val": " = 0.1"}, {"name": "wd_ratio", "val": " = 0.1"}, {"name": "nesterov", "val": " = False"}]</parameters></docstring>


</div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.optim.Adan</name><anchor>timm.optim.Adan</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/adan.py#L46</source><parameters>[{"name": "params", "val": ""}, {"name": "lr", "val": ": float = 0.001"}, {"name": "betas", "val": ": typing.Tuple[float, float, float] = (0.98, 0.92, 0.99)"}, {"name": "eps", "val": ": float = 1e-08"}, {"name": "weight_decay", "val": ": float = 0.0"}, {"name": "no_prox", "val": ": bool = False"}, {"name": "caution", "val": ": bool = False"}, {"name": "foreach", "val": ": typing.Optional[bool] = None"}]</parameters><paramsdesc>- **params** -- Iterable of parameters to optimize or dicts defining parameter groups.
- **lr** -- Learning rate.
- **betas** -- Coefficients used for first- and second-order moments.
- **eps** -- Term added to the denominator to improve numerical stability.
- **weight_decay** -- Decoupled weight decay (L2 penalty)
- **no_prox** -- How to perform the weight decay
- **caution** -- Enable caution from 'Cautious Optimizers'
- **foreach** -- If True would use torch._foreach implementation. Faster but uses slightly more memory.</paramsdesc><paramgroups>0</paramgroups></docstring>
Implements a pytorch variant of Adan.

Adan was proposed in Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models
https://arxiv.org/abs/2208.06677





<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>step</name><anchor>timm.optim.Adan.step</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/adan.py#L117</source><parameters>[{"name": "closure", "val": " = None"}]</parameters></docstring>
Performs a single optimization step.

</div></div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.optim.Adopt</name><anchor>timm.optim.Adopt</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/adopt.py#L59</source><parameters>[{"name": "params", "val": ": typing.Union[collections.abc.Iterable[torch.Tensor], collections.abc.Iterable[dict[str, typing.Any]], collections.abc.Iterable[tuple[str, torch.Tensor]]]"}, {"name": "lr", "val": ": typing.Union[float, torch.Tensor] = 0.001"}, {"name": "betas", "val": ": typing.Tuple[float, float] = (0.9, 0.9999)"}, {"name": "eps", "val": ": float = 1e-06"}, {"name": "clip_exp", "val": ": typing.Optional[float] = 0.333"}, {"name": "weight_decay", "val": ": float = 0.0"}, {"name": "decoupled", "val": ": bool = False"}, {"name": "corrected_weight_decay", "val": ": bool = False"}, {"name": "caution", "val": ": bool = False"}, {"name": "foreach", "val": ": typing.Optional[bool] = False"}, {"name": "maximize", "val": ": bool = False"}, {"name": "capturable", "val": ": bool = False"}, {"name": "differentiable", "val": ": bool = False"}]</parameters></docstring>

ADOPT: Modified Adam Can Converge with Any β2 with the Optimal Rate: https://arxiv.org/abs/2411.02853




<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>step</name><anchor>timm.optim.Adopt.step</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/adopt.py#L188</source><parameters>[{"name": "closure", "val": " = None"}]</parameters><paramsdesc>- **closure** (Callable, optional) -- A closure that reevaluates the model
  and returns the loss.</paramsdesc><paramgroups>0</paramgroups></docstring>
Perform a single optimization step.




</div></div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.optim.Lamb</name><anchor>timm.optim.Lamb</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/lamb.py#L67</source><parameters>[{"name": "params", "val": ": typing.Union[collections.abc.Iterable[torch.Tensor], collections.abc.Iterable[dict[str, typing.Any]], collections.abc.Iterable[tuple[str, torch.Tensor]]]"}, {"name": "lr", "val": ": float = 0.001"}, {"name": "bias_correction", "val": ": bool = True"}, {"name": "betas", "val": ": typing.Tuple[float, float] = (0.9, 0.999)"}, {"name": "eps", "val": ": float = 1e-06"}, {"name": "weight_decay", "val": ": float = 0.01"}, {"name": "grad_averaging", "val": ": bool = True"}, {"name": "max_grad_norm", "val": ": typing.Optional[float] = 1.0"}, {"name": "trust_clip", "val": ": bool = False"}, {"name": "always_adapt", "val": ": bool = False"}, {"name": "caution", "val": ": bool = False"}, {"name": "decoupled_decay", "val": ": bool = False"}, {"name": "corrected_weight_decay", "val": ": bool = False"}]</parameters><paramsdesc>- **params** -- Iterable of parameters to optimize or dicts defining parameter groups.
- **lr** -- Learning rate
- **betas** -- Coefficients used for computing running averages of gradient and its norm.
- **eps** -- Term added to the denominator to improve numerical stability.
- **weight_decay** -- Weight decay
- **grad_averaging** -- Whether apply (1-beta2) to grad when calculating running averages of gradient.
- **max_grad_norm** -- Value used to clip global grad norm.
- **trust_clip** -- Enable LAMBC trust ratio clipping.
- **always_adapt** -- Apply adaptive learning rate to 0.0 weight decay parameter.
- **caution** -- Apply caution.
- **decoupled** -- apply decoupled weight decay
- **corrected_weight_decay** -- apply corrected weight decay (lr**2 / max_lr) when using decoupled_decay</paramsdesc><paramgroups>0</paramgroups></docstring>
Implements a pure pytorch variant of FuseLAMB (NvLamb variant) optimizer from apex.optimizers.FusedLAMB
reference: https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/LanguageModeling/Transformer-XL/pytorch/lamb.py

LAMB was proposed in:
- Large Batch Optimization for Deep Learning - Training BERT in 76 minutes:  https://arxiv.org/abs/1904.00962
- On the Convergence of Adam and Beyond: https://openreview.net/forum?id=ryQu7f-RZ





<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>step</name><anchor>timm.optim.Lamb.step</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/lamb.py#L147</source><parameters>[{"name": "closure", "val": " = None"}]</parameters><paramsdesc>- **closure** (callable, optional) -- A closure that reevaluates the model
  and returns the loss.</paramsdesc><paramgroups>0</paramgroups></docstring>
Performs a single optimization step.



</div></div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.optim.LaProp</name><anchor>timm.optim.LaProp</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/laprop.py#L27</source><parameters>[{"name": "params", "val": ": typing.Union[collections.abc.Iterable[torch.Tensor], collections.abc.Iterable[dict[str, typing.Any]], collections.abc.Iterable[tuple[str, torch.Tensor]]]"}, {"name": "lr", "val": ": float = 0.0004"}, {"name": "betas", "val": ": typing.Tuple[float, float] = (0.9, 0.999)"}, {"name": "eps", "val": ": float = 1e-15"}, {"name": "weight_decay", "val": ": float = 0.0"}, {"name": "caution", "val": ": bool = False"}, {"name": "corrected_weight_decay", "val": ": bool = False"}]</parameters></docstring>
LaProp Optimizer

Paper: LaProp: Separating Momentum and Adaptivity in Adam, https://arxiv.org/abs/2002.04839



<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>step</name><anchor>timm.optim.LaProp.step</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/laprop.py#L66</source><parameters>[{"name": "closure", "val": " = None"}]</parameters><paramsdesc>- **closure** (callable, optional) -- A closure that reevaluates the model
  and returns the loss.</paramsdesc><paramgroups>0</paramgroups></docstring>
Performs a single optimization step.




</div></div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.optim.Lars</name><anchor>timm.optim.Lars</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/lars.py#L17</source><parameters>[{"name": "params", "val": ""}, {"name": "lr", "val": " = 1.0"}, {"name": "momentum", "val": " = 0"}, {"name": "dampening", "val": " = 0"}, {"name": "weight_decay", "val": " = 0"}, {"name": "nesterov", "val": " = False"}, {"name": "trust_coeff", "val": " = 0.001"}, {"name": "eps", "val": " = 1e-08"}, {"name": "trust_clip", "val": " = False"}, {"name": "always_adapt", "val": " = False"}]</parameters><paramsdesc>- **params** (iterable) -- iterable of parameters to optimize or dicts defining parameter groups.
- **lr** (float, optional) -- learning rate (default: 1.0).
- **momentum** (float, optional) -- momentum factor (default: 0)
- **weight_decay** (float, optional) -- weight decay (L2 penalty) (default: 0)
- **dampening** (float, optional) -- dampening for momentum (default: 0)
- **nesterov** (bool, optional) -- enables Nesterov momentum (default: False)
- **trust_coeff** (float) -- trust coefficient for computing adaptive lr / trust_ratio (default: 0.001)
- **eps** (float) -- eps for division denominator (default: 1e-8)
- **trust_clip** (bool) -- enable LARC trust ratio clipping (default: False)
- **always_adapt** (bool) -- always apply LARS LR adapt, otherwise only when group weight_decay != 0 (default: False)</paramsdesc><paramgroups>0</paramgroups></docstring>
LARS for PyTorch

Paper: `Large batch training of Convolutional Networks` - https://arxiv.org/pdf/1708.03888.pdf





<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>step</name><anchor>timm.optim.Lars.step</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/lars.py#L75</source><parameters>[{"name": "closure", "val": " = None"}]</parameters><paramsdesc>- **closure** (callable, optional) -- A closure that reevaluates the model and returns the loss.</paramsdesc><paramgroups>0</paramgroups></docstring>
Performs a single optimization step.




</div></div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.optim.Lion</name><anchor>timm.optim.Lion</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/lion.py#L31</source><parameters>[{"name": "params", "val": ": typing.Union[collections.abc.Iterable[torch.Tensor], collections.abc.Iterable[dict[str, typing.Any]], collections.abc.Iterable[tuple[str, torch.Tensor]]]"}, {"name": "lr", "val": ": float = 0.0001"}, {"name": "betas", "val": ": typing.Tuple[float, float] = (0.9, 0.99)"}, {"name": "weight_decay", "val": ": float = 0.0"}, {"name": "caution", "val": ": bool = False"}, {"name": "corrected_weight_decay", "val": ": bool = False"}, {"name": "maximize", "val": ": bool = False"}, {"name": "foreach", "val": ": typing.Optional[bool] = None"}]</parameters></docstring>
Implements Lion algorithm.


<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>step</name><anchor>timm.optim.Lion.step</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/lion.py#L81</source><parameters>[{"name": "closure", "val": " = None"}]</parameters><paramsdesc>- **closure** -- A closure that reevaluates the model and returns the loss.</paramsdesc><paramgroups>0</paramgroups><retdesc>the loss.</retdesc></docstring>
Performs a single optimization step.






</div></div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.optim.Lookahead</name><anchor>timm.optim.Lookahead</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/lookahead.py#L15</source><parameters>[{"name": "base_optimizer", "val": ""}, {"name": "alpha", "val": " = 0.5"}, {"name": "k", "val": " = 6"}]</parameters></docstring>


</div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.optim.MADGRAD</name><anchor>timm.optim.MADGRAD</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/madgrad.py#L24</source><parameters>[{"name": "params", "val": ": typing.Any"}, {"name": "lr", "val": ": float = 0.01"}, {"name": "momentum", "val": ": float = 0.9"}, {"name": "weight_decay", "val": ": float = 0"}, {"name": "eps", "val": ": float = 1e-06"}, {"name": "decoupled_decay", "val": ": bool = False"}]</parameters><paramsdesc>- **params** (iterable) --
  Iterable of parameters to optimize or dicts defining parameter groups.
- **lr** (float) --
  Learning rate (default: 1e-2).
- **momentum** (float) --
  Momentum value in  the range [0,1) (default: 0.9).
- **weight_decay** (float) --
  Weight decay, i.e. a L2 penalty (default: 0).
- **eps** (float) --
  Term added to the denominator outside of the root operation to improve numerical stability. (default: 1e-6).</paramsdesc><paramgroups>0</paramgroups></docstring>

MADGRAD_: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic
Optimization.

.. _MADGRAD: https://arxiv.org/abs/2101.11075

MADGRAD is a general purpose optimizer that can be used in place of SGD or
Adam may converge faster and generalize better. Currently GPU-only.
Typically, the same learning rate schedule that is used for SGD or Adam may
be used. The overall learning rate is not comparable to either method and
should be determined by a hyper-parameter sweep.

MADGRAD requires less weight decay than other methods, often as little as
zero. Momentum values used for SGD or Adam's beta1 should work here also.

On sparse problems both weight_decay and momentum should be set to 0.





<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>step</name><anchor>timm.optim.MADGRAD.step</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/madgrad.py#L90</source><parameters>[{"name": "closure", "val": ": typing.Optional[typing.Callable[[], float]] = None"}]</parameters><paramsdesc>- **closure** (callable, optional) -- A closure that reevaluates the model and returns the loss.</paramsdesc><paramgroups>0</paramgroups></docstring>
Performs a single optimization step.




</div></div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.optim.Mars</name><anchor>timm.optim.Mars</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/mars.py#L91</source><parameters>[{"name": "params", "val": ": typing.Union[collections.abc.Iterable[torch.Tensor], collections.abc.Iterable[dict[str, typing.Any]], collections.abc.Iterable[tuple[str, torch.Tensor]]]"}, {"name": "lr", "val": ": float = 0.003"}, {"name": "betas", "val": ": typing.Tuple[float, float] = (0.9, 0.99)"}, {"name": "eps", "val": ": float = 1e-08"}, {"name": "weight_decay", "val": ": float = 0.0"}, {"name": "gamma", "val": ": float = 0.025"}, {"name": "mars_type", "val": ": str = 'adamw'"}, {"name": "optimize_1d", "val": ": bool = False"}, {"name": "lr_1d_factor", "val": ": float = 1.0"}, {"name": "betas_1d", "val": ": typing.Optional[typing.Tuple[float, float]] = None"}, {"name": "caution", "val": ": bool = False"}]</parameters></docstring>
MARS Optimizer

Paper: MARS: Unleashing the Power of Variance Reduction for Training Large Models
https://arxiv.org/abs/2411.10438




<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>step</name><anchor>timm.optim.Mars.step</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/mars.py#L141</source><parameters>[{"name": "closure", "val": " = None"}]</parameters><paramsdesc>- **closure** (callable, optional) -- A closure that reevaluates the model
  and returns the loss.</paramsdesc><paramgroups>0</paramgroups></docstring>
Performs a single optimization step.




</div></div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.optim.NAdamW</name><anchor>timm.optim.NAdamW</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/nadamw.py#L21</source><parameters>[{"name": "params", "val": ": typing.Union[collections.abc.Iterable[torch.Tensor], collections.abc.Iterable[dict[str, typing.Any]], collections.abc.Iterable[tuple[str, torch.Tensor]]]"}, {"name": "lr", "val": ": float = 0.001"}, {"name": "betas", "val": ": typing.Tuple[float, float] = (0.9, 0.999)"}, {"name": "eps", "val": ": float = 1e-08"}, {"name": "weight_decay", "val": ": float = 0.01"}, {"name": "caution", "val": ": bool = False"}, {"name": "corrected_weight_decay", "val": ": bool = False"}, {"name": "maximize", "val": ": bool = False"}, {"name": "foreach", "val": ": typing.Optional[bool] = None"}, {"name": "capturable", "val": ": bool = False"}]</parameters><paramsdesc>- **params** -- iterable of parameters to optimize or dicts defining parameter groups
- **lr** -- learning rate
- **betas** -- coefficients used for computing running averages of gradient and its square
- **eps** -- term added to the denominator to improve numerical stability
- **weight_decay** -- weight decay coefficient
- **caution** -- enable caution
- **corrected_weight_decay** -- apply corrected weight decay (lr**2 / max_lr)</paramsdesc><paramgroups>0</paramgroups></docstring>
Implements NAdamW algorithm.

See Table 1 in https://arxiv.org/abs/1910.05446 for the implementation of
the NAdam algorithm (there is also a comment in the code which highlights
the only difference of NAdamW and AdamW).

For further details regarding the algorithm we refer to
- Decoupled Weight Decay Regularization: https://arxiv.org/abs/1711.05101
- On the Convergence of Adam and Beyond: https://openreview.net/forum?id=ryQu7f-RZ





<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>step</name><anchor>timm.optim.NAdamW.step</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/nadamw.py#L89</source><parameters>[{"name": "closure", "val": " = None"}]</parameters><paramsdesc>- **closure** (callable, optional) -- A closure that reevaluates the model
  and returns the loss.</paramsdesc><paramgroups>0</paramgroups></docstring>
Performs a single optimization step.




</div></div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.optim.NvNovoGrad</name><anchor>timm.optim.NvNovoGrad</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/nvnovograd.py#L13</source><parameters>[{"name": "params", "val": ""}, {"name": "lr", "val": " = 0.001"}, {"name": "betas", "val": " = (0.95, 0.98)"}, {"name": "eps", "val": " = 1e-08"}, {"name": "weight_decay", "val": " = 0"}, {"name": "grad_averaging", "val": " = False"}, {"name": "amsgrad", "val": " = False"}]</parameters><paramsdesc>- **params** (iterable) -- iterable of parameters to optimize or dicts defining
  parameter groups
- **lr** (float, optional) -- learning rate (default: 1e-3)
- **betas** (Tuple[float, float], optional) -- coefficients used for computing
  running averages of gradient and its square (default: (0.95, 0.98))
- **eps** (float, optional) -- term added to the denominator to improve
  numerical stability (default: 1e-8)
- **weight_decay** (float, optional) -- weight decay (L2 penalty) (default: 0)
- **grad_averaging** -- gradient averaging
- **amsgrad** (boolean, optional) -- whether to use the AMSGrad variant of this
  algorithm from the paper `On the Convergence of Adam and Beyond`_
  (default: False)</paramsdesc><paramgroups>0</paramgroups></docstring>

Implements Novograd algorithm.





<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>step</name><anchor>timm.optim.NvNovoGrad.step</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/nvnovograd.py#L66</source><parameters>[{"name": "closure", "val": " = None"}]</parameters><paramsdesc>- **closure** (callable, optional) -- A closure that reevaluates the model
- **and** returns the loss. --</paramsdesc><paramgroups>0</paramgroups></docstring>
Performs a single optimization step.




</div></div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.optim.RMSpropTF</name><anchor>timm.optim.RMSpropTF</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/rmsprop_tf.py#L20</source><parameters>[{"name": "params", "val": ": typing.Union[collections.abc.Iterable[torch.Tensor], collections.abc.Iterable[dict[str, typing.Any]], collections.abc.Iterable[tuple[str, torch.Tensor]]]"}, {"name": "lr", "val": ": float = 0.01"}, {"name": "alpha", "val": ": float = 0.9"}, {"name": "eps", "val": ": float = 1e-10"}, {"name": "weight_decay", "val": ": float = 0"}, {"name": "momentum", "val": ": float = 0.0"}, {"name": "centered", "val": ": bool = False"}, {"name": "decoupled_decay", "val": ": bool = False"}, {"name": "corrected_weight_decay", "val": ": bool = False"}, {"name": "lr_in_momentum", "val": ": bool = True"}, {"name": "caution", "val": ": bool = False"}]</parameters><paramsdesc>- **params** -- iterable of parameters to optimize or dicts defining parameter groups
- **lr** -- learning rate
- **momentum** -- momentum factor
- **alpha** -- smoothing (decay) constant
- **eps** -- term added to the denominator to improve numerical stability
- **centered** -- if `True`, compute the centered RMSProp, the gradient is normalized by an estimation of its variance
- **weight_decay** -- weight decay (L2 penalty) (default: 0)
- **decoupled_decay** -- decoupled weight decay as per https://arxiv.org/abs/1711.05101
- **corrected_weight_decay** -- apply corrected weight decay (lr**2 / max_lr) when decoupled_decay is True
- **lr_in_momentum** -- learning rate scaling is included in the momentum buffer update as per defaults in Tensorflow
- **caution** -- apply caution</paramsdesc><paramgroups>0</paramgroups></docstring>
Implements RMSprop algorithm (TensorFlow style epsilon)

NOTE: This is a direct cut-and-paste of PyTorch RMSprop with eps applied before sqrt
and a few other modifications to closer match Tensorflow for matching hyper-params.

Noteworthy changes include:
1. Epsilon applied inside square-root
2. square_avg initialized to ones
3. LR scaling of update accumulated in momentum buffer

Proposed by G. Hinton in his
[course](http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf).

The centered version first appears in [Generating Sequences
With Recurrent Neural Networks](https://arxiv.org/pdf/1308.0850v5.pdf).





<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>step</name><anchor>timm.optim.RMSpropTF.step</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/rmsprop_tf.py#L98</source><parameters>[{"name": "closure", "val": " = None"}]</parameters><paramsdesc>- **closure** (callable, optional) -- A closure that reevaluates the model
  and returns the loss.</paramsdesc><paramgroups>0</paramgroups></docstring>
Performs a single optimization step.




</div></div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.optim.SGDP</name><anchor>timm.optim.SGDP</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/sgdp.py#L19</source><parameters>[{"name": "params", "val": ""}, {"name": "lr", "val": " = <required parameter>"}, {"name": "momentum", "val": " = 0"}, {"name": "dampening", "val": " = 0"}, {"name": "weight_decay", "val": " = 0"}, {"name": "nesterov", "val": " = False"}, {"name": "eps", "val": " = 1e-08"}, {"name": "delta", "val": " = 0.1"}, {"name": "wd_ratio", "val": " = 0.1"}]</parameters></docstring>


</div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.optim.SGDW</name><anchor>timm.optim.SGDW</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/sgdw.py#L25</source><parameters>[{"name": "params", "val": ": typing.Union[collections.abc.Iterable[torch.Tensor], collections.abc.Iterable[dict[str, typing.Any]], collections.abc.Iterable[tuple[str, torch.Tensor]]]"}, {"name": "lr", "val": ": float = 0.001"}, {"name": "momentum", "val": ": float = 0.0"}, {"name": "dampening", "val": ": float = 0.0"}, {"name": "weight_decay", "val": ": float = 0.0"}, {"name": "nesterov", "val": ": bool = False"}, {"name": "caution", "val": ": bool = False"}, {"name": "corrected_weight_decay", "val": ": bool = False"}, {"name": "maximize", "val": ": bool = False"}, {"name": "foreach", "val": ": typing.Optional[bool] = None"}, {"name": "differentiable", "val": ": bool = False"}]</parameters></docstring>



<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>step</name><anchor>timm.optim.SGDW.step</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/sgdw.py#L94</source><parameters>[{"name": "closure", "val": " = None"}]</parameters><paramsdesc>- **closure** (Callable, optional) -- A closure that reevaluates the model
  and returns the loss.</paramsdesc><paramgroups>0</paramgroups></docstring>
Performs a single optimization step.




</div></div>

<EditOnGithub source="https://github.com/huggingface/pytorch-image-models/blob/main/hfdocs/source/reference/optimizers.mdx" />