# Learning Rate Schedulers

This page contains the API reference documentation for learning rate schedulers included in `timm`.

## Schedulers

### Factory functions[[timm.scheduler.create_scheduler]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>timm.scheduler.create_scheduler</name><anchor>timm.scheduler.create_scheduler</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/scheduler/scheduler_factory.py#L51</source><parameters>[{"name": "args", "val": ""}, {"name": "optimizer", "val": ": Optimizer"}, {"name": "updates_per_epoch", "val": ": int = 0"}]</parameters></docstring>


</div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>timm.scheduler.create_scheduler_v2</name><anchor>timm.scheduler.create_scheduler_v2</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/scheduler/scheduler_factory.py#L63</source><parameters>[{"name": "optimizer", "val": ": Optimizer"}, {"name": "sched", "val": ": str = 'cosine'"}, {"name": "num_epochs", "val": ": int = 300"}, {"name": "decay_epochs", "val": ": int = 90"}, {"name": "decay_milestones", "val": ": typing.List[int] = (90, 180, 270)"}, {"name": "cooldown_epochs", "val": ": int = 0"}, {"name": "patience_epochs", "val": ": int = 10"}, {"name": "decay_rate", "val": ": float = 0.1"}, {"name": "min_lr", "val": ": float = 0"}, {"name": "warmup_lr", "val": ": float = 1e-05"}, {"name": "warmup_epochs", "val": ": int = 0"}, {"name": "warmup_prefix", "val": ": bool = False"}, {"name": "noise", "val": ": typing.Union[float, typing.List[float]] = None"}, {"name": "noise_pct", "val": ": float = 0.67"}, {"name": "noise_std", "val": ": float = 1.0"}, {"name": "noise_seed", "val": ": int = 42"}, {"name": "cycle_mul", "val": ": float = 1.0"}, {"name": "cycle_decay", "val": ": float = 0.1"}, {"name": "cycle_limit", "val": ": int = 1"}, {"name": "k_decay", "val": ": float = 1.0"}, {"name": "plateau_mode", "val": ": str = 'max'"}, {"name": "step_on_epochs", "val": ": bool = True"}, {"name": "updates_per_epoch", "val": ": int = 0"}]</parameters></docstring>


</div>

### Scheduler Classes[[timm.scheduler.CosineLRScheduler]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.scheduler.CosineLRScheduler</name><anchor>timm.scheduler.CosineLRScheduler</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/scheduler/cosine_lr.py#L19</source><parameters>[{"name": "optimizer", "val": ": Optimizer"}, {"name": "t_initial", "val": ": int"}, {"name": "lr_min", "val": ": float = 0.0"}, {"name": "cycle_mul", "val": ": float = 1.0"}, {"name": "cycle_decay", "val": ": float = 1.0"}, {"name": "cycle_limit", "val": ": int = 1"}, {"name": "warmup_t", "val": " = 0"}, {"name": "warmup_lr_init", "val": " = 0"}, {"name": "warmup_prefix", "val": " = False"}, {"name": "t_in_epochs", "val": " = True"}, {"name": "noise_range_t", "val": " = None"}, {"name": "noise_pct", "val": " = 0.67"}, {"name": "noise_std", "val": " = 1.0"}, {"name": "noise_seed", "val": " = 42"}, {"name": "k_decay", "val": " = 1.0"}, {"name": "initialize", "val": " = True"}]</parameters></docstring>

Cosine decay with restarts.
This is described in the paper https://arxiv.org/abs/1608.03983.

Inspiration from
https://github.com/allenai/allennlp/blob/master/allennlp/training/learning_rate_schedulers/cosine.py

k-decay option based on `k-decay: A New Method For Learning Rate Schedule` - https://arxiv.org/abs/2004.05909


</div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.scheduler.MultiStepLRScheduler</name><anchor>timm.scheduler.MultiStepLRScheduler</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/scheduler/multistep_lr.py#L10</source><parameters>[{"name": "optimizer", "val": ": Optimizer"}, {"name": "decay_t", "val": ": typing.List[int]"}, {"name": "decay_rate", "val": ": float = 1.0"}, {"name": "warmup_t", "val": " = 0"}, {"name": "warmup_lr_init", "val": " = 0"}, {"name": "warmup_prefix", "val": " = True"}, {"name": "t_in_epochs", "val": " = True"}, {"name": "noise_range_t", "val": " = None"}, {"name": "noise_pct", "val": " = 0.67"}, {"name": "noise_std", "val": " = 1.0"}, {"name": "noise_seed", "val": " = 42"}, {"name": "initialize", "val": " = True"}]</parameters></docstring>



</div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.scheduler.PlateauLRScheduler</name><anchor>timm.scheduler.PlateauLRScheduler</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/scheduler/plateau_lr.py#L13</source><parameters>[{"name": "optimizer", "val": ""}, {"name": "decay_rate", "val": " = 0.1"}, {"name": "patience_t", "val": " = 10"}, {"name": "verbose", "val": " = True"}, {"name": "threshold", "val": " = 0.0001"}, {"name": "cooldown_t", "val": " = 0"}, {"name": "warmup_t", "val": " = 0"}, {"name": "warmup_lr_init", "val": " = 0"}, {"name": "lr_min", "val": " = 0"}, {"name": "mode", "val": " = 'max'"}, {"name": "noise_range_t", "val": " = None"}, {"name": "noise_type", "val": " = 'normal'"}, {"name": "noise_pct", "val": " = 0.67"}, {"name": "noise_std", "val": " = 1.0"}, {"name": "noise_seed", "val": " = None"}, {"name": "initialize", "val": " = True"}]</parameters></docstring>
Decay the LR by a factor every time the validation loss plateaus.

</div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.scheduler.PolyLRScheduler</name><anchor>timm.scheduler.PolyLRScheduler</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/scheduler/poly_lr.py#L19</source><parameters>[{"name": "optimizer", "val": ": Optimizer"}, {"name": "t_initial", "val": ": int"}, {"name": "power", "val": ": float = 0.5"}, {"name": "lr_min", "val": ": float = 0.0"}, {"name": "cycle_mul", "val": ": float = 1.0"}, {"name": "cycle_decay", "val": ": float = 1.0"}, {"name": "cycle_limit", "val": ": int = 1"}, {"name": "warmup_t", "val": " = 0"}, {"name": "warmup_lr_init", "val": " = 0"}, {"name": "warmup_prefix", "val": " = False"}, {"name": "t_in_epochs", "val": " = True"}, {"name": "noise_range_t", "val": " = None"}, {"name": "noise_pct", "val": " = 0.67"}, {"name": "noise_std", "val": " = 1.0"}, {"name": "noise_seed", "val": " = 42"}, {"name": "k_decay", "val": " = 1.0"}, {"name": "initialize", "val": " = True"}]</parameters></docstring>
Polynomial LR Scheduler w/ warmup, noise, and k-decay

k-decay option based on `k-decay: A New Method For Learning Rate Schedule` - https://arxiv.org/abs/2004.05909


</div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.scheduler.StepLRScheduler</name><anchor>timm.scheduler.StepLRScheduler</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/scheduler/step_lr.py#L15</source><parameters>[{"name": "optimizer", "val": ": Optimizer"}, {"name": "decay_t", "val": ": float"}, {"name": "decay_rate", "val": ": float = 1.0"}, {"name": "warmup_t", "val": " = 0"}, {"name": "warmup_lr_init", "val": " = 0"}, {"name": "warmup_prefix", "val": " = True"}, {"name": "t_in_epochs", "val": " = True"}, {"name": "noise_range_t", "val": " = None"}, {"name": "noise_pct", "val": " = 0.67"}, {"name": "noise_std", "val": " = 1.0"}, {"name": "noise_seed", "val": " = 42"}, {"name": "initialize", "val": " = True"}]</parameters></docstring>



</div>

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class timm.scheduler.TanhLRScheduler</name><anchor>timm.scheduler.TanhLRScheduler</anchor><source>https://github.com/huggingface/pytorch-image-models/blob/main/timm/scheduler/tanh_lr.py#L19</source><parameters>[{"name": "optimizer", "val": ": Optimizer"}, {"name": "t_initial", "val": ": int"}, {"name": "lb", "val": ": float = -7.0"}, {"name": "ub", "val": ": float = 3.0"}, {"name": "lr_min", "val": ": float = 0.0"}, {"name": "cycle_mul", "val": ": float = 1.0"}, {"name": "cycle_decay", "val": ": float = 1.0"}, {"name": "cycle_limit", "val": ": int = 1"}, {"name": "warmup_t", "val": " = 0"}, {"name": "warmup_lr_init", "val": " = 0"}, {"name": "warmup_prefix", "val": " = False"}, {"name": "t_in_epochs", "val": " = True"}, {"name": "noise_range_t", "val": " = None"}, {"name": "noise_pct", "val": " = 0.67"}, {"name": "noise_std", "val": " = 1.0"}, {"name": "noise_seed", "val": " = 42"}, {"name": "initialize", "val": " = True"}]</parameters></docstring>

Hyberbolic-Tangent decay with restarts.
This is described in the paper https://arxiv.org/abs/1806.01593


</div>

<EditOnGithub source="https://github.com/huggingface/pytorch-image-models/blob/main/hfdocs/source/reference/schedulers.mdx" />