#
Learning Rate Schedulers

This page contains the API reference documentation for learning rate schedulers included in `timm`.

##
Schedulers

###
Factory functions

####
timm.scheduler.create_scheduler

< source >

( args optimizer: Optimizer updates_per_epoch: int = 0 )

####
timm.scheduler.create_scheduler_v2

< source >

( optimizer: Optimizer sched: str = 'cosine' num_epochs: int = 300 decay_epochs: int = 90 decay_milestones: typing.List[int] = (90, 180, 270) cooldown_epochs: int = 0 patience_epochs: int = 10 decay_rate: float = 0.1 min_lr: float = 0 warmup_lr: float = 1e-05 warmup_epochs: int = 0 warmup_prefix: bool = False noise: typing.Union[float, typing.List[float]] = None noise_pct: float = 0.67 noise_std: float = 1.0 noise_seed: int = 42 cycle_mul: float = 1.0 cycle_decay: float = 0.1 cycle_limit: int = 1 k_decay: float = 1.0 plateau_mode: str = 'max' step_on_epochs: bool = True updates_per_epoch: int = 0 )

###
Scheduler Classes

###
class timm.scheduler.CosineLRScheduler

< source >

( optimizer: Optimizer t_initial: int lr_min: float = 0.0 cycle_mul: float = 1.0 cycle_decay: float = 1.0 cycle_limit: int = 1 warmup_t = 0 warmup_lr_init = 0 warmup_prefix = False t_in_epochs = True noise_range_t = None noise_pct = 0.67 noise_std = 1.0 noise_seed = 42 k_decay = 1.0 initialize = True )

Cosine decay with restarts. This is described in the paper https://arxiv.org/abs/1608.03983.

Inspiration from https://github.com/allenai/allennlp/blob/master/allennlp/training/learning_rate_schedulers/cosine.py

k-decay option based on `k-decay: A New Method For Learning Rate Schedule` - https://arxiv.org/abs/2004.05909

###
class timm.scheduler.MultiStepLRScheduler

< source >

( optimizer: Optimizer decay_t: typing.List[int] decay_rate: float = 1.0 warmup_t = 0 warmup_lr_init = 0 warmup_prefix = True t_in_epochs = True noise_range_t = None noise_pct = 0.67 noise_std = 1.0 noise_seed = 42 initialize = True )

###
class timm.scheduler.PlateauLRScheduler

< source >

( optimizer decay_rate = 0.1 patience_t = 10 verbose = True threshold = 0.0001 cooldown_t = 0 warmup_t = 0 warmup_lr_init = 0 lr_min = 0 mode = 'max' noise_range_t = None noise_type = 'normal' noise_pct = 0.67 noise_std = 1.0 noise_seed = None initialize = True )

Decay the LR by a factor every time the validation loss plateaus.

###
class timm.scheduler.PolyLRScheduler

< source >

( optimizer: Optimizer t_initial: int power: float = 0.5 lr_min: float = 0.0 cycle_mul: float = 1.0 cycle_decay: float = 1.0 cycle_limit: int = 1 warmup_t = 0 warmup_lr_init = 0 warmup_prefix = False t_in_epochs = True noise_range_t = None noise_pct = 0.67 noise_std = 1.0 noise_seed = 42 k_decay = 1.0 initialize = True )

Polynomial LR Scheduler w/ warmup, noise, and k-decay

k-decay option based on `k-decay: A New Method For Learning Rate Schedule` - https://arxiv.org/abs/2004.05909

###
class timm.scheduler.StepLRScheduler

< source >

( optimizer: Optimizer decay_t: float decay_rate: float = 1.0 warmup_t = 0 warmup_lr_init = 0 warmup_prefix = True t_in_epochs = True noise_range_t = None noise_pct = 0.67 noise_std = 1.0 noise_seed = 42 initialize = True )

###
class timm.scheduler.TanhLRScheduler

< source >

( optimizer: Optimizer t_initial: int lb: float = -7.0 ub: float = 3.0 lr_min: float = 0.0 cycle_mul: float = 1.0 cycle_decay: float = 1.0 cycle_limit: int = 1 warmup_t = 0 warmup_lr_init = 0 warmup_prefix = False t_in_epochs = True noise_range_t = None noise_pct = 0.67 noise_std = 1.0 noise_seed = 42 initialize = True )

Hyberbolic-Tangent decay with restarts. This is described in the paper https://arxiv.org/abs/1806.01593

Update on GitHub