Learning rate
lastmod 2022-11-25created 2022-11-259 edits
In machine learning and statistics, the learning rate is a tuning parameter in an optimization algorithm that determines the step size at each iteration while moving toward a minimum of a loss function. Since it influences to what extent newly acquired information overrides old information, it metaphorically represents the speed at which a machine learning model "learns".
Learning Rate Scheduler
- How to optimize the Learning Rate in deep learning :: Steady Wins the Race
- Considerations for improving deep learning training – GIS Developer
# 학습률 스케줄러는 옵티마이져를 통해 적용된다.
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
# 지정한 스텝 단위로 학습률에 감마를 곱해 학습률을 감소시키는 방식
scheduler = lr_scheduler.StepLR(optimizer, step_size=1, gamma= 0.99)
# 지정한 스텝 지점(예시에서는 10,20,40)마다 학습률에 감마를 곱해줘서 감소시키는 방식
scheduler = lr_scheduler.MultiStepLR(optimizer, milestones=[10,20,40], gamma= 0.1)
# 매 epoch마다 학습률에 감마를 곱해줌으로써 감소시키는 방식
scheduler = lr_scheduler.ExponentialLR(optimizer, gamma= 0.99)
# 원하는 에폭마다, 이전 학습률 대비 변경폭에 따라 학습률을 감소시켜주는 방식
scheduler = lr_scheduler.ReduceLROnPlateau(optimizer,threshold=1,patience=1,mode='min')