Fixed Learning Rate

Fixed learning rate is a hyperparameter in machine learning optimization algorithms, such as gradient descent, that determines the step size taken during each iteration to update model parameters. It remains constant throughout the training process, providing a straightforward approach to controlling how quickly a model learns from data. This method is simple to implement but requires careful tuning to balance convergence speed and stability.

Also known as: Constant Learning Rate, Static Learning Rate, Fixed Step Size, Fixed LR, Fixed Learning Rate Schedule

Why learn Fixed Learning Rate?

Developers should use a fixed learning rate when training simple models or in scenarios where computational resources are limited, as it reduces complexity and overhead compared to adaptive methods. It is particularly useful for educational purposes, baseline experiments, or in stable optimization landscapes where a constant step size suffices for convergence without oscillation or divergence.

Learning Resources

• tutorial: Understanding Learning Rates in Deep Learning
• docs: Gradient Descent and Learning Rate - Stanford CS229 Notes
• course: Machine Learning Course by Andrew Ng - Learning Rate Section
• video: Fixed vs. Adaptive Learning Rates in Optimization
• book: Deep Learning Book by Goodfellow et al. - Optimization Chapter