Fixed Learning Rate
A fixed learning rate is a hyperparameter in machine learning optimization algorithms, such as gradient descent, that determines the step size at each iteration while updating model parameters. It remains constant throughout the training process, providing a straightforward approach to controlling how quickly a model learns from data. This simplicity makes it easy to implement but can lead to suboptimal convergence if not carefully tuned.
Developers should use fixed learning rates when training simple models or in scenarios where computational resources are limited, as it reduces complexity and overhead compared to adaptive methods. It is particularly useful for educational purposes, baseline experiments, or in stable optimization problems where the loss landscape is smooth and predictable, such as linear regression or shallow neural networks.