Gradient Descent in Linear Models: Learning Rate, Initialization, and Scaling

Gradient descent is a fundamental optimization method for training linear and logistic regression models. Although the underlying objectives are convex, practitioners may observe unstable behaviour such as divergence, oscillations, and extremely slow convergence. In this work, we conduct a systematic investigation of the optimization dynamics of vanilla gradient descent for a simple linear regression model. Using controlled synthetic datasets, we study how convergence depends on learning rate selection, feature scaling, and the magnitude of parameter initialization. We also compare batch gradient descent (BGD), stochastic gradient descent (SGD), and mini-batch gradient descent. Our results illustrate clear stability regions for the learning rate, highlight the critical role of feature scaling, and show how different optimization variants trade off stability and noise.