Why Feature Scaling Works: Condition Number as the Key to Gradient Descent Convergence
Rishi Khandelwal
28/09/2026
Feature scaling is universally recommended in machine learning practice, yet the choice among competing methods is usually made heuristically. This paper uses the condition number of the Hessian as the diagnostic quantity for that choice. Part I restates the established link between feature variance disparity and ill-conditioning as a formal proposition under explicit assumptions—centred data, full column rank, and a least-squares objective—and validates it empirically on synthetic functions with controlled condition numbers. Part II states hypotheses about how four normalisation methods—standardisation, min-max scaling, robust scaling, and PCA whitening—perform across varying dataset sizes and feature scale imbalances, then tests them experimentally, reporting measured alongside convergence counts. We then report a negative result: a proposed two-stage robust-then-whiten pipeline performs identically to whitening alone under outlier contamination, because robust scaling is a diagonal transformation and therefore cannot alter the correlation structure that whitening inverts. Measuring against an effective condition number—the fitted transform evaluated on clean data—rather than the in-sample value shows that whitening attains =1 on contaminated data by construction while conditioning the underlying signal one to two orders of magnitude worse than robust scaling alone. Part II also reports an exploratory study of gradient-side factor scaling, including a negative result for sample-side preconditioning on least-squares objectives.