Optimization for AI: From Gradient Descent to Modern Optimizers

Stop treating optimization like a black box.Optimization for AI: From Gradient Descent to Modern Optimizers is a practical and rigorous guide to the algorithms that make machine learning work. Every training run depends on optimization, yet many engineers rely on optimizer.step() without understanding why a model converges, stalls, diverges, or overfits. This book closes that gap.Written for ML engineers, researchers, and serious students, this volume shows how optimization connects mathematical theory to real training behavior. You will build intuition for loss landscapes, understand what convexity does and does not guarantee, and learn how modern optimizers behave in practice.Inside this book, you will learn how to:Understand convex optimization and why it matters for machine learningImplement gradient descent, SGD, momentum, and Adam from first principlesChoose between SGD, momentum methods, and adaptive optimizers such as AdaGrad, RMSprop, Adam, and AdamWDesign and compare loss functions for classification, regression, and generative settingsApply regularization techniques such as weight decay, dropout, and early stopping with clear intuitionDiagnose unstable or slow training using convergence ideas instead of trial and errorTune learning rates, schedules, and optimization settings with more confidenceWith worked examples, practical guidance, convergence analysis, and a capstone optimizer showdown project, this book helps you move beyond cookbook training recipes and develop real optimization judgment.If you build, study, or debug machine learning systems, Optimization for AI gives you the foundation to understand what your training loop is really doing. Read more