Optimizers in Deep Learning

Optimizers are methods (algo.) used to adjust the weights and biases of a neural network during training, aiming to minimize the loss function and improve model performance.

If the "loss function" calculates how wrong your model's predictions are, the optimizer is the engine that decides how to update the model's internal "parameters" — "weights" and "biases" to make it less wrong the next time.

Key Aspects:

Weight Updates: Optimizers modify weights of the neural network to minimize the cost function.
Learning Rate: A critical parameter that defines how much the model weights update during optimization. If too large, it may overshoot; if too small, it may take excessive time to converge.
Convergence: Optimizers aim to achieve faster and stable convergence while avoiding challenges like overshooting or getting stuck in local minima.

Different optimizers offer various techniques for updating model parameters during training. By studying optimizers, we can adapt our learning algorithms to specific problem domains, enhance model convergence, and address challenges such as vanishing or exploding gradients.

1. Gradient Descent

Batch Gradient Descent (BGD)

Updates parameters by computing the gradient of the loss function over the entire dataset.

Stochastic Gradient Descent (SGD)

Instead of calculating the error for the entire dataset at once (like in BGD), SGD calculates the error and updates the network parameters — weights and biases for just one single training example at a time.

SGD Mini-batch (MSGD)

Updates network parameters using the average gradient over a small batch (e.g., 32 or 64 images) of training samples

Pros of GD:

Easy to understand and implement.
Performs well when the loss function is convex.
Basic foundation for other advanced optimizers.

Cons of GD:

Slow Convergence, especially for large datasets (with Batch Gradient Descent).
Its gets stuck in Local Minima thus not suited for non-convex loss surfaces (common in deep learning).
Requires feature scaling for consistent performance.

Comparison of BGD, SGD and MSG

	Learning Rate	Memory Requirement	Convergence Speed	Sensitivity to Learning Rate	Sensitivity to Noise
BGD	Large, Fixed: Can use a relatively large, fixed learning rate because the gradient direction is highly accurate.	Extremely High. You must process the entire dataset in memory to compute one single update. Often impossible for large datasets.	Slow. Taking one step requires massive computation. The path is direct, but the time taken per step is enormous.	Moderate. If set too high, the model will diverge (explode). If too low, training will take an eternity.	Very Low. By averaging the entire dataset, the impact of outliers and noisy data is completely smoothed out.
SGD	Small, Fixed: Requires a much smaller, decaying learning rate to prevent wild, erratic jumps.	Very Low. Only requires enough memory to hold a single training example and the network's weights.	Fast initially, but struggles to settle. Frequent updates drop the loss quickly early on, but it constantly overshoots the absolute minimum.	Extremely High. Because updates are so noisy, a slightly oversized learning rate will cause the model to bounce around endlessly without ever converging.	Very High. A single mislabeled image or outlier will instantly pull the model's weights in the wrong direction.
MSGD	Moderate, Fixed: Uses a moderate learning rate, usually combined with a decay schedule or a modern optimizer (like Adam).	Moderate / Optimized. Perfectly balances memory by loading small batches that fit exactly into standard GPU VRAM.	Fastest overall. Benefits from the speed of frequent updates and the hardware efficiency of matrix multiplication (vectorization).	Moderate. Still requires tuning, but the averaged gradients make it slightly more forgiving than pure SGD.	Moderate. Outliers are diluted by the rest of the batch, but there is still enough "healthy noise" to help the model escape shallow local minima.

2. SGD with Momentum

Momentum modifies SGD by adding a fraction of the previous weight update to the current one. It essentially builds up "inertia."

Pros:

Helps Faster Convergence, especially in high-curvature areas.
Avoids Oscillations as it smooths gradients using the momentum term.
Can escape saddle points faster than plain SGD.

Cons:

Requires tuning the momentum coefficient ( $β$ ).
If momentum is too high, it may overshoot the minima.

Learning Rate	Memory Requirement	Convergence Speed	Sensitivity to Learning Rate	Sensitivity to Noise
Fixed: Learning rate ( $η$ ) remains constant but combined with a momentum term to stabilize updates.	Slightly higher than SGD since it needs to store the momentum term.	ue to momentum, which reduces oscillations and helps navigate plateaus.	Less sensitive than SGD due to the smoothing effect of the momentum term.	Handles noise better than plain SGD by incorporating past gradients to smooth updates.

3. AdaGrad

Instead of using one global learning rate $α$ for all parameters, AdaGrad adapts the learning rate for each parameter individually. Adapts learning rates based on the frequency of updates for each parameter. It performs larger updates for infrequent parameters and smaller updates for frequent ones.

Pros:

Excellent for handling sparse data (like NLP, where some words appear rarely). It eliminates the need to manually tune the learning rate.
Generally good performance

Cons:

It continuously accumulates squared gradients in the denominator, which causes the learning rate to shrink to zero over time. Eventually, the model completely stops learning.
Higher memory requirement, may not generalize well

Learning Rate	Memory Requirement	Convergence Speed	Sensitivity to Learning Rate	Sensitivity to Noise
Adaptive (decays): Starts with a global learning rate but diminishes parameter-wise as training progresses. $η_{effective} = \frac{η}{\sqrt{\sum_{t = 1}^{T} g_{i, t}^{2}}}$	High: High memory usage to store squared gradients for each parameter.	Initially fast due to adaptive learning rates but slows significantly over time due to diminishing updates.	Less sensitive to initial learning rate as it adjusts automatically during training.	Stable for sparse data but more sensitive to noise in dense-feature spaces due to gradient accumulation.

4. RMSProp

Modifies AdaGrad by maintaining a moving average of the squared gradients, preventing rapid decay.

Pros:

Resolves AdaGrad's Issues and maintains adaptive learning rates without diminishing them excessively.
Frequently used for recurrent neural networks (RNN) due to its ability to deal with exploding/vanishing gradients.

Cons:

Hyperparameter Complexity: Requires tuning the decay rate ( $ρ$ ) and learning rate ( $η$ ).
Gradient Dependency: Relies on squared gradients, making it sensitive to initialization.

Learning Rate	Memory Requirement	Convergence Speed	Sensitivity to Learning Rate	Sensitivity to Noise
Adaptive per parameter: Decay factor ( $ρ$ ) ensures stability. $η_{effective} = \frac{η}{\sqrt{E [g^{2}]}}$	Medium: Similar to AdaGrad, requires storing moving averages of squared gradients.	Faster than Gradient Descent approaches due to adaptive learning rates.	Less sensitive due to decayed gradient accumulation.	Handles noisy gradients well by leveraging moving averages of squared gradients.

5. Adam (Adaptive Moment Estimation)

Combines the advantages of AdaGrad and RMSProp, adaptively adjusting the learning rates with momentum.

Pros:

Combines Momentum and RMSProp: Smooths updates using momentum while retaining adaptive learning rates.
Fast Convergence: Efficient for non-convex optimization problems.
Robust Across Tasks: General-purpose optimizer that handles various types of architectures and datasets.

Cons:

Computationally Expensive: Requires additional memory and computation.
Not Guaranteed to Converge: May fail to converge in certain scenarios without proper parameter tuning.

Learning Rate	Memory Requirement	Convergence Speed	Sensitivity to Learning Rate	Sensitivity to Noise
Adaptive per parameter using first and second moments: $η_{t} = \frac{η}{\sqrt{{\hat{v}}_{t}} + ϵ}$ , where ${\hat{v}}_{t}$ is the corrected EWMA of squared gradients.	High: Stores first and second momentum terms for each parameter.	Faster than SGD, RMSProp, and other basic optimizers; ideal for deep, complex architectures.	Less sensitive due to the adaptive mechanism, but the initial learning rate still requires careful tuning.	Robust to noise due to its adaptive mechanism and momentum-driven smoothing; performs well on noisy datasets.

8. Nadam (Nesterov-accelerated Adam)

Combines Nesterov Accelerated Gradient (NAG) with Adam, looking ahead in the gradient adaptation for more accurate updates.

Pros:

Improves Adam: Combines Nesterov Accelerated Gradient (NAG) with Adam to achieve better performance.
More Accurate Updates: Looks ahead in gradient adaptation, avoiding overshooting.
Good Convergence Properties: Ideal for both convex and non-convex objective functions.

Cons:

More Complexity: Requires understanding Nesterov updates in addition to Adam's principles.
Higher Computational Cost: Expands on Adam, making it even more computationally expensive.

Learning Rate	Memory Requirement	Convergence Speed	Sensitivity to Learning Rate	Sensitivity to Noise
Adaptive, similar to Adam with an additional Nesterov acceleration term: $η_{t} = \frac{η}{\sqrt{{\hat{v}}_{t}} + ϵ}$	High: Stores multiple terms for each parameter.	Generally faster than Adam due to lookahead gradient updates.	Minimal, as Nadam inherits Adam's adaptive learning rate mechanics.	Robust to noise; effectively handles noisy datasets better than standard Adam.

Questions and Answers

Optimizer in NN vs. Ensemble: What is the Difference?

Aspect	Gradient Descent in Neural Networks	Gradient Descent in Ensembles
Objective	Minimize the loss function by updating weights and biases.	Minimize residual errors by sequentially adding weak learners.
Focus	Focuses on individual model parameters (weights and biases).	Focuses on combining predictions or reducing errors in the ensemble.
Input for Gradient Computation	Gradients are computed on weights after backpropagation.	Gradients are computed on residual errors to train subsequent models.
Operational Scope	Involves a single network architecture (multi-layered).	Operates across multiple weak models (decision trees, etc.).
Optimization Target	Optimizes learnable weights using the computed gradients.	Optimizes ensemble loss or residual errors using weak learners.
Parameter Update	Updates weights in the neural network using gradients.	Updates predictions by combining weak learners iteratively.
Learning Rate	Controls how much weights change per iteration.	Scales the contribution of new weak learners (shrinkage factor).
Requirement	Requires backpropagation to compute weight gradients.	Requires fitting weak learners based on gradient residuals.