Hyperparameters and Regularization

Introduction

Hyperparameters and regularization are critical concepts in deep learning that influence how your models learn, generalize, and perform.

1️⃣ What are Hyperparameters?

Hyperparameters are configurations set before training that determine how a model learns.

They are not learned from the data but defined manually.

Common Hyperparameters

✅ Learning Rate ((\eta)): Controls how big the weight updates are during training.
✅ Batch Size: Number of samples used to compute gradients per update.
✅ Number of Epochs: Number of complete passes over the training data.
✅ Number of Layers and Units: Defines model architecture.
✅ Optimizer Type: SGD, Adam, RMSProp, etc.
✅ Dropout Rate: Fraction of neurons dropped during training for regularization.

Why Hyperparameters Matter

✅ Correct hyperparameters can improve training speed and model accuracy.
✅ Poor choices can lead to underfitting, overfitting, or slow training.

Hyperparameter tuning involves systematically experimenting with different values to find the best setup for your model.

2️⃣ What is Regularization?

Regularization is a set of techniques to prevent overfitting, ensuring your model generalizes well to new, unseen data.

Overfitting happens when:

✅ Your model learns noise in the training data instead of general patterns.
✅ It performs well on training data but poorly on test data.

Common Regularization Techniques

L1 and L2 Regularization

• L1 Regularization (Lasso): Adds the sum of absolute weights to the loss function, promoting sparsity.
• L2 Regularization (Ridge): Adds the sum of squared weights to the loss function, discouraging large weights.

Dropout

Randomly drops a fraction of neurons during training to prevent reliance on specific neurons, improving generalization.

Early Stopping

Stops training when the validation loss stops improving, preventing overfitting.

Example: Adding L2 Regularization in TensorFlow

from tensorflow.keras import regularizers

model.add(tf.keras.layers.Dense(64, activation='relu',
    kernel_regularizer=regularizers.l2(0.01)))

Example: Using Dropout

model.add(tf.keras.layers.Dropout(0.5))  # Drops 50% of neurons during training

Conclusion

✅ Hyperparameters control how your models learn.
✅ Regularization ensures your models generalize well.
✅ Understanding and tuning these will significantly improve your deep learning projects.

What’s Next?

✅ Practice tuning hyperparameters using a small dataset.
✅ Experiment with dropout and L2 regularization to see their effects.
✅ Continue your structured learning on superml.org to build strong DL foundations.

Join the SuperML Community to share your tuning experiments and get personalized feedback.

Happy Learning! 🛠️