In the previous article, we prepared everything needed to optimize our neural network and find the ideal value for the final bias.
In this article, we will begin implementing the optimization process.
Creating the Optimizer
First, we create an optimizer object.
We will use Stochastic Gradient Descent (SGD) to optimize final_bias.
optimizer = SGD(model.parameters(), lr=0.1)
To optimize final_bias, we pass:
model.parameters()
to SGD.
PyTorch will automatically optimize every parameter where:
requires_grad=True
In our case, only final_bias has requires_grad=True, so that is the only parameter that will be updated during training.
Here, lr stands for learning rate, which is set to 0.1.
The learning rate controls how large each update step is during optimization.
Understanding Epochs
Before continuing, there is one important term we need to understand: epoch.
An epoch is one complete pass through the entire training dataset.
In this example, our training data contains 3 data points.
Every time all 3 training points are passed through the model once, we call it one epoch.
Running the Optimization Loop
We can now start the optimization process using a for loop that counts the number of epochs.
for epoch in range(100):
...
This loop will run the training process 100 times.
In other words, the model will see the full training dataset 100 times.
Tracking the Loss
Next, we initialize a variable called total_loss.
This stores the loss, which is a measure of how well the model fits the training data.
To better understand total_loss, let us look at an example.
In the figure above, the unoptimized model fits the training data poorly.
The residuals (the difference between what the model predicts and what we know is true) are large.
Because the residuals are large, the loss will also be relatively large.
Now imagine the model improves and fits the training data more closely.
The residuals become smaller.
In this case, the loss becomes smaller because the model predictions are closer to the correct values.
So, during each epoch, we use total_loss to keep track of how well the model fits the training data.
We will continue building the optimization process in the next article.
AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.
git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.
Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.
Give it a ⭐ star on Github