Hello, I'm Ganesh. I'm building git-lrc, an AI code reviewer that runs on every commit. It is free, unlimited, and source-available on Github. Star git-lrc on GitHub to help more developers discover the project. Do give it a try and share your feedback for improving the product.
In the previous article, we derived formulas for updating the output layer weights w3, w4, and bias b3. Now, we will understand how to calculate the gradients for the hidden layer parameters: w1, b1, w2, and b2.
How are w1, b1, w2, and b2 connected to the prediction?
To find the gradients of the parameters in the hidden layer, we need to trace how changing these values affects the final prediction and the error (SSR).
Let's recall the structure of our neural network:
For the top neuron:
x1 = input * w1 + b1
y1 = f(x1) = log(1 + e^x1) (using the softplus function)
For the bottom neuron:
x2 = input * w2 + b2
y2 = f(x2) = log(1 + e^x2) (using the softplus function)
Finally, the prediction:
Predicted = y1 * w3 + y2 * w4 + b3
And the prediction error:
SSR = Σ (observed − predicted)²
Since w1, b1, w2, and b2 are not directly connected to the output prediction, we must use the chain rule to backpropagate the error from the output layer back to the hidden layer.
Applying the Chain Rule to the Hidden Layer
Let's calculate the gradient for the top neuron's weight w1 first.
A change in w1 affects x1, which affects the output y1, which affects the predicted value, which finally affects the SSR.
So, by the chain rule:
dSSR/dw1 = dSSR/d(predicted) * d(predicted)/dy1 * dy1/dx1 * dx1/dw1
Let's calculate each of these values:
1. dSSR/d(predicted)
As we saw in the previous articles, this is the derivative of SSR with respect to the predicted value:
dSSR/d(predicted) = -2 * (Observed - Predicted)
2. d(predicted)/dy1
Since Predicted = y1 * w3 + y2 * w4 + b3, and all other terms are treated as constants w.r.t y1:
d(predicted)/dy1 = w3
3. dy1/dx1
Since y1 = log(1 + e^x1), the derivative of the softplus function is the logistic sigmoid function:
dy1/dx1 = e^x1 / (1 + e^x1)
4. dx1/dw1
Since x1 = input * w1 + b1, differentiating w.r.t w1 gives:
dx1/dw1 = input
Final formula for dSSR/dw1:
Multiplying these parts together, we get:
dSSR/dw1 = -2 * (Observed - Predicted) * w3 * (e^x1 / (1 + e^x1)) * input
Deriving the Gradient for Bias b1
Similarly, for the top neuron's bias b1:
dSSR/db1 = dSSR/d(predicted) * d(predicted)/dy1 * dy1/dx1 * dx1/db1
The only term that changes here is the last one:
dx1/db1 = 1 (since x1 = input * w1 + b1, derivative w.r.t b1 is 1)
So:
dSSR/db1 = -2 * (Observed - Predicted) * w3 * (e^x1 / (1 + e^x1)) * 1
Deriving the Gradients for the Bottom Neuron (w2 and b2)
Following the same logic, we can find the gradients for the bottom neuron's parameters:
For weight w2:
dSSR/dw2 = dSSR/d(predicted) * d(predicted)/dy2 * dy2/dx2 * dx2/dw2
dSSR/dw2 = -2 * (Observed - Predicted) * w4 * (e^x2 / (1 + e^x2)) * input
For bias b2:
dSSR/db2 = dSSR/d(predicted) * d(predicted)/dy2 * dy2/dx2 * dx2/db2
dSSR/db2 = -2 * (Observed - Predicted) * w4 * (e^x2 / (1 + e^x2)) * 1
Improving Prediction with self Learning
Once we calculate all these derivatives (dSSR/dw1, dSSR/db1, dSSR/dw2, dSSR/db2), we can update the hidden layer weights and biases using gradient descent:
Step size w1 = derivation w1 * Learning rate
New w1 = old w1 - Step size w1
Step size b1 = derivation b1 * Learning rate
New b1 = old b1 - Step size b1
Step size w2 = derivation w2 * Learning rate
New w2 = old w2 - Step size w2
Step size b2 = derivation b2 * Learning rate
New b2 = old b2 - Step size b2
By doing this repeatedly, the model minimizes the error and converges to the optimal values for all weights and biases.
Conclusion
We have successfully derived the formulas to calculate the gradients for w1, b1, w2, and b2. Combined with the output layer derivations, we now have the math for the entire neural network's backpropagation!
In the next article, we will see how to implement this in code.
Any feedback or contributors are welcome! It’s online, source-available, and ready for anyone to use.