Cross Entropy Derivatives, Part 6: Using gradient descent to reach the final result

In the previous article, We plotted a curve which would help us optimize the bias.

In this article, we will get the accurate value for b3.

We begin with the derivative of the cross-entropy loss with respect to ( b_3 ).

Since we have three observations and make one prediction per observation, the total derivative is obtained by summing one term per prediction.

First observation

Let us focus on the prediction for the first observation.

Because the observed species is Setosa, we compute the cross entropy using the predicted probability for Setosa. We already know that

For petal width ( 0.04 ) and sepal width ( 0.42 ), the predicted probability is

So the contribution from the first observation is

0.15 - 1

Second observation

Now consider the second observation.

In this case, the observed species is Virginica, so the derivative is

Using petal width ( 1.0 ) and sepal width ( 0.54 ), the predicted probability for Setosa is

Third observation

Finally, consider the third observation.

The observed species is Versicolor, so the derivative is again

For petal width ( 0.50 ) and sepal width ( 0.37 ), the predicted probability for Setosa is

Total derivative

Adding all three contributions gives

(0.15 - 1) + 0.04 + 0.04 = -0.77

This value represents the slope of the tangent line to the loss curve at ( b_3 = -2 ).

Gradient descent update

We now plug this slope into the gradient descent update rule.

If we set the learning rate to ( 1 ), then

We update ( b_3 ) as follows:

We then repeat this process, using the updated value of ( b_3 ) each time, until the predictions stop improving, a maximum number of steps is reached, or another stopping criterion is met.

In this case, the predictions stop improving when ( b_3 = -0.03 ).

Looking for an easier way to install tools, libraries, or entire repositories?

Try Installerpedia: a community-driven, structured installation platform that lets you install almost anything with minimal hassle and clear, reliable guidance.

Just run:

ipm install repo-name

… and you’re done! 🚀

🔗 Explore Installerpedia here