Mild Introduction to Modern Sequence Processing. Part 2: Recurrent Neural Networks Training

In our previous article, we learned the fundamental concepts of RNN and examined the core logic of forward pass. In this one, we are going to circle back to forward pass to review the formulas and to recollect the intuition on what the RNN is doing, and immediately after that, we turn our attention to the training process.

Many-to-One RNN: What Is It and How Does It Work

First, let’s examine in detail one of the variants of the RNN, a slightly simplified version of the model from the previous entry - many-to-one RNN. This one only outputs once the whole sequence (all the tokens of the sequence) is processed.

The result of the function above is the error that we need to propagate back through the net to tweak the model's parameters W, U, and V to lower the error.

Backward Pass

The Backward Pass is the essence and the most complex part of backpropagation. It operates on the result of loss, applying calculus and linear algebra, to deduce how to update weights to make the next Forward Pass output less wrong results.

Once we get the gradient at hand, the little part of the backpropagation remains to complete - weights update. We refine W, U, and V using the same technique: weight matrix = weight matrix - learning rate * weight matrix gradient. Subtract happens ‘cause we should move in the direction of anti-gradient. Learning rate is a critical network hyperparameter that controls how large a step you take towards minimizing the loss function. Without this coefficient involved, the gradient descent might take too large steps, and it typically results in very slow convergence (training time increases too much).

Recap

In this entry, we’ve walked you through the model training process, backpropagation, and its foundational steps. Also, we got a glimpse of calculus terms that form the basis of ML. But let’s move past the dull theories; the true way to learn any data-related aspect is through hands-on experience.

It’s time to create something genuinely working! In the next session, let’s build an advanced and extensively trained text generation RNN model using cloud GPU, and evaluate the outcomes. Stay tuned!