GPT generates text by predicting the next word. It reads left to right.

BERT does something different. It masks random words in a sentence and tries to predict what they are. To do that well, it has to understand every word in relation to every other word simultaneously. Left and right context both matter.

That bidirectional understanding is why...