What You'll Build


The attention mechanism: the only place in a transformer where a token at position t gets to look at tokens at positions 0..t-1. This is specifically self-attention, where the token attends to other tokens in the same sequence. (You might encounter "cross-attention" in other materials, which is used in encoder-decoder models...