🔧 Building a Tokenizer from Scratch [part 2]

Parser Theory: Q/A with Claude Opus


In part 1, we built a working FSM that recognizes <div>text</div> using just 7 primitives mapped 1:1 to assembly opcodes. But FSMs have a hard limit: they can't handle nested structures like <div><div>hello</div></div>. In this post, we climb the Chomsky hierarchy from...