Tokens per second is a memory bandwidth number. Measure it in 2 seconds.

A 40B model that does 125 tok/s on a laptop, and a 7B model that does 1.5 tok/s on a 2011 CPU with the same amount of RAM. Same ballpark of memory, ~80× difference in speed. Almost every explanation you will read for that gap is wrong, and the wrong explanation costs people money. Here is the number that actually decides it: how many bytes your... Weiterlesen: Tokens per second is a memory bandwidth number. Measure it in 2 seconds.