Why Speculative Decoding Slows Hybrid Mamba Models at Large Batch Sizes

Engineer's view. This is a database that rewrote the whole table file on every transaction so it could roll back. Real databases append each change to a log, checkpoint now and then, and roll back by truncating the log. A Mamba layer can store its state the same way. TL;DR. A Mamba layer overwrites its state every token, so undoing a rejected... Weiterlesen: Why Speculative Decoding Slows Hybrid Mamba Models at Large Batch Sizes