How to Run an AI Model on Almost Any Hardware: A Practical Guide to Tiny Local LLMs

This month a developer ran a working language model on a Sony PSP-2000. That is a 333 MHz MIPS handheld from 2007 with 64 MB of RAM, and it is streaming English text onto its LCD at one to two tokens per second. The model is Karpathy's stories15M, int8-quantized down to about 17 MB, running on roughly 1,100 lines of pure C. A 64-token completion... Weiterlesen