I Built a Neural Network Inference Engine From Scratch in C++ (No PyTorch, No ONNX, Just AVX2)

Why does inference need a framework at all?

Every time I ran a tiny linear model through PyTorch, I felt like I was driving a go-kart with a jet engine strapped to it. The model was a few hundred KB. PyTorch's runtime was gigabytes. Somewhere between model(x) and the actual floating-point math, an entire universe of abstraction — autograd graphs, dispatch layers, tensor metadata — was quietly eating my CPU cycles.

So I asked a simple question: what does inference actually look like with nothing in the way?

That question turned into

Linkedin: www.linkedin.com/in/shaurya-aditya-0563a0377

Shaurya Aditya — B.Tech ECE, IIT BHU