What I Learned Quantizing DistilBERT to ONNX for Browser Inference

What I Learned Quantizing DistilBERT to ONNX for Browser Inference I built a support-ticket classifier — 77 banking intents, fine-tuned DistilBERT, Banking77 — and got it to 92.2% accuracy. Then I tried to ship it, and ran into the actual problem: the checkpoint was 256 MB and took ~9 ms per query on CPU. That's fine on a GPU server. It is not... Weiterlesen