Cutting ASR Inference Cost with NVIDIA MPS on Amazon EC2

When an ASR pipeline is pushed to production, the interesting question is not only how fast it runs, but how much throughput you can extract from each GPU before latency starts to break. In the setup described here, that tradeoff was the main lever for reducing inference cost by 75% using NVIDIA MPS on Amazon EC2. This post is a collaboration...