Fail Closed, Not Open: Designing an AI Gateway for Regulated Enterprises

"Why did this request go to that model?"

If your company uses LLMs across more than one team, someone will eventually ask you this question. If you work in financial services, healthcare, or anywhere else with a regulator, someone will ask it under oath-adjacent conditions.

Most organizations can't answer it. Teams integrate directly with whatever provider they like, sensitive data handling varies by team, regional restrictions are applied inconsistently, and there's no record of why a model was selected — or, more importantly, why the alternatives weren't.

I recently built . The scenario I'd point you at first: mark every compliant model as down and watch it refuse to route while healthy external models sit right there. That refusal is the entire architecture in one response.

What's your organization's answer to "why did this request go to that model?" I'd genuinely like to hear how others are handling this — especially the task-classification contract, which I think is the least-settled part of this space.