Deliver Completions With a Job Row, Not a Held Connection

Last Tuesday I sat with a teammate who clicked Generate on a supposedly simple report screen. The spinner behaved like a prayer, nginx hit sixty seconds, and the gateway returned 504 while a model process kept talking to nobody. Did the user get a paragraph? No. Did a refresh spend another free inference call on the same prompt? Yes, immediately,... Weiterlesen