In March 2023, GPT-4 could tell you whether a number was prime with 97.6% accuracy. By June of the same year, the same model name answered those same questions correctly 2.4% of the time. Nobody pushed a bad commit. No prompt changed in your repo. The thing behind the API just... moved.

That number comes from a Stanford and Berkeley study, "How...