AI Mistakes Are Very Different from Human Mistakes

Bruce Schneier and Nathan E. Sanders, writing in a post: Someone who makes calculus mistakes is also likely to respond "I don't know" to calculus-related questions. To the extent that AI systems make these human-like mistakes, we can bring all of our mistake-correcting systems to bear on their output. But the current crop of AI models -- particularly LLMs -- make mistakes differently.

AI errors come at seemingly random times, without any clustering around particular topics. LLM mistakes tend to be more evenly distributed through the knowledge space. A model might be equally likely to make a mistake on a calculus question as it is to propose that cabbages eat goats. And AI mistakes aren't accompanied by ignorance. A LLM will be just as confident when saying something completely wrong -- and obviously so, to a human -- as it will be when saying something true. The seemingly random inconsistency of LLMs makes it hard to trust their reasoning in complex, multi-step problems. If you want to use an AI model to help with a business problem, it's not enough to see that it understands what factors make a product profitable; you need to be sure it won't forget what money is.

[...] Humans may occasionally make seemingly random, incomprehensible, and inconsistent mistakes, but such occurrences are rare and often indicative of more serious problems. We also tend not to put people exhibiting these behaviors in decision-making positions. Likewise, we should confine AI decision-making systems to applications that suit their actual abilities -- while keeping the potential ramifications of their mistakes firmly in mind.

Read more of this story at Slashdot.