The painful paradox of using AI for high-stakes decisions is that while it democratizes output, it actually increases the premium on deep domain knowledge, because you still need the expertise to spot when a reassuringly confident answer is completely wrong.
And that's the point myself and my guest contributors keep on circling: AI can make the first answer easier to get, but tou still need someone who knows when that answer is wrong.
"The expertise you need to check an AI's work is roughly the expertise you were hoping it would replace" is the cost that never makes the invoice. The working rule I take from it: the less I can judge an answer myself, the more it needs a second check and a named person who signs off, because the confidence reads the same whether it is right or wrong.
The "could I catch it if it were wrong?" question is the useful part. The point in this thread that some knowledge was never written down is why it's hard to answer honestly, because the person who can catch the error often holds that knowledge in their head. That makes the saving depend on them staying and being asked. Worth naming who that is in each process before anything gets automated.
That’s a pretty striking bit of evidence for this, Alicia. The error rate gets worse as the questions get harder, at exactly the point where the person asking is less able to call BS.
No one should rely on the general AI - you have highlighted the problem - it is specialisation. Only specialist AI can provide closest to the truth answers and solutions, or well instructed specialist AI that is based on detailed and trustworthy data. In the examples provided in the post people asked mass AI very specific industry related questions, hence the outcome.
Specialist AI definitely helps, but I’m not sure it solves the whole problem. Some of the knowledge in these examples simply wasn’t written down anywhere for the AI to learn from.
This is exactly the point - it has not been written in a public domain because it is a specialist knowledge. And if they used mass AI like ChatGPT it is trained on what is written and available freely. Some things are connected to very specific knowledge and experiences that cannot be found in public domain or they are not accessible for AI.
Yep, we’re pretty close on that. I think the bit I’d add is that some knowledge isn’t written down anywhere, public or private. It only exists because someone has done the job for years.
I think this is one of the better framings of this problem I've read, because "the expertise you need to check an AI's work is roughly the expertise you were hoping it would replace" is the actual trap, stated cleanly. Most AI advice skips straight to use cases and never asks who's qualified to catch a failure.
Daniel, curious when the answer sounds right and you can't argue with it, what your go-to next step typically is? Make it show its sources, leave that work alone, validate with a colleague?
The painful paradox of using AI for high-stakes decisions is that while it democratizes output, it actually increases the premium on deep domain knowledge, because you still need the expertise to spot when a reassuringly confident answer is completely wrong.
And that's the point myself and my guest contributors keep on circling: AI can make the first answer easier to get, but tou still need someone who knows when that answer is wrong.
Sometimes AI isn’t the answer. Great examples.
Thanks, Stephan. I agree, the useful part is knowing when AI is helping and when you need someone who’s done the job before.
"The expertise you need to check an AI's work is roughly the expertise you were hoping it would replace" is the cost that never makes the invoice. The working rule I take from it: the less I can judge an answer myself, the more it needs a second check and a named person who signs off, because the confidence reads the same whether it is right or wrong.
It feels backwards at first, but the stuff you know least about probably needs the most checking, not the least.
Very informative and interesting.
Big decisions still deserve someone who has done the job before.
Yep. Especially when getting it wrong is expensive.
The "could I catch it if it were wrong?" question is the useful part. The point in this thread that some knowledge was never written down is why it's hard to answer honestly, because the person who can catch the error often holds that knowledge in their head. That makes the saving depend on them staying and being asked. Worth naming who that is in each process before anything gets automated.
That’s a really good point. If the only person who can catch the mistake leaves, the AI can keep doing the job and nobody knows what’s gone missing.
It's also something two recent guest posts touched on:
• Your AI reports to nobody / Heather Baker https://millennialmasters.net/p/who-owns-your-ai-systems
• AI moved the hard part somewhere else / Om Prakash Pant 🔗 https://millennialmasters.net/p/ai-human-judgement
Thanks Daniel. Both posts describe the catch going missing. I just read a study this month about why it matters most on the hard questions.
AI models gave incorrect information 57% of the time on financial advice, and 88% of the time on advanced questions. That's where the person asking is least able to spot the error, as they don't know the answer either. https://ibsintelligence.com/ibsi-news/ai-chatbots-give-incorrect-financial-advice-57-of-the-time-study-finds/
That’s a pretty striking bit of evidence for this, Alicia. The error rate gets worse as the questions get harder, at exactly the point where the person asking is less able to call BS.
No one should rely on the general AI - you have highlighted the problem - it is specialisation. Only specialist AI can provide closest to the truth answers and solutions, or well instructed specialist AI that is based on detailed and trustworthy data. In the examples provided in the post people asked mass AI very specific industry related questions, hence the outcome.
Specialist AI definitely helps, but I’m not sure it solves the whole problem. Some of the knowledge in these examples simply wasn’t written down anywhere for the AI to learn from.
This is exactly the point - it has not been written in a public domain because it is a specialist knowledge. And if they used mass AI like ChatGPT it is trained on what is written and available freely. Some things are connected to very specific knowledge and experiences that cannot be found in public domain or they are not accessible for AI.
Yep, we’re pretty close on that. I think the bit I’d add is that some knowledge isn’t written down anywhere, public or private. It only exists because someone has done the job for years.
Absolutely!
I think this is one of the better framings of this problem I've read, because "the expertise you need to check an AI's work is roughly the expertise you were hoping it would replace" is the actual trap, stated cleanly. Most AI advice skips straight to use cases and never asks who's qualified to catch a failure.
Exactly. The key question is who knows enough to catch the bad answer before it goes any further.
Daniel, curious when the answer sounds right and you can't argue with it, what your go-to next step typically is? Make it show its sources, leave that work alone, validate with a colleague?
If something feels fishy about an AI answer and I can’t quite put my finger on why, I’ll usually:
1. Open a fresh chat and ask for a second opinion
2. I’ll ask for the sources and review key ones
3. And/or run the same question past another model to see where the answers differ