I’ve used AI to do more work than I expected to this year, and one problem hasn’t gone away: knowing when it’s wrong.
I can catch a bad edit or an automation misbehaving because I know how it should work and when the output looks wrong. But when I use AI in an area I don’t know that well, there’s no safety net.
I’m relying on an answer I’m not really qualified to challenge, especially when ChatGPT or Claude can sound so reassuringly confident and convincing.
Galina Fendikevich spent years working in AI in Silicon Valley and on Wall Street before launching her podcast, Artificial Reality. Now she asks operators across industries about the AI they use and what they’d pay for that nobody has built yet.
Because nobody on her show sells AI, they answer honestly. And in those conversations, she spotted the cost nobody puts on the AI invoice: you still need enough expertise to catch a bad answer. 👇🏻
A husband-and-wife team walked into an M&A meeting holding a $5 million valuation of their business. It had made $125,000 in net profit that year, with no intellectual property or real estate. Claude had produced the number, and they believed it because nothing in the report suggested they shouldn’t.
Christine McDannell, who runs the sell-side M&A firm The Magnolia Firm, knew it was wrong before she finished reading. She told them the multiple for that business was three to four times net profit, emailing them before the call because some things shouldn’t be said to someone’s face on Zoom.
Then she read the report behind the number. Claude had assigned a dollar value to every department in the company based on payroll. It valued the accounting department at $200,000. It had counted expenses as assets. That’s how it arrived at $5 million.
The real problem is who can catch it
Christine caught it in seconds because she values businesses for a living. The sellers couldn’t, because they don’t. They’d used a capable tool, presumably prompted it well, and received a confident, well-formatted document about the most consequential financial decision of their lives.
They’d bought precisely the thing they had no way to check. It points to a cost of this technology almost nobody prices in:
The expertise you need to check an AI’s work is roughly the expertise you were hoping it would replace.
Follow that where it goes. The safest uses are the Tuesday admin you could’ve done yourself, because you’d spot an error instantly.
The risk rises when AI moves into a once-a-decade decision in a field you don’t practise, where you may never know it got something wrong.
The failures rhyme
A charter client told his captain it was safe to anchor in 25-knot winds. He was certain because ChatGPT had said so. He’d screenshotted the answer and circled it in pen before the conversation started.
Jennifer Kerum, who runs a luxury yacht charter in Croatia, was the one who had to repair that relationship afterwards, not the model. Her line: “I don’t care what ChatGPT says. You need to trust the captain.”
Why did the model say yes? Because whether it’s safe to anchor depends on the seabed, the shelter of that particular bay, the boat, the crew and the forecast at four in the morning. That knowledge lives in captains. It isn’t written down anywhere on the internet.
Nikki Siso opened a restaurant in Austin. She ran a 45-page lease through ChatGPT, got eight risk flags in three minutes, and had her lawyer review it in 10 minutes for free instead of billing her $1,000. Steal that one, it works.
But nothing told her the sequence: that the mechanical, electrical and plumbing engineer has to come before the permit application. That cost her a month and $30,000. The sequence isn’t written down either. It lives in the head of the inspector, the contractor and whoever opened a restaurant in that town last year.
Sofiane Ghorbel, with 25 years in luxury hospitality, gave me the sentence I’ve quoted in every talk since: “We don’t sell rooms, we sell trust.” I asked him about Marriott’s RENAI virtual concierge, which struggled with a guest’s request for a nearby Italian restaurant with bar seating, so the guest was sent back to the human concierge. Sofiane said AI was already proving more useful behind the scenes in forecasting, revenue management and operations.
Line the cases up and the shape is hard to miss:
Across the seven industries I looked at, every AI win was invisible to the customer. Every failure was visible.
The knowledge AI doesn’t have
I spent two and a half years deep in Silicon Valley. The engineers I worked with were exceptional, but they’d never done the jobs they were building for. They hadn’t walked a factory floor or sat with someone being told what their life’s work is worth.
So the tools ship generic answers. They can cover a huge amount of ground without having the experience that comes from doing the job.
Those legacy industry firms doing twenty million a year on sticky notes have looked at the offer and decided it isn’t good enough.
The audit, which takes an afternoon
You can use plenty of AI as long as you’re pointing it at work you can defend.
Start by asking yourself:
Could I catch it if it were wrong? If not, you’ve got a bet taken with confidence the vendor supplied.
Is the output visible to a customer, or only to me? Start in the back of house, where a bad output can be caught before a customer sees it.
What does the error cost and who pays it? A wrong draft costs a minute. A wrong valuation can cost someone their retirement, and the person paying isn’t you.
Then run this exercise against your processes:
Map the work. Write down the actual contents of a week. Every recurring task, however trivial it feels. Most people haven’t done this, and the list runs longer than they expect.
Split it in two. Put the work that needs your judgement in one column and the manual work in the other. Chasing an invoice is manual. Deciding who gets an extension needs judgement. Formatting a report is manual; knowing which number matters takes expertise. Be honest, because the column that flatters you is the one you’ll get wrong.
Go shopping in the manual column. That’s your automation list because you can check the output at a glance. Run the same test every time: could I catch it if it were wrong?
Keep the judgement work with you
It’s the part you’re qualified to verify and the part people are actually paying you for.
Automate the annoying parts without mercy
The chasing, formatting, transcribing and seventh version of the same email.
That’s where the hours actually are, and none of it asks you to be right about something you’ve never studied.
👤 Galina Fendikevich hosts Artificial Reality, where she talks to operators about how they’re actually using AI in their businesses. She previously worked in machine learning on Wall Street, co-founded SpotX Games, whose team joined Niantic in 2022, and led US go-to-market at AI company Upstage. Connect with Galina on LinkedIn.
The painful paradox of using AI for high-stakes decisions is that while it democratizes output, it actually increases the premium on deep domain knowledge, because you still need the expertise to spot when a reassuringly confident answer is completely wrong.
Sometimes AI isn’t the answer. Great examples.