Automation may lead to better documentation of how decisions are made. But it cannot capture every lesson someone has built through years of experience. Conditions change, and someone still needs to decide when the usual approach no longer makes sense.
I’ve made calls where I couldn’t have written down the rule beforehand. Something just felt off because I’d seen a similar mess years earlier. That’s the part I’m wary of handing over.
I think this is where it gets tricky. We can document more and more of the decision process, but sometimes we only realise which bit of experience mattered after something goes wrong.
And by then, the person who would have caught it may not even be in the loop.
AI can scale information, but it can’t always replace the context someone has built through years of experience, especially when things don’t fit the expected pattern.
Yes. I think that’s where the experience matters most. If everything looks normal, the system can do a lot & the difficult ones are when the recommendation looks right, but something doesn’t quite add up.
The work gets faster, but reviewing it properly doesn't necessarily get faster with it. You can end up with a lot more output, while the hard part becomes knowing which parts actually need to be questioned.
AI can automate decisions without automating judgment, which makes capturing tacit knowledge and building human oversight essential to scaling reliable systems.
This is basically why so many AI implementations quietly fail after the pilot stage. The model performs fine on clean historical data, then hits an edge case that the one retiring expert would've flagged in two seconds, and nobody in the room knows why the output looks off. I'd add that the real fix isn't documentation, it's building review checkpoints where that expert's judgment gets tested against the model's output before they leave, not after.
This makes me think the last few weeks of a handover should involve deliberately trying to trip the system up. Better to find out what it doesn’t know while the person with the answer is still sitting there.
Thank you for this. The pricing manager story illustrates it better than any framework could. His replacement inherited the output without the months of testing behind it.
Automation may lead to better documentation of how decisions are made. But it cannot capture every lesson someone has built through years of experience. Conditions change, and someone still needs to decide when the usual approach no longer makes sense.
I’ve made calls where I couldn’t have written down the rule beforehand. Something just felt off because I’d seen a similar mess years earlier. That’s the part I’m wary of handing over.
I think this is where it gets tricky. We can document more and more of the decision process, but sometimes we only realise which bit of experience mattered after something goes wrong.
And by then, the person who would have caught it may not even be in the loop.
AI can scale information, but it can’t always replace the context someone has built through years of experience, especially when things don’t fit the expected pattern.
Yes. I think that’s where the experience matters most. If everything looks normal, the system can do a lot & the difficult ones are when the recommendation looks right, but something doesn’t quite add up.
We're seeing a similar pattern with AI and remote teams.
The output gets faster.
The first draft gets easier.
The research takes less time.
But someone still needs enough context to review the work properly.
AI doesn't eliminate the need for good people.
It raises the cost of having people who don't understand what they're looking at.
This is probably one of the less obvious changes.
The work gets faster, but reviewing it properly doesn't necessarily get faster with it. You can end up with a lot more output, while the hard part becomes knowing which parts actually need to be questioned.
I’ve had AI give me work that looked perfectly fine and was still completely off.
If nobody knows the subject well enough to catch that, you’re in trouble.
AI can automate decisions without automating judgment, which makes capturing tacit knowledge and building human oversight essential to scaling reliable systems.
Exactly. The awkward cases are usually where the real experience lives, and they’re rarely the ones anyone bothered to document.
This is basically why so many AI implementations quietly fail after the pilot stage. The model performs fine on clean historical data, then hits an edge case that the one retiring expert would've flagged in two seconds, and nobody in the room knows why the output looks off. I'd add that the real fix isn't documentation, it's building review checkpoints where that expert's judgment gets tested against the model's output before they leave, not after.
This makes me think the last few weeks of a handover should involve deliberately trying to trip the system up. Better to find out what it doesn’t know while the person with the answer is still sitting there.
AI moved the research work but not the part that knew when something looked wrong.
Yep. The system can do the legwork. I still want someone there who’s seen enough weird cases to know when the answer smells wrong.
Thank you for this. The pricing manager story illustrates it better than any framework could. His replacement inherited the output without the months of testing behind it.
That’s what got me too. The new manager had the same system in front of him, but none of the history of when it had been wrong before.