Once AI starts making recommendations, someone still has to know when the answer doesn’t quite add up.
A lot of that comes from experience that was never written down in the first place. Sometimes you only realise how much one person knew until they’re gone.
Om Prakash Pant writes Between Roles, drawing on his experience across AI, technology, consulting, products and retail.
His guest post looks at what happens when the system keeps running after the person who understood all the quirks has moved on. 👇🏻
A clothing retailer asked us to build a promotion analyser.
The job sounded straightforward: scrape competitor websites, pull out their discounts and help the pricing team understand what was happening in the market.
Their pricing manager had been doing much of that manually. Browsing competitor sites, checking prices and watching seasonal promotions took two or three hours every week. We built a system to handle it.
The pricing manager helped shape the POC for three or four weeks, then worked with us for another two or three months as we refined the system. He could see which recommendations made sense and which needed another look.
Then he moved to another store. That was when we discovered how much of the system still depended on what he knew.
New here? Millennial Masters shares lessons from founders who’ve built and grown companies. Subscribe for ideas on AI, leadership and the decisions that rarely come with a playbook.
What the system couldn’t see
The system could see competitor prices. The pricing manager knew what those prices meant for this particular business.
He knew the seasonal baseline. If the store sold a certain number of winter coats last year at a particular margin, he had a feel for what a realistic increase might look like this year.
He also knew where the data could mislead us. At one point, the system recommended buying 1,500 units of a particular T-shirt colour for the next 15 days.
The confidence score was high. The manager wasn’t convinced. He had already seen the recent orders and knew the recommendation didn’t fit.
The same happened with women’s office wear. The system saw strong demand and suggested increasing quantity.
When the manager looked closer, the spike came from a promotion that bundled the product with other items. It didn’t mean underlying demand had increased.
That judgement had been built through experience. Much of it had never been written down because one person already knew it.
What left with the manager
The replacement manager came from a different region, with different weather and buying patterns.
He didn’t automatically trust the system. He wanted its recommendations checked against historical data before trusting them.
That mattered because the original manager had spent months working with the POC. He had seen where it behaved well and where it needed another look. The new manager inherited the output without having gone through that same learning process.
Their approach to margin showed the difference. The original manager had become comfortable testing some promotions at profit margins of around 1–3%.
The new manager kept profit margins closer to 3–6% while the recommendations were being validated. He was more cautious because he didn’t yet trust the POC in the same way.
The system itself also had limits. It included some procurement-cost data, but it didn’t yet contain the full cost picture. Delivery costs and wider sourcing inputs were due to be added later.
The recommendation could still help, but someone needed to understand what wasn’t in it yet.
Where the work went
The visible task had largely disappeared. Someone no longer had to spend two or three hours every week browsing competitor websites and manually collecting prices.
The new manager had to compare recommendations with historical data. Finance still needed to understand the effect on margin. Inventory still had to judge whether the quantities made sense.
Somebody still had to decide whether a recommendation that looked reasonable on screen actually made sense for that store.
The research was largely automated. People still had to judge whether the recommendations made sense for that store. That distinction only became obvious when the original manager moved.
What to capture before you automate
When someone has done the same job for years, plenty of their knowledge never appears in a process document.
They recognise when a spike looks odd because they remember what happened after the last promotion. They can also tell when a number looks plausible on screen but still deserves another look.
AI forces some of that knowledge into the open.
If the person teaching the system leaves before you’ve captured enough of it, the task may still run. The next person then has to work out how much to trust the answer.
That’s what happened here. The scraper worked and the competitor research took far less time.
What didn’t transfer as easily was the confidence the original manager had built over months of testing the recommendations against what he already knew about the store.
When he moved, the next manager had to rebuild some of that confidence for himself.
The task was easier to hand over than the judgement around it.
Who still knows when this recommendation is wrong?
👤 Om Prakash Pant writes Between Roles, where he explores AI, technology, consulting, products and retail through the experience of someone who’s worked across them. His writing focuses on what changes when new technology meets the messy reality of how work actually gets done. Connect with Om on LinkedIn.
Automation may lead to better documentation of how decisions are made. But it cannot capture every lesson someone has built through years of experience. Conditions change, and someone still needs to decide when the usual approach no longer makes sense.
AI can scale information, but it can’t always replace the context someone has built through years of experience, especially when things don’t fit the expected pattern.