The promise of hyper-efficient, AI-powered customer service is facing a significant reality check. Despite heavy investment from tech giants like Amazon and Salesforce, new research from July 2025 reveals that AI assistants in call centers often create more work for their human counterparts.
This finding is bolstered by a recent Gartner forecast scaling back predictions of AI replacing human agents. High-profile failures, such as a support bot inventing company policy, underscore the growing gap between the technology’s marketing and its real-world performance.
These flawed systems can burden employees, damage trust, and alienate the very customers they are meant to serve. The core issue is the disconnect between promised automation and the reality of human oversight still being essential.
AI’s Efficiency Promise vs. The Messy Reality
A recent academic study highlights this gap. Research accepted to the ACM CSCW 2025 conference examined AI assistants at a Chinese utility’s call center. The findings were stark. The AI struggled with basic tasks like transcribing accents and numbers accurately.
Specific failures included misinterpreting homophones, breaking up phone numbers into useless fragments, and failing to understand callers with strong regional accents. The AI’s emotion-recognition system was also found to be unreliable, often misclassifying normal speech as negative.
This forced human agents into a new role: constant fact-checkers. They had to manually correct inaccurate summaries and delete redundant text. The study’s authors noted this created new “learning” and “compliance” burdens for the customer service representatives (CSRs).
The researchers concluded that “The mismatch between technological expectations and actual implementation reflects a common oversight among technology designers, who overestimate efficiency gains while underestimating the implicit learning burdens of adapting to new systems.” This hidden labor directly contradicts the efficiency gains that are often the primary justification for adopting such expensive AI systems in the first place.
When Good Bots Go Bad: The High Cost of Hallucinations
The risks extend beyond inefficiency and into brand damage. In April 2025, AI code editor company Cursor experienced this firsthand when its own support bot “hallucinated” a fake policy. The bot incorrectly told users their subscriptions were limited to a single device.
The false information spread rapidly, causing user backlash. Cursor co-founder Michael Truell had to issue a public apology on platforms like Hacker News, confirming, “We have no such policy.” He attributed the error to their “front-line AI support bot.”
To compound the issue, a genuine technical bug related to session security was causing separate login problems, creating a perfect storm of user frustration. The bot’s confident but false answers only amplified the chaos.
The incident serves as a cautionary tale. Simply labeling AI responses is not enough to let users know that a response was generated by AI is an inadequate measure to recover user loyalty.
From Replacement to Reinforcement: A Shifting Industry Consensus
This blend of academic findings and public failures is fueling a broader shift in industry thinking. In June 2025, analyst firm Gartner made a significant revision to a previous forecast. It now predicts that by 2027, half of all organizations that expected to replace support staff with AI will abandon those plans.
The new consensus points toward a hybrid model. Kathy Ross, a senior director analyst at Gartner, argues that “A hybrid approach, where AI and human agents work in tandem, is the most effective strategy for delivering exceptional customer experiences.” This approach leverages AI for what it does well—data retrieval and routine tasks—while preserving the essential human element for complex or sensitive interactions.
This pivot acknowledges that the initial vision of fully autonomous, human-free customer service was premature. The focus is now on creating tools that augment human agents, making them more effective rather than making them obsolete.
Big Tech’s Unabated Push into AI Agents
Despite these challenges, major technology firms are not slowing down. They continue to market and develop sophisticated AI agent platforms, framing them as the future of digital labor. Salesforce has been particularly aggressive with its Agentforce platform.
In May 2025, the company launched Agentforce for HR Service, designed to automate internal employee support. This followed the general availability of the core Agentforce platform in late 2024.
Salesforce’s Kishan Chetan, EVP & GM of Service Cloud, claimed, “Salesforce is leading this digital labor revolution across industries, and now, we’re bringing Agentforce skills and actions to HR Service so companies can expedite requests and scale employee support when resources are limited.” The company even touts a 96% self-service resolution rate from its own internal deployment, positioning itself as a primary user of its own technology.
Amazon is also a key player, having enhanced its Amazon Connect platform with new generative AI features. Meanwhile, well-funded startups like Decagon, which raised $35 million in June 2024, are also vying for a piece of the growing market. The central conflict remains unresolved: a market flush with investment and ambition, yet a technology still struggling with fundamental reliability.