AI startup Anysphere has been riding high over the past two months, thanks to the skyrocketing popularity of its AI-powered software coding assistant, Cursor. The company, which was eyed for acquisition by OpenAI and has reportedly been in funding talks for a valuation of nearly $10 billion, hit $100 million in annual revenue since Cursor launched in 2023. But this week, Cursor went viral for all the wrong reasons: Its customer support AI went rogue, triggering a wave of cancellations and serving as a cautionary tale for other startups betting big on automation.
It all started earlier this week, when a Cursor user posted on Hacker News and Reddit that customers had started getting mysteriously logged out when switching between devices. Confused, they contacted customer support, only to be told in an emailed response from “Sam” that the logouts were “expected behavior” under a new login policy.
But there was no new policy—and no human behind the support email. The response had come from an AI-powered bot and the new policy was a “hallucination,” an entirely made-up explanation.
The news spread rapidly in the developer community, leading to reports of users cancelling their subscriptions, while some complained about the lack of transparency. Cofounder Michael Truell finally posted on Reddit acknowledging the “incorrect response from a front-line AI support bot” and said it was investigating a bug that logged users out. “Apologies about the confusion here,” he wrote.
But the response seemed like too little, too late—with the AI support bot hallucination leaving Cursor seemingly flat-footed. Fortune contacted Cursor for comment but did not receive a response.
LinkedIn tech influencers are already using the example to warn other startups. “Cursor…just landed itself in a viral hot mess because it failed to tell users that its customer support “person” Sam is actually a hallucinating bot,” Cassie Kozyrkov, an AI advisor and Google’s former chief decision scientist, wrote in a LinkedIn post. “This mess could have been avoided if leaders understood that (1) AI makes mistakes, (2) AI can’t take responsibility for those mistakes (so it falls on you), and (3) users hate being tricked by a machine posing as a human,” she continued.
While customer support has been considered a top use case for generative AI ever since OpenAI’s ChatGPT chatbot debuted in 2022, it is also fraught with risks, said Sanketh Balakrishna, an engineering manager at cloud security platform Datadog. Customer support requires a level of empathy, nuance, and problem-solving that AI alone currently struggles to deliver, he explained. “I would be cautious about depending entirely on an AI bot for addressing user issues,” he told Fortune by email. “If I know that the only support available was an AI system prone to occasional errors, it could undermine confidence and impact our ability to resolve problems effectively.”
Amiran Shachar, CEO of security company Upwind, pointed out that this isn’t the first time hallucinating AI chatbots have made headlines. Air Canada’s AI chatbot told customers about a refund policy that didn’t exist, while payment service provider Klarna did an about-face on replacing all of its human customer service agents in February after complaints about its AI chatbot’s hallucinations. “Fundamentally, AI doesn’t understand your users or how they work,” he told Fortune by email. “Without the right constraints, it will ‘confidently’ fill in gaps with unsupported information.” For highly-technical users like Cursor’s developer customers, “there’s no margin for sloppy explanations.”
One Cursor customer agreed: Melanie Warrick, co-founder of healthcare AI startup Fight Health Insurance, said she found that Cursor offers great support to developers, but also growing friction. “Last week, I hit a persistent agent error (“try again in a few minutes”) that never cleared,” she told Fortune by email. “Support gave the same canned, likely AI-generated response multiple times, without resolving the issue. I stopped using it — the agent wasn’t working, and chasing a fix was too disruptive.”
For now, headline-making hallucinations have been limited to AI chatbots, but experts warn that as more enterprises adopt autonomous AI agents, the consequences for companies could be far worse. That’s particularly true in highly-regulated industries like healthcare, financial, or legal—such as a wire transfer that the counterparty refuses to return or a miscommunication that impacts patient health.
The prospect of hallucinating agents “is a critical piece of the puzzle that our industry absolutely must solve before agentic AI can actually achieve widespread adoption,” said Amr Awadallah, CEO and cofounder of Vectara, a company that offers tools to help businesses reduce risks from AI hallucinations.
While not the fault of an autonomous AI agent completing tasks, Cursor’s snafu is “exactly the worst-case scenario” that is slowing adoption of agentic AI, he told Fortune by email. Businesses love the idea of autonomous AI agents that can take action on their own, thereby saving on labor costs. But, he said, companies are also “extremely worried that one incident of hallucination, paired with a disruptive or even destructive action by the AI, could deeply impact their business.”
This story was originally featured on Fortune.com