Stocks in US AI technology companies fell in value at the close of trading yesterday, with the NASDAQ Composite index down 1.4%. Among those losing value were Palantir, down 9.4% and Arm Holdings down 5%. According to the Financial Times [paywall], Tuesday saw the biggest one-day fall in the market since the beginning of August.
Some traders put the falls down to a report released [PDF] by an AI company, NANDA, which noted the high failure rate of many generative AI projects in commercial organisations. Project NANDA originated at the Massachusetts Institute of Technology Media Lab and describes itself as an organisation that’s building an “agentic web.” The paper has, since publication, been placed behind a survey wall, but is available for download from this site.
The research authors state only 5% of gen AI pilots reach production and actually produce measurable monetary value, with the vast majority of projects creating little impact on profit & loss metrics. The research undertaken by NANDA comprised of the content of 52 structured interviews with enterprise decision-makers, researchers’ analysis of 300+ public AI initiatives and announcements, and a survey questionnaire completed by 153 company leaders. It measured return on investment over six months after gen AI projects left pilot status.
While many organisations deploy AI in front-office or customer-facing business functions, successful projects tend to be found among back-office workflows, the paper says. It’s in the mundane tasks of the back office where savings are accrued, largely from a lowered need for third-party agencies and BPOs. The survey found there was little impact by AI projects on overall internal staff levels.
While 90% of staff stated they have personally benefited from using publicly-available AIs, typically in the form of large language models like ChatGPT, those subjective gains are not translated at institution level. Around 40% of the companies surveyed pay for a subscription to LLMs.
Many of the failed projects’ owners cited the lack of contextual awareness exhibited by generative AI models – that is, adapting to circumstances, changing over time, and remembering previous enquiries. NANDA states that forming a partnership with an organisation that can supply such a system and ensure it adapts to an organisation’s specific circumstances is the critical element for success. The paper highlights several quotes “derived from interviews” that include between 60%-70% agreeing with the statements, “[The AI system] doesn’t learn from our feedback,” and “Too much manual context required each time.”
The vertical most positively affected by gen AI was media & telecom, followed by professional services, healthcare & pharma, consumer & retail, and financial services. The energy & materials sector’s rate of generative AI project launch is currently negligible, the paper says. In terms of business units, sales & marketing is where most projects are or were based, with finance & procurement least popular as a place where AI projects might be begun.
The area in a typical organisation where generative AI is deployed most is in sales & marketing, with finance and procurement being the least popular site. And complex tasks are those least likely to be expected to be completed by AI; managers would assign projects like client management to an AI only 10% of the time, while tasks like summarising a report or writing an email would go to a human on 70% of occasions.
The language of the published report and its lack of academic rigour suggest that its provenance and purpose are more akin to marketing than intellectual and technological discussion. The paper’s authors urge for strategic partnerships with a knowledgeable vendor to increase the chances of generative AI projects’ success, a partnership which NANDA is, purely coincidentally, able to form one half of. There are “unprecedented opportunities for vendors who can deliver learning-capable, deeply integrated AI systems,” the paper’s conclusions state.
The headlines from the NANDA report make for sobering reading among decision-makers tasked with generative AI implementations, yet the paper’s underlying messages are weakened by the intentions behind its publication. Stock prices this week could have been affected by partisan surveys from authors with obvious skin in the game, but it seems more likely that the NANDA publication simply reflects trading floors’ concerns about generative AI’s practical effectiveness as a business tool.
(Image source: “Arthur Daley” by Tim Dennell is licensed under CC BY-NC-ND 2.0.)