But, first, consider the blockbuster first sentence of the 26-page MIT study: “Despite $30–40 billion in enterprise investment into GenAI, this report uncovers a surprising result in that 95 percent of organizations are getting zero return.”
The conclusion was made after researchers spoke to 153 senior company leaders, reviewed 300 publicly disclosed AI projects, and interviewed an unspecified number of workers at 52 organizations, the report said. Coauthors Pradyumna Chari and Ramesh Raskar at MIT did not respond to emails seeking further comment. Two other coauthors could not be reached.
But the alleged 95 percent failure rate appears to be based on some strange math that reminded me of the old trick of counting backwards how many fingers are on one hand to fool a child into thinking they have 11 fingers.
In talking to people at the 52 companies, the researchers found that only 5 percent reported productivity improvements or cost savings from using customized AI apps specially created for their projects. If 5 percent succeeded, that would mean 95 percent failed?
But only 60 percent of companies in the survey said they considered using custom AI tools and only 20 percent piloted an actual project. If 95 percent failed out of the 20 percent who tried, the vast bulk of companies in the survey never started a customized AI project. A more accurate summary might report that 19 percent of companies failed and 80 percent didn’t try.
The study’s conclusion looks even more questionable when considering that more of the companies surveyed decided to use off-the-shelf AI apps like Microsoft Copilot or ChatGPT than custom apps for their projects. Using those standard AI apps, 80 percent considered a project, 50 percent piloted a project, and 40 percent successfully implemented their projects. But those findings were dismissed as not affecting bottom-line profits and ignored at the beginning of the report.
But that might help explain the conclusions of the Stanford study, which analyzed anonymous payroll data from a few million workers at tens of thousands of employers over several years. The data, from payroll administrator ADP, provided a window into how many people were working in different kinds of jobs as AI entered the workplace.
Over the past three years, in jobs “highly exposed” to AI, such as programmers and customer service reps, the number of younger workers declined while the number of older workers continued to grow, the study found. In fields less exposed to AI, such as stock clerks and health aides, the number of younger workers was steady or grew.
The results held even when the researchers accounted for changes in interest rates or possible over-hiring at the beginning of the pandemic. And the trend hurting some younger workers was not evident in the four years before ChatGPT emerged at the end of 2022.
“Since the widespread adoption of generative AI, early-career workers (ages 22-25) in the most AI-exposed occupations have experienced a 13 percent relative decline in employment,” the study noted. But “employment for workers in less exposed fields and more experienced workers in the same occupations has remained stable or continued to grow.”
While this doesn’t prove AI is eating jobs, it’s an important point that job seekers, policymakers, and future researchers will need to consider.
Aaron Pressman can be reached at aaron.pressman@globe.com. Follow him @ampressman.