A funny thing happened on the way to AGI: Model ‘supersizing’ has hit a wall

Welcome to AI Decoded, Fast Company’s weekly newsletter that breaks down the most important news in the world of AI. You can sign up to receive this newsletter every week here.

Model ‘supersizing’ may have hit a wall

The impressive intelligence gains of large models like OpenAI’s GPT-4 and Anthropic’s Claude came about after researchers figured out that they could reap progressively and predictably better intelligence by increasing the size of the models, and training them with more data and more computing power for longer periods of time. That realization yielded some impressive chatbots and coding assistants.

The big question now is how far the “supersizing” approach can carry researchers toward the holy grail of artificial general intelligence (i.e., AI that’s generally smarter than human beings). In fact, newly departed OpenAI cofounder Ilya Sutskever recently told Reuters that the jig is up for massive scaling. Similarly, billionaire investor and accelerationist Marc Andreessen said on his podcast with Ben Horowitz that the AI researchers he talks to are hitting the limits of massive scale.

“If you look at the improvement from GPT-2 to GPT-3 to 3.5, and then compare that from like 3.5 to 4, you know we really slowed down in terms of the amount of improvement,” Horowitz says.

And the gains from scaling are getting more and more expensive, he points out. “The thing to note is that the GPU increase was comparable, so we’re increasing GPUs at the same rate but we’re not getting the intelligence improvements at all out of it.” Pretraining a large model can cost upwards of $100 million, and Anthropic CEO Dario Amodei says that number could hit a billion dollars in the near future. And, of course, those Nvidia GPUs are expensive—the new H200 chips cost up to $40,000—and in high demand.

Several AI researchers have told me that the research is shifting away from scaling up computing power to focusing on the composition and quality of the training data. When an AI company puts together a corpus of data to train a new model, it sources it from thousands of contractors, publishers, and providers of domain-specific data such as healthcare data. It even hires doctors, lawyers, and other professionals to add expert labels to the data. The company also goes to providers of synthetic data to fill in gaps in the training corpus.

Most importantly, researchers are looking at new places outside of the training process to add computing power. An OpenAI researcher who worked on the company’s new o1 model tells me that his team has been spending a lot more computing power at inference time, when the model has already been trained and deployed and is processing user queries. For difficult, multistep questions, he says, the model is trained to pursue an answer in a more human, trial-and-error fashion (pursuing promising lines of reasoning, hitting a wall, backtracking, trying something else). All the while the model is storing and reprocessing all the data from the process, which requires lots of computing power.

Not everybody agrees that brute-force scaling has reached diminishing returns. Some argue that it’s too early to say, because we’ve yet to see the size and performance of OpenAI’s forthcoming GPT-5 model, or xAI’s forthcoming Grok-3 model. If those models end up costing far more to train than earlier models and yield unimpressive intelligence gains, the industry might truly be facing an inflection point.

Do Trump and Musk intend to fire, rehire, and rewire the Pentagon?

As promised, President-elect Donald Trump tapped Elon Musk (along with Vivek Ramaswamy) to co-lead a new government agency called the Department of Government Efficiency (DOGE).

So far, there’s been no indication that the Pentagon might be exempt from changes suggested by Musk and Ramaswamy. This could include mass firings. Trump has called for the termination of up to 50,000 career civil servants working within government agencies and replacing many of them with Trump loyalists. The Defense Department employs 24,000 civilian and military personnel.

DOGE’s work could affect Silicon Valley companies in a number of ways. Regulations might be rolled back, and the government’s ability to enforce the ones that survive could be neutered (the Federal Trade Commission, for example, could lose its ability to bring antitrust cases). It could also affect Silicon Valley’s relationship with the Pentagon, which has been gradually improving as the defense establishment works to build an industrial base that includes tech and AI suppliers. The Valley itself has arguably warmed to the idea of selling tech for defense over the past five years after Project Maven imploded at Google. But it remains a work in progress. Here’s how former Google CEO and defense tech investor Eric Schmidt explained it to me back in 2021:

“Remaking the Pentagon in Silicon Valley’s image will be far more difficult a public-private challenge than, say, Uber and Lyft steamrolling municipal taxi commissions. You have to think about the military as a large bureaucracy that has existing relationships with primes, and they all live in a symbiosis that makes sense to them.”

Starting with Defense Secretary Ash Carter in the 2010s, the Pentagon tried numerous times to modernize its technology procurement practices, with the goal of funding new defense tech faster, and giving the tech supplier some leeway to define the product’s capabilities. But these efforts have had limited success. Less than 1% of the Pentagon’s $411 billion in contract awards last year went to venture-backed defense tech companies.

Can Musk bring about such a broad cultural change within the Pentagon? (His companies are already major suppliers to U.S. defense and intelligence agencies.) There are signs he’d like to try. Late on election night he tweeted to Anduril’s Palmer Luckey: “Very important to open DoD/Intel to entrepreneurial companies like yours. Pay for outcomes, not requirements documents!”

Perhaps he intends to do it with the same kind of scorched earth “efficiency” that he brought to Twitter in 2022. Musk fired 80% of the company’s workforce, precipitating a dramatic fall in content moderation and general civility, and an exodus of paying advertisers.

That’s just one of the Pentagon’s worries. Trump has nominated Pete Hegseth, Fox & Friends cohost and Iraq vet with scant military leadership experience, to be his defense secretary. Trump’s transition team is also reportedly considering an executive order that would establish a “warrior board” of retired military personnel who would review three- and four-star generals, and recommend firing those they deem unfit for leadership. The president-elect has repeatedly called U.S. generals “woke” and “weak.”

Is China pulling away in robotics?

China trails the U.S. in the development of large foundational AI models, but it may be pulling ahead in making robots. Chinese companies appear to be making robots that have arguably better utility and awareness of the world than the robots from U.S. companies such as Figure, Tesla, and Boston Dynamics.

The latest case in point is a DEEP Robotics robot that can roll or climb on two legs or four, with enough dexterity to quickly navigate up and down a rocky mountainside. DEEP Robotics, which is based in Hangzhou, Zhejiang province, primarily supplies quadruped robots (similar to Boston Dynamics’ Spot robot dog) for monitoring or securing infrastructure, or for entering dangerous environments where humans can’t go. Some of the people on the company’s research team earned PhDs at Chinese universities, while others earned theirs at top U.S. universities, including NYU, the University of Illinois Urbana-Champaign, and Georgia Tech.

Actually, reports say that China is flush with robotics talent. Chinese companies also get more help from their government than U.S. companies do. As part of China’s Military-Civil Fusion strategy, Chinese AI and robotics companies can get funding directly from the government. In return, the government gets access to cutting-edge technologies for use in defense.

More AI coverage from Fast Company:

• Mozilla is now offering free AI voice training data in 180 languages
• This hyper-smelling AI can sniff out counterfeit sneakers—and that’s only the beginning
• 9 ways AI can make your banking job easier
• Can AI therapists save us from the global mental health crisis?

Want exclusive reporting and trend analysis on technology, business innovation, future of work, and design? Sign up for Fast Company Premium.