How Anyscale cofounder Robert Nishihara is smoothing AI’s complexities

As businesses reinvent workflows and products with AI, many such initiatives have stalled or failed to reach production. It can be really hard to implement the technology. AI requires massive quantities of data and computational power, and running AI at scale introduces major costs and software engineering challenges.

Robert Nishihara saw this problem coming when he was a student at the University of California, Berkeley in 2016. At the time he had no plans to build AI infrastructure, but he was often distracted from his research because of time spent getting his models to run well across the server clusters he used. “We were doing AI research, but we were bottlenecked by the tooling,” he says. “We found ourselves spending all of our time managing clusters, wrangling data, and solving distributed systems challenges.”

So, Nishihara and his partners Philipp Moritz and Berkeley professor Ion Stoica set out to build a unified computing framework to take care of the infrastructure headaches. Ray is an open-source AI compute engine that solves the hard software engineering challenges of scaling AI applications across many machines and GPUs in the cloud, Nishihara says.

A developer can start creating a model using the popular Python language on their laptop, then rely on Ray to scale the model up to run on any brand of GPU, CPU, or other accelerator. Ray also provides a set of tools for managing and accelerating AI workloads. “Working on Ray, we’ve been privileged to help enable businesses to bring AI to production and really begin realizing the potential of AI to build better products and solve important problems in medicine, agriculture, entertainment, transportation, finance, and so much more,” Nishihara says.

Little did he know that in just a few years the need for Ray would grow even larger with the release of ChatGPT and the dawn of the generative AI boom. “Generative AI took the complexity problem and made it 10 times worse,” Nishihara says.

The popularity and energy around Ray prompted Nishihara, Moritz, and Stoica to found Anyscale in 2019 based on their initial open source project. Anyscale offers a managed version of Ray. It also offers an optimized version of Ray called RayTurbo, which features increased reliability, efficiency, and fault-tolerance. (Nishihara served as CEO until earlier this year; now his focus is working with the product and customers.)

The Ray concept has gathered plenty of adherents. More than a thousand developers have contributed to the open source project. And developers at more than 10,000 organizations are now building AI models and apps on the platform. These include some well-known names. After moving to Ray, Instacart is now training models with 100 times more data. Design tool maker Canva says it was able to cut its cloud costs in half after deploying Ray. (Instacart and Canva are also Anyscale customers.) Pokemon Go creator Niantic found that it could run its AI using 85% fewer lines of code.Apple, Uber, eBay, Ford, Lockheed Martin, Nvidia, Adobe, and LinkedIn also use Ray. OpenAI cofounder Greg Brockman says his company uses Ray to train its largest models, including GPT-4.

“We’re at the outset of tremendous value being created by AI,” Nishihara says. “To realize that value, much of the hard work ahead involves taking existing (and future) capabilities, making them incredibly reliable in the real world, and building out the underlying hardware and software infrastructure to enable them throughout every industry.”

This story is part of AI 20, our monthlong series of profiles spotlighting the most interesting technologists, entrepreneurs, corporate leaders, and creative thinkers shaping the world of artificial intelligence.