If Congress won’t regulate AI, Scott Wiener believes California must

Last summer, Scott Wiener found himself in an uncomfortable position for any California Democrat: openly quarreling with Nancy Pelosi and garnering support from Elon Musk.

The reason for this strange bedfellows moment? SB 1047, a controversial bill Wiener introduced in the state senate that would have required developers of the most advanced AI models to mitigate the risk of “catastrophic harm” caused by their creations—and would have created new liabilities for those who failed to do so.

The bill pitted the AI industry against itself, with OpenAI, Google, and Meta all arguing it would create insurmountable obstacles to innovation and AI luminaries like Yoshua Bengio, Geoffrey Hinton, and, eventually, Musk, talking up the bill’s merits. It also created a rift in Wiener’s own party, sailing easily through California’s Democrat-controlled legislature, while meeting unprecedented opposition from Pelosi and other California Democrats in Congress. In the kind of withering critique that has become her calling card, Pelosi called the bill “well-intentioned but ill informed.”

California governor Gavin Newsom ultimately vetoed the bill, arguing it focused too much on regulating models based on their size and not on the context in which they were deployed. But the conversation it sparked established Wiener as a key player in the movement to overcome Washington’s inaction and create AI guardrails at the state level. “If any state can do it, it’s California,” Wiener tells Fast Company.

That Wiener would be the one to lead this movement was not at all a given. Though he represents the California district that includes the tech mecca of San Francisco, he’s best known for advancing policies to create more affordable housing in the state. But Wiener has also fought in the past to regulate the tech sector on issues where Washington wouldn’t. In 2018, after the Trump administration overturned federal net neutrality protections, Wiener co-wrote the law that restored those protections in California.

As the AI industry boomed throughout 2023, Wiener began hosting a series of salons with AI leaders to once again explore the possibility of doing for AI safety what California had already done for net neutrality and privacy protections. “Congress has not passed a data privacy law. We passed one in California in 2018, and at the time, the industry told us that we were going to destroy innovation in California,” Wiener says, noting that the prediction turned out to be wrong. “Reasonable regulation is perfectly consistent with strong innovation.”

While SB 1047 ultimately failed, Wiener says he was encouraged by Governor Newsom’s veto message, in which he said he agreed that California “cannot afford to wait for a major catastrophe to occur before taking action to protect the public.” Wiener says he plans to introduce new AI safety legislation next year. For California, the battle over SB 1047 was just the beginning.

This story is part of AI 20, our monthlong series of profiles spotlighting the most interesting technologists, entrepreneurs, corporate leaders, and creative thinkers shaping the world of artificial intelligence.