New York Passes Groundbreaking Bill to Curb AI Catastrophes

New York Passes Groundbreaking Bill to Curb AI Catastrophes image

Image courtesy of neilsahota.com

GOOGL+0.58%

New York state lawmakers passed the RAISE Act on Thursday, a landmark bill designed to prevent advanced AI models from companies like OpenAI, Google, and Anthropic from causing large-scale disasters — including incidents resulting in more than 100 deaths or over $1 billion in damage.

The legislation marks a major victory for the AI safety movement, which has struggled to gain traction amid a national focus on speed and innovation, particularly under the Trump administration. Prominent voices such as Nobel laureate Geoffrey Hinton and AI pioneer Yoshua Bengio have backed the RAISE Act. If enacted, the bill would establish the first legally binding transparency mandates for frontier AI labs in the United States.

Though the RAISE Act shares similarities with California’s controversial SB 1047 — which was ultimately vetoed — New York State Senator Andrew Gounardes, a co-sponsor of the bill, told TechCrunch he carefully structured the legislation to avoid stifling startups and academic researchers.

“The window to put in place guardrails is rapidly shrinking given how fast this technology is evolving,” said Senator Gounardes. “The people that know [AI] the best say that these risks are incredibly likely […] That’s alarming.”

The bill now awaits action from Governor Kathy Hochul, who can either sign it into law, request changes, or veto it.

Should the RAISE Act become law, major AI developers would be required to release detailed safety and security evaluations of their frontier models. The law would also mandate the reporting of safety incidents, including problematic model behavior or the theft of AI systems. Noncompliance could result in civil penalties of up to $30 million, enforceable by New York’s attorney general.

The bill specifically targets the largest AI labs — including those in California (like OpenAI and Google) and China (like Alibaba and DeepSeek) — whose models were trained using at least $100 million in computing power and are accessible to New York residents.

Nathan Calvin, vice president of State Affairs and general counsel at Encode, who also contributed to California’s SB 1047, said the RAISE Act was crafted to address criticisms of earlier efforts. Notably, it does not require a “kill switch” on AI models, nor does it hold companies accountable for harm stemming from post-trained models. Still, the bill has met strong opposition from Silicon Valley, according to Assemblymember Alex Bores, a co-sponsor.

“The NY RAISE Act is yet another stupid, stupid state level AI bill that will only hurt the US at a time when our adversaries are racing ahead,” said Andreessen Horowitz general partner Anjney Midha in a Friday post on X. Venture firms like Andreessen Horowitz and incubator Y Combinator were vocal critics of SB 1047 as well.

Anthropic, an AI lab that recently advocated for national transparency rules, hasn’t taken an official position on the RAISE Act. But in a Friday post on X, co-founder Jack Clark expressed concern over its broad scope, warning that it “could present a risk to ‘smaller companies.’”

In response, Senator Gounardes said Clark’s critique “misses the mark,” reiterating that the bill was intentionally designed not to apply to smaller developers.

OpenAI, Google, and Meta did not respond to TechCrunch’s request for comment.

A recurring criticism of the RAISE Act is that developers might choose not to offer their most advanced AI models in New York, similar to industry responses to strict tech regulations in Europe. That concern also echoed around SB 1047 in California.

However, Assemblymember Bores told TechCrunch the regulatory requirements are minimal and shouldn’t dissuade companies from operating in New York. With the state ranking third in GDP nationwide, he argued, companies likely won’t walk away.

“I don’t want to underestimate the political pettiness that might happen, but I am very confident that there is no economic reason for [AI companies] to not make their models available in New York,” said Assemblymember Bores.

Related Posts