The rare New York City Council Committee of the Whole hearing brought all 51 council members together to examine AI safety, corporate accountability, and proposed regulations. Representatives from OpenAI, Anthropic, Google, and Meta testified alongside former employees who raised concerns about how frontier AI systems are being developed and evaluated.
Former AI Researchers Warn of Loss of Control
Jacob Coxon, a former researcher at Anthropic and OpenAI, delivered one of the strongest warnings during the hearing. He argued that the industry’s rapid-development culture is poorly suited to increasingly capable AI systems.
“On the current path, I think it is more likely than not that humanity loses control to these AIs and it could end in human extinction,” Coxon told lawmakers.
Coxon warned that the startup-style “move fast and break things” culture at AI firms is ill-suited to superintelligence and could leave humanity unable to control it, with potential extinction risks. Source: @NYCCouncil via X
Coxon also criticized what he described as a startup mentality centered on moving quickly and correcting problems later. He said that approach may work for consumer software but is inappropriate when companies are developing increasingly powerful AI systems.
His testimony followed his September resignation from Anthropic, where he worked on AI pretraining. Coxon has said AI could provide substantial benefits in science, medicine and the economy, while arguing that stronger safeguards are needed as developers pursue more capable and potentially self-improving systems.
Former OpenAI researcher Daniel Kokotajlo also testified about AI safety. He warned that developers may struggle to identify problems involving AI behavior before systems become more capable, adding to concerns about whether existing evaluation methods can reliably detect dangerous failures.
OpenAI, Anthropic and Google Pressed on AI Risks
The hearing placed representatives of the largest AI companies on the other side of the debate. OpenAI, Anthropic, Google and Meta emphasized the benefits of AI and their focus on safety, but lawmakers pressed them for more specific answers about catastrophic scenarios.
Morgan Dwyer, OpenAI’s head of policy development and operations, was asked by Council Speaker Julie Menin to quantify the potential risk in a worst-case scenario.
“I don’t know,” Dwyer said, arguing that it was less important whether the theoretical probability was 1%, 10% or 20% than ensuring companies do not train models they cannot make a strong case for keeping under human control.
Coxon recognizes AI’s potential economic, scientific, and medical benefits but criticizes firms for recklessly developing self-improving systems. Source: @NYCCouncil via X
Menin criticized the response, saying she was “incredulous” at the answer. The exchange highlighted a central issue in the AI safety debate: companies may acknowledge that severe risks are unacceptable without agreeing on a precise way to measure those risks.
Alice Friend, Google’s director of AI and emerging technology policy, said the company takes catastrophic risks seriously. Anthropic policy executive Logan Graham similarly expressed support for what he called “smart regulation,” while emphasizing the need for continued research into AI safety.
The testimony did not resolve disagreements over how much risk is acceptable, how it should be measured or when an AI system should be prevented from deployment.
NYC Considers New AI Safety Rules
The Council is considering a package of AI regulations covering safety testing, transparency, liability and whistleblower protections.
One proposal would require AI systems deployed or marketed in New York City to undergo independent third-party validation. The proposed framework would examine areas including data quality, bias, decision outputs, privacy and security.
The same proposal would require AI systems to have a human-controlled “kill switch” capable of shutting them down. Businesses and validators could face a $25,000 penalty for deploying systems without the required validation or for falsifying validation.
Another proposal would create financial incentives for whistleblowers who report violations involving AI companies. A separate measure would establish a private right of action for people who suffer foreseeable harm linked to malicious use or the circumvention of AI safety controls.
The Council is also considering rules for reporting AI safety incidents, emergency response planning, chatbot privacy and transparency, manipulated political media, and the impact of algorithmic systems on city employment.
These proposals remain part of the legislative process and have not become city law.
SpaceXAI Faces Separate Legal Pressure
Elon Musk’s SpaceXAI was notably absent from the hearing despite receiving a subpoena from the Council. Speaker Julie Menin said the Council would pursue judicial enforcement after the company failed to appear.
The dispute adds another layer to the broader question of whether AI companies should face stronger legal obligations when their systems cause harm or operate outside expected safeguards.
Labour MP Jess Asato shared testimony on her UK lawsuit against xAI, alleging privacy violations and Grok’s generation of unprompted sexualized content. Source: @Jess4Lowestoft via X
The issue extends beyond catastrophic AI scenarios. Lawmakers and regulators are also dealing with more immediate problems involving privacy, deepfakes, cybersecurity, automated decisions, and harmful content.
That broader concern was reflected in testimony from Jess Asato, a Labour member of the UK Parliament who has separately sued xAI over alleged misuse of private information and sexualized content generated by Grok. Her case argues that AI developers should be held accountable for design decisions that make harmful outputs possible.
Asato’s lawsuit is being pursued in the UK High Court, and concerns alleged breaches of data protection law and misuse of private information. xAI has faced legal and regulatory scrutiny in several jurisdictions over Grok’s handling of generated content.
AI Regulation Moves Beyond Voluntary Safeguards
The New York hearing comes as governments debate whether voluntary commitments from AI companies are sufficient as models become more capable.
New York City already has laws covering algorithmic accountability for city agencies, including requirements involving risk assessments, privacy, civil rights, and monitoring. The new proposals would extend the city’s approach toward private-sector AI systems.
Speaker Menin has argued that local governments cannot simply wait for federal action.
“The idea that AI companies are simply going to self-regulate is not going to work,” Menin said before the hearing.
For AI companies, the challenge is balancing rapid development with safeguards that can be independently evaluated and enforced. For lawmakers, the question is how to regulate systems whose capabilities and applications are changing faster than existing rules.
The NYC AI hearing did not establish that advanced AI will cause catastrophic harm. It did, however, expose a widening disagreement over how those risks should be assessed, who should be responsible when safeguards fail, and how much oversight governments should impose on the companies developing the technology.
As OpenAI, Anthropic, Google, and other AI developers push toward increasingly capable systems, those questions are likely to remain central to the AI regulation debate.
