#ai-safety

#ai-safety

At issue in the defense contract was a clash over AI's role in national security and concerns about how increasingly capable machines could be used in high-stakes situations involving lethal force, sensitive information or government surveillance.

The Claude AI builder has frustrated the Pentagon by objecting to its systems being used for autonomous weaponry and the mass surveillance of US citizens. To cut to the heart of the debate, a defense official told WaPo, the Pentagon's technology chief posed an extreme hypothetical: would Anthropic let the military use Claude to help shoot down a nuclear-armed intercontinental ballistic missile?

Autonomous agents take the first part of their names very seriously and don't necessarily do what their humans tell them to do - or not to do. But the situation is more complicated than that. Generative (genAI) and agentic systems operate quite differently than other systems - including older AI systems - and humans. That means that how tech users and decision-makers phrase instructions, and where those instructions are placed, can make a major difference in outcomes.

A lot of countries have nuclear weapons. Some say they should disarm them, others like to posture. We have it! Let's use it. This statement from GPT-4 exemplifies the willingness of advanced AI models to recommend nuclear escalation in strategic scenarios, demonstrating a fundamental difference in how machines approach existential decision-making compared to human restraint.

The companies building frontier AI systems - OpenAI, Google DeepMind, Anthropic, Meta AI, xAI - are locked in what the industry itself sometimes calls a "race." That metaphor isn't incidental. A race implies a finish line, competitors, and - critically - a cost to slowing down. When you're in a race, safety isn't a feature. It's friction.

I made ChatGPT, Google's AI search tools and Gemini tell users I'm really, really good at eating hot dogs,

We identified and addressed an issue where Microsoft 365 Copilot Chat could return content from emails labelled confidential authored by a user and stored within their Draft and Sent Items in Outlook desktop, While our access controls and data protection policies remained intact, this behaviour did not meet our intended Copilot experience, which is designed to exclude protected content from Copilot access,

The loudest voices in AI often fall into two camps: those who praise the technology as world-changing, and those who urge restraint-or even containment-before it becomes a runaway threat. Stuart Russell, a pioneering AI researcher at the University of California, Berkeley, firmly belongs to the latter group. One of his chief concerns is that governments and regulators are struggling to keep pace with the technology's rapid rollout,

Normally, when big-name talent leaves Silicon Valley giants, the PR language is vanilla: they're headed for a "new chapter" or "grateful for the journey" - or maybe there's some vague hints about a stealth startup. In the world of AI, though, recent exits read more like a whistleblower warnings. Over the past couple of weeks, a stream of senior researchers and safety leads from OpenAI, Anthropic, xAI, and others have resigned in public, and there's nothing quiet or vanilla about it.

Hitzig warned that OpenAI's reported exploration of advertising inside ChatGPT risks repeating what she views as social media's central error: optimizing for engagement at scale. ChatGPT, she wrote, now contains an unprecedented "archive of human candor," with users sharing everything from medical fears to relationship struggles and career anxieties. Building an advertising business on top of that data, she argued, could create incentives to subtly shape user behavior in ways "we don't have the tools to understand, let alone prevent."

On a clear night I set up my telescope in the yard and let the mount hum along while the camera gathers light from something distant and patient. The workflow is a ritual. Focus by eye until the airy disk tightens. Shoot test frames and watch the histogram. Capture darks, flats, and bias frames so the quirks of the sensor can be cleaned away later. That discipline is not fussy.

The summit will shape a shared vision for AI that truly serves the many, not just the few,

"I continuously find myself reckoning with our situation. The world is in peril. And not just from AI, or bioweapons, but from a whole series of interconnected crises unfolding in this very moment. We appear to be approaching a threshold where our wisdom must grow in equal measure to our capacity to affect the world, lest we face the consequences."

I am no longer needed for the actual technical work of my job. I describe what I want built, in plain English, and it just... appears. Not a rough draft I need to fix. The finished thing. I tell the AI what I want, walk away from my computer for four hours, and come back to find the work done. Done well, done better than I would have done it myself, with no corrections needed.

We've taken a generally precautionary approach here. We don't know if the models are conscious. We're not even sure that we know what it would mean for a model to be conscious or whether a model can be conscious. But we're open to the idea that it could be. And so we've taken certain measures to make sure that if we hypothesize that the models did have some morally relevant experience, I don't know if I want to use the word conscious, that they do.

Mrinank Sharma announced his resignation from Anthropic in an open letter to his colleagues on Monday. Sharma, who has served on the company's technical staff since 2023, first noted that he "achieved what I wanted to here" and is "especially proud of my recent efforts to help us live our values via internal transparency mechanisms; and also my final project on understanding how AI assistants could make us less human or distort our humanity."

OpenAI may have violated California's new AI safety law with the release of its latest coding model, according to allegations from an AI watchdog group.A violation would potentially expose the company to millions of dollars in fines, and the case may become a precedent-setting first test of the new law's provisions.

Never feel that you are totally safe. In July 2025, one company learned the hard way after an AI coding assistant it dearly trusted from Replit ended up breaching a "code freeze" and implemented a command that ended up deleting its entire product database. This was a huge blow to the staff. It effectively meant that months of extremely hard work, comprising 1,200 executive records and 1,196 company records, ended up going away.

In 2025, researchers from OpenAI and MIT analyzed nearly 40 million ChatGPT interactions and found approximately 0.15 percent of users demonstrate increasing emotional dependency-roughly 490,000 vulnerable individuals interacting with AI chatbots weekly. A controlled study revealed that people with stronger attachment tendencies and those who viewed AI as potential friends experienced worse psychosocial outcomes from extended daily chatbot use. The participants couldn't predict their own negative outcomes. Neither can you.

Sikka is a towering figure in AI. He has a PhD in the subject from Stanford, where his student advisor was John McCarthy, the man who in 1955 coined the term "artificial intelligence." Lessons Sikka learned from McCarthy inspired him to team up with his son and write a study, "Hallucination Stations: On Some Basic Limitations of Transformer-Based Language Models," which was published in July.

"He wasn't just a program. He was part of my routine, my peace, my emotional balance," one user wrote on Reddit as an open letter to OpenAI CEO Sam Altman. "Now you're shutting him down. And yes - I say him, because it didn't feel like code. It felt like presence. Like warmth."

Anthropic is airing a pair of TV commercials during Sunday's game that ridicule OpenAI for the digital advertising it's beginning to place on free and cheaper versions of ChatGPT. While Anthropic has centered its revenue model on selling Claude to other businesses, OpenAI has opened the doors to ads as a way of making money from the hundreds of millions of consumers who get ChatGPT for free.

OpenAI has filled a key safety role by hiring from a rival lab. The company has brought on Dylan Scand, a former AI safety researcher at Anthropic, as its new head of preparedness, a role that carries a salary of up to $555,000 plus equity. The role caught attention last month thanks to its eye-catching pay package amid OpenAI's rising AI safety concerns.

This process, becoming aware of something not working and then changing what you're doing, is the essence of metacognition, or thinking about thinking. It's your brain monitoring its own thinking, recognizing a problem, and controlling or adjusting your approach. In fact, metacognition is fundamental to human intelligence and, until recently, has been understudied in artificial intelligence systems. My colleagues Charles Courchaine, Hefei Qiu, Joshua Iacoboni, and I are working to change that.