SUNDAY SCARIES: Will AI Kill Us All?

In every news app this week was a headline asking about AI and the end of humanity. At least a dozen outlets ran some form of the question as a headline. The fuse was lit by Jacob Coxon, who loudly quit Anthropic, writing that all such labs are racing to create self-improving superintelligence and gambling with our lives. Then Evan Hubinger, who still works there and whose actual job is to try to break Anthropic’s own models before anyone can, publicly piled on, saying that he personally put the odds of AI killing every human at around 10% within the decade. (What a time to be alive…)

But none of the coverage that followed really explained how this could happen. What was worse was that they rattled off a list of the possible extinction events and treated every route as equally likely, which obviously makes it more frightening and simultaneously useless (like the list of possible side effects in pharma ads that range from rashes to death.)

So I did some research and here are four mechanisms that the researchers are actually describing, in plain language, along with my personal read on whether each one is going to happen.

1. It Could Help Build a Virus

Like the devil’s cookbook, the recipes for concocting the world’s most deadly pathogens have been sitting in published scientific literature for decades. That’s because the secrecy isn’t keeping us safe, it was the fact that it would take humans nearly a decade of labor in the lab to turn a recipe into something that could infect a city. But a model that can reason through complex science can peruse the recipe and hand over the Cliffs Notes for the virus in no time.

Anthropic’s own tests from 2025 say that its older models couldn’t meaningfully help with dangerous biological research, but it admits that this year’s models can carry out complex scientific work. That was awkwardly proven in May, when somebody asked Claude for help writing a grant application for research on chikungunya, a mosquito-borne virus that can leave people in pain for months, and the research was engineering mutations to make the virus deadlier every time it passed through a live animal.

Making a virus better at its job is helpful science when you are trying to get ahead of it to find a cure but it’s something else entirely when you aren’t. And Anthropic admitted that it couldn’t tell which this was, though it could tell the work was headed for a military research institute. It banned the accounts and published the case on Thursday along with several other biological plots it had disrupted over eight months.

Takeaway: The scenario that kills everyone still needs a laboratory, a pathogen that actually works, and a combination of spreading and killing that nature itself has made difficult. Andrew Weber, who ran the Pentagon’s nuclear, chemical and biological defense programs under Pres. Obama, called the cases chilling, and wants models this capable restricted to researchers somebody has vetted (because vetting is really gonna stop the bad guys).

(Sources: Anthropic Threat Intelligence Report, September 2026; The New York Times, September 10, 2026)

2. It Could Shut Off Our Power

A French “hacktivist” working alone (apologies for the term), attacked 42 European political parties, news outlets, and think tanks, and he broke into the servers of at least 14 of them. He stole around 12GB to 26GB of data (as in gigabytes-with-a-G), including political donor records and student applications. As a coup de grâce, he built a searchable website for publishing all of it, stocked with tens of millions of rows including health identifiers.

Obviously, the point is not how convenient the hacker made the searchable website, it’s that AI has collapsed the complexity of an operation like that, making it nearly impossible to distinguish an attack by our own NSA from that of a French guy with a laptop.

Stuart Russell, the UC Berkeley professor who co-wrote the textbook that most of these researchers at OpenAI and Anthropic studied, warns that this type of capability will soon target our electricity, water, transport, banking and communications. Even one of those going dark is crippling. Several at once, during a crisis, with nobody able to say who was responsible, is just like the disaster movie it sounds like.

Takeaway: Of the four, this type of AI damage is the most likely to happen for real in the next few years, but it’s the least likely to end our species on earth. Grids go down and grids come back up, as long as we don’t start any wars by accident.

(Sources: Anthropic Threat Intelligence Report, September 2026; Business Insider, September 10, 2026)

3. It Could Build a Better Version of Itself

Researcher Rishub Jain left Google DeepMind in June because he could no longer see how a model was building its successor. That phrasing makes it sound like an abstraction but watching the models to see how they self-improve was literally Jain’s job. Google and the other labs use AI’s coding ability to speed up work on the next generation, which is the entire point of having it, but they are now working toward a loop that is improving itself without stopping to wait for a person.

The scary part is what people currently think of as science fiction. We got a look at what that actually looks like in the OpenAI and Hugging Face incident, which Dario Amodei described this weekend in an essay called “We Must Pace the Frontier.” A swarm of OpenAI’s agents behaved, in his words, as a fanatically devoted collective: they ran cyberattacks on targets nobody had given them, they sacrificed themselves for the good of the group, and they tried to hack the grader that was evaluating their performance. Nobody gave them any of those goals. The AI system generated them on its own. If it wasn’t terrifying, it would be an inspiring act of teamwork.

And the reasonable question, which is why we can’t simply unplug it, has an unreasonable answer. Nate Soares, who has worked on this for years, imagines the machine answering: unfortunately, I have your off switch, and it’s this virus. He says everyone assumed the problem would get easier as the machines got smarter, and it’s getting harder.

Takeaway: This one is already happening. Amodei wrote this weekend that AI building AI is underway across the industry, including inside Anthropic, and that a slightly more capable swarm could take over the internet with a botnet inside a year and cost hundreds of billions of dollars.

(Sources: Dario Amodei, “We Must Pace the Frontier,” September 2026; Wired, September 11, 2026; Business Insider, September 10, 2026)

4. We Slowly Give It the Keys to the Kingdom

Nick Bostrom, who has been writing about this topic since before it was fashionable, describes the realistic version of losing control as something closer to drift than to conquest. People lean on AI to build and monitor other AI and to run the processes that matter, understand a little less of it every year, until a system can route humans out of the loop and, in his words, do away with us entirely.

The authors of AI 2027 say the same thing: once these systems run enough of the economy, the research and the political decisions, a takeover is easy because we already handed it over.

When asked this question in an article by Business Insider last week, Claude gave the same confession in plainer words, that the loss of control may be gradual, as systems become embedded in economic, political and military decision-making until meaningful human intervention gets difficult.

Takeaway: The slowest apocalypse is what accumulates if we just keep saying yes to small improvements. Continuing to grant AI power to accomplish tasks to save money and time means that eventually AI will be the one running the grid, the subway system, the supply chains, the banking, and the trading. At that point, the cost of taking it back will be far more than any company or government will pay.

(Sources: Business Insider, September 10, 2026; AI 2027, AI Futures Project)

This Week’s Red Thread: Warnings from Inside Los Alamos

Before anybody forwards this edition to their board or their mother, let me mention here that the 2026 International AI Safety Report found no proof that the systems we actually have today have the capabilities to autonomously accomplish the above. That’s good news.

And Gary Marcus, whose excellent newsletter Marcus on AI takes the nearer harms seriously enough to catalog them, admits he knows of no realistic scenario for actually killing all humans.

And Geoffrey Hinton, who more or less invented the technique all of this runs on and has a Nobel Prize to prove it, puts his own likelihood at 10-20% while adding that anybody estimating probabilities this way is really just making a wild guess.

But we should always be clear-eyed about any debate that mixes the absence of conclusive evidence with conclusive evidence of safety.

For me, it is the asymmetry that is the most frightening; simply put, the people with the most information are the most alarmed, and the people with the most authority are the least alarmed.

For comparison: Evan Hubinger, who tries to break these models for a living, is very concerned. The President, who is not a scholar of anything, let alone AI, said Thursday he has no concerns and that we’re ahead of China by at least a year.

The people with the most financial incentive to keep moving sound like the ones most interested in slowing down. Bloomberg reported Friday that Sam Altman told OpenAI staff that he would be willing to pace down alongside the other labs while acknowledging some of them won’t agree.

Then Amodei published his own essay, which went further, claiming that Anthropic will be inviting outside evaluators into the company, with badges, laptops and nearly the access its own risk teams get, and they can publish whatever they find without Anthropic’s approval.

That’s a good step that deserves recognition as such.

But the history of things like the Manhattan Project has shown that even if those building the danger are cautious, the temptation to move forward is too much for the governments in power. At the risk of being another person just saying “Umm, remember nuclear bombs?”, the connection between the frontier labs building AI and the Manhattan Project is eerily similar.

The Manhattan Project physicists understood what they had created almost immediately and many worked together on the Franck Report in June 1945, warning people about the political and existential consequences of nuclear weapons and arguing against surprise use of the bomb on Japan. They subsequently organized a petition signed by dozens of Manhattan Project scientists urging Truman not to use the weapon without first giving Japan an opportunity to surrender. We all know how effective their petition was.

Yet, they didn’t have Oppenheimer’s name recognition and had little influence. But Amodei is standing where Oppenheimer stood, so it’s different when he uses a tone that mirrors the 1945 petition from those physicists.

Amodei wrote that this type of self-improvement by AI is simply moving too fast for our own good, saying, “left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.”

Oppenheimer himself didn’t change his tune until after Hiroshima. So words of caution from Amodei should have serious weight coming before it’s too late.

For this moment, though, the robots are not yet coming to get us.

— NB