About eight years ago, we were at Snowshoe Mountain when my daughter Leia (yes, she’s named after Star Wars), who was eight at the time, made an observation:
“You know what’s good about this place? For the zombie apocalypse, zombies don’t want to climb mountains.”
I asked, “Do you worry about the zombie apocalypse a lot?”
“Not that much.”
As best I can remember, that was probably the last time I spent much time thinking about apocalypses.
Lately, apparently, I have more than one to worry about.
For the past year or so, I’ve been living through what some have called the “SaaSpocalypse,” the idea that AI will fundamentally disrupt the software industry.
The predictions vary. AI agents will replace some of the software applications we use today. Companies will need fewer software licenses because AI will do work that previously required people. Software will become dramatically cheaper to build, allowing new competitors to appear almost overnight. Companies that don’t adapt quickly enough will be left behind.
And people are taking this seriously. Companies have changed strategies. Investors have repriced businesses. Product roadmaps have been rewritten. “AI-native” (essentially, built around AI rather than simply adding it to existing products) has become practically mandatory vocabulary. People have lost their jobs because companies decided they weren’t moving toward AI quickly enough or, perhaps just as importantly, because they weren’t giving the appearance of moving quickly enough.
So I’ve spent a fair amount of time thinking about the possibility that AI might destroy the software industry I’ve spent most of my career in. I’ve also experienced firsthand how predictions about what AI might do tomorrow can have very real consequences today.
Then I started reading warnings from people working on the most advanced AI systems. Some were warning that the race among the companies building them could push the technology forward faster than our ability to understand or control it, potentially creating an existential risk to humanity. Around the same time, my wife heard a discussion about it on the radio and started asking me questions.
Apparently, I wasn’t thinking big enough about apocalypses.
Wait. I Thought You Said It Doesn’t Think.
Her question was pretty reasonable.
I’ve spent a lot of time telling her that today’s generative AI doesn’t really “think,” at least not in the way we normally use that word. That’s admittedly a strange thing to say about something you can have a conversation with.
Ask a good AI model to analyze a problem, write software, explain quantum physics, create a business plan, or critique something you’ve written, and it certainly looks like it’s thinking. For a growing number of tasks, the distinction can feel almost academic. It’s close enough to thinking to be extraordinarily useful.
But that doesn’t necessarily mean it understands the world the way we do. Yann LeCun has argued for years that today’s generative AI is still missing important pieces of the kind of understanding humans develop about the world. Welch Labs did a great interview with him recently about his work on “world models” and why he believes they may be necessary to get there.
Which leads back to my wife’s question:
“If these things don’t really understand what they’re doing, how exactly are they going to kill all of us?”
Unfortunately, they don’t necessarily have to understand what they’re doing to be very good at doing it.
From Adviser to Actor
Until fairly recently, most people’s experience with generative AI was pretty simple. You asked it something. It answered.
If the answer was wrong, that could certainly be a problem. But you still had to decide what to do with it.
That’s changing.
The easiest way I know to explain an AI agent is that instead of giving an AI a question, you give it a goal and a set of allowed actions. It figures out the next step, takes it, looks at what happened, and figures out what to do next. Real systems can be considerably more sophisticated, but the important change is simple: we’ve taken something that used to give us answers and started giving it permission to act.
There’s a pretty important difference between an AI saying, “Here’s what I think you should do,” and an AI saying, “I did it.”
A Job Application Gone Wrong
A friend of mine recently built an agent to automate his job search. It would find jobs that looked like a match and apply to them on his behalf.
One day, he applied for a job himself. Unbeknownst to him, his agent had applied for the same job.
That was awkward enough, but there was a bigger problem. The agent hadn’t answered all of the application questions exactly the way he had. In particular, it gave a different answer about his salary requirements.
The recruiter understandably wanted to know which answer was the real one. My friend had to fess up and explain what had happened.
He didn’t get the job.
It’s a funny story now, although probably less funny to my friend at the time. But what’s interesting is that the agent hadn’t gone rogue. It hadn’t become conscious or decided to sabotage his career.
It was doing exactly the kind of thing it had been designed to do. It had a goal, instructions, access to a website, and permission to take action on his behalf.
So it did.
The problem was that the action wasn’t entirely consistent with what he wanted, and he didn’t even know it had happened.
In this case, the consequence was the loss of a job opportunity. But it’s easy to imagine the same thing happening in systems businesses are beginning to deploy today. A customer service agent issues a refund that shouldn’t have been issued. A sales agent quotes the wrong price. A coding agent changes something without anyone realizing it.
Most of those mistakes are recoverable. Some could be expensive or embarrassing. But the mechanism isn’t fundamentally different from what happened to my friend. We gave the system a goal, access to something, and permission to act. It did something we didn’t quite intend.
Now keep increasing the stakes.
What happens when the thing we’ve connected it to is a municipal water system? Air traffic? The electrical grid? A laboratory? A weapons system?
At some point, “the AI did something we didn’t expect” stops being a funny story.
It Doesn’t Have to Hate Us
This is where I think science fiction may have trained us to think about the problem the wrong way.
The familiar version is that the computer becomes intelligent, becomes self-aware, decides humans are the problem, and bad things happen. We’ve made a few movies about this. In 2001: A Space Odyssey, HAL turns against the astronauts. In WarGames, a computer comes frighteningly close to starting a nuclear war.
But there’s another possibility that requires considerably less science fiction.
The AI doesn’t have to become evil or angry. It can simply become capable enough to take consequential actions, make a mistake, and have enough authority for that mistake to matter.
You don’t necessarily need a superintelligence plotting humanity’s demise. You need a sufficiently capable system, access to something sufficiently dangerous, an objective that produces an outcome its creators didn’t anticipate, and inadequate checks between deciding and doing.
Or you need a bad actor who deliberately doesn’t care about those checks.
That’s a much less cinematic path to disaster. It’s also one I find easier to imagine.
When the Two Apocalypses Meet
And this is where my two AI apocalypses start running into each other.
The fear of the SaaSpocalypse isn’t merely pushing software companies to use AI. It’s pushing them to let AI do more.
Don’t just help the customer write the email; send it. Don’t just identify the problem; fix it. Don’t just recommend what should happen next; do it.
That’s an incredibly compelling promise: faster work, fewer manual steps, lower costs, and software that doesn’t just help people do their jobs but actually does portions of the job for them.
I find all of this exciting. I use AI every day. I’ve spent much of the last year building software with it and experimenting with what these systems can do. Hell, I’m using AI models to help me write these articles. I’m hardly arguing that we should stop using them.
But every step from adviser toward actor changes the consequences of being wrong. There’s a big difference between an AI suggesting a sentence I don’t like and an AI taking an action I don’t know about.
And here’s where the two apocalypses really do collide. The fear of the first one creates pressure to do exactly the things that could increase the risk of the second: move faster, automate more, remove friction, and give AI more responsibility.
At the same time, boards want an AI strategy. Investors want an AI story. CEOs want AI roadmaps. Nobody wants to be the company that wakes up two years from now and discovers everyone else understood the transition first.
That pressure isn’t irrational. AI really is changing software. But there’s a difference between moving quickly because the technology demands it and feeling pressure to demonstrate that you’re moving quickly.
If the prevailing message is that moving too slowly poses an existential threat to your business, “Let’s make sure we understand the risks before we automate this” can start to sound suspiciously like resistance to change.
The instincts required to manage the risk can point in the opposite direction: understand how the system might fail, limit what it’s allowed to do, keep appropriate checks in place, and ensure a human is still involved when the consequences warrant it.
Companies really do need to move quickly. But we’re making the transition from adviser to actor at exactly the same time the industry is being told that moving too slowly may be an existential threat.
Maybe WarGames Had It Right
I’ve always thought of WarGames as a very 1980s movie: nuclear war, Cold War paranoia, giant computers with blinking lights, and a synthesized voice.
But its central idea has aged pretty well.
The dangerous decision wasn’t really creating a computer capable of playing nuclear war games. It was connecting that computer to the system capable of fighting one.
We spend enormous amounts of time debating how intelligent AI will become. Will it someday be smarter than humans? Does it reason? Does it understand? Could it become conscious?
I don’t know the answers to those questions. But there’s another one we don’t need to wait to answer:
What are we willing to connect it to?
We don’t need to settle the philosophical question of whether an AI truly thinks before we give it the ability to act.
We’re already doing that.
So Which Apocalypse Should I Worry About?
I still think AI is fundamentally reshaping the software industry. I don’t know whether that means the SaaSpocalypse actually happens. Software companies have survived supposedly existential technology transitions before, usually by changing along with them.
I also don’t know whether AI represents an existential threat to humanity. People who understand the most advanced AI systems far better than I do appear to disagree about that risk.
But I do think there’s something strange about the moment we’re in. We’re simultaneously being warned that AI is advancing so quickly that companies need to move faster or risk becoming irrelevant, and that we need to be extraordinarily careful about what we allow it to do.
Both might be true.
I don’t think the answer is to stop. I certainly intend to keep building with AI and exploring what these systems can do.
But capability isn’t the same thing as judgment. As we give AI more autonomy and connect it to systems where mistakes have greater consequences, the checks and balances probably need to get stronger rather than disappear in the name of efficiency.
Twelve months ago, I thought the AI apocalypse I needed to understand was the one coming for SaaS.
Now apparently I have another one to worry about.
Not to mention the zombie apocalypse.
Maybe Leia had the right approach.
Worry. But maybe not that much.