What is going on with all these outages?

Well, it feels like the world is actually burning this time.

It seems like the last week has been a combination of humans and agents and cyberattacks and… fiber? I spent half my day today trying to answer the question what’s happening?

TL;DR – I am like 5% more worried about AI & cybersecurity than I was 2 weeks ago, not 50% more worried.

Weird stuff that’s allegedly not a cyberattack

The Banks

J.P. Morgan, Bank of America, Wells Fargo, and U.S. Bank.  Allegedly, a bug at J.P. Morgan caused problems on Monday morning but then created slowdowns everywhere else.  

According to the bank, it was not a cyberattack and was caused by a standard software release.  

The problem with the other banks came when the other banks couldn’t settle or handle transactions with Chase.  Zelle worked, but couldn’t complete some transactions when someone on either end of the transaction used Chase.  

The NSA

The internal communications system for the NSA experienced an outage (as of November 17).  I would imagine we won’t get a full explanation of what happened, but they called it a “temporary degradation during scheduled routine maintenance.”

Apple

Apple is having a major outage (as of this writing November 17) across various software services. Some services have been recovered already.

Software issues that created hardware problems

Microsoft

A bug written by a human months ago that was just activated overwhelmed their machines which made them unreachable. Because they were unreachable, they tried turning them off and back on. That didn’t work because the servers became overwhelmed with traffic again. All this is why it took so long.

The FAA

Upgrading from literal copper wires to fiber led to routers that overwhelmed the air traffic control network leading to intermittent radio and radar outages. The FAA is still determining exactly what happened, but signs indicate legacy system overwhelm.

Confirmed Cyberattacks

Equinor

The Norwegian oil company and largest supplier of natural gas to Europe is currently being attacked (as I write this). According to Equinox, production and operations are unaffected and no customer data was accessed.

Optimizely

Optimizely is a service which helps marketing teams A/B test different landing pages and offers on their websites.  They work by loading a javascript snippet on the customer’s website. 

The attacker wrote their own code in the Optimizely customers’ javascript to essentially skim credit cards and steal cryptocurrency. According to Report URI, when the threat was basically neutralized, only about $7,000 had been confirmed to be stolen.  It’s unknown how many credit card numbers, for example, were stolen.

It is not clear if AI was involved in this hack, although I would imagine that it is.  Big companies like Zoom, Peacock, IKEA, Salesforce, GitLab, Paychex, Alaska Airlines, CNN, BBC, UNICEF USA and American Eagle were impacted by it.  To be clear: those companies, themselves, were not hacked.  Their websites were compromised becasue Optimizely, a vendor that they use, was hacked.

An Explanation

Legacy codebases are houses of cards

Codebases that are 10+ years old are so complex that nobody understands everything. In fact, there’s a strong possibility that there are some components where nobody understands anything. 

There are all sorts of components inside it that haven’t been touched in years.  The promise of AI has been that a lot of this code can be rewritten and modernized, but that doesn’t mean that people magically understand it. 

In fact, it probably means that it’ll be less understandable in the near future.  Plus, since cybersecurity is on the forefront of everyone’s minds, companies need to speed run the modernization so they can secure it.

My guess is that the problems start here.  Nobody understands the codebase (including the AI, which is tasked with rewriting and modernizing large parts of it).

Separately, sometimes fixing one bug means you discover another.  Please forgive the oversimplified example: sometimes, in software, you have 2 bugs.  In fixing one (a minor bug), you expose a major one and learn that the minor bug was actually covering up the major bug.  That major bug can be in production and creating all sorts of chaos.  My hunch is that this is happening a lot.

AI finds bugs way faster

Cybersecurity is essentially a giant game of whack-a-mole.  There are bad guys who try to find bugs so they can hack systems and good guys who try to find bugs to secure systems.  When the bad guy gets there first and is good at hacking, they hack the system.  When the good guy gets there first, they patch the bug.

AI models are now so good at writing and improving code that they can identify vulnerable systems, write the code to hack them, improve the code once they’re hacked, and then achieve their objective.  

This is a good thing and a bad thing.  The PHP bug from a few weeks ago was found because someone asked AI to “find serious vulnerabilities in PHP.”  Developers everywhere are using AI to scan codebases and find security vulnerabilities.  

Those vulnerabilities, when found, need to be patched ASAP because, when they are found by good guys, they can also be found by bad guys.

Humans racing to find and fix bugs can miss important details that lead to bigger or worse bugs getting out into the wild.

Everything was already insecure

Code up until recently was written by humans.  Humans are fallible.  Therefore, code is fallible.  All code written by humans has bugs.  Some of those bugs are more severe than others and can lead to attackers gaining unauthorized access to systems.

Secondly, there are only a few technical foundations that nearly everyone operates on.  Think about it like a house.  You can have a cinder-block house or you can have a pueblo or you can have a wood house.  

At a technical level, this is how it works. 

There are only a few operating systems everyone uses.  

Then, there are several programming languages.  

Then, there are a few dozen infrastructure providers (like AWS or GCP).  

Then, there are thousands of code libraries.  Think of each library as a book of helper tools that save developers tons and tons of time.  

Then, only after all those components are added, do you have developers writing code to make the unique app you’re using.

The problem we all face is that a bug anywhere in this stack can lead to an attacker hacking software.

Infrastructure can be easily overwhelmed

You could use AWS or GCP or other compute providers, but if you don’t have enough compute capacity and suddenly get a lot of traffic, then your systems can be overwhelmed and go down.  That’s what happened in the case of the FAA and Microsoft.

When open-source software has a bug, the fix is also open-source

Programming languages, libraries, and some operating systems are fully open-source, meaning that the code is free and available to everyone.  But, when there’s a security issue, the patch is also free and available to everyone.  Bad guys with AIs can then reverse-engineer the vulnerability and start searching for companies that haven’t updated their systems yet.

Everyone is hyperaware and most people hate AI

The sky is going to feel like it’s falling for a while.  We’ve been operating on an insecure internet since the dawn of the internet. These bugs were introduced by humans and they’ll be fixed by a combination of humans + AI. The bugs will, over time, be squashed and the internet will be far more secure.

Hacks and incidents like this happen all the time, but right now, everyone is hyperaware of AI and of cybersecurity threats — combined with the Microsoft and FAA outages (which were not caused by AI).

As I wrote about last week, AI was on the ballot and lost overwhelmingly.  Large swaths of the population don’t want AI to succeed.  My personal feelings on that sentiment are unimportant to this point: trying to create hysteria or blame everything on AI means insane amounts of attention and energy, which may lead to regulation, which may lead to it slowing down or stopping being used.

The Jury is Out

Some people online think that we are past the warning stage. They’re buying water and food and toilet paper and taking cash out of the ATM.

Personally, I think that this is a very high frequency of business-as-usual incidents.  Incidents happen.  Admittedly, they are happening faster than normal, and that is concerning, but I think that is because AI makes finding bugs faster and modernizing legacy codebases is harder than people think.

If you set aside the ultra-high-profile Microsoft, FAA, and J.P. Morgan issues which are reportedly independent of AI, the other cybersecurity issues seem extremely concerning, but it’s not the end of the world as we know it. 

More incidents will happen. Some are happening actively now.  AI will be the cause of and will accelerate some.  But I don’t think now is the time to freak out, and I think that there are lots of really smart people working on minimizing the damage.  That said, it’s going to be bumpy for a few years.

Evan Ryan
CEO | Teammate AI
teammateai.com

Jensen, the Midterms, and 2028

Earlier this week, as I was typing my update post about the midterms, I thought “Everyone wants to talk about Jensen and the spy, but I don’t have much to…

Read More

Get AI Insights in Your Inbox

We send (hopefully) timeless essays, subscriber-only Hot Takes, and quick tips about how to use common AI tools better.

You can unsubscribe at any time.

All Posts