Cloudflare outage on February 20, 2026 (blog.cloudflare.com)

111 points by nomaxx117 3 hours ago

kgeist 31 minutes ago

It's something we debated in our team: if there's an API that returns data based on filters, what's the better behavior if no filters are provided - return everything or return nothing?

The consensus was that returning everything is rarely what's desired, for two reasons: first, if the system grows, allowing API users to return everything at once can be a problem both for our server (lots of data in RAM when fetching from the DB => OOM, and additional stress on the DB) and for the user (the same problem on their side). Second, it's easy to forget to specify filters, especially in cases like "let's delete something based on some filters."

So the standard practice now is to return nothing if no filters are provided, and we pay attention to it during code reviews. If the user does really want all the data, you can add pagination to your API. With pagination, it's very unlikely for the user to accidentally fetch everything because they must explicitly work with pagination tokens, etc.

Another option, if you don't want pagination, is to have a separate method named accordingly, like ListAllObjects, without any filters.

MobileVet 22 minutes ago

I like your thought process around the ‘empty’ case. While the opposite of a filter is no filter, to your point, that is probably not really the desire when it comes to data retrieval. We might have to revisit that ourselves.

CommonGuy 2 hours ago

Insufficient mock data in the staging environment? Like no BYOIP prefixes at all? Since even one prefix should have shown that it would be deleted by that subtask...

From all the recent outages, it sounds like Cloudflare is barely tested at all. Maybe they have lots of unit tests etc, but they do not seem to test their whole system... I get that their whole setup is vast, but even testing that subtask manually would have surfaced the bug

dabinat 2 hours ago

I think Cloudflare does not sufficiently test lesser-used options. I lurk in the R2 Discord and a lot of users seem to have problems with custom domains.

martinald an hour ago

Just crazy. Why does a staging environment matter? They should be running some integration tests against eg an in memory database for these kinds of tasks surely?

asciii 2 hours ago

It was also merged 15 days prior to production release...however, you're spot on with the empty test. That's a basic scenario that if it returned all...is like oh no.

otar 2 hours ago

Reliability was/is CF's label.

It's alarming already. Too many outages in the past months. CF should fix it, or it becomes unacceptable and people will leave the platform.

I really hope they will figure things out.

argestes an hour ago

I have many things dependent on Cloudflare. That makes me root for Cloudflare and I think I'm not the only one. Instead of finding better options we're getting stuck on an already failing HA solution. I wonder what caused this.

slothsarecool an hour ago

There are no alternatives, and those alternatives that did exist back in the day, had to shut down due to either going out of business or not being able to keep a paygo model.

Not everybody needs cloudflare, but those that need it and aren't major enterprises, have no other option.

pocksuppet 40 minutes ago

Sanzig 39 minutes ago

arcatech an hour ago

Do you not feel concern about you and everybody else deciding to put ALL of their eggs into one basket like this?

alansaber an hour ago

Not sure why everyone is complaining, new MCP features are more important than uptime

atty 2 hours ago

I do not work in the space at all, but it seems like Cloudflare has been having more network disruptions lately than they used to. To anyone who deals with this sort of thing, is that just recency bias?

Icathian 2 hours ago

It is not. They went about 5 years without one of these, and had a handful over the last 6 months. They're really going to need to figure out what's going wrong and clean up shop.

NinjaTrance 2 hours ago

Engineers have been vibe coding a lot recently...

jsheard 2 hours ago

dakiol 2 hours ago

Ylpertnodi 2 hours ago

Typo: "shop", should have been with an 'el'.

(: phonetically, because 'l's are hard to read.

dazc 2 hours ago

Launching a new service every 5 minutes is obviously stretching their resources.

lysace 2 hours ago

It has been roughly speaking five and a half years since the IPO. The original CTO (John Graham-Cumming) left about a year ago.

jacquesm 2 hours ago

They coasted on momentum for half a year. I don't even think it says anything negative about the current CTO, but more of what an exception JGC is relative to what is normal. A CTO leaving would never show up the next day in the stats, the position is strategic after all. But you'd expect to see the effect after a while, 6 months is longer than I would have expected, but short enough that cause and effect are undeniable.

Even so, it is a strong reminder not to rely on any one vendor for critical stuff, in case that wasn't clear enough yet.

dazc 2 hours ago

I wondered what happened to him?

jgrahamc an hour ago

brcmthrowaway 2 hours ago

slophater 2 hours ago

been at cf for 7 yrs but thinking of gtfo soon. the ceo is a manchild, new cto is an idiot, rest of leadership was replaced by yes-men, and the push for AI-first is being a disaster. c levels pretend they care about reliability but pressure teams to constantly ship, cto vibe codes terraform changes without warning anyone, and it's overall a bigger and bigger mess

even the blog, that used to be a respected source of technical content, has morphed into a garbage fire of slop and vaporware announcements since jgc left.

goalieca an hour ago

I’ve had a lot of problems lately. Basic things are failing and it’s like product isn’t involved at all in the dash. What’s worse? The support.. the chat is the buggiest thing I’ve ever seen.

__turbobrew__ an hour ago

You know what they say, shit rolls downhill. I don't personally know the CEO, but the feeling I have got from their public fits on social media doesn't instill confidence.

If I was a CF customer I would be migrating off now.

a24446ff87 an hour ago

GSD! GSD!! ship! ship! ship!

**everything breaks**

...

**everything breaks again**

oh fuck! Code Orange! I repeat, Code Orange! we need to rebuild trust(R)(TM)! we've let our customers down!

...

**everything breaks again**

Code Orangier! I repeat, Code Orangier!

slophater 2 hours ago

amazing how my comment was flagged in 30 seconds... keep bootlicking

Betelbuddy 2 hours ago

Cloudflare Outages are as predictable, as the Sun coming up tomorrow. Its their engineering culture.

https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

candiddevmike 2 hours ago

Wait till you see the drama around their horrible terraform provider update/rewrite:

https://github.com/cloudflare/terraform-provider-cloudflare/...

blibble 2 hours ago

is this blog post LLM generated?

the explanation makes no sense:

> Because the client is passing pending_delete with no value, the result of Query().Get(“pending_delete”) here will be an empty string (“”), so the API server interprets this as a request for all BYOIP prefixes instead of just those prefixes that were supposed to be removed. The system interpreted this as all returned prefixes being queued for deletion.

client:

     resp, err := d.doRequest(ctx, http.MethodGet, `/v1/prefixes?pending_delete`, nil)
server:

    if v := req.URL.Query().Get("pending_delete"); v != "" {
        // ignore other behavior and fetch pending objects from the ip_prefixes_deleted table
        prefixes, err := c.RO().IPPrefixes().FetchPrefixesPendingDeletion(ctx)
        if err != nil {
            api.RenderError(ctx, w, ErrInternalError)
            return
        }

        api.Render(ctx, w, http.StatusOK, renderIPPrefixAPIResponse(prefixes, nil))
        return
    }
even if the client had passed a value it would have still done exactly the same thing, as the value of "v" (or anything from the request) is not used in that block

subscribed 2 minutes ago

That's weird. They only removed some 6 of our prefixes out of perhaps 40 we have with them, so something seems off in this explanation.

bretthoerner 2 hours ago

> even if the client had passed a value it would have still done exactly the same thing, as the value of "v" (or anything from the request) is not used in that block

If they passed in any value, they would have entered the block and returned early with the results of FetchPrefixesPendingDeletion.

From the post:

> this was implemented as part of a regularly running sub-task that checks for BYOIP prefixes that should be removed, and then removes them.

They expected to drop into the block of code above, but since they didn't, they returned all routes.

blibble 2 hours ago

okay so the code which returned everything isn't there

actual explanation: the API server by default returns everything. the client attempted to make a request to return "pending_deletes", but as the request was malformed, the API instead went down the default path, which returned everything. then the client deleted everything.

makes sense now

but is that explanation is even worse

because that means the code path was never tested?

jbxntuehineoh an hour ago

bstsb 2 hours ago

doesn't look AI-generated. even if they have made a mistake, it's probably just from the rush of getting a postmortem out prior to root cause analysis

himata4113 2 hours ago

yep, no mention that re-advertised prefixes would be withdrawn again as well during the entire impact even after they shut it down.

NinjaTrance 2 hours ago

The irony is that the outage was caused by a change from the "Code Orange: Fail Small initiative".

They definitely failed big this time.

himata4113 2 hours ago

This blog post is inaccurate, the prefixes were being revoked over and over - to keep your prefixes advertised you had to have a script that would readd them or else it would be withdrawn again. The way they seemed to word it is really dishonest.

boarush 3 hours ago

While neither am I nor the company I work for directly impacted by this outage, I wonder how long can Cloudflare take these hits and keep apologizing for it. Truly appreciate them being transparent about it, but businesses care more about SLAs and uptime than the incident report.

llama052 2 hours ago

I’ll take clarity and actual RCAs than Microsoft’s approach of not notifying customers and keeping their status page green until enough people notice.

One thing I do appreciate about cloudflare is their actual use of their status page. That’s not to say these outages are okay. They aren’t. However I’m pretty confident in saying that a lot of providers would have a big paper trail of outages if they were more honest to the same degree or more so than cloudflare. At least from what I’ve noticed, especially this year.

boarush 2 hours ago

Azure straight up refuses to show me if there's even an incident even if I can literally not access shit.

But last few months has been quite rough for Cloudflare, and a few outages on their Workers platform that didn't quite make the headlines too. Can't wait for Code Orange to get to production.

jacquesm 2 hours ago

Bluntly: they expended that credit a while ago. Those that can will move on. Those that can't have a real problem.

As for your last sentence:

Businesses really do care about the incident reports because they give good insight into whether they can trust the company going forward. Full transparency and a clear path to non-repetition due to process or software changes are called for. You be the judge of whether or not you think that standard has been met.

boarush 2 hours ago

I might be looking at it differently, but aren't decisions over a certain provider of service being made by the management. Incident reports don't ever reach there in my experience.

samrus an hour ago

jaboostin 2 hours ago

Hindsight is 20/20 but why not dry run this change in production and monitor the logs/metrics before enabling it? Seems prudent for any new “delete something in prod” change.

anurag 2 hours ago

The one redeeming feature of this failure is staged rollouts. As someone advertising routes through CF, we were quite happy to be spared from the initial 25%.

user205738 an hour ago

They should have rewritten this code in Rust using these brilliant language models. /jk

djfobbz an hour ago

I'm honestly amazed that a company CF's size doesn't have a neat little cluster of Mac Minis running OpenClaw and quietly taking care of this for them.

ssiddharth 2 hours ago

The eternal tech outage aphorism: It's always DNS, except for when it's BGP.

dilyevsky an hour ago

> Because the client is passing pending_delete with no value, the result of Query().Get(“pending_delete”) here will be an empty string (“”), so the API server interprets this as a request for all BYOIP prefixes instead of just those prefixes that were supposed to be removed.

Lmao, iirc long time ago Google's internal system had the same exact bug (treating empty as "all" in the delete call) that took down all their edges. Surprisingly there was little impact as traffic just routed through the next set of proxies.

vimda an hour ago

One has to wonder when the board realises Dane was a bad replacement for JGC. These outages are getting ridiculous

tokyobreakfast 2 hours ago

Is this trend of oversharing code snippets and TMI postmortems done purposely to distract their customers from raging over the outage and the next impending fuckup?

samrus an hour ago

Just seems like transparency. I agree that we should also judge them based on the frequency of these incidents and amwhether they provide a path to non-repeatability, but i wouldnt criticize them for the transparency per se

alansaber 2 hours ago

Well I still appreciate a good postmortem even if I have no doubt it'll happen again imminently

bdangubic 2 hours ago

and if they didn’t we’d posting about lack of transparency. damned if you do, damned if you don’t

wa008 an hour ago

This transparent report can earn my trust

VirusNewbie 2 hours ago

If you track large SaaS and Cloud uptime, it seem to correlate pretty highly with compensation for big companies. Is cloudflare getting top talent?

bombcar 2 hours ago

Based on IPO date and lockups, I suspect top talent is moving on.

henning 2 hours ago

Sure vibe-coded slop that has not been properly peer reviewed or tested prior to deployment is leading to major outages, but the point is they are producing lots of code. More code is good, that means you are a good programmer. Reading code would just slow things down.

sp00chy an hour ago

that’s my feeling also. We will get this more and more in future.

NooneAtAll3 2 hours ago

again?

dryarzeg 2 hours ago

DaaS - Downtime as a Service©

Just joking, no offence :)

logicchains an hour ago

DaaS is good ja