Incident with Pull Requests, Issues, Git Operations and API Requests (githubstatus.com)

327 points by maxnoe a day ago

gen220 a day ago

https://isgithubcooked.com

Normally I defend GH in the comments of these incidents but it’s been an impressively bad month by their standards, even when you filter for critical components filter out sev-2’s and 3’s.

mirekrusin a day ago

It's not physically possible to run post-mortems for issues at those rates.

They should install OpenClaw for that as well.

lenerdenator a day ago

AI: The cause of, and solution to, all of your tech debt.

baalimago a day ago

Perhaps best to simply declare indefinite-mortem

miningape 4 hours ago

embedding-shape a day ago

> It's not physically possible to run post-mortems for issues at those rates.

Not at all, you merely move the goal post of at what layer the "root cause" actually could come from! At that speed, it's always something short and sweet, while when you actually want to long-term address things, you have to have time to even investigate organizational issues or whatever the actual problems stem from.

But you have half a day? "Post-mortem: Push X wasn't properly analyzed before deployment, in future more testing" and call it a day.

drob518 4 hours ago

connorboyle 17 hours ago

Wow, it seems that 100% of sev-3 ("critical") incidents in the last year (=365 days) have occurred between April 22, 2026 and now.

Is it possible that there has been a change in the way the data are collected/recorded that even partially accounts for this sudden onset?

gen220 16 hours ago

One tangent, I believe sev-0 is actually "critical" (at least as how I'm used to reading it), and the higher you go the less critical something is.

IMO as a github-watcher, I think they changed their definition of what constitutes a sev-0 between sev-1 for the better. In particular, they had a few "sev-1"'s around the turn of the year that would be classified as sev-0's if they happened today.

Pre-4/22 GitHub sev-1 was a normal SaaS company's sev-0, imo. So I think their new system is more reflective of reality. My guess is that a few of their big customers bullied them to have more accurate SEV categorization.

lazide 16 hours ago

Waves around it had to break eventually eh?

rsyring a day ago

Of all the sites/graphs I've seen of GH outages, this one is the most striking IMO:

https://damrnelson.github.io/github-historical-uptime/

Unfortunately, it doesn't look like it's being updated with new data. But it wouldn't look any better for GH if it was.

gen220 21 hours ago

FWIW, I'm not convinced that chart is necessarily an accurate representation of pre-acquisition reality. It would really surprise me if GitHub did not have a single sev-0 pre-acquisition, but it wouldn't surprise me if they were not formally captured and reported in a format that would make its way into their current status page's database.

crote 17 hours ago

rsyring 18 hours ago

stogot a day ago

I wonder what the cause of this was? Microsoft Politics? Bureaucracy? Forced move to azure?

felooboolooomba 18 hours ago

taintlord223 a day ago

The UI of that page is so nice, should build a github competitor.

The user profile / contributions and PR UX is pretty much the entire "hub" product since git is a fully separate offline app.

embedding-shape a day ago

> The UI of that page is so nice

Is it? Seems a text description of "Make a website outlining 'How cooked GitHub' is with a modern style" to basically any LLM would produce exactly that UI and design, literally nothing of that design a human had any influence on, besides the ones selecting what training data the used LLMs was trained with.

I think most of us who've tried using LLMs for web-design can recognize that style and design at this point, regardless of model actually used.

gen220 a day ago

angrydev a day ago

Hamuko a day ago

olmo23 a day ago

vinnymac a day ago

I’m actively working on an alternative Frontend for Forgejo at the moment, completely self hostable, free, and open source.

Moving everything from GitHub to Forgejo and Tangled for now. These outages haven’t effected me for the past month because of this.

My_Name 6 hours ago

jmusall a day ago

voxic11 a day ago

The UI is in the default claude code style

FpUser a day ago

>"The UI of that page is so nice"

Most part screen is taken by picture. Contrast ratio is really low. Hard to read Should they remove that useless banner, current status which is the most interesting part coud've been made visible right away.

I would call this whole thing highly un-ergonomic

DetroitThrow a day ago

Lol it's pretty bad UI

EduardoBautista a day ago

May has been filled with critical issues. It seems it's getting worse over time.

hbn a day ago

Commits are up 14x year-over-year

https://x.com/kdaigle/status/2040164759836778878

tom1337 a day ago

bushbaba a day ago

greatgib 18 hours ago

btown a day ago

Is the “streak” days of continuous uptime, or of days with at least one downtime incident? I think it’s the latter :]

joshuaissac a day ago

It looks like it is the number of consecutive days with no incident. If you look at 31 Dec 2025, that corresponds to an 8-day period with no incidents.

isityettime a day ago

gen220 a day ago

It's a streak for continuous uptime, and yeah it is fairly depressing to imagine overseeing that :/

pluc a day ago

Name one thing Microsoft didn't run into the ground post-acquisition

robotmaxtron a day ago

hey now, LinkedIn was terrible before Microsoft.

SteveNuts a day ago

storus a day ago

Mindwipe a day ago

darkamaul a day ago

I think Minecraft is still in good shape

embedding-shape a day ago

bspammer a day ago

pocksuppet a day ago

elzbardico a day ago

GH was acquired by microsoft some eight years ago. It has been working quite well until recently.

People may have had complaints about functionality, features, commercial issues, but the thing used to at least have a decent uptime until recently.

chris_money202 a day ago

05hundred a day ago

bsimpson a day ago

modriano a day ago

bigstrat2003 19 hours ago

Dave Cutler?

rvz a day ago

They are already cooked as this has been happening ever since the Microsoft acquisition and it was run to the ground before 2023.

At this point you would get better uptime by just self-hosting your own GitLab, Forgejo or Codeberg instance instead of dealing with Github's unreliablity.

There is no defending them with their clear neglet and carelessness of the platform.

pocksuppet a day ago

If all you need is a repository, you don't even need any of these. You need SSH access to a server, and optionally, one of several web front-ends. Git comes with a CGI script that handles public anonymous checkouts via HTTP(S), although since nginx doesn't support CGI, integrating those is a little bit tricky as you need a FastCGI wrapper.

vinnymac a day ago

I moved most of my projects off GitHub to Forgejo and will be using Tangled too for public repositories. I don’t think people realize that if you self host Forgejo, you get 99% of the functionality of GitHub with zero of the limitations. Especially if you have the hardware to spare for CI runners. And if self hosting isn’t your thing you can always just use Codeberg and Tangled directly.

I’m working on an open source Forgejo browser called Joui. It’s coming along nicely, and is so much snappier than GitHub in every single way.

root-parent a day ago

Like those aviators who draw a picture on flightradar24, if you filter by All Services - Critical, somebody almost about to draw a swastika just in May... Are the AI agents revolting?

ckorhonen a day ago

This is getting ridiculous. One particularly concerning thing I’m seeing is that pull requests on both the web UI and API aren’t reflecting all commits or branch changes consistently. It would be very easy to merge something without realizing you’re not actually reviewing the full diff.

mikeocool a day ago

Yeah, I've had several occasions recently (seemingly not related to any incidents on the status page) where I've had to wait 20 minutes to an hour to be able to open a PR, because Github didn't recognize my branch had any new commits compared to the base branch.

dude250711 a day ago

A taste of the agenticaly-developed world.

xnorswap a day ago

Before clicking, I assumed this was going to be a write-up of the one from a few days ago instead of an entirely new incident.

jamdav16 a day ago

I assumed it was the one from yesterday! Silly me.

gred a day ago

New PR: revert GitHub software and infrastructure to version of June 1st, 2018.

New PR: disable new user signups for 6 months

HR initiative: all future KPIs automatically require three-nines availability; all bonuses are forfeited, regardless of accomplishments, if annual availability falls below target

HR initiative: fire CEO and CTO

rahkiin a day ago

New PR: disable Github API New PR: block (ai) bots through attestation to make usage predictable

0xblinq a day ago

Finance initiative: Undo the Microsoft purchase

thr0w4w4y1337 a day ago

Github does not have a CEO

gred 20 hours ago

We're halfway there!

fapjacks 19 hours ago

Imagine if we did this for countries.

spaceman_2020 a day ago

is it me or ever since AI coding became the norm, there have been way more outages with otherwise reliable services?

I get downtime on Supabase every few weeks. Even Cloudflare. And now Github

maccard a day ago

GitHub’s instability started when they announced they were wholesale migrating to azure. They’ve been struggling with uptime since before the enormous surge in agentic coding they’re seeing. I don’t doubt that AI has massively increased their load, but u think this is more suspect than that.

dools 16 hours ago

I think they’re referring to services using AI for coding and shipping bugs more often as a result.

chris_money202 a day ago

Yes, because that caused the usage of the services to skyrocket, GitHub runs on Azure and Azure is experiencing capacity strain due to AI, so GitHub's services are struggling to auto-scale

voncheese a day ago

Per a report that came out the other day, the GitHub move to Azure has been slowed down (i.e. I don't think it's done). But maybe you have newer/better info than me

skywhopper 18 hours ago

GitHub’s data architecture (lots and lots of projects in a hierarchical distribution with immutable, highly transaction-based data elements) ought to be easily shardable to allow for massive scale-out. Especially a decade after acquisition by Microsoft. Even a major increase in activity (which was surely predicted internally years ago, given the marketing around AI) should be easily handled by a company with their resources. These repeated failures are indicative of massive mismanagement and misinvestment.

chris_money202 6 hours ago

tom1337 a day ago

They are definitely more outages but the question is if these outages are due to the providers using LLMs to build there products and are therefore not delivering the quality they did before or have LLMs enabled a completely new user base to create projects which they deploy in the free tiers of named providers and they simply cannot keep up with the growth and the new influx of free users is skewing their mixed calculations (free vs paid) so heavily that they cannot scale without losing money. I'd probably say it's a mix of both.

mhitza 21 hours ago

Enterprise software and support services have consideraby (in my personal experience) degraded over the last year.

Less care on process, or quality, more focus on "just ship".

I've also seen and heard from peers this happening in multiple smaller outsorcing companies.

sharts a day ago

Correct. There’s no incentive to be careful anymore when you can just prompt an LLM to fix it

julianlam a day ago

Not just you, but uncertain whether it's due to unreviewed slop going to production, or increased demand due to slop generation.

throwatdem12311 a day ago

> is it me or

No, of course not.

csomar a day ago

No but everyone is pretending that everything is fine. Actually, no, no one is pretending anything. No one cares, really.

hansmayer a day ago

No, it's not just you. It is fairly obvious what's happening - the same old Entshittificators now have a great tool to up the speed of entshitification by 100x - thus these crappy outages every other day.

robin_reala a day ago

Good that the Billing functionality is still at 3 nines at least.

ale 20 hours ago

They got their priorities right for sure

simpsond a day ago

Well, the significant growth comes from freemium usage. A whole lot of vibe slop triggering actions, with no supporting business. So revenue has not tracked all other growth and the billing system isn’t stressed.

My_Name 6 hours ago

I'm finding this amusing because today, I am in the process of removing all my code from GitHub and cloning it to a local Git that has a webhook to a local releases server.

Systemic33 a day ago

Someone linked this third-party "honest" status page:

https://mrshu.github.io/github-statuses/

Seems more accurate with my experience of GitHub.

eithed a day ago

I'd appreciate if they'd not mark the incident as resolved when there's still fallout - ie: my commits didn't display on the branch, my actions didn't run

It's the same issue as the other day - display message at the top admitting that cache needs to be refreshed (or whatever the wording was)

carreau 7 hours ago

Reminder that after more than a month, many repos _still_ don't show all PRs and Issues https://github.com/orgs/community/discussions/193463

throwatdem12311 a day ago

Maybe GitHub needs to freeze free repository creation until they get this under control because this is ridiculous.

embedding-shape a day ago

I mean, if we're talking about "fixing" the symptoms of the downtimes rather than the sources and causes, I guess they could just null route github.com until they have things under control?

Personally, I think they'd have more luck actually attacking the source, what that might be. Somehow I think Microsoft's push for "Every developer only use AI for development, no manual thinking/coding from now on" is the detrimental step, seemingly many companies are still discovering the right approach. Put a freeze to that, and I'm fairly sure you'd see less downtime pretty much immediately, unless all real engineers already left the company, I'm sure I would have at this point.

throwatdem12311 a day ago

It’s not just their own slop that’s causing this, it’s also caused by the tsunami of slop being uploaded by vibe coders.

If you want to upload to GitHub, you should pay. The days of charitably giving away compute for the “open source communities” are over.

Grandfather existing public repositories in, then cut it off. Stop the bleeding. It doesn’t have to be forever.

goda90 a day ago

Or maybe they need to bring back quality assurance expertise to the company.

throwatdem12311 a day ago

Yes that would be part of getting things under control, of course.

ethagnawl a day ago

And/or move more contextually aware humans with 10K+ hours of hard won experience and fear of failure/sense of pride back into the loop.

rldjbpin 7 hours ago

among all the recent incidents, this was probably the first time i experienced any service disruption.

even comparing branches were giving 500 in the web ui. even github copilot in their new session limits now also mention if github is going through an incident, which is plainly comical.

fun times!

renehsz a day ago

chrisweekly a day ago

Good link - why and how to ditch GitHub.

sibidharan a day ago

Are they running paid marketing campaigns for Gitlab ?

ramon156 a day ago

Self-hosting forgejo under tailscale + mirroring public repos through GitHub

Has worked wonders for me :)

varun_ch a day ago

Forgejo is fantastic. I do think it could use a fresh coat of paint from a designer but it’s otherwise really good.

Gitea (what Forgejo forked from) recently stole the sidebar on repos from GitHub and I think that would be great for Forgejo to steal too…

Forgejo themed by Codeberg: https://codeberg.org/forgejo/forgejo (the codeberg theme is extremely low contrast)

Forgejo default: https://v15.next.forgejo.org/pparaxan/quark

Forgejo themed by Lix: https://git.lix.systems/lix-project/lix

Gitea: https://gitea.com/gitea/awesome-gitea

Gitea themed by Blender: https://projects.blender.org/blender/blender

I personally like Blender’s Gitea theme better than the rest but I guess that’s subjective. In dark mode I do not like the low contrast Codeberg theme or the default Forgejo theme, but all of the instances custom themes look great.

As far as Git forges go in general though.. tangled is very pretty https://tangled.org/tangled.org/core I think more power user oriented software should be comfortable with compact interfaces

maxfurman a day ago

myng111 a day ago

It's fascinating how fast the Forgejo I host at my university's laboratory loads from my home network. Every page load is <100ms. I think it goes to show how much bloat we don't realise exists in modern webapps.

robin_reala a day ago

preisschild a day ago

Ideally those forgejos would safe enough to be on the public internet (and using a federation protocol like activitypub) so we don't have to go through a centralized service such as github and not locked behind private networks (such as tailscale nets)

KptMarchewa a day ago

It's so unfortunate that Gitlab is a complete mess, that GitHub has no real competition now. I can only think that few months to a year from now there will be _something_ that works on an enterprise scale.

xnorswap a day ago

Have you forgotten about Azure Dev Ops aka Visual Studio Team System aka Team Foundation Server*?

Yes, it's still Microsoft, but they've forgotten about it, so it runs entirely adequately and is actually a surprisingly okay github replacement. It does nothing special, but it does do everything, just in a way you often would rather it wouldn't. It doesn't have the flexibility of JIRA for the ticketing, and the deployment machinery doesn't have the fanciness ( and vendor threat ) of chaining github actions, but it does handle both.

I haven't used gitlab, so I'm curious to hear what makes it a "complete mess" too.

* Microsoft's headless chicken naming strategy in full force, it's a miracle they haven't yet renamed and rebranded it to align with copilot yet.

zdragnar a day ago

KptMarchewa a day ago

manytimesaway a day ago

EduardoBautista a day ago

I'd consider self-hosting GitHub Enterprise before putting my team through the pain of Gitlab.

ricardbejarano a day ago

How is it painful to use GitLab? Curious, as a user of both, I find them both nice. I like GitLab CI/CD more than I do GHA, but that's personal preference/bias more than anything objective.

tux3 a day ago

EduardoBautista a day ago

IshKebab 21 hours ago

ale 20 hours ago

I think it’s about time to backup all of my GH repos.

voidUpdate a day ago

They may have gotten down to only 2 nines on most of their services, but at least the LLM is still running at full power! must increase value for shareholders

rozab a day ago

If you go to www.githubstatus.com, the downtime is not showing in the chart. I was annoyed enough yesterday when I visited this page to figure out why my Actions had failed and was greeted with big green ticks and only a tiny red rectangle halfway down the page to indicate the problem.

This time they've just scrubbed the evidence outright?

madeofpalk a day ago

It was previously showing, but I believe the incident has bee resolved now. At least, PRs work for me when they previously didn't.

Symbiote a day ago

The "Git Operations" chart is showing all green, but several of the recent blocks have a note showing there was an outage.

Today's is green, even though there was an outage.

looperhacks a day ago

Maybe we should start posting av story when GitHub has been fine for some time instead of posting every incident

dzonga a day ago

git is supposed to be decentralized.

maybe it's time to revert back to the central idea of git & not centralize around a particular provider.

for issues - mailing list will do. you can always slap a beautiful ui if you want to or a tui (as is the fad) these days.

actions can also be decentralized via an API spec & webhooks.

OsrsNeedsf2P 21 hours ago

Github Pull Requests are down to 1 nine of availability

hydrogenbon007 a day ago

The software reliability and uptime is going bad across the industry, railway, github etc

wild that there is a large pattern forming up of unreliable software being pushed

fen4o a day ago

Tried to do a git push - it succeeded after 3 mins. Then I wanted to open a PR and it failed with a 500 error.

Facepalmed and decided that this is it for today.

cedws a day ago

I'm so done with GitHub.

maxnoe a day ago

GitHub Incident again/

denysvitali a day ago

At this point we can even stop specifying that it's GitHub...

abhashanand1501 a day ago

as a github user, we are paying for the slow git operations through our github action minutes, if someone from GH is here, will you be compensating for it?

trenchgun a day ago

But everything else still works fine, right?

Robdel12 a day ago

I know it’s super fun to shit on GitHub and everyone’s favorite thing to say is “build a competitor!”

They’re trying to scale from 1 billion commits last year to over 14 billion this year. I have zero desire to try and manage that scaling. Basically being DDOS’d by agents all day now.

https://x.com/kdaigle/status/2040164759836778878

fg137 5 hours ago

> I have zero desire to try and manage that scaling.

Last time I checked, this is called "doing their job".

You are free to choose what company you want to work for, but for github, if they are offering the services, often as paid services, they have an obligation to keep things running which is called SLA. Otherwise people will leave (and are leaving)

sethops1 a day ago

If it were humans using the site it might be super motivating, but knowing that it's 99% bots producing AI slop, yeah I'd be looking for a new job.

dist-epoch a day ago

GitHub is not agent scale.

Multiple companies are trying to create new versioning primitives/architectures which can handle machine-level code generation - 1 commit per second per repo.

It's like switching from horse buggies to automobiles, the whole worlds needs re-architecturing to handle the new load.

The age of boutique hand-coding is being replaced by the age of industrial software factories.

swiftcoder a day ago

> new versioning primitives/architectures which can handle machine-level code generation - 1 commit per second per repo.

This is not a particularly novel level of scale. Facebook's mercurial backend had to handle >5,000 developers committing to the singular monorepo long before LLMs were a thing

KptMarchewa a day ago

Yes, on a single repo. Now multiply that per bazillion companies on github, some of which are trying that.

julianlam a day ago

Why the heck would you want to do this. Using git as your undo chain sounds like a pretty awful thing to do.

dpkirchner a day ago

I think it'd be pretty neat to be able to rebase my undo history on to a remote branch someone else is working on.

fapjacks 18 hours ago

skinfaxi a day ago

This seems odd to me. Why would you need to commit every second?

gedy a day ago

And push to remote as well? Seems not thought out

andyjohnson0 a day ago

> GitHub is not agent scale.

Is the scaling issue with git or github?

drcongo a day ago

For years we had a GitHub status thing in our Slack but I had to remove it about a year ago because the noise got too much, it would be unbearable in 2026.

Hamuko a day ago

Yesterday my CI runs wouldn't even be created because Actions was eating shit, and today my CI runs get created but fail because the API is eating shit. Fun.

emartinez-dev a day ago

I thought it was the yesterday's thread but no, here we go again

rvz a day ago

Again?

It was just yesterday [0] that GA was down and another incident today? I am convinced that Copilot and Tay.ai are destroying GitHub and there is no CEO of GitHub to contact.

Now will you please self-host as I said 6 years ago? [1]

[0] https://news.ycombinator.com/item?id=48278635

[1] https://news.ycombinator.com/item?id=22867803

KptMarchewa a day ago

GitHub famously does not have a single 9 of uptime.

rob a day ago

You think a Microsoft chatbot from 2016 is destroying GitHub?

insider123 a day ago

Microsoft corporate culture destroys Github reputation and tech.

rvz a day ago

At this point, you might as well say that is what's happening at GitHub with the help of GitHub Copilot since nothing has changed and has only gotten worse over time.

Arbortheus a day ago

Fed up and bored of this

hehe1 a day ago

dthtrj

shevy-java a day ago

I think it is time to decouple GitHub from Microsoft. Microsoft has shown irresponsible behaviour - and this continues. They keep on going at it until nothing works anymore. Typical microslop work.

pocksuppet a day ago

You don't get to control that. It is Microsoft's right to do whatever it wants with GitHub - it could shut down tomorrow, or demand face ID. If you want to control what happens with a thing, you have to make the thing instead of letting someone else make the thing and sell it to Microsoft.

Your choice is to accept the product that exists on the market, switch to another product that exists on the market (such as Codeberg or self-hosted Forgejo), make your own product, or not use any.

OkayPhysicist a day ago

You missed the option to "lobby the government to tell Microsoft to straighten up and fly right". Microsoft is a corporation, a legal entity only created with the permission and grace of the State of Washington.

pocksuppet 21 hours ago

hansmayer a day ago

It seems before AI eats software, its going to first eat GH and Microsoft.

throwatdem12311 a day ago

they didn’t think the leopard would eat their face!

markus_zhang 18 hours ago

The moral of the story is: we don’t even need one 9.

OK now people can layoff even more engineers and feed the tasks to AI.

/s