Hacker News

Sympathy? It's a Microsoft company that is being ran with a consistency of a startup in early seed rounds. Their downtime is abhorrent and unacceptable as far as enterprise goes. Their engineers look like absolute amateurs allowing for such low class work it results in their customers experiencing industry leading downtime.

> I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers.

I don't think this is commendable at all. I give GitHub a lot of money and I'm tired of it being wasted with downtime.

I think somewhere down the line you have an answer to problem itself. Its for a while that companies are not into the business of sympathies, we should probably not get into that side for the topic.

1. Github has enterprise users who paid for the service, their day job requires Github to be available and working

2. Github has generous free tier which is the one which is exploring a lot more with the AI generated code.

It is a complexity in itself but the traffic should have been separated, the free users should not be allowed to bring down Github for enterprise customers (Just to clarify, I am free user myself). And if they do not have capacity it would have been perfectly fine to push back or throttle new users/repositories.

We have a business relationship so no we shouldn’t have any sympathy. They sell a service and they’re failing to provide it.

They did that to themselves, and doubly so.

They decided they needed to capture the whole open source ecosystem by turning open source work into social networking... on a proprietary platform (because open source is great, especially when it's others' software). That was before they joined Microsoft.

And then Microsoft pushed AI everywhere, including on GitHub itself with copilot.

I would have liked if they had left the open source projects alone and didn't create that FOMO for not using them.

I have no sympathy.

I don't typically have sympathy for businesses that fail to deliver a service as advertised.

I can have sympathy for the humans caught in the crossfire but only managing one nine of availability on a commercial service is not acceptable.

> they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI.

That is an active choice they are making, over and instead of any reliability for the rest of their users.

It is a problem they are embracing, and actively encouraging, for themselves.

I have a lot of sympathy for the people working at GitHub trying to keep it going, but I do not have sympathy for GitHub. That's what they're paid for. I have things to get done, and it has no sympathy for me.

> have a little more sympathy for the billion dollar company

No.

Their leadership went all in on AI and in the last blog post essentially admitted some missing test coverage for a critical path. Time to learn lessons and fix your vibe coded shitslop and stop using "user graph go up" as some kind of excuse.

"Leave the big trillion dollar corporation alone" is not the right idea.

Yeah, all those 1100 commits per second. Like, is this the standard for 'scale of traffic' that we're impressed by these days? And it's not like this snuck up on them. The graph shows a very consistent trend that gave them months and months to respond.

Maybe for a free account.

But we pay enterprise license and GitHub is a big dependency in our software flow.

If this continues to be a problem as an enterprise product they need to do something. Otherwise theyre are going to to start losing business

Microsoft made $134B in net profit last year, and is currently valued at nearly $4T. They have the resources to solve this if they wanted to.

If you do the math, their numbers are something like 1000 commits per second. That's not pushes per second either, it's just the total number of commits summed up over a week, not all of which trigger much of anything to happen.

Yes, that is scale. And yes, that's not actual requests per second. But it's the sort of scale that big (and even mid-sized) tech has known how to deal with for decades. Microsoft doesn't have an excuse.

idk man, we pay stupid amounts of money to microsoft, we are an enterprise customer, expecting better availablity compared to my laptop isnt really a high bar.

Constant building with AI is something that they (Microsoft) promote and are heavily invested in.

For that matter, I admire their relative transparency about their incidents. I can think of other big players who will cheerfully show green statuses across the board while everyone can see that their pants are down...

lol they are Microsoft, and the AI stuff has only multiplied usage like 2x, they haven't had sudden exponential growth

I strongly suspect that quite a few HN members are busy creating agents that are furiously stuffing GH.

As I said before (and it was not a popular comment), it's easy to be an "armchair quarterback," with these services, as I think the brittleness was already baked in, and just waiting for the right time to crack. The only true way to have a robust platform, is to design something that will scale, from the start, and many startups don't do that, because they are feverishly trying to get out an MVP; even if it is a mess of bubblegum and baling wire.

They always say "We'll get it done right, once we get funding," but that never happens.

But my sympathy is limited by the fact that MS paid a lot of money for this, quite a while ago, and that is one company that knows all about issues of scale. They should have seen this coming.

Paying enterprise clients who can not push/pull code and collaborate on PRs (table-stakes) disagree with having a little more sympathy.

It’s commendable that they let people give them training data for free?

> I think we need to have a little more sympathy for GitHub

We certainly do not.

Our company pays for GitHub. We're paying for a broken product that stops our work. I don't have sympathy for the companies whom I pay for a product and give me broken shit in return. This is entirely preventable and their own fault.

A restaurant makes pizzas. They suddenly get 100x more popular. They can't make 100x more pizzas. But they are still taking orders from 100x more people. Not only are they not getting enough pizzas delivered that they took orders for, but in their rush to make and deliver more pizzas, they set the kitchen on fire, which makes an even longer wait for pizzas.

When the pizza you ordered doesn't get delivered, do you have sympathy for the restaurant? Or do you tell them to stop taking orders they can't fill and try not to set the kitchen on fire?

Now consider the pizza restaurant has 21 billion dollars in cash, is taking your money, and not giving you pizza.

No, we don’t.

Depends who you are aiming sympathy at. C-levels or Engineering?

The question is, do we think Google would have the same problems running the service? (I'm biased from having worked there tho.)

I’d have sympathy for the pre-Microsoft GitHub. For the thing they are now, backed by a top-5 company with trillions of dollars? Cry me a river.

GitHub literally went all in on doing almost nothing but AI-related feature additions for Copilot. So they built their own prison.

I don't have sympathy for them. They have disrupted my work so much in the past month that it's ridiculous, and my company likely pays them millions of dollars. And you can see from this site's plotting that there is a trend towards more frequent and more critical outages. It has been extremely bad the past month.

"sympathy" for one of the worst corporations in history is some insane corp bootlicking logic.

This is a multi billion dollar corporation. History of robbing and stealing from others of their labor or IP. History of enshittifying once great services.

Give me a break. Sympathy? For microsoft? That might have flown when github was like seven people, but they have nearly unlimited resources to make it better. They're just choosing not to. Let's talk contracts and money before we pull the sympathy card.

I used to be on-call in a high-traffic environment where single customers pushed more bits than entire nations. I chose the role. I didn't want people's sympathy, if anything, I wanted them to complain to management.

If it gets too bad they can quit. Maybe that would be for the best, just wear the thing down until it outright fails and no one wants to touch it. One less bullshit service sucking all of the oxygen out.

Gets me wondering… if I get used to the fact that some location is busy, I stop going there. Other people might do the same. Could this create a detectable wavy signal over time? People stop going because it’s busy. It stops being busy. People go because it’s not busy. Repeat. Is this a known phenomenon?

Many people think we're on the path to having separate webs for AIs and humans. This may be the beginning of that.

Random AI slop code pushers don't pay the bills. Traffic isn't revenue.

I've started suggesting to my CTO that we move to something else because we do pay the Github bills and yet we are also affected by other people's activity. Avoiding that is not a hard engineering problem for an alternative.

A world where a record amount of code is pushed to GH but a large amount of paying customers leave would be "cooked"

Ahhh, I think the author mixed up two values here. That value seems to actually be the average over the past 3 months.

    const incidents = e.detail.incidents;
    
    // ...snip...
    
    const now = new Date();
    const threeMonthsAgo = new Date(now);
    threeMonthsAgo.setMonth(threeMonthsAgo.getMonth() - 3);
    
    // ...snip...
    
    var currentFreq = recentIncidents.length / 3; // <- We out here, smoking these guns with our homeboy Claude
    
    // ...snip...
    
    var earliest = null;
    for (var j = 0; j < incidents.length; j++) {
      var dd = new Date(incidents[j].started_at); // <- eventually incidents[j].started_at is "2016-03-01T07:07:37.000Z"
      if (!earliest || dd < earliest) earliest = dd;
    }
    
    // ...snip...
    
    document.getElementById('n-since').textContent = earliest
      ? earliest.toLocaleDateString('en-US', { month: 'long', year: 'numeric' })
      : '?';
    document.getElementById('n-rate').textContent = Math.round(currentFreq * 10) / 10;

#n-rate has nothing to do with the #n-since month, it's just the last trailing 3 months. And even then, it's sort of underbaked? It's moving the date back by 3 calendar months not taking into account differing numbers of days, so it'll under-report short months.

I wouldn't trust the stats here.

Edit: whoops, author updated the template while I was writing this! It now says "Over the last 3 months", though that's still calendar months.

Not sure whether it has been updated since your comment, but the sentence now reads:

    GitHub has had 1125 incidents since March 2016. Over the last 3 months, they've averaged 24 incidents per month

> GitHub has had 1125 incidents since March 2016. Over the last 3 months, they've averaged 24 incidents per month (↓ 5% vs prev 3mo).

Looks like they fixed it already

Prob used Copilot to write the excel formula...

Relevant xkcd: https://xkcd.com/2236/

I wonder if this is a Microsoft Excel situation where everyone

1. Hates the status quo

2. Only uses a handful of the 10,000 features the status quo has

3. Uses a different handful than everyone else

Because that model kept Excel as the center of buisnesses around the world for 30 years and counting

Actions aren't secondary to most paid GH users; and if they are down it usually means no deployments and no tests, which can often halt work.

GitHub Actions is really, really badly designed. The security model is fundamentally broken, the YAML hell is as bad as any, the log streaming lags like hell, they charge self-hosted runners for using their coordination plane, jobs queuing is really slow, and their software for actually running jobs is cursed and designed in a way that is practically hostile to self-hosting.

GitHub Actions is god-awful. Have you ever used any other CI tools?

My first thought today seeing GH actions were degraded was "some good those 3 million cores did"

As a daily GitHub user in Australia I still haven’t figured out why everyone’s complaining about uptime. :)

> Their incident-free uptime on 9-5 EST, Mon-Fri, is probably like 60%.

And it may be even worse in EDT, which is currently in effect!

what's normal working hours? very US centric comment IMO. Europe, India, China, Latam etc. don't fall into your 9-5 EST normal working hour bucket.

The GitHub Outrage Tracker is at this link: https://news.ycombinator.com/from?site=githubstatus.com

That one would be like iscaliforniaonfire.com

6 years is a good run. For comparison, every 6 years there's a not insignificant chance you die. A restaurant that lasted 6 years would be an outstanding success. 6 years ago the very first LLMs were a novelty toy. 6 years before that was just after HTTPS became popular. 6 years before that was about when the iPhone came out.

Hey! isgithubcooked.com is my site; I made this site in February '26, I think

The contribution graph as an outage calendar idea is a commonly recurring one :). I definitely saw it somewhere else as a static asset first before I made this site.

https://news.ycombinator.com/item?id=48034587