The August 17 outage, and the work ahead (github.blog)
45 points by 0xedb an hour ago
blakesterz an hour ago
"Since April, monthly commits have grown from 1.4 billion to 2.9 billion. "
brookst an hour ago
It really is. I know I've gone from tens a month to thousands a month. They have to be projecting >100B/month in the next year or two.
xyzsparetimexyz 34 minutes ago
wow. they should really institute a maximum amount of individual pushes per-month per-user.
jdm2212 an hour ago
> Errors in those services triggered a client-side retry loop that increased traffic during recovery.
The worst outages I've been part of always have some version of this :(
k33P1Tr3aL an hour ago
the 'ol thundering herd problem...
pixl97 41 minutes ago
Exponential backoff is your friend... too few people use it.
ivraatiems an hour ago
"We are committed to fixing these problems, as long as it doesn't involve buying things other than AI computers, hiring humans, or using non-Microsoft products."
Calling Azure the solution to this problem when it is in fact the source of most of these problems is just fantastic doublespeak.
Github is ripe for disruption and I hope it is disrupted soon.
dcrazy an hour ago
> We installed as much hardware as available power allowed in our existing data centers while accelerating our migration to Azure.
And from the RCA [1]:
> The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic.
ivraatiems an hour ago
"While accelerating our migration to Azure," meaning, they will only solve problems if it helps them also use Azure more.
It is unbelivable that aload of 2.8b commits was totally fine, and a load of 2.9b was a sitewide outage, unless they have no reporting or their tooling is completely incompetent. If things can fall apart so easily, throwing more capacity at the problem won't fix it.
dcrazy an hour ago
cyberax an hour ago
rcleveng 39 minutes ago
bpavuk 44 minutes ago
I'm betting on Tangled and Codeberg. Tangled has a better press and in general is a dark horse, Codeberg has the "brand" and some network effects from projects that moved to there. (famously, Zig.) I heard that Sourcehut is having a moment as well, and I love the idea of email-based workflow and not having to have an account to contribute to someone's project hosted there, but I'm not maintaining anything worthwhile paying the $4/mo sub.
kjellsbells 41 minutes ago
Ok, but there's no universe where a major Microsoft-owned property is not being forced to run on Azure. Just like AWS pushing to get off Oracle back in the day. It would be career-destroying to suggest otherwise regardless of technical merit (and tbf, no infrastructure is bulletproof, unless you want to port GitHub to z/OS on mainframe)
mort96 an hour ago
If you're a big company, you can afford having one engineer spend one or two days per year to maintain your self-hosted GitLab or Forgejo. On top of better reliability than GitHub, you'll get the additional bonus that your source code won't accidentally leak through being in Copilot's training set.
If you're a hobbyist, Codeberg is great, has a nice community and automatically shields you from slop contributions.
ivraatiems an hour ago
The issue with these systems is that they lack Github's sophistication for issue tracking, knowledge transfer, and automation. I think Gitlab is a mature product in its own space and unlikey to change, for instance, at this point.
Codeberg also has the issue of having a political stance which means they will not accept just anyone's use of the platform. That is absolutely their right and I have no issue with it, but it's unattractive to me - as someone who agrees with most of their current politics - because the day they decide they don't like me, I'm screwed.
mort96 an hour ago
Shish2k 40 minutes ago
rcleveng 38 minutes ago
unrented7977 37 minutes ago
Speaking from experience, it cost mW about a week or two per year to maintain GitLab for the startup I worked at.
My personal GitLab on the other hand really does take only a day or two per year.
That said, a week or two per year is just what it costs to maintain any one thing period. I spent about that much time maintaining PCs in the office, or my personal proxmox setup. It's not onerous at all.
GitLab is super bloated and a little sucky to admin, but it's not too bad all things considered. I'm admin in my new job's GitHub org and it sucks a whole lot more to maintain.
awesome_dude an hour ago
> Github is ripe for disruption and I hope it is disrupted soon.
It's an expensive, low revenue generating site.
There are, and have always been, competitors, including "host it all yourself" solutions, but nothing has really stuck.
How is it "ripe" for disruption?
ivraatiems an hour ago
They had $1b revenue in 2023 and now probably more than $2b in revenue... do you have cost figures showing what their expenses are?
bluedino an hour ago
awesome_dude 44 minutes ago
kjuulh 31 minutes ago
Reading this port-mortem / plan shocks me, this doesn't look like a service that has been serving high-throughput services for more than a decade. In fact it is almost like they've barely started. It seems the solution has been capacity, capacity rather than architectural or data changes.
> Our next milestone is an architecture that scales read capacity linearly with the number of readers, enabling unlimited read operations
How do you not have read-replicas / read caches at this scale yet? Which is what I am reading from this statement. You can of course get really far with sharding and whatnot. But at some point it might become worth it to engineer your data into a model that scales better.
dpweb 41 minutes ago
Sorry to suggest this but if they charged everyone say $1/mo. it would absolutely help the massive surge from AI coding they seem to have had.
I don't like paying for free stuff but gh certainly worth it.
Shish2k 32 minutes ago
Unfortunately time and time again, the overwhelming majority of people show that they would rather deal with an unreliable scummy company for "free" than pay $1/mo for a reliable service which treats them with respect :(
chrisjj 22 minutes ago
Surely that's the point. Shed users.
dcrazy 34 minutes ago
Pay per issue and commit. Buy 1,000 commit credits at a time.
Might force people to review their slop before pushing it.
sajithdilshan 30 minutes ago
The comments just shows how entitled people have become. Most people use GitHub and features for free and have the audacity to complain.
The outage is due to massive load increase. In 4 months the number of commits doubled to 2.9 Billions. Anyone worked with high load systems knows that’s it’s not a normal growth and how difficult even to keep on horizontally scaling in a short time period such a complex system.
GitHub should charge at least maybe 5$ monthly fee and most of the entitled freeloaders would leave the platform and it would free up resources
yipinwong an hour ago
AWS CloudWatch has an option to show the trend and what it will be like after x-period.
Doesn't Azure have such options so that engineers can predict to scale better? Seems like engineers are not ready for this per postmortem
jdm2212 an hour ago
The trend line does not tell you what will actually happen at scale, even if you think you're perfectly prepared for the next 10% or 20% growth. As Mike Tyson put it, "everyone has a plan until they get punched in the face".
rawland 34 minutes ago
dang and friends, can you lock (some) threads? This is getting out of hand, it seems to me?
GitHub outages seem to be very normal and even more emotional these days. Yet, that’s no reason to treat each other destructively. It’s sad.
So let’s turn this energy into creating something better.
rcleveng an hour ago
Great read - I'm glad they realize there's work ahead but what I'm missing is: * Paid customers: we know you pay us often a ton of money, and we burn your month on actions during these outages - we'll refund you for the days we spent your money and gave you no value. * Paid customer: We know you put your trust in us, so we'll ensure we have a separate pool of capacity to ensure we can keep that trust. * Paid customer: we'll proactively refund you when we miss our SLA.
What I read from this is: * Scaling is hard, we don't have enough capacity * We give away a shitton of compute for free * I have to talk about Azure not being a steaming pile of poop, otherwise my bonus will get tweaked downward in the next comp cycle.
Notice there's nothing about paid customers, I'll add in what they are missing:
Paid customers: Go F*ck yourself, you don't pays us enough to be an interesting line item compared to windows server.
rcleveng an hour ago
[dead]
iSloth 38 minutes ago
Has to be one of the most vague outage summaries of the year
dcrazy 32 minutes ago
The root cause analysis is separate, but linked to from the blog post: https://www.githubstatus.com/incidents/zkxwbgr0cnmx
It is specific and technical.
undefined 32 minutes ago
kvemkon an hour ago
I fear to ask, how archive.org keeps up to catch all those events for archiving...
ChrisArchitect 16 minutes ago
Related recently:
GitHub has alternatives, but no replacement
https://news.ycombinator.com/item?id=49135365
Why developers are ditching GitHub for Codeberg and self-hosting alternatives
https://news.ycombinator.com/item?id=48842611
and new entry:
Cursor Origin Code Hosting
undefined an hour ago
annoyingnoob an hour ago
Github down, no hard drives available, no memory available, thanks AI!
Seems like we are headed for Tech Gridlock.
yipinwong an hour ago
What they can implement is to slowdown the commit rate, rate limt or just queue-up messages not to overburden their downstream service.
I don't think GH has any of those, but just keep scaling, but that scaling failed.
Just bad architectural decisions from the postmortem.
--
It will only get worse due to AIs spawning massive commits, and they don't have unlimited cloud resource.
They can scale but not scalable in terms of effort, resources, and $
jdm2212 an hour ago
How would any of what you're saying help with this?
> The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers. Pausing HAProxy on those nodes simultaneously produced immediate broad recovery.
jdm2212 an hour ago
This stuff is good! This is what a booming economy looks like. There are people out there competing with you for resources because they have cool ideas they want to implement.
a2ff6eeb0 31 minutes ago
Or at least they asked the AI to come up with cool ideas, which is even more interesting. It's exciting watching the world transition away from humanity being in the driver's seat!
lenerdenator an hour ago
We need to have a package of FLOSsoftware that you could run on the cloud of your choice that offers most of what GitHub does (niceties on top of Git) without the centralization.
GitLab was close last I remember but there was some sort of enterprise tier when I tried hosting stuff on a local server years ago. I want true FLOSS, not another SaaS equivalent of the coke dealer giving clients the good uncut stuff when they're just starting out only to sell crap when they're addicted.
cschep an hour ago
https://forgejo.org/ promises to be this, have only lightly used it on https://codeberg.org/ but it seems nice?
0x457 32 minutes ago
Forgejo is pretty neat. Their CI story is sad because it's based on act and it lacks features like GH Apps so service accounts are not possible (using a user account as service is barf).
I self-host it and its pretty easy to have uptime higher than github when you have 3 users total.
Absolutely do not recommend using GitLab.
0xblinq 41 minutes ago
"Forgejo is a self-hosted lightweight software forge"
That says absolutely nothing. The "What is Forgejo?" question is unanswered and instead you get a lot of words about their values, their inclusivity, etc. And the next thing in the docs is how to install it. It's ridiculous.
I still don't know what it is or what it does.
denzen an hour ago
Have you looked into Forgejo?
amazingamazing an hour ago
Exponential growth. No company could handle that without some issues. Good luck to them. And for those who cannot tolerate this, there are many self hosted options.
rvz an hour ago
And another outage. [0] Looking forward to the subsequent post-mortem on that one.
You might want to not go all in on GitHub anymore since it is very unstable to use. A self-hosted instance would have a far better uptime than GitHub over the years.
6 years ahead [1] on not going all in an centralizing everything on GitHub.
djha-skin 17 minutes ago
[dead]
nycpig an hour ago
Almost 8 hours of downtime across all core workflows, and the word "sorry" or "apologize" appears nowhere in this post.
"If you were trying to ship software that day, we let you down" is classic corporate non-apology speak.
I’m done.
sajithdilshan 27 minutes ago
Bye Felicia
codegeek an hour ago
[dead]
monlockandkey an hour ago
They should rewrite their Ruby code to a performant language.