Hacker News

Are these outages caused by introduced bugs, though, or by load issues?

As someone who has spent many years working in high load environments, this is not an uncommon pattern.

You design a system and it works great. It can handle failures, load spikes, it is horizontally scalable, things are great. You think you figured it out.

And then load keeps increasing and you suddenly hit a tipping point where everything keeps failing, and you cant keep up. The things that you thought were perfectly horizontally scalable turn out to have a bottleneck you didn’t even think about until you got to a truly massive scale. Your systems suddenly don’t have the excess capacity to handle load spikes or catchup work, so suddenly any failure cascades and recovery is more and more difficult. You can’t solve the problem with additional hardware, and your perfect scalable design actually can’t scale any more.

This doesn’t have to be about GitHub using LLMs in their code to still be related to LLMs. GitHub gets a lot more commits now because of LLMs and probably get a lot more reads because of LLMs as well.

It could be that the extra usage just pushed them past one of those capacity thresholds.

To me this correlates more to them being bought by Microsoft, a company known for being seemingly incapable of creating quality software to the point that it's not even funny anymore, and also known for sloppifying all the products they touch.

I guess github is kind of a shared garden. Interesting that, like in game theory, if everyone is using it too much, no one gets to use it.

It's probably not attributable to AI in the way that you're thinking - Github has been absorbing an exponential increase in usage, and that increase is mostly due to new AI-related projects being created and worked on.

Though I'm sure some of the blame can go to internal slop code.

For the longest time, I didn't appreciate the "AI-induced traffic" excuse. But seriously, I checked the rough github egress for our lab versus an old log from 2024, and there's an order of magnitude or two difference. From asking around, it seems people all have the gh cli tool and let it loose with parallel tool calls and e.g LLM's polling Actions in a background bash while loop with sleep $TOO_FEW_SECONDS. Some people use a variety of skills where the agent makes a commit every few code changes, and uses Issues for its memory/log. And they have O(5) sessions at the same time doing all kinds of crap. It's the same with PR checks/PRs. Recently we also saw continued usage throughout the night as well, which did not exist pre coding agents. Loops or whatever they call cron jobs in the harnesses these days is the reason. It must be adding up.

#0-nines

I once saw a sign at a railway station in the UK that actually did say "normal service will be operating between the 12th of July and 18th of August" (or something like that). Made me laugh. At the time Anglia Railways had a rail replacement bus literally every weekend.

I thought they should rename to Anglia Busways and have bus replacement trains instead.

GHA still beats Circle CI or trying to run your own Jenkins - even with all the downtime. I'd move in a heartbeat if there were good alternatives.

To offer a data point (not a tribalist argument): GitHub Actions being down this often is still less expensive than the cost of switching everything somewhere else. Would rather have my team walk away from the PRs and do other tasks than have engineering effort, meetings, design pages, scheduling, etc. for switching over.

This is annoying and I'm here because it's down. But it would have to be far worse to come close to actually being worth changing.

Our deployments are triggered by GitHub Actions, and it is a pain to deploy without it (specifically collecting the credentials and putting them into variables the shell will read).

This is multiple times this month that this has been a problem.

Has GitHub completed it's internal migration to Azure yet? Or is it still ongoing? None of our devs want to switch away from GH, but we will have to at this point.

I would conjecture that the long tail of GH users are not dependent on any of the CI stuff in the first place.

What would you guys recommend as an alternative? Both hosted and/or self hosted

I seriously want to. I'm just waiting for the day where I have enough extra energy to simply bite the bullet and do it.

To what? That’s kind of the problem.

how hard could it be these days for a mid to large size eng company to have their own gitlab/hub type solution hosted in aws

My company moved to GH Actions 6 months ago even after I pushed back with a "Are you sure given their reliability issues of late?"

Now I'm stuck twiddling my thumbs with PR checks stuck/failing...

Until someone makes a better PR experience you guys are stuck with GitHub.

Nobody cares about ATProto or whether your commits are a damn NFT or some bs just literally improve upon the experience.

That’s it.

It’s as if no company is focusing on the product experience or anybody’s experience anymore. It’s all ooo look what I got I got this I can do that too me me me but nobody will ever buy that.

Say what you want about huge companies like Microsoft or Walmart but they spend a lot of energy understanding the human experience to sell products and less on their own perceived self-aggrandizement.

GitHub is the best version control online and it’s not even close.

yepsies

I’ve been self-hosting lore and gitea the past 6 months or so and it’s been a breeze. For anything I really care to share I throw up on Tangled.

Except anything I'd self host would be down much more often. I've hardly ever actually experienced down time issues with github when I needed it.

Can't they haven't hooked up CoPilot to their Xitter account just yet.

Sorry folks, guess my GHCP query was too large...