Hacker News

I second this. GitHub used to be a fantastic product. Now it barely even works. Even basic functionality like the timeline updating when I push commits is unreliable. The other day I opened a PR diff (not even a particularly large one) and it took fully 15 seconds after the page visually finished loading -- on a $2,000 dev machine -- before any UI elements became clickable. This happened repeatedly.

It is fairly stunning to me that we've come to accept this level of non-functional software as normal.

“ I enjoy GH copilot as much as the next person”

So not at all?

Not going to happen. This is terminal decline. Next step is to kill off free repos, and then they'll start ratcheting up the price to the point that they have one small dedicated engineering team supporting each customer they have. They will have exactly one customer. At some point they'll end up owned by Broadcom, OpenText, Rocket, or Progress.

Github used to publish some pretty interesting postmortems. Maybe they still do. IIRC that they were struggling with scaling their SQL db and were starting to hit the limits. It's a tough position to be in because you have to either to a massive migration to a data layer with much different semantics, or you have to keep desperately squeezing performance and skirting on the edge of outages with a DB that wasn't really meant to handle what you're doing with it now. The OpenAI blog post on "scaling" Postgres to their current scale has much the same flavor, although I think they're doing it better than Github appears to be doing.

> Can someone in GitHub senior leadership please start paying attention and reprioritise towards actually delivering a product that's at least relatively reliable?

It's Microsoft. A reliable product is not a reasonable expectation.

Maybe take the initiative and move your own first? It definitely would have a bigger effect than begging here.

> Can someone in GitHub senior leadership please start paying attention and reprioritise towards actually delivering a product that's at least relatively reliable?

They claim that is what they are doing right now. [1]

[1] https://thenewstack.io/github-will-prioritize-migrating-to-a...

As an aside, God, Azure DevOps, what a total pile of crap that product is

My "favourite" restriction that an Azure DevOps PR description is limited to a pathetic 4000 characters.

You might as well self-host at this point as that is far more reliable than depending on GitHub.

Additionally, there is no CEO of GitHub this time that is going to save us here.

So as I said many years ago [0] in the long term, a better way is to self host or use alternatives such as Codeberg or GitLab which at least you can self host your own.

[0] https://news.ycombinator.com/item?id=22867803

IME this all started after MSFT acquired GitHub but well before vibe coding took the world by storm.

ETA: Tangentially, private repos became free under Microsoft ownership in 2019. If they hadn't done that, they could've extracted $4 per month from every vibe coder forever(!)

I was wondering about that the other day, the sheer amount of code, repos, and commits being generated now with AI. And probably more large datasets as well.

Live by the AI Agent hype, die by the AI Agent crush.

This is the real scenario behind the scenes. They are struggling with scale.

I don't think any of this was a mistake ;) Lock-in was by design.

We can run a Forgejo instance for you with Firecracker VM runners on bare metal. We can also support it and provide an SLA. We're running it internally and it is very solid. We're running the runners on bare metal, with a whole lot of large CI/CD jobs (mostly Rust compilation).

The down side is that the starting price is kinda high, so the math probably only works out if you also have a number of other workloads to run on the same cluster. Or if you need to run a really huge Forgejo server!

I suspect my comment history will provide the best details and overview of what we do. We'll be offering the Firecracker runner back to the Forgejo community very soon in any case.

https://lithus.eu

Long time GitLab fan myself. The platform itself is quite solid, and GitLab CI is extremely straightforward but allows for a lot of complexity if you need it. They have registries as well, though admittedly the permission stuff around them is a bit wonky. But it definitely works and integrates nicely when you use everything all in one!

Should our repos be responsible for CI in the first place? Seems like we keep losing the idea of simple tools to do specific jobs well (unix-like) and keep growing tools to be larger while attempting to do more things much less well (microsoft-like).

https://nix-ci.com/

Gitea / forgejo. It supports GitHub actions.

GitLab, best ci i’ve ever used.

Lots of dedicated CI/CD out there that works well. CircleCI has worked for me

GitLab can be selfhosted with container based CI and fairly easy to setup CE

Yay for GitLab and Forgejo/Gitea.

My previous two startups used GitLab successfully. The smaller startup used paid-tier hosted by gitlab.com. The bigger startup (with strategic cutting-edge IP, and multinational security sensitivity) used the expensive on-prem enterprise GitLab.

(The latter startup, I spent some principal engineer political capital to move us to GitLab, after our software team was crippled by the Microsoft Azure-branded thing that non-software people had purchased by default. It helped that GitLab had a testimonial from Nvidia, since we were also in the AI hardware space.)

If you prefer to use fully open source, or have $0 budget, there's also Forgejo (forked from Gitea). I'm using it for my current one-person side-startup, and it's mostly as good as GitLab for Git, issues, boards, and wiki. The "scoped" issue labels, which I use heavily, are standard in Foregejo, but paid-tier in GitLab. I haven't yet exercised the CI features.

Self-hosted Gitea is a good time if you're comfortable taking care of backups and other self-hosting stuff.

Self hosted GitLab is absolutely worth it.

I bet there's other factors that are correlated as well!

So far it feels they are vibe coding it day and night lol…probably with GitHub Copilot

Does redundancy even matter if the end result is still poorer uptime?

> TLDR: don't use GitHub Action for CI/CD!

You should reach the same conclusion by trying to use it for this purpose, but also indeed for any purpose at all. Incidents that make you unable to deploy making all your CD efforts pointless are only the cherry on top.

Sure!

Do you allow me to run the following command?

    cd project; find -type f | while read f; do mv "$f" /dev/null; done

Not quite, that one is an earlier outage while this one started at (or a bit before) 19:01 UTC.

The history for today is a bit of a mess really: https://www.githubstatus.com/history

No, it's a new outage -- that's the point! Check the URLs.