As with everything in software engineering, any average dev can fire up a few containers with Docker, but running anything in a real environment takes a whole team of people who actually understand the platforms involved in order to deliver a fully functional service, hopefully via properly designed code (TF) for ease of reusability. Most devs simply don't care to think about that part, and then throw it over the fence for "someone else" to deal with. Just as long as they can say "DONE!" (I created a thing!), that's all that matters.
I'm starting to think all these languages having their own pet "philosophies" that is "totally better than X" is a shitshow and just personal preference masquerading as standards.
Nothing is pissing me off more than GitHub's stability going down the tubes RIGHT as work is migrating everything, and I mean everything, from CircleCI to GH.
The wildest thing is that Azure Repos/Pipelines was better than this.
Their one caveat is also that they are still migrating it to Azure infra, so it's possible that's still in a one foot in one foot out kinda scenario, from what I've heard. But, this isn't inspiring confidence.
Could be. But 99% of the repos are static garbage with no PR nor actions.
They mentioned they have some elasticsearch reindexing going to, I would guess they needed to regard or move stuff and something didn't work well. But if I understood it right they mentioned the PRs ES index which they didn't shared proof increased as the number of repos.
It might be anything. It seems they lost huge chunks due to layoffs and structural changes and MS which has the reverse golden Midas touch.
This is just pure speculation but also now there is no reason for MS to keep GH working. They absorbed all code they wanted. Now they can let it burn. Would be even better for them if that happened
> Could be. But 99% of the repos are static garbage with no PR nor actions.
But the 1% of repos that do have PRs and actions are likely going to be seeing enormous increases in volumes
I have been a part of two very large companies with self hosted gits and I've seen enough to be confident that this is an incredibly hard thing to manage
Ya but they are owned by freaking microsoft and have billions of dollars and employees to throw at the problem. The outage problems shouldn't be happening period.
Easy to say that! Some problems are legitimately hard to solve though. Github is likely seeing usage patterns that have never been seen before and I bet some of these failure modes are novel
If you are at the limits of your architecture you may need to re-write things, and if you are rewriting things you can not arbitrarily speed that up by throwing dollars at it.
That's entirely a predicament of Microsoft's own making, though. Don't forget that they're the ones who launched "AI" programming into the hype cycle to begin with. So it's entirely reasonable to hold them as a company responsible for the resulting outages, which indeed shouldn't be happening. Dogs, fleas, and so on.
Serious question, have you been part of an org that had to scale orders of magnitude very quickly?
Anyone who has been part of that journey knows how painful it really is. A lot of times the systems to fail at all levels, and you have to redesign it from the first principles.
> Serious question, have you been part of an org that had to scale orders of magnitude very quickly?
I have, but it depends what you mean.
Scenario 1: e-commerce SaaS (think: Amazon but whitelabel, and before CPUs even had AES instructions); Christmas was "fun".
Scenario 2: Video Games. The first day is the worst day when it comes to scale. Everything has to be flawless from day 0 and you get no warning as to what can go wrong.
Yet, somehow, I managed to make highly reliable systems.
In scenario 1; I had an existing system that had to scale up and down with load, this was before there was cloud and hardware had a 3-4 month lead time, so most of the effort was around optimising existing code, increasing job timeouts and "quenching" sources that were expensive. We used to also do so 'magic' when it came to serving requests that had session token or shopping cart cookie.
In scenario 2; we have a clean-room implementation and no legacy, which is a blessing but also a curse, there's no possibility to sample real usage: but you also don't need to worry about making breaking changes that are for the better. With legacy you have to figure out how to migrate to the new behaviour gradually.
So, pro's and con's... but it's not like handling huge load hasn't been done before, computers are faster than they ever have been and while my personal opinion is that operational knowledge is dying (due to general distain for people who actually used to run systems that scale: not just write hopeful "eventually consistent" yaml that they call deterministic) - the systems that do exist today hold your hand much better than they did for me 20 years ago.
And I ran 1% of web traffic with an ops team of 5 back then. So, idk what's going on here.
EDIT: Likely people are flagging me because I sound arrogant (or I hurt their feelings by talking bad about YAML-ops), but all I am doing is answering the question presented based on my experience.
It really, really depends on what you mean. Specifically, it depends on the application and its various compute, I/O and access patterns. Scaling ecommerce and games is well-known by now (e.g. Amazon and Blizzard have been dealing with insane scale for two decades now.) However, anything outside a well-known pattern can be very tricky to scale.
I once worked on a team had to 100x scale a system whose downstream dependencies were various 3rd party APIs and data sources, most of which had no real SLAs to speak of and had extremely high variance in latencies and data transfer patterns. This basically required rearchitecting everything including our clients because the typical transactional request/response access pattern was too tightly coupled, and any hiccup in an external API quickly rippled up through the call-tree and caused outages 3+ services removed from ours. In some cases, the re-architecting went all the way to the UI.
Years later, I led a company-wide effort to optimize our entire user-facing application infrastructure to not fall over from sharply spiking user traffic, touching dozens of services across dozens of teams. We did a brief study and realized there was not a single common solution recommendation (like "tune your caches") we could give that would help all the teams because each one had very different resource usage patterns and hence different bottlenecks. Our approach was basically to farm the task out to each team and say "here are some common metrics to look into and some common issues to look for and some common solutions, get back to us if you need help." We spent a lot of time on the help.
I have no idea what the patterns for GitHub are, but I'll note its much more than just a DB and it has a dependency (Actions) with extremely high variance in latencies and resource usage.
I wrote this in response to the below comment, which is now edited and unfortunately dead, so posting here:
I understand, that wasn't a comment on your efforts back then, just that it is a solved problem today. But that does not mean other scaling problems are comparable or comparably solved. The universe of scaling problems is immense!
Worse, different problems occur at different scales. In the 3rd party API system, years after the first re-architecting, some use-cases developed issues at scale that exceeded the already high operational parameters we benchmarked at, and required us to re-architect the service again, including building out a whole new cluster so we could isolate that traffic entirely.
It is really hard to predict how things will break until they do.
(As an aside, I remember reading a lot of interesting things about Blizzard's technology, even if Blizzard didn't publish those themselves. There were many people who researched their products and published their findings. For instance, someone analyzed wireshark traces and published a very detailed report about how they tuned their server-side networking stack. One thing that stood out was Blizzard used TCP for WoW, whereas the conventional wisdom was UDP for real-time multiplayer!)
We used TCP for The Division, this was a major mistake and I don't think it was something people should repeat.
For example, if you have TCP_NODELAY and a few thousand players, you'll be swimming in about 1.2M packets per second pretty quickly.
This is enough to completely crush any stateful firewalls (UDP would pass through because no need to check state), so we had to do ACLs in network hardware instead, and append a magic number so that we could prevent flooding instead.
Another thing we found was that Windows networking activity only happens on Core0 (Windows 2012 R2); and that at 1.2M PPS: the driver crashes.
Logging in to a Windows machine which is AD connected when its network interface is dead is not ideal.
Makes sense, and that was the surprising thing about WoW using TCP. I wonder if Blizzard chose to put in all that extra effort to make TCP work because they encountered enough crappy home routers out there that mangled any non-TCP traffic...
The root comment asked if I'd been part of an org scaling orders of magnitude quickly, so I'll actually answer it: Venda at Christmas peak (pre-cloud, hardware on 4 month lead times, ~1% of global web traffic at peak) and The Division at launch (new IP, day-zero always-online AAA, ops team of 2). Different shapes, same playbook, both worked. So with the credentialing question out of the way..
GitHub's own April post-mortem names the causes in their own words: tight coupling allowing localised failures to cascade, and inability to shed load from misbehaving clients. Their March report says one of the March outages "shared the same underlying cause" as a February one - i.e. they hit the same rake twice in two months. Cascade isolation has a dedicated chapter in the SRE book from 2016. Load shedding is older than that; the Erlang/OTP people were writing about it in the 80s. This isn't research territory, it's a syllabus, and GitHub is fumbling it with Microsoft's chequebook behind them.
Amazon and Blizzard aren't the slam-dunk examples you want them to be either. Prime Day 2018 fell over because their auto-scaling failed and they had to manually add servers - that's not "well-known by now", that's a company at literal planetary scale getting caught short on the one day of the year it was guaranteed to matter. And Blizzard's Lord of Hatred launch this week is doing the exact same login-queue routine Diablo's done at every launch in living memory. If those are your "two decades of solved problems", the bar is on the floor.
Your 100x rearchitecture story actually argues my position, by the way. You described tight coupling causing cascading failures across services, and the fix was to decouple. That is the boring operational discipline I'm saying has atrophied - you and your team did the work. The point is GitHub, a decade later, with Microsoft's resources and thirty times the headcount, is putting out post-mortems that read like undergraduate distributed systems coursework.
So no - the question isn't whether GitHub's problem is hard. Every scaling problem looks hard from inside. The question is whether the operational discipline that solved this class of problem in the 2000s and 2010s is still being practised, or whether the industry has quietly decided "it's complicated" is sufficient cover.
Agreed, the techniques in general (caching, backpressure, exponential backoffs, etc.) are well-known, but a couple of things:
1) The general cause of issues in these cases is that certain assumptions no longer hold, and above a certain level of complexity, there are too many assumptions to keep track of, and so things fail in surprising ways. Like, the need for auto-scaling was well-known and Amazon did have that solution in place. But I recall the 2018 Prime Day was record-breaking, so it is likely the very same auto-scaling service that was supposed to save them fell over because they forecast too conservatively! (As an aside, I follow a senior AMZN engineer who's made his career out of load-testing their services, and he has many fun war stories.)
2) The resiliency work is not done upfront because it is additional complexity that may not be needed. "You're not Google" and YAGNI is sound advice most of the times. So the system is designed with some "reasonable" assumptions (which... see above!) At larger companies, resiliency mechanisms (load-shedding etc.) are built into standard components, but then...
3) Different performance profiles require different resiliency mechanisms, and it's not always clear what they would be.
Going back to the example of the 3rd party API service, when we inherited it around ~2012, it was built on standard infrastructure components with in-built resiliency mechanisms... but those were designed for internal services with latencies expected in milliseconds, whereas our downstream calls could go into seconds or even minutes. Still, with the traffic then, with a little tuning it worked fine and served the company well... until we (or the 3rd party APIs!) hit a certain scale and started seeing issues. At this point we extrapolated the trends, benchmarked heavily, and re-architected. And then we hit new scales and new use-cases that surfaced new issues, so we had to re-architect again!
The point is, the system's performance profile was very different from typical web services (the primary culprits being extremely high variance in downstream characteristics and very non-linear growth) and it was non-obvious to scale with conventional wisdom. I do not know what's happening at GitHub, but I suspect they have some similarly unique performance aspects.
I think you meant "green fields" and not "clean room"? Clean room refers to reverse engineering an existing program to create specifications, then having another team implement the specifications without legal risk from involving the original.
They say it is at least one order of magnitude[1]; "our plan to increase GitHub’s capacity by 10X in October 2025 .. By February 2026, it was clear that we needed to design for a future that requires 30X today’s scale."
I wouldn't be surprised. Have you not noticed the sheer volume of slop being posted everywhere these days? Almost all of that is hosted on Github. And some of those repos have insane commit frequencies.
At that point, make it lazy indexing? Who cares that I can't find a repo that was made 10 seconds ago, or even 15 minutes ago? No seriously, who cares? Search to that level of nuance is not mission critical, I don't care what anyone says, you'll live if you wait another 15 minutes or even an hour. Their search has been terrible since their last major set of search changes where they overhauled it completely either way.
They can claim that...but if you've built a public SaaS before you know the job is not to host the software, it's to put rails around people taking it down. They've had since 2008 to build those rails, and they're just now hitting places that take the service down on the regular?
Two weeks ago I had a commission to explore migrating from selfhosted gitlab to github for better AI integration. Last night that project was cancelled due to github outages and we're going to upgrade the self hosted server instead. I'd be tempted to use something like forgejo but there are a dozen devs and honestly I've only ever used it solo.
I didn't challenge them on this but it's because of Claude integrations with github. I'm not sure what that gives them over just running it against the codebase, but I didn't want to lose the opportunity to finally move them from that EoL server
Azure repos are kinda fine. It's really basic and there is nothing to break. I actually really really like their ticketing thingy for the same reason. It has the necessary stuff and the management types can't add a million of fields to it and annoy me with reporting, burndown charts or what not.
It has an annoying bug where approving PR's from the cli won't delete branches when you squash commit, while clicking the button in the UI does it perfectly fine. It's been a bug for a while (as in several years), and if you find something like that, don't expect it to ever be fixed. As a whole it's not a bad tool though.
As you say it's limited, but that can be both good and bad.
I'm on the other side of the fence. We're just about done migrating from GitHub to GitLab (self-hosted) and it's been refreshing to DGAF about any of the GH outages I read about.
Similar boat myself too, finished moving all important stuff from GitHub to self-hosted Forgejo with cross-platform builds. Not only do I avoid all the downtime stuff, but E2E builds also takes ~20% of the completion time it used to take, since now my runners have dedicated hardware hosted at home.
To maintain a fair comparison, GitHub has supported self-hosted runners for several years (maybe that doesn’t work for your specific usage, for whatever reason).
> To maintain a fair comparison, GitHub has supported self-hosted runners for several years
Yeah, tried that first, as I didn't want to move to Forgejo, I just wanted to keep working when I wanted to work.
The GitHub runner on Linux seemed fine, but the ones for macOS and Windows seemingly did something that made them a hell lot slower than even running VMs and then executing stuff inside those. I'm not sure what the runner is doing, if there is some built-in sandboxing or what not for those platforms, but it wasn't feasible to rely on for me as the builds took way too long time.
We were on self-hosted Gitlab but after a merger were forced to Github. Navigation feels painful in comparison and basic features such as commit graph are now behind more expensive tiers.
> We were on self-hosted Gitlab but after a merger were forced to Github. Navigation feels painful in comparison and basic features such as commit graph are now behind more expensive tiers.
Same experience here. Add to that that even on Enterprise tier:
- 1 Enterprise : 1 namespace - although you can segment it with Orgs, we were advised not to do it because we're too small (~2k people) (GL: groups, subgroups, sub-subgroups, ...)
- SSH deploy keys are singletons across the entire instance and repo-bound (and Weblate for instance can only use its own key), so you need a service account for that (GL: instance-wide SSH deploy keys that you can activate in specific repos)
Interesting! I worked with Gitlab and I also thought it was quite clunky. If it was not for the stability issues GitHub is fine. Any other alternatives to GH or GL?
We ended up an Azure Pipelines kinda by default because it was there and mostly paid for with the intention of later migrating, but it's been fine. Boring but stable and functional.
I think you’ve got it backwards. GitHub is by far the market leader for hosted repositories and maybe for CI too. This is like asking “Why are companies interested in using AWS?”
When one firm is so dominant for so long, the question is more like “Why shouldn’t we just use GitHub like 80% of software companies do?”
The issues they’ve had are almost all very recent. Very few companies have reevaluated that decision, because moving a big and well-integrated part of infrastructure is a huge project that delivers no value to the business. Speculating that you’ll have fewer development-slowing outages is not the most convincing when asking for the budget to do this. Plus, self-hosted isn’t necessarily going to have better uptime - mistakes happen.
I think before Actions, it would have been a lot easier to migrate off GH though. You’d just need to change a lot of repo URLs and find a way to set up webhooks from the new place to poke CI. Now with Actions, a lot lives in GH and in a proprietary flavor that doesn’t just ‘lift and shift.’
I think I worked at one company that used BitBucket instead of GitHub but GitHub has been the main place for the internal company repos everywhere else I’ve worked. GitHub is quite popular for any sort of git hosting.
More than half the companies I have worked for use Github. The others used Atlassian tools which were at least as bad from a reliability perspective and much less nice to use (IMO).
> The issues they’ve had are almost all very recent.
It has been bad for at least 18mo, maybe longer? I recall multiple work impacting outages at my previous employer extending back into 2024. Maybe even earlier than that?
But on the timescale of tech it's a long time. When a service has serious availability issues for like 7 quarters straight that's.. I mean, it's why we're all talking about it.
> Honest question, why are companies interested in hosting on github?
Mostly boils down to marketing and easier to establish a community. Almost every developer has an account there, leading to network effects being much larger, so if you're a new FOSS project, finding contributors and getting your project in front of other's eyes is much easier when you're on GitHub compared to your own Forgejo instance.
With that said, I'd question if chasing "most external one-time contributors" or GitHub stars is the right way to actually run a FOSS project, personally I'd avoid thinking about those vanity-numbers as much as possible and focus on the project, code and contributors themselves.
But, I've literally heard those two arguments for "why GitHub" countless of times over the years.
Go with the flow, don't rock the boat and use what developers already know, are probably the most cited reasons I've heard.
I've tried so many times in the past to argue for self-hosted setup that you fully control if you can afford it, things just get so much smoother and if you're a software development company, you probably want to own the software development workflow E2E so you can actually ship as fast as you want.
I’ve argued the opposite most of the time in build vs buy. Buy in almost every case unless it’s a real competitive advantage to you.
I know developers love to build, but do you think:
1) self-hosting git provides any competitive edge to the business over letting someone manage it?
2) it provides so much value that you’re willing to fund engineers to build, secure, support this on an ongoing basis?
I’ve found the answer to those is No in both cases.
The same reason you wouldn’t build your own internal chat tool, you’d use Slack. And you wouldn’t bother self-hosting your own Jira or documentation.
Code hosting is code hosting, there’s no difference where it's hosted. There’s no slowdown in delivery with using GitHub - their March uptime was 99.5% which annoys some commenters but it’s fine. That’s 45 minutes downtime per month which is tolerable.
You would spend way more effort and money building a jenky self-hosted solution to end up with a worse result.
You don't have to 'build' anything. Just spin up a GitLab docker container. Bonus: If you put it behind a VPN, you never have to worry about updating it.
Edit: to be clear for anyone reading, don’t do this, it’s a really bad idea. You need to patch your stuff, even behind a VPN. If you’re unsure, just google: “examples of security breaches of unpatched software behind VPNs”
Yeah I literally work on that side of things, Platform/etc. Managing GHE, CI across so many CI platforms over the past almost two decades, etc. It's a disaster thinking you can "just spin up X". Sure, if you're a two person operation, but scale becomes a factor real fast. Too many people on HN are giving away they have little to no experience. And I'm the one that made the initial comment in this thread, lol. It's frustrating, but I don't want to go back to "managing build agents" and the control plane (aka whatever your central Git Host is) for everything and all jobs, etc. That's a pain in the ass and 90% of environments have no fucking clue how to maintain that shit.
If someone wants to pay me to do lead such an effort, I'd do it, and yes, it'll all be done and managed in code, and it will be done better than 90% of folks could do (I've seen the production build systems most environments have built, some big names too, and they're trash...in no small part because it's simply hard to do), but it'll take a while to smooth the experience out for all of your requirements. You're gonna be paying, and very well, for a while.
Usually, at large enough corporations, it's one of two things. Some random project gets open sourced, and it ends up on Github(see, for example, Salesforce) - or, more commonly, some subsidiary or acquisition had github and has either refused to migrate to the internal source system or the hassle of migration isn't worth it.
I don’t know why you would even really need hosted git or why you’d be affected by its downtime. Git is decentralized by design. One node going down should not stop development. You don’t need a “central hub” to keep working.
I guess it’s all the other non-git stuff like issue tracking and other (unfortunately) centralized products on GitHub that causes disruption when they go down.
Weird how GitHub built itself around a distributed VC system and then made all its other services centralized.
Yes, you want to run automated builds, unit test, end to end test, UI tests, make it easy for testers to deploy specific versions / tags to internal server. Also kick off builds for iOS on mac computers. We use Teamcity for that.
Tracking of issues, feature and epics. Maybe also knowledge base / wiki. We use Jira.
i did the circleci --> github actions migration for my job 1.5 years ago, and things seemed great... at first. at the time, we'd been dealing w/circleci's semi-regular (but thankfully short) outages for over two years, and we were excited to move to a more stable system.
From CircleCI here. A big effort and investment went into resolving those outage issues you're referring to. Results have been stellar for a while now. Here's the latest: https://status.circleci.com
Apologies for shameless plug... have you considered Buildkite? Our statuspage is a sea of green even as we hit 1.3b minutes/wk (GHA sits at 2.1/wk now). Much less maintenance overhead than Jenkins, more dynamic featureset. The trial is all-access, unlocks the full product and can be extended past 30 days. Real human eng on standby throughout.
Why do you care about github? It’s Just another corporation doing what they know best: harvesting money. The software ecosystem can live without github just fine
I mean this in a good way, but I'm slightly chuckling to myself that it reads like people are just discovering IaC...on HN. That's all Nix configs are, at the end of the day.
No matter the tool, manage your environment in code, your life becomes much easier. People start and then get addicted to the ClickOps for the initial hit and then end up in a packed closet with a one way ticket to Narnia.
This happens in large environments too, so not at all just a home lab thing.
A NixOS config is a bit different because it’s lower level and is configuring the OS through a first-party interface. It is more like extending the distro itself as opposed to configuring an existing distro after the fact.
The other big difference is that it is purely declarative vs. a simulation of a declarative config a la Ansible and other tools. Again, because the distro is config aware at all levels, starting from early boot.
The last difference is atomicity. You can (in theory) rely on an all or nothing config switch as well as the ability to rollback at any time (even at boot).
On top of all this are the niceties enabled by Nix and nixpkgs. Shared binary caches, run a config on a VM, bake a live ISO or cloud VM image from a config (Packer style), the NixOS test framework, etc.
Nah, models can be fine tuned and trained on anything. Common consumer products like ChatGPT and Gemini have particular styles, very polite and helpful, but there are models trained to be combatative, models trained to write in the style of shakespeare, all sorts of things. Someone could train a model to reply to posts in the style of HN comments and you’d probably never know.
lmao found the dev who's only ever worked on the dev side of things.