Almost 3 hours in and still "We are still working to identify the root cause"
Can't even view diffs in the web interface!
I'm out! I am willing to pay ~$5-10/mo for a reliable host I can swap as painlessly as possible. Mostly just building little static sites and little PWAs, no more than 500mb a repo and 10gb total. Need a repo with PRs and issues, where people can register free account and contribute or raise issues, GH actions CI, and pages host to deploy static sites. I don't use more than 100 mins of CI a month. Any recommendations?
edit: GitLab free tier has all I need, but I kind of want to pay and have some kind of reliability (not necessarily an SLA, but). But their next tier is $29/mo, which is too much.
I was in a very similar situation and this is what I did:
- put Forgejo on a little $5 Vultr VPS (I share the VPS with other things, Forgejo doesn't need much resources. I also added some block storage to host my Docker containers in the Forgejo Docker registry)
- got a 8GB ram cheaper Hetzner VPS (you have to wait for their availability) for around 8 EUR and used that to install the CI runner
- Use Netlify for hosting static sites and PWA
It takes a bit to set up but everything works snappy, clean, and flawlessly. I have unlimited CI minutes. As a bonus I get free Docker image and registry hosting because it's included with Foregejo. I am so, so happy with it all.
Also with Forgejo you just feel like you're using something clean and solid. GitHub started to feel more and more bloated, slow, and unstable.
I've been very happy with forgejo for a personal, private server.
I don't care much about the social aspects of GitHub, but I do care a ton about hosting code for long periods of time, long after I've completely changed my personal homelab.
GitHub's most essential service for me is it being the equivalent of the Internet Archive, but for code.
Similar! I was tired of the outages and crazy runner bills so I just finished moving everything to my own server hosting Forgejo. I've had to recreate a couple things (like a dependabot replacement) but it's been a week so far and it's going well - I haven't had any outages nor have I been finding ways to lessen my Runner billed minutes.
Forgejo has some quirks (like no ability to share workflows between private repos) but I'm happy with my choice so far.
Great setup, two questions, do you have 2 VPNs now or all on one Hetzner VPS? And how do you setup security?
Just recommend swapping vercel/netlify with something else, we had quite a few issues and high costs with them. Not opiniated here, currently using Cloudflare and BunnyCDN (which is great)
I just SSH into the servers when I need to. I don't have any VPNs with these servers. I just try to follow normal server security setup practices, disabling password authentication etc.
I worried being behind in server-security, since all those 0days now being so easy to pull from claude/chatgpt, that maybe you guys found a risk-minimizing front-end to reduce the attack-surface.
That’s what I do. Sensible firewall rules with ufw, turn off services I don’t use, follow the docs for the services I do. For a handful of servers the maintenance overhead is fine.
Unless you want those people to have merging rights, collaboration can be easily done with exporting a read-only version of the git repo via http (even a browsable one using cgit). Then anyone can work on their copy, generate patches (git format-patch) and send the files via any means of communication (email, slack,...).
This, in a nutshell, is why Github is the way it is and there's a lack of good alternatives. I'm not knocking you in particular because we're all pretty much in the same boat here.
"I am willing to pay ~$5-10/mo for a reliable host... Need a repo with PRs and issues, where people can register free account and contribute or raise issues, GH actions CI, and pages host to deploy static sites."
Who wants to build something to acquire that $5-10/mo business? A VPS powerful enough to host a small repository costs more than that these days. There's no opportunity here for someone to build a Github alternative and make any money. The best parts of Github that make it worth using in the first place all come down to the size of the network and that would be extremely difficult to replicate.
It comes down to wanting to develop a SCM hosting service and hitting a $5-10/mo price point. To put that in perspective, people pay more than $5 for a latte and COGS on that is about $1.50 or less. A good customer will come in weekly or more often. I'd rather sell coffee than SaaS SCM.
It becomes worth it with scale. A VPS costs more than $5 but you can host a very large number of repos, a dormant repo takes no resources but disk space to host, its only activity that consumes resources, which is less than $5-$10 worth
Maybe. You would either have to buy and colocate physical servers and pack the hell of them or develop something on AWS or GCP that aggressively scales down to zero.
Anyway, how are you going to pay engineer salaries and 24/7 support staff on a $5-10/mo product? Not to mention, you'd be hosting someone's critical infrastructure so they will have a meltdown when you inevitably make a mistake. How long are you willing to burn money before you've reached a scale where it's profitable?
It's paradoxical how all the money in tech goes to the most useless things that are a net drain on society (social media, ad tech, etc.) and we're willing to pay less for the most important things than we would for a cup of coffee.
At the prices people are willing to pay for SCM, it really has to be a complement to some other product you're offering or part of an integrated suite. It's most likely a dead end on its own.
I guess you could do it... but why? There are a thousand ways to make more money with fewer headaches. I wouldn't get into this business myself and I wouldn't fund someone else either.
I can assure you that they are not hosting for free because the cost is zero and goodwill pays the bills. Read the TOS and privacy policy. If they’re not monetizing the data in their custody then it’s just more overhead that has to be covered by paying customers, which makes the business even more difficult.
Yes, you can host a repository for less than $10, but that’s not the question, is it? Can you host it cheaply enough that a customer paying $10 is a viable and worthwhile business? Can you do it for $5 all in with infrastructure, staff, and general and administrative costs? You won’t have much for a marketing budget so you’ll have to offer those free accounts with some features for viral network growth. Oh, and you’ll have to offer better uptime than GitHub. Still sound like a good idea?
Anyone can re-create the hosting part of GitHub. This is not the difficult nor the expensive part.
It's the actions. The compute to run them isn't cheap at scale nor is the amount of auditing you need to do for security and compliance if you want enterprise/government customers.
I don't think people asking for github alternatives are expecting unlimited actions for $5/month. It would be acceptable if they charged credits on top or let you host the runner on your own machine.
Good points. I guess I keep just wanting 'you are the product' / enshitification / free tier capture to not be the dominant business model of our time. You're right that a YC VC would likely not funds this. There are many alternative business models though. (edit: https://sourcehut.org seems promising if very barebones and oldschool)
> The best parts of Github that make it worth using in the first place all come down to the size of the network
Not intrinsically, only as a side-effect of economies of scale. At least for me. The 'social' features make it more compelling and explain its widespread adoption and thus economies of scale, but not what made github worth using for me at least. It Just Worked (TM) to unite all the work we used to do in the good ol' days via repos, patches, tickets, mailing lists, IRC, build servers, and host servers.
I don't need my own VPS running 24/7, I said maybe 100 mins of CI a month, even if I'm pushing and building while others are commenting and PRing multiple times a day, it's going to still take a fraction of that VPS capacity.
Same. Currently, I'm spending $10/mo on a VPS from OVH to host Forgejo. I have hourly backups to an rsync.net account I was already paying for. It's not the most efficient way to do things, but Forgejo is actually pretty nice and I have data sovereignty.
Update 16:36 UTC - We identified the problematic component and have taken corrective actions. There are strong signs of recovery but we are still working to completely restore service, with error rates still remaining slightly elevated. We will post further updates as recovery continues.
My blackest Monday today.
Deepseek is off/too expensive, Claude tokens out for 3 days, kimi refuses, and Github down for the whole afternoon. Maybe I'll take the week off
It's crazy that it wasn't that long ago that all we worked with was a compiler on the local machine and maybe a SVN server you'd interact with every few hours
Disadvantage: You have to coordinate with teammates to make sure you don't need the same file at the same time.
Advantage: You have to coordinate with teammates to make sure you don't need the same file at the same time.
Sometimes, not being able to work independently can be an advantage. Two people discovering they're trying to work on the same file can lead to useful conversations.
That is so cool. I bet that was an absolute blast. Can I ask for a war story? Something hard or goofy that I as I player (and a software engineer) probably had no idea you had to solve? I really loved that game.
Firstly, if you email me at shawnpresser@gmail.com or DM me on twitter, I'll be happy to tell you as much as you'd like.
The whole time period was goofy. I loved it.
Imagine being a 22yo, disillusioned with your first gamedev job. Then you spend all your nights playing HoN with friends.
I played hard support each game. Back then, there was one particular bug: you used keyboard hotkeys to buy things, but the shortcut for Logger's Hatchet was very close to the shortcut for wards. And hatchet was a special item, because you weren't allowed to sell it once you bought it. So at least three or four times, I was playing crystal maiden and accidentally bought this damn hatchet -- something which only benefits melee heroes, so it's completely useless to me. And I couldn't sell it back. So I'd have to start down 225 gold. When I finally got a job at S2, that was by far the most satisfying bug fix of my life. (I changed it so that you can sell it within the initial 15 second sellback window.)
Let's see, what else...
I remember a time in 2010 when we knew Valve was coming to eat our lunch with Dota 2. So I was trying very hard to think of ways to avoid this impending doom. One day I said, I got it! Custom games! Custom games had made WC3 live long beyond the lifetime of the original game. Ditto for starcraft and a bunch of others. So, being very 22, the same night I had this idea, I wanted to test it out. I didn't ask anybody. I fired up an EC2 server (which was a new thing back then) and ran the server software on it for exactly one match, which I was observing carefully, to see if EC2 could support our server. The match went well, someone called gg, and once their stats were submitted I shut down the server and went into the office the next day about to pitch my "brilliant" plan.
When I got to the office, Sean (the admin guy, and #2 at the company) was looking a little frustrated. I told him hey, if you saw any weird activity last night, that was just me testing something. He said that was you? And my heart damn near dropped out of my chest. I said yeah. He said ok, and I go to my desk.
About 20 minutes later he pulls me into a side office where he gives me the dressing down of the century. He's like, "you didn't ask anyone or let them know what you were doing? Don't you know EC2 is so much more expensive than self hosting? And by the way, something like that is completely unrelated to what we're doing here anyway." (All of which were true!) I profusely apologized, and that was the end of custom games on HoN.
If you want some more, I can probably think of a couple others.
That is fucking amazing. I’ll email you. Thank you so much for even typing that out. I also played hard support almost exclusively, back when buying wards was not a thing.
hmm very interesting combination of services being down. I've been using DeepSeek v4 Pro and Qwen3.8 now daily for close to 2 weeks. Damn good and reliable. Anytime I need to verify math I check with Kimi k3 tho, it still feels like being superior to these both, not sure if actually true.
Well, it would be a mistake on your part to assume that everything in an organization, especially a large one like github, is a reflection of the capabilities of the state of the art in anything (AI, architecture, programming language, organizational management, operations, etc.).
It's clear that updating the status page as quickly as possible isn't even something github cares about.
> It's clear that updating the status page as quickly as possible isn't even something github cares about.
Status pages for large companies are social, political and legal tools, and not technical indicators. It has always been this way, and likely always will unless some laws were passed.
My company is heavy on AI promotion, but many many many people still do things sub-optimally, and many systems are still not well integrated/able with AI.
“The future is here it’s just not evenly distributed”, and that applies inside companies too.
Hard to actually judge without being in the room and knowing the team that owns the incident. And as other said, it’s pretty common IME to have a root cause but slow roll communication of it until remediation has more confidence.
It's very easy to judge in the context of how all other service providers are performing on average.
It's very easy to hold any service provider responsible for their own words and deeds that no one else demanded nor commanded (selling themselves to MS, moving onto Azure, copilot stuffed into everything, the architectural garbage that is gh actions). It's very easy to judge when most of their difficulties are self-inflicted.
You excuse and understand the exception, not the norm.
You're basically saying that before you can know if AI is useful to you and your dealings, AI must solve all of a given corporation's problems including all the human/organizational components and even the trivial ones like how serious Github is about updating their status page, something that probably came down to one guy.
IDK, when a system is very complex — which GH certainly is — this phrasing sounds kinda fine to me? It's recovering, but not yet operating fully as intended.
I mean to be fair it’s a huge production system, there are many moving parts. Having worked on outages before in complex systems it is very difficult to identify root causes under pressure
Are you paying for GitHub? Are you complaining about an outage from a product you aren't even paying for? You know the whole point of git is you don't _need_ a central server.
I don't understand the entitlement in your post here.
Are you the user or the product? You can be both with GitHub!
GitHub provide a compelling product, enough to become the defacto forge for many. But their reputation has taken a lot of hits, as a grey beard I'm salty since the Microsoft takeover, the downtime is just the leaky diaper icing.
To diagnose at a distance it feels like they need a stronger QA process, it's a product that scales horizontally, but somehow they find a way to break it for everyone on the regular! They need less hub and more node!
> I don't understand the entitlement in your post here.
Excuse me, I'm literally asking where I can pay to get the specific cloud/web-based features I listed. I'm trying hard to NOT rely on a free tier because I want to pay to have the entitlement to complain when I don't get what I paid for. You know, the way that commerce has existed for millennia. [0]
I don't know where this aggression is coming from. You seem to want me to stay on the free tier and shut up or go all locally-hosted. Great if either of those work for you, buddy.
Not OP, but I pay. And it was an annoyance today for several hours that did impact some work.
I don't need one, but I do have that because it's doing a bit more work than just git hosting (administratable access for others on the team, build actions, etc) and yes... I'm paying for it.
This is the closest I'm finding! CI, web-based ticket submission for anyone with a site-wide login, static hosting, good price... web-based patching would be nice, good to see it in development.
Some honest feedback: 10 minutes of searching and reading left my ADHD brain confused about pricing and quotas. It doesn't have any numbers for what I'm getting for $4/8/12/mo. And it took me 3 reads of that page to notice that the tiers are not actually the standard tiers, but pay-what-you-can. I first jumped to the prominent table then hunted for the fine print for the quotas, didn't even the read "there is no difference between the subscriptions besides price." Then read it but my mind thought "OK, all features are available to all tiers, but at what quotas?" A normal VC backed company might hide this because they're trying to pull one over on you...
Also, "Sourcehut’s public alpha... What we consider an alpha passes for a production-ready system to some." That's a choice, I get it, but it makes me feel like I'd be jumping from one unreliable platform to another. Maybe link to your uptime? (And the real github uptime side by side :P)
Because another honest thought: unlike GitLab, which pretty much has the same shaped features and affordances that you can one-click import and things mostly work, this would be a pretty breaking change to that would be difficult to revert. I do see full data autonomy is on the roadmap too though.
Forgejo is a great fit for what you're describing. I run Fjord, which is managed Forgejo. Our dedicated instances are above your $5-10 budget, but we also run a free public Forgejo instance if you want to try it without operating anything yourself.
Forgejo has been great for me too. I liked it so much, I added a feature to make it relatively easy to get up and running on exe.dev. Support for Gitea and Tangle.org is also available.
Sounds like they don’t have a fine enough lens on some particular class of problems. Been there before. The source of the meta problem is so old few people will learn the lesson that hadn’t known it already.
You can basically self-host your repos on a cheap VPS. I did that some time ago. You'll have to create a ssh user that has only 'git' as a shell, iirc, on that box. Works okay. But doesn't provide a web ui.
Architecturally, how are you storing Git data? If it isn't fundamentally different from bare-repo-on-disk with libgit2 or wrapping the git binary, you're not going to scale any better.
Blobs are unique. Whether you have 700 forks or not, data doesn’t increase. I’m unpacking the git stuff, keeping all the blobs on disk as alternates. Then syncing to object storage. This proved really slow so the blobs are still repacked to sync to object storage. It was fun finding out that some of the repos I test with have 500k blobs, I didn’t know either, but half a million individual put calls made it excessively obvious that wasn’t going to work.
Between the git hosts things move around based on available space, IO pressure and whether it’s been recently accessed or not (spinning disks are far cheaper these days). Theoretically there will be a point where the routing becomes a problem, but probably not until there’s several hundred git hosts (I’ll probably have bandwidth problems before that happens).
I’m not sure what the largest git repo is that github has, but I can theoretically keep repos up to 1TB around (plus a sheer unlimited number of forks for the same). I imagine the API’s get problematic before git does at that point (e.g. shuffling data around is much harder than storing it).
$work just moved a few dozen repos to GitLab using their import tool, including issues, PRs, comments, etc. It got everything over but stumbled on a few comments.
Not a prop to GitLab, it's kindof table stakes now, just saying whichever place you go, you can find or write a migrator that will capture most of what's in GitHub, assuming their API is up long enough...
Can't even view diffs in the web interface!
I'm out! I am willing to pay ~$5-10/mo for a reliable host I can swap as painlessly as possible. Mostly just building little static sites and little PWAs, no more than 500mb a repo and 10gb total. Need a repo with PRs and issues, where people can register free account and contribute or raise issues, GH actions CI, and pages host to deploy static sites. I don't use more than 100 mins of CI a month. Any recommendations?
edit: GitLab free tier has all I need, but I kind of want to pay and have some kind of reliability (not necessarily an SLA, but). But their next tier is $29/mo, which is too much.