Me and a colleague were just ranting about how there's no way in hell Actions has 99.6% uptime but yet that's what their status page says. Maybe it's just bias and I am only remembering the minutes that it's down what feels like every week or multiple times a week. But I'm suspicious of their stats, even if they declare an incident on the status page.
I would suggest every enterprise customer that is not keeping track of github downtime and it's effect on their systems and workforce independently for presentation to their sales team every billing cycle for discounts is leaving a lot of money on the table.
Willing to Relocate: Probably not but open to PNW/Europe
Technologies: Web (React/TS/etc), backend (node/python/go), infra (everything from ansible to Terraform). When I say full stack I mean I've shipped and owned pager across the stack
Resume: via email
Email: hn@mck-p.com
I'm looking to move from IC into EM/engineering leadership. Over a decade experience across both.
We're starting to have the conversation of "do we need to replace GitHub as our CI/CD pipeline?"
No idea if/when we decide to pull that bandaid but I think once we can't merge a hot fix into main because of GH being down, it'll be a big org push to replace it.
Not the same company but similar discussions here:
- It was believed that GitHub would be more reliable than our self hosted Gitlab CI runners. This has proven to be very false.
- It was believed it would be easier to write tooling. I’m not on the dev tooling team, so can’t confirm if they found it true or not, but from the outside I see a lot more “Github can’t do that” than I did “Gitlab can’t do it”
Those are both hilarious to me. In my last FTE role we had the exact opposite beliefs - we wanted to leave GitLabs increases in pricing and move to annual sub, but the reliability of self-hosted runners and the flexibility (and clarity) of the CI system was fantastic.
The hilarious part is more the absurdity and cognitive disconnect experienced when reading about serious evaluations that went in a completely opposite direction to those directly participated in.
It's a ~year old product that started on GH. So this is the first time this team has had to think through "okay, so this isn't as reliable as it was at $oldjob."
Jenkins was (and still is) a PITA to maintain. Moved two different companies off from it to gh actions (self-managed runners in case of a second company), never had any regrets.
blacksmith.sh just raised tens of millions on the core premise that self-hosting CI/CD is hard [0].
they've had multiple outages in the last week, so i guess they are proving themselves correct.
in practice, giving ssh/sudo to AI (codex) and telling it to setup ephemeral self-hosted runners on your own host VM (qemu), takes about 30 minutes and works extremely well. it bangs out all the shell scripts. moved my own business over to it. zero problems.
you might even find it is kind of neat too. you can customize your VM with software pre-installed. speeds up your builds quite a bit.
Worth being upfront with, I'm an employee, but; Buildkite.
Frontier labs and global saas heavyweights run CI through us, plus a crop of open source tooling like Bazel, vLLM, Firecracker, Rails, Ray, Sorbet and Rust.
Shopify, Uber, Airbnb, Canva stresstested the platform ten years ago for scaling issues.
You need to take a little time to design upfront and take advantage of dynamically programmable primitives. It isn't the same ease/convenience as Actions. Bit more available though.
How would I be able to run my CI via Buildkite when I still host in GitHub and their API is down? We are currently already using self-hosted runners and they also don't work because either they aren't started by GitHub or can't download the code.
We ran a mirror of our git repo on codecommit when things started to get really hairy, and pointed out CI to that instead of GitHub. It’s still a moving target though as we were using GitHub releases to publish to. We replaced that with an s3 bucket and mirrored _those_ to GH via actions, which had the added benefit of only failing when GitHub was down!
I want to second build kite - it’s an excellent piece of tech and is best described as “sane Jenkins” as a compliment. My only gripe is the pricing - it’s not designed for “small teams”. But I’d use it again in a heartbeat.
This week we've released a free plan and overhauled the mid-tier; am keen to hear whether this alleviates the 'small team' pain against others in market
I’m not working for a small team anymore but I can say this would have gotten us to use Buildkite again.
I think an in between tier would help, as it’s 4.5x the cost of GitHub per person, plus the cost of the runners. The idea of spending $42/mo on GitHub plus $180 on buuldkite is a hard sell, but the economics aren’t there.
That's good feedback, there's been a lot of discussion internally about how pricing has affected sentiment and what the trade-offs were from previous decision makers there over the last couple of years. Without it seeming like lip service, there is a 'startup' plan in the works, to sit between free and pro, sounds like it may be the right move as this isn't the first time we've heard it.
ultimately, folks need to stop writing massive YAML files to define their builds. CI should schedule jobs, your build system should define them. my workflow files look like this:
steps:
name: Do the thing
run: nix build .#
at any point, if my CI is down or i need to switch or i need to run builds locally, i can do so trivially.
If you're seriously looking at Gitea/Forgejo, I run Fjord, dedicated managed Forgejo, so you don't have to operate it yourself. We can help with migration and runners too. We also run a free public Forgejo instance if you just want to kick the tires first.
That's really good to hear, thank you. If you run into anything that's unclear while comparing the two, I'd genuinely like to hear it. Happy to answer questions here too.
It's worked great for hosting my smaller personal projects so far, and I've already received a few helpful pull requests from the community.
Tangled.org really been working to ensure all their features work from data hosted on your personal data server (it's based on ATproto) which means should their performance, reliability, or monetization ever get you down it's easy to switch to a competitor since you're not locked in.
If your app is not too large you can do CI/CD perfectly fine from your laptop. Put it in a docker container if you want to ensure it is perfectly reproducible. You still need Github as the origin repo to pull from, but the "git part" of GitHub seems to be (almost) always up and running. (We use "capistrano" for this)
> If your app is not too large you can do CI/CD perfectly fine from your laptop.
Not if you want to make sure it compiles and works on Windows (which you don't have) or macOS (which you don't have and can't emulate). That's something github gives for free, and for me, it's the only reason to stay on github for small/tiny free software projects (at least as a secondary host).
It's not the most elegant thing ever, but I've been using (self-hosted) WoodpeckerCI and it's been dead simple. It doesn't have many bells and whistles, but honestly I think that's good. Forced me to put complex behavior in better systems as opposed to doing bash scripts in YAML.
When you self host, what happens with your compute costs? CI can be super expensive depending upon your test pipeline. Do you end up spending more than you would with GitHub actions for example?
Just get some x86 and ARM runners off netcup, manage Woodpecker CI + Woodpecker Agent. Generally pretty hands-off once you get it set up. That plus a cheap VPS running Forgejo with postgres off-site backups, you are looking at a small fraction of the cost compared to per-seat GitHub pricing for orgs + Action Minutes.
He is likely referring to the CI runners. Gitea has their own version of Actions runner. It is very much usable, though currently inferior to GitHub Actions (not always a 1:1 replacement, even though they aim for it to be):
https://docs.gitea.com/usage/actions/
Cost as always depends on your usage, but it will likely be less than your current cost of GitHub Actions.
Thank you. This is exactly what I meant. I work for a large company where our testing budget is massive and most of it goes towards these CI runners. Provisioning enough and consistent compute for testing pipelines sometimes becomes the bane of my existence.
I remember in 2024 passing over codeberg as the host for some of my personal projects because of the uptime. I don’t think Codeberg’s uptime has gotten any better but despite that it’s getting a lead over GitHub now.
Tinfoil hat time but I think they definitely did _something_ at that time that "changed" the system. It's the Cambridge Analytica/Trump time and I believe that FB definitely "changed" at the same time.
After Cambridge Analytica, my "intelligent" friends basically abandoned the platform, while my distant conspiracy inclined uncles started posting baseless slop.
This is what I don't understand about this policy. There's no way a senior has enough spare capacity to be the gate keeper on every PR made by AI below them. So now we are just making it so the senior people use more AI to keep up but now they're to blame for letting it happen.
It sounds like a piss poor deal for seniors unless senior engineer now means professional code reviewer.
That's amazon in a nutshell though. Create conflicting metrics for performance, push credit up and responsibility down, punish everyone below you for not meeting the double standards
> Create conflicting metrics for performance, push credit up and responsibility down, punish everyone below you for not meeting the double standards
This resonates with my experience.
The only thing you forgot is that you can also use the 12^H^H 14 leadership principles to argue whatever you want (and then the opposite of what you argued last month, still using the same leadership principles).
Got a project finished early? Well, you didn't insist on the highest standards. Made sure things were held to a high standard? Well, you weren't biased for action.
Were you a knowledge source for the entire team? Well, you weren't learning and being curious. Did you ask a lot of questions to learn everything? Well, then you weren't "are right a lot".
Did you think big and come up with an architecture that saved Amazon a lot of money? Then you weren't inventing and simplifying. Build something simple to get out out the door quick? Well, you weren't thinking big.
Did you act quickly without consulting others to fix an issue? Well you weren't earning trust. Did you consult people to make sure they were happy with the solution? Well you weren't biased for action.
Very nice, I can imagine someone turning it into a little satirical webpage, which implements a kind of decision tree:
1. Choose from a set of challenge types (e.g. meeting a deadline, reliability)
2. Choose whether the challenge was "met" or "failed".
3. Choose whether you want to make the person look good or bad, by following/ignoring a principle.
4. Results: A list of relevant principles with short rationalizations.
I'm almost tempted to try, except perhaps I should treasure my ignorance.
If a tool like that gets popular enough that most employees are using it for office-politics, it might even start to deflate the whole Leadership Principles thing.
Early in my career I tried very hard to "be concrete, cold, and direct" because that's what I thought a good communicator would do. It was seen as attacking to anyone below me and confusing to anyone above me. I was naive and I suffered for it.
I definitely agree with what you're saying here where these words actually do mean something, but it's completely opaque to those outside the "know". I also have found that there's not any better way to express information to those in the group than in this coded language, even if it makes completely no sense to me.
I wish younger me understood that the way I'm being perceived is the only important thing, not choosing the "best" words to technically describe a situation
This feels like such a valid solution and is how past $dayjobs released things: send to the free users, rollout to Paying Users once that's proven to not blow up.
If your target is security, then _assuming your patch is actually valid_ you're giving better security coverage for free customers than to your paying ones.
Cloudflare is both, and their tradeoffs seem to be set on maximizing security at cost of availability. And it makes sense. A fully unavailable system is perfectly secure.
reply