Your question seems to indicate that you have a runtime dependency on github. If true, this is a problem no matter where your git repo resides. You need to architect a solution wherein you do not have a runtime dependency, especially if it is a single point of failure, or accept that your reliability will never be better than the sum of your services' reli abilities (there was an article on the "calculus of availability" or something like that recently).
What I personally do in AWS is bake my artifacts or other git-sourced data into AMI's. If you want a middle ground, you can instead push your artifacts to an s3 target -- s3 has better reliability / track record than github for this purpose.
Thing is, runtime dependencies on Github is something that Github is promoting and selling now (https://github.com/marketplace/). Maybe this incident puts that value proposition in a more visceral context.
How is the GitHub marketplace a runtime dependency? Code review, project management, build bots or whatever else should NOT affect the availability of your website.
see now you're just splitting hairs and getting into philosophy. if your website will be stale tomorrow because you can't update it today, then that is a runtime dependency imo.
Yeah, I should have clarified I was talking with my operations hat on. From a development perspective, you might call it a "deploy-time" or "scale up" dependency.
naive tip: give your app that is relying on github a SKIPGIT environment variable to ignore the bits around git operations, and just make local changes as an interim hack.
>
Your question seems to indicate that you have a runtime dependency on github.
Really he is speaking to the problem with SAAS. Sure it's cheaper to rely on someone else to do the heavy lifting for you and they can do this because they 'make it up in volume'. But the other side of that coin is no one really knows how to do that anymore. If you wanted to roll your own it's going to be very hard for your company to do the basics because you've become dependent upon the cloud, and someone else paying employees to do the basics for you... The basics have never been hard.
But a positiv counter argument would be that github/gitlab also does the sharing part of the jobs and you (hopefully) pay them to handle all the security aspect of that public sharing.
Opening a port to your local instance of git implies more thinking and security consideration. Of course you could host a mirror on a cloud instance, but then it's saas anyway and you have even more work to do before ever starting coding.
So yeah github/gitlab does the heavy lifting for you, but for small organisations that might be cheaper overall than to pay somebody mastering all required stack to implement and maintain your own instance(s). For big company no question that an in house git team is probably more efficient.
Is it still considered a runtime dependency if all you do is merge code and deploy while relying on GitHub? I assume most apps will not go down if GitHub goes down, but their ability to move code to production gets stalled.
Git supports multiple remote repositories. Have github be your main repo, keep a mirror locally or on another service (updated using hooks), and switch to it if github fails.
Speaking from an operations perspective, yes, it's a runtime dependency. From a development perspective, you might think of it as a "deploy-time" dependency. Think of it in contrast to compile errors. If your program has errors that your compiler can catch, then it will refuse to compile the program and you must fix the error before executing the program. A runtime dependency is one that may change after your program is compiled -- these are harder to test, especially when the program may fail depending on its environment. In the case of dependencies on services such as Github at deploy time, it's a "runtime dependency" for the operations programs. Since the operations programs are the ones with the runtime dependency, it's also in the operations team's purview (and responsibility) to revise their program(s) and eliminate the runtime dependency such that testing (and reproducibility of) deployment is deterministic and reliable.
There's the convenience reason. Teams making that trade-off (not having a local mirror) need to understand they made GitHub itself a runtime dependency.
But in theory in a pinch you could spin up a git server somewhere publicly accessible and switch your site to pull from there, and push to there yourself.
Agreed, that shouldn't be too hard if you're really in a emergency and can't wait. What I see though is some deep integration with their API with automatic tagging, releases, etc... That's hard to work around but still not impossible if you're willing to live with some chaos in your process until things are back.
Not if you're using more than git from github, for example downloading releases as zip files for deployment.
There are probably a lot of other tempting features to put on your deployment critical path but I'm not in an environment that uses github so I've forgotten.
Webhooks are a big one: rather than polling a git server for changes it is very easy for CI/CD systems to rely on GitHub's HTTP webhooks in their critical path to kick things off on push/PR/branch/merge. Especially because GitHub does a great job of populating its webhooks with tons of useful information about the event that's tough to replicate with just git post-hooks.
"Becoming"? Can you point to any large project (at any point in the history of Git) that doesn't use some centralized service as a single point of distribution/source of truth?
The beauty of Git's design, though, is that you've still got everything to use it distributed even if you, under normal circumstances, just use it centralized like a fancy SVN.
> Is it still considered a runtime dependency if all you do is merge code and deploy while relying on GitHub?
It's not a runtime dependency, it's a stupidity to rely on random
repositories from the internets when building and deploying (remember the
left-pad farca that halted work for half of the web programmers). You should
always have your local copy of necessary sources or use repositories that can
trivially be swapped with some other mirror.
We used to have the infrastructure for those things (in the form of binary and
source package repositories), but it was mainly used by sysadmins, and
programmers usually have an alergy for such infrastructure.
> you can instead push your artifacts to an s3 target
This is our tactic. I tried to do baked AMIs at one point, but the 10-15 minute turnaround in registering them meant that we couldn't use them for staging or testing (too long to iterate changes).
Previously we were capistrano-deploying with git from bitbucket - every server had to individually git pull from servers on the other side of the continent, over the public internet. Susceptible to all sorts of problems.
You can speed up AMI baking by using chroot + a mounted EBS volume on a running instance. It should drastically reduce the time it takes to produce a new image. Also be sure that you're building EBS-backed AMI's, as S3-backed AMI's will require a lot of time for copy.
A common solution is to decide that github is down rarely enough that this isn't a problem. Eg if your only dependency on github is at deploy time, it might be ok if you can't deploy for a few hours a year when github is down. If you have a bigger (eg runtime) dependency or if being unable to deploy for a few hours is a problem, then yes– you run your own git server.
Our stats show a 1:239K request error rate for S3, and a near 100% availability. There has only been one outage of note in years. GH on the other hand is down with shocking frequency like today and yesterday, though not as much as BitBucket.
Some companies I know write to GCS and S3 in parallel, which is easy given many tools and API's work directly with both (e.g. AWSCLI/gsutil).
Others will say "work locally," but the problem is more in deployment. There's still a lot of app and CI servers out there where the workflow is "push to Github, then some automated script will tell the cluster to pull from Github" rather than "push the changes directly to the cluster." Containerization, especially projects like Kubernetes, make the latter much more viable; your cluster handles its own orchestration and deployment. It's still early days, though, and there's a lot that a new user can get wrong in setting such a system up. Ansible telling servers to git pull from Github may give you better uptime once you factor in your own mistakes.
> then some automated script will tell the cluster to pull from Github
GitHub isn't a deployment tool. It's a source control tool. Keep your deployment artifacts somewhere else (e.g. S3/GCS) more durable so you can still deploy/rollback during a GH outage.
Could you elaborate on how Kubernetes makes it easier to deploy code changes directly to the cluster? Are you talking about the Docker/container image build?
It's obviously possible to manage very large projects stored in Git repos with no dependency on a site like GitHub at all. Just look at what Git was designed for, after all.
However, I'd say one of the biggest practical problems with most of the popular DVCS tools is still that they don't also have a simple, out-of-the-box way to set up a centralised repo. As you say, using Git+SSH works up to a point, but it's still irritating and somewhat time-consuming to set up if you work with a lot of projects that might each have different contributors. Last time I checked, none of the main alternatives was any better.
It depends on what your deployment system is; it's not always trivial to just change the target git origin to a new server. For someone who knows what they're doing and has the time to focus on doing it, it won't be too hard, but Murphy's Law gets in the way.
It depends on what your deployment system is; it's not always trivial to just change the target git origin to a new server.
This seems to be a fundamental weakness in whatever deployment system is used, then. Relying on a single point of failure outside of your organisation's own control for anything critical is always risky, and in this case it seems to be entirely avoidable.
You need to either ensure you control the resources you depend on, or plan for outages.
At my gig, we use GH Enterprise (on-prem) as well as some smaller plain-vanilla git repos (mainly for CM and other critical services). Any third-party code and all other dependencies live in an artifact manager or private mirror repos.
Personally, I don't use any cloud provider to store my data. Cliche, but true: if you don't control the hardware, you don't control the data on it.
With each GitHub failure people learn some good lessons. Redundancy is the key. Push to multiple remotes and fall back to backup remotes in your scripts.
Github has added all sorts of proprietary integration points to prevent it from being this easy, like push based checkin notifications. Then developers get lure by the drag'n'drop CI servers that integrate with them.
Unfortunately people only care about API's now instead of standards.
Most items of ours that have a production dependency on GitHub actually point at a simpler (and much stupider) mirror. When commits are made pushes are mirrored to those repos. These repos can even be naively replicated since git's nature allows for this.
If you _need_ Github to be up, mirror it. If your build or deployment tools _need_ Github to be up, and you _need_ them to be always available make simple dumb mirrors.
Also make sure your tools don't lock on only one item.
It was in lunchtime in New Zealand, so doesnt really matter. ;) I am just back with our world famous wellingtonian flat white, and looks everything perfect. :)
At this point Github is a national security risk. The amount of passwords, keys, hashes, code, data uploaded to that system is massive. I am sure that government agencies, contractors, multinational defense departments, etc are dumping things on there hourly. Will it get hacked? Get your popcorn out. It will happen.
There's e.g. https://aws.amazon.com/govcloud-us/details/ these days. Azure and Google have government offerings as well. Safe to say neither would be bothering if the government didn't use cloud products these days.
For some projects, yes. The biggest well known US examples are the 18F open source efforts and http://code.gov / http://code.mil open source efforts, most of which are public, open source on GitHub.
Reinvent VMS-style clustering on BSD/Linux boxes or just buy some VMS servers. They measured uptime in years to decades for tech that started in the 80's. There should be several, modern solutions like that given companies were developing them a long time ago when I first looked into such things. I just haven't surveyed to see what exists, is reliable, and is affordable.
They could also have a base version of the site that's read-only for source or binaries to allow customers access to the data to fall back to whenever problems happen in more complex functions. Keep it running in some form while they fix things. They might also similarly sell Git appliances w/ high-availability that themselves plug into multiple colo's or cloud providers mirroring changes in the repo to the Github site. Just a wild idea as I have no idea if that's marketable.
When you build a docker image, you copy all your runtime dependencies (`node_modules/`) and build artifacts into the image. When you push a deployment you are deploying that static image with the dependencies baked in, instead of trying to install the dependencies at startup.
You can run a local copy on a server of something like Github. For example BitBucket Server (which used to be called Stash). I worked somewhere that was on GitHub but cried when it was down, then went to BitBucket and cried when it was down. It's a big point of failure.
Isn't every git client a server? Since git is peer-peer, when GitHub is inaccessible we just merge back and forth (or just commit locally -- it's never been down for days).
Does anyone know an alternative to githost.io? We don't want to use gitlab.org as it's painfully slow but also don't want to manage our own gitlab instance.
edit: We don't like the recent changes made in githost. ( Replaced the $35 option with $149 ) We're a team of 15 people.
edit2: should have clarified; I'm hooked to GitLab, just looking for another service that does managed hosting. Not looking for GitLab alternatives.
Gitea sadly has some issues (for example, the "Public Activity" tab on user profiles 500's atm: https://try.gitea.io/crackedmind?tab=activity; it throws 500's everywhere once you throw on a modern Xeon more than 30 users/second at it, because it doesn't cache database accesses, at all, etc).
Also, I don’t seem to be able to properly run it in docker (I’m trying to deploy it on kubernetes), do you, by random chance, have experience with deploying gitea?
Sorry, I don't know about running it in docker. My usecase is personal lightweight repos management, so I just download the binary and run it backed by SQLite. Never faced any issues with that. I never needed to go fancy with docker.
