If something hit production and caused a major fuck-up because there was no peer review process, then in all the places I've worked at the first action item in the post-mortem would be "we should introduce peer review." Otherwise someone would ask how we could ensure it wouldn't happen again, and no one would be able to say "because we trust them," and leave it at that. It would sound more like "I trust you will never make that mistake again," which sounds more like a threat. You also can't go to your clients and say "sorry, we let our team deploy stuff without reviewing because we trust them... so please continue to deal with our avoidable problems."
Thing is, I've lead my fair share of teams and I trust my team-mates and colleagues implicitly. It makes for a strong team. But we still do code review (peer review) because we want to build software that works well and support each other. We're not in the job to simply deploy code to prod as quickly as possible.
I'd also add that I think any software engineer should try and have the experience of working in a highly regulated field, like healthcare. It's hard to get an appreciation for why these things exist until you realise you're held to a much higher level of accountability because your oopsie moments can have much larger consequences. For me, it's been hard to go back to my old, cavalier attitude after that.
Exactly. I want my code peer reviewed partly because it makes it clear and formal that while we succeed as a team we also fail as a team. It’s much easier to talk about failures when a failure doesn’t have a single person attached to it.
I don't believe that to be true. I believe the expected outcome is that people will still want code reviews on anything vaguely complex as insurance against fuckups. "Shit, WE missed something" is a nicer place to be than "I was sure I was perfect and I'm actually an idiot".
It does however mean you don't sit around waiting for someone to approve a three line delete PR that you know is safe. The "Rubber Stamp" review is a thing and it does waste time.
Since I am currently working in an environment that resembles GP's premise, I have to say that I find this take to be way too optimistic.
I'd be exactly in the boat you describe (and actually: I've yet to see a situation where someone reviewing my code did not lead to improvements). But: If I want my code reviewed, I have to actually fight for it. Since everybody's workload is too high, even the willing often simply don't have the time.
From higher up, at our place, there's no one that actively opposes code review as a practice, but they don't seem to get that it requires investment (i.e. time), either. That means that if management refuses to change their ways of planning or explicitly enshrine reviews as priority tasks, it simply won't happen except for maybe the gnarliest cases.
And that's from a perspective of the change's author being willing and proactive. I work with individuals that produce very questionable code at times. And I'm not talking style here: Code in 2021 that still is prone to SQL injection at every turn. Methods that are hundreds of lines long with a gigantic cyclomatic complexity that no one (including the author) will ever really understand. Method names like "process" and "process2". And so on.
Needless to say, those programmers will never push for their changes to be reviewed. The rest of the team simply discovers them after they blow up (which they do regularly) and we're debugging.
Now, these people are luckily a tiny minority (which at least makes this somewhat bearable), but I over and over again would wish for someone with more authority to show that they're interested in the matter. Which a policy on code review would do -- if only as a signal. Yes, we want to check each other's code. Yes, we want collaboration. Yes, that's more important than some arbitrary deadline for minor feature X. Yes, we expect our programmers to be professionals.
Another symptom of such an environment also is the fact that everyone is the king of their particular hill. No one every familiarizes themselves with another person's code unless they really have to. By now, I've jumped to insisting that every change on a critical component I'll ever make will be done through pair programming, and so far, at least that seemed to stick. But again -- that only happens if both parties involved actively fight for it instead of everybody understanding that this is just an expected part of work.
This is a fantastic comment, thank you for writing it. I especially appreciate the explanation of how management decisions produce engineering practices.
I appreciate the context and the different point of view! I live in a bubble of how I perceive my coworkers (and indeed myself), alongside the culture of my employer where "getting it right is more important than getting it out"
Was interesting to hear your experiences and maybe update my expectations of workplaces somewhat
> [...] alongside the culture of my employer [...]
I think you've hit the nail on the head here. As many others have said in different HN posts, culture is hard to change. And I would also understand the argument that one can also enforce culture changes with policies; my hope up to now simply has always been that engineers could "overrule" some artificial time limits in favor of more collaboration by referencing such a policy.
I guess if you're starting from a context where code reviewing practices "just work", introducing explicit policies also can't improve much, anyway. On that note: If you happen to know a good heuristic to detect such cultures during interviews or the like, I would be extremely interested :)
This is where properly done scrum shines, provided the nature of the work lends itself to greenfield work (cycle time of one iteration). Good scrum is a dysfunction surfacing machine, so if you have a good system of risk management in place, the team can address in a constructive way. For all the scrum haters out there, come up with some system that codifies certainty, stability, and quality before your leadership starts pointing fingers.
> If code reviews aren't an expected part of the development process and there's pressure on delivering, it won't happen.
This hits at the heart of why I strongly agree with this post and think PRs and code reviews are ultimately destructive.
Reviews are going to get dropped _regardless_ of whether they are in the process or not. Pressure makes code reviews and quality worse, not better.
More than that, the need to "appear" to review is going to slow things down and can cause an exponential back up. I saw this happen about a mouth ago: 3 day release cycle got bogged down and then took _3 weeks_ to get a single release out.
Process goes out the window when pressure hits. You are then operating in foreign environment at the most critical point and your safety net of useless.
Your safety net when things need to go fast should be highly automated fast build and deploy, coupled with great test coverage, from unit to end to end.
The only exception might be if you work in highly regulated field where you legal required to have a very low Mean Time To Failure. In that case things move slow unless you are very well resources.
Otherwise (ie the majority of products) focus that effort on having a very low Mean Time to Resolve. If you absolutely need some form of quality assurance, Pair. Do continuous PRs and knowledge sharing rather than out of context after the fact reviews.
This sounds similar to the peer review process in publishing. Typically, getting other people to actually agree to voluntarily review your work is indeed a significant hurdle if the requested paper/book/chapter/white paper/code review is lengthy.
There is an easy way to resolve this:
Request peer review for SMALLER chunks of code.
Git makes this exceptionally easy -- it's certainly far easier than reviewing a large document with Word track changes!
The problem is that small changes are barely worth reviewing. Also a series of small changes can obfuscate the larger context of all those changes.
Each small change on its own might make sense and be ok to a reviewer, but if you step back and look at the series of changes together there might be some issues that could have been noticed.
Sure. But if you read my previous message this relies on people being comfortable asking. I am but not of my colleagues are. The formal process would let them use the benefit that I can use when I think it’s necessary.
I agree with you there, but I'm also sure that one reason for this is that many people still see reviews as the task that only happens after everything else.
I'm convinced there's a lot of useful middle ground to be had if reviews aren't just "here's 500 LOC, please review", but maybe more… staged.
Say: Rummage around in the code a bit for yourself (ask questions if you have to), come up with a strategy of solving your problem, and submit that for review. Not in code, but maybe using markdown, preudocode, small diagrams or the like. In person (i.e. via screenshare nowadays). And if the other party doesn't find obvious holes in your strategy, then go implementing it.
I remember an interview with some Linux kernel maintainer that mentioned the same effect -- people submitting a gigantic patch for something that will simply never be merged. And all that would have been necessary to avoid this wasted work was an e-mail asking if the maintainers are at all interested in the change.
This post sounds like wisdom, but after looking at it hard, it seems that the point is no more than "peer review processes are popular". Not to say that code review is bad in any way (I do it at my current role), but, this argument for them is not very good.
One downside of code review is reduced velocity. There are teams out there who use code review so effectively that they never ship big fuck-ups to production. They never have to tell clients that they have bad practices. Instead, they struggle to get clients because they pay much more per line of code shipped and therefore don't deliver as good of a value as their competitors.
This where the argument needs to be: whether and how to have code reviews be a net positive.
Ultimately I think it's down to where you want to pay for that decision (for want of a better term), or where to put the bottleneck. I think that putting the bottleneck in production (which would block all work unrelated to a fix if a catastrophic fuckup took place) might as well be a hedge against the possibility of things fucking up so badly in a given timeframe. You could also make code review a bottleneck, but I haven't seen that work well in practice compared to putting it at QA or as part of a release process (unless you combine them in a continuous delivery workflow).
I can see how that equation balances in favour of no code review when it's in the context of a startup that is probably still figuring out how to make money, especially with a subscription based app launcher. As far as prototyping and early stage iterations go, you're probably just seeing what works and reviewing the code isn't so important at that point. There would still be one or two things you'd ask someone to check, e.g. security related stuff.
I think you make a good point though: what makes an effective code review process? I don't think there's a single answer to that because it depends on the circumstances.
I would at least say that it goes better if it's treated as a priority and issues are raised and dealt with promptly, rather than leaving it as a chore you eventually get around to. And you would have to see value in it beyond it being a sanity check, as others in this thread have described (knowledge transfer, for example).
I'd argue that social proof often makes for a good argument. Perhaps not the best argument, but still a good one. "Best practices" still carry weight. Of course you can make an argument that the thing everyone does is wrong, and win. That's happened millions of times in this industry. But if the argument is nearly strictly between "this is what everyone does" vs "this is what I do", the former is going to win.
Sure, I mostly meant automated processes. They won't always save you from yourself but doing blue-green, having an extensive test suite, etc, are all things that will help reduce the risk of a bad deployment.
Yeah exactly, code review isn't a good place to catch actual bugs. Humans are terrible at catching bugs consistently... Humans probably introduced those bugs in the first place. It's also an exceedingly poor use of human time, which costs a heck of a lot more than machine time.
We really should be relying on automated systems to catch bugs for the most part, and leave code review for what it's good for: making sure a different human can understand and maintain the code.
In fact, if we manage to catch a bug in a code review that didn't also trigger our automated systems to alert us, that should probably prompt us to look into how we can expand the coverage of that automated system, similar to what we'd do in a postmortem if the bug wasn't caught in code review and reached production, because we can't rely on that happening the next time.
> code review isn't a good place to catch actual bugs.
Disagree, but it's also a good place to be ensuring that the system is evolving in a way that is reasonable and consistent — or that it is unreasonable and inconsistent for good reasons.
How do you systematically (and proactively) ensure your testing is thorough? With testing you often run into unknown unknown cases.
I’ve always code reviewed/gotten code reviewed on both the implementation and the tests, with completeness as the main quality you’re reviewing test code for. If that’s how we do it, we’re right back at code reviews.
That's definitely an unsolved (unsolvable?) problem. The best we can really do is start with a common sense set of initial tests based on the user observable behaviors we want from the system and then add more tests organically as we learn more about the system's edge cases through incidents, bugs and regressions.
I'm not saying code review can't find any bugs (in practice bugs are found in code review all the time), I'm just saying that if we do find a bug through code review, it means we got lucky, and we shouldn't count on getting lucky again, so better add a test/linter rule for the next time when we might not get so lucky. That's just one more way to grow your automated systems organically.
If there's one thing I've learned over many years writing lots and lots of automated tests for everything is that no matter how far you go with your tests, barring formal proofs or equivalent, the tests cannot save you from, at some point, breaking production badly. One small, tiny thing that slipped through your tests and boom - huge fuck up (even small things can easily cause your whole new feature to fail miserably on the big launch).
What big launch? :) Is useful to feature flag new functionality so that you can release gradually and spot the bugs - often via error monitoring and user feedback - before they affect everyone. Yes, realize it can happen but there are techniques to minimize the chance of it happening.
If you want to learn more about this, read up on 'continuous deployment'. I remember when I was first exposed to it it seemed dangerous/scary but with the right culture changes it can work really well; lots of companies do it.
Sure, but then you are just begging the question: if you have a better process for release verification you might not be need any per-commit code review at all.
It's the same in the financial sector, you can't just push code without a review.
However, the review process is far from perfect and can create a false sense of security.
To review a piece of code, that code should ideally be small in scope. For larger pieces it's rather common that the reviewer don't have enough time to do a good job.
If there were better incentives for doing a review, then it would greatly improve the situation.
I've had a lot of trouble getting people to do proper reviews. A minimum to me is that you actually compile and execute the code in some way to check its sane. Better would be the reviewer actually adds to the test suite for the code to prove their expectations of how it works.
In almost all cases it's very hard to get people to look outside the web browser for the diffs. Diffs show you something, but never the whole context.
The other problem I've found is stamping down on style arguments: they have no place in code reviews, and a lot of the time you get a round of "please change this variable to be named something else" - which might be valid, but raises the question of why the reviewer didn't do it, and more often leaves a weird authority gap - I have no power to just reject such a request because I think it's invalid, nor does the business have anyway to resolve the issue. In fact if there's a dispute at all, very rarely is there any process for bringing in a third party to mediate or break the stalemate (which is hilarious in some ways since everyone by now should realize 2-party systems can't achieve consensus).
Code reviews are very, very cargo-culty and I've not seen them ever include a thorough design: people give up on the very near edge cases, and so the whole system falls apart because it's just a hurdle to get through and not a collaborative or constructive process.
Why should it be on the reviewer to compile and execute the code, rather than the author?
Or are you saying that one can actually find more bugs when two people compile and execute the same code rather than just one?
When I review code, I want the author to tell me how they have verified that the code has the desired effect -- but if they do that, I'm going to trust that they did the actual verification and I'm not going to simply retrace their steps. That seems like a waste of time to me.
Not OP here. Code reviews, to me, are opportunities to
1) Catch edge use case / business requirements misunderstandings
2) Mentor new hires/junior engineers on the ways things should be done (e.g. internal culture about some architecture/design).
BUT, I absolutely despise code reviews that nitpick on non documented or 'because I prefer that way' stuff:
1) Task tickets that only have a vague description, have their own 'play detective' about the requirements, but a lot of new ones surface during code review.
2) Requirements are met, but there are internal/personal technical directives that are written nowhere, are not enforced with automated checks and may dramatically change a feature/patch is implemented (I can only think about that guy that worked in a company in which only LEFT JOIN sql statements were allowed).
Compiling and running the code is handled by a build server and automated tests. That's not the point of code reviews.
The main purpose of code reviews is checking the general structure of the code and tests. Is the code well designed? Are error cases considered and tested?
Another important purpose of code reviews is knowledge sharing.
If you don't catch major errors until the QA team has time to do a full comprehensive test of the product then those errors are probably sitting in the codebase for days or weeks, and additional changes are being piled on top of them which will make it harder to identify which commit was the problem. Do a code review and you'll catch a bunch of stuff before it even hits QA.
Correct. Any time you need a completely separate team to handle any aspect of the lifecycle of the product you're introducing delays and miscommunication. Besides, a separate team to do a part of the job scales badly: it needs to grow at least linearly with the number of developers, sometimes superlinearly.
You should absolutely have a small QA team, but their job shouldn't be doing the QA -- it should be helping others do their own QA.
Major errors are easy to spot and will likely take QA minutes to find. They're also probably not going to do a full test of the entire system with every change unless you've got a very small product.
One thing I've seen that I like a lot is individual changes creating individual front ends. You test the one thing in that change, and it's done. This tends to cause a lot of problems with CORS, but then what doesn't?
Agreed with you though that good code reviews (whatever that means) obviates most of this.
The actual job is always fast. What takes time is for the job to sit in the queue outside the QA department. The figurative papers moving between desks is always what kills your efficiency, not people doing work slowly.
I don't see that as mutually exclusive. A QA team can be more effective if you're taking steps beforehand to limit the scope of their work, which means your engineers need to start getting more hands on with that kind of work (code reviews, automated testing, whatever...)
Your quality problems might well come from poor or non-existent architectural decisions and tech-debt, and QA won't pick up on that stuff unless it manifests as a faulty requirement.
The earlier you catch a problem, the easier it is to fix.
Our tests have caught 10s of thousands of bugs. We couldn't move forward without them. Maybe they just fit our use case more than yours.
In our case we have a public API and several backends. The tests run against the public API. To know that a new backend is working it has to pass all the tests. Getting any new backend working correctly without the tests would be nearly impossible.
> For me, it's been hard to go back to my old, cavalier attitude after that.
I’ve heard from my workplace’s software team that this makes hiring from certain sectors particularly difficult because they don’t get this. Too used to move fast and break things and MVP.
Thing is, I've lead my fair share of teams and I trust my team-mates and colleagues implicitly. It makes for a strong team. But we still do code review (peer review) because we want to build software that works well and support each other. We're not in the job to simply deploy code to prod as quickly as possible.
I'd also add that I think any software engineer should try and have the experience of working in a highly regulated field, like healthcare. It's hard to get an appreciation for why these things exist until you realise you're held to a much higher level of accountability because your oopsie moments can have much larger consequences. For me, it's been hard to go back to my old, cavalier attitude after that.