> Increasing architectural complexity to enforce boundaries is never a solution to a lack of organizational discipline,
And yet we do this all the time. Your CI/CD blocking your PRs until tests pass? That's a costly technical solution to solve an issue of organizational discipline.
That's technical, and not architectural. I'm _all about_ technical solutions to lack of discipline, and in fact I think technical and process solutions are the only immediate way to create cultural solutions (which are the long-term ones). I'd even consider minor increases to architectural complexity for that purpose justifiable - it's a real problem, and trading to solve it is reasonable.
But architectural complexity has outsized long-term cost, and service-orientation in particular has a _lot_ of it. And in this particular case, it doesn't actually solve the problem, since you _can't_ successfully enforce those domain boundaries unless you already have them well-defined.
Can you explain the salient distinction between a "technical" versus "architectural" solution? Candidly, I'm not convinced that there is one.
> But architectural complexity has outsized long-term cost
As do technical solutions, of course. CI/CD systems are very expensive, just from a monetary perspective, but also impose significant burdens to developers in terms of blocking PRs, especially if there are flaky or expensive tests.
> And in this particular case, it doesn't actually solve the problem, since you _can't_ successfully enforce those domain boundaries unless you already have them well-defined.
Ignoring microservices, just focusing on underlying SoA for a moment, the boundary is the process. That is an enforceable boundary. I think what you're saying amounts to, in microservice parlance, that there is no way to prevent a single microservice from crossing multiple bounded contexts, that it ultimately relies on developers. This is true, but it's also just as true for good monolithic designs around modules - there is no technical constraint for a module to not expand into domains, becoming cluttered and overly complex.
Microservices do not make that problem harder, but SoA does give you a powerful technical tool for isolation.
> Can you explain the salient distinction between a "technical" versus "architectural" solution? Candidly, I'm not convinced that there is one.
Not concisely in the general case, but in this case the difference is fairly straightforward - CI/CD doesn't affect the structure of your executing application at all, only the surrounding context. I don't want to spend the hours it would take to characterize architecture as distinct from implementation, but the vast number of textbooks on the topic all generally agree that there is one, though they draw the lines in slightly different places.
> I think what you're saying amounts to,
Very much no - my point is about the process of implementation. The services.. _do_ enforce boundaries, but the boundaries they enforce may not be good ones.
In order to successfully extract services from a monolith, you have to go through a process that includes finding and creating those domain boundaries for the domain being extracted. If it's your first time, you might be doing that implicitly and without realizing it's what you're doing, but under the hood it's the bulk of the mental effort.
The part where you actually _introduce a service_ can be anywhere from a tenth to half of the work (that fraction varies a lot by technical stack and depending on how coupled the domain in question is to the central behavioral tangle in the monolith), but by the time you've gotten the domain boundary created you've _already solved_ the original problem. Now you're facing a trade of "extract this well-factored domain out to a separate service application, to prevent its now-clear boundaries from being violated in the future". And I contend that that's a trade that should rarely be made.
Which is exactly why the real challenges around (micro) services is to a large extent an organisational challenge mixed how those boundaries are belonging to which business capabilities.
The technical part of services is easy enough really but if the organisation leaks its boundaries, which always flow downstream into the actual teams, you are proper fucked.
It takes a different level of maturity, both in organisation and team, to build a service oriented software.
Right! What I'm fundamentally saying is that the majority of orgs trying to adopt SOA are doing it for the wrong reasons, and will deeply regret it. In general, the "right way" to adopt SOA is to extract services one at a time because you have to, to solve various problems you've encountered (scaling problems, technical problems, stability issues), and then realize that you are in fact currently using a service-oriented architecture, and have been for several years.
I would hope that there is more process in place protecting against downtime than code review - for example automated tests across several levels, burn-in testing, etc.
People are not reliable enough to leave them as the only protection against system failure...
Did you mean to reply to somebody else? I'm a huge believer in automated testing, and if I said something that can be interpreted otherwise I'd like to clarify it.
I guess the GP's issue is because automated tests (and every other kind of validation) imposes architectural constraints on your system, and thus are an exception to your rule.
I don't think that rule can be applied as universally as you stated it. But then, I have never seen anybody breaking it in a bad way that did also break it in a good way, so the people that need to hear it will have no problem with the simplified version until they grow a bit.
Anyway, that problem is very general of software development methods. Almost every one of them is contextual. And people start without the maturity to discern the context from the advice, so they tend to overgeneralize what they see.
Hm. I think maybe you're using "system" to mean a different thing than I am? I thinking of "the system" as the thing that is executing in production - it provides the business behavior we are trying to provide; there is a larger "system" surrounding it, that includes the processes and engineers, and the CI/CD pipelines - it too has an "architecture", and _that_ architecture gets (moderately) more complex when you add CI/CD. Is that where our communication is clashing?
Because the complexity of that outer system is also important, but there are a few very major differences between the two that are probably too obvious to belabor. But in general, architectural complexity in the inner system costs a lot more than it does in the outer system, because it's both higher churn (development practices change much slower than most products) and higher risk (taking production systems offline is much less permissible than freezing deployments)
> I think maybe you're using "system" to mean a different thing than I am?
No, I'm not. Are you overlooking some of the impacts of your tests and most of the impact of static verification?
Those do absolutely impact your system, not only your environment. For tests it's good to keep those impacts at a minimum (for static verification you want to maximize them), but they still have some.
I don't think I'm overlooking any major ones, but we are probably working in quite different types of systems - I'm not aware of any type of static verification I'd use in a rails application that would affect _architecture_ in a meaningful way (unless I would write quite terrifying code without the verifier I suppose).
I'm not sure about the tests - it depends on what you'd consider an impact possibly; I've been trained by my context to design code to be decomposable in a way that _is_ easily testable, but I'm not sure that is really an 'impact of the tests' (and I'm fairly confident that it makes the abstractions less complex instead of more).
Would you mind explaining what you mean in more detail?
No, not at all. CI/CD blocking pull requests is in place because large systems have large test suites and challenging dependencies which mean that individual developers literally can't run every test on their local machine and can often break things without realising it. It's not about organisational discipline, it's about ensuring correctness.
I can run every test on my machine if I want. It would be a manual effort, but wouldn't be hard to automate if I cared to try. However it would take about 5 days to finish. It isn't worth it when such tests rarely fail - the CI system just spin them off to many AWS nodes and if something fails then run just that test locally and I get results in a few hours (some of the tests are end to end integration that need more than half an hour).
Like any good test system I have a large suite of "unit" tests that run quick that I run locally before committing code - it takes a few minutes to get high code coverage if you care about that metric. Even then I just run the tests for x86-64, if they fail on arm that is something for my CI system to figure out for me.
The other problem is that these self-imposed roadblocks are so engrained in the modern SDLC that developers literally cannot imagine a world where they do not exist. I got _reamed_ by some "senior" engineers for merging a small PR without an approval recently. And we're not some megacorp, we're a 12 person engineering startup! We can make our own rules! We don't even have any customers...
Your 'senior' engineer is likely right: they are trying to get some kind of process going and you are actively sabotaging that. This could come back to haunt you later on when you by your lonesome decide to merge a 'small PR' with massive downtime as a result of not having your code reviewed. Ok, you say, I'm perfect. And I believe you. But now you have another problem: the other junior devs on your team who see vrosas commit and merge stuff by themselves will see you as their shining example. And as a result they by their lonesomes decide to merge 'small PR's with massive downtime as a result.
If you got _reamed_ you got off lucky: in plenty of places you'd be out on the street.
It may well be that you had it right but from context as given I hope this shows you some alternative perspective that might give you pause the next time you decide to throw out the rulebook, even in emergencies - especially in emergencies - these rules are there to keep you, your team and the company safe. In regulated industries you can multiply all of that by a factor of five or so.
> Your 'senior' engineer is likely right: they are trying to get some kind of process going and you are actively sabotaging that
Why? Because it's a "good practice"? They have 12 people and no customers, they can almost certainly adopt a very aggressive developer cycle that optimizes almost exclusively for happy-path velocity. You'd never do that at 50+ engineers with customers but for 12 engineers who have no customers? It's fine, in fact it's ideal.
> with massive downtime as a result.
They have no customers, downtime literally does not exist for them. You are following a dogmatic practice that is optimizing for a situation that literally does not exist within their company.
You establish a process before you need it, and code review, especially when starting up is a fantastic way to make sure that everybody is on the same page and that you don't end up with a bunch of latent issues further down the line. The fact that they have no customers today doesn't mean that they won't have any in the future and mistakes made today can cause downtime further down the line.
If you're wondering why software is crap: it is because every new generation of coders insists on making all the same mistakes all over again. Learn from the past, understand that 'good practice' has been established over many years of very expensive mistakes. 12 engineers is already a nice little recipe for pulling in 12 directions at once and even if they're all perfect they can still learn from looking at each others code and it will ensure that there are no critical dependencies on single individuals (which can bite you hard if one of them decides to leave, not unheard of in a startup) and that if need be labor can be re-divided without too much hassle.
I'm not advocating for having no processes, I'm advocating for a process that matches their situation. A company with no customers should not be worrying about causing a production outage, they should be worried about getting a demoable product out.
Dogmatic adherence to a process that limits developer velocity and optimizes for correct code is very likely the wrong call when you have no customers.
If it is dogmatic, then yes: but you have no knowledge of that and besides there are always people who believe there is too much process and there are people that there is too little. If you want to challenge the process you do that by talking about it not by breaking the process on purpose. That's an excellent way to get fired.
I don't know the context and I don't know the particular business the OP is talking about. What I do know is that if you feel that your management is cargo culting development methodology (which really does happen) you can either engage them constructively or you can leave for a better company. Going in with a confrontational mindset isn't going to be a good experience for anybody involved. Case in point: the OP is still upset enough that he feels it necessary to vent about this in an online forum.
Note that this is the same person who in another comment wrote:
"On the flip side I’m trying to convince my CTO to fire half our engineering team - a group of jokers he hired during the run-up who are now wildly overpaid and massively under-delivering. With all the tech talent out there I’m convinced we’d replace them all within a week."
Heh, both can be true. Process doesn't make good engineers any better. Bad code gets approved and merged every day. I'd rather have a team I could trust to merge and steward their code to production on their own instead of bureaucracy giving people a false sense of security.
> Case in point: the OP is still upset enough that he feels it necessary to vent about this in an online forum.
With no customers, one of the purposes of code-review is removed, but it's the lesser one anyway. The primary goal of code-review should _not_ be to "catch mistakes" in a well-functioning engineering team - that's a thing that happens, but mostly your CI handles that. Code-review is about unifying approaches, cross-pollinating strategies and techniques, and helping each other to improve as engineers.
Your attitude towards code-review on the other hand is one I've seen before several times, and I was glad when each of those people were fired.
We did post-merge code reviews. But half our team was on the other side of the planet from the other half (4 people on the team, US, EU. APAC, and AU).
If we waited for reviews before merging, we’d be waiting weeks to merge a single PR. Thus, you wrote your code, opened a PR, did a self-review, then deployed it. We had millions of customers, downtime was a real possibility. So you’d watch metrics and revert if anything looked slightly off.
You would wake up to your PR being reviewed. Sometimes there would be mistakes pointed out, suggestions to improve it, etc. Sometimes it was just a thumbs up emoji.
The point is, there are many ways to skin this cat and to “ream” someone for merging without deploying is incredibly immature and uncreative. You can still review a merged PR.
That process sounds fine to me, especially in a context with either good integration coverage or low downtime cost.
> to “ream” someone for merging without deploying is incredibly immature and uncreative.
I'd agree, but I _highly_ doubt that description was an accurate one. Read through the other comments by the same person and you'll get a picture of their personality pretty quickly.
It's likely that there was already an ongoing conflict either in general or specifically between them about this issue. They probably got a moderately harsh comment to the effect of "hey, you're expected to wait for code-reviews now, knock it off"
I suggested it's possible to write, commit and own code without others' approval to increase productivity and people get _extremely_ defensive about it. It's so odd. It happened in real life and it's happening in this thread now, too. They even attack your character over it.
Yes. Some people get personally attached to code. It’s incredibly frustrating. Some people use reviews to push dogmatic approaches to architecture and/or exert some kind of control over things. Whenever I meet these people in a code review, and they make unnecessary suggestions or whatever, my favorite phrase to say is, “I can get behind that, but I don’t think it’s worth the time to do that right now,” or, “I disagree, can you give an argument grounded in computer science.” With the latter only being used twice in my career, when someone left a shitload of comments suggesting variable name changes, and then again, when someone suggested rewriting something that was O(n) to O(n^2) and claimed it was better and wouldn’t give up.
You want to get the team to a point where you can disagree and commit, no code will ever be perfect and there is no reason spending 3-4 rounds of change requests trying. I think the worst code review I ever had, ended with me saying, “if you’re going to be this nitpicky, why don’t you take the ticket?” (It was extremely complex and hard to read — and there wasn’t any getting around it, lots of math, bit shifting, and other shenanigans. The reviewer kept making suggestions that would result in bugs, and then make more suggestions…)
He came back the next day and approved my PR once he understood the problem I was trying to solve.
Even these days, where I work on a close team IRL, I’ve been known to say, “if there are no objections, I’m merging this unreviewed code.” And then I usually get a thumbs up from the team, or they say something like “oh, I wanted to take a look at that. Give me a few mins I got sidetracked!” And I’ve even heard, “I already reviewed it, I just forgot to push approve!”
Communication is key in a team. Often, if the team is taking a long time to review, give them the benefit of the doubt, but don’t let yourself get blocked by a review.
If the code work/it's tested, review is for sanity checking/looking for obvious bugs.
Anything else is un-needed grooming that's more about the other developer's ego, not about good code (sometimes its to follow some other constraint, but its a good sign the person has a personality issue).
Well, partly that was a mistaken impression because I thought that your comment was also from vrosas. But I think there's enough in there to assess your attitude toward code-review at least a _bit_:
> They have 12 people and no customers, they can almost certainly adopt a very aggressive developer cycle that optimizes almost exclusively for happy-path velocity. You'd never do that at 50+ engineers with customers but for 12 engineers who have no customers? It's fine, in fact it's ideal.
12 engineers churning out code with no code-review at all? That'll produce velocity, for sure. It'll also produce not just an unmaintainable mess, but an interesting experiment, in which you get to find out which of your engineers are socially capable enough to initiate technical communication independently and construct technical rapport _without_ that process helping them to do so. Hope none of them hold strong technical opinions that clash!
No, because you're not infallible, you'll merge some crap, some things that are outright wrong, that your reviewer might have caught and that slight delay is less painful than dealing with that commited mistake - whether it be an incident in production or 'just' confusion when the next person in that area has to work out if your bug was for some reason intentional and what might break if they fix it.
I'm challenging my team to actually think about that process, why it's in place, how it's helping (or actively hurting!) us. Comparing ourselves to companies that have regulatory requirements (spoiler: we don't and likely won't for a long, long time) just furthers my point that no one really thinks about these things. They just cargo cult how everyone else does it.
You can challenge them without actually violating established process. I wasn't comparing you to companies that have regulatory requirements, I was merely saying that all of the above will factor in much, much stronger still in a regulated industry.
But not being in a regulated industry doesn't mean there isn't a very good reason to have a code review in your process, assuming it is used effectively and not for nitpicking.
Not having a code review step is usually a bad idea, unless everybody on your team is of absolutely amazing quality and they never make silly mistakes. I've yet to come across a team like that, but maybe you are the exception to the rule.
Then... the people responsible for this should have blocked PRs without a review. Or protected the target branch. Or... something. If it's sacrosanct to do what OP did, but the 'senior' folks didn't put in actual guardrails to prevent it... OP is not entirely at fault.
Really man. I have almost two decades developing software and yet, I feel a lot more comfortable having all my code reviewed. If anything I get annoyed by junior developers in my team when they just rub-stamp my PRs because supposedly I am this super senior guy that can't err. Code Reviews are supposed to give you peace of mind, not being a hassle.
During all this time, I've seen plenty of "small changes" having completely unexpected consequences, and sometimes all it would take to avoid would someone else seeing it from another perspective.
At 30 years of coding professionally in great engineering-focused organizations, and that on top of nearly a lifetime of having coded for myself, I’ve concluded code reviews barely work.
I agree with everything you say here, but honestly it’s quite ineffective. I wish we had found something better than unit tests (often misleading and fragile) and the ability of a qualified reviewer to maintain attention and context that a real review entails.
IMO catching bugs is a nice side-effect of code reviews.
The primary value I've seen across teams has been more on having shared team context across a codebase, if something goes bump in the night you've got a sense on how that part of the codebase works. It's also a great opportunity for other engineers to ask "why" and explain parts that aren't obvious or other context that's relevant. We'll find the occasional architectural mismatch(although we like to catch those much earlier in the design process) and certainly prevented bugs from shipping but if that's the primary focus I think a team is missing a lot of the value from regular code reviews.
Yes, I don’t disagree, I just think in practice they do the job very poorly. I’ve tried many things over the years trying to make this work but honestly have found nothing that works at the high end.
At the low end of engineering, yeah, code reviews matter a ton and do catch bugs even if they’re basically just peephole inspections.
I’m not convinced bad code would get merged more often if we didn’t require approvals. I am convinced we’d deliver code faster though, and that’s what I’m trying to optimize for. Your company and engineering problems are not the same as mine.
Indeed, dogmatic adherence to arbitrary patterns is a huge problem in our field. People have strong beliefs, "X good" or "X bad", with almost no idea of what X even is, what the alternatives are, why X was something people did or did not like, etc.
What you think is usually the limit of your experience, which effectively makes it anecdata. I've looked at enough companies and dealt with the aftermath of enough such instances that I beg to differ. It's possible that due to the nature of my business my experience is skewed in the other direction which makes that anecdata as well. But it is probably more important than you think (and less important than I think).
And yet we do this all the time. Your CI/CD blocking your PRs until tests pass? That's a costly technical solution to solve an issue of organizational discipline.