With more understanding one has more potential for optimization. And I don't mean performance or memory. I mean conceptual optimization. Reducing a solution of a problem to the minimum amount of concepts needed and finding a set of correct concepts to use in the solution. Once such a set is found, one might not need to think more about their details. The beauty is, that one could! Provided, that those concepts are implemented in the same clean minimalistic ways, understanding them is possible as well.
When one does not understand the details and instead just pokes around in the code "until it works", one will forever be blind for that kind of optimization and the solution created tends to have leaky abstractions and become even less composable when trying to use it to solve future problems. Reusability is reduced and we keep reinventing things over and over, but not getting a clean version out of it. This is part of why a lot of software we have is so brittle and resource hungry. We do not bother with proper understanding enough and cluelessly throw together balls of mud.
EDIT: On a second thought: Software engineering should not be like sciences in the way that we poke the "system" and see what happens. Why? Because sciences only do this, because the world existed before humans existed. We have not made it and are are inside of it. We have no choice but to poke and try to figure it all out. With computers this is not so. We have build these machines. We can and should understand what we make them do.
>With more understanding one has more potential for optimization. And I don't mean performance or memory. I mean conceptual optimization. Reducing a solution of a problem to the minimum amount of concepts needed and finding a set of correct concepts to use in the solution
Except what I'm saying is that human brain has biological limits on how much and what it can understand. Eventually a system can become so complex the only way it can be developed is through the efforts of multiple people specializing on specific components.
>When one does not understand the details and instead just pokes around in the code "until it works", one will forever be blind for that kind of optimization and the solution created tends to have leaky abstractions and become even less composable when trying to use it to solve future problems.
For a sufficiently complex system This cannot be Avoided, simply because the system is too complex to be fully understood by a single person. I am 100% talking about conceptual optimization.
I get what you are saying and it is definitely true, that systems can become too big to keep in mind completely. However, with proper and non-leaky abstractions it should be possible to reason about the system at each level of abstraction, without having to think about the lower level. The thing is then, that one could descend into the level 1 lower than one is looking at, if needed, and find out, that a simpler primitive could be used to cover more ground.
For example: Exceptions can be implemented using continuations. A return statement can also be implemented using continuations. Both can be implemented using the same concept. On the layer of exceptions and return statements, they seem like different things, but we can look under the hood and see, that we could use continuations, a single concept, to express both, if we wanted. With such optimizations one can reduce the number of concepts on needs to keep in mind lower and the system more understandable.
When one does not understand the details and instead just pokes around in the code "until it works", one will forever be blind for that kind of optimization and the solution created tends to have leaky abstractions and become even less composable when trying to use it to solve future problems. Reusability is reduced and we keep reinventing things over and over, but not getting a clean version out of it. This is part of why a lot of software we have is so brittle and resource hungry. We do not bother with proper understanding enough and cluelessly throw together balls of mud.
EDIT: On a second thought: Software engineering should not be like sciences in the way that we poke the "system" and see what happens. Why? Because sciences only do this, because the world existed before humans existed. We have not made it and are are inside of it. We have no choice but to poke and try to figure it all out. With computers this is not so. We have build these machines. We can and should understand what we make them do.