If you're doing it right, you are solving very different problems with shell vs any other language. Shell is best used as a tool for orchestrating other programs, you should not be implementing your programs in shell.
Syscalls, in general, are used in lieu of objects or other abstractions because they more accurate mirror what the underlying hardware is doing. This isn't always the case, some syscalls are maintained for POSIX-compatibility and add a lot of complexity to emulate behavior that is no longer reflective of the hardware.
At the end of the day, you'll find that it's very difficult to maintain the highest levels of performance while also presenting an API that has a high level of abstraction. Things like dynamically-resizable lists, using hash tables for everything, runtime metaprogramming, and other such niceties of modern HLLs aren't free from a performance perspective.
If you really want to know more, I would suggest reading one of McKusick books on operating system design (the most recent being The Design and Implementation of the FreeBSD Operating System 2/e, but even the older ones are still largely relevant).
Maintaining this "useless POSIX compatibility trap" has a certain amount of utility; I for one like not having to re-write all of my programs every few years. I imagine others feel the same.
In closing, some projects that are pushing the boundaries of OS design which you may want to check out include:
When I first learned one could write programs in Bash, I got very excited. It didn't take long for that to where off :) My Bash got a lot better though!
I think everyone who become proficient in shell tries to implement something large in it, gets burned, and realizes that it’s a bad idea. I certainly had that experience.
I think parent means (generally speaking) not to implement larger programs in shell.
If the tasks performed by your program can be expressed maintainably and performantly in shell, go for it.
For programs one size up, something like Perl or Ruby or any other language that allows easy execution of shell commands can be a great solution.
Of course, this all depends on any number of variables. Team competencies, target environments, etc. And with some effort even larger bash scripts can be kept maintainable-ish, and in some target environments (can't always count on other languages being available!) shell scripts might be the only available programming environment.
It depends. I don’t think there is a language which is categorically the best. The solution to a given problem may be harder or easier to express in different languages. Some problems are better solved in C, some in Python, ..., and some in Shell.
Syscalls, in general, are used in lieu of objects or other abstractions because they more accurate mirror what the underlying hardware is doing. This isn't always the case, some syscalls are maintained for POSIX-compatibility and add a lot of complexity to emulate behavior that is no longer reflective of the hardware.
At the end of the day, you'll find that it's very difficult to maintain the highest levels of performance while also presenting an API that has a high level of abstraction. Things like dynamically-resizable lists, using hash tables for everything, runtime metaprogramming, and other such niceties of modern HLLs aren't free from a performance perspective.
If you really want to know more, I would suggest reading one of McKusick books on operating system design (the most recent being The Design and Implementation of the FreeBSD Operating System 2/e, but even the older ones are still largely relevant).
Maintaining this "useless POSIX compatibility trap" has a certain amount of utility; I for one like not having to re-write all of my programs every few years. I imagine others feel the same.
In closing, some projects that are pushing the boundaries of OS design which you may want to check out include:
* Redox OS (https://www.redox-os.org/) - a UNIX-like OS done from scratch in Rust
* OpenBSD (https://www.openbsd.org/) - one of the old-school Unices, written in plain old C, but with some modern security tricks up it's sleeves
* Helen OS (http://www.helenos.org/) - a new microkernel OS written from scratch in C++, Helen OS is not UNIX-like
* DragonFlyBSD (https://www.dragonflybsd.org/) - a FreeBSD fork focused on file systems research
* Haiku (https://www.haiku-os.org/) - binary and source compatible with BeOS, mostly written in C++, but also has a POSIX compatibility layer