On the multi-core systems of today, why are we using memory and the cache as a communication device between threads running on different cores? Don't we have the spare transistors and space on the die to implement purpose-built communication hardware between cpu cores which does not fall victim to false sharing, and the like? Isn't the current setup a bit hacky?
Almost all communications between devices is memory mapped i/o these days. x86 has i/o ports, but that's mostly used for devices that were specified long ago (although some things are still relevant).
If there was a core to core bulk messaging system, it would be memory mapped as well, and you'd need to store the messages in some sort of memory, so why add a specialized message queue, when you can just use memory?
You can do things like add a message to a mailbox in memory, and then send an interrupt to the other processor to indicate it's ready.
If there was a core to core bulk messaging system, it would be memory mapped as well, and you'd need to store the messages in some sort of memory, so why add a specialized message queue, when you can just use memory?
So one could code inter-thread communication without having to think about inducing large numbers of cache misses and degrading performance.
I think there are some embedded platforms that used message passing as a primary form of communication. However those are really out there and not at all mainstream. I think what you're talking about is coming. For example, for ARM there is a network on chip architecture that I could see being used as a back bone for this type of system. However, you need software support for it. It's not enough to just build the hardware.
I think this is primarily about backwards compatibility. New CPUs, especially from Intel, were about running existing software faster. The shared memory paradigm goes back to single core (and mainframes) and is pervasive, in code, in languages, in libraries. It's really hard to rewrite all the software on this planet to use some channel mechanism between cores and that also makes it tricky when the OS needs to schedule threads. I think there were some historical examples of this sort of architecture (Transputers?) and generally they didn't fare that well...
If you made a new CPU where there's no shared memory between cores I'm not quite sure who would use it. It's also not that clear how much simpler it makes things on the chip and for the developers.
I think you have to choose between easy and fast in this case. A separate dedicated memory controller that all of the cores are connected to that handles all of the cache would make this problem a lot easier, but would defeat the purpose of the L1 cache.
The question is, why are we dealing with cross-core cache invalidation at all. When core A writes to memory location X, then core B reads from memory location X, why do we insist that core B reads what core A wrote? Almost by definition, the only reason to make such a requirement is that we want to use main memory as a mechanism to communicate between cores.
An alternative architecture would be to say that different cores should not be using the same region of memory. If they attempt to do so, it is an error and exactly what happens is undefined. We only need to worry about cache invalidation when we transfer ownership of a region of memory between cores.
If we had a non memory method of cross core communication, we would need to pass memory between cores far less often, which would greatly simplyfy the problem of memory cache invaldiation.
> different cores should not be using the same region of memory
Generally known as "NUMA"; this is a viable programmer's model, but it's different from what people are used to, and requires either software changes or a lot of performance-impacting compatibility layers when software accesses pre-existing global variables.
(You could certainly get NUMA multiprocessing systems back in the day - to make effective use you had to pin processes to cores, because the cost of migrating memory about was considerable otherwise)
> the only reason to make such a requirement is that we want to use main memory as a mechanism to communicate between cores.
For a very liberal definition of "communicate" that statement is true. The only reason for multiple threads to be sharing an address space is if they mutually depend on how the others are interacting with the data in there.
The reason for cross-core cache invalidation is that otherwise it becomes impossible to reason about behavior of any data structures written and read by different threads. Imagine trying to build multiple consumer queue without any cache coherence protocols.
