Perhaps in theory. Almost every application under the sun I've encountered expects 1 to be red, or 4 to be blue, or 3 to be yellow, or whatever.
As an example, take solarized, or really any of the base16 themes. When I've used solarized in the past, many terminal programs that expect to print something in red are actually printing in a slightly-dimmed-background color, or purple, or something else where the intention of the tool author and the intention of the theme author conflict.
The less you know about them, the higher the abstraction. A program can distribute meaning over colorN in any way it wants. Any fixed semantic mapping would prove insufficient in the universe of all possible programs.
