> Imagine the "pollution" caused by civilizations [...] Destabilizing solar systems or even entire galaxies, maybe even violating causality..

If you are interested in such a plotline you should read the "The Three-Body Problem" triology by Liu Cixin.

Without going to much into detail: This topic pops up several times in the third (and perhaps already in the second) book of the series.