Aside: I used to run a small ISP, a 200-300 dedicated+virtual machines. We set up our router to alert us if outbound SSH connections from a host went above a certain threshold, which was a super reliable way of detecting if a host was compromised. I think we had a near 100% success rate, because once a host is compromised they use it to start trying to compromise other hosts.
But, we also had every customer on a VLAN, limited to only being able to send traffic from their IPs, and also blocking incoming and outgoing bogon traffic.
Years ago I attended a presentation by Evi Nemeth (RIP) related to CAIDA and one thing they found in auditing "backbone" traffic was that some huge percentage of it was bogon traffic (I don't recall the exact number, but lets say 10% +/- 6%). Nobody wanted to filter that traffic because the pipes were less expensive than the routers to handle filtering packets at high pps rates.
What I meant by that statement was that of the system compromises that we detected, nearly 100% of them were detected through the SSH outgoing connections alert.
Yes, there could have been compromises that went entirely undetected for years (we had a really high retention, so most customers stayed with us for 5+ years), so we had a good window to detect issues.
Probably the next biggest notification of compromise was alerts about spam on our network. Mostly that was an e-mail account compromise rather than system level. But there are a ton of false positives on spam alerts, particularly AOL alerts were almost always about legitimately sent e-mails.
> Nobody wanted to filter that traffic because the pipes were less expensive than the routers to handle filtering packets at high pps rates
Situation is better nowadays, modern routers like Juniper's Trio-based MX platform do things like bogon filtering and reverse patch checks at line rate without performance impact.
But, we also had every customer on a VLAN, limited to only being able to send traffic from their IPs, and also blocking incoming and outgoing bogon traffic.
Years ago I attended a presentation by Evi Nemeth (RIP) related to CAIDA and one thing they found in auditing "backbone" traffic was that some huge percentage of it was bogon traffic (I don't recall the exact number, but lets say 10% +/- 6%). Nobody wanted to filter that traffic because the pipes were less expensive than the routers to handle filtering packets at high pps rates.