> the full IPv4 range is scanned regularly by multiple entities

Single packet authorization. Server just drops any and all packets unless you send a cryptographically signed packet to it first. To all these observers, it's like the server is not even there.