In terms of hardening against SSH attacks, the principles are quite simple. Your business case might mean that some of the following cannot be applied, but there's plenty of measures below that you can use to harden SSH.
1. firewall - only allow SSH connections from trusted static IPs
2. Use SSH keys then disable password logins. Lots of guides online to create keys, so I'll just cover the 2nd point: as root or sudo, edit /etc/ssh/sshd_config
PasswordAuthentication no
ChallengeResponseAuthentication no
# restart sshd
(edit: make sure the SSH keys have passphrases. That way you have an extra bit of security in case any workstations get compromised)
3. Disable root access. edit /etc/ssh/sshd_config
PermitRootLogin no
# restart sshd
4. Limit SSH access to specific user accounts. This prevents users creating their own key (in the case of mountable home directories) or other machine accounts with passwords (if you've not done #2):
groupadd sshaccess
# add all users to the sshaccess group. lots of different ways to do this. The following will work on some flavours of Linux but not all:
usermod -a -G sshaccess $USER
# now edit /etc/ssh/sshd_config and add the following in (it wont already exist)
AllowGroups sshaccess
# restart sshd
5. Install auto-firewalling for failed SSH logins. I personally favour fail2ban as that covers other scenarios too, but I've also used denyhosts and that's worked well for SSH.
6. Lastly, and by far the best option, don't enable SSH on any internet facing IPs.
If you need SFTP enabled, then let me know and I'll post some details on how to harden SFTP so attackers cannot gain an SSH shell.
Keep in mind though that while securing SSH is a good approach, SSH itself is very unlikely to be the route of compromise unless an extremely insecure account were present with weak password auth.
It's far more likely that the attacker got legit credentials via another means, web application vulnerability, social engineering or malware attack on company machines, etc. I'd look at the less common applications you run, particularly anything that doesn't particularly look like it was designed to run facing the internet. For example, the elasticsearch guys decided that it would be a great idea to allow anyone who can access it to run java code on the server at one point...
You assume the breach happened over SSH. This is valuable information to securing SSH, but it's entirely possible the original breach happened over some other service, and there were some other steps involved in the breach before the SSH screenshot was taken.
True but I'm working from the angle that If the breach happened via some other means then they'd need some way to remotely execute code to enable SSH, create valid login credentials, and disable the firewall; in which case they already have a more convenient shell access so gaining access to SSH becomes redundant.
However it's possible that the attacker's screenshot was of a remote shell initiated via some other means and the OP assumed it was via SSH.
Edit: why was this downvoted? If there's an error then I need to be educated. I've spent enough years of my professional life hardening servers to have some idea what I'm talking about, but I'd be an idiot if I didn't listen to the expertise of others. So please correct me rather than downvote me :)
Best not to ask why downvoted. Those people's responses will rarely teach you anything. The kind that would will usually reply instead of downvote. Plus, a few already explained to me it's common for a post to get hit with a few negative votes followed by corrective action as other, open-minded people show up. Happens all the time with mine.
IMHO you can skip most of this except #2. Key-only login is very important, blocking access with fail2ban not so much. But you definitely need to send your logs to another box so you got a copy (that is unreachable for the attacker) of the logs that this guy definitely removed from the hacked system.
I do agree but a lot of it depends on the business case. Sometimes the business requirements dictate you have password authentication enabled. :(
You're right that key only logins are the easiest quick win, but it's also worth remembering that they can still be vulnerable if the box needs passphraseless keys (eg lazy automation) or if you cannot guarantee the security of those keys (eg they're generated by third parties and/or stored on third party systems). In those situations it would pay to firewall SSH and specifically whitelist the third parties IP address.
Good suggestion about the logs too. That's an often forgotten step yet invaluable when it comes to forensics.
One extra tip (if you aren't confident in the above process, and are performing it over SSH, and don't have physical access to the server):
Test the new access before closing the current session! If you made a mistake in any of the steps you will still be able to fix it. If you close the SSH session before testing, and then find that you made a mistake (e.g. forgot to `chmod 600` the private SSH key or something similar) you're stuck!
Also, to get rid of 60% of the script kiddies, change the port to something high 1000+ (or even better 10000+). Its not a great option, but it does lower the amount of automated attacks that you get.
I'd be very cautious before moving SSH to a non-privileged port (over 1024). Any user on the server might start their own SSH server on the port assuming the real SSH server is dead. While this is hard to exploit (needs access to normal user, needs to kill real SSH server, need to get around SSH server key checking), it still is at least a theoretical reduction in security.
Besides local iptables you can forward the port at the firewall level. Many people (myself included) have observed failed ssh logins going from many thousands daily to on average zero just by changing the port on a netfacing server. Of course a determined hacker who is after you can trivially portscan. But why not block all that noise and a huge percentage of shotgun attacks? If someone is out just to find servers to root with a new zero day they're liable to spray the net searching on the target port rather than portscanning all IPv4 space. Why not buy yourself some free time?
It is basically zero inconvenience to add an extra argument or shell setting.
A user can always start their own SSH server. Just because you've decided to move it to a different port doesn't really encourage them. I suppose you could make this a bit more difficult for them by removing compilers (no really, you don't need compilers everywhere) and making sshd owned by root, mode 700...
However, proper ingress filtering or local iptables/pf rules would stop any unwanted inbound traffic from reaching your server, and you should definitely be using ingress and egress filtering on your network.
removing firewalls has effectively no benefit; a non-root user can trivially download and run an arbitrary distro or package manager (e.g. nix from nixos, portage from a gentoo prefix, etc) and effectively do a chroot + package management without root.
I'm not a fan of that option because it's causes an inconvenience for legitimate usage without offering any real security benefits. If you've hardened sshd then the biggest benefit you get from changing the port number is reducing the size of your log files. If you've not hardened sshd then you're just one nmap (et al) scan away from being in the same boat as you were previously.
If you really want to hide SSH from script kiddies then you're better off setting up firewall rules to whitelist trusted IPs. Heck, even port knocking[1] would be a better option than changing the port number.
How is port knocking not, I dunno, at least 10x more inconvenient for users?
FWIW, I've never seen a brute force attack on SSH other than on port 22. Most likely, a targeted attacker would realise if you change the port, you're probably not going to have a trivial password.
I agree, I set up port knocking and it's a huge pain in the butt (so much so that I never moved it out of testing). It also lacks cryptographic verification. It's a lot of trouble for a not quite ideal solution.
Port knocking is more inconvenient but its also more secure. I'm not advocating it though, just example how low down the scale I consider non-standard ports as a sensible recommendation.
Random IPs are unlikely to be port scanned but any reasonably popular site will.
1. firewall - only allow SSH connections from trusted static IPs
2. Use SSH keys then disable password logins. Lots of guides online to create keys, so I'll just cover the 2nd point: as root or sudo, edit /etc/ssh/sshd_config
(edit: make sure the SSH keys have passphrases. That way you have an extra bit of security in case any workstations get compromised)3. Disable root access. edit /etc/ssh/sshd_config
4. Limit SSH access to specific user accounts. This prevents users creating their own key (in the case of mountable home directories) or other machine accounts with passwords (if you've not done #2): 5. Install auto-firewalling for failed SSH logins. I personally favour fail2ban as that covers other scenarios too, but I've also used denyhosts and that's worked well for SSH.6. Lastly, and by far the best option, don't enable SSH on any internet facing IPs.
If you need SFTP enabled, then let me know and I'll post some details on how to harden SFTP so attackers cannot gain an SSH shell.