d2's lair

From time to time I see some scanning on my website. This usualy dies after 5 or so minutes, but it’s pretty annoying and space-consuming so I thought recently that it would be nice to get rid of those scans.

An example – let say – attack looks like:

** IP ** - - [02/Jun/2010:14:35:59 +0200] "HEAD /index.html HTTP/1.1" 404 - "-" "-"
** IP ** - - [02/Jun/2010:14:35:59 +0200] "HEAD /index.shtml HTTP/1.1" 404 - "-" "-"
** IP ** - - [02/Jun/2010:14:35:59 +0200] "HEAD /index.aspx HTTP/1.1" 404 - "-" "-"
** IP ** - - [02/Jun/2010:14:36:00 +0200] "HEAD /index.cgi HTTP/1.1" 404 - "-" "-"
** IP ** - - [02/Jun/2010:14:36:00 +0200] "HEAD /index.php3 HTTP/1.1" 404 - "-" "-"
** IP ** - - [02/Jun/2010:14:36:00 +0200] "HEAD /index HTTP/1.1" 404 - "-" "-"
** IP ** - - [02/Jun/2010:14:36:00 +0200] "HEAD //cgi.cgi// HTTP/1.1" 404 - "-" "-"
** IP ** - - [02/Jun/2010:14:36:01 +0200] "HEAD //webcgi// HTTP/1.1" 404 - "-" "-"
** IP ** - - [02/Jun/2010:14:36:01 +0200] "HEAD //cgi-914// HTTP/1.1" 404 - "-" "-"
** IP ** - - [02/Jun/2010:14:36:01 +0200] "HEAD //cgi-915// HTTP/1.1" 404 - "-" "-"
** IP ** - - [02/Jun/2010:14:36:01 +0200] "HEAD //bin// HTTP/1.1" 404 - "-" "-"
** IP ** - - [02/Jun/2010:14:36:01 +0200] "HEAD //cgi// HTTP/1.1" 404 - "-" "-"
** IP ** - - [02/Jun/2010:14:36:02 +0200] "HEAD //mpcgi// HTTP/1.1" 404 - "-" "-"
** IP ** - - [02/Jun/2010:14:36:02 +0200] "HEAD //cgi-bin// HTTP/1.1" 404 - "-" "-"
** IP ** - - [02/Jun/2010:14:36:02 +0200] "HEAD //ows-bin// HTTP/1.1" 404 - "-" "-"
** IP ** - - [02/Jun/2010:14:36:02 +0200] "HEAD //cgi-sys// HTTP/1.1" 404 - "-" "-"

Sad, isn’t it? This can be done by human, but most probably it’s scanner. The one I know is called “Indir”, written (and still developed?) by WillBe (at least this is what I get from README.TXT file). Well, first of all we have to change Apache’s default ErrorDocument for certain error codes, so open up apache2.conf and paste this:

 ErrorDocument 401 "/error/index.html"
 ErrorDocument 403 "http://website.com/error/index.html"
 ErrorDocument 404 "http://website.com/error/index.html"
 ErrorDocument 405 "http://website.com/error/index.html"
 ErrorDocument 413 "http://website.com/error/index.html"
 ErrorDocument 500 "http://website.com/error/index.html"
 ErrorDocument 414 "http://website.com/error/index.html"

This can be also done inside .htaccess file. Just paste above code and it should be OK. Code 401 can’t be replaced by full URL (explained here: http://httpd.apache.org/docs/2.2/mod/core.html#errordocument) but all the rest can. Now what will happen when user hits non-existing content? He will get a custom index.html page from error directory, sure, but before he will, Apache will perform REDIRECT on such request and will serve 302 to client. Having relative path (/error/index.html) instead of full will cause error code to stay as is.

OK, so now every error will be displayed as 302 and we will recognize it i.e by content of served page. BTW – would be good to have some kind of parser for various codes instead of writting multiple *.html files. Maybe I will post some of my sources later on, but for now, just to illustrate, I will keep *.html.

Is it all we can do about above scans? Well, of course not! Next, I should take care of this “HEAD” method. Unless you will run SVN via http you should consider restricting all methods to GET/POST only. This is also done inside apache2.conf and mod_rewrite:

RewriteEngine on
RewriteCond %{REQUEST_METHOD} !^(GET|POST)
RewriteRule ^(.*)$ - [F,L]

Simple, and easy. Now HEAD won’t work and end user will get 302 (not 403 because we use full URLs in ErrorDocument, remember?). OK, so this will keep Indir from getting it’s dirty hands on our server but if there is a clever script kiddie, he will notice that indir/core/connect.pl uses this to check if server can be scanned:

# line 17

if (!$head_get) {
   $head_get = "HEAD";
 }

Why? Explanation is simple. Almost all scanners uses this technique to find difference between existing file/directory and the one that doesn’t exist. Here it uses HEAD to performs checks and basing on response codes (200, 404) it decides whether or not to scan the server. We just disable HEAD method by restricting all of them to GET/POST only, but what if one change HEAD to GET? Of course Indir will work again, but we can handle this. Trick is to respond to any non-existing content with 200 code (found). You can use .htaccess in Apache to achieve this:

RewriteEngine On
RewriteBase     /
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*) http://website.com/$1 [R=200]

This will do the trick, but unfortunately you will never know if you have a missing content in your webserver. This can be fixed by adding to RewriteRule custom handler mentioned above and by producing custom logs, like that:

RewriteRule ^(.*) http://website.com/handler.php?type=$1 [R=200]

I think you will figure it out.

Well, that’s it for now. Have fun with testing and share your thoughts in comments.