DDoS protection guide – Part 2

We have discussed about DDoS hitting Webservers in this post. It can also attack the DNS servers, which is often called as DNS Amplification Attacks.

In Simple words, the attack can be explained as follows :

Someone makes an inquiry to you, on how to reach a particular destination. You are not actually sure of the location either, so you ask your friends nearer to you, and if you don’t get an answer from them, you are determined to somehow get an answer and you start inquiring further until you get one. ( Basically you do not know this ‘someone’ who requested your help)

And this ‘someone’ has not stopped there. He has asked this same question to lots many other people whom like you are determined to get an answer. He would conclude by saying, if you get an answer, please ring me to 111 – a fake number of some unknown poor guy.

Similarly, an attacker spoofs IP addresses ( he might spoof it to an IP to which he would like to carry a DDoS attack – called as the target – like the fake 111 number ) and sends a request to your DNS server asking to resolve a domain. Your DNS server would not have any details about it in your local db’s. So it goes around the internet trying to resolve the domain and as a result the request-queries and the reply-queries increase beyond a limit as the attacker sends more and more request queries.

Now, remember your server might be 1 in 10000 out of which the attacker would direct the reply’s to a target. ( If source IP of the DNS query was spoofed to that of the target’s IP )

So basically, this sort of DDoS attacks, not only affects the ‘target’ but also all the DNS server’s participating in this attack, as they are flooded with queries ( request and reply )

Attackers will typically submit a request for as much zone information as possible to maximize the amplification effect. In most attacks of this type, the spoofed queries sent by the attacker are of the type, “ANY,” which returns all known information about a DNS zone in a single request. Because the size of the response is considerably larger than the request, the attacker is able to increase the amount of traffic being generated by these DNS services and in the end- the amount of traffic directed at the target would be huge.

So, how can we prevent this from happening ?

Going back to our previous illustration, when that ‘someone’ asked you for a help, its you who sought to find an answer. You could have said :

“Im sorry, I dont know the route to that destination. Neither do i know you, so i cant spend my time/energy in assisting you.

This is where you can make your DNS server a closed resolver.

More on this is found in this post

And suppose, consider this, your DNS server is closed, still it would receive the queries from the attacker and your server would have to reply to those DNS queries. Just that it is not a part of the attack. These replies too might hinder your services if too much requests are being directed to your server.

Here you can use iptables to set a rate-limit on the queries reaching your DNS port.

First make sure the recent module is loaded in the server
This module is needed to get this particular aspect of iptables working.

First rule is set to move all the packets received in port 53 to a new chain

# iptables -N blocklist ( create a new chain )
# iptables -A INPUT -p udp --dport 53 -j blocklist

Then,

# iptables -A blocklist -m recent --set --name DNSQF --rsource ( creating a db DNSQF to capture the packets )

# iptables -A blocklist -m recent --update --seconds 5 --hitcount 15 --name DNSQF --rsource -j DROP

( set the rule for the db DNSQF which stores recent IPs )

The above rule implies to drop every packets after the 15th one, in a time-frame of 5 seconds.

Availing these rules in iptables, can in way help to reduce the traffic in your server, when DNS queries are made to your server, even when it is a closed resolver.

DDoS protection guide – Part 1

What is a DDoS attack ?

DDoS, short for Distributed Denial of Service, is a type of DOS attack where multiple compromised systems — which are usually infected with a Trojan (70% of the time)– are used to target a single system causing a Denial of Service (DoS) attack. Victims of a DDoS attack consist of both the end targeted system and all systems maliciously used and controlled by the hacker in the distributed attack.

In a DDoS attack, the incoming traffic flooding the victim originates from many different sources – potentially hundreds of thousands or more. This effectively makes it impossible to stop the attack simply by blocking a single IP address.

Honestly, it would be so difficult to protect against a DDoS attack. But we can follow some steps to make our servers more watchful against them.

=====================================

CSF firewall can be fine tuned as follows :

If you have a cPanel server, navigate to WHM as : ConfigServer Security & Firewall from WHM >> Firewall Configuration. If you do not use WHM, it would be good if you install CSF and manually plug in the following setting :

Connection Tracking : This option enables tracking of all connections from IP addresses to the server. If the total number of connections is greater than this value then the offending IP address is blocked. This can be used to help prevent some types of DOS attack.

=====================================

If you see your server is a bit on the slower side, check the number of connections to it using the following command.

# netstat -anp |grep 'tcp|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

As a preliminary step, block the IPs which doesn look valid and are offending ones using csf commands.

Another option is to go for the the MOD_EVASIVE module in the httpd configuration.

Mod_evasive is an evasive maneuvers module for Apache to provide evasive action in the event of an HTTP DoS or DDoS attack or brute force attack. It is also designed to be a detection tool and can be easily configured to talk to ipchains, firewalls.

Mod_evasive have got many many options to gun down our requirements to handle the IPs connecting to our server.

Steps to install mod_evasive is given below :

# cd /usr/local/src/

# Download the mod_evasive_xx.xx.tar.gz file

# tar -xvzf mod_evasive_xx.xx.tar.gz

# cd mod_evasive/

# /usr/local/apache/bin/apxs -cia mod_evasive20.c

Now create a file named /usr/local/apache/conf/mod_evasive.conf and add your custom settings.

For eg :

# cat /usr/local/apache/conf/mod_evasive.conf

LoadModule evasive20_module modules/mod_evasive20.so
<IfModule mod_evasive20.c>

DOSHashTableSize 3097

//The hash table size defines the number of top-level nodes for each child’s hash table. Increasing this number will provide faster performance by decreasing the number of iterations required to get to the record, but consume more memory for table space. You should increase this if you have a busy web server. The value you specify will automatically be tiered up to the next prime number in the primes list

DOSPageCount 2

//This is the threshhold for the number of requests for the same page (or URI) per page interval. Once the threshhold for that interval has been exceeded, the IP address of the client will be added to the blocking list

DOSSiteCount 50

//This is the threshhold for the total number of requests for any object by the same client on the same listener per site interval. Once the threshhold for that interval has been exceeded, the IP address of the client will be added to the blocking list

DOSPageInterval 1

//The interval for the page count threshhold; defaults to 1 second intervals

DOSSiteInterval 1

//The interval for the site count threshhold; defaults to 1 second intervals

DOSBlockingPeriod 10

//The blocking period is the amount of time (in seconds) that a client will be blocked for if they are added to the blocking list. During this time, all subsequent requests from the client will result in a 403 (Forbidden) and the timer being reset (e.g. another 10 seconds). Since the timer is reset for every subsequent request, it is not necessary to have a long blocking period; in the event of a DoS attack, this timer will keep getting reset

</IfModule>

Now include the above file inside :

/usr/local/apache/conf/includes/pre_main_global.conf

Include “/usr/local/apache/conf/mod_evasive.conf“

Now rebuild httpd.conf :

# /scripts/rebuildhttpdconf

Now restart apache :

# /scripts/restartsrv httpd

Click here to read more on DDoS and its protection !

Open DNS resolvers and patching them !

Before getting to know what is an open resolver, you need to know what is recursion, ie recursive queries !

Suppose you have a DNS server configured and a local machine which uses your DNS server queries for a website. Imagine this query is a new one and its not in the local cache of the machine which made the request. Once this request reaches your DNS server, it will attempt to find the website in question in its local cache. If it cannot find an answer it will query other DNS servers on your behalf until it finds the address. It will then respond to the original request with the results from each server’s query. This scenario is fine, because the local machine which made the initial request is trusted by you. What if another machine which isn’t trusted by you, queries your DNS server for the same ? Then your DNS is an Open resolver.

An open DNS resolver is a name server that provides a recursive name resolution for non local clients or users. Basically it’s a name server that provides recursive replies for every system on the internet. Local users or “authorized” clients are users on networks that you control and/or that you trust. Recursive replies are the result of following the chain of delegations found in DNS, ending up at the domain name that was requested by the original user.

Open DNS resolvers are frequently being abused to conduct efficient DDoS attacks towards websites, infrastructure and services. In a DNS amplification DDoS attack, the attacker sends a DNS name lookup request to an open DNS resolver with the source address spoofed to be the victim’s address.

When the DNS server sends the DNS record response, it is sent
to the victim (the source address that was used in the spoofed request). Because the size of the response is typically considerably larger than the request, the attacker is able to amplify the volume of traffic directed at the victim. Dont think it would affect just the victim. Essentially this means that your equipment is taking part in a botnet leveraging a DDoS attack towards other systems, potentially causing disruption of services and harm.

If your systems take part in such a DDoS attack then your own network, server and services infrastructure too can easily become congested.

To get around this issue, configure your DNS server to either disable recursion or allow recursion from trusted set of IPs.

For named, recursion can be disabled by adding the following line to your /etc/named.conf file :

options {

recursion no;

};

You can allow recursion from a trusted set of IPs by giving the following

options {

allow-recursion { 127.0.0.1; IP1; IP2; }; //include your server IPs and
your provider’s nameserver IPs and whichever IPs you feel can be trusted.
};

Suppose you have a DNS server and you have configured your named as :

allow-recursion { IP1;IP2; } ;

Try the following from the machine with IP1,

# nslookup google.com x.x.x.x ( x.x.x.x is the DNS server IP )

The result would be :

……………………

(root) nameserver = J.ROOT-SERVERS.NET
(root) nameserver = K.ROOT-SERVERS.NET
(root) nameserver = L.ROOT-SERVERS.NET
(root) nameserver = M.ROOT-SERVERS.NET
(root) nameserver = A.ROOT-SERVERS.NET
(root) nameserver = B.ROOT-SERVERS.NET
(root) nameserver = C.ROOT-SERVERS.NET
(root) nameserver = D.ROOT-SERVERS.NET
(root) nameserver = E.ROOT-SERVERS.NET
(root) nameserver = F.ROOT-SERVERS.NET
(root) nameserver = G.ROOT-SERVERS.NET
(root) nameserver = H.ROOT-SERVERS.NET
(root) nameserver = I.ROOT-SERVERS.NET

……………………

Suppose you made the same query from an IP which is not defined in allow-recursion, then you get the following :

Server: x.x.x.x                                                                                 Address: x.x.x.x#53

** server can’t find google.com: REFUSED

So consider about tweaking your DNS server, if its an Open resolver.

 

cPanel – Blocking spam – in addition to SpamAssassin !

SpamAssassin is just a great tool to block the incoming spams to your server. However, you might just need some other tools in addition to SpamAssassin, if you feel lotta spams are making it past SA.

Two of my suggestions from WHM are :

1. Enable the RBL option from WHM , which will check each of the IPs that is attempting to send mails to your server and if the IP is in the RBL list , it will automatically block the mails.

This option can be found from WHM as follows :

Home »Service Configuration »Exim Configuration Manager>> RBLs>>Manage Custom RBLs

You can enable the in-build lists or add your RBL lists from above.

2. Enable the setting to Reject SPF failures , which will reject emails from domain names without valid SPF records.

You can enable this option from :

WHM » Service Configuration » Exim Configuration Manager>>ACL

These two tweaks are seen as very effective when it comes to reject unwanted spams from hitting your server !

Still getting incoming spam ? Then you might need to think of training SpamAssassin. You can check this post here on more info on that.

 

Exim – Dropping SMTP connection at HELO/EHLO !

We are observing a brute-force attack towards SMTP connections from different IP addresses with the same machine name – “ylmf-pc“

It could be many malware affected machines involved or an extended IP spoofing.

If you have CSF configured properly, the IPs would be blocked at the firewall level.

Another solution is to drop the SMTP connection at HELO so that no further processing is carried out and no packet states of different IPs are examined. If CSF was to block these IPs, it could be a very large list and it could affect the performance of the server.

Add the following to EXIM ACL configuration file.

# vi /etc/exim.conf

acl_smtp_helo = acl_smtp_helo

#BEGIN ACL_SMTP_HELO_BLOCK
 drop
 condition = ${if eq {$sender_helo_name}{ylmf-pc} {yes}{no}}
 log_message = HELO/EHLO - ylmf-pc blocked
 message = I Nailed You at HELO
 accept

#END ACL_SMTP_HELO_BLOCK

Restart exim once this is done.

# service exim restart

This would make sure that the connections from these ylmf-pc ‘s are dropped before further processing !

Update : If you want to block connections from other domains too, give the following piece of code in exim.conf instead of the above :

drop
   condition = ${lookup{$sender_helo_name}lsearch{/etc/heloblocks}{yes}{no}}
   log_message = HELO/EHLO - HELO on heloblocks Blocklist
   message = HELO on heloblocks Blocklist
accept

Once the above config is given, create a new file ‘/etc/heloblocks’ and give in the domain name one by one.

Dont forget to restart exim once this is done.

Click here to read more about DDoS protection !!

 


Issue when trying to upgrade cPanel from an old version !

It would be a difficult job to upgrade cPanel from versions 11.30.x to the latest ones trending now. Under usual circumstances, upgrading cPanel can be as easy as running the script :

# scripts/upcp

However, if your box is running an old version of cPanel, you might need to carry this upgrade incrementally.

For example, if your current cPanel version is 11.30 and if you need to upgrade this,  manually change the “CPANEL=release/current” line within /etc/cpupdate.conf to “CPANEL=11.32” and then run a cPanel update.

Once this is done, edit the line CPANEL to 11.36 and then go ahead with the upgrade using the script # /scripts/upcp

Note : Recently i had to upgrade a cPanel box from 11.1.0 and when trying to follow the above steps, I got the error which stated like , cant fetch the current version from /usr/local/cpanel/version file, correct it and re-run the update. On checking the file, could see that the version was set as “11.4.19-RELEASE_14379”. I went through WHM and could see WHM – 11.1.0 and cPanel as 11.4.19. Manually edited /usr/local/cpanel/version file and corrected it as 11.1.0 and re-ran the upgrade script. ! Hurray, it worked 🙂