Category Archives: Servers

Virtuozzo – Not able to start/stop a mounted VPS ?

There are many situations in which you may find a VPS in its mounted state. Usually a # vzctl start VEID, would attempt to start the container back. But there are situations in which this task would be hung in the system memory.

You may get something an error similar to this when attempting to start/stop the VE :

# vzctl stop VEID

Cannot lock container.

If you encounter this issue, check for the VE lock file, you can view the file and note the process which is being run :

# cat /vz/lock/VEID.lck

You can find a process id and the task name from the above command.  You can check what the process is by initiating :

# ps ax | grep PID

If this is some hung process waiting infinitely, try to kill the process, remove the lock file and attempt to start/stop the VE.

If the process is related to quota calculation, run the following command :

# veid=VE_ID; vzctl stop $veid; vzctl quotaoff $veid; vzctl quotainit $veid; vzctl start $veid; vzctl enter $veid

( just specify the VEID in the field VE_ID and run the above )

– Still, if you face this error, do the following on the mounted VE :

# vzctl --skiplock umount VEID

Now once the VE is unmounted, restart it.

 

Finding .lesshts/kthread processes? – Shellshock bug !

Are you seeing weird kthread processes being executed from user’s home directory ?

For example, if you are on cPanel, are you seeing something like the file /home/user/.lesshts/kthread running and consuming the server resources ?

This looks suspicious as a kthread running with the home directory for a long time with the CPU load shooting up !

This is due to an exploitation of the ShellShock vulnerability and you should consider patching your server’s against this bug.

You check if your server is vulnerable to ShellShock by initiating the following command via SSH :

# env x='() { :;}; echo vulnerable' bash -c 'echo hello'

If you see the o/p as vulnerable hello, you would need to patch it.

On latest versions of CentOS/Redhat/Fedora, you might try with

# yum upgrade bash , to update the bash.

If you are running an older version of these, you would want to manually download the RPM and upgrade.

For CentOS/RHEL 4, you might use this RPM which i have attached here and then initiate a

# rpm -Uvh bash-3.0-27.0.3.el4.i386.rpm

Get relived from the Shell-Shock !

The SSLv3 vulnerability – What do I need to do ? !

It looks like opensource is constantly being hit with vulnerabilities these days ! but yea, as a wise man once said, the more people use and learn on a stuff, the more loopholes you get to find and fix.

So the recent vulnerability is with the SSLv3 protocol which has been tagged as a secure protocol for establishing secure communication between the client and the server until now.

You can check if your services are bound to this vulnerability by checking using this online server tester at :

https://access.redhat.com/labs/poodle/

( you may need a redhat login to get through )

Or you can check using the following one-liner from a shell :

# openssl s_client -connect 'ServerIP or hostname':<'port'> -ssl3

eg,

# openssl s_client -connect xx.xx.xx.xx:443 -ssl3

The above command when initiated should result something like this if its not vulnerable :

=========

CONNECTED(00000003)

xxxxxxxxxx :error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:s3_pkt.c:1256:SSL alert number 40

xxxxxxxxxxx :error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:596:

=========

If your service bound on the port is vulnerable to SSLv3 vulnerability, you would see a SSL handshake being established.

You would need to individually disable SSL v3 for each of the services.

To disable SSLv3 for httpd follow the steps given below :

Open your SSL directive file, ( if configured ), ie, the file /etc/httpd/conf.d/ssl.conf or the top-level configuration file, or inside the default virtual host configuration for an address and specify the following :

SSLProtocol All -SSLv2 -SSLv3

The above SSLProtocol directive disables SSLv2 and SSLv3

If you use a WHM/cPanel server, this can be done from WHM as follows :

WHM » Service Configuration » Apache Configuration » Include Editor » Pre Main Include

SSLProtocol all -SSLv2 -SSLv3
SSLHonorCipherOrder on

Save the above lines and restart the service.

For the moment, patches are still being released for each of the services.  As of now, consider patching your httpd service as the first step and then move on to other services once fixes are available.

Also, there a lot of suggestions in the forums to disable the SSL ciphers for SSLv3 in cPanel configuration so that all the services would get disabled in using SSLv3. However, if you are on centos 5, the base SSL version would be 0.98.e and there is no other ciphers included in it, ie, there are no TLS protocols along with it , which would mean if you change your cPanel to disable SSLv3, you wont be able to access anything over the browser.

You can change /var/cpanel/conf/cpsrvd/ssl_socket_args and give the following to disable SSLv3 :

ALL:!ADH:+HIGH:+MEDIUM:-LOW:-SSLv2:-SSLv3:-EXP

As i said, if you are on openssl 0.9.8.e, giving the following would break everything and you would need to give back what was originally in the file /var/cpanel/conf/cpsrvd/ssl_socket_args, ie,

ALL:!ADH:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP

You can check your current openssl version and the available ciphers in your installation using the following commands :

# openssl version -a
# openssl ciphers -v

If you need to upgrade your Openssl to a latest version, check this post here.

So now, as a server admin you should disable SSLv3 ( first and foremost for your httpd service ) for the security of your users.

As a user, you should disable SSLv3 in your browser now to secure yourself when visiting websites which still  support SSLv3.

Upgrading openssl – for Centos !

With the recent openSSL vulnerabilities,  upgrading openssl to a more latest and stable version, openssl-1.0.1g can be done using the following easy steps. Applicable for CentOS server’s !

# wget ftp://ftp.openssl.org/source/openssl-1.0.1g.tar.gz
# tar -zxf openssl-1.0.1g.tar.gz
# cd openssl-1.0.1g
# ./config --prefix=/usr/local
# make
# make test
# make install

Check the current version using the command :

# openssl version -a

 

DDoS protection guide – Part 1

What is a DDoS attack ?

DDoS, short for Distributed Denial of Service, is a type of DOS attack where multiple compromised systems — which are usually infected with a Trojan (70% of the time)– are used to target a single system causing a Denial of Service (DoS) attack. Victims of a DDoS attack consist of both the end targeted system and all systems maliciously used and controlled by the hacker in the distributed attack.

In a DDoS attack, the incoming traffic flooding the victim originates from many different sources – potentially hundreds of thousands or more. This effectively makes it impossible to stop the attack simply by blocking a single IP address.

Honestly, it would be so difficult to protect against a DDoS attack. But we can follow some steps to make our servers more watchful against them.

=====================================

CSF firewall can be fine tuned as follows :

If you have a cPanel server, navigate to WHM as : ConfigServer Security & Firewall from WHM >> Firewall Configuration. If you do not use WHM, it would be good if you install CSF and manually plug in the following setting :

Connection Tracking : This option enables tracking of all connections from IP addresses to the server. If the total number of connections is greater than this value then the offending IP address is blocked. This can be used to help prevent some types of DOS attack.

=====================================

If you see your server is a bit on the slower side, check the number of connections to it using the following command.

# netstat -anp |grep 'tcp|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

As a preliminary step, block the IPs which doesn look valid and are offending ones using csf commands.

Another option is to go for the the MOD_EVASIVE module in the httpd configuration.

Mod_evasive is an evasive maneuvers module for Apache to provide evasive action in the event of an HTTP DoS or DDoS attack or brute force attack. It is also designed to be a detection tool and can be easily configured to talk to ipchains, firewalls.

Mod_evasive have got many many options to gun down our requirements to handle the IPs connecting to our server.

Steps to install mod_evasive is given below :

# cd /usr/local/src/

# Download the mod_evasive_xx.xx.tar.gz file

# tar -xvzf mod_evasive_xx.xx.tar.gz

# cd mod_evasive/

# /usr/local/apache/bin/apxs -cia mod_evasive20.c

Now create a file named /usr/local/apache/conf/mod_evasive.conf and add your custom settings.

For eg :

# cat /usr/local/apache/conf/mod_evasive.conf

LoadModule evasive20_module modules/mod_evasive20.so
<IfModule mod_evasive20.c>

DOSHashTableSize 3097

//The hash table size defines the number of top-level nodes for each child’s hash table. Increasing this number will provide faster performance by decreasing the number of iterations required to get to the record, but consume more memory for table space. You should increase this if you have a busy web server. The value you specify will automatically be tiered up to the next prime number in the primes list

DOSPageCount 2

//This is the threshhold for the number of requests for the same page (or URI) per page interval. Once the threshhold for that interval has been exceeded, the IP address of the client will be added to the blocking list

DOSSiteCount 50

//This is the threshhold for the total number of requests for any object by the same client on the same listener per site interval. Once the threshhold for that interval has been exceeded, the IP address of the client will be added to the blocking list

DOSPageInterval 1

//The interval for the page count threshhold; defaults to 1 second intervals

DOSSiteInterval 1

//The interval for the site count threshhold; defaults to 1 second intervals

DOSBlockingPeriod 10

//The blocking period is the amount of time (in seconds) that a client will be blocked for if they are added to the blocking list. During this time, all subsequent requests from the client will result in a 403 (Forbidden) and the timer being reset (e.g. another 10 seconds). Since the timer is reset for every subsequent request, it is not necessary to have a long blocking period; in the event of a DoS attack, this timer will keep getting reset

</IfModule>

Now include the above file inside :

/usr/local/apache/conf/includes/pre_main_global.conf

Include “/usr/local/apache/conf/mod_evasive.conf“

Now rebuild httpd.conf :

# /scripts/rebuildhttpdconf

Now restart apache :

# /scripts/restartsrv httpd

Click here to read more on DDoS and its protection !

Exim – Dropping SMTP connection at HELO/EHLO !

We are observing a brute-force attack towards SMTP connections from different IP addresses with the same machine name – “ylmf-pc“

It could be many malware affected machines involved or an extended IP spoofing.

If you have CSF configured properly, the IPs would be blocked at the firewall level.

Another solution is to drop the SMTP connection at HELO so that no further processing is carried out and no packet states of different IPs are examined. If CSF was to block these IPs, it could be a very large list and it could affect the performance of the server.

Add the following to EXIM ACL configuration file.

# vi /etc/exim.conf

acl_smtp_helo = acl_smtp_helo

#BEGIN ACL_SMTP_HELO_BLOCK
 drop
 condition = ${if eq {$sender_helo_name}{ylmf-pc} {yes}{no}}
 log_message = HELO/EHLO - ylmf-pc blocked
 message = I Nailed You at HELO
 accept

#END ACL_SMTP_HELO_BLOCK

Restart exim once this is done.

# service exim restart

This would make sure that the connections from these ylmf-pc ‘s are dropped before further processing !

Update : If you want to block connections from other domains too, give the following piece of code in exim.conf instead of the above :

drop
   condition = ${lookup{$sender_helo_name}lsearch{/etc/heloblocks}{yes}{no}}
   log_message = HELO/EHLO - HELO on heloblocks Blocklist
   message = HELO on heloblocks Blocklist
accept

Once the above config is given, create a new file ‘/etc/heloblocks’ and give in the domain name one by one.

Dont forget to restart exim once this is done.

Click here to read more about DDoS protection !!

 


Complete installation of Virtualmin – CentOS 6.x

Once your new server is provisioned , install Perl in it which stands as the base for all further installations. You can use the following command to install Perl :

# yum install perl -y

When this is complete, download the script to install Virtualmin :

wget http://software.virtualmin.com/gpl/scripts/install.sh

Run the above script :

sh install.sh

Once the installation is complete, you will be able to login to the Virtualmin control panel using the address :

https://your-IP:10000

The username would be ‘root’ and the password would be your root password for the server.

As soon as you login, complete the Post-Installation Wizard, by specifying your requirements.

You will be asked to enter the name-server configuration too.

Enter the ns1/ns2 of the domain you intend to make as the name-server. If the domain is not yet registered, you would have to check the button to skip resolvability check.

Once the post-installation wizrd is complete with, you might observe this error –

Virtualmin’s configuration has not been checked since it was last updated. Click the button below to verify it now.

re-configure

Click the button listed there to find if there are any errors.

Once you run the re-check, do you face this error ?

The mailman queue processor /usr/lib/mailman/bin/qrunner is not running on your system. It can be started in the Bootup and Shutdown module.

To fix this, navigate to : Webmin > Servers > Virtualmin Mailman Mailing Lists

You will see: Warning – Mailman will not operate properly unless a list named mailman has been created. Use the form below to create it now.

mail-man-error

Add any email id and a password to configure it.  Or if you do not intend to use mailman , you can disable it which would fix the original issue.

— System Settings -> Features and Plugins,  – you can disable the Mailman feature.

Once this is complete, re-run the check and you can confirm that your Virtualmin is about to be used.

Create a new domain, here you refer a new domain as a new virtual server.

You can find this from Virtualmin. Find the image given below :

jo2

Specify the domain name, the administration password. You can also configure the IP on which your domain should reside on, by selecting the option from ‘Network interface ‘ > IP address and forwarding.

Once the domain is created, you can edit the DNS records from Virtualmin.

 

jo1

 

If this is the domain which you intend to be the name-server, add the respective NS records and the A records for the NS records.

Update this values at your registrar end to make sure the DNS records are synced-in.

– You can carry on adding new domains ( new virtual server ).  Ensure that proper DNS records are added.

This is all about the installation/configuration of Virtualmin.

 

Error – /bin/rm: Argument list too long !

When trying to delete the files of a folder ( using # rm ) with lots of contents in it , you might get this error :

/bin/rm : Argument list too long

The traditional # rm command will not be able to delete too many files in a directory.

To get around this, use the command given below. Please note that the below command will delete all the files in the current directory in which you are logged into.

# find . -name '*' | xargs rm

( Again, make sure your pwd is the directory which you want to delete the files from, else, i have no words to describe it 😀 )

 

Facing the error – ip_conntrack full & packets dropped !

Facing an issue with the kernel module, ‘ip_conntrack‘ ?

Checking /var/log/messages gives something like this ?

host kernel: ip_conntrack: VPS xxx table full, dropping packet.
host kernel: ip_conntrack: VPS xxx table full, dropping packet.

If you run an iptables firewall and have rules that act upon the state of a packet, then the kernel uses ‘ip_conntrack’ to keep track of what state what connections are in so that the firewall rule logic can be applied against them.

If you have a system that’s getting a lot of network activity then the table will accumulate entries.

Increase ip_conntrack to a higher value by editing /etc/sysctl.conf

Add/edit this line,

net.ipv4.ip_conntrack_max=xxxx

Run # sysctl -p after making the changes.

Check the current value using the command,

# sysctl net.ipv4.netfilter.ip_conntrack_max

Do not keep on increasing the above value beyond a limit, if you still see the error after the increase.

This error might indicate the start of something more destructive attack on your server’s network, something like a DDoS attack. The amount of packets sent/received during this period would be on the higher side and as a result the kernel module is not able to process them all, which results in the above error.

So check for the server traffic using commands like iftop or tcpdump and isolate if the issue is related to any attacks.