MySQL server not starting – Part 4 !

When trying to restart the MySQL service, do you happen to find the following error in the logs ?

/usr/sbin/mysqld: Can’t create/write to file ‘/tmp/xx’ (Errcode: 122)
date InnoDB: Error: unable to create temporary file; errno: 122
date [ERROR] Can’t init databases
date [ERROR] Aborting

Under usual circumstances, this error can come up, when the permission of /tmp is inappropriate. The right one should be :

drwxrwxrwt  4 root root size date /tmp

( ie, it should be equivalent to # chmod 1777 /tmp )

You make sure this is the right one set for /tmp, you ensure that disk space aint full, still do you face this issue ?

– If so, check for the number of files and nature of them in /tmp. Delete unnecessary/unwanted hidden/temporary files and restart the service. This should fix it.

 

Tracking down spamming in Plesk – 2

This post will help you to track down spamming from a Plesk server, if its employed using PHP scripts in any of the domains.

– Use this one to view the folders which have mail PHP scripts enabled and running.

# vi /var/qmail/bin/sendmail-wrapper

#!/bin/sh
(echo X-Additional-Header: $PWD ;cat) | tee -a /var/tmp/mail.send|/var/qmail/bin/sendmail-qmail "$@"

# touch /var/tmp/mail.send
# chmod a+rw /var/tmp/mail.send
# chmod a+x /var/qmail/bin/sendmail-wrapper
# mv /var/qmail/bin/sendmail /var/qmail/bin/sendmail-qmail
# ln -s /var/qmail/bin/sendmail-wrapper /var/qmail/bin/sendmail

– Wait for at some time and then change sendmail back:

# rm -f /var/qmail/bin/sendmail
# mv /var/qmail/bin/sendmail-qmail /var/qmail/bin/sendmail

Once this is completed, run the following command, which will show you all the folders from where mail PHP scripts were run :

# grep X-Additional /var/tmp/mail.send | grep `cat /etc/psa/psa.conf | grep HTTPD_VHOSTS_D | sed -e 's/HTTPD_VHOSTS_D//' `

– If your PHP version is greater than 5.3, you can also consider about enabling extended logging which will help to add a header to all outgoing email and that will help you to track the location of the script which is involved in spamming.

Add the following line to your php.ini file :

mail.add_x_header = On

– Check out the headers ( check this post know about finding the headers from the queue ) and spot the script involved.

 

Tracking down spamming in Plesk – 1

Finding the source of spamming in a server provisioned with Plesk is a tough job.

Some of the useful commands which might help you are given down.

  • Find the number of mails hung in the queue :

# /var/qmail/bin/qmail-qstat

  • To get an idea about the the message headers of mails in queue :

# /var/qmail/bin/qmail-qread

The above one shows the senders and recipients of messages. Now try to find this message in the queue by its ID

# find /var/qmail/queue/mess/ -name XXXXXX ( <- Message ID )

cat the o/p file of the above command and inspect the message headers closely.

Examine the message and find the line “Received” to find out from where it was sent for the first time.

For example, if you find:

1-> Received: (qmail 19514 invoked by uid xxxx ); 13 Sep 2005 17:48:22 +0700

It means that this message was sent via a CGI by user with UID xxxx . Using this UID, it is possible to find the domain:

# grep xxxx /etc/passwd

2-> Received: (qmail 19622 invoked from network); date/time
Received: from external_domain.com (xx.xx.xx.xx)

It means that the message has been accepted and delivered via SMTP, and that the sender is an authorized mail user. This might mean that the password of the email account has been compromised.

You can use the following command to find the users which have attempted to login via authentication. If you find lots of authentication attempts to a particular user/from a particular IP, then it might be the vulnerability present in your server.

# cat /usr/local/psa/var/log/maillog |grep -I smtp_auth |grep -I user

Also, when you check the headers of the mail in the queue, if you find that the mails are received from a particular IP address, like :

Received: (qmail 10728 invoked from network); date

Received: from unknown (HELO User) (xx.xx.xx.xx)

by domain.com with SMTP ; date

We can use the tool tcpdump to find out what is being communicated over the network from/to the IP in question :

# tcpdump -i venet0:0 -n src xx.xx.xx.xx \or dst xx.xx.xx.xx -s 2048 -w /home/wiresharklog.pcap

– Replace  venet0:0 with your appropriate interface

– Replace xx.xx.xx.xx with the IP in question.

You will obtain the logs in /home/wiresharklog.pcap. Open this pcap file using wireshark ( or any related softwares ) and have a glance through the ‘Statistics -> Flow graph’ . Check this if you can spot the connections/packets being sent over.

3-> If the “Received” line contains a UID of the user “apache” (for example, invoked by UID 48), it means that spam was sent through a PHP script. Find this post useful for dealing with this.

There is another case of spamming which has been noticed.

– When checking the qmail maillogs (usr/local/psa/var/log/maillog) :

date  xxxx smtp_auth: SMTP connect from (null)@(null) [xx.xx.xx.xx]
date  xxxx smtp_auth: smtp_auth: SMTP user xxxxx: logged in from (null)@(null) [xx.xx.xx.xx]

We can see that spamming is being done by brute forcing Plesk
email passwords and then authenticating using base 64 encoding on the username.

The built in qmail logging cannot handle this encoding and as a result the logs will just show (null) instead of the username used. This is applicable for servers running on older versions of Plesk.

The solution would be to upgrade Plesk to a more stable version.

Note : You can also check if there are any email accounts within your hosting environments which uses the mail name ‘test’. Around 90% of the accounts created as test are employed with weak passwords which make it easier for hackers to brute-force attack them.

Use this query to find if any such ones are there :

# mysql -uadmin -p`cat /etc/psa/.psa.shadow` psa

# SELECT m.mail_name, d.name FROM mail AS m LEFT JOIN (domains AS d, accounts AS a) ON (m.dom_id = d.id AND m.account_id = a.id) WHERE m.mail_name='test' ;

Hope this was helpful 🙂

Databases missing from cPanel after MySQL upgrade !

A weird situation is when you can see the databases, its tables and so on from phpmyadmin of the concerned cPanel account or even from the backend, but is actually found missing under the ‘Databases’ section.

This issue occurs due to the lack of mapping of databases. Use these commands to re-create the mapping :

# /usr/local/cpanel/bin/setupdbmap : To recreate the dbusermap

# /scripts/update_db_cache : To recreate DB cache

Help with cPHulk Brute Force Protection !

Brute force is an attack that involves using an automated system to guess the password to your web server or services. cPHulk provides protection against brute force attacks.

Some useful commands to deal with cPHulk Brute Force Protection from the back-end :

First get access to the db cphulkd via mysql

#  mysql -u root -p

> use cphulkd

You can view the list of IPs which are blocked as per the brutes table :

> select * from brutes;

To select the IP’s alone and not any other data, use this :

> select IP, BRUTETIME from brutes order by BRUTETIME;

To delete the entire list of IPs in brutes :

> DELETE FROM brutes;

If you brutes lists is too large and you need to find if any particular IP is in the list, you can use this :

> SELECT * FROM `brutes` WHERE `IP`='x.x.x.x';

To delete that particular IP,

> DELETE FROM `brutes` WHERE `IP`='x.x.x.x';

To whitelist a particular IP,

# /scripts/cphulkdwhitelist IP

To disable/enable cPHulk :

# /usr/local/cpanel/bin/cphulk_pam_ctl --enable ( enable the service )
# /usr/local/cpanel/bin/cphulk_pam_ctl --disable ( disable the service )

You can also check the cphulk logs from

# /usr/local/cpanel/logs/cphulkd.log

 

Clearing huge eximstats db !

The eximstats database might tend to grow in size if there is high amount of mailing from your server.

Check if the following value from WHM is set to a higher interval :

Home »Server Configuration »Tweak Settings

>> “The interval, in days, to retain Exim stats in the database”

You might need to reduce the time interval to retain the eximstats.

Under usual situation, eximstats will grow huge in size when there is spamming carried out from your server. First check if your server is involved in spamming and if so, find the source of spamming and eradicate it.

You can remove the eximstats mysql db as follows :

# mysql -u root -p
# use eximstats
# delete from sends;
# delete from smtp;
# delete from failures;
# delete from defers;

If the above tends to consume lots of time, you can use the below commands to clear eximstats :

# mysqladmin drop eximstats

# mysqladmin create eximstats

# mysql eximstats < /usr/local/cpanel/etc/eximstats_db.sql

 

Troubleshooting issues with named !

When you have a server without any control panel, it becomes a pain in the arse to manage the services, especially the DNS service.

You have got the tedious job of installing bind, altering the configuration file, manually creating the zone files and so on.

Here are some tips which might help you :

— Navigate to your named.conf file and first make sure that your named service is set to listen to interfaces other than localhost alone.

The default setting which would come along in named.conf would be the following :

# listen-on port 53 { 127.0.0.1; };

# allow-query  { 127.0.0.1; };

Change the above lines to :

# listen-on port 53 { any; };

# allow-query  { any; };

If the service is allowed to bind to 127.0.0.1, external queries would not be replied back. Attempting to connect to port 53 from external hosts using tools such as telnet would result in a ‘connection refused’

You will only be able to resolve the zones internally. For eg,

dig @localhost domain.com would only work with named pinned to 127.0.0.1 alone

— To check if there are any errors associated with the named.conf file, run the following

# named-checkconf /path-to-named.conf

– Once the zones are created, you can check if they posses any configuration errors :

# named-checkzone IP /path-to-zone-file-to-be-checked

You may avail online zone file creators if you are not familiar with the syntax. Click here to create a zone file of your requirement.

Note on serial numbers :

When adding the serial number in a zone file, you must use the format — YYYYMMDDNN

NN stands for increment number starting from 01, whenever you edit the zone file increase its value by +1

 

Complete installation of Virtualmin – CentOS 6.x

Once your new server is provisioned , install Perl in it which stands as the base for all further installations. You can use the following command to install Perl :

# yum install perl -y

When this is complete, download the script to install Virtualmin :

wget http://software.virtualmin.com/gpl/scripts/install.sh

Run the above script :

sh install.sh

Once the installation is complete, you will be able to login to the Virtualmin control panel using the address :

https://your-IP:10000

The username would be ‘root’ and the password would be your root password for the server.

As soon as you login, complete the Post-Installation Wizard, by specifying your requirements.

You will be asked to enter the name-server configuration too.

Enter the ns1/ns2 of the domain you intend to make as the name-server. If the domain is not yet registered, you would have to check the button to skip resolvability check.

Once the post-installation wizrd is complete with, you might observe this error –

Virtualmin’s configuration has not been checked since it was last updated. Click the button below to verify it now.

re-configure

Click the button listed there to find if there are any errors.

Once you run the re-check, do you face this error ?

The mailman queue processor /usr/lib/mailman/bin/qrunner is not running on your system. It can be started in the Bootup and Shutdown module.

To fix this, navigate to : Webmin > Servers > Virtualmin Mailman Mailing Lists

You will see: Warning – Mailman will not operate properly unless a list named mailman has been created. Use the form below to create it now.

mail-man-error

Add any email id and a password to configure it.  Or if you do not intend to use mailman , you can disable it which would fix the original issue.

— System Settings -> Features and Plugins,  – you can disable the Mailman feature.

Once this is complete, re-run the check and you can confirm that your Virtualmin is about to be used.

Create a new domain, here you refer a new domain as a new virtual server.

You can find this from Virtualmin. Find the image given below :

jo2

Specify the domain name, the administration password. You can also configure the IP on which your domain should reside on, by selecting the option from ‘Network interface ‘ > IP address and forwarding.

Once the domain is created, you can edit the DNS records from Virtualmin.

 

jo1

 

If this is the domain which you intend to be the name-server, add the respective NS records and the A records for the NS records.

Update this values at your registrar end to make sure the DNS records are synced-in.

– You can carry on adding new domains ( new virtual server ).  Ensure that proper DNS records are added.

This is all about the installation/configuration of Virtualmin.

 

Error – /bin/rm: Argument list too long !

When trying to delete the files of a folder ( using # rm ) with lots of contents in it , you might get this error :

/bin/rm : Argument list too long

The traditional # rm command will not be able to delete too many files in a directory.

To get around this, use the command given below. Please note that the below command will delete all the files in the current directory in which you are logged into.

# find . -name '*' | xargs rm

( Again, make sure your pwd is the directory which you want to delete the files from, else, i have no words to describe it 😀 )

 

MySQL server not starting – Part 3

A common issue when MySQL versions are upgraded is the presence of depreciated commands in /etc/my.cnf that would prevent MySQL from restarting.  Other than the presence of depreciated ones, incomplete commands too can cause the issues.

When MySQL is upgraded to 5.5 from 5.1 or so, you might see that the service doesnt get restarted. Looking at the configuration file at /etc/my.cnf do you find something like this ?

slow_query_log_file

Here the path is not referenced, which is the issue confronted. Usually if slow_query_log is enabled in the configuration, then the path to log file should be mentioned, like :

slow_query_log_file=/path-to-file

Either comment off the incorrect line or if you are depending on slow_query_log mention the path to the log file as shown above. Once this is given, your service would restart fine.