The world has gone to dogs in the online arena.
The attacks are increasing and they’re coming fast and furious from Turkey, India, Iran, Romania, Thailand, France, China, Ukraine, Russia, US, Canada and all over the globe.
Both private hackers and government agencies are on the rampage.
You cannot stop all of the attacks but with some effort you can prevent a lot of them before they inflict catastrophic damage on your web site or network.
Log File Analysis
Studying log files on your Linux server is a good way to spot suspicious activity early and stop attacks before they start or get out of hand.
In this post, we will examine different log files that can help to identify malicious or potentially bad actors based mostly on IP addresses.
As always in tackling web security, there’ll be some false positives but that’s the price you pay for security.
Not every visitor from Ukraine, Iran, France, UK or Turkey nor every Amazon Web services IP address is a bad element.
Types of Log Files
Linux systems log everything that happens on the server and store the activity in various files under etc/var/log and etc/var/log/httpd folders.
From a security perspective, the following logs (on a CentOS server) are of interest to us.
1) vsftpd.log – stores all FTP login attempts (including failed attempts) to your server
2) maillog – includes all e-mail sent and received by various users on your server, connections to mail server and postfix/smtpd login failures
3) messages – saves all attempts like relaying, DDOS attacks (mod_evasive attacks,) SMTP attacks, fail2ban actions, invalid smtp saslauthd logins
4) access_log – saved in /log/httpd folder and tells the story of who accessed your server including various bots and their IP addresses
5) error_log – located in /log/httpd folder provides useful information about errors generated on your web server
Now how do we access these various log files.
You can access the log files either through command line or via GUI panels like Webmin.
From the command line you can find out malicious visitors or potential attackers by running netstat or ss commands.
While these are extremely useful, they have some limitations as they will not tell you clearly who did what a few hours back, yesterday or two days back on your web site.
Also, you may have to run netstat command multiple times with different options to figure out the attacking IP.
It is here that analyzing log files directly comes in handy.
Log files can be analyzed in two ways:
1) With the command less
2) Using tailf
The less command lets you examine log files for four or five days.
Whereas tailf allows you to monitor log files live as they are being generated.
To access log files on your Linux system, you have to first log in as root or superuser.
Once you are in the log folder, you can read the log files through less command.
[root@christyserver]# cd /etc/var/log [root@christyserver log]# less vsftpd.log
When the log file opens in command line it starts at the beginning of the file. The document may contain logs for three or four days.
The beginning of the log file contains logs for older dates. This is because when the logs are saved, the newest entries are appended at the end of existing log file entries.
You can use various options to navigate the log file opened through less.
a) Page Up or b – Scroll back one page
b) Page Down or press space bar – scroll forward one page
c) Up arrow – scroll up one line
d) Down arrow – scroll down one line
e) G – Move to the end of the text file
f) 1G or g – move to the beginning of the text file
g) /characters – search forward to the next occurrence of characters
h) n -search for the next occurrence of previous search
i) h – display help screen for less
j) q – quit less
Here’s an example of useful stuff gleaned from the less command:
[root@christyserver httpd]# less access_log 188.8.131.52 - - [15/Dec/2014:18:55:14 -0500] "GET /ghgh/ghg/gh.php HTTP/1.1" 301 329 "-" "-" 184.108.40.206 - - [15/Dec/2014:18:55:14 -0500] "GET /phpMyAdmin/scripts/setup.php HTTP/1.1" 301 342 "-" "-" 220.127.116.11 - - [15/Dec/2014:18:55:15 -0500] "GET /pma/scripts/setup.php HTTP/1.1" 301 335 "-" "-" 18.104.22.168 - - [15/Dec/2014:18:55:15 -0500] "GET /myadmin/scripts/setup.php HTTP/1.1" 301 339 "-" "-"
In the above example, a bad element from Hong Kong is trying to play mischief with my PHP setup. So our next step ought to be to drop this IP and prevent it from accessing the server again.
To monitor log files live (as they are being generated), tailf is the more useful command.
To access log files on your Linux system, you have to first log in as root or superuser and then change directory to etc.
Once you are in etc folder, run the following tailf command
[root@christyserver]# cd /etc/ [root@christyserver etc]# tailf /var/log/vsftpd.log
Alternatively, you can navigate to the folder and then run tailf.
tailf provides updates as they happen and its output includes time, date, Pid number and IP number.
To stop tailf from continuously generating the logs, press control c on your keyboard.
Log Files in Webmin
Log files can also be analyzed through the Webmin GUI interface on your Linux server.
To check log files via Webmin:
1) Log into Webmin
2) Click on Others (in left panel)
3) Go to File Manager
4) Then navigate to etc/var/log folder to see the various log files.
5) Select the log file you wish to analyze and either double click so that it opens in a browser or click on edit on the top menu bar.
When you click on the edit button from the top menu bar, the file opens in a pop-up window. The loading time of the file depends on the size of the log file.
6) After analyzing the logs you can click close, save or save and close options at the right bottom of the pop-up window. In order to avoid messing with your log files you should just click close after analyzing and identifying attack IPs.
Besides going to File Manager, you can also check log files in Webmin under the tab System Logs.
However, System Logs tab will allow you to check only mail logs, message logs, kernel logs and Apache error logs.
You can’t check vsftpd logs that provide details on ftp logins or apache access logs under the System Logs tab.
What is Suspicious Activity
Suspicious activity as seen through log files can be of different kinds.
Say you manage a web site belonging to a small town called Middletown in Wisconsin and you start seeing multiple IP addresses of visitors from China or Kazakhstan. That’s certainly not normal.
Or the malicious activity can be IPs trying to log into your FTP server (you can see such activity when examining vsftpd.log files).
If you think an IP address is suspicious, grep the IP number and see the number of occurrences. Here’s an example of an IP address trying to access non-existent files.
[root@christyserver httpd]# grep 22.214.171.124 error_log [Mon Dec 15 01:35:05 2014] [error] [client 126.96.36.199] File does not exist: /home/sites/site96/web/contact.htm [Mon Dec 15 01:35:05 2014] [error] [client 188.8.131.52] File does not exist: /home/sites/site85/web/contact.html [Mon Dec 15 01:35:05 2014] [error] [client 184.108.40.206] File does not exist: /home/sites/site69/web/contact.asp [Mon Dec 15 01:35:05 2014] [error] [client 220.127.116.11] File does not exist: /home/sites/site86/web/contact.aspx [Mon Dec 15 01:35:05 2014] [error] [client 18.104.22.168] script '/home/sites/site298/web/contact.php' not found or unable to stat [Mon Dec 15 01:35:05 2014] [error] [client 22.214.171.124] File does not exist: /home/sites/site136/web/main.jsp [Mon Dec 15 01:35:05 2014] [error] [client 126.96.36.199] File does not exist: /home/sites/site99/web/main.asp [Mon Dec 15 01:35:05 2014] [error] [client 188.8.131.52] File does not exist: /home/sites/site98/web/main.htm
Once you find suspicious behavior based on study of logs, you can then ban the IPs via IPtables.
[root@christyserver etc]# iptables -A INPUT -s xx.xxx.xxx.xx -j DROP
Which One to Use
Which logs to analyze depends on your output expectations.
I suggest tailf if you’re looking for monitoring logs as they’re being generated.
And use less if you are looking for patterns of attacks from various IP addresses.
Between Webmin and the command line, I’d recommend you use command line either to run less or tailf so that you can quickly drop the suspicious IP via IP tables and stop the attack via IP tables.