Ubuntu Server 18.04 – Viewing system logs

How to turn off IE enhanced security on Windows Server 2019

After you identify the problem space, you can attack the potential origin of the problem. Often, this will involve reviewing log files on your server. Linux has great logging, and many of the applications you may be running are writing log files as events happen. If there’s an issue, you may be able to find information about it in an application’s log file.

Inside the /var/log directory, you’ll see a handful of logs you can view, which differs from server to server depending on which applications are installed. In quite a few cases, an installed application will create its own log file somewhere within /var/log, either in a log file or a log file within a sub-directory of /var/log. For example, once you install Apache, it will create log files in the /var/log/apache2 directory, which may give you a hint as to what may be going on if the problem is related to your web server. MySQL and MariaDB create their log files in the /var/log/mysql directory. These are known as Application Logs, which are basically log files created by an application and not the distribution. There are also System Logs, such as the authorization or system logs. System logs are the log files created by the distribution and allow you to view system events.

Viewing a log file can be done in several ways. One way is to use the cat command along with the path and filename of a log file. For example, the Apache access log can be viewed with the following command:

    cat /var/log/apache2/access.log
Some log files are restricted and need root privileges in order to access them. If you get a permission denied error when attempting to view a log, use sudo in front of any of the following commands to view the file.

One problem with the cat command is that it will print out the entire file, no matter how big it is. It will scroll by your terminal and if the file is large, you won’t be able to see all of it. In addition, if your server is already taxed when it comes to performance, using cat can actually tie up the server for a bit in a case where the log file is massive. This will cause you to lose control of your shell until the file stops printing. You can press Ctrl + C to stop printing the log file, but the server may end up being too busy to respond to Ctrl + C and show the entire file anyway.

Another method is to use the tail command. By default, the tail command shows you the last ten lines of a file:

    tail /var/log/apache2/access.log 

If you wish to see more than the last ten lines, you can use the -n option to specify a different amount. To view the last 100 lines, we would use the following:

    tail -n 100 /var/log/apache2/access.log  

Perhaps one of the most useful features of the tail command is the -f option, which allows you to follow a log file. Basically, this means that as entries are written to the log file, it will scroll by in front of you. It’s close to watching the log file in real time:

    tail -f /var/log/apache2/access.log  

Once you start using the follow option, you’ll wonder how you ever lived without it. If you’re having a specific problem that you are able to reproduce, you can watch the log file for that application and see the log entries as they appear while you’re reproducing the issue. In the case of a DHCP server not providing IP addresses to clients, you can view the output of the /var/log/syslog file (the isc-dhcp-server daemon doesn’t have its own log file), and you can see any errors that come up as your clients try to re-establish their DHCP lease, allowing you to see the problem as it is happens.

Another useful command for viewing logs is less. The less command allows you to scroll through a log file with the page up and page down keys on your keyboard, which makes it more useful for viewing log files than the cat command. You can press Q to exit the file:

    less /var/log/apache2/access.log  

So now that you know a few ways in which you can view these files, which files should you inspect? Unfortunately, there’s no one rule, as each application handles their logging differently. Some daemons have their own log file stored somewhere in /var/log. Therefore, a good place to check is in that directory, to see if there is a log file with the name of the daemon. Some daemons don’t even have their own log file and will use /var/log/syslog instead. You may try viewing the contents of the file, while using grep to find messages related to the daemon you’re troubleshooting. In regard to the isc-dhcp-server daemon, the following would narrow down the syslog to messages from that specific daemon:

    cat /var/log/syslog |grep dhcp  

While troubleshooting security issues, the log file you’ll definitely want to look at is the authorization log, located at /var/log/auth.log. You’ll need to use the root account or sudo to view this file. The authorization log includes information regarding authentication attempts to the server, including logins from the server itself, as well as over OpenSSH. This is useful for several reasons, among them the fact that if something really bad happens on your server, you can find out who logged in to the server around that same time. In addition, if you or one of your users is having trouble accessing the server via OpenSSH, you may want to look at the authorization log for clues, as additional information for OpenSSH failures will be logged there. Often, the ssh command may complain about permissions of key files not being correct, which would give you an answer as to why public key authentication stopped working, as OpenSSH expects specific permissions for its files. For example, the private key file (typically /home/<user>/.ssh/id_rsa) should not be readable or writable by anyone other than its owning user. You’d see errors within the /var/log/auth.log mentioning such if that were the case.

Another use case for checking the /var/log/auth.log is for security, as a high number of login attempts may indicate an intrusion attempt. (Hopefully, you have Fail2ban installed, which we went over in the last chapter.) An unusually high number of failed password attempts may indicate someone trying brute-force logging in to the server. That would definitely be a cause for concern, and you’d want to block their IP address immediately.

The system log, located in /var/log/syslog, contains logging information for quite a few different things. It’s essentially the Swiss Army knife of Ubuntu’s logs. If a daemon doesn’t have its own log file, chances are its logs are being written to this file. In addition, information regarding C ron jobs will be written here, which makes it a candidate to check when a Cron job isn’t being executed properly. The dhclient daemon, which is responsible for grabbing an IP address from a DHCP server, is also important. You’ll be able to see from dhclient events within the system log when an IP address is renewed, and you can also see messages relating to failures if it’s not able to obtain an IP address. Also, the systemd init daemon itself logs here, which allows you to see messages related to server startup as well as applications it’s trying to run.

Another useful log is the /var/log/dpkg.log file, which records log entries relating to installing and upgrading packages. If a server starts misbehaving after you roll out updates across your network, you can view this log to see which packages were recently updated. This log will not only give you a list of updated or installed packages, but also a time-stamp from when the installation occurred. If a user installed an unauthorized application, you can correlate this log to the authentication log to determine who logged in around that time, and then you can check that user’s bash history to confirm.

Often, log files will get rotated after some time by a utility known as logrotate. Inside the /var/log directory, you’ll see several log files with a .gz extension, which means that the original log file was compressed and renamed, and a new log file created in its place. For example, you’ll see the syslog file for the system log in the /var/log directory, but you’ll also see files named with a number and a .gz extension as well, such as syslog.2.gz. These are compressed logs. Normally, you’d view these logs by uncompressing them and then opening them via any of the methods mentioned in this section. An easier way to do so is with the zcat command, which allows you to view compressed files immediately:

    zcat /var/log/syslog.2.gz  
There’s also zless, which serves a similar purpose as the less command.

Another useful command for checking logging information is dmesg. Unlike other log files, dmesg is literally its own command. You can execute it from anywhere in the filesystem, and you don’t even need root privileges to do so. The dmesg command allows you to view log entries from the Linux kernel’s ring buffer, which can be very useful when troubleshooting hardware issues (such as seeing which disks were recognized by the kernel). When troubleshooting hardware, the system log is also helpful, but using the dmesg command may be a good place to checkĀ as well.

As I mentioned earlier, on an Ubuntu system there are two types of log files, system logs and application logs. System logs, such as the auth.log and the dpkg.log, detail important system events and aren’t specific to any one particular application. Application logs become installed when you install their parent package, such as Apache or MariaDB. Application logs create log entries into their own log file. Some daemons you install will not create their own application log, such as keepalived and isc-dhcp-server. Since there’s no general rule when it comes to which applications log is where, the first step in finding a log file is to see if the application you want log entries from creates its own log file. If not, it’s likely using a system log.

When faced with a problem, it’s important to practice viewing log files at the same time you try and reproduce the problem. Using follow mode with tail (tail -f) works very well for this, as you can watch the log file generate new entries as you try and reproduce the issue. This technique works very well in almost any situation where you’re dealing with a misbehaving daemon. This technique can also help narrow down hardware issues. For example, I once dealt with an Ubuntu system where when I plugged in a flash drive, nothing happened. When I followed the log as I inserted and removed the flash drive, I saw the system log update and recognize each insertion and removal. So, clearly the Linux kernel itself saw the hardware and was prepared to use it. This helped me narrow down the problem to being that the desktop environment I was using wasn’t updating to show the inserted flash drive, but my hardware and USB ports were operating perfectly fine. With one command, I was able to determine that the issue was a software problem and not related to hardware.

As you can see, Ubuntu contains very helpful log files which will aid you in troubleshooting your servers. Often, when you’re faced with a problem, viewing relevant log entries and then conducting a Google search regarding them will result in a useful answer, or at least bring you to a bug report to let you know the problem isn’t just limited to you or your configuration. Hopefully, your search results will lead you right to the answer, or at least to a workaround. From there, you can continue to work through the problem until it is solved.

Comments are closed.