Linux Logs and System Monitoring | journalctl, syslog

Linux Logs and System Monitoring

Log files are the primary diagnostic tool for Linux systems. Every service, kernel event, authentication attempt, and system error is recorded in log files. Understanding how to read, filter, and monitor logs is a critical Linux skill for troubleshooting production issues, auditing security events, and maintaining system health.

The /var/log Directory

System log files are stored in /var/log. The most important log files include syslog (general system messages), auth.log (authentication events), kern.log (kernel messages), and application-specific logs in subdirectories like /var/log/nginx/ and /var/log/mysql/.

ls /var/log
tail -f /var/log/syslog
grep "error" /var/log/syslog
cat /var/log/auth.log | grep "Failed"

Using journalctl

On systemd-based Linux systems, journalctl is the primary tool for reading logs from the systemd journal. It provides powerful filtering options for time, service, priority level, and more.

journalctl
journalctl -xe
journalctl -u nginx
journalctl --since "2024-01-01" --until "2024-01-02"
journalctl -p err
journalctl -f

The -u flag filters by systemd unit (service name). The -p err flag shows only error-level messages. The -f flag follows the journal in real time, similar to tail -f.

Kernel Messages with dmesg

The dmesg command displays messages from the kernel ring buffer. It is extremely useful for diagnosing hardware issues, driver errors, and boot-time problems.

dmesg
dmesg | grep "error"
dmesg | tail -20
dmesg --level=err

Scheduling Tasks with cron

Cron is the Linux task scheduler. It runs commands at specified times and intervals. The crontab -e command opens the user's cron table for editing. The cron schedule format is: minute, hour, day of month, month, day of week, followed by the command.

crontab -e
crontab -l

# Run backup.sh every day at 2:30 AM
30 2 * * * /home/alice/scripts/backup.sh

# Run cleanup every Sunday at midnight
0 0 * * 0 /home/alice/scripts/cleanup.sh >> /var/log/cleanup.log 2>&1

Logrotate

Logrotate is a tool that automatically rotates, compresses, and deletes old log files to prevent disks from filling up. Its configuration is in /etc/logrotate.conf and /etc/logrotate.d/. Most packages install their own logrotate configuration automatically.

cat /etc/logrotate.conf
ls /etc/logrotate.d/
sudo logrotate -f /etc/logrotate.conf