RED HAT ENTERPRISE LINUX

System Logs
and Journals

Locate and interpret system log files and journals

CIS126RH | RHEL System Administration 1
Mesa Community College

When something goes wrong on a RHEL system, the logs tell the story. Understanding where logs live, which tool reads which log, and how to filter log output to find relevant events is the most fundamental troubleshooting skill in Linux system administration. This module covers the two parallel logging systems on RHEL 9 — the systemd journal and the rsyslog text files — and the tools used to read and filter them. These skills are tested on the RHCSA exam.

Learning Objectives

1. Identify the key log files in /var/log/ — Know which log contains which type of event and when to look at each
2. Read and filter the systemd journal with journalctl — Use journalctl flags to filter by unit, priority, time range, and boot session
3. Read and search rsyslog text files — Use tail, grep, and less to navigate and search log files in /var/log/
4. Interpret log entries and syslog priority levels — Decode the structure of a log line and understand the severity levels from emergency to debug

The Two Logging Systems on RHEL 9

RHEL 9 runs two complementary logging systems simultaneously.

• The journal collects log entries from the kernel, systemd units, and applications
• rsyslog reads from the journal and writes formatted text to /var/log/
• Both systems receive the same events — they are complementary, not competing
• The journal's structured data enables powerful filtering that text files cannot match

Key Log Files in /var/log/

Reading Text Log Files

Standard Unix text tools are the right choice for reading /var/log/ text files.

# Follow /var/log/messages in real time (like top for logs)
$ sudo tail -f /var/log/messages

# Show the last 50 lines
$ sudo tail -n 50 /var/log/messages

# Search for errors in the secure log
$ sudo grep -i error /var/log/secure

# Search for failed SSH logins
$ sudo grep "Failed password" /var/log/secure

# Search across rotated log files
$ sudo zgrep "Failed password" /var/log/secure*

# Page through messages interactively
$ sudo less /var/log/messages
# Inside less: /pattern to search, n for next match, G for end

Syslog Priority Levels

Every log entry has a priority (severity) level. Understanding these levels helps filter log output and identify serious issues quickly.

Introduction to journalctl

journalctl queries the systemd journal — the structured binary log that captures all kernel, service, and application messages.

# Show the full journal — all entries since last boot
$ journalctl

# Show the most recent entries first (reverse order)
$ journalctl -r

# Follow the journal in real time (like tail -f)
$ journalctl -f

# Show only kernel messages
$ journalctl -k

# Show the last N lines
$ journalctl -n 50

# Show entries without paging (output directly to stdout)
$ journalctl --no-pager

Filtering by Unit and Priority

The most useful journalctl filters narrow output to a specific service or severity level.

# Show all messages from the sshd service
$ journalctl -u sshd

# Follow sshd messages in real time
$ journalctl -u sshd -f

# Show messages for multiple units
$ journalctl -u sshd -u httpd

# Show only error and higher priority messages
$ journalctl -p err

# Show warning and higher across all services
$ journalctl -p warning

# Combine unit and priority filters
$ journalctl -u httpd -p err

# Priority levels: emerg alert crit err warning notice info debug
# A level includes that level and all more severe levels

Filtering by Time

journalctl accepts human-readable time specifications to narrow output to a specific time window.

# Show entries since a specific time
$ journalctl --since "2026-05-25 09:00:00"

# Show entries since a relative time
$ journalctl --since "1 hour ago"
$ journalctl --since yesterday
$ journalctl --since today

# Show entries in a time range
$ journalctl --since "2026-05-25 08:00" --until "2026-05-25 10:00"

# Show entries from a specific date
$ journalctl --since "2026-05-25" --until "2026-05-26"

# Combine time and unit filters
$ journalctl -u sshd --since "1 hour ago"

Filtering by Boot Session and Other Metadata

# Show entries from the current boot only (default)
$ journalctl -b

# Show entries from the previous boot
$ journalctl -b -1

# Show entries from two boots ago
$ journalctl -b -2

# List available boot sessions
$ journalctl --list-boots
 -2 a1b2... Mon 2026-05-23 07:00 — Tue 2026-05-24 22:00
 -1 c3d4... Wed 2026-05-24 22:05 — Thu 2026-05-25 06:55
  0 e5f6... Thu 2026-05-25 07:00 — present

# Show messages from a specific executable
$ journalctl /usr/sbin/sshd

# Show messages from a specific PID
$ journalctl _PID=4251

# Show messages from a specific user ID
$ journalctl _UID=1000

Anatomy of a Log Entry

Understanding each part of a log line makes interpreting log output faster and more accurate.

# A typical entry from /var/log/messages (rsyslog format)
May 25 10:15:33 servera sshd[4251]: Failed password for student from 192.168.1.5 port 44132 ssh2
#  ^timestamp   ^host   ^proc[pid] ^message

# A typical journalctl entry
May 25 10:15:33 servera sshd[4251]: Failed password for student from 192.168.1.5 port 44132 ssh2

# journalctl with verbose output -- shows full structured fields
$ journalctl -o verbose -u sshd -n 1
Thu 2026-05-25 10:15:33.123456 MST [s=abc...]
    _HOSTNAME=servera
    _TRANSPORT=syslog
    PRIORITY=5
    SYSLOG_FACILITY=4
    _SYSTEMD_UNIT=sshd.service
    MESSAGE=Failed password for student from 192.168.1.5 port 44132 ssh2
    _PID=4251
    _UID=0

Making the Journal Persistent

By default, the systemd journal stores entries in /run/log/journal/ which is cleared on reboot. Enabling persistent storage saves the journal across boots.

# Check whether the journal is currently persistent
$ ls /var/log/journal/
# If this directory is empty or absent, journal is volatile

# Enable persistent journal storage
$ sudo mkdir -p /var/log/journal
$ sudo systemd-tmpfiles --create --prefix /var/log/journal

# Alternatively — set Storage=persistent in the journal config
$ sudo vim /etc/systemd/journald.conf
# Change: #Storage=auto  to: Storage=persistent

# Restart journald to apply the change
$ sudo systemctl restart systemd-journald

# Confirm the change — journal now writes to /var/log/journal/
$ ls /var/log/journal/
6e3a...   # a directory named after the machine ID

Practical Log Investigation Workflow

When a service fails or the system behaves unexpectedly, a systematic approach finds the relevant log entries quickly.

Service failed to start

# 1. Check the service status for a summary
$ systemctl status httpd
# The last few log lines appear in the status output

# 2. Read the full journal for the service
$ journalctl -u httpd --since "10 minutes ago"

# 3. If the journal is insufficient, check the text log
$ sudo tail -n 50 /var/log/httpd/error_log

Failed SSH logins — security investigation

# Count failed passwords by source IP
$ sudo grep "Failed password" /var/log/secure | \
  awk '{print $11}' | sort | uniq -c | sort -rn | head

rsyslog Configuration

rsyslog controls which log messages are written to which text files. Its configuration determines what appears in /var/log/.

# Main configuration file
$ cat /etc/rsyslog.conf

# Additional configuration files (drop-in format)
$ ls /etc/rsyslog.d/

# rsyslog rule format: FACILITY.PRIORITY  /path/to/logfile
kern.*              /dev/console
*.info;mail.none;authpriv.none;cron.none  /var/log/messages
authpriv.*          /var/log/secure
mail.*              -/var/log/maillog
cron.*              /var/log/cron

# Restart rsyslog after config changes
$ sudo systemctl restart rsyslog

# Test that logging is working
$ logger "Test message from $(whoami)"
$ tail -1 /var/log/messages

journalctl Quick Reference

Common Mistakes

Knowledge Check

Answer these before moving to the next slide.

1. You need to watch the system log in real time while restarting a service. Write the command using both a text file approach and a journalctl approach.
2. Write the journalctl command to show only error-level and more severe messages from the sshd service since yesterday.
3. Which /var/log/ file would you check first to investigate failed SSH login attempts?
4. What do the eight syslog priority levels mean, and which two are the most important to act on immediately?
5. The system rebooted unexpectedly last night. Write the journalctl command to view log entries from the boot session before the current one.
6. What is the difference between the systemd journal and rsyslog on RHEL 9? Are they alternatives or complementary?

Knowledge Check — Answers

1. Text file approach: sudo tail -f /var/log/messages
   Journal approach: journalctl -f or journalctl -u SERVICENAME -f for a specific service
2. journalctl -u sshd -p err --since yesterday — the -p err filter shows err, crit, alert, and emerg messages; the -u sshd filter limits to the sshd service; --since yesterday limits to events since midnight.
3. /var/log/secure — this file records all authentication events including SSH logins, failed password attempts, and sudo usage.
4. The eight levels from most to least severe: emerg (0), alert (1), crit (2), err (3), warning (4), notice (5), info (6), debug (7). The two most important to act on immediately are emerg (system is unusable) and alert (immediate action required). In practice, err and above deserves prompt investigation.
5. journalctl -b -1 — the -b -1 flag selects the boot session one before the current one. Use journalctl --list-boots to confirm available boot sessions.
6. The systemd journal and rsyslog are complementary — both run simultaneously on RHEL 9 and receive the same log data. rsyslog reads from the journal and writes plain text files to /var/log/. The journal provides structured binary storage with powerful filtering via journalctl; the text files provide compatibility with traditional Unix tools and survive reboots.

Key Takeaways

1. RHEL 9 has two complementary logging systems. The systemd journal (read with journalctl) provides structured, filterable log data. rsyslog writes plain text to /var/log/ (read with tail, grep, less). Both receive the same events.
2. Know which log file to check for which event type. /var/log/messages — general system events. /var/log/secure — authentication and security. /var/log/audit/audit.log — SELinux denials. Service logs in /var/log/SERVICENAME/ for application details.
3. The four most important journalctl flags are -u, -p, --since, and -f. -u UNIT filters by service. -p PRIORITY filters by severity (and includes all more severe levels). --since limits by time. -f follows in real time. -b -1 shows the previous boot.
4. Syslog has eight priority levels — err and above require action. emerg(0) alert(1) crit(2) err(3) warning(4) notice(5) info(6) debug(7). journalctl -p err shows err through emerg — the actionable events.

Graded Lab

• List the contents of /var/log/. Identify and open /var/log/messages with sudo less. Search for the word "error" using /error inside less.
• Run journalctl --list-boots to see available boot sessions. Run journalctl -b to view entries from the current boot and note the first entry's timestamp.
• Run journalctl -u sshd --since "30 minutes ago" to view recent sshd activity. Then add -f and try an SSH login from another terminal to watch the journal update in real time.
• Run journalctl -p err --since today to find any error-level or worse messages from today. Note which services generated them.
• Use sudo tail -f /var/log/secure and sudo tail -f /var/log/messages in two separate terminals simultaneously to watch both logs while triggering an event (such as a sudo command).
• Use logger "Lab test message from $(whoami)" to inject a test entry. Confirm it appears in both /var/log/messages and in journalctl -n 5.

Next: Preserve system journals