RED HAT ENTERPRISE LINUX

System Logging

Locating and Interpreting Logs for Troubleshooting

CIS238RH | RHEL System Administration 2
Mesa Community College

Welcome to this essential module on system logging in Red Hat Enterprise Linux. Logs are your primary tool for troubleshooting - they record what happened, when it happened, and often why something failed. In this module, you'll learn to navigate RHEL's dual logging architecture: the systemd journal (journalctl) and traditional syslog (rsyslog). You'll master finding logs, filtering them effectively, and interpreting their contents. We'll also cover time synchronization with chronyd, because accurate timestamps are crucial - a log entry with the wrong time is worse than useless during incident investigation. These skills are fundamental for any system administrator and are heavily tested on the RHCSA exam. By the end, you'll confidently locate, filter, and interpret system logs to diagnose and resolve issues.

Learning Objectives

Use journalctl to query the systemd journal

Filter logs by time, unit, priority, and other criteria

Navigate traditional log files in /var/log

Locate and read rsyslog-managed log files

Understand log priorities and facilities

Interpret severity levels from debug to emergency

Configure time synchronization with chronyd

Ensure accurate timestamps across all log entries

We have four main learning objectives. First, you'll master journalctl, the command for querying systemd's journal. The journal captures output from all systemd services and provides powerful filtering capabilities. Second, you'll learn to navigate traditional log files in /var/log. While the journal is the future, many logs still go to text files managed by rsyslog. Third, you'll understand syslog priorities and facilities - the classification system that helps you filter for critical issues versus routine information. Fourth, you'll configure time synchronization using chronyd. Accurate time is essential - during an incident, you need to correlate events across multiple systems and services, and that's impossible if clocks disagree. These skills combine to make you effective at finding the information you need when troubleshooting.

Logging Architecture

RHEL uses two complementary logging systems: the systemd journal for structured logging and rsyslog for traditional text-based log files.

systemd Journal

Binary format, indexed, structured data. Query with journalctl. Captures all systemd service output.

rsyslog

Traditional text log files in /var/log. Human-readable. Configured in /etc/rsyslog.conf.

Facilities & Priorities

Classification system. Facility = source type. Priority = severity level (0-7).

chronyd

NTP client/server for time sync. Ensures accurate timestamps. Replaces ntpd.

Dual System: Messages often go to both journal AND rsyslog. Use whichever is most convenient for your task.

RHEL's logging architecture combines modern and traditional approaches. The systemd journal stores logs in a binary, indexed format. It's fast to search, stores structured metadata, and automatically captures stdout/stderr from all systemd services. You query it with journalctl. rsyslog is the traditional syslog daemon. It writes plain text files to /var/log that you can read with cat, less, grep, and other standard tools. Many administrators prefer text files for their simplicity and compatibility. Both systems coexist - by default, rsyslog reads from the journal and writes to text files. So the same message often exists in both places. Use the journal when you need powerful filtering or service-specific logs. Use text files when you want simplicity or need to use standard Unix tools. Facilities and priorities classify messages by source and severity. Chronyd keeps system time accurate, which is critical for meaningful timestamps.

The systemd Journal

The journal is a system service (systemd-journald) that collects and stores logging data in a structured, indexed binary format.

# View all journal entries (most recent last)
[root@server ~]# journalctl

# View entries in reverse order (newest first)
[root@server ~]# journalctl -r

# Follow new entries in real-time (like tail -f)
[root@server ~]# journalctl -f

# Show only the last 20 entries
[root@server ~]# journalctl -n 20

# Show entries since last boot
[root@server ~]# journalctl -b

# Check journal disk usage
[root@server ~]# journalctl --disk-usage
Archived and active journals take up 48.0M in the file system.

The systemd journal, managed by systemd-journald, is the modern logging system in RHEL. It stores logs in binary format under /run/log/journal (volatile) or /var/log/journal (persistent). The binary format enables fast indexed searching across all fields. journalctl is your interface to the journal. Without arguments, it shows all entries from oldest to newest - usually too much. Common options: -r reverses the order (newest first), useful for seeing recent events. -f follows the log in real-time, like tail -f. -n limits output to the last N entries. -b shows only entries from the current boot, filtering out historical noise. The journal automatically indexes entries by time, service, priority, and many other fields. This is what makes filtering so powerful - the data is already organized. The --disk-usage option shows how much space the journal consumes. By default, the journal limits itself to 10% of the filesystem or 4GB, whichever is smaller.

Filtering by Unit

# Show logs for a specific service
[root@server ~]# journalctl -u sshd.service
Jan 20 10:15:23 server sshd[1234]: Server listening on 0.0.0.0 port 22.
Jan 20 10:15:23 server sshd[1234]: Server listening on :: port 22.
Jan 20 14:32:01 server sshd[5678]: Accepted publickey for student from 192.168.1.100
Jan 20 14:32:01 server sshd[5678]: pam_unix(sshd:session): session opened for user student

# Follow logs for a service in real-time
[root@server ~]# journalctl -u httpd.service -f

# Multiple units
[root@server ~]# journalctl -u sshd.service -u crond.service

# Show logs for a unit since last boot
[root@server ~]# journalctl -u NetworkManager.service -b

# Units can be shortened (without .service)
[root@server ~]# journalctl -u sshd

Tip: The -u filter is your most common tool. When troubleshooting a service, always start with journalctl -u servicename.

Filtering by unit is the most common journalctl operation. When a service isn't working, "journalctl -u servicename" shows exactly what that service logged. The -u option accepts the full unit name (sshd.service) or the short form (sshd). You can specify multiple -u options to see logs from several services together - useful when services interact. Combine with other options: -u plus -f follows a service's logs in real-time, perfect for watching during troubleshooting. -u plus -b limits to the current boot. The output shows timestamp, hostname, service name with PID, and the message. This structured format comes from the journal's binary storage. For RHCSA exam purposes, know that -u is the primary way to view service-specific logs. It's faster and more targeted than grep through text files.

Filtering by Time

# Logs since a specific time
[root@server ~]# journalctl --since "2024-01-20 09:00:00"

# Logs until a specific time
[root@server ~]# journalctl --until "2024-01-20 17:00:00"

# Time range
[root@server ~]# journalctl --since "2024-01-20 09:00" --until "2024-01-20 12:00"

# Relative time expressions
[root@server ~]# journalctl --since "1 hour ago"
[root@server ~]# journalctl --since "yesterday"
[root@server ~]# journalctl --since "2 days ago" --until "1 day ago"

# Today's logs
[root@server ~]# journalctl --since today

# Combine with unit filter
[root@server ~]# journalctl -u sshd --since "1 hour ago"

Time Formats: Use "YYYY-MM-DD HH:MM:SS" or relative expressions like "yesterday", "1 hour ago", "today".

Time filtering narrows logs to a specific period - essential during incident investigation when you know approximately when a problem occurred. The --since option shows logs from that time forward. The --until option shows logs up to that time. Combine them for a precise window. Time can be specified absolutely (2024-01-20 09:00:00) or relatively. Relative expressions are powerful: "1 hour ago", "yesterday", "2 days ago", "today". These calculate from the current time. You can combine time filters with other filters. "journalctl -u httpd --since '1 hour ago'" shows only httpd logs from the last hour - very targeted. During troubleshooting, start broad and narrow down. If you know the problem happened this morning, use --since today. Then narrow to a specific hour once you identify the timeframe. The journal's indexed storage makes time-based queries fast even on systems with extensive logs.

Log Priorities

emerg

System unusable

alert

Immediate action

crit

Critical condition

err

Error condition

warning

Warning condition

notice

Normal but significant

info

Informational

debug

Debug messages

# Show only error and higher priority (0-3)
[root@server ~]# journalctl -p err

# Show warnings and higher (0-4)
[root@server ~]# journalctl -p warning

# Priority range (warning through error)
[root@server ~]# journalctl -p warning..err

# Combine with other filters
[root@server ~]# journalctl -p err -u httpd --since today

Syslog priorities (also called severity levels) range from 0 (most severe) to 7 (least severe). Understanding these helps you filter for the most important messages. Emergency (0) means the system is unusable - kernel panics, critical hardware failures. Alert (1) requires immediate action - security breaches, imminent data loss. Critical (2) indicates critical conditions - disk failures, application crashes. Error (3) shows error conditions - failed operations, service problems. Warning (4) indicates potential problems - disk nearly full, deprecated configs. Notice (5) is normal but significant - service started, user logged in. Info (6) is informational - routine operations, status updates. Debug (7) is detailed debug information - usually only enabled for troubleshooting. The -p option filters by priority. It shows messages at that level AND all more severe levels. So -p err shows emergency, alert, critical, and error messages. For quick troubleshooting, "journalctl -p err --since today" shows today's errors - a great starting point.

Additional Filters

# Filter by executable path
[root@server ~]# journalctl /usr/sbin/sshd

# Filter by PID
[root@server ~]# journalctl _PID=1234

# Filter by UID (user ID)
[root@server ~]# journalctl _UID=1000

# Filter by hostname (useful for aggregated logs)
[root@server ~]# journalctl _HOSTNAME=webserver

# Kernel messages only
[root@server ~]# journalctl -k
[root@server ~]# journalctl --dmesg

# Show available field values
[root@server ~]# journalctl -F _SYSTEMD_UNIT
sshd.service
crond.service
NetworkManager.service
...

# Output in different formats
[root@server ~]# journalctl -o verbose    # All fields
[root@server ~]# journalctl -o json        # JSON format

Beyond unit, time, and priority, journalctl offers many other filters. You can filter by the full path to an executable - useful when you know the program but not the service name. Underscore-prefixed fields filter by metadata: _PID for process ID, _UID for user ID, _HOSTNAME for hostname. These are stored automatically by journald. The -k option (or --dmesg) shows only kernel messages - the same as the dmesg command but with journalctl's filtering capabilities. The -F option lists unique values for a field, helpful for discovering what units or hosts have logged. You can see all available systemd units that have journal entries. Output formats: default is human-readable, -o verbose shows all metadata fields, -o json exports to JSON for parsing. The verbose format reveals all the structured data the journal stores - much more than what you see in text logs.

Persistent Journal Storage

Default Behavior: On RHEL, the journal is stored in memory (/run/log/journal) and lost on reboot unless configured for persistence.

# Check current journal location
[root@server ~]# ls -la /var/log/journal/
ls: cannot access '/var/log/journal/': No such file or directory  # Not persistent

# Enable persistent storage
[root@server ~]# mkdir -p /var/log/journal
[root@server ~]# systemd-tmpfiles --create --prefix /var/log/journal
[root@server ~]# systemctl restart systemd-journald

# Or set in configuration
[root@server ~]# vim /etc/systemd/journald.conf
[Journal]
Storage=persistent

# List available boots (only with persistent storage)
[root@server ~]# journalctl --list-boots
-2 abc123 Fri 2024-01-18 09:00:00 EST—Fri 2024-01-18 18:00:00 EST
-1 def456 Sat 2024-01-19 08:00:00 EST—Sat 2024-01-19 23:00:00 EST
 0 ghi789 Sun 2024-01-20 07:00:00 EST—Sun 2024-01-20 15:30:00 EST

# View logs from previous boot
[root@server ~]# journalctl -b -1

By default, RHEL stores the journal only in memory under /run/log/journal. This means logs are lost when the system reboots - you can't investigate what happened before a crash or previous boot. For persistent storage, create /var/log/journal directory. When this directory exists, journald automatically stores logs there. Use systemd-tmpfiles to create it with proper permissions, then restart journald. Alternatively, edit /etc/systemd/journald.conf and set Storage=persistent. Other options: volatile (memory only), auto (persistent if /var/log/journal exists), none (no storage). With persistent storage, --list-boots shows all stored boots. Each boot gets an ID. Use -b -1 to see the previous boot's logs, -b -2 for two boots ago, etc. This is invaluable for diagnosing crash causes or issues that occurred before the last reboot. For servers, persistent journal storage is strongly recommended.

Traditional Log Files

/var/log/
├── messages ← General system messages
├── secure ← Authentication, sudo, sshd
├── maillog ← Mail server logs
├── cron ← Cron job execution
├── boot.log ← Boot process messages
├── dmesg ← Kernel ring buffer
├── audit/ ← SELinux audit logs
│ └── audit.log
├── httpd/ ← Apache logs (if installed)
│ ├── access_log
│ └── error_log
└── nginx/ ← Nginx logs (if installed)

# View recent entries in messages
[root@server ~]# tail -f /var/log/messages

# Search for errors in secure log
[root@server ~]# grep -i "failed" /var/log/secure

While the journal is powerful, traditional text log files in /var/log remain essential. Many administrators prefer them for simplicity, and some tools expect text files. Key files: /var/log/messages contains general system messages - your first stop for many issues. /var/log/secure holds authentication events including SSH, sudo, and login failures - check here for security incidents. /var/log/cron records cron job execution - verify your scheduled jobs ran. /var/log/boot.log captures boot process messages. /var/log/audit/audit.log contains SELinux audit messages. Applications often create their own subdirectories: httpd, nginx, samba, etc. These files are plain text, so use standard tools: tail -f to follow in real-time, grep to search, less to page through. Rotated logs have numeric or date suffixes (messages.1, messages-20240120). The rsyslog daemon writes these files based on /etc/rsyslog.conf configuration.

rsyslog Configuration

[root@server ~]# cat /etc/rsyslog.conf
...
#### RULES ####

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none     /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                    /var/log/secure

# Log all the mail messages in one place.
mail.*                                        /var/log/maillog

# Log cron stuff
cron.*                                        /var/log/cron

# Everybody gets emergency messages
*.emerg                                       :omusrmsg:*

# Save boot messages to boot.log
local7.*                                      /var/log/boot.log

Rule Format: facility.priority action
Example: authpriv.* = all priorities from authpriv facility → write to /var/log/secure

The rsyslog daemon is configured in /etc/rsyslog.conf. Rules specify which messages go where based on facility and priority. Each rule has a selector (facility.priority) and an action (usually a file path). The asterisk means "all" - authpriv.* means all priorities from the authpriv facility. Multiple selectors can be combined with semicolons. The line "*.info;mail.none;authpriv.none;cron.none /var/log/messages" means: all facilities at info and above, except mail, authpriv, and cron (which go elsewhere). Special actions: :omusrmsg:* sends to all logged-in users' terminals (for emergencies). Files starting with - disable sync after write (faster but less safe). Additional configuration in /etc/rsyslog.d/*.conf allows modular configuration. Packages can drop files there. After editing rsyslog configuration, restart the service: systemctl restart rsyslog. Most of the time you won't edit this, but understanding it helps you know where to find logs.

Syslog Facilities

Facility	Description	Default Log
kern	Kernel messages	/var/log/messages
user	User-level messages	/var/log/messages
mail	Mail system	/var/log/maillog
daemon	System daemons	/var/log/messages
auth	Security/authorization	/var/log/secure
authpriv	Private auth messages	/var/log/secure
cron	Cron daemon	/var/log/cron
local0-7	Custom/local use	Configurable

# Send test message to a facility
[root@server ~]# logger -p local0.info "Test message to local0"
[root@server ~]# logger -p authpriv.warning "Security test warning"

# Check where it went
[root@server ~]# tail /var/log/messages

Facilities classify the source of log messages. They're predefined categories that help route messages to appropriate files. Kern is for kernel messages only. User is for user-space programs without a specific facility. Mail is for mail transfer agents and related services. Daemon is for system services without their own facility. Auth and authpriv are for authentication - authpriv is for more sensitive messages with restricted file permissions. Cron is for cron and at daemons. Local0 through local7 are reserved for local/custom use - applications can use these, and you can configure where they go. The logger command sends test messages to syslog - useful for testing configuration. Specify facility and priority with -p: logger -p local0.info "message". Combined with priorities, facilities give fine-grained control over where each type of message is stored. Most default configurations work well, but understanding facilities helps when you need custom log routing.

Reading Log Entries

Jan 20 14:32:01 server sshd[5678]: Accepted publickey for student from 192.168.1.100 port 52341 ssh2

Timestamp: Jan 20 14:32:01
When the event occurred

Hostname: server
Which system generated the log

Service[PID]: sshd[5678]
Process name and ID

Message: Accepted publickey...
The actual log content

# Common log patterns to search for
[root@server ~]# grep -i "failed\|error\|denied" /var/log/secure
Jan 20 14:15:33 server sshd[4521]: Failed password for student from 192.168.1.50
Jan 20 14:15:35 server sshd[4521]: Failed password for student from 192.168.1.50
Jan 20 14:15:38 server sudo[4599]: student : command not allowed

Understanding log entry format helps you quickly extract information. The standard syslog format has four parts. Timestamp shows when the event occurred - critical for correlating events across services. Traditional syslog timestamps lack the year, which can be problematic for old logs. The journal stores full timestamps. Hostname identifies the source system - important when aggregating logs from multiple servers. Service and PID identify what process generated the message. The PID helps track specific process instances. Message is the actual content - what happened. Messages vary widely by application. When searching logs, look for keywords: "failed", "error", "denied", "unable", "warning", "critical". Case-insensitive search (-i) catches variations. Multiple patterns with \| (OR) find different types of problems. Reading logs is a skill. With practice, you'll recognize patterns: authentication failures, permission denials, service crashes, resource exhaustion. Context matters - look at surrounding entries to understand the full story.

Log Analysis Techniques

# Count occurrences of patterns
[root@server ~]# grep -c "Failed password" /var/log/secure
47

# Find unique IP addresses attempting SSH
[root@server ~]# grep "Failed password" /var/log/secure | \
    awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head
     23 192.168.1.50
     12 10.0.0.100
      8 172.16.0.25

# Timeline of errors from journal
[root@server ~]# journalctl -p err --since today --no-pager | head -20

# Watch for specific patterns in real-time
[root@server ~]# tail -f /var/log/secure | grep --line-buffered "Failed"

# Correlate across multiple logs
[root@server ~]# grep "14:32" /var/log/messages /var/log/secure

# Journal output suitable for scripting
[root@server ~]# journalctl -u sshd -o json --since "1 hour ago" | \
    jq '.MESSAGE' | head

Effective log analysis combines tools and techniques. Counting patterns with grep -c gives quick metrics - 47 failed passwords today might indicate a brute force attack. Pipeline analysis extracts specific fields. The awk command pulls out IP addresses from failed login attempts, sort and uniq -c counts occurrences, sort -rn orders by frequency. This quickly identifies the worst offenders. For real-time monitoring, tail -f with grep watches for specific patterns. The --line-buffered option ensures grep outputs immediately rather than buffering. Cross-referencing logs helps correlate events. If something happened at 14:32, check multiple logs for that timestamp. You might find an authentication event in secure, a service error in messages, and an SELinux denial in audit.log - all related. The journal's JSON output (-o json) enables scripting. Pipe to jq for parsing. This is useful for automated analysis, alerting, or feeding data to monitoring systems.

Time Synchronization

Accurate timestamps are critical for log analysis. During incidents, you correlate events across multiple systems - impossible if clocks disagree.

chronyd

RHEL's default NTP client. Fast sync, handles intermittent connectivity, low resource usage.

NTP Protocol

Network Time Protocol. Synchronizes clocks over the network to reference time servers.

# Check current time settings
[root@server ~]# timedatectl
               Local time: Sun 2024-01-20 15:45:30 EST
           Universal time: Sun 2024-01-20 20:45:30 UTC
                 RTC time: Sun 2024-01-20 20:45:30
                Time zone: America/New_York (EST, -0500)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

Scenario: Server A logs an error at 14:30:00, Server B shows related event at 14:29:55. Did B cause A, or vice versa? Without sync, you can't know.

Time synchronization might seem mundane, but it's critical for effective logging. When investigating incidents, you correlate events across multiple systems and services. If server A's clock is 2 minutes ahead of server B's, a causal chain becomes impossible to trace. Imagine debugging a distributed application where the database logs an error, the web server logs a failure, and the load balancer logs a disconnect. You need accurate timestamps to determine the sequence - what triggered what. chronyd is RHEL's default NTP implementation, replacing the older ntpd. It synchronizes the system clock using the Network Time Protocol (NTP), typically to internet time servers or local time servers. timedatectl shows current time configuration. Key indicators: "System clock synchronized: yes" means NTP is working. "NTP service: active" means chronyd is running. The timezone should match your location for readable local times in logs.

chronyd Configuration

# Check chronyd service status
[root@server ~]# systemctl status chronyd
● chronyd.service - NTP client/server
     Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled)
     Active: active (running)

# View current time sources
[root@server ~]# chronyc sources
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* time.cloudflare.com           3   6   377    34   -234us[ -312us] +/-   15ms
^+ ntp.ubuntu.com                2   6   377    35  +1234us[+1156us] +/-   28ms
^+ time.google.com               1   6   377    36   +892us[ +814us] +/-   12ms

# Check synchronization status
[root@server ~]# chronyc tracking
Reference ID    : CF8A9A87 (time.cloudflare.com)
Stratum         : 4
Ref time (UTC)  : Sun Jan 20 20:45:30 2024
System time     : 0.000000234 seconds fast of NTP time
Last offset     : -0.000078234 seconds
RMS offset      : 0.000125432 seconds
...

chronyd is the daemon; chronyc is the command-line client for querying and controlling it. First, verify chronyd is running with systemctl status. It should be enabled to start at boot. chronyc sources shows the time servers being used. Key columns: M shows mode (^ means server), S shows state (* means current sync source, + means acceptable alternative). Stratum is distance from reference clock (1 is best, atomic clock; each hop adds 1). Reach is an octal number showing recent connectivity (377 means all recent polls succeeded). chronyc tracking shows detailed sync status. Key metrics: System time shows offset from NTP time (smaller is better, microseconds is excellent). Stratum 4 means we're 4 hops from a reference clock. The configuration file /etc/chrony.conf specifies time servers. Default RHEL configuration uses Red Hat's time servers, but you can add or change servers. For enterprise environments, use internal NTP servers for consistency.

Configuring Time Servers

# Edit chronyd configuration
[root@server ~]# vim /etc/chrony.conf
# Use public servers from the pool.ntp.org project.
pool 2.rhel.pool.ntp.org iburst

# Use specific servers (enterprise environment)
server time.company.internal iburst prefer
server ntp1.company.internal iburst
server ntp2.company.internal iburst

# Allow stepping clock at startup if offset > 1 second
makestep 1.0 3

# Record the rate at which the system clock gains/loses time.
driftfile /var/lib/chrony/drift

# After editing, restart chronyd
[root@server ~]# systemctl restart chronyd

# Force immediate sync
[root@server ~]# chronyc makestep
200 OK

# Verify new sources
[root@server ~]# chronyc sources -v

The /etc/chrony.conf file controls time synchronization. Pool entries use DNS to find multiple servers from a pool - good for general use. Server entries specify individual servers - use these for enterprise environments where you have internal NTP servers. The iburst option sends multiple requests at startup for faster initial sync. The prefer option marks a server as preferred when multiple are available. makestep allows the clock to jump (step) if the offset is too large. The default "1.0 3" means: if offset exceeds 1 second, allow stepping during the first 3 clock updates after start. After that, only slewing (gradual adjustment) is allowed. The driftfile records the system clock's tendency to gain or lose time, allowing chrony to compensate even when network is unavailable. After configuration changes, restart chronyd. Use chronyc makestep to force an immediate sync if needed. The -v option with sources shows verbose output explaining column meanings.

Setting Timezone

# View current timezone
[root@server ~]# timedatectl show --property=Timezone
Timezone=America/New_York

# List available timezones
[root@server ~]# timedatectl list-timezones | grep America
America/Chicago
America/Denver
America/Los_Angeles
America/New_York
...

# Set timezone
[root@server ~]# timedatectl set-timezone America/Los_Angeles

# Verify change
[root@server ~]# timedatectl
               Local time: Sun 2024-01-20 12:45:30 PST
           Universal time: Sun 2024-01-20 20:45:30 UTC
                Time zone: America/Los_Angeles (PST, -0800)

# Alternative: use symlink directly
[root@server ~]# ls -l /etc/localtime
lrwxrwxrwx. 1 root root 38 Jan 20 12:45 /etc/localtime -> ../usr/share/zoneinfo/America/Los_Angeles

While NTP syncs to the correct absolute time, the timezone determines how that time is displayed. Servers often use UTC for consistency (especially in global deployments), while user-facing systems use local time. timedatectl manages timezone settings. The list-timezones option shows all available zones - there are hundreds. Use grep to filter. Format is Region/City (America/New_York, Europe/London, Asia/Tokyo). set-timezone changes the system's timezone. This affects how times are displayed in logs and to users. The underlying UTC time doesn't change - just the presentation. Technically, /etc/localtime is a symlink to a file in /usr/share/zoneinfo/. timedatectl manages this symlink, but you can also manipulate it directly if needed. For servers, many administrators prefer UTC to avoid daylight saving time issues and simplify correlation across systems in different timezones. For user workstations, local timezone is usually more appropriate.

Troubleshooting Workflow

Identify the timeframe

When did the problem occur? Use --since and --until to narrow scope.

Check service-specific logs

journalctl -u servicename -p err --since "1 hour ago"

Check system-wide logs

journalctl -p err --since "1 hour ago" or check /var/log/messages

Look for related events

Check /var/log/secure for auth, /var/log/audit/audit.log for SELinux

Correlate across systems

Check logs on related servers at the same timestamp

# Quick troubleshooting sequence
[root@server ~]# journalctl -u httpd -p err --since "30 min ago"  # Service errors
[root@server ~]# journalctl -p err --since "30 min ago"          # All errors
[root@server ~]# tail -100 /var/log/messages | grep -i error     # Text files
[root@server ~]# ausearch -m AVC -ts recent                      # SELinux denials

A systematic troubleshooting workflow helps you find issues faster. Start by identifying when the problem occurred. Users often report "it's not working" - probe for when it last worked, when they noticed the problem. This gives you a time window. Then check service-specific logs for the affected service. Use -p err to focus on errors and warnings. Broaden to system-wide logs if the service logs don't reveal the cause. Related services might be failing, or system-level issues might be the root cause. Check related logs - authentication issues in /var/log/secure, SELinux in audit.log. Many "mysterious" failures turn out to be permission denials. Finally, correlate across systems. If a web server can't reach a database, check both systems' logs at the same time. This is where accurate time sync pays off. The quick sequence shown covers the most common cases: service errors, system errors, text file check, and SELinux denials. Run through this sequence and you'll often find your answer.

Best Practices

✓ Do

Enable persistent journal storage on servers
Configure NTP and verify synchronization
Use UTC timezone on servers
Filter logs by time first, then by unit/priority
Check multiple log sources when troubleshooting
Monitor logs proactively, not just reactively
Set up log rotation to manage disk space
Document what normal looks like

✗ Don't

Ignore time synchronization
Delete logs without archiving
Assume one log file has all the answers
Read entire log files without filtering
Forget to check SELinux audit logs
Overlook rotated log files (.1, .gz)
Rely only on memory-based journal
Panic - be systematic

Proactive Monitoring: Set up tools like logwatch, or forward logs to a central system for alerting on errors before users report problems.

Good logging practices make troubleshooting faster and more reliable. Enable persistent journal storage on any system you care about - losing logs on reboot loses critical diagnostic information. Configure NTP on every system and verify it's working. For servers, consider UTC timezone to avoid daylight saving confusion and simplify multi-timezone correlation. When searching logs, filter aggressively - time first, then unit, then priority. Don't try to read everything. Always check multiple sources - the answer often spans multiple logs. Be proactive: know what normal looks like so you recognize abnormal. Set up monitoring to alert on errors. Don't delete logs without consideration - archive them if space is limited. Remember rotated logs exist - that error from last week might be in messages.1 or messages-20240115.gz. Don't forget SELinux - many "permission denied" issues are actually AVC denials. Most importantly, stay calm and systematic. Panic leads to missing obvious clues.

Key Takeaways

journalctl: Query journal with -u unit, -p priority, --since/--until time, -f follow, -b boot

/var/log: messages (general), secure (auth), cron, boot.log, application-specific directories

Priorities: 0=emerg through 7=debug. Use -p err for errors and above. Filter for what matters.

Time sync: chronyd for NTP, timedatectl for status/timezone. Accurate timestamps are essential.

Let's summarize the key skills for system logging. journalctl is your primary tool for the systemd journal. Key options: -u for unit filtering, -p for priority filtering, --since and --until for time ranges, -f to follow in real-time, -b for current boot. Combine these for targeted searches. Traditional log files in /var/log remain important. Know the key files: messages for general system logs, secure for authentication, cron for scheduled jobs, and application-specific directories. Priorities range from 0 (emergency) to 7 (debug). Lower numbers are more severe. Use -p err to focus on errors and above during troubleshooting. Time synchronization through chronyd ensures your timestamps are accurate. Use chronyc sources to verify sync status, timedatectl to check and set timezone.

Graded Lab

Update the time zone on an existing server.
Configure system journals to persist to storage and to rotate more often.
Configure a new log file to store all messages for authentication failures.

Next: Managing Security with SELinux