Read and Analyze Linux Logs with journalctl

In this tutorial, we'll learn how to read and analyze Linux logs with journalctl.

We’ll walk through how systemd-journald stores logs, how it differs from traditional /var/log text files, and practical journalctl commands for troubleshooting on modern systemd-based systems (Ubuntu 24.04 used as reference). This is a hands-on guide for beginner → mid-level sysadmins: short explanations, real commands, and concrete use cases so we can actually diagnose problems instead of guessing.

What systemd-journald is (and how it differs from /var/log)

systemd-journald is the systemd logging service. It collects structured, timestamped, indexed log entries from the kernel, system services, and user processes and stores them in a binary journal. That gives us metadata fields (unit, process id, executable path, priority, boot id) and fast, precise filtering.

Contrast with classic /var/log/*.log text files:

• /var/log files are plain text, easy to grep, but unstructured and sometimes duplicated by syslog daemons.
• The journal is binary and indexed — faster, richer queries, and supports structured formats (json, export).
• Many systems still run an rsyslog/syslog service that forwards selected entries to /var/log for compatibility and centralized text logs.

Key consequences:

• We usually use journalctl to read the journal (binary — not plain cat).
• We can export to text/JSON when needed for tools or backups.

Quick sanity checks (first steps)

Run these to confirm what we’re dealing with:

# Show whether journald is running
systemctl status systemd-journald

# See how much disk space the journal is using
journalctl --disk-usage

# See if persistent journal directory exists (persistent = survives reboot)
ls -ld /var/log/journal

If /var/log/journal is missing, the system is likely using volatile storage (in-memory) by default. That’s common on cloud images.

Basic journalctl usage (everyday commands)

We prefer short, memorable commands that tell a clear story.

# Show entire journal (paged)
sudo journalctl

# Jump to the end (most recent entries)
sudo journalctl -e

# Tail the last 100 lines
sudo journalctl -n 100

# Follow new log entries in real time (like `tail -f`)
sudo journalctl -f

# Show logs for a specific systemd unit (service)
sudo journalctl -u nginx.service

# Limit to the last boot only
sudo journalctl -b

# Show logs for previous boot
sudo journalctl -b -1

# Filter by time range
sudo journalctl --since "2025-10-07 09:00" --until "2025-10-07 10:00"

# Use relative times
sudo journalctl --since "1 hour ago"

# Filter by priority (severity)
sudo journalctl -p err        # show error and higher-priority messages

# Show kernel messages (dmesg equivalent, but from journal)
sudo journalctl -k

Output format: default short looks like:

Oct 07 09:12:45 hostname sshd[1234]: Accepted password for alice from 10.0.0.1 port 55022 ssh2

Fields include timestamp, hostname, unit/process and PID, and the message. Use -o to change format.

Useful output formats (pick what fits)

# Human friendly (timestamp precise)
sudo journalctl -o short-iso

# JSON lines (easy to parse)
sudo journalctl -o json

# Pretty JSON
sudo journalctl -o json-pretty

# Export binary journal for import on another host
sudo journalctl -o export > /tmp/journal.export

We use -o json when passing logs to tools (jq, ELK, etc.), and -o short-iso for humans.

Advanced filtering (selectors and fields)

The journal has indexed fields we can use directly:

# Filter by systemd unit field (same as -u but usable in field form)
sudo journalctl _SYSTEMD_UNIT=nginx.service

# Filter by executable path or command
sudo journalctl _EXE=/usr/sbin/sshd
sudo journalctl _COMM=sshd

# Filter by PID
sudo journalctl _PID=1234

# Combine filters: unit + priority + time
sudo journalctl -u nginx.service -p err --since "2025-10-07"

Field names (common): _SYSTEMD_UNIT, _PID, _COMM, _EXE, SYSLOG_IDENTIFIER, MESSAGE_ID. Use these when we need very specific queries.

Persistent vs temporary journals — how to enable persistent logging

By default many distros use volatile storage (in-memory, lost after reboot). To enable persistent journaling:

Edit /etc/systemd/journald.conf and set:

[Journal]
Storage=persistent

Create the directory and restart journald:

sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald

Check:

journalctl --disk-usage
ls -ld /var/log/journal

That makes logs survive reboots and be available for postmortem analysis. On servers where we need historical logs (hosting, debugging production), persistent storage is usually required.

Managing disk usage and retention

systemd-journald respects configuration in /etc/systemd/journald.conf (options like SystemMaxUse, SystemKeepFree, RuntimeMaxUse, MaxFileSec). Typical workflow:

# Check current journal size
journalctl --disk-usage

# Rotate the current journal (ask journald to rotate files)
sudo journalctl --rotate

# Shrink by size (remove old entries until under limit)
sudo journalctl --vacuum-size=500M

# Remove entries older than a time span
sudo journalctl --vacuum-time=2weeks

# Keep only N files
sudo journalctl --vacuum-files=3

If the journal is consuming too much disk, we prefer configuration (SystemMaxUse) plus periodic vacuums rather than manual deletions. journalctl --vacuum-* is safe and supported.

Practical use cases (step-by-step examples)

1) Investigate a failed systemd service (example: nginx)

Steps:

# Check service status and recent error hints
sudo systemctl status nginx.service

# Inspect unit logs (current boot)
sudo journalctl -u nginx.service -b --no-pager -n 200

# If the service dies during start, show logs around that start time
sudo journalctl -u nginx.service --since "5 minutes ago"

Look for lines such as bind() failed: Address already in use or permission denied. The unit's log often contains the exact failure message.

2) Kernel errors and hardware issues

Steps:

# Kernel messages from current boot
sudo journalctl -k -b

# Only show kernel errors
sudo journalctl -k -p err

# If a recent boot had kernel oops, inspect previous boot
sudo journalctl -k -b -1

Kernel logs include dmesg-origin messages (device disconnects, driver OOPS, I/O errors).

3) Boot troubleshooting (service never starts after reboot)

Steps:

# See all boots with IDs and timestamps
journalctl --list-boots

# Pick the boot index (e.g., -1 for previous) and inspect
sudo journalctl -b -1 -e   # jump to end
sudo journalctl -b -1 -u some.service

--list-boots helps map when the failure occurred; then inspect that boot’s logs.

Exporting and moving journal data

For sharing or analysis:

# Text/JSON export for tools
sudo journalctl -u nginx.service -o json > nginx.json

# Binary export/import between systems
sudo journalctl -o export > /tmp/journal.export
# on target: journalctl --file=/tmp/journal.export
sudo journalctl --file=/tmp/journal.export --no-pager -o short

We export JSON when sending to ELK/Graylog or processing with jq.

Permissions and who can read journals

By default only root (and members of group systemd-journal if persistent logs exist and group-readable) can access the binary journal. We usually run sudo journalctl for full access. To grant non-root read access, add the user to systemd-journal group (consider security implications):

sudo usermod -aG systemd-journal <username>

Quick journalctl cheat sheet (copyable)

journalctl                 # all logs
journalctl -e              # jump to the end (recent)
journalctl -f              # follow in real time
journalctl -u foo.service  # logs for unit
journalctl -k              # kernel logs
journalctl --list-boots    # see available boots
journalctl -b              # current boot
journalctl -b -1           # previous boot
journalctl -p err          # errors only
journalctl --since "2 hours ago" --until "now"
journalctl --disk-usage
journalctl --vacuum-size=500M

Best practices and tips

• Prefer -u and indexed fields for precise queries; avoid grep-ing huge journal dumps where possible.
• Use persistent journaling on production servers so logs survive reboots.
• Configure SystemMaxUse / SystemKeepFree in /etc/systemd/journald.conf to prevent journals from filling disks.
• Export JSON for automated parsing or central logging ingestion.
• Use --list-boots and -b for boot-related issues; kernel problems are best viewed via journalctl -k.
• Beware of giving broad read access to logs — they may contain sensitive info (credentials, tokens).

This guide gives the practical journalctl commands and workflows we rely on for Linux troubleshooting: quick checks, targeted filters, disk management, and real use cases (service failures, kernel errors, boot problems). For hosting and operations teams on Ubuntu 24.04, enabling persistent journals and tuning retention saves a lot of time during postmortems and reduces guesswork.

Checkout our dedicated servers India, Instant KVM VPS, and cPanel Hosting India