Day 08: Reading the journal

journalctl with no arguments is useless — it hands you everything since installation. Every useful invocation is a filter, and there are only about five worth memorising.

1. Generate something to read

logger -p user.info "lab: informational message"
logger -p user.err "lab: something went wrong"
logger -t labtag -p user.warning "lab: tagged warning"

logger writes to the system log as if a service had. -p sets facility.priority and -t sets the tag.

2. The five filters worth knowing

journalctl -n 20 --no-pager
journalctl -u ssh -n 20 --no-pager
journalctl --since "10 minutes ago" --no-pager | tail -n 20
journalctl -p err --since today --no-pager | tail -n 20
journalctl -b --no-pager | head -n 20

In order: last N lines, one unit, a time window, errors and worse, and this boot only. --since understands "2024-01-01 10:00", "yesterday", and "1 hour ago".

-p err is the highest-value filter on a machine you have just been handed. Priorities run emerg alert crit err warning notice info debug, and -p err means "err and everything more severe".

3. Follow a service live

journalctl -f -u ssh

-f is tail -f for the journal. Ctrl-C to stop. Combined with a unit this is what you leave running in one pane while you reproduce a bug in another — far better than restarting the service and hoping you catch the output.

4. Compare boots

journalctl --list-boots | tail -n 5
journalctl -b -1 -p err --no-pager | tail -n 20

-b -1 is the previous boot. This is how you investigate a crash that took the machine with it: the evidence is in the boot before the current one, and after a reboot -b alone will never show it.

If --list-boots shows only one entry, the journal is not persistent. Fix it:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

Without this, every reboot discards the journal — and the log you most need is the one from before the reboot.

5. Get the fields behind a line

journalctl -t labtag -n 1 -o json-pretty --no-pager

Every entry is structured, not a string. _PID, _UID, _SYSTEMD_UNIT, PRIORITY and _HOSTNAME are indexed fields you can filter on directly:

journalctl _UID=0 -n 5 --no-pager
journalctl PRIORITY=3 --since today --no-pager | tail -n 5

That is the real difference from a text log: you are querying a database, not grepping a file.

Clean up

The only lasting change this lab made is persistent journal storage, and you should keep it — a journal that vanishes on reboot loses the log you will most need. To undo it anyway:

sudo rm -rf /var/log/journal
sudo systemctl restart systemd-journald
journalctl --list-boots

The test messages age out on their own under the retention limits, or remove them now:

sudo journalctl --vacuum-time=1s

Where this goes next

You can find what a machine is telling you. Tomorrow: keeping its software current without breaking it, and pinning the one package you must not change.