When looking at logs, you should almost always add the --utc flag or check server's local
time (timedatectl status). It should always be UTC (timedatectl set-timezone UTC).
Why? Because when you look at logs, you often compare them with other logs/events. E.g.
Sentry issues, papertrail logs, other systems, other servers, ...
Let's talk about the most frequent journalctl commands.
Usually, you don't want to see all the logs. You want to see logs from a specific unit
(e.g., HAProxy, MySQL...). You also want to specify the time range. For this, we need to
add 2 flags. --unit (-u for short) and
--since (-S for short).
Generally, I like to use full flag names because IMO it's a good practice to do so. Why?
You learn them faster, and you will remember them longer because they are words. This is
the reason why we invented domains.
You also often copy-paste those commands to your co-workers, to your documentation, etc.
And if a developer isn't familiar with those flags, they will need to check the docs.
If we use full flag names, this is often unnecessary because full flag names are more
descriptive.
$ journalctl --unit haproxy --since "2 days ago"
$ journalctl --unit haproxy --since "2 hours ago"
$ journalctl --unit haproxy --since "2018-06-26"
$ journalctl --unit haproxy --since "2018-06-26 23:00"
$ journalctl --unit haproxy --since "2018-06-26" --until "2018-06-27"You can also select multiple units.
$ journalctl --unit haproxy --unit mysqlSometimes you want to search through logs. You have 2 options. You can use the --grep
flag or you can pipe the journalctl output to grep.
I usually prefer to pipe the output because of 2 things. 1. I can do things like count
the results (grep --count ...). Also, results are printed to your terminal, and you
can copy-paste them.
The bad thing with piping the results is that you don't see some information like
e.g., system reboot.
journalctl --unit haproxy | grep "Segmentation fault"
journalctl --unit haproxy | grep --count "Segmentation fault"
journalctl --unit haproxy --grep "Segmentation fault"If you want to immediately see the last, e.g., 1000 lines of the journal log, you can use
--pager-end (-e for short) to jump to the end and --lines 1000 (-n) to show you
the last 1000 lines.
$ journalctl --pager-end --lines 1000Show kernel messages from the last boot:
$ journalctl --bootTo see system log messages, we need to filter by identifier. You can also use --identifier
for other services (e.g., haproxy).
$ systemctl --identifier kernel
$ systemctl --identifier haproxyWhen you are doing live monitoring, you will want to use the --follow (-f) flag:
$ systemctl --unit haproxy --followRef: