How to Follow Logs Beyond tail -f: Choosing Between grep Filtering and journalctl

First Determine Where the Logs Come From

The first step in following logs is not choosing a command, but confirming where the logs land. When a service is started by systemd, content the application writes to stdout and stderr goes into the journal by default, and journalctl -u can query it directly; if the application configures file output itself (log4j, logback, zap, Python logging's FileHandler, etc.), the logs are in files, and only then do tail and grep have a target. The two may also coexist: some services both write files and hand part of their output to systemd. In that case, first confirm which copy you are looking at, to avoid switching back and forth between the two logs and to avoid mistaking “not in the file” for “the service did not log”.

File Logs: Prefer tail -F, Not Just -f

tail -f follows the file descriptor. During log rotation, if logrotate uses the “rename + create new” approach, the old file is moved away and the new file has a new inode; tail -f keeps watching the old inode, so it looks as if logging has stopped. tail -F is equivalent to --follow=name --retry, reopens by file name, and can continue following after rotation.

tail -n 200 -F /var/log/myapp/app.log

-n 200 first shows the most recent 200 lines as context, avoiding seeing only new logs without knowing what happened before. If there are multiple files in a directory, tail -F also supports following multiple files at once, and the output is separated with file names.

But tail -F does not solve “looking back by time window.” To check logs from yesterday 14:00 to 15:00, the file may already have been rotated to app.log.1 or compressed to .gz, so you need to process files one by one. zcat with grep can search compressed history, but stitching a timeline across multiple files is still cumbersome. This is an inherent cost of file logs: they leave “storage” to the application and logrotate, so query capabilities must be added yourself.

grep Filtering: Turn “Following” into “Filtering”

The most direct combination is:

tail -n 500 -F /var/log/myapp/app.log | grep --line-buffered -E 'ERROR|Exception|timeout'

--line-buffered is critical. When grep outputs to a pipe or file, it uses block buffering by default, so log lines may accumulate to several KB before being emitted, which looks like “log delay.” With it enabled, each matching line is output immediately, making it suitable for real-time observation. The cost is more frequent system calls, adding a little CPU when log volume is extremely high, but this is usually acceptable.

Add -C when you need context:

tail -n 500 -F /var/log/myapp/app.log | grep --line-buffered -C 3 -E 'ERROR|Exception'

Note that in streaming scenarios, -C only applies to lines that have already arrived. Exception stack traces often appear as multiple lines; grep matches by line, so it may only hit the first Exception line, and subsequent at ... lines will not be brought out automatically. To preserve the full stack, either increase -C, or do not filter first—write the raw stream to a file and inspect it with less—or make the application output single-line structured logs (JSON) and filter with jq.

grep also has several useful options: -n shows line numbers for locating positions back in the file; --color=auto highlights matches in the terminal; -v inverts the filter to exclude noise such as health checks and heartbeats; -F matches fixed strings to avoid interference from regex metacharacters; -m limits the number of matches and is suitable for searching historical files rather than -f streaming scenarios.

If you just want to browse history while watching, less +F is often more comfortable than tail -f | grep:

less +F /var/log/myapp/app.log

It first enters follow mode; after pressing Ctrl+C to pause, you can search with / and page up and down, then press Shift+F to return to following. The downside is weak filtering capability, making it suitable for manual browsing rather than scripted filtering.

journalctl: Slice by Unit, Time, and Priority

When a service is managed by systemd, journalctl's advantage is not “following” but “querying with an index.” The common entry point is the unit:

journalctl -u myapp.service -f

This command follows new logs for that unit. To query a historical time window:

journalctl -u myapp.service \
  --since '2024-05-01 09:00:00' \
  --until '2024-05-01 10:00:00'

--since and --until also accept relative times, such as --since '1 hour ago' and --since today. When troubleshooting, first lock down the time window, then add priority:

journalctl -u myapp.service -p warning --since '30 min ago'

-p filters by syslog priority, and warning includes more severe levels. Note that the application must pass level information in as syslog priority or journal fields for -p to be meaningful; if the application only writes WARN in the message body, -p will not recognize it, and you still need -g or a pipe to grep.

Newer versions of journalctl support -g/--grep for regex filtering on messages:

journalctl -u myapp.service -g 'trace_id=abc123' --since '1 hour ago'

When structured output is needed, use -o json, then pass it to jq:

journalctl -u myapp.service -o json --since '1 hour ago' \
  | jq -r 'select(.MESSAGE | contains("timeout")) | [.__REALTIME_TIMESTAMP, .MESSAGE] | @tsv'

There are two caveats here: journal field names are subject to the systemd version's documentation and may differ between versions; -g is also not supported by all older versions. In production, first confirm with journalctl --version, or fall back to journalctl ... | grep.

journalctl also has several switches commonly used during troubleshooting: -b shows only the current boot, -b -1 shows the previous boot; -k shows kernel messages; -n 100 shows only the last 100 lines; --no-pager avoids entering a pager and facilitates pipeline processing; -o short-iso-precise outputs ISO timestamps with milliseconds for time comparison with other systems; _PID=1234 or _SYSTEMD_UNIT=... filters by field.

Blind Spots of Both Approaches

First, the file-log side. tail -F solves the problem of reopening after rotation, but does not solve: history is scattered across .1, .2, and .gz, and filtering by time across files is not intuitive; grep works by line, so multiline events such as Java stack traces and Python tracebacks are easily cut apart; there is no unified priority field, so you can only guess ERROR and WARN by regex, and the rules break as soon as the log format changes; if write buffering is enabled on the application side, logs may not appear in the file for a long time, and no matter how fast tail is, you cannot see them; in containers where PID 1 is not the application and logs are collected by a sidecar, directly tailing files on the host may not be the same copy as the actual service.

Now the journalctl side. It is also not true that “all logs on the system can be queried”: it covers only logs that enter the journal, and parts where the application writes files itself and does not hand stdout/stderr to systemd are invisible; by default it may not be persistent—on many distributions, when /var/log/journal does not exist, only runtime logs are kept and lost on reboot—so you need to confirm the Storage= configuration; systemd-journald rate-limits large amounts of logs in a short period, and messages may be dropped under burst traffic, so you need to check the RateLimitIntervalSec and RateLimitBurst settings; old logs are cleaned up by journalctl --vacuum-time, --vacuum-size, or disk pressure, so a time window from long ago may already have no data; the receipt time recorded by the journal may differ from the time the application itself prints, and across time zones, bulk imports, or container clock skew, the boundaries of --since/--until should be based on the actual fields; macOS, most container base images, and Windows do not apply, and in those cases it is still files plus grep.

An Actionable Decision Path

The choice can be reduced to the following steps:

1. First ask whether the service is a systemd unit. If yes, try journalctl -u <unit> -f first; if not, look directly at the log file configured by the application.
2. Do you need real-time following or time-based lookback? For real-time following: use tail -F for files and journalctl -f for the journal. For time-based lookback: use --since/--until for the journal; for files, use grep by time prefix, or awk/sed to extract by format.
3. Do you need to filter by priority, unit, or PID? If yes, prefer journalctl's -p, -u, and _PID=; if not, grep on the file side is more direct.
4. Do you need structured fields? If yes, use journalctl -o json with jq; on the file side, you can only rely on the application outputting JSON and then use jq.
5. Are logs rotated, compressed, rate-limited, or cleaned up? On the file side, pay attention to tail -F and .gz; on the journal side, pay attention to persistence, rate limit, and vacuum.

A practical combination is: first use journalctl to pin down “which point in time, which unit, what level,” then use file logs to inspect the detailed internal context of the application. The two are not substitutes, but an index and a body of text.

Several Commands You Can Use Directly

View service errors in real time while excluding health-check noise:

journalctl -u myapp.service -f -p warning

Filter file logs in real time while preserving surrounding context:

tail -n 300 -F /var/log/myapp/app.log \
  | grep --line-buffered -C 2 -E 'ERROR|Exception|timeout'

Query the full trace of a request ID in the journal:

journalctl -u myapp.service --since '2 hours ago' --no-pager \
  | grep 'request_id=abc123'

Query records in file logs for a certain hour range (assuming the line starts with a time such as 2024-05-01 14:):

grep '^2024-05-01 14:' /var/log/myapp/app.log

If you need precision down to the minute, grep is not enough; you need to write awk conditions based on the log format. There is no universal one-liner that fits all log formats, which is why file logs always require “knowing the format.”

One final reminder: when following logs, preserve the raw stream first. Filtering commands are good for locating issues, not as sole evidence; redirect the complete output of tail -F to a temporary file, or keep a copy with tee, then grep on the copy, and you will save a lot of trouble after troubleshooting.

tail -n 500 -F /var/log/myapp/app.log | tee /tmp/myapp-follow.log

This way, even if the grep rule misses key lines, the original record is still there.