Security Log Analysis

Log Analysis covered the general mechanics of extracting and correlating data from logs with grep, awk, and journalctl — techniques that apply to debugging a slow endpoint just as well as to a security question. This article applies those same mechanics to a narrower, security-specific set of sources: authentication logs, sudo usage, fail2ban's ban history, and auditd's records, with the goal of answering a different kind of question — not "why is this endpoint slow," but "did anyone actually try to break in, and if so, what did they get."

What You Will Learn

By the end of this article you will be able to:

• identify the specific log sources that matter for a security investigation, on both Debian/Ubuntu and RHEL-family systems;
• recognize the log signature of a brute-force attempt, a successful login after failures, and privilege escalation;
• correlate fail2ban's ban history with the authentication log entries that triggered it;
• pull the same information from auditd when application-level logs are absent or insufficient;
• reconstruct a rough incident timeline from multiple sources agreeing on a timeframe.

Where Security-Relevant Events Actually Live

Starting in the wrong file is a common time sink during an actual investigation — on a RHEL-family system, grepping auth.log finds nothing because the file doesn't exist there at all; the same information is in /var/log/secure. Confirming which log family a given distribution uses, as covered generally in Linux Logs, is the first step before writing a single grep.

Recognizing an SSH Brute-Force Pattern

grep 'Failed password' /var/log/auth.log | tail -n 5

Jul 29 03:14:02 web01 sshd[4021]: Failed password for root from 198.51.100.7 port 51422 ssh2
Jul 29 03:14:03 web01 sshd[4023]: Failed password for admin from 198.51.100.7 port 51430 ssh2
Jul 29 03:14:04 web01 sshd[4025]: Failed password for oracle from 198.51.100.7 port 51438 ssh2

Failures against multiple usernames (root, admin, oracle) from the same source, seconds apart, is the classic signature of an automated scan trying common account names — not a person mistyping their own password, which would show repeated failures against a single, consistent username. Counting failures per source address quickly separates the two:

grep 'Failed password' /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head -n 5

 847 198.51.100.7
  312 203.0.113.44
   3 192.0.2.19

An address with 847 failures in a day and one with 3 warrant very different levels of attention — 3 failures is consistent with an ordinary mistyped password, 847 is not.

Finding a Successful Login That Followed Failures

The event that actually matters is not the failed attempts themselves — fail2ban and pam_faillock are already designed to contain those — it's whether any of them was eventually followed by a success from the same source:

grep -E 'Failed password|Accepted (password|publickey)' /var/log/auth.log | grep -A1 -B0 '198.51.100.7' | tail -n 10

Jul 29 03:14:59 web01 sshd[4099]: Failed password for root from 198.51.100.7 port 51899 ssh2
Jul 29 03:15:02 web01 sshd[4101]: Accepted password for root from 198.51.100.7 port 51910 ssh2

An Accepted password line immediately following a burst of Failed password lines from the same address, especially for a privileged account like root, is a genuine incident indicator, not routine noise — this is the exact pattern that turns "we got scanned, as every internet-facing server constantly does" into "we need to investigate what that session actually did."

Correlating with fail2ban's History

If fail2ban is running (see fail2ban), its own log confirms whether the offending address was already known and banned, and exactly when:

grep '198.51.100.7' /var/log/fail2ban.log

2026-07-29 03:14:10,442 fail2ban.actions [812]: NOTICE  [sshd] Ban 198.51.100.7
2026-07-29 03:22:47,105 fail2ban.actions [812]: NOTICE  [sshd] Unban 198.51.100.7

If a successful login from that address (found above) happened before the Ban timestamp, fail2ban reacted correctly but too late to prevent that one session — worth noting in an incident report, since it changes the recommended fix (a shorter findtime/lower maxretry, tightened in fail2ban's configuration) from "fail2ban isn't working" to "fail2ban's threshold needs tuning."

Auditing Sudo and Privilege Escalation

A compromised low-privilege account is a smaller problem than a compromised account that also has sudo access. Checking what an account actually did with elevated privileges:

grep 'sudo:.*COMMAND' /var/log/auth.log | grep 'compromised_user'

Jul 29 03:16:04 web01 sudo:  compromised_user : TTY=pts/1 ; PWD=/home/compromised_user ; USER=root ; COMMAND=/usr/bin/cat /etc/shadow

This single line is far more informative than "the account was compromised" in isolation — it identifies exactly what the attacker attempted (reading /etc/shadow, almost certainly to attempt offline password cracking on every account on the box), giving a concrete, prioritized next step: rotate every credential on the system, not just the one account that was initially compromised. If sudo I/O logging was enabled beforehand, sudoreplay reconstructs the full session, not just the command line.

When Application Logs Aren't Enough: auditd

Application-level logs only record what the application chose to log. If a question requires more — "was this specific file actually read, regardless of which process did it" — auditd's kernel-level record, covered in auditd Basics, is the source that can answer it when auth.log/secure can't:

sudo ausearch -f /etc/shadow -ts today -i

type=SYSCALL msg=audit(07/29/2026 03:16:04.201:5521) : ... auid=compromised_user uid=root exe=/usr/bin/cat

Confirms independently, at the kernel level, that /etc/shadow was in fact opened by that session — valuable corroboration when building a timeline for an incident report that needs to withstand scrutiny.

Reconstructing a Rough Timeline

An incident investigation is rarely answered by one log line in isolation — it's answered by several independent sources agreeing on a sequence of events. Pulling matching timestamps from each source into one ordered view:

{
    grep '198.51.100.7\|compromised_user' /var/log/auth.log
    grep '198.51.100.7' /var/log/fail2ban.log
} | sort -k1,3 -M

Jul 29 03:14:02 web01 sshd[4021]: Failed password for root from 198.51.100.7 ...
Jul 29 03:14:10,442 fail2ban.actions [812]: NOTICE  [sshd] Ban 198.51.100.7
Jul 29 03:15:02 web01 sshd[4101]: Accepted password for root from 198.51.100.7 ...
Jul 29 03:16:04 web01 sudo:  compromised_user : ... COMMAND=/usr/bin/cat /etc/shadow

Reading this merged sequence top to bottom tells the actual story: the successful login (03:15:02) happened before fail2ban's ban took effect based on the earlier failures, meaning the attacker got in during the exact window fail2ban's threshold hadn't yet closed — a specific, actionable finding that reading any one log in isolation would have missed.

Common Mistakes

• Checking /var/log/auth.log on a RHEL-family system. The file doesn't exist there; the equivalent is /var/log/secure, or journalctl -u sshd on either family.
• Treating every failed login as equally significant. A handful of failures against one consistent username is routine; a burst against many usernames from one source is a scan, and either pattern followed by a success is the event that actually warrants urgent action.
• Investigating only the account, not what it did with sudo. A compromised low-privilege account that never escalates is a much smaller incident than one that read /etc/shadow or modified /etc/sudoers — the sudo log line is what distinguishes the two.
• Relying on a single log source for an incident timeline. As the timeline example shows, auth.log alone missed the timing relationship between the ban and the successful login; combining sources is what surfaced it.
• Assuming application logs record everything relevant. As covered above, some questions (was this exact file read, by which process) can only be answered by auditd's kernel-level record.

Exercises

1. On a lab system's auth.log (or a sample log file), write a command that counts failed SSH login attempts per source IP and sorts by count, descending.
2. Explain, using the pattern shown in this article, how to distinguish an automated username-scanning attack from an ordinary user repeatedly mistyping their own password.
3. Given a fail2ban ban timestamp and a successful-login timestamp from the same IP, explain what it means for security posture if the login happened before the ban versus after it.
4. Describe, in your own words, a security question that auth.log cannot answer but ausearch can, and why.

Summary

Security log analysis reuses the same grep/awk/journalctl mechanics from general log analysis, aimed at a narrower set of sources — authentication logs, sudo records, fail2ban's history, and auditd's kernel-level trail — and a specific goal: recognizing attack patterns, confirming whether an attempt succeeded, and reconstructing a timeline from multiple sources that agree on a sequence of events. The final article in this module, Server Hardening, assembles every control covered so far — from least privilege through log analysis — into a single, ordered checklist for preparing a server for production.

References

• man7.org: sshd(8) — log message formats
• man7.org: ausearch(8)
• Fail2ban Official Documentation
• Red Hat: Reviewing Audit Records