Linux Forensics: Finding Attacker Traces After a Breach

In 2024, attackers exploiting CVE-2023-4911 (Looney Tunables) dropped backdoors on Linux hosts within minutes of initial access — and most teams had no idea until weeks later. When you inherit a potentially compromised server, you need a repeatable process for finding what the attacker left behind. Here is exactly that process.

Step 1: Find Suspicious Processes and Network Connections

Your first job is to understand what is running right now. Attackers frequently leave reverse shells, miners, or C2 beacons active. Start with ss — the modern replacement for netstat — combined with ps to correlate open connections back to specific processes.

# Run on the suspect host: prod-web01.internal (192.0.2.45)
$ ss -tulpn
Netid  State   Recv-Q  Send-Q  Local Address:Port  Peer Address:Port  Process
tcp    LISTEN  0       128     0.0.0.0:22            0.0.0.0:*          users:(("sshd",pid=812,fd=3))
tcp    LISTEN  0       128     0.0.0.0:80            0.0.0.0:*          users:(("nginx",pid=1024,fd=6))
tcp    ESTAB   0       0       192.0.2.45:443        192.0.2.201:54312  users:(("python3",pid=3847,fd=4))

$ ps -p 3847 -o pid,ppid,user,cmd
  PID  PPID USER     CMD
 3847  3846 www-data python3 /tmp/.cache/update

That third line is the problem. python3 owns an established HTTPS connection to 192.0.2.201 — an external IP — and it is running as www-data from /tmp/.cache/update. Legitimate web server processes do not initiate outbound connections to random IPs. The hidden-ish path under /tmp is a classic attacker tell.

Your next move: kill the process immediately if you are doing live response, then capture the binary first. Run cp /proc/3847/exe /evidence/update.bin before you kill it — once the process dies, /proc/3847/ disappears. Then block 192.0.2.201 at your firewall and pivot to figuring out how it got there.

Step 2: Audit Authentication Logs for Lateral Movement

Attackers almost always touch /var/log/auth.log (Debian/Ubuntu) or /var/log/secure (RHEL/CentOS). Parsing these manually is painful. Use grep and awk to surface the signal fast.

# Find successful logins and their source IPs — last 7 days
$ grep "Accepted" /var/log/auth.log | awk '{print $1,$2,$3,$9,$11}' | sort | uniq -c | sort -rn

      8 Aug 18 02:14 jmorales 192.0.2.88
      1 Aug 19 14:32 deploy   192.0.2.10
      1 Aug 21 09:01 svcadmin 192.0.2.33

# Check what jmorales did after login
$ grep "jmorales" /var/log/auth.log | grep -E "sudo|su |COMMAND"
Aug 18 02:15:44 prod-web01 sudo: jmorales : TTY=pts/1 ; COMMAND=/bin/bash
Aug 18 02:16:02 prod-web01 sudo: jmorales : TTY=pts/1 ; COMMAND=/usr/bin/chmod 4755 /tmp/.cache/update

Eight logins from 192.0.2.88 at 2 AM is suspicious on its own. But the sudo entries confirm it: someone used jmorales credentials to run /bin/bash as root, then set the SUID bit on that same malicious binary from Step 1. That chmod 4755 means the binary can now execute as root regardless of who runs it.

Cross-check whether 192.0.2.88 is a known internal IP. If it is not in your asset inventory, you are likely looking at a compromised jump host or direct external access through a VPN credential that was stolen. Pull the ~/.bash_history for jmorales and check /etc/passwd and /etc/sudoers for any new accounts or privilege escalations added during that window.

Step 3: Hunt for Persistence Mechanisms

Attackers need to survive reboots. The most common persistence locations on Linux are crontabs, systemd units, SSH authorized keys, and LD_PRELOAD hijacks. Check all of them — do not assume because you killed the process the threat is gone.

# Check all user crontabs and system cron directories
$ for user in $(cut -f1 -d: /etc/passwd); do 
    crontab -l -u $user 2>/dev/null && echo "--- $user ---"; 
  done

--- www-data ---
*/5 * * * * /tmp/.cache/update --reconnect > /dev/null 2>&1

# Check for rogue SSH keys
$ find /home /root -name "authorized_keys" -exec cat {} \;
sssh-rsa AAAAB3NzaC1yc2EAAAA... attacker@kali

# Check systemd for new or modified units
$ find /etc/systemd /lib/systemd -name "*.service" \
  -newer /var/log/dpkg.log -ls
-rw-r--r-- root root 312 Aug 18 02:22 /etc/systemd/system/cache-update.service

Three persistence mechanisms in one sweep. The cron job re-launches the backdoor every five minutes. The rogue SSH key gives the attacker direct key-based access to root or www-data. The systemd service — created at 02:22, seven minutes after the initial login — survives reboots. Remove all three, then rotate every credential on the host. Do not just delete the files; audit what else touched the filesystem in that same two-minute window using find / -newer /tmp/.cache/update -ls 2>/dev/null.

What To Do Right Now

Pick one Linux host in your environment — preferably internet-facing — and run this single command to surface hidden outbound connections owned by unexpected processes:

$ ss -tulpn | grep ESTAB | grep -v -E "(sshd|nginx|apache2|mysqld)"

If anything shows up that you cannot immediately explain, treat it as a compromise until proven otherwise. Capture the process with cp /proc/PID/exe, hash it with sha256sum, and run the hash against VirusTotal. That ten-second check has caught real backdoors in production environments. Start there.