In 2022, the Symbiote rootkit infected Linux servers across Latin American financial institutions — hiding processes, intercepting credentials, and living undetected for over a year. It worked by injecting itself into every running process via LD_PRELOAD, making standard tools like ps and netstat blind to its activity. That’s the rootkit problem in a nutshell: the OS you’re using to investigate has been compromised.
\n\n
How Rootkits Hide: Two Core Techniques
\n\n
Rootkits operate at two levels. User-space rootkits replace or hook system binaries and libraries — think a malicious ls that simply omits certain filenames. Kernel-space rootkits (LKMs, or loadable kernel modules) go deeper, patching the kernel’s own data structures so that the OS itself lies to you.
\n\n
The Diamorphine LKM rootkit is a textbook example. It hooks the getdents64 syscall to hide files prefixed with a magic string, and removes itself from /proc/modules so lsmod won’t list it. From inside the compromised host, you see nothing. The kernel is doing the hiding for the attacker.
\n\n
User-space rootkits are simpler but still effective. The LD_PRELOAD trick works by forcing a malicious shared library to load before any other, allowing it to intercept and modify the output of standard C library calls like readdir(). Any tool that relies on the standard library — which is almost everything — becomes unreliable.
\n\n
Detecting Rootkits with chkrootkit and rkhunter
\n\n
chkrootkit — a classic Unix rootkit scanner — checks for known binary tampering, suspicious network interfaces, and signs of LKM infection. Run it from a clean, statically-linked binary or a trusted live CD whenever possible.
\n\n
analyst@prod-web-01:~$ sudo chkrootkit\nCHECKING `ifpromisc'... eth0: PACKET SNIFFER(/sbin/dhclient[1142])\nCHECKING `bindshell'... INFECTED (PORTS: 31337)\nCHECKING `lkm'... You have 1 process hidden for readdir command\nCHECKING `chfn'... not infected\nCHECKING `login'... not infected\n\n
Two hits worth acting on immediately. Port 31337 flagged under bindshell means a process is listening on a well-known backdoor port — the attacker likely has persistent shell access. The lkm check confirms a process is being hidden by a kernel module. At this point, do not trust anything running on this host.
\n\n
Your next step: isolate the machine from the network, then boot from a trusted external medium. Use rkhunter for a second opinion — it checks file hashes against a known-good baseline.
\n\n
analyst@prod-web-01:~$ sudo rkhunter --check --sk\n[14:32:11] Checking for Diamorphine LKM [ Warning ]\n[14:32:11] Checking /sbin/ifconfig [ Warning ]\n[14:32:11] File hash: FAILED\n[14:32:11] Expected: a3f1c29e8b7d...\n[14:32:11] Found: 9d4e7f2a1c88...\n[14:32:11] Checking /bin/ps [ Warning ]\n[14:32:11] File hash: FAILED\n[14:32:12] Rootkit checks finished: 3 warnings\n\n
The hash mismatches on /sbin/ifconfig and /bin/ps confirm binary replacement — classic user-space rootkit behavior. The attacker swapped these binaries with trojaned versions that filter their output. Do not use these tools to investigate. Pull clean static binaries from your incident response toolkit instead.
\n\n
Cross-Checking with a Live Comparison: Volatility and /proc
\n\n
When you suspect kernel manipulation, compare what the kernel reports to what you can read directly from /proc. This technique bypasses any hooked system calls.
\n\n
analyst@ir-station:~$ sudo ss -tulnp | grep LISTEN\nTcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=812))
Tcp LISTEN 0 128 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=944))
analyst@ir-station:~$ sudo cat /proc/net/tcp | awk '{print $2}' | \
while read hex; do printf '%d\n' "0x${hex##*:}"; done | sort -un
22\n80\n31337\n\n
The ss command — which uses the kernel’s netlink socket interface, potentially hooked — shows only ports 22 and 80. Reading /proc/net/tcp directly reveals port 31337 is also open. The rootkit is hiding it from netlink but missed the raw proc entry. That discrepancy is your smoking gun.
\n\n
For memory forensics on acquired images, Volatility 3 can enumerate kernel modules and running processes from a memory dump without relying on the compromised OS at all — use linux.lsmod and linux.pslist plugins to cross-reference what the kernel actually has loaded versus what lsmod reported on the live system.
\n\n
What To Do Now
\n\n
Pick one production Linux host you manage and run rkhunter --propupd right now to record a clean file-hash baseline. Store the resulting rkhunter.dat file somewhere off the host — a read-only NFS share or your SIEM. Next time you run rkhunter --check, any tampered binary will fail its hash comparison immediately. Baselines you take before an incident are the ones that actually help you during one.
