Container Runtime Security: Detecting Threats Inside Running Containers with Falco

Image scanning operates on a fundamentally different threat model than runtime security. A scanner examines the static contents of a container image: it reads the package manifest, compares installed packages against CVE databases, and reports on known vulnerabilities in the code that will run when the container starts. This is valuable for preventing known vulnerable software from reaching production, but the scanner's threat model assumes that the primary attack vector is known vulnerable packages. Real attacks frequently bypass this model entirely.

Web shell attacks are the canonical example. An attacker exploits a deserialization vulnerability or SQL injection in a web application to achieve remote code execution. The application is running in a container that passed all image scans because the vulnerability is in the application logic, not a CVE-tagged package. The attacker's first action inside the container is typically spawning a shell process (bash, sh, python -c, perl -e) from the web server process. This shell spawn is visible as a syscall event: the web server process calls execve() with a shell binary path. Falco detects this syscall pattern immediately and generates an alert regardless of the vulnerability class that enabled the code execution.

Credential harvesting from environment variables is another attack pattern that image scanning cannot detect. Containers commonly receive secrets through environment variables (database passwords, API keys, cloud credentials). An attacker with code execution inside a container can read environment variables from /proc/1/environ or through the process's own environment. Falco can detect reads of /proc/*/environ from unexpected processes or reads of sensitive environment variable files from processes that should not require them.

Privilege escalation attempts inside containers are detectable through syscall monitoring in ways that network-based detection cannot achieve. A process inside a container that calls setuid(), setgid(), or capset() is attempting to change its own privileges, which is unusual behavior for application processes. Similarly, a container process that attempts to access host filesystem paths (mounting host volumes, reading host /etc through a path traversal) generates syscall patterns that Falco's default ruleset flags. The syscall-level visibility layer is the only layer that can observe these behaviors without instrumenting the application code.

Falco's architecture has three main components: the kernel-level data source that captures syscall events from running processes, the Falco engine that evaluates detection rules against the event stream, and the output plugins that deliver alerts to downstream consumers.

The kernel data source is the foundational component and the most operationally sensitive. Falco needs visibility into kernel syscalls to observe what processes are doing at the OS level. Two mechanisms provide this visibility: a kernel module (kmod) that loads into the host kernel as a traditional Linux kernel module, and an eBPF probe that runs in the kernel's eBPF virtual machine. The kernel module approach requires loading compiled kernel code into the running kernel, which carries stability risk if the module encounters a kernel version mismatch or bug. The eBPF probe runs in a sandboxed environment with kernel verification that prevents the probe from crashing the kernel, making it significantly safer for production use.

The Falco engine reads the event stream from the kernel data source and evaluates each event against the loaded ruleset. Rules are written in a YAML-based syntax that expresses conditions over event fields: process name, parent process name, container image, file path being accessed, network destination, and dozens of other fields derived from the syscall context. When a rule condition matches, the engine generates an output message using the configured output template and dispatches it to the configured output channels.

Output channels define where Falco alerts go. The default channels include standard output (useful for log aggregation via container stdout), syslog, and a file output. For production deployments, Falcosidekick is the standard companion service that routes Falco alerts to an extensible set of destinations: Slack, PagerDuty, Elasticsearch, Splunk, AWS SNS, and over 50 other integrations. Falcosidekick also supports webhook output to custom endpoints, which enables response automation by calling a Kubernetes operator or Lambda function that takes containment action when specific alert types fire.

The choice between eBPF and kernel module deployment is the most consequential early decision in a Falco deployment, and the answer for most production environments is eBPF. Understanding why requires understanding the operational risk profile of each approach.

Kernel modules run in kernel space with full kernel privileges. A bug or incompatibility in a kernel module can crash the host, which in a containerized environment crashes all workloads on that node simultaneously. The kmod approach requires that the module be compiled against the exact kernel headers for the running kernel version, which creates a maintenance dependency: every kernel update potentially requires a new module compilation. In managed Kubernetes environments where the cloud provider controls kernel versions and updates, this dependency is particularly difficult to manage.

The eBPF probe runs in the kernel's eBPF virtual machine, which validates the probe code before loading it to prevent illegal memory accesses, infinite loops, and other behaviors that could destabilize the kernel. The verifier provides a safety guarantee that the kernel module approach does not. The CO-RE (Compile Once, Run Everywhere) capability in modern Falco eBPF probes further reduces operational overhead: the probe is compiled once with BPF Type Format (BTF) information that allows it to adapt to different kernel structures at load time without recompilation. This means a single eBPF probe binary works across kernel versions 5.8+ without modification.

For environments running kernels older than 5.8, the older eBPF probe (without CO-RE) is available but requires kernel headers at deployment time, similar to the kmod approach. Amazon Linux 2, which runs a 5.10 kernel backport, supports CO-RE eBPF without issues. RHEL 8 and its derivatives run kernel 4.18 with various RHEL-specific backports; CO-RE support depends on whether the specific RHEL release has BTF support compiled in, which varies by minor version.

For Kubernetes deployments, the practical deployment choice is the Falco Helm chart with the eBPF driver enabled. The DaemonSet deployment ensures one Falco pod per node, providing coverage across all workloads on all nodes. The Helm chart handles driver installation, privileges (Falco requires specific Linux capabilities: CAP_BPF, CAP_PERFMON, CAP_SYS_RESOURCE), and configuration management.

Falco ships with a default ruleset that covers the most common container attack patterns without any configuration. Understanding what the default rules cover, and more importantly what they do not cover, is essential for assessing detection gaps and prioritizing custom rule development.

The default ruleset catches shell spawning in containers (any execution of bash, sh, dash, zsh, or other common shells from a containerized process), writing to sensitive directories (/etc, /bin, /usr/bin, /usr/local/bin), reading sensitive files (/etc/shadow, /etc/passwd, SSH private keys in /root/.ssh/ or /home/*/.ssh/), and spawning unexpected network-facing processes. These rules provide immediate value with no customization, catching the most common post-exploitation behaviors.

The Falco rule structure uses five required fields: rule (a unique name), desc (human-readable description), condition (the filter expression evaluated against events), output (the alert message template with variable interpolation), and priority (EMERGENCY, ALERT, CRITICAL, ERROR, WARNING, NOTICE, INFORMATIONAL, DEBUG). An example custom rule that detects unexpected outbound connections from a web server container:

- rule: Web Server Unexpected Outbound Connection
  desc: Detects outbound TCP connections from web server containers to unexpected destinations
  condition: >
    outbound and
    container and
    container.image.repository contains "nginx" and
    not fd.sip in (trusted_outbound_ips)
  output: >
    Unexpected outbound connection from web server
    (user=%user.name container=%container.name
    image=%container.image.repository
    dest=%fd.rip:%fd.rport)
  priority: WARNING
  tags: [network, web]

Macros and lists are the abstraction mechanism for reducing duplication and maintaining context. A trusted_outbound_ips list defined separately from the rule body can be maintained by operations teams without modifying rule logic. Macros like container (expands to container.id != host) and outbound (expands to the syscall and direction conditions for outbound TCP connections) are defined in the Falco default ruleset and can be reused in custom rules.

The most impactful custom rules for each environment describe the expected process tree and network behavior of specific applications. A payment processing service that should only run a specific Java process, write to a specific log directory, and make outbound connections to a specific set of payment gateway IPs can be described in a small set of allowlist-style rules that alert on any deviation. This behavioral allowlisting approach catches both known attack patterns and novel attack techniques that do not match any signature, because the rule is based on normal behavior rather than malicious behavior.

Out-of-the-box Falco on a production Kubernetes cluster generates significant false positive volume because the default rules describe general attack patterns that legitimate workloads also trigger. A CI/CD runner that builds Docker images will spawn shells and write to filesystems in ways that the default rules flag. A database initialization container will read sensitive files during setup. Tuning Falco to reduce false positives without eliminating legitimate detections is the critical operational task after initial deployment.

The recommended tuning approach is exception-based allowlisting. Rather than modifying default rules, define exception lists and macros that exclude known-good behavior. Falco supports rule exceptions via the exceptions key in rule definitions (added in Falco 0.28):

- rule: Launch Suspicious Network Tool in Container
  exceptions:
    - name: known_network_tools_containers
      fields: [container.image.repository, proc.name]
      comps: [in, in]
      values:
        - ["registry.example.com/network-debug", ["curl", "wget", "nmap"]]

This exception structure is preferable to modifying the rule condition directly because it keeps the exception separate from the detection logic, making it easier to audit which exceptions exist and why. Exception fields should be as specific as possible: excepting a specific image repository combined with a specific process name is far better than excepting the process name alone across all containers.

For tuning at scale across a heterogeneous cluster, Falco's override mechanism allows environment-specific configuration files that extend or modify the base ruleset. A deployment pattern that works well in practice: maintain a falco_rules.yaml with the upstream default rules unchanged, a falco_rules.local.yaml with organization-wide exceptions and custom rules, and per-namespace or per-team configuration for highly specific tuning. This layered configuration is managed through the Helm chart's values.yaml and allows team-specific tuning without modifying the shared ruleset.

Falco deployed as a DaemonSet provides node-level syscall detection, but Kubernetes-specific threats also operate at the API server level: malicious or compromised workloads that call the Kubernetes API to escalate privileges, create new privileged pods, or exfiltrate secrets from the etcd store. Falco's Kubernetes audit log integration adds API-server-level detection by consuming Kubernetes audit events as an additional event source.

Configuring Kubernetes audit log integration requires configuring the Kubernetes API server to send audit events to a webhook that Falco's k8saudit plugin receives. The Helm chart includes the k8saudit plugin configuration. Once enabled, Falco evaluates rules against audit events in addition to syscall events. Default k8saudit rules detect: attempts to exec into a running pod (kubectl exec), creation of privileged pods, RBAC changes that grant broad permissions, and access to sensitive API paths. The combination of syscall detection (node-level) and audit log detection (API server-level) provides defense-in-depth across both attack surfaces.

Falcosidekick response automation converts Falco alerts into automated containment actions. The webhook output to a Kubernetes admission controller or operator enables responses like network policy isolation of the affected pod, pod deletion, or namespace isolation. A practical response automation pattern: Falcosidekick receives a CRITICAL priority alert for shell spawning in a production web server container, calls a webhook endpoint on a custom Kubernetes operator, and the operator creates a NetworkPolicy that isolates the pod from all egress traffic while preserving ingress for ongoing investigation. This prevents lateral movement without requiring human intervention in the seconds-to-minutes after initial compromise.

Alert routing through Falcosidekick should differentiate by priority and rule tag. CRITICAL and EMERGENCY alerts route to PagerDuty for immediate human response. WARNING and NOTICE alerts route to a SIEM or log aggregation system for investigation and trend analysis. This priority-based routing prevents alert fatigue in the on-call workflow while ensuring all events are preserved for investigation. The Falcosidekick configuration for multi-destination routing is straightforward through its config.yaml:

customfields:
  environment: production
  cluster: prod-us-east-1
slack:
  webhookurl: "https://hooks.slack.com/services/..."
  minimumpriority: "warning"
pagerduty:
  routingkey: "..."
  minimumpriority: "critical"
elasticsearch:
  hostport: "http://elasticsearch:9200"
  index: "falco-alerts"
  minimumpriority: "debug"

Falco provides strong coverage for runtime behavioral threats but does not replace other container security controls. Understanding what each layer of the container security stack covers and where the gaps are is essential for building a complete defense posture.

Image scanning covers known vulnerabilities in package dependencies. Admission control covers configuration-based risks (privileged containers, containers with excessive capabilities, containers that run as root). Network policy covers lateral movement between pods. RBAC and service accounts cover API-level privilege. Falco covers runtime behavioral anomalies: process execution, file system access, network connections, and system call patterns that deviate from expected application behavior.

The detection coverage mapping exercise for container runtime security: take the MITRE ATT&CK for Containers matrix and evaluate which techniques are covered by Falco default rules, which are covered by other controls, and which are not currently detected. Key coverage areas for Falco include: initial access via web shell (T1505.003), execution via container API (T1609), execution via command and scripting interpreter (T1059), privilege escalation via container escape (T1611), and discovery via container/host enumeration (T1613). Gaps commonly exist around supply chain attacks (image tampering before deployment), sophisticated lateral movement that uses legitimate Kubernetes API calls, and persistence mechanisms that operate at the image layer.

Testing Falco detection coverage requires generating the events that rules should detect in a controlled environment. Falco Test Suite and tools like Atomic Red Team for containers provide pre-built test cases that trigger specific detection rules. Running these tests confirms that rules are active and firing correctly, that the alerting pipeline from Falco through Falcosidekick to the destination is functioning, and that the rules actually cover the attack techniques they are intended to detect. Test coverage for runtime security detection should be part of the same cadence as purple team exercises for network and endpoint detection.