Kubernetes Security Hardening: Fixing the Cluster Where Everything Runs as Root

Kubernetes clusters inherited without deliberate security hardening consistently present the same four categories of misconfiguration, and they compound each other. A container running as root is more dangerous when it also has a NetworkPolicy-free path to the cluster API server. An over-privileged service account is more exploitable when containers have unrestricted network access to internal services.

Privileged containers are the most direct path from a container compromise to a node compromise. A container running with privileged: true has access to the host's kernel capabilities, device files, and namespaces. An attacker who achieves code execution in a privileged container can trivially escape to the underlying node, access other containers' filesystems, and potentially pivot to the control plane. Even without the privileged flag, containers running as root with the default capability set have capabilities like CAP_NET_ADMIN, CAP_SYS_PTRACE, and others that enable meaningful container escape paths.

Absent NetworkPolicies mean that every pod in the cluster can communicate with every other pod by default. In a Kubernetes cluster with no NetworkPolicies, a compromised frontend pod has direct network access to database pods, internal APIs, the cluster metadata service, and any other pods in any namespace. Real environments need east-west traffic controls that reflect the actual communication requirements of the application architecture, not a default-allow model.

Wildcard RBAC and cluster-admin bindings are the third category. The default Kubernetes RBAC configuration does not grant broad permissions, but in practice, clusters accumulate bindings that do. Service accounts with get, list, watch on all resources in all groups are common. ClusterRoleBindings to cluster-admin for CI/CD service accounts are common. These broad permissions mean that compromising a single application pod with an over-privileged service account gives an attacker enumeration or modification access to the entire cluster's resources.

HostPath volume mounts are the fourth category. A container with a hostPath mount to the node's root filesystem, Docker socket, or kubelet credential directory can read node-level secrets, modify node configuration, or interact with the container runtime in ways that compromise other containers running on the same node.

PodSecurityPolicy (PSP) was deprecated in Kubernetes 1.21 and removed in 1.25. The replacement is Pod Security Admission (PSA), implemented through the Pod Security Standards framework. PSA is built into the Kubernetes API server as an admission controller and does not require installing a separate admission webhook, which simplifies deployment significantly compared to PSP.

The three Pod Security Standard profiles are: Privileged (no restrictions, equivalent to no policy), Baseline (prevents the most dangerous settings: privileged containers, host namespaces, host path mounts, and dangerous capabilities), and Restricted (adds requirements for non-root containers, seccomp profiles, and a minimal capability set). The Restricted profile is the target state for most application workloads. The Baseline profile is appropriate for infrastructure workloads like logging agents and monitoring tools that need elevated permissions but not full privilege.

PSA is enforced at the namespace level using labels. The three label modes are: warn (adds a warning to API responses but allows the pod to create), audit (records a violation in the audit log but allows the pod to create), and enforce (rejects pods that violate the profile). The three modes map directly to the audit-before-enforce methodology: start with warn and audit to understand the baseline, then move to enforce. The label syntax for a namespace is: pod-security.kubernetes.io/enforce: restricted for enforcement and pod-security.kubernetes.io/audit: restricted for audit mode. Multiple modes can be active simultaneously at different levels.

Enabling the Restricted pod security profile is straightforward in configuration. The reality is that a substantial fraction of container images in production clusters cannot meet the Restricted requirement without modifications to the container image itself. Most default base images run processes as root (UID 0) unless explicitly configured otherwise, and many application images have startup scripts, file permission requirements, or listening port configurations that assume root execution.

The securityContext fields that implement non-root execution are: runAsNonRoot: true (prevents the container from running if the image would run as root), runAsUser (specifies the UID to run as), runAsGroup (specifies the GID), and allowPrivilegeEscalation: false (prevents setuid or setgid binaries from elevating privileges). The capabilities section removes capabilities from the default set: dropping ALL and then adding back only the specific capabilities needed.

For most commercial application workloads, the practical path is to add a non-root user to the Dockerfile (RUN addgroup -S appgroup && adduser -S appuser -G appgroup in Alpine-based images), change ownership of any directories the application needs to write to, configure the application to bind to a non-privileged port (above 1024) if it previously listened on port 80 or 443, and set USER appuser before the CMD or ENTRYPOINT instruction. For third-party images where you do not control the Dockerfile, check whether the vendor provides a non-root image variant, and if not, create a wrapper image that runs the third-party image under a non-root user with appropriate filesystem permissions.

NetworkPolicy resources are Kubernetes objects that define ingress and egress rules for pods based on labels, namespace selectors, and IP address ranges. Without any NetworkPolicy defined in a namespace, all pod-to-pod communication is permitted (default-allow). Once a NetworkPolicy selects a pod, that pod becomes subject to the policy, and any traffic not explicitly permitted is denied. This is the critical design behavior: applying a default-deny NetworkPolicy to a namespace immediately blocks all traffic for any pod that the policy selects.

The staged approach to NetworkPolicy rollout is essential to avoid production incidents. The sequence is: start by auditing actual traffic flows (using your CNI plugin's traffic logging if available, or temporarily enabling network flow logs), map the required communication paths for each application, create allow rules for each required path, apply the policies in audit mode if your CNI supports it (Cilium's DENY action with metrics provides this), verify that the allow rules cover all legitimate traffic, and only then apply the default-deny policy.

The default-deny manifest that should be applied to each namespace once allow rules are confirmed is a NetworkPolicy with an empty podSelector (matching all pods) and an empty ingress array (denying all ingress). A matching egress-deny policy should be applied for outbound traffic control. Apply these last, after the allow policies for each application are confirmed to be working. Never apply the default-deny as the first step in a namespace unless it contains no running workloads.

Kubernetes RBAC audit starts with enumerating all ClusterRoleBindings and RoleBindings to identify accounts with excessive permissions. The kubectl command kubectl get clusterrolebindings -o json | jq '.items[] | select(.roleRef.name == "cluster-admin")' lists every binding to the cluster-admin role, which grants unrestricted access to every API in the cluster. In most production clusters, the list will include some expected system accounts and at least a few unexpected ones: CI/CD service accounts, monitoring tools, and sometimes user accounts that were granted cluster-admin for a one-time task and were never revoked.

For non-cluster-admin bindings, look for roles with wildcards in the verbs, resources, or apiGroups fields. A role with verbs: [""] can perform any action on its resources. A role with resources: [""] can act on any resource type in its API group. The combination of wildcard verbs and wildcard resources in a role effectively grants cluster-admin for the specified API groups. These wildcard permissions are often created by infrastructure tools during installation and then never tightened.

The remediation approach for RBAC is to apply the principle of least privilege: for each service account and user account, determine what API actions it actually requires, create a new Role or ClusterRole with only those specific permissions, and replace the existing binding with a binding to the new least-privilege role. The Rakkess and kubectl-who-can tools from Aqua Security are useful for visualizing what permissions each service account currently has. Start with the CI/CD service accounts and monitoring agents that are most commonly over-privileged, and work through the list by risk impact.

Kubernetes Secrets are base64-encoded, not encrypted, in etcd by default. Any user who can read the etcd data directory or has get permissions on Secret objects can trivially decode the content. The first mitigation is enabling encryption at rest for etcd using the EncryptionConfiguration API, which encrypts Secret objects in etcd using AES-CBC or AES-GCM before writing them to disk. This protects against the scenario where the etcd data files are accessed directly (through a backup, through a node compromise, or through a misconfigured etcd API endpoint).

Encryption at rest does not protect against in-cluster access to Secret objects via the Kubernetes API. A service account with get permissions on Secrets in a namespace can read any Secret in that namespace via kubectl get secret or the API. The complete solution to this is moving secrets out of Kubernetes Secrets and into an external secrets manager: HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, or GCP Secret Manager. The External Secrets Operator provides a CRD-based interface for synchronizing secrets from these external systems into Kubernetes Secrets on demand, enabling fine-grained access control at the secrets manager level while maintaining compatibility with workloads that consume Kubernetes Secrets.

For teams not ready to migrate to an external secrets manager, the immediate steps are: enable etcd encryption at rest, audit RBAC to ensure only specific workloads have get permissions on Secrets in each namespace, and avoid mounting service account tokens as environment variables (use projected service account tokens with short TTLs instead).

Falco is the CNCF-graduated runtime security tool that monitors system calls from containers and generates alerts when application behavior deviates from expected patterns. Falco rules are written in a YAML-based condition language that evaluates system call metadata: process name, file path, network connections, user ID, container image, and namespace. The default Falco rule set covers the most common attacker behaviors observed in container environments.

The rules that have the highest signal-to-noise ratio in production clusters: Terminal shell in container (a shell being spawned in a running production container is almost always either an attacker or an operator debugging in production, and both scenarios warrant attention), Container running as root with unexpected network connections, Sensitive file read by unexpected process (reading /etc/shadow, /root/.ssh, or Kubernetes service account token files from a process that has no business doing so), and Unexpected outbound connections (containers making outbound connections to IPs that are not in their expected external dependency list). The drift detection rules that identify container filesystem modifications are also high-value for detecting active compromise.

Falco's output can be forwarded to SIEM platforms, Slack, PagerDuty, or any webhook-compatible destination via Falcosidekick. The operational challenge is alert volume: a busy cluster with many legitimate operations will generate significant Falco noise from the default rules without tuning. The recommended approach is to start with only the highest-confidence rules (terminal shells, sensitive file reads, privilege escalation attempts) and expand coverage gradually as the baseline is established and known-legitimate behaviors are allowlisted.

The hardening sequence is not arbitrary. Each step builds on the previous one, and starting in the wrong order creates either production incidents or security controls that can be bypassed because a prerequisite control is missing. The recommended sequence is: PSA audit mode first, then RBAC cleanup, then NetworkPolicies, then non-root migration last.

Start with PSA audit mode because it gives you immediate visibility into which workloads have the highest-risk security configurations without causing any disruption. Apply pod-security.kubernetes.io/audit: restricted and pod-security.kubernetes.io/warn: restricted to every namespace and run for one to two weeks. The audit log violations give you the workload-by-workload list of required changes. This data drives the prioritization of everything that follows.

RBAC cleanup comes second because it reduces the blast radius of any compromise that occurs during the remaining hardening steps. If a workload is temporarily exploitable during the non-root migration, an attacker who achieves code execution in that workload should face a service account with minimal RBAC permissions rather than one that can list secrets across the cluster.

NetworkPolicies come third because they require the most careful traffic flow analysis and the longest audit period. Start with the least sensitive namespaces (staging, development) to develop confidence in the process before applying NetworkPolicies to production namespaces. The default-deny policies for each production namespace should be the final step of the NetworkPolicy rollout.

Non-root migration comes last because it has the highest risk of breaking application functionality and requires coordination with development teams to update container images. Use the PSA audit data from step one to prioritize which workloads need immediate attention versus which can be addressed in the next release cycle.