Kubernetes Admission Control: OPA Gatekeeper and Kyverno in Production

Admission controllers intercept requests to the Kubernetes API server after authentication and authorization but before the object is persisted to etcd. They come in two flavors: mutating (which can modify the object) and validating (which can only accept or reject). Built-in admission controllers like PodSecurity, NodeRestriction, and ResourceQuota handle baseline concerns. Dynamic admission controllers, registered as ValidatingAdmissionWebhook or MutatingAdmissionWebhook resources, let you plug in arbitrary policy logic. This is where Gatekeeper and Kyverno operate. The webhook receives an AdmissionReview, evaluates the request against policy, and returns an AdmissionResponse that either permits the operation, rejects it with a reason, or returns a patched object. The performance implications matter: every CREATE, UPDATE, and sometimes DELETE for matching resources blocks on this webhook. A slow or unavailable webhook can break the cluster control plane. Production deployments need horizontal scaling, leader election, and a failurePolicy decision (Fail blocks operations on webhook outage; Ignore allows them and creates a bypass risk). Most teams run failurePolicy: Fail for security-critical policies with high-availability webhook deployments.

OPA Gatekeeper uses Rego, a declarative policy language from the broader Open Policy Agent ecosystem. Rego is powerful, expressive, and lets you write policies that consider multiple resources, external data, and complex logical relationships. The cost is a learning curve; Rego is unlike anything most engineers have written, and effective policy requires understanding partial evaluation, rules, and Rego's particular take on logical AND/OR. Kyverno uses YAML for policy definition with a syntax that resembles Kubernetes manifests. Policies are dramatically easier to write and review for engineers already fluent in Kubernetes. The tradeoff is reduced expressiveness; complex multi-resource policies that are straightforward in Rego require workarounds in Kyverno. The practical decision framework: choose Kyverno if your team is Kubernetes-native and you want policy-as-code adoption across application teams, not just platform engineers. Choose Gatekeeper if you have policy needs that span beyond Kubernetes (Rego runs everywhere from Envoy to Terraform), if you need integration with the broader OPA ecosystem, or if you have engineers comfortable with Rego. Many mature platform teams run both, with Kyverno for high-volume baseline policies and Gatekeeper for the small set of complex policies that justify Rego.

Every production cluster should have these policies enforced from day one, regardless of which engine you choose. No privileged containers (privileged: true is essentially a root shell on the node). No hostPID, hostIPC, or hostNetwork (these break the container isolation boundary). No allowPrivilegeEscalation. No running as UID 0 unless explicitly justified. Required resource requests and limits (without limits, a single pod can starve the node; without requests, the scheduler cannot make sensible decisions). Image registry allowlist (only pull from approved registries; this blocks typosquatted and malicious images). No :latest tags and no images without digests in production (you cannot audit what you cannot pin). Required labels for ownership and cost attribution. Drop ALL capabilities by default and require explicit add for any capability. ReadOnlyRootFilesystem where the workload supports it. AppArmor or seccomp profiles for any workload handling untrusted input. RBAC restrictions on ServiceAccounts to prevent token mounting where not needed. This set blocks the majority of escape paths attackers use after gaining initial pod access.

Validating policies reject non-compliant resources. Mutating policies modify resources to make them compliant. The temptation is to mutate everything because it never breaks deployments. Resist this. Mutation hides problems from application teams; the developer never learns their manifest was non-compliant because the webhook quietly fixed it. Three years later when you migrate to a different policy engine or relax the mutation, every workload breaks. Mutation is appropriate for cross-cutting concerns that should not be every team's responsibility: injecting sidecar containers (service mesh, logging agents), adding standard labels like environment or cost-center, applying default resource limits to workloads in dev namespaces, defaulting imagePullPolicy. Use validation for security policy. If a workload requests privileged: true, the team needs to know their manifest was rejected and why. Both Gatekeeper and Kyverno support audit modes that report violations without blocking; use audit mode when rolling out new policies to discover existing violations, then flip to enforce once the backlog is cleaned up. Generation policies in Kyverno (auto-creating NetworkPolicy or ResourceQuota in new namespaces) are powerful for ensuring baseline controls exist by default.

Policies are code and need the same lifecycle as application code. Store them in Git, version them, review them in pull requests, test them in CI, and deploy them through GitOps (Argo CD or Flux). For Gatekeeper, the gator CLI runs constraints against test fixtures locally and in CI, catching policy bugs before they reach a cluster. Write fixture YAML for both should-pass and should-fail cases, run gator test on every PR, and block merges on test failures. The OPA Playground is invaluable for iterating on Rego logic outside of Kubernetes context. For Kyverno, the kyverno CLI provides equivalent functionality with kyverno test running test suites against fixtures. Both engines support dry-run modes where new policies log what they would block without actually blocking, giving you data to validate policy correctness before flipping enforcement. Roll out policies in stages: start with audit/warn mode in dev clusters, review violations with affected teams, fix legitimate workloads, then move to enforce in dev. Repeat the cycle in staging and production with longer audit windows for high-impact policies. Sudden enforcement of new policies across production is how platform teams lose application team trust and slow ZTA programs to a crawl.

Attackers who land in a cluster with admission control will probe for bypasses. The first attempt is usually ephemeral containers via kubectl debug; ephemeral containers are added to existing pods through a different API path and historically bypassed pod-level admission policy. Modern Gatekeeper and Kyverno policies handle ephemeral containers, but verify your policies match on ephemeralcontainers subresource. The second attempt is CronJobs or Jobs that create privileged pods; ensure policies match Job and CronJob templates, not just bare Pods. The third is exploiting the gap between webhook unavailability and failurePolicy: Ignore; an attacker who can DoS the webhook can sneak through non-compliant workloads. Run webhooks with high availability and alert on webhook latency or error rates. The fourth is webhook configuration tampering; an attacker with cluster-admin can delete the ValidatingWebhookConfiguration. Treat these resources as critical, monitor for changes via audit logs, and consider tools like Kyverno's policy reports or external policy verification that runs out-of-band. The fifth is exempt namespaces; admission control typically exempts kube-system and the gatekeeper-system namespace itself. Attackers who can create workloads in exempt namespaces bypass everything. Lock down who can create resources in exempt namespaces via RBAC, and audit the exemption list quarterly.