Using Kubernetes API Server Audit Logs for Threat Detection and Hunting

Kubernetes audit logging is configured via an audit policy file (a YAML specification) that maps resource types and verbs to audit levels. The four audit levels: None (do not log), Metadata (log only request metadata), Request (log metadata and request body), and RequestResponse (log metadata, request body, and response body).

A policy that logs everything at RequestResponse produces unmanageable volume for any non-trivial cluster. An effective policy logs sensitive resources at RequestResponse and high-volume, low-security-value resources at Metadata or None.

Resources to log at RequestResponse level (full request and response bodies):

• secrets (all verbs) -- secret access is a primary attacker goal; you need the full response to detect mass enumeration
• rolebindings and clusterrolebindings (create, update, patch, delete) -- privilege escalation via role binding
• pods/exec and pods/attach (create) -- exec into running containers
• pods with privileged security context (create) -- privileged container creation
• serviceaccounts and serviceaccounts/token (create) -- service account token generation

Resources to log at Metadata level (high volume, lower security value):

• All other pod operations (create, delete without exec)
• ConfigMap operations
• Service and ingress operations

Resources to log at None (exclude from audit):

• events (watch) -- high volume, low security value
• pods/log (get) -- high volume for operational logging
• Readiness and liveness probe checks (health check endpoints)

The policy file is applied to the API server via the --audit-policy-file flag. In managed Kubernetes environments (EKS, GKE, AKS), audit logging is configured through the cluster management console or CLI rather than directly on the API server.

Kubernetes audit logs are written to a file path specified by the --audit-log-path API server flag or, for managed Kubernetes, exported to the cloud provider's logging service. The ingestion path depends on your environment:

Self-managed clusters: Audit logs are written to a local file (e.g., /var/log/kubernetes/audit.log). Use a log agent (Fluentd, Fluent Bit, or Filebeat) to tail the file and forward to your SIEM. Each log line is a JSON-formatted audit event. Parse the JSON and extract the key fields for SIEM ingestion: timestamp, user.username, user.groups, sourceIPs, verb, objectRef.namespace, objectRef.resource, objectRef.name, responseStatus.code, and requestURI.

AWS EKS: Enable control plane logging (API server, audit) in the EKS cluster configuration. Logs appear in CloudWatch Logs under the log group /aws/eks/cluster-name/cluster. Use CloudWatch Logs Insights or a CloudWatch Logs subscription to forward to your SIEM (or query directly in Athena if you export to S3).

GKE: Enable Cloud Audit Logs for the Kubernetes Engine API. Logs appear in Cloud Logging under cloudaudit.googleapis.com/activity. Forward to your SIEM via Pub/Sub export or query directly in BigQuery.

AKS: Enable diagnostic settings to export audit logs to a Log Analytics Workspace. Query with KQL in Microsoft Sentinel or forward to another SIEM via Azure Event Hub.

For all ingestion paths: confirm that the ingestion pipeline has a maximum acceptable latency (target: 5 minutes from event to searchable in SIEM). Detection rules that fire on real-time Kubernetes events need near-real-time log ingestion to be actionable.

Rule 1: Secret enumeration. Alert when a principal reads 10 or more secrets within a 5-minute window in the same namespace or across namespaces. Query: verb = "get" AND objectRef.resource = "secrets" | group by user.username | count > 10 in 5 minutes. The threshold of 10 distinguishes automated secret enumeration from a developer who reads 2 or 3 secrets during normal operations.

Rule 2: ClusterRoleBinding creation granting cluster-admin. Alert when a ClusterRoleBinding is created that references the cluster-admin ClusterRole. This is one of the highest-impact privilege escalation actions in a Kubernetes cluster. Query: verb = "create" AND objectRef.resource = "clusterrolebindings" AND requestObject.roleRef.name = "cluster-admin". Any result is a true positive in most environments.

Rule 3: Exec into a running container. Alert when a pod exec or attach event is recorded for a production namespace, especially by service account principals (automated workloads) or from unexpected user principals. Query: verb = "create" AND objectRef.resource = "pods/exec" AND objectRef.namespace = "production". Exec from a service account principal (not a human user) is especially suspicious.

Rule 4: Privileged pod creation. Alert when a pod is created with securityContext.privileged = true or hostPID = true or hostNetwork = true. These are container escape prerequisites. In environments using PodSecurity Standards or OPA/Gatekeeper, these should be blocked -- an audit log showing they were allowed indicates a policy gap.

Rule 5: Service account token creation. Alert when a service account token is created via the TokenRequest API (verb = create, resource = serviceaccounts/token). Attackers who compromise a pod with limited permissions may attempt to generate a long-lived token for a more privileged service account. Any token creation in a production namespace by a non-CI/CD principal should be investigated.

Rule 6: Anonymous authentication to the API server. Alert when audit log events show user.username = "system:anonymous". Anonymous authentication should be disabled in production clusters. Any anonymous request that reaches the API server indicates that the --anonymous-auth=false flag is not set or was overridden.

Rule 7: Cross-namespace secret access. Alert when a principal in namespace A reads a secret in namespace B. Kubernetes RBAC can prevent this if properly configured -- but audit log detection catches cases where RBAC misconfiguration allows cross-namespace access. Filter: verb = "get" AND objectRef.resource = "secrets" AND user namespace != objectRef.namespace (where the user's service account namespace is derivable from the username pattern system:serviceaccount:namespace:name).

Rule 8: Workload identity modification of RBAC at cluster scope. Alert when a ClusterRole or ClusterRoleBinding is modified by a principal whose username starts with system:serviceaccount: (a service account rather than a human user or CI/CD system). Service accounts modifying cluster-level RBAC is almost always malicious -- it is the pattern of an attacker who has compromised a pod and is attempting to escalate privileges.

Raw Kubernetes audit log detection rules fire on legitimate activity if not tuned. The most common false positive sources and how to address each:

CI/CD service accounts. Deployment pipelines run as service accounts with elevated permissions that perform create, update, and delete operations at high frequency. Exclude known CI/CD service account names (e.g., system:serviceaccount:ci:jenkins) from most detection rules, or create a separate detection profile for CI/CD activity with a higher threshold for alert triggering.

Cluster autoscalers and node management. The cluster autoscaler, node provisioners, and managed node group controllers create and delete pods and modify resource limits at scale. Their service account identities should be identified and excluded from pod creation and deletion rules.

Monitoring and observability agents. Prometheus, Datadog, and similar monitoring agents list pods and secrets at high frequency. The list verb on pods and configmaps at high frequency is legitimate monitoring traffic. Tune the secret enumeration rule to target get (reading specific secrets by name) rather than list (listing all secrets in a namespace) if monitoring agents are causing false positives.

Namespace-scoped Helm releases. Helm stores release state in secrets (in newer Helm versions) or configmaps. A Helm upgrade operation reads and writes multiple secrets in rapid succession. If Helm is used extensively, tune the secret enumeration rule to exclude the Helm service account principal and secrets with names matching the sh.helm.release.v1.* pattern.

Beyond alerting on known-bad patterns, Kubernetes audit logs support proactive threat hunting -- querying historical data to find patterns of activity that a detection rule may not have been in place to catch.

Hunt 1: Historical anonymous access. Query the last 30 days of audit logs for user.username = "system:anonymous". Even if anonymous auth is now disabled, historical anonymous access may indicate a period when the cluster was reachable without authentication.

Hunt 2: Unusual user agents. The userAgent field in audit logs records the tool that made the API request (kubectl/1.28, kube-controller-manager, helm/3.12). Automated attacker tools have recognizable user agents. Query for user agents that do not match your known set: userAgent NOT IN ["kubectl/1.28*", "helm/3.12*", "kubelet*", "kube-controller-manager*"]. Unusual user agents warrant investigation regardless of what resource they accessed.

Hunt 3: Source IP anomalies. Query for API server requests from sourceIPs outside your known cluster IP ranges (pod CIDR, service CIDR, known jump host IPs, CI/CD runner IPs). An API server request from an IP outside these ranges may indicate direct API server access from outside the cluster -- a sign that either the API server is exposed to the internet or a pod has unusual external connectivity.

Hunt 4: High-privilege operations in unusual hours. Query for ClusterRoleBinding creation, secret writes, and pod exec events that occurred outside business hours. While legitimate operations can occur at any time, attackers who have persistent access tend to operate during off-hours when SOC coverage may be lower. Compare high-privilege operations in the last 30 days against the typical distribution by hour of day.

Kubernetes audit log detection coverage degrades over time if not actively maintained: new namespaces are added that detection rules need to cover, new service accounts are created that should be excluded from CI/CD filters, and cluster upgrades can change how certain operations appear in the audit log.

Three maintenance practices keep detection effective:

Namespace-aware rule updates. When a new production namespace is created, update namespace-scoped detection rules to include it. Keep a registry of namespaces that should be covered by each detection rule and review it quarterly against the current cluster namespace list.

Service account allowlist management. Maintain an explicit allowlist of service account identities that are excluded from detection rules (CI/CD, autoscalers, monitoring agents). Review the allowlist quarterly to remove service accounts that have been decommissioned and add new infrastructure service accounts that have been provisioned.

Audit policy coverage review. When a new resource type or API version is introduced in a cluster upgrade, review whether the audit policy covers it at the appropriate level. Kubernetes GA resources that should be audited at RequestResponse but are not in the audit policy represent detection gaps. Review the audit policy against the cluster's supported API resources (kubectl api-resources) annually.