HashiCorp Vault Agent Sidecar for Kubernetes: Dynamic Secrets, Auto-Auth, and Eliminating Static Credentials from Containers
Retool's new app builder is where AI-generated code ships safely
Building apps with AI is easy. Getting them to production safely is another story.
Audited Kubernetes Secrets for a client's production cluster and found 23 distinct database passwords in plaintext base64-encoded Kubernetes Secrets, none of which had been rotated since they were initially created. The oldest password was 4 years old. The passwords were encoded in base64 in the Secrets (not encrypted), visible to anyone with get/list access to Kubernetes Secrets in the namespace (which included all namespace admins), and backed up in plaintext in the cluster etcd snapshots.
The migration to Vault dynamic secrets took 6 weeks across 8 applications. By the end, each application had database credentials that Vault automatically generated at Pod startup with a 24-hour TTL, automatically renewed while the Pod was running, and automatically revoked when the Pod was deleted. The oldest credential in the system became 24 hours old by definition. The 4-year-old passwords became a data point in the migration postmortem rather than an ongoing risk.
Policy design: least privilege Vault policies for each application
Vault policy design for Kubernetes workloads should follow the same principle as Kubernetes RBAC: each application gets a policy that allows access only to the specific secret paths that application requires. A single broad Vault policy granting read access to all database credentials defeats the purpose of dynamic secrets — a compromised application that can read all database credentials is nearly as damaging as a leaked Kubernetes Secret containing all database passwords.
Create one Vault policy per application rather than shared policies across application groups
Create individual Vault policies for each application using a naming convention that reflects the application's team and environment: policy name team-name/app-name/environment (payments/checkout-service/production). The policy allows only the specific Vault paths that application requires: path "database/creds/checkout-service-role" { capabilities = ["read"] } for the database credential and path "secret/data/payments/checkout-service/*" { capabilities = ["read"] } for the KV secrets the application needs. A compromised checkout-service Pod can authenticate to Vault and read only the checkout-service role's database credentials and its own KV secrets — not the credentials for any other service. Create the Vault Kubernetes auth role with the same specificity: bound_service_account_names=["checkout-service"] bound_service_account_namespaces=["payments-production"] policies=["payments/checkout-service/production"] which ensures the checkout-service service account in the payments-production namespace can only get the checkout-service policy, not any other application's policy. Enforce this policy creation pattern through a Vault governance policy or IaC template that prevents broad wildcard policies from being attached to Kubernetes-facing roles.
Use Vault namespaces to provide organizational isolation for secrets and policies across teams
Use Vault Enterprise namespaces (or Vault OSS admin namespaces) to isolate secrets, policies, and auth methods between organizational teams, preventing a compromised application in one team's namespace from accessing secrets belonging to another team's Vault namespace. Create a Vault namespace for each team: vault namespace create payments creates a namespace where all secrets, policies, and auth methods are completely isolated from other namespaces. Configure the Vault Kubernetes auth method within the team's namespace: vault auth enable -namespace=payments kubernetes and vault write -namespace=payments auth/kubernetes/config with the cluster credentials. The Vault Agent Injector communicates with the team's namespace by setting the annotation vault.hashicorp.com/namespace: payments on Pods — Vault Agent uses this namespace for all authentication and secret retrieval operations. Team namespace isolation means that even if an attacker obtained a Vault token from a compromised Pod in the payments namespace, that token has no access to the engineering or hr namespaces' secrets — the Vault namespace boundary is a stronger isolation boundary than a Vault policy alone.
Operational hardening: securing the Vault Agent Injector deployment
The Vault Agent Injector mutating webhook has elevated cluster access — it reads Pod annotations and injects containers into all Pods in annotated namespaces. Hardening the injector's own deployment is as important as configuring the secrets it injects. A compromised injector could inject malicious sidecar containers into every new Pod in the cluster, making it a high-value target for Kubernetes supply chain attacks.
Restrict the Vault Agent Injector to specific namespaces using namespace selector configuration
Configure the Vault Agent Injector's MutatingWebhookConfiguration to only intercept Pod creation events in namespaces that have opted into Vault injection, preventing the webhook from running against system namespaces (kube-system, kube-public) and namespaces that do not use Vault secrets. Edit the vault-agent-injector-cfg MutatingWebhookConfiguration and add a namespaceSelector that matches namespaces labeled vault-injection: enabled: namespaceSelector: matchLabels: vault-injection: enabled. Add the label to namespaces that use Vault: kubectl label namespace production vault-injection=enabled. System namespaces (kube-system, monitoring) should not have the label and therefore the Vault Agent Injector webhook does not run for Pods in those namespaces. This restriction reduces the injector's scope from cluster-wide to specific namespaces, limiting the blast radius if the injector itself were compromised, and preventing misconfigured Vault annotations in system namespace Pods from unexpectedly triggering injection attempts.
Configure Vault audit logging and monitor for unusual authentication patterns from Kubernetes service accounts
Enable Vault audit logging to a file or Splunk/SIEM destination and create detection rules for unusual authentication patterns that may indicate a compromised Pod or credential theft from the /vault/secrets/ volume. Enable Vault audit log: vault audit enable file file_path=/vault/audit/vault-audit.log which logs all authentication requests and secret retrievals in JSON format including the Kubernetes service account identity, namespace, and the specific secret paths accessed. Monitor for authentication requests from service accounts to Vault roles they are not normally associated with (a payments service account authenticating to the engineering Vault role), unusually high frequency of database credential generation (may indicate a crash loop that generates new credentials faster than normal), and authentication from Kubernetes service accounts outside their registered namespaces (possible service account token theft and replay from outside the cluster). Forward Vault audit logs to the SIEM via a Fluent Bit DaemonSet log collector and create correlation rules that join Vault auth events with Kubernetes Pod lifecycle events to detect Vault credential requests that do not have a corresponding Pod creation event — a possible indicator of credential theft and external replay.
Briefings like this, every morning before 9am.
Threat intel, active CVEs, and campaign alerts, distilled for practitioners. 50,000+ subscribers. No noise.
The bottom line
HashiCorp Vault Agent sidecar injection eliminates static credentials from Kubernetes Pods by generating dynamic secrets on demand and automatically renewing them throughout each Pod's lifetime. Install the Vault Agent Injector via Helm, configure the Kubernetes auth method with a reviewer service account token, and create Vault roles that map specific service accounts and namespaces to least-privilege Vault policies. Configure the Vault database secrets engine to generate new PostgreSQL or MySQL credentials for each Pod at startup with a 24-hour TTL and automatic Vault-side revocation on Pod deletion. Add vault.hashicorp.com/agent-inject annotations to existing Deployments and use agent-inject-template annotations to render secrets into application-compatible file formats. Restrict the Injector webhook to labeled namespaces using MutatingWebhookConfiguration namespace selectors. Enable Vault audit logging and monitor for unusual authentication patterns from Kubernetes service accounts in the SIEM. Migrate applications from Kubernetes Secrets to Vault injection by running both credential sources in parallel during testing and removing the Kubernetes Secret only after confirming successful Vault credential usage.
Frequently asked questions
How do I install and configure the Vault Agent Injector on a Kubernetes cluster?
Install the Vault Agent Injector using the HashiCorp Vault Helm chart which deploys the injector alongside a Vault server (or configures the injector to connect to an existing Vault server). Add the HashiCorp Helm repository: helm repo add hashicorp https://helm.releases.hashicorp.com && helm repo update. Install the Vault chart with injector enabled: helm install vault hashicorp/vault --namespace vault --set injector.enabled=true --set server.enabled=false --set injector.externalVaultAddr=https://vault.example.com for an existing external Vault server. The Helm chart creates: a Vault Agent Injector Deployment that runs the mutating webhook server, a MutatingWebhookConfiguration that intercepts Pod creation events and modifies Pods with vault.hashicorp.com/agent-inject: true annotations, and a ServiceAccount for the injector to call the Vault API. Verify the injector is running: kubectl get pods -n vault which should show the vault-agent-injector Pod in Running state. Verify the mutating webhook is registered: kubectl get mutatingwebhookconfigurations which should include vault-agent-injector-cfg. Test injection by creating a Pod with the vault.hashicorp.com/agent-inject: true annotation and verifying that kubectl describe pod shows the vault-agent-init init container and vault-agent sidecar container were added by the webhook.
How do I configure the Vault Kubernetes auth method to authenticate Pods?
Configure the Vault Kubernetes auth method to trust your Kubernetes cluster's service account tokens by providing the Kubernetes API server address, the cluster's CA certificate, and a service account token that Vault uses to validate incoming authentication requests. Enable the auth method: vault auth enable kubernetes. Retrieve the Kubernetes API server address, CA certificate, and a long-lived service account token (in Kubernetes 1.24+ you must explicitly create a long-lived token for Vault to use for validation): kubectl config view --raw --minify --flatten --output 'jsonpath={.clusters[].cluster.server}' and kubectl get secret vault-auth-token -o jsonpath='{.data.ca\.crt}' | base64 -d. Configure the auth method: vault write auth/kubernetes/config kubernetes_host=https://KUBERNETES_API_SERVER kubernetes_ca_cert=@ca.crt token_reviewer_jwt=@reviewer-token.jwt. Create a Vault role that maps a Kubernetes service account and namespace to a Vault policy: vault write auth/kubernetes/role my-app-role bound_service_account_names=my-app-service-account bound_service_account_namespaces=production policies=my-app-policy ttl=1h. This role allows the my-app-service-account in the production namespace to authenticate to Vault and receive a token with the my-app-policy permissions. The TTL=1h means the Vault Agent automatically renews the token before it expires, maintaining continuous secret access for the pod's lifetime.
How do I configure Pod annotations to inject Vault secrets into a container?
Configure Pod annotations in the Deployment spec's template.metadata.annotations section to instruct the Vault Agent Injector on which secrets to retrieve and where to write them. The minimum required annotations are vault.hashicorp.com/agent-inject: 'true' (enables injection), vault.hashicorp.com/role: 'my-app-role' (the Vault Kubernetes auth role to authenticate as), and vault.hashicorp.com/agent-inject-secret-database: 'database/creds/my-app-role' (the Vault secret path to retrieve, with database being the file name suffix). With these annotations, Vault Agent writes the secret to /vault/secrets/database in JSON format: {data: {username: v-my-app-abc123, password: A1b2C3d4}}. The application reads credentials from the mounted file at /vault/secrets/database. To customize the output format for applications expecting specific config file formats, add the template annotation: vault.hashicorp.com/agent-inject-template-database: |{{ with secret 'database/creds/my-app-role' }}DB_USER={{ .Data.data.username }} DB_PASS={{ .Data.data.password }}{{ end }} which renders the secret as an environment file that the application can source. The /vault/secrets/ volume is automatically mounted into all containers in the Pod by the Vault Agent Injector, making the rendered files accessible to the application container without any volume configuration in the Pod spec.
How do I configure the Vault database secrets engine for dynamic PostgreSQL credentials?
Configure the Vault database secrets engine for PostgreSQL to generate dynamic database credentials that are automatically created when a Pod starts and revoked when the Pod stops or the credential TTL expires. Enable the database secrets engine: vault secrets enable database. Configure the PostgreSQL connection: vault write database/config/my-postgres-db plugin_name=postgresql-database-plugin connection_url='postgresql://{{username}}:{{password}}@postgres.production.svc.cluster.local:5432/mydb' allowed_roles=my-app-role username=vault-admin password=vault-admin-password. The vault-admin user in PostgreSQL needs permission to CREATE USER and GRANT to generate dynamic credentials — create it in PostgreSQL with: CREATE USER vault_admin WITH SUPERUSER LOGIN PASSWORD 'vault-admin-password' or with more restrictive permissions that only allow user creation and role grants. Create the Vault database role that specifies the SQL statements for credential creation: vault write database/roles/my-app-role db_name=my-postgres-db creation_statements="CREATE USER \"{{name}}\" WITH PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" default_ttl=24h max_ttl=48h. Test credential generation: vault read database/creds/my-app-role which should return a new username and password — verify the credentials work by connecting to PostgreSQL with the returned values.
How do I debug Vault Agent injection failures when Pods fail to start?
Debug Vault Agent injection failures by examining the Pod's init container logs (the vault-agent-init container), the Vault audit log for authentication failure events, and the injector webhook logs. First, check if the Vault Agent init container ran and what it logged: kubectl logs POD_NAME -c vault-agent-init -n NAMESPACE. Common init container failures: Error authenticating: error looking up token means the Kubernetes auth role configuration does not match the Pod's service account or namespace — verify the Vault role's bound_service_account_names and bound_service_account_namespaces match the Pod's spec.serviceAccountName and namespace. Error reading secret: permission denied means the Vault policy attached to the role does not grant read access to the secret path — verify the policy with vault policy read my-app-policy and ensure it includes a path "database/creds/my-app-role" { capabilities = ["read"] } rule. For template rendering failures: vault-agent-init logs the template rendering error which typically identifies a typo in the Consul Template syntax or a missing field reference. If the Pod is stuck in Init:0/1 status, the init container failed — kubectl describe pod POD_NAME shows the init container exit code and last exit message in the Containers section. Check the Vault audit log (if enabled) for the authentication attempt from the Pod's IP address to identify server-side failures that are not fully reported in the init container logs.
How do I migrate an existing application from Kubernetes Secrets to Vault dynamic secrets?
Migrate an application from Kubernetes Secrets to Vault dynamic secrets by deploying the Vault Agent Injector, creating the Vault auth role and policy for the application's service account, adding Vault annotations to the Deployment, and updating the application to read credentials from the /vault/secrets/ file path rather than environment variables from the Kubernetes Secret. First, verify the application can read credentials from a file rather than environment variables — most database libraries support credential file configuration. If the application requires environment variables: use Vault Agent's exec mode with vault.hashicorp.com/agent-pre-populate-only: 'true' annotation to have the init container write secrets before the application container starts, then use an entrypoint script in the application container that sources the /vault/secrets/database environment file before starting the application process. Add the Vault annotations to the existing Deployment alongside the existing Kubernetes Secret references — this runs both credential sources in parallel during testing. Verify the application successfully reads Vault-injected credentials by checking application logs for database connection success with the dynamically generated credentials. After confirming successful Vault credential usage, remove the secretRef from the application container's env block and delete the Kubernetes Secret — the migration is complete when the application runs without any static secrets in Kubernetes.
How do I configure Vault Agent to handle secret renewal and what happens when renewal fails?
Configure Vault Agent secret renewal by understanding the renewal lifecycle and configuring appropriate response behaviors for renewal failure scenarios. Vault Agent continuously monitors the TTL of all retrieved secrets and automatically renews them when they reach the renewal threshold (typically 2/3 of the TTL elapsed). For a database credential with a 24h TTL, Vault Agent attempts renewal at approximately hour 16. If renewal succeeds, the credential's TTL is extended and the rendered file is updated with the new expiry. If renewal fails (Vault server unreachable, credential was revoked, network partition), Vault Agent behavior depends on the agent configuration: with exit_after_auth: false (the default sidecar configuration), Vault Agent continues retrying renewal until success or the TTL expires. When the TTL expires without renewal, the credential is revoked by Vault — the application encounters authentication failures when attempting to use the expired credential. Configure the Vault Agent to restart the Pod when credentials expire by using the template rendering's exec option: configure an exec block in the Vault Agent template that runs a command when the rendered file changes, such as calling the application's credential reload API if it supports hot credential reloading without restart. For applications that cannot reload credentials without restart, configure a Vault policy with a long max_ttl (7 days or 30 days) that gives the operations team time to restore Vault connectivity before credentials expire.
Sources & references
Free resources
Critical CVE Reference Card 2025–2026
25 actively exploited vulnerabilities with CVSS scores, exploit status, and patch availability. Print it, pin it, share it with your SOC team.
Ransomware Incident Response Playbook
Step-by-step 24-hour IR checklist covering detection, containment, eradication, and recovery. Built for SOC teams, IR leads, and CISOs.
Get threat intel before your inbox does.
50,000+ security professionals read Decryption Digest for early warnings on zero-days, ransomware, and nation-state campaigns. Free, daily, no spam.
Unsubscribe anytime. We never sell your data.
Founder & Cybersecurity Evangelist, Decryption Digest
Cybersecurity professional with expertise in threat intelligence, vulnerability research, and enterprise security. Covers zero-days, ransomware, and nation-state operations for 50,000+ security professionals every morning.
Win a $2,495 Black Hat pass.
Full-access to Black Hat USA 2026 in Las Vegas. Subscribe free to enter.