Secrets Rotation Automation: Building a Pipeline That Actually Rotates

A working secrets rotation pipeline has four components and you need all of them. First, a secrets store that is the canonical source of truth: HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, or CyberArk Conjur. The store enforces access control, versioning, and audit logging. Without a canonical store, secrets sprawl into config files, environment variables in CI, and chat messages, and rotation becomes impossible because you do not know where the secret lives. Second, a rotation function: the code that generates a new credential, updates the target system (database, IAM user, SaaS API), and writes the new value back to the store. AWS Secrets Manager calls these rotation Lambda functions; Vault calls them database secret engines or rotate-root endpoints; you can also build them yourself as scheduled jobs. Third, a versioning and sync mechanism that exposes new credential versions to consumers without requiring them to be restarted simultaneously, which is what makes zero-downtime rotation possible. Fourth, audit logging that records who accessed which secret version and when, both for compliance and for incident response when you need to know which version was active at the time of a suspected leak. Skipping any one of these turns rotation into a manual process with automation wrapper, which is the worst of both worlds.

Database credentials are the canonical secrets rotation problem because they are high-value, widely consumed, and break things loudly when mishandled. Two patterns dominate. Dynamic secrets, pioneered by Vault, generate a fresh database user with a short TTL (typically 1 hour to 24 hours) on every request. The application asks Vault for credentials, gets a unique user and password, uses it for the duration of the TTL, and Vault revokes the user when the TTL expires. There is nothing to rotate because nothing is long-lived. The constraint is that the application must be Vault-aware and able to refresh credentials before expiry; legacy applications with hardcoded connection strings cannot use this pattern. The dual-credential window pattern handles legacy applications and shared credentials. The database has two users, alpha and beta. At any given time, all applications use alpha. To rotate: generate a new password for beta, update the secrets store, wait for the propagation window (typically 5 to 30 minutes depending on consumer refresh interval), verify all consumers have picked up beta credentials and are connecting successfully, then disable alpha. On the next rotation, beta is the old credential and alpha is regenerated. This gives you zero-downtime rotation for legacy stacks. AWS Secrets Manager's built-in RDS rotation Lambda implements exactly this pattern with the AWSCURRENT and AWSPENDING staging labels.

Database rotation is tractable because you control the database. API keys for external services (Stripe, Twilio, SendGrid, payment gateways, third-party data providers) are harder because the rotation interface, if one exists, is bespoke per vendor, and many vendors offer no programmatic key rotation at all. The pragmatic approach is to build a per-service rotation runbook stored alongside the secret in your secrets manager, and to track external secrets in a separate inventory with explicit rotation owners. For services with rotation APIs (AWS IAM access keys via CreateAccessKey/DeleteAccessKey, GitHub personal access tokens via the REST API, Stripe restricted keys, Datadog API keys), automate the rotation the same way you do database credentials, with a dual-credential window. For services without rotation APIs, document the manual procedure, schedule a quarterly review, and accept that rotation requires human action; track the last rotation date as a metric and alert when it exceeds policy. Service-to-service tokens (OAuth client secrets, mutual TLS certificates, JWT signing keys) should be rotated more aggressively (30 to 90 days) because they typically have broad scopes and are widely cached. For internal service tokens, prefer short-lived workload identities (SPIFFE/SPIRE, cloud workload identity federation, Kubernetes service account tokens with audience restrictions) over long-lived bearer tokens; this turns the rotation problem into an identity problem, which is easier to solve.

Secrets that are baked into container images, committed to Git, or stored in CI environment variable settings are not really secrets, they are public knowledge with extra steps. Every image layer is cached, every Git commit is replicated, every CI run log is searchable. The rule is simple: secrets enter the running process at runtime, from the secrets store, never sooner. Implementation patterns: Vault Agent runs as a sidecar or init container, authenticates the workload identity to Vault, fetches secrets, and writes them to a tmpfs volume or templates them into config files; the application reads from the file at startup and Vault Agent rewrites the file when secrets rotate. AWS Parameter Store and Secrets Manager integrate with ECS task definitions and EKS via the Secrets Store CSI Driver; the secret is materialized at task or pod start. Kubernetes External Secrets Operator (ESO) syncs from an external store into native Kubernetes Secret objects, which is convenient but means the secret lives in etcd; encrypt etcd at rest and tightly control RBAC on Secret resources, or use the CSI driver pattern that bypasses Secret objects entirely. For CI/CD, use OIDC federation: GitHub Actions, GitLab CI, and CircleCI all support exchanging a workload identity token for short-lived cloud credentials, which eliminates static cloud access keys in CI variables. The end state is that grep -r 'password' across your repos and images returns nothing meaningful.

One-size-fits-all rotation policy is wrong because the cost and risk of rotation varies dramatically by secret class. A working policy: database passwords on a 90-day rotation, service-to-service bearer tokens on 30 days, externally-issued long-lived API keys on 180 days with mandatory access review at each rotation, code signing keys and root CA keys on annual rotation with a planned overlap period so old signatures remain verifiable, encryption keys for data at rest on annual rotation with re-encryption on read or a key derivation hierarchy that avoids re-encrypting everything. Workload identities and short-lived tokens (TTL under 24 hours) do not need a rotation policy because they rotate on every use. Before rolling rotation out to production, do canary rotation. Pick one non-critical workload that uses the secret, rotate just that workload's credentials, monitor for connection failures, latency anomalies, and error rate spikes for 24 hours, then proceed. For shared credentials with wide blast radius, do a blast radius analysis first: query the audit log of your secrets store for every principal that has read the secret in the last 90 days, validate each consumer can handle rotation, and only then schedule. The goal is to make rotation routine and uneventful, not heroic. Heroic rotation events are how rotation policy dies the next year.