How to Rotate Secrets in Live Production Without Causing an Outage
Retool's new app builder is where AI-generated code ships safely
Building apps with AI is easy. Getting them to production safely is another story.
The security team that never rotates production secrets knows exactly why: the last time someone tried, a service failed because a consumer was missed, the old secret was revoked before the new one propagated, or nobody had a complete inventory of which systems used the credential being rotated.
Secrets rotation is a coordination problem as much as a security one. The cryptographic replacement itself takes seconds. The failure modes are all operational: incomplete consumer inventory, wrong rotation sequence, no verification step between propagation and revocation, and no rollback plan when something breaks.
This guide covers the rotation sequence that prevents most rotation-induced outages, how to build the consumer inventory that makes that sequence possible, the specific patterns for API keys versus database passwords versus TLS certificates, and how automated rotation tools change the workflow without eliminating the coordination requirement.
The Rotation-Induced Outage: What Actually Goes Wrong
Most secrets rotation outages follow one of three patterns:
Pattern 1: Premature revocation. The new secret is generated and deployed to the primary application. The old secret is revoked. A secondary consumer -- a batch job, a monitoring agent, a reporting service -- was using the old secret and was not updated. That consumer now fails. If it fails silently (a background job with no alerting), the failure may not be discovered for hours.
Pattern 2: Incomplete propagation. The new secret is deployed to the primary application but not to all replicas. In a horizontally scaled deployment, some instances continue using the cached old secret. When the old secret is revoked, those instances fail intermittently -- the requests that hit updated instances succeed, the requests that hit stale instances fail -- producing a partial outage that is difficult to diagnose.
Pattern 3: Missing verification. The new secret is deployed to all consumers. The old secret is revoked. But the new secret was generated with a typo, missing a permission, or was rotated in the wrong environment (staging secret deployed to production). The verification step -- actually calling the API or database with the new secret and confirming a successful response -- would have caught this. Skipping it means the first indication of the problem is service failure after revocation.
Each of these failures has the same root cause: the revocation step happened before complete, verified propagation. The safe rotation sequence prevents all three patterns.
The Pre-Rotation Consumer Inventory: The Step Most Teams Skip
Rotation cannot be safely executed without a complete inventory of every system consuming the secret being rotated. This inventory is the step that most teams treat as optional and that causes most rotation failures.
A secret consumer inventory has four columns: consumer name (the service, application, or job using the secret), consumer location (where the secret is stored in that consumer: environment variable, config file, secrets manager reference, or hardcoded), rotation trigger method (how the consumer receives the new secret: restart, config reload, or live reload via a secrets SDK), and verification method (how you confirm the consumer is using the new secret successfully).
For a database password being rotated, a typical consumer inventory might include: the primary application service (environment variable, requires restart), the reporting service (config file, requires restart), the data pipeline job (Kubernetes secret reference, requires pod restart), the monitoring agent (direct connection, requires agent restart), and the disaster recovery read replica connection (environment variable on the DR environment, requires separate deployment).
Build this inventory before rotation, not during. For services you do not control directly (third-party integrations that authenticate to your database), the inventory process may reveal that rotation is blocked by a dependency that cannot be restarted on your schedule -- which is itself a security finding worth documenting.
The inventory is also the rollback plan: if something breaks after revocation, you know which consumer to check first and what it takes to restore the old secret or redeploy the new one.
Briefings like this, every morning before 9am.
Threat intel, active CVEs, and campaign alerts, distilled for practitioners. 50,000+ subscribers. No noise.
The Four-Step Safe Rotation Sequence
The safe rotation sequence is not complex. What makes it safe is the verification step between propagation and revocation, and the explicit wait period that ensures propagation is complete before verification begins.
Step 1: Provision the new secret. Generate the new credential with the same permissions as the old one. Store it in your secrets manager (AWS Secrets Manager, HashiCorp Vault, Azure Key Vault, or equivalent). Do not revoke the old secret. At this point, both the old and new secrets are valid. This is the dual-credential window.
Step 2: Propagate to all consumers. Update every entry in your consumer inventory to use the new secret. For environment-variable-based consumers, this means redeploying or restarting each service. For secrets-manager-SDK consumers (services using the AWS SDK or Vault SDK to fetch secrets at runtime), trigger a cache invalidation or wait for the TTL to expire. For Kubernetes secrets, update the secret object and restart the affected pods. The key: propagate to every consumer before proceeding.
Step 3: Verify the new secret is active and functional in every consumer. Call a health check endpoint or run a validation query against each consumer. For a database password, run a test query against the database using the new credential. For an API key, make an authenticated API call and confirm a 200 response. Do not proceed to revocation until every consumer shows a verified successful response with the new secret.
Step 4: Revoke the old secret. Only after every consumer is verified, revoke the old credential. Update your secrets rotation log with the rotation date and the next scheduled rotation date.
Rotation Patterns for API Keys, Database Passwords, and TLS Certificates
Different secret types have different rotation mechanics, but all follow the same four-step sequence.
API keys: Most external APIs support multiple active keys per account. Use this to implement the dual-credential rotation pattern: generate the new key, propagate it to all consumers, verify each consumer is successfully authenticating with the new key, then delete the old key. For APIs that only support a single active key per account, the rotation window is riskier -- you must propagate and verify faster. For these cases, test the complete rotation sequence in a staging environment first to confirm propagation time, and schedule rotation during low-traffic periods.
Database passwords: Most databases support password rotation through an ALTER USER or equivalent command that can be applied without terminating existing connections. The new password is effective immediately for new connections, while existing connections using the old password remain open until they close. This allows you to propagate the new password to application services, wait for old connections to drain naturally, and verify new connections are using the new password before expiring the old one. AWS RDS with Secrets Manager automated rotation handles this pattern natively for MySQL, PostgreSQL, MariaDB, and Oracle.
TLS certificates: Certificate rotation has a different risk profile because expiration is the forcing function. The rotation sequence for certificates is: generate the new certificate and private key pair (or request from your CA), deploy the new certificate to all servers that present it, verify TLS handshakes succeed with the new certificate (using curl or openssl s_client against each server), and let the old certificate expire naturally (no revocation step required for expiration). CRL or OCSP revocation is only needed when a certificate is compromised before expiration.
Automated Rotation: What It Changes and What It Does Not
AWS Secrets Manager, HashiCorp Vault, Azure Key Vault, and Google Cloud Secret Manager all support automated rotation. Automated rotation executes the same four steps -- provision, propagate, verify, revoke -- but through a Lambda function or rotation script rather than manual steps.
What automated rotation changes: the rotation trigger (scheduled rather than manual), the provisioning step (performed by the Lambda function rather than a human), and the documentation burden (rotation events are logged automatically). What it does not change: the need for a complete consumer inventory, the propagation step (consumers still need to be designed to fetch new secrets rather than cache them indefinitely), and the verification step (the Lambda function runs the same validation queries you would run manually).
The most common automated rotation failure: a consumer caches the secret in memory and does not check for updates until restart. When automated rotation replaces the secret in the secrets manager, the caching consumer continues to use the old (now revoked) secret until it restarts. Fix: use the AWS Secrets Manager SDK with the built-in refresh interval (default 5 minutes for the client-side cache) or design consumers to catch authentication errors and re-fetch the secret.
For HashiCorp Vault, dynamic secrets solve the caching problem by design: instead of rotating a static credential, Vault issues a new credential on each request with a short TTL (typically 1 hour for database credentials). When the TTL expires, the credential is automatically revoked. No manual rotation cycle is needed because credentials never persist long enough to accumulate rotation debt.
Building a Rotation Schedule That Actually Gets Followed
A secrets rotation policy that specifies "rotate every 90 days" without automation or a tracked schedule becomes a policy that is followed at audit time and ignored otherwise. A rotation schedule that gets followed has three properties:
Automation for everything automatable. Database passwords, API keys for services that support automated rotation (AWS, GitHub, Stripe), and TLS certificates should all be on automated rotation. Manual rotation should be reserved for secrets that cannot be rotated automatically: legacy systems without an API for credential management, third-party services that do not support multiple active credentials, and physical hardware authentication tokens.
A calendar or ticketing system entry for every manual rotation. Manual rotation that is not scheduled on a calendar gets deferred indefinitely. Create a recurring ticket or calendar event for each manually rotated secret, with the current rotation date and the next scheduled date. Tag the ticket with the consumer inventory and the verification method so the engineer executing the rotation has everything they need without research.
A rotation audit as part of security review cycles. At your quarterly security review, run a query against your secrets management system to identify credentials that have not been rotated within their scheduled period. Treat any credential that is overdue for rotation the same way you would treat an open vulnerability: assign an owner and a due date. The audit also catches newly created credentials that were never added to the rotation schedule.
The practical reality: full rotation coverage typically takes 6 to 12 months to achieve in an organization that is starting from minimal rotation practice. Prioritize by risk: credentials with access to customer data or production infrastructure first, internal tooling and staging credentials second.
The bottom line
Secrets rotation outages are caused by operational failures, not cryptographic ones. The credential replacement itself is trivial. What causes outages is skipping the consumer inventory (missing a dependent service), skipping the verification step (discovering the new secret has a problem only after revoking the old one), or executing propagation and revocation too close together without a verification window. The four-step safe rotation sequence -- provision, propagate, verify, revoke -- prevents all three failure patterns. Automated rotation tools execute this sequence more reliably than manual processes, but they still require consumers to be designed to fetch new secrets rather than cache them indefinitely. Build the consumer inventory before each rotation, verify before revoking, and automate everything automatable. The goal is a rotation cadence that is executed reliably enough that a credential compromise does not represent a months-long exposure window.
Frequently asked questions
How do you rotate a secret in production without causing downtime?
Use the dual-credential rotation sequence: first provision the new secret without revoking the old one, then propagate the new secret to every consumer service in your inventory, then verify each consumer is successfully using the new secret by making an authenticated test call, and only then revoke the old secret. The verification step -- actively confirming the new credential works in every consumer before revoking the old one -- is the step that prevents most rotation-induced outages. Never revoke the old secret before completing propagation and verification.
What should a secrets consumer inventory include?
A consumer inventory lists every service, application, background job, and integration consuming the secret being rotated. For each consumer, document: the consumer name and team ownership, where the secret is stored (environment variable, config file, or secrets manager reference), how the consumer receives the new secret (restart, config reload, or live fetch via secrets SDK), and how you verify the consumer is using the new secret successfully. This inventory is built before rotation, not during. It also serves as the rollback plan if something fails after revocation.
What is the difference between automated rotation and dynamic secrets?
Automated rotation replaces a static credential on a scheduled basis using a rotation function that provisions a new credential, propagates it, verifies it, and revokes the old one. Dynamic secrets (used by HashiCorp Vault) eliminate static credentials entirely by issuing a new credential on each request with a short TTL (typically 1 hour for database credentials). When the TTL expires, the credential is automatically revoked. Dynamic secrets prevent credential accumulation and rotation debt by design -- there is no credential old enough to need rotating because credentials are ephemeral.
How do I rotate a database password without disconnecting existing sessions?
Most databases support changing a user's password without terminating existing authenticated sessions -- existing connections using the old password stay open until they close naturally, while new connections must use the new password. The rotation sequence: apply the new password to the database user, propagate the new password to application services, wait for old connections to drain (or force a rolling restart of application services), verify new connections are successfully authenticating with the new password, then optionally expire any remaining old-password sessions. AWS RDS with Secrets Manager automated rotation handles this native drain-and-verify pattern for supported database engines.
How often should API keys and database passwords be rotated?
NIST SP 800-57 and CIS Controls v8 both recommend rotating privileged credentials (those with access to production data or infrastructure) at least every 90 days, with 30 days recommended for credentials with broad data access. The practical rotation interval also depends on your threat model: credentials exposed to the internet or used by many consumers should rotate more frequently than internal service-to-service credentials in a private network. Rotate immediately upon any confirmed or suspected compromise, regardless of the scheduled rotation date. For credentials using dynamic secrets (Vault dynamic database credentials), standard rotation schedules do not apply because credentials are already ephemeral by design.
What causes most secrets rotation outages and how do I prevent them?
Three patterns cause most rotation outages: premature revocation before propagation is complete (missing a consumer service that was still using the old secret), incomplete propagation in horizontally scaled services (some instances receive the new secret, others do not, producing intermittent failures after revocation), and missing verification (deploying a new secret with a typo or missing permission, then discovering the problem only after revoking the old one). Prevent all three by: building a complete consumer inventory before rotation, propagating to every consumer before proceeding, waiting for each consumer to complete its reload or restart, making a verified test call with the new credential against each consumer, and only then revoking the old credential.
How do I build a secrets rotation schedule that actually gets followed?
Automate everything automatable: database passwords for RDS or Cloud SQL, API keys for services supporting automated rotation (AWS, GitHub, Stripe, and most major platforms), and TLS certificates via cert-manager or Let's Encrypt with auto-renewal. For secrets that cannot be automated, create a recurring calendar event or ticketing system entry with the consumer inventory and verification steps attached -- manual rotation without a scheduled reminder gets deferred indefinitely. Add a rotation audit to your quarterly security review process: query your secrets manager for credentials overdue for rotation and treat them as open vulnerabilities with assigned owners and due dates.
Sources & references
Free resources
Critical CVE Reference Card 2025–2026
25 actively exploited vulnerabilities with CVSS scores, exploit status, and patch availability. Print it, pin it, share it with your SOC team.
Ransomware Incident Response Playbook
Step-by-step 24-hour IR checklist covering detection, containment, eradication, and recovery. Built for SOC teams, IR leads, and CISOs.
Get threat intel before your inbox does.
50,000+ security professionals read Decryption Digest for early warnings on zero-days, ransomware, and nation-state campaigns. Free, daily, no spam.
Unsubscribe anytime. We never sell your data.
Founder & Cybersecurity Evangelist, Decryption Digest
Cybersecurity professional with expertise in threat intelligence, vulnerability research, and enterprise security. Covers zero-days, ransomware, and nation-state operations for 50,000+ security professionals every morning.
Win a $2,495 Black Hat pass.
Full-access to Black Hat USA 2026 in Las Vegas. Subscribe free to enter.