Rotating API Keys Across a Distributed System Without Causing an Outage

The most common rotation mistake is generating a new key and deploying it to the services you remember, leaving the services you forgot still using the old key, which you then delete, causing outages.

Before creating the new credential, build a complete consumer inventory:

For AWS IAM access keys:

# Find all references to the access key ID across your infrastructure
KEY_ID="AKIAIOSFODNN7EXAMPLE"

# Search in AWS Systems Manager Parameter Store
aws ssm describe-parameters --query "Parameters[*].Name" | \
  xargs -I{} aws ssm get-parameter --name {} --with-decryption \
  --query "Parameter.Value" 2>/dev/null | grep -l "$KEY_ID"

# Search in Secrets Manager
aws secretsmanager list-secrets --query "SecretList[*].Name" | \
  xargs -I{} aws secretsmanager get-secret-value --secret-id {} \
  --query "SecretString" 2>/dev/null | grep -l "$KEY_ID"

# Search in EC2 user data and Lambda environment variables
aws lambda list-functions --query "Functions[*].FunctionName" | \
  xargs -I{} aws lambda get-function-configuration --function-name {} \
  --query "Environment.Variables" 2>/dev/null | grep -l "$KEY_ID"

Also search your git history, CI/CD environment variable configurations, Kubernetes secrets, Helm values files, and any configuration management system (Ansible, Chef, Puppet) you use.

For most credential types, you can have both old and new active simultaneously during the transition.

AWS IAM access keys: IAM allows up to 2 access keys per user simultaneously. Generate the new key while the old one is still active:

new_key=$(aws iam create-access-key --user-name SERVICE-USER \
  --query 'AccessKey.[AccessKeyId,SecretAccessKey]' --output text)
echo "New Key ID: $(echo $new_key | awk '{print $1}')"
echo "New Secret: $(echo $new_key | awk '{print $2}')"

Do not deactivate the old key yet. Both keys are now valid.

GitHub Personal Access Tokens: Create a new token with identical scopes. GitHub allows unlimited PATs. Both old and new tokens work in parallel.

Stripe / third-party API keys: Most SaaS platforms allow creating a new key before revoking the old one. If the platform only allows one active key, this rotation will cause brief downtime, factor in a maintenance window.

Database credentials (PostgreSQL/MySQL): Create a new database user with identical permissions rather than rotating the existing user's password:

CREATE USER new_service_user WITH PASSWORD 'new-password';
GRANT ALL PRIVILEGES ON DATABASE mydb TO new_service_user;
-- Old user remains active during transition

Deploy the new credential to your central secrets store first, then let services pick it up, do not inject it directly into individual services.

AWS Secrets Manager:

# Update the secret with the new value
aws secretsmanager put-secret-value \
  --secret-id /prod/service/api-key \
  --secret-string '{"api_key": "NEW_KEY", "old_api_key": "OLD_KEY"}'

# AWS Secrets Manager automatically versions this
# Services reading the secret will get the new value on their next read

HashiCorp Vault:

# Write new version, old version remains accessible at version N-1
vault kv put secret/prod/service api_key=NEW_KEY

# Services using vault agent will pick this up on next lease renewal
# Services using vault SDK will get new value on next read

Kubernetes Secrets:

kubectl create secret generic service-api-key \
  --from-literal=api_key=NEW_KEY \
  --dry-run=client -o yaml | kubectl apply -f -

# Pods reading from environment variables need restart to pick up new value
# Pods using mounted secret volumes get updates without restart (within ~1 minute)

Know your injection method, environment variables require a pod restart, mounted volumes update automatically. This difference determines your rollout sequencing.

Deploy the new credential to services in order of their dependencies, services that other services call must be updated first.

For Kubernetes deployments:

# Trigger rolling restart of deployments that use the secret as env var
kubectl rollout restart deployment/SERVICE-NAME -n NAMESPACE

# Monitor the rollout
kubectl rollout status deployment/SERVICE-NAME -n NAMESPACE

# Verify the new pod is using the new credential
kubectl exec -it POD-NAME -- printenv API_KEY | head -c 8

For Lambda functions:

# Update the environment variable
aws lambda update-function-configuration \
  --function-name FUNCTION-NAME \
  --environment Variables={API_KEY=NEW_KEY}

# Lambda deploys immediately, running invocations use old key until complete
# No cold start required; the environment update takes effect on next invocation

For services with in-memory credential caches: Services that cache credentials at startup or on a TTL will continue using the old key until their cache expires or the service restarts. Check each service's credential refresh logic. If the cache TTL is 15 minutes and you need the old key revoked sooner, force a restart.

Do not disable the old key on a timer. Disable it only after you have confirmed all consumers have migrated.

For AWS IAM keys, monitor last used:

# Check when the old key was last used
aws iam get-access-key-last-used --access-key-id OLD_KEY_ID

# If LastUsedDate is more than 5 minutes ago and activity was before rotation time,
# all services have migrated

For API keys with access logs: Check your API gateway or the third-party service's usage dashboard. Filter requests by the old key, if the last request using the old key is more than 1 cache-TTL ago, rotation is complete.

Deactivate before deleting:

# Deactivate first (can be reactivated if something breaks)
aws iam update-access-key \
  --access-key-id OLD_KEY_ID \
  --status Inactive

# Monitor for 24 hours for any authentication failures
# If none: delete
aws iam delete-access-key --access-key-id OLD_KEY_ID

Deactivating before deleting gives you a 24-hour recovery window. If a service you missed starts failing, you can reactivate the old key immediately.