Terraform Remote State with S3 and DynamoDB Locking: Setup, State File Security, and Concurrent Run Prevention
Retool's new app builder is where AI-generated code ships safely
Building apps with AI is easy. Getting them to production safely is another story.
Lost a Terraform state file for a staging environment once — not through disk failure but through a well-intentioned engineer who ran rm terraform.tfstate to 'clean up' before re-running terraform init on a machine with local state. The infrastructure was still running. The state was gone. Rebuilding the state required three hours of terraform import commands, manual resource identification from the AWS console, and careful reconstruction of the dependency graph. S3-backed remote state with versioning would have made this a two-minute rollback.
That incident was the inflection point that made remote state non-negotiable for any Terraform project with more than one contributor. The S3 backend with DynamoDB locking solves three distinct problems: it prevents state file loss through S3 durability and versioning, it prevents concurrent corruption through DynamoDB locking, and it enables team collaboration by making state accessible to all authorized users and CI/CD pipelines from a shared location.
Backend configuration: S3 bucket hardening and encryption requirements
The S3 state bucket requires specific security configuration beyond the default S3 bucket settings. A default S3 bucket created through the console or a basic aws_s3_bucket resource has versioning disabled, no encryption, and potentially public access unless the account-level public access block is configured. All three defaults must be explicitly overridden for the state bucket to provide the durability and confidentiality that production Terraform state requires.
Enable S3 Object Lock or bucket versioning with MFA delete for maximum state file protection
Configure S3 versioning on the state bucket as the minimum protection against accidental state deletion, and optionally enable MFA Delete which requires multi-factor authentication to permanently delete object versions or disable versioning. MFA Delete prevents even an AWS administrator with full S3 permissions from permanently deleting state file versions without the MFA device — protection against both accidental deletion and insider threats. Enable MFA Delete using the AWS CLI with the root account (MFA Delete cannot be enabled from IAM users): aws s3api put-bucket-versioning --bucket your-company-terraform-state --versioning-configuration Status=Enabled,MFADelete=Enabled --mfa 'arn:aws:iam::ACCOUNT_ID:mfa/root-account-mfa-device TOKEN'. With versioning enabled, every terraform apply creates a new state version in S3, and previous versions are accessible for recovery by listing object versions: aws s3api list-object-versions --bucket your-company-terraform-state --prefix environments/production/terraform.tfstate.
Use AWS KMS customer-managed key encryption for state files containing sensitive resource attributes
Configure AWS KMS customer-managed key (CMK) encryption for the state bucket instead of S3-managed encryption (SSE-S3) when the Terraform state contains sensitive values such as database passwords, private keys, or API tokens that appear as plaintext in resource state. With KMS CMK encryption, access to state file contents requires both S3 bucket permissions and KMS decrypt permission on the specific key — two independent permission checks that an attacker must bypass to read the plaintext state. Create a KMS key with a key policy that grants decrypt permission only to the Terraform execution role: the key policy Deny for kms:Decrypt from any principal not in the approved role list complements the S3 bucket policy restriction. Configure the Terraform backend to use the KMS key: add kms_key_id = "arn:aws:kms:us-east-1:ACCOUNT_ID:key/KEY_ID" to the backend block and change the encrypt parameter value to true. Track KMS key usage in CloudTrail — every state file read by an unauthorized principal generates a KMS decrypt API call that appears in the CloudTrail event log.
State operations: inspecting, modifying, and recovering state
Terraform state operations beyond init, plan, and apply are required when resources drift from what Terraform expects, when resources are renamed or moved between modules, or when recovering from corrupted or incomplete state. The terraform state subcommands provide surgical access to individual resource records without requiring a full destroy and re-apply cycle. Used correctly they resolve state drift in minutes; used incorrectly they create new inconsistencies that compound the original problem.
Use terraform state mv to rename resources or refactor module structure without destroying and recreating infrastructure
Use terraform state mv to rename a resource in state when the Terraform configuration resource address changes due to refactoring, preventing Terraform from destroying the old resource and creating a new one. When refactoring a resource from module.network.aws_vpc.main to module.core_network.aws_vpc.main, running terraform plan without moving the state first shows the old resource scheduled for destruction and a new resource scheduled for creation — which would destroy and recreate the VPC. Instead, run terraform state mv module.network.aws_vpc.main module.core_network.aws_vpc.main which updates the state address without touching the actual infrastructure, after which terraform plan shows no changes. Use terraform state list to get the current full list of resource addresses before and after the move to verify the rename completed correctly. For moving resources between workspaces or state files, use terraform state pull to export state as JSON, terraform state push to import modified JSON, and terraform state mv -state=source.tfstate -state-out=destination.tfstate to move individual resources between state files.
Recover from state drift using terraform import to bring manually created resources under Terraform management
Use terraform import to bring resources that were created manually (outside of Terraform) under Terraform state management without destroying and recreating them. First write the Terraform resource block for the existing resource with the correct configuration, then run terraform import RESOURCE_ADDRESS RESOURCE_ID where RESOURCE_ID is the cloud provider's identifier for the resource: terraform import aws_security_group.main sg-0abc123def456789 imports the existing security group into the aws_security_group.main state address. After import, run terraform plan to verify that the imported resource's configuration matches the Terraform resource block — any differences appear as planned changes. Resolve differences by updating the Terraform resource block to match the actual resource configuration (rather than letting Terraform modify the resource to match the block) when the existing manual configuration is the intended state. terraform import does not generate a Terraform configuration block — it only creates the state entry. The resource block must be written manually before running import, which requires knowing the resource's configuration parameters from the AWS console or CLI.
Briefings like this, every morning before 9am.
Threat intel, active CVEs, and campaign alerts, distilled for practitioners. 50,000+ subscribers. No noise.
The bottom line
Terraform remote state with S3 and DynamoDB locking is the minimum viable state management configuration for any team environment or production infrastructure. Create the S3 bucket with versioning enabled, server-side encryption configured, and public access blocked. Create the DynamoDB table with the LockID partition key using PAY_PER_REQUEST billing. Configure the backend block with the bucket, key path, region, encrypt = true, and dynamodb_table parameters. Migrate existing local state with terraform init -migrate-state. Restrict state bucket access with an S3 bucket policy allowing only Terraform execution roles and denying all other principals including IAM administrators. Use workspace-based or key-path-based state isolation for multi-environment deployments. Diagnose lock errors by reading the DynamoDB table directly and force-unlock only after verifying no active apply is running. Treat the state bucket as one of the most sensitive assets in the AWS account given that state files contain plaintext sensitive resource attributes.
Frequently asked questions
How do I create the S3 bucket and DynamoDB table for Terraform remote state?
Create the S3 bucket and DynamoDB table for Terraform remote state using the AWS CLI or a bootstrap Terraform configuration that is applied once and never stored in the remote backend it creates. Using the AWS CLI: aws s3api create-bucket --bucket your-company-terraform-state --region us-east-1 to create the bucket, then aws s3api put-bucket-versioning --bucket your-company-terraform-state --versioning-configuration Status=Enabled to enable versioning, then aws s3api put-bucket-encryption --bucket your-company-terraform-state --server-side-encryption-configuration '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}' to enable at-rest encryption, then aws s3api put-public-access-block --bucket your-company-terraform-state --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true to block all public access. For DynamoDB: aws dynamodb create-table --table-name terraform-state-locks --attribute-definitions AttributeName=LockID,AttributeType=S --key-schema AttributeName=LockID,KeyType=HASH --billing-mode PAY_PER_REQUEST --region us-east-1. The LockID attribute name and String type are mandatory — any deviation causes Terraform to fail to acquire or release locks. After creating both resources, configure the Terraform backend block referencing them.
How do I configure the Terraform S3 backend block in my configuration?
Configure the S3 backend in a backend.tf file (or within the main terraform block) using the backend "s3" block with the required parameters. The minimal working configuration: terraform { backend "s3" { bucket = "your-company-terraform-state" key = "environments/production/terraform.tfstate" region = "us-east-1" encrypt = true dynamodb_table = "terraform-state-locks" } }. The key parameter is the S3 object path within the bucket — use a path that includes the environment and project to create isolated state per deployment: environments/staging/network/terraform.tfstate and environments/production/network/terraform.tfstate are separate state files for staging and production network infrastructure. The encrypt = true parameter requires that Terraform encrypt the state file using the bucket's server-side encryption configuration before uploading. Run terraform init after adding the backend block — if existing local state exists, Terraform prompts to migrate it to the remote backend with the -migrate-state option. After initialization, verify the backend is working with terraform plan which should show Acquiring state lock before computing the plan.
How do I isolate state for multiple environments using Terraform workspaces or separate state files?
Isolate multi-environment Terraform state using either separate S3 key paths per environment (recommended for production workloads) or Terraform workspaces (appropriate for ephemeral or test environments). For separate state files per environment, create distinct backend configurations for each environment: a production backend.tf pointing to key = environments/production/terraform.tfstate and a staging backend.tf pointing to key = environments/staging/terraform.tfstate. This requires either separate Terraform root modules per environment or Terraform partial backend configuration with -backend-config flags at init time: terraform init -backend-config='key=environments/production/terraform.tfstate'. For Terraform workspaces, a single backend configuration creates workspace-specific state paths automatically: when the workspace is named production, Terraform stores state at env:/production/terraform.tfstate in the S3 bucket. Create and switch workspaces with terraform workspace new production and terraform workspace select production. Workspaces are better suited for ephemeral environments (feature branches, temporary test environments) where the infrastructure is similar but isolated, rather than for permanent long-running environments where separate root modules provide cleaner separation of concerns and access control.
How do I diagnose and resolve a Terraform state lock error?
Diagnose a Terraform state lock error by reading the error message which includes the lock ID, the identity that holds the lock, the timestamp the lock was acquired, and the operation that acquired it. The error looks like: Error acquiring the state lock, Error message: ConditionalCheckFailedException: The conditional request failed, Lock Info: ID: abc123, Path: environments/production/terraform.tfstate, Operation: OperationTypeApply, Who: user@host, Created: 2026-07-14 10:30:00. If the lock is held by an active terraform apply in another terminal or CI/CD pipeline, wait for it to complete — force-unlocking an active apply corrupts state. Verify the lock holder by checking the DynamoDB table directly: aws dynamodb get-item --table-name terraform-state-locks --key ''{"LockID":{"S":"your-bucket/environments/production/terraform.tfstate"}}' which shows the lock details. If the lock was left by a crashed or canceled process with no active operation, run terraform force-unlock LOCK_ID where LOCK_ID is the ID from the error message. Force-unlock deletes the DynamoDB lock record, allowing the next terraform operation to acquire a fresh lock. Document force-unlock usage with a timestamp and reason in the team's incident log — repeated force-unlocks indicate a process issue where Terraform operations are being killed without proper cleanup.
How do I restrict access to the Terraform state S3 bucket?
Restrict Terraform state S3 bucket access using an S3 bucket policy and IAM policies that limit access to specific IAM roles and deny access to all other principals including the account root. The S3 bucket policy should include an explicit Deny for s3:GetObject and s3:PutObject from any principal not in the approved Terraform execution role list: { "Effect": "Deny", "Principal": "*", "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"], "Resource": "arn:aws:s3:::your-company-terraform-state/*", "Condition": { "StringNotLike": { "aws:PrincipalArn": ["arn:aws:iam::ACCOUNT_ID:role/terraform-execution-role", "arn:aws:iam::ACCOUNT_ID:role/ci-cd-role"] } } }. This explicit deny blocks even IAM administrators with s3:* permissions from reading state files unless they assume one of the listed roles. Enable S3 server access logging on the state bucket to record every access to the state file — access logs reveal if state was read by unauthorized principals. State files contain all Terraform-managed resource attributes including sensitive values like database passwords, private keys, and API tokens stored in plaintext in resource state; this makes the state bucket one of the most sensitive objects in the AWS account and warrants the strictest access controls.
How do I migrate existing local Terraform state to a remote S3 backend?
Migrate local Terraform state to a remote S3 backend by adding the backend configuration and running terraform init -migrate-state, which copies the local state to the remote backend without destroying any infrastructure. First, add the backend block to your Terraform configuration pointing to the S3 bucket and DynamoDB table. Run terraform init — Terraform detects that a backend configuration has been added and asks Do you want to copy existing state to the new backend? Enter yes to trigger the migration. Terraform copies the terraform.tfstate local file to the S3 key path specified in the backend configuration and removes the local state file (leaving a terraform.tfstate.backup for safety). Verify the migration succeeded by running terraform state list — if it returns the expected list of managed resources, the remote state is correctly populated. After a successful migration, commit the backend.tf file to version control but do not commit terraform.tfstate or terraform.tfstate.backup files — add both to .gitignore. If the migration fails midway, the local state file is preserved and the migration can be retried after fixing the issue. Never delete the local state backup file until you have verified that the remote state is complete and correct by comparing resource counts with terraform state list.
How do I configure Terraform state for CI/CD pipelines with multiple concurrent environments?
Configure Terraform state for CI/CD pipelines by using partial backend configuration with environment-specific -backend-config parameters at init time, ensuring each pipeline run targets the correct isolated state file without hardcoding environment-specific values in the Terraform configuration files. In the backend block, leave the key parameter empty or use a placeholder: terraform { backend "s3" { bucket = "your-company-terraform-state" region = "us-east-1" encrypt = true dynamodb_table = "terraform-state-locks" } }. In the CI/CD pipeline, pass the environment-specific key at init time: terraform init -backend-config="key=environments/${ENVIRONMENT}/terraform.tfstate" where ENVIRONMENT is a pipeline variable set per environment. This pattern allows a single Terraform configuration to be applied to multiple environments with isolated state by passing different keys at init time. For concurrent environment deployments, the DynamoDB locking ensures that two pipeline runs targeting the same environment key cannot run simultaneously — the second pipeline run waits for the first to release the lock. Configure a -lock-timeout=10m flag in CI/CD terraform plan and apply commands to allow the waiting pipeline to retry lock acquisition for 10 minutes rather than immediately failing when another run holds the lock.
Sources & references
Free resources
Critical CVE Reference Card 2025–2026
25 actively exploited vulnerabilities with CVSS scores, exploit status, and patch availability. Print it, pin it, share it with your SOC team.
Ransomware Incident Response Playbook
Step-by-step 24-hour IR checklist covering detection, containment, eradication, and recovery. Built for SOC teams, IR leads, and CISOs.
Get threat intel before your inbox does.
50,000+ security professionals read Decryption Digest for early warnings on zero-days, ransomware, and nation-state campaigns. Free, daily, no spam.
Unsubscribe anytime. We never sell your data.
Founder & Cybersecurity Evangelist, Decryption Digest
Cybersecurity professional with expertise in threat intelligence, vulnerability research, and enterprise security. Covers zero-days, ransomware, and nation-state operations for 50,000+ security professionals every morning.
Win a $2,495 Black Hat pass.
Full-access to Black Hat USA 2026 in Las Vegas. Subscribe free to enter.