Hardcoded Secrets Remediation: What to Do After Your Repo Scan Returns 3,000 Results

The high finding count in most legacy repositories is not the result of deliberate negligence. It reflects a set of development practices that were standard before secrets scanning tools existed and before the security consequences were well understood. Understanding the root causes helps with triage because different causes produce different risk profiles.

The most common source is local configuration files committed to the repository. Before environment variable management was a standard practice, the easiest way to configure a development environment was to include a config file in the repository that contained database URLs, API keys, and other connection strings. These files were meant to represent the developer's local setup, but they frequently contained real credentials because the developer was testing against a shared development or staging environment. The file was committed, pushed, reviewed by no one, and forgotten.

The second major source is copy-paste culture in CI/CD pipeline configuration. Configuring Jenkins, GitHub Actions, or GitLab CI requires credentials for deployment targets, cloud providers, and external services. Early pipeline configurations often included these credentials inline in the YAML or pipeline script because the developer was following a tutorial that showed inline credentials for simplicity. The pipeline worked, so no one revisited the credential handling approach. Over several years, each new service added to the pipeline added another inline credential.

The third source is test and mock data that evolved into real credentials. Developers create placeholder values like api_key = "test123" during development. At some point, testing against the real external service becomes necessary and the placeholder is replaced with a real key, committed as a minor change, and never moved to a proper secrets store. The scanner correctly identifies this as a real secret.

The fourth source is merge commit history. When branches with secrets are merged, the secret exists in the merge commit even if it was removed in a subsequent commit. Scanners that check full git history, which Trufflehog and Gitleaks do by default, will find every version of every secret that was ever committed to any branch that was ever merged into your main branch. A secret that was committed, caught in code review, and immediately removed in the next commit will still appear in the scan results because it exists in the git object database.

The purpose of triage is to determine which findings represent immediate active risk versus which can be addressed through normal remediation cycles. The classification that matters most is whether a credential is currently active and whether the systems it accesses are sensitive enough to constitute a breach scenario if compromised.

Tier 1 (rotate immediately without verifying): Cloud provider access keys (AWS, GCP, Azure) with broad permissions, production database credentials, OAuth client secrets for production systems, payment processor API keys, and identity provider service account credentials. These categories warrant immediate rotation before verifying whether they are currently in use because the downside of rotating an active key is temporary service disruption, while the downside of leaving an active key in a scannable location is potentially a full account or data breach. For cloud provider keys specifically, the assumption should always be that the key is active and has been accessed by threat actors, given that automated scanners continuously index public repositories.

Tier 2 (verify activeness, then rotate): Internal service API keys for non-sensitive services, SMTP credentials for internal alerting, monitoring platform tokens, and analytics service API keys. For these, a quick check against the service's audit log or last-used timestamp is worth doing before rotating, because rotation may require coordination with other teams and you want to prioritize the coordination effort.

Tier 3 (low risk, schedule for cleanup): Credentials that appear to be for development or staging environments, credentials that scanner confidence scoring suggests may be false positives, old version-controlled config templates with placeholders that match secret patterns, and credentials for decommissioned services. These should be documented and scheduled for cleanup but do not require emergency response.

The practical tool for this classification is to export scan results from Trufflehog or Gitleaks into a spreadsheet or ticketing system and add columns for: service type, environment (production versus staging versus dev), last git activity date for the file, and scanner-reported confidence. Group by service type first, then by environment. All production cloud keys go to Tier 1 regardless of last activity date. Apply the tier criteria as rules to the grouped view rather than reviewing each finding individually, which will not scale at 3,000 results.

One of the most operationally significant decisions in secrets remediation is whether to rotate a credential immediately or to first verify whether it is currently in use. The risk calculus depends on the credential type, the sensitivity of the systems it accesses, and the blast radius of both a breach and a rotation incident.

For cloud provider credentials, the answer is always rotate immediately. AWS IAM access keys, GCP service account keys, and Azure service principal credentials with any level of permissions should be treated as compromised the moment they are found in a repository with a history that extends beyond your organization's network. Automated threat actor tooling continuously monitors public repositories (and some private repositories via token theft) for cloud provider credentials. The average time between a credential being pushed to a public GitHub repository and its first unauthorized use is less than five minutes. Even for private repositories, you do not know how many developers have cloned the repo to personal machines or how many CI systems have cached the repository content.

For database credentials, the immediate rotation argument is also strong, but the operational complexity is higher. Rotating a production database password requires coordination with every application that uses that password. In environments without centralized secret injection (which is precisely the kind of environment that has 3,000 hardcoded secrets), rotating a database password means finding every place it is referenced and updating it simultaneously. The operational approach that minimizes downtime is: create a new database user with identical permissions, update all application configurations to use the new user's credentials (deploying configuration changes before disabling the old credentials), verify that all services are connecting with the new credentials, then revoke the old user. Do not simply change the password on the existing database user without first auditing every connection that uses it.

For API keys to external services, most providers offer key rotation without downtime by allowing you to create a new key, use it in parallel with the old one, then revoke the old key after confirming successful operation. This parallel rotation approach is the standard for services like Stripe, Twilio, SendGrid, and most SaaS APIs. Check the provider's documentation for the specific rotation procedure, as some providers invalidate all sessions when a key is revoked.

The "verify first" approach is appropriate when the credential is for an internal service where you have full visibility into connection logs and can determine within minutes whether the credential is active. If your internal monitoring service's API token shows no successful authentications in the last 90 days, verification takes thirty seconds and confirms whether an emergency rotation is warranted versus a normal sprint cycle.

After rotating credentials, teams face a decision about the git history that contains the now-revoked secrets. The exposure already occurred in the sense that the credentials existed in a scannable location. The question is whether rewriting history to remove the secrets from git reduces ongoing risk or creates more problems than it solves.

The tool for git history rewriting is git-filter-repo, which replaces the older and slower git filter-branch. The basic operation finds all occurrences of a specific string across all commits in the repository and replaces them with a placeholder, rewriting the commit hashes of all affected commits. This effectively produces a new history without the secrets. However, it also invalidates all existing clones of the repository because the commit hashes no longer match. Every developer, every CI system, every deployment pipeline that has cloned the repository needs to reclone from the rewritten origin.

The practical cases where history rewriting is worth the disruption are: secrets in public repositories where the exposure is ongoing because anyone can clone and scan, and secrets that were so sensitive (master encryption keys, root credentials, private keys) that their presence in any readable location constitutes an ongoing risk even after revocation. For a private repository where the credentials have been rotated and access is controlled, the risk reduction from history rewriting is marginal.

The alternative position is to accept that the historical exposure occurred, document it as a remediating circumstance that the credentials were rotated, and focus operational effort on ensuring the new credential storage method is secure and that the git history exposure cannot be repeated. This is the position taken by most mature security programs for private repositories: revoke the credential, migrate to proper secrets management, and move forward without the disruption of a force-push that invalidates everyone's local checkouts.

One middle ground is to scrub secrets from the active branches that are regularly cloned and reviewed while leaving older archived branches untouched. This does not produce a perfectly clean history but reduces the likelihood that a routine clone or shallow clone of the main branch will expose credentials. For monorepos with multi-year histories, this pragmatic approach is often the most operationally sustainable.

Regardless of whether you rewrite history, rotate the credentials and update secret detection tooling immediately. History rewriting is a hygiene measure. Credential rotation is the security control.

Rotating credentials is necessary but not sufficient. The structural problem is that your codebase treats credentials as configuration values that belong in code or config files. Fixing this requires moving to a centralized secrets management system that provides runtime credential injection, audit logging, and automatic rotation. The three dominant options in enterprise environments are HashiCorp Vault, AWS Secrets Manager, and Azure Key Vault.

HashiCorp Vault is the most flexible option and works across cloud providers and on-premises environments. Its core value proposition for secrets remediation is dynamic secrets: rather than storing a static database password that an application reads, Vault generates a unique, short-lived set of database credentials for each application that requests them, with a TTL of minutes to hours. When the TTL expires, the credentials are automatically revoked. This means a leaked secret is useless after its TTL expires, which fundamentally changes the risk profile of credential exposure. Vault also supports static secret storage with automatic rotation policies, which is the migration path for credentials that cannot be dynamically generated.

AWS Secrets Manager is the path of least resistance for AWS-native workloads. It integrates directly with RDS, Redshift, and DocumentDB for automatic password rotation, and with IAM for access control. Applications running on EC2 or Lambda retrieve secrets via the AWS SDK without managing credentials in environment variables. The important migration step is replacing hardcoded credential references in application code with AWS SDK calls to retrieve the secret by ARN at runtime. AWS Secrets Manager also supports cross-account access for organizations that need to share secrets between AWS accounts.

Azure Key Vault performs the equivalent function for Azure workloads and integrates with Azure Managed Identity, which eliminates the need for any application credential at all. Applications running in Azure App Service, AKS, or Azure Functions with a managed identity assigned can retrieve Key Vault secrets without authenticating themselves. The identity is managed by the Azure control plane. This is the strongest secrets management pattern because there is no application credential to leak.

For teams migrating from hardcoded secrets, the operational path is: identify the application components that consume each rotated credential, update the application code to retrieve the secret from the secrets manager at startup or at the time of use, test the new retrieval path in staging, deploy the updated application, and remove the hardcoded credential from the configuration file. The last step requires that the configuration file change happen simultaneously with the application deployment, or you create a brief window where neither the hardcoded nor the vault-retrieved credential is active.

After completing the initial remediation, the priority shifts to ensuring the finding count does not return to 3,000. Prevention operates at two points: before code reaches the repository (pre-commit hooks) and after code is pushed (CI/CD pipeline scanning). Both layers are necessary because pre-commit hooks can be bypassed or disabled by individual developers, while CI scanning catches everything but has higher latency.

For pre-commit hooks, the two most widely deployed tools are detect-secrets from Yelp and gitleaks. Detect-secrets generates a .secrets.baseline file that records all approved secrets (test credentials, example values that match secret patterns but are not real) in the repository. The hook compares each commit against the baseline and fails the commit if new secret-pattern matches are found that are not in the baseline. This design prevents blocking developers who are working with intentional test fixtures or documentation examples. Gitleaks as a pre-commit hook performs a similar function with different detection rules and is the better choice for teams already using Gitleaks for their repository scanning, since it ensures rule consistency between developer environments and the CI pipeline.

The most common failure mode for pre-commit hooks is inconsistent installation across the developer population. Pre-commit hooks live in the .git/hooks directory, which is not version controlled. A developer who clones the repository fresh does not automatically have the hooks installed unless there is an onboarding step that runs pre-commit install or the equivalent. Solve this by including hook installation in your development environment setup script and documenting it as a required step in the repository's contribution guide.

CI/CD pipeline scanning catches commits where the pre-commit hook was bypassed (using git commit --no-verify), secrets in CI pipeline configuration files themselves, and secrets introduced through automated dependency updates or code generation tools. For GitHub Actions repositories, Trufflehog has a GitHub Action that runs on every push and pull request and comments on the PR with any findings. For GitLab CI, both Trufflehog and Gitleaks publish Docker images that can be included as pipeline stages. The scanning job should fail the pipeline on new findings, meaning findings that were not present in the base branch.

The false positive management question arises in every secrets scanning deployment. Engineers will object to detection rules that flag things like RSA key headers in documentation, AWS account IDs that match key patterns, or JWT examples in test files. The correct response is to build a suppression process rather than disabling detection rules. Both Trufflehog and Gitleaks support inline suppression comments (# nosec style annotations) that tell the scanner to ignore a specific line, along with explanations that are reviewable during code review. Suppression of a real secret should fail code review; suppression of a documented false positive should be approved quickly and documented in the commit message.