Let's Encrypt and Certbot Automated Certificate Renewal: ACME Protocol, DNS Challenges, and Nginx/Apache Integration
Retool's new app builder is where AI-generated code ships safely
Building apps with AI is easy. Getting them to production safely is another story.
Had a production Nginx server serving an expired Let's Encrypt certificate for 4 days before a customer reported it. The certbot.timer was running on schedule, the renewal was being attempted, but the HTTP-01 challenge was failing silently — a recent Nginx configuration change had moved the document root and the /.well-known/acme-challenge/ path was no longer being served from the correct directory. Certbot logged the failure to /var/log/letsencrypt/letsencrypt.log, but nobody was monitoring that log or the resulting certificate expiry. The Certbot renewal failure email sent to the Let's Encrypt account email address was in a shared mailbox that had not been checked in three weeks.
The fix was straightforward: update the webroot path in the renewal configuration and add certificate expiry monitoring via Prometheus blackbox_exporter. But the lesson was that Certbot automation removes the operational overhead of manual renewal without removing the need for monitoring that verifies the automation is actually working. Silent failure is the primary operational risk of automated certificate renewal.
Installation and challenge configuration: HTTP-01 vs DNS-01
The choice between HTTP-01 and DNS-01 challenges determines where the certificate can be issued and what infrastructure it can protect. HTTP-01 works for any domain with a publicly accessible HTTP server but cannot issue wildcard certificates and fails when the server is behind a load balancer that does not forward the challenge path correctly. DNS-01 works for any domain regardless of HTTP accessibility and is required for wildcard certificates, but requires API access to the DNS provider for automated TXT record creation.
Use DNS-01 challenge for all certificate issuance when possible to eliminate web server dependency on renewal
The DNS-01 challenge is more reliable than HTTP-01 for production environments because it decouples certificate renewal from web server availability and configuration. With HTTP-01, any change to the Nginx configuration (document root change, new redirect rule, maintenance mode) can silently break the /.well-known/acme-challenge/ path and cause renewal failure. With DNS-01, renewal happens through the DNS API independently of the web server — even if Nginx is stopped for maintenance, the certificate renews correctly. Configure the DNS plugin API credentials with minimum necessary permissions: for Cloudflare, create an API token scoped to DNS edit permission for the specific zone only (not a global API key) and store it in the credentials file referenced in the renewal configuration. The security improvement from DNS-01 (no certificate issuance risk from HTTP server compromise) plus the reliability improvement (no web server dependency) makes DNS-01 the preferred choice when the DNS provider offers a supported Certbot plugin.
Test the full renewal workflow including deploy hooks using --force-renewal before the first certificate expires
Run a forced renewal test on every newly configured certificate within the first week of issuance to verify the complete workflow — challenge completion, certificate issuance, deploy hook execution, and web server reload — before the first automated renewal deadline. Run certbot renew --force-renewal --cert-name domain.com which renews the certificate immediately regardless of remaining validity. Verify that the renewal completed with Congratulations in the log output, that the deploy hook executed with the expected reload command, and that Nginx (or Apache) is serving the new certificate by checking openssl s_client output for the new certificate's serial number. This forced-renewal test catches the most common configuration issues: deploy hook not executable, web server not reloading correctly, DNS API credentials expired, or rate limit issues from repeated testing. Running this test within the first week (while the certificate is brand new) means you have the full 90-day period to fix any issues found before they affect an expiring certificate.
Monitoring: detecting silent renewal failures before certificate expiry
Certbot's automation is reliable when the environment it operates in is stable — the DNS API credentials are valid, the web server is serving the challenge path correctly, and the Let's Encrypt rate limits have not been hit. Environmental changes (DNS provider API key rotation, web server configuration changes, account email address changes) silently break the renewal automation without producing any immediately visible error. Monitoring that checks actual certificate expiry dates rather than assuming Certbot is working is the defense against this failure mode.
Configure external certificate expiry monitoring with 21-day alert threshold and 7-day escalation
Configure two-tier certificate expiry monitoring: a 21-day warning alert that notifies the infrastructure team via email that an upcoming certificate expiry needs attention, and a 7-day critical alert that triggers on-call notification via PagerDuty. The 21-day window gives time to investigate and fix a broken renewal automation, re-run the renewal manually, or escalate to the DNS provider if API credentials have expired. The 7-day window triggers on-call because a certificate with 7 days remaining has likely already failed multiple automated renewal attempts and requires immediate manual intervention. Set up both tiers using the Prometheus blackbox_exporter ssl probe with two Grafana alerting rules at different thresholds, or use a monitoring service with configurable multi-stage alerts. Verify both alert tiers fire correctly by checking the alert configuration with a domain whose certificate expiry date falls within the alert window from the test environment's perspective.
Monitor the Certbot renewal log for failure messages using a log monitoring tool
Forward the Certbot renewal log at /var/log/letsencrypt/letsencrypt.log to a log monitoring system and create an alert for lines containing Failed, Error, or Problem with the domain validation that indicate a failed renewal attempt. In Grafana Loki or Splunk, create a log alert that triggers when the letsencrypt.log contains a failure pattern during the scheduled renewal time window (Certbot typically runs at a randomized time between midnight and 12pm based on the systemd timer randomized delay setting). A failure log alert at renewal time gives a 30-day warning before expiry — the certificate failed to renew while still in the 30-day renewal threshold — which is the maximum lead time available for investigation and remediation. Pair the log failure alert with the external SSL expiry monitoring for defense-in-depth: the log alert catches failures early with 30 days remaining, and the external expiry monitoring catches failures that were somehow not logged or where the log alert was missed.
Briefings like this, every morning before 9am.
Threat intel, active CVEs, and campaign alerts, distilled for practitioners. 50,000+ subscribers. No noise.
The bottom line
Let's Encrypt and Certbot provide fully automated TLS certificate management that eliminates manual certificate renewal as an operational task, but the automation requires active monitoring to detect silent failures. Install Certbot via snap for the latest version, prefer DNS-01 challenge for production certificates to eliminate web server dependency on renewal, and install the DNS plugin for your provider (certbot-dns-cloudflare, certbot-dns-route53) for automated TXT record management. Configure deploy hooks in /etc/letsencrypt/renewal-hooks/deploy/ that reload Nginx or Apache after each successful renewal. Test the complete renewal workflow with --force-renewal within the first week of issuance. Monitor certificate expiry with a 21-day warning and 7-day critical alert using Prometheus blackbox_exporter or an external monitoring service. Monitor the Certbot log for failure messages at renewal time. The four-minute time investment in monitoring configuration is what separates automated certificate management that is genuinely hands-off from automated certificate management that fails silently until a customer reports an untrusted certificate error.
Frequently asked questions
How do I install Certbot and obtain a Let's Encrypt certificate for Nginx?
Install Certbot on Ubuntu/Debian with snap install --classic certbot which installs the latest version with all official plugins. For Nginx, run certbot --nginx -d domain.com -d www.domain.com which uses the Nginx plugin to automatically configure Nginx with the certificate, modify the Nginx configuration to redirect HTTP to HTTPS, and set up the renewal hook to reload Nginx after renewal. The Certbot Nginx plugin modifies the existing Nginx server block by adding ssl_certificate and ssl_certificate_key directives pointing to /etc/letsencrypt/live/domain.com/fullchain.pem and privkey.pem, and adding a managed TLS configuration block with modern cipher settings. If you prefer to maintain full control of the Nginx configuration, use certbot certonly --nginx -d domain.com instead — this obtains the certificate without modifying Nginx, leaving the Nginx configuration entirely in your control. After obtaining the certificate, verify it is accessible with openssl s_client -connect domain.com:443 -servername domain.com which shows the certificate chain, issuer (Let's Encrypt), and expiry date.
How do I obtain a wildcard certificate with Certbot using DNS-01 challenge?
Obtain a wildcard certificate using DNS-01 challenge with a Certbot DNS plugin that automates TXT record creation for your DNS provider. For Cloudflare-managed DNS, install the plugin with snap install certbot-dns-cloudflare and create a Cloudflare API token credentials file at /etc/letsencrypt/cloudflare.ini containing dns_cloudflare_api_token = YOUR_CLOUDFLARE_API_TOKEN with permissions 600. Run certbot certonly --dns-cloudflare --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini -d '*.domain.com' -d domain.com which issues a certificate covering the bare domain and all first-level subdomains. For AWS Route 53, use the certbot-dns-route53 plugin with IAM credentials configured for Route 53 TXT record creation. The DNS-01 challenge requires the DNS plugin to create a _acme-challenge TXT record in your domain's DNS zone, wait for DNS propagation, and then validate the record with Let's Encrypt — this takes 1-3 minutes per domain depending on DNS TTL and propagation speed. Wildcard certificates require DNS-01 because the HTTP-01 challenge cannot verify ownership for all possible subdomains without an HTTP server at every subdomain.
How do I configure Certbot to automatically reload Nginx after certificate renewal?
Configure Certbot post-renewal hooks to automatically reload Nginx after a certificate is renewed, ensuring the web server loads the new certificate without requiring manual intervention. Create an executable shell script in /etc/letsencrypt/renewal-hooks/deploy/ — any script in this directory is automatically run by Certbot after each successful renewal. Create /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh with content: #!/bin/bash and systemctl reload nginx. Set permissions with chmod +x /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh. The deploy hooks run after every successful renewal for any certificate on the system, so if you host multiple domains on one server, the hook reloads Nginx for each renewed certificate. Alternatively, use pre and post hooks for more control: pre hooks run before the ACME challenge (useful for maintenance mode), post hooks run after the challenge regardless of success or failure, and deploy hooks run only after successful renewal. Test the hook is working by running certbot renew --force-renewal --cert-name domain.com --dry-run which simulates renewal and shows whether the deploy hook would be executed.
How do I verify that Certbot's automated renewal is working correctly?
Verify Certbot automated renewal by checking the systemd timer status, reviewing the Certbot renewal log, and running a manual dry-run test. Check the systemd timer status with systemctl status certbot.timer — it should show Active: active (waiting) and the next trigger time (typically 12 hours from the last run). View the actual renewal schedule with systemctl list-timers | grep certbot. Check the Certbot renewal log at /var/log/letsencrypt/letsencrypt.log or with journalctl -u certbot to see the last renewal attempt results — look for Cert not yet due for renewal (certificates with more than 30 days remaining) or Congratulations, all renewals succeeded (certificates renewed in this run). Run a dry-run test annually or after any infrastructure change: certbot renew --dry-run which tests the full renewal workflow for all certificates using Let's Encrypt's staging environment without modifying actual certificates. If the dry-run fails, the error message identifies the specific failure — DNS challenge credentials expired, web server not responding to HTTP-01 challenge, rate limit reached — allowing remediation before the actual certificate expiry deadline.
How do I monitor Let's Encrypt certificate expiry and get alerted before they expire?
Monitor certificate expiry using the Prometheus blackbox_exporter for automated SSL expiry metrics or a dedicated certificate monitoring service. Configure the Prometheus blackbox_exporter with an HTTPS probe module and add targets for all your domains in the Prometheus scrape configuration. The exporter exposes the ssl_earliest_cert_expiry metric as a Unix timestamp of the earliest certificate expiry in the chain. Create a Grafana alert that fires when ssl_earliest_cert_expiry - time() < 86400 * 14 (less than 14 days remaining) for any monitored domain — this provides a 14-day warning window before the automated renewal threshold of 30 days, giving time to investigate and fix renewal failures before the certificate actually expires. For simpler monitoring without Prometheus, use an external certificate monitoring service (Uptime Kuma self-hosted, Checkly, or StatusCake) that checks your domains and sends email or Slack alerts when the SSL certificate has fewer than 21 days remaining. Direct email alerts from the monitoring service to the security and infrastructure team aliases so certificate expiry warnings are not lost in a single engineer's inbox.
How do I handle Let's Encrypt rate limits when deploying certificates for many domains?
Handle Let's Encrypt rate limits by understanding the limits and planning certificate requests to avoid hitting them. The most important limits: 50 certificates per registered domain per week (example.com and all subdomains count toward this limit — 50 certificates is the total across all subdomains), 5 failures per account per hostname per hour (5 failed certificate requests for domain.com from one account blocks further attempts for 1 hour), and 5 duplicate certificates per domain per week (requesting the same certificate with the same SANs more than 5 times in a week). For organizations deploying certificates for many subdomains, use wildcard certificates (*.domain.com) to cover all subdomains with a single certificate rather than individual certificates per subdomain, which consumes 50x fewer rate limit quota. Use Let's Encrypt's staging environment (--staging flag) for all testing and development to avoid consuming production rate limit quota: staging certificates are signed by a test CA (not trusted by browsers) but undergo the same ACME validation process, making them suitable for testing the full automation workflow. If rate limits are hit in production, check the Boulder status page at status.letsencrypt.org and use certbot certificates to list all current certificates and identify any duplicate requests that are consuming quota unnecessarily.
How do I migrate from manually managed certificates to Certbot automated renewal?
Migrate from manually managed certificates to Certbot automated renewal by first running Certbot alongside the existing certificate to verify the automation works, then switching the web server to use Certbot-managed certificates only after confirmation. On the first pass, run certbot certonly --webroot --webroot-path /var/www/html -d domain.com -d www.domain.com to obtain a Let's Encrypt certificate without modifying the web server configuration, while the existing manually managed certificate remains in use. Verify the new certificate was issued successfully with certbot certificates and check the expiry. Configure the web server to load the Certbot-managed certificate (from /etc/letsencrypt/live/domain.com/) in a test Nginx configuration, verify the TLS handshake works, then update the production Nginx configuration to point to the Certbot certificate paths. After the switch, test automated renewal with certbot renew --dry-run and verify the deploy hook reloads Nginx. Remove the old manually managed certificate files after two successful automated renewal cycles to confirm the automation is reliable. Document the certificate paths (/etc/letsencrypt/live/domain.com/fullchain.pem and privkey.pem) and the renewal configuration location (/etc/letsencrypt/renewal/domain.com.conf) for any infrastructure-as-code configurations that reference certificate paths.
Sources & references
Free resources
Critical CVE Reference Card 2025–2026
25 actively exploited vulnerabilities with CVSS scores, exploit status, and patch availability. Print it, pin it, share it with your SOC team.
Ransomware Incident Response Playbook
Step-by-step 24-hour IR checklist covering detection, containment, eradication, and recovery. Built for SOC teams, IR leads, and CISOs.
Get threat intel before your inbox does.
50,000+ security professionals read Decryption Digest for early warnings on zero-days, ransomware, and nation-state campaigns. Free, daily, no spam.
Unsubscribe anytime. We never sell your data.
Founder & Cybersecurity Evangelist, Decryption Digest
Cybersecurity professional with expertise in threat intelligence, vulnerability research, and enterprise security. Covers zero-days, ransomware, and nation-state operations for 50,000+ security professionals every morning.
Win a $2,495 Black Hat pass.
Full-access to Black Hat USA 2026 in Las Vegas. Subscribe free to enter.