ACME certificate renewal automation that does not fail quietly

Automated certificate renewal with the ACME protocol replaces a certificate before it expires with no person involved: a client proves control of the domain, receives the new certificate, installs it and reloads the server. It rarely fails loudly. It stops running or stops completing, the old certificate keeps working, and the outage arrives weeks later when it expires. This page covers how renewal automation breaks, how to test it, and how to monitor it so that a failure shows up while there is still time to act.

Quick facts

When to renew

when about a third of the lifetime remains, or when the authority's ARI says so

Test it

certbot renew --dry-run or your client's equivalent, against the staging environment

Reload after renewal

a deploy hook, for example --deploy-hook "systemctl reload nginx"

Monitor

the certificate each hostname serves, checked from outside

Why it matters more now

public certificate lifetimes fall to 200, 100 and then 47 days (SC-081v3)

The four steps that can fail

Automated renewal is a chain of four steps, and each can stop working without affecting the others.

• The schedule. Something must run the client periodically, usually a systemd timer or a cron job. Timers get disabled during maintenance, containers get rebuilt without them, and servers get replaced by ones where nobody set them up. systemctl list-timers shows whether a timer is active and when it last ran.

• The challenge. Validation must still succeed. A redirect added months ago, an application that takes over the /.well-known/ path, a firewall change, or rotated DNS credentials each break it silently. How the challenge types differ is on ACME.

• The deployment. The new certificate must reach the place that serves it. On a single server the client writes the files; with a load balancer, a container image or several servers, a separate step copies them. Renewal that succeeds while deployment fails looks most like everything working.

• The reload. The server must load the new files. Without a reload, a correctly renewed and correctly deployed certificate is not the one visitors receive. In Certbot, a deploy hook runs only after a successful renewal:

$ certbot renew --deploy-hook "systemctl reload nginx"

Only the last two are visible from outside, and only once the old certificate is close to expiry. That asymmetry is the whole problem.

When renewal should happen

Renew early enough to survive a few failed attempts. A common rule is to renew when a third of the lifetime remains: about 30 days before expiry for a 90-day certificate. Most clients do this by default and try twice a day, so a temporary failure is retried long before it matters.

Authorities can also tell clients when to renew. ACME Renewal Information, ARI, defined in RFC 9773, lets the authority suggest a renewal window for each certificate, and ask for earlier renewal when it has to revoke certificates in bulk. Clients that support ARI follow that window instead of a fixed number of days. Renewing a Let's Encrypt SSL certificate works this way: the authority asks clients to check ARI, renew at random times and retry with backoff.

Shorter lifetimes make this more pressing. Under the CA/Browser Forum ballot SC-081v3, the maximum lifetime falls to 47 days by March 2029. At that point each certificate renews roughly every month, and a renewal that silently stops is an outage within weeks.

Test the whole cycle before relying on it

Waiting to see whether automation renews on its own is a slow way to discover a broken deployment step. Test it on the day you set it up:

$ certbot renew --dry-run

A dry run performs the full renewal against the authority's staging environment, without replacing your certificate and without counting against production rate limits. Other clients have the same option under a different name.

Then force one real renewal and confirm from outside that the new certificate is the one being served, by its dates:

$ openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null | openssl x509 -noout -dates

That exercises all four steps in one run. Repeat the test after any change to the server, the routing, the DNS provider or the credentials, since those are what break the challenge and the deployment.

Monitor the outcome, not the process

Monitoring the renewal job tells you whether it thinks it succeeded. Monitoring the certificate tells you what visitors get. They differ in exactly the cases that matter: a job that stopped being scheduled reports nothing, and nothing looks like success.

The check that works is external and simple. Ask each hostname which certificate it presents and how many days it has left, from a machine that does not share credentials or hardware with the renewal system:

$ openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null | openssl x509 -noout -checkend 1209600

The command exits with status 1 when the certificate expires within 14 days. Any monitoring system can run it, or an equivalent HTTPS check, on a schedule.

Alert on the right threshold

Alerting the day before expiry is too late. Alerting on every failed renewal attempt is noise, because transient failures are normal and are retried.

Set the alert where renewal should already have happened. With 90-day certificates renewed 30 days ahead, a certificate with fewer than about 20 days left means automation has not worked for ten days, and there is still time to fix it calmly. Scale the threshold with the lifetime: with 47-day certificates, alert at about a week left.

That one alert catches all four failing steps, because they all end the same way: the served certificate stops being replaced.

Cover every certificate

Monitoring the website's certificate confirms the one most likely to be noticed anyway. The certificates that lapse are the ones nobody watches: mail servers, APIs, load balancers, internal dashboards, devices.

Keep a list of every certificate in service, where it terminates and what renews it, and monitor the list rather than one hostname. Where the list is kept by hand it drifts, so derive it from what is deployed where possible. Certificate Transparency logs also show every public certificate issued for your domains, which helps find ones you forgot.

Rate limits and retries

Authorities limit how often certificates can be requested for the same names. A homemade script stuck in a loop, retrying immediately, can use up that allowance and stay blocked after the original problem is fixed. Use a client that backs off between attempts, which standard clients do by default, and debug issuance against the staging environment rather than production.

What to do when it does fail

1. Restore service first. Obtain and install a certificate by whatever works now, as described on renewal.

2. Find which step failed. The client's log usually tells: no entry at all points at the schedule, a validation error at the challenge, a successful renewal with an unchanged served certificate at deployment or reload.

3. Ask why nobody noticed. That is a different question from why it failed. If monitoring existed and did not fire, fix the monitoring.

Choosing a client that can deploy and reload on its own removes two of the four steps; see ACME clients.

FAQ