case study 06 / 11Atlas Labs · Security & PKI · 2026

Registry Certificate Auto-Renewal

The registry's certificate was renewed by hand. When it lapsed, every deploy that pulled an image failed until someone noticed.

Certificate renewal for a private container registry that runs itself: DNS-01 validation through Cloudflare and an install hook that verifies the live certificate and restores the old one on failure.

30 DAYS90 DAYSDNS-01INSTALLVERIFYNO OPEN PORTS · RESTORE ON FAILURE
outage that started it
1.5 houtage that started it
inbound ports needed to renew
0inbound ports needed to renew
blip per renewal, every ~60 days
~2 sblip per renewal, every ~60 days
scenarios simulated before go-live
6scenarios simulated before go-live

01The situation

Every deploy pulls its container images from a private Harbor registry. Its TLS certificate was renewed by hand, and one morning it simply expired. From that moment every image pull failed, and deploys stopped until it was renewed about an hour and a half later.

Renewing it by hand that morning showed why it had always been manual, and why the manual way was a trap.

02What it had to do

  • Renew without anyone remembering to, and without opening any inbound port.
  • Never leave the registry serving a bad certificate: verify what is actually served, and roll back if it's wrong.
  • Touch as little of the registry as possible: no regenerated config, no full restart.
  • Keep the credential it needs as narrow as possible.

03Architecture

  1. Renewal

    • certbot timer

      Runs twice a day and renews once the certificate has less than 30 days left.

    • Cloudflare DNS-01

      Proves ownership with a temporary TXT record through an API token scoped to one zone. Nothing connects in.

    Downward link: new certificate and key

  2. Install hook

    • Check

      Refuses a key that doesn't match its certificate.

    • Back up and copy

      Keeps timestamped copies, then installs the pair with the original owner and mode.

    • Restart only nginx

      About two seconds where the registry doesn't answer. Nothing else in the registry is touched.

    Downward link: verify the served serial, or restore

  3. Registry

    • Served certificate

      The hook reads back the serial the registry now serves. If it isn't the new one, the backup goes back in.

Every step that changes something has a backup or a way back.

No inbound port, ever: all traffic goes out to Cloudflare and Let's Encrypt.

04How it works

  1. Why renewal never worked on its own

    Three problems sat on top of each other. The renewal method checked ownership by connecting in on port 80, which the firewall blocks and the registry's web server occupies, so the timer on the box could never succeed. The registry serves a copy of the certificate, not the file certbot renews, so renewing alone changed nothing users see. And the official way to refresh that copy also regenerates the compose file, which would drop a hand-made storage mount and make every image look missing.

  2. Prove ownership through DNS

    certbot now uses DNS-01: it creates a temporary TXT record through the Cloudflare API, Let's Encrypt reads it from public DNS, and certbot deletes it. No port is opened, then or later. The API token can edit DNS in one zone only and sits in a root-only file.

  3. Install exactly what's needed

    A deploy hook runs after each renewal. It checks the key matches the certificate, backs up the current pair, copies the new one in with the right owner and mode, and restarts only the web server container. It never runs the regenerate step and never touches the compose file.

  4. Believe the live endpoint, not the log

    The hook reads the serial the registry is now serving. If it isn't the new certificate, it puts the backup back, restarts again and reports a failure. A status command shows the three serials side by side: the renewed file, the registry's copy and what port 443 serves.

  5. Test before touching production

    I simulated the script on my laptop with test certificates, a stub container runtime running a real TLS server and a stub certbot: install, no-op, key mismatch, automatic restore, hook mode, and a manual renewal with Ctrl-C and a failing certbot. On the server, setup ran a staging dry run first and would stop with nothing changed if that failed.

05What I found along the way

certbot succeeded, and the old certificate was still served

During the manual renewal I followed the handover document, which didn't make clear that the copy step was required. certbot reported success, but a live check from outside still showed the expired serial. That is why the hook verifies the served certificate instead of trusting the renewal.

The handover document had errors

Its restart step said to stop Docker where it meant start, and stopping all of Docker wasn't needed at all. I wrote a corrected runbook with rollback steps and a manual fallback.

Renewals can fail silently

Let's Encrypt stopped sending expiry emails in 2025, so a revoked token would go unnoticed until the next outage. An outside-in scan flags any certificate with less than 30 days left, which leaves a month of retries before it matters.

06What changed

  • The registry is set to renew itself about 30 days before expiry, with no one involved. The cost is a two-second blip every couple of months instead of an outage.
  • Port 80 is closed for good. The temporary firewall rule from the manual renewal was removed.
  • Verified after setup: the new serial served from outside, the registry health check green, an image pulled with its digest checked, and a renewal dry run passing without port 80.

07Takeaways

  • The first automatic renewal is due in mid-December 2026. I'll confirm the expiry moved, not assume it.
  • Still open: make the storage mount survive the regenerate step, so registry upgrades stop being a trap.
  • Check what is actually served. A green renewal log only proved that a file changed on disk.