Cert renewal suddenly stopped working

My web server and certs have been working for a long time until recently. I haven't touched anything on the server and I only found out recently when I was told by the browser that the cert was invalid.

I run a shell script to add and update certs though I haven't used it in close to two years. I use the script because I have Opennic domains so I add conventional domains manually. It has worked in the past.

This is the script:

certbot --nginx\
    -d r3df0x.com\
    -d www.r3df0x.com\
    -d analytics.r3df0x.com\
    -d mail.dns.blacklist.r3df0x.com\
    -d cccrwiki.r3df0x.com\
    -d www.cccrwiki.r3df0x.com\
    -d coc.r3df0x.com\
    -d code-of-conduct.r3df0x.com\
    -d codeofconduct.r3df0x.com\
    -d conservativesjw.r3df0x.com\
    -d global.r3df0x.com\
    -d guns.r3df0x.com\
    -d i.r3df0x.com\
    -d metawiki.r3df0x.com\
    -d www.metawiki.r3df0x.com\
    -d opennic.r3df0x.com\
    -d www.opennic.r3df0x.com\
    -d resources.r3df0x.com\
    -d shitlordsinaction.r3df0x.com\
    -d sturmovik.r3df0x.com\
    -d www.sturmovik.r3df0x.com\
    -d template.r3df0x.com\
    -d thesexynerd.r3df0x.com\
    -d cccrwiki.rfx.fi\
    -d www.cccrwiki.rfx.fi\
    -d metawiki.rfx.fi\
    -d www.metawiki.rfx.fi\
    -d opennic.rfx.fi\
    -d www.opennic.rfx.fi\
    -d sturmovik.rfx.fi\
    -d www.sturmovik.rfx.fi\
    -d voynawiki.rfx.fi\
    -d www.voynawiki.rfx.fi\
    -d rfx.fi\
    -d www.rfx.fi\

This is the output of the script:

root@f0x-kali:/etc/nginx/sites-available# /root/certbot.sh

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for r3df0x.com
http-01 challenge for www.r3df0x.com
http-01 challenge for analytics.r3df0x.com
http-01 challenge for mail.dns.blacklist.r3df0x.com
http-01 challenge for cccrwiki.r3df0x.com
http-01 challenge for www.cccrwiki.r3df0x.com
http-01 challenge for coc.r3df0x.com
http-01 challenge for code-of-conduct.r3df0x.com
http-01 challenge for codeofconduct.r3df0x.com
http-01 challenge for conservativesjw.r3df0x.com
http-01 challenge for global.r3df0x.com
http-01 challenge for guns.r3df0x.com
http-01 challenge for i.r3df0x.com
http-01 challenge for metawiki.r3df0x.com
http-01 challenge for www.metawiki.r3df0x.com
http-01 challenge for opennic.r3df0x.com
http-01 challenge for www.opennic.r3df0x.com
http-01 challenge for resources.r3df0x.com
http-01 challenge for shitlordsinaction.r3df0x.com
http-01 challenge for sturmovik.r3df0x.com
http-01 challenge for www.sturmovik.r3df0x.com
http-01 challenge for template.r3df0x.com
http-01 challenge for thesexynerd.r3df0x.com
http-01 challenge for cccrwiki.rfx.fi
http-01 challenge for www.cccrwiki.rfx.fi
http-01 challenge for metawiki.rfx.fi
http-01 challenge for www.metawiki.rfx.fi
http-01 challenge for opennic.rfx.fi
http-01 challenge for www.opennic.rfx.fi
http-01 challenge for sturmovik.rfx.fi
http-01 challenge for www.sturmovik.rfx.fi
http-01 challenge for voynawiki.rfx.fi
http-01 challenge for www.voynawiki.rfx.fi
http-01 challenge for rfx.fi
http-01 challenge for www.rfx.fi
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. global.r3df0x.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://global.r3df0x.com/.well-known/acme-challenge/Vt_j56IBrtzS1nsHzzPJ8j3MLwjNJo59AbLwToMACbY [192.241.132.174]: "global.r3df0x.com\n<html lang="en" dir="ltr" class="client-nojs">\n\n<meta charset="UTF-8"/>\n.well-know", coc.r3df0x.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://coc.r3df0x.com/.well-known/acme-challenge/PC8jBDi8y2dn8Alj48TrYvWFRtIMeO7RK-OH5mLmX-Y [192.241.132.174]: "\r\n404 Not Found\r\n<body bgcolor="white">\r\n

404 Not Found

\r\n
", cccrwiki.r3df0x.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://cccrwiki.r3df0x.com/.well-known/acme-challenge/RYrVe-ljOzyQYiCHsjzxRMxzH9nKcxHH4km5MikWA_Q [192.241.132.174]: "\n<html lang="en" dir="ltr" class="client-nojs">\n\n<meta charset="UTF-8"/>\nLogin required - CCCR Wiki<", codeofconduct.r3df0x.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://codeofconduct.r3df0x.com/.well-known/acme-challenge/Hhtes1QBISN5l1sxGRVjI-CSJQ7YJmZbLaofctz272Q [192.241.132.174]: "\r\n404 Not Found\r\n<body bgcolor="white">\r\n

404 Not Found

\r\n
", conservativesjw.r3df0x.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://conservativesjw.r3df0x.com/.well-known/acme-challenge/yo5sn6F2OrGEZwNPYVs_ZBrCuFWzRyPNmZVh76aIyT8 [192.241.132.174]: "\n<html xmlns="http://www.w3.org/1999/xhtml" lang="en-US">\n\n\t<meta http-equiv="Content-Type" content="text/h", guns.r3df0x.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://guns.r3df0x.com/.well-known/acme-challenge/3kbzS5YIJ4QD_QG0Zim_vaobJnE1BmD82xVM8I-jWMc [192.241.132.174]: "\n<html xmlns="http://www.w3.org/1999/xhtml" lang="en-US">\n\n\t<meta http-equiv="Content-Type" content="text/h", cccrwiki.rfx.fi (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://cccrwiki.rfx.fi/.well-known/acme-challenge/2Tncs0p5M3dHXE8toZd7soth7oKdA4yKR6505mwZXOU [192.241.132.174]: "\n<html lang="en" dir="ltr" class="client-nojs">\n\n<meta charset="UTF-8"/>\nLogin required - CCCR Wiki<", metawiki.r3df0x.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://metawiki.r3df0x.com/.well-known/acme-challenge/idcl3gRxNcEuGPWsG06Uhpshn8-eRqssRot4ob9hQH4 [192.241.132.174]: "\n<html lang="en" dir="ltr" class="client-nojs">\n\n<meta charset="UTF-8"/>\nLogin required - Metawiki</", code-of-conduct.r3df0x.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://code-of-conduct.r3df0x.com/.well-known/acme-challenge/wQF3qd5SlXEti6J0nIvv8e_K0kzIBLkI5ALDWQD1Js0 [192.241.132.174]: "\r\n404 Not Found\r\n<body bgcolor="white">\r\n

404 Not Found

\r\n
", mail.dns.blacklist.r3df0x.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://mail.dns.blacklist.r3df0x.com/.well-known/acme-challenge/d-syJZGG7BQc1EHyK0tFeAoRU4o1t5Wv6riFMWvu7do [192.241.132.174]: "\r\n404 Not Found\r\n<body bgcolor="white">\r\n

404 Not Found

\r\n
", i.r3df0x.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://i.r3df0x.com/.well-known/acme-challenge/QWjJb-XxTY65h41vrU1RrdDiPTlzIiVKP0SXaPPuA1Q [192.241.132.174]: "\r\n404 Not Found\r\n<body bgcolor="white">\r\n

404 Not Found

\r\n
", r3df0x.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://r3df0x.com/.well-known/acme-challenge/eCzFXZeDkoNlrISwH4Dpwr0V2xAOwUzEMxsF2WvSI2k [192.241.132.174]: "\r\n\r\n<html lang="en-US">\r\n \r\n \r\n\r\n \r\n <!-- Meta Tags -", analytics.r3df0x.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://analytics.r3df0x.com/.well-known/acme-challenge/E7EStoQKK_EKfYGl0muAmLNymCUrMb8v5z0Ae_CbsOo [192.241.132.174]: "\r\n404 Not Found\r\n<body bgcolor="white">\r\n

404 Not Found

\r\n
"

IMPORTANT NOTES:

This is on Ubuntu and Nginx.

nginx servers with a large number of virtual hosts sometimes suffer from a problem where nginx doesn't reload quickly enough when being reconfigured by Certbot to respond to the challenges.

That's my initial guess as to what the problem would be.

What is your version of Certbot?

3 Likes

The Certbot version is 0.22.2.

Ubuntu is 14.04

Thanks for editing to include your Ubuntu release ^^. Unfortunately 0.22 is quite ancient.

There is a workaround available in much newer versions of Certbot (1.6.0+). However, with Ubuntu 14.04 being end-of-life, those newer releases are not available to you.

What I can suggest on your version of Certbot is trying the webroot authenticator. It will avoid queuing up 35 nginx reloads, which I suspect is what causes the issue in this case.

It's a little involved though. Your command would have to be changed to be something like:

certbot run -i nginx -a webroot \
-w /path/to/r3df0x/ -d r3df0x.com \
-w /path/to/r3df0x/ -d www.r3df0x.com \
-w /path/to/analytics/ -d analytics.r3df0x.com \

I can appreciate this is quite annoying but I don't know of a less annoying fix, assuming the diagnosis is right.

2 Likes

Will upgrading Ubuntu allow Certbot to be updated?

Yes, absolutely. Even upgrading to just 16.04 would be enough to get to the latest version of Certbot (via snaps).

Though I would suggest upgrading even further, since 16.04 is end-of-life in April 2021.

1 Like