Need help renewing expired cert

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
icesat-2-scf.gsfc.nasa.gov
gs615-icesat2scf-pz.gsfc.nasa.gov

I ran this command:
/usr/bin/certbot certonly -q --webroot -w /var/www/html -d gs615-icesat2scf-pz.gsfc.nasa.gov -w /var/www/webscf -d icesat-2-scf.gsfc.nasa.gov --post-hook "echo LetsEncrypt Certificate Renewed; /usr/sbin/service apache2 restart"

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Attempting to parse the version 0.31.0 renewal configuration file found at /etc/letsencrypt/renewal/gs615-icesat2scf-pz.gsfc.nasa.gov-0001.conf with version 0.27.0 of Certbot. This might not work.
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for gs615-icesat2scf-pz.gsfc.nasa.gov
http-01 challenge for icesat-2-scf.gsfc.nasa.gov
Using the webroot path /var/www/webscf for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Running post-hook command: echo LetsEncrypt Certificate Renewed; /usr/sbin/service apache2 restart
Output from echo:
LetsEncrypt Certificate Renewed

Failed authorization procedure. icesat-2-scf.gsfc.nasa.gov (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching https://icesat-2-scf.gsfc.nasa.gov/.well-known/acme-challenge/TqjjKeCTwQbp_iZHR4X2Eo9u0AggDl6zu_HUtuvMZdo: Timeout during connect (likely firewall problem), gs615-icesat2scf-pz.gsfc.nasa.gov (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching https://gs615-icesat2scf-pz.gsfc.nasa.gov/.well-known/acme-challenge/cGYn1rPGAaA6hEdP8yHKaAVp0R8tzqlrRbE0PiUG4Bk: Timeout during connect (likely firewall problem)

IMPORTANT NOTES:

My web server is (include version):
Apache/2.4.29 (Ubuntu)

The operating system my web server runs on is (include version):
Ubuntu 18.04

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 0.27.0 (from the Ubuntu 18.04 repository)

Additional info:

  1. This server is behind an outbound squid proxy. I've received conflicting instructions on configuring our server to work with this squid proxy, so have tested a variety of configs. The above output is the furthest certbot has proceeded with any of the configs.

  2. This server has had a working LetsEncrypt cert/conf for a few years. Recently a failed OS upgrade resulted in a fresh OS install and restore of the /etc/apache2 and /etc/letsencrypt configuration directories from backup.

The certbot cron job has been failing since the OS upgrade, and now the cert has expired. I don't know if the fresh OS install might have affectd the LetsEncrypt cert renewal (new/different server certs?)

1 Like
2

Welcome Back to the Let's Encrypt Community, John :slightly_smiling_face:

I can connect to these:

http://icesat-2-scf.gsfc.nasa.gov/.well-known/acme-challenge/test
http://gs615-icesat2scf-pz.gsfc.nasa.gov/.well-known/acme-challenge/test

over IPv4 via port 80 with a correct 404 response, but not over IPv6 via port 80, which is what Let's Encrypt will do when verifying the ACME challenge files.

I recommend checking:

  • the Apache configuration files to ensure that the two websites are listening over IPv6
  • the DNS records to ensure that the IPv6 addresses (AAAA records) point to the same servers as the IPv4 addresses (A records)
4 Likes
3

Your IPv6 address 2001:4d0:2310:170::41 isn't reachable at all, not by ping, port 80 or port 443.

2 Likes
4

OK, ipv6 address should be working now. [As part of the fresh OS install, I had to do it on a different network, and when I moved back to the operational network, forgot to enable the ipv6 config.] But I still get the same errors when running certbot.

1 Like
5

Exactly the same error? Because now most if the errors at LetsDebug are gone, just the rate limit warning for nasa.gov.

Could you paste the complete output of a new attempt?

2 Likes
6

The requests that fail are HTTPS.
That implies that a redirect was reached for the initial HTTP request.
If so, then all you need do is to handle the challenge requests within HTTP (don't redirect them).

1 Like
7

You make a very interesting point, @rg305. How did the original certbot errors contain "https" when the timeout was occurring during the "http" request (per the original Let's Debug links that I posted)?

1 Like
8

The HTTP issue was with IPv6 and the IPv4 failure shows HTTPS:

image
image927×410 20.3 KB

1 Like
9

It's working and I now have a valid cert. I suspect the initial problem was caused by the incorrect IPv6 config, and at some point someone recommended I try using the acme.sh script. I installed it, but it didn't work any better. Someone else asked to look at the config files, and while looking in apache2.conf noticed a bunch of Alias statements pointing /.well-known/acme-challenge to /home/.acme. I commented those out, bounced apache, and it worked.

2 Likes
10

Excellent! :smiley:

Glad you got it working! :partying_face:

2 Likes
11

Not sure who to credit for the Solution. You were all a great help. Will this message board let me credit more than one?

2 Likes
12

It won't allow more than one solution to be marked. We usually go with the earliest response that correctly describes the problem and means to alleviate it. Often the problem is multifaceted (as was the case here), which makes marking a single post as "the" solution not really accurate anyhow.

1 Like
13

By the way...

  1. Why are you using the certbot webroot authenticator rather than the apache authenticator?

  2. Gracefully reloading apache with apachectl -k graceful would be far less disruptive than a hard restart.

  3. You should be using a --deploy-hook (which only executes when a certificate actually renews) as opposed to a --post-hook (which executes every time a certificate renewal is attempted).

2 Likes
14

You might try this instead:

/usr/bin/certbot certonly -q --apache -d "gs615-icesat2scf-pz.gsfc.nasa.gov,icesat-2-scf.gsfc.nasa.gov" --deploy-hook "echo LetsEncrypt Certificate Renewed; apachectl -k graceful"

You can add --dry-run to the end of the given command to have certbot use the Let's Encrypt staging environment to test if the given command will work for you. Your current certificate will be unaffected.

You can add --force-renewal to the end of the given command to have certbot update your renewal configuration. Be careful with this parameter. It can cause you to hit the duplicate certificate rate limit in short order if used excessively.

1 Like
15

OK, well griffin was the first to mention the IPv6 issue, which apparently was part of the problem, so I'll credit him. But again, thanks to all. I'll also look into changing the options I'm using for certbot, testing with the --dry-run first.

I haven't looked at Let's Encrypt or certs in general for a few years, and had to re-learn a lot when it stopped working. That's the problem when something works so well for such a long time. :wink:

3 Likes
16

So very true:

image
image845×74 2.69 KB

Cheers from Miami :beers:

3 Likes
17

Absolutely, @rg305! We aim to (silently) please. :grin:

Cheers from the Rockies :beers:

2 Likes
18

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.