Renewal works on staging fails on production

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: mail.rubinsoftware.com

I ran this command: certbot renew -v --break-my-certs --force-renew --server “https://acme-v02.api.letsencrypt.org/directory

It produced this output:
Attempting to renew cert (mail.rubinsoftware.com) from /etc/certbot/renewal/mail.rubinsoftware.com.conf produced an unexpected error: Failed authorization procedure. mail.rubinsoftware.com (tls-sni-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Timeout during connect (likely firewall problem). Skipping.

My web server is (include version): There is an apache on this machine, but the certificate is for the mail server.

The operating system my web server runs on is (include version):
Linux mail 4.16.0-1-default #1 SMP PREEMPT Wed Apr 4 13:35:56 UTC 2018 (e16f96d) x86_64 x86_64 x86_64 GNU/Linux

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know): Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):

Basically this command works flawlessly, it just give me bad certs:

certbot renew -v --break-my-certs --force-renew --server “https://acme-staging-v02.api.letsencrypt.org/directory

This started when the auto renewal process failed. I have been trying to debug the process since. That it works on staging and fail on production is very confusing. I tried specifying the server because when I ran the debug I noticed that the url was ver 1 instead of 2 and since 2 was staging I decided to try ver 2 production

Hi,

The staging environment uses http-01… But because your server have requested an certificate before tls-sni-01 being disabled, your renewal challenges in production environment is by tls-sni.

Hense, please run the renew with the flag --preferred-challenge http to switch your challenges to http

Thank you

FWIW, you can use "cerbot renew --dry-run" to do roughly the same thing. (It issues staging certificates, but doesn't save them.)

https://mail.rubinsoftware.com/ times out for me. http://mail.rubinsoftware.com/ works. Is there a firewall blocking port 443?

Thank you, I got the certificate fixed for today. Now I want to make sure that the autorenewal will not break next time.

  1. I figured out the firewall that was blocking 443.
  2. I used the long form parameters to get my certificates updated.
  3. I cannot dry-run without getting en error:

mail:/etc/certbot # certbot renew -v --dry-run
–server value conflicts with --dry-run

I am guessing that my conf file is the issue,

mail:/etc/certbot # cat renewal/mail.rubinsoftware.com.conf

renew_before_expiry = 30 days

version = 0.26.1
archive_dir = /etc/certbot/archive/mail.rubinsoftware.com
cert = /etc/certbot/live/mail.rubinsoftware.com/cert.pem
privkey = /etc/certbot/live/mail.rubinsoftware.com/privkey.pem
chain = /etc/certbot/live/mail.rubinsoftware.com/chain.pem
fullchain = /etc/certbot/live/mail.rubinsoftware.com/fullchain.pem

Options used in the renewal process

[renewalparams]
authenticator = standalone
rsa_key_size = 4096
account = XXXXXXXXXXXXX
renew_hook = /root/bin/fixcourier.sh
pre_hook = /root/bin/stopApache.sh
post_hook = /root/bin/fixcourier.sh

Suggestions? Ideas?

Is that exactly what you're running? Is /etc/letsencrypt/cli.ini or /etc/certbot/cli.ini or a shell alias adding a --server argument?

1 Like

You are correct in cli.ini there is a

server = https://acme-v01.api.letsencrypt.org/directory

Taking that out allowed the dry run to run fine.

While I do not use this machine as webserver it is hosting some web based mail utilities so there is a copy of apache. What happens about 1/2 the time is that the pre_hook which is supposed to stop apache fails to do blocking access to the standalone authenticator.

I am wondering if I would be wiser to go ahead and let certbot add https to use the apache agent instead? If I understand correctly it uses apache itself instead of the stand alone to verify the domain and as such the shutdown should not be required?

You can also use certbot certonly -a apache if you don’t have a corresponding Apache virtual host or if you don’t want Certbot to install the certificate in your Apache configuration.

Thank you for all your help so far. This is getting better at every step.

So I allowed it to update https for the main virtual host mail.rubinsoftware.com, and now it is happy. When I tried adding the 2 extra virtual hosts figuring why not, I got this:

certbot -v --apache -d mail.rubinsoftware.com -d www1.rubinsoftware.com -d www1.goodstein.org


You have an existing certificate that contains a portion of the domains you
requested (ref: /etc/certbot/renewal/mail.rubinsoftware.com.conf)

It contains these names: mail.rubinsoftware.com

You requested these names for the new certificate: mail.rubinsoftware.com,
www1.rubinsoftware.com, www1.goodstein.org.

Do you want to expand and replace this existing certificate with the new
certificate?


(E)xpand/©ancel: E
Renewing an existing certificate

Reporting to user: The following errors were reported by the server:

Domain: www1.goodstein.org
Type: unauthorized
Detail: Invalid response from http://www1.goodstein.org/.well-known/acme-challenge/eryOEAeZ4siatbCZ0Hjkh4CxhJFAVNP4niToqO5K8bk: "<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/D" To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address. So I read the debug log and I can see where it modifies the apache configuration: Performing the following challenges: tls-sni-01 challenge for mail.rubinsoftware.com http-01 challenge for www1.goodstein.org http-01 challenge for www1.rubinsoftware.com Adding a temporary challenge validation Include for name: www1.rubinsoftware.com in: /etc/apache2/vhosts.d/www1.rubinsoftware.com.conf writing a pre config file with text: RewriteEngine on RewriteRule ^/\.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/certbot/http_challenges/$1 [END] writing a post config file with text:
        Require all granted
    </Directory>
    <Location /.well-known/acme-challenge>
        Require all granted
    </Location>

Creating backup of /etc/apache2/vhosts.d/www1.rubinsoftware.com.conf
Creating backup of /etc/apache2/httpd.conf
Adding Include /etc/apache2/vhosts.d/le_tls_sni_01_cert_challenge.conf to /files/etc/apache2/httpd.conf
writing a config file with text:

<VirtualHost *>
ServerName 7b581bb8a9c994fb6f4335502e602e82.68ef8cf57367d6b8a09ef2a99e2ad55d.acme.invalid
UseCanonicalName on
SSLStrictSNIVHostCheck on

LimitRequestBody 1048576

Include /etc/certbot/options-ssl-apache.conf
SSLCertificateFile /var/lib/certbot/gSI8gtBB_R3mCsMKUk3xZw5v4EOENfGZTvHFCrSae1k.crt
SSLCertificateKeyFile /var/lib/certbot/gSI8gtBB_R3mCsMKUk3xZw5v4EOENfGZTvHFCrSae1k.pem

DocumentRoot /var/lib/certbot/tls_sni_01_page/

Creating backup of /etc/apache2/httpd.conf
Waiting for verification…

But it seems that those changes may not have been in the proper format / location because they seem to be ignored:

{
  "type": "http-01",
  "status": "invalid",
  "error": {
    "type": "urn:ietf:params:acme:error:unauthorized",
    "detail": "Invalid response from http://www1.goodstein.org/.well-known/acme-challenge/eryOEAeZ4siatbCZ0Hjkh4CxhJFAVNP4niToqO5K8bk: \"\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003c!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\"\n  \"http://www.w3.org/TR/xhtml1/D\"",
    "status": 403
  },

  "type": "http-01",
  "status": "invalid",
  "error": {
    "type": "urn:ietf:params:acme:error:unauthorized",
    "detail": "Invalid response from http://www1.rubinsoftware.com/.well-known/acme-challenge/YXbwUulIK_2AjwzsDC1lG-MVZeRf7eE1UoHVVQhwNiI: \"\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003c!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\"\n  \"http://www.w3.org/TR/xhtml1/D\"",
    "status": 403
  },

Is there an east way for me to see the modified files as apposed to the snipets?

Do you have a <VirtualHost *> somewhere in your normal configuration?

If so, does it work correctly if you change that to <VirtualHost *:80> (assuming it's a HTTP virtual host)?

If you add the --debug-challenges option to the certbot command it will pause and wait for input between modifying the configuration and asking the CA to verify the challenge, so you can check how things are set up at that point.

2 Likes

DING! DING! DING!

Thank you. That was the last issue.

You have all been very helpful.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.