Trying to expand certs to other subdomains

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

I ran this command:
/usr/bin/certbot --apache --expand -d louise.ingber.com

It produced this output:
/usr/bin/certbot --apache --expand -d louise.ingber.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for louise.ingber.com

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority repo:
Domain: louise.ingber.com
Type: unauthorized
Detail: Invalid response from http://louise.ingber.com/.well-known/acme-challenge/6NX90bboyT57cIf"

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log.

My web server is (include version):
Linode.com
arch (GNU coreutils) 8.30

The operating system my web server runs on is (include version):
Distributor ID: Ubuntu
Description: Ubuntu 20.04.3 LTS
Release: 20.04
Codename: focal

My hosting provider, if applicable, is:
Linode.com

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 1.23.0

See my previous topic
" Can’t expand certs to other subdomains"

for submissions of additional info requested.

First of all, unfortunately the certificate managing options of Certbot are terrible. Adding --expand to the command line options with only the new hostname does not expand an existing certificate: Certbot wouldn't know what certificate to pick to expand.

To properly expand a certificate, you'd need to, unfortunately, do the following:

  • Run certbot certificates and find the certificate you want to expand;
  • Note the "certificate name" from the previous step, let's call that name CERTIFICATE NAME;
  • Note the list of hostnames after "Domains:" from the first step, lets call those hostnames OLDDOMAINS;
  • Run certbot --apache --expand --cert-name CERTIFICATENAME -d "OLDDOMAINS" -d NEWDOMAIN

This will make sure you're expanding the correct certificate and you include the previous domains and the new domain. I really wish there was an easier way, but unfortunately, there is none.

As for the actual error: I'm not sure why that's happening. Except for the certificate error, it seems IPv4 and IPv6 are returning the proper site.

What's the output of:

apachectl -t -D DUMP_VHOSTS
2 Likes

Hi. I attach the output of
apachectl -t -D DUMP_VHOSTS
as DUMP_VHOSTS.txt

Thanks.

Lester
DUMP_VHOSTS.txt (3.6 KB)

Hm, it probably has something to do with the usage of IP addresses for the VirtualHosts.

Could you do the same command (apachectl -t -D DUMP_VHOSTS) but now when Certbot has loaded the challenge?

You can do that by running the Certbot command you'd like to succeed (see my post above about expanding an existing certificate), but now add the option --debug-challenges. That option will pause Certbot when the challenge has been loaded into Apache. When Certbot is paused, please run the apachectl -t -D DUMP_VHOSTS command and show the output again.

2 Likes

I did not notice any "pause" but I believe I followed your directions. I attach DUMP_VHOSTS-debug.txt .

DUMP_VHOSTS.txt (3.6 KB)

The name did not take with a "-" so here is the file with a "_".

DUMP_VHOSTS.txt (3.6 KB)

That did not work:

09:16:48 ingber@linode# ~: /usr/bin/certbot --apache --expand --cert-name www.ingber.com -d "www.ingber.com blog.ingber.com default.ingber.com ingber.com lester.ingber.com lin.ingber.com" -d louise.ingber.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log


You are updating certificate www.ingber.com to include new domain(s):

You are also removing previously included domain(s):

Did you intend to make this change?


(U)pdate certificate/(C)ancel: U
Renewing an existing certificate for www.ingber.com blog.ingber.com default.ingber.com ingber.com lester.ingber.com lin.ingber.com and louise.ingber.com
An unexpected error occurred:
The server will not issue certificates for the identifier :: Error creating new order :: Cannot issue for "www.ingber.com blog.ingber.com default.ingber.com ingber.com lester.ingber.com lin.ingber.com": Domain name contains an invalid character
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

I tried with commas:

09:22:03 ingber@linode# ~: /usr/bin/certbot --apache --expand --cert-name www.ingber.com -d www.ingber.com,blog.ingber.com,default.ingber.com,ingber.com,lester.ingber.com,lin.ingber.com -d louise.ingber.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log


You are updating certificate www.ingber.com to include new domain(s):

You are also removing previously included domain(s):

Did you intend to make this change?


(U)pdate certificate/(C)ancel: U
Renewing an existing certificate for www.ingber.com and 6 more domains

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
Domain: blog.ingber.com
Type: unauthorized
Detail: Invalid response from http://blog.ingber.com/.well-known/acme-challenge/nUuL5xeb0OSrokqF6aILSWDIUqQEAwxoTLVIiE5e_QE [2600:3c01::f03c:91ff:fe93:e6f3]: "\n\n403 Forbidden\n\n

Forbidden

\n<p"

Domain: default.ingber.com
Type: unauthorized
Detail: Invalid response from http://default.ingber.com/.well-known/acme-challenge/CR2pyON3J1Imn2uhUC2KbFNlpbWiqdtnmthiA2hnDKs [2600:3c01::f03c:91ff:fe93:e6f3]: "\n\n403 Forbidden\n\n

Forbidden

\n<p"

Domain: ingber.com
Type: unauthorized
Detail: Invalid response from http://ingber.com/.well-known/acme-challenge/bc3neix-2cmN_nFfZlhAXgI8LWJ8aiM81nnDQl9r1F0 [2600:3c01::f03c:91ff:fe93:e6f3]: "\n\n403 Forbidden\n\n

Forbidden

\n<p"

Domain: lester.ingber.com
Type: unauthorized
Detail: Invalid response from http://lester.ingber.com/.well-known/acme-challenge/oSpiyp-B9qr1UJriLZhbc2Je8wjd4gpce5-aN2mtn74 [2600:3c01::f03c:91ff:fe93:e6f3]: "\n\n403 Forbidden\n\n

Forbidden

\n<p"

Domain: lin.ingber.com
Type: unauthorized
Detail: Invalid response from http://lin.ingber.com/.well-known/acme-challenge/WDBDwYjlYEaseBzb9ohpZhx5QgeEhGB47X1k1EqjFkA [2600:3c01::f03c:91ff:fe93:e6f3]: "\n\n403 Forbidden\n\n

Forbidden

\n<p"

Domain: louise.ingber.com
Type: unauthorized
Detail: Invalid response from http://louise.ingber.com/.well-known/acme-challenge/XXmBqMY-_-7293tjC2gXXDrPNi0QWNa2cMxh1QNp7Bs [2600:3c01::f03c:91ff:fe93:e6f3]: "\n\n403 Forbidden\n\n

Forbidden

\n<p"

Domain: www.ingber.com
Type: unauthorized
Detail: Invalid response from http://www.ingber.com/.well-known/acme-challenge/rZX4uTZvDJilUFKSunCQxOkxhuhegWrxj8kZWXfpsRY [2600:3c01::f03c:91ff:fe93:e6f3]: "\n\n403 Forbidden\n\n

Forbidden

\n<p"

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

Certbot often doesn't play nice when <VirtualHost> sections are using IP addresses instead of a wildcard (*). Do you actually require IP based virtual hosts?

If not, it might be better to combine the two <VirtualHost 173.255.212.226:80> and <VirtualHost [2600:3c01::f03c:91ff:fe93:e6f3]:80> sections into a single <VirtualHost *:80> section. And the same for port 443.

3 Likes

Certbot has been working fine with current VirtualHost setup for 7 domain names. I just want to add one more.

Lester

This I cannot reproduce.

# curl -6 -IL -A "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" http://lin.ingber.com/.well-known/acme-challenge/WDBDwYjlYEaseBzb9ohpZhx5QgeEhGB47X1k1EqjFkA
HTTP/1.1 302 Found
Date: Fri, 04 Mar 2022 18:56:56 GMT
Server: Apache
Location: https://lin.ingber.com/.well-known/acme-challenge/WDBDwYjlYEaseBzb9ohpZhx5QgeEhGB47X1k1EqjFkA
Content-Type: text/html; charset=iso-8859-1

HTTP/1.1 404 Not Found
Date: Fri, 04 Mar 2022 18:56:57 GMT
Server: Apache
Last-Modified: Sat, 25 Jan 2020 17:25:28 GMT
ETag: "859-59cfa290c9200;5d967ee104a83"
Accept-Ranges: bytes
Content-Length: 2137
Content-Type: text/html

# curl -4 -IL -A "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" http://lin.ingber.com/.well-known/acme-challenge/WDBDwYjlYEaseBzb9ohpZhx5QgeEhGB47X1k1EqjFkA
HTTP/1.1 302 Found
Date: Fri, 04 Mar 2022 18:57:11 GMT
Server: Apache
Location: https://lin.ingber.com/.well-known/acme-challenge/WDBDwYjlYEaseBzb9ohpZhx5QgeEhGB47X1k1EqjFkA
Content-Type: text/html; charset=iso-8859-1

HTTP/1.1 404 Not Found
Date: Fri, 04 Mar 2022 18:57:11 GMT
Server: Apache
Last-Modified: Sat, 25 Jan 2020 17:25:28 GMT
ETag: "859-59cfa290c9200;5d967ee104a83"
Accept-Ranges: bytes
Content-Length: 2137
Content-Type: text/html

Is your firewall playing nice? Is fail2ban messing with apache?

1 Like

It's the only suggestion I currently have. Alternatively, you could post your Apache configuration files.

2 Likes

If that were my apache, I would do it like this:

  • a file would define all the options common to the virtualhosts, ready to be included in each of them.
  • I would have a single port 80 virtualhost, *:80 and its only job would be to redirect to https on the same domain name. Optionally, you could make it answer acme challenges with a dedicated webroot (only for requests hitting .well-known/acme-challenge using a Location block)
  • I would have one file for each virtualhost (or one for all of them, it only matters if you want to use a2ensite and a2dissite) with the bare minimum, like:
    • ServerName
    • DocumentRoot
    • Include /etc/apache2/snippets/common_options.conf (this includes the certificate, because yours covers all names. If you get a certificate for each name, then every virtualhost gets its own certificate and key)
1 Like

Why are you using "-d" twice?
[I don't think that is critical - just wondering]

2 Likes

A missed opportunity:

curl -Ii6 http://louise.ingber.com/.well-known/acme-challenge/Test_File-1234
HTTP/1.1 302 Found
Date: Fri, 04 Mar 2022 23:09:31 GMT
Server: Apache
Location: https://louise.ingber.com/.well-known/acme-challenge/Test_File-1234
Content-Type: text/html; charset=iso-8859-1

A failed connection:

curl -Ii6 https://louise.ingber.com/.well-known/acme-challenge/Test_File-1234
curl: (51) SSL: no alternative certificate subject name matches target host name 'louise.ingber.com'
1 Like

Boulder doesn't check certificates, add a -k switch :wink:

(Also: I and i together? Experimentally it behaves like I alone -- just headers. i should be headers then body.)

# curl -I6k https://louise.ingber.com/.well-known/acme-challenge/Test_File-1234
HTTP/1.1 404 Not Found
Date: Fri, 04 Mar 2022 23:15:51 GMT
Server: Apache
Last-Modified: Tue, 25 Oct 2016 00:03:39 GMT
ETag: "529-53fa53d96ccc0;5d967f0c8dac9"
Accept-Ranges: bytes
Content-Length: 1321
Content-Type: text/html

If the server has no match for that FQDN, then I doubt it would find the right response.

1 Like

It will respond with the default virtualhost, which will have a certificate for a different FQDN. (In this case, it does)

1 Like

Correct "a response" and "the right response" are two very different things.
I suppose the IP based virtual host is largely to blame, but since it is Apache, well... you should already know how I feel about that.

1 Like