Unable to Create Certificates -- Is this related to today's service disruption?

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: billkochman.com, www.billkochman.com

I ran this command: sudo certbot certonly --webroot --webroot-path /Applications/MAMP/htdocs/Bill-Kochman-Com

It produced this output: Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Enter email address (used for urgent renewal and security notices) (Enter ‘c’ to
cancel): wordweaver777@gmail.com

Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf. You must
agree in order to register with the ACME server at
https://acme-v02.api.letsencrypt.org/directory

(A)gree/©ancel: a

Would you be willing to share your email address with the Electronic Frontier
Foundation, a founding partner of the Let’s Encrypt project and the non-profit
organization that develops Certbot? We’d like to send you email about our work
encrypting the web, EFF news, campaigns, and ways to support digital freedom.

(Y)es/(N)o: n
Please enter in your domain name(s) (comma and/or space separated) (Enter ‘c’
to cancel): billkochman.com www.billkochman.com
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for billkochman.com
http-01 challenge for www.billkochman.com
Using the webroot path /Applications/MAMP/htdocs/Bill-Kochman-Com for all unmatched domains.
Waiting for verification

Cleaning up challenges
Failed authorization procedure. billkochman.com (http-01): urn:ietf:params:acme:error:unknownHost :: The server could not resolve a domain name :: No valid IP addresses found for billkochman.com, www.billkochman.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.billkochman.com/.well-known/acme-challenge/RqFbPQQzN_tG5CB3fNtqYYPjl2_MtwX7a91ABw_BMgA: “\r<html amp=”" lang=“en”>\r \r <meta charset=“utf-8”>\r <script async src=“https://cdn.ampproject.org/v0”

IMPORTANT NOTES:

  • The following errors were reported by the server:

    Domain: billkochman.com
    Type: unknownHost
    Detail: No valid IP addresses found for billkochman.com

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A/AAAA record(s) for that domain
    contain(s) the right IP address.

  • The following errors were reported by the server:

    Domain: www.billkochman.com
    Type: unauthorized
    Detail: Invalid response from
    http://www.billkochman.com/.well-known/acme-challenge/RqFbPQQzN_tG5CB3fNtqYYPjl2_MtwX7a91ABw_BMgA:
    “\r<html amp=”" lang=“en”>\r \r <meta
    charset=“utf-8”>\r <script async
    src=“https://cdn.ampproject.org/v0”

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A/AAAA record(s) for that domain
    contain(s) the right IP address.

  • Your account credentials have been saved in your Certbot
    configuration directory at /etc/letsencrypt. You should make a
    secure backup of this folder now. This configuration directory will
    also contain certificates and private keys obtained by Certbot so
    making regular backups of this folder is ideal.

My web server is (include version): MAMP PRO 5.1.1

The operating system my web server runs on is (include version): OS X 10.11.6 El Capitan

My hosting provider, if applicable, is: Docomo Pacific

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

2

This was not related to today’s incident and appears to be a normal mis-configuration on the serverside.

3

Hi,

You’ll need to also set the correct IP address to billkochman.com


For your www site, I’m not sure what is the issue
 but it seems that your server is publishing a error page (with AMP enabled)

Maybe your web server does not have permission to use the webroot folder?

Thank you

4

cpu and stevenzhu,

As far as I can tell, this is false reporting on Certbot’s part. Although I did not mention it in my previous post, when I began having this problem earlier this morning, I did a number of things to try to resolve it.

  1. I verified the server path to billkochman.com and www.billkochman.com in my server software setup – MAMP PRO. It is correct for both.

  2. I visited my registrar’s website to make sure that my external static IP address had not somehow become corrupted in my “A” records. It is fine, and has been the same for quite a few years now.

  3. I verified my internal LAN IP address for the server machine. Again, no problem.

  4. I updated Homebrew – it is used to install Certbot on OS X.

  5. I updated Certbot to the lateset version.

When nothing failed to fix Certbot’s reported problems, I decided to delete my LetsEncrypt installation altogether, and start from scratch. So, I deleted the entire “letsencrypt” folder in “private/etc” as well as the logs for LetsEncrypt in “private/var/log/letsencrypt”.

After rebooting my iMac, and restarting the web server, I tried again from scratch to create brand new certificates – not renewals – being as I had already deleted the aforementioned two folders from my hard drive and rebooted the machine and server. The results are noted in my first post above.

If I type https://www.billkochman.com or https://billkochman.com, or http://www.billkochman.com or http://billkochman.com in my web browser, I am taken to the SSL version of my website without problems. I am not receiving any kind of server errors, including no AMP errors. My website has been AMP-compliant for a number of years now.

I have been using CertBot successfully for a number of years now without a problem, until this morning. Other than updating my server software when required, I have made no configuration changes to my web server, and renewing my certificates every 90 days has worked fine until now.

So I honestly don’t know what to think, or how to fix this. My current certificates expire in nine days. Thankfully, my SSL works with copies of the LetsEncrypt certificates installed in my actual MAMP PRO installation, and not with the originals in the LetsEncrypt folder in “private/etc”. So, while my web server’s SSL is not currently being affected by these problems, it will be when the current certificates expire in nine days.

Steven, you mentioned that you are being sent to an error page. Is it a 404 page, or something else? I am wondering if I see no such errors because of the localhost loopback 127.0.0.1?

Any further assistance would be greatly appreciated. I don’t know what else to do. Thanks!

5

You definitely don't have an A record for billkochman.com, which is the first problem:

$ dig @dns113.c.register.com billkochman.com A

; <<>> DiG 9.9.4-RedHat-9.9.4-51.el7_4.2 <<>> @dns113.c.register.com billkochman.com A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59611
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1680
;; QUESTION SECTION:
;billkochman.com.               IN      A

;; AUTHORITY SECTION:
billkochman.com.        14400   IN      SOA     dns038.a.register.com. root.register.com. 2014090808 28800 7200 604800 14400

;; Query time: 31 msec
;; SERVER: 216.21.235.113#53(216.21.235.113)
;; WHEN: Tue Oct 02 23:14:45 EDT 2018
;; MSG SIZE  rcvd: 103
1 Like
6

For the second, www.billkochman.com, it seems that your web server is responding with some HTML page instead of the challenge file. The bit we can see looks kind of like some CDN’s DDOS protection. Can you put a test file in /.well-known/acme-challenge and call it test so that we can attempt to access it via www.billkochman.com/.well-known/acme-challenge? Seeing the full response your server/CDN gives would be helpful in determining what’s messing up this process.

7

Jared,

Let me tackle your first comment first. I am very confused by it, perhaps due to my lack of expertise in this area.

If I log in to my account at register.com, click on the link for my domain and scroll down to the section called “ADVANCED TECHNICAL SETTINGS”, and click on the “Edit IP Address Records” link next to “A”, it takes me to a new page where it shows this:

  • .billkochman.com points to 202.128.4.177

If I understand that correctly, that means that ALL requests should go to my static IP address.

Isn’t the above my “A” record, or am I mistaken?

Whatever the case may be, that setting has worked for me for the past eighteen years, although the IP address has changed a few times over the years due to changes at my ISP.

8

Normally it should look like this
@ points to 202.128.4.177

The docs for register.com is unclear... But i managed to find this:
https://support.gospacecraft.com/hc/en-us/articles/206455944-Change-DNS-records-on-Register

Thank you

9

Just to clarify, in case it isn’t apparent from my previous post, that is actually an asterisk – not a bullet – before the domain name. In other words, it is a catch-all/wildcard to direct everything for my domain to my static IP address.

10

However, please note that wildcard record will only set all first-level subdomains (e.g. www.example.com, abc.example.com, ahsd.example.com, a-really-long-domain.example.com) to this IP address, example.com is not covered in the *, you still need to set the root domain manually.

11

Okay, if I understand you correctly, when you say “you still need to set the root domain manually”, you mean that the “A” record also needs to have this, correct?:

.billkochman.com points to 202.128.4.177

In other words, there needs to be an entry with the asterisk, and one without it, correct?

After I added that just now, on the page I then saw this:

  • points to 202.128.4.177
    @ points to 202.128.4.177

That first one is an asterisk.

If that is all correct now, I think what may have happened is that earlier this morning when I checked my “A” record, I removed the second entry, because I didn’t think it was necessary, being as I erroneously assumed that the asterisk covered everything.

Please let me know if that fixes things from your end. Then we can work on your second comment above. BTW, it may take 24 hours for that registrar record change to kick in. I hope not.

12

Yup!

Congratulations, now the first half of the issue is resolved. (root domain not reporting unknownHost)...

It's time for the second issue... (which might also be applied to the root domain when you run certbot again)

Let's Encrypt query the authoritative DNS server directly, so the IP (unknownHost issue is confirmed to resolved)

Thank you

13

It looks like all URLs on http://www.billkochman.com/ get redirected to https://www.billkochman.com/errors/404.html.

It looks like that’s the page Let’s Encrypt got when they tried to access http://www.billkochman.com/.well-known/acme-challenge/RqFbPQQzN_tG5CB3fNtqYYPjl2_MtwX7a91ABw_BMgA earlier.

You need to check the web server configuration, and make sure that the ACME validation requests either don’t get redirected, or do get redirected to the appropriate file.

14

Yes, due to AMP’s strict requirements, I had to add a directive to my httpd.conf file to forward all http requests to https instead. Otherwise, Google will drop me from SERPs and kill my web presence, which I have worked so hard to build over the past twenty-one years.

I am going to need some help here. Does this mean that I need to add another redirect directive to my httpd.conf file so that ACME validation requests are sent to the right place? If so, exactly what would such a directive look like?

Please note that I am really a newbie when it comes to httpd.conf files and directives. It took me a long time – and a lot of frustration and research – just to figure out what directive to use in MAMP PRO to get everything to redirect from http to https. Thanks!

15

Okay, after making the above change in my “A” record, I ran the Certbot command again. Now it is spitting out the following:

Performing the following challenges:
http-01 challenge for billkochman.com
http-01 challenge for www.billkochman.com
Using the webroot path /Applications/MAMP/htdocs/Bill-Kochman-Com for all unmatched domains.
Waiting for verification

Cleaning up challenges
Failed authorization procedure. www.billkochman.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.billkochman.com/.well-known/acme-challenge/K8xndCQLkZwR6Yc_ymkJ-4MSECzrAXOExTs7O2FAtvI: “\r<html amp=”" lang=“en”>\r \r <meta charset=“utf-8”>\r <script async src=“https://cdn.ampproject.org/v0”

IMPORTANT NOTES:

Jared, I just found an invisible folder inside of my public html folder called “well-known”. There is no subfolder in it called “acme-challenge”. Should I manually create it, or does Certbot need to create that itself?

When you say to create a test file, do you mean an HTML file?

BTW, I do not use any CDNs, at least not directly. I was formerly using the CDN that comes with Jetpack for my WP blog, but I discontinued using it because it created more problems than it was worth. Besides, that was just for my blog, and not for my entire website.

Of course, I don’t know what may be occurring further upstream with my ISP or further up the line.

16

Certbot automatically creates -- and subsequently deletes -- the acme-challenge directory. That part is probably working okay.

The problem is the HTTP redirect to https://www.billkochman.com/errors/404.html -- that page isn't the file Let's Encrypt is looking for.

How is the redirect being done in the web server configuration?

If you use something like "Redirect / https://www.billkochman.com/", Let's Encrypt should have no problem, since e.g. http://www.billkochman.com/.well-known/acme-challenge/K8xndCQLkZwR6Yc_ymkJ-4MSECzrAXOExTs7O2FAtvI would be redirected to https://www.billkochman.com/.well-known/acme-challenge/K8xndCQLkZwR6Yc_ymkJ-4MSECzrAXOExTs7O2FAtvI. But if you need the current redirect for other reasons, it gets a little more complicated.

17

I just looked at my httpd.conf file, and while I thought I had added the redirect directive for AMP, as it turns out, I actually added it so my domain name could be included in the HSTS Preload List. I tried all kinds of Apache redirect codes that I found on the web, but the only one that I could get to work in MAMP PRO’s httpd.conf file is the following one:

For HSTS preload list, we must redirect to https://billkochman.com first,

and then to https://www.billkochman.com after that

ServerAdmin wordweaver777@gmail.com
ServerName billkochman.com
RedirectMatch 301 (.) https://billkochman.com$1
RedirectMatch 301 (.) https://www.billkochman.com$1

So that is how I am redirecting from http to https.

On a side note, oddly enough, my domain has still not been added to the HSTS Preload List because their website is reporting an error:

Response error: Multiple HSTS headers (number of HSTS headers: 2).

But in my httpd.conf file, I only have the following ONE time:

##Add this to enable HSTS on my site

Header set Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"

Anyway, at this point, the main issue is getting Certbot working again. So, if you have any idea regarding how I can get the ACME challenge working right, please let me know.

Thanks again for your time.

18

Regarding the side note, it’s true:

< HTTP/1.1 200 OK
< Date: Wed, 03 Oct 2018 04:41:38 GMT
< Server: Apache
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Last-Modified: Fri, 10 Aug 2018 04:21:16 GMT
< ETag: "6c98eab-773d-5730d131d2700"
< Accept-Ranges: bytes
< Vary: Accept-Encoding,User-Agent
< Content-Encoding: gzip
< Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
< Content-Length: 7080
< Content-Type: text/html

Maybe the second Strict-Transport-Security header is being set in a .htaccess file or something?

19

Mnordhoff, you were correct regarding the HSTS issue. As it turns out, the web server developer had ALREADY included an HSTS directive in the “httpd-ssl.conf” file that I was not aware of. Once I realized that, I updated its length, and then I deleted my own HSTS directive from the “httpd.conf” file. However, despite that, both the HSTS Preload List website and SSL Labs were still reporting double HSTS headers. That is when I discovered that I had also added the same HSTS directive to my domain’s top level .htaccess file. Once I removed that, the HSTS Preload List website approved my submission. However, the SSL Labs is still complaining about two HSTS headers. However, I suspect that it may be because it is slow to update its database, since I had just tested on that site.

But getting back to the main issue, after fixing that problem, I ran Certbot’s certificate command in the OS X Terminal again, hoping that it would now be fixed. However, sadly, the same redirect problem continues to exist:

The following errors were reported by the server:

Domain: www.billkochman.com
Type: unauthorized
Detail: Invalid response from
http://www.billkochman.com/.well-known/acme-challenge/VWdULXgS1z32Hd-TBNf3Uxkg4vU_N-D1fmUpPaQKHhA

So, do you – or anyone else here – see anything wrong with my redirect code in my “httpd.conf” file which might be affecting Certbot? This code has worked for a long time, so I don’t understand why Certbot would suddenly start having problems with it:

ServerAdmin wordweaver777@gmail.com
ServerName billkochman.com
RedirectMatch 301 (. ) https://billkochman.com$1
RedirectMatch 301 (.) https://www.billkochman.com$1

NOTE: In the above directive there is actually a period and an asterisk between the parentheses.

Is there something I can append to the above directive which specifically targets Certbot and sends it to the right file in the invisible “well-known” folder?

Thanks again.

20

Yup...

Could you please make a test file under your document root/.well-known/acme-challenge/ ?

Can you also please check if your program allows hidden directory being served?

Thank you