Lost in large error message and weird traceroute

Hi all!
My letsencrypt certificate suddenly was no longer renewed after working without issues for a very long time. I checked and got the error, that the acme-0x... servers are not reachable. I read through the forum and found that I should check traceroute:

traceroute acme-v01.api.letsencrypt.org
traceroute to acme-v01.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
1 fritz.box (192.168.10.1) 0.420 ms 0.469 ms 0.784 ms
2 62.155.246.182 (62.155.246.182) 8.232 ms 10.989 ms 11.017 ms
...
8 * * *
9 * * *
...
30 * * *
So it ends in stars.
Using my Notebook connected to the internet via my phone traces the same IP but even it sends the packet through different routers it ends up in stars.

And I found that I should try again with curl:

curl https://acme-v01.api.letsencrypt.org/directory
{
"LKblUkqa_oE": "Adding random entries to the directory",
"key-change": "https://acme-v01.api.letsencrypt.org/acme/key-change",
"meta": {
"caaIdentities": [
"letsencrypt.org"
],
"terms-of-service": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
"website": "https://letsencrypt.org"
},
"new-authz": "https://acme-v01.api.letsencrypt.org/acme/new-authz",
"new-cert": "https://acme-v01.api.letsencrypt.org/acme/new-cert",
"new-reg": "https://acme-v01.api.letsencrypt.org/acme/new-reg",
"revoke-cert": "https://acme-v01.api.letsencrypt.org/acme/revoke-cert"
So this looks like I can address something on the other side, even I cannot traceroute it?

I have no extra firewall hardware running, ports 80 and 443 are forwarded to the server and there were no configuration changes since letsencrypt had successfully updated it's certificates last time. But I have to admit that I regularly install updates on that platform as they are provided by the OMV team.

There where some reports like mine in the forums, unfortunately they end in "I got it solved!" without proper description or they where running on a totally different platform through real firewalls and such. I just have a home-router.


My domain is: astralix.mooo.com

I ran this command: sudo certbot renew

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/omv.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for astralix.mooo.com
Waiting for verification...
Challenge failed for domain astralix.mooo.com
Cleaning up challenges
Attempting to renew cert (omv) from /etc/letsencrypt/renewal/omv.conf produced an unexpected error: Challenges failed for all domains. Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/omv/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/omv/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

I do have the full log, but I am not sure which parts have to be crossed out to not to expose my certificates or keys. But the only error that is repeated I copy this part:

2019-10-27 21:46:17,126:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2019-10-27 21:46:17,129:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org:443
2019-10-27 21:46:17,131:WARNING:certbot.renewal:Attempting to renew cert (omv) from /etc/letsencrypt/renewal/omv.conf produced an unexpected error: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fcefccee550>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)). Skipping.
2019-10-27 21:46:17,140:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 57, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/lib/python3.5/socket.py", line 733, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

My web server is (include version):
nginx 1.10.3-1+deb9u3

The operating system my web server runs on is (include version):
openmediavault 4.1.26-1 (Arrakis)
based on Debian Stretch amd64

My hosting provider, if applicable, is:
n/a dynamic IP service

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
command line and openmediavault control panel

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 0.28.0

Hi @Astralix

that's another problem, your client can't find an ip address.

Perhaps use the static ip address

172.65.32.248 acme-v02.api.letsencrypt.org

in your hosts file.

If tracert doesn't work, looks like a problem of your configuration. Should be something like

D:\temp>tracert -4 acme-v02.api.letsencrypt.org.

1 <1 ms <1 ms <1 ms fritz.box [192.168.0.1]
2 5 ms 5 ms 4 ms 62.155.240.117
3 6 ms 6 ms 6 ms 217.239.55.2
4 6 ms 6 ms 6 ms 217.239.55.2
5 6 ms 6 ms 7 ms lag-10.edge4.Berlin1.Level3.net [4.68.73.5]
6 7 ms 6 ms 14 ms ae-1-3502.edge3.Berlin1.Level3.net [4.69.159.1]
7 7 ms 6 ms 6 ms unknown.Level3.net [212.162.40.34]
8 6 ms 6 ms 6 ms 172.65.32.248

Perhaps reduce your MTU 1500 -> 1400, that has helped sometimes.

Hi Juergen!
Thanks for your reply. The DNS resolution seems to work, as the name is correctly resolved to the same IP, testing the traceroute command from two totally different networks. Adding the IP to the hosts file, doesn't change anything.

And in case that is important: Internally IPv6 is completely disabled.

MTU of the NIC was set to default (0) so I set it to 1400 manually. Traceroute still fails, however now the letsencrypt server sent something:

"identifier": {
"type": "dns",
"value": "astralix.mooo.com"
},
"status": "invalid",
"expires": "2019-11-04T09:41:41Z",
"challenges": [
{
"type": "http-01",
"status": "invalid",
"error": {
"type": "urn:ietf:params:acme:error:unauthorized",
"detail": "Invalid response from http://astralix.mooo.com/.well-known/acme-challenge/Ln3zlsMsELw0ZxjYN3qyk9cJu-Kt8fswMpu2kF4ThKU [84.163.24.174]: "\u003chtml\u003e\r\n\u003chead\u003e\u003ctitle\u003e400 The plain HTTP request was sent to HTTPS port\u003c/title\u003e\u003c/head\u003e\r\n\u003cbody bgcolor=\"white\"\u003e\r\n\u003ccenter\u003e\u003ch1\u003e400 B"",
"status": 403
},
So my problem shifted closer to my location....

@Astralix A few things:

If your curl works, but your traceroute doesn't, I think it's best to conclude that making a connection to the ACME server in essence works, but there is a problem with your traceroute. This could be by many things. For example, by default, traceroute uses ICMP probes. These probes can behave differently compared to TCP packets to port 443. For example, by firewalls. You can however generate TCP SYN probes on port 443 with: traceroute -T -p 443 That should give proper results.

Another thing: your non-debugging-output gives a few signals it can connect to the ACME server. For example, it mentions:

http-01 challenge for astralix.mooo.com
Waiting for verification...
Challenge failed for domain astralix.mooo.com

This would suggest certbot successfully connected to the ACME server to get a challenge type and token.
However, the reason why your challenge failed isn't copy/pasted in your log. Certbot should give you some more details about that, even without the debugging log.

The name resolution failure is strange. It would suggest certbot can't connect to the ACME server because of a DNS failure, but in the log above it clearly can connect to the ACME server.. And it doesn't have anything to do with failing challenges..
The debug log shouldn't provide sensitive information like private keys.

Your nginx configuration is completely wrong. It seems nginx is configured to "speak" HTTPS over port 80. This isn't correct, it should only speak HTTPS over port 443. It should speak HTTP over port 80.

1 Like

Does “curl -v https://acme-v01.api.letsencrypt.org/directory” work? What’s the output?

And what does “curl -v https://acme-v02.api.letsencrypt.org/directory” output?

Edit: Also:

dig acme-v01.api.letsencrypt.org
dig acme-v01.api.letsencrypt.org aaaa
dig acme-v02.api.letsencrypt.org
dig acme-v02.api.letsencrypt.org aaaa
1 Like

He already successfully curl'ed the directory. I think the errors aren't related with ACME server connectivity.

ACME is reachable and the error message changed in the large log after modifying the default MTU set to 0 to 1400 fixed.

@Osiris Yes, it looks like something messed up ngnix. I had the option to force https enabled in my OMV via GUI configuration. I disabled it, just to test the access from outside and found chrome is still rejecting the access cause of an invalid certificate.

I wonder how that has happened...

Now

the connection problem is fixed, a new order is created, but the validation doesn't work.

Checked your domain - https://check-your-website.server-daten.de/?q=astralix.mooo.com - now there is a Bad Gateway, http status 502.

Checking something like

http://astralix.mooo.com/.well-known/acme-challenge/1234

a http status 404 - Not Found is expected.

1 Like

Thank you all for pointing me into the right direction!

The issue was caused by something that added a redirection on port 80 to my nextcloud. So OMV was no longer responsible for port 80. I moved the nextcloud out of the way and now letsencrypt has successfully renewed its certificate as OMV has control over ports 80 and 443 again.

After waiting two minutes, my page is available again.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.