Always getting fallback-*.perm files!

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: asp-services.de

I ran this command:
https://letsdebug.net/asp-services.de/122682

It produced this output:
## est result for asp-services.de using http-01
All OK!
OK
No issues were found with asp-services.de

My web server is (include version):
Apache/2.4.43

The operating system my web server runs on is (include version):
FreeBSD 11.3

My hosting provider, if applicable, is:
Hetzner

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):
mod_md (version 2.2.7)

+++ PROBLEM +++

I am always only getting back FALLBACK-perm-files although the tests mentioned before were positive.

Kind regards
testit

======================================================

CURL

Blockquote
curl -v https://acme-v02.api.letsencrypt.org/directory

  • Trying 172.65.32.248:443…
  • Connected to acme-v02.api.letsencrypt.org (172.65.32.248) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
  • successfully set certificate verify locations:
  • CAfile: /usr/local/share/certs/ca-root-nss.crt
    CApath: none
  • TLSv1.2 (OUT), TLS header, Certificate Status (22):
  • TLSv1.2 (OUT), TLS handshake, Client hello (1):
  • TLSv1.2 (IN), TLS handshake, Server hello (2):
  • TLSv1.2 (IN), TLS handshake, Certificate (11):
  • TLSv1.2 (IN), TLS handshake, Server key exchange (12):
  • TLSv1.2 (IN), TLS handshake, Server finished (14):
  • TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
  • TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.2 (OUT), TLS handshake, Finished (20):
  • TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
  • TLSv1.2 (IN), TLS handshake, Finished (20):
  • SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
  • ALPN, server accepted to use h2
  • Server certificate:
  • subject: CN=acme-v01.api.letsencrypt.org
  • start date: Mar 12 18:07:07 2020 GMT
  • expire date: Jun 10 18:07:07 2020 GMT
  • subjectAltName: host “acme-v02.api.letsencrypt.org” matched cert’s “acme-v02.api.letsencrypt.org
  • issuer: C=US; O=Let’s Encrypt; CN=Let’s Encrypt Authority X3
  • SSL certificate verify ok.
  • Using HTTP2, server supports multi-use
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  • Using Stream ID: 1 (easy handle 0x803aa5800)

GET /directory HTTP/2
Host: acme-v02.api.letsencrypt.org
user-agent: curl/7.69.1
accept: /

=======================================================

md-status-handler output of mod_md

{
“version”: “2.2.7”,
“managed-domains”: [
{
“name”: “www.asp-services.de”,
“domains”: [
www.asp-services.de”,
asp-services.de
],
“contacts”: [
mailto:you@example.com”,
mailto:anomized@anonmail.de
],
“transitive”: 1,
“ca”: {
“proto”: “ACME”,
“url”: “https://acme-v02.api.letsencrypt.org/directory”,
“agreement”: “accepted”
},
“state”: 1,
“renew-mode”: 1,
“renew-window”: “33%”,
“warn-window”: “10%”,
“must-staple”: true,
“proto”: {
“acme-tls/1”:
},
“stapling”: false,
“watched”: true,
“renew”: true,
“renewal”: {
“name”: “www.asp-services.de”,
“finished”: false,
“notified”: false,
“next-run”: “Mon, 13 Apr 2020 17:15:34 GMT”,
“last-run”: “Mon, 13 Apr 2020 17:04:54 GMT”,
“errors”: 8,
“last”: {
“status”: 20014,
“status-description”: “Internal error (specific information not available)”,
“detail”: “Unsuccessful in contacting ACME server at https://acme-v02.api.letsencrypt.org/directory. If this problem persists, please check your network connectivity from your Apache server to the ACME server. Also, older servers might have trouble verifying the certificates of the ACME server. You can check if you are able to contact it manually via the curl command. Sometimes, the ACME server might be down for maintenance, so failing to contact it is not an immediate problem. Apache will continue retrying this.”,
“activity”: “Contacting ACME server for www.asp-services.de at https://acme-v02.api.letsencrypt.org/directory
}
}
}
]
}

Hi @testit

what says

traceroute acme-v02.api.letsencrypt.org

What client do you use? Is there a better log?

He already told us:


I’m not sure how much experience there is on this community with mod_md. Luckily, the developer has a presence here: @icing Perhaps he might be of assistance.

Hallo Jürgen,

obviously I have the same problem like another poster here:

Because

traceroute acme-v02.api.letsencrypt.org

prints out following:

In httpd-error-log I find the same detailed information like in the posted output od mod_md (detail:):

[Mon Apr 13 19:38:56.803885 2020] [md:error] [pid 5899] (20014)Internal error (specific information not available): AH10056: processing www.asp-services.de: Unsuccessful in contacting ACME server at https://acme-v02.api.letsencrypt.org/directory. If this problem persists, please check your network connectivity from your Apache server to the ACME server. Also, older servers might have trouble verifying the certificates of the ACME server. You can check if you are able to contact it manually via the curl command. Sometimes, the ACME server might be down for maintenance, so failing to contact it is not an immediate problem. Apache will continue retrying this.

Viele Grüße
testit

Thanks!

I already sent an email to icing this afternoon.

Best regards
testit

I have two servers provided by Hetzner.

Starting a
traceroute acme-v02.api.letsencrypt.org

from the other server produces the same output (stopping after 5 162.158.84.254).

But what does that mean now for my problem?
I ask because
curl -v https://acme-v02.api.letsencrypt.org/directory
obviously works fine (look at my first post please).

Thanks and regards
testit

There are other users (other providers) with the same blocked traceroute.

Reducing the MTU to 1100 or lower has helped.

Play with the

ping -M do -s 1453 -c 4 acme-v02.api.letsencrypt.org

command. -s is the packet size.

If you see an error message, reduce the -s value. If it works, change your TCP config.

Hallo Jürgen,

 ping -M do -s 1453 -c 4 acme-v02.api.letsencrypt.org

does not work correcty for me:

ping: invalid message: `d’

Maybe it depends on a special ping implementation in FreeBSD?

On my other server at Hetzner - on which
traceroute acme-v02.api.letsencrypt.org
also stops after
5 162.158.84.254 (162.158.84.254)

I nevertheless do not have the same problems and the certificates were renewed yesterday.

So how important is it that
traceroute acme-v02.api.letsencrypt.org
does not stop when the curl connection DOES obviously work?

Thanks in advance and best regards
testit

Obviously the FreeBSD command of your LINUX ping is the following one:
ping -D -s 1453 -c 4 acme-v02.api.letsencrypt.org

RESULT:

**ping -D -s 1453** -c 4 acme-v02.api.letsencrypt.org

PING ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com (172.65.32.248): 1453 data bytes
1461 bytes from 172.65.32.248: icmp_seq=0 ttl=59 time=5.822 ms
1461 bytes from 172.65.32.248: icmp_seq=1 ttl=59 time=5.216 ms
1461 bytes from 172.65.32.248: icmp_seq=2 ttl=59 time=5.219 ms
1461 bytes from 172.65.32.248: icmp_seq=3 ttl=59 time=5.191 ms

--- ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 5.191/5.362/5.822/0.266 ms

Greetings
testit

I reduced the MTU stepwise down to 1000 - without any other result for the traceroute result.

ifconfig vtnet0 mtu 1000

# traceroute acme-v02.api.letsencrypt.org

traceroute to ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com (172.65.32.248), 64 hops max, 40 byte packets

 1  static.161.89.9.5.clients.your-server.de (5.9.89.161)  1.794 ms  12.359 ms  2.336 ms
 2  core23.fsn1.hetzner.com (213.239.245.109)  1.350 ms
    core24.fsn1.hetzner.com (213.239.245.149)  0.783 ms  0.864 ms
 3  core0.fra.hetzner.com (213.239.252.41)  5.541 ms
    core4.fra.hetzner.com (213.239.229.73)  10.456 ms  5.811 ms
 4  core9.fra.hetzner.com (213.239.224.178)  5.865 ms
    core9.fra.hetzner.com (213.239.252.18)  5.848 ms  6.301 ms
 5  162.158.84.254 (162.158.84.254)  6.826 ms  5.911 ms  6.399 ms
 6  * * *
 7  * * *
 8  * * *
 9  * * *
...

Regards
testit

The other article suspects that it might be some networking issue with Hetzner. But maybe we can add some more debugging from our end.

First, if you say that curl is working against https://acme-v02.api.letsencrypt.org/directory, did you test it from the same machine? If there is a source ip block somewhere, this would confirm or debunk it.

Second, you can trace in mod_md up to the byte of what is send and received by mod_md. Set LogLevel md:trace8 to see this in Apache’s error log.

The error mentioned happens right at the start, e.g. the first request fails, so the amount of log should be limited. There will be a lot of chatter by the module regading its regular checks on all your domains, though.

Hi,

thanks for your reply!

Yes - the curl command was started on the same machine.

Currently I set LogLevel md:trace2
I will try it with LogLevel md:trace8 during this day.

On another Hetzner server I am using md_mod v1.x under Apache 2.4 and do not have the same problems although I get the same traceroute problem there. Strange!

Kind regards
testit

The v1.x mod_md will use the older ACME API (https://acme-v01.api.letsencrypt.org/directory) which may be resolved differently for you.

But for testing, you could configured that url for the v2.x module with MDCertificateAuthority.

Hi,

Making use of LogLevel md:trace8 does not result in another output of md-status or in the apache error log.

Kind regards
testit

Apache error.log

[Wed Apr 15 08:34:19.860787 2020] [md:error] [pid 68453] (20014)Internal error (specific information not available): AH10056: processing www.asp-services.de: Unsuccessful in contacting ACME server at <https://acme-v02.api.letsencrypt.org/directory>. If this problem persists, please check your network connectivity from your Apache server to the ACME server. Also, older servers might have trouble verifying the certificates of the ACME server. You can check if you are able to contact it manually via the curl command. Sometimes, the ACME server might be down for maintenance, so failing to contact it is not an immediate problem. Apache will continue retrying this.

/md-status

{
  "version": "2.2.7",
  "managed-domains": [
    {
      "name": "www.asp-services.de",
      "domains": [
        "www.asp-services.de",
        "asp-services.de"
      ],
      "contacts": [
        "mailto:you@example.com",
        "mailto:test@asp-services.de"
      ],
      "transitive": 1,
      "ca": {
        "proto": "ACME",
        "url": "https://acme-v02.api.letsencrypt.org/directory",
        "agreement": "accepted"
      },
      "state": 1,
      "renew-mode": 1,
      "renew-window": "33%",
      "warn-window": "10%",
      "must-staple": true,
      "proto": {
        "acme-tls/1": []
      },
      "stapling": false,
      "watched": true,
      "renew": true,
      "renewal": {
        "name": "www.asp-services.de",
        "finished": false,
        "notified": false,
        "next-run": "Wed, 15 Apr 2020 06:38:34 GMT",
        "last-run": "Wed, 15 Apr 2020 06:38:24 GMT",
        "errors": 2,
        "last": {
          "status": 20014,
          "status-description": "Internal error (specific information not available)",
          "detail": "Unsuccessful in contacting ACME server at <https://acme-v02.api.letsencrypt.org/directory>. If this problem persists, please check your network connectivity from your Apache server to the ACME server. Also, older servers might have trouble verifying the certificates of the ACME server. You can check if you are able to contact it manually via the curl command. Sometimes, the ACME server might be down for maintenance, so failing to contact it is not an immediate problem. Apache will continue retrying this.",
          "activity": "Contacting ACME server for www.asp-services.de at https://acme-v02.api.letsencrypt.org/directory"
        }
      }
    }
  ]
}

Testing it with the older API https://acme-v01.api.letsencrypt.org/directory
leads to the same result.

Kind regards
testit

Now unfortunately I have the next problem:

I accidentally deleted the folder on my 2nd server with the certificates that had been created with ACME v1. When I try to generate them again, I get the following message:

[md:warn] [pid 87291] (13)Permission denied: acme problem urn:acme:error:unauthorized: Account creation on ACMEv1 is disabled. Please upgrade your ACME client to a version that supports ACMEv2 / RFC 8555. see https ://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430 for details.

And since ACME v2 - as described - also leads to the errors I described at the beginning, none of my domains has a Letsencrypt SSL certificate anymore.

Regards
testit

I move to certbot meanwhile which works correctly for me!

Kind regards
testit

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.