ASN1_CHECK_TLEN:wrong tag

Takios · March 8, 2018, 10:28am

Hello,

we experienced a problem with a certificate today. It got issued at 09:32:17 CET.
After including it into our server configuration, the server refused to start with the error message in the topic title.

My domain is:
globaltrends.sef-bonn.org

I ran this command:
/usr/bin/acme.sh --issue -d globaltrends.sef-bonn.org -w DOCROOT --server https://acme-v01.api.letsencrypt.org/directory --syslog 6 --reloadcmd /etc/init.d/apache2 reload

It produced this output:
[…]
[Thu Mar 8 09:32:17 CET 2018] Your cert is in /etc/acme.sh/globaltrends.sef-bonn.org/globaltrends.sef-bonn.org.cer
[Thu Mar 8 09:32:17 CET 2018] Your cert key is in /etc/acme.sh/globaltrends.sef-bonn.org/globaltrends.sef-bonn.org.key
[Thu Mar 8 09:32:17 CET 2018] The intermediate CA cert is in /etc/acme.sh/globaltrends.sef-bonn.org/ca.cer
[Thu Mar 8 09:32:17 CET 2018] And the full chain certs is there: /etc/acme.sh/globaltrends.sef-bonn.org/fullchain.cer
[…]

The apache log shows when trying to start:
[ssl:emerg] [pid 32149:tid 139410412] SSL Library Error: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag
[ssl:emerg] [pid 32149:tid 139410412] SSL Library Error: error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error (Type=X509)
[ssl:emerg] [pid 32149:tid 139410412] SSL Library Error: error:0906700D:PEM routines:PEM_ASN1_read_bio:ASN1 lib

My web server is (include version):
apache 2.4.29

SSL config in vhost:
SSLEngine on
SSLCertificateFile "/etc/acme.sh/globaltrends.sef-bonn.org/fullchain.cer"
SSLCertificateKeyFile “/etc/acme.sh/globaltrends.sef-bonn.org/globaltrends.sef-bonn.org.key”

I can login to a root shell on my machine (yes or no, or I don’t know):
yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
no

contents of fullchain.cer:
-----BEGIN CERTIFICATE-----
MIIFFTCCA/2gAwIBAgISA5w6T2F7fWoRSM/9pWXDdviVMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xODAzMDgwNzMyMTZaFw0x
ODA2MDYwNzMyMTZaMCQxIjAgBgNVBAMTGWdsb2JhbHRyZW5kcy5zZWYtYm9ubi5v
cmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC+jlIvCTzyrYHS5J+U
tNvxhVN6bzP9GtlOmxyAyify4/TmCGsSTZZDRJJ6+DBOM1ZoZAWyR0RsdmGyVhvD
OUlrEc1olB2IicGLg42jyMl9z/gJNxwtnDqP4VBx3/fQJyfGRc2dOMNyQ/PVWsuM
C0s5jSUQ9ep8gaQwS6jYcLaY+EvPEQ6W6fvimac7kkpGn1CiuciJNrkW+qvhoOw1
QKuHmdW8ikLNCunNp5vyDkYXHZRJ8REqT2c8TIVNJ9NILsxLW+J9PQtTrzmFZL/9
fG/8TYonjqDhxplY3sstUH/luMDQnINaab9NXYme3QYYoFToWVZk4Vxlcrz3j0RF
5vnvAgMBAAGjggIZMIICFTAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYB
BQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFCovIFMNoUkW
4o7CEBfMz4WkFBaqMB8GA1UdIwQYMBaAFKhKamMEfd265tE5t6ZFZe/zqOyhMG8G
CCsGAQUFBwEBBGMwYTAuBggrBgEFBQcwAYYiaHR0cDovL29jc3AuaW50LXgzLmxl
dHNlbmNyeXB0Lm9yZzAvBggrBgEFBQcwAoYjaHR0cDovL2NlcnQuaW50LXgzLmxl
dHNlbmNyeXB0Lm9yZy8wJAYDVR0RBB0wG4IZZ2xvYmFsdHJlbmRzLnNlZi1ib25u
Lm9yZzCB/gYDVR0gBIH2MIHzMAgGBmeBDAECATCB5gYLKwYBBAGC3xMBAQEwgdYw
JgYIKwYBBQUHAgEWGmh0dHA6Ly9jcHMubGV0c2VuY3J5cHQub3JnMIGrBggrBgEF
BQcCAjCBngyBm1RoaXMgQ2VydGlmaWNhdGUgbWF5IG9ubHkgYmUgcmVsaWVkIHVw
b24gYnkgUmVseWluZyBQYXJ0aWVzIGFuZCBvbmx5IGluIGFjY29yZGFuY2Ugd2l0
aCB0aGUgQ2VydGlmaWNhdGUgUG9saWN5IGZvdW5kIGF0IGh0dHBzOi8vbGV0c2Vu
Y3J5cHQub3JnL3JlcG9zaXRvcnkvMA0GCSqGSIb3DQEBCwUAA4IBAQAPcK4Bfhay
hJJfiGOy6Z1SwZ/6iFZwlqWBVi/pBZGpUusVNa1uIC90lJTuKVV60139vvLMUf7S
LTnLK0pFD0HO8teqVBiF9WAqmWUKM/v7qxu9Oca2wL6xtZBldaBeTkn7rchm/TYL
+x40ojcshiQjpQ101x45gQMZni4WCGldGR9tzWIJixMP0wwR6QSwVTAll/E4sj59
jRAEj6tDYik5DpQIfMQWEJwx6D4bwPrrvwtW+NUN6KosBWj4kyQSfSyDZXEOcTcr
dk93Wh5l5rA4YVtc0jSWKcUDLEL6lSv7s3l9Q7dnVIJWaKOj+yTXFwMrysJczQa1
9O8h+bt/24ei
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
ewogICJpZGVudGlmaWVyIjogewogICAgInR5cGUiOiAiZG5zIiwKICAgICJ2YWx1
ZSI6ICJ3d3cuZ2xvYmFsLXRyZW5kcy1pbmZvLm9yZyIKICB9LAogICJzdGF0dXMi
OiAiaW52YWxpZCIsCiAgImV4cGlyZXMiOiAiMjAxOC0wMy0xNVQwODozMjoxMloi
LAogICJjaGFsbGVuZ2VzIjogWwogICAgewogICAgICAidHlwZSI6ICJkbnMtMDEi
LAogICAgICAic3RhdHVzIjogInBlbmRpbmciLAogICAgICAidXJpIjogImh0dHBz
Oi8vYWNtZS12MDEuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2NoYWxsZW5nZS9V
M0RWYjRzd2F5VGpCRDVhRG4zSGlNUmFXX0dwd0xMcWU2ckt6dUxmLXRFLzM3MzUw
OTIyNjYiLAogICAgICAidG9rZW4iOiAid2xVN3JXcU1kY2U5UWJVVk9ZUzFxZWty
b1pkOHhpWGtGNnNvdXlfUG1WZyIKICAgIH0sCiAgICB7CiAgICAgICJ0eXBlIjog
Imh0dHAtMDEiLAogICAgICAic3RhdHVzIjogImludmFsaWQiLAogICAgICAiZXJy
b3IiOiB7CiAgICAgICAgInR5cGUiOiAidXJuOmFjbWU6ZXJyb3I6dW5hdXRob3Jp
emVkIiwKICAgICAgICAiZGV0YWlsIjogIkludmFsaWQgcmVzcG9uc2UgZnJvbSBo
dHRwOi8vd3d3Lmdsb2JhbC10cmVuZHMtaW5mby5vcmcvLndlbGwta25vd24vYWNt
ZS1jaGFsbGVuZ2UvYXF4SlRlM19QdmVZS3BmYTdnQXg2azVOMmJFWENUcHhWMFRI
d3JEZm1abzogXCJcdTAwM2MhRE9DVFlQRSBIVE1MIFBVQkxJQyBcIi0vL0lFVEYv
L0RURCBIVE1MIDIuMC8vRU5cIlx1MDAzZVxuXHUwMDNjaHRtbFx1MDAzZVx1MDAz
Y2hlYWRcdTAwM2Vcblx1MDAzY3RpdGxlXHUwMDNlNDA0IE5vdCBGb3VuZFx1MDAz
Yy90aXRsZVx1MDAzZVxuXHUwMDNjL2hlYWRcdTAwM2VcdTAwM2Nib2R5XHUwMDNl
XG5cdTAwM2NoMVx1MDAzZU5vdCBGb3VuZFx1MDAzYy9oMVx1MDAzZVxuXHUwMDNj
cFwiIiwKICAgICAgICAic3RhdHVzIjogNDAzCiAgICAgIH0sCiAgICAgICJ1cmki
OiAiaHR0cHM6Ly9hY21lLXYwMS5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvY2hh
bGxlbmdlL1UzRFZiNHN3YXlUakJENWFEbjNIaU1SYVdfR3B3TExxZTZyS3p1TGYt
dEUvMzczNTA5MjI2OCIsCiAgICAgICJ0b2tlbiI6ICJhcXhKVGUzX1B2ZVlLcGZh
N2dBeDZrNU4yYkVYQ1RweFYwVEh3ckRmbVpvIiwKICAgICAgImtleUF1dGhvcml6
YXRpb24iOiAiYXF4SlRlM19QdmVZS3BmYTdnQXg2azVOMmJFWENUcHhWMFRId3JE
Zm1aby5JU2MzOG1pRi1rLW5MM0pvcThNU1dJSkp2YjdCTHo1di1MdnFCektCcTd3
IiwKICAgICAgInZhbGlkYXRpb25SZWNvcmQiOiBbCiAgICAgICAgewogICAgICAg
ICAgInVybCI6ICJodHRwOi8vd3d3Lmdsb2JhbC10cmVuZHMtaW5mby5vcmcvLndl
bGwta25vd24vYWNtZS1jaGFsbGVuZ2UvYXF4SlRlM19QdmVZS3BmYTdnQXg2azVO
MmJFWENUcHhWMFRId3JEZm1abyIsCiAgICAgICAgICAiaG9zdG5hbWUiOiAid3d3
Lmdsb2JhbC10cmVuZHMtaW5mby5vcmciLAogICAgICAgICAgInBvcnQiOiAiODAi
LAogICAgICAgICAgImFkZHJlc3Nlc1Jlc29sdmVkIjogWwogICAgICAgICAgICAi
MTc2LjkuMTIuOTUiCiAgICAgICAgICBdLAogICAgICAgICAgImFkZHJlc3NVc2Vk
IjogIjE3Ni45LjEyLjk1IgogICAgICAgIH0KICAgICAgXQogICAgfQogIF0sCiAg
ImNvbWJpbmF0aW9ucyI6IFsKICAgIFsKICAgICAgMQogICAgXSwKICAgIFsKICAg
ICAgMAogICAgXQogIF0KfQ==
-----END CERTIFICATE-----

I’m not sure if this is a Let’s Encrypt issue or an issue with acme.sh, but any help would be appreciated.

Greetings

_az · March 8, 2018, 10:47am

For sure the problem is reproducible:

$ openssl x509 -in a -noout -text
unable to load certificate
140275967294208:error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1197:
140275967294208:error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:374:Type=X509
140275967294208:error:0906700D:PEM routines:PEM_ASN1_read_bio:ASN1 lib:pem_oth.c:83:

You could just go get your certificate from https://crt.sh/?d=349971185 and replace the first certificate in the fullchain.cer file with the contents of the downloaded one.

I can’t tell whether the problem is the way you posted your certificate on the forum or not, but when I copy what you pasted into vim, the file leads with 0x20 (which is a space). Check that there is no space at the start of your file:

$ xxd -l 10 /etc/acme.sh/globaltrends.sef-bonn.org/fullchain.cer

Should be:

00000000: 2d2d 2d2d 2d42 4547 494e                 -----BEGIN

not

00000000: 202d 2d2d 2d2d 4245 4749                  -----BEGI

Takios · March 8, 2018, 12:17pm

Thanks for the input. Getting the cert from crt.sh is a nice idea!

The space isn’t in the file on the server though, so I’m really curious what’s wrong with it.

jmorahan · March 8, 2018, 8:43pm

Well that’s intriguing…

$ cat chain.pem | tail -n +2 | head -n -1 | base64 -d
{
  "identifier": {
    "type": "dns",
    "value": "www.global-trends-info.org"
  },
  "status": "invalid",
  "expires": "2018-03-15T08:32:12Z",
  "challenges": [
    {
      "type": "dns-01",
      "status": "pending",
      "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/U3DVb4swayTjBD5aDn3HiMRaW_GpwLLqe6rKzuLf-tE/3735092266",
      "token": "wlU7rWqMdce9QbUVOYS1qekroZd8xiXkF6souy_PmVg"
    },
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:acme:error:unauthorized",
        "detail": "Invalid response from http://www.global-trends-info.org/.well-known/acme-challenge/aqxJTe3_PveYKpfa7gAx6k5N2bEXCTpxV0THwrDfmZo: \"\u003c!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\"\u003e\n\u003chtml\u003e\u003chead\u003e\n\u003ctitle\u003e404 Not Found\u003c/title\u003e\n\u003c/head\u003e\u003cbody\u003e\n\u003ch1\u003eNot Found\u003c/h1\u003e\n\u003cp\"",
        "status": 403
      },
      "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/U3DVb4swayTjBD5aDn3HiMRaW_GpwLLqe6rKzuLf-tE/3735092268",
      "token": "aqxJTe3_PveYKpfa7gAx6k5N2bEXCTpxV0THwrDfmZo",
      "keyAuthorization": "aqxJTe3_PveYKpfa7gAx6k5N2bEXCTpxV0THwrDfmZo.ISc38miF-k-nL3Joq8MSWIJJvb7BLz5v-LvqBzKBq7w",
      "validationRecord": [
        {
          "url": "http://www.global-trends-info.org/.well-known/acme-challenge/aqxJTe3_PveYKpfa7gAx6k5N2bEXCTpxV0THwrDfmZo",
          "hostname": "www.global-trends-info.org",
          "port": "80",
          "addressesResolved": [
            "176.9.12.95"
          ],
          "addressUsed": "176.9.12.95"
        }
      ]
    }
  ],
  "combinations": [
    [
      1
    ],
    [
      0
    ]
  ]
}

cpu · March 8, 2018, 8:55pm

@Neilpang - This seems like it might be an acme.sh bug. What do you think? Pretty odd!

jmorahan · March 8, 2018, 11:22pm

I wonder if it could be related to this - is it possible that Boulder could return two different Link headers, or one header with multiple values, and acme.sh could pick the wrong one?

Neilpang · March 8, 2018, 11:27pm

@cpu @jmorahan
Yes, it seems like a bug.

Neilpang · March 8, 2018, 11:37pm

@jmorahan Is that possible ?

Neilpang · March 8, 2018, 11:41pm

Can you please show me the config file:

/etc/acme.sh/globaltrends.sef-bonn.org/globaltrends.sef-bonn.org.conf

jmorahan · March 8, 2018, 11:45pm

I don’t know, I was hoping @cpu would know

Neilpang · March 8, 2018, 11:46pm

please also show me the debug log with --debug 2

Takios · March 9, 2018, 11:04am

the current config has been cleaned up by out automation already, sorry >.<

but I have the suspicion that acme.sh was executed simultaneously multiple times and that led to this behaviour. I found this in the logs:

[Thu Mar  8 09:32:07 CET 2018] Creating domain key
[Thu Mar  8 09:32:09 CET 2018] The domain key is here: /etc/acme.sh/globaltrends.sef-bonn.org/globaltrends.sef-bonn.org.key
[Thu Mar  8 09:32:09 CET 2018] Single domain='globaltrends.sef-bonn.org'
[Thu Mar  8 09:32:10 CET 2018] Getting domain auth token for each domain
[Thu Mar  8 09:32:10 CET 2018] Getting webroot for domain='globaltrends.sef-bonn.org'
[Thu Mar  8 09:32:10 CET 2018] Getting new-authz for domain='globaltrends.sef-bonn.org'
[Thu Mar  8 09:32:11 CET 2018] Single domain='global-trends-info.org'
[Thu Mar  8 09:32:11 CET 2018] Getting domain auth token for each domain
[Thu Mar  8 09:32:11 CET 2018] Getting webroot for domain='global-trends-info.org'
[Thu Mar  8 09:32:11 CET 2018] Single domain='www.global-trends-info.org'
[Thu Mar  8 09:32:11 CET 2018] Getting new-authz for domain='global-trends-info.org'
[Thu Mar  8 09:32:11 CET 2018] Getting domain auth token for each domain
[Thu Mar  8 09:32:11 CET 2018] Getting webroot for domain='www.global-trends-info.org'
[Thu Mar  8 09:32:11 CET 2018] Getting new-authz for domain='www.global-trends-info.org'
[Thu Mar  8 09:32:11 CET 2018] The new-authz request is ok.
[Thu Mar  8 09:32:11 CET 2018] Verifying:globaltrends.sef-bonn.org
[Thu Mar  8 09:32:12 CET 2018] The new-authz request is ok.
[Thu Mar  8 09:32:13 CET 2018] The new-authz request is ok.
[Thu Mar  8 09:32:13 CET 2018] Verifying:global-trends-info.org
[Thu Mar  8 09:32:13 CET 2018] Verifying:www.global-trends-info.org
[Thu Mar  8 09:32:15 CET 2018] Success
[Thu Mar  8 09:32:15 CET 2018] Verify finished, start to sign.
[Thu Mar  8 09:32:17 CET 2018] Cert success.
[Thu Mar  8 09:32:17 CET 2018] Your cert is in  /etc/acme.sh/globaltrends.sef-bonn.org/globaltrends.sef-bonn.org.cer
[Thu Mar  8 09:32:17 CET 2018] Your cert key is in  /etc/acme.sh/globaltrends.sef-bonn.org/globaltrends.sef-bonn.org.key
[Thu Mar  8 09:32:17 CET 2018] Pending
[Thu Mar  8 09:32:17 CET 2018] www.global-trends-info.org:Verify error:Invalid response from http://www.global-trends-info.org/.well-known/ac
HwrDfmZo:
[Thu Mar  8 09:32:17 CET 2018] Please check log file for more details: /dev/null
[Thu Mar  8 09:32:17 CET 2018] The intermediate CA cert is in  /etc/acme.sh/globaltrends.sef-bonn.org/ca.cer
[Thu Mar  8 09:32:17 CET 2018] And the full chain certs is there:  /etc/acme.sh/globaltrends.sef-bonn.org/fullchain.cer
[Thu Mar  8 09:32:17 CET 2018] Run reload cmd: /etc/init.d/apache2 reload
[Thu Mar  8 09:32:19 CET 2018] Reload success

See how there are three “Verifying:” lines for different domains at almost the same time.
Could this be the cause of this behaviour, @Neilpang ?

Another try with debug 2 yielded no errors and a correct certificate.

Neilpang · March 9, 2018, 12:09pm

yes, do not run multiple instance of acme.sh at the same time.

system · April 8, 2018, 12:09pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.