InternalServerError when attempting to create certificate


#1

Hi,

I’m getting the following error when attempting to create a new certificate. Other certificates are created successfully, however, this specific one fails.

Unexpected error
+Response from server:

  • Code: InternalServerError
  • Content: Error
    An error occurred while processing your request.


    Reference #179.c8453c17.1488450658.2ec4c12e

This occurs during initiation of the challenges - i.e. before any authorization takes place. I’m trying to use HTTP authorization.

Does anyone know what could cause this?

EDIT: Seems to be failing for certain other domains as well. Some will work fine, others not.


#2

What client are you using ? Can you provide the debug log for the attempt ?


#3

I’m hitting the same issue at the moment, with different domains, and at different steps during the process. I’m using dehydrated with DNS-based challenge.

dehydrated doesn’t offer an option for a debug log. Here’s the output generated by one such failing run:

Processing linet.services with alternative names: www.linet.services
 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting challenge for linet.services...
  + ERROR: An error occurred while sending post-request to https://acme-v01.api.letsencrypt.org/acme/new-authz (Status 500)

Details:
<HTML><HEAD><TITLE>Error</TITLE></HEAD><BODY>
An error occurred while processing your request.<p>
Reference&#32;&#35;179&#46;2e8f1402&#46;1488455641&#46;92b4b2
</BODY></HTML>

Sometimes the first “Requesting challenge for…” works, but subsequent ones for the same domain (different sub-domain, of course) don’t.

It happens both with our main domain that we’ve requested for already, and it happens with a brand-new domain (the linet.services shown above) that I’ve never requested LE certificates for.


#4

We’re using the .Net library from ACMESharp. It’s highly integrated within our own internal APIs, so a debug log isn’t easilly produced - if ACMESharp even provides this (I’ll have to do some digging).

Is it possible to look up the Reference # within your systems to see what went wrong?


#5

There may be someone around in a few hours that has access - I can’t (I’m just a moderator here )

From my perspective I know GetSSL best ( having written it ) - if you fancy doing a quick test for one domain using that ( it’s bash like dehydrated), it does have a full debug option though, which might help track down the issue.


#6

Hello,
also getting lot’s of Errors on Cert-Generation (some dozen Domains for now) - using our custom PHP Client:

2017-03-02 13:43:10 [error] HTTP Challenge for metzgerei-gasser.at is not available. Whole response: “Error</TITLE></HEAD>\nAn error occurred while processing your request.

\nReference #179.a4055368.1488458590.1d651331\n</BODY></HTML>\n”

Andreas Schnederle-Wagner


#7

If you can provide a debug log of exactly what command was being used that would be helpful. Perhaps @cpu may have some ideas.


#8

Here is the unencoded Payload which is sent to LE - sensitive Data masked …:

URL: https://acme-v01.api.letsencrypt.org/acme/challenge/XXXXXXXX/XXXXXXXX
Data:

Array (
[header] => Array
(
[alg] => RS256
[jwk] => Array
(
[kty] => RSA
[n] => XXXXXXXX
[e] => XXXXXXXX
)

    )
[protected] => XXXXXXXX
[payload] => XXXXXXXX
[signature] => XXXXXXXX

)

Payload:

[resource] => challenge
[type] => http-01
[keyAuthorization] => XXXXXXXX
[token] => XXXXXXXX

Response:

Error<\/TITLE><\/HEAD>\nAn error occurred while processing your request.

\nReference #179.a4055368.1488461497.1d96d010\n<\/BODY><\/HTML>\n


#9

More logs would definitely be helpful.

We’re looking into it. Thanks for the tag.


#10

@cpu - you mean more logs from different people or mor logs from me? :wink:
If there is anything I can do help speeding up locating the problem - just tell me what … :slight_smile:
Andreas


#11

Thinking about it more I think it would actually be most useful if yourself and @hognevevle, @mbunkus could provide some more information about where you’re generating these requests from geographically/source IP-wise.

Could you also run the following (Akamai always asks anyway):

curl http://ipv4.whatismyip.akamai.com/ ; echo
curl http://ipv6.whatismyip.akamai.com/ ; echo
dig +short whoami.ipv4.akahelp.net TXT
dig +short whoami.ipv6.akahelp.net TXT
dig +short whoami.ds.akahelp.net TXT
dig +short whoami.ds.akahelp.net TXT
dig +short whoami.ds.akahelp.net TXT
mtr -c 20 -w -r acme-v01.api.letsencrypt.org

#12

alright @cpu - the Server the problem is occurring is located in VIENNA, AUSTRIA - IPV4 only hosted @ UPC / VIX2 DC.

curl http://ipv4.whatismyip.akamai.com/ ; echo
83.65.246.198
dig +short whoami.ipv4.akahelp.net TXT
"ns" “83.64.177.250"
dig +short whoami.ds.akahelp.net TXT
"ns” "83.64.177.250"
mtr -c 20 -w -r acme-v01.api.letsencrypt.org
Start: Thu Mar 2 16:14:56 2017
HOST: ortsinfo.futurehosting.at Loss% Snt Last Avg Best Wrst StDev
1.|-- 83-64-189-9.static.upcbusiness.at 0.0% 20 2.4 1.3 0.8 2.6 0.4
2.|-- at-vie15a-rd1-ae8-2127.aorta.net 0.0% 20 2.0 2.1 1.1 10.5 2.4
3.|-- at-vie05b-ri2-ae12-0.aorta.net 0.0% 20 1.4 2.1 1.2 14.7 2.9
4.|-- a104-83-5-242.deploy.static.akamaitechnologies.com 0.0% 20 1.9 1.8 1.6 2.7 0.0
5.|-- a104-87-17-95.deploy.static.akamaitechnologies.com 0.0% 20 1.3 1.3 1.1 1.7 0.0

But I don’t think that it’s a Network Issue - as I immediately get back a 500 Server Error with Reference Number …


#13

True - but hopefully it can help narrow down which server is to blame :slight_smile: Thanks for the info!


#14

@cpu sure. Here you go:

[0 mbunkus@bellerophon ~] curl http://ipv4.whatismyip.akamai.com/ ; echo
82.100.242.26
[0 mbunkus@bellerophon ~] curl http://ipv6.whatismyip.akamai.com/ ; echo
2001:1640:141:3::16
[0 mbunkus@bellerophon ~] dig +short whoami.ipv4.akahelp.net TXT
"ns" "212.90.153.68"
[0 mbunkus@bellerophon ~] dig +short whoami.ipv6.akahelp.net TXT
"ns" "2001:1640:141:e::feed:1"
[0 mbunkus@bellerophon ~] dig +short whoami.ds.akahelp.net TXT
"ns" "212.90.153.68"
[0 mbunkus@bellerophon ~] dig +short whoami.ds.akahelp.net TXT
"ns" "212.90.153.68"
[0 mbunkus@bellerophon ~] dig +short whoami.ds.akahelp.net TXT
"ns" "212.90.153.68"
[0 mbunkus@bellerophon ~] mtr -c 20 -w -r acme-v01.api.letsencrypt.org
HOST: bellerophon                   Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2001:1640:141:3::feed:1        0.0%    20    0.3   0.2   0.2   0.3   0.0
  2.|-- dsl-gate.mk.de                 0.0%    20   24.2  24.3  23.7  24.8   0.3
  3.|-- vl78.c1.mk.de                  0.0%    20   41.5  28.7  23.9  73.2  11.6
  4.|-- e1-ae3.i.mk.de                 0.0%    20   23.8  24.4  23.8  27.9   0.9
  5.|-- decix-fra5.netarch.akamai.com  0.0%    20   24.4  25.4  24.1  34.8   2.4
  6.|-- 2a02:26f0:10:28d::3d5          0.0%    20   24.6  24.7  24.1  30.2   1.3

#15

One more thing to run if you would be so kind, futureweb (please un-obfuscate your XXXX when you actually run it, of course):
curl -vv -H “Pragma: akamai-x-get-cache-key, akamai-x-cache-on, akamai-x-cache-remote-on, akamai-x-get-true-cache-key, akamai-x-get-extracted-values, akamai-x-check-cacheable, akamai-x-get-request-id, akamai-x-serial-no, akamai-x-get-ssl-client-session-id, akamai-x-feo-trace” https://acme-v01.api.letsencrypt.org/acme/challenge/XXXXXXXX/XXXXXXXX


#16

@isk there you go … “unfortunately” it succeeded this time …

curl -vv -H “Pragma: akamai-x-get-cache-key, akamai-x-cache-on, akamai-x-cache-remote-on, akamai-x-get-true-cache-key, akamai-x-get-extracted-values, akamai-x-check-cacheable, akamai-x-get-request-id, akamai-x-serial-no, akamai-x-get-ssl-client-session-id, akamai-x-feo-trace” https://acme-v01.api.letsencrypt.org/acme/challenge/XXXXXXXX/XXXXXXXX

  • About to connect() to acme-v01.api.letsencrypt.org port 443 (#0)
  • Trying 104.87.17.95…
  • Connected to acme-v01.api.letsencrypt.org (104.87.17.95) port 443 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • CAfile: /etc/pki/tls/certs/ca-bundle.crt
    CApath: none
  • SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  • Server certificate:
  •   subject: C=US,ST=California,L=Mountain View,O=INTERNET SECURITY RESEARCH GROUP,CN=*.api.letsencrypt.org
    
  •   start date: Jun 26 17:05:45 2015 GMT
    
  •   expire date: Jun 25 17:05:45 2018 GMT
    
  •   common name: *.api.letsencrypt.org
    
  •   issuer: CN=TrustID Server CA A52,OU=TrustID Server,O=IdenTrust,C=US
    

GET /acme/challenge/XXXXXXXX/XXXXXXXX HTTP/1.1
User-Agent: curl/7.29.0
Host: acme-v01.api.letsencrypt.org
Accept: /
Pragma: akamai-x-get-cache-key, akamai-x-cache-on, akamai-x-cache-remote-on, akamai-x-get-true-cache-key, akamai-x-get-extracted-values, akamai-x-check-cacheable, akamai-x-get-request-id, akamai-x-serial-no, akamai-x-get-ssl-client-session-id, akamai-x-feo-trace

< HTTP/1.1 202 Accepted
< Server: nginx
< Content-Type: application/json
< Content-Length: 630
< Boulder-Request-Id: XXXXXXXX
< Link: https://acme-v01.api.letsencrypt.org/acme/authz/XXXXXXXX;rel=“up”
< Location: https://acme-v01.api.letsencrypt.org/acme/challenge/XXXXXXXX/XXXXXXXX
< Replay-Nonce: XXXXXXXX
< X-Akamai-SSL-Client-Sid: XXXXXXXX
< X-Check-Cacheable: NO
< X-Akamai-Request-ID: XXXXXXXX
< Expires: Thu, 02 Mar 2017 15:33:07 GMT
< Cache-Control: max-age=0, no-cache, no-store
< Pragma: no-cache
< Date: Thu, 02 Mar 2017 15:33:07 GMT
< X-Cache: TCP_MISS from a104-83-5-164.deploy.akamaitechnologies.com (AkamaiGHost/8.2.4-19356466) (-)
< X-Cache-Key: S/D/981/432721/000/origin-2pvah7paghah4iu6P.api.letsencrypt.org/acme/challenge/XXXXXXXX/XXXXXXXX
< X-True-Cache-Key: /D/000/origin-2pvah7paghah4iu6P.api.letsencrypt.org/acme/challenge/XXXXXXXX/XXXXXXXX
< X-Akamai-Session-Info: name=AKA_PM_BASEDIR; value=
< X-Akamai-Session-Info: name=AKA_PM_CACHEABLE_OBJECT; value=false
< X-Akamai-Session-Info: name=AKA_PM_FWD_URL; value=/acme/challenge/XXXXXXXX/XXXXXXXX
< X-Akamai-Session-Info: name=AKA_PM_NETSTORAGE_ROOT; value=
< X-Akamai-Session-Info: name=AKA_PM_PREFETCH_ON; value=true
< X-Akamai-Session-Info: name=AKA_PM_RUM_ENABLED; value=off
< X-Akamai-Session-Info: name=AKA_PM_SR_ENABLED; value=false
< X-Akamai-Session-Info: name=AKA_PM_TD_ENABLED; value=false
< X-Akamai-Session-Info: name=AKA_PM_TD_MAP_PREFIX; value=ch2
< X-Akamai-Session-Info: name=DO_EDGECONNECT_PUBLISH; value=on
< X-Akamai-Session-Info: name=EDGECONNECT_API_CONNECTOR; value=default
< X-Akamai-Session-Info: name=EDGECONNECT_API_CONNECTOR_VERSION; value=1.0
< X-Akamai-Session-Info: name=EDGECONNECT_API_DATA_ELEMENTS; value=http apm
< X-Akamai-Session-Info: name=EDGECONNECT_API_NAME; value=cloud_monitor
< X-Akamai-Session-Info: name=EDGECONNECT_API_NAME_VERSION; value=1.0
< X-Akamai-Session-Info: name=EDGECONNECT_ENDPOINT_HOST; value=cloudmonitor.api.letsencrypt.org
< X-Akamai-Session-Info: name=EDGECONNECT_ENDPOINT_PATH; value=/receiver/v1/http/XXXXXXXX
< X-Akamai-Session-Info: name=EDGECONNECT_ENDPOINT_TYPE; value=custom_origin
< X-Akamai-Session-Info: name=EDGECONNECT_EVENT_SCOPE; value=all
< X-Akamai-Session-Info: name=EDGECONNECT_RULE_ID; value=1
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_API_NAME_VERSION; value=1.0
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_CACHE_STATUS; value=0
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_DATA_APM; value=on
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_DATA_GEO; value=on
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_DATA_HTTP; value=on
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_DATA_NETWORK; value=off
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_DATA_REQHEADER; value=on
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_DATA_RESPHEADER; value=on
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_DATA_SEC_APPV2; value=off
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_DATA_SEC_RATE_DENYV2; value=off
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_DATA_SEC_RATE_WARNV2; value=off
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_DATA_WAFV2; value=off
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_END_CLIENT_REQUEST; value=on
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_IP_FIRST; value=104
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_IP_FIRST_HEX; value=68
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_IP_FOURTH; value=95
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_IP_FOURTH_HEX; value=5f
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_IP_SECOND; value=87
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_IP_SECOND_HEX; value=57
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_IP_THIRD; value=17
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_IP_THIRD_HEX; value=11
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_MIDMILE_LATENCY; value=22
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_MIDMILE_RTT; value=22; full_location_id=X-EdgeConnect-MidMile-RTT
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_NETORIGIN_LATENCY; value=159; full_location_id=X-EdgeConnect-Origin-MEX-Latency
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_SRV_ERROR; value=0
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_TIME_HEX; value=58b83b33
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_UID; value=XXXXXXXX
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_UID_FWD_HEADER; value=XXXXXXXX
< X-Akamai-Session-Info: name=ENT_EDGECONNECT_UID_FWD_HEADER_ADDED; value=on
< X-Akamai-Session-Info: name=FASTTCP_RENO_FALLBACK_DISABLE_OPTOUT; value=on
< X-Akamai-Session-Info: name=HEADER_NAMES; value=User-Agent%3aHost%3aAccept%3aPragma; full_location_id=
< X-Akamai-Session-Info: name=IP_HASH; value=678
< X-Akamai-Session-Info: name=OVERRIDE_HTTPS_IE_CACHE_BUST; value=all
< X-Akamai-Session-Info: name=STRICT_BASELINE_V1ARL_CHECKS; value=<>
< X-Akamai-Session-Info: name=TCP_OPT_APPLIED; value=medium
< X-Serial: 981
< X-Akamai-SSL-Client-Sid: XXXXXXXX
< Connection: keep-alive
< X-Cache-Remote: TCP_MISS from a2-18-240-87.deploy.akamaitechnologies.com (AkamaiGHost/8.2.4-19356466) (-)
<
{
“type”: “http-01”,
“status”: “valid”,
“uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/XXXXXXXX/XXXXXXXX”,
“token”: “XXXXXXXX”,
“keyAuthorization”: “XXXXXXXX”,
“validationRecord”: [
{
“url”: “http://dandler.at/.well-known/acme-challenge/XXXXXXXX”,
“hostname”: “dandler.at”,
“port”: “80”,
“addressesResolved”: [
“83.65.246.198”
],
“addressUsed”: “83.65.246.198”
}
]


#17

Thanks, futureweb. The team has opened an issue with Akamai (the CDN in front of the API) as the problem seems to be within their internal communication. We’ll post back when we’ve got news.


#18

It seems this may be resolved. Is anyone on this thread still encountering issues?


#19

@isk - tested on ~10 Domains … all renewed without problems!

thx :slight_smile:


#20

Excellent. Thanks for checking.