ERROR about Issue SSL let's Encrypt in cPanel

Hi Luu,

https://acme-v01.api.letsencrypt.org is the ONLY external server that the plugin connects to. You could test connectivity by running:

curl -i https://acme-v01.api.letsencrypt.org/directory

I have responded to your email with a follow-up question, btw.

1 Like

Hi _az,
I received email from you
I will response
Thanks

@cpu could we please get your assistance?

I am not able to get to the bottom of a seemingly network-related issue with @luuvuong .

I have written a reproduction program, which is at the bottom of this post.

It seems like large request bodies are not making it to Boulder via Akamai in south east Asia.

Their output to the program is as follows, and I can’t repro on any other networks, and they can send large request bodies to other hosts (cpanel.com, example.org) just fine.

If we send with a small request body (e.g. 1K) it works OK.

[root@webhost-0104 ~]# ./network-test
Trying against https://cpanel.com with 50012 bytes body
        [1515405940593163588] Got conn to 208.74.121.58:443
        [1515405940593546450] Wrote request
        [1515405941274465754] TTFB
        [1515405941275096792] Response: 200 OK (24707 bytes): [PCFET0NUWVBFIGh0bWw+CjxodG1sIGxhbmc9ImVuIj4KCiAgPGhlYWQ+CiAgICA8bWV0YSBjaGFyc2V0PSJ1dGYtOCI+CiAgICA8bWV0YSBodHRwLWVxdWl2PSJYLVVBLUNvbQ==]...

Trying against https://example.org with 50012 bytes body
        [1515405941862972818] Got conn to 93.184.216.34:443
        [1515405941863449239] Wrote request
        [1515405942413049915] TTFB
        [1515405942413448127] Response: 200 OK (1270 bytes): [PCFkb2N0eXBlIGh0bWw+CjxodG1sPgo8aGVhZD4KICAgIDx0aXRsZT5FeGFtcGxlIERvbWFpbjwvdGl0bGU+CgogICAgPG1ldGEgY2hhcnNldD0idXRmLTgiIC8+CiAgICA8bQ==]...

Trying against https://127.0.0.1:2083 with 50012 bytes body
        [1515405942421597395] Got conn to 127.0.0.1:2083
        [1515405942421896672] Wrote request
        [1515405942462921579] TTFB
        [1515405942463802530] Response: 401 Access Denied (42225 bytes): [CjwhRE9DVFlQRSBodG1sPgo8aHRtbCBsYW5nPSJlbiIgZGlyPSJsdHIiPgo8aGVhZD4KICAgIDxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0idGV4dA==]...

Trying against https://acme-v01.api.letsencrypt.org/acme/new-cert with 50012 bytes body
        [1515405942604677618] Got conn to 104.111.200.137:443
        [1515405942604864840] Wrote request
        [1515405972463974115] Experienced request error: Post https://acme-v01.api.letsencrypt.org/acme/new-cert: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Trying against https://acme-staging.api.letsencrypt.org/acme/new-cert with 50012 bytes body
        [1515405972590767451] Got conn to 104.111.200.137:443
        [1515405972590974514] Wrote request
        [1515406002464191275] Experienced request error: Post https://acme-staging.api.letsencrypt.org/acme/new-cert: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
package main

import (
	"bytes"
	"crypto/tls"
	"encoding/base64"
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"
	"net/http/httptrace"
	"os"
	"strconv"
	"strings"
	"time"
)

func main() {
	bufSize, _ := strconv.Atoi(os.Getenv("NETWORK_TEST_SIZE"))
	if bufSize == 0 {
		bufSize = 500
	}

	buf, _ := json.Marshal(map[string]string{
		"dummy": strings.Repeat("x", bufSize*100),
	})

	urls := []string{
		"https://cpanel.com",
		"https://example.org",
		"https://127.0.0.1:2083",
		"https://acme-v01.api.letsencrypt.org/acme/new-cert",
		"https://acme-staging.api.letsencrypt.org/acme/new-cert",
	}

	for _, u := range urls {
		hit(u, buf)
	}
}

func hit(u string, payload []byte) {
	defer fmt.Println()

	cl := &http.Client{
		Timeout: 30 * time.Second,
		Transport: &http.Transport{
			TLSClientConfig: &tls.Config{
				InsecureSkipVerify: true,
			},
		},
	}

	req, err := http.NewRequest("POST", u, bytes.NewReader(payload))
	if err != nil {
		panic(err)
	}

	trace := &httptrace.ClientTrace{
		GotConn: func(connInfo httptrace.GotConnInfo) {
			fmt.Printf("\t[%d] Got conn to %s\n", time.Now().UnixNano(), connInfo.Conn.RemoteAddr())
		},
		WroteRequest: func(i httptrace.WroteRequestInfo) {
			fmt.Printf("\t[%d] Wrote request\n", time.Now().UnixNano())
		},
		Got100Continue: func() {
			fmt.Printf("\t[%d] Got 100-continue\n", time.Now().UnixNano())
		},
		GotFirstResponseByte: func() {
			fmt.Printf("\t[%d] TTFB\n", time.Now().UnixNano())
		},
	}
	req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))

	fmt.Printf("Trying against %s with %d bytes body\n", u, len(payload))

	resp, err := cl.Do(req)
	if err != nil {
		fmt.Printf("\t[%d] Experienced request error: %v\n", time.Now().UnixNano(), err)
		return
	}

	defer resp.Body.Close()

	var buf []byte
	if buf, err = ioutil.ReadAll(resp.Body); err != nil {
		fmt.Printf("\t[%d] Experienced read error: %v\n", time.Now().UnixNano(), err)
	}

	fmt.Printf("\t[%d] Response: %s (%d bytes): [%v]...\n", time.Now().UnixNano(),
		resp.Status, len(buf), base64.StdEncoding.EncodeToString(buf[:100]))

}
1 Like

Hi @_az,

Thanks for tagging me (and the reproduction program! that's great!)

I will pass this to our operations team so they can contact Akamai support to see if there's something that needs to change in our configuration or if this is an issue on the CDN-side that needs resolving.

1 Like

Hi cpu,
Please support for me!
Thanks

Hi @luuvuong, please be patient :slight_smile: Our operations team will be working on this problem and will reply on-thread if there is anything they need from you or if there are any updates to provide.

Thanks!

1 Like

Hi @cpu, sure! Thanks so much!

1 Like

After looking through the available diagnostic tools available from the CDN, I have not been able to determine the problem. I have opened a case with our CDN to look further into this and will report any updates and findings here.

3 Likes

Hi jillian,
Thank for reply!
Please notify to me when you had more infor

Hi @jillian,

An additional data point that may help Akamai pin down the problem:

I have provided @luuvuong with a different IP (via /etc/hosts) to use to talk to acme-v01 (specifically 23.49.216.139, an Akamai POP in Australia), and they report that this restores the ability to send large request bodies.

Hopefully this helps narrow down any CDN issues!

1 Like

@_az thanks for providing a work around while we look for the bigger problem. I’ve asked @luuvuong in a DM for his ip address and what region is host is located in to provide the CDN with a narrower area. Are you able to provide an ip address in SE Asia that experiences this problem where large request bodies fails?

1 Like

Sure, sent you a DM.

1 Like

Thanks @_az I have forwarded the information along.

@luuvuong, will you please run the following command from your hosts that were experience problems with large request bodies curl -I https://acme-v01.api.letsencrypt.org -H "Pragma: akamai-x-cache-on, akamai-x-get-cache-key, akamai-x-get-true-cache-key, akamai-x-get-request-id" and provide the results here. Please make sure that you when run these commands you remove or comment the workaround in your /etc/hosts. The results of this command will provide information to the CDN to help diagnose the problem.

2 Likes

Hi Jillian,

I already command line above which you provide. I send result bellow when run it. Sure i already remove in /ect/hosts

HTTP/1.1 200 OK
Server: nginx
Content-Type: text/html
Content-Length: 2175
Last-Modified: Tue, 13 Oct 2015 15:42:36 GMT
ETag: "561d266c-87f"
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Accept-Ranges: bytes
X-Akamai-Request-ID: 11f39561.131b538a
Expires: Thu, 11 Jan 2018 01:40:50 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Thu, 11 Jan 2018 01:40:50 GMT
X-Cache: TCP_MISS from a184-28-218-215.deploy.akamaitechnologies.com (AkamaiGHost/9.1.4.4-21488373) (-)
X-Cache-Key: S/D/981/432721/000/origin-2pvah7paghah4iu6P.api.letsencrypt.org/
X-True-Cache-Key: /D/000/origin-2pvah7paghah4iu6P.api.letsencrypt.org/
Connection: keep-alive

Thank you for providing your results. I have passed them onto our CDN to examine. I will keep you updated with any more requests or results.

1 Like

@luuvuong Apologies for a lag in getting back to you.

I would also like to see the POST request. You can do this by adding -X GET after the curl -I part of the command

You can also modify small script written by @_az to include the Pragma headers and then run the updated script.

1 Like

@luuvuong to make things a little easier for you I am attaching 10kb and 50kb payloads that you can use with curl to manifest the issue.

10k.txt (10.0 KB)
50k.txt (48.8 KB)

It would be nice if you could run three commands to fulfill this request for info (using those files, without /etc/hosts):

Small POST

curl -v --data '{"dummy":"x"}' -H 'Expect:' https://acme-v01.api.letsencrypt.org/acme/new-cert -H "Pragma: akamai-x-cache-on, akamai-x-get-cache-key, akamai-x-get-true-cache-key, akamai-x-get-request-id"

10k POST

curl -v --data @10k.txt -H 'Expect:' https://acme-v01.api.letsencrypt.org/acme/new-cert -H "Pragma: akamai-x-cache-on, akamai-x-get-cache-key, akamai-x-get-true-cache-key, akamai-x-get-request-id"

50k POST

curl -v --data @50k.txt -H 'Expect:' https://acme-v01.api.letsencrypt.org/acme/new-cert -H "Pragma: akamai-x-cache-on, akamai-x-get-cache-key, akamai-x-get-true-cache-key, akamai-x-get-request-id"

(Expect included to disable curl’s Expect: 100-Continue behavior).

1 Like

Perhaps <(head -c 10000 /dev/zero) would work in place of the 10k.txt file? Like curl -v --data @<(head -c 10000 /dev/zero) -H 'Expect:' https://acme-v01.api.letsencrypt.org/acme/new-cert -H "Pragma: akamai-x-cache-on, akamai-x-get-cache-key, akamai-x-get-true-cache-key, akamai-x-get-request-id"?

2 Likes

Sure, was just worried that Akamai may have had edge filtering for non-JSON request bodies or something :smiley: .

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.